Subject: [NNTP] CAPABILITIES problem!

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 1 Aug 2005 09:33:23 +0100

I can't believe we all missed this. The IANA registry people have just
pointed it out to me.

In 3.3.1 we define a capability label as being a keyword.
In 2 we state that a keyword MUST be at least 3 characters and MUST NOT
  exceed 12 characters.
[Both these are repeated in the formal syntax.]
In 3.3.2 we specify the capability label IMPLEMENTATION.

Which is 14 characters long.

Possible solutions:
(1) Relax the 12 character limit to something longer (where did it come
from anyway?).
(2) Relax the requirement for the label to be a keyword.
(3) Change the keyword to something shorter, like SERVER_INFO.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: Ken Murchison <[email-masked]>
Date: Mon, 01 Aug 2005 08:07:30 -0400

> (2) Relax the requirement for the label to be a keyword.

Either (1) or (2) works for me.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 1 Aug 2005 13:14:58 +0100

> Either (1) or (2) works for me.

I should have been clear that (1) would have effects beyond CAPABILITIES -
command names are keywords, for example.

---

From: "maurizio codogno" <[email-masked]>
Date: Mon, 1 Aug 2005 13:10:10 +0000

>(3) Change the keyword to something shorter, like SERVER_INFO.

If there is no current implementation of IMPLEMENTATION :-), probably
option 3 is the best, but the others are fine too.

ciao, .mau.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 01 Aug 2005 09:41:30 -0400

> command names are keywords, for example.

Right.  Then (2) or (3), with a preference for (2).  I think the only 
reason we settled on IMPLEMENTATION is that its also used in POP3. 
FWIW, POP3 capability tags (1 or more chars) are defined separately from 
command keywords (3 or 4 chars).

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 1 Aug 2005 14:59:41 +0100

Ken Murchison said:
> Right.  Then (2) or (3), with a preference for (2).

Of course, it all depends what Russ thinks is acceptable at this stage.

> I think the only 

Sounds familiar.

> command keywords (3 or 4 chars).

Certainly we can do that. I reused the definition of "keyword" because it
was convenient and reserved other tokens for future concepts.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 01 Aug 2005 09:54:56 -0700

> (3) Change the keyword to something shorter, like SERVER_INFO.

I think the easiest thing to do at this point is just change it, since I
don't think anyone has started implementing CAPABILITIES yet.  Since the
choice of keywords is fairly arbitrary, whereas lengths might (probably
won't, but might) have subtle consequences, changing the keyword at this
stage seems like the lowest impact change.

Ned, Scott, is this something that we can take care of in author's 48, or
do we need to do something more formal?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Ned Freed <[email-masked]>
Date: Mon, 01 Aug 2005 12:29:59 -0700 (PDT)

> (3) Change the keyword to something shorter, like SERVER_INFO.

My personal preference would be to bump the limit to 16 or even 32 characters.
12 seems a bit short.

				Ned

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 2 Aug 2005 06:35:11 +0100

> My personal preference would be to bump the limit to 16 or even 32 characters.

As I said, I'm not sure where the limit comes from in the first place.

But, as Russ asks, what's the formal process?

---

From: Ned Freed <[email-masked]>
Date: Mon, 01 Aug 2005 23:10:09 -0700 (PDT)

> But, as Russ asks, what's the formal process?

IMO this can be handled during AUTH48. When the RFC Editor is ready you
as author will be contacted and asked if the RFC-to-be is OK. You submit
a correction then. The RFC Editor will run it past the relevant AD (Scott
in this case). If Scott is OK with the change then we should be good to go.

				Ned

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 2 Aug 2005 02:16:25 -0400

> -----Original Message-----
> From: Russ Allbery [mailto:[email-masked]] 
> Sent: Monday, August 01, 2005 12:55 PM
> To: [email-masked]
> Subject: Re: [NNTP] CAPABILITIES problem!
> 

Given that this is something more than an editorial change, it needs to be
made with some notice.  Let's do this:

Russ, please figure out what the best change is by floating and confirming a
proposal on the WG mailing list.  Once we know what to do, please reply to
Michelle's request for info to tell her what IANA should do.  Then make the
document change in auth48.

Sound reasonable?

-Scott-

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 2 Aug 2005 07:17:54 +0100

> in this case). If Scott is OK with the change then we should be good to go.

Thanks.

One final question: would you say this applies for all three options, or
only for 2 and 3? Option 1 may have wider implications, because it affects
other items (e.g. command names).

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 2 Aug 2005 02:19:50 -0400

> From: Ned Freed [mailto:[email-masked]] 
> Sent: Tuesday, August 02, 2005 2:10 AM
> To: Clive DW Feather
> Cc: Ned Freed; IETF NNTP mailing list

Agreed.  All I would add to Ned's note is that I'd like to see a proposal
confirmed by the WG before I agree with the change.  Ned's suggestion to
bump the limit seems like the least intrusive document change, but I don't
know what the impact would be on implementations.

-Scott-

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 2 Aug 2005 07:54:03 +0100

> know what the impact would be on implementations.

Okay, here's some strawmen.

Option (1):

The term "keyword" is used for four things:
- command names
- command second word names (notably LIST)
- the argument to CAPABILITIES
- capability labels

The relevant wording is:

| A "keyword" MUST consist only of US-ASCII letters, digits, and the
| characters dot (".") and dash ("-"), and MUST begin with a letter.
| Keywords MUST be at least three characters and MUST NOT exceed 12
| characters.

and the syntax:

|   keyword = ALPHA 2*11(ALPHA / DIGIT / "." / "-")

We can change or remove either or both limits. For example:

  A "keyword" MUST consist only of US-ASCII letters, digits, and the
  characters dot (".") and dash ("-"), and MUST begin with a letter.
  Keywords MUST be at least three characters and MUST NOT exceed 16
  characters.

    keyword = ALPHA 2*15(ALPHA / DIGIT / "." / "-")

or:

  A "keyword" MUST consist only of US-ASCII letters, digits, and the
  characters dot (".") and dash ("-"), and MUST begin with a letter.
  Keywords MUST be at least two characters.

    keyword = ALPHA 1*(ALPHA / DIGIT / "." / "-")

No other wording changes would be needed. I don't know what the
implications are of changing the minimum or maximum command length. There
is no such limit in RFC 977 and I don't know where it came from.

Option (2):

The current wording is:

| The capability label, which is a keyword indicating the capability. 

| IANA is requested to maintain a registry of NNTP capability labels.
| All capability labels in the registry MUST be keywords and MUST NOT
| begin with X.

and the syntax:

|  capability-label = keyword

There are two sub-options:
(2A): remove the length restriction
(2B): remove all restrictions, except that it must begin with a letter
      other than X.

Option (2A):

New wording:

  The capability label, which is a token indicating the capability. 
  The label must be a keyword except that it can be longer than 12
  characters.

  IANA is requested to maintain a registry of NNTP capability labels.
  All capability labels in the registry MUST be keywords (except that
  they can be longer than 12 characters) and MUST NOT begin with X.

and the syntax:

   capability-label = ALPHA 2*(ALPHA / DIGIT / "." / "-")

Alternative wording, with no minimum length either:

  The capability label, which is a token indicating the capability. 
  It MUST consist only of US-ASCII letters, digits, and the characters
  dot (".") and dash ("-"), and MUST begin with a letter.

  IANA is requested to maintain a registry of NNTP capability labels.
  All capability labels in the registry MUST consist only of US-ASCII
  letters, digits, and the characters dot (".") and dash ("-"), and MUST
  begin with a letter other than X.

   capability-label = ALPHA *(ALPHA / DIGIT / "." / "-")

Option (2B):

New wording:

  The capability label, which is a token indicating the capability. 
  It MUST begin with a letter.

  IANA is requested to maintain a registry of NNTP capability labels.
  All capability labels in the registry MUST begin with a letter other
  than X.

   capability-label = ALPHA *P-CHAR

Option (3):

Change "IMPLEMENTATION" to "SERVER-INFO" wherever it occurs. Reorder the
table of predefined capabilities. In the syntax, change
"implementation-capability" to "server-info-capability" and reorder lists
as appropriate.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 02 Aug 2005 10:04:24 -0400

>     keyword = ALPHA 1*(ALPHA / DIGIT / "." / "-")

My preference would be one of these two changes.  If we do limit the 
length, is 16 still too short?  How about 20?  I still like 3 as the 
lower limit.

> is no such limit in RFC 977 and I don't know where it came from.

It looks like it was in Stan's -01 draft, but I can't find any 
discussion leading up to this change.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     2495 Main St. - Suite 401
716-604-0088 x26      Buffalo, NY 14214
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Aug 2005 09:32:56 -0700

Scott Hollenbeck <[email-masked]> writes:

> Given that this is something more than an editorial change, it needs to
> be made with some notice.  Let's do this:

> Russ, please figure out what the best change is by floating and
> confirming a proposal on the WG mailing list.  Once we know what to do,
> please reply to Michelle's request for info to tell her what IANA should
> do.  Then make the document change in auth48.

> Sound reasonable?

Yup, this sounds good to me.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Aug 2005 09:37:36 -0700

> implications are of changing the minimum or maximum command
> length. There is no such limit in RFC 977 and I don't know where it came
> from.

I have no idea why there's a limit either.  We've had a limit, as Ken
noted, for the entirety of this process, but I don't know why it was added
in the first place since RFC 977 doesn't have one.

Given that, it seems cleanest to just remove the limit; that's a fairly
minimal change to the document.  It's a more aggressive change to the
standard, however.  Can anyone on the list think of any reason why
removing this length limit entirely would cause any problems with existing
software?

Note that, of course, removing the limit doesn't cause incredibly long
keywords to suddenly appear; someone would still have to define a command
that uses an extra-long keyword.  So the only place that I could see this
even potentially being a problem would be at the generic parsing layer,
and I really don't see how a length limit on keywords helps there in any
practical sense.

(I guess I do see *one* reason for a limit -- the length of commands that
take message IDs as arguments directly affects the maximum message ID
length that NNTP can deal with, given the overall limit on the length of
commands.  But I'm inclined to let people defining new commands that take
message ID arguments worry about that, rather than limiting everything
just for the sake of that one issue.)

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Aug 2005 09:38:09 -0700

> length, is 16 still too short?  How about 20?  I still like 3 as the

This is a good point that I missed in my last message.  I would like to
keep 3 as a lower limit as well.

---

From: Ned Freed <[email-masked]>
Date: Tue, 02 Aug 2005 09:59:39 -0700 (PDT)

> keep 3 as a lower limit as well.

I like the lower limit too. Removing the upper limit entirely is fine with
me.

				Ned

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 2 Aug 2005 19:09:14 +0100

> removing this length limit entirely would cause any problems with existing

Clients, no, because they only deal with existing commands.

Servers that are broken by it aren't sufficiently defensive.

> just for the sake of that one issue.)

Note that message IDs are limited to 250 characters including the < and >,
while command lines are limited to 510 plus CRLF.

Okay, so the proposed wording (having seen your later message) is:`

    A "keyword" MUST consist only of US-ASCII letters, digits, and the
    characters dot (".") and dash ("-"), and MUST begin with a letter.
    Keywords MUST be at least three characters.

      keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-")

---

From: [email-masked] (Peter Robinson)
Date: Tue, 2 Aug 2005 23:19:46 +0100

Clive D.W. Feather <[email-masked]> wrote:

> > take message IDs as arguments [...]

> while command lines are limited to 510 plus CRLF.

So there's plenty of headroom for command names.

>       keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-")

I support this change.

Bumping into the old limit before CAPABILITIES is even out of the door
is a good indication that this would have been a future annoyance for
extensions.  I had assumed that the 12 character command name limit came
from RFC977, but since that isn't the case I see no reason at all to set
a limit.

Regards,

Peter

---

From: Ade Lovett <[email-masked]>
Date: Tue, 02 Aug 2005 22:33:14 -0700

> from RFC977, but since that isn't the case I see no reason at all to set

Likewise.  AIUI, the impact of having no limit is server-side, and thus
if it DOES come up as an implementation issue, is likely to be far more
solvable than a client-side change.

Indeed, I'm toying with the idea of an extension to allow for retrieval
of an article by an MD5 hash of the message ID in all places where
<messageid> is currently used.  Given that such an implementation would
be of completely fixed length, it would seem to deal nicely with the
edge case of ultra-long message IDs potentially messing things up.

My vote is to remove the limit on the capabilities keywords entirely.  I
could possibly be persuaded for a 64 (or even 128 character) limit, but
the current limit is far too small.

-aDe

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 3 Aug 2005 08:30:50 +0100

> edge case of ultra-long message IDs potentially messing things up.

I don't follow. Message IDs are limited to 250 characters, so where's the
edge case? If someone puts an illegally-long message ID in a command, why
couldn't they put an illegally-long hash as well? [Both should just
generate a 501 response.]

> the current limit is far too small.

I'm happy with no limit if nobody can see a problem. Remember that the
limit we're talking about is command names and capability labels (the first
word on a capability line), not other arguments, which are already
unlimited.

---

From: Ade Lovett <[email-masked]>
Date: Wed, 03 Aug 2005 00:49:43 -0700

> generate a 501 response.]

MD5 is 128-bit.  Period.

> word on a capability line), not other arguments, which are already

In which case, I really cannot see a situation where an overly long
capability *and* an overly long command name, could result in an
overflow of 510 characters (+ \r\n).

I've been trying to avoid this particular thread for a reason.  If
someone can show a situation where a capability of, say, 128 bytes in
length could potentially result in an overflow, then that bears further
consideration.  As it stands right now, the issue that came to mind was
the message-ID (which is by far and away the largest single entity we
have to deal with).

Unless there's a compelling reason otherwise, I would suggest a 128 byte
limit (perhaps 64 if really necessary) for a capability, and be done
with it.  640k^W128 bytes is enough for anyone....

-aDe

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Aug 2005 10:20:22 GMT

>in the first place since RFC 977 doesn't have one.

Life is always simpler with strings of a known maximum length (no need to
malloc space every time a new string is encountered). For simple keywords,
intended primarily for internal protocol use, a fixed length is quite
enough. Unlimited length is really only needed to suit the convenience of
human readers/writers (which might apply in the Capabilities case, but not
for command names).

Note that the only effect of the current restriction is to constraint
authors of future extensions (which might simplify parsers).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Aug 2005 10:10:38 GMT

>- the argument to CAPABILITIES

Seems a lot of features to change at this late stage.

>  The label must be a keyword except that it can be longer than 12

Kludge!

>  It MUST begin with a letter.

That's better!

>Change "IMPLEMENTATION" to "SERVER-INFO" wherever it occurs.

And for a last minute change, that seems far enough.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 3 Aug 2005 12:15:44 +0100

Charles Lindsey said:
> malloc space every time a new string is encountered).

But, in this day and age, what's wrong with the already-known maximum of
510?

---

From: Russ Allbery <[email-masked]>
Date: Wed, 03 Aug 2005 10:15:10 -0700

Charles Lindsey <[email-masked]> writes:

> Life is always simpler with strings of a known maximum length (no need
> to malloc space every time a new string is encountered). For simple
> keywords, intended primarily for internal protocol use, a fixed length
> is quite enough.

Having written a fair number of NNTP parsers at this point, I'm at a loss
to understand where someone would use a fixed-length buffer specifically
for keywords.

You already have the whole command in memory since you have to read it
from the network.  At that point, why would you copy the keyword into yet
another buffer instead of just parsing it in-place?

---

From: [email-masked] (Kai Henningsen)
Date: 06 Aug 2005 19:01:00 +0200

> human readers/writers (which might apply in the Capabilities case, but not

But note that a keyword length limit in the standard does nothing  
whatsoever to help you there.

First, there obviously is a maximum length to the set of keywords you  
actually understand. If you want to take advantage of such a length, you  
already have a candidate length.

And second, there's no way a rule in the standard can prevent the other  
side from trying to use a longer string, so you already have to be able to  
cope with that.

> authors of future extensions (which might simplify parsers).

Again, no. Changing that maximum length doesn't make the parser either  
simpler or the opposite.

MfG Kai