Subject: [NNTP] Changes from RFC 977 / RFC 2980

From: Ken Murchison <[email-masked]>
Date: Fri, 13 May 2005 13:50:17 -0400

Since I suggested that we add an appendix that lists the changes that 
we're making to RFC 977 and RFC 2980, I thought I'd take a first stab at 
it.  Here's what I came up with, which may not be an exhaustive list.

Changes from RFC 977:

-  SLAVE command deprecated

-  all articles MUST have a message-id (no "<0>" placeholder allowed in
    responses)

-  UTF-8 character set used for commands and responses [section 3.1]

-  x8x response codes are reserved for authentication and privacy
    extensions (formerly used for non-standard/private extensions)
    [section 3.2]

-  x9x response codes are reserved for non-standard/private extensions
    (formerly used for debugging output) [section 3.2]

-  new 401, 403, 480, 483, and 504 generic response codes
    [section 3.2.1]

-  redefined 502 and 503 response codes [section 3.2.1]

-  servers MUST support pipelining of commands (where possible)
    [section 3.5]

-  wildmat format standarized [section 4]

-  new CAPABILITIES command [section 5.2]

-  new MODE READER command [section 5.3]

-  group argument to the GROUP command is now optional
    [section 6.1.1]

-  new LISTGROUP command [section 6.1.2]

-  new DATE command [section 7.1]

-  4 digit years allowed for NEWGROUPS and NEWNEWS commands
    [sections 7.3 and 7.4]

-  new ACTIVE, ACTIVE.TIMES, DISTRIB.PATS, and NEWGROUPS variants of
    LIST command [sections 7.6.3 - 7.6.6]

-  new "m" status flag for LIST / LIST ACTIVE response [section 7.6.3]

-  new OVER and LIST OVERVIEW.FMT commands [sections 8.3 and 8.4]

-  new HDR and LIST HEADERS commands [sections 8.5 and 8.6]

Changes from RFC 2980:

-  LISTGROUP command can accept an optional range argument,
    and the 211 initial response has the same format as the
    GROUP 211 response [section 6.1.2]

-  :bytes and :lines metadata items deprecate Bytes and Lines
    (pseudo)headers [sections 8.1.1 and 8.1.2]

-  XOVER command formalized as OVER command [section 8.3]

-  OVER command can accept a message-id argument [section 8.3]

-  XHDR command formalized as HDR command [section 8.5]

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 May 2005 11:31:20 GMT

>-  wildmat format standarized [section 4]

Which allows some new usages (involving ',', for example) in places where
the old wildmats would not have done so.

>-  4 digit years allowed for NEWGROUPS and NEWNEWS commands

No distribution parameter to NEWNEWS.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Wed, 18 May 2005 19:12:28 -0700

> it.  Here's what I came up with, which may not be an exhaustive list.

This list looks good to me, combined with the additions from Charles.
Clive, if you're comfortable with this, could you include it in the draft?
I don't want to spend a lot of additional time on this (see the message
that I'm about to send), so we should probably note at the beginning of
the list that it may not be exhaustive.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 19 May 2005 10:54:26 +0100

> Clive, if you're comfortable with this, could you include it in the draft?

I've used that list as the basis for a new appendix (I think I spotted one
or two other matters).

Section 1, paragraphs 3 and 4 are now largely redundant and have been
replaced by:

   While the protocol specification in this document is largely
   compatible with the version specified in RFC 977 [RFC977], there are
   a number of changes which are summarised in Appendix D.  In
   particular: 
   o  the default character set is changed from US-ASCII [ANSI1986] to
      UTF-8 [RFC3629] (note that US-ASCII is a subset of UTF-8);
   o  a number of commands that were optional in RFC 977 or have been
      taken from RFC 2980 [RFC2980] are now mandatory;
   o  a CAPABILITIES command has been added to allow clients to
      determine what functionality is available from a server.

Then the new Appendix D:

   Appendix D.  Changes from RFC 977
   
   In general every attempt has been made to ensure that the protocol
   specification in this document is compatible with the version
   specified in RFC 977 [RFC977] and the various facilities adopted from
   RFC 2980 [RFC2980].  However, there have been a number of changes,
   some compatible and some not.
   
   This appendix lists these changes.  It is not guaranteed to be
   exhaustive or correct and MUST NOT be relied on.
         
   o  A formal syntax specification (Section 9) has been added.
      
   o  The default character set is changed from US-ASCII [ANSI1986] to
      UTF-8 [RFC3629] (note that US-ASCII is a subset of UTF-8).  This
      matter is discussed further in Section 10.
      
   o  All articles are required to have a message-id, eliminating the
      "<0>" placeholder used in RFC 977 in some responses.
   
   o  The newsgroup name matching capabilities already documented in
      RFC 977 ("wildmats" (Section 4)) are clarified and extended.  The
      new facilities (e.g. the use of commas and exclamation marks) are
      allowed wherever wildmats appear in the protocol.
   
   o  Support for pipelining of commands (Section 3.5) is made
      mandatory.
   
   o  The principles behind response codes (Section 3.2) have been
      tidied up.  In particular:
      *  the x8x response code family, formerly used for private
         extensions, is now reserved for authentication and privacy
         extensions;
      *  the x9x response code family, formerly intended for debugging
         facilities, are now reserved for private extensions;
      *  the 502 and 503 generic response codes (Section 3.2.1) have  
         been redefined;
      *  new 401, 403, 480, 483, and 504 generic response codes have
         been added.
   
   o  The rules for article numbering (Section 6) have been clarified.
   
   o  The SLAVE command (which was ill-defined) is removed from the 
      protocol.
      
   o  Four-digit years are permitted in the NEWNEWS (Section 7.4) and
      NEWGROUPS (Section 7.3) commands (two-digit years are still
      permitted).  The optional distribution parameter to these commands
      has been removed.
   
   o  The argument to GROUP (Section 6.1.1) is optional.
      
   o  The LIST (Section 7.6.1) command is greatly extended; the original
      is available as LIST ACTIVE, while new variants include
      ACTIVE.TIMES, DISTRIB.PATS, and NEWSGROUPS.  A new "m" status flag
      is added to the LIST ACTIVE response.
   
   o  A new CAPABILITIES (Section 5.2) command allows clients to
      determine what facilities are supported by a server.
         
   o  The DATE (Section 7.1) command is adopted from RFC 2980
      effectively unchanged.
   
   o  The LISTGROUP (Section 6.1.2) command is adopted from RFC 2980.
      An optional range argument has been added, and the 211 initial
      response line now has the same format as the 211 response from the
      GROUP command.
      
   o  The MODE READER (Section 5.3) command is adopted from RFC 2980 and
      its meaning and effects clarified.
   
   o  The XHDR command in RFC 2980 has been formalised as the new HDR
      (Section 8.5) and LIST HEADERS (Section 8.6) commands.
   
   o  The XOVER command in RFC 2980 has been formalised as the new OVER
      (Section 8.3) and LIST OVERVIEW.FMT (Section 8.4) commands.  The  
      former can be applied to a message-id as well as to a range.
   
   o  The concept of article metadata (Section 8.1) has been formalised,
      allowing the Bytes and Lines pseudo-headers to be deprecated.
         
   Client authors should note in particular that lack of support for the
   CAPABILITIES command is a good indication that the server does not
   support this specification.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 19 May 2005 11:30:39 +0100

>    o  The rules for article numbering (Section 6) have been clarified.

I've added "(also see Section 6.1.1.2)" because that's got quite a lot in
it.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 May 2005 14:42:50 GMT

There are now two drafts for your consideration:

draft-lindsey-news-nntp-uri-00.txt
draft-ellermann-news-nntp-uri-00.txt

The first is exactly what I put before you a few months back, except that
the '!' has been removed as a possible newsgroup-name character. You all
seemed happy with that text last time.

It is based on our present NNTP drafts.
It uses the definitions of newsgroup-name and message-id from our NNTP drafts.
It allows wildmats as in our NNTP drafts, so you can write:
   <news://news.example.com/*.test,!de.test>

It is based on RFC 977.
It uses the definition of newsgroup-name from USEFOR.
It uses Yet Another Definition of message-id, loosely based on RFC 1036.
It supports only a single '*' as a wildcard, as in RFC 1738 and meaning
only "all available newsgroups", and points out that if you omit it
entirely the meaning is unchanged. A vague mention that some clients might
support "different forms of wildcard notations", which is a nonsense
because it will usually be the server that determines what is supported.

Take your pick. COmments should really be sent to the [email-masked] list, but
I will pass on anything that is posted here.

---

From: Urs Janßen <[email-masked]>
Date: Tue, 24 May 2005 12:06:21 +0200

What happend to LIST SUBSCRIPTIONS (RFC 2980) and LIST MOTD?
<[email-masked]> ff. didn't
show up any rejections (and e.g. tin uses "LIST SUBSCRIPTIONS" for
ages). At last if LIST SUBSCRIPTIONS now is deprecated this should
be noted in the appendix "Changes from RFC 977 / RFC 2980".

urs

---

From: Russ Allbery <[email-masked]>
Date: Tue, 24 May 2005 09:34:25 -0700

> be noted in the appendix "Changes from RFC 977 / RFC 2980".

I'm not sure where to find that message ID.

I don't think that either were ever in our draft, or at least I don't
recall them being there.  They at least weren't in the draft as far back
as base-13.  Which is odd, since as you point out, LIST SUBSCRIPTIONS is
in RFC 2980.  The intent was not to deprecate LIST SUBSCRIPTIONS at least
that I can recall, although we didn't end up standardizing it either.

There is a generic mechanism for software to include additional LIST
keywords, so we haven't outlawed either.  If I'd realized that LIST
SUBSCRIPTIONS wasn't in there a year back, I probably would have argued
for inclusion; at this point, I'd argue for a supplemental RFC documenting
it and other LIST extensions if folks want.

---

From: Urs Janßen <[email-masked]>
Date: Tue, 24 May 2005 18:57:35 +0200

> I'm not sure where to find that message ID.

<nntp://news.gmane.org/[email-masked]>
<nntp://news.gmane.org/gmane.ietf.nntp/787>

| From: "Jeffrey M. Vinocur" <[email-masked]>
| Date: Sun, 27 Apr 2003 00:17:33 -0400 (EDT)
| Subject: ietf-nntp RFC 2980 replacement
[...]
|    - LIST SUBSCRIPTIONS
|
|    Do we want an extension for this?  Does anybody actually use it?
|    Anyway, certainly easy to include if desired.

> keywords, so we haven't outlawed either.  If I'd realized that LIST

draft26 3.3.3 says
| Each extension MUST define at least one new capability label
[...]
| An extension is either a private extension or else its capabilities
| are included in the IANA registry of capabilities
[...]
| A private extension MAY or MAY NOT be included in the capabilities
| list. If it is, the capability label MUST begin with "X"

I would read this as 'either LIST SUBSCRIPTIONS must be deprecated or
renamed or defined in this (or another RFC (in which case the draft
can't supersede RFC2980 till LIST SUBSCRIPTIONS is defined elsewhere ,-)',
but this point iof view might be too strong.

urs

---

From: Russ Allbery <[email-masked]>
Date: Tue, 24 May 2005 10:11:58 -0700

> <nntp://news.gmane.org/gmane.ietf.nntp/787>

Hm.  It looks like we dropped the ball on this a couple of years back,
since I don't recall any follow-up to that as well.

> can't supersede RFC2980 till LIST SUBSCRIPTIONS is defined elsewhere
> ,-)', but this point iof view might be too strong.

In this case (and Clive, please correct me if I'm wrong, since you're more
familiar with the wording), LIST is the capability label and there's a
general, blanket permission granted to add additional list types to that
capability label.  Those additions don't have to begin with X.

I don't entirely trust my memory, but I think that was done that way
intentionally since we know that existing servers support LIST extensions
that we didn't want to either document or outlaw.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 24 May 2005 13:17:33 -0400

> that we didn't want to either document or outlaw.

This sounds right.  The LIST SUBSCRIPTIONS command (and its 
corresponding "SUBSCRIPTIONS" argument to the "LIST" capability) 
*should* be specified in a new document referencing the upcoming RFC 
3977.  Eventually, anything in RFC 2980 that's still useful should be 
specified in new document(s) and RFC 2980 moved to historical.

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Tue, 24 May 2005 20:20:27 -0400

> specified in new document(s) and RFC 2980 moved to historical.

Right.

I can't recall if there was actually consensus about this (or even 
discussion beyond the bit quoted upthread), but when I started the 
STREAMING draft I intentionally didn't include all the "useful" bits of 
2980.  My impression was that lumping so many unrelated commands into 
one extension document was prone to cause trouble in the future (e.g. 
to update or obsolete a single extension) with little real benefit.

Thus, although the four documents we have now do account for much of 
the functionality described by 2980, I'm not sure we can retire it yet. 
  For example, at some point we need to tackle XPAT...

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Tue, 24 May 2005 17:24:32 -0700

Jeffrey M Vinocur <[email-masked]> writes:

> Thus, although the four documents we have now do account for much of the
> functionality described by 2980, I'm not sure we can retire it yet. For
> example, at some point we need to tackle XPAT...

Oh.  Yes.  I completely forgot about XPAT.

Scott, that's a definitive answer.  RFC 2980 is the only current
specification for XPAT, which definitely implemented and used in the real
world and which we are not at all prepared to standardize further at this
point.  We can't obsolete it completely or relegate it to historical
status until we've dealt with that.

I do think we should mark RFC 2980 as updated by the new base
specification, AUTHINFO, and STREAMING, though.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 24 May 2005 21:32:43 -0400

> specification, AUTHINFO, and STREAMING, though.

AUTHINFO and STREAMING already say that they update 2980 (actually they 
incorrectly state 2970 in the boilerplate, but I've fixed it in my 
updated versions).  I note that the new base doc doesn't say anything 
about updating 2980 or obsoleting 977 in the boilerplate, but the editor 
can probably fix that.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 25 May 2005 11:17:35 +0100

> capability label.  Those additions don't have to begin with X.

That's correct.

> that we didn't want to either document or outlaw.

Indeed, in my time we've dropped LIST EXTENSIONS and LIST DISTRIBUTIONS,
knowing that implementations can still provide them.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 25 May 2005 11:35:10 +0100

Ken Murchison said:
> I note that the new base doc doesn't say anything 

It turns out that xml2rfc lets me put the "updates" and "obsoletes" clauses
in and it Just Works, putting "(if approved)" in the header. This will go
into any future versions, including what finally goes to the RFC editor.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 31 May 2005 20:28:40 -0700

> The first is exactly what I put before you a few months back, except
> that the '!' has been removed as a possible newsgroup-name
> character. You all seemed happy with that text last time.

I personally would prefer to see the NNTP URI scheme wait on the
publication of the new base document and then follow the approach that
you're taking of picking up the new NNTP work.  I'm still not sure that
the full wildmat support in the URI is really the best way to go, but I
can't come up with any strong argument against it other than "it looks odd
and I'm not sure how many people will bother using it," which doesn't seem
strong enough.