Subject: [NNTP] LIST EXTENSIONS (again)

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 04:16:06 2004

Okay, let me have another go, this time assuming that we only want a
minimal version indication for LIST EXTENSIONS. Here's a proposal:

(0) Adopt the "restrict characters in command names and extension labels
proposal".

(1) Make LIST EXTENSIONS mandatory.

(2) Require a version entry in the output from LIST EXTENSIONS. However,
rather than "NNTPv2", I would suggest:

    _RFC_ 3977

(this assumes that _ is not a permitted character); 3977 is the number of
the final RFC for NNTP. [This approach has been used successfully in other
standards, and allows more flexibility than just an arbitrary string.]

(3) A conforming server MUST NOT advertise any extension label, other
than one beginning with X, not in the IANA registry.

(4) Any specification in extensions should be phrased along the lines of:

    This specification requires that the server conform to [NNTP],
    indicated by "_RFC_ 3977" or some higher number in the output from
    LIST EXTENSIONS.

or:

    If the server conforms to [NNTP], indicated by "_RFC_ 3977" or some
    higher number in the output from LIST EXTENSIONS, then ....
    Otherwise ....

(5) I'd *like* to see this entry also used to indicate at least some
of the optional things that are in the core protocol. For example:

    _RFC_ 3977 MODE_READER         indicates MODE READER is significant

    _RFC_ 3977 LIST_OPTIONAL       indicates all four optional LIST
                                   commands (excludes OVERVIEW.FMT) are
                                   supported

[Actually, those are the only two significant options I can think of.] But
don't let this derail the previous points.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 2 06:47:42 2004

> (0) Adopt the "restrict characters in command names and extension labels

If this is the proposal to allow upper/lower and numeric chars (with a 
few extras), I agree.

> (1) Make LIST EXTENSIONS mandatory.

Absolutely.

> rather than "NNTPv2", I would suggest:

My only problem with tying the capability to a particular document 
revision, is that the document may be updated in the future to fix 
errata, ambiguities, etc without changing the underlying functionality. 
  For example, RFC3977 gets updated by RFC4321 without change of 
functionality, should the server then advertise "_RFC_ 4321" or continue 
to advertise "_RFC_ 3977"?  I don't think either case is optimal.  By 
choosing a somewhat arbitrary string like "NNTPv2", we're RFC 
revision-proof, and still have the intended behavior.

> standards, and allows more flexibility than just an arbitrary string.]

Has RFC3977 actually been reserved for the new NNTP spec (ala 
RFC282[12]), or did you just pick this as an example?  I think reserving 
3977 it is a good idea, but we'd need to move fast if it hasn't already 
been done, since the RFCs are up to at least 3955 already.

> than one beginning with X, not in the IANA registry.

Agreed.

>     higher number in the output from LIST EXTENSIONS, then ....

Yes, we would need some sort of language along these lines.

>                                    commands (excludes OVERVIEW.FMT) are

I like the idea of advertising whether MODE READER is required and which 
optional LIST commands are supported, but I don't see where our new 
token needs to be used as part of this.

Is it realistic to assume that if a server supports one of the optional 
LIST commands, that it would support all of them?  I don't know, I'm 
just asking (especially since Cyrus nntpd which I wrote only currently 
supports LIST NEWSGROUPS).  It seems like it would be better to 
enumerate them as part of the extension label, e.g.:

[C] LIST EXTENSIONS
[S] MODE_READER
[S] LIST ACTIVE.TIMES DISTRIBUTIONS DISTRIB.PATS NEWSGROUPS
[S] .

Obviously, any combination of the four LIST arguments is valid.

If we nail down capabilities for these two items, then a client can 
learn everything it needs to from LIST EXTENSIONS and no longer has to 
guess or do a trial-and-error.  This is a "good thing" IMHO.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 07:37:22 2004

> If this is the proposal to allow upper/lower and numeric chars (with a 

It is.

> revision-proof, and still have the intended behavior.

Two things. Firstly, I want the number separate from the pseudo
extension-label, to make it clear that it is a number and not an arbitrary
string. Whether it's "_RFC_ 3977" or "_NNTP_ 2" is secondary.

Secondly, we're back the the versioning issue again.

There are three basic cases to consider and, as far as I can see, three
approaches we could follow.

Case 1: a new specification makes absolutely no changes to what's on the
wire; it just clarifies things.

Case 2: a new specification is completely upwards compatible but adds new
facilities. Note: a MODE NEW command that *changed* the semantics of
existing commands is fine, so long as an unsuspecting client doesn't see
the changes.

Case 3: a new specification is not upwards compatible for whatever reason.

Approach A: the server lists the latest RFC to which it conforms. The
client needs a list of RFCs that it can handle.
- Following a case 1 change, the number does *not* change; the server
  continues to reply "3977" and not "4321".
- Following a case 2 change, the number does change; the server replies
  "4321". The client needs to know that it can talk to either 3977 or 4321
  servers even if it doesn't grok the extra features of the latter.
- Following a case 3 change, the number does change; the server replies
  "4321". The client needs to know that it can talk to 3977 but not 4321
  servers, or it needs separate code paths contingent on the two.

Approach B: the server lists all the RFCs to which it conforms. The client
needs only know a single RFC (unless it has multiple code paths).
- Following a case 1 change, the number does *not* change; the server
  continues to reply "3977".
- Following a case 2 change, the number does change; the server replies
  "3977 4321". The client recognises 3977 and works, or it recognises
  4321 and uses the additional features.
- Following a case 3 change, the number does change; the server replies
  "4321". The client needs to know that it can talk to 3977 but not 4321
  servers, or it needs separate code paths contingent on the two.

Approach C: there are major and minor version numbers; what we have now
is 2.0. The client needs only know its current version number.
- Following a case 1 change, the number does *not* change; the server
  continues to reply "2.0".
- Following a case 2 change, the minor number changes and the server
  replies "2.1". A 2.0 client knows that 2.1 is upwards compatible and
  talks 2.0. A 2.1 client can use the extra features with a 2.1 server,
  and ignores them with a 2.0 server.
- Following a case 3 change, the major number changes; the server replies
  "3.0". A 2.x client knows that it can't handle this, or it has separate
  2.x and 3.x code paths.

Approach C is my preference and matches (e.g.) HTTP. I only suggested an
alternative because people choked on the major.minor idea previously.

> been done, since the RFCs are up to at least 3955 already.

I haven't done it. I'd very much like it to be done, but the RFC editors
don't seem to be talking to me - they still haven't answered my comment
about them mangling the last few drafts (as Ken, I think, first realized).

Russ? Can you have a go at this?

>> than one beginning with X, not in the IANA registry.

Note, I think that's stronger than what we had before.

> token needs to be used as part of this.

These are features of the core protocol, not extensions. Therefore, in my
opinion, they belong with the support information for the core extension.
Note how we have "OVER MSGID" rather than separate "OVER" and "OVER-MSGID"
capabilities.

> LIST commands, that it would support all of them?  I don't know,

Nor do I. I have no objection to splitting them; I'm just getting chary of
making complex-looking proposals (even when they're actually conceptually
simple).

> It seems like it would be better to 

I'd prefer:

  [C] LIST EXTENSIONS
  [S] _NNTP_ 2.0 MODE_READER LIST_ACTIVE_TIMES LIST_NEWSGROUPS
  [S] STREAMING
  [S] .

for the reasons given above.

> Obviously, any combination of the four LIST arguments is valid.

Of course.

Agreed.

Um, I'm not going to swear there's no other significant options that are
worth noting here. Can anyone think of any?

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 2 08:38:23 2004

> string. Whether it's "_RFC_ 3977" or "_NNTP_ 2" is secondary.

Ah, OK.  I didn't notice the space.  I'm not sure I like it, but I'm not 
going to make a big deal of it.

The main point is that we need a way to specify the core functionality 
and behavior to the client.  I *think* we are all in agreement on this 
point.

> Case 3: a new specification is not upwards compatible for whatever reason.

Hopefully case 3 is never allowed to happen without *very* good reason, 
but discussing it OK.

> alternative because people choked on the major.minor idea previously.

A and C seem to be functionally identical to me and just differ in the 
syntax.  I could live with either.  FWIW, A is similar to IMAP.  IMAP 
has both "IMAP4" and "IMAP4bis" capabilities.

> Note how we have "OVER MSGID" rather than separate "OVER" and "OVER-MSGID"

Agreed, since they are not extensions they do belong as arguments to the 
NNTP version capability.  My main point was that we should enumerate the 
optional LIST commands.

> making complex-looking proposals (even when they're actually conceptually

Sorry, didn't mean to scare you off.  ;)

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 2 11:50:53 2004

> worth noting here. Can anyone think of any?

I was just scanning nntp-24 and noticed that LIST OVERVIEW.FMT is listed 
as optional.  This seems somewhat questionable to me.  If the server 
advertises support for OVER, what shouldn't it support LIST 
OVERVIEW.FMT?  If the answer is because some legacy implementations 
don't support it, my response is who cares?  OVER is new and we can and 
should be able to enforce support for LIST OVERVIEW.FMT.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 12:16:20 2004

Ken Murchison said:
> as optional.

Yes, but it's part of the OVER extension which is why I left it out of my
previous messages.

> This seems somewhat questionable to me.  If the server 

Fine with me, but we need rough consensus on the list.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 4 09:09:04 2004

> and behavior to the client.  I *think* we are all in agreement on this 

Indeed (well, I agree).

> Hopefully case 3 is never allowed to happen without *very* good reason, 

We need to think about it, otherwise we risk being hit over the head later
when it happens. Look how OVER is incompatible with the old XOVER, and some
of the similar debates of the past.

> syntax.

In A, case 2, the client needs to know that 4321 exists and is upwards
compatible; otherwise it will barf when it meets a 4321 server. In C, a
2.0 client automatically knows that 2.1 is upwards compatible.

*B* and C are functionally equivalent - the server indicates all the client
versions it can work with and the client needs to know only its own
version. B does it explicitly, while C does it implicitly.

Let me present it another way. Let's suppose that we have RFC 3977, a
compatible update in 4321, and an incompatible one in 5555. There are
three types of server, which issue responses:

  Server type:       3977            4321            5555
                   ---------------------------------------
  Approach A         3977            4321            5555
  Approach B         3977            3977 4321       5555
  Approach C         2.0             2.1             3.0

There are *four* types of client:
  (P) 3977 that doesn't know 4321 exists ) this distinction only
  (Q) 3977 that *does*  know 4321 exists ) matters in approach A
  (R) 4321
  (S) 5555

When they meet, we get:

  Server type:       3977            4321            5555
                   ---------------------------------------
  Client type P:     works          [1][3]             reject
  Client type Q:     works          [2][3]             reject
  Client type R:      [4]            works           reject
  Client type S:    reject          reject           works

[1] Reject in approach A, because 3977 != 4321.
[2] Works in approach A because the client knows that 4321 encompasses
    3977 as a subset.
[3] Works in approach B because server response includes 3977; works in
    approach C because 2.1 >= 2.0 but indicates compatibility.
[4] Works in approaches A and B because the client knows 3977 was a subset
    of 4321; works in approach C because 2.0 indicates compatibility. In
    both cases the client needs to know which features to avoid.

> My main point was that we should enumerate the 
> optional LIST commands.

Agreed (again).

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 11:43:00 2004

> 2.0 client automatically knows that 2.1 is upwards compatible.

I think you're splitting hairs here.  Why does a 2.0 client know that 
2.1 is upwards compatible?  Just because of the version number?  Whether 
it seems obvious or not given your choice of syntax, it still MUST be 
explicitly document somewhere that 2.1 retains 2.0 functionality (AFAIC 
2.0 and 2.1 are still meaningless tokens at a protocol level).  The same 
would be true in the A case, RFC 4321 MUST explicitly state that it 
retains RFC 3977 functionality.  In either case, the client knows this 
because the author read the specs and programmed accordingly.

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 22:00:16 2004

>> This seems somewhat questionable to me.  If the server advertises
>> support for OVER, what shouldn't it support LIST OVERVIEW.FMT?  If the
>> answer is because some legacy implementations don't support it, my
>> response is who cares?  OVER is new and we can and should be able to
>> enforce support for LIST OVERVIEW.FMT.

> Fine with me, but we need rough consensus on the list.

I have no problems with making it mandatory.  Mark has convinced me to
fear and hate optional commands.  :)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 22:17:04 2004

>> I like the idea of advertising whether MODE READER is required and
>> which optional LIST commands are supported, but I don't see where our
>> new token needs to be used as part of this.

> These are features of the core protocol, not extensions. Therefore, in
> my opinion, they belong with the support information for the core
> extension.  Note how we have "OVER MSGID" rather than separate "OVER"
> and "OVER-MSGID" capabilities.

So, here's a question.  Why *aren't* they extensions?  If it's just
because they were in RFC 977, I don't think that's a very good reason;
we've already eliminated one command that no one used (SLAVE), and I'm
happy to take other little-used commands and move them into the extension
category.

Here's a proposal:  Why don't we make LIST NEWSGROUPS mandatory (pretty
much everyone implements it, even if they don't have any data to return,
and the case of not having any data to return is already supported in the
description of the command), and make LIST ACTIVE.TIMES, LIST
DISTRIB.PATS, and LIST DISTRIBUTIONS separate extensions?

(Actually, I'd be quite happy if we dropped LIST DISTRIBUTIONS entirely,
but that may be too much hassle at this point.  It's a rather pointless
command that could be replaced by a web page at sites that care.)

They're optional commands, we want to advertise them in LIST EXTENSIONS,
we have a well-defined way of doing that for actual extensions... I think
this would be the simplest approach.

> Nor do I.

They're unrelated.  Many servers support one of them and not the others.

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 22:53:25 2004

Okay, I've read and I think digested all of the long threads about the
NNTPv2 label and related issues.  I'm pleasantly surprised that everyone
agreed that we should have the protocol version advertised; I thought that
was going to be controversial!

Here is my take on this:

I think we need to be able to say what version of the base protocol
specification the server follows (everyone agrees on this).  I would
prefer to copy IMAP as closely as possible here because IMAP has done this
and we know that it works in practice, which means that unless we can see
a really significant problem with IMAP's approach, I don't think we should
think too hard.  It's possible that IMAP just hasn't encountered the
problems that could come up, but I think it's also equally possible that
IMAP's track record shows that the problems don't actually come up.

I don't think we need to know what version of an extension the server
supports because I don't expect extensions to have versions.  Note that no
documented standard command has ever changed its behavior in a significant
and non-backwards-compatible way over the life of the NNTP protocol; the
only things we've had to change were for commands like XOVER where
implementations diverged without a specification.  (Well, I guess that the
dropping of the optional distribution argument to NEWNEWS could be called
a non-backward-compatible change, but I think that's a stretch.)

If a command changes in a non-backward-compatible way, I think a new
command should be introduced instead and the old one left alone and
deprecated.  This is analogous to header fields.  New names are cheap;
backward compatibility problems aren't.  And even if we had a versioning
mechanism, I know there would be problems with older clients that didn't
support it quite right.  That's code that's likely to break since it's
never exercised in practice and won't be for years.

If a command changes but in a backward-compatible way, this is why we have
additional arguments to the extension advertisement.  Like with OVER
MSGID, we can define a way to advertise the new capability with more
specificity than is offered by a version number.

I'm comfortable setting the basic parameters of the extension in stone
once it's published, at least barring another major NNTP revision.  And I
agree with Ken that if it turns out we desperately need this, we can add
it later, with a bit of trouble.

I see what Clive is getting at by making the extension label "NNTP 2" or
the like instead of "NNTPv2", but I think it's better to go the route that
lets us advertise compatibility with multiple NNTP versions rather than
going the route that lets clients compare numbers.  Again, I'd rather
follow IMAP here.  I don't see a need for minor protocol revisions
indicating backward-compatible changes; I think that is subsumed by the
ability to advertise compatibility with multiple revisions (listing both
NNTPv2 and NNTPv3 extension labels, for instance).

I don't think using an RFC number instead of an arbitrary version number
is a good idea for the reasons that Ken stated.  There's too much chance
that RFCs may have significant errata, there may be multiple RFCs that
update each other, etc.  We're safer with an arbitrary version number like
v2.  (And again, let's follow IMAP.)

I do think that we want to adopt something out of the discussion that
Andrew and Mark had about avoiding round trips.  NNTP is too heavy on
round trips right now, and that's been bothering me when we list all the
places clients need to issue LIST EXTENSIONS again.  At the least, we need
to advertise whether MODE READER is required.  I think there are four
possible states the server can be in:  all commands supported; reader
commands not supported until after MODE READER; transit commands not
supported at all; and reader commands not supported at all.  I think the
best way to indicate those states would be with two extensions:

    NOREADER [MODE-READER]
    NOTRANSIT

(those may not be the best names).  NOTRANSIT is a negative extension
saying that IHAVE is not supported, which saves a client the need to try
it and get 502.  NOREADER is a negative extension saying that reader
commands (ARTICLE, BODY, POST, GROUP, LAST, NEXT, NEWGROUPS, and NEWNEWS)
are not available.  If the optional MODE-READER parameter is given, it
indicates that they will be available (possibly still requiring STARTTLS
and authentication) after issuing MODE READER.  Note that this label
doesn't need to imply anything about extensions, since that can be done
with the presence or absence of the extension labels themselves.

After Mark's messages, I've been convinced that we should mandate the
order of the state change commands.  The correct order as near as I can
tell would be:

    MODE READER
    STARTTLS
    AUTHINFO

(where you can skip the initial MODE READER if your initial LIST
EXTENSIONS doesn't have the NOREADER extension listed).  That still means
that a connection with TLS and authentication requires seven commands to
initiate in the worst case, though, which is rather obscene:

    LIST EXTENSIONS
    MODE READER
    LIST EXTENSIONS
    STARTTLS
    LIST EXTENSIONS
    AUTHINFO SASL
    LIST EXTENSIONS

Of course, nearly all servers won't require MODE READER, which cuts out
the second and third commands.

One could cut this down by one more command by adding some sort of initial
extension advertisement to the greeting, but this worries me for the
reasons stated by Clive and Andrew, just in a cleanliness sense.  It's
worth considering, though.

One can also gain a round trip with a different idea, namely having
successful successful AUTHINFO SASL (which is a new commands, so we can do
whatever we want with it) include LIST EXTENSIONS output in its response.
I'm not sure what I think of that; I'm guessing it's a bad idea since no
one else does it.

Assuming most servers don't have MODE READER, the normal case with TLS and
auth will be five round trips (plus however many SASL needs).  Adding
[STARTTLS] or some such thing to the welcome banner would make that four.
Assuming that sending extension information after AUTHINFO is a bad idea,
I don't see any way to get it lower than four -- two of the commands have
to be issued, the LIST EXTENSIONS after STARTTLS is required for security
reasons since you can't trust the results of it beforehand, and the LIST
EXTENSIONS after AUTHINFO is required if authenticating changes anything
about what you can do.  (Although I'm wondering how often authenticating
is really going to change anything other than noting that you can no
longer run STARTTLS or AUTHINFO again, and if clients would really need to
check there.)

>>> (0) Adopt the "restrict characters in command names and extension
>>> labels proposal".

> It is.

I agree with this as well.

>   servers, or it needs separate code paths contingent on the two.

This is equivalent to having NNTPv2, NNTPv3, etc. extension labels that
can all be advertised together if the revisions are compatible somehow.  I
think this is the best approach to take.

>> RFC282[12]), or did you just pick this as an example?  I think
>> reserving 3977 it is a good idea, but we'd need to move fast if it
>> hasn't already been done, since the RFCs are up to at least 3955
>> already.

> Russ? Can you have a go at this?

I'll ask our AD.

> Note, I think that's stronger than what we had before.

Yes, but I think it's the right thing to do.

Optional LIST commands discussed in another message.  (Mark, to give you
the summary, I'm proposing making one of them mandatory and the other
three regular extensions with regular extension labels; the argument I
gave you before about them is sounding less persuasive to me.)

> worth noting here. Can anyone think of any?

Do we want to get into advertising whether commands are permitted for a
particular client?  (POST and NEWNEWS are the ones most frequently not
permitted.)

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Nov 5 00:55:29 2004

Russ Allbery said:
> NNTPv2 label and related issues.
[...]

I have a number of issues I want to talk through before we make final
decisions on this. I'm going to be off-net for most of today and Monday,
but I'll try to write some stuff ASAP.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 06:44:57 2004

> because they were in RFC 977, I don't think that's a very good reason;

Actually, to strengthen the argument, they weren't in RFC 977, they are 
only documented in RFC 2980, which IMHO definitely puts them in the 
extension category.

> DISTRIB.PATS, and LIST DISTRIBUTIONS separate extensions?

Works for me.  I'd [re]suggest a "LIST" extension with one or more 
arguments, e.g.:

[C] LIST EXTENSIONS
[S] LIST ACTIVE.TIMES DISTRIBUTIONS
[S] .

Of course, if we don't allow '.' in arguments, we could use '-' instead.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 07:50:33 2004

Russ Allbery wrote:

I agree with all of Russ' points.  No surprise, since he has agreed with 
most of my opinions on this topic  ;)

> NNTPv2 and NNTPv3 extension labels, for instance).

Just to clarify the way that I envision that this would work (and the 
way it currently works in IMAP).  If NNTPv3 adds to NNTPv2 and retains 
NNTPv2 functionality, then only NNTPv3 would have to be advertised by 
the server (although there would be no harm in also advertising NNTPv2). 
  If NNTPv3 removes functionality from NNTPv2, but the server retains 
this functionality, then both NNTPv3 and NNTPv2 would have to be advertised.

 > At the least, we need

Works for me.  The only question that I have is whether these 
capabilities are advertised as arguments to NNTPv2 or as standalone 
extension labels?  The only alternative names that I have for the 
capabilities would be something like "READINGDISABLED" and 
"TRANSPORTDISABLED"

> order of the state change commands.  The correct order as near as I can

If we do this, then I can add the same language to STARTTLS and AUTHINFO 
that I proposed for yesterday for OVER, etc.

> worth considering, though.

I understand that most folks find this proposal somewhat "distasteful" 
(for lack of a better term), but does anyone have any *technical* 
reasons why this can't work?

> I'm not sure what I think of that; I'm guessing it's a bad idea since no

I don't think IMAP disallows it.  Mark, does your server do this?

I don't see any technical reasons why we can't.  The only issue would be 
that if a SASL security layer is negotiated, any extensions returned in 
the response would probably be ignored by the client anyways, since the 
response is sent in the clear.  Only subsequent commands (e.g. LIST 
EXTENSION) are protected by the SASL security layer.

So, assuming capabilities are advertised in 200, 201, 281 and 283, this 
would look like:

	[MODE READER]
	STARTTLS
	LIST EXTENSIONS
	AUTHINFO
	[LIST EXTENSIONS]

MODE READER would only be used if required by the server and the second 
LIST EXTENSIONS would only be used if a SASL security layer was negotiated.

> and the LIST

I'm probably not qualified to answer this, but might there be a case 
where "lightweight" reading is offered for free (w/o authentication), 
but more "expensive" commands such as NEWNEWS or OVER or some future 
XSEARCH are only allowed to (paying) customers that authenticate?

> particular client?  (POST and NEWNEWS are the ones most frequently not

Doesn't the 201 response signal that POST is disabled, or I'm I 
completely clueless?  NEWNEWS can probably just spit back 503, correct?

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 08:19:54 2004

On Fri, 5 Nov 2004, Ken Murchison wrote:
> I agree with all of Russ' points.  No surprise, since he has agreed with most 
> of my opinions on this topic  ;)

I think that I do as well, but I need to digest his message more 
thoroughly.

> Works for me.  The only question that I have is whether these capabilities 
> are advertised as arguments to NNTPv2 or as standalone extension labels?

IMHO, standalone labels are easier to deal with.  Ease of client 
implementation has priority over before syntactic sugar.

> If we do this, then I can add the same language to STARTTLS and AUTHINFO that 
> I proposed for yesterday for OVER, etc.

I wish that I could sign off on this, but I don't recall precisely which 
server(s) issued a 480 to MODE READER.  I won't sign off on this unless 
that 480 is explicitly forbidden.

> I understand that most folks find this proposal somewhat "distasteful" (for 
> lack of a better term), but does anyone have any *technical* reasons why this 
> can't work?

IMAP status response codes and SMTP extended response codes both prove 
that it can (and does work).

> I don't think IMAP disallows it.  Mark, does your server do this?

My IMAP server used to issue an untagged CAPABILITY response to a LOGIN or 
AUTHENTICATE command, but this annoys IMAP2 clients (believe it or not, 
such things still exist in the world!).  As a result, it's now carried in 
the command response (which is completely compatible with IMAP2).

Here is a startup from a real IMAP4rev1 session:

S: * OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS LOGINDISABLED AUTH=CRAM-MD5 AUTH=ANONYMOUS] Ikkoku-Kan.Panda.COM IMAP4rev1 2004.355 at Fri, 5 Nov 2004 08:06:25 -0800 (PST)
C: 00000000 STARTTLS
S: 00000000 OK STARTTLS completed
C: 00000001 CAPABILITY
S: * CAPABILITY IMAP4REV1 LITERAL+ IDLE NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND SASL-IR LOGIN-REFERRALS AUTH=CRAM-MD5 AUTH=PLAIN AUTH=LOGIN AUTH=ANONYMOUS
S: 00000001 OK CAPABILITY completed
C: 00000002 AUTHENTICATE CRAM-MD5
S: + PDQyMzguMTA5OTY3MDc5MUBJa2tva3UtS2FuLlBhbmRhLkNPTT4=
C: bXJjIDM2M2I1NzU3YjVmMDY3ODI1YjkxZGVlODM5Y2IwZmJl
S: 00000002 OK [CAPABILITY IMAP4REV1 LITERAL+ IDLE NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User mrc authenticated

Note that this entire exchange was just 4 RTTs.  It would have been 3, 
except that the CRAM-MD5 SASL authenticator requires a server challenge.

Note also that there are many more capabilities, and a much more complex 
protocol, than in NNTP; yet the responses were still much smaller than 512 
octets.

If we had been clever with TLS, we would have designed STARTTLS so that 
there was an intermediate response (not under TLS) and a final response 
(under TLS) so the exchange looked like:

C: 001 STARTTLS
S: + TLS starting, switch now
S: 001 OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS AUTH=CRAM-MD5 AUTH=PLAIN AUTH=LOGIN AUTH=ANONYMOUS] If you see this we are talking TLS

This would have eliminated the need for the extra CAPABILITY command and 
thus reduced another RTT.

> I don't see any technical reasons why we can't.  The only issue would be that 
> if a SASL security layer is negotiated, any extensions returned in the 
> response would probably be ignored by the client anyways, since the response 
> is sent in the clear.  Only subsequent commands (e.g. LIST EXTENSION) are 
> protected by the SASL security layer.

Same comment about more cleverness applies here too.

>> (Although I'm wondering how often authenticating
> I'm probably not qualified to answer this, but might there be a case where 
> "lightweight" reading is offered for free (w/o authentication), but more 
> "expensive" commands such as NEWNEWS or OVER or some future XSEARCH are only 
> allowed to (paying) customers that authenticate?

Right.

Also, servers which require authentication may decline to advertise their 
feature set to hackers who don't have an account.

> Doesn't the 201 response signal that POST is disabled, or I'm I completely 
> clueless?  NEWNEWS can probably just spit back 503, correct?

Yes!!  There should be a POST and NEWNEWS capability, and less reliance 
upon magic response codes.

Done right, clients should be able to care only about the first digit of 
the response code.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 08:46:51 2004

Mark Crispin wrote:
>> If we do this, then I can add the same language to STARTTLS and 
>> AUTHINFO that I proposed for yesterday for OVER, etc.
> 
> 

If it was my Cyrus NNTP server, its a mistake.

I'll defer to others on whether forbidding 480 will work in the wild.

> upon magic response codes.

So, you're proposing "POSTDISABLED" and "NEWNEWSDISABLED" capabilities?

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 09:39:52 2004

> So, you're proposing "POSTDISABLED" and "NEWNEWSDISABLED" capabilities?

Either that, or POST and NEWNEWS capabilities.  Pick your poison.

Remember, with NNTPv2 we have a chance to redefine the base behavior as we 
please.

Internally, it's a check for "post" than "!postdisabled".

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Nov 5 10:48:16 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> I wish that I could sign off on this, but I don't recall
 Mark> precisely which server(s) issued a 480 to MODE READER.  I won't
 Mark> sign off on this unless that 480 is explicitly forbidden.

A quick survey of sites suggests that currently the Diablo reader has
this issue.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Fri Nov 5 13:21:25 2004

> was going to be controversial!

I've also been impressed by the amount of headway we've made; this is 
probably the most functional the WG has been in quite a while :-)

I agree strongly with most of what Russ has said.  A few comments 
follow.

> order of the state change commands.  The correct order as near as I can

I agree with that order; I found some of the orders discussed up-thread 
rather odd.

> longer run STARTTLS or AUTHINFO again, and if clients would really 

I wonder this as well.  But even if we manage to say something like "no 
previously advertised extensions can be removed except for the 
following" and "clients only need to send LIST EXTENSIONS again if they 
might be interested in some extension not previously advertised," then 
as soon as there's one common extension that becomes available 
mid-session, everything will degenerate into the "must send LIST 
EXTENSIONS" case again.

Also, I have the impression that NNTP sessions are somewhat 
longer-lived than most protocols, and thus that startup overhead is 
relatively less important.  But I could be wrong about that.

> particular client?  (POST and NEWNEWS are the ones most frequently not

If we're going to start advertising whether reading commands are 
available, and then whether various other commands like NEWNEWS are 
available (and doing such advertising seems very reasonable to me), and 
then we get into whether they're available only after MODE READER, and 
then we get into whether they're available after 
authentication...perhaps we should try to tackle this in some uniform 
way.

Part of my concern is for cleanliness, but part of it is also for its 
effect on behavior.  I mean, suppose the unauthenticated client sees 
that it has general newsreading capabilities (NOREADER is not 
advertised), but doesn't see NEWNEWS advertised.  Suppose the client 
prefers NEWNEWS, but can fall back to alternative core commands if 
NEWNEWS is not available (I believe this is a common client 
implementation choice).  Now the client software has no way of knowing 
(without explicit user configuration) whether it should prompt the user 
to authenticate, or give up on NEWNEWS and start downloading articles 
immediately using the alternative mechanisms.

And yes, I realize this could also balloon into something quite 
complicated.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Fri Nov 5 13:25:35 2004

> So, you're proposing "POSTDISABLED" and "NEWNEWSDISABLED" capabilities?

Oof, these extension names (and NOREADER, NOTRANSIT) are kinda making 
my head spin.  I think we should try to stick to non-negated names 
where possible.  Even something like:

TRANSIT YES
READER MODE-READER-REQUIRED
POST NO
NEWNEWS NO

would be less confusing, I think?

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 13:55:41 2004

> A quick survey of sites suggests that currently the Diablo reader has

This strikes me as obviously wrong behavior that should be explicitly
banned in the standard.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 15:10:12 2004

On Thu, 4 Nov 2004, Russ Allbery wrote:
> As the maintainer of INN, I can assure you with 100% certainty that this
> order will NOT work in the unusual situation where MODE READER is
> required.  Your AUTHINFO command will fail before MODE READER, and after
> nnrpd spawns it will have no idea that you ever attempted it.

> A quick survey of sites suggests that currently the Diablo reader has

Well, then, we have the worst-case situation.

Some servers require MODE READER first, and some other servers require 
AUTHINFO first.

I respectfully submit that this situation is unacceptable from the 
viewpoint of a client author.  NNTPv2 should rule and either determine One 
True Order, or declare that the order is totally dependent upon the 
client.

Since Russ is here with us, there may be more hope in expecting inn to 
submit to the new world order.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 15:13:42 2004

> Some servers require MODE READER first, and some other servers require

I think the situation is not actually as bad as that, as I believe Diablo
doesn't require MODE READER at all (just "helpfully" returns 480 if you
happen to use it before you're authenticated).  They're equivalently bad
from a client writer's standpoint, but it means that it should be very
easy to get Diablo fixed; it's probably just one line of code.

> viewpoint of a client author.  NNTPv2 should rule and either determine
> One True Order, or declare that the order is totally dependent upon the
> client.

I certainly agree with you.  I think we can declare one true order.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 15:47:18 2004

> I certainly agree with you.  I think we can declare one true order.

OK.  Assuming that we're sticking with

     MODE READER
     STARTTLS
     AUTHINFO

I will fix the STARTTLS and AUTHINFO docs to state:

"On servers that require the use of the MODE READER command, the
STARTTLS/AUTHINFO command can only be used after the MODE READER command."

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 15:52:10 2004

> STARTTLS/AUTHINFO command can only be used after the MODE READER command."

I have a problem with that choice.

The effect is to break tens (hundreds?) of thousands of deployed Pine 
clients (and other clients based upon the c-client library), and some 
unknown number of Diablo servers.

My impression is that most inn servers will *not* be broken if we choose:
 	STARTTLS
 	AUTHINFO
 	MODE READER
Indeed, even though many people use inn servers with Pine, *nobody* has 
complained about Pine having this order.

How many inn servers would be broken?  Just a handful?  Wouldn't it be 
easier to fix those handful of inn servers?

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 16:22:22 2004

> easier to fix those handful of inn servers?

Every server that actually requires MODE READER will be broken by that
order.  Fixing the servers that require that order is equivalent to
eliminating MODE READER entirely.

You can see why by observing what MODE READER does on a system that
actually needs it:  it causes a separate program to be spawned and the
connection passed to that program, destroying all established session
state.  There's no way to preserve a TLS connection and an authentication
session across that sort of change.

I'm really sorry that it looked like this was the right thing to do.  :/
This is absolutely the fault of the NNTP community for never having
written a proper specification for MODE READER.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Nov 5 16:24:37 2004

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 >> A quick survey of sites suggests that currently the Diablo reader
 >> has this issue.

 Russ> This strikes me as obviously wrong behavior that should be
 Russ> explicitly banned in the standard.

I agree.

The types of authentication available, and the process by which
authentication happens in the server are at least potentially
dependent on whether MODE READER has been issued. So requiring
authentication to come first doesn't seem to make any sense at all.
(However, I do not know enough about the authentication model in the
Diablo reader to know if this is something easily fixable in it.)

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 16:35:12 2004

> eliminating MODE READER entirely.

Only those servers which *also* have STARTTLS and/or AUTHINFO.

How many such servers are there in the wild?  I suspect that this is a 
very small number.

> session across that sort of change.

I certainly understand the technical difficulty (not impossibility) of 
fixing this.  That doesn't mean that this isn't the right solution, 
especially when we consider the task of changing a handful of deployed 
servers (if any) against tens of thousands of deployed clients.

It may be possible to help inn by making some simplifying assumptions. 
For example, if STARTTLS and AUTHINFO are assumed to be done only by 
readers, then these commands can do an implicit MODE READER and then make 
the subsequent MODE READER be a no-op.

If it's necessary for peers to do TLS and/or authentication, perhaps we 
could have a separate command for peer TLS and/or authentication.

Of course, I'm happy with the notion of eliminating MODE READER entirely 
(which I agree is what this all implies), so my position has to be 
considered with that in mind... :-)

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 16:43:53 2004

> How many such servers are there in the wild?  I suspect that this is a

The point is more that there isn't any way to fix this *in software* to
make your order work other than eliminating the MODE READER command
altogether.  There is nothing that I can do to INN to make the current
Pine work properly for servers that require MODE READER; all I could do is
make it impossible to install INN in a currently supported configuration.

I'm not saying this is impossible; I'm just saying that if we're going to
do this, we should simply eliminate MODE READER entirely.  It makes no
sense to continue to describe it but allow it to be sent after
authentication.

> It may be possible to help inn by making some simplifying
> assumptions. For example, if STARTTLS and AUTHINFO are assumed to be
> done only by readers, then these commands can do an implicit MODE READER
> and then make the subsequent MODE READER be a no-op.

AUTHINFO, unfortunately, is not only used by readers.

> considered with that in mind... :-)

Right.  :)

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 17:18:38 2004

On Fri, 5 Nov 2004, Russ Allbery wrote:
> do this, we should simply eliminate MODE READER entirely.

I would shed no tears over this outcome.

But!!!

If it's just the AUTHINFO userid, why can't it be passed as part of the 
invocation to the MODE READER server via environment variable or argv?

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 17:23:54 2004

> invocation to the MODE READER server via environment variable or argv?

Privacy layers negotiated by AUTHINFO SASL.  You're right that if it's
just an AUTHINFO USER case, it's solvable, but it doesn't work with the
more general case going forward.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 17:38:28 2004

> more general case going forward.

Are SASL privacy layers actually implemented in inn yet?  If so, which 
SASL methods?

If not, we're not talking about breaking deployed software.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 17:42:16 2004

> Are SASL privacy layers actually implemented in inn yet?  If so, which

Yes, privacy layers are implemented for every SASL method that has them
and is supported by the Cyrus SASL library (which is at least GSSAPI I
know).  Regardless of implementation, though, the standard allows for them
and I don't really want to fiddle with SASL.

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 17:49:02 2004

Ken Murchison <[email-masked]> writes:

> the server (although there would be no harm in also advertising
> NNTPv2). If NNTPv3 removes functionality from NNTPv2, but the server
> retains this functionality, then both NNTPv3 and NNTPv2 would have to be
> advertised.

Hm.  That strikes me as odd -- how does an NNTPv2 client know if a server
advertising NNTPv3 is okay or not?  This would seem to require that
clients know whether NNTPv3 is backward-compatible when NNTPv3 may not
have even been written when the client was.

I must be missing something.

> capabilities would be something like "READINGDISABLED" and

I'd rather make them stand-alone labels.

Jeff's point is well-taken, though, and I'll write a separate reply to
that going into more depth about extension labels, particularly with
relation to commands that the client may not be allowed to issue.

>> One could cut this down by one more command by adding some sort of
>> initial extension advertisement to the greeting, but this worries me
>> for the reasons stated by Clive and Andrew, just in a cleanliness
>> sense.  It's worth considering, though.

> reasons why this can't work?

I don't.

I'd prefer to include only a subset of extensions in the initial reply
just because NNTP has a response length limit for single-line responses
and existing clients do rely on that.  It may be paranoia, but I know that
particularly with private extensions, extension lists can get rather long.

STARTTLS and AUTHINFO seem like the main ones that should be advertised,
and maybe a POST and IHAVE extension.

> XSEARCH are only allowed to (paying) customers that authenticate?

Yeah, but in that case, wouldn't you advertise the extensions but reply
with a 480 code to attempts to use them, prompting the client to
authenticate?  That was what I'd been assuming in previous discussions of
extension advertisements, but maybe that's not the best way to do it?

The problem with not advertising extensions that the client is allowed to
use after authentication is that the client then has no idea whether the
extension is available but needs authentication or if it's just not there
at all.  Advertising it and then responding with 480 is annoying in the
terms of round trips, but at least lets the client figure everything out
eventually.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 17:50:35 2004

Well, then, I guess that it comes to whether more harm is caused by having 
to change:

  . a configuration (installed on a handful of sites at most) of one
    server implementation,

  . break until changed every client based upon the c-client library (most
    notably Pine -- we're talking about at least tens of thousands if not
    hundreds of thousands of separate implementations -- as well as break
    the fixed version of these clients against Diablo servers.

I agree that the choice is not unpalatable, and it would have been better 
if we could have avoided it.  But we didn't, and so we're screwed now.

If it helps matters, I would agree to a separate AUTHINFO/STARTLS command 
for clients vs. peers in return for the deprecation of MODE READER 
(replacing it with a no-op and servers MUST NOT require).

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 17:53:14 2004

On Fri, 5 Nov 2004, Russ Allbery wrote:
> advertising NNTPv3 is okay or not?

The server must advertise both NNTPv2 and NNTPv3.  A server which just 
advertises NNTPv3 is not OK to an NNTPv2 client.

This will place a chilling effect on any effort to create NNTPv3, perhaps 
permanently preventing it.  Good.

> extension advertisements, but maybe that's not the best way to do it?

I think that it's better to move away from magic response codes and use 
extensions exclusively.  The fact that authentication is offered is enough 
of a hint to anonymous clients that more can be gained through 
authentication.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Nov 5 18:11:57 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 >> STARTTLS/AUTHINFO command can only be used after the MODE READER

Complication: a server can potentially require authentication for
_transit_ connections too, in which case the client would do AUTHINFO
without ever doing MODE READER at all. Having authenticated, it would
only do IHAVE. With INN, it could in theory do MODE READER later, but
that would, as a side effect, negate the previous authentication.

 Mark> I have a problem with that choice.

It happens to be logically necessary, since (from the server
perspective) authentication may depend on knowing whether MODE READER
has been done but not the reverse. For example, the set of available
authentication methods may be different.

This happens not to be an issue with Diablo, since the only effect
that MODE READER has on a Diablo reader is to disable some
functionality.

 Mark> Indeed, even though many people use inn servers with Pine,
 Mark> *nobody* has complained about Pine having this order.

It's quite likely that no-one has used it in a situation where it
matters (INN, transfer access _and_ authentication).

I got curious about what other clients were doing. The first two
that came to hand were slrn (because I use it) and knews (because
I recently had to fix its auth code for one of my users). In both
cases, they can cope with _either_ the INN or Diablo behaviour,
because they do this:

 - connect
 - issue MODE READER
 - if the response is 480, then authenticate and redo the command
   (this is generic handling of 480 in any case)
 - if configured to authenticate and not authenticated yet,
   then authenticate now

 Mark> How many inn servers would be broken?  Just a handful?
 Mark> Wouldn't it be easier to fix those handful of inn servers?

There is no "fix".

-- 
Andrew, Supernews
http://www.supernews.com

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Nov 5 18:35:31 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> Well, then, I guess that it comes to whether more harm is
 Mark> caused by having to change:

 Mark>   . a configuration (installed on a handful of sites at most)
 Mark>   of one server implementation,

 Mark>   . break until changed every client based upon the c-client
 Mark>   library (most notably Pine -- we're talking about at least
 Mark>   tens of thousands if not hundreds of thousands of separate
 Mark>   implementations -- as well as break the fixed version of
 Mark>   these clients against Diablo servers.

You don't seem to understand that this is _already broken_. None of
the MODE READER-related stuff in the spec is anything more than trying
to document _current practice_ in such a way as to get interoperability.
Trying to do STARTTLS or AUTHINFO before doing MODE READER (assuming a
reader client) does not interoperate with existing software in certain
known (but rare) cases and there is no workaround.

(Proper handling of 480 responses allows a client to cope
transparently with the INN vs. Diablo differences. Even if we mandate
the order, we should probably point this out so that new clients can
handle old noncompliant servers.)

 Mark> If it helps matters, I would agree to a separate
 Mark> AUTHINFO/STARTLS command for clients vs. peers in return for
 Mark> the deprecation of MODE READER (replacing it with a no-op and
 Mark> servers MUST NOT require).

You still don't seem to be getting the point here. If MODE READER could
be disposed of that easily, we'd have done it long ago. Nobody likes it;
it's a really nasty wart on the protocol.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 19:15:52 2004

> advertises NNTPv3 is not OK to an NNTPv2 client.

Okay.  Ken's statement was that if NNTPv3 adds to but retains all of
NNTPv2's functionality, the server need only advertise NNTPv3.  Your
statement above implies to me that this is not correct and it does need to
continue to advertise NNTPv2.

> extensions exclusively.  The fact that authentication is offered is
> enough of a hint to anonymous clients that more can be gained through
> authentication.

Hm.  Yeah, maybe so.  Certainly, the 200/201 distinction in the opening
banner is pretty much a disaster in practice and a POST capability that
changes after authentication is a heck of a lot simpler and more accurate.

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 19:33:22 2004

Jeffrey M Vinocur <[email-masked]> writes:

> Oof, these extension names (and NOREADER, NOTRANSIT) are kinda making my
> head spin.  I think we should try to stick to non-negated names where
> possible.  Even something like:

> would be less confusing, I think?

I'd rather leave off the YES/NO part if possible.  What about something
like:

    TRANSIT

and then after MODE READER something like:

    READER
    POST
    NEWNEWS

That seems like it would work to me.

---

From: rra at stanford.edu (Russ Allbery)
Date: Fri Nov 5 19:37:16 2004

> authentication...perhaps we should try to tackle this in some uniform

I think we may be able to just assume that if AUTHINFO is advertised,
clients know that authenticating may get them more commands.  Likewise
with STARTTLS if a server wants to require TLS before allowing something.

MODE READER is a special case since most servers don't require it, so it's
worthwhile to say that it's needed if it is.  But for the rest, just not
advertising NEWNEWS and POST and advertising AUTHINFO seems okay to me.

This does mean that we need to be clear that a server is allowed to return
480 to a command that isn't advertised, to deal with existing clients that
don't understand the whole capability system.

> immediately using the alternative mechanisms.

Yeah, this does concern me as well, but I wonder how many clients really
do reactive authentication and how useful reactive authentication is.

Although we should probably be clear that if we go down the route of
advertising capabilities only when they're available given the current
authentication and privacy state, we're essentially abandoning any attempt
to support reactive authentication (since clients won't try the commands
if they're not advertised).

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 19:54:04 2004

> continue to advertise NNTPv2.

Mark is correct.  I was trying to multi-task when I wrote that and had a 
brain fart.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 20:13:36 2004

> My impression is that most inn servers will *not* be broken if we choose:

Would Pine gracefully handle "401 MODE-READER" in response to STARTTLS 
or AUTHINFO?  If so, then we already have a simple way out of this.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 20:19:38 2004

> more general case going forward.

Hmm, there *may* be a more elegant way of doing this (I'd have to ponder 
it some more to see if we could transfer the current security layer 
state between processes), but OTH if nntpd negotiated a TLS layer and/or 
SASL security layer before MODE READER, nntpd could stick around and 
proxy the traffic from the client to nnrpd via stdio or a UNIX socket. 
Its brute force, but it should work.

I might need another beer to analyze the performance ramifications of this.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 20:28:18 2004

On Fri, 5 Nov 2004, Ken Murchison wrote:
> Would Pine gracefully handle "401 MODE-READER" in response to STARTTLS or 
> AUTHINFO?  If so, then we already have a simple way out of this.

No.  It does the startup sequence in a particular order, based upon 
empirical evidence that some servers would not accept MODE READER unless 
you did AUTHINFO first.

-- Mark --

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Sat Nov 6 01:32:11 2004

On Nov 5, 2004, at 11:19 PM, Ken Murchison wrote:

> Hmm, there *may* be a more elegant way of doing this (I'd have to 
> ponder it some more to see if we could transfer the current security 
> layer state between processes), but OTH if nntpd negotiated a TLS 
> layer and/or SASL security layer before MODE READER, nntpd could stick 
> around and proxy the traffic from the client to nnrpd via stdio or a 
> UNIX socket. Its brute force, but it should work.

Unfortunately, innd cannot be capable of those negotiations without 
being able to parse readers.conf (in order to find out what 
authentication methods are permitted for a given connection, and what 
user databases should be consulted to determine success/failure).  I 
can't even imagine trying to fold all of that code into innd -- a rough 
estimate is that about a third to half of all of nnrpd would be 
involved.  At that point, we may as well ditch MODE READER entirely, I 
think.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:21:43 2004

In message <[email-masked]>, Russ Allbery
<[email-masked]> writes
>fear and hate optional commands.  :)

I shall Make It So.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:28:06 2004

In message <[email-masked]>, Ken Murchison <[email-masked]>
writes
>Of course, if we don't allow '.' in arguments, we could use '-' instead.

We allow anything (except space) in arguments; we allow alphanumerics,
dot, and dash in command names and extension labels.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:31:20 2004

In message <[email-masked]>, Ken Murchison <[email-masked]> writes
>2.1 is upwards compatible?  Just because of the version number?

Yes.

>Whether it seems obvious or not given your choice of syntax, it still
>MUST be explicitly document somewhere that 2.1 retains 2.0
>functionality (AFAIC 2.0 and 2.1 are still meaningless tokens at a
>protocol level).

No. That's the difference between A and C. If we went down path C, the
base specification would explicitly state that x.1 is upwards compatible
with x.0 and so on. The token is *structured*, unlike A and B.

In other words, the process is *reversed*; the author of RFC 4321 picks
a version number of 2.1 *because* it is upwards compatible, while the
author of 5555 picks 3.0 *because* it is not.

>The same would be true in the A case, RFC 4321 MUST explicitly state
>that it retains RFC 3977 functionality.  In either case, the client
>knows this because the author read the specs and programmed accordingly.

Not so.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:39:54 2004

Yes.

Not so.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:39:59 2004

I shall Make It So.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:40:00 2004

In message <[email-masked]>, Russ Allbery
<[email-masked]> writes
>because they were in RFC 977, I don't think that's a very good reason;
[Ken says they weren't]

Don't look at me: I thought you lot had sorted all this out before I
came on the scene.

>(Actually, I'd be quite happy if we dropped LIST DISTRIBUTIONS entirely,

Enthusiastically supported.

Taking this and Ken's message, here's what I'll do if nobody screams:

* Make LIST NEWSGROUPS mandatory.
* Create a new standard extension called LIST.
* All commands in this extension have two-part keywords; the only
  restriction is that they can't be the same as the mandatory LIST
  commands, obviously.
* An implementation can support any number of such keywords but:
  - they are only for returning information;
  - no LIST command can change the server state;
  - they all use the same, specified, set of response codes.
* LIST EXTENSIONS lists all of the supported keywords.
* We document meanings for ACTIVE.TIMES and DISTRIB.PATS.

[If you don't understand all these points, wait for some draft text.]

>this would be the simplest approach.

Indeed.

---

From: binaryboy at binaryboy.com (Howard Swinehart)
Date: Sun Nov 7 06:49:40 2004

From: "Mark Crispin" <[email-masked]>
> advertises NNTPv3 is not OK to an NNTPv2 client.

This returns us to the original problem of identifying broken servers.

NNTPV2 was supposed to distinguish between Netscape Collabra (has LIST 
EXTENSIONS but wrongly advertises OVER) and a truly compliant server. 
Without NNTPV2 and without a version number, a client based on this proposal 
won't see a difference between a future incompatible NNTPVx server and 
Netscape Collabra because neither will return NNTPV2.

With the version number proposal, we can do this:

if version is 2
    it's compatible
if version is larger than 2
    it's incompatible
if LIST EXTENSIONS works but no NNTP version
    it's Netscape Collabra

---

From: binaryboy at binaryboy.com (Howard Swinehart)
Date: Sun Nov 7 07:27:18 2004

From: "Clive D.W. Feather" <[email-masked]>
> alternative because people choked on the major.minor idea previously.

I agree with approach C (NNTP major.minor).

---

From: rra at stanford.edu (Russ Allbery)
Date: Sun Nov 7 09:20:01 2004

> EXTENSIONS but wrongly advertises OVER) and a truly compliant
> server. Without NNTPV2 and without a version number, a client based on
> this proposal won't see a difference between a future incompatible
> NNTPVx server and Netscape Collabra because neither will return NNTPV2.

I think that's actually the correct behavior.  That future server is
speaking a protocol that the client doesn't understand, which is, if one
thinks about it in the right direction, exactly the same as the case for
Netscape Collabra.  The client can give up, or fall back on RFC 977 and
try using commands at random and hope, which is pretty much the case for
Collabra today.

---

From: binaryboy at binaryboy.com (Howard Swinehart)
Date: Sun Nov 7 10:24:48 2004

From: "Russ Allbery" <[email-masked]>
> try using commands at random and hope, which is pretty much the case for

The difference is that a client author can potentially experiment and create
a workaround for non-compliant servers that exist today while the future
protocol is completely unknown.  A client might want to give up immediately
when it sees a backwards incompatible version number.

---

From: rra at stanford.edu (Russ Allbery)
Date: Sun Nov 7 14:43:46 2004

Howard Swinehart <[email-masked]> writes:

> The difference is that a client author can potentially experiment and
> create a workaround for non-compliant servers that exist today while the
> future protocol is completely unknown.  A client might want to give up
> immediately when it sees a backwards incompatible version number.

Well, I'm dubious that anyone would want to produce a version of NNTP that
was that completely incompatible so that the same sort of fallback that
you'd use for Netscape Collabra wouldn't also work.

I'm not completely opposed to the idea, but I'm very suspicious of it
because other protocols haven't needed it.  If I had my druthers, we'd
just copy IMAP and not worry too much about it.

---

From: ken at oceana.com (Ken Murchison)
Date: Sun Nov 7 15:12:50 2004

> just copy IMAP and not worry too much about it.

Right.  I'm old, but not old enough to have taken part in the IMAP 
discussions, but my guess is that all of these arguments have been made 
before and have ended up being inconsequential, otherwise we'd see such 
version info in IMAP or POP or SMTP.

I've finally come to realize that NNTP is different from a sheer 
load/bandwidth standpoint, but from a protocol standpoint, its really no 
different from the other messaging protocols (other than the fact that 
transport and retrieval have been intermingled, which is really poor 
design).

---

From: binaryboy at binaryboy.com (Howard Swinehart)
Date: Sun Nov 7 15:50:42 2004

> just copy IMAP and not worry too much about it.

How about if we use the NNTPVx tokens exactly as you described with one
additional note that tokens beginning with the 5 letters "NNTPV" are
reserved for indicating the protocol version.  This way clients can reliably
detect a NNTPV2 or higher protocol by checking for NNTPV*, they can detect
current broken servers by the lack of NNTPV* and they can detect future
incompatible versions by the lack of NNTPV2.  RFC977 servers are detected by
an error on LIST EXTENSIONS as before.

---

From: rra at stanford.edu (Russ Allbery)
Date: Sun Nov 7 20:30:04 2004

Howard Swinehart <[email-masked]> writes:

> reserved for indicating the protocol version.  This way clients can
> reliably detect a NNTPV2 or higher protocol by checking for NNTPV*, they
> can detect current broken servers by the lack of NNTPV* and they can
> detect future incompatible versions by the lack of NNTPV2.  RFC977
> servers are detected by an error on LIST EXTENSIONS as before.

I don't have a problem with that; that seems reasonable to me.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Mon Nov 8 09:42:44 2004

[Splitting off a sub-thread.]

> was going to be controversial!

You weren't going to get an argument from me.

> and we know that it works in practice,

Can you summarize?

> supports because I don't expect extensions to have versions.

Okay, I can live with that.

> deprecated.

I can live with that as well.

> specificity than is offered by a version number.

True.

> going the route that lets clients compare numbers.

There's two or three points that are getting conflated here.

Firstly, given that we're hacking this into LIST EXTENSIONS rather than
working through a capability system from scratch, I'd much much rather
that everythign about the core protocol was in one entry. I see Mark has
made comments about parsing and syntactic sugar, but I think he's wrong
here.

So what I'm suggesting is that there is a line with pseudo-extension-
label "_NNTP_" (or "*", or anything that couldn't be a real extension)
that represents "the core protocol". Then the arguments to that label give
the various properties we're documenting. I'd further make it simple by
saying:
- arguments beginning with a digit represent the supported version(s);
- arguments beginning with a letter indicate support for options.

> Again, I'd rather

As I've already said, there are three ways to do this.

(A) Version numbers are magic cookies. Server advertises its own version
and the client has to know which versions it can work with.
(B) Version numbers are magic cookies. Server advertises all the
versions it supports and the client is happy if any of them appear.
(C) Version numbers are structured. Server advertises its own version
and client deduces whether it can work with that.

If version 3 is upwards compatible with version 2, then the server
information line for a v3 server is:

  (A)      "_NNTP_ 3"          Old client needs to learn about 3.
  (B)      "_NNTP_ 2 3"        Old client sees the 3.
  (C)      "_NNTP_ 2.1"        Old client knows it supports 2.N
                               for all N (since it was written for 2.0).

I personally think C is better than B and I think that A is really
horrible in our problem domain (many client implementations, few server
implementations).

> is a good idea for the reasons that Ken stated.

Okay; it seemed a good idea at the time.

I'm proposing numbers, rather than "V2", because they're less likely to
be used as extension or capability labels and so we can reserve the
entire namespace right now. Using "V2" means we have more risk of
running into another use of that name-format (e.g. "V21" is or was an
ISP in the UK). It's not a huge deal, but on the other hand versioning
*is* something I've had experience of in the past and this seems within
my editorial purview.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Mon Nov 8 10:00:58 2004

I strongly suggest that before even contemplating a version number 
mechanism such as Clive proposes, that you look into the experience of 
other protocols.  In particular:

. Why is the MIME version number forever fixed at 1.0?

. Why is the MIME version number forever fixed at 4rev1?  [Hint: read the
   next question.]

. Why do the IMAP people shudder whenever they refer to the transition
   from 4 to 4rev1?  Does much IMAP client software actually work with a 4
   server?

. Why do HTTP people shudder when asked about HTTP versioning?

In my experience, a version number facility does nothing except to add 
substantial complexity and open the door to mischief.

What we need is One True definition of NNTP, an indicator of compliance 
with this One True definition, and perhaps a slight crack left open for 
extensions.  We need to *abolish* the myriad non-interoperable variants 
that currently exist, not bless them.  We need to *simplify* the requires 
on clients, not make them worse.

NNTP is a dinky stupid little protocol.  If it was a car, it would be a 
Yugo.  It does not need Ferrari gauges.  We can try to make this Yugo be 
reliable (unlike the car...) and easy to implement.  We're not going to 
accomplish this with complex versioning.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Nov 8 10:05:02 2004

> . Why is the MIME version number forever fixed at 4rev1?  [Hint: read the

Just to clarify, I think Mark meant IMAP instead of MIME.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Nov 8 14:39:11 2004

> authentication methods may be different.

Fine, then; have a different command for client vs. peer authentication.

> matters (INN, transfer access _and_ authentication).

In which case, we should stop worrying about being compatible with this 
and do what is right.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Nov 8 15:13:55 2004

On Sat, 6 Nov 2004, Andrew - Supernews wrote:
> You don't seem to understand that this is _already broken_.

It works for all of our users, unlike doing MODE READER first.

> known (but rare) cases and there is no workaround.

Rare, or non-existant?  We have no reports of such a server existing in 
reality in the wild.

> it's a really nasty wart on the protocol.

I didn't say that it was easy to get rid of MODE READER; just that it was 
the right thing to do.

Everything that I have heard indicates that this is ultimately a matter of 
implementation elegance in inn rather than an insurmountable barrier.  It 
isn't a "doing that will make my server three orders of magnitude slower" 
issue.

I don't mean to belittle the work that Russ will have to do (in fact, I'll 
buy him a six-pack of his favorite brew as recompense).  I'm just claiming 
that the greater good is to move forward in the NNTPv2 world without MODE 
READER.

Yes, for the time being, clients will have to worry about MODE READER. 
But we can fix things today so that clients 5 years from now won't have 
this burden.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Nov 8 15:34:08 2004

Mark Crispin <[email-masked]> writes:

> I didn't say that it was easy to get rid of MODE READER; just that it
> was the right thing to do.

What would you think of documenting port 433 for transit connections in
the standard and saying that transit feeds SHOULD use port 433 instead of
port 119?  That to me seems like the most viable way of eliminating MODE
READER.

> Everything that I have heard indicates that this is ultimately a matter
> of implementation elegance in inn rather than an insurmountable barrier.
> It isn't a "doing that will make my server three orders of magnitude
> slower" issue.

Using separate ports is something that would be fairly straightforward,
since that's mostly just a change in installation documentation without a
lot of code change.  Some people are already using port 433, and of course
people wouldn't have to change right away.  (We may want to add some
functionality to innd to allow it to listen on both ports, as a transition
aid.)

Redesigning INN so that it can handle reader and transit connections with
the same daemon on the same port is something that would essentially
require rewriting INN completely.  This isn't just an incidental design
decision -- this was one of the core portions of Rich Salz's original
design, and it would probably be easier to write a completely new server
from scratch than to eliminate INN's split innd/nnrpd structure.

Not that that would necessarily be a horrible idea, given that INN is a
pretty old code base, and I've considered it from time to time myself, but
as Andrew points out, splitting transit and reader functions is still a
very attractive design from a server architecture perspective.  I do think
there are arguments to be made in favor of allowing that design even in
the absence of any consideration of what INN does.

Basically, you're talking about a design change on the order of merging an
MTA and an IMAP server into a single daemon.  (In retrospect, having the
transit and reader components of NNTP be the same protocol was a huge
mistake.)

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Nov 8 15:50:37 2004

> port 119?  That to me seems like the most viable way of eliminating MODE

I think that this is an *excellent* solution.

> require rewriting INN completely.

Well...there *are* shortcuts that allow this to be accomplished, as long 
as you don't care about the cosmetics of the result... :-)

> MTA and an IMAP server into a single daemon.

Believe me, many people have tried to do just that to IMAP.

It can be argued that client NNTP is effectively a merge of POP3 and SMTP 
because of the POST command in NNTP.

> (In retrospect, having the

You'll hear no dispute from me on that point!!

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Nov 8 16:10:17 2004

>> the standard and saying that transit feeds SHOULD use port 433 instead
>> of port 119?  That to me seems like the most viable way of eliminating
>> MODE READER.

> I think that this is an *excellent* solution.

Okay, there was already some consensus in the working group for talking
about 433, so I think this is really our best route forward.  Andrew, what
do you think about taking this approach?

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Tue Nov 9 03:24:19 2004

>. Why is the MIME version number forever fixed at 1.0?

Because they made no syntactic provision for a header of the form

MIME-Version: 1.0,2.0

which would indicate that clients of both flavours should be able to
handle the message.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Tue Nov 9 03:24:22 2004

>> matters (INN, transfer access _and_ authentication).

When I first started to use NNTP (some years back, now), I used Stan
Barber's model implementation which followed RFC 997 closely and knew
nothing of "MODE READER".

My first attempts to read news from a server failed because I had not
given MODE READER. It was some early version of INN, and came up in innd
mode even though the IP address I was coming from had no permission to use
IHAVE.

The server I currently upload to comes up in innd mode (because it
recognizes my IP and knows it has permission to use IHAVE). It I wanted to
read from it, I would have to give MODE READER first (I don't, because I
actually read news from yet another server).

So yes, there have been, and still are, servers where MODE READER is
required. I grant you that neither of the cases I mentioned require
AUTHINFO, because they rely(ied) on recognizing my IP. But I see no reason
why they should not choose to use AUTHINFO (I think the capability is
there in INN if they choose to use it).

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:31:53 2004

Russ Allbery said:
> at all.

Um, I hate to sound sour, but that was one of my points when I proposed
the STATUS command. Yet everyone said this feature was unnecessary.

In particular, I asked Mark:
| So you don't want a "this could be available", only "this is available
| right now"?
and he confirmed that this was his view.

So colour me confused.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:34:08 2004

Howard Swinehart said:
>> because other protocols haven't needed it.

Yet.

My experience in other areas (X, C, POSIX) is that eventually the need
comes back and bites you.

> incompatible versions by the lack of NNTPV2.

Except that I'm writing it differently, this is what I'm proposing (my
"approach B") and what Russ seemed to agree with in a previous message.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:39:36 2004

> Done right, clients should be able to care only about the first digit of 

Excuse me? Do you really mean this as a generic comment?

If so, what is the point of having these detailed codes? I can understand
the approach of "use the first digit if you don't recognise the code", but
we've explicitly forbidden unknown responses to existing commands.

Anyway, what should you do with a 377 response? Or a 123? If you get 299,
is there following dot-stuffed text or not? Etc.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:42:47 2004

> Fine, then; have a different command for client vs. peer authentication.

I still don't see how this helps, because the server throws away all state
on receipt of a (needed) MODE READER. This includes the authentication ID
(and encryption keys).

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:54:19 2004

> substantial complexity and open the door to mischief.

That's not been my experience with (e.g.) C.

> extensions.

So HTTP should have stuck with the One True definition of HTTP/1.0? Never
mind that nobody thought about virtual servers until too late.

I'm stunned by the hubris that says "we've got it completely and absolutely
right and will never need to change things".

And I'm not asking for complex versioning. Just a version number scheme
that allows for change when it turns out we need it.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 03:57:51 2004

Okay, this is going to be a rather portmanteau answer to the long LIST
EXTENSIONS thread sitting in my inbox.

Russ says:
> Andrew and Mark had about avoiding round trips.

I've posted separately on this.

> At the least, we need
> to advertise whether MODE READER is required.

And this.

Right.

> whatever we want with it) include LIST EXTENSIONS output in its response.

This is a specific example of my LEAN suggestion.

> think this is the best approach to take.

Right. Except that it would be "_NNTP_ 2 3".

[3977]
>> Russ? Can you have a go at this?

Thanks.

> Yes, but I think it's the right thing to do.

Done.

> particular client?  (POST and NEWNEWS are the ones most frequently not

We've been through this before. NEWNEWS was mandatory in 977 and we agreed
that there was no reason to change it.

We indicate POST through the initial greeting (and MODE READER). However,
authentication could change this, so I'd be in favour of a POST argument to
the _NNTP_ extension label.

Ken:
> The only question that I have is whether these                   
> extension labels?

They're arguments. The model I have is a single "core capabilities" line in
the LIST EXTENSIONS output.

> The only issue would be        

Um, just out of interest, doesn't this mean that the argument of a 283
response is insecure as well?

Can I make a proposal here? It might sound odd, but bear with me.

*IF* an SASL security layer is negotiated, the server replies with two
responses:
* a 180 response meaning "security layer negotiated"
* the 281 or 283 response as normal
with the security layer coming into effect at the end of the 180, *not* the
28x.

So, using {} to indicate a secure connection:

    [C] AUTHINFO SASL WHATEVER
    [S] 383 base64+stuff
    [C] base64+stuff/finishing+the+negotiation+of+a+security+layer
    [S] 180 security layer starting after this line
    {S} 283 base64/additional+data

You can then send the capabilities in a secure way, either Russ's way:

    [C] AUTHINFO SASL WHATEVER
    [S] 383 base64+stuff
    [C] base64+stuff/finishing+the+negotiation+of+a+security+layer
    [S] 180 security layer starting after this line
    {S} 283 base64/additional+data LIST EXTENSIONS data follows
    {S} _NNTP_ 2
    {S} STREAMING
    {S} .

or in accordance with my LEAN proposal:

    [C] LIST EXTENSIONS AUTO
    [S] 101 auto mode activated
    [S] _NNTP_ 2
    [S] AUTHINFO SASL:WHATEVER
    [S] .
    [S] 203 dummy response
    [C] AUTHINFO SASL WHATEVER
    [S] 383 base64+stuff
    [C] base64+stuff/finishing+the+negotiation+of+a+security+layer
    [S] 180 security layer starting after this line
    {S} 101 Updated LIST EXTENSIONS data follows
    {S} _NNTP_ 2
    {S} STREAMING
    {S} .
    {S} 283 base64/additional+data

Incidentally, I know that TLS puts security into effect before NNTP
restarts and that we've said that the *client* gets to go next, but I
wonder if we should put in a 180 response here to provide a hook for server
capabilities, however done:

    [C] STARTTLS
    [S] 382 negotiating
    [TLS layer is negotiated]
    {S} 180 TLS successful

or with LEAN:

    [C] LIST EXTENSIONS AUTO
    [S] 101 auto mode activated
    [S] _NNTP_ 2
    [S] STARTTLS
    [S] .
    [S] 203 dummy response
    [C] STARTTLS
    [S] 382 negotiating
    [TLS layer is negotiated]
    {S} 101 Updated LIST EXTENSIONS data follows
    {S} _NNTP_ 2
    {S} AUTHINFO USER SASL:PLAIN
    {S} .
    {S} 180 TLS successful

I note that Mark writes:
> (under TLS)
so I guess he agrees.

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 06:46:28 2004

> response is insecure as well?

Yes.  Its designed this way because the client may not yet know what 
security layer(s) the server has selected.  Just like TLS, the security 
layer can't take effect until negotiation has completed.

Typically nothing in the server's final challenge needs to be protected 
(at least nothing in the mechs that I'm familiar with).

> with the security layer coming into effect at the end of the 180, *not* the

We'd be straying from the beaten path (IMAP, POP, SMTP) with this 
proposal which is something that I'd personally like to avoid.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 06:51:28 2004

> layer can't take effect until negotiation has completed.

But that's nothing to do with the content of the 283, is it? It's simply
that the client needs to be told that negotiation is complete. So if we did
split it as I suggest, the 180 would give that message.

[...]
> proposal which is something that I'd personally like to avoid.

Though Mark says that this is the way he wished he'd done it in IMAP.
Are we never allowed to learn from our predecessors' mistakes?

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 07:10:30 2004

Clive D.W. Feather wrote:

> But that's nothing to do with the content of the 283, is it?

It may.  The last server "challenge" is more than a "negotiation 
complete" message.  Its typically used for the server to authenticate 
itself to the client, but *could* be used to say "here's the cipher/MAC 
that I've agreed to use for the rest of the session".

> Are we never allowed to learn from our predecessors' mistakes?

Actually, now that I think about it none of IMAP, POP or SMTP support 
"success data" in the response.  Only newer SASL-enabled protocols 
(LDAP, BEEP?) support this in their SASL profiles.  Nonetheless, the 
SASL spec states:

"The security layer takes effect immediately following the last response 
of the authentication exchange for data sent by the client and the 
completion indication for data sent by the server."

I guess the term "completion indication" could be somewhat misleading, 
but I *think* this is pretty clear that the "success data" response 
should NOT be encrypted in any way.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 07:16:36 2004

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 Russ> What would you think of documenting port 433 for transit
 Russ> connections in the standard and saying that transit feeds
 Russ> SHOULD use port 433 instead of port 119?  That to me seems like
 Russ> the most viable way of eliminating MODE READER.

The problem with this is that it requires changing the behaviour of a
lot of feeder sites. This solution has been around for many years, and
people have been suggesting it should be the standard for many years,
but it's just not getting any traction (and putting it in the spec
won't change that).

We have one (1) peer out of hundreds that's using port 433. Many sites
don't bother to support alternate ports in their own configuration or
administrative scripts, making it harder to get peers if you restrict
yourself to ports other than 119.

 Russ> Redesigning INN so that it can handle reader and transit
 Russ> connections with the same daemon on the same port is something
 Russ> that would essentially require rewriting INN completely.  This
 Russ> isn't just an incidental design decision -- this was one of the
 Russ> core portions of Rich Salz's original design, and it would
 Russ> probably be easier to write a completely new server from
 Russ> scratch than to eliminate INN's split innd/nnrpd structure.

It's also worth noting that other servers' solutions to this either
aren't applicable to INN or aren't even solutions at all. Diablo, for
example, requires that the transit and reader functionality is in
completely separate processes - running both on a single machine
involves considerable contortions to happen at all and doing both on
the same port, INN-style, is simply not possible. Typhoon solves the
problem by using threads. For DNews I don't know the internal details,
but neither do I know anyone getting decent results from running it
on a large feed.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 09:00:18 2004

Yes.

> If so, what is the point of having these detailed codes?

Documentation only.

Checking for explicit codes has a long history of being problematic in 
SMTP and FTP.  NNTP has a dismal history as well (need I remind you of 380 
vs 480)?

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 10:12:26 2004

Mark Crispin <[email-masked]> writes:

> SMTP and FTP.  NNTP has a dismal history as well (need I remind you of
> 380 vs 480)?

Well, I don't know about the other folks on this list, but I have to admit
that I'm not feeling particularly guilty, nor think the NNTP community as
a whole should be blamed, for status codes that were never documented
anywhere, that no one else had ever heard of, and that have not been
enountered by anyone else that I've talked to other than you.

I don't know what gave you 380, but it was broken by any documentation the
NNTP community has ever produced.  Either it was broken as written, or
it's some highly obscure or experimental server from before AUTHINFO, or
something else strange was going on there.  I certainly wouldn't blame you
if you just gave up entirely on any server that returned a 380 status
code.  People write hideously broken software; I don't think the IMAP
community is responsible for, say, some of the things that Courier does.

That being said, it has been documented in NNTP for quite some time that
clients should respond to unknown codes based on the first digit, so
clearly this is indeed a design goal and status codes should be chosen to
enable that behavior.  I think we're doing fairly well so far with this.
Note that such things as reactive authentication or reactive TLS
negotiation aren't going to work with that paradigm, but I think we're
moving away from that towards LIST EXTENSIONS advertisements, so that
isn't really a problem.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 10:13:50 2004

>> The problem with not advertising extensions that the client is allowed
>> to use after authentication is that the client then has no idea whether
>> the extension is available but needs authentication or if it's just not
>> there at all.

> the STATUS command. Yet everyone said this feature was unnecessary.

I was reiterating your point to make sure that people continued to think
about it.  I'm mostly convinced that it's unnecessary to distinguish
between an extension that's not available at all and one that will become
available after authentication or TLS negotiation, and Mark certainly
seems to think that it's unnecessary, but I don't want us to forget that
we're making that decision and I want to give people plenty of opportunity
to object if that sounds wrong.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 10:35:42 2004

> on receipt of a (needed) MODE READER. This includes the authentication ID

If there is a different command for client vs. peer authentication, then 
there is no need for MODE READER.

Actually, if you take the position that RFC 977 is anything more than 
worthless, then MODE READER is not needed because it is not in RFC 977.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 10:52:15 2004

On Tue, 9 Nov 2004, Russ Allbery wrote:
> something else strange was going on there.

Support for 380 and 480 appeared in my code between April 14, 1994 and 
August 1, 1995; and apparently 380 came first.

I respectfully point out that RFC 2980 was not issued until October 2000. 
Therefore, it can not be claimed that 380 was "broken" prior to that time.

I agree that 480 makes much more sense than 380; I have no intention of 
defending 380.  On the other hand, the fact that 381 exists tends to 
suggest that, at least at some point in history, 380 existed.

Actually, 580 makes more sense than 480.  Ditto 481 and 482.  Someone 
evidentally misunderstands the meaning of 4xx response codes.  I've kept 
silent on this problem because NNTP is so broken in other respects that 
are more important to fix.  The world will continue to survive even if 
NNTP continues to misuse of 4xx.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 11:17:16 2004

> August 1, 1995; and apparently 380 came first.

I did some more hunting.  That hypothesis makes sense, given that the
reference NNTP implementation does have 380 in its nntp.h, but the
1.5.12.1 release doesn't use it anywhere, using only 480.  INN 1.0 used
only 480 in 1991, so presumably the change, if the guess is correct, was
made sometime before that.  If there was a change, it's not mentioned in
the changelog of the reference NNTP implementation, so it's hard to date.

That puts the date of a change somewhere between 1986, when RFC 977 was
published (which was right about the time that NNTP software was first
being deployed) and 1990, since INN has never had a mention of it at all.

My guess would be that you added the code for interoperability with a
fairly old C News site.  That would explain why I've never heard of it; at
the time it was still in use, I was using VMS and had never heard of the
Internet.  I don't expect there are many NNTP implementors who are still
writing code who remember the history.  Stan or Henry might remember.

> I respectfully point out that RFC 2980 was not issued until October
> 2000. Therefore, it can not be claimed that 380 was "broken" prior to
> that time.

Actually, RFC 2980 can't even declare things broken; it's not a standard.
RFC 2980 is just documentation.

In some sense, anything that anyone does with NNTP outside of the bounds
of RFC 977 can't be declared "wrong"; in practice, I'm pretty comfortable
calling things wrong if they don't behave like the reference NNTP
implementation and INN, until we get a new standard released.  That's the
best that we've had for quite a while now (which is, of course, why this
working group).

> NNTP continues to misuse of 4xx.

4xx has never, from the very beginning of the NNTP protocol, meant the
same thing that it meant in other protocols.  So that one you can blame on
the original RFC 977 authors, but all subsequent status codes are just
staying consistent with RFC 977.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 11:30:31 2004

>> I still don't see how this helps, because the server throws away all
>> state on receipt of a (needed) MODE READER. This includes the
>> authentication ID (and encryption keys).

> there is no need for MODE READER.

I don't understand why this would be the case.  The need or lack thereof
of MODE READER has nothing to do with authentication.  You can make the
client authentication command do an implicit MODE READER, but that does
nothing for the (numerous) sites that don't use any authentication at all.

> worthless, then MODE READER is not needed because it is not in RFC 977.

I would take the position that RFC 977 is largely worthless, not because
of that, but because of quite a few other omissions and inconsistencies
with what news servers actually do.  Most of them are, like MODE READER,
ones that only bite you in rare circumstances, although there are
certainly a fair number of core commands missing entirely.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 13:37:41 2004

> nothing for the (numerous) sites that don't use any authentication at all.

If the site does not use authentication, then the mode switch can be made 
on the first IHAVE command (to set peer mode) or the first command other 
than HEAD, STAT, LIST ACTIVE, or LIST EXTENSIONS (to set client mode).

Or we can simply define a no-credentials authentication method (ala SASL 
anonymous) that sets peer mode and declare client mode to otherwise be the 
default.

> certainly a fair number of core commands missing entirely.

I agree; which proves the point that "it's not compatible with RFC 977" is 
a silly argument.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 13:47:21 2004

On Tue, 9 Nov 2004, Russ Allbery wrote:
> Internet.

The term "C News" sure sounds familiar.

For what it is worth, I have been on the Internet (or its ARPAnet 
predecessor) for 30 years.

> staying consistent with RFC 977.

As I said, NNTP is broken in many respects.  It is a particularly bad form 
of breakage to copy, but misuse, a protocol architecture from other 
protocols (e.g. SMTP, FTP).  NNTP's misuse of 4xx seems to have been 
unintentional rather than open defiance of the architecture.

Nevertheless, I don't think that it's worth trying to fix this.  There are 
other matters which are more important to fix.  It's much higher priority 
from my perspective to get MODE READER flushed.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 14:01:43 2004

Mark Crispin <[email-masked]> writes:

> If the site does not use authentication, then the mode switch can be
> made on the first IHAVE command (to set peer mode) or the first command
> other than HEAD, STAT, LIST ACTIVE, or LIST EXTENSIONS (to set client
> mode).

That's not a bad idea and wouldn't break existing software that I'm aware
of.  I think it's even uglier than MODE READER from a protocol standpoint,
since it means that LIST EXTENSIONS may change as the result of issuing
quite a few different commands, but if you really think that's cleaner, we
can at least discuss it.

> anonymous) that sets peer mode and declare client mode to otherwise be
> the default.

This doesn't help since it's not backward compatible with existing
clients.  Whatever we come up with has to be backward compatible with
existing clients (see the working group charter).

I'll expand on that thought in a little bit as soon as I get out of some
meetings.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 14:33:38 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> If the site does not use authentication, then the mode switch
 Mark> can be made on the first IHAVE command (to set peer mode) or
 Mark> the first command other than HEAD, STAT, LIST ACTIVE, or LIST
 Mark> EXTENSIONS (to set client mode).

No, it can't. One word: pipelining.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 14:42:35 2004

> quite a few different commands, but if you really think that's cleaner, we

Well, I personally don't think that there should be a mode at all; either 
the protocol should be split into two different port services, or the 
modality should be abolished.

I'm just trying to come up with alternative strawman proposals for what 
could work for inn while relieving clients of the burden.

A third variant (as an inn implementation solution) is to have the 
heuristic to set the mode, but if a command in the other mode is given 
then you just run the other daemon in a subfork.  Since this should never 
happen in real life as long as the heuristic isn't broken, it's alright 
that this be inefficient.

> clients.

Why?  We're not changing clients, we're changing NNTP peers.  I think that 
we can claim this to be in-charter.

In this strawman, MODE READER becomes a noop.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 14:43:25 2004

> No, it can't. One word: pipelining.

Pipelining isn't relevant.  The mode switch is an internal concept in the 
server, and doesn't affect the client/server protocol at all.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 14:45:26 2004

>>> Or we can simply define a no-credentials authentication method
 >>> (ala SASL anonymous) that sets peer mode and declare client mode
 >>> to otherwise be the default.

 >> This doesn't help since it's not backward compatible with existing

 Mark> Why?  We're not changing clients, we're changing NNTP peers.  I
 Mark> think that we can claim this to be in-charter.

No. Peers are clients too (and if anything they are harder to change than
reader clients).

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 14:49:11 2004

> No. Peers are clients too (and if anything they are harder to change than

It only matters when there is peer authentication.  Just how many such 
peers are there.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 14:56:00 2004

>> No, it can't. One word: pipelining.

 Mark> Pipelining isn't relevant.  The mode switch is an internal
 Mark> concept in the server, and doesn't affect the client/server
 Mark> protocol at all.

Of course it's relevent. Notice that MODE READER is explicitly defined
as not being pipelinable?

If you want the switch to happen in response to a pipelinable command,
then you have to start worrying about preserving the pipeline over the
handoff (which in the transit->reader direction is a fork+exec of a new
program; there is no way back).

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 14:59:59 2004

> program; there is no way back).

That is a server implementation issue.  It is not something that should be 
inflicted on the client.

NNTP sure stands out as being a protocol of, by, and for servers and 
server developers.  Perhaps the NNTP community would be happier if there 
weren't any clients.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 16:30:29 2004

> Why?  We're not changing clients, we're changing NNTP peers.  I think
> that we can claim this to be in-charter.

That's not my reading of the charter.  Note particularly this part:

    The first concern of this working group shall be for the
    interoperability of the various NNTP implementations, and therefore
    for clear and explicit specification of the protocol. It is very
    important that we document the existing situation before taking up any
    new work.

I really don't think we should be doing anything that breaks existing
NNTP clients if we can possibly avoid it, which includes NNTP peers.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 16:45:34 2004

> NNTP clients if we can possibly avoid it, which includes NNTP peers.

I don't see how that is inconsistant with creating a "compatibility with 
pre-NNTPv2 servers" document and then moving ahead.

As far as interoperability, I respectfully suggest that the current 
situation, in which MODE READER can respond with "can't do this until you 
AUTHINFO first", and AUTHINFO can respond with "can't do this until you 
MODE READER first" is not condusive to interoperability.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 16:47:41 2004

Mark Crispin <[email-masked]> writes:

> situation, in which MODE READER can respond with "can't do this until
> you AUTHINFO first", and AUTHINFO can respond with "can't do this until
> you MODE READER first" is not condusive to interoperability.

We've already reached a list consensus that 480 responses to MODE READER
will be explicitly banned by the standard.  You may not have gotten a copy
of that, though (I think you're getting about 75% of the list mail right
now but not all of it).

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 17:08:14 2004

> * We document meanings for ACTIVE.TIMES and DISTRIB.PATS.

I'm inclined to say that this sounds like the right thing to do.  It's a
little weird, but then the LIST command in general is rather weird.  The
only thing that worries me is that I wonder if we need an IANA registry or
something for different LIST types, and if so that looks like a very odd
thing to ask them to maintain.

The alternative, I suppose, is to define each additional list command as a
separate extension and advertise it as such (LIST-ACTIVE.TIMES,
LIST-DISTRIB.PATS, etc.).  That's a little more consistent, but feels even
uglier and too verbose to me.

I'm not really convinced one way or the other, though.

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 17:20:32 2004

Mark Crispin wrote:
> Well, I personally don't think that there should be a mode at all; 
> either the protocol should be split into two different port services, or 
> the modality should be abolished.
> 

Correct me if I'm wrong in my logic.

The problem that we're trying to solve at the moment is that at least 
one reader client (Pine and c-client derivatives) does STARTTLS/AUTHINFO 
before MODE READER.  This works in all cases except those in which MODE 
READER is significant (certain INN configs for one).

IMO, the cat is already out of the bag on this one.  We have deployed 
client code which will not work with a presumably small set of deployed 
servers.  We can only hope to fix this going forward.  We have two 
choices: either fix the client or fix the server.

If we fix the server, we have to remove the significance MODE READER so 
it can either be abolished or be used in any order with 
AUTHINFO/STARTTLS.  This will work, but given that INN was designed 
around the use of MODE READER, changing this willmost likely result in a 
non-trivial development effort.

If we simply change the client to use MODE READER before 
STARTTLS/AUTHINFO (which was its intended, although undocumented, 
intent), we solve the problem with a trivial fix and AFAIK don't 
introduce any new compatibilties with deployed servers.

I'm sure I'm going to get some heat from Mark on this, but I thought I'd 
throw it out there.  Please set me straight on why this solution is not 
as simple as it appears.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 17:22:12 2004

Clive D W Feather <[email-masked]> writes:

> We've been through this before. NEWNEWS was mandatory in 977 and we
> agreed that there was no reason to change it.

Good point.  Maybe there's no reason to allow servers to disable it any
more (or at least to disable it frequently enough that it's worth special
note in LIST EXTENSIONS).

> We indicate POST through the initial greeting (and MODE
> READER). However, authentication could change this, so I'd be in favour
> of a POST argument to the _NNTP_ extension label.

The initial greeting has been a complete mess for years.  An extension
would be much more reliable and much more consistent.

> They're arguments. The model I have is a single "core capabilities" line
> in the LIST EXTENSIONS output.

This I'm not sure I understand the reason for.

Looking at this from the perspective of a client, I just want to know
what's available to me to use.  I don't really care about the distinction
between the core protocol and extensions; for all that I care, POST could
be considered an extension.  I just want to know whether I can use it or
not.

It's really easy to take a list of extensions and parse it.  It's a bit
more annoying when you have to special-case the _NNTP_ extension and parse
its arguments and use that to decide what commands are available, when for
OVER it's just a regular extension.  It seems like it would be a lot more
straightforward to just say that there's a POST capability that we
advertise if POST is accepted, there's an IHAVE capability that we
advertise if IHAVE is accepted, and there's a READER capability that we
advertise if reader commands are allowed.

In my mind, extension arguments are there to express some sort of
additional information about the extension other than just whether or not
it's available.  For AUTHINFO, this is what authentication mechanisms are
supported.  For OVER, this is whether OVER on a message ID is supported.
Whether POST, IHAVE, or reader commands are available doesn't feel to me
like the same sort of thing at all; in fact, that seems parallel to OVER
or HDR or the other extensions that are part of the list like any other.

The whole distinction of what's an extension and what isn't is
fundamentally arbitrary anyway -- the only real distinction is that
extensions are things that are optional to implement.  In that sense,
IHAVE is actually an extension, since you can write a standalone NNTP
server that doesn't take any external feeds without ever writing an IHAVE
implementation.  We wouldn't generally call that a fully-functional NNTP
server, but from a client perspective it looks just like any other one (it
just doesn't advertise the IHAVE capability).

> with the security layer coming into effect at the end of the 180, *not* the

I'm really leery of sending two responses in response to a command.  Other
NNTP commands don't do that; we'd be adding a new style of response (since
you're not sending the terminating .CRLF of a multiline response either).

After reading all this discussion, I'm becoming inclined to just give
further attempts to reduce the RTTs for an initial client connection a
pass for the time being, except possibly for putting something into the
200 response.  I know that in an ideal world STARTTLS or SASL should have
had some way of communicating this sort of information after completion,
but none of the other protocols do this (except IMAP, which has a
capability for out of order responses that we don't have).

We've been brainstorming, which is good, but we also need to limit our
sights a little so that we can get something published, because right now
the situation with NNTP is hideous and we need to at least get some sort
of standard out there for people to use now.  Most of these decisions
aren't final, and we can always revisit this and come up with some neat
solution later on provided we don't trap ourselves in a corner.

It sounds like providing extensions information as part of 281 or 283
doesn't work from a protocol standpoint because that response is not
protected, which means that there really isn't anything that we can say in
the existing documents (without changing NNTP in fundamental ways like
returning multiple responses) that helps or hurts for solving this problem
down the road.

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Tue Nov 9 17:56:39 2004

> Right. Except that it would be "_NNTP_ 2 3".

Is everyone in agreement that we need the special underscores here?  It 
strikes me as a strange and unnecessary distinction, or perhaps I just 
haven't thought about it hard enough.

On a related note, I don't think we need to cram everything under 
discussion this week into a single line of LIST EXTENSIONS; if we want 
to indicate specification revision and command sets available, can't we 
just make them two separate lines?

(Oh, and would VERSION be a better name?)

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 18:06:11 2004

On Tue, 9 Nov 2004, Russ Allbery wrote:
> of that, though

That's great, but it declares at least one existing deployed 
implementation (Diablo?) to be broken, right?

> (I think you're getting about 75% of the list mail right
> now but not all of it).

Probably.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:09:34 2004

>> We've already reached a list consensus that 480 responses to MODE
>> READER will be explicitly banned by the standard.  You may not have
>> gotten a copy of that, though

> implementation (Diablo?) to be broken, right?

Yes.  I can take this forward with the Diablo maintainers and get their
reactions on it.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 18:18:04 2004

On Tue, 9 Nov 2004, Ken Murchison wrote:
> If we simply change the client to use MODE READER before STARTTLS/AUTHINFO 
> (which was its intended, although undocumented, intent), we solve the problem 
> with a trivial fix and AFAIK don't introduce any new compatibilties with 
> deployed servers.

Actually it does introduce a problem.  We have at least one server 
(Diablo?) in which this will not work.

I used to do MODE READER first, and had to change my code due to user 
complaints that it didn't work.

More precisely, here is what my code does:

I do STARTTLS first.  [Sorry if you think that this is wrong.  The TLS 
architecture says that this is mandatory.]

Then, if the user configures authentication (by specifying /user=xxx in 
the NNTP specification), I do AUTHINFO.

Then, I do MODE READER, and if it responds 480 I do AUTHINFO.

I did not do this out of a desire to be gothic.  I did this because it was 
the only thing that worked with all the servers that real users 
encountered.  There are, indeed, users who could not use NNTP with Pine 
until they added the /user=xxx to force authentication before MODE READER.

It sounds to me as if you telling me that I should do MODE READER first, 
and tell my users who set /user=xxx to force AUTHINFO first "sorry, the 
IETF NNTPEXT working group says this is broken, tell your server admin to 
fix his server."

This ain't gonna happen.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:21:27 2004

Mark Crispin <[email-masked]> writes:

> I did not do this out of a desire to be gothic.  I did this because it
> was the only thing that worked with all the servers that real users

> IETF NNTPEXT working group says this is broken, tell your server admin
> to fix his server."

> This ain't gonna happen.

I think that, for your software, what makes the most sense is to just
document that Pine doesn't work with any server that requires MODE READER
unless that server implements the new LIST EXTENSIONS protocol that we're
hammering out now (at which point you can use that to decide whether you
have to do MODE READER).

The point, however, that several of us have been trying to make is that
you could currently simply remove MODE READER from Pine entirely and it
would be functionally equivalent to what you're doing right now for any
server that requires either TLS or AUTHINFO.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 18:23:38 2004

>> READER will be explicitly banned by the standard.  You may not
 >> have gotten a copy of that, though

 Mark> That's great, but it declares at least one existing deployed
 Mark> implementation (Diablo?) to be broken, right?

Yes.

The workaround for clients (recognising the 480 response to MODE
READER and applying the generic meaning of "authenticate and do that
again") should at least be mentioned as a backward compatibility note,
though, since all the other clients I've looked at so far seem to be
doing that.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 18:29:18 2004

> hammering out now (at which point you can use that to decide whether you

Except that Pine does work for such servers, now, in *real* installations 
instead of all the theoretical installations we are talking about.

> server that requires either TLS or AUTHINFO.

No, because there are servers which require MODE READER, and servers which 
require AUTHINFO before MODE READER, and Pine as it exists today works 
with these servers.

I think that, as a client implementor, I'm a bit more aware of what 
clients encounter in the field.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 18:32:42 2004

> haven't thought about it hard enough.

I'm not particularly fond of the '_'.  I'm also not fond of listing the 
version number(s) as argument(s) to the token.  I prefer what Russ used 
above: individual tokens for each revision.  I don't really care what 
the tokens are; call them "RUSS", "JEFF", "KEN" as far as I'm concerned. 
  They are just arbitrary tokens whose meaning is specified in the 
relevant documents (just like "STREAMING").

That being said, the driving force should be what is easier for clients 
to parse/use.  Its doesn't *really* matter how [un]attractive the 
mechanism looks because the protcol really isn't for human consumption 
anyways (other than us hackers).

Agreed.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 18:32:46 2004

> Mark> implementation (Diablo?) to be broken, right?

Well, then, we are all agreed that MODE READER is broken.

Therefore, it is pointless to argue that the charter prohibits action. 
The charter does not prohibit fixing things that are broken.

Otherwise, I can argue that requiring MODE READER before AUTHINFO breaks a 
widely deployed client (Pine), and therefore the WG is prohibited from 
doing that by the charter.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:40:40 2004

>> document that Pine doesn't work with any server that requires MODE
>> READER unless that server implements the new LIST EXTENSIONS protocol
>> that we're hammering out now (at which point you can use that to decide
>> whether you have to do MODE READER).

> Except that Pine does work for such servers, now, in *real*
> installations instead of all the theoretical installations we are
> talking about.

Pine in its current implemention as you described it does not work with
any INN installation that requires both AUTHINFO and MODE READER.

> No, because there are servers which require MODE READER, and servers
> which require AUTHINFO before MODE READER, and Pine as it exists today
> works with these servers.

No server which requires AUTHINFO before MODE READER actually requires
MODE READER at all; you don't need to send MODE READER to such a server.
(I can make this statement because I believe, with a fairly high degree of
certainty, that only INN actually requires MODE READER, and INN does not
require AUTHINFO before it.  I also know that Diablo, the only server
known to require AUTHINFO before MODE READER, doesn't require MODE READER
at all.)

> clients encounter in the field.

I do know how INN works, though.  :)

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 18:46:49 2004

> complaints that it didn't work.

Argh!!!  I didn't realize that we have two contradicting servers, one 
which requires MODE READER before AUTHINFO (INN) and one that requires 
AUTHINFO before MODE READER (Diablo).

Unless this can be worked out with some combination of existing failure 
responses (which I'm sure Mark has tried), we may just have to declare 
some implementation as broken and move on.  Otherwise maybe we are stuck 
trying to remove INN's dependency on MODE READER.

Does anyone know if the Diablo design has the same dependency on MODE 
READER as INN?  Andrew may have covered this, but I don't recall what he 
said.

Hmm, thinking on my feet...

What about this heuristic for INN?  AFAIK, most sites have a list of 
peers that they are willing to accept an NNTP feed from.  When innd 
accepts a connection, it checks the remote hostname/ip against its table 
and if it finds a match, it stays in transit mode.  If it doesn't find a 
match, it automatically switches to reader mode (nnrpd).  I would 
imagine that a feeding host wouldn't also have reading clients 
originating from it.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:51:35 2004

> READER as INN?  Andrew may have covered this, but I don't recall what he

Andrew said that Diablo does not require MODE READER.  Nothing about
Diablo's architecture should require it, since they don't support the
configuration that INN supports at all.

> imagine that a feeding host wouldn't also have reading clients

INN already does this.  This is the reason why so few people encounter
this problem; you have to have both transit and reader access from the
same host.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 19:11:03 2004

> this problem; you have to have both transit and reader access from the

This is why I think we should just abolish MODE READER and be done with 
it.  Retaining MODE READER defends a very small set of circumstances at 
the cost of inflicting amazing complexity on clients forever.

I also find it remarkable that people glibly talk about changing Pine, a 
widely-deployed client, in order to avoid changing what is probably a 
handful of peers.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 19:26:23 2004

> any INN installation that requires both AUTHINFO and MODE READER.

Please explain how.

If the user does not specify a /user=xxx specifier, it will not do 
AUTHINFO, and eventually it will hit a command that sends a 480 and Pine 
will AUTHINFO at that time.

The fact still remains that MODE READER itself is a broken, and that it is 
not permissible to require MODE READER before STARTTLS since TLS does a 
complete reversion to initial connection state.

That's right; STARTTLS has to undo MODE READER and restore the mode as 
unspecified.  If NNTP TLS does not say this, then it's in violation of 
TLS.

There's no escape.  MODE READER is broken.  The only way to keep that 
function is to split the service on separate ports; or, just give up on 
the function.  Either way, MODE READER can not be salvaged.

You can't say that MODE READER is required but not really required; that 
MODE READER is required before AUTHINFO but actually required after 
AUTHINFO if not really required.  But that's what you're trying to say if 
you attempt to retain the status quo.

TLS is just the straw that breaks the MODE READER camel's back.

If we talk about recommendations in a "compatibility with pre-NNTPv2 
servers" document, I'm all for talking about MODE READER in all its 
ugliness.  But let's free the NNTPv2 world from this abomination.  Let's 
save future clients from having to deal with it.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 19:45:28 2004

> will AUTHINFO at that time.

Ah, I didn't realize that you did also do reactive authentication.  (A
misconception of mine from a previous discussion.)  Yes, if you also do
reactive authentication, this implementation works in that mode.  If,
however, the user does specify /user=xxx to force authentication, that
authentication would fail in this situation since it would be done against
the transit server component.

If your client does have everything enabled and sends, as you described
earlier:

    STARTTLS
    AUTHINFO
    MODE READER

in that order to an INN server in a situation that actually requires MODE
READER, STARTTLS would fail, AUTHINFO would fail, and then MODE READER
would actually give you a reader connection.  I expect that this would
throw some sort of error to the user in Pine (and, in fact, I expect that
this configuration of Pine wouldn't proceed after STARTTLS failed).

That MODE READER that you're sending in the order above doesn't work
properly and *never* worked properly with INN unless neither of the
commands before it were actually sent.  (Or, to be more pedantically
correct, the two commands sent before it would not have worked properly,
but it amounts to the same thing.)

This is a great illustration of why we needed a real NNTP standard years
ago, since the implementation that you (quite reasonably) arrived at via
experimentation was, in fact, always broken and never would have worked
properly in the situation you were attempting to allow for.  (That
situation is just quite rare.)  It is indeed the fault of the NNTP
community and the INN developers for not documenting MODE READER clearly
enough so that both you and Diablo made different, apparently reasonable
mistakes in implementation (although Diablo's mistake is frankly less
reasonable than yours).

> The fact still remains that MODE READER itself is a broken, and that it
> is not permissible to require MODE READER before STARTTLS since TLS does
> a complete reversion to initial connection state.

Could you point me at where this is documented?  I'm aware that it
requires discarding all information about server capabilities and all
authentication credentials, but this is a stronger statement that I don't
recall seeing in the documents that I've read.

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 20:49:07 2004

> widely-deployed client, in order to avoid changing what is probably a 

If you're referring to me, I wasn't intentionally being glib at all.  I 
was simply trying to find the least painful solution which would yield 
the greatest amount of functionality amongst current and future deployed 
software.

Given the fact that Diablo does something completely different from INN 
and the number of INN deployments which *require* MODE READER would seem 
to be a lot smaller than I first imagined, I'm now in favor of 
eliminating MODE READER entirely (or at least making it a NOOP which has 
no dependency on other commands).

For those edge cases where a server needs to support both transit and 
reader functions from the same client we can provide port 433 as the 
alternative.

Is this where we are, or have I completely missed the boat (again)?

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Tue Nov 9 20:50:54 2004

>>>>> "Ken" == Ken Murchison <[email-masked]> writes:

 Ken> Argh!!!  I didn't realize that we have two contradicting
 Ken> servers, one which requires MODE READER before AUTHINFO (INN)
 Ken> and one that requires AUTHINFO before MODE READER (Diablo).

Diablo does not require MODE READER at all. But it will give a 480
response to it if authentication is required and the client hasn't
authenticated yet.

 Ken> Unless this can be worked out with some combination of existing
 Ken> failure responses (which I'm sure Mark has tried),

This algorithm works for slrn, knews, gnus at minimum, against all known
servers:

0) connect to server and get initial banner
1) send MODE READER
   - if response is 480, then:
      - execute authentication procedure
      - send MODE READER again
   (the handling of 480 there can be generic code used for all commands)
2) if the configuration specifies that we should authenticate preemptively,
   and we've not authenticated yet, then execute the authentication
   procedure now
3) proceed with reader commands

What this would end up being replaced with, after adding in all the
new stuff from the draft, is something along these lines:

0) connect to server and get initial banner
1) send LIST EXTENSIONS
2) if the extensions output indicated that MODE READER is required,
   _or_ this is an old (non NNTPv2 or whatever we call it) server,
   then:
       send MODE READER
         - if response is 480 (should only happen on an old server),
           then:
            - execute authentication procedure
            - send MODE READER again
         mark old LIST EXTENSIONS data invalid
3) execute authentication procedure unless we already did it
4) proceed with reader commands

The authentication procedure would be something like:

0) do LIST EXTENSIONS if we don't have valid data yet
1) if TLS is desired, or if the available auth schemes don't
   include a desired one, then do STARTTLS and get the
   LIST EXTENSIONS data again
2) if no suitable auth scheme available, fail
3) do AUTHINFO with the desired scheme

This should mean that when a new client is talking to a new server,
assuming the client wants both tls and auth, the sequence is:

 LIST EXTENSIONS
 STARTTLS
 LIST EXTENSIONS
 AUTHINFO
 LIST EXTENSIONS (if needed)
 ...normal commands...

 Ken> Does anyone know if the Diablo design has the same dependency on
 Ken> MODE READER as INN?  Andrew may have covered this, but I don't
 Ken> recall what he said.

You could always read my messages again.

 Ken> Hmm, thinking on my feet...

 Ken> What about this heuristic for INN?  AFAIK, most sites have a
 Ken> list of peers that they are willing to accept an NNTP feed from.
 Ken> When innd accepts a connection, it checks the remote hostname/ip
 Ken> against its table and if it finds a match, it stays in transit
 Ken> mode.  If it doesn't find a match, it automatically switches to
 Ken> reader mode (nnrpd).

This is exactly what INN always did.

 Ken> I would imagine that a feeding host wouldn't also have reading
 Ken> clients originating from it.

Well, that's the problem; your imagination isn't covering the
real-world cases. The most common case, for example, is when you run a
client directly on the server itself (either because you're the admin,
or it's a single-user box, or the client is some automated script or a
web interface or similar); by default INN gives transfer access to
localhost. (On something like a multiuser shell box you'd disable
that.)

Then you might give reader access to a peer for diagnostic or
monitoring purposes. You might give reader access to a "peer" who is
actually a suck-feeding client because you want them feeding you posts
back properly rather than using rpost. Then there are some
caching/proxy setups where you want both reader and transfer access
from the cache to the real server. Then there are probably more
scenarios I've not thought of (Russ might have some).

You'll notice in all of the above that the chances of something like
Pine running into the issue are quite low (really only the single-user
host case). OTOH, clients like perl's Net::NNTP are more likely to need
correct MODE READER handling.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 21:02:25 2004

"Andrew - Supernews" <[email-masked]> writes:

> Then you might give reader access to a peer for diagnostic or monitoring
> purposes. You might give reader access to a "peer" who is actually a
> suck-feeding client because you want them feeding you posts back
> properly rather than using rpost. Then there are some caching/proxy
> setups where you want both reader and transfer access from the cache to
> the real server. Then there are probably more scenarios I've not thought
> of (Russ might have some).

News administrators often open up transfer access to their own
workstations for testing purposes.  I have several hosts with both
transfer and reader access because some jobs that run on those hosts want
to inject Usenet messages without adding the standard trace headers for
various reasons.  Some providers offer both IHAVE and reader access to
clients, and one way to do this is to use the MODE READER support in INN.

Andrew's case of the single host is the most common, but there are a bunch
of other edge cases.  Usually they only involve fairly sophisticated
users.

> correct MODE READER handling.

And indeed there have been various bugs in that area and work put into it
at various times.  It always sends MODE READER at the beginning of the
connection unless explicitly configured not to, but it ignores any failure
of the command (which is how it works around Diablo's issue).

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 21:49:19 2004

> haven't thought about it hard enough.

The reason for the underscores is precisely because it *isn't* an
extension. We agreed a namespace for extension labels (alphanumeric plus
dot and dash), and that namespace ought to be used only for that purpose.
However we do it, lines in the LIST EXTENSIONS output that are not related
to real live extensions ought not to use that namespace.

[Put another way, it's better to say 'an extension label is a keyword' than
'an extension label is a keyword other than "NNTPV" followed by one or more
digits'.]

It costs nothing in the text, it costs nothing for in software (which won't
care), and it eliminates a possible future problem (C and POSIX have had
*endless* aggro because of namespace troubles).

> just make them two separate lines?

I want to think on this some more.

> (Oh, and would VERSION be a better name?)

If it's a separate line, then "_VERSION" is fine by me.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 22:43:24 2004

On Tue, 9 Nov 2004, Russ Allbery wrote:
> Ah, I didn't realize that you did also do reactive authentication.

Indeed I do, both in NNTP and SMTP.  AFAIK, I coined the term.

> Yes, if you also do

Yup, and when it happens I blame the server.

> this configuration of Pine wouldn't proceed after STARTTLS failed).

Right.  Servers should not advertise STARTTLS if they don't accept it.

> properly in the situation you were attempting to allow for.

My client implementation is not broken.  It is forced to send AUTHINFO 
first by the Diablo server, and (in real life) inn servers don't force 
MODE READER first.

Nor is my client broken because inn can be configured to break my client. 
inn is not the arbitrar on what is broken.

Nor is inn broken.

Nor is Diablo broken.

The NNTP protocol is what is broken.  This is the part that many people 
seem not to understand.  Protocols that create non-interoperability are 
broken.  That is why we need to fix NNTP.

> It is indeed the fault of the NNTP

It was not a mistake, for the simple reason that MODE READER was not 
documented!

It is a painful lesson to learn that, by failing to document a facility, 
the original deployer forfeits the ability to call differing 
implementations broken.  Trust me; I learned *that* particular lesson the 
hard way.  Let's you and I down that sixpack and we'll commiserate with 
our respective war stories.  We can play that fun game of "who has the 
worse tale of woe"... :-)

> recall seeing in the documents that I've read.

That's what I was told when similar issues came up in IMAP and SMTP. 
Everything had to be reset.

If you think about it, it only makes sense.  Otherwise, you don't really 
know that MODE READER actually happened, as opposed to having been seized 
by an MITM.  Although I don't think there is much (beyond harassment) that 
could happen by attacking this, a bad guy could certainly cause a great 
deal of confusion!

-- Mark --

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 00:36:58 2004

Russ Allbery said:
> clients should respond to unknown codes based on the first digit,

On the other hand, it also says that a server MUST return only the codes
permitted in this specification or in an extension *that has been activated*
(that's why 401, 480, and 483 had to be in the core specification).

Yes, the wording you mention is still there, but I wonder if we should be
removing it, or at least changing it to make it clear it's only a hint on
how to deal with broken servers.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 00:41:30 2004

> that I've agreed to use for the rest of the session".

Okay.

> should NOT be encrypted in any way.

I disagree. I read "data", in that quote, as being the low-level exchanges
of the protocol. It says that the client turns on the layer immediately
after sending the last response, and the server immediately after sending
the "completion indication", whatever that is.

Now in NNTP as currently written, the completion indication happens to be
the 281/283 response, so the extra data happens to be within it. But
there's no reason why a protocol can't have the server send data after the
completion response, precisely so that it *is* protected.

I can see that you sometimes need that data to complete setting up the
security layer, which is why I proposed the two-stage response.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 00:56:30 2004

Agreed.

> OVER it's just a regular extension.

Is it? If I want to use LIST HEADERS, I have to look for HDR in the list,
whereas if I want to use LIST ACTIVE.TIMES, I have to look for LIST with
ACTIVE.TIMES as an argument.

> It seems like it would be a lot more

One concern I have is that structuring the document (or, in particular,
"READER") as a set of extensions will make it incomprehensible to the
average reader. I believe there is a fundamental distinction between
capabilities and extensions. POST isn't an extension that some servers
offer; it's a core facility that is sometimes locked away from the
dangerous users.

To be selfish for a moment, I'm the Editor and this document is going out
with my name on every page. I am not willing to see it be a mess.

> implementation.

Except that if you're writing a transit server, you *have* to provide IHAVE
to make things work at all.

However, I'll offer you a compromise.

I'll accept that we have capabilities and extensions, and each gets its own
line in the LIST EXTENSIONS output (which really really needs renaming, but
never mind). So there will be separate lines for IHAVE and for reading
capability, for example. [I think that POST looks better as a property of
READER, just like HDR MSGID, since it's a subset of reading; you can't have
POST on its own.]

In return, you accept my concerns about namespace issues and all these
core capabilities have names outside the extension-label namespace. So that
means leading underscores, asterisk, !, or anything else that people think
gives the message "core capability as opposed to extension".

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 01:30:11 2004

>> with the security layer coming into effect at the end of the 180, *not* the

[Following comments from Ken, I think, I modified this to do them the other
way round: 281/283 as the last step of SASL negotiation, and 180 after the
security layer is in place.]

> NNTP commands don't do that; we'd be adding a new style of response

I can see that it looks like two responses, but actually it isn't.

What I'm suggesting - as modified - is that after a change to the
underlying transport encoding, the server sends a new initial greeting.
I'm suggesting 180 as the code, but it could even be 200/201. This would
happen in two circumstances in our present situation:
- successful TLS negotiation
- successful SASL security layer negotiation
though I could envisage other places where it could happen (e.g. a COMPRESS
extension that starts compression on the transport layer).

The primary purpose of this is to provide a response on to which hang any
extension capability information (however done), but it also improves
consistency by starting each protected session in the same way as the
initial unprotected one.

> (since

It's not a multiline response, which is something different.

> pass for the time being,

That's what I suggested, though I note that Mark is violently against it.

> except possibly for putting something into the
> 200 response.

Sigh. See rant below.

> I know that in an ideal world STARTTLS or SASL should have

See above. And see also Mark's comments to the effect that this is how he
would have done it in hindsight.

> of standard out there for people to use now.

Agreed.

> Most of these decisions

However, having STARTTLS and SASL leave the protocol in client-next state
is, I think, going to be one of those corner traps.

> returning multiple responses) that helps or hurts for solving this problem

Well, I'd argue the "fundamental".

====

Rant follows.

I wish you people would make your bloody minds up as to how you want to do
things.

When I argue for putting core capabilities in one line in the extensions
output, I get told that we need to make things easy and consistent for
clients to parse, so we need to put each capability on a separate line.

Yet at the same time you're proposing putting capability information in two
totally separate places (LIST EXTENSIONS output and the free-text part of
response codes) in two totally different formats that will need to be parsed
in two totally different ways and will carry two different levels of
completeness. When I suggest a way to make the two arrangments consistent
with each other, I get told it's too complex. You've got a strange idea of
what is and isn't simple.

[End rant. I now have to go off and try to get the Department of Trade and
Industry to do something more about spam. I may be some time.]

---

From: rvtol at isolution.nl (Ruud H.G. van Tol)
Date: Wed Nov 10 04:17:53 2004

Toen wij Andrew - Supernews kietelden, kwam er dit uit:

> [...] Then there are probably more

Maybe: an ssh-tunnel to that host.

-- 
Grtz, Ruud

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 10:21:11 2004

>> It is indeed the fault of the NNTP community and the INN developers for
>> not documenting MODE READER clearly enough so that both you and Diablo
>> made different, apparently reasonable mistakes in implementation
>> (although Diablo's mistake is frankly less reasonable than yours).

> implementations broken.  Trust me; I learned *that* particular lesson
> the hard way.  Let's you and I down that sixpack and we'll commiserate
> with our respective war stories.  We can play that fun game of "who has
> the worse tale of woe"... :-)

I think we're splitting hairs.  Let me try phrasing it a different way:

In the case where the initial connection involves one or more of STARTTLS
or AUTHINFO as well as MODE READER, Pine and INN do not interoperate.
Since there was no standard for this command, it's not possible to declare
one or the other of them broken in any formal sense, but given that INN
introduced the MODE READER command and continues to be the only server
that actually requires it, there is no point in implementing MODE READER
in a client unless you're trying to interoperate with INN.  INN's behavior
in this respect has not changed over the lifetime of the server.

> know that MODE READER actually happened, as opposed to having been
> seized by an MITM.  Although I don't think there is much (beyond
> harassment) that could happen by attacking this, a bad guy could
> certainly cause a great deal of confusion!

I can see how an attacker could use a MITM attack to launch a denial of
service attack, but an attacker capable of a MITM attack being able to
deny service isn't particularly uncommon.  IMAP and SMTP don't have this
bizarre mode switch that NNTP does (thankfully!), so I'm not sure that I
see how their experience implies that this is not allowed.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 10 10:48:19 2004

> or AUTHINFO as well as MODE READER, Pine and INN do not interoperate.

And since neither Pine nor inn are broken, MODE READER itself is broken.

MODE READER has multiple, non-interoperable, definitions.  It doesn't 
matter that only one of those definitions actually is used by a server 
that needs it.  There is a server that enforces the other definition's 
order (even though it doesn't otherwise care), inn does not generally 
enforce its definition's order, and Pine complied with what worked in all 
the cases its encountered.

Of course, there will be objection from those who have to change to comply 
with the future world.  It can't be avoided.  The only course is to work 
out what will be the best choice in the long term.

> in a client unless you're trying to interoperate with INN.

That may be true; but it conflicts with what I've been told for over 10 
years, which was to the effect that MODE READER was mandatory with all 
servers and that only a cretin would write a client that uses RFC 977 
instead of what the netnews weenies said was right.  Then I found that 
every server which had AUTHINFO did something different.

Given this particular history, I am not terribly sympathetic to one 
server's rarely-used configuration.

> deny service isn't particularly uncommon.

This particular type of attack has unbelievable costs in lost user and 
support staff time.  When you and I get a chance to split that sixpack, 
ask me about Norton Anti-Virus and SMTP.  We'll need a second sixpack.

The lesson learned is that you can't believe or trust anything that 
happens before TLS takes hold.

> IMAP and SMTP don't have this

Well, it's arguable that SMTP has mode switches, and there are ongoing 
efforts (which I have been opposing) to put in such things into IMAP. 
It's an attractive nuisance.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 12:45:47 2004

>> more annoying when you have to special-case the _NNTP_ extension and
>> parse its arguments and use that to decide what commands are available,
>> when for OVER it's just a regular extension.

> Is it? If I want to use LIST HEADERS, I have to look for HDR in the
> list, whereas if I want to use LIST ACTIVE.TIMES, I have to look for
> LIST with ACTIVE.TIMES as an argument.

Right, but there's a mapping between the extension labels and the
commands.  If I see HDR, I know I can use these commands.  If I see OVER,
I know I can use those commands.  I don't have to look at the parameters
to figure out what commands I can use except for AUTHINFO or LIST
subcommands, which are something of a special case.

> To be selfish for a moment, I'm the Editor and this document is going
> out with my name on every page. I am not willing to see it be a mess.

I think you were on the right track with a STATUS command, getting away
from the whole extension concept which has turned out to be too limiting
for what we're trying to represent.  How do you feel about going to
CAPABILITY (just because that matches closer what other protocols use) and
calling these capabilities instead of extensions?  Does that make it feel
like less of a mess?

> I'll accept that we have capabilities and extensions, and each gets its
> own line in the LIST EXTENSIONS output (which really really needs
> renaming, but never mind). So there will be separate lines for IHAVE and
> for reading capability, for example. [I think that POST looks better as
> a property of READER, just like HDR MSGID, since it's a subset of
> reading; you can't have POST on its own.]

You can -- you can configure INN that way -- but it's an uncommon
configuration for only very special cases that likely wouldn't be hurt by
just returning 502 to the reader commands.  So that's not necessarily an
argument against what you're proposing, although the reader commands and
the POST command really are independent.

But, for example, an automated system that generates reports by posting
them to a local newsgroup doesn't need read access and is sometimes
configured to only have posting access.

> core capabilities have names outside the extension-label namespace. So
> that means leading underscores, asterisk, !, or anything else that
> people think gives the message "core capability as opposed to
> extension".

The reason why I'm not as worried about the namespace issues is that we
have something that C and POSIX don't have, namely IANA and the ability to
require that people register their use of the namespace.  In essence,
anything that *doesn't* begin with an X is reserved for standard published
by the IETF, which significantly reduces the worries about conflicting
with extensions.  Does that change how you feel at all?

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 14:37:54 2004

> calling these capabilities instead of extensions?  Does that make it feel

Extracting this one point - I'm very happy with a new command, and
CAPABILITY is just as good as STATUS as far as I'm concerned.

I'm not clear, though: is the intent to use this only for core
capabilities, or is it going to list extensions as well, thus replacing
LIST EXTENSIONS?

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 15:37:35 2004

> capabilities, or is it going to list extensions as well, thus replacing

The intention would be to replace LIST EXTENSIONS completely.

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Wed Nov 10 17:18:56 2004

> to real live extensions ought not to use that namespace.

I thought about this a lot this morning, walking in during the cold 
dark hours before dawn, and came to the conclusion that I disagree with 
the above, that I have a lot of trouble drawing a meaningful 
distinction between core capabilities (not all of which will 
necessarily be available) and extensions (some of which will likely be 
nigh universal), and that there's no reason they shouldn't all be 
listed together.

And then I come home, and I find out that Russ has beaten me to the 
punch.  Suffice it to say that I'm all behind the CAPABILITY/LIST 
CAPABILITIES proposal, whatever it's called.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:17 2004

> server developers.  Perhaps the NNTP community would be happier if there

That sounds to me more like a social attitude problem than like a  
technical argument.

Similar for the argument "MODE READER is only useful for INN, two other  
people implemented a broken version, and the fact that their version is  
broken (except that because one is pine's and therefore it cannot be  
broken, no matter that it is completely useless) means that INN is the one  
who needs to change".

Frankly, it's hard to imagine a less convincing way of argumentation.

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:22 2004

[...]

> scenarios I've not thought of (Russ might have some).

UUCP needs transfer access from localhost. Well, actually, it's rnews that  
needs this; UUCP uses rnews. IIRC, however, nnrpd also uses rnews if innd  
can't currently accept messages - it puts those in rnews' in.coming queue  
directory and relies on a cronjob calling rnews -U to unspool them into  
innd via - of course - a transfer mode connection.

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:26 2004

> transport and retrieval have been intermingled, which is really poor

It's a very similar problem to the SMTP submission/transfer intermingling  
(which is why there's now a separate submission protocol & port) - at  
least it avoids the mistake of SMTP/POP-or-IMAP of separating retrieval  
from submission. Now that is a truely bizarre architecture!

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:31 2004

> port 119?  That to me seems like the most viable way of eliminating MODE

Something like the following:

* Document that 433 is the standard port for transfer
* Document that for historical reasons, very many servers use 119 just as
  for client access
* Document that some few servers use both client and transfer access on
  port 119, and that this case is SHOULD NOT, and that there's an old
  mechanism of "MODE READER" for coping with this, and that, *if* that is
  needed, that command needs to be the very first thing sent, and if the
  server doesn't like that command in that position, not send it at all.
* And document that standard servers might, to be compatible with
  pre-standard clients, want to support MODE READER on port 119 as a no-op
  compatibility command - they ought to accept it whenever any client
  command is allowed, for maximum compatibility.

* Modify INN to be able to have innd simultaneously listen on 119 and 433
  for migration purposes, with MODE READER not working on 433. Also make
  433 the default transfer port for new installs.

Open question: what should a modern INN in migration configuration do, and  
should that be documented - that is, a server that supports LIST  
EXTENSIONS but does have a combined transit/reader port while it hasn't  
yet mirated all transfer connections off to 433?

Possibly document a MODE_READER extension that asks for MODE READER and is  
documented purely for that case, with SHOULD NOT and so forth? Ok, let's  
spell that out:

# MODE_READER means if you want reader access, your very next move MUST be
  to issue MODE READER and reset everything you know. You really cannot
  have TLS running for that.

# Servers SHOULD NOT use this extension. It SHOULD NOT be used except
  while migrating existing transfer peers to 433.

# Pre-NNTPv2 Servers may need that functionality, or not (see above).

# It only ever affects port 119, never port 433.

Just to make sure to avoid confusion: this is port 433 "nnsp", not port  
563 "nntps" or "snntp" (NNTP over SSL) which ought to die.

>From port-numbers:

nntp            119/tcp    Network News Transfer Protocol
nntp            119/udp    Network News Transfer Protocol
#                          Phil Lapsley <[email-masked]>

nnsp            433/tcp    NNSP
nnsp            433/udp    NNSP
#                          Rob Robertson <[email-masked]>

nntps           563/tcp    nntp protocol over TLS/SSL (was snntp)
nntps           563/udp    nntp protocol over TLS/SSL (was snntp)
#                          Kipp E.B. Hickman <[email-masked]>

MfG Kai

---

From: rra at stanford.edu (Russ Allbery)
Date: Sun Nov 14 12:00:27 2004

Kai Henningsen <[email-masked]> writes:

> UUCP needs transfer access from localhost. Well, actually, it's rnews
> that needs this; UUCP uses rnews. IIRC, however, nnrpd also uses rnews
> if innd can't currently accept messages - it puts those in rnews'
> in.coming queue directory and relies on a cronjob calling rnews -U to
> unspool them into innd via - of course - a transfer mode connection.

rnews (and nnrpd) normally both use the local Unix domain socket rather
than a regular localhost connection, although they do fall back on a
localhost connection if Unix domain sockets aren't available for some
reason.

---

From: dsr at mail.lns.cornell.edu (Dan Riley)
Date: Sun Nov 14 14:24:58 2004

> > port 119?  That to me seems like the most viable way of eliminating MODE

I'd hum for that.

> Something like the following:

I mostly agree with what Kai proposed--it's close to what I was
consider proposing.  I do have a few quibbles...

>   server doesn't like that command in that position, not send it at all.

Seems like a chicken/egg problem--how does the client know if the server
likes "MODE READER" or not, if it needs to be the very first thing sent?
I'd prefer:

 * Clients SHOULD NOT send an unsolicited "MODE READER" by default
 * Clients MAY offer an option to send an unsolicited "MODE READER"
 * Clients MAY send "MODE READER" in response to specific 401 and
   502 errors
 * Clients SHOULD send "MODE READER" in response to an advertised
   "MODE READER" capability or extension.

>   command is allowed, for maximum compatibility.

At first I liked "MODE READER" as a noop, but on further thought I
think it's a bad idea.  Client issues a "GROUP" command, gets a 502,
issues a "MODE READER", and gets success--how does it interpret that?
Has something changed that means a reader command might succeed, or
hasn't it?  I'd like to hear client authors weigh in on this, but my
current prejudice is that encouraging "MODE READER" as a noop is a
mistake.

I would add:

 * The interaction between "MODE READER" and "AUTHINFO" or "STARTTLS"
   is *unspecified*

which I believe is the only accurate characterization of existing
practice.

>   while migrating existing transfer peers to 433.

I dislike the way this combines a client MUST with a server SHOULD
NOT.  I'd rather make issuing a "MODE READER" a SHOULD or even a MAY,
not a MUST.

-- 
	  The 10/8 that can be pinged is not the true 10/8.