Subject: [NNTP] Notes on auxiliary documents

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 01:58:59 2004

A cluster of notes on the three auxiliary documents (authinfo, tls, and 
streaming). Mostly editorial.

General
=======
The word "draft" shouldn't be used to refer to the document itself. It 
should be "specification" or "document".

The core specification never includes generic responses in the pro-forma 
sections for each command (unless they have some unusual meaning, such 
as 502 for MODE READER). They *are* referred to in the description to 
explain how to report various situations. The one case I have 
specifically noted is in STARTTLS, but there may be others.

authinfo-05
===========
Section 1 para 1: change "has traditionally provided public access" to 
"has traditionally been used to provide public access". [NNTP is the 
mechanism, not the agent.]

Next para, last sentence, I would reword as:

    Due to their ubiquity they are formalized in this specification but,
    because of their insecurity, only for use ...

Section 2.2: "Servers are not required to accept unsolicited 
authentication". I don't follow this. Since the server advertises its 
conformance with this document (through CAPABILITY / LIST EXTENSIONS), 
it is soliciting authentication. The only way I can read this is that, 
despite that, authentication is not permitted *until* a 480 happens. Um:

    [C] LIST EXTENSIONS
    [S] 202
    [S] AUTHINFO USER SASL:EXTERNAL
    [S] .
    [C] AUTHINFO USER fred flintstone
    [S] 481 unsolicited authentication rejected (or is it 503?)
    [C] GROUP secret.group
    [S] 480 need to authenticate
    [C] AUTHINFO USER fred flintstone
    [S] 281 Hello Fred.

Surely that can't be desirable?

Section 2.4.2: I know that we've discussed this before, but I think the 
code for "don't handle this mechanism" shouldn't be 501 but, rather, 503 
or something AUTHINFO-specific.

Just above the wording you changed, I suggested that "if a security 
layer as part of the authentication," be deleted after "In agreement 
with [SASL]". If this ability is useful (I'm still agnostic), then it 
could be useful in other circumstances as well.

Sections 2.4.2 and 6: we say that the server MUST discard all knowledge. 
Does this include NNTP state? E.g.

    [C] GROUP a.group
    [S] 211 10 555 666 a.group selected
    [C] AUTHINFO SASL MECHANISM
    [SASL details omitted, security layer created]
    [S] 281
    [C] ARTICLE 10

Does the server respond with 220 or 412? Whichever the answer is, it 
needs to be made more clear.

Section 3.1: in the last paragraph, insert ", both exclusive, after 
"between the first ... CRLF".

Section 6, last paragraph: add something like "The minimum strength 
acceptable will, of course, depend on specific circumstances and is 
outside the scope of this specification." Otherwise it implies there's 
some definition of "weak" which we're hiding (just to be perverse?).

streaming-02
============

I notice that you switch between "pipeline", "stream", and "asynchronous 
transfer" with no apparent consistency. We agreed to call the basic 
concept "pipelining", and that's what the core document calls it. You 
may want to revisit this.

Section 1 (and possibly elsewhere) talks about "server-to-server". NNTP 
doesn't have server-to-server links. Again, we should be consistent and 
only say "peer-to-peer" and "client-to-server".

Section 2.3.2 gives too much emphasis on non-conforming implementations. 
Replace it completely with something very similar to:

    On a server which conforms to this specification, the MODE STREAM
    command MUST return a 203 response (or 501 if an argument is given)
    and MUST NOT have any other effect.

    Historic note: some old server implementations did not advertise
    streaming in the response to LIST EXTENSIONS but provided CHECK and
    TAKETHIS commands similar or identical to those in this document.
    Clients could test for this by issuing the MODE STREAM command; if
    it succeeded, pipelining was possible. However, even if the command
    succeeded it does not mean that the CHECK and TAKETHIS commands
    conform to this specification. Clients SHOULD use the CAPABILITY
    command to determine server capabilities.

Section 2.4.2: is "another client is sending the same article" the best 
example? Surely "disc full" or something like that would be better.

tls-03
======

Section 2.2.2 para 6: The text "The TLS negotiation begins ... response" 
only handles one direction; compare the corresponding text in AUTHINFO 
SASL. It should be something like "For the server to client direction, 
the TLS negotiation ... response; for client to server it begins with 
the next octet sent after the CRLF at the end of the command."

In the following paragraph, surely "MUST discard" should be "SHOULD NOT 
rely on", since the client can still make heuristic decisions.

Section 5 para 3: you've got two sentences beginning "Further,"; I would 
omit the first instance. And the last sentence needs something like "or 
vice versa" at the end.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 11 04:13:49 2004

>>>>> "Clive" == Clive D W Feather <[email-masked]> writes:

 Clive> streaming-02
 Clive> ============

 Clive> I notice that you switch between "pipeline", "stream", and
 Clive> "asynchronous transfer" with no apparent consistency. We
 Clive> agreed to call the basic concept "pipelining", and that's what
 Clive> the core document calls it. You may want to revisit this.

The term for article transfer using check/takethis is "streaming feed"
(and has been since the concept was invented). The fact that the main
document uses "pipelining" to refer to sending multiple commands before
receiving responses isn't relevent to that (and in fact the main doc
deliberately doesn't use the term "streaming" for that precisely because
that term has always had the specific meaning of "streaming feed").

 Clive> Section 2.4.2: is "another client is sending the same article"
 Clive> the best example? Surely "disc full" or something like that
 Clive> would be better.

You really don't understand how streaming feeds work at all, do you?

"Another client is sending the same article" is the SOLE REASON WHY THE
DEFERRAL RESPONSE CODE FOR "CHECK" EXISTS. The normal case is that the
server receives CHECK commands for a given article from several peers
more or less simultaneously; it answers the first one with 238 and all
the rest with 431, until either it receives the article via TAKETHIS or
a timeout expires (in which case it answers the next CHECK for that
article with 238 again). This is necessary to avoid wasting bandwidth
by getting multiple copies of the same article.

The usual (and most reliable) way to handle "disc full" is to throttle
the server, i.e.  to stop accepting feeder connections at all.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 04:23:34 2004

> that term has always had the specific meaning of "streaming feed").

Okay. In which case, we should be careful that each use of those two terms
is correct, and that "asynchronous" is removed completely.

> You really don't understand how streaming feeds work at all, do you?

No. That's why I asked the question.

> a timeout expires

Okay. But in which case "has offered to send the same article" would be
better wording. As phrased, it looked like "another client has already
started an IHAVE or TAKETHIS for the article, but it hasn't completed"
and seemed an odd reason for giving a defer code (if it fails, that other
client can re-send).

> (in which case it answers the next CHECK for that

Understood.

Note: this may be obvious to you who does this every day, but the spec
should attempt to minimize the specialist knowledge required.

> the server, i.e.  to stop accepting feeder connections at all.

Um, I thought you said that deferring made sense after CHECK and the first
stage of IHAVE; it was only after TAKETHIS (and the second stage of IHAVE)
that it didn't.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 11 06:17:48 2004

>> "Another client is sending the same article" is the SOLE REASON
 >> WHY THE DEFERRAL RESPONSE CODE FOR "CHECK" EXISTS. The normal case
 >> is that the server receives CHECK commands for a given article
 >> from several peers more or less simultaneously; it answers the
 >> first one with 238 and all the rest with 431, until either it
 >> receives the article via TAKETHIS or a timeout expires

 Clive> Okay. But in which case "has offered to send the same article"
 Clive> would be better wording.

Yeah.

 Clive> As phrased, it looked like "another client has already started
 Clive> an IHAVE or TAKETHIS for the article, but it hasn't completed"
 Clive> and seemed an odd reason for giving a defer code (if it fails,
 Clive> that other client can re-send).

That case applies as well. The article can't be considered "received"
until the IHAVE or TAKETHIS is completed; and during the interval
between first seeing the message-id (whether by CHECK, IHAVE or
TAKETHIS) and the completion of the IHAVE or TAKETHIS, the server will
(generally speaking) return deferrals rather than refusals simply
because it has no way to ensure that a failed in-progress transfer
will ever be retried by the sending end.

 >> The usual (and most reliable) way to handle "disc full" is to
 >> throttle the server, i.e.  to stop accepting feeder connections at
 >> all.

 Clive> Um, I thought you said that deferring made sense after CHECK
 Clive> and the first stage of IHAVE; it was only after TAKETHIS (and
 Clive> the second stage of IHAVE) that it didn't.

It makes sense in the context of the normal case described above. But
it's _not_ OK to simply respond to _every_ CHECK or IHAVE with a
defer, because that means that both ends of the peering session will
just waste bandwidth and CPU on a continuous stream of deferrals.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 11 08:37:13 2004

> should be "specification" or "document".

Fixed the one instance (in AUTHINFO) that I found.

> specifically noted is in STARTTLS, but there may be others.

502 is listed in both AUTHINFO and STARTTLS pro-forma sections to make 
it clear that it also applies after these commands have been used (per 
the notes).  I'm pretty sure this was discussed before and it was signed 
off on.  I really don't care if we keep it or remove it, but at this 
point, I'd like to change as little as possible since we appear to be in 
a "final tweaking" stage.

> authinfo-05

Done.

>     because of their insecurity, only for use ...

Done.

> Surely that can't be desirable?

Correct, I think this is what Jeff is trying to say.  That the client 
can go ahead and authenticate without having received a 480 response and 
the server must accept authentication even is not prompted by a 480. 
Essentially, this sentence is to be paired up with the first sentence of 
the previous paragraph.

I'll gladly take some suggested text, unless you feel that this sentence 
can be deleted entirely.

> or something AUTHINFO-specific.

This has already been changed.  Did you see my "proposed changes" posts?

> could be useful in other circumstances as well.

Continuing to advertise only makes sense when a security layer is 
advertised, since the info can be trusted otherwise (a MiM attack could 
still be in progress).

> needs to be made more clear.

That's a good question.  This wording comes from the other SASL profile 
docs (IMAP, POP, SMTP) in which authentication usually (always?) happens 
before other commands.

I don't *think* that server has to go back to the "unselected group" 
state after auth, but my guess is that if the client authenticates in 
the middle of a session its because they are trying to access a group 
that returned a 480 and therefore will be changing groups immediately 
after auth.

Does anyone have any string feelings on this?

> "between the first ... CRLF".

I don't understand what change your are requesting (you only have 3 
quotes in your suggestion).

> some definition of "weak" which we're hiding (just to be perverse?).

I added: "The minimum strength acceptable is a policy decision which is 
outside the scope of this specification."

> streaming-02

Please look at my "proposed changes" post and see if it addresses this. 
  Of course, based on Andrew's post, we may have to change this yet again.

> only say "peer-to-peer" and "client-to-server".

I fixed the two instances in Section 1.  I believe the wording is 
correct elsewhere.

>     command to determine server capabilities.

First off, I think Russ signed off on the current text, but I'm willing 
to clarify if needed.  In fact, I like most of what you have above.

This is a similar situation to MODE READER.  We are trying to steer new 
implementations away from MODE STREAM but document its prior use.

Russ, can you provide some guidance?

> example? Surely "disc full" or something like that would be better.

I used the wording from your subsequent post.

> the next octet sent after the CRLF at the end of the command."

I don't see the confusion.  For both client and server the negotiation 
starts after the CRLF.  The same wording (except for "octet") is in IMAP 
and POP.

> rely on", since the client can still make heuristic decisions.

I'm not sure about this.  Based on Mark's comments from the MODE READER 
thread, I *think* that this is almost a requirement of TLS (whether 
implicitly or explicity, I don't know).

Russ, guidance?

> omit the first instance. And the last sentence needs something like "or 

Sentence removed, not sure what "vice versa" gets us.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 09:17:49 2004

Ken Murchison said:
>> as 502 for MODE READER).

> off on.

Hmm, I don't recall offhand.

> the server must accept authentication even is not prompted by a 480. 

Okay, that's good.

> Essentially, this sentence is to be paired up with the first sentence of 

So by "unsolicited" you actually mean something like AUTHINFO PASS after
the AUTHINFO USER was rejected, or carrying on the SASL exchange after a
48x. The latter is just going to confuse the server, and the former has an
error code already.

> I'll gladly take some suggested text, unless you feel that this sentence 

If I'm still on track, I don't think we need it. It's like saying that the
client mustn't send the article body if POST doesn't get the 354 code.

> This has already been changed.  Did you see my "proposed changes" posts?

Yes, but obviously overlooked this one when writing this message.

> advertised, since the info can be trusted otherwise (a MiM attack could 

I still think this is wrong (and, indeed, how we advertise this is the
wrong approach), but I'll leave it.

> that returned a 480 and therefore will be changing groups immediately 

Well, yes, but we can't guarantee it and so can't rely on it. There may
also be other server state either now or in the future.

I don't mind whether it's "server must forget all state", or "server must
retain all NNTP state", but we need to be absolutely clear. Oh, and TLS
needs to be consistent with SASL on this.

[I'd prefer "retain state", because it may not be obvious to the main NNTP
engine in the client whether a security layer was negotiated or not by some
other module, and therefore whether the state was retained or not.]

> quotes in your suggestion).

Sorry, it comes of writing the material on a cramped aircraft.

Change:
    "between the first ... CRLF ..."
to:
    "between the first ... CRLF, both exclusive, ...

> outside the scope of this specification."

Fine.

> Of course, based on Andrew's post, we may have to change this yet again.

I think it might be best to wait until you circulate the next draft (or a
snapshot in the way I do) and then do a full sweep.

> I fixed the two instances in Section 1.  I believe the wording is 

I didn't make a full scan for this, just noted it as I was writing (I blame
the cramped plane again).

> implementations away from MODE STREAM but document its prior use.

Right. One thing that's happened since that text was written is that we've
gone for a much more solid distinction between conforming-to-our-texts and
older servers. For example, an NNTPv2 server now MUST NOT advertise
STREAMING in LIST EXTENSIONS unless it conforms to your document. So we can
sensibly say it's a no-op, conforming clients don't need to use it with
conforming servers, but there's a historic compatibility issue.

I'll be taking the same approach with MODE READER if you lot decide to
change INN not to need it.

> I used the wording from your subsequent post.

Okay.

> starts after the CRLF.  The same wording (except for "octet") is in IMAP 

Hmm.

> implicitly or explicity, I don't know).

We discussed this a while ago when talking about whether clients had to
resend LIST EXTENSIONS in various places. Security considerations mean that
you mustn't rely on previous information, but it's sensible to use it for
heuristic decisions - so long as you're prepared for failure - and even for
security checks. That "do not rely on" wording is in the core document
IIRC.

> Sentence removed,

Um, I didn't mean remove the sentence, just the word "Further" (before
"because delivery").

> not sure what "vice versa" gets us.

The sentence basically tells server authors that they can't trust the
ultimate origin of material sent via IHAVE/POST/TAKETHIS just because
they know who the client is. Fine.

But don't TLS and SASL allow the client to check it's talking to the true
server? If so, the fact that it is doesn't mean that the server has checked
the origin of articles sent via ARTICLE either. Hence "or vice versa".

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Thu Nov 11 15:56:24 2004

On Nov 11, 2004, at 11:37 AM, Ken Murchison wrote:

>> conformance with this document (through CAPABILITY / LIST 
>> EXTENSIONS), it is soliciting authentication. The only way I can read 
>> this is that, despite that, authentication is not permitted *until* a 
>> 480 happens.
>
> can go ahead and authenticate without having received a 480 response 
> and the server must accept authentication even is not prompted by a 
> 480. Essentially, this sentence is to be paired up with the first 
> sentence of the previous paragraph.

Well, Clive's initial read is correct.  Before we had the strong 
extension discovery mechanism we do now, the original text was trying 
to document existing practice.  And presently there's quite a mess 
about AUTHINFO, in that some servers will return a non-empty group list 
(and thus not issue 480) despite having additional private groups that 
appear after authentication.  Thus many newsreaders have an "always 
send username/password even when not prompted" option.

And from that point of view, without LIST EXTENSIONS, Ken's comment is 
correct that

     A client MAY attempt the first step of authentication at any time
     during a session to acquire additional privileges without having
     received a 480 response. [...] Servers are not required to accept
     unsolicited authentication information from the client, therefore
     clients MUST accommodate servers that reject such authentication
     information.

was intended to be one coherent thought.  (And I think it is, in fact, 
quite sensible, as documentation of current practice.)

But I think Clive's observation is correct, that this doesn't make much 
sense with extension discovery.  So I guess this is another spot that 
needs rewording to distinguish interoperability with old servers as 
separate.

Ken, given the above, can you edit, or do I have to come up with 
something concrete?  :-)

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 11 15:58:06 2004

> quite sensible, as documentation of current practice.)

Is there any reason not to simply require that servers support unsolicited
authentication, without any note about older servers?  Are there any
servers that don't support AUTHINFO at any point during the session?  (I
think it would actually be harder to write a server that behaved that way
than to support AUTHINFO at any time.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 11 16:13:55 2004

> than to support AUTHINFO at any time.)

That's a good question.  Do we really need to discuss this at all? 
Isn't assumed that any command can be used unprompted?

If we can get a consensus on what we want this to say, I have some 
airplane time tomorrow and Saturday to work on this.

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Thu Nov 11 16:15:31 2004

On Nov 11, 2004, at 6:58 PM, Russ Allbery wrote:

> authentication, without any note about older servers?

I think we should be able to say that any NNTPv2 server that lists an 
AUTHINFO extension should support unsolicited authentication, sure.

Y'know, there's a subtly here I can't quite wrap my head around.  
Everyone seems to like the "advertising AUTHINFO is enough to 
effectively eliminate the 480 response" idea, if I'm reading the list 
right.  But have we come to a conclusion that reactive authentication 
is bad?  I mean, what if the server advertises AUTHINFO because some 
users can authenticate, but most users won't need to.  Then the client 
software still has no way to know if it should prompt the user for 
authentication parameters or not.  So I guess we still need 480.

But regardless, yeah, unsolicited authentication can be mandatory when 
AUTHINFO is advertised.

> Are there any

I think it would be hard to do so as well.  But in the "current 
practice" realm, the client has no way of telling if the server 
supports AUTHINFO at all, and so it might appear as a "failure of 
unsolicited authentication" in that case.  If you think we don't need 
to document this, I certainly don't feel strongly about it...the 
existing text reads as it does really by accident.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Fri Nov 12 09:12:26 2004

>that returned a 480 and therefore will be changing groups immediately 

Not necessarily.

Suppose you are in Group A, happily reading articles. Suppose you now
encounter an article cross-posted to the restricted group B. The server
invites you to authenticate, which you do. You would now like to be in the
same NNTP state as before, still in Group A with the crossposted article
still the current article, so you can now read it.

OTOH, what happens if you are unable to authenticate? Is there some way
that you can remain in the same group, ignore that article, and continue
reading the rest of the articles in the group?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 17 02:24:15 2004

Charles Lindsey said:
>>> Does this include NNTP state?

I see that the current wording (06pre1) says that it does.

> still the current article, so you can now read it.

Very true. This suggests that the state should *not* be reset.

On the other hand, all the client needs to do is to re-issue the GROUP and
ARTICLE commands to get back to where it was. And this ensures that the
state really is what the client thought, even if someone was spoofing the
connection previously.

Um, if the ARTICLE fails for some other reason the current article number
will be wrong. There's no way to set it in this case; there are always edge
cases that break things.

> reading the rest of the articles in the group?

If authentication fails, the server state should be unaffected; you ignore
the 480 error and just issue a NEXT (or whatever).