Subject: [NNTP] LIST EXTENSIONS and an NNTPv2 capability

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 03:36:40 2004

Ken Murchison said:
>> But I'd certainly like to encourage extensions to indicate both what is
>> available and what is currently-unavailable, rather than just the former.
>> And this is something that I'd like to see even if we don't adopt this
>> STATUS proposal.
> I don't see why this is necessary.  We really need client authors to 
> weigh in here.  We, as server authors shouldn't be deciding what's good 
> for clients in a vacuum.

I'd still like to hear from client authors here.

>> Or we can bite the bullet and say that servers must advertise every command
>> that they implement.
> *Every* command?  Why?  The NNTPv2 capability states all of the base 
> commands that the server supports (other than the optional LIST xxx 
> stuff -- which is still a klugde IMHO).  Any other commands MUST be 
> advertised by an extension label.

Um, I didn't mean that each command should be listed separately in LIST
EXTENSIONS (or STATUS). I meant that every command supported (except for
the mandatory ones) should be *indicated* by something in the output. In
many cases several commands are indicated at once.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 07:40:30 2004

Mark Crispin said:
> I am an NNTP client author, *not* an NNTP server author.

Um, yes. But Ken (I think) was saying you're the only one.

> I agree with Ken Murchison.

So you don't want a "this could be available", only "this is available
right now"? Just want to be clear I'm understanding you.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 07:44:33 2004

Mark Crispin said:
>> As far as MODE READER is concerned, I personally agree with you. However,
>> server implementers will tell you otherwise. If you want to argue for this
>> change, please do so and let's see what happens.
> It certainly can be argued that this is a server internal detail which 
> should not burden the protocol.  It also adds an additional RTT (6 by my 
> count).
[...]

Again, I'm in complete sympathy.

I note that no server authors have chipped in here (or I missed them), but
my memory is that, at the time we first discussed this, they were adamant
that it was necessary.

Russ, where do we go from here?

> Better would be something like the following, which uses an IMAP-like 
> mechanism to carry capabilities in responses, and flushes MODE READER:
> 
> 	S: 200 [STARTTLS] NNTP server, negotiate TLS now

I'm still thinking on this. As I said before, it isn't compatible with NNTP
as currently specified but that's not irretrievable.

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 2 08:00:08 2004

> Um, yes. But Ken (I think) was saying you're the only one.

Yes, I was pointing out that he was the only client author that we've 
heard from.  This is actually disconcerting that with all of the NNTP 
clients out there (it seems everyone has their own favorite) that only 
one has chosen to take part in this list (although somewhat late).

I don't know if this is because they don't know that work is being done 
to update NNTP, they became frustrated with past work and stopped paying 
attention (which is where Mark was for a while), or they just don't care.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 2 12:48:31 2004

> as currently specified but that's not irretrievable.

I'm not sure why this would be incompatible.  Obviously the ABNF would 
have to change to accomodate this, but from a client perspective it 
wouldn't be a big deal.  For a client which handles this, it parses the 
contents of "[ ... ]", otherwise, it looks just like trailing-comment.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 2 15:15:16 2004

> contents of "[ ... ]", otherwise, it looks just like trailing-comment.

The problem is the other way round: a conforming RFC 977 server can already
produce such responses with meaningless contents.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:06 2004

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:07 2004

> right now"? Just want to be clear I'm understanding you.

Yes.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:07 2004

> produce such responses with meaningless contents.

It is not, however, particularly likely that an RFC 977 is going to 
produce such a response which contains the "NNTPv2" capability.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 03:07:02 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 >> As far as MODE READER is concerned, I personally agree with
 >> you. However, server implementers will tell you otherwise. If you
 >> want to argue for this change, please do so and let's see what
 >> happens.

 Mark> It certainly can be argued that this is a server internal
 Mark> detail which should not burden the protocol.  It also adds an
 Mark> additional RTT (6 by my count).  Consider the normal startup
 Mark> situation:

 [long exchange of commands]

OK, it occurs to me that we can simplify that a lot by allowing the
LIST EXTENSIONS output to indicate one of three explicit conditions:

1) that MODE READER isn't necessary (which will be a very common case)
2) that MODE READER really is necessary (but might fail)
3) that MODE READER isn't possible, and nor are any reader commands (which
will be true for transit servers with no reader support at all, or where
the connecting IP can never obtain reader access)

with the absence of any of the above indications, the existing rule
applies (that the server is allowed to reject reader commands if MODE
READER is not issued first).

In existing practice, ignoring the above proposal, it's actually not
unreasonable for clients to do MODE READER before the first LIST
EXTENSIONS, since the MODE READER shouldn't get a 480 response (and
if it does, you can do LIST EXTENSIONS then before authenticating).
But the spec certainly doesn't lead an implementor to do it that way.

Other issues:

 - the client doesn't have to do LIST EXTENSIONS before STARTTLS if it
   wants to initiate TLS on its own initiative rather than at the
   server's request.

 - if the client doesn't want to initiate TLS but the server wants to
   force it to do so, your suggested exchange doesn't make any sense.
   (It's important that servers not encourage clients to initiate TLS
   unnecessarily, since that leads to overhead that can only be avoided
   by disabling TLS altogether.)

 Mark> Better would be something like the following, which uses an
 Mark> IMAP-like mechanism to carry capabilities in responses, and
 Mark> flushes MODE READER:

There are two fundamental objections to that. First is that you're
assigning semantics to fields which in the deployed specification are
explicitly designated as containing arbitrary non-semantic text. The
second is that you can't just handwave MODE READER out of existence;
if you make it impossible to conform to the spec, then people will
just not bother conforming to the spec and will stick with the
established practice.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 4 07:13:19 2004

> the connecting IP can never obtain reader access)

What you mean by 3 is "MODE READER will fail with 502", right? Number 2
means that reading commands can fail "401 MODE-READER" while number 1
means they won't.

> READER is not issued first).

Um, let's do it right: the server will be required to include this
information in the LIST EXTENSIONS response.

> explicitly designated as containing arbitrary non-semantic text.

Right: even if it's unlikely in practice, that is not a foundation to build
a standard on.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 4 08:16:32 2004

> produce such a response which contains the "NNTPv2" capability.

It doesn't matter; you can't rely on it. That's building on sand, not
drawing a line in it :-)

There's also the problem that response lines are limited in length and
there may not be room for the message as well as the existing response.

====

I've done some thinking about this and come to some conclusions.

What you're talking about is a specific use of what I would call
"asynchronous messages" - messages not in immediate response to a command.
IIRC, IMAP has these as a general possibility, as does (in a different
area) the X11 protocol.

Our existing response code model doesn't have anywhere to insert
asynchronous messages from the server. It can't go in the free text area
for the reasons given above, so it needs to go somewhere else. For example,
a response line beginning "!" could indicate an asynchronous message (I
thought of using a response code range like 7xx, but it doesn't work well).
However, if we start sending these to existing clients they will get very
very confused.  This means that the client needs to explicitly indicate
that it is willing to receive these messages.

At which point I realize that we're talking about an extension to the
protocol which can be specified just like any other extension. So we don't
have to include it in the core document and we don't even have to agree it
right now. That means we don't need to let it affect the present work.

====

Having said that, and purely as a proof of concept, I'll sketch out such an
extension to show that it does what you want. I'll call it "EVENTS", having
pinched the name from X11.

If the extension is supported, LIST EXTENSIONS reports it as EVENTS
followed by one or more keywords indicating the event types available.
The initial set is:
  EXTENSIONS - report changes to the available set of extensions since
               the last LIST EXTENSIONS in this session.
  GROUPSIZE  - report a change in the low and/or high water marks of the
               current group.
  TELLUSER   - the server wishes the user to be informed of something.

A client indicates its willingness to receive events by issuing the EVENTS
command followed by the set of events it wants. "*" indicates all events;
an empty argument list cancels all previous requests. Response codes:
  206 - request successful
  503 - unsupported event type

An event is indicated by a response line beginning "!", instead of a three
digit response code, at *any* place where the server can send a response
(including both places in two-stage commands). A single-line event begins
with "! ", while a multi-line one begins "!. " and is dot-stuffed and
-terminated. The event is followed by another event (without limit) or
the actual response.

Initial list of events:

EXTENSIONS events. In the following, the "current extensions knowledge"
is the server's understanding of what the client knows about the available
extensions. This is an empty set at connection setup, and is set to the
response whenever a LIST EXTENSIONS command is successfully issued.

!. EXTENSIONS !
    The current extensions knowledge is replaced by the following material.
    This MUST be used after successful authentication or security level
    changes and whenever the server knows that the knowledge is wrong; this
    implies it will occur as part of the response to an EVENTS command if
    no LIST EXTENSIONS has been issued.

!. EXTENSIONS +
    The following material is added to the current extensions knowledge.

!. EXTENSIONS -
    The following material is removed from the current extensions knowledge.

GROUPSIZE events:

! GROUPSIZE number low high group
    If a GROUP command were issued with the name of the current group, the
    server would return a 211 response with these parameters.

TELLUSER events:

! TELLUSER text
    The server wishes the user to be informed of the contents of the text
    without delay.

Example:

    [C] EVENTS GROUPSIZE
    [S] 206
    [C] GROUP local.group
    [S] 211 5 100 107 local.group selected
    [C] STAT 107
    [S] 223 107 <a.message@id> selected
    [C] NEXT
    [S] 421 no next article
    [C] LAST
    [S] 223 106 <[email-masked]> selected
    [C] NEXT
    [S] ! GROUPSIZE 6 100 108 local.group
    [S] 223 107 <a.message@id> selected
    [C] NEXT
    [S] 223 108 <new.message@id> selected

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 09:27:29 2004

> the connecting IP can never obtain reader access)

This is a good idea.

> But the spec certainly doesn't lead an implementor to do it that way.

Sigh.  Once again, we fall into the trap of talking about how to work 
around existing warts, rather than moving forward.

>   wants to initiate TLS on its own initiative rather than at the

No!!!  The client must know that the server supports a non-mandatory 
command before issuing it.  That is the entire point of a capability 
announcement mechanism.

Please expunge this notion of "just try the command and see if it works." 
It is *extremely* harmful to a protocol to have a mixed mode of capability 
announcement and "just try."

>   force it to do so, your suggested exchange doesn't make any sense.

In all protocols, a server forces TLS by requiring authentication and not 
advertising any authentication mechanisms until TLS is in force.

In all protocols with the exception of NNTP, TLS is forced unless the 
server has an authentication mechanism that does not exchange passwords in 
the clear.  Even then, it is advisable to use TLS since it is absolutely 
trivial to hijack sessions.

>   by disabling TLS altogether.)

TLS is here.  TLS is mandatory.  We need to live with it.

Statements to the effect of "security isn't as important in this protocol" 
are going to be met with very harsh responses from people who have fought 
this battle many times before.

Yes, indeed, this battle has been fought many times before; your argument 
has been advanced many times before; and it's been thoroughly crushed many 
times before.  Hard feelings have very commonly resulted.

It would be a benefit for all of our blood pressures if we can learn the 
lessons from the past, decide not to fight this battle, and accept the 
outcome from this battle in every other protocol.

> explicitly designated as containing arbitrary non-semantic text.

There is ample precedent in other protocols: MIME and IMAP have both done 
it, and quite successfully.

Removal of extraneous RTTs is always a good thing.

> The

Is it really just one server, inn, that has this issue?

It certainly is not "impossible to conform to the spec"; and adding an 
unnecessary RTT to every NNTP interchange for the benefit of a single 
server implementation is an extremely poor idea.

-- Mark --

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 09:34:43 2004

> information in the LIST EXTENSIONS response.

Yes.  If MODE READER is required, there should be an extension that 
indicates that fact (or an extension to indicate that MODE READER is not 
required -- I don't care).

The cases then degenerate into:

1) MODE READER isn't required
2) MODE READER is required, but you must authenticate before you can do
    any reading.  The extension for MODE READER is not listed until after
    authentication.
3) MODE READER is required, and authentication is not required before you
    can do any reading.  The extension for MODE READER is listed.

> Right: even if it's unlikely in practice, that is not a foundation to build

Why?  MIME did it.  IMAP did it.

These are standards which are arguably quite a bit technically superior 
to, and much less messy than, NNTP.

-- Mark --

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 09:50:19 2004

> It doesn't matter; you can't rely on it. That's building on sand, not

Sometimes you have to make compromises in order to accomplish a desirable 
result.  The elimination of several RTTs is a highly desirable result.

I know that I'm sounding like a broken record, but both MIME and IMAP 
exist as precedents.  I know because I was personally involved in both; 
and in both cases there were objections much as you have raised.

Nevertheless, the predicted problems never came to pass.

Actually, now that I think about it, SMTP has *also* done this!!!!!!!

> there may not be room for the message as well as the existing response.

That, now, is a real issue.

However, I note that the the response line limit is 512 octets.  This is 
enough for the existing set of extensions, although I would recommend a 
larger limit.

A way that this could be gotten around may be to restrict the 
auto-announced extensions to a certain set which are meaningful for 
session startup; that is, the extensions for TLS, authentication, and 
(sigh) MODE READER.  So, the client still has to do a single LIST 
EXTENSIONS when it's actually ready to do actual reading.  That's one 
addtional RTT, which is still better than the status quo.

> "asynchronous messages" - messages not in immediate response to a command.

No.  It's piggy-backing additional status information on an existing 
synchronous response.  Both IMAP and SMTP does this.  SMTP has no 
"asynchronous messages".

> for the reasons given above, so it needs to go somewhere else.

What is so special about NNTP that it can not do exactly what IMAP and 
SMTP did?

Both IMAP and SMTP have piggy-backing, and both did it on what was 
previously a free-text area.

I repeat: what is so special about NNTP?!?!

> pinched the name from X11.

If you're going to add what IMAP calls "unsolicited data" to NNTP, then 
you have a l-o-n-g way to go.  Don't forget that I've been in that 
business with IMAP for nearly 20 years.

At that point, the question becomes "why not flush NNTP in favor of IMAP."

Fortunately, you don't need unsolicted data.  You just need piggybacking.

-- Mark --

---

From: ma at dt.e-technik.uni-dortmund.de (Matthias Andree)
Date: Thu Nov 4 10:11:49 2004

> change, please do so and let's see what happens.

I am a server implementor (leafnode) and can state that "mode reader" is
a no-op, printing 200, not switching state, in my software, but leafnode
is best described as store-and-forward proxy in its current shape, so it
won't stack up to INN (I don't care for what INN implementors might
think until I've read it), DNews or Diablo.

MODE READER to me has been the means that told INN to exec NNRPD... and
this documents there appears to be a desire to distinguish streamlined
transport facilities (within innd) from more expensive and colorful
reader facilities (within nnrpd).

NNTP is, in this respect, different from mail, for instance, which has
separate protocols for transport (SMTP, UUCP) and retrieval (POP3, IMAP,
ETRN, ODMR, perhaps something else I forgot).

I don't have strong opinions either way, on one hand the startup
overhead is considerable, on the other hand we have a lot more mode
switching around, STARTTLS the most prominent in this thread albeit in
different context, so if thinking this to the very end, the first step
would be to drop STARTTLS and other obvious mode switching, and the
second step would be to drop the hidden mode switching, such as
statefulness of GROUP command, which switches from the mode/state
"cannot retrieve by article number" into "can switch by article number".

So let's please do away with mode selection arguments.

-- 
Matthias Andree

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 10:36:37 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 >> - the client doesn't have to do LIST EXTENSIONS before STARTTLS if
 >> it wants to initiate TLS on its own initiative rather than at the
 >> server's request.

 Mark> No!!!  The client must know that the server supports a
 Mark> non-mandatory command before issuing it.  That is the entire
 Mark> point of a capability announcement mechanism.

 Mark> Please expunge this notion of "just try the command and see if
 Mark> it works." It is *extremely* harmful to a protocol to have a
 Mark> mixed mode of capability announcement and "just try."

I can see we're not going to agree on this.

 >> - if the client doesn't want to initiate TLS but the server wants
 >> to force it to do so, your suggested exchange doesn't make any
 >> sense.

 Mark> In all protocols, a server forces TLS by requiring
 Mark> authentication and not advertising any authentication
 Mark> mechanisms until TLS is in force.

It's very common with NNTP to not require authentication at all.
That's why we have a response code specifically saying "you can't do
that without starting TLS first" in addition to the one that says "you
can't do that without authenticating".

 Mark> In all protocols with the exception of NNTP, TLS is forced
 Mark> unless the server has an authentication mechanism that does not
 Mark> exchange passwords in the clear.  Even then, it is advisable to
 Mark> use TLS since it is absolutely trivial to hijack sessions.

When authentication isn't necessary, it's extremely rare (by proportion
of traffic) for TLS to have any value at all for an NNTP session.

 >> unnecessarily, since that leads to overhead that can only be
 >> avoided by disabling TLS altogether.)

 Mark> TLS is here.  TLS is mandatory.  We need to live with it.

So you'll be happy to foot the bill?

Let me ask you this question. What would you say is a typical range
for peak (or 95-percentile) traffic in megabits/sec (client -> server
and server -> client combined) for a large POP3 or IMAP installation?

 Mark> Yes, indeed, this battle has been fought many times before;
 Mark> your argument has been advanced many times before; and it's
 Mark> been thoroughly crushed many times before.  Hard feelings have
 Mark> very commonly resulted.

I suggest you read what I actually said.

 >> assigning semantics to fields which in the deployed specification
 >> are explicitly designated as containing arbitrary non-semantic
 >> text.

 Mark> There is ample precedent in other protocols: MIME and IMAP have
 Mark> both done it, and quite successfully.

MIME did it? where?

 Mark> Removal of extraneous RTTs is always a good thing.

Only where it doesn't break stuff.

 >> The second is that you can't just handwave MODE READER out of
 >> existence; if you make it impossible to conform to the spec, then
 >> people will just not bother conforming to the spec and will stick
 >> with the established practice.

 Mark> Is it really just one server, inn, that has this issue?

I believe so. However, you have to understand the _reason_ why that is
so - other servers, such as Diablo for example (or indeed our
proprietary one) simply split the two functions (transit and reader)
apart and require that they be run on separate servers or separate
ports, thus adding an order of magnitude to the difficulty of
installing and setting up. That's tolerable at large sites, but INN
is still usually the favoured (free) solution for small to medium
sites.

Furthermore, the problem is architectural in nature and not a mere
implementation issue - good design usually requires separating the
transit and reader functions since they have almost no overlap, and
forcing use of an alternate port number for transit is not an
acceptable solution.  That means that any INN-like implementation
will run into exactly the same issue.

 Mark> It certainly is not "impossible to conform to the spec";

It would be impossible to conform to the spec (sans MODE READER)
without taking away features that people are, in fact, using quite
widely. 

I agree that MODE READER is an extremely annoying wart, but it exists
for a specific reason, and you can't legislate that reason out of
existence. Which is why I proposed an approach that actually _is_
implementable.

 Mark> Science does not emerge from voting, party politics, or public
 Mark> debate.

I think you should read your own signature more often.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 10:48:39 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> The cases then degenerate into:

 Mark> 1) MODE READER isn't required

This is, by far, the most common case.

 Mark> 2) MODE READER is required, but you must authenticate before
 Mark> you can do any reading.  The extension for MODE READER is not
 Mark> listed until after authentication.

That's backwards. Where MODE READER is required, it has to come
_before_ authentication (or TLS). I should have spotted that error in
your previous post. (Implementations where MODE READER is an issue
almost certainly won't be advertising auth or TLS in the LIST
EXTENSIONS output until after the MODE READER command, and even if
they do, the MODE READER has to come first because authentication
is likely to be handled differently for transit clients.)

-- 
Andrew, Supernews
http://www.supernews.com

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 11:07:44 2004

> This is, by far, the most common case.

Excellent.

> your previous post.

That is *not* (repeat, NOT!!) the case in implementations today.

I have encountered multiple server implementations which do NOT (repeat, 
NOT) permit MODE READER until after authentication is negotiated.

If you are seriously advocating requiring MODE READER before 
authentication, then in effect you are declaring these deployed servers as 
being non-compliant.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 11:15:17 2004

> It doesn't matter; you can't rely on it. That's building on sand, not

How/why wouldn't it be reliable?  If we use a construct that is fairly 
unique (e.g. "[ ... ]") and pick an extension label for our undated spec 
(e.g. "NNTPv2" or "_NNTP_2") that is unlikely to be used in comment 
text, the likely hood of them being used together in a specific format 
in a response (e.g. "[NNTPv2 STARTTLS ...]" is *very* unlikely. 
Therefore any client which is aware of NNTPv2 and wishes to use this 
facility can easily spot the capability list in the response.  I'd also 
like to point out, that only four of our current responses (200, 201, 
281, 283) would be realistic candidates for this facility (although I 
could be missing one).

If a client is not aware of NNTPv2 or this capability announcement 
facility, it will simply ignore it as comment text.

> there may not be room for the message as well as the existing response.

This isn't a big deal.  Either the server pares down the list as Mark 
suggests, or it simply doesn't include it at all (forcing the client to 
do an explicit LIST EXTENSIONS).  We're doing a similar thing with the 
SASL initial response.  If the initial response would cause the command 
line to be too long, the client must omit it and force the server to 
request it with an empty challenge.

> IIRC, IMAP has these as a general possibility, as does (in a different

Mark has already addressed this as *not* what we're doing.  All we're 
really doing here is anticipating the fact the the client will be doing 
a LIST EXTENSIONS after the current command and merging the responses 
into a single one.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 11:18:24 2004

>> _before_ authentication (or TLS). I should have spotted that error
 >> in your previous post.

 Mark> That is *not* (repeat, NOT!!) the case in implementations today.

 Mark> I have encountered multiple server implementations which do NOT
 Mark> (repeat, NOT) permit MODE READER until after authentication is
 Mark> negotiated.

Interesting. Do you have data on what implementations those are?
Because INN certainly doesn't allow authentication to happen before a
_required_ MODE READER, so either you're seeing this from non-INN
implementations (in which case MODE READER isn't just a
one-implementation artifact) or you're confusing the (rare) required
use of MODE READER with the no-op implementation.

Traces of actual conversations would be useful.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 11:35:11 2004

> is likely to be handled differently for transit clients.)

Given INN's use of MODE READER, I would expect this to be the case. 
Assuming that this is the expected use, I think we can address these 
issues with 2 capabilities (these are actually arguments to the NNTPv2 
extension label since we're talking about base functionality):

1. MODE_READER - (as already proposed by Clive) specifies that MODE 
READER is required before using any reader commands

2. MODE_FEEDER (or FEED_ONLY or NO_READING, etc) - specifies that the 
server doesn't permit *any* reader commands

The absence of both of these options means that the server permits 
reader commands and MODE READER isn't required.  The two options MUST 
NOT be advertised at the same time.

The only case which isn't addressed is a reader-only server, which 
really only effects IHAVE and I think returning a 503 solves this.  I'm 
guessing most reading clients wouldn't bother with IHAVE anyways.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 11:41:32 2004

> I can see we're not going to agree on this.

This is not a matter of opinion.  It is proven fact.

IMAP, POP3, SMTP, ACAP, etc. all have a capability mechanism that works as 
I state.

NNTP, on the other hand, has indeterminate states and an incomplete 
capability mechanism.  To put it mildly, NNTP sucks.

I was hoping that this group was going to fix that.

> can't do that without authenticating".

That is an obsolete protocol design.  The problem is (as was realized 
during the IMAP2 -> IMAP4 transition) is that the client is obliged to 
undertake many unnecessary RTTs.

RTTs are not your friend.

> of traffic) for TLS to have any value at all for an NNTP session.

We are rapidly approaching a world in which authentication is always 
necessary.

Yes, I know that Supernews (like many other NNTP service provides) uses 
quick and dirty authentication mechanisms such as IP address validation.

> So you'll be happy to foot the bill?

I'm sorry to be the bearer of bad news, but you will foot the bill, after 
you find itself the victim of an attack that your current authentication 
mechanisms do not prevent.  Your are, in effect, hiding behind a security 
through obscurity shield.

The means of attack are already well-known and implemented.  Currently, 
there are much easier targets for the bad guys.  However, as those targets 
are one-by-one closing their holes, your day of reckoning approaches.

We all have the choice of being reactive or proactive.  We're forced in 
some cases to be reactive.  However, there are other cases in which we can 
be proactive now to save having to be reactive later.

> Mark> both done it, and quite successfully.

MIME boundary delimiters.  I know because I invented them.

All the same arguments were made against MIME boundary delimiters because 
the string in the delimiter could appear in the MIME segment (those who 
held this position argued for a count of "lines").  The successful 
counter-argument was that it was easily possible to generate a boundary 
delimiter with the chance of a false hit mathematically small enough to be 
disregarded.

Later, BASE64 and QUOTED-PRINTABLE made it possible to create boundary 
delimiters which could not appear in MIME segments that used those 
transport encodings.  However, since not all segments use these transport 
encodings, it has to fall back on unlikelihood.

> Only where it doesn't break stuff.

However, in this case nothing is being broken.  The client can still do 
the RTT-costly method.

> so

I understand perfectly.  It's an implementation issue in one server. 
That is a remarkably poor excuse for something which costs multiple RTTs 
per session for every client.

If a mode switch was needed to separate clients from peers, why isn't the 
burden on peers?

> without taking away features that people are, in fact, using quite

Just what "features" are "taken away"?

> existence.

No, but competant programmers can fix the servers so that they do not have 
this problem.

-- Mark --

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 11:44:23 2004

On Thu, 4 Nov 2004, Andrew - Supernews wrote:
> _required_ MODE READER

In that case, INN is not compliant with the AUTHINFO draft.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 11:47:48 2004

> In that case, INN is not compliant with the AUTHINFO draft.

If there is text in AUTHINFO which either explicitly or implicitly 
states that MODE READER can't be used before authentication, please 
point it out so I can fix it.  This certainly wasn't the intent of the 
document.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 11:50:35 2004

On Thu, 4 Nov 2004, Ken Murchison wrote:
> If there is text in AUTHINFO which either explicitly or implicitly states 
> that MODE READER can't be used before authentication, please point it out so 
> I can fix it.  This certainly wasn't the intent of the document.

There is text in AUTHINFO which states that authentication can be used 
before MODE READER.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 11:52:52 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 [snip]

I'll address your other points separately, but I'll repeat this
question since you failed to answer it:

-- 
Andrew, Supernews
http://www.supernews.com

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 12:00:36 2004

>>>>> "Ken" == Ken Murchison <[email-masked]> writes:

 Ken> The only case which isn't addressed is a reader-only server,
 Ken> which really only effects IHAVE and I think returning a 503
 Ken> solves this.  I'm guessing most reading clients wouldn't bother
 Ken> with IHAVE anyways.

It might be possible to make a case that IHAVE should really be an
extension.

Traditionally, reader-only servers have responded with 480 to IHAVE,
though that's slightly misleading since no possible authentication
could make the command available. We return 502 at the moment. It's
mostly a non-issue precisely because reader clients don't use it.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 12:01:41 2004

> There is text in AUTHINFO which states that authentication can be used 

Right, it says:

"The AUTHINFO commands can be used before or after the MODE READER
command, with the same semantics."

It doesn't say that AUTHINFO MUST be used prior to mode reader.  How is 
INN non-compliant?  Maybe I'm being thick.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 12:01:56 2004

> and server -> client combined) for a large POP3 or IMAP installation?

Even if I had an answer for this question, it is irrelevant for the 
discussion at hand.

If a facility's lack of security is used to enable a major attack, the 
number of megabits/second without security ceases to be relevant.  The 
facility will do what is necessary to ameliorate the attack.

You are not the only person to squawk about the overhead of TLS.  The cell 
phone providers also vigorously object; currently use of TLS cuts battery 
time in half.

-- Mark --

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 12:12:01 2004

On Thu, 4 Nov 2004, Ken Murchison wrote:
>>> that MODE READER can't be used before authentication, please point it out 
>>> so I can fix it.  This certainly wasn't the intent of the document.
> It doesn't say that AUTHINFO MUST be used prior to mode reader.  How is INN 
> non-compliant?  Maybe I'm being thick.

> command, with the same semantics."

You're thinking about the other case, which is of a server which issues a 
480 to MODE READER, which was the (2) example of server handling of mode 
reader.  AUTHINFO seems to prohibit this behavior as well as inn's 
behavior.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 12:32:55 2004

>> command, with the same semantics."

OK, I see your point.

The intent of the statement is to allow servers to decide whether to 
permit AUTHINFO before and/or after MODE READER.

You're looking from the client's perspective in which the statement 
seems to say that the client can use AUTHINFO whenever it feels like it. 
  But as long as INN doesn't advertise AUTHINFO before MODE READER, 
doesn't this solve the problem?

Do you have any suggested text which might clarify this?

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 12:51:37 2004

>> Let me ask you this question. What would you say is a typical
 >> range for peak (or 95-percentile) traffic in megabits/sec (client
 >> -> server and server -> client combined) for a large POP3 or IMAP
 >> installation?

 Mark> Even if I had an answer for this question, it is irrelevant for
 Mark> the discussion at hand.

It's relevent, but the reasons for that may not become clear unless you
answer the question.

 Mark> If a facility's lack of security is used to enable a major
 Mark> attack, the number of megabits/second without security ceases
 Mark> to be relevant.  The facility will do what is necessary to
 Mark> ameliorate the attack.

I don't think you really understand the threat model that applies to
much of the common usage of NNTP.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 13:03:56 2004

> It's relevent, but the reasons for that may not become clear unless you

I know, you're going to give me the "TLS is too expensive" argument.

The fallacy is that this assumes current conditions.  When engineering a 
protocol, we must consider the conditions of the future.

> much of the common usage of NNTP.

The phrase "much of the common usage" is quite a prevarication.

Threats occur from many sources.  DDOS is a threat, but so are session 
hijacking and IP address spoofing.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 13:05:09 2004

> Do you have any suggested text which might clarify this?

Answering my own question.  Would this text be better?

"A server MAY allow the AUTHINFO commands to be used before or after the 
MODE READER command, with the same semantics."

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 13:20:14 2004

On Thu, 4 Nov 2004, Ken Murchison wrote:
> The intent of the statement is to allow servers to decide whether to permit 
> AUTHINFO before and/or after MODE READER.

That is an extremely bad design (to put it mildly).

> You're looking from the client's perspective in which the statement seems to 
> say that the client can use AUTHINFO whenever it feels like it.

That is the only reasonable interpretation of:
 	The AUTHINFO commands can be used before or after the MODE READER
 	command, with the same semantics.

> But as long 
> as INN doesn't advertise AUTHINFO before MODE READER, doesn't this solve the 
> problem?

At the cost of considerable client complexity, which will get worse if 
authentication is required for peers as well as clients.

As a client author, I vehemently object to allowing the server to dictate 
the order of a protocol sequence.  In EVERY well-designed protocol, the 
protocol sequence is dictated first by the specification, and then by the 
client.

> Do you have any suggested text which might clarify this?

Yes.  Do not give the server the option of imposing an order on the 
client.  Either leave the order as random (under client direction), or 
impose One True Order and *prohibit* all other orders.

My released client code implements this order, which works on every NNTP 
server that I tested:
 	STARTTLS (if needed - note that plaintext password authentication
 		  requires TLS)
 	AUTHINFO (if needed)
 	MODE READER

To re-confirm, I just verified on an inn server that inn *does* allow 
AUTHINFO before MODE READER.  I also determined that Supernews' NNTP 
server allows AUTHINFO before MODE READER.

Perhaps there are clients which implement a different order.  If so, then 
the *only* choice is to require that servers MUST allow either order.

Life's tough if you're in the server business.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 13:32:23 2004

>     command, with the same semantics.

Yes, I came to this conclusion myself after re-reading it.

> As a client author, I vehemently object to allowing the server to 
> dictate the order of a protocol sequence.  In EVERY well-designed 
> protocol, the protocol sequence is dictated first by the specification, 
> and then by the client.
> 

I'm tending to agree with this recommendation.  My gut it to leave the 
order as random, but I'll wait to see if anybody squeals before I make 
this change.

> server allows AUTHINFO before MODE READER.

OK, so the current language in AUTHINFO does not make either server 
non-compliant.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 13:34:59 2004

On Thu, 4 Nov 2004, Ken Murchison wrote:
>> AUTHINFO before MODE READER.  I also determined that Supernews' NNTP server 
>> allows AUTHINFO before MODE READER.

Unless there is a version of inn server (or Supernews server) that forces 
one or the other.  If so, that version is non-compliant.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 4 13:38:18 2004

Mark Crispin wrote:

> Unless there is a version of inn server (or Supernews server) that 
> forces one or the other.  If so, that version is non-compliant.

The intent is to document existing deployment, so any text which makes 
existing implementations non-compliant is not a good thing.  I'd like to 
rewrite the text, if necessary, solve any problems.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Thu Nov 4 13:43:15 2004

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> My released client code implements this order, which works on
 Mark> every NNTP server that I tested:
 Mark>  	STARTTLS (if needed - note that plaintext password authentication
 Mark>  		  requires TLS)
 Mark>  	AUTHINFO (if needed)
 Mark>  	MODE READER

This will fail on INN in those few cases where MODE READER is actually
needed (rather than being a no-op) and where authentication (or TLS)
is also required. However, I think you'll be hard-pressed to find such
a case other than by setting it up explicitly.  (In most of the few
cases where MODE READER is needed, authentication would be pointless.)

 Mark> To re-confirm, I just verified on an inn server that inn *does*
 Mark> allow AUTHINFO before MODE READER.

This is only true when the MODE READER was never necessary in the
first place (which as I said before is the most common case by far).

In the case where you actually need MODE READER, some AUTHINFO
commands will work before MODE READER is used but the results will
be thrown away once you're talking to nnrpd rather than innd.

If you want to test it, you need to arrange to have transfer as well
as reader access to the INN server you're testing against.

 Mark> I also determined that Supernews' NNTP server allows AUTHINFO
 Mark> before MODE READER.

MODE READER is a no-op on our server.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 16:33:50 2004

> rewrite the text, if necessary, solve any problems.

Well, I haven't seen any deployments that do that.

-- Mark --

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 16:37:14 2004

> cases where MODE READER is needed, authentication would be pointless.)

In that case, I think that it is safe to say that such cases don't exist 
and *won't exist* in NNTPv2.

> be thrown away once you're talking to nnrpd rather than innd.

Remember that we are documenting NNTPv2.  This would be something that 
would have to be fixed in inn before it can advertise itself as being 
NNTPv2 compliant.  Fortunately, it seems that the fix affect few if any 
deployed sites.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 20:36:04 2004

> AUTHINFO before MODE READER.

Then your INN server is configured in a way that doesn't require MODE
READER.  Please note that almost all INN configurations do not require
MODE READER, since even when the initial connection is handled by innd, it
is immediately shunted to nnrpd unless the connecting host has permission
to issue transit commands.

As the maintainer of INN, I can assure you with 100% certainty that this
order will NOT work in the unusual situation where MODE READER is
required.  Your AUTHINFO command will fail before MODE READER, and after
nnrpd spawns it will have no idea that you ever attempted it.

Note that most servers have eliminated the need for MODE READER by
ensuring that one of the following conditions is true:

 * The same server process is used for both transit and reader
   connections.  I think this design makes it considerably harder to
   maintain the server, for a wide host of reasons, but admittedly those
   reasons are internal to server design.

 * There are no connecting hosts which are both allowed to issue transit
   commands and are allowed to read news.  This is the common case and is
   the reason why one can completely ignore MODE READER with nearly all
   INN installations and get away with it.

 * The transit server is running on the separate port reserved for that
   purpose (433) and the reader-only server is running on 119.

Of the solutions, I actually prefer the last one, since I've encountered
rare but significant instances where I really needed to give both transit
and reader permissions to the same client host.  It's obnoxious in that
NNTP consumes an extra low-numbered port, which is really highly rude of
it, but the damage (in terms of the port registration) was already done
many years ago.

I don't like MODE READER either.  Personally, I implement the third
solution for all of my servers.  But mandating one of the above cases
makes me pretty nervous.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 20:38:15 2004

"Andrew - Supernews" <[email-masked]> writes:

> though that's slightly misleading since no possible authentication could
> make the command available. We return 502 at the moment. It's mostly a
> non-issue precisely because reader clients don't use it.

Note that INN optionally allows IHAVE via the reader-only server due to
constant demand for the feature complete with provided patches from server
administrators who had to implement it.  I believe the primary motivation
was support for Microsoft products that wanted to pull news with NEWNEWS
and send it back with IHAVE.

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 20:59:01 2004

> protocol, we must consider the conditions of the future.

It's probably a mistake to stick my ore in here, but in the hope that I
can try to explain where large site administrators like Andrew are coming
from....

I understand your point about security, but the content transmitted via
NNTP is public information to several nines.  Sites that deal with private
newsgroups do indeed have to worry about the things that you talk about;
sites that deal with the public Usenet simply don't, and won't, because
it's unnecessary work.  The data that security measures would normally
protect is not worth protecting.  The cost/benefit tradeoff for using TLS
routinely, at least for reading (posting can be a separate issue) is
wrong, and nothing we say in a standard will change that.

I *wholeheartedly* agree with you that NNTP has wandered far out in the
woods, ignored lessons from other protocols, and made a mess of itself by
doing things its own way.  It is, by and large, not special, not
different, and doesn't need its own solutions to problems.  However, there
is one way in which NNTP actually is legitimately different than other
protocols mentioned in this discussion, and that's that nearly all traffic
it transmits is public information neither requiring nor wanting any sort
of protection.  In this respect, the best analog to another protocol is
actually to HTTP, and the common case for HTTP traffic is to not encrypt
it and not protect it for very similar reasons.  NNTP does do more
authentication than HTTP (*particularly* for posting), but I don't think
the benefits of authentication for the typical NNTP server even reach the
level of SMTP, let alone something like IMAP or POP.

Yes, the facilities have to exist to deal with the difference between
almost and all, but NNTP does have a significantly different common case,
and hence when one tries to optimize for the common case, one ends up
taking different tradeoffs.

Anyway, I don't think that we have to agree on this to figure out the
right protocol design for the current document, so we probably don't
really need to argue over it.

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 21:01:32 2004

>>> I guess the only additional char (beyond what you've outlined) which
>>> sticks out as potentially usable for keywords is '_'

>> I think we only want one of '-' and '_'. I don't greatly care which,
>> though I think the former is slightly preferable.

> I don't care enough to make a big issue out of it.  I'll let the rest of
> the WG (you and Russ) decide which one(s) will be used.

I've been engaging in a completely irrational war against underscores for
years, so I'm quite happy with a dash.  :)  (Really, I don't much care one
way or the other, and can't see omitting '_' causing any real problems for
anyone.)

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 4 22:23:04 2004

On Thu, 4 Nov 2004, Russ Allbery wrote:
> The cost/benefit tradeoff for using TLS

Posting is an issue.  Session hijacking and IP address spoofing are 
currently too much trouble for spammers to use, given all the much easier 
ways to spam.  That will not always be the case.

Unfortunately, it is not safe to infer that the absence of attacks in the 
past is a predictor of the absence of attacks in the future.

As for reading, I'll note that Supernews apparently considers read 
security to be important enough to deny me access to their servers when my 
client IP address is my office workstation at UW, but to permit it when my 
client IP address is one of my home network sites.  So read security is 
not unimportant.

Note, however, that TLS protects more than public posting.  It also 
protects the exchange of authentication credentials.  For better or worse, 
the overwhelming majority of authentications involve userid and password, 
transmitted to the server in the clear.  Every NNTP server that has 
password-based authentication but not mandatory TLS exposes that user's 
password to every bad guy in the universe.

I am reasonably confident that IESG will not permit AUTHINFO USER or 
AUTHINFO SASL PLAIN to be standardized without mandatory TLS.

Now, you could have AUTHINFO SASL CRAM-MD5 or AUTHINFO SASL GSSAPI or any 
other SASL authenticator that does not allow an eavesdropping attacker to 
authenticate as the user, and get away without using TLS.  But that still 
doesn't address session hijacking problems.  Not using AUTHINFO at all, 
and instead relying upon client IP address validation, doesn't address IP 
address spoofing.

-- Mark --

---

From: rra at stanford.edu (Russ Allbery)
Date: Thu Nov 4 23:04:23 2004

Mark Crispin <[email-masked]> writes:

> protects the exchange of authentication credentials.  For better or
> worse, the overwhelming majority of authentications involve userid and
> password, transmitted to the server in the clear.

Well, the majority (but probably not overwhelming any more) of NNTP
authentications are based on the IP address of the connecting client and
don't involve any protocol commands at all, since anything heavier-weight
isn't judged to be worth the tradeoff.  (Sites like Supernews are a
special case since they have clients all over the Internet, but most news
servers are for particular organizations, and just accept all connections
coming from the IP addresses of those organizations.)  I'm not sure how
long this will continue to be viable, particularly for protecting posting,
but my experience with helping people install INN is that this is what the
significant majority of INN installations use.

Part of the problem, of course, is the lack of support in clients for
sensible authentication mechanisms that don't involve handing your
passwords out to every server you talk to and sticking them into every
client program you run, but if I start ranting about Kerberos and lack of
support thereof even with network protocols that have allowed for it for
years, I'll be here all day.  :)

> AUTHINFO SASL PLAIN to be standardized without mandatory TLS.

Yup, I agree with you.

The restriction on AUTHINFO USER will likely be ignored by everyone,
however, even if we put it into the protocol.  I doubt we're going to see
anyone change existing practice for AUTHINFO USER; our best hope is to get
them to do the right thing when they go to AUTHINFO SASL.

But that doesn't mean I think we should take it out.  I'm just saying that
what the standard will say for good, valid security reasons and what
people will do is going to differ for a while, due to where we're at right
now and the slow speed of change.

I don't expect anyone to be convinced that session hijacking is a real
problem until spammers start doing it on a broad basis.  This is one of
those things that is likely to require negative things to happen in
practice before installation configurations will change.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Nov 5 06:35:31 2004

> way or the other, and can't see omitting '_' causing any real problems for

Whatever we do, we should make sure that we're consistent with the use 
of the extension label as documented for the 401 response (which 
currently includes "MODE-READER").

---

From: jeff at litech.org (Jeffrey M. Vinocur)
Date: Fri Nov 5 13:03:22 2004

On Nov 5, 2004, at 1:23 AM, Mark Crispin wrote:

> password, transmitted to the server in the clear.  Every NNTP server 
> that has password-based authentication but not mandatory TLS exposes 
> that user's password to every bad guy in the universe.

The overwhelming majority of authentications involve cleartext 
passwords because up until now that's essentially all there was.  I 
expect that SASL will play a substantial role in the future.  And over 
time, I believe NNTP (and hopefully many other protocols) are going to 
subsume their authentication and encryption into SASL, and be able to 
fold all of that into use of a SASL library.  I don't think this is 
naive, but I guess I could be wrong.

Regardless, the mere fact that authentication at present is cleartext 
unless protected by TLS is insufficient justification to distinguish 
STARTTLS as coming strangely before MODE READER.

> Now, you could have AUTHINFO SASL CRAM-MD5 or AUTHINFO SASL GSSAPI or 
> any other SASL authenticator that does not allow an eavesdropping 
> attacker to authenticate as the user, and get away without using TLS.  
> But that still doesn't address session hijacking problems.

Although, a SASL mechanism that establishes a security layer would 
suffice for this.

> Not using AUTHINFO at all, and instead relying upon client IP address 
> validation, doesn't address IP address spoofing.

Although I don't have any data, my impression is that most sites using 
IP addresses for authorization are in control of the entire block of 
IPs in question, the network paths between those IPs and the server, 
and the network paths to the server from the outside world.  Thus I 
expect the level of concern these administrators have for IP spoofing 
is quite low.  Even if such attacks became more common, I'd expect the 
easiest solution to be proper configuration of the involved network 
hardware...

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Fri Nov 5 15:13:33 2004

On Thu, 4 Nov 2004, Russ Allbery wrote:
> I'm not sure how

I suspect that you and I are loudly agreeing.

> practice before installation configurations will change.

I fear that you're right on this point.  On the other hand, the fact that 
the right thing isn't taken seriously today doesn't mean that we should 
stop saying it.  Otherwise, when the problem *does* start happening, we're 
going to need a new working group to devise the standards to react to it.

Proactive is always better than reactive.

-- Mark --

---

From: clive at demon.net (Clive D.W. Feather)
Date: Sun Nov 7 03:41:27 2004

In message <[email-masked]>, Ken Murchison <[email-masked]> writes

>>way or the other, and can't see omitting '_' causing any real problems for

Okay, that's settled then and I'll make the appropriate changes.

>currently includes "MODE-READER").

Since this was our own, recent, invention I'm going to alter it to
"MODE_READER"; we can consistently use underscores wherever we need to
join two separate words into one.

---

From: rvtol at isolution.nl (Ruud H.G. van Tol)
Date: Sun Nov 7 04:00:41 2004

Toen wij Clive D.W. Feather kietelden, kwam er dit uit:
> Ken Murchison:

>> [dashes-yes, underscores_no]
>> Whatever we do, we should make sure that we're consistent with the
>> use of the extension label as documented for the 401 response (which

> join two separate words into one.

Why can't that <401-response-extension-label> remain "MODE-READER"?

-- 
Grtz, Ruud

---

From: binaryboy at binaryboy.com (Howard Swinehart)
Date: Sun Nov 7 07:27:17 2004

From: "Ken Murchison" <[email-masked]>
> clients out there (it seems everyone has their own favorite) that only one 
> has chosen to take part in this list (although somewhat late).
>
> I don't know if this is because they don't know that work is being done to 
> update NNTP, they became frustrated with past work and stopped paying 

I'm a client author and I've been reading semi-regularly for several years. 
Usually the discussion is moving in a direction I agree with so I feel no 
need to add anything.  My silence is agreement.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Mon Nov 8 06:29:58 2004

> Why can't that <401-response-extension-label> remain "MODE-READER"?

Because we're permitting (in theory) an extension called MODE-READER. Using
an underscore to represent a space costs nothing and prevents a potential
confusion in the future.

I only used hyphen in the first place because we didn't used to permit
hyphens in extension labels.

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Mon Nov 8 09:12:46 2004

>it, but the damage (in terms of the port registration) was already done

Does our present draft document the use of port 433? And if not should it
do so? If it does, then the last solution seems the cleanest.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 01:20:24 2004

> Why?  MIME did it.  IMAP did it.

I don't know about IMAP, but your MIME example isn't relevant - the MIME
delimiter field was *not* replacing arbitrary non-semantic text, since
delimited objects were a new feature. Rather, you made it a requirement
that the delimiter not appear in the delimited text, which *can* be done
(even if you have to scan the complete object before choosing the
delimiter).

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 01:37:35 2004

[...]

Having worked through the MODE READER discussion, here's what I think is
the situation.

There are three classes of commands: "transit", "reader", and "general".
Of the core commands apart from MODE READER:
* IHAVE is transit
* HEAD, STAT, LIST ACTIVE, LIST EXTENSIONS are general
  (that's what the present text implies; I think at least QUIT needs
   to be added to that list)
* everything else is reader.

There are *four* possible types of connection. The connection type depends
on the server, the connecting IP, the connection port, and possibly the
phase of the moon :-)

(1) Full-functional. MODE READER is a no-op. The client can use any command
at any time (I'm ignoring authentication etc. for the moment).

(2) Reader-only. MODE READER is a no-op. The client can *not* use any
transit command (it gets 502), but can use all other commands at any time.

(3) Transit-only. MODE READER is a no-op. The client can *not* use any
reader command (it gets 502), but can use all other commands at any time.

(4) Switching. Initially reader commands cannot be used; they get a 401
response. Once MODE READER has been issued, transit commands cannot be
used; they get a 502 response. MODE READER destroys all server state apart
from the switch and therefore LIST EXTENSIONS output can change in this
one case.

Is this correct, in people's opinion?

If so, then why don't I revamp the description to express it like that?
MODE READER can then be rewritten in these terms, which I expect will be
far less confusing. LIST EXTENSIONS can indicate which of the four cases
applies (I would make (1) the default, with the others being READER_ONLY,
TRANSIT_ONLY, and MODE_READER). We'll need to decide which commands are
"transit" and which are "reader", but that won't hold up the rewrite.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 01:40:48 2004

Charles Lindsey said:
> Does our present draft document the use of port 433?

It does now.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 01:45:12 2004

> reader command (it gets 502), but can use all other commands at any time.

These two should be 503 ("not supported") rather than 502 ("wrong
authority").

> used; they get a 502 response.

This one *is* 502, since reconnecting makes transit commands available.

> LIST EXTENSIONS can indicate which of the four cases
> TRANSIT_ONLY, and MODE_READER).

Ken writes:
> really only effects IHAVE and I think returning a 503 solves this.

Actually, I agree with Mark here. If we're going to have core capabilities
described in LIST EXTENSIONS, let's not leave *anything* as "suck it and
see".

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 02:21:13 2004

Ken Murchison said:
> How/why wouldn't it be reliable?

Apart from anything else, because a malicious implementation could cause
all kinds of hassle while remaining technically conforming to 977.

Let's see if we can find a cleaner answer.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Nov 9 02:48:44 2004

> result.  The elimination of several RTTs is a highly desirable result.

I agree. The question is whether we can do so without breaking anything
already there. And I think I can come close.

> (sigh) MODE READER.

Right now, yes. But will this change? Who knows.

> synchronous response.  Both IMAP and SMTP does this.  SMTP has no 

I think this is a terminology thing. What we're looking at is status
changes that are caused by, rather than specified by, commands. I see
notification of these as independent of the main responses; I agree that
technically they're not asynchronous, but a capability for the latter can
be used for the former as well.

> What is so special about NNTP that it can not do exactly what IMAP and 

What exactly was done, and what was the installed base?

And if a cleaner solution exists, why not use it? [Please answer this
generically, before arguing whether my proposals are cleaner.]

====

Okay, accepting for the moment your view that all we need is piggybacking,
and agreeing that the aim is to reduce the number of round trips, let's see
if we can find a workable compromise.

First, I'd like to reinforce a point which people - including Russ - seem
to have missed (perhaps I didn't put it well). This is *VERY* late in the
day to be making significant changes to the wire protocol, particular
when these changes are forced on all clients whether they like it or not,
and when there seem to be ramifications (e.g. piggybacking on the SASL
completion response) that still aren't clear. Technically these changes
are probably out of scope, and do we really want to be including them
without any test implementations (both client and server)?

Fortunately, it isn't necessary to do this in the core protocol unless.
There's a solution which:
- decouples this from the core standardisation path;
- only invokes the feature on client demand, rather than always;
- doesn't require restricting output to a few "important" extensions
- meets my concerns about uncleanliness;
- costs only one RTT at most compared with your approach, while still
  saving round trips compared with the existing protocol.

The answer is to define this facility as an extension, in a separate
document (though it could be rolled back into the core at a later date).
That does the decoupling. The specific proposal meets the other three
objectives.

====

The proposed extension is called LEAN ("LIST EXTENSIONS automatic
notification"). It defines no new commands.

If LIST EXTENSIONS is invoked with the argument AUTO and the server
supports LEAN, auto-notify mode is activated; this is described below.
[We can either allow NOAUTO to turn off auto mode, or we can not have any
way to turn it off.]

[Um, I've just realized that we need to allow "LIST EXTENSIONS AUTO" in
the core protocol to avoid the need to see if LEAN is supported before
using it.]

In auto mode, any command can generate a 101 response code. This indicates
that the LIST EXTENSIONS output has changed. The new output follows the 101
code, followed by the original response line. LIST EXTENSIONS shows that
auto mode is in effect by using the 101 response itself, followed by a
203 response (since the normal 202 means the list would follow).

    [S] 200 NNTP server ready
    [C] LIST EXTENSIONS AUTO
    [S] 101 Auto mode selected
    [S] _NNTP_ 2
    [S] LEAN
    [S] AUTHINFO USER SASL
    [S] .
    [S] 203
    [C] AUTHINFO USER fred
    [S] 101 New extension list
    [S] _NNTP_ 2
    [S] LEAN
    [S] STREAMING
    [S] .
    [S] 281 Authentication accepted

A POSSIBLE optimisation is to provide a "delta" mode where the server sends
changes to the list rather than a new list. This could be done by flagging
the response, for example with a FULL/DELTA argument to the 101 response
and then + and - prefixes:

    [S] 200 NNTP server ready
    [C] LIST EXTENSIONS AUTO
    [S] 101 FULL Auto mode selected
    [S] _NNTP_ 2
    [S] LEAN
    [S] AUTHINFO USER SASL
    [S] .
    [S] 203
    [C] AUTHINFO USER fred
    [S] 101 DELTA Change extension list
    [S] - AUTHINFO USER SASL
    [S] + STREAMING
    [S] .
    [S] 281 Authentication accepted

But this is an add-on; it's not a core part of my proposal.

====

If Russ is confident that a significant change to the core protocol can be
put through at this stage, my proposal can be rolled into the core
protocol simply by removing the fact that it's an extension!

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 08:32:11 2004

> all kinds of hassle while remaining technically conforming to 977.

Pshaw.  Strawman argument.

Nothing, and I do mean nothing, will stop a malicious implementation no 
matter what we do.

There is plenty of precedent to the practice of placing protocol in what 
was previously free-text.  SMTP does it (extended response codes, "ESMTP" 
in the greeting).  POP3 does it (APOP authentication data in the 
greeting).  IMAP does it (status response codes, including capabilities in 
the greeting).

NNTP is not unique and different, nor does it have special vulnerabilities 
that require special practices different from every other protocol to 
overcome malicious implementations.

> Let's see if we can find a cleaner answer.

Somehow, I find it difficult to hear the phrase "clean" associated with a 
protocol that gave us MODE READER, incomplete LIST EXTENSIONS, etc.

-- Mark --

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 08:53:46 2004

> What exactly was done, and what was the installed base?

What is the installed base of SMTP, POP3, and IMAP?

SMTP added extended response codes.  SMTP also requires that an ESMTP 
capable server include "ESMTP" somewhere in the greeting.

POP3 added the APOP authentication token in the greeting.

IMAP added status response codes, among which are capabilities in the 
greetings.

All of these protocols either have incredibly stupid designers who can not 
see the problems that are the NNTP designers see -- or just maybe the 
problems are not substantial enough to be of concern.

> generically, before arguing whether my proposals are cleaner.]

I'm afraid that you may not be seeing the higher-level picture.

The single aspect of NNTP and news header formats that sucks the worst is 
where they go off and do something different from email, all in the name 
of doing what someone in the news community thought was "cleaner".  This 
is an ongoing problem; the news community has a dismal history of 
insisting upon reinventing wheels that are already long-invented and 
well-understood.

The work involved to examine and consider reinvented wheels, and to create 
new tools for these wheels (as opposed to reusing existing tools) should 
not be underestimated.

It isn't as if there is no other work to do in NNTP, and thus there is the 
luxury of thinking about how to reinvent a better wheel.  It isn't as if 
the IESG, not to mention the entire community of client implementors, are 
twiddling their thumbs with nothing to do other than think about the 
latest wheel reinvention.

> This is *VERY* late in the
> completion response) that still aren't clear.

Now you are making me quite angry.

There is a reason why productive people haven't paid much attention to 
what is going on in the NNTPEXT and USEFOR working groups.  These groups 
have been so dysfunctional in the past that most of us stopped tracking 
them.  I wouldn't be here if Ken hadn't dragged me in with assurances that 
there was new leadership to stop it.

"Force changes on all clients"?!?  What the hell are you talking about? 
Every existing client is fully compatible with this change.  That's the 
advantage of using a free text field.

"SASL ramifications still aren't clear"?!?  Do you really think that NNTP 
is so unique that this is a problem that hasn't already been addressed?

> Technically these changes

A client and server implementation could probably be produced in a few 
hours.

>  saving round trips compared with the existing protocol.

It's still a wasted RTT.

Your proposed AUTO mechanism is quite complicated and as such is extremely 
vulnerable to implementation problems.  I doubt that I would ever 
implement it.

Among other things, it adds the equivalent of IMAP unsolicited data to 
NNTP.  This is something that is extremely difficult to get right.  IMAP 
absolutely needs it.  NNTP does not.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 09:45:05 2004

> IMAP added status response codes, among which are capabilities in the 

I don't have any hard numbers, but my guess is that if you just look at 
open source products (Sendmail, Qpopper and UW IMAP respectively), we're 
talking about thousands (if not 10K or 100K) of servers that were 
deployed with these changes without the client world coming to an abrupt 
end.

The reason being that the above changes as well as those being proposed 
for NNTP and *completely* backwards compatible with an RFC 977 client. 
Its a completely "take it or leave it" facility, not a "take it or else 
you're screwed" facility.

>> completion response) that still aren't clear.

We're not *forcing* anything on the client.  We're simply putting some 
structured text in the free-text area that a client is free to use or 
ignore as they see fit.  An "enlightened" client such as Pine would use 
the capability info in these responses to reduce roundtrips.  An 
"unenlightened" client (such as legacy clients or those written by 
authors that don't care to read specs) wouldn't be impacted as all.  My 
guess is that most currnt clients ignore the free-text that comes with 
"success" responses anyways.

Please point out the unclear SASL ramifications, so we can address them. 
  If we're talking about the 283 response, I don't see any problems.  If 
we're talking about SASL security layers, this is nothing new and isn't 
a show stopper.

> A client and server implementation could probably be produced in a few 

I'd volunteer to put this in the Cyrus code and Mark could test against 
it with Pine, as we've done in the past.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 19:00:06 2004

>    to be added to that list)

QUIT is indeed also a general command.  (In practice, STAT probably is
too, but we didn't put it into that category for some reason that I don't
recall off-hand and which isn't important enough to revisit.)

> * everything else is reader.

Right.

> There are *four* possible types of connection. The connection type
> depends on the server, the connecting IP, the connection port, and
> possibly the phase of the moon :-)

> (1) Full-functional. MODE READER is a no-op. The client can use any
> command at any time (I'm ignoring authentication etc. for the moment).

> used; they get a 502 response. MODE READER destroys all server state
> apart from the switch and therefore LIST EXTENSIONS output can change in
> this one case.

> Is this correct, in people's opinion?

This is correct according to what's specified in our standard.  It's worth
noting that, as 401 is an innovation of this document, the last point
isn't what happens right now in practice; instead, 502 is returned.

> applies (I would make (1) the default, with the others being
> READER_ONLY, TRANSIT_ONLY, and MODE_READER). We'll need to decide which
> commands are "transit" and which are "reader", but that won't hold up
> the rewrite.

I'm not entirely sure that this will really be an improvement, but it may
be.  In any case, let's hold off until we decide what extensions we're
using to advertise the availability of IHAVE, POST, and reader commands,
since that will have a lot of influence on the wording of the MODE READER
description.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 19:05:00 2004

> These two should be 503 ("not supported") rather than 502 ("wrong

502 is existing practice.  503 has the following description:

   If the server recognizes the command but does not provide an optional
   feature (for example because it does not store the required
   information), or only handles a subset of legitimate cases (see the HDR
   command (Section 8.6.1) for an example), the response code 503 MUST be
   returned.  Note that where a command is optional (e.g.  LIST
   ACTIVE.TIMES) and is not provided by a server, this MAY be treated as
   an unimplemented command (response code 500 or 501 as appropriate) or
   as a working command where the information is not available (response
   code 503).

This doesn't sound like the right code to use to me, since we're not
talking about optional commands and we're not talking about a client
request for command functionality that isn't supported.  Actually, a 500
or 501 response makes more sense to me than 503, but that means we're
returning unimplemented status codes for mandatory commands.

Of course, once we have LIST EXTENSIONS labels for them, the commands
aren't clearly mandatory, and I'm not sure if this whole mandatory
vs. optional command distinction is really one to hang on to.  The
distinction is really between commands associated with some advertised
extension (which includes all the reader commands) and commands that
everyone has to implement, period, no matter what capabilities they
advertise (like QUIT and LIST EXTENSIONS).

Right.

> Actually, I agree with Mark here. If we're going to have core
> capabilities described in LIST EXTENSIONS, let's not leave *anything* as
> "suck it and see".

I think we should go ahead and add an IHAVE capability, yes.  Among other
things, some servers offer IHAVE to readers.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 00:24:24 2004

> recall off-hand and which isn't important enough to revisit.)

Um, STAT *is* in that list. Did you mean something else?

>> There are *four* possible types of connection.
[...]
>> Is this correct, in people's opinion?

But is it what we think happens in real life?

> isn't what happens right now in practice; instead, 502 is returned.

Yes, and we comment that as a historical point.

> since that will have a lot of influence on the wording of the MODE READER

Um, no it won't.

My intention is to rewrite various parts of the document to talk about
"reader" and "transit" connections. Some commands will be tagged as reader
commands, which need not be supported on transit connections, and transit
commands, which need not be supported on reader connections. Then we talk
about using different ports, heuristics, both-transit-and-reader servers,
and, finally, MODE READER (if we don't kill it). I already have a fair
amount sketched out in my head.

How we advertise this becomes the least of our concerns. Really.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Nov 10 00:32:45 2004

Russ Allbery said:
>    returned.

> request for command functionality that isn't supported.

I don't follow. On a reader-only server, IHAVE is a documented command that
will be rejected by the server because it only handles a subset of
legitimate cases - it doesn't handle the transit case. Or transit is an
optional feature that isn't provided.

> Actually, a 500

Argh. 500/501 should mean "you've said something nonsensical".

502 doesn't make sense, because it implies that reconnecting can give you
the appropriate authority.

>    Note that where a command is optional (e.g.  LIST

Note that this text has now been deleted, because we don't have any
optional commands any more, just extensions that aren't available.

> vs. optional command distinction is really one to hang on to.

Indeed. On the other hand, I don't like thinking of "reader-mode" or
"transit-mode" as an extension, because after all you need to have one or
the other.

> advertise (like QUIT and LIST EXTENSIONS).

I really really really don't like thinking of the reader commands as an
extension. Apart from anything else, it will make a real mess of the
document and confuse the readers.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 10:25:13 2004

>> too, but we didn't put it into that category for some reason that I
>> don't recall off-hand and which isn't important enough to revisit.)

> Um, STAT *is* in that list. Did you mean something else?

No, sorry, I just missed it and confused myself.

>>> (4) Switching. Initially reader commands cannot be used; they get a
>>> 401 response. Once MODE READER has been issued, transit commands
>>> cannot be used; they get a 502 response. MODE READER destroys all
>>> server state apart from the switch and therefore LIST EXTENSIONS
>>> output can change in this one case.

> But is it what we think happens in real life?

Mostly.  INN's nnrpd can be configured to support IHAVE, however.
(Although I'm not particularly happy with that IHAVE support; it's
intended for the use of broken Microsoft clients that insist on feeding
with IHAVE, so it still adds trace headers.  It may be better to ignore it
as something outside of the standard protocol.  It's not enabled by
default.)

> "reader" and "transit" connections. Some commands will be tagged as
> reader commands, which need not be supported on transit connections, and
> transit commands, which need not be supported on reader
> connections. Then we talk about using different ports, heuristics,
> both-transit-and-reader servers, and, finally, MODE READER (if we don't
> kill it). I already have a fair amount sketched out in my head.

> How we advertise this becomes the least of our concerns. Really.

Oh, okay, I see what you're getting at.

As long as readers can still advertise transit commands and vice versa, I
think that sounds great.  And that would let us introduce 433 in a more
comprehensive fashion and maybe push people a little towards using that in
situations where MODE READER is currently used.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 10:32:55 2004

> I don't follow. On a reader-only server, IHAVE is a documented command
> that will be rejected by the server because it only handles a subset of

I think what's making me leery is that, prior to our document, 503 meant
"something went wrong in processing your command that was probably
transient."  This is a somewhat more permanent error than what people have
been using 503 to mean in the past.

503 is, for example, the return code that INN returns to AUTHINFO if it
can't fork an external authenticator, the return code that it returns to
LIST if the active file can't be found, or the return code that it returns
to DATE if localtime() fails.

> 502 doesn't make sense, because it implies that reconnecting can give
> you the appropriate authority.

I'm not sure why it would imply that.  502 means "permission denied"; it
is certainly entirely reasonable for that state to be permanent for that
client.  For example, a read-only reader server is going to return 502 to
POST from now until the end of time, and reconnecting all you want won't
help.

> "transit-mode" as an extension, because after all you need to have one
> or the other.

This is true.

> document and confuse the readers.

Well, one problem is that we're talking about extensions, whereas the
closest example to what we're actually trying to do here (IMAP) talks
about capabilities.  You're right that it doesn't make any sense to talk
about reader commands as an extension, but it does make sense to talk
about them as a capability.

I wonder if it might not be worth changing LIST EXTENSIONS to LIST
CAPABILITIES.  It's certainly clearer given what we're now using it for,
it may mean that we could skip the whole business of introducing a version
number entirely because no one has implemented LIST CAPABILITIES before
and thereby defer that discussion until the next time we have to change
the core protocol, and it would let us distinguish between an extension
(something added on to NNTP that isn't defined in the core protocol
document) from a capability (something advertised with LIST CAPABILITIES
that may not necessarily be present, which includes extensions and some
other things).

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 10:40:21 2004

Russ Allbery <[email-masked]> writes:

> As long as readers can still advertise transit commands and vice versa,
> I think that sounds great.  And that would let us introduce 433 in a
> more comprehensive fashion and maybe push people a little towards using
> that in situations where MODE READER is currently used.

One thing that I do want to add here, though, is that we don't want to go
down the path of making too many changes, and this is borderline -- it
sort of deals with the LIST EXTENSIONS stuff we're working on, but also
goes a little bit beyond it.

In other words, we have to be careful not to reopen too much, although if
we can make this part clearer, maybe that would be worthwhile.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Nov 10 11:18:57 2004

> that may not necessarily be present, which includes extensions and some

I'd have no problem with this.  In fact I think Mark's originally 
suggestion was that we should create a new command.  In an effort to 
remain compatible we decided to just add an "NNTPv2" capability to LIST 
EXTENSIONS.  Given our inability to agree on the syntax of this new 
"NNTPv2" capability, let's try the LIST CAPABILITIES (or just 
CAPABILITY) track.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 11:21:07 2004

> "NNTPv2" capability, let's try the LIST CAPABILITIES (or just

CAPABILITY is a good point.  The LIST command is already a mess, and while
taking this out of it won't completely clean it up, at least we're not
adding to the problem.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 10 11:23:25 2004

> taking this out of it won't completely clean it up, at least we're not

I'm fine with that idea.

-- Mark --

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Nov 10 11:37:13 2004

> taking this out of it won't completely clean it up, at least we're not

As an aside, what, if any, legacy extensions should a server list via 
LIST EXTENSIONS for backwards compatibility with existing clients?  Or 
is the number of clients which use LIST EXTENSIONS too small to worry 
about, especially if the current LIST EXTENSIONS output can't be trusted?

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Nov 10 11:43:38 2004

> taking this out of it won't completely clean it up, at least we're not

Keeping with the cleanup theme, is there any reason why LIST HEADERS 
can't be changed to something different (e.g. HDR.LIST or HEADERS), 
especially since we're defining it for the first time in our doc (at my 
suggestion)?  Similarly LIST OVERVIEW.FMT could be changed to just 
OVERVIEW.FMT, but for backwards compatibility this may or may not be wise.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 10 12:33:45 2004

On Wed, 10 Nov 2004, Ken Murchison wrote:
> As an aside, what, if any, legacy extensions should a server list via LIST 
> EXTENSIONS for backwards compatibility with existing clients?  Or is the 
> number of clients which use LIST EXTENSIONS too small to worry about, 
> especially if the current LIST EXTENSIONS output can't be trusted?

Pine uses LIST EXTENSIONS, but since the output is untrustworthy there are 
all sorts of bizarre heuristics to decide whether or rely upon it for a 
particular extension or not.

I would be happy to abandon LIST EXTENSIONS support entirely in Pine in 
favor of support for a trustworthy CAPABILITY command.

-- Mark --

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Wed Nov 10 13:08:45 2004

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 Russ> CAPABILITY is a good point.  The LIST command is already a
 Russ> mess, and while taking this out of it won't completely clean it
 Russ> up, at least we're not adding to the problem.

I actually disagree on this point - I prefer to add a new LIST
subcommand than an entire new basic command, both because it's easier
in the code (probably is in the client, too, given that LIST output is
reasonably consistent) and because it's logical.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 10 13:27:05 2004

> reasonably consistent) and because it's logical.

At least in my client code, it makes absolutely no difference.  Each 
flavor of LIST response has to be parsed differently.

It seems bizarre to me that LIST has all these seemingly orthogonal 
functions, but that's just syntactic sugar.

-- Mark --

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 05:20:32 2004

> we can make this part clearer, maybe that would be worthwhile.

This is going to look (I've started on it, so I know) like a major revision
of parts of the document. But in actual fact, it changes very little. The
wording of MODE READER, if you know how to read it, already allows for the
four different server types we discussed. It's just so badly presented that
everyone sits looking confused.

The only substantive changes will be the capability reporting (which is a
separate matter anyway) and *possibly* the elimination of MODE READER,
which is also a separate sub-thread.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 05:25:16 2004

Mark Crispin said:
>> in the code

Huh? Surely you have a basic command switch table somewhere.

>> (probably is in the client, too, given that LIST output is
>> reasonably consistent)

Not really, no. About the only consistent thing is that it's multi-line.

>> and because it's logical.

To paraphrase Mark: how can you use LIST and logical in the same sentence
with a straight face?

> functions, but that's just syntactic sugar.

"seemingly orthogonal" is right. There are all kinds of bizarre
inconsistencies in the detail.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 05:27:39 2004

> about, especially if the current LIST EXTENSIONS output can't be trusted?

My intent, when adding CAPABILITY, is to either remove LIST EXTENSIONS
entirely or, for the benefit of existing clients, to require its output to
be the extensions part of the CAPABILITY output. If that isn't clear,
please wait for the draft.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 05:32:40 2004

Ken Murchison said:
> especially since we're defining it for the first time in our doc

Fine with me. HDR.AVAIL would be my first choice (HEADERS on its own is a
bit confusing), since it lists what's available to HDR, but it's not a
big deal.

> Similarly LIST OVERVIEW.FMT could be changed to just 

This is a bit less clear-cut. It would allow elimination of the legacy
quirk in the specification.

What getting rid of one or both also does is makes it reasonable to say
"no more LIST variants in extensions", limiting LIST to the "LIST keyword"
command in 25.pre-2 and making it more self-consistent.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Thu Nov 11 07:24:31 2004

> Huh? Surely you have a basic command switch table somewhere.

I did not say the above comment.  Please attribute your quotes correctly.

It is of no consequence to me whether capabilities are a subcommand of 
LIST or a new basic command.

> inconsistencies in the detail.

I did say the above comment.

-- Mark --

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 08:22:42 2004

> I did not say the above comment.  Please attribute your quotes correctly.

Count the arrows. They were attributed correctly.

---

From: ken at oceana.com (Ken Murchison)
Date: Thu Nov 11 08:47:00 2004

> Count the arrows. They were attributed correctly.

Nope.  The statement above was made by Andrew.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Thu Nov 11 08:56:45 2004

> Nope.  The statement above was made by Andrew.

Oh, good grief.

In the above:
>>>    is Mark
>>     is Clive
>      is Ken
As far as I and many (most) Usenet posters are concerned, this is a correct
attribution. Stuff with one arrow is the attributed person; stuff with more
than one is material that person quoted.

Similarly, in the original post I attributed Mark because I was quoting his
post, but some of the text in my post began ">>", because it was Mark
quoting Andrew.

Happy now?

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:36 2004

[email-masked] (Clive D.W. Feather)  wrote on 08.11.04 in <[email-masked]>:

> Because we're permitting (in theory) an extension called MODE-READER.

And it's this thing. There's no conflict.

> hyphens in extension labels.

There should not be a syntactic difference here. That way lies madness.

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:39 2004

> capable server include "ESMTP" somewhere in the greeting.

SMTP requires no such thing.

I was there when it was discussed and rejected; nevertheless I had a look  
at RFC 2821 to make sure it's not in there.

> POP3 added the APOP authentication token in the greeting.

One of the ugliest mechanisms I ever saw ...

> IMAP added status response codes, among which are capabilities in the

I'll not talk about that incredibly overcomplicated protocol.

> problems are not substantial enough to be of concern.

Or maybe some of them *did* see it.

> insisting upon reinventing wheels that are already long-invented and

That is well-balanced by a consistent trend of the mail community to  
refuse to even consider where circumstances for news are plainly  
different, and the mail solution has serious problems for news.

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:42 2004

> please wait for the draft.

But please use the plural: CAPABILITIES.

MfG Kai

---

From: kaih at khms.westfalen.de (Kai Henningsen)
Date: Sun Nov 14 11:58:45 2004

[email-masked] (Clive D.W. Feather)  wrote on 11.11.04 in <[email-masked]>:

> attribution.

Nope.

The *correct* Usenet-style attribution involves not snipping out the names  
of the other authors in the first place; that's just rude, and you seem to  
do it consistently.

See how my reply starts with *two* attribution lines, and yours started  
with just one.

That's where all the trouble comes from.

>Stuff with one arrow is the attributed person; stuff with more

*This* part is correct. It's necessary, but not sufficient.

> post, but some of the text in my post began ">>", because it was Mark

And you were at fault for snipping the line attributing it to Andrew.

> Happy now?

Not really.

MfG Kai