Subject: [NNTP] NNTP working group status

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:08:25 2004

We've been doing a lot of brainstorming here for a while, and I've been
letting it run because I think we've been getting a lot of productive work
done, but I'm overdue for sending out some sort of status message saying
where we're at in terms of the overall goals of the working group.

I'm also overdue on a status update to Scott on where we are, so this
message will double as that.

First of all, where we are formally:

 * The base protocol document (version -23, I believe) passed IETF last
   call and has now passed IESG approval.  If we believed it was the right
   thing to do, it's actually at the stage where we would send it to the
   RFC editor for publication.  I don't think that's the right thing to
   do (see below), but I wanted everyone to know what the actual status of
   it is.

 * We have three other documents in progress.  The AUTHINFO and STARTTLS
   documents are, I believe, done changing, except for the outcome of the
   current discussions (more on that in a moment).  Except for the whole
   LIST EXTENSIONS issue that we're debating now, they would be ready for
   working group and then IETF Last Call.

 * I believe that the STREAMING document is essentially done, although
   since it hasn't baked as much as the other two, it's more likely to
   turn up some additional comments in a working group last call.

 * I think it is to our advantage to publish all four of these documents
   around the same time, since together they comprise pretty much all of
   the significant and useful extensions that are currently in use by
   deployed NNTP clients and servers.

 * We are significantly overdue on all of our outstanding milestones,
   many of which have already been moved multiple times.

Now, here's why we're not currently just pushing forward with publishing
the work we've already done, which was fully my intention as of a month
ago:

Due to a wide host of reasons, including massively extended working group
life, the tendency for NNTP to work in a vacuum anyway, and lack of broad
working group participation, we've been working in a bit of a vacuum for a
while.  I was pretty much resigned to this, and resigned to publishing
something that not everyone would like, on the grounds that it was better
to have *some* published standard than basically nothing (which is what we
have right now).  In the course of reviewing a detail of the AUTHINFO
draft, however, Mark Crispin made several excellent comments about the
NNTP draft that seemed worth considering.

The issues that we reopened discussion on in the mailing list were:

 * There are some completely broken implementations of LIST EXTENSIONS and
   some implementations that don't list extensions that they actually
   provide.  From a client perspective, it would be nice to have a clear
   indication of whether LIST EXTENSIONS can be trusted.  IMAP solved a
   similar problem by introducing a capability that stated the protocol
   version number, and it was proposed that NNTP do the same (adding an
   NNTPv2 extension, or something similar).

 * The NNTP specification still allowed for optional commands, where one
   wouldn't know if a command was available without trying to execute it.
   This is not the model that other Internet protocols with extension
   mechanisms use, so it was proposed that we go back and add clearly
   advertised extensions for all of the optional commands, indicating
   whether they were available.

 * There was no indication whether a client had to issue MODE READER at
   the beginning of a session, and it's normally not needed.  It was
   proposed that we add some extension indicating whether MODE READER is
   required, since normally it won't be.

In the process of that discussion, we've also talked about other things (a
more expressive alternative to LIST EXTENSIONS, whether we can eliminate
MODE READER entirely, whether we can provide a short list of extensions as
part of the opening banner in order to reduce round trips for clients).

Now, I view the role of a working group chair to be somewhat similar to
the role of a release manager, in that I have to try to balance the
competing priorities of making the standard as good as we can and getting
something out the door.  While we have a lot of energy and activity right
now, I mistrust (based on past experience) our ability to maintain that
energy level for very long, and therefore think we need to get documents
out the door before we lose energy again.  We have to strike a balance
between getting the documents just right and getting them published so
that people can actually start benefiting from them.

I do think that if we can get an extension mechanism that would make Mark
happy, that is a significant improvement to the document and is worth
delaying it for another three months (which is probably going to be the
minimum delay if we go back, change it in a fundamental way, and put it
forward again for Last Call, which I assume would be the procedure).
Mark's points about nailing down exactly what the server will let one do
in the LIST EXTENSIONS response are very well-taken, as is the point about
adding some sort of protocol token to avoid known-bad implementations.

Accordingly, I think it's right to ask Scott if he also feels like we
should reopen discussion on LIST EXTENSIONS (more formally than we have so
far) and make modifications to the standard in order to meet the above
three goals.  This would likely also result in some changes to the LIST
command to try to resolve the issues with the current draft where some of
the commands are optional and there's no advertised way of knowing whether
they will work.  It will also probably mean dropping LIST DISTRIBUTIONS
from the standard entirely and leaving it as a separate extension that we
don't document in this initial set of documents, and making LIST
NEWSGROUPS mandatory.

However, I think that if we're going to go back and revisit things we've
already decided, even for very good reasons, we need to sharply limit the
scope of that discussion so that we can complete it and move forward.
This means that we're not going to fix everything that could be fixed, and
it means that the next version of the standard is likely to require more
work, but it does mean that we'll get the standard out the door faster so
that people can start using it.

Accordingly, I would strongly prefer to reopen discussion *only* on LIST
EXTENSIONS and what should be contained in it and on the LIST command and
what LIST subcommands we choose to document (so that we can fix up the
issues related to optional commands there).  This means leaving MODE
READER alone for now, which includes the following admonition:

   Servers are encouraged to not require this command even though clients
   SHOULD send it when appropriate.  It is present to support some news
   architectures that switch between modes based on whether a given
   connection is a peer-to-peer connection with another server or a news
   reading client.

but not removing the command from the standard entirely.  I do think we
can document 433 as an alternate port for NNTP servers that can be used
only by peers, which is one way in which some servers have avoided MODE
READER, but I don't really want to have an extended discussion about that
either.

Now, in the interests of full disclosure, please note that I maintain a
free news server (INN) which is essentially the only software package that
requires MODE READER at present.  I didn't add it, don't particularly like
it, and configure my own servers to not require it, but there is certainly
the possibility of bias in my own feelings on the subject.  I'm
uncomfortable making a working group chair decision on this because of
that, and I want to make sure Scott knows about that and also say that I'm
quite happy for some third party to make the decision about whether to
reopen discussion of or remove MODE READER from the NNTP standard, and
I'll happily abide by the decision of that person.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 18:25:17 2004

> I'll happily abide by the decision of that person.

I admit that this is twisting the knife (I really do owe you that 
sixpack), but there *is* a way that you can make the decision for the 
entire working group without anyone criticizing your actions.

That would be if you were to declare that the NNTPv2 version of INN will 
never require MODE READER no matter how it is configured.  This would 
remove the need for MODE READER, and move it into the "compatibility with 
pre-NNTPv2 servers" document.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Nov 9 18:37:26 2004

> remove the need for MODE READER, and move it into the "compatibility
> with pre-NNTPv2 servers" document.

Well, I personally agree with the messages that Andrew has sent on the
topic -- this isn't just a question of a particular implementation choice,
but rather is fairly fundamental to the way that NNTP works.  Diablo
doesn't require MODE READER just because Diablo doesn't even allow the
feeder and reader components to run on the same *system* without a lot of
careful work.

That being said, and regardless of the merits of that argument, the amount
of time that I think it would require to implement something exceeds the
amount of time I have available to spend on INN in the forseeable future.
I wish that weren't the case, but right now INN is a hobby rather than a
job and I just don't have the resources to devote to trying to figure out
a good solution.

Certainly if someone came up with some killer patches that just made it
all work properly, that would be great and I'd happily accept them, but
I think it's unlikely for the reasons that Andrew stated.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Nov 9 19:17:45 2004

On Tue, 9 Nov 2004, Russ Allbery wrote:
> but rather is fairly fundamental to the way that NNTP works.

The only "fundamental to NNTP" is RFC 977.  Everything else is folklore.

If I wanted to be a real jerk, I could say that MODE READER violates RFC 
977 because it adds a client requirement that is not in RFC 977; and 
therefore the inclusion of MODE READER to the new NNTP base document 
violates the charter.

I'm a jerk, but not that much of a jerk.

On the other hand, the conclusion seems right.

> job and I just don't have the resources to devote to trying to figure out

If "death to MODE READER in NNTPv2" is agreed, perhaps Ken and I might be 
talkable into offering some cycles to beat INN into submission for NNTPv2. 
I don't know if we can promise that the result will be pretty.

-- Mark --

---

From: ned.freed at mrochek.com ([email-masked])
Date: Tue Nov 9 20:02:41 2004

>    do (see below), but I wanted everyone to know what the actual status of

As you say, we've waited this long, if waiting a bit longer will result
in substantial improvement, I'm for it.

>    working group and then IETF Last Call.

I agree with this assessment, although I have yet to review the STARTTLS
draft to see if it meets all of the newer things the security folks want
out of TLS in more recent protocols.

>    turn up some additional comments in a working group last call.

Sure.

>    deployed NNTP clients and servers.

As a purely practical point, if all of them get to the RFC Editor within a
reasonably close period of time the RFC Editor is likely to lump them together
in terms of the RFC numbers they get. They're smart this way... Although if we
got a move on we might even be able to get RFC 3977 as the number for the
revised NNTP specification ;-)

>    many of which have already been moved multiple times.

Yes, well, I suppose its not PC to say, but we're far from the only group with
this problem!

> ...

> that people can actually start benefiting from them.

I'm nominally the process weenie here, but FWIW I agree.

> forward again for Last Call, which I assume would be the procedure).

That's correct, although it will probably end up being mostly a formality.

> don't document in this initial set of documents, and making LIST

FWIW, I have no problem with pursuing this course. I note, however, that
I have precious little skin in this particular game, and others that
do may feel differently. (And if so should chime in now...)

> that people can start using it.

Absolutely right.

> I'll happily abide by the decision of that person.

All things being equal I'd prefer to only reopen MODE READER if there
was obviously potential benefit from doing so. I see the potential
benefit for LIST EXTENSIONS, but not for MODE READER.

				Ned

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 20:54:13 2004

Mark Crispin wrote:

> If "death to MODE READER in NNTPv2" is agreed, perhaps Ken and I might 
> be talkable into offering some cycles to beat INN into submission for 
> NNTPv2. I don't know if we can promise that the result will be pretty.

Since I spent a few days hacking AUTHINFO SASL support into INN, why not 
spend a few more killing MODE READER?  If that's what we decide and we 
have a viable plan for INN, count me in.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Nov 9 20:58:14 2004

> out of TLS in more recent protocols.

Ned, do you have a reference to these "newer things" so I can get a jump 
on them?

> revised NNTP specification ;-)

This has already been suggested and I believe Russ was looking into it.

---

From: ned.freed at mrochek.com ([email-masked])
Date: Tue Nov 9 21:36:38 2004

> Ned, do you have a reference to these "newer things" so I can get a jump

They came up in the MSGTRK group and ended up in RFC 3887. Basically what was
called for was a parameter on STARTTLS specifying the domain name of the server
the client believes it is talking to. In the words of RFC 3887:

   The parameter MUST be a fully qualified domain name (FQDN).  A client
   MUST specify the hostname it believes it is speaking with so that the
   server may respond with the proper TLS certificate.  This is useful
   for virtual servers that provide message tracking for multiple
   domains (i.e., virtual hosting).

This is really more of a matter for draft-ietf-nntpext-tls-nntp-03.txt,
but as there are some STARTTLS examples in draft-ietf-nntpext-tls-nntp-03.txt
there is some impact there as well.

> This has already been suggested and I believe Russ was looking into it.

Cool, I must have missed it.

				Ned

---

From: fantec at proxad.net (Francois Petillon)
Date: Wed Nov 10 02:49:26 2004

> feeder and reader components to run on the same *system* without a lot of

I do not know if Diablo may be a good example here. Reader side receive 
a feed from specified IPs. From these IPs, you can not access to mode 
reader and you are only allowed to feed the server : there is no way to 
switch from one mode to the other (and in fact, I have never see any 
complaints about it).

   Fran?ois

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Wed Nov 10 04:29:44 2004

>>>>> "Francois" == Francois Petillon <[email-masked]> writes:

 Francois> I do not know if Diablo may be a good example here. Reader
 Francois> side receive a feed from specified IPs. From these IPs, you
 Francois> can not access to mode reader and you are only allowed to
 Francois> feed the server : there is no way to switch from one mode
 Francois> to the other (and in fact, I have never see any complaints
 Francois> about it).

You're not reading the code closely enough, I think.

While it's even less typical in a Diablo environment for a single IP to
have both reader and (header-only) feeder access, it is possible. The
only effect of MODE READER is to disable the feeder commands (and thus
is never actually necessary).

I can't think of any case other than running both the spool and the
reader on the same host where this would ever be an issue, though.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: sah at 428cobrajet.net (Scott Hollenbeck)
Date: Wed Nov 10 04:35:47 2004

>    do (see below), but I wanted everyone to know what the 

It hasn't yet been approved by the IESG.  It's going to be held until
progress is made on the tls and authinfo documents.

[snip]

> don't document in this initial set of documents, and making LIST

Well, if I'm going to hold the document I don't mind if it's revised again.
I agree, though, with the need to limit the scope.  It will probably be
necessary to run the document through the IESG again if substantive changes
are made.

-Scott-

---

From: fantec at proxad.net (Francois Petillon)
Date: Wed Nov 10 06:21:09 2004

> You're not reading the code closely enough, I think.

Grmpf, you are right :-)
(in fact, I have not read this part of the code _yet_ ;-)

> is never actually necessary).

When you are allowed to feed & read, you need to issue a MODE READER to 
access reader commands (thus, just like Inn). Feed-only connections are 
managed by a dedicated process (unlike reader-allowed connections). At 
first sight, the differences between both modes are related to a flag 
and allowing a feeder/reader to use reader commands should not be 
difficult (_at_first_sight_).

   Fran?ois

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Wed Nov 10 06:41:04 2004

>> While it's even less typical in a Diablo environment for a single
 >> IP to have both reader and (header-only) feeder access, it is
 >> possible. The only effect of MODE READER is to disable the feeder
 >> commands (and thus is never actually necessary).

 Francois> When you are allowed to feed & read, you need to issue a
 Francois> MODE READER to access reader commands (thus, just like
 Francois> Inn). Feed-only connections are managed by a dedicated
 Francois> process (unlike reader-allowed connections). At first
 Francois> sight, the differences between both modes are related to a
 Francois> flag and allowing a feeder/reader to use reader commands
 Francois> should not be difficult (_at_first_sight_).

The main point with diablo, though, is that unlike INN, it never
actually makes sense to give feeder access on the reader port to
anything except the appropriate spool backend server.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Nov 10 06:59:07 2004

>   domains (i.e., virtual hosting).

Doh!  My co-author originally had such a parameter for NNTP STARTTLS, 
but I told him that it was unnecessary given the "Server Name 
Indication" TLS extension (RFC 3546, section 3.1).  We mention 
implementation of this as a SHOULD in the draft.  Is this not 
sufficient?  It seems to make more sense to me to handle TLS specific 
stuff within TLS itself, but I could be wrong.  I'm curious why MSGTRK 
was forced to do this at the application level rather than within TLS.

One problem with the wording in RFC 3887 as far as NNTP is concerned is 
that there already is existing deployment of STARTTLS without such an 
argument.  At the very least, if we add a domain argument to STARTTLS it 
would probably have to be optional for backwards compatibility.

I realize you're probably busy at IETF, but would you be able to query a 
couple of security guys on this and let me know if we are going to have 
to change the doc?  I'd prefer to do it before we send it off to IESG 
and ultimiately have it bounced.

Thanks,
Ken

---

From: ned.freed at mrochek.com ([email-masked])
Date: Wed Nov 10 07:25:20 2004

> sufficient?

I honestly don't know.

> It seems to make more sense to me to handle TLS specific

I agree. I don't recall a justification being given for MSGTRK doing it
this way.

> and ultimiately have it bounced.

I agree, however, since I'm not in DC someone else will have to do it.
Any volunteers? If memory serves the pushback on this in MSGTRK came
from Eric Rescorla so you might try him.

				Ned

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Nov 10 10:00:32 2004

Scott Hollenbeck <[email-masked]> writes:

>>  call and has now passed IESG approval.  If we believed it was the
>>  right thing to do, it's actually at the stage where we would send it
>>  to the RFC editor for publication.  I don't think that's the right
>>  thing to do (see below), but I wanted everyone to know what the actual
>>  status of it is.

> progress is made on the tls and authinfo documents.

Whoops, sorry, I misunderstood or used the wrong terminology for something
there.

> Well, if I'm going to hold the document I don't mind if it's revised
> again.  I agree, though, with the need to limit the scope.  It will
> probably be necessary to run the document through the IESG again if
> substantive changes are made.

*nod*.  That was what I was expecting.  Thanks!