Subject: [NNTP] Additions to LIST commands

From: Julien ÉLIE <[email-masked]>
Date: Tue, 17 Nov 2009 20:12:27 +0100

Hi,

I have just posted the following proposal for:

    * LIST DISTRIBUTIONS
    * LIST MODERATORS
    * LIST MOTD
    * LIST SUBSCRIPTIONS
    * new status for newsgroups returned by LIST ACTIVE

Comments are welcome!

Especially on:

    * how to advertise the meaning of new status;
    * how to write an extension of RFC 3977 without defining
      a new capability;
    * how to deal with the fact that RFC 3696 limits the local part
      of an e-mail address to 64 characters whereas USEAGE mentions
      that a newsgroup name is limited to 66 characters (therefore,
      what for the submission address of a moderated newsgroup whose
      length is 66 characters?).

Follow up on either the NNTP mailing-list or the USEFOR mailing-list,
depending on the subject you want to tackle.

> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
>
>
> Title : Network News Transfer Protocol (NNTP) Additions to LIST Command
> Author(s) : J. Elie
> Filename : draft-elie-nntp-list-additions-00.txt
> Pages : 21
> Date : 2009-11-17
>
> This document defines a set of enhancements to the Network News
>   Transfer Protocol (NNTP) that allows a client to request extended
>   information maintained by NNTP servers as for local use and
>   distribution policy.  These enhancements are made as new keywords to
>   the existing LIST capability described in RFC 3977.
>
>   This memo updates and formalizes the LIST DISTRIBUTIONS and LIST
>   SUBSCRIPTIONS commands defined in RFC 2980.  It also adds the LIST
>   MODERATORS and LIST MOTD commands, and specifies additional values
>   returned by the existing LIST ACTIVE command for the status of a
>   newsgroup.
>
>
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-elie-nntp-list-additions-00.txt

-- 
Julien ÉLIE

« Un seul être vous manque et tout est dépeuplé. » (Alphonse de Lamartine)

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Tue, 17 Nov 2009 21:51:02 +0200

> Comments are welcome!

    "j"  Articles are filed under the "junk" newsgroup.

Is there any reason to constrain the implementation so much?  It seems to me
that "j" could easily be implemented as "accept posts but do not file them",
without the use of, or existence of, a "junk" newsgroup.   Would that break
existing clients, or common usage? 

-- 
Antti-Juhani Kaijanaho, Jyväskylä, Finland
http://antti-juhani.kaijanaho.fi/newblog/
http://www.flickr.com/photos/antti-juhani/

---

From: Russ Allbery <[email-masked]>
Date: Tue, 17 Nov 2009 11:57:18 -0800

> existing clients, or common usage? 

The junk group is sometimes readable and sometimes not.  There is the
problem that if you want it to be readable, j currently tells the client
where the article can be found, whereas if it's not filed at all or in
some other newsgroup, the client can't find it that way.  But I think the
idea of reading the junk group via a regular GROUP junk command is a bit
dubious.  I'm not sure how many people would do that.

Currently in the INN implementation, j is roughly equivalent to =junk.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Julien ÉLIE <[email-masked]>
Date: Tue, 17 Nov 2009 21:20:20 +0100

Hi Russ,

> But I think the

At least on fr.*, a few people like to add "Followup-To: junk" on certain
occasions; and read (and sometimes respond) to answers which kept that
follow up.

> Currently in the INN implementation, j is roughly equivalent to =junk.

"roughly" :)

As I wrote in news.software.nntp, if:

 - group1 is status "j",
 - group2 is status "=junk",
 - group3 is status "y",

then :

 * a local post to group1 only will be accepted (and filed into "junk")
   whereas a local post to group2 only will not be accepted;

 * a crosspost to group1,group3 will be filed into only group3 whereas
   a crosspost to group2,group3 will be rejected if it is a local post
   and filed into "junk" and group3 if it comes from a peer.

-- 
Julien ÉLIE

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Tue, 17 Nov 2009 22:33:08 +0200

> dubious.  I'm not sure how many people would do that.

Sounds like a SHOULD, or even a MAY.  Certainly, the relation to the junk group
should be documented.

---

From: Julien ÉLIE <[email-masked]>
Date: Tue, 17 Nov 2009 21:42:44 +0100

Hu Antti-Juhani,

> Is there any reason to constrain the implementation so much?

I just documented what INN has always done.
Diablo and C News, to cite only them, do not use "j".

> It seems to me

So your suggestion is to decorrelate "j" with the existence of anything
else?  I believe it is fine to do that (and what the post become is
an implementation issue -- either it is stored nowhere or somewhere,
like the junk newsgroup).

-- 
Julien ÉLIE

« I don't know if it's what you want, but it's what you get. » (Larry Wall)

---

From: Julien ÉLIE <[email-masked]>
Date: Tue, 17 Nov 2009 21:46:22 +0100

Hi Antti-Juhani,

> Sounds like a SHOULD, or even a MAY.  Certainly, the relation to the junk group

What do you want to document exactly?
What should be a SHOULD?

Isn't the note I put in the Internet-Draft what you were looking for?
Isn't it clear enough?  (If no, what should be changed?)

      NOTE:  The status "j" is used only by news servers on which the
      newsgroup "junk" exists.  This group has special meaning.  It
      usually contains all the postings which cannot file under a
      newsgroup name.  Instead of rejecting an article which contains an
      invalid Newsgroups header or which is posted to newsgroups it does
      not carry, a news server may accept such an article and file it
      under a generic newsgroup named "junk".  This newsgroup may be
      available to news readers and is often used by a news server as a
      way to locally store an article with the view to transmitting it
      to its peers (which may carry some of the newsgroups the article
      was posted to).

      Depending on the configuration of the news server, mentioning a
      newsgroup with status "j" is different than simply not listing the
      group, since articles arriving in unknown newsgroups MAY be
      rejected.

-- 
Julien ÉLIE

« Ad iura renunciata, non datur regressus. »

---

From: Russ Allbery <[email-masked]>
Date: Tue, 17 Nov 2009 17:27:38 -0800

>      length is 66 characters?).

This looks very good.  A few comments:

      NOTE:  Periods are changed to dashes, and dashes are left alone.
      It implies that two moderated newsgroups whose names differ only
      by changing a period to a dash would go to the same address.
      Therefore, such moderated newsgroup pairs SHOULD NOT be created on
      a news server.

I think this is too strong, since this problem only occurs if %s patterns
are used for determining the moderation address.  That's the case for
world-wide hierarchies, but often not the case for local hierarchies.
Also, this draft is probably the wrong location for a general SHOULD about
moderated group naming.

I would instead say that if such moderated newsgroup pairs exist with
different submission addresses, a %s pattern rule cannot be used for the
moderation submission addresses for those groups and explicit entries
without a pattern will be required.

I wouldn't worry about +; any special interpretation is applied by the MTA
or MDA of the receiving site, and if they want to host a submission
address for a moderated group determined by a %s pattern rule, they'll
need to suppress that interpretation.

      Similarly, a news server SHOULD NOT carry two moderated newsgroups
      whose names differ only by the case of their characters.

USEFOR already covers this:

   A <component> SHOULD NOT consist solely of digits and SHOULD NOT
   contain uppercase letters.  Such <component>s MAY be used only to
   refer to existing groups that do not conform to this naming scheme,
   but MUST NOT be used otherwise.

so you probably want to reference USEFOR here (and here also this only
applies to groups using %s pattern rules).

For the new LIST ACTIVE flags, no CAPABILITIES line should be required,
since the client doesn't need to know whether or not the server might send
these flags before the server sends them.  That was intentional in how we
defined LIST ACTIVE in RFC 3977:

   Other values for the status may exist; the definition of these other
   values and the circumstances under which they are returned may be
   specified in an extension or may be private to the server.  A client
   SHOULD treat an unrecognized status as giving no information.

Since client behavior when seeing unknown flags is specified, there's no
need to do capability negotiation with them or to have a way to advertise
which flags might be used.  When there's an available specification, the
server can just start sending those flags; clients that don't implement
the specification are required to assume nothing about the group based on
the flag they see.

Both your "l" and "r" status flag proposals seem reasonable, but on the
other hand I don't recall anyone specifically asking for either feature,
so I'm not sure I'd go to the effort unless someone wants to use it.  "r"
does seem quite useful, though.

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Tue, 17 Nov 2009 22:01:36 -0500 (EST)

>    SHOULD treat an unrecognized status as giving no information.

Although, what if there are two separate specifications that define a 
given flag differently?  In that case, the client will not know which 
meaning the server intends.

So perhaps it's worth having the server advertise what addon 
specifications it implements?

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Tue, 17 Nov 2009 19:04:42 -0800

>> since the client doesn't need to know whether or not the server might
>> send these flags before the server sends them.  That was intentional in
>> how we defined LIST ACTIVE in RFC 3977:

> specifications it implements?

Hm, yeah, that's true, but it feels like it's rather unlikely that anyone
is going to implement those flags in a way substantially different than
INN did, since they've been around forever with basically the same
meaning.  If there were existing use with different meanings, we'd need
something to clarify what the meanings are, but I think any time you see
them they're going to have basically the same meaning.

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Wed, 18 Nov 2009 07:45:35 +0200

> What do you want to document exactly?

I think it is a good idea to document that if "j" posts are filed anywhere,
they are filed in "junk", since that is what the current servers are doing.

> What should be a SHOULD?

That messages posted to "j" are filed to the "junk" newsgroup.

I'm looking at this from the point of view of server implementor pondering
whether I should implement "j", and the "file to 'junk'" sounds to me like an
implementation detail that should not be mandated by a (proposed) standard.

The main question is, and I tried to ask this in my original message,
whether the essential property of "j" is "not filed here" or "filed in junk".
If clients and users expect the latter, then messages to "j" MUST or SHOULD
be filed to "junk"; if the former, then messages to "j" MAY be filed to "junk".

> Isn't the note I put in the Internet-Draft what you were looking for?

No.  It currently says, without actually using the keyword, that "j" posts MUST
be filed in "junk".  I'm questioning the validity of that absolute requirement.

>      newsgroup "junk" exists.

This implies that a server MUST not implement "j" if it does not have "junk".

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Wed, 18 Nov 2009 07:51:28 +0200

> Diablo and C News, to cite only them, do not use "j".

It is fine from the point of view of client authors to document what the
existing big server implementors do.  However, the question is, whether you
really want to constrain *new* server implementors in this particular way.

> else?

I would prefer to allow a server to use "j" to indicate "I will accept messages
to this newsgroup but I will not file them anywhere".  That many servers file
them to "junk" should be documented for the benefit of client authors and
users (and perhaps made a SHOULD, if clients expect it), but I don't think it
should be an implementation requirement for servers.

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 20 Nov 2009 18:55:02 +0100

Hi Russ,

> without a pattern will be required.

Yes, you're right.  Thanks.

> need to suppress that interpretation.

OK.

> applies to groups using %s pattern rules).

OK.

> For the new LIST ACTIVE flags, no CAPABILITIES line should be required

But what for that MUST in RFC 3977?

   "Each extension MUST define at least one new capability label (this
   will often, but need not, be the name of one of these new commands)."

> does seem quite useful, though.

While we're at it, it does not take much time to define them (especially "l").
One could have the need for it in the future.

I do not know whether "l" can be used with INN because innd injects the
article and it does not always have the information about the source
(depending on the presence of Unix domain sockets, or a different
nnrpdposthost) to decide if it is local.  (?)
Anyway, other news servers may know that better.

As for "r", is it OK to have either "r" or "r1258739643" (time since epoch)?

-- 
Julien ÉLIE

« Les idées fausses ne sont pas toujours mauvaises. » (Marcel Aymé)

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 20 Nov 2009 19:09:35 +0100

Hi Antti-Juhani,

> implementation detail that should not be mandated by a (proposed) standard.

OK, I got it.

> should be an implementation requirement for servers.

I think what you suggest for "j" is better.  Let's mention that no messages
are filed into a newsgroup with status "j".
The note would then document "junk" (explaining why the status is called "j"),
saying that if a news server files somewhere the articles, it SHOULD be into
"junk".  And a news server MAY not file the articles at all.

Is it fine with these modals?

-- 
Julien ÉLIE

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Fri, 20 Nov 2009 20:11:15 +0200

> Is it fine with these modals?

Sounds good.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 20 Nov 2009 10:50:38 -0800

>   will often, but need not, be the name of one of these new commands)."

Oh, good point.  I'm not sure that defining the meaning of those flags
quite counts as an extension in the way that we were thinking of, but I
suppose that we could do something like:

    ACTIVE FLAG-J FLAG-N FLAG-ALIAS

as an extension label, which would expand to any other new features of the
active file that one wants to specify later.

> nnrpdposthost) to decide if it is local.  (?)  Anyway, other news
> servers may know that better.

It would be a bit tricky to implement in INN, yes.

> As for "r", is it OK to have either "r" or "r1258739643" (time since

That would be kind of a neat feature, but I'm not sure that encoding it in
the active file is the best place to return that information.  I'd be more
tempted to add a new list, like LIST REMOVALS [wildmat], that shows
upcoming groups to be removed to avoid putting too much different data
into the active file.

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 20 Nov 2009 20:27:42 +0100

Hi Russ,

>    ACTIVE FLAG-J FLAG-X FLAG-ALIAS
>

The problem is that if something like that is done, it makes more difficult
a possible integration of these flags into a future update to RFC 3977
(VERSION 3 of the protocol).
And it doesn't seem homogeneous to have all these keywords if we do not
also have FLAG-Y, FLAG-N and FLAG-M.

Besides, it would need to have two separate documents:  one for the ACTIVE
extension, and one for the additions to LIST.

Last but not least, we will have the same problem for an extension of wildmats
and metadata items...
It is very unclear how to extend them too.

> It would be a bit tricky to implement in INN, yes.

Unless we say, in the documentation of INN, that "l" works only
if the server has Unix domain sockets.  And, for other cases (or to
handle inews, nnrpdposthost...), we could add a parameter in incoming.conf
(localpost: yes) to allow postings to "l" groups for the connection
which has localpost set to yes.  One could add here "localhost" or the IP
of his server.

> upcoming groups to be removed to avoid putting too much different data

So we would have "r" in the active file.
And LIST REMOVALS would return the relevant information as for the removal.
That seems great!  Thanks for the idea.

(I would suggest LIST ACTIVE.REMOVALS but LIST REMOVALS seems better.)

-- 
Julien ÉLIE

« Rien n'est plus agaçant que de ne pas se rappeler ce dont on ne parvient
  pas à se souvenir et rien n'est plus énervant que de se souvenir de ce qu'on
  voudrait parvenir à oublier. » (Pierre Dac)

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 09:53:14 +0100

Hi Russ,

> upcoming groups to be removed to avoid putting too much different data

On second thoughts, I do not see well the use readers will have
of LIST REMOVALS.
Maybe it should be internally handled.

That is to say, when a rmgroup control article arrives, the newsgroup
is marked as "x" (no local posting, no article from peer).  And the news
server has a file (probably named removals) which keeps track of when
the newsgroup should really be deleted.

This way, if a newgroup control article comes, the newsgroup is marked
as "y" and no longer listed in removals.

Thus, no need to have an "r" status and LIST REMOVALS.
It can be handled internally by a news server (and maybe mentioned in USEAGE
as best practice).

Unless readers have a need for that information?  Please speak up if
that's the case.  Thanks,

-- 
Julien ÉLIE

« Et cette même nuit, c'est-à-dire trois semaines plus tard... » (Astérix)

---

From: Russ Allbery <[email-masked]>
Date: Sat, 28 Nov 2009 01:32:54 -0800

>> That would be kind of a neat feature, but I'm not sure that encoding it
>> in the active file is the best place to return that information.  I'd
>> be more tempted to add a new list, like LIST REMOVALS [wildmat], that
>> shows upcoming groups to be removed to avoid putting too much different
>> data into the active file.

> On second thoughts, I do not see well the use readers will have of LIST
> REMOVALS.  Maybe it should be internally handled.

> That is to say, when a rmgroup control article arrives, the newsgroup is
> marked as "x" (no local posting, no article from peer).  And the news

The only additional feature that I can think of LIST REMOVALS providing is
letting readers know when that 'x' group is going to disappear completely,
so that they can, for example, archive postings they want to keep.

But unless someone asks for it, I'm not sure it's worth writing a protocol
specification for it.

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Sat, 28 Nov 2009 11:34:46 +0200

> Unless readers have a need for that information?  Please speak up if

I suspect any group suddenly changing from y to x is going to cause questions,
which the server admin will have to answer in some out-of-band way.  Better to
provide the information in-band (assuming clients start telling users that "r"
happened and what it means).

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 11:12:00 +0100

Hi Antti-Juhani,

> happened and what it means).

OK, I understand your point.

I'm wondering why we do not have something like LIST EXPIRE which could tell
readers how long articles are kept by the news server in its newsgroups...

Other currently out-of-band information is the list of hierarchies for which
control articles are (supposed to be) honoured, the list of NoCeM issuers whose
notices are (supposed to be) honoured...

But maybe no one cares about having these lists in-band?
LIST EXPIRE seems to have the same interest as LIST REMOVALS, doesn't it?

-- 
Julien ÉLIE

« En voyant le lit vide, il le devint. » (Ponson du Terrail)

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Sat, 28 Nov 2009 12:27:01 +0200

> OK, I understand your point.

It occurs to me now that much more useful for that purpose is just to have a
customized message along with the POST rejection.  LIST ACTIVE is not going to
be commonly checked, unless there's some what I'm forgetting for the client to
request updates in the style of NEWGROUPS.

Thus, my argument for "r" vanishes.

Perhaps there should be a way for a server to provide in-band a group-specific
motd message, but this can be simulated fairly easily by having the admin post
a local-only message to the newsgroup.

And in any case, I'm a bit wary about adding unimplemented features in
standards, even though a Proposed Standard is allowed them.

> LIST EXPIRE seems to have the same interest as LIST REMOVALS, doesn't it?

These differ from removals in that the data is stable - you look it up once,
and you're likely not going to get into trouble for using outdated info.  In
contrast, a delayed removal (y->x before actual removal) is going to be a
sudden and (in general) unforeseen  event, which, when it happens, will raise
questions.  But my comment above applies as to what's the best way of providing
the answers.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 28 Nov 2009 02:44:31 -0800

Antti-Juhani Kaijanaho <[email-masked]> writes:

> It occurs to me now that much more useful for that purpose is just to
> have a customized message along with the POST rejection.  LIST ACTIVE is
> not going to be commonly checked, unless there's some what I'm
> forgetting for the client to request updates in the style of NEWGROUPS.

The drawback to reporting this with a POST rejection is that by the time
the client gets a POST rejection, the user has already spent a bunch of
time composing the post.

> standards, even though a Proposed Standard is allowed them.

This is a good point.  It might be best to stick with just the things INN
has already implemented for the Proposed Standard document and add
anything new in a separate document.  It's more work and more writing, but
it's the more conservative standardization approach.  The other document
can be on a different track (experimental, for instance) until the new
ideas prove themselves useful.

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 11:52:08 +0100

Hi Russ,

> anything new in a separate document.

And what for LIST COUNTS?  It is implemented in Highwinds products
(e.g. Cyclone, Tornado, Typhoon, Twister -- when they support readers).

Should it be put in the Proposed Standard document?

It behaves like LIST ACTIVE except that it also returns the
number of articles in every newsgroup.

LIST ACTIVE xs4all.*test
215 NewsGroups Follow
xs4all.be.test 0000000274 0000000275 y
xs4all.test 0000003629 0000003623 y
.
LIST COUNTS xs4all.*test
215 NewsGroups Follow
xs4all.be.test 0000000274 0000000275 0000000000 y
xs4all.test 0000003629 0000003623 0000000007 y
.
GROUP xs4all.test
211 7 3623 3629 xs4all.test

It can be useful for news readers which indicate the number of
articles in a newsgroup (otherwise, they have to send both LIST ACTIVE
and GROUP for every subscribed newsgroup at startup).

> It's more work and more writing, but

OK.

-- 
Julien ÉLIE

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 28 Nov 2009 20:08:43 +0100

Hi Antti-Juhani,

> standards, even though a Proposed Standard is allowed them.

But then, I still do not understand how I can write an extension for
LIST REMOVALS.

RFC 3977 mentions:

   An extension is a package of associated facilities, often but not
   always including one or more new commands.  Each extension MUST
   define at least one new capability label (this will often, but need
   not, be the name of one of these new commands).  While any additional
   capability information can normally be specified using arguments to
   that label, an extension MAY define more than one capability label.
   However, this SHOULD be limited to exceptional circumstances.

The capability label is LIST.  It already exists!

Is is true that LIST HEADERS is normally added by HDR, and LIST OVERVIEW.FMT
by OVER.
What for a new LIST variant only, without any other base command?

Should we write an erratum for RFC 3977 so as to permit extensions
without new capability labels?  Or am I missing something?

-- 
Julien ÉLIE

« The hardest thing is to go to sleep at night, when there are
  so many urgent things needing to be done.  A huge gap
  exists between what we know is possible with today's machines
  and what we have so far been able to finish. »  (Donald Knuth)

---

From: Russ Allbery <[email-masked]>
Date: Sat, 28 Nov 2009 11:34:10 -0800

> Should it be put in the Proposed Standard document?

Oh, yeah, I forgot about that.  I think that would be fine.  I was more
saying that we might want to stick with things that have an
implementation for Proposed Standard.