Subject: [NNTP] Suggestions for NNTP extensions (CAPABILITIES)

From: Julien ÉLIE <[email-masked]>
Date: Sat, 20 Sep 2008 15:46:25 +0200

Hi,

In order to properly advertise a few (legacy) recognized commands in CAPABILITIES,
some more extensions are needed, as you all know.  They should be standardized.
Here is what I suggest:

1/ Additions to LIST.

    * LIST DISTRIBUTIONS (with a wildmat for the area?)
    * LIST MODERATORS
    * LIST MOTD
    * LIST SUBSCRIPTIONS

    * Define the meaning of some other status fields of LIST ACTIVE.
      I do not know how to advertise it.  Maybe we could have LIST ACTIVE.STATUS
      where the return would be:
        y Posting is permitted.
        n Posting is not permitted.
        =* Alias group.
      because it might be different in implementations.  Or should we have another
      capability name (like EXTENDED-STATUS) which imposes the meaning of the defined
      status(?)
      Perhaps EXTENDED-STATUS is a better idea than LIST ACTIVE.STATUS because only
      humans will be able to understand the result of LIST ACTIVE.STATUS.

    * Say in the introduction that LIST EXTENSIONS is deprecated in favour of CAPABILITIES.

Available references:
    http://www.eyrie.org/~eagle/nntp/drafts/draft-hernacki-nntplist-02.txt
    http://www.eyrie.org/~eagle/nntp/rfcs/rfc2980.txt

2/ PAT capability.

    * The legacy syntax is kept:
      PAT header range|message-ID pattern [pattern ...]

    * New metadata :body to specify that PAT will search in bodies.
      It would be advertised as PAT BODY in CAPABILITIES.  (Or by default in PAT?)

    * Maybe another metadata :text to search in the whole article (headers+body)?
      It would be advertised as PAT TEXT in CAPABILITIES.   (Or by default in PAT?)

    * Support for existing :bytes and :lines metadata, like what is currently
      implemented in INN 2.5 (for XPAT):
        XPAT :lines 112- *
        221 Header or metadata information for :lines follows (from overview)
        112 7
        113 8
        .
        XPAT lines 112- *
        221 Header information for lines follows (from articles)
        112 Not a number!
        113 1789
        .

    * I do not know how to deal with encoding problems...  For instance,
      if I search the word "trouvé", it can be encoded in lots of ways
      in bodies, as well as in headers (QP...).

    * What for a search in folded headers?  May we assume they are in the same
      format as in the overview database?  Therefore, we cannot search for
      tabulations...

    * Hmm...  Should PAT be more extensible?  Allowing to find articles matching
      several conditions?  (header Path: contains "server" AND metadata :lines
      is LARGER than 20...)

Available references:
    http://www.eyrie.org/~eagle/nntp/drafts/draft-ballou-nntpsrch-03.txt

3/ Something to deal with large article numbers.  What can be done?
   An extension?  But what kind of capability and use?

Available references:
    http://lists.eyrie.org/pipermail/ietf-nntp/2005-July/005720.html
    http://lists.eyrie.org/pipermail/ietf-nntp/2005-July/005802.html
    [...]

4/ Something to deal with compressed feeds.
   Maybe we could define BATCH there (the legacy XBATCH command) along
   with another command(?)

5/ Something to deal with cancels.  Maybe the CANCEL capability?
   A basic CANCEL <message-ID> command.

6/ Something to deal with headers feeds.  It should be great to have
   a standard for that.
   MODE HEADFEED is normally used.  I thought MODE commands began to
   be deprecated (MODE STREAM and MODE READER) but I do not see well
   how we could have a headers feed without a special mode...

7/ I also see QUOTA there:
    http://lists.eyrie.org/pipermail/ietf-nntp/2007-February/005989.html

   Is it useful to go on with it?

8/ INN also implements a wider syntax for wildmats (with "[" and "]" for
   instance).  Ranges like "-12" and "-" are also recognized in order to
   be more symmetric with the already defined "12-".
   Notwithstanding, I do not think it needs an extension for that...

If someone knows other references or thinks about other extensions,
please complete this mail.

Could someone *briefly* explain me the process to follow in order to
write a draft for these extensions?
I admit I am a bit lost when I try to find information on rfc-editor.org
when there is an available working group.

Thanks beforehand,

-- 
Julien ÉLIE

« Omnia uincit Amor et nos cedamus Amori. » (Virgile)

---

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Sep 2008 12:43:24 -0700

Julien ÉLIE <[email-masked]> writes:

> In order to properly advertise a few (legacy) recognized commands in
> CAPABILITIES, some more extensions are needed, as you all know.  They
> should be standardized.  Here is what I suggest:
>
> 1/ Additions to LIST.
>

I certainly agree with standardizing all of these based on the INN
implementation.  The implementations of these haven't changed in quite
some time and for whatever they're worth to the world, we may as well
write a specification for them.

>    * Define the meaning of some other status fields of LIST ACTIVE.  I
>      do not know how to advertise it.  Maybe we could have LIST
>      ACTIVE.STATUS where the return would be:
>        y Posting is permitted.

>      because it might be different in implementations.  Or should we
>      have another capability name (like EXTENDED-STATUS) which imposes
>      the meaning of the defined status(?)

>      Perhaps EXTENDED-STATUS is a better idea than LIST ACTIVE.STATUS
>      because only humans will be able to understand the result of LIST
>      ACTIVE.STATUS.

y, n, and m are already standardized.  The flags that INN adds are:

    j   No posting allowed, incoming articles filed into junk
    x   No posting allowed, remote postings rejected
    =*  Group aliasing

Thinking about this, I think we may want a combination of standardization
and a change in INN's behavior here.

To a reader, j groups really don't exist.  They will never contain any
articles, and are really just an internal hint to the serving agent to
move articles into a junk group.  nnrpd should ideally just suppress them
from LIST ACTIVE output.

To a reader, x groups are completely equivalent to n groups.  The only
difference is the behavior of incoming new messages from peers.  nnrpd
should probably change x to n when responding to LIST ACTIVE.

So I think INN should actually not return any flags other than the ones
already standardized and the alias flag.  Standardizing aliasing requires
deciding how it's supposed to work, but I think it's a useful capability.

>    * Say in the introduction that LIST EXTENSIONS is deprecated in

I'm not sure it's worth bothering with this.  LIST EXTENSIONS existed only
in Internet-Drafts.

>      PAT header range|message-ID pattern [pattern ...]

There are two main reasons why we didn't standardize PAT.  The first and
most serious is whitespace handling in patterns.  Given the way that NNTP
command parsing works, how would you search for a pattern containing two
spaces?  And what do the spaces between patterns really mean?

The second, as you mention below, is the encoding problem, which is very
hairy and difficult to deal with.

IMAP has addressed the search problem at some length, and my impression
was that it wasn't at all simple to deal with.  I'm afraid that doing a
good job of it is going to require quite a lot of work.

It would be worthwhile writing up an informational (?) I-D explaining what
INN actually does with XPAT without trying to change anything about how it
currently works and specifying an extension name of XPAT rather than a
standardized one.

>    * New metadata :body to specify that PAT will search in bodies.  It
>    would be advertised as PAT BODY in CAPABILITIES.  (Or by default in
>    PAT?)
>
>    * Maybe another metadata :text to search in the whole article
>    (headers+body)?  It would be advertised as PAT TEXT in CAPABILITIES.
>    (Or by default in PAT?)

Anything involving full body search should definitely be optional, since
the server may not hold the full article locally until the user requests
reading it and may not want to support expensive full search operations.

>        221 Header information for lines follows (from articles)

Seems like a reasonable thing to do.

>    * What for a search in folded headers?  May we assume they are in the
>    same format as in the overview database?  Therefore, we cannot search
>    for tabulations...

This is more of the whitespace problem.

>    * Hmm...  Should PAT be more extensible?  Allowing to find articles
>    matching several conditions?  (header Path: contains "server" AND
>    metadata :lines is LARGER than 20...)

Yeah, you can start adding almost arbitrary complexity when you get into
the more general search problem.

>    http://lists.eyrie.org/pipermail/ietf-nntp/2005-July/005802.html

IIRC, Clive had an extension proposal for how to deal with this.

>   Maybe we could define BATCH there (the legacy XBATCH command) along

Compressed feeds in general require something that replaces the current
data conveyance method so that you can convey arbitrary binary data over
NNTP.  The problem with the existing NNTP protocol for binary data is that
the data MUST end in CRLF, which is considered part of the data.  We need
some way of avoiding that, or some set of commands that say that the first
CRLF of CRLF.CRLF isn't part of the data.

XBATCH is worth documenting.  I don't know if it's worth standardizing as
BATCH, but I wouldn't mind at all.  Most of the work there will be
defining the batch format, and that will depend on how many different
transforms people feel like writing up (c7unbatch, cunbatch, gunbatch,
etc.).

>   A basic CANCEL <message-ID> command.

For those not familiar with INN's implementation, this is referring to
the existing INN MODE CANCEL hack, which is only supported on local
sockets and which lets the client just spew message IDs at the server and
have them treated as cancels.  The right way to do this is to introduce a
new CANCEL command that can be governed by normal authorization and
capability negotiation.

This is primarily used by other software on the same system, such as NoCeM
processors.

>   how we could have a headers feed without a special mode...

You have to introduce new commands or at least options to existing
commands (and new commands is probably cleaner).  Does the Diablo
implementation do streaming for header feeds?  If so, we need a
header-only equivalent for TAKETHIS, maybe TAKEHEADER.

>   Is it useful to go on with it?

I think we'd need someone who would actually use it to write it up.

>   Notwithstanding, I do not think it needs an extension for that...

The wildmat extensions would be nice to write up formally, but I'm not
sure it's worth a full extension.

> write a draft for these extensions?  I admit I am a bit lost when I try
> to find information on rfc-editor.org when there is an available working
> group.

Since there is no active NNTP working group (and I doubt there's enough
interest to create one), any drafts would need to be individual
submissions.  This isn't really a problem; the IETF publishes
standards-track RFCs from individual submissions.  However, there will be
more burden on IETF Last Call and on expert review to be sure that the
drafts are reasonable, in the absence of a working group process.  I think
the IETF ADs realize that the resources for NNTP are limited, though.

To write an I-D, I recommend using xml2rfc:

    http://xml.resource.org/

It makes the whole process much easier.  I thought there was a repository
somewhere of the XML versions of RFCs written with xml2rfc, but I haven't
been able to find it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 22 Sep 2008 00:03:06 +0200

Hi Russ,

> write a specification for them.

The same as what there is in INN?  So no second argument for LIST DISTRIBUTIONS
or the others?
No new LIST keywords?

> should probably change x to n when responding to LIST ACTIVE.

All right.  Easy to do for nnrpd.
I believe the same applies to innd (junk for j and ignore for x).

> deciding how it's supposed to work, but I think it's a useful capability.

Isn't what INN does right for them?

    If the <flag> field begins with an equal sign, the newsgroup is an alias.
    Articles cannot be posted to that newsgroup, but they can be received
    from other sites.  Any articles received from peers for that newsgroup
    are treated as if they were actually posted to the group named after
    the equal sign.  Note that the Newsgroups: header of the articles are
    not modified.  (Alias groups are typically used during a transition
    and are typically created manually with ctlinnd(8).)  An alias should
    not point to another alias.

Just a bit of rewording for an RFC is needed.

> spaces?  And what do the spaces between patterns really mean?

We could say that a server which complies with PAT does not parse the
pattern argument the same way it would for other commands.

RFC 4643 suggests that something like that could be done:

   Note that a server MAY (but is not required to) allow white space
   characters in usernames and passwords.  A server implementation MAY
   blindly split command arguments at white space and therefore may not
   preserve the exact sequence of white space characters in the username
   or password.

It would lift the problem of spaces for a real good PAT.

> hairy and difficult to deal with.

Only compare octets then.  It is what XPAT does by the way.
Otherwise, if we care about encoding, we will not find a decent solution...

> good job of it is going to require quite a lot of work.

It would be another extension (SEARCH for instance, if anybody wishes to
write it) and not tackled by PAT.
It would then allow to standardize a real PAT, and not only an informational
draft for XPAT; I do not know what is the best thing to do.

> IIRC, Clive had an extension proposal for how to deal with this.

Oh, great.  It should then be proposed to Giganews, as we discussed
last month on news.software.nntp.

> transforms people feel like writing up (c7unbatch, cunbatch, gunbatch,

Why should it be specified what the user does with the batch?
Let's give an opaque stream of n octets.  What the news server does with
that is of no concern.  POST does not define storage methods for instance.
Or is there something I am missing?

> Does the Diablo

I think it does streaming but I am not sure.
And CHECK also needs to be split in order to know whether the news
server only has the headers (?)
And IHAVEHDR?

It makes me think that another extension could be to change the newsgroups
subscriptions of the remote peer.  FEEDADD wildmat, FEEDDEL wildmat
for instance.  Server A connects to server B and *asks* for the feed.
But the protocol will not be proper because A needs to ask for an article
and then wait.  Server B could answer a code within n seconds to say
it has no articles.  And A could ask again...
A bit complicated, though.

> I think we'd need someone who would actually use it to write it up.

Sure.

> sure it's worth a full extension.

But in what kind of document could it be written?
An informational draft for only 12 lines of real content?  :-)
Or in a new man page for INN?

> somewhere of the XML versions of RFCs written with xml2rfc, but I haven't

Thanks, Russ.
I saw XML templates, which are a good starting point:
    http://tools.ietf.org/tools/templates/

-- 
Julien ÉLIE

« On ne va jamais si loin que lorsque l'on ne sait pas où l'on va. »

---

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Sep 2008 20:45:04 -0700

Julien ÉLIE <[email-masked]> writes:

> The same as what there is in INN?  So no second argument for LIST
> DISTRIBUTIONS or the others?  No new LIST keywords?

There definitely should be LIST capabilities added.  I didn't mean to
imply not doing that.

For any changes to the commands, well, if you feel inspired to change what
INN is currently doing and then document the new behavior, I think that's
certainly fine.  I think that these commands have existed for a long time
without a lot of interest in adding new parameters, though, so I don't
think it would be a big loss to just end up with what INN does now.  But I
suppose while we're writing standards, we should make them as general as
possible.

> I believe the same applies to innd (junk for j and ignore for x).

Yes.

The only possible concern is what it would do to some actsync setups, if
people are relying on replication of those flags.

>> already standardized and the alias flag.  Standardizing aliasing
>> requires deciding how it's supposed to work, but I think it's a useful
>> capability.
>
>    If the <flag> field begins with an equal sign, the newsgroup is an
>    alias.  Articles cannot be posted to that newsgroup, but they can be
>    received from other sites.  Any articles received from peers for that
>    newsgroup are treated as if they were actually posted to the group
>    named after the equal sign.  Note that the Newsgroups: header of the
>    articles are not modified.  (Alias groups are typically used during a
>    transition and are typically created manually with ctlinnd(8).)  An
>    alias should not point to another alias.
>

INN doesn't currently return the aliased group in the POST error, I think,
which might be nice.  It used to, but we lost that with the new overview
mechanism.  Otherwise I think INN's implementation is probably a good
reference.

> It would lift the problem of spaces for a real good PAT.

Yeah, we could do that.  I don't really like it -- note that occurs only
in the deprecated AUTHINFO command and we talked about that some at the
time, IIRC -- but it's probably the only sensible thing that can be done
without changing the command completely.

>> The second, as you mention below, is the encoding problem, which is
>> very hairy and difficult to deal with.
>

Well, you can require the pattern be in UTF-8 and do RFC 2047 decoding of
the header before doing the matching, which would then work properly for
all RFC-2822-compatible articles.  Doesn't help with articles with 8-bit
headers that aren't in UTF-8, but the standards say you're not supposed to
do that anyway.

The body is way harder.  You'd need a full-blown MIME implementation in
the server.

> write it) and not tackled by PAT.  It would then allow to standardize a
> real PAT, and not only an informational draft for XPAT; I do not know
> what is the best thing to do.

Well, I think that if we're going to wait for someone to write up SEARCH
for NNTP, we're going to be in for a long wait.  So if we want to
standardize PAT, we probably have to do it with most of its existing
limitations.  I think the main question is whether it's work worth doing,
but I can't see how it would hurt.

> Oh, great.  It should then be proposed to Giganews, as we discussed last
> month on news.software.nntp.

It would have been on the list shortly before the final publication, which
was when we discussed at the last minute what to do about this problem.
I'm pretty sure he wrote something up, although I don't remember how far
we got in discussing it.

>> XBATCH is worth documenting.  I don't know if it's worth standardizing
>> as BATCH, but I wouldn't mind at all.  Most of the work there will be

> Why should it be specified what the user does with the batch?  Let's
> give an opaque stream of n octets.  What the news server does with that
> is of no concern.  POST does not define storage methods for instance.

A standard needs to specify how the data is used.  I suppose we could
standardize a pure transport mechanism, but that's not horribly
interesting and doesn't include the information required to write real
interoperable implementations that do anything useful.

In practice, there are five interesting batch formats: straight rnews,
the binary compressed formats for compress, gzip, and bzip2 (cunbatch,
gunbatch, and bunbatch respectively), and c7unbatch, which is cunbatch
with a transport encoding that I'm not sure anything else uses

    The encoding uses characters from 0x20 (' ') through 0x7A ('z').
    (That fits nicely into the UUCP 'f' protocol by Piet Beertema.)
    First, expand three eight-bit charcters into four six-bit ones.
    Collect until we have 13, and spread the last one over the first 12,
    so that we have 12 6.5-bit characters.  Since there are very few
    half-bit machines, collect them into pairs, making six 13-bit
    characters.  We can do this as A * 91 + B where A and B are less then
    91 after we add 0x20 to make it printable.

All of them except c7unbatch are fairly easy to document.  c7unbatch we
might be able to get away with just declaring obsolete.

>> Does the Diablo implementation do streaming for header feeds?  If so,
>> we need a header-only equivalent for TAKETHIS, maybe TAKEHEADER.
>
> I think it does streaming but I am not sure.  And CHECK also needs to be
> split in order to know whether the news server only has the headers (?)
> And IHAVEHDR?

We'd need more information about how it's used.  Does it rely on being
able to distinguish between CHECK for a header and CHECK for a full
article?  What does a Diablo implementation with a header feed do if it
gets a traditional CHECK for an article for which it has only the header?
It may be that it's not that complicated.

Given that it's a new protocol, it's not clear to me that there's any
reason to bother with an IHAVE equivalent.  Again, though, we'd have to
look at Diablo and see if it implements both.

> It makes me think that another extension could be to change the
> newsgroups subscriptions of the remote peer.  FEEDADD wildmat, FEEDDEL
> wildmat for instance.  Server A connects to server B and *asks* for the
> feed.  But the protocol will not be proper because A needs to ask for an
> article and then wait.  Server B could answer a code within n seconds to
> say it has no articles.  And A could ask again...  A bit complicated,
> though.

Yeah, this has been kicked around for a while, although I don't think
anyone's gone as far as writing up an I-D for it.

Also see http://packages.qa.debian.org/g/gup.html

> But in what kind of document could it be written?  An informational
> draft for only 12 lines of real content?  :-) Or in a new man page for
> INN?

Well, one of the things that would be nice to do is to move RFC 2980 to
Historic.  One of the things that's currently only in RFC 2980 is the
extended wildmat syntax with character sets.  (Another are the LIST
extensions that you mention above, and PAT.)  If we got those into some
other published RFCs, we could declare RFC 2980 historic; the other stuff
in it is all deprecated and no longer generally implemented.

>    http://tools.ietf.org/tools/templates/

You can also grab the XML source of the current USEPRO draft as another
example at:

http://www.eyrie.org/~eagle/usefor/drafts/draft-ietf-usefor-usepro-12.xml

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 22 Sep 2008 20:48:44 +0200

Hi Russ,

> and a change in INN's behavior here.

As for innd's problem, it is triggered off by the fact that:

    7.6.3.  LIST ACTIVE

       This keyword MUST be supported by servers advertising the READER
       capability.

LIST ACTIVE is typically a reader command.  Therefore, "j" and "x" do not
have meaning for it.  However, they do have one for peers.  When "n" is
used for instance, it means that articles from peers are accepted.
Thus, "n", "j" and "x" are more precise.

I think that only "n" should be used by nnrpd but innd should go on using
and advertising "j" and "x".
Maybe it is something we can specify in the document for other LIST methods.
We define "j" and "x" which have special meanings for peers.  And also
for "n".

> headers that aren't in UTF-8, but the standards say you're not supposed to

Interesting idea, indeed.  Thanks!
I see that there are C (GPL) librairies to do the decoding so it is feasible.

> The body is way harder.  You'd need a full-blown MIME implementation in

I do not know if there is really a need to search inside bodies.  But it
might still be useful to standardize.

> It may be that it's not that complicated.

I do not know.  I shall see when trying to deal with that.
Let's focus on LIST and PAT before.

> http://www.eyrie.org/~eagle/usefor/drafts/draft-ietf-usefor-usepro-12.xml

Thanks.

-- 
Julien ÉLIE

« Quand on demande aux gens d'observer le silence, au lieu de l'observer
  comme on observe une éclipse de lune, ils l'écoutent ! » (Raymond Devos)

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 23 Sep 2008 18:11:26 GMT

>spaces?  And what do the spaces between patterns really mean?

Another reason is that wildmats are not really the right tool (and it is
not clear whether/where they are supposed to be anchored); and they may
not play very well with UTF-8. I think the view was that, to do the job
properly, we needed some kind of decent regular expression, but unless
there is some likelihood that someone would actually implement such a
thing, I think it better just to leave XPAT as it is (but ensure it is
documented somewhere, which it actually is). Leave PAT for some future.

>reading it and may not want to support expensive full search operations.

+1

If you want to search in bodies, then leave it to Google :-) .

>IIRC, Clive had an extension proposal for how to deal with this.

IIRC there was a scheme which involved setting the article numbers for an
about-to-overflow group back to some low value (but NOT zero) that might
not cause too much confusion with existing servers (for sure, ANY solution
will cause pain to SOME users, at least).

>sure it's worth a full extension.

IIRC, the reason we removed the [...] feature was that it might not play
well with UTF-8.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Tue, 23 Sep 2008 21:24:24 -0700

"Charles Lindsey" <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> IIRC, the reason we removed the [...] feature was that it might not play

It works fine in INN (after I fixed the wildmat support in general to
support UTF-8).  Most of the problems with UTF-8 support in [...] you
have to fix to properly support ? anyway, and some edge cases with *.
It's a bit more complicated and rules out some simple implementation
strategies, but it's not horribly difficult.

There's a test suite in INN that's free for anyone to use which tests
wildmat functions including character classes against various UTF-8 cases
and malformed UTF-8.