Subject: [NNTP] Re: Nearly there

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 10 Aug 2006 10:26:51 +0100

> I have a vague memory that we had a couple of changes we were considering
> making at this stage, but can't remember exactly what they were. If anyone
> else can, this would be a good time to carry out a sanity check in the
> group.

Okay, trawling through the archives ...

(1) Keywords (which include capability names) are limited to 12 characters,
but we already have a keyword ("IMPLEMENTATION") which is longer than that.
The majority view was to remove the limit completely:

    A "keyword" MUST consist only of US-ASCII letters, digits, and the
    characters dot (".") and dash ("-"), and MUST begin with a letter.
    Keywords MUST be at least three characters.

      keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-")

(2) What should the limit be on article numbers? At present it is 2^32-1,
but there were proposals:
- to make it 2^31-1, because some existing implementations use signed 32
  bit types to hold the article number and we should be documenting
  existing practice;
- to make it 2^64-1, either as a SHOULD or a MUST;
- to have no limit, but warn that existing implementations have limits.
There were also proposals for solving the limit (my BIGNUM proposal and
someone else's wrapping proposal), but these were not intended to go into
this document.

I'm very nervous about raising this, after all the arguments this time last
year, but I believe that changing it to 2^31-1 is the correct thing to do
*FOR THIS VERSION OF THE NNTP SPECIFICATION*. In support of this, I note
that Russ quoted our charter:

| The first concern of this working group shall be for the interoperability
| of the various NNTP implementations, and therefore for clear and explicit
| specification of the protocol. It is very important that we document the
| existing situation before taking up any new work.

and I also note something else he wrote earlier in the discussion:

| (I wonder how much software will fail at 31 bits, too.  I dug a lot of
| signed integers for article numbers out of INN when I started doing
| cleanups, and I think there are still a few left in the corners.)

(3) It was noted that some servers may be getting near this limit and we
don't yet have a good mechanism for dealing with this. It was therefore
proposed to add wording saying that this is unsatisfactory but is being
worked on ready for the next version.

    It is likely that at some stage at least one NNTP server will reach the
    limit of [4,294,967,295] in at least one newsgroup. Ways to deal with
    this while not gratuitously breaking compatibility are still being
    investigated and are likely to result in publication of a revision or
    extension to this specification at some future date.

    While servers MUST NOT send article numbers greater than this limit,
    client and server developers are advised to use internal structures
    and datatypes capable of handling larger values in anticipation of
    such a change.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Ken Murchison <[email-masked]>
Date: Thu, 10 Aug 2006 08:56:23 -0400

>       keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-")

Agreed.

> - to have no limit, but warn that existing implementations have limits.

In the absence of any defined limit in RFC 977, I'd say we need to 
document minimum limit which gives us interoperability but leave it open 
ended for implementations that have gone higher.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: Russ Allbery <[email-masked]>
Date: Thu, 10 Aug 2006 07:27:11 -0700

Agreed.

> I'm very nervous about raising this, after all the arguments this time
> last year, but I believe that changing it to 2^31-1 is the correct thing
> to do *FOR THIS VERSION OF THE NNTP SPECIFICATION*.

I won't argue against either 2^31-1 or 2^32-1, but I'm not sure how much
we can get away with changing at this stage in the actual protocol
requirements.

>     It is likely that at some stage at least one NNTP server will reach
>     the limit of [4,294,967,295] in at least one newsgroup. Ways to deal
>     with this while not gratuitously breaking compatibility are still
>     being investigated and are likely to result in publication of a
>     revision or extension to this specification at some future date.

>     and datatypes capable of handling larger values in anticipation of

I'm not sure what to think about this.  I think overall it's not a bad
idea, though, and might be helpful.  Since the above doesn't change the
protocol specification, it's more palatable during AUTH48 to me.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 10 Aug 2006 15:47:21 +0100

> we can get away with changing at this stage in the actual protocol

If I understood what Scott said, we can make such changes at AUTH48
provided it's been through the working group.

> protocol specification, it's more palatable during AUTH48 to me.

The first part was in lieu of anyone coming up with a mutually acceptable
proposal in the timescale. The second part is a blatant hint to developers
not to use 32-bit variables in the future.

---

From: Ken Murchison <[email-masked]>
Date: Thu, 10 Aug 2006 14:39:39 -0400

> not to use 32-bit variables in the future.

The first part seems too wordy to me and gets into more detail than I 
think is necessary.  How about something like this instead:

"Note that it is likely that the article number limit of [4,294,967,295] 
will be increased by a future revision or extension to this 
specification.  While servers MUST NOT send article numbers greater than 
this current limit, client and server developers are advised to use 
internal structures and datatypes capable of handling larger values in 
anticipation of such a change."

---

From: Russ Allbery <[email-masked]>
Date: Thu, 10 Aug 2006 14:38:54 -0700

> anticipation of such a change."

That seems better to me.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 11 Aug 2006 10:12:02 +0100

>> anticipation of such a change."

I can live with that.

So, when I get the alert from the RFC Editor:
- change 1 from my original email (remove limit of 12 characters) to be made;
- wording above to be added;
- either:
  - article number limit left unchanged at 2^32-1
  - article number limit changed to 2^31-1.

But which? I still prefer the change, as it better represents what servers
can rely on the client understanding.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 11 Aug 2006 07:38:03 -0700

> But which? I still prefer the change, as it better represents what
> servers can rely on the client understanding.

A brief review of Unix NNTP software revealed a lot of programs that
assumed 2^31-1.  Given the current state of NNTP software development in
general (not very quick), I do think it's safer to tell people 2^31-1.
The argument the other direction is that 2^31-1 is a simple bug and a
fixable one at that, but I'm not sure that's a strong enough argument.

I really would like to see multiple people weigh in on this, preferrably
more than just Ken, Clive, and I.

---

From: [email-masked] (Peter Robinson)
Date: Sat, 12 Aug 2006 15:51:59 +0100

> - change 1 from my original email (remove limit of 12 characters) to be made;

Yes.

> can rely on the client understanding.

The realistic limit is 2^31-1.  It's much more important to document the
current situation realistically than to start making improvements at
this stage.

Peter

---

From: Jim Beebe <[email-masked]>
Date: Sat, 12 Aug 2006 16:27:39 -0400

>current situation realistically than to start making improvements at

A long-time lurker to the mailing list agrees fully.

Jim Beebe

---

From: Ned Freed <[email-masked]>
Date: Sat, 12 Aug 2006 16:54:48 -0700 (PDT)

> A long-time lurker to the mailing list agrees fully.

(former cochair hat off) As do I. Our charter was fairly clear on this IMO.

				Ned

---

From: Thomas Gschwind <[email-masked]>
Date: Sun, 13 Aug 2006 18:20:48 +0200

Hi all!

> current situation realistically than to start making improvements at

Putting in 2^31-1 is fine with me especially considering our charter.

However, I am also very much in favor of putting in a paragraph stating
that RFC977 did not specify a limit and that there are news servers that
will generate article numbers larger then that (especially considering
that some newsgroups may run out of numbers soon) and that clients
SHOULD be able to handle numbers larger then that limit.  IMHO, we
cannot suddendly make those compliant news servers non-compliant.

Thomas
-- 
Thomas Gschwind                      Email: [email-masked]
IBM Zurich Research Lab
Saeumerstrasse 4                     Tel: +41-1-724-8990
CH-8803 Rueschlikon, Switzerland     Fax: +41-1-724-8953

---

From: Ade Lovett <[email-masked]>
Date: Thu, 17 Aug 2006 12:24:03 -0700

> Putting in 2^31-1 is fine with me especially considering our charter.

Agreed.  Let's get this done, out the door, and start work on the  
next steps.

> cannot suddendly make those compliant news servers non-compliant.

This will make things needlessly wordy.

Right now[1], we're looking at 8.3m articles/day.  Assuming every  
single article went into one group, that gives 258 days before  
wraparound.  The actual worst case[2] is about a tenth of that,  
875000 articles/day, 2454 days, or 6.7 years.  One can certainly make  
the argument for the small percentage of news servers that have to  
deal with this, steps have already been taken to move to (at least)  
64-bit clean.

Provided we get an update document out, mandating (at least) 64-bit  
clean inside of a couple of years (honestly, this is do-able, folks),  
we'll be fine.

977->ThisDoc *cannot* clean up everything.  977->ThisDoc->NewDoc will.

-aDe

[1] http://www.supernews.com/stats/feedvol/
[2] http://www.supernews.com/stats/retention.cgi? 
group=alt.binaries.boneless&type=Exact

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 18 Aug 2006 09:34:48 +0100

> Agreed.  Let's get this done, out the door, and start work on the  

I'm currently doing the AUTH48 walk through. I've made that particular
change.

>> will generate article numbers larger then that

> This will make things needlessly wordy.

Whether or not it does, I've not done this since we had no
previously-agreed wording.

> wraparound.

While there was a general feeling that we shouldn't pander to it, previous
discussions did bring up a server that allocated unique article *numbers*
to every article, producing the same effect.

Agreed.

> 977->ThisDoc *cannot* clean up everything.  977->ThisDoc->NewDoc will.

Doubleplusagreed.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 Aug 2006 10:44:47 GMT

In <[email-masked]> "Clive D.W. Feather" <[email-masked]> writes:

Just returned from 10 days holiday, and find this list a hive of activity!

>(1)  (which include capability names) are limited to 12 characters,

Agreed.

>someone else's wrapping proposal), but these were not intended to go into

I still favour some sort of wrapping rather than BIGNUM (but obviously not
for this draft). So please avoid anything that appears to commit us to one
or the other solution (the modified explanatory text might be read as
committing us to BIGNUM).

I am happy to make it 2^31-1, which seems to be the consensus. Any
wrapping scheme that might be proposed later will just have to fit within
that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5