Subject: [NNTP] Article number wording

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 11:19:49 -0700

To have something more concrete to talk about, here's what I think several
people are asking for.  Note that I still haven't been convinced this is
the right thing to do, particularly now, but it's easier to discuss the
proposal when there's text to look at.

    arrive earlier.  The server SHOULD allocate the next sequential
    unused number to each new article.
 
-   Article numbers MUST lie between 1 and 4,294,967,295 inclusive.  The
-   client and server MAY use leading zeroes in specifying article
-   numbers, but MUST NOT use more than 16 digits.  In some situations,
-   the value zero replaces an article number to show some special
-   situation.
+   Article numbers MUST be positive integers.  The client and server
+   MAY use leading zeroes in specifying article numbers.  In some
+   situations, the value zero replaces an article number to show some
+   special situation.
+
+   This standard imposes no maximum limit on article numbers other than
+   the maximum length of an NNTP command.  Implementors are encouraged
+   to support arbitrarily large article numbers and SHOULD use at least
+   the largest integer type their software can support.  However, note
+   that many widely deployed NNTP clients cannot handle article numbers
+   larger than 2,147,483,647 or 4,294,967,295 and article numbers higher
+   than that are likely to cause interoperability problems.
 
 6.1.  Group and article selection

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 18 Jul 2005 14:51:47 -0400

> +   than that are likely to cause interoperability problems.

Yes, as an implementor, I'd read that as indicating bignums.  It doesn't
give any indication of Steve Walker's implementation being in the wild.

How about....

+   This standard imposes no maximum limit on article numbers other than
+   the maximum length of an NNTP command.  Note that many widely deployed
+   NNTP clients do not handle article numbers larger than 2,147,483,647 or
+   4,294,967,295, and no widely known clients or servers handle article
+   numbers larger than 18,446,744,073,709,551,615.

Check that 2^64 -1.  I don't trust that I or the calculator did that correctly.

I wouldn't mind seeing 2^31, etc expressions instead of the numbers.  But
the format should match other RFCs.

---

From: "Howard Swinehart" <[email-masked]>
Date: Mon, 18 Jul 2005 16:25:08 -0400

> the format should match other RFCs.

I agree.  Knowing which data type can hold 2^x or 2^x-1 is a lot easier than 
figuring out the wall of digits.  If showing the full number is necessary, 
then I suggest including the 2^x notation in parentheses.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 14:37:58 -0700

> +   numbers larger than 18,446,744,073,709,551,615.

You lost the only SHOULD in the paragraph, as well as any idea of using
the largest native integer type.  I don't like this as well; it doesn't
seem to give the implementor any real guidance.

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 00:42:19 +0200

> proposal when there's text to look at.

I strongly object, and I expect that the better part of the
implementations won't follow this.

I understand that putting up arbitrary limits is tricky as the limit
might be chosen too low, but article numbers are THE primary index -
introducing arbitrary length integers here is going to hurt big time -
implementors and execution speed.

-- 
Matthias Andree

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 15:43:52 -0700

> implementors and execution speed.

What do you think of an explicit 64-bit limit?

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 01:40:12 +0200

> What do you think of an explicit 64-bit limit?

Fine with me.

I can't imagine how we could possibly bump into a 2^64 > 10^19 article
limit - assuming a group with a post per second, and a year to have
10^7.5 seconds this will suffice for 10^11.5 years.

Current projections how long life on planet Earth will be possible range
between 10^9 and 10^10 years, before The Sun will turn into a Red Giant
and swallow the Inner Planets.

Even if we logarithmize these figures because we hopelessly
underestimate the growth, we'll have almost a dozen years to standardize
and implement NNTP 3.

My NNTP software leafnode uses "unsigned long", so if you compile it on
a 64-bit machine (say, UltraSPARC), so I'm not concerned about such a
limit. By the time 64-bit article number will get interesting, 64-bit
CPUs will have become a commodity.

Note #1: I am assuming that humans are going to read NNTP-transported
articles.

Note #2: I don't care the least about binary file distribution or other
forms of NNTP abuse. If the protocol has a known limitation, and this
one is documented, the users will have to make sure they're not bumping
into it through using the protocol unwisely.

-- 
Matthias Andree

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 18 Jul 2005 23:15:53 -0400

Russ Allbery wrote:
> the largest native integer type.  

Dropped, not lost.  It was not an accident.

 > I don't like this as well; it doesn't

I think it replaces a sentence which did not give real guidance.

I wrote to give the server implementor an idea of what to expect in client
behavior at the time we write it, and the client implementor an idea of
what to expect in server behavior.

I don't like "SHOULD" here at all. Interoperating Article numbering is waaay
more important than what the "extra features" you typically get by following
a SHOULD.   SHOULD doesn't say anything definite about what to expect from
the other side.

It would be cleaner to provide a hard limit in the protocol, with
a MUST, but there's no consensus.  There are rational voices here to not
break clients by rollover at 2^31, and there are rational voices here to
support 2^64.

I think they both have good arguments.  It certainly isn't clear to me what's
the right solution.

So, I say punt, outline the problem-space and then implementors will do
the right thing.  No need for the RFC2119 language, because it doesn't
really help, unless it is MUST, (and even then that might not help define
what legacy clients will do.)

---

From: Ade Lovett <[email-masked]>
Date: Tue, 19 Jul 2005 01:30:32 -0700

Matthias Andree wrote:
> might be chosen too low

The primary problem with limits is the requirement for the associated
wording of the text around what happens when (not if) those limits are
reached.

> but article numbers are THE primary index

Absolutely, utterly, and totally, *wrong*.  Article numbers are most
certainly *an* index, but they are not *the* index.  The fact that a
given article can have completely different numbers for each of its
groups across every single Usenet server should make that clear.

*THE* primary index of Usenet is the message ID.  There is even language
in the text to indicate that it SHOULD (RFC-speak) be globally unique.
I can go to pretty much any Usenet server on the planet, and request
<[email-masked]> and get the same article.  The same cannot be said
for article #123456789 in news.software.nntp.

> implementors and execution speed.

I see absolutely no compelling reason to move away from what we have
right now in the final draft that is due to become a *Proposed* Standard
in the near future.

We have spent far too much time already in getting this out and *that*
is hurting implementors far more than 32 vs 64 vs 128 vs n-bit ethereal
article number representation.

By all means continue to discuss this, and other, issues that have
arisen.  Remember, we can always issue a new document in the future with
more stringent guidelines (and maybe it won't take 7+ *years* like this
one).

But for now, we have a good solid document.  Sure, it's not perfect,
nothing ever is, but it's a damn sight better than what is out there
right now.

Things can always evolve.  It is their nature.

For the record, I strongly object to any specific mention of
32/64/whatever-bit text in the document as it stands now.  I firmly
believe we would be doing a major disservice to the Internet community
at large by delaying publishing this over one of a myriad of
implementation issues.

-aDe

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 11:09:37 +0200

> groups across every single Usenet server should make that clear.

In SQL language, consider a group a table (all of the same schema =
columns), one row per article.

The article number is THE de-facto index, ALTHOUGH it is only unique per
(server or site, group) pair. Most NNTP clients access articles by
(group, article number) tuple, NOT by Message-ID. Regardless of the
Message-ID being globally unique for two years or something like that.

Message-ID is used to reference articles, not to access them in the
first place.

Paying attention to interoperability is important, and as you say, not
standardizing is hurting implementors, give a clear and strong
guideline, with "MUST" language, 

-- 
Matthias Andree

---

From: Steve Walker <[email-masked]>
Date: Tue, 19 Jul 2005 04:32:02 -0500

>at large by delaying publishing this over one of a myriad of
Are you suggesting removing the existing text that says numbers must be 
32 bit and they can't wrap around?  That's my problem with the existing 
proposal.  It added a new 32 bit limit and forbid the wrapping of 
numbers.  That's why I say it is so broken that it shouldn't be used as 
is.  It's a time bomb waiting for everyone to do their own thing.  
Personally I'm going to push 64 bit numbers, the next guy may wrap 
around.  We don't have a clear, correct path for what to do when it 
happens and the client authors will be confused.  They know they have to 
support numbers larger the 32 bit, but the RFC says to not do it.

Personally I think 64 bit numbers solves the problem with the least 
amount of problems.  It's easy to fix software to use 64 bit numbers.  
Doing unlimited text based math, while fine in a perfect world, will 
cause a lot more issues and there is little to really gain compared to 
64 bit numbers.

Steve.

---

From: Ade Lovett <[email-masked]>
Date: Tue, 19 Jul 2005 10:42:34 -0700

Matthias Andree wrote:
>>Article numbers are most
>>certainly *an* index, but they are not *the* index.
> 

Please read what I wrote, article numbers are *an* index, not *the* index.

> Message-ID is used to reference articles, not to access them in the

Huh?  ARTICLE|HEAD|BODY|STAT <msgid> must be a figment of my imagination
then.

> guideline, with "MUST" language, 

We are hurting implementors considerably more by not releasing the
document for publication with this eleventh-hour discussion.  That is
not to say that this discussion is in any way invalid, but having read
the entire thread, it seems very unlikely that any kind of consensus is
going to be reached quickly.  Taking one point from the above, I would
have to completely disagree on the use of MUST.  At most, it whould be a
SHOULD.

I again appeal to the WG chair to publish the document as-is, and we can
continue these discussions in the meantime, along with any other nits
that will result in the far bigger audience reading the document as a
result of it being published as a Proposed Standard.

-aDe

---

From: Ken Murchison <[email-masked]>
Date: Tue, 19 Jul 2005 13:43:46 -0400

There haven't been any direct responses to my proposed text change, so 
I'll repeat a variation of it here.  IMO, this doesn't break existing 
implementations, but urges that they switch over to using 64-bits.  When 
some server does eventually split back >32-bit article numbers, a 32-bit 
client is no worse off than if the server rolled over, since there is no 
documented standard for handling roll over.  The bottom line is that we 
won't see article numbers exceed 32-bits for several years, so starting 
the 64-bit ball rolling now seems like a safe, wise, and prudent move.

-   Article numbers MUST lie between 1 and 4,294,967,295 inclusive.  The
-   client and server MAY use leading zeroes in specifying article
-   numbers, but MUST NOT use more than 16 digits.  In some situations,
-   the value zero replaces an article number to show some special
-   situation.
+   Article numbers MUST be positive integers.  Clients and servers
+   MUST support article numbers up to, and including 4,294,967,295
+   (2^32 - 1).  However, due to the exponential growth of Netnews
+   traffic, it is RECOMMENDED that clients and servers support article
+   numbers up to, and including 18,446,744,073,709,551,615 (2^64 - 1).
+   The client and server MAY use leading zeroes in specifying article
+   numbers.  In some situations, the value zero replaces an article
+   number to show some special situation.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Russ Allbery <[email-masked]>
Date: Tue, 19 Jul 2005 10:53:30 -0700

> result of it being published as a Proposed Standard.

I haven't put any sort of hold on the documents, so for right now the
process is continuing.  Nothing is likely to happen to the base document
before the extension documents are approved, which will hopefully happen
at the July 21st IESG telechat.

So the discussion can happen without worrying that it's delaying anything.
The RFC Editor queue also looks fairly long, so I'm not sure how long the
documents will wait there before they're actually published.

---

From: Chris Caputo <[email-masked]>
Date: Tue, 19 Jul 2005 18:47:54 +0000 (GMT)

On Tue, 19 Jul 2005, Ken Murchison wrote:
> There haven't been any direct responses to my proposed text change, so I'll 
> repeat a variation of it here.  IMO, this doesn't break existing 
> implementations, but urges that they switch over to using 64-bits.  When some 
> server does eventually split back >32-bit article numbers, a 32-bit client is 
> no worse off than if the server rolled over, since there is no documented 
> standard for handling roll over.  The bottom line is that we won't see 
> article numbers exceed 32-bits for several years, so starting the 64-bit ball 
> rolling now seems like a safe, wise, and prudent move.
>
>

Works for me.  I like it.

Chris

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 23:35:31 +0200

> Huh?  ARTICLE|HEAD|BODY|STAT <msgid> must be a figment of my imagination

They are part of the protocol and I'd say an indispensable one, but not
in wide use by clients as their primary access method. The primary
access method is GROUP <name> then XOVER or similar, then
{ARTICLE|HEAD|BODY|STAT} <number>.

> have to completely disagree on the use of MUST.  At most, it whould be a

Then we can just as well release the document without semantic change,
just spell out "2^31 - 1" in addition to that 10-digit number, and leave
the 2^31..2^64-1 issue for NNTP 2 revision 1 or NNTP 3.

I don't mind personally, I am not reading groups that might approach
that limit soon, and I haven't yet heard of leafnode user concerns in
this regard.

-- 
Matthias Andree

---

From: Ade Lovett <[email-masked]>
Date: Tue, 19 Jul 2005 18:21:57 -0700

> {ARTICLE|HEAD|BODY|STAT} <number>.

Maybe not in "true" Usenet clients (ie: text), but that's more due to
ancient history with certain NEWNEWS implementations being extremely
slow, resulting in a dramatic decline in its use (which is a shame,
since it is the perfect tool for a client operating across multiple news
servers, and removes the issues associated with a server failing
catastrophically, and resetting article numbers).

However, regardless of whether you view binaries-over-Usenet as "abuse",
this is the real world we're talking about, and it does happen.  A lot.
 A particular usage of Usenet cannot be sidelined on the basis of a
purist view, we have to be realists.  Usenet *is* used for binary
transmission, and there are certainly multiple-server clients out there
that use message ID as the primary access method.  Look up NZB files for
a case in point.

As and when a concrete situation is presented, with associated wording,
that makes sense, I'll certainly evaluate it on its merits, see this
mailing list passim for examples of that.

Right now, I'm not sure that the proposed wording does actually improve
the situation to the point where updates would have to take place,
possibly resulting in the entire document having to go back through the
whole IESG process (Russ?  How much flexibility do we have before the
document is changed to an extent that it would require this?).

-aDe