Subject: [NNTP] Article number wording

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 26 Jul 2005 14:53:18 +0100

[Typically, this would blow up while I'm on holiday. My apologies for
starting a separate thread, but there are several points scattered around
that I want to address.]

Firstly, I believe we *MUST* get this document out without any more delays.
It is ridiculous how long it has taken already. If major changes are
needed, we can address them in the next review cycle.

Secondly, I don't accept the "977 had no limits" argument. RFC 977 was
written at a time when people were far more sloppy about specifications
than they are now. There are lots of places where it failed to make matters
clear enough, and this is one of them. In those days everyone would have
seen 32 bits as being "obvious". The lack of Bignum implementations within
news software is evidence of this.

Thirdly, I do actually sympathise with those who want something done. When
we agreed that wording 8 or so years ago, we decided that the 32 bit limit
would be hit rarely enough that local "flag days" would be acceptable. Had
this topic been raised a year ago, I would have been very much in favour of
solving it before publication (given we were waiting on plenty of other
things). But it's been dormant for many years; now is not the time. This is
*not* the same as saying "never fix it".

Fourthly, it isn't "a simple matter of programming", as Ken implies with:
> If I change my server to use 64-bits today (actually it might be
> already, I'd have to look), its not going to break any existing clients,

What you'll be left with is a problem waiting to happen *silently*, as old
clients that thought they conformed suddenly run into servers that use
large numbers. Or vice versa (though that's rather less likely).

Fifthly, when we worked on the text originally, nobody noted the signed
versus unsigned problem. The limit we have in the text is 2^32-1 (spelled
out, of course). Should this be changed to 2^31-1 before publication
(presuming it's acceptable as a minor change)?

Russ suggested the wording:
> Implementors are encouraged
> to support arbitrarily large article numbers and SHOULD use at least
> the largest integer type their software can support.  However, note
> that many widely deployed NNTP clients cannot handle article numbers
> larger than 2,147,483,647 or 4,294,967,295 and article numbers higher
> than that are likely to cause interoperability problems.

I don't think this helps. A maximally portable server will have to limit
itself to 2^31-1. Better to find a way for the two parties to negotiate the
maximum they can cope with and have a defined way to deal with overflow. An
extension is then the right way to do it.

[Incidentally, "can support" should be "can support efficiently", because
everyone *can* support bignums with enough effort.]

For now, put in some wording like:

    It is likely that at some stage at least one NNTP server will reach the
    limit of 4,294,967,295 in at least one newsgroup. Ways to deal with
    this while not gratuitously breaking compatibility are still being
    investigated and are likely to result in publication of a revision or
    extension to this specification at some future date.

and let that be a warning to implementers.

[Rather than making this message any longer, I'll address the extension
idea separately.]

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 26 Jul 2005 09:57:04 -0400

> -----Original Message-----
> From: Clive D.W. Feather [mailto:[email-masked]] 
> Sent: Tuesday, July 26, 2005 9:53 AM
> To: IETF NNTP mailing list
> Subject: [NNTP] Article number wording
> 

[snip]

Which document are you referring to?  nntp-base has already been approved by
the IESG.  There is no "next review cycle", other than auth48, and it's
generally not appropriate to make non-editorial changes in auth48.

-Scott-

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 26 Jul 2005 15:36:52 +0100

> generally not appropriate to make non-editorial changes in auth48.

By "next review cycle" I mean the work after nntp-base is published as an
RFC. Whether this means publishing extensions, a revised nntp-base, or
whatever. But *after* is the key word.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 26 Jul 2005 10:37:45 -0700

Scott Hollenbeck <[email-masked]> writes:

> Which document are you referring to?  nntp-base has already been
> approved by the IESG.  There is no "next review cycle", other than
> auth48, and it's generally not appropriate to make non-editorial changes
> in auth48.

It's my intention to accumulate implementation information and start draft
standard work a year or two after we release nntp-base, which would give
us another chance to revisit these issues (hopefully we won't have so many
changes that we have to do another round at proposed, but it's always
possible).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Steve Walker <[email-masked]>
Date: Thu, 28 Jul 2005 23:53:32 +0700

>changes that we have to do another round at proposed, but it's always
What about the servers like mine that will exceed the 32 bit limit 
before your two year plan?  I will be pushing >32bit number before 2 years.

Steve.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 28 Jul 2005 10:05:51 -0700

>> It's my intention to accumulate implementation information and start
>> draft standard work a year or two after we release nntp-base, which
>> would give us another chance to revisit these issues (hopefully we
>> won't have so many changes that we have to do another round at
>> proposed, but it's always possible).

> before your two year plan?  I will be pushing >32bit number before 2

If you're asking my personal opinion, I think you made a poor choice in
implementation strategies without first checking whether clients could
handle your setup.

No matter what we say or don't say in this draft, you're still going to
have serious problems in two years with regular news reader clients.

---

From: Steve Walker <[email-masked]>
Date: Fri, 29 Jul 2005 00:14:53 +0700

>needed, we can address them in the next review cycle.
There isn't time.  People will be pushing >32 bit number before V3 could 
be released.  V2 is the perfect time to fix it before it happens.  Any 
more delays and it will be too late. Why push something forward knowing 
it has a major flaw what will prevent many people from using this 
proposed standard.  You are also adding wording that will require many 
clients to choose from downgrading to 31/32 bits or not be compliant.  
Keep in mind that there are 64 bit clients in use today.

>would be hit rarely enough that local "flag days" would be acceptable. Had

That "flag day" is forbidden by the new proposal.  It explicitly forbids 
rolling over.

>*not* the same as saying "never fix it".
Waiting until it's too late to avoid a problem is not the correct way to 
release a new RFC.

>large numbers. Or vice versa (though that's rather less likely).
Leaving the new 32 bit limit will certainly break much more.  Servers 
have no choice but to either roll over or exceed 64 bits.  You can't 
make both a MUST NOT and expect it to work knowing that it will happen.

>(presuming it's acceptable as a minor change)?
Programmers will always do the minimum required to get the job done.  
That is why we now have 31, 32, 63 and 64 bit clients/servers.  Changing 
the wording to 64 bits is the least likely to cause major problems.

>extension is then the right way to do it.
How does negotiating solve anything other than the client knows it won't 
work.  It won't make anything work that wouldn't have already worked.  
Servers that exceed 31 bits will have problems with broken clients, but 
it's the clients that need fixed.  We should not re-write 977 to the 
lowest common denominator knowing that it will break good software. No 
matter what is done many clients will have to be fixed.  Why not really 
fix them instead of just adding a one/two year band-aid.

Steve.

---

From: Steve Walker <[email-masked]>
Date: Fri, 29 Jul 2005 00:49:50 +0700

>have serious problems in two years with regular news reader clients.

I think that the clients that currently don't support 64 bit numbers 
will be fixed soon since it's a very easy process.  That is, if the RFC 
doesn't expressly forbid 64 bit numbers.  If the new RFC forbids 64 bit 
numbers then the client developers will have to choose between being 
non-compliant and actually working on all servers.  This is a potential 
mess.

Steve.

---

From: Matthias Andree <[email-masked]>
Date: Fri, 29 Jul 2005 11:29:22 +0200

> (presuming it's acceptable as a minor change)?

Don't change. 2^32 - 1 is valid, if hackers abuse half of their number
space to fold some ridiculous flags in, they can suffer.

> and let that be a warning to implementers.

Fine.

-- 
Matthias Andree

---

From: [email-masked] (Thomas Gschwind)
Date: Fri, 29 Jul 2005 11:38:26 +0200

> > and let that be a warning to implementers.

To me this sounds as good as no warning at all, I as implementor of any
news client/server software would not know how to interpret this.  Does
it mean using 64 bit ints as article numbers in the future or does it
mean a new mechanism will be developed or does it mean some wrap-around
mechanism will be introduced?  In the end, I guess I would be doing
nothing and leave my implementation to use ints.

I would at least recommend that news client developers SHOULD use
64 bit unsigned integers for storing article numbers in order to
facilitate a possible expansion of the article number range.

This way nothing breaks, we still have all the options available, and
it makes it easier if we want to go that direction for the next version
of the draft.

Regards,
Thomas
-- 
Thomas Gschwind                      Email: [email-masked]
IBM Zurich Research Lab
Saeumerstrasse 4                     Tel: +41-1-724-8990
CH-8803 Rueschlikon, Switzerland     Fax: +41-1-724-8953

---

From: Urs Janßen <[email-masked]>
Date: Fri, 29 Jul 2005 11:55:55 +0200

> will be fixed soon since it's a very easy process.  That is, if the RFC 

I (as a client author) don't think so - it's not easy to fix code which
runns on a bunch of different (some very old and/or obscure) unices.
exceeding 32bit artnums within the next ~5 years means asking for trouble,
no matter that the RFCs say.

> non-compliant and actually working on all servers.  This is a potential 

as most servers (>99%) won't exceed the 32bit artnums within the next ~5
years client authors do not have fix it emediately...

IMHO we should leave the artnum stuff for the next RFC.

urs
-- 
"Only whimps use tape backup: _real_ men just upload their important stuff
 on ftp, and let the rest of the world mirror it ;)" - Linus

---

From: Steve Walker <[email-masked]>
Date: Fri, 29 Jul 2005 09:29:43 -0500

>IMHO we should leave the artnum stuff for the next RFC.

The problem is that the current draft has addressed the issue.  It says 
32 bit numbers and no roll over.  If you wish to leave it open as 977 
did, you need to remove the current text on the matter.  In either case 
you still have to make a change to the current draft.  Either leave it 
open, or specify 64 bit numbers.  I think it's just as easy to change to 
64 bit numbers as it is to remove the current 32 bit limit. 

The very old unix clients are not normally used on modern binary 
servers.  Any 64 bit change would not effect them for several years and 
by that time 32 bit computers will not be in use.

Steve.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jul 2005 11:00:16 -0700

Steve Walker <[email-masked]> writes:

> did, you need to remove the current text on the matter.

It's not acceptable to leave it unspecified.  RFC 977 was deficient in
this regard.  Whatever we do, we have to be clear and specific about it;
that's the whole point of a standard.

> servers.

We're not talking about old Unix clients.  The example I gave was from
slrn.

---

From: [email-masked] (Thomas Gschwind)
Date: Fri, 29 Jul 2005 20:25:32 +0200

Hi all!

> that's the whole point of a standard.

I perfectly agree that this should be specified.  At the same time,
since we are aware of the fact that article numbers will eventually
overflow, I think a solution SHOULD be pointed out in the RFC.
Otherwise the RFC renders itself obsolete before we have a better
solution and everybody will come up with a different solution.  This I
would consider a bad thing.

I think the best solution would be to recommend news servers
implementors to limit their article numbers to unsigned 32-bit numbers
for the next 2(?) years and at the same time, we should recommend news
client implementors to update their clients to handle unsigned 64-bit
article numbers.  This gives them 2 years time which to update their
clients which should be sufficient.

Specifying a wrap-around strategy similar to the one that we have
(almost) worked out already, would be another/additional option.
However, I agree that at this time adding the wrap-around strategy might
be too big a change.

I don't think using an extension will provide an adequate solution since
again old clients would be blocked from reading news unless they
implement the extension (which they don't) or unless the server provides
two different numberings (in which case the extension is obviously
unnecessary).  Please correct me, if I am wrong.

Thomas

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jul 2005 11:35:17 -0700

> would consider a bad thing.

I continue to disagree with the perceived severity of this problem.

> clients which should be sufficient.

Flag days are strongly discouraged in IETF standards, mostly because they
just don't work very well.

> unnecessary).  Please correct me, if I am wrong.

They would only be blocked from reading groups that had overflowed 2^32-1,
not all groups.

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Fri, 29 Jul 2005 21:49:42 +0200

Russ Allbery:

> [article numbers, limits]
> this regard.  Whatever we do, we have to be clear and specific about
> it; that's the whole point of a standard.

I would just mention in a side-note that in 2005 it is no longer good to
have a limit in the software that is less than 2^64, and that any limit
will be proven too small some time.

-- 
Grtz, Ruud

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jul 2005 14:05:46 -0700

> will be proven too small some time.

Making this statement specifically about article numbers is a valid, or at
least arguable, point.  The statement as you made it, as a general
statement about all standards, makes no sense to me.  The above would
argue, for example, that Unicode is not an acceptable standard because it
has a 2^32 bit limit on the code space, when so far as I can tell pretty
much everyone involved in character set issues believes that will be
sufficient for the forseeable future.

In other words, it depends on what the limit is for and how much utility
is gained by having a limit.

Also, not specifically on this point but with relation to this discussion
in general, I feel obligated at this point to once again ask everyone
participating in the working group to remember our charter:

| The IETF NNTP extensions Working Group shall:
| 
|    1. Revise and publish a standards-track successor to RFC 977 that
|       removes ambiguities from the original document, defines a mechanism
|       for adding extensions to the protocol, and provides a mechanism for
|       the server to inform the client of the extensions which it supports.
| 
|    2. Include in the same document some reasonable group of existing
|       commonly used extensions forming a new base functionality for NNTP.
| 
|    3. Upon completion of the RFC977 successor document, and presuming that
|       proposals for extensions to the NNTP protocol have been submitted
|       for consideration by IESG, the working group may be asked by the
|       IESG Applications Area Directors to review one or more extensions
|       for NNTP. Part of the purpose of such a review will be to test the
|       newly established mechanism for adding protocol extensions.
| 
| The first concern of this working group shall be for the interoperability
| of the various NNTP implementations, and therefore for clear and explicit
| specification of the protocol. It is very important that we document the
| existing situation before taking up any new work.

Article number handling falls into point 1, and the caveats in the final
paragraph applies.  I think the first sentence of the last paragraph makes
it clear (in reference to other messages in this thread) why leaving the
limit unspecified is not acceptable.  (Which is not the same thing as
specifying that there is no limit -- that solution may have other
problems, but it would be clear and explicit.)

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Sat, 30 Jul 2005 01:05:17 +0200

Russ Allbery:
> Ruud H G van Tol:

>>> It's not acceptable to leave it unspecified.  RFC 977 was deficient
>>> in this regard.  Whatever we do, we have to be clear and specific
>>> about it; that's the whole point of a standard.
>
>> I would just mention in a side-note that in 2005 it is no longer
>> good to have a limit in the software that is less than 2^64, and
>> that any limit will be proven too small some time.
>
> Making this statement specifically about article numbers is a valid,
> or at least arguable, point.

Yes, that was mainly how I meant it.

> The statement as you made it, as a
> general statement about all standards, makes no sense to me.  The
> above would argue, for example, that Unicode is not an acceptable
> standard because it has a 2^32 bit limit on the code space, when so
> far as I can tell pretty much everyone involved in character set
> issues believes that will be sufficient for the forseeable future.

Unicode escapes 2^32 with a mechanism for modifying character shape,
such as combining diacritical marks. The result does not need to have a
codepoint as a precomposed character. And who knows how many other
writing systems there are in the universe, so a better name for it is
Terracode.

Maybe it is the moment to give article number 2^32 a special meaning.
;)

> In other words, it depends on what the limit is for and how much
> utility is gained by having a limit.

I agree.

>> [charter]
>> The first concern of this working group shall be for the
>> interoperability of the various NNTP implementations, and therefore
>> for clear and explicit specification of the protocol. It is very
>> important that we document the existing situation before taking up
>> any new work.
>
> Article number handling falls into point 1, and the caveats in the
> final paragraph applies.  I think the first sentence of the last
> paragraph makes it clear (in reference to other messages in this
> thread) why leaving the limit unspecified is not acceptable.  (Which
> is not the same thing as specifying that there is no limit -- that
> solution may have other problems, but it would be clear and explicit.)

I would specify that there is no limit in the standard, but that 2^31
and 2^32 (actually 2^31-1 and 2^32-1) have been (and still are)
practical limits in a lot of existing software, and that it is time to
move on to at least support of 2^64-1.

-- 
Grtz, Ruud