Subject: [NNTP] 64-bit article counter extension strawman

From: Ken Murchison <[email-masked]>
Date: Sun, 17 Jul 2005 20:44:42 -0400

I gave some more thought to this today and came up with the following:

- Servers which support 64-bit (large) article counters advertise a 
capability such as 'LARGEGROUP' (or some better named capability).

- If a group has exceeded the 32-bit article count, and the server 
supports the extension, then it uses a new 212 response code to 
GROUP/LISTGROUP which tells the client that the response may contain 
64-bit numbers.

- Similarly, a new 216 response code is used for LIST ACTIVE if any of 
the requested groups has exceeded the 32-bit article count.

- Clients which don't support this extension will not understand the new 
response codes and *should* fail elegantly.

As an alternative:

- Servers which support 64-bit (large) article counters advertise a 
capability such as 'LARGEGROUP' (or some better named capability).

- Servers always hide (411 response for [LIST]GROUP, ommission from LIST 
ACTIVE response) groups which have exceeded the 32-bit article count 
from clients until the client tells the server that it can support such 
groups.

- A client tells the server that it supports large groups by using the 
new keyword 'LARGEGROUP' as an argument to the CAPABILITY command.

- The server can then use the existing response codes for the 
[LIST]GROUP and LIST ACTIVE commands.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Russ Allbery <[email-masked]>
Date: Sun, 17 Jul 2005 17:59:50 -0700

> response codes and *should* fail elegantly.

This is pretty nasty from a backward compatibility standpoint.  Old
clients that rely on LIST ACTIVE would potentially be unable to use the
server at all.  Plus, I hate to change the response code for a standard
command.

> [LIST]GROUP and LIST ACTIVE commands.

I like this a lot better; this was the sort of extension that I had in
mind.

One possible variation would be to add a new command other than GROUP for
accessing large-numbered groups and a new command other than LIST ACTIVE
for retrieving active file information, which would obviate the need for
client-side negotiation and storing additional server state to remember if
the client supports large-numbered groups.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Sun, 17 Jul 2005 21:21:01 -0400

> the client supports large-numbered groups.

I think I agree with this.  The one thing bugging me is that, for 
clients that are already able to handle >32-bit article numbers, it's 
much more hassle this way -- rather than just fiddle the session 
initiation stuff, the client author has to keep state around to know 
which commands to use.

But I really don't want to start piling commands onto the front of 
every NNTP session, and having separate GROUP64 and LIST ACTIVE64 (or 
whatever we call them) seems to be the only way to avoid that.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Steve Walker <[email-masked]>
Date: Sun, 17 Jul 2005 22:01:33 -0500

Jeffrey M. Vinocur wrote:
 > But I really don't want to start piling commands onto the 
front > of every NNTP session, and having separate GROUP64 and LIST
 > ACTIVE64 (or whatever we call them) seems to be the only way to
 > avoid that.

Two commands that do the exact same thing are redundant.  The 
existing 64 bit clean software works fine with the existing 
command set.  What is needed is a method to tell the 32 bit 
software that it won't work and it should go away or accept what 
happens.

It's common practice for servers to zero-pad article numbers. 
What if we just said something like:

Servers that have the capability to handle article numbers 
greater than 2^32-1 MUST zero pad the ranges published by the 
list and group commands.  Clients should expect article numbers 
greater than 2^32-1 if they receive article number responses with 
a text length greater than 10 digits.

The number 2^32-1 fits in a 10 digit text string.  Any zero 
padded number larger than 10 digits is already implying a 
potential article number greater than 10 billion.  Zero padding 
numbers in the list and group output is a very easy and simple 
way to inform the client that article numbers may be given that 
exceed 2^32-1.

As a bonus it fixes the problem without specify any new absolute 
limit, which IMO, a text protocol shouldn't.  If any existing 32 
bit software zero pads greater than 10 digits we could increase 
this number with causing any problems up to 20 digits (64 bit 
number).

Steve.

---

From: Russ Allbery <[email-masked]>
Date: Sun, 17 Jul 2005 20:01:18 -0700

Steve Walker <[email-masked]> writes:

> Two commands that do the exact same thing are redundant.  The existing
> 64 bit clean software works fine with the existing command set.

I would be curious to see a list of software known to already be able to
handle 64-bit article numbers.  You mentioned earlier that you know of at
least one client, but you didn't say what client that was.

> What is needed is a method to tell the 32 bit software that it won't
> work and it should go away or accept what happens.

> It's common practice for servers to zero-pad article numbers. What if we
> just said something like:

> Servers that have the capability to handle article numbers greater than
> 2^32-1 MUST zero pad the ranges published by the list and group
> commands.  Clients should expect article numbers greater than 2^32-1 if
> they receive article number responses with a text length greater than 10
> digits.

(You only want to do this for groups that actually have large article
numbers; there's no reason to prevent 32-bit software from reading all the
groups that haven't needed such large numbers.  Plus, zero-padding all of
LIST ACTIVE would noticably increase the size of the response
unnecessarily.)

This doesn't do what you say is needed above, as existing 32-bit software
will have no idea that additional zero padding means this.  The software
would have to be modified to add this interpretation, which means it will
be years, perhaps decades, before one can be reasonably assured that all
client software will recognize this signal.

I think it would be far better to have a signal that's recognized by
existing software without requiring any modifications.  Both of Ken's
proposals would do that.

> The number 2^32-1 fits in a 10 digit text string.  Any zero padded
> number larger than 10 digits is already implying a potential article
> number greater than 10 billion.  Zero padding numbers in the list and
> group output is a very easy and simple way to inform the client that
> article numbers may be given that exceed 2^32-1.

> As a bonus it fixes the problem without specify any new absolute limit,
> which IMO, a text protocol shouldn't.  If any existing 32 bit software
> zero pads greater than 10 digits we could increase this number with
> causing any problems up to 20 digits (64 bit number).

This *is* a nice property, and this part of it I do like.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 18 Jul 2005 08:45:04 -0400

> the client supports large-numbered groups.

OK, we have a start.  I think this is an important enough issue (I might 
  be in the minority with Steve on this) that we should have a 
reasonable solution fleshed out before we close the door on making this 
part of NNTP-base.  I still don't see the problem with making 64-bit a 
SHOULD or RECOMMENDation in NNTP-base since no current groups have 
exceeded 32-bits that we know of, but if this is a non-starter, let's at 
least have a viable alternative in sight, so we don't paint ourselves 
into a corner and end up with some crufty hack down the road.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 08:31:13 -0700

Ken Murchison <[email-masked]> writes:

> be in the minority with Steve on this) that we should have a reasonable
> solution fleshed out before we close the door on making this part of
> NNTP-base.  I still don't see the problem with making 64-bit a SHOULD or
> RECOMMENDation in NNTP-base since no current groups have exceeded
> 32-bits that we know of, but if this is a non-starter, let's at least
> have a viable alternative in sight, so we don't paint ourselves into a
> corner and end up with some crufty hack down the road.

My problem with making 64-bit support a SHOULD is twofold: I think it's a
landmine for implementors, and I really, really don't want to make any
further changes to the document at this point, now that we're *finally*
done.  There's always going to be just one more thing.

An extension such as what we're discussing here could be written up in a
separate draft and go forward on its own schedule, and then be considered
for inclusion into the base document during the draft standard revision (I
think we're allowed to do that sort of thing).

---

From: Ken Murchison <[email-masked]>
Date: Mon, 18 Jul 2005 12:16:53 -0400

> think we're allowed to do that sort of thing).

Right.  I just think its important to make sure that we have something 
that we believe is workable sooner rather than later.  If it turns out 
that we can't come up with a viable extension without changes to the 
base spec (which I do NOT believe is the case at this point), then we 
should re-evaluate.

---

From: Steve Walker <[email-masked]>
Date: Mon, 18 Jul 2005 11:24:00 -0500

> landmine for implementors, and I really, really don't want to make any

The same thing can be said for mandating 32 bit support.  I don't 
think you can even claim that most software is 32 bit clean as 
anything that used 'int' is going to break at 31 bits.  We 
currently have a mix of 31, 32 and 64 bit software.  You would be 
better off leaving out any limit if your plans are to address the 
issue latter.  No matter what limit is used, most software needs 
to be reviewed and updated.

> done.  There's always going to be just one more thing.

That's just being lazy.  This is a major problem that is 
guaranteed to cause the complete malfunction of the Usenet system 
at some point.  It's irresponsible to call this a completed 
protocol if it adds a new article number limit and doesn't 
specify what to do when it's reached.

Steve.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 09:37:49 -0700

> base spec (which I do NOT believe is the case at this point), then we

I'd like Clive and, ideally, Andrew to weigh in, and would love to hear
from other working group members as well, but that sounds good to me.
(Andrew's input would be very useful on this.)

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 09:57:22 -0700

Steve Walker <[email-masked]> writes:

> The same thing can be said for mandating 32 bit support.  I don't think
> you can even claim that most software is 32 bit clean as anything that
> used 'int' is going to break at 31 bits.

And either change requires changing all the printf formats, so there is
something to what you say.  However, fixing the 31-bit issue only requires
changing int to unsigned int, which doesn't change the size of the
variable and therefore makes quite a bit of the work easier.  One also
doesn't have to deal with the portability issues around 64-bit types, nor
does one have to figure out how to deal with systems that don't have a
64-bit type.

> We currently have a mix of 31, 32 and 64 bit software.  You would be
> better off leaving out any limit if your plans are to address the issue
> latter.  No matter what limit is used, most software needs to be
> reviewed and updated.

I don't agree.  At the most, software that deals with exceptionally
high-volume groups (read: software that deals with binary groups) will
have to be reviewed and updated sometime in the next ten years or so.
Even 31-bit implementations that are used for reading text groups are
probably fine for the lifetime of Usenet as a protocol.

> That's just being lazy.

You're certainly entitled to your opinion, but insulting me is unlikely to
change mine.  :)

> This is a major problem that is guaranteed to cause the complete
> malfunction of the Usenet system at some point.

Well, it is a major problem for your particular implementation.  It is not
a major problem for the rest of Usenet and will not be for at least
another 10-20 years.

Part of the job of working group chair is to be a release manager.  Part
of the job of a release manager is to recognize that there's always going
to be one more bug fix and apply increasing pressure to stop doing work
and start releasing work.  The way I view this, it's not a veto -- it's
accelerating pressure, which means that the farther we go along, the more
strongly I argue against making changes, but if the group consensus says
make the changes anyway, we still do that.

> It's irresponsible to call this a completed protocol if it adds a new
> article number limit and doesn't specify what to do when it's reached.

We aren't calling it a completed protocol.  That would be the Standard
phase, whereas this is only at the Proposed Standard phase.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 18 Jul 2005 13:03:34 -0400

> (Andrew's input would be very useful on this.)

I dislike specified limits in standards.

At the time this WG group started, the 32 bit rollover seemed like a
good way of dealing with article number that are too high.  I do not
agree that it is appropriate now.  It seems that people are willing
to declare software that overflows at 31-bits as "poor implementations
we don't support" but they are not willing to declare software that
overflows at 32 bits in the same class.  That seems a hard position
to hold rationally.

MSVC has 64 bit integer types for what, 8 years now, and that generates
code that runs on 386's.  GCC has supported "long long" for just about
that long or longer.

A note about breakage seems appropriate.  Servers MAY roll over at
31 bits seems appropriate to me.

Now that we see GigE common, TCP sequence numbers of 32 bits are
kind of uncomfortable, due to the blind RST and similar spoofs.
32 bits seemed like a good idea at the time, then, I'm sure, and
there is the advantage of saving bits in the TCP header.

But we don't need to save bits in NNTP, so I'd not even specify
64 bits...except that leaving it open-ended seems to promote abuse of
article number as a structured number, instead of a plain sequence
number.  (E.g. some scheme that gives you filtered articles if
you add 2^56 to the article number, or other kludges.)

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 10:20:29 -0700

> code that runs on 386's.  GCC has supported "long long" for just about

I'm not really disagreeing with what you're saying, just making sure that
everyone understands what I'm talking about.

Support of the 64-bit type itself is, as previously noted, not really the
gating factor.  It's the supporting library functions that are more of a
problem, and GCC can't help with your printf implementation (for example).

> A note about breakage seems appropriate.  Servers MAY roll over at 31
> bits seems appropriate to me.

Do we want to allow rolling over?  Rolling over comes with its own set of
bugs, in particular triggering the low > high logic that causes many
clients to consider the group empty.  I'm not sure we want to allow *both*
rolling over *and* longer article numbers.  We would get all the different
breakage scenarios at the same time.

> But we don't need to save bits in NNTP, so I'd not even specify 64
> bits...except that leaving it open-ended seems to promote abuse of
> number.

Saying "there's no limit and you should be prepared for any length of
article number" says to client authors "you must include a bignum
library."  That seems a little excessive, and in practice I doubt clients
will actually do this.  I think we need to be very careful about what we
say about allowed or recommended client behavior if we really don't
specify any limit.

---

From: Steve Walker <[email-masked]>
Date: Mon, 18 Jul 2005 13:04:42 -0500

> problem, and GCC can't help with your printf implementation (for example).

Glibc supports 64 bit numbers as does the Microsoft stuff as far 
as I'm aware.  There really isn't much out there that couldn't 
support at least 64 bit numbers.  And if a system didn't it 
wouldn't break until the 31/32 bit limit was reached so why would 
it really matter.  It was doomed to break anyway.

> bugs, in particular triggering the low > high logic that causes many

I think rolling over is prone to problems and it's not worth it 
when there is there is a good clean method of using 64 bit numbers.

> say about allowed or recommended client behavior if we really don't

It worked for 977 so why wouldn't it continue to work.  It would 
be just like file systems limits.  Each file system is rated to x 
number terabytes.  We would have servers/clients that claim we're 
rated to article numbers up to x billion.  I'm not saying this is 
a good thing.  I'm just saying this is what will happen as soon 
as servers start exceeding 31 bits.

If you specify a 32 bit limit in the protocol client authors will 
not know what is the proper method to fix the overflow problem. 
Everyone will be forced to guess and we will have a mess.

Steve.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 18 Jul 2005 13:55:40 -0400

> say about allowed or recommended client behavior if we really don't

Put in a NOTE: that the WG considered 64-bit article numbers to be adequate
to handle X articles per day at 20% growth/year until YYYY.

I suppose we could argue about the growth rate, but really, the use of
the protocol cannot grow faster than available bandwidth or CPU power.

If BIGNUMs weren't so messy to store and load, they would be a good choice.
They are not terribly expensive in code size, execution data size, complexity,
or CPU cycles.   Using them creates portable code, instead of the mish-mosh of
ways to declare integer types in C wider than 32 bits on different platforms
and compilers, not to mention all the different ways sscanf and printf and
friends do it on different platforms.

But my guess is that with a note like the above, most implementors would
do 64-bits in the likelihood that they would be deceased before the limit were
reached, modulo any successful cryogenic or life-extending technologies.

---

From: Chin Chee-Kai <[email-masked]>
Date: Tue, 19 Jul 2005 01:29:50 +0800 (SGT)

Hi,

Sorry for jumping in, but I was having some interest reading about
the 32bit/64bit discussion.  I thought I'd give a view from a 
bit of different angle hopefully it could help.

Basically I think RFC977 gives more freedom in terms of number
range than a specific limit as proposed (either 32bit, 64bit, 128bit,
etc).  To be specific, RFC977 doesn't really say "no limit".
It does restrict command line to a usable of 510 bytes.
So ignoring some 4 to 10+ bytes of command word and using all 510 bytes
to print a decimal number, RFC977 implies a number that is no
more than 1,695 bits (about 212 bytes).  Technically, if you
have a fixed precision counter package that can hold that size
(rather than a more elaborate arbitrary precision package), it 
would suffice to be compliant with the RFC's number size requirement.

So all 31bit, 32bit, 64bit etc implementations are approximations
to deal with this 1,695bit requirement.  And like Steve said,
they need to be reviewed from time to time.  And when that happens,
a 10-year-old client might just be upgraded to 64bit in preparation
for future, and an 8-year-old server might be armed with 128bit
counter to take care of "just-in-cases", for example.

So the "problem" isn't so much with the standard, the protocol or 
limits, but with the way implementations have somewhat arbitrarily 
taken their own approximation methods to cope with the size requirement.
So assuming with me this for a moment, then, trying to "fix the
problem" by adding commands, options, changing semantics of existing 
commands, etc wouldn't seem to be a right approach.

I agree with Russ on the need for stable specs and with 
interoperability in mind, if there's really any thing that needs
to be done.  It might be the case that clients that have implemented
with only 31bits of 'int' (using Steve's example) would be a 
"too bad" case - the user would notice lesser articles and
start to upgrade or change.  This is reminiscent of Y2K changeoever.

If, just if, there's anything that needs to be done, I think it
would have to be a non-obliging signal.  "Non-obliging" in the
sense that the server/client sends out this signal of >32bit number
voluntarily, such as MAY respond with an unused 2xx success code.
The receiving client/server that is cognizant of the code would
either deal with it, or prompt user that it can't handle large
articles and give a graceful termination.

Just some thoughts.

Best Regards,
Chin Chee-Kai
SoftML
Tel: +65-6820-2979
Fax: +65-6743-7875
Email: [email-masked]
http://SoftML.Net/

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 11:04:33 -0700

>> Support of the 64-bit type itself is, as previously noted, not really
>> the gating factor.  It's the supporting library functions that are more
>> of a problem, and GCC can't help with your printf implementation (for
>> example).

> Glibc supports 64 bit numbers as does the Microsoft stuff as far as I'm
> aware.  There really isn't much out there that couldn't support at least
> 64 bit numbers.

I just refuted this statement in the paragraph immediately above your
response.  If you want more details or have questions about the specifics
of what I said, could you please ask them, rather than simply asserting
that I'm wrong without any additional support?

>> library."  That seems a little excessive, and in practice I doubt
>> clients will actually do this.  I think we need to be very careful
>> about what we say about allowed or recommended client behavior if we
>> really don't specify any limit.

> It worked for 977 so why wouldn't it continue to work.

No one actually implemented this aspect of RFC 977 that I am aware of.  If
you know of a client implementation that supports arbitrarily long article
numbers, I'd love to hear about it.  If not, then saying that it worked
for 977 isn't a particularly useful statement, since RFC 977 itself didn't
actually work as a standard in this area.

One of the things that we're trying to *not* do is emulate RFC 977's
problems, one of which being that portions of it are tacitly ignored by
all known implementations without any warning in the standard for future
implementors that this is the case.

---

From: Steve Walker <[email-masked]>
Date: Mon, 18 Jul 2005 13:38:44 -0500

> that I'm wrong without any additional support?

I didn't mean to step on your toe's.  I apologize if you saw it 
that way, it was not my intention.  The question (or I guess it 
was more of a statement) was are there any modern systems out 
there that can not use either the glibc or windows libraries? 
Note that the glibc libraries are not the same as a gcc compiler.

> actually work as a standard in this area.

It work in the sense that everyone used a binary number that 
would hold a number large enough to get the job done.  Why didn't 
people use 8 or 16 bits numbers?  They choose a number that 
matched the current needs.  If we left it open the same would 
happen again.  People would slowly migrate to the next larger 
binary number, in this case 63/64 bit.

Steve.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Jul 2005 11:34:29 -0700

Steve Walker <[email-masked]> writes:

> I didn't mean to step on your toe's.  I apologize if you saw it that
> way, it was not my intention.

Thank you... that was graciously said.  No troubles.

> The question (or I guess it was more of a statement) was are there any
> modern systems out there that can not use either the glibc or windows
> libraries?

Yes -- all versions of Unix other than Linux do not use glibc.  glibc is
not portable to versions of Unix other than Linux; some people have done
occasional one-off ports, but the maintainers of glibc disavow all
knowledge of them and refuse to support them in any way.  And extracting
and using portions of glibc is prohibitively difficult.

Trying to port old code and introduce things like 64-bit types is
something that unfortunately I have some experience with.  It's still very
obnoxious to do in a sane fashion.  If one just assumes Linux, or maybe
Linux and *BSD, it's much easier, but most Unix software maintainers are
loathe to do that.

> It work in the sense that everyone used a binary number that would hold
> a number large enough to get the job done.  Why didn't people use 8 or
> 16 bits numbers?  They choose a number that matched the current needs.
> If we left it open the same would happen again.  People would slowly
> migrate to the next larger binary number, in this case 63/64 bit.

The part of this that bothers me is that a server has no way of knowing
whether incrementing that article number just one more is going to cause a
bunch of clients to break (and possibly not just for that group).  But
perhaps I'm too fixated on this, and it's not really as much of an issue
as it feels like to me.

---

From: "Andrew - Supernews" <[email-masked]>
Date: Mon, 18 Jul 2005 23:26:32 +0100

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 Russ> I'd like Clive and, ideally, Andrew to weigh in, and would love
 Russ> to hear from other working group members as well, but that
 Russ> sounds good to me.  (Andrew's input would be very useful on
 Russ> this.)

I did some statistics on our active file (which has been running
without any number resets for a good many years) and current group
traffic levels.

If you assume unchecked exponential growth at a rate extrapolated from
past growth in each specific group, then we have maybe 10-20 groups
that are likely to run into problems within the 5 to 7 year timeframe
(none sooner). It's worth noting that if those assumptions hold, then
the feed volume at that time will be perhaps 48 TB/day, and the
busiest groups will be getting on the order of 10 million
articles/day, which has obvious implications for any scheme based on
resetting article numbers. (It also means that downloading the
overview alone for one such group will consume 4GB of transfer per
day, taking half an hour or so even at 20 megabits.)

It's obviously debatable how useful these assumptions are.

If you assume anything other than exponential growth, then the problem
is of course much further off (10 years or so even for the worst cases).

On the other hand, I would be quite astonished if there were more than
two clients in existence that allowed for 64-bit article numbers,
especially given that there are no servers generating such numbers.
Given the extent to which existing clients break at 2^31 or 2^32, it
is clearly unwise for servers to do so.

So there are several (non-independent) choices:

1) Leave the issue for now and work on it for Draft Standard. This is,
in my opinion, highly desirable.

2) Add language along these lines:
   -  clients SHOULD accept article numbers up to 2^63-1 (NOT 2^64-1)
   -  servers SHOULD NOT generate article numbers over 2^31-1
   -  servers MUST NOT generate article numbers over 2^63-1

3) Keep the 2^32 limit but add an extension for large numbers.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 00:46:24 +0200

Russ Allbery <[email-masked]> writes:

> changing int to unsigned int,

and again all printf from %[l]d to %[l]u...

-- 
Matthias Andree

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 00:56:54 +0200

> problem, and GCC can't help with your printf implementation (for

practically, software will state minimum requirements to the environment
- and uint64_t has been around since SUS v2 (1997) at the latest, I
don't bother to check how old it really is.

If printf doesn't cut it, well, you'll find dozens of liberally licensed
implementations to format 64-bit unsigned integers into strings on the
net. Besides that, the standard won't specify implementors have to use
printf...

> say about allowed or recommended client behavior if we really don't

Which is a compelling reason /not/ to allow numbers of arbitrary length.

-- 
Matthias Andree

---

From: Matthias Andree <[email-masked]>
Date: Tue, 19 Jul 2005 01:10:14 +0200

> It's obviously debatable how useful these assumptions are.

...and how usefuls groups with 10^7 articles/day (that's 100 per second)
are and who is going to read them in 10 ms...

> is clearly unwise for servers to do so.

Here's another assumption: a client that uses the "unsigned long" C type
(or equivalent in other languages), anything that is the natural word
size of the system, will transparently support 64-bit numbers when
compiled for 64-bit - and this magic "64" has been at least sort of
natural type for ages. Same for the server, even more likely there.

>    -  clients SHOULD accept article numbers up to 2^63-1 (NOT 2^64-1)

I see no particular reason why 2^63-1 is used here rather than
2^64-1. It only lets implementors get away with inconsistent code.

>    -  servers SHOULD NOT generate article numbers over 2^31-1

Maybe. I'll happily ignore it on grounds that I'm not putting up
arbitrary limits if my code does 2^32-1 or 2^64-1 perfectly.

>    -  servers MUST NOT generate article numbers over 2^63-1

Which is nonsense. We may as well allow 2^64-1.

> 3) Keep the 2^32 limit but add an extension for large numbers.

One word: Ugh. Translated: this is bloat for no good reason.

-- 
Matthias Andree

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 19 Jul 2005 09:51:59 GMT

>when there is there is a good clean method of using 64 bit numbers.

And I think exactly the opposite. It is ridiculous using huge numbers
(approaching 64 bits) to identify articles in a group where the maximum
number of articles actually existing at any one time is unlikely to exceed
100,000, and probably nowhere near that.

So we should look at wrap-around solutions (several variants have been
mentioned). What are the problems? Special meanings for zero, and several
ways of detecting whether a group is empty. Is that it?

If that is so, then we should be exploring whichever wrap-around scheme
will cause least disruption to introduce. For example, would reducing the
number of ways to indicate an empty group help? If so, then you say that
method X is deprecated, _and_will_be_removed_in_a_future_standard_. We are
in no hurry to solve this problem (we have at least 10 years), and so all
we need is to devise a scheme that could safely be standardized in 10
years time, provided we give notice of intention NOW.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Steve Walker <[email-masked]>
Date: Tue, 19 Jul 2005 10:14:26 -0500

>100,000, and probably nowhere near that.

A few years ago and I would have agreed.  Today, it's not unreasonable 
to have a binary group with 100 million active articles.  If your server 
software assigns unique article numbers you also will use numbers much 
faster as all articles are in a single "master" group.  This server 
design has some very good benefits with the only downside being that you 
will reach the 32 bit limit in about 2 years on a full feed server.

>ways of detecting whether a group is empty. Is that it?

Clients have to add a lot of logic for detecting the wraparound.  The 
number zero is a reserved article number, but to do simple % math we 
really need it as a valid number. The xover command has to support a 
high < low number request.   In general, it will be prone to errors. 
Wrapping can work, but it would be complex.

Steve.

---

From: "Andrew - Supernews" <[email-masked]>
Date: Tue, 19 Jul 2005 16:24:40 +0100

>>>>> "Charles" == Charles Lindsey <[email-masked]> writes:

 Charles> And I think exactly the opposite. It is ridiculous using
 Charles> huge numbers (approaching 64 bits) to identify articles in a
 Charles> group where the maximum number of articles actually existing
 Charles> at any one time is unlikely to exceed 100,000, and probably
 Charles> nowhere near that.

If the number of existing articles isn't likely to exceed 100,000 then
non-cyclic 32-bit numbers will suffice anyway.

If exponential growth continues to the point where 32-bit numbers are
no longer enough, then you're looking at groups with perhaps 100 to
200 _million_ articles in at any given time (10 million per day, 10 to
20 days retention). Furthermore, five more years of the same growth
pattern and you're looking at groups with more than 2^31 articles in -
and so cyclic numbering breaks down.

Cyclic numbering would only be a win if you think that exponential
growth will continue long enough to break the 32-bit limit but then
suddenly stop or slow down. That seems improbable.

(63-bit numbering is safe for at least a further 40 years even
assuming that exponential growth continues unchecked.)

-- 
Andrew, Supernews
http://www.supernews.com

---

From: Matthias Andree <[email-masked]>
Date: Wed, 20 Jul 2005 00:19:26 +0200

> 100,000, and probably nowhere near that.

More like a million in groups that have been in existence for several
years and have constantly been carrying high traffic volumes. As I'm
typing this, I'm offline so I cannot check the de.rec.fotografie
counters, but the group has certainly seen much more than 100,000 posts
so far.

> ways of detecting whether a group is empty. Is that it?

Wrap-around is just nonsense and should be expressly forbidden, with a
"MUST" (or MUST NOT, as appropriate).

It causes an enormous complexity for getting all the tiny implementation
details right, and it requires carrying around lot of code for a special
case.  While the same argument can be mounted against a 31->64-bit
switchover when we're in fact using, say, 20 bits, the 64-bit code
requires no changes to the logic, no adding ifs and buts to detect
wrap-around, unwrap, and all that.

In the easiest case, the article number is implemented as a type or
class in its own right and a programmer just picking a wider type is
done with the job.

> will cause least disruption to introduce.

That scheme would be "no wrap-around allowed", and I am serious about
this, no matter how much advance warning you give.

-- 
Matthias Andree

---

From: Russ Allbery <[email-masked]>
Date: Tue, 19 Jul 2005 18:42:04 -0700

> "MUST" (or MUST NOT, as appropriate).

Wrap-around is currently expressly forbidden.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 20 Jul 2005 11:39:04 GMT

>Wrapping can work, but it would be complex.

Whatever means we use to solve this, implementations are going to need to
change. Currently, some implementations use high=low=0 to indicate an
emtpy group. That already breaches a SHOULD in the present draft. If we
were to say _now_ that the next standard would prohibit that method
entirely, that gives people a few years to phase it out.

AFAICS, once that is gone wrap-around solutions at least become
_possible_. But we don't need to make the final dicision now - that is for
the next standard. All I am suggesting is that we open up the possibility
now so as to leave more room for manoevre later.

As I have already said, current implementations will not cope either way
in a few years time.

---

From: Steve Walker <[email-masked]>
Date: Fri, 22 Jul 2005 09:15:10 -0500

There seems to be a general agreement that a limit is wanted for 
practical reasons.  Is this correct?  If that is the case, would 
changing the exiting wording to replace the 10 character/ 32 bit number 
with 20 character 64 bit numbers be acceptable?  Maybe placing a short 
warning that some clients may not handle numbers larger than 31 bit?

Steve.

---

From: Matthias Andree <[email-masked]>
Date: Sun, 24 Jul 2005 03:04:34 +0200

> warning that some clients may not handle numbers larger than 31 bit?

Works for me.

-- 
Matthias Andree

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 25 Jul 2005 10:25:42 +0100

> from other working group members as well, but that sounds good to me.

Clive has been on holiday for the last week, but will weigh in after he
catches up with incoming email.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 26 Jul 2005 16:06:49 +0100

> don't bother to check how old it really is.

It's been around, but it's not been supported by all platforms.

uint421_t has been around since 1999 formally, and since 1996 in drafts,
but it still isn't supported by many platforms.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 26 Jul 2005 16:09:14 +0100

> I dislike specified limits in standards.

Agreed, though sometimes you have to live with them.

> good way of dealing with article number that are too high.

We never agreed text with a rollover mechanism in it. The final consensus
was that the client would have to have a one-off breakage if it hit the
limit.

> It seems that people are willing
> overflows at 32 bits in the same class.

I'm not doing that. Indeed, if anyone had pointed out to me - 8 years ago -
that code used signed ints rather than unsigned, I would have used 2^31-1
instead in the text I wrote.

> code that runs on 386's.  GCC has supported "long long" for just about

Not all of us use only Windows and GCC.