Subject: [NNTP] Large article numbers extension

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 26 Jul 2005 15:59:38 +0100

Firstly, let us learn from the past and *not* call this - or make it - a
"64 bit" facility. Hence the new subject line.

For the purposes of this message, I call the proposed extension BIGNUM.

Let's start by defining a bit of notation: {MC} is the largest article
number that the client can cope with, {MS} is the largest article number
that the server can cope with, and {MA} is the largest article number
currently in use in the server. Draft 27 requires that {MC} and {MS} both
be no less than, and that {MA} be no greater than, 4294967295. {MU} is the
minimum of {MC} and {MS}.

Clients normally get article numbers from the server, rather than making
them up out of whole cloth, so that direction is less of a problem. If it
does send a number greater than {MS}, it has violated a MUST and the server
can respond appropriately (probably a 501 or 400).

So, as I see it, we have three problems to address:
(1) How does a server tell a client that it has article numbers greater
    than 2^32-1 (or 2^31-1) sitting around?
(2) How does a client tell a server that it can cope with article numbers
    greater than 2^32-1?
(3) What does a server do if it has an article number greater than the
    client can cope with?

Question 1 is the easiest. In the CAPABILITIES output, the server includes
the label BIGNUM with argument {MS}:

    [C] CAPABILITIES
    [S] 101 Capability list:
    [S] VERSION 2
    [S] READER
    [S] NEWNEWS
    [S] LIST ACTIVE NEWSGROUPS
    [S] BIGNUM 9876543210987654321
    [S] .

In the absence of this capability, the client assumes {MS} is 4294967295.

Question 2 is also pretty easy. {MC} is sent as a command and {MU} returned
in the response:

    [C] BIGNUM 314159265358979323
    [S] 202 314159265358979323 is our mutual minimum
    [C] BIGNUM 31415926535897932386468
    [S] 202 9876543210987654321 is our mutual minimum

Until this command is used, the server assumes {MC} is 4294967295.

We're now left with question 3. The server provides article numbers in the
following places:

    211 response from GROUP/LISTGROUP
    LIST ACTIVE 215 response content
    NEWGROUPS 231 response content

    22x response from ARTICLE/HEAD/BODY/STAT/LAST/NEXT
    HDR 225 response content
    LISTGROUP 211 response content
    OVER 224 response content

The division is for a reason. The first three cases report the high water
mark of a newsgroup, and are therefore the first place the problem will
appear. The other four cases are less problematic because a number greater
than {MC} can only appear in four situations I can think of:
- GROUP or LISTGROUP was used with a group whose high water mark is greater
  than {MC};
- the high water mark on entry was {MC} but a further article appeared
  later;
- the server is sending an article number, rather than 0, when the client
  specified a message-ID;
- the client used an article number greater than {MC} as an argument.

The third of these can simply be forbidden, and the fourth is already a
MUST NOT. The other two can be addressed simply by using the generic
response
    401 BIGNUM
(possibly with a second argument giving the maximum article number in the
group).

So now we're down to one basic situation: how to report a group with large
article numbers in GROUP, LISTGROUP, LIST ACTIVE, and NEWGROUPS. Here is
what I would propose:

* If the client has not used the BIGNUM command, the server acts as if
  the high water mark is 4294967295, except that NEXT fails with 401 BIGNUM
  instead of 421 ("no next article").
* If the client *has* used the BIGNUM command, article numbers greater than
  the current {MC} are prefixed with a tag (e.g. a plus sign) to indicate
  that they are too big.

Thoughts?

----

A couple of bits of syntactic sugar.

I could live with the BIGNUM command *also* being permitted as an argument
to CAPABILITIES:
    [C] CAPABILITIES BIGNUM 314159265358979323
though I don't see it as a significant need.

I could also live with a notation like "^31" being allowed, in both
directions, as a synonym for 2 to that power minus 1; that is:

    BIGNUM ^40

is equivalent (if my calculator isn't lying) to:

    BIGNUM 1099511627775

though, again, it isn't a major issue.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 27 Jul 2005 10:04:09 GMT

>Thoughts?

The chief problem is that the old client will *never* be able to read the
recent articles in the overflowed group. That is not good.

How does that compare with the alternatives?

1. Wrap-around.

The client would get mightily confused. But, after a while, all the
articles prior to the wrap would have expired, as which point we are back
to the present situation. At that point, the client would be able to read
the group normally. Not good, but surely better than *never* seeing
articles in the group again.

2. Have a flag day.

Has anyone actually looked into how a flag day would be organized?

Presumably the server goes off line for a few hours while it renumbers all
the articles in the affected group, starting from 1 (or perhaps from
some large constant).

What does the client see when it comes back? Presumably its .newsrc file
is all wrong. Somehow it has to reset that line of its .newsrc file, at
which point it will be offered all the already-read articles in the group
as it they were new. Tedious, but manageable, and no articles are lost.
And with a bit of assistance from the server (for example, a list mapping
old numbers to new ones, or making the new numbers differ by a large
constant from the old ones, or keeping the old group around in a renamed
form, with Xref headers of old articles pointing to both groups) things
could be made almost tolerable.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: [email-masked] (Thomas Gschwind)
Date: Wed, 27 Jul 2005 14:00:55 +0200

Hi all!

> Has anyone actually looked into how a flag day would be organized?

Sounds like a good option, if done right.

> could be made almost tolerable.

It is necessary to provide an explicit renumbering table or to keep the
old newsgroup around.  As you say if the numbers differ by a large
constant, the implementation should be fairly simple.  As soon as the
low and high watermark exceed 2^30, simply subtract 2^30 from all the
articles (or 2^31 in case of unsigned article numbers).  Note the
constant SHOULD be highest possible watermark divided by 2.

New clients would see that there was a wrap-around and would do the same
thing.

Old clients would automatically see that the newsgroup was "reset" and
would display many of the read articles again.  That could be solved
with a dirty hack.  If one has a client that uses a .newsrc file, use a
script that copies the .newsrc file away before, starts the news client,
and after termination of the client compares the .newsrc, and "fixes"
the .newsrc file.  Sure one has to restart the client to get see the
articles one has read but that should be OK.

Only problem, if the news server administrator has to reset a newsgroup
and resets the low watermark to one, when the server is close to a
wrap-around.  In this case, new clients would think a wrap-around has
occurred.  But I guess any wrap-around implementation might have this
problem.

Regards,
Thomas
-- 
Thomas Gschwind                      Email: [email-masked]
IBM Zurich Research Lab
Saeumerstrasse 4                     Tel: +41-1-724-8990
CH-8803 Rueschlikon, Switzerland     Fax: +41-1-724-8953

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Wed, 27 Jul 2005 14:24:07 +0200

Thomas Gschwind schreef:

> Only problem, if the news server administrator has to reset a
> newsgroup and resets the low watermark to one

The #1-article could be a special wrap-around-article.

-- 
Grtz, Ruud

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 27 Jul 2005 14:02:42 +0100

> recent articles in the overflowed group. That is not good.

I agree. It isn't.

> the group normally.

Not so. If the client is tracking articles by number, it will see the new
articles as if they were reincarnations of old articles. If the new ones
have numbers less than the low water mark for the group, it's likely to
ignore them completely.

Consider a client using a .newsrc file which contains lists of "read"
articles, and assume that the client uses the LWM to prevent the list
growing unboundedly with never-to-appear articles.

> Has anyone actually looked into how a flag day would be organized?

I believe we discussed it here briefly.

> the articles in the affected group, starting from 1 (or perhaps from

Or by subtracting a large constant from the numbers.

It need not be "a few hours". For example, a traditional Unix spool could
build a shadow directory with the new numbers in, then the transition just
requires two directory renames.

> is all wrong.

Indeed.

> Somehow it has to reset that line of its .newsrc file,

This may be a software option if it isn't a text editing job.

> at

The same is true with wraparound.

> could be made almost tolerable.

Also true with wrap-around. Indeed, it's probably easier, because you just
mark the low half of the number space as unread.

However, in either situation, what if there isn't a .newsrc file you can
edit? This appears to be the situation in Turnpike, for example. [I
normally use it in Newnews mode, so I'm not completely familiar with this
case.]

What it comes to is that, in both cases, clients need to do something
special to recover from the overflow situation. If this is going to be
coded in, it might as well be done properly so as to eliminate the issue.

If we're going to have a BIGNUM extension, I don't see the benefit in
getting too fancy with those clients that don't implement it. Note that the
above wording does not prevent the server having a flag day if too many
customers complain.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 27 Jul 2005 14:06:25 +0100

> constant SHOULD be highest possible watermark divided by 2.

Actually, a better approach is to reduce the numbers by 2^30 when both the
watermarks exceed 2^31 (assuming a 2^32-1 maximum). This means a
wrap-around is much less likely to be confused with a reset.

However, I don't think either approach is something we should be
supporting.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 28 Jul 2005 10:24:24 GMT

>mark the low half of the number space as unread.

Actually, what we are seeing is that there is little difference between a
wrap-around and an intelligently performed flag day.

Indeed, the essential difference is that a wrap-around is written into
some future extension of our standard, so that clients that understand the
extension can detect it and make adjustments to .newsrc files
automatically; whereas with a flag day the server has somehow to warn all
of its clients that they need to take some action (but posting some
articles to the group, to be distributed only locally to that server and
its clients, would be the way to achieve that).

Also, it seems clear that action should be taken well before the upper
limit is reached, and that wrapping around to something rather larger than
'1' would be sensible.

>normally use it in Newnews mode, so I'm not completely familiar with this

Surely, there must be some equivalent of the .newsrc file. OTOH if the
client is relying entirely on NEWNEWS, they it doesn't really care about
article numbers anyway.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 28 Jul 2005 16:20:25 +0100

> wrap-around and an intelligently performed flag day.

Right.

> automatically;

But, I would argue, any work to change clients would be better done by
implementing a large numbers extension instead.

> client is relying entirely on NEWNEWS, they it doesn't really care about

You misunderstand. Turnpike can run (on a per-server basis) in either
NEWNEWS or article number mode. In each case the record of the current
state is stored in binary files, not in a text file that can easily be
edited. My only available options are:
* NEWNEWS mode: change the "last fetched" time for a group to:
  - now
  - the last time a fetch was done
  - a specific time
* Article number mode:
  - fetch all new articles
  - fetch all new articles and refetch the last N already-fetched articles
  - fetch the N most recent articles only
Also note that these relate to fetching articles into the common database;
each user has a separate "articles read" list.

My point, though, is that all this information is hidden away somewhere in
binary files which I have no idea how to edit.

---

From: Steve Walker <[email-masked]>
Date: Thu, 28 Jul 2005 23:56:11 +0700

>> the current {MC} are prefixed with a tag (e.g. a plus sign) to indicate

It doesn't matter what the client can support.  If the numbers can't 
roll over they have no choice but to exceed 32 bits.  It's pointless 
trying to tell the client that it's broken.

Steve.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 29 Jul 2005 11:39:39 GMT

>implementing a large numbers extension instead.

At which point we will have to agree to disagree.

In any case, now is not the time to decide the future solution. All we can
do in this draft is minor tweaks to ensure that the future possibilities
are left as open as possible.

I would not be averse to retreating from unsigned 32bit to signed 32 bit,
and I would not be averse to saying that any special meaning for article
#0 might be removed in a future standard.

If, in the meantime, Steve runs out of numbers, he will just have to
negotiate a flag day with his own clients.

>edited.

Yes, I have that problem too with the Opera Mail engine, which uses a
totally opaque format for its files. Which is fine until the format gets
corrupted, and then you have no means of fixing it.

BTW, another way of dealing with a flag day is to delete that group
entirely from your system, and then resubscribe to it. Than way, even
Turnpike should cope with the change, but you still have a massive
'catchup' to do.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 29 Jul 2005 17:19:41 +0100

> At which point we will have to agree to disagree.

Okay.

> are left as open as possible.

Right. I'm with you, but not everyone here is.

> I would not be averse to retreating from unsigned 32bit to signed 32 bit,

Fine.

> #0 might be removed in a future standard.

<FX: sharp intake of breath>

Consider ARTICLE <message-id>.

> Turnpike should cope with the change, but you still have a massive

[Going off-topic]

Turnpike would actually cope quite well. So long as no expiry run is done
between the delete and resubscribe, the articles will remain in its
internal spool. When the resubscribe causes all the articles to be fetched
again, they'll be discarded as duplicates so will only be in the spool
once.

Meanwhile, users won't be presented with them a second time. That's because
Turnpike allocates a unique number internally to each article, and puts
*those* numbers in the .newsrc equivalent.