Subject: [NNTP] Thoughts on article number resetting/rollover

From: Ken Murchison <[email-masked]>
Date: Thu, 11 Aug 2005 09:44:09 -0400

The whole discussion of the 2^32 limit for article numbers and rollover 
got me thinking a little.  Rolling the numbers over is really no 
different than if the server unilaterally resets the article numbers for 
some reason (server crash, software change/upgrade).  Obviously admins 
try to avoid this at all cost, but I'm sure it happens.  In either case, 
the server should have some mechanism to tell the client that the 
article numbers have changed, and that any cached data that it has is 
invalid.

We need look no farther than IMAP (which also has a 2^32 limit for 
article/message numbers BTW) for a solution.  Each mailbox on an IMAP 
server has a UIDVALIDITY metadata item (usually a timestamp).  Any time 
the existing UIDs of a mailbox are reset/reordered, the server changes 
the UIDVALIDITY on the mailbox (new messages appended to the mailbox 
does NOT cause any change to UIDVALIDITY).  When the client sees a 
change in the UIDVALIDITY, it flushes its cache and re-reads the mailbox 
to get the new message UIDs and contents.

A mechanism like this seems like a pretty trivial extension to NNTP (one 
extra param in the GROUP, LISTGROUP, and LIST ACTIVE responses) and 
would solve both the rollover and resetting of article numbers problems. 
  This extension would be orthogonal to any BIGNUM extension and could 
co-exist peacefully.

Thoughts?

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     2495 Main St. - Suite 401
716-604-0088 x26      Buffalo, NY 14214
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Steve Walker <[email-masked]>
Date: Thu, 11 Aug 2005 11:53:39 -0500

Ken Murchison wrote:

> The whole discussion of the 2^32 limit for article numbers and 
> rollover got me thinking a little.  Rolling the numbers over is really 
> no different than if the server unilaterally resets the article 
> numbers for some reason (server crash, software change/upgrade).  
> Obviously admins try to avoid this at all cost, but I'm sure it 
> happens.  In either case, the server should have some mechanism to 
> tell the client that the article numbers have changed, and that any 
> cached data that it has is invalid.

The wording in the current proposal expressly forbids reseting the 
article numbers....

>6. The final type of key is the arrival timestamp, giving the time that
>   the article arrived at the server.  The server MUST ensure that
>   article numbers are issued in order of arrival timestamp; that is,
>   articles arriving later MUST have higher numbers than those that
>   arrive earlier.  The server SHOULD allocate the next sequential
>   unused number to each new article.

Any reset or wrapping is a violation of the MUST above.  This is why I said
this RFC is completely broken.  People are forced to violate it in one 
way or the other.

>   Article numbers MUST lie between 1 and 4,294,967,295 inclusive.  The
>   client and server MAY use leading zeros in specifying article
>   numbers, but MUST NOT use more than 16 digits.  In some situations,
>   the value zero replaces an article number to show some special
>   situation.

The two MUST's together cause a deadlock.

Personally I really can't see why anyone would have a problem with just 
changing the limit above to 2^64-1. It doesn't break anything that 
wasn't already going to eventually be break and it allows everyone the 
chance to prepare for the future.

Clive ask for wording so here goes:

>   Software MUST be able to correctly process article numbers between 1 and 
>   4,294,967,295 inclusive and SHOULD be capable of processing article 
>   numbers up to 2^64-1.  The client and server MAY use leading zeros 
>   in specifying article numbers, but MUST NOT use more than 20 digits.
>   In some situations, the value zero replaces an article number to show 
>   some special situation.

If anyone is really worried about detecting if a server is using 64 bit 
numbers you could add something like
this so a client knows as soon as it issues a GROUP command if the range 
is beyond it's capabilities.

 > Clients that convert article numbers into internal binary data should 
use 64 bit unsigned
 > integers if the server returns any zero padded article number or 
range component with a
 > length greater than 16 octets and less than or equal to 20 octets.
 >
 > Servers that offer articles with numbers greater than 2^32-1 MUST 
zero pad all
 > article numbers or ranges in command responses to a total  length 
greater than 16 octets
 > and less than or equal to 20 octets.

A 32 bit client still can't read 64 bit article numbers, but at least 
one person seem concerned
that a client would not know if their internal binary conversion was 
invalid.  I personally think a
good atoi function that returned MAXINT as an overflow error would be 
simpler.

Steve Walker

---

From: "Andrew - Supernews" <[email-masked]>
Date: Fri, 12 Aug 2005 03:00:27 +0100

>>>>> "Ken" == Ken Murchison <[email-masked]> writes:

 Ken> The whole discussion of the 2^32 limit for article numbers and
 Ken> rollover got me thinking a little.  Rolling the numbers over is
 Ken> really no different than if the server unilaterally resets the
 Ken> article numbers for some reason (server crash, software
 Ken> change/upgrade).

It's actually very different.

The most common cases where article numbers are reset are (in roughly
descending order of probability):

1) the client ends up talking to a completely different server, where
there is no relationship between the old and new numbers at all
(e.g. when a service is outsourced, or the outsource provider is
changed; ISPs are _incredibly_ bad at managing this changeover, and
will almost never give users any warning)

2) the group is reset completely on the server with all articles
removed, and the numbering reset either to 0 or to some arbitrary
point

Everything else is likely to be down in the noise; in particular,
renumbering of existing articles within a group is something that
servers as a rule simply do not do.

 Ken> We need look no farther than IMAP (which also has a 2^32 limit
 Ken> for article/message numbers BTW) for a solution.  Each mailbox
 Ken> on an IMAP server has a UIDVALIDITY metadata item (usually a
 Ken> timestamp).  Any time the existing UIDs of a mailbox are
 Ken> reset/reordered, the server changes the UIDVALIDITY on the
 Ken> mailbox (new messages appended to the mailbox does NOT cause any
 Ken> change to UIDVALIDITY).  When the client sees a change in the
 Ken> UIDVALIDITY, it flushes its cache and re-reads the mailbox to
 Ken> get the new message UIDs and contents.

Hard to coordinate this between independent, non-cooperating servers
(see the outsource case above).

 Ken> A mechanism like this seems like a pretty trivial extension to
 Ken> NNTP (one extra param in the GROUP, LISTGROUP, and LIST ACTIVE
 Ken> responses)

You think you can alter those responses without breaking _every single
client_ in existence?

-- 
Andrew, Supernews
http://www.supernews.com

---

From: Ken Murchison <[email-masked]>
Date: Fri, 12 Aug 2005 09:36:22 -0400

> (see the outsource case above).

I wasn't trying to solve any inter-server problems.

> You think you can alter those responses without breaking _every single

I believe so, unless those clients are already broken.  We *should* be 
able to add extra arguments to these responses without breaking existing 
clients.  NNTP-BASE says: "The server MAY add any text after the 
response code or last argument as appropriate, and the client MUST NOT 
make decisions based on this text."  In fact, I was told that this is 
the exact reason why we couldn't add an auto-capabilities response to 
the "free text" area of a response.

Here's a potential strawman:

- Add a new UIDVALIDITY capability to the CAPABILTIES response, which 
tells clients that the GROUP/LISTGROUP and LIST ACTIVE responses contain 
UIDVALIDITY as an extra argument.

-Extend GROUP/LISTGROUP response:
211 number low high group uidvalidity

- Extend LIST ACTIVE response:
215 list follows:
group high low status uidvalidity
...

Clients that don't support the UIDVALIDITY extension, *should* ignore 
the extra arguments (not sure about the LIST ACTIVE multi-line response 
though).

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 12 Aug 2005 16:21:17 +0100

> 211 number low high group uidvalidity

<FX: blinks>

That would work. It breaks, or at least seriously bends, the paradigm of
"responses have a fixed number of arguments", but it's completely backwards
compatible.

[The name needs changing, of course.]

> group high low status uidvalidity

I'm much less sure about this, though. I don't know what clients will do if
there's an extra field.

On the other hand, if you made the status be "yu=1234", "nu=1234", or
"mu=1234", where 1234 is the uidvalidity value, this would conform to the
existing spec:

    Other values for the
    status may exist; the definition of these other values and the
    circumstances under which they are returned may be specified in an
    extension or may be private to the server.  A client SHOULD treat an
    unrecognized status as giving no information.

Something akin to this would provide a good basis for a extension for
"group reset", complementary to BIGNUM.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: Russ Allbery <[email-masked]>
Date: Fri, 12 Aug 2005 10:04:11 -0700

> I'm much less sure about this, though. I don't know what clients will do if

I haven't actually checked, but I'm pretty sure at least some clients will
treat the uidvalidity as part of the status flag.  Others will probably
cope.  I'm not sure if any will treat the line as malformed, but it's
possible.

BTW, while I know that the standard requires that the server be able to
add arbitrary text after positional parameters in replies, I would be
leery of relying too heavily on that requirement for widely used replies
with positional parameters.  I know it's supposed to work, but we haven't
had a complete formal standard in a long time and clients written against
existing practice by experimentation likely won't handle that properly.

> "mu=1234", where 1234 is the uidvalidity value, this would conform to
> the existing spec:

>     Other values for the status may exist; the definition of these other
>     values and the circumstances under which they are returned may be
>     specified in an extension or may be private to the server.  A client
>     SHOULD treat an unrecognized status as giving no information.

Which means that clients not supporting this extension will lose
information about whether the group is moderated.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>