Subject: [NNTP] Article Reinstatement, Was Article Numbers Becoming Invalid (RFC 3977)

From: Sabahattin Gucukoglu <[email-masked]>
Date: Sun, 1 May 2011 20:32:15 +0100

Okay, so I *finally* read through all the postings for the thread which I started.  Sorry about that!  Now I am happy with the outcome, it confirms my understanding of the specification, and all the errata are good.  But as far as I can see there is still one more problem ...

Article reinstatement, besides the stated uses of gradual cluster-wide synchronisation (which makes perfect sense) poses a small problem for many clients using articles as intended, monotonically increasing unique per-server references to articles.  If a client should not be aware that an article has been removed pending reinstatement, it has no reason to suspect that the low watermark will ever decrease.  Even if it wanted to, it could not straightforwardly obtain any articles that were added, and then removed pending reinstatement, both in the client's absence, when they were finally reinstated.

Example: news.test has 5 messages, min=1 max=5.  Article 1 is removed.  Server indicates on group entry to news.test, min=2 max=5 no=4.  The client updates its counters to indicate that article 5 was the last article seen.  Article 1 is replaced.  Client connects, server indicates min=1 max=5 no=5.  How is the client to ensure the retrieval of article 1?  The low watermark may certainly indicate its presence, but it can hardly assume that an article was reinstated simply because on this occasion min=1 is the smallest value seen; the article numbered 1 may already have been fetched under normal conditions, or the heuristic will not work at all if group expiration on criteria other than number of articles has occurred, for instance.

Does anybody have any ideas?

Cheers,
Sabahattin

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 13 May 2011 19:06:11 +0200

Hi Sabahattin,

> If a client should not be aware that an article has been removed
> pending reinstatement, it has no reason to suspect that the low
> watermark will ever decrease.

The low watermark can not decrease.

> Example: news.test has 5 messages, min=1 max=5.  Article 1 is
> removed.  Server indicates on group entry to news.test, min=2 max=5
> no=4.  The client updates its counters to indicate that article 5 was
> the last article seen.  Article 1 is replaced.  Client connects,
> server indicates min=1 max=5 no=5.  How is the client to ensure the
> retrieval of article 1?

It is not an RFC-compliant behaviour.
min could not have been set to 2.

RFC 3977, Section 6.1.1.2:

    o  Articles may be reinstated in the group with the same article
       number, but those articles MUST have numbers no less than the
       reported low water mark (note that this is a reinstatement of the
       previous article, not a new article reusing the number).

[...]
    No similar
    assumption can be made about the high water mark, as this can
    decrease if an article is removed and then increase again if it is
    reinstated or if new articles arrive.

> The low watermark may certainly indicate its
> presence, but it can hardly assume that an article was reinstated
> simply because on this occasion min=1 is the smallest value seen; the
> article numbered 1 may already have been fetched under normal
> conditions, or the heuristic will not work at all if group expiration
> on criteria other than number of articles has occurred, for
> instance.
>

I do not understand your point.

Wouldn't it rather be:

news.test has 5 messages, min=1 max=5 count=5
1. remove #1
min=1 max=5 count=4
2. reinstate #1 and remove #3
min=1 max=5 count=4

So a client that comes between action 1 and action 2 will never see #1; 
it won't ask for #1 afterwards because the high water mark is 5.  The 
same way as a client coming after action 2 will never see #3.
But is it really a problem?

-- 
Julien ÉLIE

« Il faut que j'en parle à notre chef ; ce n'est pas normal que des
   Romains bravent les dangers de la forêt, surtout quand les dangers,
   c'est nous ! » (Astérix)

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Sat, 14 May 2011 20:08:51 +0100

> The low watermark can not decrease.

Even when the client disconnects?

> min could not have been set to 2.

So what does a server do when the lowest-numbered article is pending reinstatement and the client requests it?

>      previous article, not a new article reusing the number).

This does make sense, but doesn't apply to the high watermark.

>   reinstated or if new articles arrive.

Looking back at my original message, I see that I made an assumption that the high and low watermarks were always valid articles, but I can see now that the RFC has us covered in the special case of reinstatement.  Good. :-)

> 2. reinstate #1 and remove #3

By which we understand that an invalid article response will result from requesting a removed article, yes?

> So a client that comes between action 1 and action 2 will never see #1; it won't ask for #1 afterwards because the high water mark is 5.  The same way as a client coming after action 2 will never see #3.

Taking into account the use of article reinstatement, probably not in many cases, but you can imagine that if you are a news puller or offline reader, you want to go forward in the article stream and not worry about what came before.  Any circumstance that results in article deletion should be permanent, from the client's perspective.  But yes, it's just me, picking nits. :-)

Cheers,
Sabahattin

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 14 May 2011 21:37:52 +0200

Hi Sabahattin,

> Even when the client disconnects?

Yes, the low water mark SHOULD NOT decrease.

Section 6.1.1.2:

    Except when the group is empty and all three numbers are zero,
    whenever a subsequent GROUP command for the same newsgroup is issued,
    either by the same client or a different client, the reported low
    water mark in the response MUST be no less than that in any previous
    response for that newsgroup in this session, and it SHOULD be no less
    than that in any previous response for that newsgroup ever sent to
    any client.  Any failure to meet the latter condition SHOULD be
    transient only.  The client may make use of the low water mark to
    remove all remembered information about articles with lower numbers,
    as these will never recur.

>> It is not an RFC-compliant behaviour. min could not have been set
>> to 2.
>
> So what does a server do when the lowest-numbered article is pending
> reinstatement and the client requests it?

The news server just answers the article does not exist:

    First form (message-id specified)
      430                   No article with that message-id

    Second form (article number specified)
      423                   No article with that number

    Third form (current article number used)
      423                   No article with that number

Note that reinstatement can be done on other articles than the 
lowest-numbered article.

> This does make sense, but doesn't apply to the high watermark.

The high water mark can decrease.
It can happen with cancels for instance.  Note that such a behaviour is 
implementation-specific.  A news server can keep the high water mark 
without decreasing it.

>> [...] No similar assumption can be made about the high water mark,
>> as this can decrease if an article is removed and then increase
>> again if it is reinstated or if new articles arrive.
>
> Looking back at my original message, I see that I made an assumption
> that the high and low watermarks were always valid articles, but I
> can see now that the RFC has us covered in the special case of
> reinstatement.  Good. :-)

And also the fact that articles can expire and no longer be present in 
the news spool.  The low water mark does not necessarily point to a 
still retrievable article at the time an ARTICLE command is sent.

>> news.test has 5 messages, min=1 max=5 count=5 1. remove #1 min=1
>> max=5 count=4 2. reinstate #1 and remove #3 min=1 max=5 count=4
>

Not "invalid", but "inexistent".

>> So a client that comes between action 1 and action 2 will never see
>> #1; it won't ask for #1 afterwards because the high water mark is
>> 5.  The same way as a client coming after action 2 will never see
>> #3. But is it really a problem?
>
> Taking into account the use of article reinstatement, probably not in
> many cases, but you can imagine that if you are a news puller or
> offline reader, you want to go forward in the article stream and not
> worry about what came before.  Any circumstance that results in
> article deletion should be permanent, from the client's perspective.
> But yes, it's just me, picking nits. :-)

It is up to the client to decide how to implement its retrieval of 
articles.  If it assumes the article stream always go forward (which 
happens nearly all the time), then it will miss reinstatements.  That's 
all.  The client just prefers rapidity to exhaustivity (sending a 
LISTGROUP every time, or every n times, it connects to a news group).

-- 
Julien ÉLIE

« Pour une personne optimiste, le verre est à moitié plein. Pour une
   personne pessimiste, il est à moitié vide. Pour l'informaticien, il
   est deux fois plus grand que nécessaire. »

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Sat, 14 May 2011 21:00:22 +0100

>    The client may make use of the low water mark to

This was written like that so as to support the common behaviour of a
.newsrc file that contained lines like:

    uk.railway: 1-85623,85625,85629-85641

which listed the articles to be treated as read. If the server offers a
low-water mark of 85627, the client knows that 85624 and 85626 are *never*
going to get read and can change the line to:

    uk.railway: 1-85626,85629-85641

-- 
Clive D.W. Feather          | If you lie to the compiler,
Email: [email-masked]     | it will get its revenge.
Web: http://www.davros.org  |   - Henry Spencer
Mobile: +44 7973 377646