Subject: [NNTP] STREAMING diffs (take 2)

From: Ken Murchison <[email-masked]>
Date: Fri, 10 Jun 2005 11:18:04 -0400

2 small editorial tweaks from Jeff and Clive.  Do people feel this this 
has addressed all important issues brought up in last call?

--- streaming-5.txt	2005-06-09 14:17:54.000000000 -0400
+++ streaming-6.txt	2005-06-10 11:15:19.000000000 -0400
@@ -6,13 +6,13 @@

  NNTP Extensions Working Group                                 J. Vinocur
  Internet Draft                                        Cornell University
-Updates: 2970 (if approved)                                 K. Murchison
+Updates: 2980 (if approved)                                 K. Murchison
  Expires: December 2005                                Oceana Matrix Ltd.
                                                                 June 2005

                     NNTP Extension for Streaming Feeds
-                    draft-ietf-nntpext-streaming-05
+                    draft-ietf-nntpext-streaming-06

  Status of this memo
@@ -46,23 +46,31 @@
  Abstract

       This memo defines an extension to the Network News Transport
-     Protocol [NNTP] to provide asynchronous (otherwise known as
+     Protocol (NNTP) to provide asynchronous (otherwise known as
       "streaming") transfer of articles.  This allows servers to transfer
       articles to other servers with much greater efficiency.

-     Section 1 of [NNTP-COMMON] summarizes some ad-hoc transport
-     extensions currently used in the NNTP protocol.  This document
-     updates and formalizes the CHECK and TAKETHIS commands and
-     deprecates the MODE STREAM command.
+     This document updates and formalizes the CHECK and TAKETHIS
+     commands specified in RFC 2980 and deprecates the MODE STREAM
+     command.
+
+Note to the RFC Editor
+
+     The normative reference to RFC 2234 may be replaced by
+     draft-crocker-abnf-rfc2234bis should it reach RFC status before
+     this one.
+
+     The normative reference to [NNTP] is a document which is expected
+     to be published simultaneously with this one and so can be replaced
+     by reference to the resulting RFC.

  Table of Contents

-     0. Changes from Previous Version ............................  2
       1. Introduction .............................................  3
          1.1. Conventions Used in this Document ...................  3
       2. The STREAMING Extension ..................................  3
-        2.1. Advertising the STREAMING Extension .................  3
-        2.2. Streaming Article Transfer ..........................  4
+        2.1. Streaming Article Transfer ..........................  3
+        2.2. Advertising the STREAMING Extension .................  4
          2.3. MODE STREAM Command .................................  5
             2.3.1. Usage ..........................................  5
             2.3.2. Description ....................................  5
@@ -76,44 +84,38 @@
             2.5.2. Description ....................................  7
             2.5.3. Examples .......................................  8
       3. Augmented BNF Syntax for the STREAMING Extension .........  9
-        3.1. Commands ............................................  9
+        3.1. Commands ............................................ 10
          3.2. Command Datastream .................................. 10
          3.3. Responses ........................................... 10
          3.4. Capability entries .................................. 10
-     4. Summary of Response Codes ................................ 10
+     4. Summary of Response Codes ................................ 11
       5. Security Considerations .................................. 11
       6. IANA Considerations ...................................... 11
       7. References ............................................... 12
          7.1. Normative References ................................ 12
-        7.2. Informative References .............................. 12
+        7.2. Informative References .............................. 13
       8. Authors' Addresses ....................................... 13
       9. Acknowledgments .......................................... 13
       10. Intellectual Property Rights ............................ 13
       11. Copyright ............................................... 14

-0. Changes from Previous Version
-
-     Changed:
-     o  Updated to RFC 3978/3979 boilerplate.
-     o  Uses new ABNF tokens from [NNTP].
-     o  Uses new "multi-line data block" term from [NNTP].
-     o  POST has its own capability.
-
-     Clarified:
-     o  Section 3: This document extends the ABNF in [NNTP], and the
-        [NNTP] ABNF must be imported first before validating the
-        STREAMING ABNF (based on recommendations of AD regarding
-        IMAPEXT I-Ds).
-
  1. Introduction

       According to the NNTP specification [NNTP], a peer uses the IHAVE
       command to query whether a server wants a particular article.
       Because the IHAVE command cannot be pipelined, the need to stop and
       wait for the remote end's response greatly restricts the throughput
-     that can be achieved.  The alternative method of peer-to-peer
-     article transfer described in this document permits a more
-     consistent use of network bandwidth.
+     that can be achieved.
+
+     The ad-hoc CHECK and TAKETHIS commands, originally documented in
+     [NNTP-COMMON], provide an alternative method of peer-to-peer
+     article transfer which permits a more effective use of network
+     bandwidth.  Due to their proven usefulness and wide deployment,
+     they are formalized in this specification.
+
+     The ad-hoc MODE STREAM command, also documented in [NNTP-COMMON],
+     is deprecated by this specification, but due to its ubiquity is
+     documented here for backwards compatibility.

  1.1. Conventions Used in this Document

@@ -139,7 +141,36 @@
       This extension provides three new commands: MODE STREAM, CHECK, and
       TAKETHIS.  The capability label for this extension is STREAMING.

-2.1. Advertising the STREAMING Extension
+2.1. Streaming Article Transfer
+
+     The STREAMING extension provides the same functionality as the
+     IHAVE command ([NNTP] section 6.3.2) but splits the query and
+     transfer functionality into the CHECK and TAKETHIS commands
+     respectively.  This allows the CHECK and TAKETHIS commands to be
+     pipelined ([NNTP] section 3.5) and provides for "streaming" article
+     transfer.
+
+     A streaming client will often pipeline many CHECK commands and use
+     the responses to construct a list of articles to be sent by a
+     pipelined sequence of TAKETHIS commands, thus increasing the
+     fraction of time spent transferring articles.  The CHECK and
+     TAKETHIS commands utilize distinct response codes so that these
+     commands can be intermingled in a pipeline and the response to any
+     single command can be definitively identified by the client.
+
+     The client MAY send articles via TAKETHIS without first querying
+     the server with CHECK.  The client SHOULD NOT send every article in
+     this fashion unless explicitly configured to do so by the site
+     administrator based on out-of-band information.  However, the
+     client MAY use an adaptive strategy where it initially sends CHECK
+     commands for all articles, but switches to using TAKETHIS without
+     CHECK if most articles are being accepted (over 95% acceptance
+     might be a reasonable metric in some configurations).  If the
+     client uses such a strategy, it SHOULD also switch back to using
+     CHECK on all articles if the acceptance rate ever falls much below
+     the threshold.
+
+2.2. Advertising the STREAMING Extension

       A server supporting the streaming commands described in this
       document will advertise the "STREAMING" capability label in
@@ -180,35 +211,6 @@
          [S] HDR
          [S] .

-2.2. Streaming Article Transfer
-
-     The STREAMING extension provides the same functionality as the
-     IHAVE command ([NNTP] section 6.3.2) but splits the query and
-     transfer functionality into the CHECK and TAKETHIS commands
-     respectively.  This allows the CHECK and TAKETHIS commands to be
-     pipelined (unlike IHAVE) and provides for "streaming" article
-     transfer.
-
-     A streaming client will often pipeline many CHECK commands and use
-     the responses to construct a list of articles to be sent by a
-     pipelined sequence of TAKETHIS commands, thus increasing the
-     fraction of time spent transferring articles.  The CHECK and
-     TAKETHIS commands utilize distinct response codes so that these
-     commands can be intermingled in a pipeline and the response to any
-     single command can be definitively identified by the client.
-
-     The client MAY send articles via TAKETHIS without first querying
-     the server with CHECK.  The client SHOULD NOT send every article in
-     this fashion unless explicitly configured to do so by the site
-     administrator based on out-of-band information.  However, the
-     client MAY use an adaptive strategy where it initially sends CHECK
-     commands for all articles, but switches to using TAKETHIS without
-     CHECK if most articles are being accepted (over 95% acceptance may
-     be a reasonable metric in some configurations).  If the client uses
-     such a strategy, it SHOULD also switch back to using CHECK on all
-     articles if the acceptance rate ever falls much below the
-     threshold.
-
  2.3. MODE STREAM Command

       Historically this command was used by a client to discover if a
@@ -240,9 +242,11 @@

       If a server supports this extension, it MUST return a 203 response
       to the MODE STREAM command (or 501 if an argument is given).  The
-     MODE STREAM command MUST NOT affect the server state in any way.  A
-     server MUST NOT require that the MODE STREAM command be issued by
-     the client before accepting the CHECK or TAKETHIS commands.
+     MODE STREAM command MUST NOT affect the server state in any way
+     (that is, it is not a mode change despite the name), therefore this
+     command MAY be pipelined.  A server MUST NOT require that the MODE
+     STREAM command be issued by the client before accepting the CHECK
+     or TAKETHIS commands.

  2.3.3. Examples

@@ -339,16 +343,16 @@
       message-id to the server.  The article is sent immediately
       following the CRLF at the end of the TAKETHIS command line.  The
       client MUST send the entire article, including headers and body, to
-     the server as a multi-line data block (see Section 3.1.1 of
-     [NNTP]).  Thus a single dot (".") on a line indicates the end of
-     the text, and lines starting with a dot in the original text have
-     that dot doubled during transmission.  The server MUST return
-     either a 239 response, indicating that the article was successfully
-     transferred, or a 439 response, indicating that the article was
-     rejected.  If the server encounters a temporary error that prevents
-     it from processing the article but does not want to reject the
-     article, it MUST reply with a 400 response to the client and close
-     the connection.
+     the server as a multi-line data block ([NNTP] section 3.1.1).  Thus
+     a single dot (".") on a line indicates the end of the text, and
+     lines starting with a dot in the original text have that dot
+     doubled during transmission.  The server MUST return either a 239
+     response, indicating that the article was successfully transferred,
+     or a 439 response, indicating that the article was rejected.  If
+     the server encounters a temporary error that prevents it from
+     processing the article but does not want to reject the article, it
+     MUST reply with a 400 response to the client and close the
+     connection.

       This function differs from the POST command in that it is intended
       for use in transferring already-posted articles between hosts.  It
@@ -602,9 +606,9 @@
       [NNTP-COMMON], by Stan Barber.

       Special acknowledgment also goes to Russ Allbery, Clive Feather,
-     and others who commented privately on intermediate revisions of
-     this document, as well as the members of the IETF NNTP Working
-     Group for continual (yet sporadic) insight in discussion.
+     Andrew Gierth, and others who commented privately on intermediate
+     revisions of this document, as well as the members of the IETF NNTP
+     Working Group for continual (yet sporadic) insight in discussion.

  10. Intellectual Property Rights

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 01:37:41 -0700

> has addressed all important issues brought up in last call?

Looks good to me.  I think we should go for it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 01:49:01 -0700

> Looks good to me.  I think we should go for it.

Oh, forgot -- the only other issue (for this and the other extension
drafts) is whether we want to go look at the uses of the "header"
terminology as mentioned in my other message.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 10:48:34 -0400

>>Looks good to me.  I think we should go for it.

What about the DoS discussions?  Have you decided that no new text is 
needed?

> terminology as mentioned in my other message.

I'll steal whatever text Clive uses for IHAVE.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 09:19:07 -0700

> What about the DoS discussions?  Have you decided that no new text is

The only thing that really might be worth covering is a warning about
peers that send CHECK but never send the actual article, not for DoS
reasons so much as because this is something that happens anyway even
between well-meaning peers.  We do have the one sentence in the CHECK
description, but it's a bit confused (it's saying the client can't rely on
the TAKETHIS being accepted and that the server can't rely on the TAKETHIS
being sent, both at the same time).

How about this -- what if we replace that last sentence of the paragraph
with something like:

    Servers handling multiple streaming feeds MAY remember that one client
    sent CHECK for an article and respond with 431 to other CHECK commands
    for the same article from other clients, making the assumption that
    the first client will send the article with TAKETHIS.  The client is
    not required to send the eventual TAKETHIS, however, and the server
    SHOULD therefore discard such information after a short timeout (a few
    seconds is generally sufficient) and accept the article from another
    client.

    The responses to CHECK are advisory.  Even if the server returns 238
    to CHECK, it may still reject TAKETHIS for the same article.

and then in security considerations, just for completeness, add:

    A malicious client with knowledge of the message-ids a server will be
    receiving could use a flood of CHECK commands to cause the server to
    defer offers of those articles from its other peers.  If client
    authentication is not sufficient to protect against this attack, the
    server SHOULD watch for clients that send excessive CHECK commands
    without a following TAKETHIS and reject further CHECK commands (with
    438 or 431) from that client.

    TAKETHIS is designed to maximize the speed of article transfer and
    therefore is the only client command followed immediately by a
    multi-line data block.  Servers concerned with possible excessive
    bandwidth utilization by clients using TAKETHIS SHOULD restrict use of
    the STREAMING extension to trusted and authenticated clients.

What do people think of this?  If we go this route, we should probably add
an informative reference to the authinfo draft and cite it under the
mentions of authentication in the security considerations above.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 13:49:37 -0400

>     seconds is generally sufficient) and accept the article from another

The above seems like we're treading into implementation details. 
Whether an implementation decides to cache CHECK message-ids or whether 
it treats connections individually (and has a race between TAKETHIS 
commands) really isn't a protocol matter.

>     438 or 431) from that client.

This problem isn't really a protocol problem, but an implementation 
problem.  I'm not saying that we shouldn't discuss it, I'm just pointing 
out that the protocol itself doesn't necessarily create this problem.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 10:51:59 -0700

>>     Servers handling multiple streaming feeds MAY remember that one
>>     client sent CHECK for an article and respond with 431 to other
>>     CHECK commands for the same article from other clients, making the
>>     assumption that the first client will send the article with
>>     TAKETHIS.  The client is not required to send the eventual
>>     TAKETHIS, however, and the server SHOULD therefore discard such
>>     information after a short timeout (a few seconds is generally
>>     sufficient) and accept the article from another client.

> The above seems like we're treading into implementation details. Whether
> an implementation decides to cache CHECK message-ids or whether it
> treats connections individually (and has a race between TAKETHIS

Er, it sure seems like a protocol issue to me....

>>     A malicious client with knowledge of the message-ids a server will
>>     be receiving could use a flood of CHECK commands to cause the
>>     server to defer offers of those articles from its other peers.  If
>>     client authentication is not sufficient to protect against this
>>     attack, the server SHOULD watch for clients that send excessive
>>     CHECK commands without a following TAKETHIS and reject further
>>     CHECK commands (with 438 or 431) from that client.

> out that the protocol itself doesn't necessarily create this problem.

Same here.  I don't understand.  Surely the use of deferrals and the
interaction between CHECK and TAKETHIS are protocol issues?

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 14:27:42 -0400

> interaction between CHECK and TAKETHIS are protocol issues?

The vulnerability that we're discussing isn't inherent in the design of 
the CHECK command.  Its only present in server implementations which 
choose to "lock" a particular message-id that it receives from a CHECK 
command.  You can certainly write a server which isn't subject to this 
attack.

As I said, I'm not going to argue strongly against adding your suggested 
text, but it seems like BCP stuff to me.  Then again, I might be full of 
shit.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 15:58:31 -0400

> mentions of authentication in the security considerations above.

As an alternative, what if we leave the existing sentence in 2.4.2 alone 
and add something more a little more generic in 5 (stealing some of 
Russ' text):

"A malicious client could use the STREAMING extension to launch a denial 
of service attack on a server.  For instance, a client could cause the 
server to indefinitely defer offers of articles from its peers by 
issuing CHECK commands with specific message-ids and never sending the 
corresponding articles, or it could use a flood of TAKETHIS commands 
with unwanted articles to consume excessive bandwidth.

To prevent such attacks, servers SHOULD be configurable to restrict the 
use of the STREAMING extension to trusted and authenticated clients 
(either via an out-of-band arrangement, or via the AUTHINFO extension 
[NNTP-AUTH]).  In the absence of such a trust relationship, the server:

   - SHOULD watch for clients that send excessive CHECK commands without 
corresponding TAKETHIS commands and reject further CHECK commands (with 
438 or 431 response) from those clients

   - SHOULD watch for clients that send excessive TAKETHIS commands with 
unwanted articles and close the connection (possibly with a 400 
response) with those clients"

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 16:07:05 -0400

> with unwanted articles to consume excessive bandwidth.

I might add this to the end of the first paragraph:

"Note that the vulnerability of a server to such attacks is dependent 
upon its implementation."

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 13:16:43 -0700

>    response) with those clients"

This would also work for me.  (400 could also be listed as a response for
dealing with excessive CHECKs.)

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 13:18:23 -0700

> command.  You can certainly write a server which isn't subject to this

Yeah, but does anyone?  I thought that approach was pretty much universal
in servers that implement streaming, since otherwise you get a ton of
duplicates, particularly of large articles.  Or you end up not getting
articles if you go the other direction and refuse articles promised but
not sent by another connection.

> text, but it seems like BCP stuff to me.  Then again, I might be full of

I can see the point about it being BCP stuff, but we aren't splitting
implementation best practices from protocol issues currently in the NNTP
documents.  It feels very borderline to me, I guess.

---

From: Ade Lovett <[email-masked]>
Date: Mon, 13 Jun 2005 13:47:37 -0700

> dealing with excessive CHECKs.)

I still absolutely do not see the need to explicitly mention this,
unless EVERY possible case for doing bad things to a server is enumerated.

For example, with a number of pre-commit history file caches, STAT can
be trivially used to stomp all over the cache, rendering
lookup-via-message-ID slow elsewhere.

Likewise, depending on how an article count is generated, flip-flopping
GROUP requests could similarly bring a server to its knees.

The point I'm making is not so much an issue with the text itself, but
the case that if we do it here, then we would need to seriously extend
the document to cover other cases where the protocol could be abused.
We also then leave ourselves open to then having to deal with those
situations that aren't explicitly mentioned.

If someone(tm) wants to go ahead and write up a formal BCP on how
servers might want to interact in the real world when faced with such
protocol abuses, that would probably be a Good Thing.  But it does not
belong in the protocol description document itself.

-aDe

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 16:50:41 -0400

> unless EVERY possible case for doing bad things to a server is enumerated.

Russ,

What's the call from the chair?  I'm ready to submit streaming-06.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 14:17:01 -0700

> What's the call from the chair?  I'm ready to submit streaming-06.

I think I'm outvoted, so let's go ahead with what we've got.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 18:34:23 -0400

> I think I'm outvoted, so let's go ahead with what we've got.

What's the vote count?  Ade was the only one dead set against the text, 
all others were in for it or on the fence, right?

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Mon, 13 Jun 2005 18:45:13 -0400

On Jun 13, 2005, at 2:27 PM, Ken Murchison wrote:

> The vulnerability that we're discussing isn't inherent in the design 
> of the CHECK command.  Its only present in server implementations 
> which choose to "lock" a particular message-id that it receives from a 
> CHECK command.  You can certainly write a server which isn't subject 
> to this attack.

I agree with Russ.  The concept of "locking" is inherent in the 
protocol; what is the purpose of the 431 response otherwise?

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 19:03:27 -0400

> protocol; what is the purpose of the 431 response otherwise?

I don't see how anyone reading our document for the first time would 
necessarily think that locking is inherent in the protocol.  In fact, if 
we said that the server SHOULD/MUST do some kind of locking, we'd 
definitely be getting too involved in implementation details and I'd object.

The 431 response could mean any number of things.  It could mean that 
the database which keeps track of message-ids, is temporarily offline, 
or it has too many outdtanding transactions, or too many concurrent 
connections, etc.  I don't see where its "only" meaning is that "we've 
already seen this message-id via IHAVE/CHECK, but haven't received the 
article yet".

That being said, as evidenced by my alternate suggested text, I'm 
convinced that saying something in the security considerations is 
worthwhile as long as we don't make it too implementation specific.

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Mon, 13 Jun 2005 19:29:02 -0400

On Jun 13, 2005, at 7:03 PM, Ken Murchison wrote:

> necessarily think that locking is inherent in the protocol.

That's potentially a flaw in the document, then, rather than the 
protocol.

> In fact, if we said that the server SHOULD/MUST do some kind of 
> locking, we'd definitely be getting too involved in implementation 
> details and I'd object.

For the record, I wouldn't suggest saying anything stronger than MAY.

> already seen this message-id via IHAVE/CHECK, but haven't received the 

Sure, these are ways that you -could- use 431, now that it exists.  But 
the one and only reason[*] that 431 was created in the first place was 
to provide for this locking, because real-world use of CHECK / TAKETHIS 
is fatally flawed without it.

[*] not that I was there at the time, but as far as I understand it

> worthwhile as long as we don't make it too implementation specific.

Maybe we need to hear from some implementors here, about what things 
are universal?

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: Ade Lovett <[email-masked]>
Date: Mon, 13 Jun 2005 16:56:37 -0700

> all others were in for it or on the fence, right?

Not quite.

I am against this text *in isolation* since it would be the only part of
the document that deals specifically with how to handle protocol abuse,
particularly as it is part of a capability that would, in current
practice, be available only to otherwise trusted clients who could do
significantly more damage much more easily.

If the *each aspect* of the protocol that could be abused to render
service impairment is (a) identified [hard] and (b) given equivalent
treatment [long-winded], then I would withdraw my objection.

I see absolutely no reason why CHECK/TAKETHIS/431 should be given such
special treatment.

-aDe

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 17:01:50 -0700

> significantly more damage much more easily.

So, the whole reason why I wrote the text this morning is that, thinking
about it, I felt like there was an issue regarding pre-commit caches that
most everyone implements, that's necessary to avoid duplicate articles in
practice, but that the document doesn't talk about.  That's not actually
just a question of abuse, although it happens to also help with one
particular abuse case.  So I think we're talking about multiple things
here.

One question is do we want to say something in the protocol description
about pre-commit caching, or do we want to leave that to a followup BCP?

The other question is whether we need to say something in security
considerations.  I don't think we want to describe pre-commit caching in
the security considerations section, since it's really not a security
issue precisely.

Ken's text captures the security bits quite well, but doesn't really talk
about the protocol bit (understandably, since he didn't feel like it was
properly part of the protocol).  I'm now somewhat confused as to who
thinks we should talk about the pre-commit cache and who thinks we should
talk about the abuse prevention issues.

Could folks weigh in on what they think about both?  Apparently it wasn't
as clear-cut as I'd thought; I thought I was the only one who was seeing
much utility in either and that other people were just going along but not
feeling strongly about it, but maybe that's not the case.

---

From: Ade Lovett <[email-masked]>
Date: Mon, 13 Jun 2005 17:17:44 -0700

> about pre-commit caching, or do we want to leave that to a followup BCP?

I don't see why.  It's entirely possible to write a CHECK/TAKETHIS
module simply using the yes/no responses, as opposed to the "431 maybe".

Indeed, for purely internal article transmission, when machines are
connected via considerably higher-speed pipes than at large, the
overhead of dealing with receiving articles twice or more may be
considerably less than the code complexity of handling the "maybe" at
both receiving and, more likely, sending ends.

Pre-commit caches are an implementation optimization around making
transit boxes deal with insane article-per-second transaction rates, and
as such are best left to a BCP.

Having the distinction between the protocol document and the BCP is
important to my mind, since the latter could change quite significantly
with no change in the former.

-aDe

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 17:29:03 -0700

> module simply using the yes/no responses, as opposed to the "431 maybe".

Yes, and for small numbers of peers, it's easier.

Okay, that's one person who is saying (please correct me if I'm wrong)
basically "no changes" -- unless we want to go back and discuss lots of
things that at this point we don't have the time to do.

Other folks?

---

From: Ken Murchison <[email-masked]>
Date: Mon, 13 Jun 2005 20:42:08 -0400

> about pre-commit caching, or do we want to leave that to a followup BCP?

I'd say no, since this is purely an implementation detail.

> the security considerations section, since it's really not a security

I'm not sure if there is precedent for discussing security w.r.t. an 
implementation detail in a protocol document, so I don't have a good 
answer for this.

Do we still want to mention TAKETHIS, since it truely is unique and I 
think its potential for abuse aren't implementation specific?

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 14 Jun 2005 05:13:30 +0100

> Maybe we need to hear from some implementors here, about what things 

Um, Ade and Andrew *are* (major) implementers.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 14 Jun 2005 05:16:46 +0100

> things that at this point we don't have the time to do.

I'm going to side with Ade. This would be a good section for an
Informational RFC ("Dummy's Guide to writing a major News Server"?), but I
don't believe we need it here.

---

From: "Jeffrey M. Vinocur" <[email-masked]>
Date: Tue, 14 Jun 2005 06:24:59 -0400

> [...] Could folks weigh in on what they think about both?

Okay, I've lost track of what the original proposed addition was.

I -do- think we should talk about the pre-commit caching a little more 
than we currently do (which is simply "if, for example, another client 
has offered to send the same article to the server").

I agree implementors are -not required- to have such a cache.  But this 
is one of those cases we've discussed previously, where we wrote the 
document with domain-specific knowledge that has not been encoded in 
the text.  I think our justification for 431 should be clear from the 
specification without any special background.  (Does anyone feel that 
this -isn't- the reason for 431?)

However, I don't feel that we need to say much more about the security 
implications of this vs other commands.  I was originally somewhat 
convinced by the argument that a broken STREAMING client can 
inadvertently deny service by offering a message-id and then crashing 
before sending it, but with Andrew's description of a more intelligent 
caching system, I'm now leaning towards considering that an 
implementation detail.

-- 
Jeffrey M. Vinocur
[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 11:21:03 GMT

>and add something more a little more generic in 5 (stealing some of 

Better, but still making a big issue out of one small problem.

>[NNTP-AUTH]).  In the absence of such a trust relationship, the server:

Again, mention of restricting STREAMING to trusted clients is helpful, but
with a minimum preamble to give it some context.

And mention of locking (with MAY) might be useful.

>response) with those clients"

But that lot is telling inplementors things that good implementors will
understand anyway.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 14 Jun 2005 10:10:04 -0400

> -----Original Message-----
> From: Ken Murchison [mailto:[email-masked]] 
> Sent: Monday, June 13, 2005 6:34 PM
> To: Russ Allbery
> Cc: [email-masked]
> Subject: Re: [NNTP] STREAMING diffs (take 2)
> 
> Russ Allbery wrote:
> 

Remember that we don't vote in the IETF.  Your chairs are responsible for
measuring consensus.

I assume Russ' comment used the "vote" word in an informal sense, not as a
description of the process being used to measure consensus.

-Scott-

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 08:50:53 -0700

Scott Hollenbeck <[email-masked]> writes:

> Remember that we don't vote in the IETF.  Your chairs are responsible
> for measuring consensus.

> I assume Russ' comment used the "vote" word in an informal sense, not as
> a description of the process being used to measure consensus.

Yes, sorry, that's entirely correct.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 11:59:50 -0700

> has offered to send the same article to the server").

You and I seem to be the only people who feel this way so far.  I'm
currently leaning still towards deciding that not enough other people
agree with me on this and letting it go, but maybe we can convince
others.  :)

> document with domain-specific knowledge that has not been encoded in the
> text.  I think our justification for 431 should be clear from the

I'm not sure I'd go so far to say that this is *the* reason for 431.  431
is a general mechanism for deferrals for whatever reason, including for
such reasons as very short-lived service outages or retryable errors in
storing an article.  I think that, in practice, this sort of feed
interaction problem is far and away the most common use of 431, but I'm
not sure that it's fundamental to the 431 specification.  In other words,
I could see inventing 431 even if we didn't have this problem (we added
deferrals for IHAVE, after all), but it wouldn't have happened as soon.

> caching system, I'm now leaning towards considering that an

Okay, it seems like we do have consensus on making no changes to the
security section, and the only question is over whether or not to add
additional documentation on the use of deferrals and a pre-commit cache.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 14 Jun 2005 16:52:05 -0400

> Remember that we don't vote in the IETF.  Your chairs are responsible for

Right.  I just was wondering how many people Russ thought were "humming" 
loudly against his proposal, because I didn't "hear" more than a couple.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 14 Jun 2005 16:58:20 -0400

> additional documentation on the use of deferrals and a pre-commit cache.

Are we certain that we don't need/want to discuss TAKETHIS in the 
security section?  I'm not pushing hard for it, I just don't want to 
skip it if we feel its justified.  Of course a malicious client could 
abuse the server in the same fashion with POST or IHAVE and just not 
wait for the initial response, so maybe TAKETHIS isn't all that unique.

I think any discussion of a pre-commit cache or "locking" is strictly an 
implementation issue and outside of the protocol.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 16:21:56 -0700

> wait for the initial response, so maybe TAKETHIS isn't all that unique.

I don't think we need to say anything.  All we'd end up saying is "only
make this service available to authenticated clients if you're worried
about abuse," which is something that could equally be said about any NNTP
command.

That is, at least, my current feeling for the group consensus.