Subject: [NNTP] Future-proofing for including capabilities in responses

From: Russ Allbery <[email-masked]>
Date: Tue, 22 Mar 2005 15:57:40 -0800

We previously had a discussion about returning new capabilities after
commands such as STARTTLS and AUTHINFO, and possibly also in the server
welcome banner, but didn't reach any consensus on how to do it.

Rather than reopening that, I'm wondering how people would feel about
adding some provisions that would make it easier to propose a way to do
this at a future date.  In particular, those provisions would be:

 * Add a final response to the STARTTLS protocol so that, after TLS
   negotiation completes, the server sends a response to the client saying
   that the TLS negotiation completed.  We discussed doing this before and
   there was some support for doing it, but we didn't, as I recall, mostly
   because other protocols didn't and there didn't seem to be a point.

 * Reserve (i.e., forbid) the use of the characters [ and ] (or some other
   pair, if another pair looks better) in all freeform text in NNTP
   responses, with a warning that those characters may be used for
   protocol purposes in later standards.

This would allow people to pursue proposals for returning capabilities in
the final responses to AUTHINFO and STARTTLS, without sticking us with the
task of hammering out exactly how to do so, how to deal with modifiers,
and so forth that we weren't able to reach a consensus around.  It would
also give server authors some advance warning around using [] (or some
other pair), giving them a chance to eliminate those characters as part of
the effort to comply with this standard and making it a bit less worrisome
if those characters are given extra meaning later.

How do people feel about this?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 24 Mar 2005 16:47:32 +0000

>    because other protocols didn't and there didn't seem to be a point.

I would be very much in favour of this. As I recall, I suggested re-sending
the 200 or 201 greeting.

Would you want a separate code for a failed TLS?

As a very minor point, it eliminates the anomaly that the 382 response is,
in effect, final rather than intermediate - there's no later 2xx/4xx/5xx
response.

> and so forth that we weren't able to reach a consensus around.

I don't think we need this and I don't like the idea. Let me explain.

Firstly, we can leave the default behaviour alone. The CAPABILITIES command
is specified as taking an optional argument, so if people want unsolicited
updates then they can ask for them using such an argument (or, for that
matter, using a separate command). That's upwards-compatible in a way that
fiddling with the free text isn't.

Secondly, if information like this is put in the free text of responses, it
has to be parsed in a totally different way. We were unhappy with this in
the past (AUTHINFO SASL, I think) and I don't see that things have changed.

Off the top of my head, I can think of three ways that the information can
be provided once the feature has been turned on:

(1) Any response can be preceded by an update response code in the 6xx
block:

    [C] STARTTLS
    [S] 382 Continue with negotiation
    [TLS negotiation occurs here]
    [S] 601 New capability list follows
    [S] VERSION 2
    [S] READER POST
    [S] LIST ACTIVE NEWSGROUPS OVERVIEW.FMT
    [S] .
    [S] 201 Welcome to the secure server

(2) A flag, either before or after the three digit code, indicates that
capability information will follow:

    [C] STARTTLS
    [S] 382 Continue with negotiation
    [TLS negotiation occurs here]
    [S] 201+ Welcome to the secure server, capability information follows
    [S] VERSION 2
    [S] READER POST
    [S] LIST ACTIVE NEWSGROUPS OVERVIEW.FMT
    [S] .

(3) A flag before the three digit code of a response indicates another
response will follow:

    [C] STARTTLS
    [S] 382 Continue with negotiation
    [TLS negotiation occurs here]
    [S] +201 Welcome to the secure server
    [S] 101 Capability information follows
    [S] VERSION 2
    [S] READER POST
    [S] LIST ACTIVE NEWSGROUPS OVERVIEW.FMT
    [S] .

[In each case I'm using STARTTLS as the example, but the same could apply
to AUTHINFO and MODE READER among others.]

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 Mar 2005 12:41:04 -0500

> I would be very much in favour of this. As I recall, I suggested re-sending

Wouldn't this break INN?  It currently has a STARTTLS implementation 
based on existing practice in IMAP/POP/SMTP which also conforms to the 
current nntp-tls draft.

Is this response sent in the clear, or is it encrypted?

> Would you want a separate code for a failed TLS?

Obviously, this response would have to be sent in the clear.  It would 
seem totally asymetric to have a command return an encrypted response in 
one case, but a plaintext response in the other.  I'm sure this is one 
of the reasons why the existing TLS profiles, don't use a response code. 
  I could probably dig through some archives to see if there is a thread 
which discusses this.

> in effect, final rather than intermediate - there's no later 2xx/4xx/5xx

Actually, there is a response, just not one that conforms to NNTP.  The 
TLS exchange will finish with a success/failure.  Adding a response to 
NNTP would just be redundant (since both the client and server already 
know the result), but I won't argue this point very strongly.

> to AUTHINFO and MODE READER among others.]

The only variants that would work with AUTHINFO are (2) and (3).  Since 
AUTHINFO SASL might negotiate a security layer, the capabilities stuff 
MUST NOT be returned until AFTER the SASL exchange has completed, which 
may not be until a 283 response (server success data).

Of the two, I prefer (2) since (3) has back-to-back response codes.  (2) 
has the feel of transforming what would normally be a single line 
response into a multi-line response, which seems cleaner IMHO.

Actually, if the client enables the "unsolicited capabilities response" 
feature do we even need an indicator like "+", since the client will 
already know to expect the capabilities after STARTTLS, AUTHINFO and 
MODE READER?

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Russ Allbery <[email-masked]>
Date: Thu, 24 Mar 2005 11:00:17 -0800

>> This would allow people to pursue proposals for returning capabilities
>> in the final responses to AUTHINFO and STARTTLS, without sticking us
>> with the task of hammering out exactly how to do so, how to deal with
>> modifiers, and so forth that we weren't able to reach a consensus
>> around.

> I don't think we need this and I don't like the idea. Let me explain.

[...]

One thing that I should have mentioned is that I was aiming for enough
leeway that would allow servers to return capabilities in the initial
greeting as well as in response to commands.  That's why the reservation
for all freeform text; that catches the initial greeting as well.  (I
certainly don't expect that all commands will be allowed to return
capabilities!  That's just the minimal change to the current draft to
reserve the right leeway.)

Sending CAPABILITIES with an additional argument or allowing multiline
replies to AUTHINFO or STARTTLS doesn't solve the initial greeting
problem.

Being able to turn any arbitrary reply into a multiline reply the way that
SMTP can do would be nice, but I think that's one of those things we're
far too late to be able to add to NNTP.

> Secondly, if information like this is put in the free text of responses,
> it has to be parsed in a totally different way. We were unhappy with
> this in the past (AUTHINFO SASL, I think) and I don't see that things
> have changed.

If one of the goals is to allow capabilities to be presented during the
initial greeting, I don't see any way around this.

However, the purpose is to reserve enough leeway that people can
experiment, not hash out the best scheme on the list (since I think we've
already established that consensus is unlikely).  Towards that end, I
would be happy to reserve enough leeway for anyone's preferred schemes, if
there are other provisions you can see that would make it easier to do
something like this later.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 24 Mar 2005 11:03:51 -0800

>> I would be very much in favour of this. As I recall, I suggested
>> re-sending the 200 or 201 greeting.

> based on existing practice in IMAP/POP/SMTP which also conforms to the

Yes, this would break (well, declare non-compliant) INN's current
behavior.  Note that INN also uses the wrong reply codes and has a few
other problems at the moment.

I'm not aware of any news client that issues STARTTLS at present, rather
than using a separate port for SSL connections.  I think we're probably
still at the point where we can change this.  On the other hand, changing
who's supposed to go next is a much more significant change than changing
response codes.

> Is this response sent in the clear, or is it encrypted?

Encrypted.  (There wouldn't be any use for using it to convey capabilities
otherwise.)

> of the reasons why the existing TLS profiles, don't use a response
> code. I could probably dig through some archives to see if there is a
> thread which discusses this.

Mark seemed to feel that adding a response code at the end wouldn't be a
problem protocol-wise, so he may have an immediate feeling on how to
handle this issue.

> know the result), but I won't argue this point very strongly.

Right, it is redundant for all of our current purposes.  The only reason
to add it from our perspective is to allow it to be used as a vehicle for
capabilities information in the future, should one of the proposals for
how to do that mature.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 25 Mar 2005 12:35:39 +0000

> greeting as well as in response to commands.
[...]
> replies to AUTHINFO or STARTTLS doesn't solve the initial greeting

Is there really a need for this? Is the initial CAPABILITIES command really
a problem? If you do it this way, you've still got the parsing problem,
in that the line has to be parsed in a totally different manner to the
normal CAPABILITIES response, and you've also got the question of what to
do if the capabilities list is too long to fit into the initial greeting.
Oh, and we haven't reserved a character that can be used to replace the
CRLF between capabilities. Finally, you're dumping a significant amount of
extra data on the client when it hasn't even asked for it.

I can see lots of mess building up and I just don't see the benefits.

> far too late to be able to add to NNTP.

That's not what I'm suggesting (I agree it's far too late). I'm suggesting
a "multi-line response" extension, or - if you prefer - a "response can
include capability information" extension. But it's an *extension* that
only gets invoked when the client asks for it (and so expects it).

> initial greeting, I don't see any way around this.

Circular argument - why is that a goal?

> something like this later.

This whole thing feels like it fits firmly in the "extensions" bucket, and
a general principle of extensions is that they don't affect anyone using
the existing protocol. Redefining the meaning of the free text just feels
absolutely and completely wrong to me.

Having said all that (yet again), let me point out that if you really do
want to put data in the free text, then using a delimiter like [] is a bad
idea for at least two reasons:
- it's not unlikely that it's been used already (having semantics in
  natural languages);
- it leads to more questions, such as whether you can have multiple []
  clauses and, if so, which ones have meaning.

A better approach is to define a character sequence WHICH IS VERY UNLIKELY
TO OCCUR IN THE WILD as an "end of free text" delimiter. Alternatively,
say that the sequence only has special meaning at the *start* of the free
text, and then indicates that the "free text" isn't.

I would suggest that suitable choices for the delimiter would be:
- %x1C (the "field separator" control character);
- %xC0.9C (which is the same thing encoded in an invalid UFT-8 way; this
  technically violates the existing syntax rules, which may or may not be
  a good idea);
- a rare control code, such as %xC2.95 (Unicode U+0095 "Message Waiting")
  or %xE2.81.A3 (U+2063 "Invisible Separator");
- %xCD.B3.CE.87 (U+0373 U+0387 "erotimatiko" followed by "ano teleia",
  which I picked because these characters are unlikely to occur in that
  order, are only two octets in UTF-8, *and* are not preserved by any of
  the four Unicode normalisation forms);
- if you really feel it has to be ASCII printable text, then something
  like "~fReE!" or something equally unreadable.

But I repeat that I feel it's a bad idea, and you've not justified the
need.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 25 Mar 2005 12:46:41 +0000

Ken Murchison said:
>>>  that the TLS negotiation completed.

> Is this response sent in the clear, or is it encrypted?

Encrypted; it would go after the existing STARTTLS transactions.

> know the result), but I won't argue this point very strongly.

The purpose of providing an NNTP-like response is purely to let us hook
other NNTP things to it; I agree it's redundant.

[...]
[...]

> may not be until a 283 response (server success data).

Well, with (1) the client could accept the results anyway, overwriting what
it knew from before, and then the 283 tells it to throw away anything except
from an immediately preceding 6xx. But I agree it's messy.

> response into a multi-line response, which seems cleaner IMHO.

Whereas I prefer (3), for reasons I'll cover below.

> already know to expect the capabilities after STARTTLS, AUTHINFO and 

If we're going to create a mechanism like this, I don't want to limit it to
just one special situation. As such:
- it should be possible for STARTTLS etc. *not* to return the information;
- it should be syntactically possible to include or exclude the data in any
  response code, so that an XENCRYPT feature can use it just as well as
  STARTTLS;
- it should be possibly to use this mechanism for things other than
  resending the entire capabilities list, so it should be extensible.

To me, these all say that we're looking for a way to provide multiple
responses to a command. The exact syntax is unimportant at this stage,
but (2) requires the use of multiple flags while (3) just requires a
"another response coming" flag.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 25 Mar 2005 10:05:27 -0800

Clive D W Feather <[email-masked]> writes:

> Is there really a need for this? Is the initial CAPABILITIES command
> really a problem?

It's a problem in the sense that it's another round trip.  I don't think
that's a huge problem, but I can see why someone might want to eliminate
it.  The point wasn't to decide what to do, since I already know that
we're not going to reach an agreement on that, but to figure out what
leeway we can insert so that people can experiment in this area later.

> If you do it this way, you've still got the parsing problem, in that the
> line has to be parsed in a totally different manner to the normal
> CAPABILITIES response, and you've also got the question of what to do if
> the capabilities list is too long to fit into the initial greeting.

Right, I understand.  Those are all the reasons why we weren't going to
reach consensus in this group around doing things that way.  Some people
feel that it's still worth it, and other people don't.

> That's not what I'm suggesting (I agree it's far too late). I'm
> suggesting a "multi-line response" extension, or - if you prefer - a
> "response can include capability information" extension. But it's an
> *extension* that only gets invoked when the client asks for it (and so
> expects it).

Yes, I can definitely see the merits of this approach.

Okay, we're going down the path of arguing about the best solution, which
is exactly what I was trying to avoid.  My understanding is that you feel
there's already enough leeway for someone to add capability responses
after STARTTLS or AUTHINFO in the existing draft provided that we add that
final response to STARTTLS?

Do we have a consensus to add that final response to STARTTLS to enable
work in this area in the future?  Ken, are you uncomfortable doing that
given the existence of earlier implementations that don't return a final
response?

If we don't have consensus on reserving characters in free-form text for
future extensions in this area, we don't have consensus, and we should
move on.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 28 Mar 2005 10:03:08 +0100

> final response to STARTTLS?

Correct.

> given the existence of earlier implementations that don't return a final

Actually, thinking further, we don't need a final response to allow us to
provide capabilities update as part of an extension - whatever mechanism is
used to provide the update can be used anyway, with a dummy response code
if necessary. We *only* need the final response if you want to piggy-back
on the free text.

[It's still neater, in my opinion, to have a final response.]

> future extensions in this area, we don't have consensus, and we should

I've been thinking a bit more over the weekend. I've had several objections
to the use of free text, but one of them has been that you can't reliably
tell free text from any new usage. RFC 977 stated that NNTP was 7 bit, and
the extension to UTF-8 is only just starting with our document. So I could
live with the approach I mentioned before, that if the "free text" begins
with a specific non-ASCII escape code, it is instead some other form of
data, reserved for future expansion. Could others live with that?

[I still think this is the wrong way to do capability updates, but I can
see other potential uses for it.]

Something like this:

    The server MAY add any text after the
    response code or last argument as appropriate, and the client MUST
    NOT make decisions based on this text.  Such text MUST be separated
    from the numeric status indicator or the last argument by at least
|   one space. The text MUST NOT begin with the three octets %xE2.81.A3
+   (the UTF-8 encoding for U+2063 "Invisible Separator"); this code is
+   reserved for future versions of this specification.

or

|   one space. Future versions of this specification may allow a server
+   to provide additional data instead of this text; any such data will
+   begin with the two octets %xC0.9B (this sequence is an invalid UTF-8
+   encoding of U+001B "Escape" and so cannot appear in a response).

and syntax change:

-   simple-response =
-       simple-response-content [SP trailing-comment] CRLF
+   simple-response = simple-response-content
+       [SP (trailing-comment/additional-response-data)] CRLF
+   additional-response-data = additional-response-flag additional-response
+   additional-response-flag = %xC0.9B
+   additional-response = X-additional-response
+   X-additional-response = *U-CHAR

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 28 Mar 2005 10:24:37 GMT

>given the existence of earlier implementations that don't return a final

I am unhappy having a response sent which is encrypted if the command had
succeeded, but unencrypted if it had failed. At least the present
situation does not have that problem.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Ken Murchison <[email-masked]>
Date: Mon, 28 Mar 2005 09:31:04 -0500

> given the existence of earlier implementations that don't return a final

Yeah, I'm uncomfortable doing it for the reasons already mentioned:

- its not done this way in similar protocols
- success/failure responses would be asymmetric (encrypted/unencrypted)
- its redundant

I'd rather go with a separate response which is enabled by the client, 
like Clive suggested.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 28 Mar 2005 09:40:40 -0500

>   clauses and, if so, which ones have meaning.

Then pick something that isn't likely to appear in the wild, e.g.:

[:  :]  or [!  !] or <?  ?>

Using non-ASCII seems overkill.

I think at this point we've reached consensus that this would only be 
necessary if we want to put the capabilities in the initial greeting (we 
can use a multiline responses elsewhere), so this discussion is probably 
moot.  But I really don't see what the big deal is with putting the 
capabilities in the free text.  IMAP added it and nobody squealed.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 29 Mar 2005 09:07:17 +0100

> Using non-ASCII seems overkill.

Seeing as non-ASCII is very unlikely to appear in the wild, and
illegal-UTF-8 even less likely, why isn't that better?

We're talking about tokens that the client is going to examine; it's not
that important that they be readable.

> moot.

Indeed.

> But I really don't see what the big deal is with putting the 

Was it free text in previous versions of the protocol? I can't be bothered
to go and check, but it's not my memory of the discussion last time.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 29 Mar 2005 10:27:34 GMT

>+   reserved for future versions of this specification.

I don't like the idea of a text which, even viewed in a UTF-8 reader with
full capabilities, contains an invisible character which has semantic
meaning.

>+   encoding of U+001B "Escape" and so cannot appear in a response).

And I like even less the idea of breaking the already-extablished UTF-8
standard.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 29 Mar 2005 13:56:05 +0100

[...]
> And I like even less the idea of breaking the already-extablished UTF-8

In both cases we're looking for a delimiter for *machine* parsing - humans
would not normally be examining this text, so why does it matter?

And we're not breaking the UTF-8 standard, we're using an octet sequence
which is *not* UTF-8 to indicate, well, that what follows is not just UTF-8
text.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 29 Mar 2005 08:35:42 -0500

> to go and check, but it's not my memory of the discussion last time.

Yes and no.  Anything after an OK, NO, or BAD response is free text 
unless it appears within "[...]" in which case its a special response 
code.  AFAIK, the original spec talked about response codes, but the 
[CAPABILITY] response code was added well after the fact, thus allowing 
capabilties to appear in a previously all free text banner.

---

From: Richard Clayton <[email-masked]>
Date: Tue, 29 Mar 2005 17:22:54 +0100

In message <[email-masked]>, Clive D.W.
Feather <[email-masked]> writes

>would not normally be examining this text, so why does it matter?

because sometimes humans DO look at the text, for example when things
are broken and they're checking out what really happens with telnet (or
whatever)   that's why IETF protocols are "readable" and aren't just
full of magic byte values to mean  "ARTICLE" or "GROUP" or whatever

>which is *not* UTF-8 to indicate, well, that what follows is not just UTF-8

you're breaking an important principle of ability-to-debug ... and
that's not something to discard lightly :(

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 30 Mar 2005 11:07:46 GMT

>full of magic byte values to mean  "ARTICLE" or "GROUP" or whatever

Absolutely. I was about to right a similar response, but Richard has
beaten me to it.