Subject: [NNTP] New NNTP drafts approaching IETF Last Call

From: Russ Allbery <[email-masked]>
Date: Sat, 12 Mar 2005 23:46:03 -0800

Hello folks,

After the previous IETF Last Call near the end of last year, problems with
the LIST EXTENSIONS specification were discovered and we spend some time
reworking that part of the proposed new NNTP standard.  The result, which
replaces the LIST EXTENSIONS command (which was implemented incorrectly by
some servers) with a new CAPABILITIES command and attempts to ensure that
all the details of what commands are implemented can be determined by the
results of that command, is currently in working group last call and
(based on the feedback so far) will likely be put forward for IETF Last
Call by the end of this month.

Please, if you have an interest in the new NNTP standard, take a look at
the current set of documents.  The base standard is at:

    <http://www.ietf.org/internet-drafts/draft-ietf-nntpext-base-25.txt>

and the current drafts of the AUTHINFO, TLS, and STREAMING extensions are
at:

<http://www.oceana.com/ftp/drafts/draft-ietf-nntpext-authinfo-07pre1.txt>
<http://www.oceana.com/ftp/drafts/draft-ietf-nntpext-streaming-04pre1.txt>
<http://www.oceana.com/ftp/drafts/draft-ietf-nntpext-tls-nntp-05pre1.txt>

(These will be submitted as regular I-Ds before the IETF Last Call, of
course, incorporating the few changes that have surfaced during the
working group last call.)

Please send feedback to the ietf-nntp mailing list at lists.eyrie.org.
You can get to the list information page at:

    <http://lists.eyrie.org/mailman/listinfo/ietf-nntp>

and to the (recent) list archives at:

    <http://lists.eyrie.org/pipermail/ietf-nntp/>

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Mar 2005 16:43:53 -0800 (Pacific Standard Time)

I found something right away:

The text in section 3.2:

    Note that texts using an encoding (such as UTF-16 or UTF-32) that may
    contain the octets NUL, LF, or CR other than a CRLF pair cannot be
    reliably conveyed in the above format (that is, they violate the MUST
    requirement above).  However, except when stated otherwise, this
    specification does not require the content to be UTF-8 and therefore
    it MAY include octets above and below 128 mixed arbitrarily.

seems silly to me.  Nobody sends UTF-16, UTF-32, UCS-2, or UCS-4 data in 
Internet protocol commands.  Viewed one way, it's a tautology; viewed 
another, it confuses contexts.

Furthermore, the second sentence, while obviously intended to maintain 
compatibility with the past, is short-sighted and will lead to 
compatibility problems forever.

Suggest the following rewrite:

    Note: implementations prior to this specification used octets other
    than CR, NUL, and LF arbitrarily; the character set of any octets
    greater than 128 is indeterminate with old servers.  Server
    implementations which comply with this specification (and thus
    advertise VERSION 2 in CAPABILITIES) MUST send UTF-8 strings in
    responses exclusively; and client implementations MUST treat any
    response string from a server which advertises VERSION 2 as being
    in UTF-8.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 14 Mar 2005 16:56:49 -0800

>     Note that texts using an encoding (such as UTF-16 or UTF-32) that
>     may contain the octets NUL, LF, or CR other than a CRLF pair cannot
>     be reliably conveyed in the above format (that is, they violate the
>     MUST requirement above).  However, except when stated otherwise,
>     this specification does not require the content to be UTF-8 and
>     therefore it MAY include octets above and below 128 mixed
>     arbitrarily.

> another, it confuses contexts.

This section is covering multiline responses, which are used (among other
things) for conveying the actual article.  While no one uses those
character sets for *commands*, they are all valid character sets to use in
a MIME object, which is why this comes up.

> compatibility problems forever.

Actually, it's not for backward compatibility.  It's for MIME
compatibility.  We can't require any specific character set be used for
all Usenet news articles since news articles are simply MIME messages and
MIME allows the use of a variety of different character sets.  I don't
think it would be the correct choice to try to force all Usenet news
articles into UTF-8.

What we're basically trying to say here is that the data conveyed via
multiline responses must comply with the MIME restrictions on the 8bit
CTE, except that NNTP doesn't have the line length limitation of the MIME
8bit CTE.  There may very well be a better way of phrasing this.  We were
a bit reluctant to reference MIME directly since NNTP as a protocol
doesn't know anything about MIME structure and doesn't actually care; it
just has some similar restrictions due to the inability of existing
software to correctly handle embedded NUL characters.

Does that make more sense?

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Mar 2005 17:09:41 -0800 (Pacific Standard Time)

> a MIME object, which is why this comes up.

I doubt that any news articles are (or will ever be) posted in UTF-16, 
UTF-32, UCS-2, or UCS-4, but I'll let that pass.

> We can't require any specific character set be used for

OK, that was not clear.

I think that you need to do two things here:
  1) Clarify that, as of this specification, all command responses are
     UTF-8.  Death to all ISO 8859-x, Shift-JIS, etc. responses.
  2) have some wording such as what section 4.3.1 in IMAP has, e.g.
 	Article texts MAY contain 8-bit or multi-octet characters,
 	but SHOULD do so when the [CHARSET] is identified via
 	[MIME-IMB] and/or [MIME-HDRS].
     [CHARSET] being a tag for RFC 2978 (or successor)
     [MIME-IMB] being a tag for RFC 2045 (or successor)
     [MIME-HDRS] being a tag for RFC 2047 (or successor).

That leaves the deplorable practice of "just send 8-bits" for headers 
instead of using [MIME-HDRS].  We need a very strong SHOULD here that 
it should be UTF-8 and/or [MIME-HDRS] compliant for server responses, and 
a MUST for client postings (that is, a client which complies with this 
specification MUST either use UTF-8 or [MIME-HDRS]).

This is adequate weasel-wording for old software while setting a good 
future direction.

-- Mark --

---

From: Russ Allbery <[email-masked]>
Date: Mon, 14 Mar 2005 17:46:16 -0800

>> We can't require any specific character set be used for all Usenet news
>> articles since news articles are simply MIME messages and MIME allows
>> the use of a variety of different character sets.  I don't think it
>> would be the correct choice to try to force all Usenet news articles
>> into UTF-8.

>      UTF-8.  Death to all ISO 8859-x, Shift-JIS, etc. responses.

Here's the practical problem:  Some of the command responses (such as OVER
and HDR in particular) are article headers.  Usenet in practice has a
significant "just send 8-bits" culture in some hierarchies, and while I
agree with you on the merits of that culture, it's pretty hard, as a news
administrator, to disallow those messages.  It's a substantial portion of
Usenet traffic, particularly in some European and Asian hierarchies.

So, given that, what can NNTP do about it?  We can say it's not allowed,
but that isn't going to cause change in the real world; it's just going to
mean that people will ignore the specification and the whole situation
gets worse for client authors because the specification they're writing to
will be widely ignored.

Maybe we have to say something about responses that are done from article
data similar to what's said below about articles.

Certainly, I agree with the above with respect to any data that isn't
taken from news articles themselves.  I think that all that data already
has ABNF in the draft that makes it clear that it's UTF-8, so I think this
is already taken care of from that perspective.  The paragraph you were
looking at was limited to multiline responses, which already rules out
nearly all of the data not taken from articles.

> instead of using [MIME-HDRS].  We need a very strong SHOULD here that it
> should be UTF-8 and/or [MIME-HDRS] compliant for server responses, and a
> MUST for client postings (that is, a client which complies with this

We were kind of hoping to bail on questions of article format by treating
NNTP as a transport mechanism that hands back an article as an opaque wad
of data and leaving the fights over character sets to the article format.
Do you think we have to jump into this area in the NNTP specification,
when NNTP itself doesn't particularly care?  The farther we can run away
from this the better, I think -- the article format standard *has* to deal
with it, but NNTP can hopefully treat some of it as opaque.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Mar 2005 17:53:48 -0800 (Pacific Standard Time)

On Mon, 14 Mar 2005, Russ Allbery wrote:
> significant "just send 8-bits" culture in some hierarchies

>>  	[MIME-IMB] and/or [MIME-HDRS].

The "SHOULD" in this article is adequate weasel-wording; it states the 
principle while allowing leeway for legacy.

> of data and leaving the fights over character sets to the article format.

In response, I can say that IMAP wasn't allowed to bail on such questions 
(I tried...).  So the precedent is there, and it's not in your favor.

Fortunately, the "bail" option isn't lost; it just requires more wording.

-- Mark --

---

From: Russ Allbery <[email-masked]>
Date: Mon, 14 Mar 2005 17:57:44 -0800

>> Here's the practical problem:  Some of the command responses (such as
>> OVER and HDR in particular) are article headers.  Usenet in practice
>> has a significant "just send 8-bits" culture in some hierarchies

> Those are "article texts", and thus are covered in point 2:

Ah, got it.  Sorry, a bit slow today.

> In response, I can say that IMAP wasn't allowed to bail on such
> questions (I tried...).  So the precedent is there, and it's not in your
> favor.

> Fortunately, the "bail" option isn't lost; it just requires more wording.

I'll try to draft something based on the IMAP language as a proposal for a
specific change in the current wording and run it by the group and see if
we can get consensus around it.  I'll try to get that finished by
tomorrow.

Thank you!

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 Mar 2005 09:20:31 +0000

[Longer message coming, but in the meantime.]

> UTF-32, UCS-2, or UCS-4, but I'll let that pass.

Well of course not, because those character sets don't conform to the
requirements on CR/LF/NUL.

>     UTF-8.  Death to all ISO 8859-x, Shift-JIS, etc. responses.

If by "command responses" you mean the initial response line, then 3.1 is
perfectly clear:

   The character set for all NNTP commands is UTF-8 [RFC3629].

and the ABNF confirms it.

> 	[MIME-IMB] and/or [MIME-HDRS].
[...]

This is a Usefor issue, not an NNTP issue.

> We need a very strong SHOULD here that 
> it should be UTF-8 and/or [MIME-HDRS] compliant for server responses,

We're talking about material transferred over Usenet. There is no way to
enforce it. The wording was written that way for very good reason.

> and 

This "must" will be completely and utterly ignored.

> This is adequate weasel-wording for old software while setting a good 

This isn't "old software", it's the entire world. The current news article
standard isn't MIME- or UTF-8 conformant, and vast areas of the world use
ISO 8859-n BECAUSE IT'S THE RIGHT THING.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 15 Mar 2005 11:56:33 GMT

>another, it confuses contexts.

Well yes and no. If someone uses one of those charsets in an article body
then it will have to use a Content-Transfer-Encoding such as Q-P or Base64
(or even, our dead bodies not withstanding, yEnc). So then what NNTP sees
is actually just ASCII (or something else in the case of yEnc, though it
is guaranteed not to contain NUL or naked CR or LF).

However, the wording in the present draft is slightly misleading, since it
seems to suggest that the arbitrary mixture of octets might contain NUL or
naked CR or LF. So perhaps the second sentence would better say:

"However, except where given a specific interpretation by this document,
the content {of this multiline response} can be any sequence of octets
(not necessarily interpretable as UTF-8), just so long as it does not
include the octets NUL, CR of LF (other than as CRLF)."

That is as far as this document should go. What headers and bodies
actually contain is a matter for USEFOR (which does address such issues)
and its proposed I18N extension (which will have to consider them
carefully). NNTP is a pure transport protocol and what is is required to
transport is (and always has been) sequences of octets excluding NUL and
naked CR and LF, and all it actually has to do with the articles it
transports is to pull message-ids out of them and maybe the contents of
certain headers for OVER, HDR, etc.

Certainly, it does not need to be aware of MIME, because by the time it
sees the articles all the MIME in them has been encoded away out of sight,
only to be seen when it is unwrapped at the final destination.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 Mar 2005 12:18:06 +0000

Charles Lindsey said:
>>    requirement above).

> naked CR or LF. So perhaps the second sentence would better say:

It doesn't need anything saying. This paragraph is a NOTE. The normative
text is item 1 of the numbered list immediately before it.

* There is a MUST requirement about NUL, CR, and LF.
* UTF-16 violates this requirement.
* The content of multi-line responses is NOT REQUIRED to be UTF-8.

What's so hard about this?

---

From: Mark Crispin <[email-masked]>
Date: Tue, 15 Mar 2005 07:48:31 -0800 (PST)

> This is a Usefor issue, not an NNTP issue.

IMAP didn't get away with that argument, and NNTP should not be allowed to 
get away with it either.

I strongly suggest that you read (and understand) RFC 2130.

> enforce it. The wording was written that way for very good reason.

There is most definitely a way to enforce it.

Once NNTPv2 is a standards-track RFC, compliance with NNTPv2 becomes part 
of procurement RFPs.  The server may not be able to do anything about 
garbage from outside, but it very well can control generation of its own 
garbage.

> This "must" will be completely and utterly ignored.

Bullshit.  Servers are fully capable of rejecting postings from 
non-compliant clients, and very well might decide to do this with the 
express purpose of killing those clients (or at least forcing them to be 
fixed).

> standard isn't MIME- or UTF-8 conformant,

This is one of the many broken aspects of NNTP that need to be fixed. 
That's the whole point of this exercise.

> and vast areas of the world use

I am appalled that any document editor would say such a thing.  If you 
really believe this, then please resign and allow someone who is willing 
to follow IETF policy to take your place.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 15 Mar 2005 09:26:00 -0800

On Tue, 15 Mar 2005, Russ Allbery wrote:
> I'd also love to get your opinion on the new CAPABILITIES command
> before we get too bogged down in the character set discussion.  Does it
> resolve the issues that you had with LIST EXTENSIONS before?  Did we miss
> anything?

I haven't reviewed that thoroughly yet.  I promise that I will, soon.

-- Mark --

---

From: Russ Allbery <[email-masked]>
Date: Tue, 15 Mar 2005 09:19:59 -0800

> IMAP didn't get away with that argument, and NNTP should not be allowed
> to get away with it either.

> I strongly suggest that you read (and understand) RFC 2130.

I'm going to try to get some guidance on this from our AD, since I can see
both sides of this and it's really a question of how the IETF as a whole
wants to coordinate and handle character set issues when updating old
transport protocols (something that cries out for a general policy
decision, IMO).

Mark, I do understand the importance of this and I don't intend to drop
it, but I'd also love to get your opinion on the new CAPABILITIES command
before we get too bogged down in the character set discussion.  Does it
resolve the issues that you had with LIST EXTENSIONS before?  Did we miss
anything?

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 Mar 2005 17:18:40 +0000

[...]

Actually it's 3.1. Anyway, let's start with the basics. Further up it says:

Moving to the command syntax in 9.1, the generic syntax is:

    keyword *(WS token)

keyword is ASCII; token is UTF-8. The syntaxes of the individual commands
meet this requirement. For responses, any arguments after the code are
required to be ASCII, while the trailing comment is UTF-8.

So the issue only applies to the subsequent lines of multi-line commands
and responses.

> another, it confuses contexts.

Firstly, this text appears in the context of multi-line responses - it's
immediately after the bit about dot-stuffing. The actual requirement is:

   1.  The response consists of a sequence of one or more "lines", each 
       being a stream of octets ending with a CRLF pair.  Apart from
       those line endings, the stream MUST NOT include the octets NUL,
       LF, or CR.

and this is echoed in the syntax. So it's not protocol commands and I don't
see where the context is confused.

> compatibility problems forever.

It's consistent with the real world, however.

>    response string from a server which advertises VERSION 2 as being

Sorry, but this is just unacceptable to me.

Firstly, the NNTP specification is written to pass around very generic
"articles" - it could front-end many different content databases, of which
Usenet is only one. If you read 3.6 and Appendix A you'll see how we've
been careful to keep it generic like that. In particular:

   The content of a header SHOULD be in UTF-8.  However, if a server
   receives an article from elsewhere that uses octets in the range 128
   to 255 in some other manner, it MAY pass it to a client without
   modification.  Therefore clients MUST be prepared to receive such
   headers and also data derived from them (e.g.  in the responses from
   the OVER (Section 8.3) command) and MUST NOT assume that they are
   always UTF-8.  How the client will then process those headers,
   including identifying the encoding used, is outside the scope of this
   document.

and, at the start of Appendix A:

   NNTP is most often used for transferring articles that conform to RFC
   1036 [RFC1036] (such articles are called "Netnews articles" here).
   It is also sometimes used for transferring email messages that
   conform to RFC 2822 [RFC2822] (such articles are called "email
   articles" here).  In this situation, articles must conform both to
   this specification and to that other one; this appendix describes
   some relevant issues.

Note particularly "must conform both to [...] that other one". Any
requirements on character set should be related to the article semantics,
not to the transfer protocol syntax.

Secondly, there is no way that your proposal is going to work. Suppose that
I upgrade my server to talk NNTPv2. You are going to require me to convert
*EVERY* article body received to UTF-8? That completely breaks the spirit
of Usenet and probably the wording of Usefor/RFC1036.

I've just run an analysis of around 64,000 articles that arrived on our
news server today. I checked two things:
  * Do they have a Content-Type: header?
  * Do they use characters coded 128 or above and, if so, is the usage
    consistent with UTF-8?

Header     Coding      Occurrences
  Yes       ASCII         1367
  Yes       UTF-8            6     one of which was my own test message
  Yes       other         1295
  No        ASCII         8133
  No        UTF-8            0
  No        other        53426

That says there's a long way to go in the adoption of UTF-8, and the right
place to start is in Usefor, not NNTP. For the same reason that it's RFC
2822, not 2821, that deals with email content.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 16 Mar 2005 12:31:08 GMT

>text is item 1 of the numbered list immediately before it.

Sure, but even NOTEs have to be well written. Clearly, Mark DID
misunderstand it. Therefore there MIGHT be some benefit in considering
alternative wordings.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 16 Mar 2005 13:18:47 GMT

>I strongly suggest that you read (and understand) RFC 2130.

I fail to see the relevance of RFC 2130 here. We are talking about
multiline responses, which means that we are talking essentially about
article text, whether from client to server of from server to client.

Whereas headers are more or less structured, bodies are not. They may not
even represent data expressed in characters (and RFC 2130 applies only to
data expressed in characters).

But even with character data, RFC 2130 provides for a mapping from
characters to integers (CCS), a mapping from integers to sets of octets
(CES) and a mapping from octets to whatever the wire will accomodate
(TES).

Examples of TESs are Quoted-Printable and Base64 (which have the
interesting property that they map the octets back into ASCII, which is
then sent over the wire as-is). But that is not a necessary property of
TESs - they may, for example, map into a stream of octets from which NUL
and naked CR and LF are carefully excluded. yEnc is an example of such a
TES and is fairly widely used on Usenet (though we could all expound at
great length on what is wrong with it). People who "just send 8 bits" are
in fact using some CCS and CES of their choice, where the CES just happens
to avoid NUL and naked CR anf LF, and so the TES is just "straight
through". Those people are not aware that is what they are doing, and life
would be much simpler if they used some notation to indicate what CCS, CES
and TES they were in fact using (by adhering to the MIME standards, for
example). But that is nothing to do with NNTP.

NNTP is a protocol designed for wires that will accomodate streams of
octets excluding NUL and naked CR and LF. What it receives and passes on
as articles (in the form of multiline responses with dot stuffing) is the
output of the TES, which has already been applied by the clients that
submit the articles.

Thus it is a complete nonsense to speak of the "character set" within the
multiline response, because it is simply not a stream of characters at all
(well, in some special cases it might be).

Thus what are the proper CCSs, CESs and TESs to use with news articles are
a matter to be addressed by the standards that address the format of such
articles. Ditto for the standards that define any other formats that one
might choose to transport over NNTP.

NNTP is of a similar status to SMTP, as definined in RFC 2821 and its
extensions. RFC 2821 makes it clear that its wire can only accomodate
US-ASCII, and so the TES used must map into US-ASCII, and it points you to
the MIME standards as a way of ensuring that (but it does not require the
use of MIME if the body is unstructured). Likewise RFC 1652 describes
8BITMIME, which permits a less restrictive TES to be used (the same as
NNTP, as it happens).

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Mar 2005 13:34:21 -0800 (Pacific Standard Time)

The following text on page 9:

    The term "character" means a single Unicode code point and
    implementations are not required to carry out normalisation.  Thus
    U+0084 (A-dieresis) is one character while U+0041 U+0308 (A composed
    with dieresis) is two; the two need not be treated as equivalent.

is problematic and is unlikely to pass muster.

Welcome to the wonderful world of stringprep.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Mar 2005 13:38:47 -0800 (Pacific Standard Time)

The text uses such terms as "letter" and "digit" apparently presupposing 
the ASCII meanings of these terms, for example, in 3.3.1 we have "a 
capability line can only begin with a letter."

Since this protocol is specified to use UTF-8, that is no longer adequate. 
If you wish to restrict to the codepoints used by ASCII for Latin letters, 
you need to say that explicitly.  The same for digits.

-- Mark --

---

From: Russ Allbery <[email-masked]>
Date: Tue, 22 Mar 2005 14:26:52 -0800

> Welcome to the wonderful world of stringprep.

I have just now (and I'm very sorry for having sat on this for a week) put
forward a question to our AD about the whole general issue of i18n and
character sets and how we should approach that at a high level, and will
report back his guidance on that.

I definitely agree that if the opaque blob approach we were trying to take
doesn't pass muster, this is another place where we're going to need to be
more specific.  I was hoping to avoid NNTP having to be the standard that
had to specify the stringprep for newsgroup names, just because that's
going to be a zoo to nail down and to some degree it's premature because
no one's really done the hard work on i18n newsgroup names yet.  But if we
have to, we have to.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 22 Mar 2005 14:35:02 -0800

> Since this protocol is specified to use UTF-8, that is no longer
> adequate. If you wish to restrict to the codepoints used by ASCII for
> Latin letters, you need to say that explicitly.  The same for digits.

We do have a general statement in 2 that all keywords must be US-ASCII,
and the ABNF and the text say that a capability label is a keyword (and
also specifies keywords as ABNF).  I'm not sure if the additional clarity
is worh reiterating the definition of a keyword from the previous section
in the description of a capability label, but I agree that putting
"US-ASCII" before the word "letter" at the end of 3.3.1 is simple and adds
clarity.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Mar 2005 16:52:41 -0800 (Pacific Standard Time)

> It's consistent with the real world, however.

There's a difference between what you write in a "compatibility with the 
past" document and what you write in a protocol specification document.

> "articles"

You are mixing message contents with responses, and saying "we can't fix 
responses because responses are used to contain message contents."

The entire problem is that you're not separating the two.

> *EVERY* article body received to UTF-8?

Eventually, yes.

> That completely breaks the spirit

It's time to break it.  It can't happen today, or tommorrow, but maybe it 
will happen and be done in 5 years if we start now.

> news server today.

This simply proves the need to roll up sleeves and establish standards for 
the future.

The difference between your approach and mine is that you're saying "it's 
broken, so it can't be fixed ever" whereas I'm saying "it's broken, let's 
start saying that we are fixing it, and here is the fix, and eventually 
the fix will go from being a tiny minority to the mainstream."

> 2822, not 2821, that deals with email content.

You're overlooking the IMAP precedent.  IMAP has done everything that I 
advocated, and did it a decade ago.

-- Mark --

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 Mar 2005 11:54:47 GMT

>This simply proves the need to roll up sleeves and establish standards for 

No, the matter of what may occur in a Usenet article is for the article
format standards. And yes, there are issues to address there. But the NNTP
standard is not the place to specify article format, except in the
broadest terms. The function of NNTP is to transport articles reliably,
including everything defined by the current format standards, eveything
that can be foreseen as likely in future format standards and, in the
interests of being liberal, anything else that is known to be around on
the existing network and which does not actually cause harm to the NNTP
process (if it causes harm further on down the chain, then that is not
NNTP's problem).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 Mar 2005 11:45:39 GMT

>Welcome to the wonderful world of stringprep.

No, stringprep is for implementing the idea of "early normalization", as
espoused by Martin Duerst and the Unicode people. He who generates a UTF-8
string that may have to be recognized by software is responsible for
normalizing. He who transports it can then assume it is normalized and is
not obliged to confirm the normalization (at much expense) en route. If
the generator of the string screws up, then "garbage in, garbage out"
applies.

So as regards article contents, it is up to the standards for the article
format to require such normalizations as may be necessary. If NNTP itself
requires certain formats within commands, then it would be in order to
specify normalizations if such were needed, and likewise for any critical
strings in responses. However, there is nothing in the present standard
where these are needed AFAICS.

For example, most text in responses, even if not written in English, is
for human consumption only. Message-ids are in any case restricted to
ASCII.

The only potential worry is newsgroup-names. Currently, these are
restricted to ASCII, but the Usefor WG is charged with producing some form
of I18N newsgroup-names after it has completed its present tasks. The
format of these is yet to be decided. The canonical form on the wire may
be UTF-8 (in which case it will need to be normalized, presumably with a
something-prep). Or it may be some encoding into ASCII. Or whatever.

As far as NNTP is concerned, it has to recognize newsgroup-names but,
following the principle of early normalization, it is entitled to assume
that any newsgroup-name presented to it is already normalized, and so a
simple comparison of the octets presented is sufficient to compare two
names, or to determine whether a given name matches a given wildmat (that
should be made clear in the text, if it is not already so). We have
already taken care to ensure that the syntax and semantics of wildmats is
UTF-8 proof.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 23 Mar 2005 11:19:00 -0800

Charles Lindsey <[email-masked]> writes:

> espoused by Martin Duerst and the Unicode people. He who generates a
> UTF-8 string that may have to be recognized by software is responsible
> for normalizing. He who transports it can then assume it is normalized
> and is not obliged to confirm the normalization (at much expense) en
> route. If the generator of the string screws up, then "garbage in,
> garbage out" applies.

Please note that this is not, as I understand it, an accurate
representation of the IETF stance on stringprep.  If we were starting from
scratch today with NNTP as a new protocol, I don't think there would even
be any useful discussion about whether or not we would have to deal with
stringprep.  It would be a requirement.

The only reason why we may be able to avoid dealing with this is because
of the stage of the process and the desire to get something out there to
replace RFC 977 which is hopelessly obsolete.

---

From: Matthias Andree <[email-masked]>
Date: Thu, 24 Mar 2005 02:10:17 +0100

> ISO 8859-n BECAUSE IT'S THE RIGHT THING.

Fallacy.

-- 
Matthias Andree

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 24 Mar 2005 17:04:37 +0000

> misunderstand it. Therefore there MIGHT be some benefit in considering

I still think it's unneeded, but I've added "(subject to that same
requirement)" before "it MAY include octets above and below 128 mixed
arbitrarily".

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 24 Mar 2005 15:58:42 GMT

>stringprep.  It would be a requirement.

I am not convinced of that. It is certainly legitimate to use
charset=utf-8 in the body of an article (or in a part of a multipart) -
usually in conjunction with CTE 8bit. There is certainly no requirement to
normalize article bodies in any way (though it might be a requirement of
some particular application types).

So any requirement for stringprep (or any other -prep) could only arise
with texts generated by the server (e.g. in responses), and not in bodies
of articles passing in either direction.

But does the IETF actually require normalization of texts that are only
intended for human consumption? I would have thought not (though it might
be considered polite to use at least some minimal normalization). AFAIK,
there are currently no places in the document where any required content
of a respnse is in other than US-ASCII.

As I said yesterday, newsgroup-names are a matter for the format standards
to sort out (in the current Usefor they are still US-ASCII). There is also
the matter of the output of the LIST NEWSGROUPS command, but that again is
a matter for Usefor or its successors (since it is closely tied with the
checkgroups control message, which again is currently restricted to
ASCII).

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 24 Mar 2005 17:17:31 +0000

Russ Allbery said:
> doesn't pass muster,

Since I was reading it for other reasons, can I please point at RFC 2821,
section 2.3.1:

    The SMTP content is sent in the SMTP DATA protocol unit and has two
    parts:  the headers and the body.  If the content conforms to other
    contemporary standards, the headers form a collection of field/value
    pairs structured as in the message format specification [32]; the
    body, if structured, is defined according to MIME [12].

(where [32] is RFC 2822).  This is the paradigm we've been following -
we're writing the transport protocol, and Usefor is writing the message
format specification. The message format has the sub-structure of headers
plus body, but we don't restrict the body.

More specifically, our article format definition says:
    header names: ASCII printable except colon;
    header values: MUST accept 33-255 unstructured,
                   SHOULD only generate UTF-8;
    body: any octets other than NUL, CR, and LF not in CRLF pairs.

> I was hoping to avoid NNTP having to be the standard that
> no one's really done the hard work on i18n newsgroup names yet.

*I* did a load of hard work in i18n newsgroup names a few years ago. I
don't know what has survived, because I left Usefor soon after, but one
thing that was clear was that people aren't willing to do normalisation
every time an article or newsgroup name wanders past. The best model was
for there to be a canonical form for newsgroup names, for the newgroup
message to enforce that form, and for everyone else to assume it had
already been done. So LIST ACTIVE, for example, would only return canonical
names.

> But if we

Do we have another 19 years?

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 24 Mar 2005 17:26:41 +0000

> capability line can only begin with a letter."

"letter" occurs in three places in the current draft:

  Section 2:
    A "keyword" MUST consist only of US-ASCII letters, digits, and the  
    characters dot (".") and dash ("-"), and MUST begin with a letter.

  Section 3.3.1:
    Each capability line consists of:
    o  The capability label, which is a keyword indicating the
       capability.
    [...]
    Note that a capability line can only begin with a letter.

  Section 8.4.2:
    [...] the first 7 lines MUST (except for the case of letters) be
    exactly:
   
       Subject:
       From:
       Date:
       Message-ID:
       References:
       :bytes
       :lines

In the full context, I believe the text doesn't need changing.

As for digits, section 2 already says:

    "digit" refers to the octets %x30-39.

We also say elsewhere:

    This syntax extends and refines the descriptions elsewhere in this
    specification, and should be given precedence when resolving apparent
    conflicts.

and the syntax is clear on these matters.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 24 Mar 2005 09:41:39 -0800

> Since I was reading it for other reasons, can I please point at RFC 2821,

The fallacy in your argument is that NNTP does not directly correspond to 
SMTP.  NNTP is both an access and a posting protocol.  Consequently, NNTP 
must consider the access protocols (POP3 and IMAP).  As soon as you do 
that, the precedent of IMAP invalidates your argument.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Thu, 24 Mar 2005 09:50:54 -0800

> "letter" occurs in three places in the current draft:

Sorry, you can not get away with using a term without defining it.  Not 
even "US-ASCII letter" is good enough; and the other two places don't even 
go that far.

>    "digit" refers to the octets %x30-39.

This is what is needed for "letter".  However, make sure that you define 
it properly.

You've already referred to UTF-16.  Is U+0030 a "digit"?  I suspect not, 
since "digit" would not appear in context where UTF-16 would appear.  But 
you need to say that, and you need to consider that in the context of 
"letter" as well.

Specifically, it appears that a "digit" (and perhaps a "letter") is always 
an "octet", and not necessarily a "character".

Don't be deluded into thinking that this is all "obvious" and doesn't need 
to be stated.  When it comes to a correct interpretation of an RFC, the 
word of the editor of that RFC is *not* considered authoritative.  Your 
only chance to adjudicate is with explicit text in the document.

-- Mark --

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 24 Mar 2005 17:54:36 +0000

> The entire problem is that you're not separating the two.

No I'm not (mixing), not it isn't (the entire problem) and yes I am
(separating).

* A response consists of an initial line followed by subsequent lines.
* The initial line consists of a 3-digit code, zero or more arguments
  surrounded by single spaces, and a trailing comment.
  - the 3-digit code is ASCII
  - the arguments are ASCII printable
  - the spaces are ASCII
  - the comment is UTF-8.
* The subsequent lines are specific to commands. There are 16 such commands
  in the core protocol.

Of the 16 commands, these 9 already meet the requirements:

    CAPABILITIES       - UTF-8
    LIST ACTIVE        - UTF-8
    LIST ACTIVE.TIMES  - UTF-8
    LIST DISTRIB.PATS  - UTF-8
    LIST HEADERS       - ASCII
    LIST OVERVIEW.FMT  - ASCII
    LISTGROUP          - ASCII
    NEWGROUPS          - UTF-8
    NEWNEWS            - ASCII

Next, ARTICLE, BODY, and HEAD return articles or parts of them. An article
is specified - as I said in another message, to consist of headers and an
unstructured body. The header names are ASCII and the header values are:
    MUST accept any encoding
    SHOULD only generate UTF-8
The responses from HDR and OVER are derived from header contents and carry
the same requirement. This requirement matches both current practice and
the principle that this is Usefor territory, not ours.

Next there's LIST NEWGROUPS. The newsgroup names are UTF-8 but the
descriptive text is again "MUST accept any, SHOULD generate UTF-8". This is
because this text is generated by national or regional authorities and we
have no practical power to order them to do anything; we can only
encourage. While a hierarchy works in KOI-8, they're hardly likely to want
this text to be UTF-8.

Finally there is HELP. This is unstructured text designed to match
purely local needs. Currently it's like an article body - any octets
allowed except NUL, CR, and LF not in CRLF pairs. It normally gets dumped
from a file on the server or some similar approach. Any change in
requirements is going to be widely ignored.

So what am I mixing or failing to separate?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 24 Mar 2005 10:54:46 -0800

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> representation of the IETF stance on stringprep.  If we were starting
>> from scratch today with NNTP as a new protocol, I don't think there
>> would even be any useful discussion about whether or not we would have
>> to deal with stringprep.  It would be a requirement.

> usually in conjunction with CTE 8bit.

Article bodies are a different issue.  Stringprep comes in to play with
command arguments, not with data transport.  I agree that we don't need
stringprep with articles.  (What we would likely be required to do is
state that articles must comply with MIME, were we starting from scratch
right now.)

> with texts generated by the server (e.g. in responses), and not in
> bodies of articles passing in either direction.

And in arguments to commands issued by clients.

> intended for human consumption?

Newsgroup names are not only intended for human consumption.  They are,
for example, used as arguments to commands issued by clients.  An obvious
analogy is to usernames, and SASL was indeed required to specify
stringprep.

> As I said yesterday, newsgroup-names are a matter for the format
> standards to sort out (in the current Usefor they are still US-ASCII).
> There is also the matter of the output of the LIST NEWSGROUPS command,
> but that again is a matter for Usefor or its successors (since it is
> closely tied with the checkgroups control message, which again is
> currently restricted to ASCII).

Note that NNTP does not restrict either of those to ASCII, which is why
this comes up.  Nor do I think we should.  NNTP is significantly in
advance of USEFOR here at the moment, which is not a bad thing, but it
does mean that we have to think about this issue and see what we need to
do to deal with it.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 24 Mar 2005 11:15:34 -0800

>> I was hoping to avoid NNTP having to be the standard that had to
>> specify the stringprep for newsgroup names, just because that's going
>> to be a zoo to nail down and to some degree it's premature because no
>> one's really done the hard work on i18n newsgroup names yet.

> *I* did a load of hard work in i18n newsgroup names a few years ago.

Yes, that's very true, and I'm sorry, I hadn't meant to imply that you
didn't.  What I was trying to drive at was that we didn't have an RFC or
even an I-D that looked likely to become an RFC, we didn't have much in
the way of implementation experience, etc.

You did indeed lay the groundwork, and I wish that would have been more
successful.

> I don't know what has survived, because I left Usefor soon after, but
> one thing that was clear was that people aren't willing to do
> normalisation every time an article or newsgroup name wanders past.

It's not clear to me that this would be something about which we have a
choice.  In following the IETF WG chairs mailing list and some of the
other IETF policy mailing lists, i18n and character set issues appear to
be matters that have increasingly been nailed down by i18n-specific
working groups and applied IETF-wide to all protocols.  Individual
protocol working groups are getting less and less leeway in this area and
are being strongly encouraged to simply reuse the efforts of the i18n
experts, due to the wealth of complex and difficult issues involved in
handling this properly and securely.  (I am generally in favor of this
trend.)

Obviously there are optimizations that one can make internal to news
clients.  If the news client already knows that the newsgroup name is in
canonical form, it doesn't have to recanonicalize it every time it's used.

> The best model was for there to be a canonical form for newsgroup names,
> for the newgroup message to enforce that form, and for everyone else to
> assume it had already been done. So LIST ACTIVE, for example, would only
> return canonical names.

This is certainly the model that makes the most sense to me, but I'm also
painfully aware of my lack of expertise in this area and would have to
defer to you, IETF i18n experts, or anyone else who has done significant
work in this area and hammered out the various issues involved.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 24 Mar 2005 11:50:35 -0800

> the principle that this is Usefor territory, not ours.

Sorry, that doesn't wash.

First, when "current practice" is broken, it needs to be fixed.  "Just 
send 8-bits" was wrong-headed 15 years ago and it remains wrong-headed 
today.  I see your persistance in this matter as being a underhanded 
attempt to preserve "just send 8-bits" and avoid making the very difficult 
decisions necessary to establish true multilingual interoperability.

Second, it doesn't work to punt a question to Usefor when Usefor has 
nothing to say on the matter.  We are talking about responses to the HDR 
and OVER commands in NNTP, not message headers.

The only way that you can do this is to establish an explicit requirement 
for Usefor to establish a means to declare the character set of 8-bit 
characters in HDR and OVER responses in NNTP.  If that is not in Usefor's 
charter, then you must convince Usefor to accept that as a work item.

I offered you an easy way out; to use SHOULD to establish UTF-8 as a goal 
state but allow other character sets in legacy uses.

Since you've rejected that, what's left is the hard way.  Either you add 
an explicit character set tag to the HDR and OVER responses, or you 
convince Usefor to add such a tag (ala MIME quoted-words) to all 8-bit 
texts.

> This is

What are these "national or regional authorities"?  In particular, is 
there *any* national or regional government which has passed a law that 
says "newsgroups are to be in KOI-8"?

> requirements is going to be widely ignored.

I don't see how a "I don't want to fix it, so I refuse to allow it to be 
fixed" attitude is at all helpful.

Quite frankly, if NNTPv2 doesn't fix the problems in NNTP, or worse, 
establishes a principle that it's alright to disregard what you don't 
like, then I see no point to having an NNTPv2 document.

Time and time again, decisions were made in IMAP that some people did not 
like.  But since the document took a position, it was clear what complied 
with the specification and what did not.

The only reason why the text in HELP is random is because the 
specification gives no guidance as to what that text should be.  Rather 
than being "widely ignored", I suspect that most implementors try to do 
the right thing if they had some guidance as to what the "right thing" may 
be.

All that's needed is a simple statement to the effect that HELP texts must 
be in UTF-8 to comply with the NNTPv2 specification.  That doesn't "must 
be in English" or any other such thing that inflames national pride.  It 
simply indicates that, if text is not ASCII, it be something that can be 
rendered propertly by a compliant NNTPv2 client.

I may not be able to read Cyrillic; but I can recognize Cyrillic text and 
I know native speakers who can interpret it for me.  I can do this because 
the text, being UTF-8, will render properly.  If it's some random 8-bit 
string, it may take a considerable amount of time to determine the correct 
character set.

-- Mark --

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 25 Mar 2005 12:22:42 GMT

>that, the precedent of IMAP invalidates your argument.

POP3 is a protocol for managing a store of email messages and delivering
them to clients. It is therefore irrelevant to Netnews.

IMAP is a protocol for managing a store of email messages and/or news
articles and delivering them to clients.

It is not usual to access an IMAP store of news articles using NNTP,
though there is no reason why some NNTP server should not offer that
service (and internally it might or might not use the IMAP access
facilities in order to do so).

But I don't see how this affects our NNTP draft in any way, since the
structure of what is contained in a current IMAP store is a subset of what
can be delivered by an NNTP server conforming to our draft (assuming
conversions from UTF-7 to UTF-8 in the proper places).

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 25 Mar 2005 12:44:19 GMT

>handling this properly and securely.  (I am generally in favor of this

My understanding is that i18n-specific working groups which need to
specify normalization restrictions on UTF-8 SHOULD do so by writing
documents in the style of stringprep, and moreover they SHOULD make use of
one of the already-defined *-prep documents unless there is a good eason
to invent yet-another-one.

But there is no requirement that every UTF-8 text permitted by every
standard must be subject to such normalization, like it or not; it depends
on the use to which that text is to be put (e.g. in newsgroup-names,
domain-names, etc.).

So I think the most that is required in our draft (except for those cases
arising in responses and commands, which Clive has alraeady analyzed for
us in some detail) is to point out that is it likely that the format
standards for the articles to be transported will have limited some of the
UTF-8 that that may be encountered by means of assorted *-prep filters.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 25 Mar 2005 16:27:21 GMT

>and OVER commands in NNTP, not message headers.

But Usefor has a lot to say on the matter. And the HDR and OVER commands
in NNTP are simply required to take whatever is in the headers of the
articles and to deliver it as-is (subject to some specified mangling in
the case of OVER). So yes, "as-is" had better be in UTF-8, but the draft
is written so that anything with at least ASCII as a subset is acceptable,
and will do no harm at least within the NNTP protocol.

>charter, then you must convince Usefor to accept that as a work item.

It is no part of the function of this WG to instruct the Usefor WG as to
what it should do.

What the Usefor WG has done, in its current drafts, to to limit headers to
be in US-ASCII (which may include things encoded according to RFC 2047, of
course). It also imposes some requirements on the transport mechanisms
(notably that they be 8-bit clean), so as to preserve the greatest scope
for future extensions (and the current NNTP draft is entirely consistent
with those requirements, as indeed are all known implementations of news
transport mechanisms).

The Usefor WG is also charged with introducing I18N as an Experimental
Protocol, but only after it has finished work on the base standards. So
I18N is off topic on the Usefor WG for now, but it will come, and it will
have to take account of the sort of issues you have been raising.

>state but allow other character sets in legacy uses.

It already is established as a goal state, but it is in the nature of
transports that they should be liberal.

>says "newsgroups are to be in KOI-8"?

Such national or regional authorities undoubtedly exist. They are
established by the consensus of the administrators of servers worldwide
who choose to acknowledge their authority (because they prefer a quiet
life, and so it in in their interest to do so). But the process of
establishing them is a political and social matter not governable by any
standard. It is an adhocratic anarchy, and it works remarkably well.

Usefor acknowledges that the exist, and provides tools for them to use and
conventions for them to follow, but it very carefully does not say how
they come to be established. Maybe some informational document will
explain it all some day.

>fixed" attitude is at all helpful.

There is nothing to fix. Like everywhere else in the document, the default
charset is UTF-8, but the sky will not fall in if some implementation
chooses to use something else. However, I tend to agree that there should
be a statement somewhere that these informational texts (the initial
greeting is another example) SHOULD be interpreted as UTF-8 texts.

---

From: [email-masked] (Kai Henningsen)
Date: 26 Mar 2005 15:59:00 +0200

> says "newsgroups are to be in KOI-8"?

Not governments. Usenet hierarchy administrators. From the point of view  
of the IETF, they count as "users".

For an incomplete list of administrators, see, for example,
        ftp://ftp.isc.org/pub/usenet/CONFIG/control.ctl
and for a (partial) current list of groups and descriptions,
        ftp://ftp.isc.org/pub/usenet/CONFIG/newsgroups
- you can see that various regions do use various charsets. To point out  
just some examples, is.*, medlux.*, sunet.* (national variant of ISO  
646?), tw.*, uk.* ...

MfG Kai

---

From: Mark Crispin <[email-masked]>
Date: Sat, 26 Mar 2005 09:30:54 -0800 (PST)

> of the IETF, they count as "users".

Exactly

While the decisions of these individuals may have been reasonable at the 
time, times change.  It is therefore inaccurate to present these decisions 
are cast-in-stone "national or regional authority" decisions that are not 
subject to influence of newer standards.

More to the point, many of these individuals would very likely welcome 
guidance from specifications, instead of being compelled to guess at 
what's right and hope for the best.  Of course, it's not going to work to 
say "on April 23, 2005, everything changes to UTF-8."  But having a 
standards track document that points strongly towards UTF-8 will lead 
presently to such migrations.

You're not doing people any favors by saying "we're not going to give you 
any guidances or any bedrock standards points that you can use to bolster 
your arguments.  You're free!"

-- Mark --