Subject: [NNTP] AD guidance on NNTP i18n issues

From: Russ Allbery <[email-masked]>
Date: Mon, 28 Mar 2005 15:59:23 -0800

I have conferred with our AD about the general question of how far we need
to go in resolving the various i18n issues around NNTP, and he had the
following guidance and help in interpreting the working group charter:

We must document that the issue exists, describe current practice, and
note that more work may need to be done.  It would be appropriate to put
this into a new "Internationalization Considerations" section.  However,
complete resolution is not part of our charter, and our charter
specifically tasks us with documenting the existing situation before
taking up new work.  Therefore, if changes to NNTP are required to address
internationalization issues, it's preferrable for us to defer that work
until our core work has been completed.

USEFOR is tasked in its charter with doing some of the
internationalization work.  If it can be done there, it should be.  If
not, USEFOR also needs to document that the issue exists and a new working
group may need to be formed specifically to deal with i18n issues around
Usenet and NNTP.

My interpretation of this guidance is:

 * Clive's changes recently posted to the list are good clarifications.
   Beyond that, the current permissive language around article content can
   stay, and we don't need to add stringprep or other canonicalization for
   newsgroup arguments.

 * We need to add an Internationalization Considerations section.  That
   section should spell out:

   o The current internationalization problems from an NNTP perspective,
     namely article data (including data taken from headers), newsgroup
     names, and LIST NEWSGROUPS output.

   o The current growing use of MIME for the article format, but also the
     substantial use of local character sets in article headers and (less
     commonly) untagged article bodies.

   o The mix of character sets used in newsgroup descriptions.

   o The current pure-ASCII convention for newsgroup names, which is
     widespread but not entirely universal.

   o The need, long-term, for standardization of character sets, tagging,
     and encoding for articles, for standardization on UTF-8 for newsgroup
     descriptions, and for standardization on UTF-8 and canonicalization
     of newsgroup names.  We should also clearly state that all of those
     issues require work done in conjunction with the article format
     standard.

 * We should strongly discourage any use of newsgroup names that would
   interfere with the long-term canonicalization goals, since that's the
   area where a bad choice may be hard to reverse later.  Personally, I'm
   leaning towards saying that newsgroup names SHOULD be US-ASCII for the
   time being (read: until a standard is in place specifying how to handle
   UTF-8 newsgroup names), just to put a significant warning in front of
   people that experimenting with UTF-8 may cause them interoperability
   problems later, while keeping the existing syntax that allows for
   UTF-8 down the road.

We need someone to write this section.  Clive, if you feel comfortable
doing so, you're probably in the best position to do so right now.  If
not, or if you need any help, please let me know and I'll try to find a
volunteer.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Mark Crispin <[email-masked]>
Date: Mon, 28 Mar 2005 16:13:16 -0800 (Pacific Standard Time)

>     commonly) untagged article bodies.

Recommendations are needed.  Suggest:
  1) untagged non-ASCII character sets SHOULD NOT appear in headers and
     bodies.
  2) to make (1) feasible, USEFOR is given a task to determine how 8-bit
     characters in headers are to be tagged.
  3) clients encountering untagged non-ASCII characters SHOULD interpret
     these as UTF-8, and MAY interpret these as the client local character
     set instead.

>   o The mix of character sets used in newsgroup descriptions.

Similar recommendations: ASCII, UTF-8, or tagged charset.  Clients should 
not be obliged to guess.

>     widespread but not entirely universal.

Recommendation: any non-ASCII expansion be Unicode, either using UTF-8 or 
punycode.

>     issues require work done in conjunction with the article format

Agreed.

>   time being

Agreed.  IMAP does the same thing for mailbox names, although allows a 
form of Unicode (Modified UTF-7 -- we would have used punycode if it had 
existed at the time).

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 29 Mar 2005 11:46:18 +0100

> not, or if you need any help, please let me know and I'll try to find a

Let me see what I can do.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 29 Mar 2005 11:40:26 GMT

Agreed.

>     names, and LIST NEWSGROUPS output.

I think you want to do that mainly by punting it to Usefor. Something like

"It is anticipated that extensions or replacements to RFC 1036 will
introduce I18N features, notably in connection with newsgroup-names, the
information needed for the LIST NEWSGROUPS output etc. etc. etc. Insofar
as these extension may make use of UTF-8, this present standard has
hopefully made suitable provision. However, it is not precluded that such
developmens may require further extensions to this standard."

>     commonly) untagged article bodies.

I think you can safely point out that your standard is completely
MIME-proof, and there is no excuse at all for untagged article bodies. By
all means point out that there is some existing (but unauthorized) use of
local charsets in headers and LIST NEWSGROUPS output, and point to the
various "MAY"s that Clive mentioned in another thread, but please do so in
the context that this is to be addressed by the "extensions or
replacements to RFC 1036" that I mentioned above, without implying either
that such practices will be brought within the standards, or that they
will be totally outlawed.

>   o The mix of character sets used in newsgroup descriptions.

Yes, that is mentioned in my suggested wording, but it may need some
amplification.

>     widespread but not entirely universal.

Actually, it is _almost_ universal (even in China).

>     issues require work done in conjunction with the article format

Yes indeed. But be careful not to commit to any particular final solution.
I am sure we all believe that UTF-8 is the way to go, but we may not
succeed in carrying the rest of the world with us.

>   problems later, while keeping the existing syntax that allows for

And the initial Usefor standard will undoubtedly stick with US-ASCII for
newsgroup-names. So it would indeed be undesirable for the world to
anticipate what might be in its proposed experimental I18N extension. But,
by and large, the less that is said in this section, the less chance that
we will turn out to have shot ourselves in the foot later on.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 29 Mar 2005 11:50:47 GMT

>     characters in headers are to be tagged.

No, that is a "Do you still beat your wife?" sort of solution. It implies
that there WILL be tagging of headers, and that prevents Usefor from
saying that "Headers are in UTF-8. Period." with all other charsets
outlawed. I don't know whether we can persuade the world to accept that,
but we should not be prevented from trying.

Tagging is a very messy thing.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 29 Mar 2005 10:41:34 -0800

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> hopefully made suitable provision. However, it is not precluded that
> such developmens may require further extensions to this standard."

We should say something about future work, yes, but we need to spell out
what the problems are specifically.  The above is not adequate in my
opinion, as I understand the guidance we've received and the issue,
although it's not *far* off.  I would rather specifically talk about why
those three things in particular are an issue and then talk about what
some future article standard might do later.

> MIME-proof, and there is no excuse at all for untagged article
> bodies.

We should describe existing practice and *then* get proscriptive; I think
mixing the two just confuses matters.  I'd rather lay out what the
problems are and then lay out what the recommendations are to avoid future
problems, as well as mention that work remains to be done in standardizing
a solution.

> Actually, it is _almost_ universal (even in China).

That's what I said.  :)

> Yes indeed. But be careful not to commit to any particular final
> solution.  I am sure we all believe that UTF-8 is the way to go, but we
> may not succeed in carrying the rest of the world with us.

If we're not going to commit to UTF-8 as the solution, we should clearly
state that if the solution ends up being something other than UTF-8,
further changes to the NNTP standard may also be required.  Maybe say
something along the lines of "this standard anticipates and allows for a
future standardization on UTF-8 for newsgroup names and descriptions; if
some other solution is chosen, further changes to the NNTP standard in
this area may be required."

> anticipate what might be in its proposed experimental I18N
> extension. But, by and large, the less that is said in this section, the
> less chance that we will turn out to have shot ourselves in the foot
> later on.

A point well-taken, although I'm not worried about descriptions of current
practice so long as they're clearly labelled as such.  We can't shoot
ourselves in the foot with that; it will be factual information regardless
of what happens (it will just hopefully become dated).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 30 Mar 2005 11:16:59 GMT

>some future article standard might do later.

Sure, that text was just intended as a skeleton on which you could hang
specific technical points that need to be incluided.

>this area may be required."

Yes that is fine (it is pretty close to part of what I wrote).

Note that one of the technical points which could follow on from that is
that if UTF-8 were to be used for newsgroup-names, some form of
normalization would be essential (via *-prep or whatever) because it would
be quite infeasible to expect NNTP implementations to recognise that two
newsgroup-names were in fact alternative 'spellings' of the same
newsgroup. Essentially the same problem as comparing message identifiers,
for which we had to put some really ugly syntax into Usefor.

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Wed, 30 Mar 2005 15:45:17 +0200

Charles Lindsey schreef:

> it would  be quite infeasible to expect NNTP implementations 
> to recognise that two newsgroup-names were in fact alternative 
> 'spellings' of the same newsgroup.

Aliasses would be nice. In that way a newsgroup can have 
more than one name, so also more than one variant of the 
same name.

A central repository, like DNS, could be set up to resolve 
aliases, but a local table will often be good enough. 

The base-name could then be restricted to a limited charset.

-- 
Grtz, Ruud

---

From: Russ Allbery <[email-masked]>
Date: Wed, 30 Mar 2005 14:01:48 -0800

> more than one name, so also more than one variant of the 

This is an extension that I think would be worthwhile to standardize at
some point (including the control messages to create the aliases).  I
think most of the work for handling it is on the USEFOR side, but there is
some work on the NNTP side as well.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 31 Mar 2005 14:04:31 GMT

>more than one name, so also more than one variant of the 

Please! No aliases! Not as a feature of NNTP that would be expected to
work at all sites (and would be useless if it only worked at some sites).

It is all right for individual sites that store news to create aliases, or
to present articles from two groups as a single group for the benefit of
their local clients, but such effects should be invisible from outside
that site.

Usefor has (I hope) made it clear that, when following up to an article,
you send it to the groups in the precursor's Newsgroups header, whatever
the name of the groups you "thought" you were reading at the time.