Subject: [NNTP] Internationalisation

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 8 Apr 2005 17:29:46 +0100

First draft. Comments welcome.

   10.  Internationalisation Considerations

   10.1  Introduction and historical situation

   RFC 977 [RFC977] was written at a time when internationalisation was
   not seen as a significant issue.  As such, it was written on the
   assumption that all communication would be in ASCII and use only a
   7-bit transport layer.

   Since then, Usenet and NNTP have spread throughout the world.  In the
   absence of standards for handling the issues of language and
   character sets, countries, newsgroup hierarchies, and individuals
   have all found different solutions which work for them but are not
   necessarily appropriate elsewhere.  For example, some have adopted a
   default 8-bit character set appropriate to their needs (such as
   ISO8859-1 in Western Europe or KOI-8 in Russia), others have used
   ASCII (either US-ASCII or national variants) in headers but local 16-
   bit character sets in article bodies, and still others have gone for
   a combination of MIME and UTF-8.  With the increased use of MIME in
   email, it is becoming more common to find MIME headers identifying
   the character set of the body, but this is far from universal.

   The resulting confusion does not help interoperability.

   One point that has been generally accepted is that articles can
   contain octets with the top bit set, and NNTP is only expected to
   operate on 8-bit clean transport paths.

   10.2  This specification

   Part of the role of this present specification is to eliminate this
   confusion and promote interoperability as far as possible.  At the
   same time, it is necessary to accept the existence of the present
   situation and not gratuitously break existing implementations and
   arrangements, even if they are less than optimal.  Therefore current
   practice has been taken into consideration while in producing this
   specification.

   The NNTP itself is extended from US-ASCII [ANSI1986] to UTF-8
   [RFC3629] in this specification.  Except in the specific areas
   discussed below, UTF-8 (which is a superset of ASCII) is mandatory
   and implementations MUST NOT use any other encoding.

   The major deviation from this requirement lies in the topic of
   articles and data derived from them.  As described in Section 3.6,
   articles consist of a set of headers and then a body.  While the
   names of headers (e.g.  "From" or "Subject") are limited to US-ASCII,
   some header values (and, of course, the article body) are generated
   by users using software which adopts local practices; for example, it
   may encode all text is in ISO 8859-1 without including a MIME header
   to that effect.

   OUTSTANDING ISSUE

      Include references to MIME?  To 8859-1?  To KOI-8?

   In an ideal world it would be possible to declare such usage non-
   conforming and ignore it, but in practice any specification that
   attempted to do so would be ignored.  Therefore this version of NNTP
   allows this practice.  More specifically, while implementations
   SHOULD only allow the creation of new articles where the headers
   conform to UTF-8, where an article is obtained from an external
   source an implementation MAY pass it on, and derive data from it
   (such as the response to the HDR command), even though the article or
   the data is not valid UTF-8.  Implementations MUST transfer such
   articles and data correctly.  (Nevertheless, a client or server MAY
   elect not to post or forward the article if, after further
   examination of the article, it deems it inappropriate to do so.)

   This requirement affects the ARTICLE (Section 6.2.1), BODY
   (Section 6.2.3), HDR (Section 8.5), HEAD (Section 6.2.2), IHAVE
   (Section 6.3.2), OVER (Section 8.3), and POST (Section 6.3.1)
   commands.

   The second area of deviation is the newsgroups list returned by the
   LIST NEWSGROUPS (Section 7.6.6) command.  The actual newsgroup name
   is required to be in UTF-8 - in practice, Usenet newsgroup names are
   almost all US-ASCII - but the descriptive text is normally generated
   according to the standards of the local hierarchy and, once again,
   may not conform to UTF-8.

   The final deviation is the HELP (Section 7.2) command.  The help text
   that this returns is typically created by server operators and is not
   presented to normal users.  It does not appear profitable to put
   restrictions on this text.

   10.3  Outstanding issues

   10.3.1  Article format

   While the primary use of NNTP is for transmitting articles that
   conform to RFC 1036 [RFC1036] (Netnews articles), it is also used for
   other formats (see Appendix A).  It is therefore most appropriate
   that internationalisation issues related to article formats be
   addressed in the relevant specifications.  For Netnews articles, this
   is any successor to RFC 1036.  For email messages, it is RFC 2822
   [RFC2822].

   Of course, any article transmitted via NNTP needs to conform to this
   specification as well.

   10.3.2  Newsgroup names and descriptions

   Newsgroup names are required by this specification to be in UTF-8
   and, in practice, are almost always US-ASCII.  The spread of
   implementations that conform to this specification should suffice to
   encourage the phasing out of the few non-conforming names in use.

   Restricting newsgroup names to UTF-8 is not a complete solution to
   the issues, of course.  In particular, when new newsgroup names are
   created or a user is asked to enter a newsgroup name, some form of
   canonicalisation will need to take place.

   Newsgroup descriptions are a more difficult problem.  The pressures
   that have restricted newsgroup names to US-ASCII - essentially that
   it is more likely to remain unaltered during transmission - do not
   apply to descriptions.  More variation has therefore sprung up and
   there will be difficult problems involved in any transition to UTF-8.

   Since the primary use of NNTP is with Netnews, and since this
   information is normally distributed through specially formatted
   articles, it is recommended that these issues be addressed in any
   successor to RFC 1036.  In the meantime:

   o  servers SHOULD by default report to their administrator any use of
      character sets other than UTF-8 in the newsgroups list data (see
      Section 7.6.6);

   o  administrators of Netnews hierarchies SHOULD NOT permit the
      creation of newsgroups with names that are not US-ASCII, as any
      name that does not conform to the eventual specifications in this
      regard is likely to be a permanent source of interoperability
      issues.

   10.3.3  Other

   While the text of the HELP response remains an open issue, it is
   unclear whether there is benefit in attempting to solve it.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 11 Apr 2005 11:20:55 GMT

>First draft. Comments welcome.

I think this pudding has been somewhat over-egged :-) .

Generally, the less we say on this issue, the less likely it is to come
back to haunt us later.

>   assumption that all communication would be in ASCII and use only a

Just to forestall the Bruce Lillys of this world, a parenthetical remark
to the effect that all known current implementations are nevertheless 8bit
clean would help.

>   the character set of the body, but this is far from universal.

Again, whilst all of these deviations have been seen somewhere, they are
not as common as this text would suggest. Successors to RFC 1036, such as
Usefor or its proposed I18N extension, may yet decide to try and put the
genii back in the bottle (or they may not, or they may weasel their way
around the problem). So you need to be completely neutral regarding such
possibilities, and not say anything which could be taken as encouraging
(or discouraging for that matter) such deviations.

In particular, I doubt any such I18N extension is going to give any
comfort at all to those who would use strange charsets in bodies without
proper MIME headers.

>   operate on 8-bit clean transport paths.

Agreed.

>   practice has been taken into consideration while in producing this

I think I would rather say:

Whilst this specification has been designed to place no gratuitous
obstacles to the continued transport of articles which exceed the
spcifications in RFC 1036 in such ways, it should not be read as
condoning such practices (which is a matter properly left to future
extensions of RFC 1036).

>   may encode all text is in ISO 8859-1 without including a MIME header

... software which may have adopted other charsets and/or practices ...

>   examination of the article, it deems it inappropriate to do so.)

I think I would omit the first two sentences.

>   may not conform to UTF-8.

Again, you need to allow for the possibility that the I18N extension to
Usefor may go in for some encoding of newsgroup-names into ASCII, rather
than expressing them directly in UTF-8 (though I think you have pretty
well precluded any other ways of handling them, which will not please the
Chinese :-( ).

At the time that Usefor decided to postpone the I18N of newsgroup-names to
a future document, there were mutterings about such encodings, even
involving the ghastly punycode. I am sure that you and I agree that would
be a terrible approach to take, but we still need to take a neutral stance
on it in this NNTP standard.

>   restrictions on this text.

By which you mean that you do not care if it is in KOI-8?

>   is any successor to RFC 1036.  For email messages, it is RFC 2822

Agreed, except that there could also be extensions/successors to RFC 2822.

>   encourage the phasing out of the few non-conforming names in use.

ITYM those that conform to the successors and extensions mentioned above.

>   canonicalisation will need to take place.

I think you need to make it clear here that implementations of this new
NNTP standard MUST consider two newsgroup-names to refer to the same group
IFF they are represented by the same sequence of octets. Thus such
canonicalization as may be needed is the responsibility of the clients, or
of the the machinery (undefined by this standard) that admits new
newsgroups to the active file of the server.

>   there will be difficult problems involved in any transition to UTF-8.

Hmmmm! Might it not be possible to allow the lattitude you have permitted
for the texts or articles to apply here also? I am not sure. You certainly
need to be able to extract a valid list of newsgroup-names from a LIST
NEWSGROUPS response, even if the remainder of the descriptions looks like
gibberish, which pretty well limits you to UTF-8 (unless the
newsgroup-names are encoded in ASCII - yech!).

>      regard is likely to be a permanent source of interoperability

Do those actually add anything to what has already been said earlier?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Sat, 16 Apr 2005 17:28:36 -0700

> First draft. Comments welcome.

I think you might be trying a bit too hard here to not put any constraints
on anything.  I think we can be more succinct and a little stronger.  I'll
try to do this on a line-by-line basis, but to sum up, I think we can say:

 o The character set of article bodies SHOULD be tagged in the article
   headers via some mechanism such as [MIME].

 o Generators of article headers are strongly encouraged to use a US-ASCII
   encoding such as [RFC2047] until such time as another approach has been
   standardized; 8-bit encodings (including UTF-8) are not forbidden by
   this standard but are likely to cause interoperability problems.

 o Generators of newsgroup descriptions are strongly encouraged to use
   US-ASCII or UTF-8 until a successor to [RFC1036] standardizes a
   particular approach.  8-bit encodings other than UTF-8 are not
   forbidden by this standard but are known to cause interoperability
   problems in practice.

 o Although UTF-8 is allowed for newsgroup names by this standard in
   anticipation of where the article format standard may go, newsgroup
   names SHOULD be restricted to US-ASCII for the time being until a
   successor to [RFC1036] standardizes some other approach.  Character
   sets other than US-ASCII and UTF-8 MUST NOT be used for newsgroup
   names.

 o In all cases, implementations which receive articles from other sources
   MUST correctly handle arbitrary 8-bit data in article headers, article
   bodies, and newsgroup descriptions and MAY pass such data along via the
   NNTP protocol even if it does not follow the above recommendations.

and in fact I would replace nearly all of section 10.2 with the above
text.

>    assumption that all communication would be in ASCII and use only a

Agree with Charles that a ", although in practice all known NNTP
implementations are 8-bit clean" wouldn't be out of line here.

>    a combination of MIME and UTF-8.

I wouldn't mention UTF-8 here.  There's very little use of UTF-8 in
practice on Usenet, and what little there is is in bodies and is tagged
via MIME and therefore already covered by the MIME reference.

>    With the increased use of MIME in email, it is becoming more common
>    to find MIME headers identifying the character set of the body, but
>    this is far from universal.

>    operate on 8-bit clean transport paths.

This seems fine.

>    and implementations MUST NOT use any other encoding.

This is a bit too chatty, I think.  I would say something like:

    This standard extends NNTP from US-ASCII [ANSI1986] to UTF-8
    [RFC3629], and for most portions of the protocol, UTF-8 (which is a
    superset of ASCII) is mandatory and implementations MUST NOT use any
    other encoding.  For article headers and bodies, use of MIME is
    strongly recommended.  However, given widely divergent existing
    practices, an attempt to require all NNTP data meet a particular
    encoding and tagging standard at this time would be premature and
    unsuccessful.

    Accordingly, this protocol allows arbitrary 8-bit data in article
    headers, article bodies, and newsgroup descriptions, subject to the
    following recommendations:

and then insert my text from above.

>    may not conform to UTF-8.

Replace these two paragraphs with:

    The recommendations for article headers and bodies affect the ARTICLE
    (Section 6.2.1), BODY (Section 6.2.3), HDR (Section 8.5), HEAD
    (Section 6.2.2), IHAVE (Section 6.3.2), OVER (Section 8.3) and POST
    (Section 6.3.1) commands.  The recommendations for newsgroup
    descriptions affect the LIST NEWSGROUPS (Section 7.6.6) command.

Do we really need to allow deviation in HELP text?  I would tend to just
require it be UTF-8 and not worry about it.

>    Of course, any article transmitted via NNTP needs to conform to this

This seems fine.

>    10.3.2  Newsgroup names and descriptions

I would drop most of this section; I think we're justifying ourselves too
much around newsgroup descriptions and the text I wrote above would cover

>       character sets other than UTF-8 in the newsgroups list data (see

isn't a good idea; what is the administrator going to do with that sort of
report?  And I think the newsgroup name issue is dealt with better by my
text above.

I would instead have a 10.3.2 section entitled Canonicalization that says
something like:

    Restricting newsgroup names to UTF-8 is not a complete solution.  When
    new newsgroup names are created or a user is asked to enter a
    newsgroup name, some form of canonicalisation will need to take place.
    This specification does not attempt to define that canonicalization;
    servers are expected to match newsgroup names octet-by-octet for the
    time being.  Further work is needed in this area in conjunction with
    the article format standard.

    In the meantime, any implementation experimenting with UTF-8 newsgroup
    names is strongly cautioned that a future specification may require
    that those names be canonicalized when used with NNTP and that
    canonicalization may not be compatible with their experimental
    newsgroup names.

Thank you very much for doing a first draft!  I hope this is helpful.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 29 Apr 2005 07:46:02 +0100

> I think this pudding has been somewhat over-egged :-) .

I've noted your comments when working on the second attempt and won't
attempt to reply in detail. But thanks for them anyway.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 29 Apr 2005 07:52:54 +0100

Russ Allbery said:
> on anything.

I was trying to describe the situation as I saw it.

> I think we can be more succinct and a little stronger.  I'll
[...]

This bit moved down.

> implementations are 8-bit clean" wouldn't be out of line here.

Added.

> via MIME and therefore already covered by the MIME reference.

This paragraph is intended to cover both headers and body. MIME doesn't
imply UTF-8, and I wanted some mention of the latter in this survey of the
current situation.

> >    practice has been taken into consideration while in producing this

Are you suggesting dropping this paragraph? It expresses the principles
used in making the decisions we have made. I think something like it
belongs here, though it could go at the end of 10.1.

>     other encoding.

"most portions" is weak; I want to be explicit that the *only* deviations
from UTF-8 are listed here.

>     For article headers and bodies, use of MIME is
[...]

I hacked it around a bit but used it as a basis.

> require it be UTF-8 and not worry about it.

Well, when we first looked at this issue (a couple of years ago, I think)
we decided not to make a decision. But I have no problem with changing this
(and, given the new stronger stance on I18N, it's probably the right thing
to do). It involves some consequential changes outside 10.2 (e.g. in the
syntax).

> I would drop most of this section;
[...]

Okay.

> I would instead have a 10.3.2 section entitled Canonicalization that says
[...]

I've mostly done this, but kept the existing title and kept a note about
descriptions.

> Thank you very much for doing a first draft!  I hope this is helpful.

It is. New text will be circulated soon.