Subject: Re: [NNTP] Internationalisation, attempt 2

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 2 May 2005 09:09:49 +0100

maurizio codogno said:
>>   o  Header values SHOULD use US-ASCII or an encoding based on it such
>>      as RFC 2047 [RFC2047] until such time as another approach has been
>>      standardised. 8-bit encodings (including UTF-8) MAY be used but
>>      are likely to cause interoperability problems.
> 
> I would move the last sentence to a separate paragraph. I don't like
> to read in the specifics that an implementation is likely to cause
> interoperability problems... At least, say "Legacy 8-bit encodings...".

I've made it "At present, 8-bit ...".

> As for the first sentence, is there a simpler way to assess that in
> future other encodings may be standardises? I am confident "Until such
> time as" is good English, but I find it difficult to understand it.

Sorry, but I can't think of another way of phrasing it succinctly.

> >   o  Where an article is obtained from an external source an
> >      implementation MAY pass it on, and derive data from it (such as
> >      the response to the HDR command), even though the article or the
> >      data does not meet the above requirements.  Implementations MUST
> >      transfer such articles and data correctly.  (Nevertheless, a
> >      client or server MAY elect not to post or forward the article if,
> >      after further examination of the article, it deems it
> >      inappropriate to do so.)
> 
> Could you state that if it decides to pass it it must do it verbatim?
> Otherwise some implementation could choose to "rectify" the non-standard
> header value, and this would be a Very Bad Thing.

Hmm. Do we want to forbid recoding of articles as they arrive? Obviously
the HDR data must correspond to the article as it would be delivered.

Opinions needed before I touch this wording.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 2 May 2005 09:54:29 GMT

In <[email-masked]> "Clive D.W. Feather" <[email-masked]> writes:

>Okay, here's the second attempt.

>   Restricting newsgroup names to UTF-8 is not a complete solution.  In
>   particular, when new newsgroup names are created or a user is asked
>   to enter a newsgroup name, some form of canonicalisation will need to
>   take place.  This specification does not attempt to define that
>   canonicalization; servers are expected to match newsgroup names
>   octet-by-octet for the time being.  Further work is needed in this
>   area in conjunction with the article format specifications.

Not so sure about that "for the time being".

Surely the intent is that matching octet-by-octet should apply
_permanently_ (in any case, our draft proposes no other way of doing it).
Normalization is for clients to get right (or at most for the injecting
agent hiding behind the POST command to check and/or fix). If some
unnormalized newsgroup-name makes it through, then it will just fail to
match. Tough!

Not the similarity with the way we deal with message-ids. The comparison
is octet by octet, in spite of implications in RFC 2822 that
<[email-masked]> is equivalent to <"[email-masked]> (and which
USEFOR is dealing with.

BTW, if the draft still includes those examples on non-equivalent
message-ids, the latest version of them (following discovery of yet
another case needing to be covered) is:

|  <[email-masked]>
|  <"ab.cd"@example.com>
|  <"ab.\cd"@example.com>

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 May 2005 09:09:33 -0700

Clive D W Feather <[email-masked]> writes:

> Hmm. Do we want to forbid recoding of articles as they arrive?

Yes.  Absolutely.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 3 May 2005 09:20:16 +0100

>> Hmm. Do we want to forbid recoding of articles as they arrive?

Okay. New wording:

    o Where an article is obtained from an external source an
      implementation MAY pass it on, and derive data from it (such as
      the response to the HDR command), even though the article or the
      data does not meet the above requirements.  Implementations MUST
      transfer such articles and data correctly and unchanged; they MUST
      NOT attempt to convert or re-encode the article or derived data.
      (Nevertheless, a client or server MAY elect not to post or forward
      the article if, after further examination of the article, it deems
      it inappropriate to do so.)

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 3 May 2005 09:34:53 +0100

> unnormalized newsgroup-name makes it through, then it will just fail to

I can sympathise, but I don't want to be too ready to make that assumption
for all uses (in particular, POST could well be an exception). Here's what
I've now got:

    Restricting newsgroup names to UTF-8 is not a complete solution.
    In particular, when new newsgroup names are created or a user is
    asked to enter a newsgroup name, some scheme of canonicalisation
    will need to take place.  This specification does not attempt to
    define that canonicalization; further work is needed in this area
    in conjunction with the article format specifications.  Until such
    specifications are published, implementations SHOULD match newsgroup
    names octet-by-octet.  It is anticipated that any approved scheme
    will be applied "at the edges" and therefore octet-by-octet comparison
    will continue to apply to most, if not all, uses of newsgroup names
    in NNTP.

> another case needing to be covered) is:

Um, how does this differ from the present example of:

    <[email-masked]>
    <"abcd"@example.com>
    <"ab\cd"@example.com>

?

---

From: "maurizio codogno" <[email-masked]>
Date: Tue, 3 May 2005 09:36:41 +0000

>      it inappropriate to do so.)

This is fine for me.
ciao, .mau.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 3 May 2005 09:42:55 GMT

>Opinions needed before I touch this wording.

Usefor/pro makes it quite clear that relaying agents either pass on
articles exactly as received, or else drop them on the floor. There are
too many agents around that 'think' they can 'improve' articles (and they
are more often than not wrong in that belief). If you drop it on the
floor, then it becomes somebody else's problem. If somebody else is
prepared to relay it, then it will flood around you and the rest of Usenet
will still see it (and form their own opinions about it).

Injecting agents OTOH might possibly fix such problems, and gateways might
be forced to do so in order to be compliant on the target medium.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 03 May 2005 08:38:34 -0700

>       it inappropriate to do so.)

Looks good to me.

---

From: Matthias Andree <[email-masked]>
Date: Tue, 03 May 2005 18:31:22 +0200

"Clive D.W. Feather" <[email-masked]> writes:

> 10.  Internationalisation Considerations
>
> 10.1  Introduction and historical situation
>
>    RFC 977 [RFC977] was written at a time when internationalisation was
>    not seen as a significant issue.

That's a pretty friendly wording of US American i18n ignorance. Nevermind.

>    As such, it was written on the
>    assumption that all communication would be in ASCII and use only a
>    7-bit transport layer, although in practice all known implementations
>    are 8-bit clean.

Suggest to replace "all known implementations" by "most known
implementations" -- unless you can produce proof you've known all at the
time of writing.

>    Since then, Usenet and NNTP have spread throughout the world.  In the
>    absence of standards for handling the issues of language and
>    character sets, countries, newsgroup hierarchies, and individuals
>    have found a variety of solutions that work for them but are not
>    necessarily appropriate elsewhere.  For example, some have adopted a
>    default 8-bit character set appropriate to their needs (such as
>    ISO8859-1 in Western Europe or KOI-8 in Russia), others have used

This should be spaced "ISO 8859-1" or even better use the official IANA
spelling "ISO-8859-1". I haven't checked on KOI-8.

>    ASCII (either US-ASCII or national variants) in headers but local 16-
>    bit character sets in article bodies, and still others have gone for
>    a combination of MIME [RFC2045] and UTF-8.  With the increased use of
>    MIME in email, it is becoming more common to find NNTP articles
>    containing MIME headers identifying the character set of the body,
>    but this is far from universal.
>
>    The resulting confusion does not help interoperability.
>
>    One point that has been generally accepted is that articles can
>    contain octets with the top bit set, and NNTP is only expected to
>    operate on 8-bit clean transport paths.
>
> 10.2  This specification
>
>    Part of the role of this present specification is to eliminate this
>    confusion and promote interoperability as far as possible.  At the
>    same time, it is necessary to accept the existence of the present
>    situation and not gratuitously break existing implementations and
>    arrangements, even if they are less than optimal.  Therefore the
>    current practice described above has been taken into consideration in
>    producing this specification.
>
>    This specification extends NNTP from US-ASCII [ANSI1986] to UTF-8
>    [RFC3629].  Except in the two areas discussed below, UTF-8 (which is
>    a superset of ASCII)

This should probably be US-ASCII not just ASCII.

> is mandatory and implementations MUST NOT use
>    any other encoding.
>
>    Firstly, the use of MIME for article headers and bodies is strongly
>    recommended.  However, given widely divergent existing practices, an
>    attempt to require a particular encoding and tagging standard would
>    be premature at this time.  Accordingly, this specification allows
>    the use of arbitrary 8-bit data in articles subject to the following
>    requirements and recommendations.
>
>    o  The names of headers (e.g.  "From" or "Subject") MUST be in US-
>       ASCII.
>

I don't think this is strong enough. 8-bit encoded Subjects are almost
guaranteed to cause trouble, I'd think the paragraph should only allow
UTF-8 and RFC-2047. Suggestion:

" Header values SHOULD use US-ASCII or an encoding based on it such
  as RFC 2047 [RFC2047] until such time as another approach has been
  standardised. 8-bit encodings other than UTF-8 SHOULD NOT be used,
  because they are likely to cause interoperability problems."

>    o  The character set of article bodies SHOULD be indicated in the
>       article headers, and this SHOULD be done in accordance with
>    MIME.

Please consider "MUST" for the second "SHOULD". I see no sane solution
other than MIME at this time, and MIME is the widest spread.

>    Secondly, the following requirements are placed on the newsgroups
>    list returned by the LIST NEWSGROUPS (Section 7.6.6) command:
>
>    o  Although this specification allows UTF-8 for newsgroup names, they
>       SHOULD be restricted to US-ASCII until a successor to RFC 1036
>       [RFC1036] standardises another approach. 8-bit encodings MAY be
>       used but are likely to cause interoperability problems.
>
>    o  The newsgroup description SHOULD be in US-ASCII or UTF-8 unless
>       and until a successor to RFC 1036 standardised other encoding
>       arrangements. 8-bit encodings other than UTF-8 MAY be used but are
>       likely to cause interoperability problems.

Again, these paragraphs ought to read "8-bit encodings other than UTF-8
SHOULD NOT be used, because they are ...(interoperability)..."

-- 
Matthias Andree

---

From: Matthias Andree <[email-masked]>
Date: Tue, 03 May 2005 18:33:31 +0200

> the HDR data must correspond to the article as it would be delivered.

Recoding in transit should be expressly forbidden, i. e. "MUST NOT",
although the initial normalization that makes the article ready for
injection (cf. the roles such as injecting agent and similar) OTOH
should expressly be required to either normalize to standards or reject
the article; this would remove the need of any later encoding.

-- 
Matthias Andree

---

From: "Andrew - Supernews" <[email-masked]>
Date: Tue, 03 May 2005 17:44:28 +0100

>>>>> "Matthias" == Matthias Andree <[email-masked]> writes:

 Matthias> I don't think this is strong enough. 8-bit encoded Subjects
 Matthias> are almost guaranteed to cause trouble,

Haven't we been through this before? 8-bit encoded Subject headers are
extremely common and have been for many years... they outnumber
2047-encoded subject headers by a factor of about 20 at the moment.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 4 May 2005 09:07:23 +0100

> That's a pretty friendly wording of US American i18n ignorance. Nevermind.

Not ignorance, disinterest.

> implementations" -- unless you can produce proof you've known all at the

All the ones known to me are.

>>    (such as

ISO themselves call it "ISO/IEC 8859-1", so that's what I'll use.

The terms appear to be "KOI-8" but "KOI8-R" (or other suffices).

> This should probably be US-ASCII not just ASCII.

True.

> UTF-8 and RFC-2047.

However, other encodings happen in real life and UTF-8 is as likely to
cause problems as any other.

> Suggestion:

The SHOULD NOT is probably better wording, but I'm not willing to call out
UTF-8 as special in this situation.

> other than MIME at this time, and MIME is the widest spread.

But it's not universal. We don't know enough about usage in (say) Japan to
say whether that MUST will be another thing for users to ignore.

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Wed, 4 May 2005 11:00:45 +0200

Clive D.W. Feather:

> [ISO8859-1,ISO 8859-1,ISO-8859-1]

See http://en.wikipedia.org/wiki/ISO_8859-1
and especially the phrase:

"The IANA has approved ISO-8859-1 (note the extra hyphen), a superset of
ISO/IEC 8859-1, for use on the Internet."

In "ISO/IEC 8859-1", the characters 00-1F and 7F-9F are unused.

-- 
Grtz, Ruud

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 May 2005 10:24:24 GMT

>    will continue to apply to most, if not all, uses of newsgroup names

Looks fine to me.

>    <"ab\cd"@example.com>

By the inclusion of that '.' in "ab.cd".

Usefor has been gradually restricting the syntax of message-ids so that
they can safely be compared as octet streams. Periodically, someone comes
up with an ingenious new problem :-( .

The latest was when it was pointed out that an otherwise unexceptionable
dot-atom-text should never be placed inside a quoted-string, and after the
syntax had been fixed for that, the standard example was changed to
illustrate the point (well, it hasn't made it into the latest draft yet,
but that is what was suggested recently). I was just suggesting that the
NNTP draft might wish to make the same change.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 May 2005 10:35:53 GMT

>implementations" -- unless you can produce proof you've known all at the

No, the present wording is correct. "all known implementations" means all
implementations known to the collective wisdom of this WG, not all
implementations that might exist. AFAIK, nobody here is aware of _any_
implementation without that property.

>  because they are likely to cause interoperability problems."

But they don't cause interoperability within NNTP. Whilst it is highly
desirable that non-standard usages within headers should cease, the NNTP
standard is not the place to fix it (we have already had that argument
with Mark Crispin :-( ). NNTP should just pass through non-standard stuff
that does not affect its own operations. Fixing it is someone else's
problem (most likely Usefor's).

>other than MIME at this time, and MIME is the widest spread.

Same again. It is not NNTP's problem. Usefor can and does say what you
want.

>SHOULD NOT be used, because they are ...(interoperability)..."

Ditto.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 4 May 2005 14:56:52 +0100

> NNTP draft might wish to make the same change.

I'm not sure what was wrong with the existing example, but I've made the
change anyway.

---

From: [email-masked] (Peter Robinson)
Date: Wed, 4 May 2005 22:53:40 +0100

Clive D.W. Feather <[email-masked]> wrote:

> >>      standardised.

> Sorry, but I can't think of another way of phrasing it succinctly.

Surely just s/such time as //
Or am I missing the point here?

Peter

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 5 May 2005 09:22:22 +0100

> Or am I missing the point here?

I feel "until such time as" emphasises it more.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 6 May 2005 11:24:25 GMT

>I feel "until such time as" emphasises it more.

I agree.

---

From: [email-masked] (Kai Henningsen)
Date: 06 May 2005 10:09:00 +0200

> I'm not sure what was wrong with the existing example, but I've made the

Is there something about <[email-masked]>? If not, there should  
probably be. (That is, while domain names are case insensitive, message  
ids are compared bytewise.)

MfG Kai

---

From: [email-masked] (Kai Henningsen)
Date: 06 May 2005 11:02:00 +0200

> implementation without that property.

Make that "any implementation in current use in the global Usenet",  
otherwise it is false.

This came up a few years ago (can't find it in Google[1], though), and  
ISTR that only one or two instances of not-8-bit-clean implementations  
were found to still be in use at that time. I can't name the software in  
question, but again I seem to recall it was definitely known to the old  
guard at that time. Maybe an implementation for some more exotic machine  
like Vax/Vms? IIRC there was doubt that those instances could last much  
longer ...

[1] Though I did find that early versions of B and C News needed patching  
for Japan because they stripped ESC (which kills ISO 2022).

MfG Kai

---

From: Russ Allbery <[email-masked]>
Date: Fri, 06 May 2005 14:28:34 -0700

> ids are compared bytewise.)

You actually can't rely on that; INN compares the RHS case-insensitively
(and the LHS case-insensitively as well iff it case-insensitively matches
"postmaster", due to a hilarious misreading of the standard that I haven't
fixed for backward compat issues).

---

From: [email-masked] (Peter Robinson)
Date: Sat, 7 May 2005 13:11:36 +0100

:   o  Header values SHOULD use US-ASCII or an encoding based on it such
:      as RFC 2047 [RFC2047] until such time as another approach has
:      been standardised.

[context (otherwise my response makes even less sense than usual):]

maurizio codogno said:
| I am confident "Until such time as" is good English, but I find it
| difficult to understand it.

Charles Lindsey <[email-masked]> wrote:

> In <[email-masked]>

> >I feel "until such time as" emphasises it more.

OK.

Peter

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 9 May 2005 18:52:56 GMT

>fixed for backward compat issues).

Then I think INN is in breach of our new draft (on both counts), and a
suitably ingenious malefactor could cause nasty things to happen on Usenet
(like different sites seeing different forms of supposedly identical
articles). Practically, it is most unlikely to arise, unless some stupid
relayer tries to "fix" a message-id by changing its case.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 9 May 2005 18:48:22 GMT

>ids are compared bytewise.)

My reading of RFC 2822 is that an <id-right> (which might well look like a
<domain>, but actually is just the sequences of characters you can get by
expanding dot-atom-text or no-fold-literal) is not case-insensitive.

Bruce Lilly disagrees, and I have had endless arguments with him about
this (I am right, of course :-) ). Note that in RFC 822 the RHS of a
Message-ID was indeed case-insensitive, and my claim was that RFC 2822 had
fixed that unfortunate state of affairs.

Sadly, it had not fixed the problems with quoted-strings and quoted-pairs.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 10 May 2005 07:15:07 +0100

> like Vax/Vms? IIRC there was doubt that those instances could last much  

Okay, I give up. I'll change it.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 10 May 2005 07:37:37 +0100

> You actually can't rely on that; INN compares the RHS case-insensitively

That's not compatible with our current text.

Are you proposing a change, or are you accepting that INN is wrong?

---

From: Russ Allbery <[email-masked]>
Date: Mon, 09 May 2005 23:55:38 -0700

> Are you proposing a change, or are you accepting that INN is wrong?

It's more a curiosity; INN is wrong.  Although it's very unlikely that INN
is going to change in the near future, since changing this requires a
complete history rebuild.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 18 May 2005 19:05:22 -0700

> Okay, here's the second attempt.

I have reviewed the pre-2 text, which I realize may be somewhat updated
from the text in the message that I'm responding to.  It looks good to
me.  The only comment that I have is that in several places the text says:

    <some phrasing about other encodings> MAY be used but are likely to
    cause interoperability problems.

following a SHOULD saying something different.  This is, as I understand
it, not the intended use of SHOULD and MAY; in particular, a MAY I don't
believe should say you can do something that a SHOULD says you shouldn't.

This problem may well have been in my proposed text.

I believe that all wording of that sort should be changed to either:

    ...are likely to cause interoperability problems.

(in other words, just dropping the "MAY be used" entirely and leaving the
SHOULD with its implicit escape clause) or:

    ...SHOULD NOT be used, as they are likely to cause interoperability
    problems.

which just reiterates the previous SHOULD.  I have a minor preference for
the first.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 18 May 2005 19:09:05 -0700

> guaranteed to cause trouble, I'd think the paragraph should only allow

I think the current text is as strong as we can get away with.  8-bit
encoded Subjects are in widespread use on Usenet currently, so clearly
someone is using them with sufficient interoperability to continue doing
so.  I agree that this should eventually stop, but RFC 2047 is
sufficiently unappealing that even though it's the right thing to do, I'm
quite sure that mandating it in our standard will just mean that the
standard says something other than what people do in practice, which is
what we're trying to avoid.

> other than MIME at this time, and MIME is the widest spread.

I think the second SHOULD is only reflecting the first, since otherwise it
implies that the whole thing is a SHOULD.  If Clive feels like trying to
reword this to indicate that *if* the character set is indicated, it MUST
be done via MIME, I would have no objections, but I'm not sure the
clarification is worth the trouble.

> SHOULD NOT be used, because they are ...(interoperability)..."

See my other message on that.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 19 May 2005 09:18:40 +0100

Russ Allbery said:
> The only comment that I have is that in several places the text says:
> 
[...]

---

From: Russ Allbery <[email-masked]>
Date: Thu, 19 May 2005 19:43:46 -0700

>>     ...SHOULD NOT be used, as they are likely to cause interoperability

Looks great.  Thanks!