Subject: [I18ndir] Fwd: Re: Working Group Last Call: Structured Headers for HTTP

From: Asmus Freytag <[email-masked]>
Date: Wed, 5 Feb 2020 14:50:58 -0800

(didn't go out to the list when I first sent it)

When I read the word "exception" I always think of this:

When we built the first Unicode-enabled OS (Windows NT), we had a long 
discussion of which "strings" in the OS needed to be Unicode.

Some thought that there was a clear dividing line between data and what 
would be called "protocol values" in another context.

Some of the latter did look like they were easily limited to ASCII; but 
everywhere we found "exceptions". There might be a set of enumerable 
tokens, but it allowed extended values that were network or file 
identifiers.

After exhaustively researching everything, the conclusion was that every 
single string in the OS had to be Unicode (and making any exceptions was 
either not possible, or not worth the effort).

However, while all strings were encoded in Unicode, not all string 
values were allowed. While file names could be localized (within the 
limits of file system syntax), some of the enumerated strings were left 
limited to the ASCII set in repertoire (even if encoded in Unicode).

Reading this discussion (and I'm sorry I don't have the time right now 
to properly delve into the details) it seems that a natural 
recommendation would be to require Unicode for any native representation 
and, if necessary (or possible), limit the repertoire.

This also requires a definition of the matching protocol for all strings 
that are to be matched as part of the protocol (or should be 
searchable). For any format, that would cover issues of casing, white 
space handling etc., but for Unicode, by necessity, that also requires 
defining the normalization form to be used.

A./

PS: given how few systems these days natively operate in any character 
set other than Unicode, I am always astonished at the length to which 
people go to justify not making something native Unicode. They just pick 
up conversion issues when they use platform libraries to do any work or 
display.

---

From: John C Klensin <[email-masked]>
Date: Wed, 05 Feb 2020 20:23:02 -0500

Asmus,

In at least my case and I assume in Patrik's and John Levine's,
when I said "ASCII", I might better have said "the ASCII graphic
subset of Unicode", or "the Basic Latin repertoire of Unicode",
or "Unicode code points in the range U+0030 (or perhaps U+0020)
through U+007E (or maybe +007A), i.e., a repertoire, not a
specific coding standard.  Given that all of the documents to
this particular discussion are pushing UTF-8 and that a Unicode
string from that repertoire in UTF-8 is indistinguishable at the
octet level from a ASCII string encoded by right-justifying each
seven-bit ASCII code point in an eight bit byte (octet) with a
leading zero bit, the shortcut should be obvious.  But you are
correct in that we were talking about repertoire, not encoding
or choice of character set.  

That said, it may be worth remembering that, independent of what
operating systems may or may not do, early Web specifications
were written assuming ISO 8859-1 and that there are almost
certainly some applications out there that assume that CCS when
they see octets with the leading bit on.  The ASCII repertoire
as described above is still a proper subset, but, because 8859-1
(and 8859-x more generally) are not UTF-8-compatible, the
ability to define the CCS and encoding in use may still be
necessary even if the necessity is waning.

best,
  john

--On Wednesday, February 5, 2020 14:50 -0800 Asmus Freytag
<[email-masked]> wrote:

> When we built the first Unicode-enabled OS (Windows NT), we
> had a long discussion of which "strings" in the OS needed to
> be Unicode.
> 
> Some thought that there was a clear dividing line between data
> and what would be called "protocol values" in another context.
> 
> Some of the latter did look like they were easily limited to
> ASCII; but everywhere we found "exceptions". There might be a
> set of enumerable tokens, but it allowed extended values that
> were network or file identifiers.
> 
> After exhaustively researching everything, the conclusion was
> that every single string in the OS had to be Unicode (and
> making any exceptions was either not possible, or not worth
> the effort).
> 
> However, while all strings were encoded in Unicode, not all
> string values were allowed. While file names could be
> localized (within the limits of file system syntax), some of
> the enumerated strings were left limited to the ASCII set in
> repertoire (even if encoded in Unicode).
> 
> Reading this discussion (and I'm sorry I don't have the time
> right now to properly delve into the details) it seems that a
> natural recommendation would be to require Unicode for any
> native representation and, if necessary (or possible), limit
> the repertoire.
> 
> This also requires a definition of the matching protocol for
> all strings that are to be matched as part of the protocol (or
> should be searchable). For any format, that would cover issues
> of casing, white space handling etc., but for Unicode, by
> necessity, that also requires defining the normalization form
> to be used.
> 
> A./
> 
> PS: given how few systems these days natively operate in any
> character set other than Unicode, I am always astonished at
> the length to which people go to justify not making something
> native Unicode. They just pick up conversion issues when they
> use platform libraries to do any work or display.
>

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 5 Feb 2020 20:18:06 -0800

On 2/5/2020 5:23 PM, John C Klensin wrote:
> specific coding standard.

Right; so far I'm with you.

I do agree that as far as repertoires go, it has some simplicity to it, 
although even with "ASCII" you run into questions of how you want to 
handle case.

>    Given that all of the documents to

I also agree that the ASCII repertoire has the nice property that it 
maps to the same octets whether UTF-8, Latin-1 of many other character sets.

Whether a protocol needs to support a CCS declaration is a separate 
question. Beyond explicitly identifying data as utf-8, that is (rather 
than implicitly via choice of protocol).

Allowing the protocol to be expressed in any other CCS than utf-8 is 
something that in today's environment should require to clear a 
substantial hurdle (by promising a really clear win in compatibility 
with key installed technology and which is not outweighed by imposing 
costs donwstream).

> necessary even if the necessity is waning.

Precisely the reason to make sure new protocols don't add to this 
problem other than with a gun to their heads, so to speak. Definitely 
not on the "there might be" level of reasoning, I'd think.

A./

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 Feb 2020 12:54:26 -0500

--On Wednesday, February 5, 2020 20:18 -0800 "Asmus Freytag (c)"
<[email-masked]> wrote:

>> In at least my case and I assume in Patrik's and John
>> Levine's, when I said "ASCII", I might better have said "the
>> ASCII graphic subset of Unicode", or "the Basic Latin
>> repertoire of Unicode", or "Unicode code points in the range
>> U+0030 (or perhaps U+0020) through U+007E (or maybe +007A),
>> i.e., a repertoire, not a specific coding standard.
> 
> I do agree that as far as repertoires go, it has some
> simplicity to it, although even with "ASCII" you run into
> questions of how you want to handle case.

Not just case.  In different situations, one runs into issues
with so-called "white space" characters and sequences of them,
interpretations and matching of punctuation characters,
line-ending conventions, and so on.  If one wanted simple and
uncomplicated, one needs to go back to fairly early versions of
BCP and even that, at least in in principle, could force one to
address string sorting and collation issues associated with
consecutive spaces and the implications of punctuation.

In addition, if one tries to map between less-extensive coding
systems and more extensive ones (or even equally-extensive
systems that have made different choices), one can easily get
into questions for which the answers that seem obvious on first
glance really are not.  If one goes from BCD (which was
upper-case-only) to ASCII, it seems obvious that the letters of
the former should be mapped to upper-case letters of the latter,
but that may be exactly wrong for some applications and
operating systems.  Should a sequence of space characters be
mapped to a horizontal tab or left alone?  And so on. The
correct and answer may be "it depends".  It may depend on the
application, the circumstances, or fairly subtle issues with the
language and/or locale involved.  A recent conversation with
people not on this list has reminded me that some of these
issues are reflections of ones that have been around for
millennia and for which we simply are not going to find "one
size fits all" solutions, much less be able to specify them in a
simple and clear way.  

I'm confident that you know all of of the above but it
reinforces several other aspects of the recent conversation
about draft-ietf-httpbis-header-structure-15, both in terms of
what we should suggest be done with that document and some
generalizations we might make:

(1) Unless it is extremely simple, any choice of character
repertoire is going to require additional conventions for some
uses and applications.  "Basic Latin upper case letters only"
may be simple enough as long as the strings are short, but, as
soon as digits are allowed, the "O" and "0" confusion/ ambiguity
issue arises (and a few here might be old enough to remember the
then-nearly-universal convention of writing digit-zero with a
slash, usually a reverse-slash, on coding sheets preparatory to
punching cards or paper tape in the 1940s and 1950s [1]).  In
any event, a corollary to the principle that making things more
complex makes them harder and more error-prone is "don't do it
unless you have to" and that is consistent with recommendations
to stick to a restricted ASCII repertoire unless there is a real
reason for allowing a significantly broader repertoire.

(2) There may well be exceptions but much of the time arguments
that there are strings -- especially in field values but often
in running text -- that we don't need to be concerned about
beyond simple character coding are likely to turn out to be
incorrect.  For field values, the most-often example cited
historically are email subject lines, but mail is sorted and
search for in archives using that field all the time and
procedures to duplicate elimination in mail archives typically
use that field as well as date and address ones as first-level
keys. 

(3) At least for i18n purposes, for a document that specifies
conventions for fields (or name-value pairs) that have been or
will be specified elsewhere (including container documents like
this one about structured headers) it is probably better to
specify what field-specific documents must specify or explain
than to write requirements, or even relatively specific
guidance, for all such fields.

>> this particular discussion are pushing UTF-8 and that a
>> Unicode string from that repertoire in UTF-8 is
>> indistinguishable at the octet level from a ASCII string
>> encoded by right-justifying each seven-bit ASCII code point
>> in an eight bit byte (octet) with a leading zero bit, the
>> shortcut should be obvious.  But you are correct in that we
>> were talking about repertoire, not encoding or choice of
>> character set.
> 
> I also agree that the ASCII repertoire has the nice property
> that it maps to the same octets whether UTF-8, Latin-1 of many
> other character sets.
> 
> Whether a protocol needs to support a CCS declaration is a
> separate question. Beyond explicitly identifying data as
> utf-8, that is (rather than implicitly via choice of protocol).

I agree that it is a separate question, but see above.  However,
when we don't know how a string will be used in the future --and
we rarely do-- supplying too much information is rarely harmful
while supplying too little and leaving things to heuristics
often turns out to be.

> Allowing the protocol to be expressed in any other CCS than
> utf-8 is something that in today's environment should require
> to clear a substantial hurdle (by promising a really clear win
> in compatibility with key installed technology and which is
> not outweighed by imposing costs donwstream).

I agree and would agree even more strongly had you said
"Unicode" with UTF-9 only as a strong preference.  The reason
for that distinction is at least one of the reasons why assorted
operating systems use UTF-16 or UTF-32 internally: the
variable-length relationship between a code point and the length
of the string (in bits or octets) needed to represent it in
UTF-8 can be a considerable nuisance in programming (as an
example from earlier in this thread, normalization is about code
points, not UTF-8 strings), even more than a nuisance than the
need to use surrogates for non-BMP code points in UTF-16.  I'm
not making that up, the "Comparison of the Advantages..."
discussion in Section 2.5 of TUS 12.0 is, IMO, quite good and
completely consistent with the above.  Now, if you had said
something like "on the wire" rather than "allowing the protocol
to be expressed", we would have been closer to agreement, but
I'd still caution that one of the things we have learned very
generally in the last 40 or 50 years is that strings that are
expected to be used only internally in operating systems tend to
leak into protocol fields.

In this context, it is perhaps relevant that IDNA talks about,
and is expressed in, code points, not an particular encoding
form.

>> That said, it may be worth remembering that, independent of
>> what operating systems may or may not do, early Web
>> specifications were written assuming ISO 8859-1 and that
>> there are almost certainly some applications out there that
>> assume that CCS when they see octets with the leading bit on.
>> The ASCII repertoire as described above is still a proper
>> subset, but, because 8859-1 (and 8859-x more generally) are
>> not UTF-8-compatible, the ability to define the CCS and
>> encoding in use may still be necessary even if the necessity
>> is waning.
> 
> Precisely the reason to make sure new protocols don't add to
> this problem other than with a gun to their heads, so to
> speak. Definitely not on the "there might be" level of
> reasoning, I'd think.

I agree, but note that the document under discussion is about
Structured Headers for HTTP and it would be very hard to argue
that HTTP is a new protocol.

best,
   john

[1] (I don't think I thought about it at the time, but I assume
that would have been required to be a very clear reverse-slash
in Norway, Denmark, etc. where U+00D8is common.  And that may be
a reasonable example of just how rapidly complexity rises.

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Thu, 6 Feb 2020 14:18:53 -0800

A couple of thoughts:

There are protocol parameters (like programming language keywords) and 
then there are object identifiers. For the former, issues like O vs. 0 
are annoyances - many of which can be avoided by proper choice of 
keywords. Object identifiers are chosen ad-hoc; only the repertoire and 
comparison rules are under the control of the protocol designer. And 
they may be directly supplied by/exposed to users.

Much of what you write below seems to me to primarily apply to such 
object identifiers and it's good to realize that even simple schemes 
(uppercase ASCII) have many edge conditions that need to be nailed down 
precisely. Even more true for more complex (internationalized) versions.

On compatibility: for a "bis" version of a protocol, there's always a 
question of whether it represents a break or a continuation from the 
original. Perhaps there are only some aspects that simply can't be kept 
compatible. However, anytime there is such a break, the question of 
limiting character set support to the universal character set ought to 
be raised.

On specifying utf-8 (not 9 :) ) vs. "Unicode". The high-level bit of 
this is the need to limit the Unicode encoding forms in the 
specification to a single one. Nobody benefits from having to support 
all of them.

UTF-8 has rather obvious advantages in interchange: it sidesteps the 
byte order problem (and therefore doesn't require any additional 
specification on how to uniquely serialize it to a byte stream). That 
should make it a good choice for any network protocol that isn't limited 
to stings of 7-bit items. (For those, some other serialization is 
needed, whether punycode or whatever).

This does not identify how the Unicode string is to be processed. You 
mention that IDNs work on the code point level. That's fine, and 
independent of how the data is transmitted. It is part of the 
specification that needs to be supplied anyway: how these protocol 
values are to be compared, sorted, etc. (The rules may well differ 
between keywords and object identifiers).

Finally, for containers, there's the question of what meta-data they 
should carry for their content. That's another issue altogether. If the 
content item is a document, defining the encoding it is in (or how to 
discover it) may be more universally applicable than defining a  
comparison method. The latter is more applicable to object identifiers.

Someone mentioned that this discussion should lead to a general 
guideline for protocol writers. I'm surprised there isn't one already, 
but if this is something to work out, count me in.

A./