Subject: [I18ndir] Fwd: Working Group Last Call: Structured Headers for HTTP

From: Martin J. Dürst <[email-masked]>
Date: Mon, 3 Feb 2020 04:38:21 +0000

Dear I18N Directorate,

I think we should do a review of the draft below. In particular, I think 
that in this day and age, something like "Strings are zero or more 
printable ASCII [RFC0020] characters" is highly outdated. See
https://tools.ietf.org/html/draft-ietf-httpbis-header-structure-15#section-3.3.3 
for details.

Regards,   Martin.

-------- Forwarded Message --------
Subject: Working Group Last Call: Structured Headers for HTTP
Resent-Date: Wed, 29 Jan 2020 17:11:30 +0000
Resent-From: [email-masked]
Date: Wed, 29 Jan 2020 09:11:20 -0800
From: Tommy Pauly <[email-masked]>
To: HTTP Working Group <[email-masked]>

Hi all,

The Structured Headers draft was updated yesterday, and Mark (as author) 
let me know that the authors believe this is ready for last call. The 
issues list is down to zero, so it's a good time for the working group 
to take a careful read-through the document, and get it progressed!

This email starts a working group last call, which will end on Friday, 
February 21. Please review the document and reply to the list with your 
comments (or file an issue), and state whether you think this document 
is ready for publication.

The current version of the document can be found here: 
https://tools.ietf.org/html/draft-ietf-httpbis-header-structure-15

Best,
Tommy

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Feb 2020 02:01:45 -0500

Martin,

Remembering that, at least in principle, the purpose of the
Directorate is just to advise the ART ADs, why not just send a
note to the WG list, during the WG LC, pointing out that, for a
document like this (and, as you point out, in this day and age),
they should either support a much broader range of characters of
include an Internationalization Considerations section that
explains why not. 

Because HTTP is not supposed to be visible to users, if the
Directorate is going to get involved, we will probably need to
have the usual discussion about protocol elements versus
user-visible strings, noting that one could easily have:

   Foo-Header: "Unicode in UTF-8 and NFKD"

Which is an all-ASCII description of a potentially non-ASCII
payload.  It seems to me that the fifth and six paragraphs of
that section (starting with "Unicode is not directly
supported...") address the issue in a clear way.  One can
disagree with the assertion there about causing interoperability
issues but that is a much more subtle and complex question than
"in this day and age".

Of course, in honor of the Rat Year and one of said rat's
favorite holes, we might then repeat the original IDNA argument
about UTF-8 discriminating against East Asian writing systems
and move toward a Bootstring-style encoding rather than
ASCII-encoded UTF-8.

And, FWIW, I am personally pleased to see that the draft is
using Base64 for byte sequences rather than %-encoded octets.    

best,
   john

--On Monday, 03 February, 2020 04:38 +0000 "Martin J. Dürst"
<[email-masked]> wrote:

> Dear I18N Directorate,
> 
> I think we should do a review of the draft below. In
> particular, I think  that in this day and age, something like
> "Strings are zero or more  printable ASCII [RFC0020]
> characters" is highly outdated. See
> https://tools.ietf.org/html/draft-ietf-httpbis-header-structur
> e-15#section-3.3.3  for details.
> 
> Regards,   Martin.
> 
> The Structured Headers draft was updated yesterday, and Mark
> (as author)  let me know that the authors believe this is
> ready for last call. The  issues list is down to zero, so it's
> a good time for the working group  to take a careful
> read-through the document, and get it progressed!
> 
> This email starts a working group last call, which will end on
> Friday,  February 21. Please review the document and reply to
> the list with your  comments (or file an issue), and state
> whether you think this document  is ready for publication.
>

---

From: "John Levine" <[email-masked]>
Date: 3 Feb 2020 12:34:04 -0500

In article <[email-masked]> you write:
>payload. ...

HTTP headers and mail headers share the same namespace.  Mail software
does not expect non-ASCII header names and Bad Things(tm) would happen
if we started sending mail with UTF-8 names.  EAI keeps ASCII field
names while allowing UTF-8 contents in headers intended to be shown to
people.

Keeping in mind that these names are all intended to be read by
computers, not by people, I don't see what problem they would solve,
but I can see a lot of problems they would cause.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Feb 2020 14:00:11 -0500

--On Monday, February 3, 2020 12:34 -0500 John Levine
<[email-masked]> wrote:

> HTTP headers and mail headers share the same namespace.  Mail
> software does not expect non-ASCII header names and Bad
> Things(tm) would happen if we started sending mail with UTF-8
> names.  EAI keeps ASCII field names while allowing UTF-8
> contents in headers intended to be shown to people.

I had assumed that Martin was concerned about HTTP header field
values, not field names.  If the concern was also about field
names, then everything I said earlier is even more the case and
I agree strongly with your comment below.

The SMTPUTF8 ("EAI") analogy doesn't help with either because,
at least when last I looked, HTTP had no equivalent of a
separate envelope in which one can negotiate options about the
header fields themselves on a "unless we both agree to this
option, the client is not allowed to send the strange stuff it
allows and the server is expected to reject it if sent".  For
SMTPUTF8, we could have allowed non-ASCII header names but
didn't, in large measure because of a list of problems they
would cause that probably at least overlaps with yours.

> computers, not by people, I don't see what problem they would
> solve, but I can see a lot of problems they would cause.

Yeah.  When I sent that note in response to Martin's, I thought
that changing the text to be more i18n-friendly was probably
unnecessary.  After a few hour's thought and your note, I think
it would be undesirable and that, were the relevant WG to change
the spec in some way, we should definitely review what was said,
and review it with something approaching extreme prejudice.

IMO at least, we get good internationalization along with good
interoperability by being very careful and thoughtful about what
should be allowed to be non-ASCII and what should be kept to as
simple a character repertoire as possible.  In addition to all
of the problems to which you refer, adopting a "Unicode
everywhere" model of the equivalent might actually hurt
localization efforts by making it much harder to translate a
field name or value into something that makes sense locally.

   john

---

From: "John R Levine" <[email-masked]>
Date: 3 Feb 2020 17:05:54 -0500

On Mon, 3 Feb 2020, John C Klensin wrote:
> values, not field names. ...

You're right, I misread it, but I still agree with you that it would not 
be a great idea to allow UTF-8.  While UTF-8 is the most popular encoding 
for web pages, it's not mandatory and I'm sure there are still plenty of 
pages in other encodings, particularly in Asian languages.

In section 3.3.3 it says that non-ASCII text is to be encoded as base64, 
which seems reasonable, "along with a character encoding (preferably 
UTF-8)" but it doesn't have any convention for where the encoding goes. 
Experience with MIME suggests that life is simpler if there's a standard 
form for an encoded text blob so libraries can decode and if need be 
transliterate in one go.

It also seems to me that this draft is reinventing JSON badly, but I don't 
think it would be helpful to point it out.  I do like section 1.1 which 
instructs implementors to fix bugs rather than working around them.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Feb 2020 20:30:10 -0500

--On Monday, February 3, 2020 17:05 -0500 John R Levine
<[email-masked]> wrote:

>...
> In section 3.3.3 it says that non-ASCII text is to be encoded
> as base64, which seems reasonable, "along with a character
> encoding (preferably UTF-8)" but it doesn't have any
> convention for where the encoding goes. Experience with MIME
> suggests that life is simpler if there's a standard form for
> an encoded text blob so libraries can decode and if need be
> transliterate in one go.

And omission of a clear model for specifying what is really what
we call a "charset" elsewhere (not just an encoding) is the one
clear defect in that particular discussion in the document.

Seems to me that, unless there is a desire to reinvent the
wheel, using the encoded word model of RFC 2231 (or something
very much like it)  but strictly with Base64, would be the right
way to go.

best,
  john

---

From: "John R Levine" <[email-masked]>
Date: 3 Feb 2020 20:32:45 -0500

> very much like it)  but strictly with Base64, would be the right

Agreed.  Anyone offering to tell them?

Regards,

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Feb 2020 20:47:22 -0500

--On Monday, February 3, 2020 20:32 -0500 John R Levine
<[email-masked]> wrote:

>> And omission of a clear model for specifying what is really
>> what we call a "charset" elsewhere (not just an encoding) is
>> the one clear defect in that particular discussion in the
>> document.
>> 
>> very much like it)  but strictly with Base64, would be the
>> right way to go.
> 

I think Martin may have volunteered, but he should confirm.   If
not, since this was posed as a question for the Directorate, I
think the burden falls on Pete (and maybe Peter) or to the ADs
on the list to pick this ball up and run with it.

Of course, were the document to go into IETF LC without that
issue addressed, it would constitute a Known Technical Omission.
That, in turn, would be grounds for either a nasty comment from
any handy individual during LC or for an appeal of the decision
to open the LC on the document since, as of now, the issue has
been brought to the attention of the ADs who would need to
initiate the Last Call.  Much better to get the word to the WG.

best,
  john

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 04 Feb 2020 13:07:02 +0100

On 4 Feb 2020, at 2:30, John C Klensin wrote:

> And omission of a clear model for specifying what is really what we call a "charset" elsewhere (not just an encoding) is the one clear defect in that particular discussion in the document.

Don't forget normalization or otherwise how comparison is to be done of the UTF-8 encoded characters in the string. Nothing about that either.

   Patrik

> wheel, using the encoded word model of RFC 2231 (or something very much like it)  but strictly with Base64, would be the right way to go.
>
> best,
>   john
>
> -- 
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2020 10:08:10 -0500

> Don't forget normalization or otherwise how comparison is to be done of the UTF-8 encoded characters in the string. Nothing about that either.

NFC is specific to UTF-8.  MIME doesn't have that as far as I know so I 
don't think it makes sense here.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Feb 2020 11:06:20 -0500

--On Tuesday, February 4, 2020 13:07 +0100 Patrik Fältström
<[email-masked]> wrote:

> Don't forget normalization or otherwise how comparison is to
> be done of the UTF-8 encoded characters in the string. Nothing
> about that either.

The omission was deliberate, so I should explain.  Let me take
advantage of John Levine's observation about email headers, with
which we have much more experience.  Very loosely and
informally, there are two types of header field data - the stuff
that is highly structured like addresses (e.g., "From:", "To:",
and "Cc:") and maybe even "Date:" and the stuff, like "Subject:"
fields, that is basically free text.  For the latter, in most
cases one doesn't care whether it can be compared to other
strings or not -- the important thing is that it can be read by
the user and not be confusing.  Of course, someone might want to
sort or select on unstructured fields, but that may not impose
as strong a requirement.   For HTTP headers that contain
structured material, strict normalization and comparison rules
might be less important -- as I suggested to John, one place
where the analogy breaks down is that many or most HTTP headers
are closer to envelope information in email than they are to
user presentation information.  

What is more important is that, as The Unicode Standard has been
suggesting all along and more recent W3C work addresses in a
more refined way, in many or most cases (and even for more
structured fields), the right answer is to normalize (and apply
other rules as needed) only at comparison time, normalizing both
(all) strings to be compared, and not even bothering to store
the normalized strings after the comparison is completed.  I
still believe that we got IDN labels right by requiring that
both the stored form and any lookup keys be normalized and
structured early, but they are, for several reasons, an unusual
case.  

So, out of concern that anything that looked like "one size fits
all" might turn out to be closer to "...fits none", I decided to
avoid touching those issues in the context of this particular
document.  Maybe that is wrong but, if it is, I'd be inclined to
suggest that any document that specifies a new HTTP header field
type and that allows characters in the value/data that are not
natively ASCII MUST include an Internationalization
Considerations section that explicitly addresses these issues.

Does that make sense?  Do we think that sort of requirement is
needed and would be helpful?

   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Feb 2020 11:27:04 -0500

--On Tuesday, February 4, 2020 10:08 -0500 John R Levine
<[email-masked]> wrote:

>> be done of the UTF-8 encoded characters in the string.
>> Nothing about that either.
> 
> NFC is specific to UTF-8.  MIME doesn't have that as far as I
> know so I don't think it makes sense here.

Huh?   NFC is certainly specific to Unicode, but there is
nothing specific to UTF-8 (or any other encoding form) about it.
My highly abstracted (but I hope not far off the mark) view of
Patrik's comments is that, if one is going to compare or sort
strings --or do much else with them other than maybe displaying
them -- is that one needs a plan other than the equivalent of
"just use UTF-8".   Fairly recent W3C discussions, a few around
ICANN, and comments in TUS about script-specific or
language-specific rendering, strongly suggest that a more
specific plan is needed even to display strings reasonably in
some scripts and circumstances.

Note that this is another example of a situation in which the
problem exists even with basic Latin characters and moving
beyond them just makes the situation more important or worse or
the examples more dramatic.  The obvious case involves sorting
strings into alphabetic order: one has to decide what to do
about case mapping or ordering, what to do about spaces or other
punctuation, and so on.

Where Patrik and I disagree (if we disagree at all - I may have
misread his intent) is whether this particular document needs to
say anything on the subject or anything beyond "definitions for
individual header fields need to include an analysis and
guidance for those issues".

I also think that, in our capacity of giving advice to the ART
ADs, there is an important lesson here and that is that _any_
document that even hints at non-ASCII characters, even if the
hint is nothing more than "don't do that unless you have to"
MUST get serious review by experts in i18n issues, not just the
WG that happens to be writing the spec.  That, in turn, almost
certainly implies the need for an organized and functioning
directorate, not just one to which one of those on the list can
post a note (as Martin did) and hope for responses from the
usual suspects.  Or, if they can't make such a directorate work,
either some other plan to accomplish the same thing or explicit
acknowledgment that IETF work on these is not getting
sufficient, and sufficiently skilled, review to be trusted.

Watch for what may be a more complicated example soon.

   john

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2020 11:51:44 -0500

> nothing specific to UTF-8 (or any other encoding form) about it.

Sorry, I meant specific to Unicode.  If there is some old page in big5 or 
ISO-2022-x, there aren't any normalization rules I'm aware of.

I think we agree, normalization is a level up from what they're describing 
here.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Feb 2020 14:24:33 -0500

--On Tuesday, February 4, 2020 11:51 -0500 John R Levine
<[email-masked]> wrote:

> Sorry, I meant specific to Unicode.  If there is some old page
> in big5 or ISO-2022-x, there aren't any normalization rules
> I'm aware of.

IIR, Big5 would not need one because it is strictly Traditional
Chinese Character.  As soon as one has something that would
contain both Traditional one for which Simplified forms exist
and those Simplified forms, one _may_ need a mechanism for
comparing them equal, but that wouldn't be normalization in the
Unicode sense either [1].   If by "ISO-2022-x" you mean either
ISO 8859/x or code pages that can be accessed or switched by ISO
2022 methods, definitely no normalization for the first case and
probably not the latter because at least most of them were
designed to avoid the problems (like composing character
sequences) that normalization addresses.

> I think we agree, normalization is a level up from what
> they're describing here.

Ack.

   best,
    john

[1] In retrospect, Unicode's using, and effectively preempting,
the terms "normalization" and, especially, "canonicalization"
for very specific purposes in the context of their standard
because it makes any use of, for example, "canonical form" in
describing a string or identifier ambiguous as to whether
Unicode canonicalization (as in NFKC) or some other set of rules
was intended.  Having, or converting to, some "normal form"
isn't much better.  Nothing can be done about that now, but it
means we should remind ourselves that we need to be very careful
to be clear when we use of those terms or similar ones.

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 04 Feb 2020 22:47:05 +0100

On 4 Feb 2020, at 17:06, John C Klensin wrote:

> Does that make sense?  Do we think that sort of requirement is needed and would be helpful?

Yes, something at least. Because from my point of view IF you use Unicode in something that is to be compared as a protocol element (header and/or value) one should specify what the comparison algorithm is. This might be that data is to be normalized or that normalization is to happen at the time when comparison happens.

Just not stating anything -- even if the field itself is ascii only as in this case -- is I think a bit naive. Just at least say something. Which in the case of "headers which are ascii only while values can be UTF-8 encoded values" could be very easy.

I just want "comparison" be mentioned when people talk about unicode. Regardless of how they then specify the comparison algorithm.

Comments on the comparison algorithm specified is a different thing.

Am I wrong?

I just see too many programmers moving from text strings to unicode strings and then they think they are done. I am so hard trying to tell them "just do one more step" and think about comparison (even if the library they use "might do the right thing") for a few seconds.

I have been burned too much for example with the NFD normalization of the Apple file system which forced use of special rsync software which of course was newer than what Apple distributed etc... We all have been down this hole.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 04 Feb 2020 22:48:10 +0100

On 4 Feb 2020, at 17:27, John C Klensin wrote:

> Where Patrik and I disagree (if we disagree at all - I may have misread his intent) is whether this particular document needs to say anything on the subject or anything beyond "definitions for individual header fields need to include an analysis and

At least for me it must be a conscious decision, and I do not see that.

   Patrik

---

From: "John R Levine" <[email-masked]>
Date: 4 Feb 2020 16:56:10 -0500

> Yes, something at least. Because from my point of view IF you use Unicode in something that is to be compared as a protocol element (header and/or value) one should specify what the comparison algo…

Take a look at the document again.  It's a container format, sort of 
half-hearted JSON.  I don't see anything that suggests that the strings 
are likely to be tokens you'd want to do exact matches on.

Regards,

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 04 Feb 2020 23:35:11 +0100

>> I think we agree, normalization is a level up from what

Ok, I have tried to really understand what they are doing here, and it feels like if Unicode was pasted in after they had designed the whole thing.

Look for example at B.1 when they compare with JSON. They say one advantage of their format is that JSON do allow Unicode data which gives interoperability issues, but they do allow it themselves as well.

Then this is 3.3.3:

> Unicode is not directly supported in strings, because it causes a
> number of interoperability issues, and - with few exceptions - header
> values do not require it.

How do they know header values do not need it -- with a few exceptions?

> When it is necessary for a field value to convey non-ASCII content, a
> byte sequence (Section 3.3.5) SHOULD be specified, along with a
> character encoding (preferably [UTF-8]).

I think that the default encoding MUST be UTF-8 OR specified explicitly.

I further think it should be noted comparison of parameter values is NOT specified in this base specification as normalization might create non-interoperability. If needed, the specification of the header must say how comparison is managed.

I also think 4.2.7 should mention the resulting sequence of the parsing of a binary structure might be a UTF-8 encoded string with a reference to 3.3.3.

I.e. I find it hard to read (and understand why I misunderstood this at first) that the only mentioning of Unicode and UTF-8 is in "string" but the only thing it does is to reference byte sequence, which in turn do never talk about it. Neither at serialization or deserialization. So where UTF-8 strings are described thet are not mentioned.

But ok...you are right.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Feb 2020 21:32:25 -0500

--On Tuesday, February 4, 2020 23:35 +0100 Patrik Fältström
<[email-masked]> wrote:

> Ok, I have tried to really understand what they are doing
> here, and it feels like if Unicode was pasted in after they
> had designed the whole thing.

Indeed.  It feels as if they started with "all ASCII" and then
someone said "well, how about cases that need non-ASCII
characters", and they responded with a lot of circling around,
much of which amounts to "just use UTF-8".  That is something we
know is almost always inadequate or worse and often a sign of
deeper problems and lack of thought.

> Look for example at B.1 when they compare with JSON. They say
> one advantage of their format is that JSON do allow Unicode
> data which gives interoperability issues, but they do allow it
> themselves as well.
> 
> Then this is 3.3.3:
> 
>> Unicode is not directly supported in strings, because it
>> causes a number of interoperability issues, and - with few
>> exceptions - header values do not require it.
> 
>> When it is necessary for a field value to convey non-ASCII
>> content, a byte sequence (Section 3.3.5) SHOULD be specified,
>> along with a character encoding (preferably [UTF-8]).
> 
> I further think it should be noted comparison of parameter
> values is NOT specified in this base specification as
> normalization might create non-interoperability. If needed,
> the specification of the header must say how comparison is
> managed.
> 
> I also think 4.2.7 should mention the resulting sequence of
> the parsing of a binary structure might be a UTF-8 encoded
> string with a reference to 3.3.3.
> 
> I.e. I find it hard to read (and understand why I
> misunderstood this at first) that the only mentioning of
> Unicode and UTF-8 is in "string" but the only thing it does is
> to reference byte sequence, which in turn do never talk about
> it. Neither at serialization or deserialization. So where
> UTF-8 strings are described thet are not mentioned.
> 
> But ok...you are right.

Right about what?  We may be in agreement after all.  I think
there are at least three different issues here.

(1) I took Martin's note, especially the "in this day and age"
part, as suggesting that the document should encourage, rather
than discourage, non-ASCII strings in header field values.   I
suggested that, in most cases, discouraging such values was
entirely appropriate although it was reasonable to make
provision for them in special cases.

(2) The we got to the question of whether non-ASCII strings
should be allowed in header field names at all.   I think we (at
least you, John Levine, and me) are in agreement that it almost
certainly a bad idea.

(3) In the situations that non-ASCII characters are allowed in
header field values, we are in agreement that the description is
inadequate in multiple ways.  They include that the spec should
either require UTF-8 everywhere (e.g., "encoding MSUT be Unicode
in UTF=8") or the description / container syntax  must specify
how and in what syntax the encoding (or "charset') is specified.
They also include a requirement that any header files defined
according to this spec, at least ones that allow non-ASCII
characters  MUST be required to include in those definitions.
Those definitions would be required to have either an
"Internationalization Considerations" section or or equivalent.
And that section MUST either explain why comparison, searching,
or ordering of all or part of the header field value is never
going to be required  or describe the mechanisms or conventions
needed to carry out the operations.

For (3), I first understood you to be suggesting a discussion
of, e.g., normalization in the container/ format document that
would apply to all header fields defined under it.  I think that
would be wrong because different header fields and their values
might require different treatment.  However, I certainly agree
that information has to be provided somewhere and that this
document is deficient in saying as little as it does about
strings, byte sequences, and non-ASCII characters without
spelling out the requirement for that type of information to be
specified somewhere.   I also think that the IETF must never
again produce a technical specification or BCP that deal with or
allows non-ASCII characters by saying the equivalent of "just
use UTF-8".

Does that put us closer together?

   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 05 Feb 2020 11:37:27 +0100

On 5 Feb 2020, at 3:32, John C Klensin wrote:

> Right about what?

That specifics about normalization should not be in a document that describes a container format.

> We may be in agreement after all.  I think there are at least three different issues here.

I hope so as well.

> part, as suggesting that the document should encourage, rather than discourage, non-ASCII strings in header field values.   I suggested that, in most cases, discouraging such values was
> should be allowed in header field names at all.   I think we (at least you, John Levine, and me) are in agreement that it almost certainly a bad idea.

I think restricting unicode to parameter values (which is even more strict than header field values, as you might have key/value pairs as part of a header field value in the draft) is correct without major redesign of the whole thing.

> (3) In the situations that non-ASCII characters are allowed in header field values, we are in agreement that the description is inadequate in multiple ways.  They include that the spec should either require UTF-8 everywhere (e.g., "encoding MSUT be Unicode in UTF=8") or the description / container syntax  must specify how and in what syntax the encoding (or "charset') is specified.
> They also include a requirement that any header files defined according to this spec, at least ones that allow non-ASCII

Agree.

> And that section MUST either explain why comparison, searching, or ordering of all or part of the header field value is never going to be required  or describe the mechanisms or conventions needed to carry out the operations.

Yes.

> For (3), I first understood you to be suggesting a discussion of, e.g., normalization in the container/ format document that would apply to all header fields defined under it.  I think that would be wrong because different header fields and their values might require different treatment.  However, I certainly agree that information has to be provided somewhere and that this
> spelling out the requirement for that type of information to be specified somewhere.   I also think that the IETF must never again produce a technical specification or BCP that deal with or allows non-ASCII characters by saying the equivalent of "just use UTF-8".
>

Yes.

My question was whether not this spec:

1. Should point out normalization is recommended as only UTF-8 is not enough.

2. Should talk about Unicode "more" when looking at byte sequences. I.e. it does not talk about it at all when looking at byte sequences. It talks about non-ascii in one place, in the definition of "text" and says "non-ascii is not next, it is binary".

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Wed, 05 Feb 2020 07:50:27 -0500

Executive summary for Murray and any current ART ADs who have
not been following the I18N Directorate (i18ndir) list:  Martin
Dürst posted a suggestion to the list several days ago that
suggested we take a careful look at
draft-ietf-httpbis-header-structure-15.  The request has led to
a lively discussion among John Levine, Patrik Fältström and
myself, during which I, at least, have learned a great deal
especially about how difficult it is to find just the right
language and terminology to talk about these areas even among
experts with decades of experience working together on them.
This note is, I hope, at or near the end of the thread rather
than at the beginning or middle and ends with a few paragraphs
of draft recommendations to you as ADs.   You can probably 
skip over the rest of it and the earlier entries in the thread
unless you are interested in either how we got her or a deeper
understanding of the issues.

> that describes a container format.

Ah.  See below.

>> part, as suggesting that the document should encourage,
>> rather than discourage, non-ASCII strings in header field
>> values.   I suggested that, in most cases, discouraging such
>> values was entirely appropriate although it was reasonable to
>> make provision for them in special cases.
>> 
>> should be allowed in header field names at all.   I think we
>> (at least you, John Levine, and me) are in agreement that it
>> almost certainly a bad idea.
> 
> I think restricting unicode to parameter values (which is even
> more strict than header field values, as you might have
> key/value pairs as part of a header field value in the draft)
> is correct without major redesign of the whole thing.

Agree about the principle.   I wonder if, some day, that might
turn out to be over-restrictive.   On the other hand, this is
about "structured headers" and not unstructured ones, so a
parameter restriction is probably reasonable.  As I read your
note and think about it more, the core issue is probably
different; see under "my question" below.

>> (3) In the situations that non-ASCII characters are allowed
>> in header field values, we are in agreement that the
>> description is inadequate in multiple ways.  They include
>> that the spec should either require UTF-8 everywhere (e.g.,
>> "encoding MSUT be Unicode in UTF=8") or the description /
>> container syntax  must specify how and in what syntax the
>> encoding (or "charset') is specified. They also include a
>> requirement that any header files defined according to this
>> spec, at least ones that allow non-ASCII characters  MUST be
>> required to include in those definitions.
> 
> Agree.
> 
>> "Internationalization Considerations" section or or
>> equivalent. And that section MUST either explain why
>> comparison, searching, or ordering of all or part of the
>> header field value is never going to be required  or describe
>> the mechanisms or conventions needed to carry out the
>> operations.
> 
> Yes.
> 
>> of, e.g., normalization in the container/ format document
>> that would apply to all header fields defined under it.  I
>> think that would be wrong because different header fields and
>> their values might require different treatment.  However, I
>> certainly agree that information has to be provided somewhere
>> and that this document is deficient in saying as little as it
>> does about strings, byte sequences, and non-ASCII characters
>> without spelling out the requirement for that type of
>> information to be specified somewhere.   I also think that
>> the IETF must never again produce a technical specification
>> or BCP that deal with or allows non-ASCII characters by
>> saying the equivalent of "just use UTF-8".
>> 

Again, aha.   This is where my thinking has evolved considerably
in the last five or six years.  Well before that, I would have
said (and did) that say "just used UTF-8" was not enough but
that normalization was sufficient.  Now, after the
non-decomposing character discovery and analysis, more complex
debates about confusable characters, Asmus's "troublesome"
efforts, vomiting cowboys and their various friends and
relatives, and many other issues, I would say that recommending
UTF-8 and normalization is not enough either.

The question of whether this document should specify what should
be done or should specify that any definition of a particular
field written to be consistent with it must specify that is
separate.  I think we are, at this point, in agreement that it
should be the second.

> 2. Should talk about Unicode "more" when looking at byte
> sequences. I.e. it does not talk about it at all when looking
> at byte sequences. It talks about non-ascii in one place, in
> the definition of "text" and says "non-ascii is not next, it
> is binary".

And I didn't catch that until now.  That is exactly got us into
trouble with the DNS, i.e., the claim that if a string was
all-ASCII (in that case, defined as none of the octets having a
non-zero leading bit), then it obeyed strict,
carefully-spelled-out rules but, if it wasn't, it was strictly a
string of octets with no defined semantics at all.  That means
two strings don't match unless they are bit-identical and that
any sorting operations effectively have to be on the full string
(not octet by octet and not considering the variable length for
different code points encoding of things like UTF-8) and that
maybe there is a big-endian versus little-endian question.  Even
without concerns about normalization (much less the other things
suggested above), as soon as one talks about Unicode (in UTF-8
or otherwise), one is talking about "characters", not byte (or
bit) sequences because any actual use or interpretation will
depend on code points, not bits or octets.   So the
recommendation to use UTF-8 and the recommendation to treat
things as byte sequences are contradictory at a fundamental
technical level and the penultimate paragraph of Section 3.3.3
is bad news.

I haven't studied things carefully enough, but there is another
fundamental problem with this document.   RFC 8187 already
specifies a way to do character handling in HTTP header fields,
one that is consistent with my/our comments about encoded words
several days ago.  It also specifies that UTF-8 is mandatory to
implement.  That this document does not either reference or
update that one but is inconsistent with it in suggesting
treating character sequences as simply byte sequences seems to
me to be a grave omission.

Martin's "in this day and age" comment applies to the
questions/comments above even if it didn't apply as an argument
for more internationalization.  Maybe that is what he meant in
the first place-- his has been surprisingly silent since this
three-way discussion started.

Is that about right?

If it is, I think it is time for two directorate
recommendations.  One is that this document, with respect to
i18n issues, is not ready for IETF LC.  It may even be that the
HTTP WG needs to clean up its act and, for i18n issues, may be
in need of adult supervision because they are contradicting
existing standards track documents within their own scope,
apparently without noticing.

The second is that it may be time to update the recommendations
about character sets, Internationalization Considerations, etc.
I don't know whether the revised recommendation should find
expression in an IESG statement, a new BCP, or some other form
of guidance or direction but it is now clear to me that, without
it, the IETF is just going to dig itself in deeper and deeper...
even more so if there is not a functioning directorate or
equivalent than if there is.   Those recommendation would be,
approximately:

(i) For any document that discusses or allows the use of
non-ASCII characters, UTF-8 is mandatory to implement.  Other
encodings of Unicode or use of other coded character sets may be
justified by circumstances but, if they are allowed, the CCS and
any additional encoding information required with it must be
identified, e.g., by the mechanisms defined in RFCs 2231 and
8187.

(ii) For any document that allows or specifies the use of
non-ASCII characters in a protocol context (even in what is
nominally free text), neither "use UTF-8" nor "Use UTF-8 and
specify a normalization" are sufficient.  The presence of such
statements without further explanation is usually an indication
that an author or WG has paid insufficient attention to i18n
issues but is, instead and intentionally or not, trying to blow
them off.  In almost all cases that will be visible to users,
character strings will eventually be compared to others, sorted,
searched and/or rendered.  For one or more languages or scripts,
each of those operations requires treatment that goes beyond
simply examining octets.   Documents allowing such strings must
either specify how those issues are addressed or must explain
why they are not applicable.  For characters or strings than are
protocol elements not visible to users, specific justification
is required if non-ASCII characters are to be allowed.

best,
   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 05 Feb 2020 14:55:50 +0100

> Is that about right?

Yes

> recommendations.  One is that this document, with respect to i18n issues, is not ready for IETF LC.  It may even be that the HTTP WG needs to clean up its act and, for i18n issues, may be in need of adult supervision because they are contradicting

Yes

> The second is that it may be time to update the recommendations about character sets, Internationalization Considerations, etc.
> expression in an IESG statement, a new BCP, or some other form of guidance or direction but it is now clear to me that, without it, the IETF is just going to dig itself in deeper and deeper...
> equivalent than if there is.

This is something IESG have to give advice on.

> Those recommendation would be, approximately:
>
> non-ASCII characters, UTF-8 is mandatory to implement.  Other encodings of Unicode or use of other coded character sets may be justified by circumstances but, if they are allowed, the CCS and any additional encoding information required with it must be identified, e.g., by the mechanisms defined in RFCs 2231 and 8187.

Yes

> nominally free text), neither "use UTF-8" nor "Use UTF-8 and specify a normalization" are sufficient.  The presence of such statements without further explanation is usually an indication that an author or WG has paid insufficient attention to i18n issues but is, instead and intentionally or not, trying to blow them off.  In almost all cases that will be visible to users, character strings will eventually be compared to others, sorted, searched and/or rendered.  For one or more languages or scripts, each of those operations requires treatment that goes beyond simply examining octets.   Documents allowing such strings must either specify how those issues are addressed or must explain why they are not applicable.  For characters or strings than are protocol elements not visible to users, specific justification is required if non-ASCII characters are to be allowed.

There is one sentence that I think could be removed, so the recommendation is (one misspelling of "than" which should be "that" in the last sentence as well I think:

(ii) For any document that allows or specifies the use of non-ASCII characters in a protocol context (even in what is nominally free text), neither "use UTF-8" nor "Use UTF-8 and specify a normalization" are sufficient.  In almost all cases that will be visible to users, character strings will eventually be compared to others, sorted, searched and/or rendered.  For one or more languages or scripts, each of those operations requires treatment that goes beyond simply examining octets.   Documents allowing such strings must either specify how those issues are addressed or must explain why they are not applicable.  For characters or strings that are protocol elements not visible to users, specific justification is required if non-ASCII characters are to be allowed.

   Patrik

---

From: "John R Levine" <[email-masked]>
Date: 5 Feb 2020 10:02:27 -0500

> understanding of the issues.

One other point worth mentioning: we are not asking them to invent new 
stuff.  For example, this draft has base64 encoded i18n text which needs a 
tag for the encoding.  MIME already does that, they can reuse the same 
syntax.

R's,
John

Regards,

---

From: John C Klensin <[email-masked]>
Date: Wed, 05 Feb 2020 12:41:30 -0500

--On Wednesday, February 5, 2020 10:02 -0500 John R Levine
<[email-masked]> wrote:

>> not been following the I18N Directorate (i18ndir) list:
>> Martin Dürst posted a suggestion to the list several days
>> ago that suggested we take a careful look at
>> draft-ietf-httpbis-header-structure-15.  The request has led
>> to a lively discussion among John Levine, Patrik Fältström
>> and myself, during which I, at least, have learned a great
>> deal especially about how difficult it is to find just the
>> right language and terminology to talk about these areas even
>> among experts with decades of experience working together on
>> them. This note is, I hope, at or near the end of the thread
>> rather than at the beginning or middle and ends with a few
>> paragraphs of draft recommendations to you as ADs.   You can
>> probably skip over the rest of it and the earlier entries in
>> the thread unless you are interested in either how we got her
>> or a deeper understanding of the issues.
> 
> One other point worth mentioning: we are not asking them to
> invent new stuff.  For example, this draft has base64 encoded
> i18n text which needs a tag for the encoding.  MIME already
> does that, they can reuse the same syntax.

Not only does MIME do that and specify a syntax, but one of the
things I noticed (and that was mentioned later in the note) is
that there is already a Standards Track specification for doing
exactly that in HTTP, specifically by including adapting the
MIME encoded word form.  See RFC 8187 and note that this spec
seems to be unaware of its existence despite, presumably, coming
out of the same WG.  

I hope no one was offended by my use of the term "adult
supervision" in that context but when a WG appears to be unaware
of existing standards track specifications that appear to fall
precisely within the same narrow area of work (HTTP header
fields in this case) and contradicts them, we move, IMO, out of
the area of technical problems and into the area of management
ones.

    john

    john

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 5 Feb 2020 10:26:10 -0800

> I just want "comparison" be mentioned when people talk about unicode. Regardless of how they then specify the comparison algorithm.

Already for ASCII there's the distinction between case sensitive or not, 
and/or white-space collapse or even, as in comparing names of Unicode 
characters (or property aliases) more elaborate equivalences.

Specifying comparison algorithms makes sense for data elements that must 
be acted on as part of the protocol, but does not necessarily make sense 
for text that is only human-readable (for example, comments).

Some of this data may eventually be something that one might search for, 
but I feel, just like the contents of a document, such search algorithms 
don't belong in the spec describing the container for that data.

> I just see too many programmers moving from text strings to unicode strings and then they think they are done. I am so hard trying to tell them "just do one more step" and think about comparison (e…

This includes knowing what items in a protocol must have a well-defined 
equivalence set that defines uniqueness for the purpose of the protocol.

> I have been burned too much for example with the NFD normalization of the Apple file system which forced use of special rsync software which of course was newer than what Apple distributed etc... W…
A./