Subject: [I18ndir] Review volunteer needed (Fwd: [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07)

From: "Pete Resnick" <[email-masked]>
Date: Tue, 28 Apr 2020 14:14:35 -0500

Folks,

DISPATCH has made a working group last call on 
draft-ietf-dispatch-javascript-mjs. Given how much i18n content there 
is, the ADs and chairs agreed that an early review wouldn't hurt. There 
is a bunch of text about formal programming languages in this draft, so 
I didn't want to just stick somebody with it. Anyone feel comfortable 
enough to take on a review?

Note that the review itself is much less horrible than it seems. This is 
really a -bis draft of RFC 4329, and most of the i18n language in here 
is unchanged from 4329. See:

https://www.ietf.org/rfcdiff?url1=rfc4329&url2=draft-ietf-dispatch-javascript-mjs-07

So it's really just reviewing any changes, and making sure that nothing 
absolutely egregious was in 4329 itself.

Thanks,

pr

Forwarded message:

> From: Ben Campbell <[email-masked]>
> To: DISPATCH WG <[email-masked]>
> Cc: dispatch chairs <[email-masked]>, ART ADs 
> <[email-masked]>, [email-masked]
> Subject: [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07
> Date: Thu, 23 Apr 2020 14:24:47 -0500
>
> Hi,
>
> This is a repeat working group last call of 
> draft-ietf-dispatch-javascript-mjs-07.
>
> https://datatracker.ietf.org/doc/draft-ietf-dispatch-javascript-mjs/
>
> The new version has changed quite a bit due to feedback from the first 
> WGLC, shepherd feedback, and feedback from others. In particular, this 
> version obsoletes RFC 4329, rather than updating as did the version 
> from the first WGLC.
>
> This WGLC will end on 8 May 20202. Please send feedback to the 
> DISPATCH list and the authors. And if you review the document and 
> think it’s ready to go, please say so.
>
> Thanks!
>
> Ben

---

From: John C Klensin <[email-masked]>
Date: Tue, 28 Apr 2020 21:41:25 -0400

Pete,

Since some hours have gone by without a response to your message
and I was in need of an excuse to delay getting to an unpleasant
task...

This note does not constitute either a review or an offer to do
one, but I just took a quick look at RFC 4329 and the diff and,
in a few seconds, found one statement/requirement in 4329 that
is unchanged (although circumstances have) that is an invitation
to a world of grief.  In addition, given Section 4.1, the
algorithm in 4.2 (both versions) is incomplete and can result in
a fairly serious error.

Moreover, if I correctly understand what seems like unnecessary
convoluted text (in both versions) a BOM is ignored in further
processing if the character encoding scheme is determined to be
UTF-8 in 4.2(2) or 4.2(3) but not ignored if charset="UTF-8" is
present and the BOM occurs anyway (something clearly allowed by
RFC 3629).  That doesn't appear to make sense.  If it was
intended, enough of an explanation would be in order that the
reader does not concluded it is just a mistake in the document
(I notice the I-D already corrected a mistake in 4329).  So,
having spent five minutes with 4329 and the diff and another
five writing this note, I can say with some confidence that the
I-D needs work and that someone is going to need to work with
the document authors to explain the issues, some of which are
substantive, not just text in need of tuning, and sort things
out.   You may be looking for a volunteer to do that job, not
just an early reviewer.

Back to lurking.
    john

--On Tuesday, April 28, 2020 14:14 -0500 Pete Resnick
<[email-masked]> wrote:

> draft-ietf-dispatch-javascript-mjs. Given how much i18n
> content there is, the ADs and chairs agreed that an early
> review wouldn't hurt. There is a bunch of text about formal
> programming languages in this draft, so I didn't want to just
> stick somebody with it. Anyone feel comfortable enough to take
> on a review?
> 
> Note that the review itself is much less horrible than it
> seems. This is really a -bis draft of RFC 4329, and most of
> the i18n language in here is unchanged from 4329. See:
> 
> So it's really just reviewing any changes, and making sure
> that nothing absolutely egregious was in 4329 itself.
> 
> Thanks,
> 
> pr
> 
> Forwarded message:
> 
>> <[email-masked]>,
>> [email-masked] Subject:
>> [dispatch] WGLC of draft-ietf-dispatch-javascript-mjs-07
>> The new version has changed quite a bit due to feedback from
>> the first  WGLC, shepherd feedback, and feedback from others.
>> In particular, this  version obsoletes RFC 4329, rather than
>> updating as did the version  from the first WGLC.
>> 
>> This WGLC will end on 8 May 20202. Please send feedback to
>> the  DISPATCH list and the authors. And if you review the
>> document and  think it's ready to go, please say so.
>> 
>> Thanks!
>> 
>> Ben

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 29 Apr 2020 12:00:07 -0500

Thanks for the quick look over and the heads up. Is anyone willing to 
take this on? I will shortly start going after individuals.

pr

-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 29 Apr 2020 20:58:51 +0200

On 29 Apr 2020, at 3:41, John C Klensin wrote:

> Since some hours have gone by without a response to your message and I was in need of an excuse to delay getting to an unpleasant task...

Now I have read the draft as well.

> Moreover, if I correctly understand what seems like unnecessary convoluted text (in both versions) a BOM is ignored in further processing if the character encoding scheme is determined to be UTF-8 in 4.2(2) or 4.2(3) but not ignored if charset="UTF-8" is present and the BOM occurs anyway (something clearly allowed by RFC 3629).  That doesn't appear to make sense.

I think I see the same thing as you, which is that even if the charset parameter states the encoding is UTF-8, if the data itself starts with a BOM, then the text is to be treated as UTF-16.

That is just so wrong.

I went to the ECMA spec and see they use UTF-16 all over the place, and have to bend over backwards to get things right. It feels like reading a BER encoding spec (again). :-)

This "problem" do already exist in RFC 4329...

But, if they update RFC 4329 I think they should clean this up, and my suggestion would be:

The encoding must be what is actually labeled. If the encoding is UTF-16 (which it seems it often is), then it should be tagged as UTF-16, not UTF-8 with BOM.

   Patrik

> If it was
> intended, enough of an explanation would be in order that the reader does not concluded it is just a mistake in the document (I notice the I-D already corrected a mistake in 4329).  So, having spent five minutes with 4329 and the diff and another five writing this note, I can say with some confidence that the I-D needs work and that someone is going to need to work with the document authors to explain the issues, some of which are substantive, not just text in need of tuning, and sort things out.   You may be looking for a volunteer to do that job, not just an early reviewer.
>
> Back to lurking.
>     john
>
>
>
>
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: John C Klensin <[email-masked]>
Date: Wed, 29 Apr 2020 18:30:35 -0400

--On Wednesday, April 29, 2020 20:58 +0200 Patrik Fältström
<[email-masked]> wrote:

>> Since some hours have gone by without a response to your
>> message and I was in need of an excuse to delay getting to an
>> unpleasant task...
> 
>> Moreover, if I correctly understand what seems like
>> unnecessary convoluted text (in both versions) a BOM is
>> ignored in further processing if the character encoding
>> scheme is determined to be UTF-8 in 4.2(2) or 4.2(3) but not
>> ignored if charset="UTF-8" is present and the BOM occurs
>> anyway (something clearly allowed by RFC 3629).  That doesn't
>> appear to make sense.
> 
> I think I see the same thing as you, which is that even if the
> charset parameter states the encoding is UTF-8, if the data
> itself starts with a BOM, then the text is to be treated as
> UTF-16.

Actually, not what I noticed, and that reinforces my view that
even if one ignores the specific i18n issues, the text is just
too convoluted.  I had read the text as suggesting that, if the
charset was labeled (part 1), that the checks in part 2 and 3
just don't get made.  Whether that is smart or not -- whether
the protocol or application should make a sanity check on
whether something labeled as charset="UTF-8" is actually
conforming UTF-8 and/or whether there is a BOF present if it is
consistent with UTF-8 -- I don't really know except that it
should probably be clear.  But, you are right: if the text is
identified with charset-"UTF-8", then it really, really, better
be UTF-8 and, if there is a BOF that suggests it is something
else, then the spec should say something quite definite about
that.

> That is just so wrong.

I was more concerned about something else (with the
understanding that it isn't my only issue).  As I read the spec,
the plan is, approximately:
  (1) Apply step one, if there is no charset parameter present,
go to step 2
  (2) Apply step 2, i.e., go looking for a BOM fingerprint.  If
there isn't one present, go to stem 3.
  (3) Step 3: decide it is UTF-8.

Now the (or at least one) problem with that is that, absent a
rule that says "MUST use Unicode in some known encoding form",
there is no practical and reliable way to distinguish UTF-8 from
any part of ISO/IEC 8859 or, for that matter, any proprietary
code page.  UTF-16, with or without the BOM heuristics of your
choice is better, but not much better.
   
> I went to the ECMA spec and see they use UTF-16 all over the
> place, and have to bend over backwards to get things right. It
> feels like reading a BER encoding spec (again). :-)

BER is at least precise about what it is talking about.  This
spec isn't.
 
> The encoding must be what is actually labeled. If the encoding
> is UTF-16 (which it seems it often is), then it should be
> tagged as UTF-16, not UTF-8 with BOM.

Absolutely.  The easiest way out of both the problem you saw and
the one I saw is to get rid of steps 2 and 3 and insist on
labeling and conformance to what is labeled.  If that is
impractical for some reason, much more specificity is needed,
starting with a firm Unicode requirement.  And, unless they
intend to confine themselves to the BMP, they probably need to
talk about surrogates and their implications (or include such an
explanation by reference).   As to whether they "should" fix a
problem left over from 4329, they have changed the text in that
area and this is a Known Technical Omission or Defect.

best,
   john

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 29 Apr 2020 16:54:21 -0700

Don't have time to delve into this deeply, but am reading along the 
discussion here.

+1 on the text being too convoluted - I know my first reading was wrong.

+2 on seeing whether we can't use the revised spec to move to the 21st 
century in terms of character encoding.

(Their step 3 really attempts that, by making it the default / fallback 
when unlabeled / or without BOM signature, but it could be more explicit 
that you can only get legacy behavior if such is labeled).

+3 on stronger language about detecting mislabeled / inconsistent cases. 
Some conditions are clear cut enough to warrant rejection. Ideally, 
you'd always reject and not convert anything to U+FFFD. But, again, 
existing legacy may make that not possible.

+4 Aren't UTF-16 BOM versions are not legal UTF-8 byte sequences, or am 
I misremebering? If so, a UTF-16 BOM should override or flag as invalid 
any UTF-8 declaration.

A./

---

From: "John Levine" <[email-masked]>
Date: 29 Apr 2020 21:45:15 -0400

I looked at it and have different concerns.

A chunk of Ecmascript can either be a script or a module, which differ
syntactically, but that you can't tell apart by looking at them, so
you need external hints.  Sec 3 says that if the file extension is
.mjs, that is the hint that it is a module.  Sec 4.1 says that modules
are always in UTF-8.

Sec 4.2 has three steps which I believe are supposed to be done in order.

Step 1 says that if you know it's a module you know it's UTF-8, so stop.

Otherwise you sniff the data to see if it starts with a BOM, and if
so, that tells you the encoding and you otherwise ignore the BOM.

Step 2 says look at the charset, and believe if if you understand it.

Step 3 says if all else fails, assume it's UTF-8.

It looks like step 1 is saying that if the text starts with a BOM, you
ignore the declared charset and sniff the BOM instead, which sounds to
me like an ancient workaround that is perhaps no longer needed.

Given that they are deprecating all of the existing javascript media
types and reviving text/javascript which 4329 declared obsolete, this
might be a good time to say if you're going to use our lovely new
(old) media type, declare the correct character set so consumers can
believe it and stop doing byte sniffing kludges.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 29 Apr 2020 21:48:29 -0400

In article <[email-masked]> you write:
>I went to the ECMA spec and see they use UTF-16 all over the place, and have to bend over backwards to get things
>right. It feels like reading a BER encoding spec (again). :-)

They painted themselves into a corner with UCS a long time ago, but their current spec says in
section 10:

 The actual encodings used to store and interchange ECMAScript source
  text is not relevant to this specification.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 30 Apr 2020 01:50:41 -0000

In article <[email-masked]>,
Asmus Freytag  <[email-masked]> wrote:
>I misremebering?

You are correct.  That's the little table in sec 4.2.

> If so, a UTF-16 BOM should override or flag as invalid 
>any UTF-8 declaration.

That's what sec 4.2 sort of says.  But it doesn't really need to,
since somewhere else it says that text with invalid characters is invalid.

R's,
John
-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 29 Apr 2020 19:37:59 -0700

> me like an ancient workaround that is perhaps no longer needed.

If data are preceded by ("start with")  a BOM, you do want to strip it; 
you never want to keep it (and the chance that a legitimate text starts 
with a BOM that has an actual function in the text is perfectly negligible).

If you have data that carries a UTF-16 bom it cannot be UTF-8, no matter 
what the charset declaration says (FF can't be a byte in UTF-8).

Therefore, you always want to look for all three.

If  it's UTF-8 you confirm you have the good character set and remove it.

If it's one of the UTF-16 ones, you switch the charset to UTF-16 (or 
proper endianness) and remove it. (Or reject the input?)

> believe it and stop doing byte sniffing kludges.

There are two issues here. One centers around the fact that BOMs are 
"invisible". You can work hard at avoiding them, but they may be added 
by some "helpful" tool.

The other is that they serve as a useful consistency check when present.

You may write the spec to reject data with initial BOM, but then you'd 
still need to check for them. You definitely don't want to admit data 
without checking. Since you already need to check for them, you are 
better off giving clear instructions of how to make use of the fact that 
you've detected one.

A./

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 29 Apr 2020 19:43:36 -0700

> You are correct.  That's the little table in sec 4.2.
Good - I've confirmed that at the source just now (having another brief 
time slice).
> since somewhere else it says that text with invalid characters is invalid.

Well, if you want to make UTF-16 presented as UTF-8 invalid, I have 
fewer problems with that that with the assumption that it's possible to 
pretend data can't ever start with the (correct) BOM for the encoding.

So, one way or another you need precise instructions how to identify and 
ignore a leading BOM (even if it matches the character encoding).

You write somewhere: "if it's a module, you know it's utf-8". If that 
knowledge includes that the text can never have a BOM (e.g. because it's 
gone through some vetting) then, by all means, base on that for 
modules.  Not sure whether it would be possible to make that stick 
usefully for scripts (and you'd have to vet them anyway).

A./

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Apr 2020 00:09:32 -0400

Asmus, good to hear from you.

Inline below.

--On Wednesday, April 29, 2020 16:54 -0700 Asmus Freytag
<[email-masked]> wrote:

>...
> +4 Aren't UTF-16 BOM versions are not legal UTF-8 byte
> sequences, or am I misremebering? If so, a UTF-16 BOM should
> override or flag as invalid any UTF-8 declaration.

Your memory is correct.  The question is whether, if
charset="UTF-8" is declared and then the first two octets are a
UTF-16 BOM, whether to let it override and treat the data as
UTF-16  -or- to figure out a way to say "this is nonsense".

Patrik interprets the document as requiring the override
behavior.  He and I both prefer the latter, I think for at least
three reasons:

(i) I think, but am not sure, that some valid UTF-16 sequences
are valid UTF-8, even if the BOMs aren't.  If so, a file that is
labeled as UTF-8 but contains a UTF-16 BOM because some idiot
(not necessarily a human one) put it there at the beginning,
could really get into a world of trouble.  For example, any
valid UTF-16 sequence, especially one that contains no zero
octets, could be a valid 8859 sequence or valid sequences in
most code pages.  So starting from "it was labeled as UTF-8 but
isn't" doesn't make it UTF-16, BOM or no BOM.

(ii) As you know, there are a number of sequences that look like
UTF-8 but are not valid,  ones that Unicode now prohibits (my
vague memory is that it took a while for that prohibition to be
clearly established).  If something is labeled as UTF-8 (and/or
starts with a UTF-8 BOM) but contains one of those sequences,
the choice is whether to say "this is nonsense" in some way or
to try to guess at what was intended.  My assumption is that
those two nonsense cases are more or less equivalent and that,
if systems are sent ill-formed files, they are better off not
trying to guess... especially if the spec does not require
Unicode (which goes to your 21st Century comment, I think).

(iii) Mislabeled content is just bad news for all sorts of
reasons as I have been discovering when one of the mail systems
of a very large vendor are identifying message bodies as
'text/plain; charset="UTF-8"' and then sending UTF-16, BOM and
all.  Anything that can be done to discourage that sort of
nonsense --or, if not, to describe carefully want should be done
when it is encountered would be, IMO, A Good Thing.

The I-D, in its present form, does none of that.

stay safe and well,
   john

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Apr 2020 00:17:47 -0400

John and Asmus,

It seem to me that we are straying into doing the design (or
redesign) work that is the responsibility of the WG.  Explaining
the problem to the WG and suggesting fixes would be entirely
reasonable.   Trying to convince each other, IMO, a lot less so.

   john

--On Wednesday, April 29, 2020 19:37 -0700 Asmus Freytag
<[email-masked]> wrote:

>> It looks like step 1 is saying that if the text starts with a
>> BOM, you ignore the declared charset and sniff the BOM
>> instead, which sounds to me like an ancient workaround that
>> is perhaps no longer needed.
> 
> If data are preceded by ("start with")  a BOM, you do want to
> strip it; you never want to keep it (and the chance that a
> legitimate text starts with a BOM that has an actual function
> in the text is perfectly negligible).
> 
> If you have data that carries a UTF-16 bom it cannot be UTF-8,
> no matter what the charset declaration says (FF can't be a
> byte in UTF-8).
> 
> If it's one of the UTF-16 ones, you switch the charset to
> UTF-16 (or proper endianness) and remove it. (Or reject the
> input?)
> 
>> Given that they are deprecating all of the existing
>> javascript media types and reviving text/javascript which
>> 4329 declared obsolete, this might be a good time to say if
>> you're going to use our lovely new (old) media type, declare
>> the correct character set so consumers can believe it and
>> stop doing byte sniffing kludges.
> 
> There are two issues here. One centers around the fact that
> BOMs are "invisible". You can work hard at avoiding them, but
> they may be added by some "helpful" tool.
> 
> You may write the spec to reject data with initial BOM, but
> then you'd still need to check for them. You definitely don't
> want to admit data without checking. Since you already need to
> check for them, you are better off giving clear instructions
> of how to make use of the fact that you've detected one.
> 
> A./
>

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 29 Apr 2020 23:38:39 -0500

> reasonable.   Trying to convince each other, IMO, a lot less so.

I think that's pretty much right. Does anybody on the list think they 
can summarize this discussion of the problems and suggestions 
sufficiently for the WG to take some action? If I don't hear from 
anyone, I'll start accosting people privately.

pr

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 30 Apr 2020 07:02:53 +0200

John, I agree with your findings regarding MIME-types as well, but did not think that should be included in an I18N review :-) He he he...

I like MIME-types and think those should, as charset, be how one differentiate between different kinds of blobs. Using a file name regex match is something I have always hated.

   Patrik

On 30 Apr 2020, at 3:45, John Levine wrote:

> A chunk of Ecmascript can either be a script or a module, which differ syntactically, but that you can't tell apart by looking at them, so you need external hints.  Sec 3 says that if the file extension is ..mjs, that is the hint that it is a module.  Sec 4.1 says that modules are always in UTF-8.
>
> Otherwise you sniff the data to see if it starts with a BOM, and if so, that tells you the encoding and you otherwise ignore the BOM.
>
> It looks like step 1 is saying that if the text starts with a BOM, you ignore the declared charset and sniff the BOM instead, which sounds to me like an ancient workaround that is perhaps no longer needed.
>
> Given that they are deprecating all of the existing javascript media types and reviving text/javascript which 4329 declared obsolete, this might be a good time to say if you're going to use our lovely new (old) media type, declare the correct character set so consumers can believe it and stop doing byte sniffing kludges.
>
> R's,
> John
>
> -- 
> I18ndir mailing list
> [email-masked]

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 30 Apr 2020 07:13:17 +0200

On 30 Apr 2020, at 6:09, John C Klensin wrote:

> He and I both prefer the latter, I think for at least three reasons:

Yet another reason:

- multipart/alternative

You MUST be able to do the selection of which body part to consume based on the content-type and encoding etc.

   Patrik

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 29 Apr 2020 23:39:20 -0700

> The I-D, in its present form, does none of that.

Identifying data that contradict the declaration is the first step.

Tossing it with prejudice would probably be better than fixing it.

However, all of that requires BOM sniffing - I don't think we can
get away from being aware what a BOM is and skipping it
if present -- an "extra" BOM is merely
a redundancy whereas an "inconsistent" one is a contradiction
in terms -- and those are two fundamentally different things.

A./

PS: tossing malformed input for other reasons than BOM might
be cleaner from a security perspective. But not an expert.

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 29 Apr 2020 23:40:33 -0700

> reasonable.   Trying to convince each other, IMO, a lot less so.
Good point - let's treat this discussion not as leading up to a
solution but to help us understand what requirements we'd

like to suggest to the WG.

A./

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 29 Apr 2020 23:44:23 -0700

Pete,

not comfortable doing that, even if I had the time slice - there are 
some things about the matrix in which these i18n questions are embedded 
that I'm not as grounded in as would be a precondition for this task.

A./

---

From: "John R Levine" <[email-masked]>
Date: 30 Apr 2020 13:22:08 -0400

> the WG to take some action? If I don't hear from anyone, I'll start accosting 
> people privately.

Nooo, not the Private Accosting.

Summary:

The i18n directorate has some concerns about character set handling in
draft-ietf-dispatch-javascript-mjs-07.

We understand it to say that if a javascript MIME element does not have a 
name that ends with .mjs, a consumer ignores the declared charset and 
looks at the first few bytes of the content for a byte order mark (BOM.) 
If it finds one, it uses the charset implied by the BOM, which can be 
UTF-16BE, UTF-16LE, or UTF-8.  If there's no BOM, it uses the declared 
charset unless there isn't one, in which case it defaults to UTF-8.

We are unaware of any other MIME type that uses this sort of trick to work 
around mislabelled content, and are concerned that it leads to failures in 
general MIME code that doesn't handle this special case.  We also don't 
know how important the workaround is in practice, e.g., how many MIME 
producers still mislabel UTF-16 as UTF-8 or vice versa.

For better interoperation it could say something like producers MUST put 
the correct charset on any media (same as any other media type) and that 
consumers SHOULD use the declared charset but MAY do the BOM trick for 
backward compatibility in certain cases.

It also says the BOM must be removed from the decoded text.  That's 
confusing since ECMAscript treats a BOM as a space which would be harmless 
at the start of a block of code.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Pete Resnick" <[email-masked]>
Date: Thu, 30 Apr 2020 12:30:26 -0500

On 30 Apr 2020, at 12:22, John R Levine wrote:

>> the WG to take some action? If I don't hear from anyone, I'll start 
>> accosting people privately.
>

Obviously you have never experienced my full-out private accosting. :-)

> We understand it to say that if a javascript MIME element does not 
> have a name that ends with .mjs, a consumer ignores the declared 
> charset and looks at the first few bytes of the content for a byte 
> order mark (BOM.) If it finds one, it uses the charset implied by the 
> BOM, which can be UTF-16BE, UTF-16LE, or UTF-8.  If there's no BOM, it 
> uses the declared charset unless there isn't one, in which case it 
> defaults to UTF-8.
>
> We are unaware of any other MIME type that uses this sort of trick to 
> work around mislabelled content, and are concerned that it leads to 
> failures in general MIME code that doesn't handle this special case.  
> We also don't know how important the workaround is in practice, e.g., 
> how many MIME producers still mislabel UTF-16 as UTF-8 or vice versa.
>
> For better interoperation it could say something like producers MUST 
> put the correct charset on any media (same as any other media type) 
> and that consumers SHOULD use the declared charset but MAY do the BOM 
> trick for backward compatibility in certain cases.
>
> confusing since ECMAscript treats a BOM as a space which would be 
> harmless at the start of a block of code.

Thanks for taking up the pen John. If folks think something needs to be 
elaborated or added, or if you have some wordsmithing, do speak up.

I'll check with Barry whether he wants this on the official review form. 
If so, I'll assign the review to you in the datatracker. Otherwise, you 
can just email the dispatch list and sign it "John, stuckee for the 
directorate" or some such.

pr

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Apr 2020 15:47:40 -0400

Pete,

I do not have time (especially today or tomorrow) to engage with
this further or try to explain again, but parts of John's
summary are, I believe, inconsistent with how Patrik, Asmus, and
myself interpreted the document.  Specifically, 

(1) We all interpreted it as "check the 'charset' parameter
first".  The question was what was to be done next.

(2) At least two of us expressed concern about the use of a file
name suffix as a classifier.  Even if that is a carry-forward
from 4329, it is a major step back from the reason why both
email and the web adopted media type labeling and, if we are
counting deployment and running code, I think it is safe to
suggest that those two applications (or, if you prefer, sets of
applications) are somewhat more broadly deployed than these
so-called "scripting media types".

(3) Other than the statement "Source text is expected to be in
Unicode Normalization Form C", there is apparently no
requirement that the underlying CCS be Unicode.  The statement
"Implementations are required to support the UTF-8 character
encoding scheme" does not impose that requirement either, it
just makes UTF-8 support mandatory to implement... if it does
that, because normative requirements of that type are normally
not buried in Security Considerations sections.  This I-D claims
authority for insisting on NFC from the Security Considerations
Section of RFC 3629, but that Section does not discuss
normalization forms at all: instead, it discusses "the same
thing" issue and then says that the problems are "amenable to
solutions" based on normalization (not just NFC) and UAX15.
Even if that was good advice based on our best understanding in
2003, it certainly is neither necessary nor sufficient now.
More generally, the current conventional wisdom (or, if you
will, "best practice") is that, except where special
circumstances apply (as they do with IDNA) normalization should
occur if needed at processing (especially comparison) time, not
at storage or transmission time.  Yet, this document specifies
NFC and does so without explanation other than, as discussed
about, blaming RFC 3629 which is not only not guilty but is not
applicable to any charset other than UTF-8.

And, more broadly and probably more important:

(5) It is not an i18n issue specifically, but as a co-author of
RFC 6838 / BCP 13, I find it deeply troubling that this document
is put forward as a media type registration when what it appears
to do is to (i) allow the charset parameter but make it optional
and specifically provide that it is to be ignored if present for
Module Goals (Section 4.1) and then provide that, even for
Script Goals, it can be ignored if heuristics are applied the
suggest the charset (and encoding form) in use is something
else.  In particular, one can conform to the SHOULD in that
Section by specifying, for example, 'charset="IEO-8859-6"' and,
since the document doesn't specify what to do with Script goal
charsets one does not recognize, support for that drags in all
of the bidi and troublesome sequence issues associated with
Arabic without any of the support Unicode the documents
surrounding it provide.  

(6) The text and section organization that I described as
"convoluted" is very troublesome.  It was bad in 4329; the
changes in the current I-D certainly does not fix the problem
and may make it worse.  When four of us (Patrik, Asmus, John
Levine, and myself) each with considerable experience in reading
technical specifications, can read the same spec and come to
four different conclusions as to what it says to do, that is a
deep, fundamental, problem independent of any details. 

What is even more troublesome is that they could rather easily
dig themselves out of most (sadly, not all) of this mess by, as
Asmus more or less put it, joining the 21st century.    For
example, well-placed requirements that state clearly:

(i) For Module goal sources, the information MUST be in Unicode,
using encoding form UTF-8.  A charset parameter SHOULD NOT be
specified, if it is, its value MUST be "UTF-8".

(ii) For Script goal sources, a charset parameter MUST be
specified and MUST be one of "UTF-8", "UTF-16BE", or "UTF-16LE".
If it is omitted, the receiving system MAY dig itself into as
deep a whole as it prefers, possibly using BOM heuristics if
there is an explicit "MUST use Unicode" requirements for Script
goals.

and getting all requirements on the spec itself moved  out of
the Security Considerations section and stated as requirements,
without relying on requirements or recommendations of documents
like RFC 3629 that are somewhat outdated and/or don't say what
the I-D claims or implies that they say and/or are not
applicable to encoding forms (and non-Unicode CCSs) that the I-D
allows.

_Recommendations_

(a) We have said in multiple places, most recently in what is
now RFC 8753, that this i18n stuff requires a collaborative
effort by people whose expertise comes from a variety of
different perspectives.  The comments from Patrik, Asmus, John
Levine, and myself illustrate the reasons for that.  So either
no review should ever go out unless it either reflects multiple
sets of eyes and consensus (at least among those who were
willing to look) or it should bear a much stronger disclaimer
than is typical for "area review team" review assignments.  The
latter might say something like "while this review was assigned
to me by the i18n directorate, it represents my opinion only and
not consensus among the experts who make up that directorate,
even consensus that my summary of their discussion is accurate".

Consider what might happen in this case without one or the
other.  A review goes off that talks about the concerns of the
directorate and John's summary of those concerns ("We understand
it to say...").  The WG addresses those issues and the document
goes to IETF LC.  Some of Patrik, Asmus, and me (and maybe
others) respond to the IETF LC pointing out the issues raised in
our earlier notes and above, strongly suggesting that the WG
should have known about most of this, that they are depending on
documents that don't say what the WG claims they say and that
violate the letter and spirit of assorted RCPs.  We point out to
IANA that this document is not a proper Media Type registration
and that 4329 wasn't either.   The WG responds with dismay
because all of this is new to them.  And the ART ADs (whom I
believe are on this list) end up with egg on their faces as does
the whole directorate and its leadership.

(b) Let's respond to the WG with the issue I think those of us
who have looked at the document are all agreed about: it is
_really_ hard to figure out just what the document specifies and
hence to comment on it in an authoritative way.   If they are
assuming Unicode, they need to make that a requirement, not hope
the reader figures it out.   The notorious Section 4 may need to
be split up into separate subsections for Module goals and
Script goals or otherwise structured to be sure it is clear what
one is to do in each case and with and without charset
parameters.  And probably (less important for this iteration
since no one else mentioned it, but I predict an extra iteration
if it is not done), normative requirements on the spec must not
appear only in the Security Considerations section and they
better check the applicability of their references.   Only when
they fix enough of those things that we can all agree about what
the documents says are they going to get a review of substantive
i18n issues.

Disgustedly,
   john

--On Thursday, April 30, 2020 12:30 -0500 Pete Resnick
<[email-masked]> wrote:

>>> the WG to take some action? If I don't hear from anyone,
>>> I'll start  accosting people privately.
>> 
>> The i18n directorate has some concerns about character set
>> handling in draft-ietf-dispatch-javascript-mjs-07.
>> 
>> We understand it to say that if a javascript MIME element
>> does not  have a name that ends with .mjs, a consumer ignores
>> the declared  charset and looks at the first few bytes of the
>> content for a byte  order mark (BOM.) If it finds one, it
>> uses the charset implied by the  BOM, which can be UTF-16BE,
>> UTF-16LE, or UTF-8.  If there's no BOM, it  uses the declared
>> We are unaware of any other MIME type that uses this sort of
>> trick to  work around mislabelled content, and are concerned
>> that it leads to  failures in general MIME code that doesn't
>> handle this special case.   We also don't know how important
>> the workaround is in practice, e.g.,  how many MIME producers
>> still mislabel UTF-16 as UTF-8 or vice versa.
>> 
>> For better interoperation it could say something like
>> producers MUST  put the correct charset on any media (same as
>> any other media type)  and that consumers SHOULD use the
>> declared charset but MAY do the BOM  trick for backward
>> compatibility in certain cases.
>> 
>> It also says the BOM must be removed from the decoded text.
>> That's  confusing since ECMAscript treats a BOM as a space
>> which would be  harmless at the start of a block of code.
> 
> Thanks for taking up the pen John. If folks think something
> needs to be elaborated or added, or if you have some
> wordsmithing, do speak up.
> 
> I'll check with Barry whether he wants this on the official
> review form. If so, I'll assign the review to you in the
> datatracker. Otherwise, you can just email the dispatch list
> and sign it "John, stuckee for the directorate" or some such.
> 
> pr
> --

---

From: "John R Levine" <[email-masked]>
Date: 30 Apr 2020 16:11:45 -0400

> first".  The question was what was to be done next.

Not really.  It says:

    The charset parameter is only used when processing a Script goal
    source; Module goal sources MUST always be processed as UTF-8.

i.e., if it ends with .mjs it's UTF-8

> name suffix as a classifier. ...

> requirement that the underlying CCS be Unicode. ...

If there were some politically viable way to say it, I'd say to throw the 
entire draft away other than the IANA registrations and replace it with 
something written by someone who understands MIME and isn't totally 
embedded in the html/javascript/UCS world.

Regards,

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 30 Apr 2020 13:33:09 -0700

I'm in broad stroke agreement with John K as to the suggested
recommendations to the WG. (found under (b)) and the other
feedback found mostly in item (6).

(Unfortunately these are buried in the meta discussion of how we
got there which we should spare the WG).

As an aside, he's correct that in this case all four of us have come
from slightly different directions and with positive effect.

As for the other John (L) I sympathize with his comment on (3) - but
it's out of my direct expertise.

His comment on (1) supports the recommendation to separate
the specifications for Script and Modules.

A./

On 4/30/2020 1:11 PM, John R Levine wrote:

> 1) We all interpreted it as "check the 'charset' parameter
> If there were some politically viable way to say it, I'd say to throw 
> the entire draft away other than the IANA registrations and replace it 
> with something written by someone who understands MIME and isn't 
> totally embedded in the html/javascript/UCS world.

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 30 Apr 2020 13:40:48 -0700

Just saw this, and think these items should be raised, but I also like 
some of the other John's feedback on document structure. While it's not 
an i18n issue per se, it really impeded the review. I also think the 
other John and some more direct suggestions on how to write the language 
(and there's the point about NFC and what it implies about Unicode - 
although legacy data almost always is in NFC once converted, per design 
of NFC).

A./

> harmless at the start of a block of code.

Good point. In that case, if BOM can happily survive then that removes 
the implicit requirement for BOM sniffing altogether, making a stronger 
case for disallowing it to override anything.

If specific legacy support is required, it should be called out as such.

A./

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Apr 2020 16:45:32 -0400

--On Thursday, April 30, 2020 16:11 -0400 John R Levine
<[email-masked]> wrote:

> If there were some politically viable way to say it, I'd say
> to throw the entire draft away other than the IANA
> registrations and replace it with something written by someone
> who understands MIME and isn't totally embedded in the
> html/javascript/UCS world.

About that, we are definitely in agreement.

Out of time today to response to your other points.

  john

---

From: "John Levine" <[email-masked]>
Date: 30 Apr 2020 17:40:50 -0400

In article <477C5A18357719590D6336D9@PSB> you write:
>just makes UTF-8 support mandatory to implement...

That's just more wrongness.  The ECMAscript spec says in the lexical
grammar that a source character can be "any Unicode code point" and
otherwise has a whole lot of discussion about how Unicode ie to be
interpreted.  Whether or not you think they've done a good job of it,
it's clear that the Unicode code point management happens at the
ECMAscript level, and this draft should not make guesses about
normalization.

Given how tightly ECMAscript is bound to Unicode, it would be
reasonable to say that the only interoperable charsets are UTF-8,
UTF-16BE, and UTF-16LE.  This is not strictly backward compatible, but
I would be pretty surprised if it broke anything that wasn't already
hopelessly broken.

Maybe I should tell Ned, the designated IANA charset expert, to take a
look and see what happens.

R's,
John

---

From: "Pete Resnick" <[email-masked]>
Date: Thu, 30 Apr 2020 18:24:06 -0500

[Secretary/chair hat off. Technical commenter hat on.]

> first".  The question was what was to be done next.

No, 4329 says check the charset first. As Levine pointed out, the draft 
says:

1a First check if it's a Module (an out-of-band determination); if so, 
it's UTF-8
1b Second check for UTF-8 or UTF-16 BOM; if you find one of them, trust 
it
2  Third check the charset; if you understand it, use it.
3  If none of the above, assume UTF-8.

> from 4329...

First, the draft registers .mjs, but the only thing it says about it is 
that environments that use file extensions will treat .mjs to mean it is 
a module. It has no normative text claiming that in a MIME contexts it 
should be used as a classifier. Second, .mjs is not a carryover from 
4329. It is true that .js is a carryover, but the only place it appears 
is in the MIME registration for the file extension associated with the 
MIME type. I don't see that as problematic.

> requirement that the underlying CCS be Unicode.

See Levine's last message on this.

> else.

I suspect there is some HTTP behavior assumption in here. But I think 
your first sentence is correct; that should probably be labeled as "Not 
an i18n issue, but this is Not Good ™."

> deep, fundamental, problem independent of any details.

Yep. Probably treated like (5).

> specified, if it is, its value MUST be "UTF-8".

Yes. It almost says that now, but that would be crystal clear.

> there is an explicit "MUST use Unicode" requirements for Script

As Asmus said at some point, checking for a BOM at the front of the data 
is probably a good idea to confirm that the charset is correct, but 
other than that, I agree.

> applicable to encoding forms (and non-Unicode CCSs) that the I-D

You're just referring to the two paragraphs that mention 3629, correct? 
I didn't see anything else in section 5 that looked like a requirement.

[Tech reviewer hat off / secretary/chair hat back on]

> even consensus that my summary of their discussion is accurate".

While I agree with the concern, I think this pushes in the wrong 
"responsibility" direction. I would hope that if I had poorly triaged 
this as "just a boring little i18n review" and not mentioned it on the 
list but simply assigned it to Levine, he would have called me out and 
said, "Whoa, there's a lot going on here. We should have a list 
discussion about this." If both he and I screwed up and didn't think a 
conversation was worth having and you all had to yell out at IETF LC, 
that would be a (painful) learning experience for John and I, and a 
bummer for the WG in question, I think that is the correct way to grow. 
If we need to depend on all of the experts to weigh in on every 
document, that's simply not sustainable. We have to individually be able 
to make calls and know (or learn) when we have to bring out the troops. 
Yes, that means there will be errors that will lead to well-egged faces 
every so often. I know of no other way to make things better in the long 
run.

pr

---

From: "John R Levine" <[email-masked]>
Date: 30 Apr 2020 20:52:48 -0400

> First, the draft registers .mjs, but the only thing it says about it is that 
> environments that use file extensions will treat .mjs to mean it is a module. 
> It has no normative text claiming that in a MIME contexts it should be used 
> as a classifier. Second, .mjs is not a carryover from 4329. It is true that 
> .js is a carryover, but the only place it appears is in the MIME registration 
> for the file extension associated with the MIME type. I don't see that as 
> problematic.

The whole distinction between a script and a module is a crock. 
Attempting to decode their strange language, it appears that a js script 
and a js module have minor syntactic differences (that's the "goal" bit) 
which presumably matter to the application consuming the text.  But that's 
no business of MIME handling.  I could have a MIME part tagged text/html 
that was in fact a Fortran program, and while it would make an application 
looking for HTML unhappy, it's a perfectly good MIME.

>> and specifically provide that it is to be ignored if present ...

> I suspect there is some HTTP behavior assumption in here. But I think your 
> first sentence is correct; that should probably be labeled as "Not an i18n 
> issue, but this is Not Good ™."

I'm wondering if the BOM hack is an ancient workaround for bugs in some 
server in the 1990s that gave you UTF-16 regardless of what you asked for, 
perhaps early versions of IIS. I hope no living web server is still that 
broken.

>> Asmus more or less put it, joining the 21st century.

Yup.

>> there is an explicit "MUST use Unicode" requirements for Script

Once again, I think that importing the semantic difference between js 
modules and scripts into MIME is a mistake.  It's OK to have .mjs as a 
hint about what it's intended to be, but not to make that affect the MIME 
processing.

Regards,

---

From: Adam Roach <[email-masked]>
Date: Thu, 30 Apr 2020 20:27:47 -0500

On 4/30/2020 7:52 PM, John R Levine wrote:
> I'm wondering if the BOM hack is an ancient workaround for bugs in 
> some server in the 1990s that gave you UTF-16 regardless of what you 
> asked for, perhaps early versions of IIS. I hope no living web server 
> is still that broken. 

For what it's worth, the vast, /vast/ majority of these kinds of 
workarounds are not due to bugs in servers.

They're due to /misconfigured /servers that early, buggy, and extremely 
popular OS-integrated clients (which broadly ignored content 
information, happily rendering, e.g., "text/plain" as a JPEG image in 
certain contexts) didn't complain about. To remain compatible with these 
servers, subsequent browsers have had to perpetuate these bugs; and 
because of this permissiveness, administrators in 2020 continue to 
persistently misconfigure servers: they don't see any errors from doing 
so. Fixing these problems would require a flag day -- coordinated with 
1.7 billion websites -- after which browsers wouldn't put up with this 
kind of content mislabeling. Even though those specific browsers that 
led us to this swamp have been EOL for nearly two decades, that kind of 
massive coordination still remains infeasible.

Nearly all of the MIME-specific issues cited on this list so far have 
this as their explanation. It's an absolute cornucopia of ugliness, and 
everyone involved with the situation is painfully aware of how bad it is.

Permissive MIME handling in early web browsers remains a vivid 
demonstration of how aggressive application of Postel's Maxim at scale 
in widely-used consumer products necessarily ossifies an ever-increasing 
set of historical product bugs into protocols indefinitely. Although not 
cited directly, this specific mess was instrumental in the development 
of the advice in draft-iab-protocol-maintenance.

In terms of what draft-ietf-dispatch-javascript-mjs should say, there 
are two disjoint paths we can decide to follow. The IETF can either 
document a fantasy world of platonic solids that we all wish existed, 
which would have virtually no practical application to implementors; or 
the IETF can document the world as it exists today, warts and all, with 
decades of baked-in unfortunate implementation decisions, and enable 
actual interoperability.

Given this history, the two goals of producing a useful document and an 
ideologically pure document are in diametric opposition. I would hope we 
ultimately decide to do the former rather than the latter, and encourage 
the directorate to steer its feedback to the working group in a way that 
helps achieve this goal.

/a

---

From: "John R Levine" <[email-masked]>
Date: 30 Apr 2020 21:58:13 -0400

> In terms of what draft-ietf-dispatch-javascript-mjs should say, there are two 
> disjoint paths we can decide to follow. The IETF can either document a 
> fantasy world of platonic solids that we all wish existed, which would have 
> virtually no practical application to implementors; or the IETF can document 
> the world as it exists today, warts and all, with decades of baked-in 
> unfortunate implementation decisions, and enable actual interoperability.

I don't think we're asking for ideologically pure documents, but this 
draft could use a lot of rewriting to say what it needs to say more 
clearly.  The "goal" stuff is particularly obtuse, and I say this as a 
guy who's written books on parsing tools.

If it is indeed the case that a lot of javascript mislabels the charset 
and you have to sniff the BOMs, so be it.  But surely we can limit the 
scope of awfulness, e.g., is anything mislabeled as other than UTF-8 or 
UTF-16?

Regards,

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Apr 2020 22:54:19 -0400

--On Thursday, April 30, 2020 21:58 -0400 John R Levine
<[email-masked]> wrote:

>> In terms of what draft-ietf-dispatch-javascript-mjs should
>> say, there are two  disjoint paths we can decide to follow.
>> The IETF can either document a  fantasy world of platonic
>> solids that we all wish existed, which would have  virtually
>> no practical application to implementors; or the IETF can
>> document  the world as it exists today, warts and all, with
>> decades of baked-in  unfortunate implementation decisions,
>> and enable actual interoperability.
> 
> I don't think we're asking for ideologically pure documents,
> but this draft could use a lot of rewriting to say what it
> needs to say more clearly.  The "goal" stuff is particularly
> obtuse, and I say this as a guy who's written books on parsing
> tools.
> 
> If it is indeed the case that a lot of javascript mislabels
> the charset and you have to sniff the BOMs, so be it.  But
> surely we can limit the scope of awfulness, e.g., is anything
> mislabeled as other than UTF-8 or UTF-16?

Although John and I still disagree on some details, +1.

And, as a small step toward ideological purity, if, in practice,
no one pays attention to what appears in the charset parameter
value, then why not either drop that parameter, say explicitly
that it is likely to be ignored, or see if a registration of
'charset="IDontHaveAClue"' would make it past Ned and Martin.
Not sure I'm serious about that, but you get the point.

Adam, the gulf between ideological purity and matching reality
isn't, IMO, quite as large as you portray it.   A
standards-track document that says "this is really how it should
be done but, if you, or those sending stuff to you, are going to
ignore that, this is what you should do to avoid nasty
interoperability problems and traps" should make us all a bit
uncomfortable, but is still lots better than convoluted text
that ends up amounting to "Do what thou wilt" (and that is not a
comment about this I-D in particular).

And Postel was (and presumably still would be if he were around)
horrified by the idea that the robustness principle would be
interpreted as "as the sender, I get to do whatever I
<obscenity> feel like and it is up to the receiver to compensate
and Do What I Meant".

best,
    john