Subject: [I18ndir] I18ndir early review of draft-ietf-dispatch-javascript-mjs-07

From: John Levine via Datatracker <[email-masked]>
Date: Fri, 08 May 2020 13:17:25 -0700

Reviewer: John Levine
Review result: Ready with Issues

This is my take on issues with this document mostly from my personal
review but also after some discussion we've had on the i18ndir list.

Some parts of this draft are quite hard to follow, so I'm giving my
understanding of the parts I'm commenting on in case I got them wrong.
I realize that a lot of this is unchanged from 4329, which we should
have reviewed more carefully 15 years ago.

Section 4 on Encoding: I believe it says that the preferred encoding
for all javascript is UTF-8, but some sources use other encodings and
sometimes mislabel them.  So for anything that you don't know is a
module, you have to sniff the contents to see if starts with a BOM,
and if so, use the BOM's encoding and delete the BOM.  If the BOM uses
an encoding the consumer doesn't support, fail.  If there's no BOM,
use the declared character set, or if it's one the consumer doesn't
understand, treat it as UTF-8 anyway.

Step 1 says "The longest matching octet sequence determines the encoding." 
which I don't understand, since none of the encodings overlap.  Does that 
mean it should interpret a partial BOM, e.g., EF BB 20 for UTF-8? Also, 
why is the BOM deleted?  ECMAscript says a BOM is a space so it should be 
harmless.

While I understand that there is a lot of history here, I'm wondering if 
the range mislabeling is really as extreme as this implies.  Is there, 
say, text labelled Shift-JIS which is really UTF-8 or UTF-16? If the 
mislabelled stuff is consistently mislabelled as one of UTF-8/16/16BE/16LE
perhaps it could say to try the BOM trick on those encodings and fail otherwise.

I don't understand step 3, "The character encoding scheme is
determined to be UTF-8."  How can it be determined to be UTF-8 other
than by steps 1 and 2?  Or is it saying that if the declared charset
is one the consumer doesn't understand such as KOI8-U, assume it's
UTF-8 anyway.

I'd suggest rewriting the section to make it clearer that if it's not
a module, you look for a BOM, use its encoding if you find one, and (I
think) otherwise use the declared encoding.

Section 4.3 on error handling: I think it says that if there's a byte
sequence that isn't a valid code point in the current encoding, it can
fail or it can turn the bytes into Unicode replacement characters, but
MUST NOT try anything else.  I agree with this advice but again it
could be clearer.

Section 3 on Modules: I believe it says that JS scripts and modules have 
different syntax but you can't easily tell them apart by inspection.  
(The term "goal" is familiar since I used to write books about compiler 
tools, and I realize it's what the ECMAscript spec uses, but it's 
confusing if you're not a programming language expert.  How about just 
saying that scripts and modules have different syntax?)

Hence some software uses a .mjs filename as a hint that something is a
module.  Again I realize that there is a bunch of existing code but
this is not great MIME practice.  If the difference matters, it's
worth providing a new MIME type such as text/jsmodule, which could
have consistently accurate content encodings.  It would coexist with
all of the other old MIME types and the .mjs hints. Since this draft
deprecates a bunch of existing types and de-deprecates another, this
seems as good a time as any to do it.

I also wonder whether it's worth making a distinction in MIME
processing between modules and scripts.  Would there be any harm in
saying to sniff everything for a BOM?  If a .mjs file turns out to
have a UTF-16 BOM, it's wrong, but is it likely to be anything other
than a javascript module in UTF-16?

---

From: Barry Leiba <[email-masked]>
Date: Fri, 8 May 2020 19:18:48 -0400

Thanks, John, for taking the time on this.  It’s a really helpful review,
and appreciated.

Barry

---

From: Myles Borins <[email-masked]>
Date: Fri, 8 May 2020 19:24:46 -0400

Regarding the mime type "text/javascript"

The HTML Specification is rather explicit
<https://html.spec.whatwg.org/multipage/scripting.html#scriptingLanguages>

Servers should use text/javascript for JavaScript resources. Servers should
not use other JavaScript MIME types for JavaScript resources, and must not
use non-JavaScript MIME types.

Browsers validate that source text being loaded is delivered with the
"text/javascript" mime type and fails if it is any other mime is used.
There is no in-band way to determine the goal of a Module, and Node.js
needed a signal to be able to determine how to interpret source text. This
was the motivation behind creating .mjs.

The problem this created was that webservers, not knowing the extension,
were not serving the test/javascript mimetype and browsers were then in
turn failing to load them as modules. This was a pretty awful developer
experience, especially for folks experimenting with a newer technology they
didn't entirely understand to begin with.

With this context in mind I don't think it is really worth going and making
a distinct mimetype for modules, at least not one that would be
strictly enforced, at the very least this probably shouldn't be
pursued without broader socialization and buy-in from browser vendors.

My gut is that we are best to use this proposal to document the existing
status quo... what people are using and is working, rather than introduce
new constraints that do not have any existing implementation or adoption.

I'll let folks more familiar with your other notes field those questions.

On Fri, May 8, 2020 at 7:19 PM Barry Leiba <[email-masked]> wrote:

> On Fri, May 8, 2020 at 4:17 PM John Levine via Datatracker <
> [email-masked]> wrote:
>
>> Reviewer: John Levine

---

From: "John R Levine" <[email-masked]>
Date: 8 May 2020 19:31:15 -0400

> new constraints that do not have any existing implementation or adoption.

I completely agree that for something like this which is widely 
implemented, the spec should describe what actually works.  But I was 
wondering if you could narrow down the range of what you describe so 
future implementors don't waste time worrying about screwups that aren't 
an issue in practice.

I'm sorry to hear that people are using filename extensions to tell what's 
in a file.  I'd hoped that all of the badness from old versions of Windows 
doing that might have been a hint.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Matthew A. Miller" <[email-masked]>
Date: Fri, 8 May 2020 18:00:00 -0600

Thank you very much for the review, John.  This is very helpful, and
will work on changes to resolve the concerns.

Regarding the encoding:

UTF-8 is strongly encouraged, and required for a source to be a module.
 Preferring BOM sniffing over a charset declaration is the result of
general purpose browsers having to deal with a myriad of misconfigured
servers; this being the defense mechanism that permitted the best
interoperability.

The part about "longest matching octet sequence" is a missed remnant
from the original document which included UTF-32BE/LE entries.  As
UTF-32BE/LE is no longer found in the wild*, those entries were removed
but I missed cleaning up that sentence.

Implementations today do ignore the BOM once the character encoding is
worked out, so this was portion from Section 4.2 was kept to reflect
that.  I'm not sure what changes to make here regarding that, although
the ending "to source text" is another missed remnant from my editing
that will be removed.

Section 4.2 still includes step #3 to deal with the (in practice quite
common) case of a missing BOM and the media type missing a charset
parameter.  There are also too many servers that set this to
"ISO-8859-1" without otherwise examining the sources being served.
We'll make it clearer this is a default/fallback case.

- m&m

Matthew A. Miller

* Web browsers haven't accepted UTF-32 encodings of JavaScript for quite
a long while
On 20/05/08 17:24, Myles Borins wrote:
>     Servers should use text/javascript for JavaScript resources. Servers
>     should not use other JavaScript MIME types for JavaScript resources,
>     and must not use non-JavaScript MIME types.
> 
> 
> needed a signal to be able to determine how to interpret source text.
> This was the motivation behind creating .mjs.
> 
> experience, especially for folks experimenting with a newer technology
> they didn't entirely understand to begin with.
> 
> With this context in mind I don't think it is really worth going and
> making a distinct mimetype for modules, at least not one that would be
> status quo... what people are using and is working, rather than
> introduce new constraints that do not have any existing implementation
> or adoption.
> 
> On Fri, May 8, 2020 at 7:19 PM Barry Leiba <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Thanks, John, for taking the time on this.  It’s a really helpful
>     review, and appreciated.
> 
>     Barry
> 
>     On Fri, May 8, 2020 at 4:17 PM John Levine via Datatracker
>     <[email-masked] <mailto:[email-masked]>> wrote:
> 
>         Reviewer: John Levine

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 00:01:42 -0400

Hi.

Given that I felt a bit attacked for stepping in earlier
(although I'm obviously pleased that John incorporated some of
my comments in his review), a few comments that can be passed
along or not as people prefer....

--On Friday, May 8, 2020 18:00 -0600 "Matthew A. Miller"
<[email-masked]> wrote:

> Thank you very much for the review, John.  This is very
> helpful, and will work on changes to resolve the concerns.
> 
> Regarding the encoding:
> 
> UTF-8 is strongly encouraged, and required for a source to be
> a module.  Preferring BOM sniffing over a charset declaration
> is the result of general purpose browsers having to deal with
> a myriad of misconfigured servers; this being the defense
> mechanism that permitted the best interoperability.
> 
> The part about "longest matching octet sequence" is a missed
> remnant from the original document which included UTF-32BE/LE
> entries.  As UTF-32BE/LE is no longer found in the wild*,
> those entries were removed but I missed cleaning up that
> sentence.

An observation with the understanding that I'm not advocating
for UTF-32, in either ordering, and especially not on the wire.
First, until Unicode deprecates it (which, AFAIK, they haven't
done), "no longer found in the wild" is a rather bold statement
(analogies to the recent IETF list discussion about POP2 come to
mind) and rather different from "no current or recent version of
any browser with which the authors are familiar accepts and
supports UTF-32 encodings" (which is what "Web browsers haven't
accepted UTF-32 encodings of JavaScript for quite a long while"
almost certainly means).   I have no information about
Javascript specifically, but I'm aware of several systems that
still use UTF-32 for internal processing because, if one does
not have storage space limitations, there are significant
advantages of having exactly the same number of octets (four)
for each code point regardless of where it falls in the Unicode
tables and hence without the need to pack and unpack UTF-8 or be
aware of and process surrogates.   

More important, my understanding of the general practices in the
IETF for many years is that we don't deprecate features that may
once have been accepted and used, even if not "for quite a long
while") by simply removing them and assuming that readers will
observe their absence and make the appropriate inferences,
whatever those might be.  I see nothing in either the body of
the document nor in Appendix B that makes it explicit that
support for UTF-32 has been dropped.   I think such a comment
should be added and explained, even if the explanation is that
no one significant is accepting it any more and it is therefore
NOT RECOMMENDED and being dropped from the spec.

> Implementations today do ignore the BOM once the character
> encoding is worked out, so this was portion from Section 4.2
> was kept to reflect that.  I'm not sure what changes to make
> here regarding that, although the ending "to source text" is
> another missed remnant from my editing that will be removed.
> 
> Section 4.2 still includes step #3 to deal with the (in
> practice quite common) case of a missing BOM and the media
> type missing a charset parameter.  There are also too many
> servers that set this to "ISO-8859-1" without otherwise
> examining the sources being served. We'll make it clearer this
> is a default/fallback case.

Noting (again) that, absent a BOM (or even with one) UTF-8
cannot be reliably distinguished from ISO-8859-X (for any
registered or unregistered value of X), I don't know what the
default/fallback case actually is or, more generally, what  the
above paragraph means.  Unless I have misunderstood something
important, the reality is that, if there is anywhere on the
Internet that a web browser or server (including decades-old
embedded servers) treat ISO/IEC 8859-1 as either the default or
legitimate, then there are two possibilities: accurate labeling
of the charset in use or use of heuristics that, by their nature
and the nature of possible CCSs (not just encoding schemes), may
fail.   That calls for at least a health warning in the
document, not proceeding as if the heuristics are foolproof.

best,
   john

p.s. Noting a recent unpleasant discussion on the IETF list and
that the review is being handled as an individual one rather
than as a conclusion by the directorate, if any ideas or text
are taken from this or my prior comments on the draft, I expect
that, consistent with BCP 78, those comments will be attributed
so they can be properly acknowledged in the I-D.

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 8 May 2020 21:21:38 -0700

On 5/8/2020 9:01 PM, John C Klensin wrote:
> almost certainly means).

There's no reason for Unicode to deprecate UTF-32.

Nobody really advocates it as an encoding form for transmission, but 
Unicode's definition of UTFs are not file formats. That's an essential 
misunderstanding. Anytime a per-character API accepts or returns 32-bit 
value that cover the entire code space, that format is effectively 
UTF-32. Deprecating this would force all those APIs to work on strings - 
not necessarily an improvement.

Whether it's advisable to have an internal string format that's UTF-32 
(rather than the more widely implemented UTF-16) is something that 
Unicode correctly is agnostic about.

It's not Unicode, but things like W3C and IETF standards that are 
supposed to define transmission / file formats. The Unicode-defined 
range of UTFs just represents the building blocks.

>   I have no information about

That's independent of the definition of a programming language. How you 
implement such a language on top of  a "system" that works natively in 
UTF-32 is a matter for the implementer of that programming language. 
Javascript has all the rights to define whether UTF-32 is or is not an 
acceptable source code format (and to deprecate it for that purpose).
> NOT RECOMMENDED and being dropped from the spec.
If that understanding is correct, I agree with the conclusion.

> document, not proceeding as if the heuristics are foolproof.

A move to the 21st century would imply making  UTF-8 the default (in the 
absence of another declaration). If that can't be done because of 
pervasive legacy, then that needs to be stated and motivated.

If a BOM is present and disagrees with the declaration, that should be 
an error condition. Hopefully, there isn't pervasive legacy to the 
contrary.

It looks like in source code a proper BOM is always treated as whitspace 
and ignored, if so, that could be mentioned as a reason why it's not 
necessary to actively skip it.

Another thing: in our discussion we mentioned the need to take the 
description of the charset handling out of the security consideration 
section. I don't see that mentioned.

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 09 May 2020 08:03:43 +0200

On 9 May 2020, at 1:31, John R Levine wrote:

> I completely agree that for something like this which is widely implemented, the spec should describe what actually works.  But I was wondering if you could narrow down the range of what you describe so future implementors don't waste time worrying about screwups that aren't an issue in practice.

It is also the case that the IETF document can explain exactly this, that the use of filename extensions is discouraged (for a multitude of reasons), and still define a new MIME-type which is the recommended use case. Collaboration with W3C is of course essential.

> I'm sorry to hear that people are using filename extensions to tell what's in a file.  I'd hoped that all of the badness from old versions of Windows doing that might have been a hint.

This is something that just must go away, and I think a big note in non-friendly letters must be added in the security considerations section is needed that explain the situation and why the current practice is not really what we want in the wild.

I.e. yes, while still documenting "what works", that should not be the goal. It will lead to even more problems getting secure mechanisms in browsers deployed.

   Patrik

---

From: Martin J. Dürst <[email-masked]>
Date: Sat, 9 May 2020 16:36:31 +0900

On 09/05/2020 13:01, John C Klensin wrote:

> registered or unregistered value of X),

Sorry, not true. Please see 
https://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, page 20.

> I don't know what the
> fail.

For people not familiar with the details, the following should be 
pointed out clearly: The heuristic of detecting UTF-8 as UTF-8 based on 
its very specific bit patterns is an extremely strong heuristic. There 
are cases of specific character combinations in specific encodings that 
can look like UTF-8 bit combinations, but that is extremely rare and 
gets rarer and rarer the longer the data gets. This is already true if 
one looks just at the available characters/bit patterns, but even more 
so if one looks at the languages that use the respective encodings and 
their letter patterns.

Regards,   Martin.

> That calls for at least a health warning in the

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 12:33:25 -0400

> I see nothing in either the body of the document nor in Appendix B that 
> makes it explicit that support for UTF-32 has been dropped.  I think 
> such a comment should be added and explained, even if the explanation is 
> that no one significant is accepting it any more and it is therefore NOT 
> RECOMMENDED and being dropped from the spec.

I agree that a comment saying that UTF-32 is dead as an interchange format 
would be reasonable.

> registered or unregistered value of X), ...

I'm wondering how much this matters in practice.  I see a lot of 
javascript libraries entirely in ASCII, since non-ASCII characters only 
appear in variable names and in string literals and comments.

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 12:36:20 -0400

> I.e. yes, while still documenting "what works", that should not be the goal. It will lead to even more problems getting secure mechanisms in browsers deployed.

I agree with Patrik, this reinvents a security hole from the floppy disk 
era and is not something we should let pass without at least making it 
clear that we believe it is a significant mistake.

It occurs to me that http clients can send an Accept header, so how about 
saying that modules SHOULD be text/jsmodule, but MAY be text/javascript 
for backward compatibiltiy with clients that don't handle text/jsmodule?

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 12:40:19 -0400

> https://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, page 20.

Well, any -X that is likely to show up on a dusty IIS server.

I don't think we're doing anyone any favors by encouraging yet more 
kludges to work around mislabelled MIME types.  If people want to 
interoperate, label the text correctly.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 12:48:59 -0400

Martin,

I stand corrected, but the point I was trying to get at remains
unchanged.

Inline.

--On Saturday, May 9, 2020 16:36 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>>> examining the sources being served. We'll make it clearer
>>> this is a default/fallback case.
>> 

Very interesting and useful analysis, especially if one starts a
page or two earlier.  And I understand it, what you are saying
is that there are some 8859 parts (and some other code pages and
CCSs) in which some of the bytes are not populated and hence
cannot be confused with some strings of UTF-8. 

The second sentence in the paragraph in the middle of page 19 is
particularly useful, IMO, in understanding what you are
suggesting:

	"To be able to assess the actual potential of confusion,
	it is important that all possible octet combinations are
	tested, within the limits of computational power and the
	repetitions that result in longer strings."

Let me come back to that but, in addition, some other
possibilities can be eliminated by careful script or
language-based analysis.  Especially if one had a reasonable
hypothesis about the language, one could even go a bit further,
assuming the text were in UTF-8 and some other candidate
encoding and CCS and then applying natural language analysis to
both.  The odds that both would make sense in that language
(and, for this case, conform to Javascript syntax rules) are
vanishingly small; one or the other would approximate gibberish
or worse.  However, even tests like that can get us only to
"could not plausibly be anything but UTF-8" as distinct from an
absolute conclusion because, I suppose, text cold be a digital
signature or encrypted, in which one would not would expect
those strings to be anything but gibberish from a normal/natural
language standpoint.   Since I have not yet read the entire
paper, perhaps you make those points elsewhere.

So, what I would (or at least should) have said if I had your
level of understanding, or even had read this paper earlier,
would have been closer to:

	"Noting (again) that, absent a BOM (or even with one)
	UTF-8 cannot be reliably distinguished from most
	ISO-8859-X (for most registered or unregistered values
	of X) and many other 8bit CCSs without the use of
	careful testing on a per-alternative basis or an
	analysis of the text that was presumably encoded.
	Exceptions, which are fairly easily detected as not
	being UTF-8 if sufficient text is available, include any
	8bit CCS that does not assign values to any of the
	octets with hex values between x'80' and x'9F' 
  inclusive."
	
There are two problems with that revised explanation.  One is
that, in practice, most people reading the I-D would not read
it.  Their eyes would glaze over.  Possibly more important, the
odds that any browser (or similar application) would try to
perform any significant fraction of the needed tests (noting
your comment about testing "all possible octet combinations" as
a starting point) in real time is, IMO, just about zero.  So,
unless there is sufficient language tag or other identifying
information present to vastly narrow the alternative CCSs to be
considered as alternate possibilities (and maybe not then), the
substance of my original text does not change significantly.

Given the history of the use of non-Unicode CCSs on the web, a
far less sweeping statement than the one I made would be more
likely and make a similar point almost equally well.  That
statement would suggest that there are, in all likelihood, only
four possibilities (ignoring the wildness, or lack thereof, of
UTF-32):
    ISO/IEC 8859-1, UTF-8, UTF-16BE, and UTF-16LE

However, even with that restriction, UTF-8 cannot reliably be
distinguished from 8859-1 unless it is possible to apply an
additional constraint that C1 code points cannot occur in
Javascript, or at least any Javascript of interest, even in
embedded strings.  And, again, making the scans and tests needed
to make that distinction might involve a higher resource cost
than browsers (etc.) would find acceptable.

Another difficulty with that restriction is that they allow a
charset parameter without specifying that it "MUST" denote a
Unicode encoding form.  I think I suggested that in my earlier
note but don't think it made it into the review.

So, perhaps I should have said "cannot be easily and reliably
distinguished" (and then perhaps define what I mean by
"reliably") as well as substituting "most" for "any".

And perhaps we should suggest including the substance of the
above discussion, referencing Martin's paper of course, as an
appendix to the I-D or as a separate I-D that could then be
referenced.  Perhaps writing such a thing would be a good
project for the directorate.
 
>> I don't know what the
>> default/fallback case actually is or, more generally, what
>> the above paragraph means.  Unless I have misunderstood
>> something important, the reality is that, if there is
>> anywhere on the Internet that a web browser or server
>> (including decades-old embedded servers) treat ISO/IEC 8859-1
>> as either the default or legitimate, then there are two
>> possibilities: accurate labeling of the charset in use or use
>> of heuristics that, by their nature and the nature of
>> possible CCSs (not just encoding schemes), may fail.
 
> For people not familiar with the details, the following should
> be pointed out clearly: The heuristic of detecting UTF-8 as
> UTF-8 based on its very specific bit patterns is an extremely
> strong heuristic. There are cases of specific character
> combinations in specific encodings that can look like UTF-8
> bit combinations, but that is extremely rare and gets rarer
> and rarer the longer the data gets. This is already true if
> one looks just at the available characters/bit patterns, but
> even more so if one looks at the languages that use the
> respective encodings and their letter patterns.

While I think that is, like my original statement, somewhat
stronger than is justified (e.g., we have been repeatedly told
that implementers are not going to do analyses that require
examining long strings and language information is not
guaranteed to be available, and noting that the current I-D
suggests looking only at  initial characters that might be a
BOM, not doing text analysis), I think a paragraph along those
lines would be helpful.

Coming back to ISO/IEC 8859-1, is the sequence "ï" "»" "¿"
(for identification, equivalent to Unicode code points U+00EF,
U+00BB, U+00BF) at the beginning of a file of unencrypted
natural language text likely?  Nope.  Is it "extremely rare",
making the heuristic "extremely strong" when the language is not
known, that is all the text one is going to look at, and when,
depending on Javascript restrictions and restrictions I haven't
found yet in the I-D, there may be no guarantee of normal
language and cleartext?  Seems to me that depends on a judgment
call about how rare or strong something needs to be to rate
"extremely" and I'd guess different of us might have different
opinions about that.   But, again, my concern in making those
comments was that the I-D not say anything that can be read as
"this heuristic will work in all cases" and there I think Martin
and I agree even if the cases in which it will fail are
"extremely rare".

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 13:14:39 -0400

--On Saturday, May 9, 2020 12:33 -0400 John R Levine
<[email-masked]> wrote:

>> I see nothing in either the body of the document nor in
>> Appendix B that  makes it explicit that support for UTF-32
>> has been dropped.  I think  such a comment should be added
>> and explained, even if the explanation is  that no one
>> significant is accepting it any more and it is therefore NOT 
> I agree that a comment saying that UTF-32 is dead as an
> interchange format would be reasonable.
> 
> I'm wondering how much this matters in practice.  I see a lot
> of javascript libraries entirely in ASCII, since non-ASCII
> characters only appear in variable names and in string
> literals and comments.

Martin's comments (IMO, helpful but somewhat pedantic relative
to the issues involved) and my (much more pedantic) response
notwithstanding, my guess (like yours) is that it is unlikely to
make much difference in practice, at least as long as the text
is not actually UTF-16 (BE or LE).  However, I think it is
inappropriate for an IETF consensus document [1] to say
something that amounts to "this will work" when the reality is
"this will work most of the time" or, if a formulation closer to
Martin's is preferred, "this will work except in extremely rare
cases" [2].  So, setting aside the issues with "extremely", a
statement somewhere to the effect of "these heuristics are
pretty good, but they are still heuristics and may fail
occasionally, so implementations should be designed to not fail
catastrophically [3] if a failure occurs" seems appropriate.  In
case it has not been clear, statements like that are what I mean
by a "heath warning".

best, 
   john

[1] Noting that is the new requirement.  At the time RFC 4329
was published, it was possible to interpret an Informational
document like that as the opinion of the author.  With the new
requirement for IETF consensus for any document published in the
IETF stream, any ignorance is the IETF's ignorance and is, at
best, a bad idea.

[2] Comments in my previous message about "extremely"
incorporated by reference.

[3] I consider a program halt (or something like it, including
the notorious bloe screens) or a presumed error message with
semantic content equivalent to "you lose" to be a catastrophic
failure, while I consider "I couldn't figure out what the
character encoding was; it certainly wasn't the UTF-8 I
expected" to be a sensible error message.  YMMD.

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 13:19:39 -0400

--On Saturday, May 9, 2020 12:36 -0400 John R Levine
<[email-masked]> wrote:

>>> I'm sorry to hear that people are using filename extensions
>>> to tell what's in a file.  I'd hoped that all of the badness
>>> from old versions of Windows doing that might have been a
>>> hint.
>> 
>> This is something that just must go away, and I think a big
>> note in non-friendly letters must be added in the security
>> considerations section is needed that explain the situation
>> and why the current practice is not really what we want in
>> the wild.
>> 
>> I.e. yes, while still documenting "what works", that should
>> not be the goal. It will lead to even more problems getting
>> secure mechanisms in browsers deployed.
> 
> I agree with Patrik, this reinvents a security hole from the
> floppy disk era and is not something we should let pass
> without at least making it clear that we believe it is a
> significant mistake.
> 
> It occurs to me that http clients can send an Accept header,
> so how about saying that modules SHOULD be text/jsmodule, but
> MAY be text/javascript for backward compatibiltiy with clients
> that don't handle text/jsmodule?

And these kinds of suggestions (both of them) along with some
caveats about heuristics that will work well most of the time
but maybe not always, and some text that should be much more
obvious to someone reading quickly to figure out what to do
(even if they are clear to a very careful reader) identify, to
me, the main deficiencies in the current I-D.

   john

---

From: Barry Leiba <[email-masked]>
Date: Sat, 9 May 2020 14:36:34 -0400

> for backward compatibiltiy with clients that don't handle text/jsmodule?

I'm very much not fond of the SHOULD... but MAY construction, because
I think the MAY contradicts the SHOULD.

I think the right way to say this is more along the line of "SHOULD
use text/jsmodule.  If that is not practical because backward
compatibility with clients that do not support text/jsmodule is
necessary, then the alternative MUST be text/javascript."

Barry

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 15:14:41 -0400

> necessary, then the alternative MUST be text/javascript."

Fine by me.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 16:28:03 -0400

+1, fwiw.
   john

--On Saturday, May 9, 2020 14:36 -0400 Barry Leiba
<[email-masked]> wrote:

>> MAY be text/javascript for backward compatibiltiy with
>> clients that don't handle text/jsmodule?
> 
> I'm very much not fond of the SHOULD... but MAY construction,
> because I think the MAY contradicts the SHOULD.
> 
> I think the right way to say this is more along the line of
> "SHOULD use text/jsmodule.  If that is not practical because
> backward compatibility with clients that do not support
> text/jsmodule is necessary, then the alternative MUST be
> text/javascript."
> 
> Barry

---

From: Asmus Freytag <[email-masked]>
Date: Sat, 9 May 2020 16:44:08 -0700

> necessary, then the alternative MUST be text/javascript."

Barry's formulation possibly implies that there's some third alternative 
that SHOULD / MAY allows without mentioning it. However, if that is the 
case, the new formulation allows this third alternative in every case 
where the implementer deviates from SHOULD for any reason other than the 
one given in the MUST phrase.

MUST be one of...

SHOULD not use text/javascript unless ...

This way the limitation to the two possible forms is explicit and the 
grounds for the exception are attached to a SHOULD phrase, which 
conceptually does imply the need for a justification.

You could also quibble that a MUST phrase that has an implementer 
defined "out" is against the spirit of the thing.

A./

---

From: "John Levine" <[email-masked]>
Date: 9 May 2020 22:11:52 -0400

>defined "out" is against the spirit of the thing.

If you look at 2119, SHOULD roughly means the same as MUST unless you
understand why you're special.

I think we all know what we want to say, I'll let you guys thrash it out.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 10 May 2020 06:49:02 +0200

> +1, fwiw.

Agree

---

From: Asmus Freytag <[email-masked]>
Date: Sat, 9 May 2020 23:17:27 -0700

> understand why you're special.

That then squares with this pattern:

MUST be <one of two alternatives>

SHOULD be <the preferred one>, unless ... <circumstances>

That way, there's no room for doing something third and the SHOULD works 
as intended.

A./

> I think we all know what we want to say, I'll let you guys thrash it out.
t

---

From: Mathias Bynens <[email-masked]>
Date: Mon, 11 May 2020 09:22:36 +0200

On Sat, May 9, 2020 at 6:36 PM John R Levine <[email-masked]> wrote:

> >> I'm sorry to hear that people are using filename extensions to tell
> what's in a file.  I'd hoped that all of the badness from old versions of
> Windows doing that might have been a hint.
> >
> > This is something that just must go away, and I think a big note in
> non-friendly letters must be added in the security considerations section
> is needed that explain the situation and why the current practice is not
> really what we want in the wild.
> >
> > I.e. yes, while still documenting "what works", that should not be the
> goal. It will lead to even more problems getting secure mechanisms in
> browsers deployed.

> for backward compatibiltiy with clients that don't handle text/jsmodule?

I agree it would have been nice if the HTML spec + browsers had restricted
modules to a new MIME type, but this proposal was discussed and rejected
<https://github.com/whatwg/html/issues/558>. In 2017, all major browsers
supported JavaScript modules using the recommended MIME type of
text/javascript (Myles already posted the details), or any other JS MIME
type <https://mimesniff.spec.whatwg.org/#javascript-mime-type> as fallback.

At this point, I’m afraid the ship has sailed. Introducing yet another MIME
type *now* is unlikely to gain adoption in the HTML Standard, browsers, or
on the web (because it wouldn’t be backwards compatible). Let’s not publish
specs that intentionally conflict with both the HTML Standard and web
reality.

Also, why is the BOM deleted?  ECMAscript says a BOM is a space so it
> should be harmless.

It’s not entirely harmless. The following JS program behaves differently if
it’s preceded by a BOM:

#!/usr/bin/env bash

Without BOM, this is a hashbang comment, and no exception is thrown. With
BOM, this is invalid syntax, resulting in a SyntaxError exception.
Stripping away the BOM is something hosts (e.g. browsers) do before
processing the source as JS.

---

From: "John R Levine" <[email-masked]>
Date: 11 May 2020 12:55:26 -0400

> specs that intentionally conflict with both the HTML Standard and web

It actually would be backward compatible because of the advice to use 
Accept media types to tell when to send which media type, but if browsers 
aren't going to do it, I guess we put a warning in the security section 
and give up.

> It’s not entirely harmless. The following JS program behaves differently if

I really don't want to know when self-running shell scripts became part of 
Ecmascript.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 May 2020 15:16:18 -0400

--On Monday, May 11, 2020 12:55 -0400 John R Levine
<[email-masked]> wrote:

>> At this point, I'm afraid the ship has sailed. Introducing
>> yet another MIME type *now* is unlikely to gain adoption in
>> the HTML Standard, browsers, or on the web (because it
>> wouldn't be backwards compatible). Let's not publish
>> specs that intentionally conflict with both the HTML Standard
>> and web reality.
> 
> It actually would be backward compatible because of the advice
> to use Accept media types to tell when to send which media
> type, but if browsers aren't going to do it, I guess we put a
> warning in the security section and give up.

Well, just a minute.  Despite approval by DISPATCH for further
discussion and processing, this is basically an individual
submission, not a draft developed by a WG.  With the
understanding that what I'm about to say is not I18n-specific
and therefore should perhaps go on the agenda of the next
DISPATCH meeting or wait for IETF Last Call, the IETF has, as
Patrik has pointed out, both a collection of standards-track and
BCP specifications in these areas and experience with what
happens when other approaches (like reliance on name suffixes)
are used instead.  Recommending something else is not just a
matter of Security Considerations and, fwiw, I note that at
least one document has been stuck after IETF Last Call completed
in September because the IESG had DISCUSS-level concerns about
telling implementers what they should do if they chose to not
conform to the IETF's normative recommendations.   If this is
really just a description of what browser vendors are doing or
will do now and in the future, then we have a long history of
publishing "for the information of the community, this is what
some vendor or cluster of vendors are doing" descriptive
Informational documents.  They are not standards track, they do
not use normative language (2119/8174 terminology included),
they just describe what is being done and are quite explicit
about that.  

So, if the decision criteria for what belongs in this I-D is
"whatever the browser vendors are going to do whether the IETF
thinks it is a good idea or not", then let's see it recast as a
"what is being done in practice" description, with the normative
language removed or transformed into "if you want to do what the
vendors are doing, then this is what you need to do" (or even
"should" do - noting lower case).  If that means updating the
registration for text/javascript (which, I note, is in the
registry at
https://www.iana.org/assignments/media-types/media-types.xhtml#text
as "OBSOLETED in favor of application/javascript" -- exactly the
opposite of which this I-D proposes) and application/javascript
to treat them as vendor registrations (applying the exception
for non-faceted names allowed by Appendix A of RFC 6838) then so
be it... because that is, apparently, the truth of the matter.

Sorry, but it seems to me that this document is trying to have
it both (or all three) ways: to be a normative document without
meeting the higher bar the IETF usually applies for standards
track documents, to be a vendor specification without making
that clear in the text (and noting that vendor specifications
are still expected to have accurate Security Considerations
sections), and to make requirements that are at variances with
significant community experience about bad results when similar
things have been done in only slightly different areas and still
expect IETF consensus for the document.  I don't think that
works.  YMMD.

And...

>> It's not entirely harmless. The following JS program
>> behaves differently if it's preceded by a BOM:
>> 
>> # !/usr/bin/env bash
> 
> I really don't want to know when self-running shell scripts
> became part of Ecmascript.

Presumably, by an extension of the comments that caused the rant
above, whatever "part of ECMAscript" means, they are a
consideration because some or all browser vendors decided to
treat them as if they were.

    john

---

From: Bradley Farias <[email-masked]>
Date: Mon, 11 May 2020 17:21:27 -0500

Forgive me, but the tone appears to be rather harsh against people who have
spent quite a few years waiting on this document to even get to a
visibility that we can discuss things. Per Adam Roach's recommendation we
did not go through a WG when I made this document years ago. The past few
emails seem rather derisive against such decisions, but I am still new to
IETF workings and would love help through the process; there seems to be
issues that cause me some personal confusion on the intent of a review.

Is there a way we can follow or be aided such that we can reach the "higher
bar" that we are unaware we were missing here such that we can go for a
normative change? I'm unclear but it seems like there is a desire to design
some solution for concerns about ECMA262 in host environments but I'm
unclear on this being the venue to do so and would be appreciative of
guidance on how to proceed.

Additionally, I have concerns with shipping things like MIMEs that are not
supported in any environment and are not discussed in the host venues they
intend to be used in and without any buy in from a host. This document is
trying to be normative to match the environment scenarios of not just
browsers but other environments that run ECMAScript such as server side
environments like Node.js are using text/javascript as the preferred
non-obsoleted form in order to coalesce around a standard that matches the
browser recommendations. The reversal of the application/javascript as the
preferred MIME was intentionally discussed in those environments. I'm
unclear on how creation of a new MIME would help here given we cannot
remove support for the status quo; creation of a new MIME seems like it
would require implementer interest to be a viable replacement.

Per things like hash bang comments and concerns about the language being
different from when older RFCs existed; I would view this document as being
a way to catch up to at least the status quo; creation of
unrelated evolution of standards paths that are not intended to match the
status quo likely won't be useful without other standards groups buy in. In
the future I'm sure people with such varied interests in the evolution of
ECMAScript certainly could participate in the evolution of the language if
they want to, but that was not the intent of this document. It seems like
an additional RFC might be desirable for other issues we are seeing brought
up here? Once again, I'm less familiar with IETF process and am unclear if
all possible design decisions need to be made in this document.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 11 May 2020 18:57:16 -0400

Thanks for this, Bradley.  A couple of clarifications, to make sure we’re
together on this:

The IETF crowd can tend to be blunt — sorry for that.  If it helps, keep in
mind that the purpose in that bluntness is not to insult, but to aim for
the highest quality documents we can get.

As to the higher bar for non-WG documents, it’s not that there’s a higher
bar for the document content — we’re looking, as I say, for high quality in
general — but a higher bar for getting reviews at this stage.  Working
group documents have time in working group development and review, and the
IETF last call process is bringing the document to the community beyond the
working group for final review.  Non-WG documents lack the work and review
in the working group, and, frequently, the first significant review is
during last call.  There is, consequently, a stronger-than-usual desire to
ensure thorough review at this stage.

I, as the sponsoring AD, am aware of the work that’s been done on the
document already, and am not concerned, in general, about the quality of
that.    That said, this particular review is from experts in
internationalization issues, and that’s a quite specialized skill set held
by a relatively few participants.  It’s really important to get this part
right, and I very much appreciate their review and discussion.  I hope we
can put all this together to make the document even better.

Barry, ART AD

On Mon, May 11, 2020 at 6:21 PM Bradley Farias <[email-masked]>
wrote:

> Forgive me, but the tone appears to be rather harsh against people who
> have spent quite a few years waiting on this document to even get to a
> Is there a way we can follow or be aided such that we can reach the
> "higher bar" that we are unaware we were missing here such that we can go
> for a normative change? I'm unclear but it seems like there is a desire to
> design some solution for concerns about ECMA262 in host environments
> but I'm unclear on this being the venue to do so and would be appreciative
> of guidance on how to proceed.
>