Subject: [DNSOP] Status of IDNA

From: Florian Weimer <[email-masked]>
Date: Wed, 12 Apr 2017 13:36:49 +0200

What's the current standardization status of IDNA?

As far as I can tell, a lot of vendors are still stuck with the original 
IDNA standard (IDNA2003).  There are three or more competing successors, 
IDNA2008 as standardized by the IETF (without any tweaks), the Unicode 
IDNA standard TS46 (<http://www.unicode.org/reports/tr46/>, which is 
configurable and allegedly compatible with IETF IDNA2008, but is not 
because it yields different results than IETF IDNA2008), and the 
Mozilla/DENIC IDNA implementation in Firefox/Thunderbird 
(<https://bugzilla.mozilla.org/show_bug.cgi?id=479520> and other sources).

These aren't compatible.  You can see this by visiting 
<https://www.buße.de/> in different browsers.

Is there an ongoing effort to reconcile application behavior?  Different 
TLDs appear to expect different IDNA implementations.

One practical problem with IDNA2003 is that it prevents having Hebrew 
domain names containing ASCII digits.  Any of the IDNA2008 variants 
mentioned above will fix that, I think, but it's difficult to pick a 
variant to implement.  And I certainly don't want to implement per-TLD 
policies.

Thanks,
Florian

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 12 Apr 2017 15:47:14 +0000

> <https://www.buße.de/> in different browsers.

Some browsers are still doing "transitional" UTS#46, it is time
they move on.  In up to date releases:

    Firefox:	Already UTS#46 non-transitional
    Safari:	Already UTS#46 non-transitional
    Chrome:	Still broken:
		https://bugs.chromium.org/p/chromium/issues/detail?id=505262
		https://bugs.chromium.org/p/chromium/issues/detail?id=61328

Postfix and Exim have also moved on to UTS#46 non-transitional.

> Any of the IDNA2008 variants mentioned above
> will fix that, I think, but it's difficult to pick a variant to implement.
> And I certainly don't want to implement per-TLD policies.

Just move on to non-transitional UTS#46.

-- 
	Viktor.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 12 Apr 2017 19:18:04 -0400

Accidentally sent this just to Florian.  Fixing that.

> What's the current standardization status of IDNA?

It's complicated.

> IDNA standard (IDNA2003).

Well, not exactly.  IDNA 2003 is defined only for Unicode 3.2, which
nobody has used for a long time.  So strictly speaking, what most of
those vendors are doing is undefined, since there's no way to know
what version of Unicode you have installed.  Bur practically, of
course, this is what's happening.

> There are three or more competing successors,
> IDNA2008 as standardized by the IETF (without any tweaks), the Unicode IDNA
> standard TS46 (<http://www.unicode.org/reports/tr46/>, which is configurable
> and allegedly compatible with IETF IDNA2008, but is not because it yields
> different results than IETF IDNA2008), and the Mozilla/DENIC IDNA
> implementation in Firefox/Thunderbird

UTS#46 is allegedly a transitional technology that is supposed to do
mapping so that things that did work in IDNA2003 but that won't under
IDNA2008 will continue to work.  As a practical matter, it does not
appear to have a mechanism by which the transition can be brought to
an end, so it's hard to see it as an actual transition mechanism.
It's also hard to understand why it has so many characters marked as
"valid" that are not valid under any version of IDNA, including many
emojis.  

> <https://www.buße.de/> in different browsers.

Yes.

> TLDs appear to expect different IDNA implementations.

ICANN's rules require IDNA2008.  There is a new version of the
guidelines out -- I can't recall whether the public comment is closed.
Many ccTLDs voluntarily conform to the guidelines also, but ICANN
can't compel it.  And of course ICANN has no control of other things
down the tree.

Part of the trouble comes because the WHATWG is apparently opposed to
IDNA2008, partly on the grounds that it makes some domain names that
were valid under 2003 invalid.

> One practical problem with IDNA2003 is that it prevents having Hebrew domain
> names containing ASCII digits.  Any of the IDNA2008 variants mentioned above

Another practical problem with IDNA2003 is that it loses information
round trip: Unicode can be sent into the algorithm, turned into
Punycode, and when it is turned back into Unicode the result is not
the same.  ß is mapped, for instance, so you get back "ss" whatever
you put in.  This issue is one of the main things that IDNA2008 fixes,
in my opinion: every valid A-label matches exactly one valid U-label,
and anything that does not have this property is not a valid A- or
U-label.

There are problems with IDNA that have prevented it being updated for
recent versions of Unicode.  Asmus Freytag, John Klensin, and I are
working on a draft to try to fix that.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 12 Apr 2017 19:45:41 -0400

> Just move on to non-transitional UTS#46.

Given that its mappings include many emojis, which aren't allowed
under either IDNA2003 or IDNA2008, I think there is an apparent
problem with UTS#46 too.  One might be forgiven for wondering why the
UTC decided that it knew something about domain names and the history
of how they are used.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 13 Apr 2017 02:39:36 +0000

> UTC decided that it knew something about domain names and the history

Well, IIRC they sensibly converged on a case-folded normal form
that ensures that https://Духовный.org maps to the same underlying
wire-form domain as https://духовный.org, i.e. both result in
queries for xn--b1adqpd3ao5c.org.  AFAIK, those would generally be
different domains under IDNA2008.

While is true that UTS#46 maps <U+1F4A9>.org to xn--ls8h.org, (see
http://www.fileformat.info/info/unicode/char/1f4a9/index.htm) that
seems to be a small price to pay for avoing surprising case-sensitive
behaviour.

-- 
	Viktor.

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 13 Apr 2017 07:33:13 -0400

> different domains under IDNA2008.

They would be different domains because the first of them is
DISALLOWED.  But everyone knew, when making IDNA2008, that removing
the case mapping from the protocol meant that clients needed to do it
before starting.  That's what https://tools.ietf.org/html/rfc5895 was
all about (a document that could have moved faster if some
participants had collaborated more enthusiastically instead of, well,
going away and making their own protocol).

One of the problems we had with IDNA2003 was that the protocol did the
caseFold operation.  The difficulty there was that there was no way to
pay attention to locale or other information that might tell you the
right thing to do, because caseFold is not nearly as simple as ASCII
always pretended it was.  IDNA2008's answer was to kick this problem
out of the protocol and into user agents, which were supposed to do
this in a sensible way.  If UTS#46 had restricted itself to that kind
of job, it could well have been an enormous contribution to the
practical use of IDNs.  Unfortunately, it didn't do just that.

> While is true that UTS#46 maps <U+1F4A9>.org to xn--ls8h.org, (see

In the way I'm using the term, that's not mapping, that's re-encoding.
The idea of "mapping" is to substitute in some more or less
predictable way some set of Unicode code points for some other set of
Unicode code points.  Then you can run the resulting final string
through the IDNA2008 algorithm.  

The problem with treaing U+1F4A9 as an acceptable character for an
identifier is not in itself -- maybe it is fine on its own -- but that
it is part of a class of characters that do not have normalizations
and are not letters or digits.  The IETF took seriously the advice
from UTC that we should use the stable categories that UTC had
invented, and derive our properties from those.  We did so, and no
emojis are in the categories that we used for the derivation.  It is
therefore more than a little frustrating to see the same UTC now
recommending that such characters be used in identifiers on the
network.  Such use is particularly bad with emojis because they have
no normalization either, and they interact in some ways with ZWNJ.
They provide a completely new playground for attackers to use in
phishing and so on, and we already have _enough_ trouble with that
without inventing new ways to cause ourselves grief.

Anyway, I doubt very much that DNSOP is the list where this ought to
be discussed (idna-update is still an open list, as is the IAB's
i18n-discuss list even though the program is closed, and precis is
still an active WG last I checked).  But any sentence about
internationalization that involves the concept "just do _x_" is, I
think, already too naïve.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]