Subject: Re: [I18nrp] [art] Use Unicode if Using Unicode?

From: John C Klensin <[email-masked]>
Date: Wed, 10 Oct 2018 22:00:59 -0400

Shawn,

I'm confused about what you are suggesting, so let me clarify
where I'm confused and then hope you can enlighten me...

--On Wednesday, October 10, 2018 19:02 +0000 Shawn Steele
<Shawn.Steele=[email-masked]> wrote:

> The draft states "It further suggests for the IETF a path
> forward regarding ensuring IDNA2008 follows the evolution of
> the Unicode Standard" and "this document requests that IANA
> update the tables to Unicode 11."

> Each Unicode version creates a data file with information from
> applying the IDNA2008 rules that can be used for IDNA mapping
> algorithms.  (Indeed, that's where Windows gets the data
> from). 

Taking a half-step back, IDNA imposes two requirements for new
versions of Unicode, both of which are addressed by the present
draft.  One is that the changes in the new version be examined
to ensure that nothing new has been done that requires changes
to IDNA (most likely adding exceptions or rules for particular
code points) or careful explanation somewhere.  No one I know of
expected that provision would be exercised very often but, for a
variety of reasons, the consensus was that it was an important
safeguard.   The other was that the IDNA ruleset be run against
the characters and properties in the Unicode Character Database
to produce a table reflecting the combination of that version of
Unicode and IDNA, as modified, at that time.  That table was
(and is) to be stored with IANA but, while we expect it to be
checked carefully for accuracy, it is not actually authoritative
-- only the rules and categories specified by IDNA (and the
Unicode properties used to support them) are.

The discussion in RFC 5895 notwithstanding, one of the
critically important properties of IDNA2008 is that U-labels and
A-labels are duals: one can get from one to the other and back
without any loss of information.  That reversibility is true in
only some cases for Unicode normalization (especially with
compatibility normalization) or case folding, much less for
other mapping scenarios.  So I don't understand what sort of
"mapping" you are talking about.

The only Unicode-created IDN data files I'm aware of are those
associated with the UTS#46 effort.  Because UTS#46 makes
recommendations that are inconsistent with IDNA2008, if
Microsoft is using those tables, its usage is non-conforming to
IDNA2008.  I certainly cannot prevent Microsoft from doing that
(and wouldn't try), but it would certainly not be consistent
with general interoperability of IDNs or what is known elsewhere
as Universal Acceptance of those domain names.

> If the goal is to "follow the evolution of the Unicode
> Standard" and the Unicode Standard is providing data that
> conforms to the IDNA rules, then why not just point directly
> to the Unicode derived tables?

The simplest answer to your question is that, unless I've missed
something, the conditional is false: the Unicode Standard is not
providing data that conforms to the IDNA2008 rules.  Instead,
the data they are providing is more like one of the creative
IDNA2003-IDNA2008 hybrids to which Patrik refers.

   best,
     john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 11 Oct 2018 07:48:38 +0000

The file Unicode generates can be used either with the UTS46 compatibility characters or without.  To use your terms, it both provides data that conforms to the IDNA2008 rules... and hybrid versions.  If you choose the appropriate information when reading the IdnaMappingTable.txt you should have exactly the table expected.  In particular the NV8 and XV8 tagged characters I believe you'd prefer to exclude.

I'm unaware of other differences.  I could also be in error, I haven't actually done an exhaustive comparison of the IndaMappingTable.txt data vs generated datasets via the rules.

The IdnaMappingTable.txt file also isn't "authoritative," however since it's generated from the Unicode data and the rules in the RFCs, the file should be as accurate as any other reference.  

Unicode is aware of the importance of the stability of the valid character set, so probably the best time to review and vet the Unicode changes for new Unicode versions would be during the Unicode process, prior to release.  After a Unicode version was released, any disagreements would lead to problematic confusion (as we've seen before.)

It seems to me that using the IdnaMappingTable.txt - perhaps with guidance as to proper usage - would reduce delays in adopting new characters.

-Shawn

-----Original Message-----
From: John C Klensin <[email-masked]> 
Sent: Wednesday, October 10, 2018 7:01 PM
To: Shawn Steele <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [art] Use Unicode if Using Unicode?

Shawn,

I'm confused about what you are suggesting, so let me clarify where I'm confused and then hope you can enlighten me...

--On Wednesday, October 10, 2018 19:02 +0000 Shawn Steele <Shawn.Steele=[email-masked]> wrote:

> The draft states "It further suggests for the IETF a path forward 
> regarding ensuring IDNA2008 follows the evolution of the Unicode 
> Standard" and "this document requests that IANA update the tables to 
> Unicode 11."

> algorithms.  (Indeed, that's where Windows gets the data from).

Taking a half-step back, IDNA imposes two requirements for new versions of Unicode, both of which are addressed by the present draft.  One is that the changes in the new version be examined to ensure that nothing new has been done that requires changes to IDNA (most likely adding exceptions or rules for particular code points) or careful explanation somewhere.  No one I know of expected that provision would be exercised very often but, for a variety of reasons, the consensus was that it was an important
safeguard.   The other was that the IDNA ruleset be run against
the characters and properties in the Unicode Character Database to produce a table reflecting the combination of that version of Unicode and IDNA, as modified, at that time.  That table was (and is) to be stored with IANA but, while we expect it to be checked carefully for accuracy, it is not actually authoritative
-- only the rules and categories specified by IDNA (and the Unicode properties used to support them) are.

The discussion in RFC 5895 notwithstanding, one of the critically important properties of IDNA2008 is that U-labels and A-labels are duals: one can get from one to the other and back without any loss of information.  That reversibility is true in only some cases for Unicode normalization (especially with compatibility normalization) or case folding, much less for other mapping scenarios.  So I don't understand what sort of "mapping" you are talking about.

The only Unicode-created IDN data files I'm aware of are those associated with the UTS#46 effort.  Because UTS#46 makes recommendations that are inconsistent with IDNA2008, if Microsoft is using those tables, its usage is non-conforming to IDNA2008.  I certainly cannot prevent Microsoft from doing that (and wouldn't try), but it would certainly not be consistent with general interoperability of IDNs or what is known elsewhere as Universal Acceptance of those domain names.

> If the goal is to "follow the evolution of the Unicode Standard" and 
> the Unicode Standard is providing data that conforms to the IDNA 
> rules, then why not just point directly to the Unicode derived tables?

The simplest answer to your question is that, unless I've missed something, the conditional is false: the Unicode Standard is not providing data that conforms to the IDNA2008 rules.  Instead, the data they are providing is more like one of the creative
IDNA2003-IDNA2008 hybrids to which Patrik refers.

   best,
     john

---

From: Larry Masinter <[email-masked]>
Date: Thu, 11 Oct 2018 21:52:28 -0700

No matter what path you take for new RFCs, implementations will lag, and some choices for strings used for domain names, (LHS of) email addresses and short URLs will have problems 
1) When put through different implementations of normalization (IDNA2008 vs 2003 vs. some exceptions)
2) When used in user interfaces for comparison or validation or comparison (in the address bar or rollover links)
3) When scanning text for picking out URLS
4) when used in print or other media (in an ad or written on a napkin)

These are in order of decreasing specificity, increasing “liberal displacement”.
I think those who are choosing names to use (as domain name, email address, short url) will want to use (4), no matter what rules the registry applies. Anyone registering a domain name, a user name at a mail host, or a (short) URL will want to choose names that they can print on a business card or an ad or a roster and believe that their target audience can enter and get the string intended.
And 1 < 2 < 3 < 4 as equivalence goes. 

I don’t think it’s practical to redo all the protocols that use these strings to pass along a tuple of string and context.

Larry
--
https://LarryMasinter.net

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 12 Oct 2018 01:12:37 -0700

On 10/11/2018 9:52 PM, Larry Masinter wrote:
>
> No matter what path you take for new RFCs, implementations will lag, 
> and some choices for strings used for domain names, (LHS of) email 
> addresses and short URLs will have problems
>
>  1. When put through different implementations of normalization
>     (IDNA2008 vs 2003 vs. some exceptions)
>  2. When used in user interfaces for comparison or validation or
>     comparison (in the address bar or rollover links)
>  3. When scanning text for picking out URLS
>  4. when used in print or other media (in an ad or written on a napkin)
>

That sounds vaguely political :)

Not sure what you mean with that expression, actually.

> I think those who are choosing names to use (as domain name, email 
> address, short url) will want to use (4), no matter what rules the 
> registry applies. Anyone registering a domain name, a user name at a 
> mail host, or a (short) URL will want to choose names that they can 
> print on a business card or an ad or a roster and believe that their 
> target audience can enter and get the string intended.
>
> I don’t think it’s practical to redo all the protocols that use these 
> strings to pass along a tuple of string and context.
>
You can avoid certain problems by limiting what a registry (or mail 
server) allow to be registered. This limitation does not belong in the 
protocol, but is a separate layer of the architecture (label generation 
rules).

With label generation rules you can restrict strings to those that can 
be rendered unambiguously, and should display technology change over 
time, you can relax those constraints without affecting other 
implementations of the protocol.

Many of these kinds of constraints are (largely) generic to certain 
scripts. It would therefore be possible to address them by informational 
RFCs laying out "best practices" that can serve as a starting point.

To some extent, similar mechanisms can be used to address discrepancies 
between IDNA2003 and IDNA2008 - in some cases affected code points could 
either be restricted or equivalenced.

Am I on the right track, or were you trying to address something different.

A./

> _______________________________________________
> i18nRP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18nrp