Subject: Re: [Idna-update] Genart telechat review of draft-faltstrom-unicode11-08

From: John C Klensin <[email-masked]>
Date: Mon, 18 Mar 2019 13:21:37 -0400

Dan,

I have read Barry's two responses, Alissa's response, and your
response to Barry, but this note seems the easiest one to which
to respond.  This particular draft is, as I understand it, OBE,
but there are (still) a few things going on that are probably
not visible from this particular I-D alone and that were even
less visible in draft-faltstrom-unicode11-07.   The draft and
those issues  has been discussed extensively in the
recently-created I18n directorate and many, perhaps most, of the
changes between -07 and -08 are the result of those discussions
(the summary below might be consistent with those directorate
discussions, but I'm not sure there is sufficient consensus
there to make such a claim, so take the summary as my personal
opinion).   A bit of that perspective follows for those who have
not been following IDNA or i18n issues closely; the IDNA-update
list is copied in the hope that I (and Patrik and others) won't
need to go through this too many more times.

(1) This review of new Unicode versions and these IANA tables
are inherently odd.  Usually, when information is sent to IANA
and put, by them, in a registry, the contents of that registry
are considered the authoritative source of the information.  As
one of the older examples still in use, if there is a list of
assigned port numbers floating around the Internet somewhere and
the values in it are different from the values in the IANA
registry (
https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml
), the IANA registry is authoritative and that list is simply
wrong.   That is not the case for these IDNA-related tables.
For them, according to the IDNA2008 specs (and RFC 5892 in
particular) the authoritative information for a particular
application or implementation is to pick the version of Unicode
that application or implementation is using (in theory, at a
given instant if that is changing) and then calculate the
derived properties as specified from RFC 5892, and then use that
information (if the implementation needs a table, that is how it
is supposed to get the table).  The WG was persuaded, after
what, IIR, was considerable demand, to create the IANA registry
and put tables in it, but that does not make the IANA tables
authoritative, nor do they distinguish between supported code
points or supported versions of Unicode, etc.  

(2) In a more perfect world, the review called for by RFC 5892
and reflected in RFC 6452 and this I-D would not be necessary at
all.  Unicode would announce a new version, assorted entities
would calculate tables for their use with that version, and
everyone would go merrily on their way.  The IDNA-update WG was
painfully aware of the imperfection of the world and, for this
particular case, was worried about two possibilities (both of
which we hoped would be very, very rare).  One was that, for a
new version of Unicode, the Unicode Consortium would make
changes in the properties or category values they assigned to
particular code points in a way that would change the derived
properties calculated by the RFC 5892 algorithm for those code
points.  This I-D primarily reflects a review designed to detect
such changes.  As I read RFC 5892, if such cases are detected,
the normal response is to introduce additional backward
compatibility exceptions into the list in RFC 5892 to preserve
stability (Patrik reads that a bit differently than I do).  The
other option is to leave the exception list in 5892 unchanged,
but that means that an application using Unicode version YY and
tables corresponding to it (and possibly a current set of IANA
tables) is not going to behave in a way that is completely
consistent (except for code points not allowed earlier because
they were unassigned) with an application using Unicode version
XX and corresponding IDNA tables).   Ultimately, the choice is
whether, for a particular code point whose Unicode properties
have changed, to maintain consistency with Unicode changes or to
maintain consistency within implementations that calculated IDNA
derived properties based on an earlier Unicode version and are
now recalculating.   

The second possible problem, to which this I-D pays little
attention, is one in which it is discovered that some action
(such as assigning a new code point) or other discovery exposes
an issue or other behavior that was not anticipated by IDNA2008.
This is obviously a much more complicated problem, harder to
review for and detect and much harder to figure out what to do
with.  It was, however, precisely the sort of situation that we
discovered when reviewing the additions in Unicode 7.0.  That
review led to the 2015 IAB statement and to
draft-klensin-idna-5892upd-unicode70.   If the issue were just
about one code point, we could either make it an exception or
ignore it, and move on.  However,  it isn't and examining it led
to a series of still-unresolved issues about assumptions
IDNA2008 made about Unicode and its evolution.  Since that
discovery in 2014, time and Unicode versions have moved on and
applications and libraries that are based on doing their own
derived property calculations (as IDNA2008 requires) have
diverged more and more from any assumption that the tables
available from IANA are somehow "current" or that other
inferences can be drawn from them and the version of Unicode
they reference.

(3) As long as there are no changes or issues of either of the
types described in (2) above, a reasonable reading of RFC 5892
is that the Designated Expert shrugs his or her shoulders,
passes a new table off to IANA (or validates a new table that
IANA calculates) and moves on, with no report to IESG or I-D
that is expected to become an RFC required.   Viewed that way,
RFC 6452 was published only to confirm that, for the first
version of Unicode to appear after the IDNA2008 docs were
finalized, things were working as predicted and no IETF action
was required.  draft-faltstrom-unicode11 is significant, less
because several years have passed and we need to catch up (and
maybe explain) but because, between Unicode 6.x and 11.0 (or
12.0) we've discovered examples of both changed derived
properties for code points that existed before the relevant
Unicode version and a case of discovering an "other problem"
that opens one or more cans of worms.

(4) Because the IDNA requirement is that applications calculate
derived properties (or find and use an appropriate table) that
corresponds to whatever version of Unicode they are using,
IANA's having a registry that consists only of the table
corresponding to one particular version of Unicode seems
inappropriate, unwise, and maybe inconsistent with that
provision of the standard.

It is fairly clear to me that sorting the above out and
clarifying things to the point that confusion is considerably
reduced and expectations of the Designated Expert(s) are clear
should now be a reasonably high priority.   It is almost equally
clear that putting those clarifications in the critical path in
front of publication of draft-faltstrom-unicode11 (or
-unicode12), much less getting relatively more current tables
into IANS's hands, would be unwise pragmatically, however much
more satisfying it might be procedurally.

Specific suggestions about how this might be untangled, text and
ideas about where to put it, etc., would be very welcome (at
least by me).

best,
    john

--On Monday, March 18, 2019 00:31 -0700 Dan Romascanu via
Datatracker <[email-masked]> wrote:

> Reviewer: Dan Romascanu
> Review result: Ready with Issues
> 
> I am the assigned Gen-ART reviewer for this draft. The General
> Area Review Team (Gen-ART) reviews all IETF documents being
> processed by the IESG for the IETF Chair. Please wait for
> direction from your document shepherd or AD before posting a
> new version of the draft.
> 
> For more information, please see the FAQ at
> 
> <https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.
> 
> Document: draft-faltstrom-unicode11-08
> Reviewer: Dan Romascanu
> Review Date: 2019-03-18
> IETF LC End Date: 2019-03-05
> IESG Telechat date: Not scheduled for a telechat
> 
> Summary:
> 
> This document is Ready. However, I have signaled one minor
> issue in my LC review which was not addressed in the revision,
> neither have I seen an answer to the clarification that I
> requested (unless I missed some mail).
> 
> Major issues:
> 
> Minor issues:
> 
> (from my previous review, not addressed)
> 
> The IANA Considerations text is not clear to me.
> 
>> IANA is requested to update the IDNA Parameters registry of
>> derived
>    property values, after the expert reviewer validates that
> the derived    property values are calculated correctly.
> 
> What does this exactly say? Are these recommendations for
> future actions, or do they refer specifically to property
> values described in this document? Are the expert reviewer
> validation and IDNA Parameters registry update required to be
> performed prior to approval of this document?
> 
> Nits/editorial comments:
> 
>

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 19 Mar 2019 07:47:01 +0100

On 18 Mar 2019, at 18:21, John C Klensin wrote:

> (2) In a more perfect world, the review called for by RFC 5892 and reflected in RFC 6452 and this I-D would not be necessary at all.  Unicode would announce a new version, assorted entities would calculate tables for their use with that version, and
> everyone would go merrily on their way.  The IDNA-update WG was painfully aware of the imperfection of the world and, for this particular case, was worried about two possibilities (both of which we hoped would be very, very rare).  One was that, for a new version of Unicode, the Unicode Consortium would make
> changes in the properties or category values they assigned to particular code points in a way that would change the derived properties calculated by the RFC 5892 algorithm for those code points.  This I-D primarily reflects a review designed to detect such changes.  As I read RFC 5892, if such cases are detected, the normal response is to introduce additional backward
> compatibility exceptions into the list in RFC 5892 to preserve stability (Patrik reads that a bit differently than I do).

Correct, and let me try to explain this in slightly different words. And yes, we do read 5892 slightly differently, but I claim the outcome is the same. Let me come back to that.

1. We MUST remember that it is the algorithm in 5892 that is normative. Not any table of derived property values. As John wrote which I agree with, having IANA having a table with non-normative data was probably a very bad idea. And maybe that registry should simply be removed. It just confuses people. If it confuses people on IESG, of course it confuses other people as well. Maybe the description of the table at IANA should be changed? Let me think and discuss with IANA.

2. The whole idea with the algorithm was that the property values that the algorithm uses would never ever be changed as Unicode Consortium said those would be stable. At least the property values IETF did use in the algorithm. So having the algorithm normative, and "just" let Unicode consortium release new versions of Unicode was EXACTLY one design criteria for IDNA2008 that is vastly different from the "pick code points" that IDNA2003 did.

3. RFC 5892 was based on Unicode 5.2.0, and since then Unicode have released versions 6.0.0, 6.1.0, 6.2.0, 6.3.0, 7.0.0, 8.0.0, 9.0.0, 10.0.0, 11.0.0, 12.0.0 and a pre-relase of 12.1.0. Of all of these versions, two include incompatible changes: 6.0.0 and 11.0.0.

4. As John wrote, the simple solution for the IDNA2008 standard to move forward would be to change the algorithm so that the IDNA2008 algorithm when applied to any Unicode version would produce the same result, and indeed did we think about that when doing RFC 5892. See section 2.7 <https://tools.ietf.org/html/rfc5892#section-2.7>. By adding code points and explicit derived property values to rule set [G], we would have backward compatibility.

5. When 6.0.0 was released, I as appointed expert did flag that for IETF (I do not remember exactly how, but I might be able to dig up those email messages) and after some discussions the conclusion was that rule set [G] should NOT be updated. I think I was convinced that at that time some software libraries where changed, and it was more important to get IDNA2008 out the door and we had the risk of two different incompatibilities:

5.1. If we added a rule to [G], the incompatibility would be between software packages that did use the original version of 5892 algorithm without the added rule and the ones which did have the updated rule.

5.2. If we did not add a rule to [G], the incompatibility would be between software packages that calculated the derived property value on Unicode 5.2.0 and the ones that did the calculation on 6.0.0.

Ultimately [5.2] above did win, and that was explained in RFC 6452. And I as the appointed expert and editor of 5892 I feel good having the discussion about this incompatible change that *would never happen* recorded in 6452.

6. The fact it was an issue was of course also brought up to the Unicode Consortium as I am (handy enough) also the liaison from IETF to Unicode Consortium. No response. They do push, as we know, TR#46 which is a different kind of animal than IDNA2008. Oh well. Things where fine, and we moved on. And IETF could focus on what IETF really should focus on, to ensure the algorithm itself was still "ok". I.e. without looking at individual code points, but more in general terms, was IDNA2008 still good enough? Lots of good feedback from people like Asmus that was fighting like mad (and still is) trying to a. convince registries they MUST come up with a subset of IDNA2008 permissible code points when they decide what can be used in their zones and b. come up with processes and rules for how that work should be done. Specifically work did go on for a long time regarding what code points can be used in the root zone, i.e. for TLDs. This is managed in the IAB document, and the Klensin draft about and more.

6. Now in version 11.0.0, the non-backward-compatible change happened again. Sigh. I was at that time on my way to just give up, and recommend IETF that what we do today, to reference something that we thought was normative, stable etc (i.e. the Unicode Standard) did not work. That IETF instead would pick up and use some relationship with ISO that should be formalized, and instead of referencing the Unicode Consortium IETF should reference ISO, and then ISO should have in its rules that backward compatibility would be a requirement. If then Unicode did ship something to ISO for "approval", then ISO would say "no, try again" or else ISO would make a process violation that members could appeal within ISO. But after sleeping on this for a bit, I felt the issue with 11.0.0 also happened with 7.0.0 and maybe one should first have a discussion within IETF again before the bolts are blown regarding relationship with Unicode Consortium. That just started when the I18N directorate was closed, and I ended up in a void. After long time, pushing and what not (lets discuss in the bar) this group finally was created were this VERY SERIOUS ISSUE could be discussed, and as John explained, there are many things to discuss, where we can boil it down to two:

A. Should the non-backward compatible change in 11.0.0 result in a change in rule [G] in RFC 5892, or should we accept a non-backward compatible change? To trigger the discussing, I proposed the same result as for 7.0.0, to NOT update [G], and simply accept it. This also because the next issue is the important one.

B. What should we do with IDNA2008? Obviously Unicode Standard is not stable enough. Or is it? What should we do with review? Should we have to start do what Martin just did with 12.0.0? Do we IETF have the expertise? Can we rely on individuals like Martin, Asmus, myself and John and very few more to be around? Can we rely on Unicode Consortium? Is this the time to instead move to ISO? Like keeping IDNA2008 algorithm BUT tie it to ISO approved charset, and then ask ISO to protect the backward compatibility? Or are we done, so that we simply freeze IDN to a specific Unicode version, and simply ignore all added code points after that version? I.e. at 5.2.0 we knew some interesting code points where still to be added, but after 11.0.0?

What I now know is that IESG have told me that we do have agreement in IETF on draft-faltstrom-unicode11-08.txt which implies moving to IDNA 11.0.0 without adding things to [G] is the path forward for now.

What to do with 12.0.0 and future versions is still up in the air.

See (some of) you in Prague.

See you in the bar...

   Patrik

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 19 Mar 2019 00:55:52 -0700

On 3/18/2019 11:47 PM, Patrik Fältström wrote:
> 6. Now in version 11.0.0, the non-backward-compatible change happened again. Sigh.

I think its worth injecting some pragmatism here.

We know that only about 35K code points out of 130K total are actually 
used in someones orthography, where that someone is a speaker of a 
language used for daily activities and with good intergenerational 
transmission (= not likely to die out soon).

All the other 100K code points are either symbols or punctuation or, 
more often, code points that are recognized only by scholars.

Very little activity in Unicode affects that first set of 35K code 
points: future changes may be possible due to orthography reforms, as 
well as some population acquiring widespread literacy for the first 
time, but other than that, there's not much going on that needs new code 
points. There's the occasional case where the properties for some of the 
lesser used code points in this set may turn out to require corrections 
as result of software implementations for these becoming more mainstream 
and thus discovering issues. Generally, these affect languages where 
digital deployment itself is still nascent; and only a small subset 
potentially affects IDNA category.

The upshot is that the practical effect of such activity in the margins 
is effectively invisible.

The overwhelming part of the additions and other changes are within the 
remaining set of 100K specialized characters. These code points are 
either misused for malicious registrations, relying on users' 
unfamiliarity with these rare code points or of use only to specialized 
audiences.

The practical effect of any incompatibilities in that set is, again, 
effectively minuscule; even given the realistic expectation that the 
property data for these won't get rigorously tested in practical 
applications and therefore could be found incorrect at some future point.

>   I was at that time on my way to just give up, and recommend IETF that what we do today, to reference something that we thought was normative, stable etc (i.e. the Unicode Standard) did not work. That IETF instead would pick up and use some relationship with ISO that should be formalized, and instead of referencing the Unicode Consortium IETF should reference ISO, and then ISO should have in its rules that backward compatibility would be a requirement.

This is a complete nonstarter.

Nobody wins if we encourage the existence of two different encoding 
standards - and the current process of cooperation works precisely 
because it doesn't work the way it's sketched below.

> If then Unicode did ship something to ISO for "approval", then ISO would say "no, try again" or else ISO would make a process violation that members could appeal within ISO. But after sleeping on this for a bit, I felt the issue with 11.0.0 also happened with 7.0.0 and maybe one should first have a discussion within IETF again before the bolts are blown regarding relationship with Unicode Consortium. That just started when the I18N directorate was closed, and I ended up in a void. After long time, pushing and what not (lets discuss in the bar) this group finally was created were this VERY SERIOUS ISSUE could be discussed, and as John explained, there are many things to discuss, where we can boil it down to two:
>
> B. What should we do with IDNA2008? Obviously Unicode Standard is not stable enough. Or is it? What should we do with review? Should we have to start do what Martin just did with 12.0.0?

what Martin did with Unicode 12.0.0 is what I call a sanity check. I 
think it's a reasonable activity (if resources exist), but I would not 
expect to find any issues that are of enough practical importance to 
actually matter. (See above).

>   Do we IETF have the expertise? Can we rely on individuals like Martin, Asmus, myself and John and very few more to be around? Can we rely on Unicode Consortium? Is this the time to instead move to ISO? Like keeping IDNA2008 algorithm BUT tie it to ISO approved charset, and then ask ISO to protect the backward compatibility? Or are we done, so that we simply freeze IDN to a specific Unicode version, and simply ignore all added code points after that version? I.e. at 5.2.0 we knew some interesting code points where still to be added, but after 11.0.0?
>

I believe the correct answer is to have a very serious discussion on how 
to help registries to make sure their support for the widely needed 35K 
is more robust and not to get sidetracked by the vast wasteland of 
obsolescent, obsolete, ancient or otherwise specialized use code points 
(whether for liturgical, phonetic, or poetic usage).

Continue to keep a watchful eye over the application of the algorithm 
for future versions, but don't expect that any cases will rise to the 
threshold where any exceptions need to be baked into the protocol - with 
all the attendant issues for updating software libraries.

But focus on solving issues that have at least practical impact.

A./

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 19 Mar 2019 10:19:34 +0000

Hello Asmus, Patrik,

On 2019/03/19 16:55, Asmus Freytag wrote:
>> 6. The fact it was an issue was of course also brought up to the 
>> Unicode Consortium as I am (handy enough) also the liaison from IETF 
>> to Unicode Consortium. No response. They do push, as we know, TR#46 
>> which is a different kind of animal than IDNA2008. Oh well. Things 
>> where fine, and we moved on. And IETF could focus on what IETF really 
>> should focus on, to ensure the algorithm itself was still "ok". I.e. 
>> without looking at individual code points, but more in general terms, 
>> was IDNA2008 still good enough? Lots of good feedback from people like 
>> Asmus that was fighting like mad (and still is) trying to a. convince 
>> registries they MUST come up with a subset of IDNA2008 permissible 
>> code points when they decide what can be used in their zones and b. 
>> come up with processes and rules for how that work should be done. 
>> Specifically work did go on for a long time regarding what code points 
>> can be used in the root zone, i.e. for TLDs. This is managed in the 
>> IAB document, and the Klensin draft about and more.
>>

I mostly agree with Asmus here. I think it's also worth looking at the 
actual cases. Looking at 
https://tools.ietf.org/html/draft-faltstrom-unicode11-08#section-4.3, in 
the 11.0.0 timeframe, we have a single previous character that changed 
from DISALLOWED to PVALID. See also 
https://tools.ietf.org/html/draft-faltstrom-unicode11-08#section-5, 
which essentially says that there's not much of a problem either way.

In the 6.0.0 timeframe, there were both more actual changes (3), and 
there was one change where we moved from PVALID to DISALLOWED, which I 
personally think is the tougher one (see 
https://tools.ietf.org/html/rfc6452#section-1.3). And looking at the 
code chart at https://www.unicode.org/charts/PDF/U1980.pdf, one may even 
reasonably argue that keeping U+19DA NEW TAI LUE THAM DIGIT ONE would 
have been the better choice, because when one compares U+19B1 NEW TAI 
LUE VOWEL SIGN AA and U+19D1 NEW TAI LUE DIGIT ONE and finds them having 
(at least on first approximation) the same glyph shape, one can easily 
guess that U+19DA NEW TAI LUE THAM _DIGIT_ ONE is used (this is only a 
quick conjection which would have to be confirmed) exactly in situations 
where the distinction between letters and digits is important, of which 
we know domain names are a good example.

>>   I was at that time on my way to just give up, and recommend IETF 
>> that what we do today, to reference something that we thought was 
>> normative, stable etc (i.e. the Unicode Standard) did not work. That 
>> IETF instead would pick up and use some relationship with ISO that 
>> should be formalized, and instead of referencing the Unicode 
>> Consortium IETF should reference ISO, and then ISO should have in its 
>> rules that backward compatibility would be a requirement.
> 
> 

I completely agree. First, ISO mostly delegates properties to Unicode. 
Second, Unicode is where the feedback from implementers and users is 
mostly coming in, and it's that feedback that leads to corrections. 
Third, while in ISO there's a chance that they would leave some 
properties untouched just because they don't understand them or are not 
interested, there's also the chance that they would tweak something if 
for whatever reason one national body or the other (or to be precise, 
the representative of that national body) insists on a change.

>> If then Unicode did ship something to ISO for "approval", then ISO 
>> would say "no, try again" or else ISO would make a process violation 
>> that members could appeal within ISO. But after sleeping on this for a 
>> bit, I felt the issue with 11.0.0 also happened with 7.0.0 and maybe 
>> one should first have a discussion within IETF again before the bolts 
>> are blown regarding relationship with Unicode Consortium. That just 
>> started when the I18N directorate was closed, and I ended up in a 
>> void. After long time, pushing and what not (lets discuss in the bar) 
>> this group finally was created were this VERY SERIOUS ISSUE could be 
>> discussed, and as John explained, there are many things to discuss, 
>> where we can boil it down to two:
>>
>> A. Should the non-backward compatible change in 11.0.0 result in a 
>> change in rule [G] in RFC 5892, or should we accept a non-backward 
>> compatible change? To trigger the discussing, I proposed the same 
>> result as for 7.0.0, to NOT update [G], and simply accept it. This 
>> also because the next issue is the important one.
>>
>> B. What should we do with IDNA2008? Obviously Unicode Standard is not 
>> stable enough. Or is it? What should we do with review? Should we have 
>> to start do what Martin just did with 12.0.0?
> 
>>   Do we IETF have the expertise? Can we rely on individuals like 
>> Martin, Asmus, myself and John and very few more to be around? Can we 
>> rely on Unicode Consortium? Is this the time to instead move to ISO? 
>> Like keeping IDNA2008 algorithm BUT tie it to ISO approved charset, 
>> and then ask ISO to protect the backward compatibility?

If we needed that, we could simply also use the rule [G] override. It 
might be a lot cheaper both for us and for others.

While at it, I'd like to discuss some assumptions about implementations. 
https://tools.ietf.org/html/draft-faltstrom-unicode11-08#section-5 says:
                 As including an exception would require implementation
    changes in deployed implementations of IDNA20008, the editor proposes
    that such a BackwardCompatible rule NOT to be added to IDNA2008.

This seems to assume the following implementation strategy: The code of 
an IDNA library is version-agnostic, relies on some lower layer for 
providing property information, and this lower layer gets updated 
automa[c/t]ically when new Unicode information is published. That would 
e.g. be the case with an implementation in pure Ruby that used \p{} 
regular expression properties (the way I did it in my script). Such an 
implementation would not need any updates except the updates to Ruby 
itself when the Unicode version changes and there are no additions to 
the rule [G] set.

While it is my understanding that e.g. fonts for emoji in new Unicode 
versions get deployed to many OSes pretty quickly, IDNA implementations 
(I just looked at https://gitlab.com/libidn/libidn2/) seem to work 
differently. They assume that they are on their own with respect to 
properties, and compile in all the property data they need (which may be 
just the end values of PVALID/DISALLOWED/...). That means that for them 
to update to a new Unicode version means to adjust the source code and 
to recompile. With such a strategy, adding a codepoint or two into the 
rule [G] set would be rather easy and trivial.

It might be good to inspect some of the implementations we know about to 
see which strategy they follow.

>> Or are we 
>> done, so that we simply freeze IDN to a specific Unicode version, and 
>> simply ignore all added code points after that version? I.e. at 5.2.0 
>> we knew some interesting code points where still to be added, but 
>> after 11.0.0?
>>
>> What I now know is that IESG have told me that we do have agreement in 
>> IETF on draft-faltstrom-unicode11-08.txt which implies moving to IDNA 
>> 11.0.0 without adding things to [G] is the path forward for now.
>>

I fully agree with Asmus here.

Regards,   Martin.

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 19 Mar 2019 09:24:16 -0700

On 3/19/2019 3:19 AM, Martin J. Dürst wrote:
> code chart athttps://www.unicode.org/charts/PDF/U1980.pdf, one may even
> guess that U+19DA NEW TAI LUE THAM_DIGIT_  ONE is used (this is only a

Here's a telling example of how a formal problem can be practically 
irrelevant:

I looked up New Tai Lue on Omniglot:

"The new script is used exclusively in Jinghong, so could be called the
   New Jinghong Tai Lue script, and is used for shop and street signs.
*Fe**w people can read it*. "   (Emphasis mine).

If you read the full text you get the impression that the whole script
represents a failed state-imposed "simplification". Unicode still needs
to encode it, so that archival texts from the period between 1950 and 1980
can be encoded, but it's surely of no practical relevance for domain names.

A./

---

From: Michel Suignard <[email-masked]>
Date: Wed, 20 Mar 2019 00:11:28 +0000

To add some here (I happen to be the project editor for ISO/IEC 10646, the ISO ‘equivalent’ of Unicode and also de facto liaison from Unicode to IETF):
ISO/IEC 10646 has been for a long time using a subset of Unicode properties by reference, and that includes the General Category (Gc) which is used by IDNA2008, same for normalization forms (NFC, NFD, NFKC, NFKD). There is zero appetite (and frankly no resource) in the ISO side to change that situation.

Concerning stability guarantee I can assure every one in this list that Unicode is way more concerned about stability guarantee than ISO. So far the only stability guaranteed in 10646 are the character names and the non-deletion of code points.

To the best of my knowledge there has been a single change: U+19DA NEW TAI LUE THAM DIGIT ONE that is problematic because it goes from PVALID to DISALLOWED, all others are from DISALLOWED to PVALID (which is not that different from the common change from UNASSIGNED to a mix of PVALID/DISALLOWED for new repertoires based on their Gc value and NFC behavior which has happened for 10,000s code points since Unicode 5.2).

Where Category G should contain that single case (or any similar case) is open to debate which we can entertain. I don’t think the other cases deserve it.

The point concerning modern scripts is well taken as well. Typically Gc changes result of a reinterpretation of a writing system because it was not well understood. This is unlikely to happen for a modern use script. There are some tension on NFC stability (recent case in Bengali), but most interested parties in Unicode and IETF know better than giving up on that kind of cases.

There is one case where a somewhat ‘modern’ script is currently under revision. That is the Mongolian script, but you would have to be a fool to create Mongolian IDNA labels at this point, even if there are many Mongolian PVALID characters and they appear in IDNA tables published by IANA. Therefore, there is a non-zero chance that Mongolian code points could show up in Category G in the future. But Mongolian has so many confusability issues that it is unlikely that it could be used for IDNA labels ever.

Finally there is some hope that the Unicode interpretation of IDNA (UTS #46) may get better aligned with IDNA 2008 and RFC5892. I have been considering putting some resources into that.

Michel

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 20 Mar 2019 04:39:52 +0000

Hello Asmus,

> can be encoded, but it's surely of no practical relevance for domain names.

Interestingly enough, their page (at 
https://omniglot.com/writing/tailue.htm) shows the U+19DA NEW TAI LUE 
_THAM_ DIGIT ONE glyph rather than the U+19D1 NEW TAI LUE DIGIT ONE 
glyph in the Numerals section.

Regards,   Martin.

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 20 Mar 2019 05:19:41 +0000

Sorry I forgot to mention it, but I think the change of the Unicode 
property for U+19DA NEW TAI LUE THAM DIGIT ONE happened because the 
General_Category value "Nd" was tightened to only apply to consecutive 
decimal digits that start with zero. See 
https://www.unicode.org/versions/Unicode12.0.0/ch04.pdf, page 175:

The Numeric_Type = Decimal property value (which is correlated with the 
General_Category = Nd property value) is limited to those numeric 
characters that are used in decimal-radix numbers and for which a full 
set of digits has been encoded in a contiguous range, with ascending 
order of Numeric_Value, and with the digit zero as the first code point 
in the range.

Either the tightening must have happened in the 6.0.0 timeframe, or 
U+19DA NEW TAI LUE THAM DIGIT ONE was originally classified wrongly.

Anyway, the DNS doesn't need decimal digits to be in a contiguous block 
starting with zero.

Regards,   Martin.

On 2019/03/20 13:39, Martin J. Dürst wrote:
> _______________________________________________
> IDNA-UPDATE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idna-update
>

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 20 Mar 2019 05:35:28 +0000

Hello Michel,

Glad to hear from you.

On 2019/03/20 09:11, Michel Suignard wrote:
> Where Category G should contain that single case (or any similar case) is open to debate which we can entertain.

Yes, we could do that. But the decision not to do it is over 8 years old 
now, and I haven't heard any actual complaint from an interested user 
about it. We are just discussing it here as part of the overall 
discussion of moving on with updating IDNA to newer Unicode versions.

> I don’t think the other cases deserve it.
> 

One attempt at understanding the Mongolian script I once made was to 
describe it as "imagine to write English with IPA codepoints, but to 
display it the way we see it every day, adding some variant selectors". 
I was quickly told by an expert that it's even more difficult.

> Finally there is some hope that the Unicode interpretation of IDNA (UTS #46) may get better aligned with IDNA 2008 and RFC5892. I have been considering putting some resources into that.

That is very good to hear. Any details worth mentioning? Anything that 
would help?

Regards,   Martin.

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 19 Mar 2019 23:22:11 -0700

Martin,

The property gc=Nd is used by software identifying strings of characters 
that can be parsed as decimal numbers using offsets. That was implicit 
in the initial design but not made explicit until the first exceptions 
started to show up with new characters.

It is indeed the case that this new code point would have made a fine 
one to use in domain names. Possibly better than the 'standard' digit 
one that is used in running numbers, because that looks like a letter (a 
common issue for digits, just think of l and 1 and o and 0, but here the 
similarities are far more striking).

However, the decision to omit this code point isn't the end of the world 
for domain names. Given what is available today as information about 
usage, any practical impact may be rather limited; the population is 
reported highly literate, but the sources admit that this may reflect 
literacy in the dominant surrounding language. And an unknown and by 
some indication growing fraction writes the old script.

One thing that I find is that the information available for many scripts 
with at least some modern use is generally getting better long after 
they have been encoded. This should be a useful reminder that striving 
for absolute perfection here is a fool's errand.

A./