Subject: [I18ndir] Request for review: draft-faltstrom-unicode11-07

From: Alexey Melnikov <[email-masked]>
Date: Sun, 03 Feb 2019 12:59:15 +0000

Dear I18N Directorate,

I will be sending this document to IETF Last Call shortly, so I would like to request a review. In particular, I would appreciate:

1) speedy review of the document, due to Unicode Consortium intent to publish Unicode 12 in March 2019 (*).
2) comments on technical content of the document, separated into major, minor and nit categories.
3) comments on whether this document should be Proposed Standard or Informational.
4) comments on dependencies between this document and others.

Thank you,
Alexey, as an ART AD.

(*) - I appreciate that finishing discussions by March doesn't give much time, but this document is already delayed by 2 months.

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 03 Feb 2019 16:34:57 +0100

As the editor of the document I want to point out the mechanisms of the document explicitly.

1. It identifies a few incompatibilities in the Unicode Standard that have been made between the versions. Including one already identified by IAB.

2. For each one of the incompatibilities two choices can be made:

2a. Add one or more exceptions to the IDNA2008 standard (i.e. updating it with new rules)

2b. Add zero exceptions and acknowledge that the incompatibilities have been made (i.e. updating it without new rules).

[2b] is the choice IETF did last time, RFC 6452.

[2b] is what this document suggests.

3. The IAB statement on code points <https://www.iab.org/documents/correspondence-reports-documents/2015-2/iab-statement-on-identifiers-and-unicode-7-0-0/> is a larger issue which can not easily be solved by "just" adding some exceptions. One can also see in for example <https://www.alvestrand.no/pipermail/idna-update/2015-February/007911.html> that the views on the overall problem (if it is a problem or not) is split. My view is that the over arching issue here to some degree have to do with "trust" between IETF and Unicode Consortium and whether IETF should continue to use various meta data in the Unicode Standard for calculating what code points are valid or not. Alternatives could be to just let Unicode Consortium do it, or for IETF to do code point selection code point by code point. The latter IETF tried in IETF 2003, and we decided then that was a stupid idea and came up with IDNA2008.

To conclude, as an editor of IDNA2008 I suggest [2b], i.e. no changes to IDNA2008 even though there are incompatibilities, and I think [3] is a larger discussion that for example should be held here.

IF, but only IF, that is the consensus of the IETF, then this draft is what should be an RFC. If any of the questions I list above is answered differently, then the draft is the wrong thing and should contain something else.

I.e. I wrote it that way to trigger some discussion as I did not feel after getting some letters from IAB that ultimately "is my boss" in my role as expert reviewer of the IDNA2008 tables at IANA I could "just" ask IETF what to do. I needed something that "is a proposal".

Here it is!

And I am happy to answer questions!

   Patrik

On 3 Feb 2019, at 13:59, Alexey Melnikov wrote:

> -- 
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 03 Feb 2019 16:40:06 +0100

On 3 Feb 2019, at 16:34, Patrik Fältström wrote:

> My view is that the over arching issue here to some degree have to do with "trust" between IETF and Unicode Consortium and whether IETF should continue to use various meta data in the Unicode Standard for calculating what code points are valid or not. Alternatives could be to just let Unicode Consortium do it, or for IETF to do code point selection code point by code point. The latter IETF tried in IETF 2003, and we decided then that was a stupid idea and came up with IDNA2008.

IETF tried it in IDNA2003...of course.

   Patrik

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Thu, 21 Feb 2019 19:50:39 +0000

I've completed my review of draft-faltstrom-unicode11 from the perspective of a domain name registry operator. In that context, I believe the document is "Ready with Nits". The nits I noted were minor issues with grammar and/or phrasing that'll be sorted out by the RFC Editor, so I don't see a lot of value in calling them out individually.

The Conclusion and Security Considerations sections contain useful information that domain registries and registrars will need to fully understand to guide implementation and deployment decisions. I also appreciated the last sentence of the Introduction ("The current document resolves this issue and suggests that the IDNA2008 standard followsO  (SAH: there's a nit) the Unicode Standard and not update RFC 5892 [RFC5892] or any other IDNA2008 RFCs") because it provides an up-front executive summary of what an implementer needs to know at a high level.

Scott

---

From: John C Klensin <[email-masked]>
Date: Sat, 23 Feb 2019 21:41:20 -0500

Scott,

Two questions that may affect, not the utility of your report
but how it is interpreted in context:

(1) I see no trace in the report of any of a recent discussions
of whether draft-faltstrom-unicode11-07 can be construed as
accepting problems that the IETF has been aware of for years,
either treating those problems as of no particular interest or
as issues on which registries should be warned and aware but, in
either event, as things for which there is no justification for
adjusting IDNA2008 and its lists or rules now or in the future.
Was it your intent to indicate that the document is ok at is and
that those issues are either of no significance or at least do
not need to be reflected in the document?

(2) You use the term "domain name registry operator" below.  To
be sure we are talking about the same thing, I tend to use that
term in the context used in RFC 1591, i.e., applying to every
entity that places names into the DNS at any level with the
possible exception of the root.   Is that your intent as well,
or were you referring to such subsets as "TLD registry
operator", "gTLD registry operator", what ICANN calls a
"contracted party", or some combination of them or others?

I'll let Pete, Peter, and Harald address the question of whether
the directorate is expected to submit one report based on a
discussion to which I think this is your first contribution or
whether Harald is preparing a summary and we are each expected
to write our own report or additions.  

best,
    john

--On Thursday, February 21, 2019 19:50 +0000 "Hollenbeck, Scott"
<shollenbeck=[email-masked]> wrote:

> I've completed my review of draft-faltstrom-unicode11 from the
> perspective of a domain name registry operator. In that
> context, I believe the document is "Ready with Nits". The nits
> I noted were minor issues with grammar and/or phrasing that'll
> be sorted out by the RFC Editor, so I don't see a lot of value
> in calling them out individually.
> 
> The Conclusion and Security Considerations sections contain
> useful information that domain registries and registrars will
> need to fully understand to guide implementation and
> deployment decisions. I also appreciated the last sentence of
> the Introduction ("The current document resolves this issue
> and suggests that the IDNA2008 standard followsO  (SAH:
> there's a nit) the Unicode Standard and not update RFC 5892
> [RFC5892] or any other IDNA2008 RFCs") because it provides an
> up-front executive summary of what an implementer needs to
> know at a high level.
> 
> Scott

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 25 Feb 2019 09:04:02 +0000

Hello everybody,

I'm responding to this mail because it's the last in the thread that 
arrived here, and because it's reasonably short.

I'm also including my detailed review.

> not need to be reflected in the document?

As to what our draft should do, I am very much of the opinion that it 
should say that it does not warrant any adjustments to the rules of 
IDNA2008 at this moment.

But I think it should not say (or give the impression) that such changes 
(or even more wide-ranging changes) are ruled out in the future.

>> in calling them out individually.

I agree with the "Nits", but I think it's better to sort out most of 
them as soon as we are aware of them to avoid confusion later. And some 
may be a bit more than just nits, depending on how you look at them. So 
here we go:

Abstract, second para:

"Some changes have been made in the Unicode Standard related to the
    algorithm IDNA2008 specifies."
->
"Some additions and changes have been made in the Unicode Standard that 
affect the algorithm IDNA2008 specifies."
[it's mostly additions, very few changes]

"IDNA2008 allows adding exceptions to
    the algorithm for backward compatibility; however, this document 
makes no such changes"
->
"IDNA2008 allows adding exceptions to
    the algorithm for backward compatibility; hovever, this document not 
add any such exceptions"
[in 'such changes', the referent may be unclear]

"Thus this document requests that IANA update
    the tables to Unicode 11."
->
"This document provides the necessary tables to IANA for the update to 
Unicode 11."
[This document would request that IANA update the tables even if we 
decided we needed some exceptions.]

"The document also recomments that all DNS registries continue the
    practice of calculating a repertoire using conservatism and inclusion
    principles."
->
"The document also recommen*d*s that all DNS registries continue the
    practice of defining their repertoires using conservatism and inclusion
    principles."
[Align to plural for 'registries' and 'repertoires'; avoid 
'calculating', because it suggests this can mostly be done 
algorithmically, without judgement, which isn't the impression I think 
we should give.]

1. Introduction, first para:

'was largely completed in 2008, and is thus known within as "IDNA2008"'
->
'was started in 2008, and is thus known as "IDNA2008"'
[within what?; "largely completed" already noted, more correct wordings 
appreciated]

"It is specified in a series of documents
    listed in Section 3.1."
->
"It is specified in the series of documents
    listed in Section 3.1."

1. Introduction, first 'o' bullet:

      "Assigning code points can create problems if the newly-assigned
       code points are compositions of code points changes (or would have
       changed) the normalization functions.  These problems can arise if
       the new code points change the matching algorithms used and this
       in turn creates problems looking up already stored strings."
[I think I know what this paragraph wants to get at, but I have problems 
parsing the text. For example, should it be "compositions of code point 
changes" or "compositions of code points that change" or what else? 
Also, what matching algorithm? As far as I'm aware, matching of labels 
is done by comparing A-Labels in the DNS.]

1. Introduction, second 'o' bullet:

"a code point that
       was not allowed (and thus is blocked in some situations) to
       suddenly end up being allowed."
->
"a code point that
       was not allowed (and thus is blocked in some situations) can
       suddenly end up being allowed."

1. Introduction, para following bullet list:

   "Historically, the IETF has accepted all implications of changes in
    the Unicode Standard even though the changes have resulted in
    problematic changes in the derived property value."
->
   "Historically, the IETF has accepted all implications of changes in
    the Unicode Standard even though the changes have resulted in
    minor changes in the derived property values."
[Let's not call these 'problematic' if we actually accepted them. If we 
really have to, "potentially problematic" might be the furthest I'd be 
willing to support.]

1. Introduction, second-to-last para:

"there is not a problem" -> "there is no problem"

"This document concludes that no
    exceptions are to be added to IDNA2008 even if changes in the derived
    property value is a result of the changes made in Unicode between
    version 6.3.0 and 11.0.0."
->
"This document concludes that no
    exceptions are to be added to IDNA2008 even though there are changes 
in the derived
    property values as a result of the changes made in Unicode between
    version 6.3.0 and 11.0.0."
[We know whether there are changes in the derived property values or 
not, so "if" isn't appropriate.

1. Introduction, last para:

"The current document
    resolves this issue and suggests that the IDNA2008 standard followsO
    the Unicode Standard and not update RFC 5892 [RFC5892] or any other
    IDNA2008 RFCs."
->
"The current document
    resolves this issue and suggests that the IDNA2008 standard follows
    the Unicode Standard and does not update RFC 5892 [RFC5892] or any other
    IDNA2008 RFCs."
[Actually, we could also address the "ARABIC LETTER BEH WITH HAMZA 
ABOVE" by adding an exception for this letter, rather than changing RFC 
5892, but we are not doing that, either. Maybe we should say so.]

Section 3.1, RFC 5892:

"an is based originally on Unicode 5.2.0"
->
"and is based on Unicode 5.2.0"
[It's still based on Unicode 5.2.0, and that will never change, so 
'originally' seems superfluous.]

"The Unicode-based rules in RFC 4892"
->
"The Unicode-based rules in RFC 5892"
[RFC 4892 happens to be about the DNS, but not what we want.]

"RFC 5892
       obsoletes RFC 3491 [RFC3491], and in particular the use of the
       tables to which it refers."
->
"RFC 5892
       obsoletes RFC 3491 [RFC3491], and in particular the use of the
       tables to which RFC 3491 refers."
[Avoid unclear referent.]

Section 3.1, RFC 5895:
     " RFC 5895 [RFC5895], informally called "Mapping", discusses the
       issue of mapping characters into other characters and that
       provides guidance for doing so when that is appropriate.  RFC 5895
       provides advice and is not a required part of IDNA."
->
      "RFC 5895 [RFC5895], informally called "Mapping", discusses the
       issue of mapping characters to other characters and
       provides guidance for doing so when that is appropriate.  RFC 5895
       provides advice only and is not a required part of IDNA."

Section 3.1, RFC 6452:
"that resulted in the derived property value change
       for the code points U+0CF1, U+0CF2 and U+19DA"
->
"that resulted in derived property value changes
       for the code points U+0CF1, U+0CF2 and U+19DA"

"The IETF concluded that no update to RFC
       5892 [RFC5892] was needed based on the changes made in Unicode
       6.0.0 [Unicode-6.0.0]."
[There was also no additional exception, and this should probably be 
mentioned here.]

"As a result, the derived property value
       remained aligned with the Unicode Standard."
->
"As a result, the derived property values
       remained aligned with the Unicode Standard."

Section 3.2, first 'o' bullet:
"IDNA2003 as specified in RFC 3490 [RFC3490] and RFC 3491 [RFC3491]
       which implies using a table within which it is said whether code
       points are allowed to be used or not, after doing the
       normalization specified in IDNA2003."
->
"IDNA2003 as specified in RFC 3490 [RFC3490] and RFC 3491 [RFC3491].
       This implies using a table that indicates whether code
       points are allowed to be used or not, after doing the
       mapping specified in IDNA2003."
[ToASCII as specified in RFC 3490 can be seen as a kind of normalization 
in the wider sense, but when Unicode is involved, I'd try to avoid this 
term unless it refers to things like NFC, NFD, NFKC,...]

Section 3.2, first para after bullet list:

"create a subset of these
    by assigning the PVALID derived property value to them."
->
"create a subset consisting of the codepoints
    that are assigned the PVALID derived property value."
[Should we talk about contextual rules here?]

Section 4.1, third paragraph:

"The character U+08A1 (ARABIC LETTER BEH WITH HAMZA ABOVE) can be 
represented with the
    sequence ARABIC LETTER BEH (U+0628) and ARABIC HAMZA ABOVE (U+0654)."
->
"The character U+08A1 can be represented with the
    sequence ARABIC LETTER BEH (U+0628) and ARABIC HAMZA ABOVE (U+0654)."
[Better give the name again, makes it easier to understand for outsiders 
(unless we explicitly want this to be under the radar, but I guess it's 
way too late for that).]

"This identical to LATIN SMALL LETTER A WITH DIAERESIS (U+00E4), that
    can be represented"
->
"This identical to LATIN SMALL LETTER A WITH DIAERESIS (U+00E4), which
    can be represented"

"One difference between
    these two sequences is how they are treated in the normalization
    forms specified by the Unicode Consortium."
->
"The important difference between
    these two sequences is how they are treated in the normalization
    forms specified by the Unicode Consortium."

4.1, last para:

"U+08A1 is discussed in draft-freytag-troublesome-characters
    [I-D.freytag-troublesome-characters] and other Internet-Drafts."
["and other Internet-Drafts" does not give any information whatsoever. 
Either remove it or provide actual references.]

"Regardless of whether the discussion of those drafts ends in
    recommendations to include the code point in the repertoire of
    characters permissable for registration or not, it is still
    acceptable to allow the code point to have a derived property value
    of PVALID."
->
"Regardless of whether the code point U+08A1 should be permitted for
registration in any specific case, it is still
    acceptable to allow the code point to have a derived property value
    of PVALID."
[I think this is a bit more open. In practice, I'd expect most serious 
registries to disallow this codepoint, but would expect registries 
supporting the Adamawa Fulfulde language in Cameroon to allow it, making 
sure there's no confusion with the decomposed version.]

Section 4.2:

"There are no changes made to Unicode between version 7.0.0 and 10.0.0
    that impact IDNA2008 calculation of the derived property value."
->
"There are no changes made to Unicode between version 7.0.0 and 10.0.0
    that impact the IDNA2008 calculation of derived property values."

Section 4.3, first para:
"The Unicode Standard Version 11.0.0 [Unicode-11.0.0] has included a
    number of changes [Changes-11.0.0] from version 10.0.0."
->
"The Unicode Standard Version 11.0.0 [Unicode-11.0.0] includes a
    number of changes [Changes-11.0.0] from version 10.0.0.
    Those relevant for IDNA2008 are listed below."

Section 4.2, first 'o' bullet of second bullet list:

"o  The newly assigned 684 characters are to have a derived property
       value as of a result of applying the IDNA2008 algorithm."
->
"o  The newly assigned 684 characters are assigned a derived property
       value as a result of applying the IDNA2008 algorithm."

Section 4.2, second 'o' bullet of second bullet list:

Add something like "The newly encoded Mtavruli letters have general 
category "Lu" and are therefore DISALLOWED."

Section 5, last para:

"All DNS registries (and other organizatios) SHOULD calculate a
    repertoire using the conservatism and inclusion principles, as
    described in SAC-084 [SAC-084] and similar documents."
[Again, I think "and similar documents" is not helpful. I'd point to UTR 
36 (Unicode Security Considerations) and UTR 39 (Unicode Security 
Mechanisms). These references might also go into the Security 
Considerations section.]

Section 7:

"This document makes recommendations regarding the use of the IDNA2008
    algorithm for calculation of derived property values, based on the
    current Unicode version."
->
"This document makes recommendations regarding the use of the IDNA2008
    algorithm for calculation of derived property values, based on 
Unicode version 11.0.0."
[Be specific about version. Makes it easier for people to read this 
document years later.]

"It also recommends that DNS registries (and
    others dealing with Internationalized Domain Names) explicitly select
    appropriate subsets of characters with the derived value of PVALID."
->
"It also recommends that DNS registries (and
    others dealing with Internationalized Domain Names) explicitly select
    appropriate subsets of the characters with the derived value of PVALID."

Again, I didn't check the tables. I might be able to do so (by writing 
my own program) in a week or two, but I understand this will be too late.

I seem to remember that Patrik wrote the program for generating the 
tables in Ruby. If that's so, I'd be glad to review it if I get a pointer.

Hope this helps.   Regards,   Martin.

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Mon, 25 Feb 2019 12:34:31 +0000

> -----Original Message-----
> From: John C Klensin <[email-masked]>
> Sent: Saturday, February 23, 2019 9:41 PM
> To: Hollenbeck, Scott <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: [EXTERNAL] Re: [I18ndir] Request for review: draft-faltstrom-
> unicode11-07
>
> Scott,
>
> Two questions that may affect, not the utility of your report but how it is
> interpreted in context:
>
> (1) I see no trace in the report of any of a recent discussions of whether
> draft-faltstrom-unicode11-07 can be construed as accepting problems that
> the IETF has been aware of for years, either treating those problems as of no
> particular interest or as issues on which registries should be warned and
> aware but, in either event, as things for which there is no justification for
> Was it your intent to indicate that the document is ok at is and that those
> issues are either of no significance or at least do not need to be reflected in
> the document?

No, that's not my intent. My intention was to note that gTLD registry operators (see below) are ultimately looking for clear guidance. The document provides that guidance. The issues you mentioned have been described already, and the document (and guidance) may change as a result of whatever is done to address them. In the end, I still need clear guidance.

> (2) You use the term "domain name registry operator" below.  To be sure we
> are talking about the same thing, I tend to use that term in the context used
> in RFC 1591, i.e., applying to every entity that places names into the DNS at
> any level with the
> or were you referring to such subsets as "TLD registry operator", "gTLD
> registry operator", what ICANN calls a "contracted party", or some
> combination of them or others?

ccTLD and gTLD registry operator.

Scott

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Feb 2019 00:43:57 -0500

Scott,

Thanks.

One further observation.   I'm fairly sure you know this, but
maybe it is worth trying a different perspective. The desire for
clear (and easy-to-follow) guidance is both clear and
understandable.  That desire isn't limited to registry
operators: different of us will have different experiences and
memories, but I keep being reminded of a long-term IETF
participant whose work has nothing to do with IDNs but who has
stood up in several meetings and made comments about i18n issues
that amounted to "I don't want to understand this stuff, just
give me a simple set of clear rules that I can follow".  For him
(at least) part of "clear guidance" is a list of "safe" code
points that can be safely used in any context and, ideally, a
list of "bad" ones and nothing in between.

The problem is that natural language is messy (or, in different
words, very diverse and with lots of situation-specific rules)
and writing systems are at least as messy. In the US, every
first or second-grader figures out parts of that when they
discover that phonetic spelling rules work except when they
don't.  IDNA2008 actually gives clear guidance and that guidance
is that, while its list of PVALID and CONTEXTx code points set
an upper bound and prohibit everything else, registries are
allowed to register and delegate only those labels whose scripts
and character sequences they actually understand.   While I
think draft-klensin-idna-rfc5891bis is worth finishing and
publishing, I don't think it actually says anything new.   That
said, many of most of us knew, when that rule was written into
IDNA2008, that it was aspirational.  I would have been
astonished if every registry in the DNS -- or even every
registry handling second-level names -- would have interpreted
it as it was intended and been that cautious and conservative
about allowed labels.

Accepting that rule as aspirational immediately raises the
questions of what can or should be done to help registries who
are, quite reasonably, trying to operate in the space between
the limits imposed by IDNA2008's rules and just allowing
whatever those limits permit without thinking about it (or even
going beyond them).  I think it is worth providing some guidance
in that space but I'm confident that it won't be crisp or clear.

The issue that you have seen Asmus and I circling around is part
of this and hinges on how useful things like the "troublesome"
list are.  It clearly provides useful warnings about code points
that need extra attention and caution. From the standpoint of a
registry that is really following the principle of "don't allow
anything you don't fully understand", it adds nothing because,
if the understanding is there, the list is not necessary or even
especially useful (Asmus may well be right that many users of
complex scripts don't understand their own writing systems but I
think trying to fix that leads to madness... YMMD and so may
his).  On the other hand, if a registry wants to (or is
obligated to) register anything that someone "wants to" register
unless they can demonstrate that some harm would likely result,
it -- and assorted label generation rules and general per-script
guidance-- provide good hints about labels about which one
should be particularly wary.  Those kinds of rules will never
cover all such cases.   As a silly and obvious example, I
suggest that allowing "g00g1e", "g00gle", or "micr0s0ft" as
labels, especially given the DNS's ASCII case-equivalence rule,
is just inviting trouble.   They aren't IDN problems, but
similar problems exist with many other scripts.  As far as I
know, they can be blocked automatically in only two ways.  One
is to ban the digits "0" and "1" entirely; the other is to adopt
some variation of a FCFS principle for any collection of labels
with certain substitutions.  I can't imagine either would go
over well.

In any event, the current I-D does not provide clear guidance.
It adds a few (percentage-wise) new code points to the PVALID
list, but it gives you no information about which of those code
points are safe to allow and in what contexts.  Given the nature
of how and in what order scripts have been added to Unicode,
most of the new ones are associated with writing systems that
are not in active use on the Internet by large populations of
people.  Were it not discriminatory, part of a chicken-and-egg
problem, inappropriate for special cases, and otherwise socially
and ethically problematic, it might make sense to ban almost all
of them at the registry level, just as it makes sense for most
registries to ban archaic scripts.

I definitely think that having the IANA tables seriously out of
synch with Unicode versions is a bad idea and a source of
confusion.  However, there is a big distance between that and
clear guidance about what should be permitted and what should
not.   I can tell you how to get to guidance of that type for
second level registrations, but it is not an IETF problem and I
don't believe it would fly politically.

   best,
    john

--On Monday, February 25, 2019 12:34 +0000 "Hollenbeck, Scott"
<[email-masked]> wrote:

>> Subject: [EXTERNAL] Re: [I18ndir] Request for review:
>> draft-faltstrom- unicode11-07
>> 
>> Scott,
>> 
>> (1) I see no trace in the report of any of a recent
>> discussions of whether draft-faltstrom-unicode11-07 can be
>> construed as accepting problems that the IETF has been aware
>> of for years, either treating those problems as of no
>> particular interest or as issues on which registries should
>> be warned and aware but, in either event, as things for which
>> there is no justification for adjusting IDNA2008 and its
>> lists or rules now or in the future. Was it your intent to
>> indicate that the document is ok at is and that those issues
>> are either of no significance or at least do not need to be
>> reflected in the document?
> 
> No, that's not my intent. My intention was to note that gTLD
> registry operators (see below) are ultimately looking for
> clear guidance. The document provides that guidance. The
> issues you mentioned have been described already, and the
> document (and guidance) may change as a result of whatever is
> done to address them. In the end, I still need clear guidance.
> 
>> (2) You use the term "domain name registry operator" below.
>> To be sure we are talking about the same thing, I tend to use
>> that term in the context used in RFC 1591, i.e., applying to
>> every entity that places names into the DNS at any level with
>> the

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 26 Feb 2019 00:44:26 -0800

> don't believe it would fly politically.

The way I see it, you can do four things:

(1) Simply extend IDNA2008 following its own inherent logic, taking into 
account not only the design but also the practice since then. In other 
words pretty much Patrik's draft as is.

(2) wag a finger at registries demanding the impossible, i.e. "only 
register what you understand". It's impossible, because with few 
exceptions, registries do not understand anything when it comes to IDNs, 
and where they do have some understanding, it (generally) breaks down at 
the edge of their own native language: most of them don't even know when 
they are leaving the reservation. This is pretty much what current 
draft-klensin sets out to do.

(3)  provide guidance on the extent of the problem; give a map of the 
reservation. Outline, but don't solve the technical problem. In some 
ways that's what the registry of "troublesome" characters started out 
as, but it's attempt to maintain a data set is too fine grained (and 
duplicates other work). At the same time, it falls short, because 
character selection is not (!) the biggest problem (it may be in the 
Latin script, but the general case needs full-fledged label generation 
*rules* not code point tables). The sketch I sent out today ("Towards a 
guidance on more restrictive LGRs") at least attempts to map the problem 
more systematically - and largely without the need for IETF maintaining 
lists of code points. As a side effect, it makes concrete what a 
conservative design should do: focus on active orthographies that users 
understand.

(4) complete guidance / turnkey solution for every script. ICANN's 
Second Level Reference LGRs or ICANN's Root Zone LGRs (and a few second 
level LGRs / IDN Tables by third parties) represent something that could 
be adopted "as is" - or very largely as is (like having to add digits to 
the Root Zone LGRs to make them work for the second level). I don't see 
value for IETF to go here, but guidance of type (3) could point to such 
existing work product.

Or you could do a combination:
- do (1) but mention that there is more AND
- do (2) but mention that "understanding" has to include knowing the 
"rules" AND
- do (3) which roughly partitions the repertoire by the types of 
policies needed AND
- have it point to (4), that is external solutions that registries can 
adopt now that they understand what "understanding code points" means 
and why they need to adopt a rather complex-looking solution.

Have we won the war?

No.

Why?

Because above the LGR is the next layer:

(5) String similarity review.  I'm not sure that anybody knows 
conclusively how to automate that for an arbitrary script. A subset of 
this problem you could treat with the variant mechanism, such as Zero 
vs. Oh, One vs Ell and so on, but pretty soon you run into less direct 
similarity and equivalence relations won't cut it: you'll be dealing 
with non-zero distances in perception space.

I've seen some demonstrations of systems that try this for Latin; not 
sure how accurate they are or how well they scale.

Trying to even fit this level of the problem into our discussion means 
that we won't ever get anywhere, so realistically, this has to remain 
out of scope - as much as we'd want registries to use such technologies 
to protect users.

Given this, it makes a lot of sense to me to proceed in order (1), (2), (3).

A./

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Feb 2019 11:02:24 -0500

TL;DR summary for those who need it: please read the last three
paragraphs of my response to Asmus.

Asmus,

For purposes of discussion, let's assume that I agree with you,
at least with two fairly minor exceptions.  It isn't quite true,
but it may make it easier to have a discussion, at least for the
present.

The exceptions are:

(i) In part because the IETF is supposed to have strong rules,
or at least traditions, about fixing observable problems (aka
"technical defects") with its standards, I think it is important
that we have/include a plan for reviewing the IDNA2008 specs
and, if appropriate, making adjustments for whatever we have
learned in the last eight or so years.   In its most simple
form, this may involve looking at the boundary between whatever
would fall into your (3) or (4) and seeing whether any of it
suggests global rules or global exclusions that should be
defined as part of IDNA2008 and making appropriate changes.  Or,
at the other extreme, it could involve pulling some rules (or
exceptions) out of IDNA2008 on the grounds that they really
should have been advisory.  Or there may be one particularly
relevant example of a needed change to IDNA2008 that has nothing
to do with rules or code points; see my penultimate paragraph
below.

(ii) I think you are falling into the (unfortunately popular)
trap of assuming that the DNS is basically flat and assuming
that nothing beyond the second level (or the first "non-public"
level) is of much importance.   I see far more zones out there
that are associated with either enterprises or other related
parties.  We can say that a different set of rules apply to
them, or they are not a problem, etc., but, if things are being
done in the IETF context, with applicability to the whole
Internet, we then need to get far more clear about applicability
than many of our discussions (and many of your comments) have
done so far.   As an alternative, we can stick with a decision
that was made around 2002 and mostly reaffirmed by practice
since.   That decision is that, for the DNS, the IETF has
responsibility for:

(a) The basic protocols and what they allow or require, both
algorithmically and operationally.   Those things apply to the
entire DNS and includes coding issues but not the details of
what can be expressed by those codes.  I note that, despite a
lot of hand-waving about "host names", the IETF has never
attempted to change the RFC 1034/1035 principle that the octets
that make up DNS labels can contain any sequence of bits.
IDNA2008 is a "if you decide to do this, this is how to do it"
specification, not "if you are going to do non-ASCII strings,
you are required to do it this way".

(b) Basic operational rules and requirements affecting the root,
mostly based on authority the IETF and IAB inherited from
pre-ICANN IANA via the Host Requirements work and, to some
extent, RFC 1591.  "No digits and no hyphens" is an example of
this.

Responsibility for everything else, particularly decisions about
what new labels were going to be created in the root and to whom
they were to be delegated, managing those delegations, decisions
or recommendations about what was allowed (or encouraged,
discouraged, or prohibited) in the second level or below, passed
directly from pre-ICANN IANA to ICANN.   As I understand it,
unless matters involve registry operations, they don't even
involve PTI (and hence should not involve the IAB or IETF).  The
tables specified by RFC 5892 and draft-faltstrom are PTI issues
wrt performance but IANA involvement in your (3) -(5) has not
been specified by anyone.

Now, with that (I hope) understood...

The more we (as the IETF) venture toward your (3), much less (4)
or (5), the more skeptical I get about how we do it and how we
do it consistent with IETF norms about maintenance, much less
informed consensus.   Remember that part of the origin of
draft-faltstrom in its current form was a January 2015 IAB
statement urging the IETF to get on with work in this area that
was perceived as critical.  That request was never implemented:
some discussion documents were posted but never led to serious
community discussion and the IAB ultimately dropped the request.
The BOF that led to this directorate was another attempt to get
the general situation with i18n work unstuck.  I don't know if
the very long time it took to get the directorate set up is a
symptom or not, but I observe that, despite requests from both
Pete and Peter that we look at and comment on each other's
reviews and comments, there has been almost none of that except
from you, me, and occasionally Patrik.   Reviews on the IETF
list have also not been substantive but, instead, have commented
that the document (independent of its content) is long overdue,
have raised issues that are almost certainly outside the
document's scope, or that are focused on nit-picking.

So things are looking a lot as if the directorate idea is
failing and that the  only way to get an i18n document approved
by the IETF is to twist some AD's arm sufficiently that the
document goes out for IETF LC and then count on enough
cheerleading and nit-picking reviews that don't address
substance (possibly because the review authors are
insufficiently expert) to slip the document through the system.
Given the quality and range of reviews in the last few years, I
think that description might include not just these IDNA issues
but the second-round PRECIS documents and the two LGR documents.
If that is really where we are, then proposing new IETF work is
probably a bad idea.  It may even be that the content of
draft-falstrom-unicode11 and how it has been received calls for
a different sort of change to that document, one that would
explicitly say that IDNA2008 is frozen, that future attempts at
IETF review are a waste of time because we are going to end up
accepting whatever Unicode has done, and that RFC 5892 should be
updated to change the review requirement so that IANA computes
the tables (with whatever advice they want to seek) as soon as a
new version of Unicode appears and posts them, possibly after a
short public comment period.  That outcome would make me very
unhappy, but maybe it is time to face reality.

Especially in the light of the combination of your comment in an
earlier note that perhaps everything the IETF does in this area
is useless and a very general concern that the IETF is not
adding value to the situation and is undermining its own
credibility when documents are posted with little or no evidence
of informed IETF community consensus, perhaps it really is time
to conclude that the IETF can't do this and we all need to
accept that an move on rather than continuing to hope for
miracles.

best,
   john

--On Tuesday, February 26, 2019 00:44 -0800 Asmus Freytag
<[email-masked]> wrote:

>> I definitely think that having the IANA tables seriously out
>> of synch with Unicode versions is a bad idea and a source of
>> second level registrations, but it is not an IETF problem and
>> I don't believe it would fly politically.
> 
>...

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 26 Feb 2019 09:47:43 -0800

> paragraphs of my response to Asmus.

To keep this shorter, I will reply globally.

I extract several issues.

(A) If you believe that IDNA2008 as currently specified has a technical 
defect that mandates a fix, or even if you believe that it is crucial to 
review the specification for the possibility of such defect, the first 
step should be to understand and come to agree what would constitute a 
defect. IDNA 2008 has not, from its inception, been able to guarantee 
visually (or otherwise) absolutely unambiguous identifiers.

Therefore, any particular instance of IDNA2008 not preventing ambiguous 
identifiers would not seem to be an isolated defect. In fact, anyone 
concluding from that, that the defect is in not making that overall 
limitation explicit in the description of IDNA 2008 might agree to fix 
that expectation instead.

(B) If there is agreement on a technical defect that mandates a fix, the 
next step is to look at what it would break, or whether you can identify 
any other obstacles to propagating the fix. Is the defect serious enough 
that fixing it outweighs the potential for follow-on issues?

(C) The need to guarantee unambiguous identifiers is distributed 
unevenly across the DNS. We do agree on that. The methods that can be 
used successfully to get to fewer ambiguous identifiers are by now 
well-known to those expert in the field. However, they are way to 
heavy-handed to incorporate into a protocol; particularly if only 
certain types of zones and registries are benefiting.

As a corollary: there's really no simple, straightforward subset that 
could be tacked onto the protocol and would be a major "win".

(D) Ambiguity in identifiers is not a black&white affair and it's not 
entirely due to IDNs: even the simplest identifier rules (LDH) are not 
free of confusable identifiers (.corn, app1e). It is a hard problem. 
Confusability is a scale and successive solutions can only ever add yet 
another partial remedy.The cumulative effect can be more and more 
robust, without ever reaching perfection. At the same time, many of the 
techniques intersect with usability issues - which removes them further 
from the protocol level.

(E) There are other actors on the field who are not constrained by 
having to cater to all users of the protocol and who have the resources 
to look deeply into the linguistic underpinning affecting individual 
code points. Something that IDNA 2008 explicitly does not do (and I'm 
not arguing it should). I think it might be a reasonable conclusion to 
combine level setting that explains why the protocol by itself cannot 
deliver on the widely expected guarantee of unambiguous identifiers with 
pointing registries that need to make such a guarantee to these external 
resources.

I would frame the "mandate" to registries differently. Rather than the 
somewhat tautological "register only what you understand", why not: 
"based on the protocol, the responsibility for guaranteeing that 
identifiers are unique and unambiguous requires active support from 
registration policies" or something like that.

(F) If my tentative conclusion seems overly pessimistic, that's 
something that I have been extrapolating from this and previous 
discussions, including very much your input, and not something that I 
started out with.

However, it applies currently only to the question of whether IETF 
specifications can guarantee unambiguous identifiers.

A./

---

From: John C Klensin <[email-masked]>
Date: Wed, 27 Feb 2019 18:35:08 -0500

--On Tuesday, February 26, 2019 09:47 -0800 "Asmus Freytag (c)"
<[email-masked]> wrote:

>> TL;DR summary for those who need it: please read the last
>> three paragraphs of my response to Asmus.
> 
> (A) If you believe that IDNA2008 as currently specified has a
> technical defect that mandates a fix, or even if you believe
> that it is crucial to review the specification for the
> possibility of such defect, the first step should be to
> understand and come to agree what would constitute a defect.
> IDNA 2008 has not, from its inception, been able to guarantee

It was never intended to do that.  I believe that repeating the
assertion that it does not, especially in a group as presumably
expert as the membership of this directorate, is a distraction
from more important issues.

As to "the first step", I would have agreed with you if we had
been having this discussion three or four years ago.   Now, with
a number of documents that raise issues or proposes fixed lying
around unprocessed, I believe the first step is to figure out
how, or if, it would be possible to process a proposed fix.

In addition, I think "what constitutes a defect" is clear.  If
reasonable people who have read the IDNA2008 specs conclude that
they are intended, without specific registry-level restrictions
and their application, that would constitute a defect because it
would mean the standards were too difficult to interpret as was
intended.  Fortunately, there is no evidence of that even though
there is some limited evidence of the standard being
deliberately misinterpreted that way for commercial advantage.  

By contrast. the standard seems quite clear that, if a review of
a new version of Unicode identifies code points that raise
issues equivalent to those that have already been treated by
exception or changes to properties of code points assigned
earlier that affect final categories for those code points, the
standard anticipates changes to track those issues and maintain
consistency, at least unless the impact of the changes is no
trivial as to have no important impact.  Because changes of that
type are fully anticipated by the standard, it is hard to
consider them defects.  However, there is some ambiguity in the
standard as to whether, if an issue shows up with a code point
assigned earlier in the course of reviews of newer versions of
Unicode, whether correcting the standard to reflect those
earlier oversights is just part of the ordinary review or is a
defect.  Once good sense is applied, I can't see that
distinction as making any possible difference and believe that
the focus should be on getting the issue identified and
resolved, but YMMD.

>...
> (B) If there is agreement on a technical defect that mandates
> a fix, the next step is to look at what it would break, or
> whether you can identify any other obstacles to propagating
> the fix. Is the defect serious enough that fixing it outweighs
> the potential for follow-on issues?

An entirely reasonable question but not what the standard, as I
read it, now says.  If you believe the standard is defective for
not identifying and requiring that criterion, I assume that
would be a defect that should be addressed.  See above.

> (C) The need to guarantee unambiguous identifiers is
> distributed unevenly across the DNS. We do agree on that. The
> methods that can be used successfully to get to fewer
> ambiguous identifiers are by now well-known to those expert in
> the field. However, they are way to heavy-handed to
> incorporate into a protocol; particularly if only certain
> types of zones and registries are benefiting.

While we agree on almost all of that, I am far less concerned
about "benefits" to zones and registries than I am about risks
to end users.  I am [almost] as interested as Scott is in making
sure that those registries who are interested have the best and
clearest guidance possible (and, again, agree that is not going
to come exclusively from the IDNA2008 protocol or rules), but my
goal is to make life easier and safer for users.  I am also
painfully aware that, in a world in which neither Unicode, nor
IDNA2008, nor ICANN or IETF guidelines specify type families or
styles in which domain names are displayed, many users will see
what they expect to see and that nothing we can do is likely to
prevent that, even if it were heavy-handed in the extreme.   One
of the consequences of those things is that, if users can see
domain names in a script that they do not read easily (or from
an unfamiliar zone) then the damage is not limited to "benefits"
(or not) for the registry.  For that and other reasons, if your
criteria is benefits to zones and registries, I think we are
looking at different problems and different criteria.   

> I would frame the "mandate" to registries differently. Rather
> than the somewhat tautological "register only what you
> understand", why not: "based on the protocol, the
> responsibility for guaranteeing that identifiers are unique
> and unambiguous requires active support from registration
> policies" or something like that.

Do you consider the need for that reframing significant enough
that you would consider the current statements ("register only
what you understand" is my summary for this list, not, IIR, what
the IDNA2008 specs actually say) to be a defect in the protocol
specification that is worth fixing?  If not, why are you
bringing this up?

> (F) If my tentative conclusion seems overly pessimistic,
> that's something that I have been extrapolating from this and
> previous discussions, including very much your input, and not
> something that I started out with.
> 
> However, it applies currently only to the question of whether
> IETF specifications can guarantee unambiguous identifiers.

Again, IDNA2008 explicitly doesn't do that or attempt to do
that, even if "unambiguous" means "not capable of being confused
with others".  It is possible that some people still believe one
can have identifiers that are tied to natural language,
abbreviations or acronyms based on natural language, or anything
else that is obviously of mnemonic value outside a very small
circle of people, but I'm not one of them and I hope that view
does not have representation on this list.  That limitation
appears to be especially true the further the language departs
from Western European ones and the further the writing system
departs from unadorned Basic Latin, but, as we know, even
English and ASCII lend themselves to problems.   I suggest that,
if one wants unambiguous and self-contained identifiers, one
gives up mnemonic value.

Even if one manages enough in the way of string similarity (or
word similarity, etc.) rules (algorithms and human intervention)
to prevent confusingly-similar strings from getting into the DNS
as labels, I don't think it creates lack of ambiguity because
not finding what a reasonable user would perceive as an
alternate form may protect against false positives but does
nothing for false negatives.

Now, whether IETF might try to develop such identifiers
--independent of what IDNA2008 does and does not do or claim --
is another issue, but I would certainly oppose trying if
"guarantees" are needed.

So, from my point of view, the answer to that question is not
only "no", but posing it as the (or even a) key question has
significant similarity to a red herring.

best,
   john

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 27 Feb 2019 16:11:57 -0800

On 2/27/2019 3:35 PM, John C Klensin wrote:
>> particularly if only certain
> to end users.

This is why I find it hard to spend further time on this discussion.

Yes, ultimately the "benefit" is to the users through increased security
and not is some stupid way to the "registries" as organizations.

I feel that you misconstrued my intent here to score a cheap point.

A./

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 27 Feb 2019 16:22:34 -0800

> specification that is worth fixing?  If not, why are you
I'm bringing this up because I find the summary somewhat misleading, or, 
if you like that term better, misdirecting. Also, if applied to code 
points, it tends to focus the reader on code point selection, which I 
keep pointing out is a problem that's not the primary problem for many 
scripts.

Being in the middle of a long discussion, like you mentioned, I take is 
as "read" that we are talking in this instance about getting the 
draft-klensin "right" without having to have a discussion each time on  
whether to do it or not.

In that context, which assumes as goal of that draft, to restate the 
registries' responsibility, it seems important to me that we describe 
these very strongly in terms of the ultimate outcome ("unique and 
unambiguous identifiers" and whatever similar outcomes we like to 
propose - like not needlessly including stuff that's "unrecognizable" to 
users because obsolete etc.).

On some level, you could say that whether the registries "understand" 
the code points is less relevant on whether they are as secure as they 
can be (for example by deploying some reference LGR recommended for the 
script).

A./

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 27 Feb 2019 16:39:28 -0800

John,

it comes down to what you see as the goal for the protocol.

Or, because this discussion is in the context of Patrik's draft, the 
part of the protocol assigning status values to code points. If we agree 
that the goal cannot be "unambiguous identifiers" then the evaluation of 
new additions to Unicode cannot primarily focus on whether they impact 
that non-goal. It has to be more differentiated: to make sure they don't 
make the situation worse.

Reading Patrik's draft, I'm convinced the new code points since 6.3.0 do 
not represent a new fundamental issue and thus back up his conclusion.

For your draft, see separate mail; but my take is that what we should do 
is to ask registries to do their bit to make the DNS more secure by 
removing cases of ambiguities - the Root Zone procedure said it in a 
nice way: the RZ-LGR is to remove ("take off the table") cases that are 
obvious and can be addressed mechanically (i.e. as matter of a policy 
that can be automated).

That's a limited goal, because we all agree that there is no perfection 
(at least I read your statement here in that way).

As to how they (registries) are supposed to do this, my conclusion from 
our discussion so far is that there is no room in IETF to provide 
specifications. The long-and-short is not only is there no definite 
end-point, but also because while you can create algorithmic rules to 
classify and compare labels, you cannot create an algorithm to create 
such rules.

The closest you might come is to create a propery-based algorithm that 
flags code points for the type of concern they represent. That might be 
a useful educational tool and could even be used to audit registry 
policies, but I don't sense that there's overwhelming interest in 
working that out as a specification. Thus here lies "troublesome" at the 
end of his lonely road.

What's left is the work carried out by other actors, outside IETF.

A./