Subject: [I18ndir] Trying to summarize discussion of draft-faltstrom

From: Harald Alvestrand <[email-masked]>
Date: Fri, 8 Mar 2019 09:53:19 +0100

I'm still struggling with what to write in the directorate review.

There's clearly two parts to the review: One of the document, and one of
the context.

Currently, my draft of the context paragraph looks like what's pasted below.

Does it capture any essential aspect of the discussion?

--------------------

**

    *Context issues*

*

The discussion of draft-faltstrom-unicode11 in the directorate has shown
that the directorate members share a number of concerns about the
current state of IDNA, only some of which are directly relevant to this
memo.

IDNA2008 considered limits to what was reasonable to register and use in
the DNS at a number of levels:

  *

    A level of “don’t register stuff that causes confusion”. This
    requires human judgment, and reasonable people may disagree about
    what causes confusion.

  *

    A level of “don’t register stuff that is structurally invalid under
    the relevant writing system”. Aspects of this can be captured in
    rulesets (ICANN’s RZ-LGR efforts fall noto this category), but
    requires deep expertise; this is captured in IDNA2008 as the “don’t
    register what you don’t understand” rule.

  *

    A level of “this is stuff that you should never register, and
    applications can reasonably choose to treat it as an error or an
    attack if it ever shows up”. This is the distinction that is
    captured in the classification of codepoints as DISALLOWED, and
    where IDNA2008 (with updates) gives precise rules.

The current document focuses on the last level only - the maintenance of
the distinction between PVALID and DISALLOWED.

It is clear from directorate discussion that work needs to be done at
the other levels outlined above too, but it is not clear from the
discussion what form that work should take or what fora that work is
reasonably performed in; the work may or may not involve a revision of
the basic IDNA2008 specifications.

*---------------------*

**

**

*

-- 
Surveillance is pervasive. Go Dark.

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 8 Mar 2019 09:11:47 +0000

Hello Harald,

> Does it capture any essential aspect of the discussion?

I'd say this captures quite a lot of the essential aspects of the 
discussion, definitely enough for the current review. In my eyes, this 
part is ready to ship. It may be tempting to add more details, but I 
don't think they are needed now.

I hope you'll do a similarly excellent job for the rest of the review 
(which I hope will be easier).

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Sun, 10 Mar 2019 06:55:44 -0400

Harald,

I decided to accelerate assembling the very long analysis I just
posted after reading this (I started a few days earlier).  That
note should make a lot more of my context clear, but...

(1) The discussion of levels is quite good although I'm
concerned that we keep forgetting the two CONTEXTx categories.
That is not an IDNA issue separate from this document because
the review and I-D could identify exceptions that belonged in
one of those categories rather than PVALID and DISALLOWED.  

(2) While I think the distinction you make about the scope of
the document should be the correct one, the document itself
wanders into a discussion of non-conforming variations on
IDNA2008 (a discussion that needs work if it is to be retained),
assertions about actions that will either reduce the number of
variations or make them less different (for which there is
little or no evidence), and assumptions about advice to
registrars and maybe that it such advice will be followed.  See
my discussion with Marc last week and the long note for details.

(3) While I agree that there is IDNA work that should be done
and that is not directly relevant to this memo, the memo makes
assertions about IETF discussions and conclusions that are
simply incorrect unless some of that work is done.  In most
cases, I can see how to adjust those assertions or take them out
without appreciably weakening the document, but, as the text
stands, it is problematic.  Again see the long note for details
and suggestions.

  best,
   john

--On Friday, March 8, 2019 09:53 +0100 Harald Alvestrand
<[email-masked]> wrote:

> There's clearly two parts to the review: One of the document,
> and one of the context.
> 
> The discussion of draft-faltstrom-unicode11 in the directorate
> has shown that the directorate members share a number of
> concerns about the current state of IDNA, only some of which
> are directly relevant to this memo.
> 
> 
> IDNA2008 considered limits to what was reasonable to register
> and use in the DNS at a number of levels:
> 
> 
>   *
> 
>     A level of "don't register stuff that causes
> confusion". This     requires human judgment, and reasonable
> people may disagree about     what causes confusion.
> 
>   *
> 
>     A level of "don't register stuff that is structurally
> invalid under     the relevant writing system". Aspects of
> this can be captured in     rulesets (ICANN's RZ-LGR efforts
> fall noto this category), but     requires deep expertise;
> this is captured in IDNA2008 as the "don't     register
> what you don't understand" rule.
> 
>   *
> 
>     A level of "this is stuff that you should never
> register, and     applications can reasonably choose to treat
> it as an error or an     attack if it ever shows up". This
> is the distinction that is     captured in the classification
> of codepoints as DISALLOWED, and     where IDNA2008 (with
> updates) gives precise rules.
> 
> 
> The current document focuses on the last level only - the
> maintenance of the distinction between PVALID and DISALLOWED.
> 
> 
> It is clear from directorate discussion that work needs to be
> done at the other levels outlined above too, but it is not
> clear from the discussion what form that work should take or
> what fora that work is reasonably performed in; the work may
> or may not involve a revision of the basic IDNA2008
> specifications.
> 
> *---------------------*
> 
> **
> 
> **
> 
> *

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 10 Mar 2019 11:25:38 -0700

> one of those categories rather than PVALID and DISALLOWED.

Not so much forgetting, but a realistic assessment that they won't play 
a role.

There's CONTEXTJ and CONTEXTO.

CONTEXTJ reflects a set of invisible characters that nevertheless are 
part of some orthographies, and in a few cases, rather hard to do 
without altogether, although that's what we do in the root. Based on the 
specific nature, I'd say the possibility of further additions to that 
that category of context is unlikely.

If there was ever a new set of such invisible code points, perhaps they 
would require their own context.

CONTEXTO reflects a "grab-bag" of characters that arguably should never 
have been defined as such in the protocol. It represents a tiny fraction 
of all code points that warrant context restrictions. The majority of 
code points that should be given such restrictions are not recent 
additions, but have been part of the repertoire all along.

In addition, many suffer from the fact that while restrictions are 
needed to allow them to function properly in identifiers, the nature and 
details of that restriction does not depend solely on the code points, 
but on the orthographies they are used with. And the latter are not 
fully stable. Neither is their digital representation.

While in principle additional code points could be identified as 
CONTEXTO, I see no obvious candidates that I'd be entirely comfortable 
baking into the protocol.

---

If you'd like, you could do a mental exercise: try to create a 
"Reference LGR" that works for all possible zones in all scripts. That's 
essentially what the full protocol represents.

Now take a look at the discussions over at ICANN where they are working 
on reference LGRs for specific scripts (and modern usage only), and 
directed at second level domains only and note, how difficult it is to 
get those defined.

A./

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 11 Mar 2019 05:26:06 +0100

On 03/10/2019 07:25 PM, Asmus Freytag wrote:
> "Reference LGR" that works for all possible zones in all scripts.
> That's essentially what the full protocol represents.
>
> Now take a look at the discussions over at ICANN where they are
> working on reference LGRs for specific scripts (and modern usage
> only), and directed at second level domains only and note, how
> difficult it is to get those defined.
>
>
Isn't that what the MSR (Maximum Script Repertoire) is supposed to
represent, while the LGRs are stating all the rules that are more
complex than "alllow this character"?

(spent some little time yesterday on the MSR/LGR documents)

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Mar 2019 01:14:46 -0400

--On Sunday, March 10, 2019 11:25 -0700 Asmus Freytag
<[email-masked]> wrote:

> Not so much forgetting, but a realistic assessment that they
> won't play a role.

As long as they are possible values, a piece of report
describing what is _possible_ should not omit them.  My
assessment of likelihood is no different from yours, but that is
a different question.

> CONTEXTJ reflects a set of invisible characters that
> nevertheless are part of some orthographies, and in a few
> cases, rather hard to do without altogether, although that's
> what we do in the root. Based on the specific nature, I'd say
> the possibility of further additions to that that category of
> context is unlikely.
> 
> If there was ever a new set of such invisible code points,
> perhaps they would require their own context.
> 
> CONTEXTO reflects a "grab-bag" of characters that arguably
> should never have been defined as such in the protocol. It
> represents a tiny fraction of all code points that warrant
> context restrictions. The majority of code points that should
> be given such restrictions are not recent additions, but have
> been part of the repertoire all along.
> 
> In addition, many suffer from the fact that while restrictions
> are needed to allow them to function properly in identifiers,
> the nature and details of that restriction does not depend
> solely on the code points, but on the orthographies they are
> used with. And the latter are not fully stable. Neither is
> their digital representation.
> 
> While in principle additional code points could be identified
> as CONTEXTO, I see no obvious candidates that I'd be entirely
> comfortable baking into the protocol.

As suggested above, I agree about the likelihood of adding more
or them and the conditions for doing so.  Your analysis suggests
something else that is out of scope for draft-faltstrom but that
should be considered at some stage.  That is whether to remove
CONTEXTO entirely from IDNA2008, leaving those issues to
registry good sense (and then hope to find that property, which
appears to be scarce among some registries).

   best,
   john

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 10 Mar 2019 22:52:55 -0700

Harald,

that is a bit of a simplification, but not that far off the mark.

MSR effectively adds to the "DISALLOWED" for the purpose of the Root 
Zone - it takes certain code points off the table. It does not "allow" 
characters for the root, that's what the repertoire section of the Root 
Zone LGRs do. It also does not define context rules or variants.

The whole Root Zone LGR consists of a combination of per-script LGRs which:

(1) define repertoire of allowed code points (subset of MSR)

(2) define context further constraining some code points

(3) define variants

     (a) across scripts (e.g. Latin/Cyrillic letters that look alike and 
many more)

     (b) in script (e.g. Chinese simplified / traditional, but also 
Latin U+0259 / U+01DD and similar cases)

[Not all scripts require context rules, not all scripts require 
variants, but many do]

A "reference LGR" would be like a "model" LGR that does (1), and where 
needed: (2) and (3b). If you think of the context rules in IDNA2008, it 
is a bit more like an LGR that is super permissive: all PVALID code 
points are included, and the contexts are minimal.

That's why I made that particular comparison:

If you begin to add more context rules into the protocol (which is 
something John hinted at as a possibility) then you run into the same 
kinds of issues that the "Reference" LGRs for the second level are 
running into: a strong disagreement on what precise restrictions are 
appropriate/correct for a given zone.

For the Root Zone, we have many scripts for which the community has 
managed to come to an agreement on what is appropriate for the root in a 
context that equally supports all languages that are commonly written in 
that script. (There are a few scripts where that consensus is still 
elusive.) The situation for 2n Level Reference LGRs is different, and if 
you tried to create contexts that are valid for *all* situations / all 
zones you could expect strong pushback.

That, to my view, makes it extremely unlikely that an attempt to update 
context rules in the protocol will result in a consensus. (And I'm not 
sure whether we are better off keeping the CONTEXTO archived, or whether 
we are better off with rescinding them in favor of a different solution 
.  . )

A./

PS: sorry not short.