Subject: [I18ndir] Draft review of draft-faltstrom-unicode

From: Harald Alvestrand <[email-masked]>
Date: Sun, 10 Mar 2019 07:43:54 +0100

I've talked to Patrik, and we think that this review text makes sense.

Comments welcome - preferably before 2300 Monday Japan time (less than
24 hours seems unreasonable). I'll incorporate suggestions (at least if
they seem obviously correct - if not, we'll discuss) - and if the
directorate seems to have consensus or someething close to it, I'll
publish the review Tuesday morning Japan time.

Patrik will prepare a new version based on this review, and submit that
either before the deadline or with an override approval from our ADs.

Then we'll see where we are.

----------------------------

**

*Directorate review of draft-faltstrom-unicode11-07*

*

Overall conclusion: Almost ready, with caveats

    Context issues

The discussion of draft-faltstrom-unicode11 in the directorate has shown
that the directorate members share a number of concerns about the
current state of IDNA, only some of which are directly relevant to this
memo.

IDNA2008 considered limits to what was reasonable to register and use in
the DNS at a number of levels:

  *

    A level of “don’t register stuff that causes confusion”. This
    requires human judgment, and reasonable people may disagree about
    what causes confusion.

  *

    A level of “don’t register stuff that is structurally invalid under
    the relevant writing system”. Aspects of this can be captured in
    rulesets (ICANN’s RZ-LGR efforts fall not this category), but
    requires deep expertise; this is captured in IDNA2008 as the “don’t
    register what you don’t understand” rule.

  *

    A level of “this is stuff that you should never register, and
    applications can reasonably choose to treat it as an error or an
    attack if it ever shows up”. This is the distinction that is
    captured in the classification of codepoints as DISALLOWED, and
    where IDNA2008 (with updates) gives precise rules.

The current document focuses on the last level only - the maintenance of
the distinction between PVALID and DISALLOWED.

It is clear from directorate discussion that work needs to be done at
the other levels outlined above too, but it is not clear from the
discussion what form that work should take or what fora that work is
reasonably performed in; the work may or may not involve a revision of
the basic IDNA2008 specifications.

We suggest to insert a paragraph in the document describing the context
of the state of IDNA2008, and explain what issues this document does not
attempt to address. Specifically that the conclusion of the document is
what to do regarding Unicode versions up to and including 11, and that
this is not to be used as expectations of future versions of Unicode.

    Content issues

Section 4.1 does not specify where to find the conclusion of the IETF
discussion on U+08A1.

It is not easy to see from the text whether the algorithms and
procedures will render U+0628 U+0654 an illegal sequence or a legal
sequence. No matter what the resolution is, the document should make it
obvious what the conclusion is (and why).

RFC 5892 states that SPHERICAL ANGLE OPENING UP is DISALLOWED not PVALID:

27D0..2B4C  ; DISALLOWED

Section 4.1 ought to include numbers for how many characters ended up in
DISALLOWED vs PVALID.

    Nits

1 intro para 3 (bullet 1): “Assigning code points can create problems if
the newly-assigned     code points are compositions of code points
changes” - this sentence ungrammatical. Presumed intent: “can create
problems if the newly-assigned code points are compositions of code
points that change (or would have changed) the normalization functions”.

Headers in 4: “between” is ambiguous on whether its endpoints are included.

“Changes after Unicode 7.0.0 up to Unicode 10.0.0” would be unambiguous.

4.1 para 3 - “This identical to LATIN….” - presumed meaning “This is the
same situation as with LATIN….”

A series of comments from Martin Dürst have been forwarded to the author
separately.

*

-- 
Surveillance is pervasive. Go Dark.

---

From: "Marc Blanchet" <[email-masked]>
Date: Sun, 10 Mar 2019 09:45:46 -0400

I have made a review and a point about some more detailed guidelines for 
the expert review for new Unicode versions. Patrick agreed. I don’t 
see anything here on this.

Marc.

> -- 
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 00:52:27 +0900

I must go to bed and have not addressed all of this yet, but working on 
it.

Here is my current beta of -08.

    Patrik

On 10 Mar 2019, at 15:43, Harald Alvestrand wrote:

> 24 hours seems unreasonable). I'll incorporate suggestions (at least 
> if they seem obviously correct - if not, we'll discuss) - and if the 
> Patrik will prepare a new version based on this review, and submit 
> that either before the deadline or with an override approval from our 
> ADs.
>
> The discussion of draft-faltstrom-unicode11 in the directorate has 
> shown that the directorate members share a number of concerns about 
> the current state of IDNA, only some of which are directly relevant to 
> this memo.
>
>
> IDNA2008 considered limits to what was reasonable to register and use 
> in the DNS at a number of levels:
>
>
>   *
>
> The current document focuses on the last level only - the maintenance 
> of the distinction between PVALID and DISALLOWED.
>
>
> We suggest to insert a paragraph in the document describing the 
> context of the state of IDNA2008, and explain what issues this 
> document does not attempt to address. Specifically that the conclusion 
> of the document is what to do regarding Unicode versions up to and 
> including 11, and that this is not to be used as expectations of 
> future versions of Unicode.
>
>
>     Content issues
>
>
> sequence. No matter what the resolution is, the document should make 
> it obvious what the conclusion is (and why).
>
>
> Section 4.1 ought to include numbers for how many characters ended up 
> in DISALLOWED vs PVALID.
>
>
>
>     Nits
>
>
> 1 intro para 3 (bullet 1): “Assigning code points can create 
> problems if the newly-assigned     code points are compositions 
> of code points changes” - this sentence ungrammatical. Presumed 
> intent: “can create problems if the newly-assigned code points are 
> compositions of code points that change (or would have changed) the 
> normalization functions”.
>
>
> 4.1 para 3 - “This identical to LATIN….” - presumed meaning 
> “This is the same situation as with LATIN….”
>
>
> A series of comments from Martin Dürst have been forwarded to the 
> author separately.
>
>
> *
>
> --

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 10 Mar 2019 13:35:27 -0700

> Here is my current beta of -08. 

Patrik,

here's my review:

Line 17: With the publication of Unicode 12.0 this draft should 
(ideally) be updated to cover the character additions since 11.0. 
Unicode  (see comments at end).

Line 174/175: change to "this document concludes" (add 's').

Line 178: I disagree with the statement that U+08A1 as such can be 
called "not understood". I believe, by now, it's one of the better 
understood cases. (see further comments below)

Line 360: The correct parallel would not be U+00E4 but instead it would 
be cases like LATIN SMALL LETTER O WITH STROKE (U+00F8) where there 
exists a combining character sequence of similar appearance involving 
COMBINING SHORT SOLIDUS OVERLAY (U+0337) but no normalization decomposition.

This parallel should be stated first, then the contrast made with U+00E4.

Line 368: I think it's fair to write the the "underlying problem 
with...is not fully understood". But I have issue with some of the 
phrasing. The passage "...fully to say that this.." doesn't work. It 
would be better to write: "..fully. Therefore it cannot be claimed that 
this ..". This is a simple copy edit.

Line 369: In the same passage, the problem is identified as one with 
"combining sequences". I think it's the opposite. The issue is the 
encoding of characters that to the naive observer appear to match a 
combining sequence but which are encoded without decomposition because 
Unicode deems the combining sequence not canonically equivalent.

The solution could be to DISALLOW a subset of combining marks: all 
Arabic combining marks and all that have the word COMBINING in their 
name but are not today part of a decomposition. Or some other means to 
encourage registries to not include those code points. However, I agree 
that there is not a consensus on that.

As result, it seems to me better to go out with the existing language.

Line 499: I think Marc has a good point about setting out some 
guidelines about when it is appropriate to "silently" progress the IANA 
registry to the new version of Unicode. That text could be inserted here.

"The Unicode Standard is being updated at least yearly and
implementors and vendors of OS, libraries and other tools are
aggressively updating their systems/libraries to the new versions of
Unicode. Therefore, IETF recognizes the burden on implementers
to support IDNA2008 corresponding to an outdated version of Unicode.

Therefore, during the beta review stage before each release of Unicode
the reviewer will apply an analysis of the changes as carried out in this
document and share it with any active i18n WG and / or the I18n Directorate
for additional guidance.

If neither the reviewer or those reviewing the results note any issue that
requires an exception (or where the decision to not make an exception
warrants IETF consensus), the IANA tables will be updated in sync with the
release of the new version of the Unicode Standard.

Otherwise, the reviewer will write a document like the current one to get
the necessary consensus."

Text like the above could be inserted at Line 499, or a separate section 
could
be added.

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 12:07:23 +0900

> here's my review:

Thanks!

> Line 17: With the publication of Unicode 12.0 this draft should (ideally) be updated to cover the character additions since 11.0. Unicode  (see comments at end).

Noted.

> Line 174/175: change to "this document concludes" (add 's').

Done

> Line 178: I disagree with the statement that U+08A1 as such can be called "not understood". I believe, by now, it's one of the better understood cases. (see further comments below)

What I mean is that this WG has clearly not consensus on how to deal with U+08A1 and similar cases (whatever similar cases are). We do not have consensus on whether IDNA2008 mechanism with rules is the way to deal with those cases. The contrary, we seem to have agreement that for this specific code point we do accept the situation but use it as a trigger for creation of this directorate and start really thinking hard.

The only way I managed to try to express this was to say "not understood".

If you have better wording, please let me know.

> Line 360: The correct parallel would not be U+00E4 but instead it would be cases like LATIN SMALL LETTER O WITH STROKE (U+00F8) where there exists a combining character sequence of similar appearance involving COMBINING SHORT SOLIDUS OVERLAY (U+0337) but no normalization decomposition.

Ok, fair. The reason why I use what I use is because the expected behaviour changes depending on what language context one is in. You are right your example is better so I will change.

> This parallel should be stated first, then the contrast made with U+00E4.

I removed the U+00E4 example.

> Line 368: I think it's fair to write the the "underlying problem with...is not fully understood". But I have issue with some of the phrasing. The passage "...fully to say that this.." doesn't work. It would be better to write: "..fully. Therefore it cannot be claimed that this ..". This is a simple copy edit.

Thanks!

> Line 369: In the same passage, the problem is identified as one with "combining sequences". I think it's the opposite. The issue is the encoding of characters that to the naive observer appear to match a combining sequence but which are encoded without decomposition because Unicode deems the combining sequence not canonically equivalent.
>
> The solution could be to DISALLOW a subset of combining marks: all Arabic combining marks and all that have the word COMBINING in their name but are not today part of a decomposition. Or some other means to encourage registries to not include those code points. However, I agree that there is not a consensus on that.
>

Ok

> Line 499: I think Marc has a good point about setting out some guidelines about when it is appropriate to "silently" progress the IANA registry to the new version of Unicode. That text could be inserted here.
>
> Text like the above could be inserted at Line 499, or a separate section could
> be added.

I have not added this as I see discussion on this issue is ongoing.

   Patrik

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 10 Mar 2019 20:30:46 -0700

> If you have better wording, please let me know.
Line 176: Change: "It should though be noted that IETF do not believe the
    underlying issue is resolved as the issue with the code point U+08A1
    is not completely understood."

To: "It should be noted that the review on U+08A1 indicated that it is
not an isolated case and that a number of PVALID code points of long 
standing
may have similar issues. A fuller resolution of this issue may require 
future
changes to IDNA2008 or additional specifications, but there is no 
understanding
yet of what would constitute the best approach."

Something like that.

A./

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Mar 2019 01:41:22 -0400

--On Sunday, March 10, 2019 20:30 -0700 Asmus Freytag
<[email-masked]> wrote:

>...
> not an isolated case and that a number of PVALID code points
> of long standing
> may have similar issues. A fuller resolution of this issue may
> require future
> changes to IDNA2008 or additional specifications, but there is
> no understanding

Patrik, for whatever it is worth, I prefer the above phrasing to
the text I suggested to you off-list although I think
"insufficient understanding" would be more accurate than "no
understanding".   But the differences are just editorial issues;
Asmus and I are in agreement about the principles.

 best,
   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 14:56:50 +0900

On 11 Mar 2019, at 14:41, John C Klensin wrote:

> Patrik, for whatever it is worth, I prefer the above phrasing to the text I suggested to you off-list although I think
> "insufficient understanding" would be more accurate than "no understanding".   But the differences are just editorial issues;

Noted.

I have added Asmus text, and will now start work through your other mail(s).

   paf

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 11 Mar 2019 06:27:51 +0000

Hello Patrik, John, Asmus,

> Asmus and I are in agreement about the principles.

Here I agree with Asmus, and with John, including John's "insufficient 
understanding".

Regards,   Martin.

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 15:47:08 +0900

On 11 Mar 2019, at 15:27, Martin J. Dürst wrote:

> Here I agree with Asmus, and with John, including John's "insufficient understanding".

Done.

   paf

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 11 Mar 2019 07:27:38 +0000

Hello Patrik,

In the draft attached to this mail, the following just hit my eyes:

    4.  Notable Changes Between Unicode 6.3.0 and 11.0.0  . . . . . .   6
      4.1.  Changes between Unicode 6.2.0 and 7.0.0 . . . . . . . . .   6

Are our changes starting at 6.2.0 or 6.3.0? This should be aligned.

Also, for the lists of character numbers that follow, I'd add a sentence 
at the start of each list such as the following:

"Summary of number of characters:" or
"Change in number of chacters in each category:"

Also, when there's no change, I'd suggest to replace
"CONTEXTO did not change from 25" to
"CONTEXTO did not change, at 25" throughout.

Regards,   Martin.

On 2019/03/11 00:52, Patrik Fältström wrote:
> I must go to bed and have not addressed all of this yet, but working on it.
> 
>> 1 intro para 3 (bullet 1): “Assigning code points can create problems 
>> if the newly-assigned     code points are compositions of code points 
>> 4.1 para 3 - “This identical to LATIN….” - presumed meaning “This is 
>> the same situation as with LATIN….”
>>
>>

-- 
Prof. Dr.sc. Martin J. Dürst
Department of Intelligent Information Technology
College of Science and Engineering
Aoyama Gakuin University
Fuchinobe 5-1-10, Chuo-ku, Sagamihara
252-5258 Japan

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 16:30:07 +0900

> Are our changes starting at 6.2.0 or 6.3.0? This should be aligned.

Thanks.

The last version we where looking at (at IETF) was 6.2.0. I presume we should say because of this "between 6.2.0 and 11.0.0", right?

> Also, for the lists of character numbers that follow, I'd add a sentence at the start of each list such as the following:
>

Ok!

> "CONTEXTO did not change, at 25" throughout.

Ok!

   paf

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 11 Mar 2019 07:34:57 +0000

Hello Patrik,

> The last version we where looking at (at IETF) was 6.2.0. I presume we should say because of this "between 6.2.0 and 11.0.0", right?

Yes.

Regards,   Martin.

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Mon, 11 Mar 2019 09:54:14 -0700

>> Here I agree with Asmus, and with John, including John's "insufficient understanding".

I think that tweak is a good improvement.
A./