Subject: [I18ndir] Version 2: Directorate review of unicode11-07

From: Harald Alvestrand <[email-masked]>
Date: Tue, 12 Mar 2019 00:59:21 +0100

Working on the document I have the pen on:

With -08 being wordsmithed on, I think "less is better" for the review
of -07, and have deleted the "nits" section altogether - but delivering
the review is still important for some transparency in the directorate's
work.

I've added some words about the weakness of the review process
description in IDNA2008, and tweaked some more words.

Do you think this is now in a shape where it doesn't make sense to
wordsmith on it any more?

Harald

**

*Directorate review of draft-faltstrom-unicode11-07*

*

Overall conclusion: Almost ready, but needs some updates

    Context issues

The discussion of draft-faltstrom-unicode11 in the directorate has shown
that the directorate members share a number of concerns about the
current state of IDNA, only some of which are directly relevant to this
memo.

IDNA2008 considered limits to what was reasonable to register and use in
the DNS at a number of levels:

  *

    A level of “don’t register stuff that causes confusion”. This
    requires human judgment, and reasonable people may disagree about
    what causes confusion.

  *

    A level of “don’t register stuff that is structurally invalid under
    the relevant writing system”. Aspects of this can be captured in
    rulesets (ICANN’s RZ-LGR efforts fall not this category), but
    requires deep expertise; this is captured in IDNA2008 as the “don’t
    register what you don’t understand” rule.

  *

    A level of “this is stuff that you should never register, and
    applications can reasonably choose to treat it as an error or an
    attack if it ever shows up”. This is the distinction that is
    captured in the classification of codepoints as DISALLOWED, and
    where IDNA2008 (with updates) gives precise rules.

The current document focuses on the last level only - the maintenance of
the distinction between PVALID and DISALLOWED. (It also considers
whether new CONTEXTO and CONTEXTJ rules are needed, but that’s only used
for special cases).

It is clear from directorate discussion that work needs to be done at
the other levels outlined above too, but it is not clear from the
discussion what form that work should take or what fora that work is
reasonably performed in; the work may or may not involve a revision of
the basic IDNA2008 specifications.

We suggest to insert a paragraph in the document describing the context
of the state of IDNA2008, and explain what issues this document does not
attempt to address. Specifically that the conclusion of the document is
what to do regarding Unicode versions up to and including 11, and that
this is not to be used as expectations of future versions of Unicode.

In addition, it’s become clear that IDNA2008 does not specify the
mechanisms and expectations of the review of new versions of Unicode in
enough detail; with the review of a number of versions of Unicode behind
us, we should be able to describe those procedures and expectations
better than IDNA2008 does. However, this may need to happen in another
document than this one.

    Content issues

Section 4.1 does not specify where to find the conclusion of the IETF
discussion on U+08A1.

It is not easy to see from the text whether the algorithms and
procedures will render U+0628 U+0654 an illegal sequence or a legal
sequence. No matter what the resolution is, the document should make it
obvious what the conclusion is (and why).

RFC 5892 states that SPHERICAL ANGLE OPENING UP is DISALLOWED not PVALID:

27D0..2B4C  ; DISALLOWED

Section 4.1 ought to include numbers for how many characters ended up in
DISALLOWED vs PVALID - ideally, for each Unicode version since IDNA2008
was issued.

    Nits

These have been submitted separately to the author, and are not
enumerated here.

*

-- 
Surveillance is pervasive. Go Dark.

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Mar 2019 20:56:43 -0400

Harald,

I will try to study this within the next few hours and get back
to you on details, but...

draft-faltstrom-unicode11-08 was just posted.  If Alexey keeps
draft-faltstrom-unicode11 on the agenda for the 14th, I can't
imagine IESG members who haven't started reading yet deciding to
read the older version, especially since, as we know, there are
extensive changes between the two.

That suggests that your review should really be against -08
because it and -07 are different enough, and -08 fixes enough of
the issues that various of us had about -07, that a report on
-07 is likely to be a lot more confusing than helpful.   There
is also a procedural issue: -08 differs enough from -07 that it
is, at best, not clear that it is proper for the IESG to hold a
vote on -08 on the basis of the IETF Last Call on -07.  

In addition, draft-faltstrom-unicode12, which contains
substantially all of the text of draft-faltstrom-unicode11-08,
has now been posted and I'm confused about what we are working
on (see note preceding this on the list).  

Best,
   john

--On Tuesday, March 12, 2019 00:59 +0100 Harald Alvestrand
<[email-masked]> wrote:

> With -08 being wordsmithed on, I think "less is better" for
> the review of -07, and have deleted the "nits" section
> altogether - but delivering the review is still important for
> some transparency in the directorate's work.
> 
> Do you think this is now in a shape where it doesn't make
> sense to wordsmith on it any more?
> 
> 
> Harald
> 
> 
> **
> 
> The discussion of draft-faltstrom-unicode11 in the directorate
> has shown that the directorate members share a number of
> concerns about the current state of IDNA, only some of which
> are directly relevant to this memo.
> 
> 
> IDNA2008 considered limits to what was reasonable to register
> and use in the DNS at a number of levels:
> 
> 
>   *
> 
>     A level of "don't register stuff that causes
> confusion". This     requires human judgment, and reasonable
> people may disagree about     what causes confusion.
> 
>   *
> 
>     A level of "don't register stuff that is structurally
> invalid under     the relevant writing system". Aspects of
> this can be captured in     rulesets (ICANN's RZ-LGR efforts
> fall not this category), but     requires deep expertise; this
> is captured in IDNA2008 as the "don't     register what
> you don't understand" rule.
> 
>   *
> 
>     A level of "this is stuff that you should never
> register, and     applications can reasonably choose to treat
> it as an error or an     attack if it ever shows up". This
> is the distinction that is     captured in the classification
> of codepoints as DISALLOWED, and     where IDNA2008 (with
> updates) gives precise rules.
> 
> 
> The current document focuses on the last level only - the
> maintenance of the distinction between PVALID and DISALLOWED.
> (It also considers whether new CONTEXTO and CONTEXTJ rules are
> needed, but that's only used for special cases).
> 
> 
> It is clear from directorate discussion that work needs to be
> done at the other levels outlined above too, but it is not
> clear from the discussion what form that work should take or
> what fora that work is reasonably performed in; the work may
> or may not involve a revision of the basic IDNA2008
> specifications.
> 
> 
> We suggest to insert a paragraph in the document describing
> the context of the state of IDNA2008, and explain what issues
> this document does not attempt to address. Specifically that
> the conclusion of the document is what to do regarding Unicode
> versions up to and including 11, and that this is not to be
> used as expectations of future versions of Unicode.
> 
> 
> In addition, it's become clear that IDNA2008 does not
> specify the mechanisms and expectations of the review of new
> versions of Unicode in enough detail; with the review of a
> number of versions of Unicode behind us, we should be able to
> describe those procedures and expectations better than
> IDNA2008 does. However, this may need to happen in another
> document than this one.
> 
> 
>     Content issues
> 
> 
> Section 4.1 does not specify where to find the conclusion of
> the IETF discussion on U+08A1.
> 
> procedures will render U+0628 U+0654 an illegal sequence or a
> legal sequence. No matter what the resolution is, the document
> should make it obvious what the conclusion is (and why).
> 
> 
> Section 4.1 ought to include numbers for how many characters
> ended up in DISALLOWED vs PVALID - ideally, for each Unicode
> version since IDNA2008 was issued.
> 
> 
>     Nits
> 
>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 12 Mar 2019 02:13:04 +0100

Top-posting, since this is an overarching item....

I'm somewhat a stickler for delivering what has been asked for - we were
asked for a review of -07, we should deliver a review of -07.
I think the rational action of the AD in this situation is to say "thank
you, I think -08 addresses your issues, can you review that?"

The review of -08 would of course be quite different.

But I like having the -07 review as part of the public record.

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Mar 2019 22:13:28 -0400

--On Tuesday, March 12, 2019 02:13 +0100 Harald Alvestrand
<[email-masked]> wrote:

> I'm somewhat a stickler for delivering what has been asked for
> - we were asked for a review of -07, we should deliver a
> review of -07. I think the rational action of the AD in this
> situation is to say "thank you, I think -08 addresses your
> issues, can you review that?"

Seems reasonable.  However the problem with that logic is that,
if we are asked to review -07 as part of an IETF Last Call/ IESG
review process, and a -08 is posted that is significantly
different, that should immediately stop the process with review
(and IETF LC) reinitiated with -08.   The present situation is
even a little more complex because i18n review resources are
very limited and asking up (or the IETF community) to approve
draft-faltstrom-unicode11 and then to almost immediately start
on draft-faltstrom-unicode12 (even in the version with all of
the unicode11 stuff stripped out that Patrik didn't have time to
construct and post) would be a significant waste of those
resources and would have the effect of preventing us from
getting to issues that have been pushed aside.   Especially
since the IESG processing and approving -08 at this point with
no IETF LC and no directorate review of that version would be in
bad taste and might cause appeals, I'd rather that we be thanked
for our extensive contributions to moving things along,
including the review that was part of that process, that the
unicode11 draft be abandoned, and that we (and the IESG) then
start looking at the unicode12 draft in a slightly more
systematic and orderly way, building on the work we've done
already. 

Just my opinion, of course.

> But I like having the -07 review as part of the public record.

Works for me and I will try to review your notes although it
feels to some extent like a waste of time unless the IESG is
actually going to vote on -07.  However (and you may have said
this already), the other side of the logic above is that one
conclusion, maybe even consensus, of the directorate (including
the document author) is that -07 simply was not ready for prime
time without at least the extensive changes reflected in -08.
That is _lots_ different from "ready with nits" or any similar
remark.  

FWIW, part of the motivation for my two very long
analyses/reviews was to document how little ready for IETF
action -07 is/was.

   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 12 Mar 2019 12:36:37 +0900

On 12 Mar 2019, at 11:13, John C Klensin wrote:

> The present situation is even a little more complex because i18n review resources are very limited and asking up (or the IETF community) to approve draft-faltstrom-unicode11 and then to almost immediately start on draft-faltstrom-unicode12 (even in the version with all of the unicode11 stuff stripped out that Patrik didn't have time to construct and post) would be a significant waste of those
> getting to issues that have been pushed aside.

The difference between 07 and 08 can be found here:

<https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-faltstrom-unicode11-08.txt>

Regarding draft-faltstrom-unicode12, yes, it is draft-faltstrom-unicode11 but with data for Unicode 12.0.0. I did produce it as I heard a few voices saying we could as well say something about Unicode 12.0.0.

Please note though a few things:

1. The algorithm is normative, not the result of applying the algorithm on any Unicode version. Because of this, people might already now apply IDNA2008 on Unicode 11.0.0 and 12.0.0 (of course).

2. Given the different steps I go through (which I did explain in a different email), publication of Unicode 12.0.0 would not trigger me notify IESG about any notable change from 11.0.0.

3. Because of this, if draft-faltstrom-unicode11 is approved (with a backward compatible exception added or not), and we because of that know how to deal with the normative algorithm due to the incompatible change made to Unicode 11.0.0, I do not see any need for IETF to say anything about Unicode 12.0.0. So a "stripped down draft-faltstrom-unicode12" is not needed.

4. What IETF should "worry" about is the meta issue on whether the model we use is still working.

5. Because of this, I personally think publication of draft-faltstrom-unicode11 is what we should do, after ensuring we really agree on the content.

   Patrik

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 12 Mar 2019 10:17:43 +0000

> asked for a review of -07, we should deliver a review of -07.

I agree with that. There are a lot of changes between -07 and -08, and 
while the changes themselves are visible to people outside this 
directorate, the reasons are not. I think it's important to show the 
reasons to the rest of the IETF.

Regards,   Martin.

-- 
Prof. Dr.sc. Martin J. Dürst
Department of Intelligent Information Technology
College of Science and Engineering
Aoyama Gakuin University
Fuchinobe 5-1-10, Chuo-ku, Sagamihara
252-5258 Japan

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 12 Mar 2019 10:30:36 +0000

Hello Patrik, John,

>> getting to issues that have been pushed aside.

I'm not that skeptical. I think here in this group, we are getting into 
something of a rhythm. Also, a draft-faltstrom-unicode12 (without the 
stuff up to Unicode 11) should be very short and easy, without too much 
time needed for review.

> 3. Because of this, if draft-faltstrom-unicode11 is approved (with a backward compatible exception added or not), and we because of that know how to deal with the normative algorithm due to the incompatible change made to Unicode 11.0.0, I do not see any need for IETF to say anything about Unicode 12.0.0.

How will you or the IETF tell everybody else that it is okay with 
Unicode 12.0.0?

It can't be that they each have to go through the same steps that you 
did (because they may not know these steps, and we should avoid useless 
duplication of work), conclude that you wouldn't have had to inform the 
IESG, and then move on.

On top of that, what everybody else remembers of the IETF is that they 
they got very nervous because of a single new character with hamza, and 
took four or so years to sort out, so nobody outside the IETF might want 
to risk a guess.

So I still see something missing here. But maybe that's just me.

Regards,   Martin.

> So a "stripped down draft-faltstrom-unicode12" is not needed.
>

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 12 Mar 2019 10:37:25 +0000

> stuff up to Unicode 11) should be very short and easy, without too much

Just for the record, I'm fairly indifferent of whether we do 
draft-faltstrom-unicode11 (with whatever followup is needed) or 
draft-faltstrom-unicode12.

What I'd very strongly advocate is that we spend as little as possible 
on this question, to save cycles for more important stuff. Ideally, I'd 
prefer our AD just tells us what's preferred.

Regards,   Martin.

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 12 Mar 2019 22:15:18 +0900

On 12 Mar 2019, at 19:30, Martin J. Dürst wrote:

> How will you or the IETF tell everybody else that it is okay with Unicode 12.0.0?

I do not have to. The algorithm is normative. I tell IESG if I see potential issues with a version. Otherwise I am quiet.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 15:00:01 -0400

--On Tuesday, March 12, 2019 22:15 +0900 Patrik Fältström
<[email-masked]> wrote:

> I do not have to. The algorithm is normative. I tell IESG if I
> see potential issues with a version. Otherwise I am quiet.

And _that_ is part of the reason I prefer to go with
draft-faltstrom-unicode12 as the comprehensive catch-up version.
It would be really easy to incorporate text into that document
that says, more or less, 

"One of the important things about this analysis is that most of
these versions of Unicode did not change derived property values
for any code points that appeared in the prior version and so,
given the provisions of RFC 5892, did not even need to be
reported to the IESG (or the community).  While such changes did
occur after 10.0 (reflected in 11.0), the analysis of 12.0
effectively confirms that those changes were the exception, as
we expected, rather than a new trend.  As a result, there will
be no future versions of this type of document and analysis
unless non-backward-compatible changes or other problems are
discovered in a new version of Unicode."

That would close the loop and adjust expectations appropriately.
I'd be a bit concerned about closing the loop with 11.0,
precisely because it wouldn't be possible to say conclusively
that the problem code points in 11.0 do not indicate a trend in
which problem code points can be predicted in future versions.
I can think of ways to get around that and put a comment like
the above into -unicode11, but they would essentially be a trick
to allow processing something called draft-faltstrom-unicode11
in spite of having changed both the scope and a large fraction
of the substantive text after IETF LC.   Someone would almost
certainly object to, and appeal, such a trick, thereby
eliminating any possible time advantage from trying it (it might
even be me if no one else got there first).

In any event, if our plan is "no more I-Ds for future versions
of Unicode unless there are problems", whichever draft the IETF
adopts now should clearly say that rather than leaving people
wondering if "no I-D" equals "no analysis yet".

I do have two other concern about that "no more I-Ds" plan (the
second one is an editorial strategy issue that I will cover in a
separate note).  While issues and problems with, or identified
by, new code points are less likely and harder to identify than
code points whose derived properties have changed, especially
given the availability of an easy mechanical search for the
latter.  I believe the search for such new code point problems
is required by RFC 5892, that is it important, and that the most
important lesson from the Hamza incident is that we cannot
pretend that such cases are impossible and hence can be ignored.
I'm concerned that if Patrik (or his successor as Designated
Expert who will, because of the passage of time, almost
certainly be less tied into the history surrounding IDNA2008
than he is) quietly and efficiently does the review for derived
property changes, decides that there is nothing to report, and
hands a table off to IANA, the review of the new code points
will simply get lost and not happen.   

Just as we need to get comments that imply the IANA tables are
normative or that there is an official/approved version of
Unicode for IDNA2008 out of our discussions and documents, we
need to stop implying that the discovery of the U+08A1 issue was
about a single code point.  If it had been, I'm convinced we
would have made a decision, perhaps an arbitrary one, about
whether it was worth leaving PVALID or DISALLOWing it by
exception, and we would have moved on.  Instead, the issue was
that examination of U+08A1 led to our discovering that a design
assumption that had a significant (at least IMO) impact on the
design of the IDNA2008 rule set was incorrect.  That design
assumption was not something we didn't know enough to think
about.  Instead, it was one about which we had asked very
specific questions and gotten what we interpreted as assurances
that the case was not significant in Unicode versions through
5.0 and would not arise at all in the future.  Whether we
misunderstood the answers to our questions, didn't ask precisely
the right questions, or were deliberately misled makes little
difference at this point -- a discovery that parts of the rule
set are based on an incorrect assumption deserves being brought
to the attention of the community and carefully evaluated, even
if the conclusion is that nothing can be done at the protocol
level.  It should not be dismissed and minimized by claims that
it is a single code point issue.

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 19:27:01 -0400

Harald, 

Notes on your draft below.  Sorry not to have it to you sooner,
but I --and others in the directorate who have been deeply
involved in IDNA2008 and these issues, obviously including
Patrik -- have been left with a choice between concentrating on
content and substance versus fussing with report drafts.  In
retrospect, I think it would have been of more benefit to the
IETF and the Internet had you been able to concentrate on the
substantive issues and text as well rather than trying to
produce a summary.  I think the report should point that out
(specifics below).  If there is a lesson in this, it is that the
roles of "review team" and "collection of experts on the subject
matter who are expected to help sort the documents out" may be
incompatible, at least unless relevant ADs are very conservative
and careful about the state of documents when IETF Last Calls or
initiated (and/or ready to pull the plug quickly on the Last
Call and IESG review process if substantive problems requiring
significant new text are uncovered).

While I accept your "we were asked to review -07 and therefore
should produce a review of -07" reasoning, I think that, given
the discussions in the directorate that led to changes from -07
to -08, it is inappropriate to say "but needs some updates" and
move on.

Specifics inline below.

> some transparency in the directorate's work.

ok.

> sense to wordsmith on it any more?

I can't really answer that question because I don't consider
most of the comments below to be "wordsmithing".  Perhaps only
our definitions differ.

> Overall conclusion: Almost ready, but needs some updates

Suggested alternate overall conclusion:  Intensive reviews of
this document by multiple directorate participants identified a
number of technical problems, missing information, and strategic
issues that strongly suggest the document is not ready for
publication.   A new version that reflects solutions to many of
the issues raised is in progress, but it is significantly
different from -07.

Comment: that is a conclusion about -07 that reflects actual
discussions.  It is not in any way a comment on -08.  If -08
were addressed, it would clearly be less harsh.   But "Almost
ready, but needs some updates" would be appropriate to a
document that needed a rewritten paragraph or two or some
relatively minor corrections, perhaps even things that could be
done by an RFC Editor note from the IETF.   Based on either the
discussions on the list or what the directorate (including
Patrik) decided to do in -08, that certainly does not describe
-07.

> are directly relevant to this memo.

Suggested addition:

The same discussions identified some material in the document
that was outside what the directorate believes are part of those
more general concerns rather than falling within the natural
scope of the document and has recommended that that text be
removed.

Comment:

Keep in mind that the differences between -07 and -08 are not
limited to additions.  Some of the statements that drifted over
the line from IDNA2008 into specific recommendations for
registries and their structure are good examples.  We also
convinced Patrik to remove some paragraphs as overreaching and
not quite right. 

> people may disagree about  what causes confusion.

Comment: 

I don't remember IDNA2008 talking about confusion at all and I
probably would have had to to write that text.   The issues with
normalization and when it might be perceived as failing are
about accurate technical matching issues, not confusion, even
though getting the matching wrong could cause confusion.  The
term "confusion" did come up several times in our discussions,
but I suggest it is one of those topics that, paraphrasing your
words above, "are NOT directly relevant to this memo".  AFAICT,
the only context in which confusion is mentioned in -07 (or -08)
is as a Security Consideration.  If we were being strict about
scope and IDNA2008, it doesn't belong there but I don't see
that as an issue worth pushing (indeed, I think it would be
undesirable to push).  

Suggestion: 

If you want to retain the able, modify the end of the first
paragraph to read "...use in the DNS at a number of levels,
specifically including levels and issues that IDNA2008 did not
believe could be addressed as part of the protocol that address
the DNS globally."   

(or words to that effect)

> you don't understand" rule.

"fall not this category" does not parse.  "Fall into..." may
have been what you intended.

Suggestion: Change "Aspects of this can be captured... this
category)" to "Many believe that aspects ... this category, but
the IETF has never taken a position of the utility and impact of
that approach)"

Comment: I know that several people on the directorate believe
that there is great value for the use of that work in zones who
are not quite up to "register only what you understand" or who
need help with that understanding.  Although I use a lot of
qualifying language, I'm one of them.   However, because there
are a number of scripts -- and hence IDNA2008 PVALID code points
-- whose active users are unlikely to meet the conditions for
establishing a generation panel (e.g., money and having been
alive in the last century or even less recently) and because
while there are a couple of RFCs describing aspects of the
approach, the IETF has never gone further into the business of
advising registries about specifics than the recommendations of
RFC 6912.  So, although probably your wording is ok because it
talks about the root zone, tread carefully here.

> updates) gives precise rules.

Comment: Ok, but note the lookup applications checking for
DISALLOWED code points in labels and rejecting those labels
rather than looking them up is not something that "applications
can reasonably choose"; Section 5.4 of RFC 5891 says "MUST be
rejected".  I'm reasonably sure that we've got all of these
details right in -08.  If -07 does not, it is still reasonable
for you to borrow from its description but not to imply that
description represents directorate consensus.  So either
explicitly identify your summary as a paraphrase of -07's
description of the situation or get it right and present it as a
directorate summary.

> or may not involve a revision of the basic IDNA2008

Ok, good.  Given documents already on the table, "a revision of
or clarifications to" would be a bit more accourate.

> IDNA2008 does. However, this may need to happen in another

Good.  Given Patrik's recent notes, it might be appropriate to
say "However, while some directorate members are ready ti
recommend approaches for the future, this may need..."

> RFC 5892 states that SPHERICAL ANGLE OPENING UP is DISALLOWED

Comment: The table in Appendix B.1 is not normative so, while
the above strong was presumably found there, "RFC 5892 states"
is inappropriate.

Suggestion: "When the rules of RFC 5892 are applied to SPHERICAL
ANGLE OPENING UP (U+29A1), they produced DISALLOWED, not PVALID,
as noted there in Appendix B.1:"   Then 27D0... line works.

> version since IDNA2008 was issued.

Ok.  It is actually not clear to me that the counts are useful,
but I defer to  those who think it is.

> These have been submitted separately to the author, and are not

      ----

My summary of the summary of both -07 and our discussions about
it is that -07 is not ready for publication as an RFC without
significant modification.  Whatever your report says, it is
vital (and consistent with directorate discussions) that point
comes across very clearly.  And, again, that isn't "Almost
ready, but needs some updates".   I'm particularly worried about
IESG members who want as little to do with i18n as possible and
who might take "Almost ready, but..." as a basis for voting "no
objection"... and I think that there is evidence that ADs who
want as little to do with i18n as possible are a significant
majority of the IESG.  My hope is that Alexey will just pull the
ballot and do whatever is needed to start moving either -08 or
-unicode12 forward, preferably not launching another IETF LC
before a decision is make about which of those paths to go down,
we've had an opportunity to do a careful review pass on
whichever I-D/ path is chosen, and, if it is 12.0, there is time
to do a serious "potentially problematic code point" review in
addition to the "changes in derived properties" review.   You
can't (and shouldn't) say any of that in a review focused
exclusively on -07 but it isn't appropriate to guess at what
Alexey is going to do before he decides and tells us either.  So
we need to be very, very, clear that the directorate consensus
is that publishing -07 would be completely inappropriate.  I
would prefer that we at least imply that approving -07 now and
letting details be sorted out at AUTH48 or doing a quick
substitution of -08 (which is not covered by your report) would
be bad ideas and probably procedurally inappropriate, but I
don't know how to do that in a -07 report.

best,
    john

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 12 Mar 2019 23:45:38 +0000

Hello Patrik, others,

> I do not have to. The algorithm is normative. I tell IESG if I see potential issues with a version. Otherwise I am quiet.

It may be that that's what the RFC(s) say, and that's what was intended, 
but at least to me, that would look like a protocol design error. I'd be 
very disappointed if the IETF made such an error.

Now after checking RFC 5892, it seems that the story is slightly better. 
To see whether Patrik checked the latest Unicode version and is okay 
with it, one has to check 
https://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml#idna-tables-properties. 
There one sees a reference which currently reads "[Unicode Character 
Database 6.3.0]". If that reference gets updated, then that's the canary 
that indicates that a newer version of Unicode has successfully passed 
Patrik's checks.

Did I get that right? If yes, it would be good to make that clear in the 
draft we are working on.

Regards,   Martin.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 13 Mar 2019 00:54:53 +0100

I think one (short) document we could write as a result of this round is
a document that describes the review procedure.

I want to have registered somewhere public (with a process that has
lower overhead than publishing an RFC) the determination that "the IETF
reviewer has looked at Unicode version X.Y and has found no problematic
changes from Unicode version A.B".

I am also happy to go with unicode-12 as the document we publish for
this round, but want instructions from our AD on that point; to my mind,
it's an AD decision, not a directorate or reviewer decision.

Formally, that will make a new Last Call necessary, I think. I don't see
that as being a big deal.

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 12 Mar 2019 18:10:07 -0600

[ no hats ]

> a document that describes the review procedure.

That would indeed be very helpful.

> it's an AD decision, not a directorate or reviewer decision.

Yes - now that 12.0 has been published, we might as well be as up to
date as possible. The reasons John adduces (essentially, there's no
pattern here) are also relevant.

> that as being a big deal.

Agreed.

Peter

---

From: "Marc Blanchet" <[email-masked]>
Date: Tue, 12 Mar 2019 20:11:38 -0400

> a document that describes the review procedure.

that is what I’ve been proposing in my initial review.

Marc.

> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 12 Mar 2019 18:17:30 -0600

One small point...

On 3/12/19 5:27 PM, John C Klensin wrote:

> I don't remember IDNA2008 talking about confusion at all 

Section 4.4 ("Visually Similar Characters") of RFC 5890 does talk about
confusion, but perhaps not in the precise context you have in mind.

Peter

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 21:43:08 -0400

--On Wednesday, March 13, 2019 00:54 +0100 Harald Alvestrand
<[email-masked]> wrote:

> I think one (short) document we could write as a result of
> this round is a document that describes the review procedure.

That would be very good.  That document could also clarify what
we want IANA to keep.   I think one implication of my note about
whether or not to keep the tables (and which ones) with
draft-faltstrom-XX when it becomes an RFC is that just keeping
the most recent version of the full tables supplied to them may
not be enough.    

> I want to have registered somewhere public (with a process
> that has lower overhead than publishing an RFC) the
> determination that "the IETF reviewer has looked at Unicode
> version X.Y and has found no problematic changes from Unicode
> version A.B".

I'm good with that.  My concern is having no record of anything
being done other than new tables showing up at IANA.   Given how
long it seems to take to get anything new into the datatracker
(remember that was the reason given for the long time it took to
set up the directorate).  That is probably not the answer.
However, especially if we want IANA to keep tables corresponding
to all versions of Unicode, if they were willing to cite an I-D
(makes me nervous, but I'd guess it has been done before) a
sequence like:

 * Post I-D with quick summary, maybe much like Section
	4.2 of draft-faltstrom-unicode11-08 and some boilerplate
	that we'd need to sort out only once.  It could include
	derived tables or not at Patrik's preference.
	
 * IANA posts table.
	
 * Stop.

Then, for the cases in which something actually is found, the
I-D gets more detail, it gets flagged to the IESG, it certainly
moves on to be an RFC if 5892 (or some other part of IDNA) is
actually updates (and we decide whether RFC publication is
worthwhile if we decide to _not_ make changes) and we move on.
That assumes that posting an I-D, which would usually begin and
end with a -00 is pretty cheap in terms of time; the time and
aggravation arises in walking something through the consensus
process and into RFC publication.

Just one quick idea. I think there are likely others that would
be as good or better, but I do think we should leave tracks.

> I am also happy to go with unicode-12 as the document we
> publish for this round, but want instructions from our AD on
> that point; to my mind, it's an AD decision, not a directorate
> or reviewer decision.

Yep.

> Formally, that will make a new Last Call necessary, I think. I
> don't see that as being a big deal.

Formally, I don't see how to avoid a new Last Call on
unicode11-08, so it is probably six of one and a half dozen of
the other.  But up to Alexey.

best,
    john

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 13 Mar 2019 11:07:40 +0900

On 13 Mar 2019, at 10:43, John C Klensin wrote:

> However, especially if we want IANA to keep tables corresponding to all versions of Unicode, if they were willing to cite an I-D (makes me nervous, but I'd guess it has been done before) a
> sequence like:
>

As the appointed expert that would be to follow this process, I have no problems with this.

It is a transparent way for me to explain what I already do.

So, from my perspective, works.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 22:29:30 -0400

--On Tuesday, March 12, 2019 18:17 -0600 Peter Saint-Andre
<[email-masked]> wrote:

> Section 4.4 ("Visually Similar Characters") of RFC 5890 does
> talk about confusion, but perhaps not in the precise context
> you have in mind.

Proving once again that I can't remember a few years (or most of
a decade) later what I wrote. :-(

The rest of Section 4 (Security Considerations) in that document
is also interesting because it goes much further in the
direction of advice to implementers and registries than I
remembered going and 4.2 contains the analysis to address a
recent argument (elsewhere) about U-label lengths and whether an
implementation restricting them to 63 octets in UTF-8 was
appropriate.  

I'd like to believe that forgetting that was a result of not
having looked carefully through the full IDNA2008 document
collection since we went into a wait state in the last half of
2014 but it might be a sign on approaching senility. 

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 22:35:30 -0400

Sorry Martin, should have read this before sending my most
recent note.  Inline below...

--On Tuesday, March 12, 2019 23:45 +0000 "Martin J. Dürst"
<[email-masked]> wrote:

>> I do not have to. The algorithm is normative. I tell IESG if
>> I see potential issues with a version. Otherwise I am quiet.
>> 
>>     Patrik
> 
> It may be that that's what the RFC(s) say, and that's what was
> intended,  but at least to me, that would look like a protocol
> design error. I'd be  very disappointed if the IETF made such
> an error.

Agreed, especially given that something should trigger the
opportunity for a review of the new code points even if that
isn't assigned to anyone in particular and we decide to let
things go if no one speaks up after a reasonable period.   On
the other hand, even "let things go" would require an update to
5982 as I read it and that is why Harald's suggestion about a
new document that describes the review procedure and our
expectations of it seems important as well as useful.

> Now after checking RFC 5892, it seems that the story is
> slightly better.  To see whether Patrik checked the latest
> Unicode version and is okay  with it, one has to check 
> https://www.iana.org/assignments/idna-tables-6.3.0/idna-tables
> -6.3.0.xhtml#idna-tables-properties.  There one sees a
> reference which currently reads "[Unicode Character  Database
> 6.3.0]". If that reference gets updated, then that's the
> canary  that indicates that a newer version of Unicode has
> successfully passed  Patrik's checks.
> 
> Did I get that right? If yes, it would be good to make that
> clear in the  draft we are working on.

Not a very good canary because there is that requirement for a
review of the new code points.    While I don't remember when
(although I can guess) or in what context, I have a very clear
memory of Patrik saying that he was willing to automate and do
the comparison for changes in derived property values but not
the review of new code points.  The latter job was, IIR, never
assigned to anyone in particular.  I looked through 6.x before
RFC 6452 was published; I have a vague recollection that maybe
Paul Hoffman did so too.  And I looked through 7.0, which is how
the apparent combining Hamza irregularity was first spotted.
Then I stopped because, if the community could not deal with the
issues raised in draft-klensin-idna-5892upd-unicode70 [1], there
seemed little point.

So, while the current information for 6.3 is consistent with
both reviews having been done, if the I-D causes a table to be
posted with information for 11.0 (and/or 12.0), it would
indicate only one of the reviews was performed.

In addition, if we expect people to be able to pull tables from
IANA that correspond to whichever version of Unicode they are
running (which is the IDNA2008 requirement) those other tables
are either not on the IANA site or are beyond my limited ability
to find.

We really need to get our ducks in a row about those files as
well as the review process.  Assuming we can figure out what we
want to do (I agree with Harald that it should not be hard),
sorting all of the needed record-keeping out should be
straightforward.  

But, again following Harald's lead, let's think in terms of a
separate document rather than putting work in the critical path
to getting draft-faltstrom finished (in either a unicode11 or
unicode12 version).  That does require being clear enough about
what we are doing going forward that we don't end up with
confusion text in draft-faltstrom but, even if we can't reach
agreement in principle quickly, I can think of several ways to
do that.

best,
    john

---

From: "Alexey Melnikov" <[email-masked]>
Date: Wed, 13 Mar 2019 12:41:11 -0400

> it's an AD decision, not a directorate or reviewer decision.

After discussing this with some of IESG members I think we should go with unicode12.

In order to help preserve document history: Patrick, can you post unicode12-00 as unicode11-09? I will authorize exception for this.

Agreed.

---

From: John C Klensin <[email-masked]>
Date: Wed, 13 Mar 2019 14:21:02 -0400

Alexey,

I'm ok either way, but why not just mark the unicode12 draft as
replacing the unicode11 one in the tracker?   That would save
Patrik a tiny bit of extra effort, make the document name
correspond to what we are actually doing, and avoid a posting
exception and whatever confusion might result from having two
documents in the archive with identical content.

If the answer is that datatracker tooling makes that complicated
because of the already-initiated IETF LC on unicode11-07, so be
it -- again, I'm ok either way and the goal of trying to
minimize extra work and confusion is not likely to be a big deal
in this case.

best,
   john

--On Wednesday, March 13, 2019 12:41 -0400 Alexey Melnikov
<[email-masked]> wrote:

>> that point; to my mind, it's an AD decision, not a
>> directorate or reviewer decision.
> 
> In order to help preserve document history: Patrick, can you
> post unicode12-00 as unicode11-09? I will authorize exception
> for this.
> 
>> Formally, that will make a new Last Call necessary, I think.
>> I don't see that as being a big deal.
> 
> Agreed. 
> 
>> > --On Tuesday, March 12, 2019 22:15 +0900 Patrik F??ltstr??m
>> > <[email-masked]> wrote:
>> > 
>> >> On 12 Mar 2019, at 19:30, Martin J. D??rst wrote:
>> >> 
>> >> I do not have to. The algorithm is normative. I tell IESG
>> >> if I see potential issues with a version. Otherwise I am
>> >> quiet.
>> > draft-faltstrom-unicode12 as the comprehensive catch-up
>> > version. It would be really easy to incorporate text into
>> > that document that says, more or less, 
>> > 
>> > "One of the important things about this analysis is that
>> > most of these versions of Unicode did not change derived
>> > property values for any code points that appeared in the
>> > prior version and so, given the provisions of RFC 5892, did
>> > not even need to be reported to the IESG (or the
>> > community).  While such changes did occur after 10.0
>> > (reflected in 11.0), the analysis of 12.0 effectively
>> > confirms that those changes were the exception, as we
>> > expected, rather than a new trend.  As a result, there will
>> > That would close the loop and adjust expectations
>> > appropriately. I'd be a bit concerned about closing the
>> > loop with 11.0, precisely because it wouldn't be possible
>> > to say conclusively that the problem code points in 11.0 do
>> > not indicate a trend in which problem code points can be
>> > predicted in future versions. I can think of ways to get
>> > around that and put a comment like the above into
>> > -unicode11, but they would essentially be a trick to allow
>> > processing something called draft-faltstrom-unicode11 in
>> > spite of having changed both the scope and a large fraction
>> > of the substantive text after IETF LC.   Someone would
>> > almost certainly object to, and appeal, such a trick,
>> > thereby eliminating any possible time advantage from trying
>> > it (it might even be me if no one else got there first).
>> > 
>> > In any event, if our plan is "no more I-Ds for future
>> > versions of Unicode unless there are problems", whichever
>> > draft the IETF adopts now should clearly say that rather
>> > than leaving people wondering if "no I-D" equals "no
>> > analysis yet".
>> > 
>> > I do have two other concern about that "no more I-Ds" plan
>> > (the second one is an editorial strategy issue that I will
>> > cover in a separate note).  While issues and problems with,
>> > or identified by, new code points are less likely and
>> > harder to identify than code points whose derived
>> > properties have changed, especially given the availability
>> > of an easy mechanical search for the latter.  I believe the
>> > search for such new code point problems is required by RFC
>> > 5892, that is it important, and that the most important
>> > lesson from the Hamza incident is that we cannot pretend
>> > that such cases are impossible and hence can be ignored.
>> > than he is) quietly and efficiently does the review for
>> > derived property changes, decides that there is nothing to
>> > report, and hands a table off to IANA, the review of the
>> > new code points will simply get lost and not happen.   
>> > 
>> > Just as we need to get comments that imply the IANA tables
>> > are normative or that there is an official/approved version
>> > of Unicode for IDNA2008 out of our discussions and
>> > documents, we need to stop implying that the discovery of
>> > the U+08A1 issue was about a single code point.  If it had
>> > been, I'm convinced we would have made a decision, perhaps
>> > an arbitrary one, about whether it was worth leaving PVALID
>> > or DISALLOWing it by exception, and we would have moved on.
>> > Instead, the issue was that examination of U+08A1 led to
>> > our discovering that a design assumption that had a
>> > significant (at least IMO) impact on the design of the
>> > IDNA2008 rule set was incorrect.  That design assumption
>> > was not something we didn't know enough to think about.
>> > Instead, it was one about which we had asked very specific
>> > questions and gotten what we interpreted as assurances that
>> > the case was not significant in Unicode versions through
>> > misunderstood the answers to our questions, didn't ask
>> > precisely the right questions, or were deliberately misled
>> > makes little difference at this point -- a discovery that
>> > parts of the rule set are based on an incorrect assumption
>> > deserves being brought to the attention of the community
>> > and carefully evaluated, even if the conclusion is that
>> > nothing can be done at the protocol level.  It should not
>> > be dismissed and minimized by claims that it is a single
>> > code point issue.
>> > 
>> >     john
>> > 
>> > 
>> > 
>> 
>> --

---

From: "Alexey Melnikov" <[email-masked]>
Date: Wed, 13 Mar 2019 15:29:46 -0400

Hi John,

> minimize extra work and confusion is not likely to be a big deal

You are right. Somebody from IESG asked to update in place to preserve IESG comments, but using datatracker "replaces draft-XXX" relationship is a cleaner and more natural way to do this with datatracker.

Best Regards,
Alexey

---

From: "Alexey Melnikov" <[email-masked]>
Date: Wed, 13 Mar 2019 15:46:22 -0400

John,

On Tue, Mar 12, 2019, at 11:27 PM, John C Klensin wrote:
> majority of the IESG.

IESG was made aware of various discussions on this mailing list.

Let's concentrate on getting draft-falstrom-unicode12 done as quickly as possible. I would expect full cooperation on getting this document done. Should you or other Directorate members decide to propose new documents to write, that would be welcomed, as long as this doesn't delay this draft.

Best Regards,
Alexey

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 14 Mar 2019 06:42:45 +0900

> In order to help preserve document history: Patrick, can you post unicode12-00 as unicode11-09? I will authorize exception for this.

It will take me a day or two as ICANN makes me quite busy for the next say 24 hours. And that I got some really good comments.

You should have it no later than Sunday. When I am in Sweden again.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 14 Mar 2019 06:47:48 +0900

So replace with -12 is what I should do?

   Patrik

P.S. For me it does not matter....

---

From: John C Klensin <[email-masked]>
Date: Wed, 13 Mar 2019 20:50:41 -0400

--On Wednesday, March 13, 2019 15:46 -0400 Alexey Melnikov
<[email-masked]> wrote:

>> My summary of the summary of both -07 and our discussions
>> about it is that -07 is not ready for publication as an RFC
>> without significant modification.  Whatever your report says,
>> it is vital (and consistent with directorate discussions)
>> that point comes across very clearly.  And, again, that isn't
>> "Almost ready, but needs some updates".   I'm particularly
>> worried about IESG members who want as little to do with i18n
>> as possible and who might take "Almost ready, but..." as a
>> basis for voting "no objection"... and I think that there is
>> evidence that ADs who want as little to do with i18n as
>> possible are a significant majority of the IESG.
> 
> Let's concentrate on getting draft-falstrom-unicode12 done as
> quickly as possible. I would expect full cooperation on
> getting this document done. Should you or other Directorate
> members decide to propose new documents to write, that would
> be welcomed, as long as this doesn't delay this draft.

Alexey,

I think we are all agreed about getting a document that reflects
Unicode 12 done as soon as possible.  The difficulty I'm
specifically concerned about is a combination of two things.
First, even if all the other issues are ignored, the other side
of "it has taken us four years to get this document out" ought
to be, IMO, "let's at least do it right if we can do so without
significant additional delay" rather than "let's rush it out
just to get something published".  I don't think anyone is going
to delay it longer: my guess is that I can speak for Patrik as
well as myself and at least most others if I tell you we are
thoroughly sick of it and understand that the best way to make
it go away is to get it approved and published.

The other reason is why I suggested trying to get things like
the Unicode version review procedures and content, and the IANA
registry plan, a little bit more under control is that the
present situation, to be accurate, has required some rather
convoluted language in draft-faltstrom-unicodeXX.  If we are at
least clear about where we are going (I don't thing finished, or
even significantly started, documents are needed to get
draft-faltstrom- out) we can clean up that language, make the
document clearer and a tad shorter, and then put it through a
Last Call process that should not produce pages of issues
(whether minor or major) in IETF review, risks of appeals
because things that are required aren't done, and, although I
think -08 (and unicode12) take care of all of them, statements
that are either inaccurate or misleading in the document.  There
are two other aspects of that reason.   One I've mentioned
before and that is that unnecessary time pressure leads to haste
and stress and haste and stress lead to completely avoidable
errors.  The other is that is we either make mistakes with the
document or push the decisions that interact with it off until
after it is published, we set ourselves up for either errata
(which might (or might not) interact with the IANA tables in
ways that would waste a lot of energy sorting out) or a need for
actual clarifications and updates.  Not a good use of time;
let's just get the thing right.

So, again, if the IESG is feeling a sense of urgency, by all
means approve Alissa's recommendation and forward the Unicode 11
tables to IANA, optionally with a note that indicates that
Unicode 12 tables will follow in short order although possibly
not until after IETF 104.  I think it is safe for IANA to
reference draft-faltstrom-unicode11-08 if they want a reference.
The same arguments that say that there is no requirement for a
new versions of Unicode to be accompanied by an RFC strongly
suggest that there is no requirement that IANA wait for a formal
report before posting tables and hence that the IESG can just
say "here they are, go post them".  In addition, and frankly,
the person most likely to object if IESG tells IANA to update
the tables without an adequate review or procedural foundation
is me and, as long as they post tables for 11 and (not 12) this
week or early next, I won't do it -- not as some sort of
compromise but because I believe there is adequate foundation
(and will say, and explain, that in public if anyone else
objects).  Then lets press ahead with getting the loose ends
tied up and getting a document ready for that second Last Call
with which we are all at least reasonably satisfied, even if,
given IETF 104 in the middle, it takes us into the first half of
April.

This is still not how I think the document should be, and should
have been, handled.  But I'm obviously in the minority, so let's
figure out something reasonable and move on.

best,
    john

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 14 Mar 2019 08:40:02 +0000

Hi Patrick,

> On 13 Mar 2019, at 21:47, Patrik Fältström <[email-masked]> wrote:
> 

After my exchange with John yesterday, I think updating unicode12-00 would be better.

Thank you,
Alexey

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 14 Mar 2019 17:36:27 +0900

> After my exchange with John yesterday, I think updating unicode12-00 would be better.

Ha ha ha!

Ok, I just sent unicode11-09...but I will send unicode12-01 instead.

   Patrik