Subject: Re: [I18ndir] Getting restarted and triage

From: John C Klensin <[email-masked]>
Date: Thu, 13 Jun 2019 21:38:20 -0400

Dear esteemed and rapidly-moving secretaries,

Noting that tomorrow makes a month since you sent your "get back
to work" note, that there has been nothing more from you or
anyone else, and that things seem to be moving at the nearly
same rate they were after last summer's BOF, I thought I'd check
back in and tell people, more or less as a heads-up, what Patrik
and I have been up to.  Some news:

(1) Patrik and I have been working on a document that clarifies
and cleans up the new Unicode version review process that was,
in retrospect, not very well specified in RFC 5892.  The I-D
identifies the two types of reviews, explained expectations, and
clarifies the role of the IANA tables.  It probably needs one
more pass by each of us and then will be posted as
draft-klensin-idna-unicode-review.  

(2) One thing that document notes is that RFC 5892 very nearly
requires that, if Unicode changes code point properties in a way
that changes the IDNA2008 derived property, we adjust IDNA2008
to preserve IDNA stability and consistency rather than following
along with Unicode.  That is consistent with the promises we
made to the registrar community about preserving stability.  It
is also the exact opposite of what we have actually done with
the Unicode 6.x, 11.0, and 12.0 reviews.   For future versions
of Unicode, we propose to revert to the specified behavior in
which stability for IDNA is at least a strong preference. 

If that is what we do, it may add yet another irony to the
collection around things that are called IDNA.  IIR, the
original core reason for the creation of UTR#46 was that
IDNA2008 wasn't forward compatible enough with IDNA2003.  The
UTR#46 model was that if a label or potential label was valid at
one point, it needed to be valid forever.  When we track Unicode
changes --as we and UTR#46 have been doing-- we create
instability for IDNA label validity.  If we start favoring the
preservation of label validity (and invalidity) as 5982
specifies, then the gap between IDNA2008 and UTR#46 almost
certainly increases.

(3) If anyone hasn't noticed, Unicode 12.1 has been published.
Only one new code point since 12.0 and it decomposes properly
and there are no other changes, but we need to decide whether we
are reviewing, publishing summaries about,  abd publishing IANA
tables for, interim (i.e., NN.x, x not zero) versions of
Unicode.   The I-D mentioned above will have some text about
that, but we are open to different conclusions than it proposes.

(4) I'm working on a draft that explicitly addresses assumptions
that were made in the IDNA2008 design that have turned out to
not be true or not work as expected.  It is partially a
follow-on to draft-klensin-idna-unicode-7-0-0 but without a lot
of the details and alternatives about non-decomposing
characters.  Watch for something that will probably be called
draft-klensin-idna-architecture-00 in the fairly near future
(unless I shift to directorate time, in which case you should
expect it in early 2020).  

(5) draft-klensin-idna-architecture, discussed above,
necessarily contains a reference to
draft-klensin-idna-rfc5891bis, the registry restrictions
document, so that should be on someone's agenda too.

(6) I suspect draft-klensin-idna-unicode-7-0-0, or at least
those details, of the non-decomposing character issues, should
be published for the historical record, so it would be good for
people to start thinking about that and what should be in it.

That brings me to a key question.  Noting that the main reason
for proposing the BOF that led to this directorate was to try to
get work on core I18N, and especially IDNA, issues under
control, my preference would be that the directorate take a look
at the drafts mentioned above (and probably Asmus's work on
troublesome characters, etc.) and make a recommendation to the
ADs about how to handle them.   An alternative would be for us
to introduce the drafts on the IDNAbis WG mailing list and then
pass them directly to the ADs with a request for AD sponsorship
and, if needed, a short-term restart of that WG, which would get
them to the directorate that way.   There is a Plan C, but I'm
quite confident that almost no one would like it.

So, Glorious Leaders and other directorate participants, how
would you like to proceed?  

     john

--On Tuesday, May 14, 2019 13:01 -0500 Pete Resnick
<[email-masked]> wrote:

> Greetings,
> 
> Your faithful secretaries are getting off of their respective
> rear-ends to get this directorate doing its regular
> directorate duties. This means reviewing documents that are
> entering Last Call (or at the request of a chair or AD) to
> look for i18n issues. To do this, we're going to want to
> triage those documents, and then assign those with potential
> issues to a reviewer; we don't have enough reviewers to look
> at every document like GenART does. The two of us can do some
> of that triage, but we could use some help, and for
> educational purposes we think it would be good to get helpers.
> 
> A few of you have mentioned that you don't feel up to doing
> detailed i18n reviews of documents, but would be interested in
> doing (or learning to do) document triage. Please drop private
> email to either or both of us and let us know. Our plan is to
> have a small group "review meeting" with potential triagers
> and triage some documents together so you get the hang of it.
> We'll also be creating a wiki of triage hints and tricks based
> on the i18n bits of the old "Typical Apps Area Issues" wiki
> <https://trac.ietf.org/trac/app/wiki/TypicalAppsAreaIssues>
> that the Apps ADs made some time ago.
> 
> For the rest of you: Expect to see some documents appear in
> your queue in the next little bit.
> 
> 2 x Pete

---

From: "Marc Blanchet" <[email-masked]>
Date: Fri, 14 Jun 2019 08:18:12 -0400

On 13 Jun 2019, at 21:38, John C Klensin wrote:

> them to the directorate that way.

your description of the drafts seems to show that these are enough 
substantive that it requires a WG.

my 2 cents,

Marc.

> There is a Plan C, but I'm
> -- 
> I18ndir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18ndir

---

From: John C Klensin <[email-masked]>
Date: Fri, 14 Jun 2019 09:52:04 -0400

--On Friday, June 14, 2019 08:18 -0400 Marc Blanchet
<[email-masked]> wrote:

>> Noting that tomorrow makes a month since you sent your "get
>> back to work" note, that there has been nothing more from you
>...
>> for proposing the BOF that led to this directorate was to try
>> to get work on core I18N, and especially IDNA, issues under
>> control, my preference would be that the directorate take a
>> look at the drafts mentioned above (and probably Asmus's work
>> on troublesome characters, etc.) and make a recommendation to
>> the ADs about how to handle them.   An alternative would be
>> for us to introduce the drafts on the IDNAbis WG mailing list
>> and then pass them directly to the ADs with a request for AD
>> sponsorship and, if needed, a short-term restart of that WG,
>> which would get them to the directorate that way.
> 
> your description of the drafts seems to show that these are
> enough substantive that it requires a WG.

Marc,

In principle, fine with me.  However, I observe that:

<rant, because the topic probably deserves one>

(1) Pete's note suggests that several people in this rather
small directorate list have told them that they "don't feel up
to doing detailed i18n reviews of documents".  That does not
bode well for their likely active participation in a WG with
several documents.

(2) The original IAB statement on what was described at the time
as a Hamza problem but which we are now calling a
non-decomposing character one asked the IESB to do something,
presumably a WG or at least some study effort.  Nothing really
happened.  And the amount of useful input on the various
iterations of draft-klensin-idna-unicode-7-0-0 --other than from
the co-authors to each other, Mark Davis, Andrew Sullivan, and
Asmus-- over the lifetime of that document was just about zero.
I note that later versions of the document both described the
issues and laid out alternatives -- rather long for a WG problem
statement, but otherwise should have been more than adequate.

(3) By the time the EAI WG effort wound down, there were, IIR,
only about two people other than the co-chairs and the author
collection actually doing work or paying attention.  The number
of substantive comments on IETF Last Call was just about zero.

(4) By the time the PREICS WG effort wound down, the number of
people who were active other than co-chairs and authors also
appeared to be fewer than a half-dozen, probably much fewer for
the second-round documents.   Again, virtually no substantive
comments on IETF Last Call other than from that group.

(5) Other than discussion within the Directorate and some
proforma Area reviews that did not show an understanding of the
issues, the IETF Last Call on draft-faltstrom-unicode11 produced
approximately nothing.   Especially because that document (and
the IAB statement that motivated it) swept the issues discussed
in draft-klensin-idna-unicode-7-0-0-05 aside (clearly "known
omissions" in RFC 2026-speak and substantive enough to cause
several years of paralysis), there was an argument similar to
yours that a WG should have been required too.  

(6) There were even fewer IETF Last Call comments on
draft-faltstrom-unicode12 because there was no Last Call,
despite the fact that it and draft-faltstrom-unicode11 cover
different material.   That substitution may have been the right
thing to do, but it violates all sorts of rules and precedents
that normally would have required a WG for both of those
documents.

(7) Speaking as one of those who proposed it, the purpose of the
BOF that led to this directorate was precisely to figure out how
to process documents like these, with becoming more systematic
about reviews of non-i18n documents with i18n content being
distinctly secondary.  The conclusion was, approximately, that a
directorate should be formed to, independent of whatever else it
might do, make recommendations to the ADs as to how to proceed.
So here we are.  

Without interpreting the months it took to get it off the
ground, the lag time between the discussions of the Unicode 11.0
and 12.0 tables and drafts and Pete's note and the month between
Pete's note and my note as indicating anything (although it
probably does), (1) - (6) above make an extremely strong case
that getting critical mass together to initiate and sustain a
WG, at least a conventional one that does not bend various
rules, is implausible.  

FWIW, if we look over the i18n drafts posted in the last few
years (since the last PRECIS ones to go to RFC), I believe that
none of the authors other than Andrew are regular IETF meeting
attendees.  That doesn't bode well for a WG, at least a
conventional one either.

So, if we are going to say "requires a WG", I think we need to
figure out, and make some recommendations about, how that WG
will be populated and function.   Saying "requires a WG" without
such a plan takes us back to the proposal for the BOF; the
reasons why, by traditional IETF criteria (although it wasn't
done in either case and I'm glad it wasn't) EAI and PRECIS
should have been shut down for lack of involved and contributing
participants before the last documents were finished; and the
IETF should not simply and formally exit the i18n area for lack
of interest and competence.  I don't know what that would do
with other documents that make i18n assertions, but it probably
wouldn't be good.

</rant>

best,
   john

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 14 Jun 2019 09:29:33 -0700

In prinicple, I'd agree. However:

> non-decomposing character one

Which description still falls short of the problem as long as you have 
several
pairs of code points with identical (by design) appearance in IDNA 2008 
-- and
not counting the Latin/Cyrillic pairs.

> asked the IESB to do something,
Seems like a reasonable reading of a the evidence to me.
> attendees.  That doesn't bode well for a WG, at least a

That's an issue.

> with other documents that make i18n assertions, but it probably

i18n is special in the way it intersects technologies. It isn't a standalone
technology, despite the fact that some technologies are i18n-specific.

In principle, the directorate model should cover the other aspects well,
except that IETF has too few people who can (or want to) understand and
review meaningfully those "generic" technologies that nevertheless have
i18n exposure. The W3C make that model work, but only because their
core participants are funded directly for that work.

A./

---

From: John C Klensin <[email-masked]>
Date: Fri, 14 Jun 2019 16:03:53 -0400

Asmus,

I think we are in substantial agreement, but a few comments
below to be sure (and with everything else elided)...

--On Friday, June 14, 2019 09:29 -0700 Asmus Freytag
<[email-masked]> wrote:

>> (2) The original IAB statement on what was described at the
>> time as a Hamza problem but which we are now calling a

> Which description still falls short of the problem as long as
> you have several pairs of code points with identical (by
> design) appearance in IDNA 2008 -- and not counting the
> Latin/Cyrillic pairs.

First of all, that issue, as described in the original IAB
statement and in draft-klensin-idna-unicode-7-0-0 (even in
version -00) is about conflicts within a script and hence does
not interact with Latin/Cyrillic, Cyrillic/Greek, or Latin/Greek
overlaps.  While we could have a long and probably amusing
discussion about whether Greek, Latin, and Cyrillic should have
been separated while the Arabic language variation of Arabic
Script and the variations based on or derived from Persian were
not, that discussion is of theoretical interest only (except
possibly for those who feel some nostalgia for the original ISO
DIS 10646 and its greater identifier focus rather than
printing/display focus).

Second, no one I know ever claimed it was a complete description
of the problem or even that draft-klensin-idna-unicode-7-0-0 was
at the most recent (public) -05.   I was just trying to map the
landscape out in a very approximate way for this list.

>> ground, the lag time between the discussions of the Unicode
>> 11.0 and 12.0 tables and drafts and Pete's note and the month
>> between Pete's note and my note as indicating anything
>> (although it probably does), (1) - (6) above make an
>> extremely strong case that getting critical mass together to
>> initiate and sustain a WG, at least a conventional one that
>> does not bend various rules, is implausible.

> Seems like a reasonable reading of a the evidence to me.

Sadly, we agree.
 
>> years (since the last PRECIS ones to go to RFC), I believe
>> that none of the authors other than Andrew are regular IETF
>> meeting attendees.  That doesn't bode well for a WG, at least
>> a conventional one either.

> That's an issue.

Given the above, just one more minor complication.  It either
means that we need to get agreement to do something mildly
unconventional or that we need to figure out how to do any work
mostly remotely _and_ in a way that keeps people engaged.  In
the latter regard, while I've pulled time out to try to write
some drafts and look again at others in the hope of either
getting things moving or proving that we cannot do so -- getting
this off my plate one way or the other, the other issue is that
I'm really busy and have zero support for this (even support I
can borrow from other things or justify on the basis of other
commitments), Patrik is really busy with other commitments,
Andrew now has a more than fill-time job, I assume I can make
inferences about the degree to which Pete and Peter are sitting
around contemplating what to do with their extensive free time
from the rate at which they have been pushing the directorate
forward, and I assume that no one has stepped forward to support
you for as much time as you feel like putting into this and in
the style to which you would like to become accustomed. That,
IIR, is entire the recent author list.

IMO that is a fairly serious problem and the reason I took
exception to "requires a WG" without a plan that addresses it.

> i18n is special in the way it intersects technologies. It
> isn't a standalone technology, despite the fact that some
> technologies are i18n-specific.

Yes.  I hope we all know and agree about that.  Certainly I do.

> In principle, the directorate model should cover the other
> aspects well, except that IETF has too few people who can (or
> want to) understand and review meaningfully those "generic"
> technologies that nevertheless have i18n exposure. The W3C
> make that model work, but only because their core participants
> are funded directly for that work.

Actually, there was a bit of a fuss when the directorate was
created about confining its role to that of traditional
directorates.  If those positions are accepted (some of which
came from comments by people who where then ADs), our sole role
is to advise that ART ADs on strategic issues.  Even reviewing
out-of-Area documents is a little marginal and some Areas have,
historically, had both directorates (for strategy and technical
advice to the Area) and what are now called Review Teams (for
out of Area document reviews) with different memberships.   

>From the perspective of someone who is part of W3C's core i18n
WG and who has been on most of the weekly calls for the last few
years --probably a higher percentage of calls over that time
than anyone other than the assigned staff member and the chair
(and who, by the way, is not funded either directly or
indirectly for that work), there are at least two other reasons
why that effort works.  One is that the core group, and the W3C
generally, have been at liberty to say "not a web problem" or
equivalent and walk away (and have done that, repeatedly).   I
cannot imagine them spending much time on, e.g., non-ASCII
identifiers in X.509 certificates or physical device
identifiers.   The second is that they are actually treated as a
group of experts that is required, or even expected, to justify
every internal decision to a pack of people with strong
opinions, loud voices, and no expertise (in our case, whether
within the IETF or to various ICANN and other industry groups).
Even the public review process is different in that respect.

So I agree with you, but the difference is much greater than
what you say above.

    john

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Fri, 14 Jun 2019 14:13:58 -0700

> landscape out in a very approximate way for this list.

I was just making the point that Unicode's own list of intentionally
identical characters contains several same-script pairs.

They also document a large number of sequences that would render
identically, except that one of each pair is considered "do not use"
by Unicode (and vendors obligingly tend to insert dotted circles into
the display - although that is not mandated).

These cases are edge cases not covered by normalization for various
good reasons, but we might need to work with them to make that data
machine readable.

Finally, there are some sequences that we discovered in the Root Zone
development process that are not normalizable (they are are distinct
spellings, even though they look alike - the correct choice depends on
the word in question). Because they are sequences,
they are not yet covered by Unicode data.

And then the larger issues is that nobody knows enough about ancient,
obsolete or archaic scripts (or rare code points in general) to be able
to map the problem space *within* that repertoire; which is why it's
best to not support any of them in "public" zones.

We should talk about the problem this way: "In the general case,
if you want a reasonably secure zone, you need to develop
a set of Label Generation Rules for your zone,
that is restricted to modern-use repertoire (to facilitate
recognition) and that uses context and other rules to restrict
labels further to stay within the structure of the script and prevent
duplicate or non-recognizable labels; or, that uses variants to
limit the number of labels that users may consider substitutes
for one another; or both. In addition, it may require the use of a
separate process to deal with certain more subtle forms of confusion
that fall short of full exchangeability."

The issue here is that not all scripts require all types of mitigation
to the same extent (or for the same reasons). Hence the, "in the
general case".

But I find that if we think of only a sub-problem, like non-normalizable
sequences with identical (or possibly identical) display, we fall short
of seeing the full issue and will try to pursue remedies that,
while well intentioned, will fall short. Or worse, will make the life
of those more difficult who are doing the right thing.

For example, you could define some kind of inverse "CONTEXT"
rule for IDNA that invalidates certain code points and feed that
process all the "do not use" sequences from Unicode. Turns out,
for the Indic scripts, where these are of an issue, a rather simple
set of positive context rules, generally based on character classes,
not individual code points, covers the 90+% of all cases including
100% of those listed by Unicode.

But you don't want to bake these into the protocol, because between
1 and 10% of the cases can be language dependent, or can depend
on whether you allow certain other code points (or not), that is
the details (and needed exceptions) can be repertoire dependent.

[We really should be having this discussion based on your updated
draft.]

>> Seems like a reasonable reading of a the evidence to me.

Which is why a different beast (or some tweaking of the rules) may
be needed to obviate any "bending".

> Yes.  I hope we all know and agree about that.  Certainly I do.

May need it's own niche in terms of process.

> out of Area document reviews) with different memberships.

Again, the way, i18n cuts across technologies seems poorly understood
by IETF in general  - excepting this group.

> why that effort works.

I've watched the process from a bit further remote, but I do monitor their
work and tend to put my oar in when issues intersect my particular 
expertise.

>   One is that the core group, and the W3C
> identifiers.

But I'm sure IETF also has a scope. Wouldn't expect this directorate to get
involved in defining issues for HTML for example ?

But some of the discussions here make me think that, for example, IETF
isn't clear about keeping character encoding issues on the outside.

>   The second is that they are actually treated as a

I can't parse that first sentence. Is there a "not" missing?
> So I agree with you, but the difference is much greater than

Well, comparisons are always approximate.

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 14 Jun 2019 23:37:25 +0200

On 14 Jun 2019, at 14:18, Marc Blanchet wrote:

> your description of the drafts seems to show that these are enough substantive that it requires a WG.

Friends,

In general I would agree with you Marc, as one would really like to have "a large bunch of people" that discusses these kind of things. In the IETF such "a large bunch of people" is called "a wg".

The sad thing, explained in Johns note, does not exist. However much we would like it.

Yes, I also would like to have a wg. Per issue. With tons of specialist experts. We just do not have them.

So, given that does not exist, we just have to face the fact that we know who the people are, on an individual level, that do have at least some clue, some time, and some interest to discuss these things. Lets call that "a small group of known people".

In the IETF, we call "a small group of known people" "a directorate".

Why do I explain this? Well, this because on 30.000 feet level, although the processes have been refined and specified and what not in the IETF, ultimately there is an internet-draft, it is sent to the IETF for last call, the IESG look at the feedback during last call and make a determination.

Our job, as "a small group of known people" is to help the shepherd IESG member to create whatever document needed, that suggest whatever we think is to be done, that will go on last call and not only pass, but more importantly "say the right thing".

Because we are here to do good shit!

That we would like to have a wg as a tool to create that, sure, I would like to have that as well. Just like a pony ;-)

To conclude, yes Marc, I agree with you, but I think we just have to face the fact we can never have an effective wg, and the best we can do regarding transparency, creation of documents when needed and produce good results that IESG can use in their determination is to have a good directorate.

   Patrik

---

From: "Marc Blanchet" <[email-masked]>
Date: Fri, 14 Jun 2019 17:41:42 -0400

On 14 Jun 2019, at 17:37, Patrik Fältström wrote:

> In general I would agree with you Marc, as one would really like to 
> have "a large bunch of people" that discusses these kind of things. In 
> the IETF such "a large bunch of people" is called "a wg".
>
> Yes, I also would like to have a wg. Per issue. With tons of 
> specialist experts. We just do not have them.
>
> So, given that does not exist, we just have to face the fact that we 
> know who the people are, on an individual level, that do have at least 
> some clue, some time, and some interest to discuss these things. Lets 
> call that "a small group of known people".
>
> Why do I explain this? Well, this because on 30.000 feet level, 
> although the processes have been refined and specified and what not in 
> the IETF, ultimately there is an internet-draft, it is sent to the 
> IETF for last call, the IESG look at the feedback during last call and 
> make a determination.
>
> Our job, as "a small group of known people" is to help the shepherd 
> IESG member to create whatever document needed, that suggest whatever 
> we think is to be done, that will go on last call and not only pass, 
> but more importantly "say the right thing".
>
> That we would like to have a wg as a tool to create that, sure, I 
> would like to have that as well. Just like a pony ;-)
>
>
> To conclude, yes Marc, I agree with you,

I know the history, been involved… I was more responding to the fact 
that from the IETF process perspective, when something of that 
substantive nature, IESG had usually highly preferred to run this 
through a wg than anything else. And I know at least one past AD here 
that was very pushy in _not_ doing AD-sponsored drafts. Having said 
that, any way is fine by me.

Marc.

> but I think we just have to face the fact we can never have an 
> effective wg, and the best we can do regarding transparency, creation 
> of documents when needed and produce good results that IESG can use in 
> their determination is to have a good directorate.
>
>    Patrik

---

From: "Pete Resnick" <[email-masked]>
Date: Fri, 21 Jun 2019 14:12:59 -0500

[Copying the ADs, as their input is also necessary.]

Before getting to your direct question, a followup on the "restart and 
triage" bit: I only got 2 messages from folks who wanted to do triage, 
and then promptly went on vacation (with a longer-than-normal vacation 
interrupt latency on the backend), so I haven't gotten a chance to 
individually harass people who said some time ago that they would help 
out. That harassment will start today. So hopefully we'll get on track 
to do triage and regular reviews quickly.

But, as for your direct question:

> would you like to proceed?

So, first let's get together the full list of "core" documents needing 
review in this group. From your message, I glean:

draft-klensin-idna-unicode-review
draft-klensin-idna-architecture (to be published)
draft-klensin-idna-rfc5891bis (expired)
draft-freytag-troublesome-characters-02 (expired)

Did I miss anything?

(I'm of course leaving aside draft-faltstrom-unicode12, which is on 
Alexey's plate.)

Do you think all of these need to be reviewed in parallel, or can we 
subdivide the task? My inclination is to get a couple of sets of 
eyeballs on each document and then discuss the reviews on the list; I 
don't think we need to have a free-for-all on all of the documents.

(On that, and to reply to some of the other comments: Dealing with these 
documents is **not at all** like a WG: In a WG, documents are 
collectively developed and documented by the doc editor, and by the time 
a document is done, we have a nice history, whether it's on the mailing 
list, in github, or whatever, of how decisions were made and why we have 
consensus. This is very different: We have documents prepared by 
individual experts and are trying to retrospectively review them, 
looking for errors or editorial flubs to make sure the document is ready 
to go. We want to get additional eyeballs on the documents, but this is 
*not* a collective effort to create these documents. That's OK, but 
that's also why it's so hard to get focus. The purpose of the 
directorate IMO is to get that focus.)

I'm inclined to hear what the ADs want out of this process as well. For 
documents like this, my presumption is that you want to hear a hum from 
the directorate that you're safe to AD sponsor the document. Is that 
right?

pr
-- 
Pete Resnick http://www.episteme.net/
All connections to the world are tenuous at best

---

From: Barry Leiba <[email-masked]>
Date: Fri, 21 Jun 2019 15:29:09 -0400

> I'm inclined to hear what the ADs want out of this process as well. For documents like
> this, my presumption is that you want to hear a hum from the directorate that you're safe
> to AD sponsor the document. Is that right?

I suppose that's one way to look at it, and it's certainly something
we need to know.  More broadly, though, it's that the content is
"right" and sensible, to the extent that something in this space can
be objectively called "right", and that it contributes to moving the
I18N vehicle in the right direction.  Which also means some sort of
assessment about how it ties in with other documents in the vehicle --
the ones that came before, and the others that are being considered
now.

For that last, it means that in addition to separate reviews of each
it would be nice to have someone look over all of them with that in
mind.  Because Alexey and I are clueful about I18N, we can certainly
handle that part this time.  In general and in future, where we're
looking at a set of ART Directors none of whom are versed in I18N, it
will be more important for the Directorate to take a holistic view.

Barry

On Fri, Jun 21, 2019 at 3:13 PM Pete Resnick <[email-masked]> wrote:
> Before getting to your direct question, a followup on the "restart and triage" bit: I only got 2 messages from folks who wanted to do triage, and then promptly went on vacation (with a longer-than-normal vacation interrupt latency on the backend), so I haven't gotten a chance to individually harass people who said some time ago that they would help out. That harassment will start today. So hopefully we'll get on track to do triage and regular reviews quickly.
>
> So, first let's get together the full list of "core" documents needing review in this group. From your message, I glean:
>
> (I'm of course leaving aside draft-faltstrom-unicode12, which is on Alexey's plate.)
>
> Do you think all of these need to be reviewed in parallel, or can we subdivide the task? My inclination is to get a couple of sets of eyeballs on each document and then discuss the reviews on the list; I don't think we need to have a free-for-all on all of the documents.
>
> (On that, and to reply to some of the other comments: Dealing with these documents is not at all like a WG: In a WG, documents are collectively developed and documented by the doc editor, and by the time a document is done, we have a nice history, whether it's on the mailing list, in github, or whatever, of how decisions were made and why we have consensus. This is very different: We have documents prepared by individual experts and are trying to retrospectively review them, looking for errors or editorial flubs to make sure the document is ready to go. We want to get additional eyeballs on the documents, but this is not a collective effort to create these documents. That's OK, but that's also why it's so hard to get focus. The purpose of the directorate IMO is to get that focus.)
>
> I'm inclined to hear what the ADs want out of this process as well. For documents like this, my presumption is that you want to hear a hum from the directorate that you're safe to AD sponsor the document. Is that right?
>
> pr
> --

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Jun 2019 18:19:57 -0400

Pete,

I've read Barry's note, which seems entirely reasonable.  As you
(and I trust most other's here) have noticed, while waiting I
did post a note to the IDNAbis list asking for review and
feedback on the theory that, if anyone on that list but not on
this one had something to say / contribute, we should listen.
So far the silence has been deafening which I can interpret
either as (i) everyone who is on that list and likely to do work
is on this list and has been waiting for your note (and word
from Barry, Alexey, or Adam) or (ii) we are in even bigger
trouble than I assumed.

More inline below.

--On Friday, June 21, 2019 14:12 -0500 Pete Resnick
<[email-masked]> wrote:

> Before getting to your direct question, a followup on the
> "restart and triage" bit: I only got 2 messages from folks who
> wanted to do triage, and then promptly went on vacation (with
> a longer-than-normal vacation interrupt latency on the
> backend), so I haven't gotten a chance to individually harass
> people who said some time ago that they would help out. That
> harassment will start today. So hopefully we'll get on track

I hope so, but with the understanding that there is an
interaction with the core documents in many (although certainly
not all) potential reviews of non-I18n documents with i18n
citations, dependencies, or other relationships.    If they
either reference IDNA or are dependent on IDNA's approach
(probably including almost everything that depends on PRECIS) or
contain any statements or logic that are sensitive to Unicode
versions, the issues that motivated the documents that are more
or less in the queue may affect them and be candidates for
"known technical omissions".  Unless the IESG is willing to
simply blow off that provision of 2026 --which I hope and assume
they would not if it were called to their attention for a
particular document-- either the document being reviewed goes
into a wait state until we clean up at least some of the i18n
core mess or someone is going to have to write an explanation of
the problem and why the document should progress without it. I'd
assume that, in most cases, that "someone" would need to be one
of us because there are not may other pockets of the relevant
expertise.   Perhaps that makes "triage" and "regular reviews" a
little more complicated than you are anticipating.

>> which would get them to the directorate that way.   There is
>> a Plan C, but I'm quite confident that almost no one would
>> like it.
>> 
> So, first let's get together the full list of "core" documents
> needing review in this group. From your message, I glean:
> 

Any of these (and those below) that have expired are expired
because the ADs or directorate advised the the authors to let
them expire.   They are easily un-expired (and will be) when we
are ready to cope with them.  And I hope you mean "posted" and
not "published".  One step at a time.

draft-klensin-idna-unicode-7-0-0.  While I assumed when I wrote
my earlier note that draft-klensin-idna-architecture would
completely subsume its contents, it now appears to me that some
of the discussion of specific issues (such as the
non-decomposing character one) in
draft-klensin-idna-unicode-7-0-0 belongs in a separate document.
There may also be a document on the special properties of
complex scripts (if we want it and can persuade Asmus to write
it).

> (I'm of course leaving aside draft-faltstrom-unicode12, which

There are some potentially serious complications with it, most
brought on by the passage of time while we've been waiting and
others that might have slipped through it is had been handled
quickly but are now hard (and probably improper) to ignore.
That deserves a separate note (and probably thread)...
forthcoming.

> Do you think all of these need to be reviewed in parallel, or
> can we subdivide the task? My inclination is to get a couple
> of sets of eyeballs on each document and then discuss the
> reviews on the list; I don't think we need to have a
> free-for-all on all of the documents.

Sure, if it is feasible.  But several of these documents are
dependent on others of them, either in the form of normative
references or on assumptions or context that depend on approval
of one or more of the others.  So, it is not obvious to me that
is feasible to separate and order them by document or, if it is,
that we would just be trading a quicker review process for a
confusing set of IETF Last Calls confusion to everyone paying
attention, which might be null set when the people on this
mailing are excluded) and documents parked with the RFC Editor
in reference hold.   I do think we could build a road map for
people threading their way through the documents, but that would
be a fair amount of work for someone and I don't think I'm going
to volunteer.

> (On that, and to reply to some of the other comments: Dealing
> with these documents is **not at all** like a WG: In a WG,
> documents are collectively developed and documented by the doc
> editor, and by the time a document is done, we have a nice
> history, whether it's on the mailing list, in github, or
> whatever, of how decisions were made and why we have
> consensus. This is very different: We have documents prepared
> by individual experts and are trying to retrospectively review
> them, looking for errors or editorial flubs to make sure the
> document is ready to go. We want to get additional eyeballs on
> the documents, but this is *not* a collective effort to create
> these documents. That's OK, but that's also why it's so hard
> to get focus. The purpose of the directorate IMO is to get
> that focus.)

Sorry, Pete, but that characterization of WG documents is, for
many of them, somewhere between a fairy tale and wishful
thinking.  Too often, especially in recent years but even before
that, a WG has started with a draft prepared entirely outside
the IETF and done nothing but tune it a bit, a possibility that
exists even with these i18n documents.  Sometimes, WGs have been
told that the protocol described by the documents that were
prepared and largely completed before the WG started was already
deployed and that any fundamental changes to that protocol were
out of scope.  Certainly we haven't had the latter here.

In addition, there was considerable discussion -- as a group --
of what the requirements for IDNA review for new versions of
Unicode were and should be while we were sorting through
draft-faltstrom-unicode11 and draft-faltstrom-unicode12.  I made
notes, I believe Patrik made notes, probably other people made
notes.   I think that several of us assumed that those notes
would be circulated, we'd discuss them, someone would volunteer
or be appointed as document editor and we'd get something
together and proceed -- not at all different from the way your
idealized WG would work.   

That didn't happen that way.  I can't speak for others, but I
wanted to get draft-faltstrom-unicode12 behind us and was
waiting for instructions from the co-chairs as to how they
wanted to move forward.   When neither happened --months after
draft-faltstrom-unicode12-00 was posted and more than a month
after your note-- I decided to put some words down, enlist
Patrik, and get a draft posted.   What do you think I/we should
have done differently?

> I'm inclined to hear what the ADs want out of this process as
> well. For documents like this, my presumption is that you want
> to hear a hum from the directorate that you're safe to AD
> sponsor the document. Is that right?
>...

best,
   john

---

From: "John Levine" <[email-masked]>
Date: 21 Jun 2019 18:42:09 -0400

In article <F2A317B2420BA48FC2A23176@PSB> you write:
>either as ...

It's on my List of Things to Do Immediately.  You know how it is.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Jun 2019 20:39:07 -0400

As indicated in my earlier note, this probably deserves a
message, and thread, of its own...

> (I'm of course leaving aside draft-faltstrom-unicode12, which

But you shouldn't because there are two special issues with it,
especially given that it has waited this long, that probably
justify (require?) opening a Last Call as if it were completely
new and reviewing the document.  The causes for both are
addressed in draft-klensin-idna-unicode-review but, of course,
it, and its conclusions, are not addressed in
draft-faltstrom-unicode12.  At the risk of introducing actual
technical topics, they area:

(1) If we are going to do a new review for every new version of
Unicode --which is more or less what I think the community was
promised and 5892 implies -- then Unicode 12.1 is out and
another review and set of tables are required.  The current
version of draft-klensin-idna-unicode-review makes that
requirement apply only to major versions, i.e., 12.0 and then
13.0 when it arrives, but, unless it is put out there and gets
community consensus, the fact that the current version of
draft-faltstrom-unicode12 does not address 12.1 is a known
technical omission.  The good news is that 12.1 is trivial, with
one new code point that, incidentally, decomposes the way RFC
5892 anticipates.  But still (and unrelated to 12.1)...

(2) RFC 5892 is reasonably clear that the intended normal
resolution when a code point is discovered whose Unicode
properties have changed enough between Unicode versions to alter
their derived property status under IDNA is to list that code
point as an exception, thereby being sure that any possible
label that was valid earlier remains valid and that any possible
label that was invalid earlier  (with the obvious exception of
ones containing previously-unassigned code points) remains
invalid.  We didn't do it that way with the Unicode 6.0 review
reflected in RFC 6452.  Instead, we decided to just accept the
new properties and stay consistent with calculations on the
current Unicode version.  We didn't really explain why, nor did
we change 5892 to encourage that choice.  I presume the reason
(or at least our excuse in retrospect) was that Unicode 6.0
followed quickly enough on the core IDNA2008 documents and came
before IDNA2008 was widely used, that consistency with Unicode
was reasonable and would cause no damage (especially given the
code points involved).

We didn't follow the RFC 5892 requirement in the review for
Unicode 11.0 or the retrospective reviews for 7.x, 8.x, 9.x. and
10.x either (some of those did not contain incompatibility that
would raise any concerns -- see draft-faltstrom-unicode12 for
discussion).  Our reason for that, or at least my understanding
of the IAB's reasoning in telling Patrik to go ahead and get the
tables out, was that we concluded that, after so much time has
passed, there was significant risk that tables that had been
calculated directly from the relevant Unicode versions would
diverge from the ones IANA was about to publish, creating a
different and harmful kind of incompatibility.
draft-faltstrom-unicode12 does discuss this (in Section 5), but
the explanation is, IMO, less than clear in the context of the
strong preference in 5892. 

There is no such excuse for 12.0 (much less 12.1): the reviews
occurred more or less contemporaneously with the release of the
new versions of Unicode and, if we were going to return to the
specification in RFC 5892, that probably deserves some comment
(keep reading before telling me what I/we already know).

But, in reviewing the discussion in the first part of the year,
the reason why 5892 had the default it did, and the various
promises the IETF made to the domain name registry and user
communities about stability, Patrik and I concluded that, rather
than change 5892 to prefer Unicode conformance, it was time to
return to the consensus at that time IDNA2008 was adopted and
prefer label stability for future reviews, at least unless there
was a really strong reason (that we were willing to document)
for doing otherwise.    Obviously, that conclusion is subject to
discussion, but I'm now convinced the argument for doing what we
concluded should be done in 2010 is quite strong.

With 12.0 (and 12.1), we lucked out and there are no code points
whose properties have changed in a way that requires action
--either staying with Unicode, its properties, and their
implications or preserving compatibility via the backwards
compatibility table in IDNA2008.  But the text of
draft-faltstrom-unicode12 implies that the way we look at and
handle code points is unchanged from 6.0 through 12.0 and it
isn't.  So the I-D probably needs either a normative reference
to draft-klensin-idna-unicode-review, a bit of discussion that
isn't there yet, or both.

So, while we can sit on that until Alexey finishes his writeup
if you (Pete, Peter, Alexey, directorate participants) like, it
will come back to get us sooner or later.  And this is another
reason why these documents are fairly seriously intertwined.

best,
    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Jun 2019 20:46:51 -0400

--On Friday, June 21, 2019 18:42 -0400 John Levine
<[email-masked]> wrote:

>> I've read Barry's note, which seems entirely reasonable.  As
>> you (and I trust most other's here) have noticed, while
>> waiting I did post a note to the IDNAbis list asking for
>> review and feedback on the theory that, if anyone on that
>> list but not on this one had something to say / contribute,
>> we should listen. So far the silence has been deafening which
>> I can interpret either as ...
> 

Oh, yes. 

But one guess as to what is causing Pete's difficulties in
getting responses to his triage question is that I18N issues
generally, and perhaps IDNA-related ones in particular, have
gotten into that state and that List for everyone here
(including Pete's and Peter's and, probably given the progress
on draft-faltstrom-unicode12-00 since March 11th, Alexey's).
While we've been saying things in terms of "insufficient
expertise" for i18n issues, it may be equally true that is just
has not been high enough priority for anyone with expertise or
otherwise in the critical path.  

If that is the case, then the question --and, again, the
question since Patrik and I requested the BoF over a year ago --
is whether we have a plan about changing that (maybe easier than
finding more expertise and maybe not) or whether should be
making another plan entirely, presumably one that gets the IETF
out of the critical path.

   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 22 Jun 2019 03:28:23 +0200

On 22 Jun 2019, at 2:39, John C Klensin wrote:

> So, while we can sit on that until Alexey finishes his writeup if you (Pete, Peter, Alexey, directorate participants) like, it will come back to get us sooner or later.  And this is another reason why these documents are fairly seriously intertwined.

1. draft-faltstrom-unicode12 have two main issues:

1.1. It does not address Unicode 12.1 (easy fix)

1.2. It suggests a resolution to the divergence between Unicode and IETF due to incompatible changes in Unicode that makes IETF follow the Unicode Standard by allowing derived property values to change (better to let 5892 rules be fixed).

2. draft-klensin-idna-unicode-review do suggest the derived property value stay fixed, which implies 5892 algorithm must be updated in the case Unicode do changes in the future like has happened before as described in draft-faltstrom-unicode12.

Because [1.2] and [2] above are incompatible with each other, text in them should and I would say MUST match each other.

What we have as a fact is that IETF have already accepted actions based on draft-faltstrom-unicode12 until Unicode 12.0 at least so applying draft-klensin-idna-unicode-review to the review process can not happen for earlier versions.

   Patrik

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 21 Jun 2019 19:34:24 -0700

John,

some of us don't have the time to thrash to death things like Patrik's 
draft over minute issues, like whether to consider the single DISALLOWED 
character for Unicode 12.1 a technical omission.

If we can't find ways to remove this artificial friction, we won't see 
more vigorous involvement.

Notably, such hyperfocus on minutiae of little practical impact is 
absent from successful projects in the i18n arena.

A./

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Jun 2019 22:36:17 -0400

--On Saturday, June 22, 2019 03:28 +0200 Patrik Fältström
<[email-masked]> wrote:

>> So, while we can sit on that until Alexey finishes his
>> writeup if you (Pete, Peter, Alexey, directorate
>> participants) like, it will come back to get us sooner or
>> later.  And this is another reason why these documents are
>> fairly seriously intertwined.
> 
> 1.2. It suggests a resolution to the divergence between
> Unicode and IETF due to incompatible changes in Unicode that
> makes IETF follow the Unicode Standard by allowing derived
> property values to change (better to let 5892 rules be fixed).
> 
> 2. draft-klensin-idna-unicode-review do suggest the derived
> property value stay fixed, which implies 5892 algorithm must
> be updated in the case Unicode do changes in the future like
> has happened before as described in draft-faltstrom-unicode12.
> 
> Because [1.2] and [2] above are incompatible with each other,
> text in them should and I would say MUST match each other.
> 
> What we have as a fact is that IETF have already accepted
> actions based on draft-faltstrom-unicode12 until Unicode 12.0
> at least so applying draft-klensin-idna-unicode-review to the
> review process can not happen for earlier versions.

In case it wasn't clear, we are in complete agreement.

Thanks,
   john

---

From: John C Klensin <[email-masked]>
Date: Sat, 22 Jun 2019 01:36:43 -0400

--On Friday, June 21, 2019 19:34 -0700 Asmus Freytag
<[email-masked]> wrote:

> John,
> 
> some of us don't have the time to thrash to death things like
> Patrik's draft over minute issues, like whether to consider
> the single DISALLOWED character for Unicode 12.1 a technical
> omission.

Which absolutely no one, AFAICT, but especially not either
Patrik or myself, has suggested.  

In more abbreviated form which I thought would be clear to
everyone here, but let me spell it out a little more, all I have
said about 12.1 is that (i) there are no changed in property
values compared to 12.0, (ii) there is only one new code point,
and (iii) that code point behaves exactly the way IDNA2008
assumes/ believes it should, i.e., the grapheme (and, because it
is a CJK code point, its approximate interpretation) can be
constructed under 12.0 (and earlier) by a code point sequence
and the next code point therefore decomposes under NFC.
Conclusions (i) and (ii) are the algorithmic review; (iii) is
the new code point review.  So, nothing to be done about that
code point and, other than maybe adjusting some terminology,
nothing that isn't already in draft-faltstrom-unicode12-00.

What does need to be adjusted -- and needs to be adjusted
whether we include 12.1 or not -- is the discussion of the basis
for making decision when they need to be made.  That has nothing
to do with the new code point.  See Patrik's note.
> 
> If we can't find ways to remove this artificial friction, we
> won't see more vigorous involvement.
> 
> Notably, such hyperfocus on minutiae of little practical
> impact is absent from successful projects in the i18n arena.

If I understand what you are concerned about -- and, given that
I believe that your complaint above is about a strawman that was
not even in sight, I may not -- let me review why this (both my
note and Patrik's) is important given the design of IDNA2008.
As we know from IDNA2003 and the Unicode Standard itself, it is
possible to define how to handle code points in a variety of
contexts by simply making a list of all of the code point in
Unicode and assigning properties (and, implicitly,
appropriateness in various contexts, to each.  The IETF
concluded that choice was inappropriate for IDNs and IDN labels
and that the best alternative (or, if you prefer, the least-bad
one) was to construct a system based on a series of rules and an
algorithm, one that had provisions for differences in
short-string, language-independent, requirements of the DNS and
requirements that are more typical of the broad range of Unicode
applicability.  Maybe that was the wrong decision or just one
that was hopelessly naive and we should un-do it, but I've seen
no proposal to do that and how to do so.  If we are to stay with
it, the algorithms and surrounding rule sets much be orderly and
stay that way, and that --and, again, not any single code point
in 12.1 or otherwise-- is what this is about.

best,
   john

---

From: Asmus Freytag <[email-masked]>
Date: Fri, 21 Jun 2019 22:38:38 -0700

>> has happened before as described in draft-faltstrom-unicode12.

If some character buried in some archaic script changes from punctuation
to letter and would become PVALID, then, at first glance, I'd say who cares
one way or another.

For modern use scripts, the code point with changed status could present
a spoofing opportunity that would now need to be addressed, even if it
doesn't affect registered labels. (Remote, but a possibility).

It could also be a code point in an emerging script, where disallowing it
could seriously cripple support for some user community. In that case,
the reason for the property change may have been precisely to recognize
that it is a member of an alphabet (part of an orthography, whatever).

Can't see the value of mandating a "one-size-fits-all" approach here.

The inverse case (new value: DISALLOWED) is clearer about the need for
always grandfathering via an exception.

For new code points, I can see some modern-use scripts where new code
points can cause unexpected issues that potentially would have a
practical impact. The vast majority of additions are in scripts that,
other than vanity labels, would not be expected to see significant use,
therefore, I'd say who cares one way or another to get the fine details
"perfect". (And the same goes for extensions to Chinese: at this point,
they are practically all in the long tail. . . and have no use other than
for spoofing a more frequent code point.)

---

From: "John R Levine" <[email-masked]>
Date: 22 Jun 2019 16:05:50 -0400

> It's on my List of Things to Do Immediately.  You know how it is.

I took a look, it all seems reasonable keeping in mind that my knowledge 
of non-Western scripts is pretty marginal.

For the IANA not-really-registry of code points, my advice would be to 
remove it from IANA and put it on github.  That is presumably pretty 
stable and can handle a lot of downloads, while nobody would mistake it 
for IANA Holy Writ.  It also shows you the history of updates which would 
in this case be useful.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Pete Resnick" <[email-masked]>
Date: Sat, 22 Jun 2019 17:58:55 -0500

Asmus (and others for future reference),

Please confine commentary on the list to the technical discussion (e.g., 
"Such-and-so is a minute issue that is not worth considering, and 
therefore the document should say this-or-that") and not personal 
comments on the participation style of other participants. Not calling 
someone out by name can still appear to be personally going after 
someone.

If you do have a concern about someone's participation style, please 
mention it to Peter or I offline and we'll try to address it. If you 
have a concern that the group more broadly is going down the wrong path 
(e.g., focusing on the wrong issues), note that to the list if you think 
it will be effective, but more likely also bring that to the chairs' 
attention. It's our job to focus discussion, and if we're not doing that 
job, we should be told so.

pr

On 21 Jun 2019, at 21:34, Asmus Freytag wrote:

> draft over minute issues, like whether to consider the single 
> DISALLOWED character for Unicode 12.1 a technical omission.
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 Jun 2019 12:24:48 -0400

Asmus (and everyone else),

Let's try to restate your (IMO very helpful) explanation in
possibly-actionable terms...

--On Friday, June 14, 2019 14:13 -0700 "Asmus Freytag (c)"
<[email-masked]> wrote:

>...
> I was just making the point that Unicode's own list of
> intentionally identical characters contains several
> same-script pairs.

>  They also document a large number of sequences that would
> render identically, except that one of each pair is considered
> "do not use" by Unicode (and vendors obligingly tend to insert
> dotted circles into the display - although that is not
> mandated).

> These cases are edge cases not covered by normalization for
> various good reasons, but we might need to work with them to
> make that data machine readable.

First question cluster: Would you advocate modifying RFC 5892 to
say something substantive about one or both members of that pair
of characters?  If so, do we need to ask Unicode, not just for
machine-readability but for a property or something similar that
IDNA can use?  Do we need to make a plan about what to do if we
ask and they say "no"?   Also see "Final, meta, questions" below.
 
> Finally, there are some sequences that we discovered in the
> Root Zone development process that are not normalizable (they
> are are distinct spellings, even though they look alike - the
> correct choice depends on the word in question). Because they
> are sequences, they are not yet covered by Unicode data.

Second question cluster: Is there something that could usefully
be done about these in IDNA and, if so, what would it be?  How
are such sequences differ from the "rn" and "m" pair that, with
the right choice of type styles and sizes, can look nearly
identical (or at least "alike")?  If there is a correct choice
and it depends on spelling of actual words, how does that
interact with the principle that DNS labels are not required to
be words in any language (and that many historically and
currently important ones are not)?  Is the Root Zone development
process imposing a "word" criterion for non-ASCII labels even
though the rest of the DNS does not have that requirement
(noting, e.g., "COM", "ORG", "BIZ", "MIL", most of the ISO
3166-1-based ccTLDs, etc., there is enough precedent for
non-words in ASCII that proposing such a change would probably
be a lost cause)?   Considering your work with and comments
about complex scripts, is it possible to construct and impose an
"orthographically plausible in a given script" rule that would
stop short of a "word" requirement and should we be exploring
that for IDNA?  Again see "Final, meta, questions" below.
 
> And then the larger issues is that nobody knows enough about
> ancient, obsolete or archaic scripts (or rare code points in
> general) to be able to map the problem space *within* that
> repertoire; which is why it's best to not support any of them
> in "public" zones.

While I think there is general agreement about that (some
fussing about the meaning of "public zone" aside), if it is a
public zone question alone, does it impact IDNA in any way?

> We should talk about the problem this way: "In the general
> case, if you want a reasonably secure zone, you need to
> develop a set of Label Generation Rules for your zone, that is
> restricted to modern-use repertoire (to facilitate
> labels further to stay within the structure of the script and
> prevent duplicate or non-recognizable labels; or, that uses
> variants to limit the number of labels that users may consider
> substitutes for one another; or both. In addition, it may
> require the use of a separate process to deal with certain
> more subtle forms of confusion that fall short of full
> exchangeability."

At one level, that seems entirely reasonable. I think --but an
not sure and would learn from a correction and explanation --
that it is actually less stringent as a requirement than the
repeated variations on "as a registry, you should register only
strings you understand in scripts that you understand
thoroughly" in the existing IDNA specs and reinforced in
draft-klensin-idna-rfc5891bis.  Would it work any better as
advice or in practice?   I think each of us --or at least many
of us-- have hypotheses about why the existing rule has not been
observed very much in zones that are operated for a profit or
even by third party registries on a cost-recovery basis (not the
same as "public domain", but with a good deal of overlap).
Most of those hypotheses seem to amount to "not profitable",
either because it would reject potentially-paying customers who
want to buy particular names or because the costs of checking
each string would be too high. 

Third question cluster:  Keeping in mind the above and that
we've, so far, avoided (or at least tried to avoid) getting into
issues of what humans can confuse, especially if others are
deliberately trying to confuse them in IDNA, what would you
suggest doing in this area?  Or is it just about reinforcing the
mandate for registries being responsible about what they do and
providing additional guidance and tutorial materials to help if
they decide, for business reasons, to substitute "have a
general, if vague, knowledge or and some helpful guidelines" for
"understand thoroughly"?
 
> The issue here is that not all scripts require all types of
> mitigation to the same extent (or for the same reasons). Hence
> the, "in the general case".

Clearly.  And I think we understood that in the IDNA2008 design
even though I now believe that we botched some of the details.
 
> But I find that if we think of only a sub-problem, like
> non-normalizable sequences with identical (or possibly
> identical) display, we fall short of seeing the full issue and
> will try to pursue remedies that, while well intentioned, will
> fall short. Or worse, will make the life of those more
> difficult who are doing the right thing.
 
> rule for IDNA that invalidates certain code points and feed
> that process all the "do not use" sequences from Unicode.
> Turns out, for the Indic scripts, where these are of an issue,
> a rather simple set of positive context rules, generally based
> on character classes, not individual code points, covers the
> 90+% of all cases including 100% of those listed by Unicode.

> But you don't want to bake these into the protocol, because
> between 1 and 10% of the cases can be language dependent, or
> can depend on whether you allow certain other code points (or
> not), that is the details (and needed exceptions) can be
> repertoire dependent.

Ok.  Whether we agree or not on the details (and you and I, at
least, mostly agree) I think our understanding of what we should
not do is improving, with your efforts and explanations being a
significant component of that improvement.  What I'm not sure
about is whether we are getting any closer to an understanding
and agreement on what we should do.

> [We really should be having this discussion based on your
> updated draft.]

Probably.  See "Final, meta, questions" below. 

>>>> 11.0 and 12.0 tables and drafts and Pete's note and the
>>>> month between Pete's note and my note as indicating anything

 
> Which is why a different beast (or some tweaking of the rules)
> may be needed to obviate any "bending".

Ok.  But what do you suggest given the constraints to which the
above points?  In a way, this is at the core of the "Final,
meta, questions" discussion below, but it states the problem at
least a bit differently.
   
> May need it's own niche in terms of process.

Yes.  However, when the directorate was under discussion, we got
back what I thought were fairly clear signals from the IESG that
our role was going to be scoped even more narrowly to advising
the ART ADs than some directorates have been in the past.  That
does not make me optimistic about trying to define a special
niche and get it accepted, but maybe things have changed.  As we
wrote to each other earlier...

>>> make that model work, but only because their core
>>> participants are funded directly for that work.

>> came from comments by people who where then ADs), our sole
>> role is to advise that ART ADs on strategic issues.  Even
>> reviewing out-of-Area documents is a little marginal and some
>> Areas have, historically, had both directorates (for strategy
>> and technical advice to the Area) and what are now called
>> Review Teams (for out of Area document reviews) with
>> different memberships.

> Again, the way, i18n cuts across technologies seems poorly
> understood by IETF in general  - excepting this group.

I agree completely.  But I'm not seeing the mechanism or
suggestions that get us unstuck from however one describes the
current situation.  See "Final, meta, questions" below.

>> WG and who has been on most of the weekly calls for the last
>> few years --probably a higher percentage of calls over that
>> time than anyone other than the assigned staff member and the
>> chair (and who, by the way, is not funded either directly or
>> indirectly for that work), there are at least two other
>> reasons why that effort works.

> I've watched the process from a bit further remote, but I do
> monitor their work and tend to put my oar in when issues
> intersect my particular expertise.

Yes.  But the claim, IIR, was the W3C effort was succeeding and
we weren't because the core activity there was fully funded and
staffed.  Staffed, yes, but, other than Richard, not much better
funded than the IETF effort.  Certainly I'm not funded for that
work and I assume you are not either.  I also note that the
level of activity of the other people on this list who are
supposedly part of the core group (and who may not be funded for
it either) has been rather close to zero and that has not
prevented work getting done either.

> But I'm sure IETF also has a scope. Wouldn't expect this
> directorate to get involved in defining issues for HTML for
> example ?

Of course.  But that isn't the point.   The point is that
analysis of, e.g., implications of of non-ASCII identifiers in
X.509 certificates or storage identifiers is fairly far from the
expertise of most of us on this list.  But we don't get to say
"out of scope" and, instead, appear to be expected to prioritize
them.
 
> But some of the discussions here make me think that, for
> example, IETF isn't clear about keeping character encoding
> issues on the outside.

More explanation about that would be helpful.  I think we are
fairly clear about it and that, when issues have arisen, they
have been about whether other groups are following their own
rules (or what they told the IDNA2008 development effort their
rules are) or about areas in which they have done work or made
decisions that are not, historically, about character encoding.

>> group of experts that is required, or even expected, to
>> justify every internal decision to a pack of people with
>> strong opinions, loud voices, and no expertise (in our case,
>> whether within the IETF or to various ICANN and other
>> industry groups). Even the public review process is different
>> in that respect.
> 

Yes.  Should have been "is not required..."  Sorry.

Final, meta, questions(s):

We are now six weeks past Pete's "get back to work" note and two
weeks past my response to it.
draft-klensin-idna-unicode-review, the direct result of
discussions in this group, will have been posted for two weeks
on Friday.

Since then, we've had Marc's note suggesting the issues ought to
have a WG and several comments that I believe add up a
conclusions that the only way we are going to get a WG is to
redefine this list as one ... and that probably won't work
either.  We've had a few exchanges between Asmus and myself that
I'm learning from but that I don't think are moving us forward.
We've had a review by John Levine that is encouraging because it
means that someone other than Patrik and myself has looked at
the document, but that doesn't show a path forward either (not
his job).  And we've had a few notes from Pete about procedures
and comment styles but, AFAICT, no direction that will move us
forward.  My (I thought) fairly simple question as to whether
I/we should expect the directorate to review that draft and
advise the ADs on what to do with it or whether I should try to
find an AD to sponsor it has not been addressed by Pete, Peter,
or any of the ADs.

I hope my being unable to find this encouraging is no surprise.
I am having a good deal of trouble prioritizing this effort,
particularly working on the various documents, ahead of work
that actually generates interest and, btw, income to pay the
bills.  I can push back, as I have above, to try to find things
that are within the IETF's scope and actionable, but if we can't
figure out how to process even the two most obviously actionable
documents we have on the table
(draft-klensin-idna-unicode-review and
draft-klensin-idna-rfc5891bis), then there doesn't appear to be
a lot of point. Unlike most of the rest of what should be in the
directorate's queue, those two documents make no substantive
changes to IDNA2008 that, e.g., change the derived properties of
code points so they should not be very controversial.   I could
make a fast pass through the "registry responsibility" document
and encourage Asmus to join me to be sure that we haven't
changed our minds about anything significant and then post a
current (non-expired) version but, right now and given the
underwhelming response to the draft posted a bit under two weeks
ago, it is questionable whether even that is worth the trouble.

So, is there a path forward or was this directorate idea a
well-intentioned failure that leaves us back where we were a
year ago?   Is there going to be a report on the directorate on
the morning of 22 July and, if so, what do the five of you
expect to say?  

best,
   john

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Wed, 26 Jun 2019 11:59:28 -0700

> possibly-actionable terms...

OK, let's.

[Note added after finishing my reply: it felt like a large part of your 
message was continuing our conversation, and others may not necessarily 
have something to add to the details. However, the "meta question" 
really addresses a different audience, and people should be sure to read 
to the end. Esp. Pete&Peter.]

> ask and they say "no"?   Also see "Final, meta, questions" below.
Machine readable for Unicode means "property". Currently they document
this in tables in the text, but that's no place for such normative 
information.

(There is other information about these scripts that also, over time, 
has turned
into properties. By its nature, the other stuff is less "stable" as it 
is concerned
with rendering and those things tend to have a feedback loop with rendering
technology (and are exposed to potential orthography reforms). This "do not
use" info, however, appears to be more fundamental and more on the
encoding level.)

I would simply ask and take it from there.

> be done about these in IDNA and, if so, what would it be?

Not sure, unless you admit a discussion of variants (or the role of variants
in defining registration policies aka. LGRs). The omission of (blocked) 
variants
after IDNA has both repertoire and context rules is one that places IDNA2008
out of step with where domain names are going.

(Note, I think "allocatable" variants, despite sharing the name 
"variant" are
a different beast - while "blocked" variants directly improve security and
only that, "allocatable" variants are often motivated by the desire to allow
users of different communities to access the same resource, no matter their
local keyboard, or input method settings. They also cut down on spoofing,
but that is secondary. There are very few "allocatable" variants that are
truly unavoidable: Tamil "sri" / "shri" comes to mind.)

> How
> identical (or at least "alike")?

Tamil "sri" / "shri" are two ways of spelling the same syllable; rendering
systems will make them come out identical, even though only the second
one is "formally" the correct one. For historic reasons (Unicode lacked a
dedicated SHA character) the "wrong" sequence was for a long time the
only way to go. However, the "correct" spelling isn't universal and both
exist side-by-side. Well, one is technically more correct than the other,
so you could add an additional normalization step that always forces the
correct one, but no software exists to do that.

This is an example where blocking alone is insufficient; users will not know
why typing a perfectly good label doesn't work. Hence, this is one of the
very few allocatable variants the root zone has for Indic scripts.

> If there is a correct choice
> currently important ones are not)?

That question seems orthogonal to the issue I'm trying to address here.

However, let me elaborate this: for complex scripts, you can have
code point sequences that are invalid, not just "not words". Neither
users nor rendering engines are prepared for doing something sensible
with structurally unsound ("impossible") syllables. Users read in syllables,
and software likewise renders syllables.

> Is the Root Zone development
> be a lost cause)?

Not really. In most Indic scripts you can put together any random string
of consonants, or consonants mixed with independent vowels. Those
subsets of strings are as close as you get to "random" letter sequences
possible in those scripts - they do exist in the wild as acronyms, at least
for some of these scripts.

Generally, we tended to push back against restrictions that seemed like
"spelling rules", preferring those that are "structural", and preferably
"deep structural" (if I may use this term).

> Considering your work with and comments

That's the way we squared that particular circle for the Root Zone (to the
best of our abilities). However, some of the rule systems represent 
compromise.
If it was easier to model an approximation, we would do that, allowing
both under and over production of labels.

In some cases, languages using different ideas of structure share a given
script. In Thai, we were unable to accommodate one of the minority
languages, because so much of Thai computing infrastructure has the
rules for the Thai language baked in, that allowing something that violated
assumptions for that language would have resulted in unstable or
ambiguous rendering for the minority language labels. So, we went
conservative and had to limit things.

That's why the protocol would not be a good fit to shoehorn all these
rules as CONTEXTO. But the IDNA2008 should be updated to mandate that
"if you support a complex script, you must provide context rules."

(And perhaps we can find a way to list those scripts and/or a minimal
set of code points for which a context rule should be mandatory, no
exceptions: it's pretty clear that, for example, all VIRAMA code points
need to have a context; after all, like joiners, they are invisible in many
situations; unlike joiners, their context rules aren't as universal).

> public zone question alone, does it impact IDNA in any way?

Yes, in the sense that we should strongly imply that applying IDNA to
archaic scripts in public zones is unsafe. "Secure" is probably a bad choice
so maybe some different term, but we should define a designation for
these partially more secure registry policies that

(1) avoid unanalyzed scripts (ancient, obsolete etc)

(2) limit code points to modern/everyday use ones (better recognition)

(3) provide required contexts (complex scripts)

(4) provide blocked variants for "identical" code points/sequences
       and/or "allocatable" ones in limited circumstances

In other words, without changing base IDNA2008, we should provide
something that can be conformed to (not least so ICANN could
potentially require this of contracted TLDs)

> draft-klensin-idna-rfc5891bis.

Compare this wishy-washy (sorry) description to my points (1) through (4)
above. Which one reads as more actionable?

Also, much of the discussion for IDNA2008 has focused on repertoire,
even RFC 6912 only talks about repertoire selection. A policy (aka LGR)
designed for a typical script needs ALL three legs: repertoire, context
and variants, because most scripts (not most languages, but most scripts)
require all three, and many require two out of the three.

We are up against the "unknown unknown" here: even well-meaning zone
administrators do not know what the don't understand. They will tend to
answer your question in the affirmative.

Part of it comes down to people not connecting the rendered string with
the encoded string. In Latin, the difference is of no practical concern.
In complex scripts, knowing the code points is meaningless, if you don't
know how they interact. But a string is just a sequence of code points.
Hence the disconnect.

>   Would it work any better as

Mechanical verification with an LGR at registration time is inexpensive.
We have solved the issue of how to encode registration policies so that
machine verification against such policies is now trivial. That didn't use
to be the case.

I'm thinking that "advice" is nice, but you want to elevate it to something
that can be conformance tested. Only that way could some body like
ICANN write it into contracts, for example.

> general, if vague, knowledge or and some helpful guidelines" for

If my morning coffee hasn't worn off and I can parse this correctly, I think
you are reaching for something similar to my points (1)-(4) above; those
could be hammered out to become a "profile" on IDNA2008 that would
add a layer of protection against cases where strings are not just similar
(like "rn" and "m") but effectively identical where even an observant user
cannot tell apart two labels that are not presented side-by-side in 22pt
fully serifed fonts.

(Many cases like U+0259 and U+01DD) are 100% identical, even in 72pt
fonts and in *every* font, but we like to slightly relax the definition of
identical to cover "effectively identical" - code points or sequences that
users will unhesitatingly "substitute" for each other. This would include
some distinctions that can be detected with a good font, at high resolution
and by people that know the subtle things to look for.)

Unicode has a list of "intentionally identical" code points; many are cross-
script, some cross the letter/digit divide, but others are the result of
disunifying the encoding of the same shape based on things like bidi
properties or casing pairs. For that list, you could easily mandate 
"blocked"
variants (or allow optionally "allocatable" variants). The profile would
then demand that your policy deal with these, but also strongly
recommend that the same mechanism be used for "effectively" identical
code points.

Does that sound like a start?

You can then further recommend that every zone have some scheme
of resolving issues around "confusables" of a wider range. Unicode has
some files on those, if you'd like being able to conformance test.

> even though I now believe that we botched some of the details.

It reflects the fact that its authors were not familiar with all scripts. As
part of the RZ-LGR project, it's been my privilege to become familiar with
tons of details of all modern, every day use scripts; that perspective is
rare. Some rendering system designers may have that also, but they don't
look at it from same angle as you would do for identifiers.

> and agreement on what we should do.

Do you like the suggestion of a "restricted profile" on IDNA208 (or whatever
name you'd like to give it) that focuses on the extra security issues for
"public" zones (however we want to define those)? That's something that
I believe we can make concrete and actionable (essentially something someone
could "conform" to).

> meta, questions" discussion below, but it states the problem at

(I would let Pete&Peter worry about how to make things happen in the IETF
process arena - I would like to focus on getting the technical aspects 
right).

> wrote to each other earlier...
(Again, good thoughts, but should be on a separate track with different 
people
making the call).
> prevented work getting done either.

(W3C participants may be funded by their own employers, who understand
their companies' dependence on these specifications. Not the case here.)

> "out of scope" and, instead, appear to be expected to prioritize
(W3C have the same problem: they are lucky to have some people with wide-
ranging abilities. I can usually give useful input if I can identify 
character
coding related issues, or if someone identifies for me where they are.)
> decisions that are not, historically, about character encoding.

Any time someone talks about "discarding Unicode" is a good example of that.
Or, defining some non-Unicode normalization, etc. Shows an unwillingness
to accept that these things are out of scope and need to be taken as they
exist (and then mitigated where they don't come tailored for identifiers
out of the box).

> discussions in this group, will have been posted for two weeks
Hey, I at least read your draft. Was busy publishing RZ-LGR-3. (It's 
done, but
there's a two-week embargo, because of the ICANN meeting).
> I'm learning from but that I don't think are moving us forward.

(If you haven't caught on, I'm using () to mark off things that are meta or
out-of-scope comments or an occasional aside. The next one is meta).

(We need to get Pete&Peter to come down unambigously in support of
us finishing that series of "core" documents +/- whatever factoring of these
documents we end up with. I'd leave it to them to find suitable additions
to our process. I don't have the IETF expertise to be productive and it
frankly wastes my time, which I could put to use on technical contents.)

> find an AD to sponsor it has not been addressed by Pete, Peter,

(I had a chat with Pete where he promised to be more supportive/proactive,
but he'll have to put that into action.)

> a lot of point.

That latter one, can we move that towards proposing a profile like I am 
suggesting.
(And you can keep the advice that all zones should be limited to 
well-understood
stuff and add that they should also follow whatever items (1)-(4) they find
applicable for their zone, to make even that advice more "actionable").

Item (5), incidentally, ought to be that the relevant zone policies are 
published
in RFC 7940 format.

For the former, I like the idea of an "under review" classification; I 
am otherwise
inclined to follow Unicode even on property changes, as long as they don't
affect, directly or via related code points, the interpretation of 
"high-use"
modern code points. Obviously, moving towards DISALLOWED would be an
issue, but not everyone I spoke to agrees that keeping code points PVALID
after we discover (strong) reasons why it shouldn't have been PVALID to
begin with is such a good idea. (That's in the details).

> Unlike most of the rest of what should be in the

I'd like that (as far as it involves me :) ).

(We should get P&P explicitly involved in creating a roadmap of how this
stuff is supposed to move forward. I feel this process could use a bit of
management).

> the morning of 22 July and, if so, what do the five of you

(Seems mostly addressed to the other members, so I'm not adding a 
response for now).

I found this rather constructive, esp. the parts that weren't meta 
(those are largely for a different audience than me in my role of 
technical expert).