Subject: Re: [I18ndir] draft-faltstrom-unicode11-07

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 10 Mar 2019 23:24:31 +0900

On 15 Feb 2019, at 2:12, Asmus Freytag wrote:

> Section 4.3 claims U+29A1 is PVALID from the beginning, but I find this in the IANA IDNA paramater files: 
>
> 2701-2B4C | DISALLOWED | UPPER BLADE SCISSORS..RIGHTWARDS ARROW ABOVE REVERSE TILDE OPERATOR
> ---|---|---
>
> There appears to be a contradiction; it should be reviewed and addressed as necessary.

The text was wrong. It has always been DISALLOWED.

   paf

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 00:20:21 +0900

On 15 Feb 2019, at 5:31, Marc Blanchet wrote:

> Hello,
>  read it (did not check the content of Appendix A). comments:
> a) it says in the intro:  « It further suggests a path forward for the IETF to ensure IDNA2008 follows the evolution of the Unicode Standard. ». Unless I skip something, it is far from clear to me that when a new version of Unicode is out (as one is expecting pretty soon and as Asmus wrote, will likely to happen before the RFC is published), what is the exact path forward? Are we going to issue a new RFC every time a new Unicode standard is published? My suggestion would be to add a section such as: « Guidance to the IANA expert reviewer » and write some rules so that we don’t need an RFC or IETF concensus for no-issue-unicode-new-version. Some text such as:

As the discussions in this group conclude as far as I can see (Harald is to correct me here) "the document is ok regarding validation of IDNA2008 but not further than Unicode 11.0.0", I have removed this text in the intro and because of this not added the text below either.

> <section title=« Guidance to the IANA Expert Reviewer »>
> If a new version of Unicode is published, the IANA expert reviewer for the IDNA2008 registries should verify that: a) no codepoint in the previous version of Unicode changes from PVALID to DISALLOWED as a result of applying the rules and properties to the new Unicode version, b) no change in normalization of sequences c) …   The result of the audit by the expert should be shared to the [email-masked] mailing list for other people to confirm. If any of the criteria is not met, then the expert should not confirm to IANA to proceed to the publication of the new registries of the new Unicode version and report to the appropriate Area Director for further instructions. If all criteria is met and the audit is confirmed by other people, then the expert can confirm to IANA to proceed with the publication of the new registries of the new Unicode version.
>
> b) I think it would be appropriate to cite the Root-zone LGR for registries to look for advice on how to limit the number of codepoints. To my knowledge, this is the only process, for most live scripts, that creates such an extensive list, with full review from script communities. This could be added at the end of third paragraph of section 5, or last paragraph of section 3.2.

As the talk about need for creation of a repertoire is removed, this is no longer a suitable change I think.

> c) as proposed by Asmus, I think given timing, we should aim for including Unicode 12 in it as it is imminent.

I do not see consensus for this, so for now it stays with Unicode 11.0.0.

   Patrik

---

From: "Marc Blanchet" <[email-masked]>
Date: Sun, 10 Mar 2019 11:34:54 -0400

On 10 Mar 2019, at 11:20, Patrik Fältström wrote:

>> a) it says in the intro:  « It further suggests a path forward for 
>> the IETF to ensure IDNA2008 follows the evolution of the Unicode 
>> Standard. ». Unless I skip something, it is far from clear to me 
>> that when a new version of Unicode is out (as one is expecting pretty 
>> soon and as Asmus wrote, will likely to happen before the RFC is 
>> published), what is the exact path forward? Are we going to issue a 
>> new RFC every time a new Unicode standard is published? My suggestion 
>> would be to add a section such as: « Guidance to the IANA expert 
>> reviewer » and write some rules so that we don’t need an RFC or 
>> IETF concensus for no-issue-unicode-new-version. Some text such as:
>
> As the discussions in this group conclude as far as I can see (Harald 
> is to correct me here) "the document is ok regarding validation of 
> IDNA2008 but not further than Unicode 11.0.0", I have removed this 
> text in the intro and because of this not added the text below either.

so we are collectively saying that we are limiting the update of the 
IANA registries up to Unicode 11.0.0 and any new version of Unicode is 
not supported by IDNA2008, as it is unspecified for IANA. It may be that 
Unicode 12.X or other new ones, including minor versions, will not have 
more or less known issues than Unicode 11.0.0 but we are saying: Unicode 
12 not supported. Hence, the text I suggested to carry the cases where a 
new Unicode version does not harm more than the previous.

Given the popularity of emojis and their new versions coming every year, 
implementors and vendors of OS, librairies and other tools are 
aggressively updating their systems/libraries to the new versions of 
Unicode. Therefore, to handle IDNA2008 with a old version of Unicode is 
just way complicated for implementors. So freezing to a specific version 
of Unicode is just not helping IDNA2008, IMHO.

And I’m not sure we have collectively cycles to issue a new 
internet-draft for every Unicode version that goes out, including minor 
versions.

Regards, Marc.

>> If a new version of Unicode is published, the IANA expert reviewer 
>> for the IDNA2008 registries should verify that: a) no codepoint in 
>> the previous version of Unicode changes from PVALID to DISALLOWED as 
>> a result of applying the rules and properties to the new Unicode 
>> version, b) no change in normalization of sequences c) …   The 
>> result of the audit by the expert should be shared to the 
>> [email-masked] mailing list for other people to confirm. If 
>> any of the criteria is not met, then the expert should not confirm to 
>> IANA to proceed to the publication of the new registries of the new 
>> Unicode version and report to the appropriate Area Director for 
>> further instructions. If all criteria is met and the audit is 
>> confirmed by other people, then the expert can confirm to IANA to 
>> proceed with the publication of the new registries of the new Unicode 
>> version.
>>
>> b) I think it would be appropriate to cite the Root-zone LGR for 
>> registries to look for advice on how to limit the number of 
>> codepoints. To my knowledge, this is the only process, for most live 
>> scripts, that creates such an extensive list, with full review from 
>> script communities. This could be added at the end of third paragraph 
>> of section 5, or last paragraph of section 3.2.
>
>> c) as proposed by Asmus, I think given timing, we should aim for 
>> including Unicode 12 in it as it is imminent.
>

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Sun, 10 Mar 2019 11:31:01 -0700

> The text was wrong. It has always been DISALLOWED.

Good to have that cleared up.

A./

---

From: Peter Saint-Andre <[email-masked]>
Date: Sun, 10 Mar 2019 13:51:19 -0600

[Without my Secretary hat on.]

On 3/10/19 9:34 AM, Marc Blanchet wrote:
>>> Standard. ». Unless I skip something, it is far from clear to me that
>>> when a new version of Unicode is out (as one is expecting pretty soon
>>> and as Asmus wrote, will likely to happen before the RFC is
>>> reviewer » and write some rules so that we don’t need an RFC or IETF
>>> concensus for no-issue-unicode-new-version. Some text such as:
>>

Unfortunately, this seems to be the situation we're in. I suppose you
could say we're "kicking the can down the road" with this update;
however, at least we're getting unstuck from a very old version of
Unicode. I agree we do need to figure out a better path forward.

Peter

---

From: "Marc Blanchet" <[email-masked]>
Date: Sun, 10 Mar 2019 15:58:04 -0400

On 10 Mar 2019, at 15:51, Peter Saint-Andre wrote:

>>>> that
>>>> IETF

I proposed one: with some guidelines to the expert reviewer, if a new 
version of Unicode comes in, the expert reviews the new version, 
following the guidelines we state, share its review with some other 
experts (might be this mailing list) and then if no issues found (more 
than the one we already know), then proceed with updating the IANA 
registry to the new Unicode version. That way we are not stuck on a 
older version that causes no more harm than what we have.

Marc.

---

From: Peter Saint-Andre <[email-masked]>
Date: Sun, 10 Mar 2019 14:12:24 -0600

Right, that seems reasonable and within the purview of the expert reviewer (who, thankfully, is an expert). Although perhaps the whole directorate could be helping...

Peter

Sent from mobile, might be terse 

> On Mar 10, 2019, at 1:58 PM, Marc Blanchet <[email-masked]> wrote:
> 
> 
> 
> I proposed one: with some guidelines to the expert reviewer, if a new version of Unicode comes in, the expert reviews the new version, following the guidelines we state, share its review with some other experts (might be this mailing list) and then if no issues found (more than the one we already know), then proceed with updating the IANA registry to the new Unicode version. That way we are not stuck on a older version that causes no more harm than what we have.
> 
> Marc.
> 
>> 
>> Peter

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 07:21:32 +0900

FWIW, what I have done as an expert is exactly this UNLESS I have discovered some (incompatible) changes in the Unicode Database that MIGHT require a backward compatible exception added to RFC5892. That was the case with the Hamza Above and that was the case with Unicode 11.0.0.

Btw, between 11.0.0 and 12.0.0 there are no such changes. And given this draft sails through, I feel "more comfortable" on approving 12.0.0 just like I have approved some 6.x.0 versions.

   Patrik

>> I proposed one: with some guidelines to the expert reviewer, if a new version of Unicode comes in, the expert reviews the new version, following the guidelines we state, share its review with some…

  Patrik Fältström
  Technical Director and Head of Security
  Netnod

---

From: John C Klensin <[email-masked]>
Date: Sun, 10 Mar 2019 20:01:45 -0400

Another over-long note, unfortunately.   

Executive summary:  Thinking about several recent notes and some
implications of draft-faltstrom-unicode11-07 has caused me to
got back to the IDNA2008 specs to try to find the basis for
these reviews.  While I hope it does not come as news to any of
us, we may need reminding of what those specs actually say.   In
particular: (1) The IANA tables are not normative, the rules in
IDNA5890-5894 are.  Normatively, IDNA2008 is on Unicode 12.0 and
has been for about five days now.  (2) While a particular
registry or other entity could make a different decision for its
own purposes, there is nothing about the IANA tables that imply
whether a particular Unicode version is supported. (3) There is
nothing in the IDNA2008 documents that explicitly specifies this
type of review, what it should consist of, or even whether it
should produced a report. (4) If there is a bias in the
documents between "follow Unicode" and "preserve backward
compatibility" is to toward the latter.  (5) The tone, and
probably some of the text, of draft-faltstrom is inconsistent
with some of the above and should be adjusted to avoid future
confusion. (6) Substantially independent of the content of
draft-faltstrom, we really need to get the review process and
its role nailed down and explain to various communit8ies what is
normative and what isn't. (7) Given the release of Unicode 12.0,
we should quickly review and make a recommendation as to whether
the I-D should be held until a report on the current version can
be incorporated.

Inline below.

--On Sunday, March 10, 2019 11:34 -0400 Marc Blanchet
<[email-masked]> wrote:

>>> a) it says in the intro:  « It further suggests a path
>>> forward for  the IETF to ensure IDNA2008 follows the
>>> evolution of the Unicode  Standard. ». Unless I skip
>>> something, it is far from clear to me  that when a new
>>> version of Unicode is out (as one is expecting pretty  soon
>>> published), what is the exact path forward?
>...
> so we are collectively saying that we are limiting the update
> of the IANA registries up to Unicode 11.0.0 and any new
> version of Unicode is not supported by IDNA2008, as it is
> unspecified for IANA. It may be that Unicode 12.X or other new
> ones, including minor versions, will not have more or less
> known issues than Unicode 11.0.0 but we are saying: Unicode 12
> not supported. Hence, the text I suggested to carry the cases
> where a new Unicode version does not harm more than the
> previous.
> 
> Given the popularity of emojis and their new versions coming
> every year, implementors and vendors of OS, librairies and
> other tools are aggressively updating their systems/libraries
> to the new versions of Unicode. Therefore, to handle IDNA2008
> with a old version of Unicode is just way complicated for
> implementors. So freezing to a specific version of Unicode is
> just not helping IDNA2008, IMHO.
> 
> And I'm not sure we have collectively cycles to issue a new
> internet-draft for every Unicode version that goes out,
> including minor versions.

Patrik and Marc,

So that the comments below can be read by all of us in the same
context, Unicode 12.0 was officially announced last Tuesday, 5
March,
   554 new characters,
   four new scripts, and, for those who care,
   61 new emoji characters
http://blog.unicode.org/2019/03/announcing-unicode-standard-version-120.html

So, if any of the sense of urgency associated with
draft-faltstrom-unicode11 was associated with "get this out
before Unicode 12 appears", we've lost that window.   Going
forward, while I want to recognize and deal with urgency if it
is real, I think we all know that trying to do things quickly
and under pressure often leads to errors and other types of
poor-quality work, so we should be careful that the urgency is
indeed real.

I've downloaded some of the tables and text for 12.0 last week
and my long note was based on it even though I downgraded a
reference or two to refer to Unicode 11.0 for consistency with
the I-D.   Based on a superficial inspection of the descriptions
of changes and a spot check of the UnicodeData table, I have
trouble convincing myself that anything about new code points in
Unicode 12.0 justifies a sense or urgency about IDNA2008
calculations and tables whether implementations upgrade to new
versions of Unicode or not.  What I see are newly-added historic
scripts and abstract characters and code points (including the
new emoji) for which there is unlikely to be a rush for
inclusion in labels for legitimate identifier uses.  The group
includes many code points that the IDNA2008 algorithm (unchanged
from 5892) would conclude were DISALLOWED.  I have not tried to
run a comparison for category changes for code points that were
defined in 11.0 or earlier but, as noted in the long review, any
such changes result in problematic incompatibilities between
versions of the IANA tables whether we decide to treat them in
some special way or not.

The release of 12.0 does raise the question of whether we should
withdraw draft-faltstrom-unicode11, apply the same reasoning and
explanations to 12.0, and then push the more current and
comprehensive document through the system. Very little of the
text, other than Section 4.3 (of version -07 of the I-D), is
specific to Unicode version 11 and we are already focused on the
I-D and underlying principles, so I'm guessing that going
directly from 7.0 to 12 would not represent a lot of extra
work... work that would be easier to do now than trying to draw
an effort together a year hence.   I'm not advocating for doing
that, especially in the moments when I think the directorate
will succeed sufficiently that doing a review for tables for
Unicode 12 in some months will not be an ordeal.  But I believe
we, in conjunction with the ART ADs and other relevant parties,
should give it a bit of thought rather than rejecting the
possibility by inertia.

However, whether we view draft-faltstrom-unicode11 as a normal,
if overdue, code point review and update or whether we see it as
a perhaps-clumsy patch to catch us up, the question of what we
are going to do about future versions of Unicode seems to me to
be critical.   My preference would have been to answer that
question and document the results as part of, or in parallel
with, draft-faltstrom-unicode11, but we don't seem to be having
that discussion.  My fallback is that draft-faltstrom-unicode11
should at least be very clear about which of the two it is.  I
note that, if it is "normal", the apparent failure to consider
and resolve the new code point review for Unicode 7.0 [1] and
the absence of such reviews for Unicode 8, 9, 10, and 11 (and
maybe 12 if we decide to incorporate it) should be a showstopper.

I also have not ever interpreted any part of IDAN2008 as
equating "no IDNA table" with "unsupported".  "Unverified",
maybe, but not "unsupported".  To be sure I wasn't making that
up, I've gone back today and done a careful rereading of
relevant parts of RFCs 5892 and 5894 (to which 5892 refers for
information about the IANA tables).  That rereading was very
illuminating.  It is reflected below and I would encourage
others to reread the specs too.    Some other body might
certainly decide that it was not interested in unverified
versions of Unicode in IDNA, but is not an IDNA2008 (or other
IETF) requirement, certainly nothing that 5982 has to say.  That
relationship is a bit of a separate question, but another one
that I think we need to sort out, be explicit about, and
probably update IDNA2008 in some normative way to reflect.   

Going forward (and _not_ as part of draft-faltstrom-unicode11),
I agree with Marc that we need to come to grips with our plan
for future review cycles (whether starting with Unicode 12 or
Unicode 13).   Because there seems (nearly a decade later and
despite the comments below) to be some uncertainty about what
IDNA2008 intended, I think whatever we agree on should be
documented in an update, probably to 5892 but perhaps elsewhere
in the set.  I think it is very important that whatever we
decide be realistic, i.e., if we cannot realistically expect to
get the energy together to generate a new I-D in a timely way
for each Unicode version, then we had best not specify that.

To be clear about the present status of things, there is no
explicit requirement in RFC 5892 for periodic updates to the
IANA registry.  It creates the registry, populates it with
Unicode 5.2 values, and while some statements in the text can be
read as implying updates of the IANA registry for new versions
of Unicode, it does not specify that or the procedure for
getting there.  It does refer to RFC 5894 for that information.
Section 7.1.2 of RFC 5984 talks about the requirements for
registries to update their own tables (see particularly the
second paragraph (starting with "Under this model, registry
tables...").  What it does not say is "that registries should
look at the tables on file with IANA and believe what they find
there".  Similarly Section 7.1.3, first bullet, says that lookup
applications "Maintain IDNA and Unicode tables that are
consistent with regard to versions,", i.e., the Unicode version
they use is up to them (not some higher authority), but they
must use IDNA tables that are consistent with it.  Again, no
statement about going off to IANA and fetching something
authoritative there.   

The IANA Considerations section of 5894 is explicitly
descriptive rather than normative and does not specify any IANA
actions (5894 is, after all, Informational) but it does say:

	"While not normative, an IANA registry of characters and
	scripts and their categories, updated for each new version
	of Unicode and the characters it contains, are convenient
	for programming and validation purposes.  The details of
	this registry are specified in the Tables document."

So, good idea to update the IANA tables but, updated or not, the
rules of IDNA2008 are  the only normative stuff around and there
is no notion of "unsupported" because the IANA tables have or
have not been updated.   For completeness and to save others
work, RFC 5890 and 5891 don't specify procedures for updating
the IANA tables or say anything about validity or supported-ness
in conjunction with them either; they just point to 5892 and
5894.

I've looked at the few notes from that period that I have
readily available, tried to search my fading memory, and reread
what seem to be the relevant sections of 5892 and 5894 and the
I-Ds that preceded the latter.  I believe our intent was that
the IANA tables would be regularly updated with new Unicode
versions and with modifications to IDNA (protocol, tables, or
exception lists in the latter) as a convenience for anyone who
wanted them but that they had no normative effect, much less an
effect about which Unicode versions were "supported".  RFC 5894
is quite explicit that a key reason for moving from IDNA2003 to
IDNA2008 was to get away from the requirement to do a
substantive and normative update to the standard for each
version of Unicode.    There is, at best, an implicit
requirement for a regular review but the remedy for a Unicode
action that is problematic for IDNA is an update to the relevant
section of one of the IDNA documents when the problem is
discovered, not the result of some IANA table-building process.
It is interesting that RFC 6452 reinforces that view: it doesn't
say "this review was done because the base IDNA2008
specifications require it".  Instead, at the end of Section 2,
it simply says  "This RFC has been produced because 6.0 is the
first version of Unicode to be released since IDNA2008 was
published".

Another point of interest is that it seems clear from the
discussion of backward  compatibility in 5892 and 5894 that, if
Unicode changed the properties of an existing code point, our
bias was to preserve the older derived property unless it could
be shown that no harm would result from the switch or at least
that the advantages of staying with Unicode clearly outweighed
the harm of an incompatible change.  As reports, both 6452 and
the current draft-faltstrom seem to ignore that preference
without comment, simply asserting that it is better to say with
Unicode, a position for which there is little or no support in
any consensus base IDNA2008-related document.  On the other
hand, if someone felt strongly about preserving backward
compatibility for any of those code points, they would be free,
at any time, to propose an update to IDNA2008 to make that
correction.  I assume (or at least hope) that such a proposal
would cause a lively debate about the tradeoffs for that
particular code point or set of code points_ between stability
and backward compatibility one one hand and consistency with
Unicode and, in many cases correctness, on the other (I assume
that most of the property changes between Uniocde versions are
because UTC was persuaded, or concluded on their own, that the
initial classification was incorrect).

Curiously, this probably also answers the document category
question.  Both RFC 6452 and draft-faltstrom-unicode11 are
reports of analyses that are not normatively required by
IDNA2008.   They do not update or otherwise modify the IDNA2008
specs.  And, while they explain the IANA tables and changes that
were (or were not) made (a very useful thing to do), neither the
tables nor the explanations have any normative effect.   Sounds
to me about as close to a description of Informational as one
can get... probably to the point that the IESG should be asked
to reclassify 6452 to Informational for consistency.

For the future, more generally, and with the understanding that
this entire "review new Unicode versions and update the IANA
tables" business is oral tradition rather than anything that
makes a particular version of Unicode more or less valid for
IDNA, I think we are at a three-way fork in the road.   One path
involves our doing a serious, mandated, review for each new
Unicode version with the serious possibility of modifying
IDNA2008 to including more code points in the exception list but
the consequence of actually tying IDNA2008 to a specific and
normative version of Unicode.  A second would involve a review,
and a report on that review, but with the review being no more
than advisory about any modifications to the IDNA2008 tables of
rules or protocol.  That is what the IDNA2008 documents seem to
call for (and is very different from the way draft-faltstrom has
been treated in recent discussions).  The questions of how bad a
problem needs to be to justify a change to IDNA2008 to, e.g.,
preserve backward compatibility and whether to do new code point
reviews almost certainly need to be addressed if we are going to
take either of those two paths and, to the extent possible, we
probably should get it written down this time.   The third path
(actually the other extreme -- there may be intermediaries) is
to decide that these reviews are a waste of time (and scarce
resources), that we are never again going to decide to deviate
from wherever the Unicode properties, or changes in property
values for existing code points, lead us and we should just
change IDNA2008 to eliminate any notion of a review and provide
for uncritical generation of new IANA tables shortly after new
Unicode versions appear.  The latter still would not prevent
proposals to change the standard in a way that would affect both
the normative rules and the tables but I'd hope we would be
confident that such proposals would be taken as seriously as
they deserve and processed, not thrown in the back of a queue
that never moves.

FWIW, I'm less pessimistic about our ability to more clearly
define and then take the first or second path than I believe
Marc to be after reading the comments above and in his earlier
notes.  First, I don't see much evidence that, as long as there
are no terrible surprises with a new version of Unicode, doing
the calculations, doing a quick pass through the new code points
(especially those that would come out PVALID or that seem
obviously controversial) should require a huge amount of effort.
Producing an I-D to summarize the process and it conclusions
should certainly not be a big deal.  The thing that held this
revision up for years was not the difficulty of running the
tables or producing I-Ds.  It was precisely that we encountered
what appeared to be a showstopper problem and couldn't manage to
engage with addressing it for years (and, based on
draft-faltstrom-unidoe11-07, we still can't).

On the other hand, if we can't get the energy together to do
these reviews and produce timely updates, it would be highly
questionable whether the IETF is capable of doing any serious
i18n work that has sufficient informed involvement and
participation to make any claims of IETF consensus about the
result plausible.   I am still trying to avoid the conclusion
that we cannot manage such work, but that is getting harder.  If
we cannot, any i18n work that leads to informed consensus, then
what we can or cannot do with IDNA2008 reviews for new versions
of Unicode is probably among the least of our problems.

best,
    john 

[1] An appropriate way to look at
draft-klensin-idna-5892upd-unicode70, especially ifs first few
versions, is that it is a report on that new code point review.
I don't imagine anyone wants to take the time, but probably the
ideal way to proceed with draft-faltstrom-unicode11 (or 12)
would be to recast and publish draft-klensin as a report on a
problem discovered while making a review, publish is, remove
most of the material about Unicode 7.0 from draft-faltstrom, and
make the latter about 8.0-11 (or 12), and not about 7.0 at all.
That would be especially advantageous because, unless Patrik has
fixed it in -08. the treatment of the 7.0 issue in the
draft-faltstrom I-D is really not very good.  On the other hand,
because Patrik is a co-author on draft-klensin, it wouldn't let
him off the hook.

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 14:26:30 +0900

On 11 Mar 2019, at 9:01, John C Klensin wrote:

> (1) The IANA tables are not normative, the rules in IDNA5890-5894 are.  Normatively, IDNA2008 is on Unicode 12.0 and has been for about five days now.

Correct.

> (2) While a particular
> registry or other entity could make a different decision for its own purposes, there is nothing about the IANA tables that imply whether a particular Unicode version is supported.

As soon as people use IDNA2008 this would be the case... ;-)

> (3) There is nothing in the IDNA2008 documents that explicitly specifies this type of review, what it should consist of, or even whether it should produced a report.

Not explicitly, but, my view is that IF a backward compatible rule is to be added to the algorithm, then that addition requires IETF consensus as RFC5892 is to be updated.

This gives for me that regardless of whether a rule actually is added or not, a discussion is needed in the IETF.

The implementation of this have been for me to flag when the following happens:

1. A new version of Unicode is released
2. Code points are added and some have changed data
3. Calculation of IDNA2008 derived property value is made
4. Validation that all derived values that have changed is from UNASSIGNED to something else, if a derived property value changes that originally was NOT UNASSIGNED, then I feel IETF discussion is needed

It can also happen, which have happened at least once, that myself or anyone else, when doing manual inspection of the Unicode Tables do see something "which is weird". This was the case that resulted in IAB talking about "the Hamza above" issue.

If applying 1-4 above, there is a block when going to Unicode 11, there was when going to Unicode 6, but for example not to 10 or 12.

> (4) If there is a bias in the
> compatibility" is to toward the latter.

This implies we SHOULD add a rule.

> (5) The tone, and
> probably some of the text, of draft-faltstrom is inconsistent with some of the above and should be adjusted to avoid future confusion.

Specifically (4).

> (6) Substantially independent of the content of
> draft-faltstrom, we really need to get the review process and its role nailed down and explain to various communit8ies what is normative and what isn't.

Yes

> (7) Given the release of Unicode 12.0, we should quickly review and make a recommendation as to whether the I-D should be held until a report on the current version can be incorporated.

Yes.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 11 Mar 2019 14:44:09 +0900

2nd response on this long email.

On 11 Mar 2019, at 9:01, John C Klensin wrote:

> Patrik and Marc,
>
> So that the comments below can be read by all of us in the same context, Unicode 12.0 was officially announced last Tuesday, 5 March,
>    554 new characters,

PVALID from 122734 to 123006 (+272)
UNASSIGNED from 837091 to 836537 (-554)
CONTEXTJ still 2
CONTEXTO still 25
DISALLOWED from 154260 to 154542 (+282)
TOTAL still 1114112

> Based on a superficial inspection of the descriptions of changes and a spot check of the UnicodeData table, I have trouble convincing myself that anything about new code points in Unicode 12.0 justifies a sense or urgency about IDNA2008
> calculations and tables whether implementations upgrade to new versions of Unicode or not.

Agree, but that can also be used as an argument to "just" say the document include update for 12. The trouble is between 10 and 11.

> I have not tried to run a comparison for category changes for code points that were defined in 11.0 or earlier but, as noted in the long review, any such changes result in problematic incompatibilities between versions of the IANA tables whether we decide to treat them in some special way or not.

I have only compared the changes that according to the IDNA2008 algorithm do change derived property values. I have not checked other data that for example might change normalization (i.e. the step before IDNA2008 takes over).

> The release of 12.0 does raise the question of whether we should withdraw draft-faltstrom-unicode11, apply the same reasoning and explanations to 12.0, and then push the more current and
> comprehensive document through the system. Very little of the text, other than Section 4.3 (of version -07 of the I-D), is specific to Unicode version 11 and we are already focused on the I-D and underlying principles, so I'm guessing that going
> work... work that would be easier to do now than trying to draw an effort together a year hence.

My preference would be:

1. Decide what to do with the incompatible change between Unicode 10 and 11
2. Ensure draft-faltstrom-unicode11 do say so
3. Update the draft for Unicode 12
4. Publish

And then the important thing:

5. Decide what to do

As John said earlier and pointed out, it is the *algorithm* that is normative, not the tables, and the longer we wait with discussing what to do, the worse the situation might get.

And it is (1) and (5) which is important.

> explicit requirement in RFC 5892 for periodic updates to the IANA registry.  It creates the registry, populates it with
> Unicode 5.2 values, and while some statements in the text can be read as implying updates of the IANA registry for new versions of Unicode, it does not specify that or the procedure for
> getting there.

Section 5.1 in RFC5892 says:

5.1.  IDNA-Derived Property Value Registry

   IANA has created a registry with the derived properties for the
   versions of Unicode released after (and including) version 5.2.  The
   derived property value is to be calculated in cooperation with a
   designated expert [RFC5226] according to the specifications in
   Sections 2 and 3 and not by copying the non-normative table found in
   Appendix B.

   If non-backward-compatible changes or other problems arise during the
   creation or designated expert review of the table of derived property
   values, they should be flagged for the IESG.  Changes to the rules
   (as specified in Sections 2 and 3), including BackwardCompatible
   (Section 2.7) (a set that is at release of this document is empty)
   require IETF Review, as described in RFC 5226 [RFC5226].

And this is what I have tried to implement.

> descriptive rather than normative and does not specify any IANA actions (5894 is, after all, Informational) but it does say:
>

Yup.

> discussion of backward  compatibility in 5892 and 5894 that, if Unicode changed the properties of an existing code point, our bias was to preserve the older derived property unless it could be shown that no harm would result from the switch or at least that the advantages of staying with Unicode clearly outweighed the harm of an incompatible change.  As reports, both 6452 and the current draft-faltstrom seem to ignore that preference
> without comment, simply asserting that it is better to say with Unicode, a position for which there is little or no support in any consensus base IDNA2008-related document.

Well, that was my proposal, and I have waited for some counter argument, and discussion. And given that I would update the document accordingly.

I could as well have suggested to add the backward compatible rule. But personally did this because the difficulties we have is the non-conformance to IDNA2008, and I thought multiple versions of IDNA2008 could be problematic. And even harder to push things to IDNA2008v2 instead of "just IDNA2008". If people just load the tables from IANA, it does not matter.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Mar 2019 14:16:39 -0400

(Part of this note is a bit of a rant, for which I apologize,
but it is also an explanation of some comments I made when the
directorate was asked to start on this I-D that some others
claimed were unreasonable... and I think Harald's summary should
reflect those as issues are _are_ related to the current I-D.
Issues on which we clearly agree and don't need further
discussion are elided to keep the note from being even longer.)

--On Monday, March 11, 2019 14:26 +0900 Patrik Fältström
<[email-masked]> wrote:

>> (3) There is nothing in the IDNA2008 documents that
>> explicitly specifies this type of review, what it should
>> consist of, or even whether it should produced a report.
> 
> Not explicitly, but, my view is that IF a backward compatible
> rule is to be added to the algorithm, then that addition
> requires IETF consensus as RFC5892 is to be updated.

The requirement for IETF Review for either of those changes is
quite explicit in RFC 5892 Sections 5.1 and 5.2, so no question
or need for an opinion there.  The only questions are what
initiates the IETF Review process (presumably an I-D proposing
the change) and, more important, what initiates a review, what
is to be reviewed, and whether there is any requirement the
proposals for changes have to be initiated by the Designated
Expert.

Borrowing from a previous set of comments before coming back to
this one,

           ------ Rant alert ----

--On Monday, March 11, 2019 14:44 +0900 Patrik Fältström
<[email-masked]> wrote:

>    IANA has created a registry with the derived properties for
> the    versions of Unicode released after (and including)
> version 5.2.  The    derived property value is to be
> calculated in cooperation with a    designated expert
> [RFC5226] according to the specifications in    Sections 2 and
> 3 and not by copying the non-normative table found in
> Appendix B.
> 
>    If non-backward-compatible changes or other problems arise
> during the    creation or designated expert review of the
> table of derived property    values, they should be flagged
> for the IESG.  Changes to the rules    (as specified in
> Sections 2 and 3), including BackwardCompatible    (Section
> 2.7) (a set that is at release of this document is empty)

And that is one of the sections I've read -- five or six times
in the last few days.  No matter how many times I read it, I
fail to find anything more than a hint about a requirement for
reviews with each new major or minor version of Unicode, formal
reports, exactly what is supposed to be reviewed, or whether the
Designated Reviewer has sole and exclusive responsibility or
authority for any of this.  What I do find, albeit also only as
a hint, is that any tables that are produced are to be submitted
to IANA but that they are more of a forcing function for reviews
rather than an event that creates something normative.  I think
we are in agreement about that, but earlier drafts of
draft-faltstrom-unicode11 may have been less clear about it than
they should have been and, more important, there has seemed to
be a good deal of confusion in the community about whatever IANA
has means vis-a-vis implementation requirements and/or
"supported" versions of Unicode.  I think we have an obligation
to try to clear that confusion up, either in
draft-faltstrom-unicode11 or in another document that we produce
and get through the system RSN (rather than dumping it at the
end of a never-moving queue).

I also find one other thing that seems to be more than a hint.
While I don't think "should be flagged for the IESG" is the
particular language we would use today given both changes in the
IETF and the deep expertise and involvement in i18n issues the
IESG has shown recently (and that I suggest may have been part
of both the reason for creating this directorate and the length
of time it took to get it together), it doesn't say "if a
non-backward-compatible change (presumably defined as a code
point whose derived property changes between Unicode versions or
other problem (presumably including the discovery that a code
point exposes an incorrect assumption in the definition of
IDNA2008), then the Designated Expert gets to decide whether the
change or problem is significant and report only if it is".
What is says, as I read it, is that all such issues are to be
"flagged for the IESG".  Given the way the IETF is supposed to
work, it is then the responsibility of the IESG to figure out,
on a timely basis, how to make a decision that is consistent
with IETF consensus.  In doing that, it is entirely reasonable
for them to consider any advice the Expert provides and any
other advice they get or solicit.  If they conclude that the
conclusion is obvious or the decision trivial, it is reasonable
for them to make the decision and announce it (so as to allow
for appeals if others seriously disagree).  Otherwise, I think
they need to ask the community through something equivalent to
an IETF Last Call, possibly with a WG along the way.

Now, "alerting the IESG" was exactly what we were trying to do
when we posted draft-klensin-idna-5892upd-unicode70-00 in the
middle of 2014.  We alerted the IAB too.  The IAB statement six
months later was, in this context, another alert to the IESG.
Arguably the requests for what became the LUCID and I18NRP BoFs
were more alert attempts even though they had other functions.
What we haven't gotten is a path to resolution of either the
issues identified 4 1/2 years ago or the changed-property code
point you have identified now.   For the latter, perhaps
approval of draft-faltstrom-unicode11 is such a resolution, but
there are problems with that approach:

(i) Nothing in IDNA2008 says that, after being alerted, the IESG
can choose to address one problem but ignore others.  I'm all
for pragmatism and not artificially blocking progress, but that
is worth keeping in mind.

(ii) While I think the explanation in -08b01 is a vast
improvement (and adequate, modulo (iii)), I don't think any of
us who find backward-incompatibilities or other problems get to
say "X is not a problem because of <rule>" and therefore no
action is required", especially when the rule is not supported
by the IDNA2008 specification.  The text in -07 and earlier was
fairly close to that.

(iii) While we've all done it and the many advantages of keeping
things in context suggest that we will continue to do so, there
is an inherent risk of burying (however unintentionally or with
good intentions for editorial clarity) a substantive action in
the middle or what is otherwise just a report -- and one that is
inevitably a little tedious at best.  If someone were to post a
report on comparative IETF traffic statistics with many sections
saying "same as last time" and then include nearly 70 pages of
appendices with, shall we say, very little plot or action, but
insert "The IETF has consensus that interplanetary networking is
all a hoax", I'd guess the odds of that document surviving IETF
Last Call, review team reports, etc., without anyone noticing
and complaining about that ridiculous statement are reasonably
high.  So I would urge that Last Call announcements for this
report and ones like it should explicitly call out the fact that
a decision is being made about whether to preserve backward
compatibility or compatibility with Unicode for a particular
code point whose derived property changed.  Unless we produce as
separate document to address just that one issue and process it
separately, I think a claim of IETF consensus about what to do
about that code point is bogus unless that is called out in the
IETF LC announcement.

And, fwiw, the above is exactly why I argued strongly that it
was unreasonable and inappropriate to move forward with Last
Call and approval of draft-faltstrom-unicode11, at least in its
-07 form, unless known issues and the documents that described
them were examined first.  In retrospect, I should have also
stressed that the IANA registry and associated tables were not
normative and therefore I didn't understand the apparent hurry
after sitting around and making no progress --and with the IESG
apparently not troubled by that -- for years.   I think -08 is
moving rapidly toward that point of incorporating workarounds
for those issues but -07 really was not ready and, whether
because the workarounds represent significant changes or because
we fold in information about Unicode 12.0, I assume we are going
to end up with a second IETF LC, probably on about the schedule
we would have had if we had looked at those other issues first.

       --- end rant alert -----

> 4. Validation that all derived values that have changed is
> from UNASSIGNED to something else, if a derived property value
> changes that originally was NOT UNASSIGNED, then I feel IETF
> discussion is needed

quite reasonable.

> It can also happen, which have happened at least once, that
> myself or anyone else, when doing manual inspection of the
> Unicode Tables do see something "which is weird". This was the
> case that resulted in IAB talking about "the Hamza above"
> issue.
> 
> If applying 1-4 above, there is a block when going to Unicode
> 11, there was when going to Unicode 6, but for example not to
> 10 or 12.

Right.  That leaves the question of whether one can progress
beyond Unicode 7 without resolving the block identified there.

> This implies we SHOULD add a rule.

Probably so, at least in the interest of clarity.  My assertion
that the specification creates a bias in favor of backward
compatibility, rather than following Unicode, is based on the
text in the middle of the Introduction that ends with "Moreover,
even if such changes were the result, the BackwardCompatible
list (Section 2.7) can be adjusted to ensure the stability of
the results" and the first sentence of 2.7, which says "includes
the code points" rather than the "can be adjusted" above and the
"can be updated" of the next sentence of 2.7.  Clearly the IETF
can decide that other considerations (track Unicode versions
included) are more important than stability but it is a choice
and "well, we will just follow Unicode" doesn't do it, at least
without some serous discussion and presumably an update to 5892
to discuss just when stability is more important.

   best,
     john