Subject: [I18ndir] draft-faltstrom-unicode and "outdated versions of Unicode" ... and the "review model" document

From: John C Klensin <[email-masked]>
Date: Sat, 16 Mar 2019 16:58:37 -0400

Hi.  In the confusion of the last week, I apparently wrote one
note that I never sent.  As we think about what needs to be said
or clarified going forward, some of that note may be useful even
though I think all of us are now in synch about it and hence,
for the purposes of the directorate of
draft-faltstrom-unicodeXX, I'm probably kicking the proverbial
dead horse.

--On Sunday, March 10, 2019 13:35 -0700 Asmus Freytag
<[email-masked]> wrote:

>...
> "The Unicode Standard is being updated at least yearly and
> implementors and vendors of OS, libraries and other tools are
> aggressively updating their systems/libraries to the new
> versions of
> Unicode. Therefore, IETF recognizes the burden on implementers
> to support IDNA2008 corresponding to an outdated version of
> Unicode.

It is probably worth having tables as part of an IANA registry.
It is probably worth keeping them up to date (see below about
both of those statements). But those tables are not normative
and do not define a particular version of Unicode for use with
IDNA2008.  The requirement on implementers is that they choose a
version of Unicode and then work either directly with the
[current] rules of IDNA2008 or derive or find a table that
corresponds to that version of Unicode.  There is absolutely
nothing about IDNA2008 that would require an implementer to work
with an outdated version of Unicode of to "support IDNA2008
corresponding with it" (whatever that means).

I say "probably" above because, if the existence of the IANA
tables going forward is to create confusion about this --
especially if we don't get IANA to keep versions of each table
corresponding to every version of Unicode from 5.2 forward -- I
think it may be critical to either add more notes to the top of
that IANA table or to get rid of the IANA tables entirely.
That is something we should think about when we are working on
the "review model" document because that would be a logical
place to adjust the instructions to IANA going forward.

Now, should some body with a claim to regulatory or contractual
authority over some collection of zones say "you are allowed to
only use Unicode version X" or "you must use the version of
Unicode corresponding to the IANA tables at the time of
registration of a label", IDNA2008 does not prohibit such a
restriction and the decision to impose it is Not An IETF
Problem.  I would, personally, consider such a restriction
rather dumb unless it turned out to be a requirement of some
more elaborate system for advising registries, but, even if
people agreed with me, that evaluation doesn't belong in
draft-falstrom or any of the core IDNA2008 documents.

I believe and hope that anything in the above that needs to be
said in draft-faltstrom-unicode11 (or 12) has already been said.
However, for work going forward, are we all in agreement about
the above?

best,
   john

    john

---

From: Asmus Freytag <[email-masked]>
Date: Sat, 16 Mar 2019 15:03:14 -0700

I find the IANA tables useful myself. While I'm fully capable of 
deriving them, not having to do that is a benefit. Having them 
unpredictably "out-of-date" adds an uncertainty.

Two things should be avoided in the future:

 1. any consistent lag between the IANA tables and Unicode, say
    something like a 1-year delay (amounting to being at least one
    version off).
 2. any arbitrary "halt" to generating the tables.

As you write, the tables aren't normative because the application of the 
current state of the algorithm to the latest version of the Unicode 
Standard defines the normative data.

The way Unicode works is that it releases the Unicode Character Database 
ahead of time for beta review. It is during this time (which now will 
occur relatively predictably on a yearly pattern for major versions) 
that the review of new characters and effect of potential property 
revisions should be performed.

At the same time a set of "beta" IANA tables can be generated (draft 
update). These can be sent for review in IETF.

If no issues are found and Unicode's beta process does not result in a 
(rare) last minute correction, then the draft tables can be released 
concurrent with Unicode's version release (or as nearly concurrent as 
practical).

If issues are found, the process of adding an exception can be started 
before final release of the Unicode version. If the process of reaching 
consensus in IETF does converge in a timely manner (either before 
Unicode's final release or in a short time window), only the IANA tables 
including the exception should be posted.

However, if the consensus on an exception does not converge, the 
unmodified tables should be posted; while this would make visible the 
"retroactive" nature of applying an exception it only documents the 
normative contents of IDNA2008: as soon as the Unicode version is 
public, the properties can be calculated, and until the RFC with the new 
exception is published, the original, unmodified calculation is what is 
defined normatively by IDNA2008. Therefore, as far as normative content 
of IDNA2008 is concerned, any exception declared after a code point was 
encoded is "retroactive". (The same applies even more so for exceptions 
based on property changes). Therefore, normatively, it doesn't help to 
hold updates of these tables.

A./

PS: freezing any particular zone to an earlier version of Unicode is 
outside the scope of IDNA2008 or the IANA tables. However, it is a major 
pain. For the Root Zone it turns out that the only code point that might 
have been added is, of all cases U+08A1, which is completely harmless in 
the root because none of the Arabic combining marks are allowed. 
However, in terms of the tools we use to create the MSR and the RZ-LGR 
tables and documentations we definitely have "leakage" from later 
versions of Unicode. For example, the tables we generate show fonts and 
annotation data from later versions, and, while marked as "excluded", 
some tables show the presence of "future" characters.

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 16 Mar 2019 23:09:03 +0100

On 16 Mar 2019, at 23:03, Asmus Freytag wrote:

> The way Unicode works is that it releases the Unicode Character Database ahead of time for beta review. It is during this time (which now will occur relatively predictably on a yearly pattern for major versions) that the review of new characters and effect of potential property revisions should be performed.
>
> At the same time a set of "beta" IANA tables can be generated (draft update). These can be sent for review in IETF.

FWIW, they *are* sent to me, and I do check them. I also do send feedback to Unicode Consortium.

So the situation is not as bad as one might think when reading your note.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 16 Mar 2019 23:10:54 +0100

One more think...as this directorate now exists, I will most certainly send notifications here and not only to IAB/IESG when I get notifications of Unicode versions.

   Patrik

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Sat, 16 Mar 2019 15:19:56 -0700

> So the situation is not as bad as one might think when reading your note.

Patrik,

I was simply describing my understanding of the ideal state; glad to 
find reality is closer to it than I had imagined.

A./