Subject: [I18ndir] draft-faltstrom-unicode12 or draft-faltstrom-unicode14?

From: Paul Hoffman <[email-masked]>
Date: Sun, 03 Oct 2021 10:35:50 -0700

Greetings again after a long time. I see that draft-faltstrom-unicode12 
is still getting ready for IESG review, but in the meantime Unicode 13 
and Unicode 14 have been released. Given that many people expect that 
the IETF is keeping up with TUC, how much effort would it be to bring 
this document up to Unicode 14? I'd be willing to contribute if I can, 
but I have been paying no attention to TUS changes for many years.

--Paul Hoffman

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 3 Oct 2021 17:16:22 -0700

On 10/3/2021 10:35 AM, Paul Hoffman wrote:
> Greetings again after a long time. I see that 
> draft-faltstrom-unicode12 is still getting ready for IESG review, but 
> in the meantime Unicode 13 and Unicode 14 have been released. Given 
> that many people expect that the IETF is keeping up with TUC, how much 
> effort would it be to bring this document up to Unicode 14? I'd be 
> willing to contribute if I can, but I have been paying no attention to 
> TUS changes for many years.
>
> --Paul Hoffman
>
Paul,

you raise a good point.

One of the things I've been trying to get the Unicode folks to do is to 
add to their process so they can better flag code points where the IDNA 
2008 property calculation has changed, whether from reserved to PVALID 
or to/from PVALID due to property changes (not that we want to encourage 
that, but better if it's flagged as part of Unicode BETA, than after the 
fact).

I'm told that the data is all there, but they need to add a step in an 
already crowded release process.

For now, getting this out as targeted would be best, otherwise we might 
never see it.

A./

---

From: John C Klensin <[email-masked]>
Date: Sun, 03 Oct 2021 21:46:02 -0400

--On Sunday, October 3, 2021 10:35 -0700 Paul Hoffman
<[email-masked]> wrote:

> draft-faltstrom-unicode12 is still getting ready for IESG
> review, but in the meantime Unicode 13 and Unicode 14 have
> been released. Given that many people expect that the IETF is
> keeping up with TUC, how much effort would it be to bring this
> document up to Unicode 14? I'd be willing to contribute if I
> can, but I have been paying no attention to TUS changes for
> many years.

Let me ask a broader and more difficult question that is
somewhat inspired by Paul's.  What we had originally  intended
(with IDNA2008) and what RFC 8753 clarifies is that we expect
two types of review or each new Unicode version, one which was
largely mechanical relative to prior versions and the other that
required examination of new code points and changes to check for
cases in which Unicode assumptions led to code points and
properties that were inconsistent with IDNA2008 assumptions.
Our, or at least my, assumption when we were putting RFC 8753
together was that we were going to have a healthy and active
directorate that could get the second type of review done and
cross-check it.  

It now appears rather clear that assumption falls somewhere
between "excessively optimistic" and "silly".  If Unicode did
part of the work as Asmus suggested, it would make things easier
but we'd had at least one situation in which our evaluation of
what was being done was different from theirs so even that would
not eliminate the need for a review.   And, ultimately, if we
don't have the will or ability to do a careful review, then it
makes little difference how hard that review is.

So I wonder if we should be thinking about another option, if
only as a clarifying thought experiment: Assume we have no
realistic expectation of being able to do serious i18n work that
requires meaningful and informed consensus in the IETF ever
again (ignoring heroic efforts by one or two people that can
make it through the process because no one else really cares):

(i) Is it worth finishing draft-faltstrom-unicode-12 and, if so,
should we make it an explicit tombstone?

(ii) Is it worth making an effort (after 745 days and counting)
to revisit draft-klensin-idna-rfc5891bis and try to get it out?
And, if so, what about the now-expired documents that are
logically queued behind it, like "troublesome".

If the answer to either of those questions is "yes", who is
going to do the work other than Patrik, me, and maybe Paul for
(i) and Asmus, me, and (possibly but I'm guessing unlikely)
Andrew for (ii).  And, even if the answer to both questions is
"yes" but that we can't manage the reviews that would be needed
to draft-faltstrom-unicode-13-and-14, do we just drop  the ball
and walk away or do we need to figure out who does the work to
document a closing-down process.

    john

---

From: Asmus Freytag <[email-masked]>
Date: Sun, 3 Oct 2021 19:26:51 -0700

> document a closing-down process.

John,

realistically, each new version of Unicode is going to encode characters 
that are more and more specialized. That's in the nature of the process. 
As such, the pressure to get a review done, and get it done "correctly" 
is grounded less and less in practical concerns.

There are three exceptions:

(1) An "new" script gets added (like Adlam) that catches on with a 
previously illiterate population (or one that repudiates scripts that 
they feel were imposed on them). Such a script, if it becomes adopted, 
definitely needs to be vetted for surprises, because it could see 
realistic use in IDNs.

(2) A "new" character gets added that sees widespread use - perhaps as 
result of a writing system reform. Getting that one right in terms of 
identifiers is paramount; again such an addition would be the exception 
that could see realistic use.

(3) An "existing" character gets re-evaluated and reclassified affecting 
the IDNA2008 calculation. Does happen, but it's pretty rare. And even 
then, fewer of these characters are expected to show up widely used in IDNs.

The remainder of any additions are "old" scripts or characters, stuff 
buried in archives and added to allow better digitization of text, or 
"rare" characters, extensions to support some languages that may have 
other issues, whether not-yet-stable writing systems or a population of 
speakers slowly dying out, or both. Expertise will be limited, as will 
the impact of not getting the review "perfect". (We unfortunately didn't 
add a "not ready for prime time" status in IDNA2008).

We looked at the additions to Unicode 13/14 (and 15) from the 
perspective of the Root Zone. The RZ is pretty comprehensive in covering 
languages that are used (and written) actively. We're missing a couple 
of scripts that may be emerging, but the remainder is in scope. Within 
that scope, we had a handful of possible additions (after subtracting 
IPA extensions). That's how limited the realistic scenario has become.

Now, my thought was, if Unicode can flag all the additions/changes in 
terms easily digestible from the IDNA2008 perspective, and does so in 
the context of its BETA process, then any glaring issue can be handled 
by whoever is interested in a follow up. Either try to get properties 
modified/changes retracted in the Unicode process via a public review 
comment, or issue an exception on the IETF side -- before the characters 
get deployed.

The remainder would then be pass-through to the IANA registry or whatever.

In any case, while we added some post 6.3.0 code points to the Maximal 
Starting Repertoire, I believe, we've yet to interest any of the 
regional Generation Panels in asking for any of them for the Root Zone.

A./

PS: as to whether it makes sense to issue any of the other RFCs, I'm not 
sure. ICANN seems to have woken up and is publishing recommended label 
generation rules for the more popular scripts and languages; those are 
actual "implementations" of the mitigation steps that "troublesome" was 
going to call for. If these reference LGRs end up getting some traction, 
they are going to do more than an RFC could in influencing actual 
practice. That's my thinking, at least.

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 04 Oct 2021 04:27:21 +0200

> you raise a good point.

The big difference is that RFC 8753 is made available that details the process for review of new Unicode versions.

I do not think we can mix the two reviews. The ad-hoc used up to including Unicode 12, and then the more formal laid out in RFC 8753.

It will not be hard to produce "the next document". The problem is the lack of people that are interested and have time to help.

   Patrik

>  One of the things I've been trying to get the Unicode folks to do is to add to their process so they can better flag code points where the IDNA 2008 property calculation has changed, whether from reserved to PVALID or to/from PVALID due to property changes (not that we want to encourage that, but better if it's flagged as part of Unicode BETA, than after the fact).
>
>  I'm told that the data is all there, but they need to add a step in an already crowded release process.
>
>  For now, getting this out as targeted would be best, otherwise we might never see it.

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 04 Oct 2021 04:31:29 +0200

On 4 Oct 2021, at 4:26, Asmus Freytag wrote:

> Now, my thought was, if Unicode can flag all the additions/changes in terms easily digestible from the IDNA2008 perspective, and does so in the context of its BETA process, then any glaring issue can be handled by whoever is interested in a follow up. Either try to get properties modified/changes retracted in the Unicode process via a public review comment, or issue an exception on the IETF side -- before the characters get deployed.

I have no issues producing a list during the BETA process. In fact I do, to prepare on the upcoming version of Unicode.

The big thing is not that, but the review process as laid out in RFC 8753.

Given some code points get added with derived property value PVALID, which ones should be PVALID and which ones should undergo a specific review?

That I feel is our job in IETF according to RFC 8753.

My suggestion:

1. Let this draft be published (I tried 6 months ago, but no response, nothing happened)

2. Unicode 13 and 14 we follow RFC 8753 and learn how it goes

   Patrik