Subject: [I18ndir] Late suggestion if we are looking at the last IDNA Unicode version review to be published (Re: Version 2: Directorate review of unicode11-07)

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 16:18:24 -0400

Sorry this is so late, but it didn't occur to me until the last
24 hours or so that Patriks's plan was to not produce more
summary report RFCs with new versions of Unicode after Unicode
versions 11 (or 12.0).

Especially if whatever document the IETF decides to publish in
the near future is expected to be the last one of this type
(unless changes in derived property values are encountered or
problems are discovered), I believe we should instruct the RFC
Editor to drop at least the final appendix of
draft-faltstrom-unicode11 (or draft-faltstrom-unicode12 as
appropriate), and perhaps all of the appendices, before
publication.    I think the appendices are important to have in
the I-D (and any future ones associated with discovery of
problems), but that leaving them in the RFCs is an invitation to
unnecessary confusion as well as unnecessary length.

I note that RFC 6452 does not contain an appendix listing code
points (or one listing differences, although that information is
present in the text) so perhaps Patrik has planned all along for
some or all of the appendices to be dropped before RFC
publication, but the I-D does not say that and such a plan
should have been a topic which the community got to consider
during IETF LC.

Discussion and motivation:

Appendix numbers (letters) below are for
draft-faltstrom-unicode11-08 (there are no differences in -07);
the corresponding ones for draft-faltstrom-unicode12 are left as
a rather trivial exercise.

(1) Appendix F: If we believe that an IANA copy of the derived
property table is important in spite of not being normative,
then let's be sure people go there rather than looking at an RFC
and wondering which one is authoritative.   If we want to
encourage people to look at the current IANA tables and we don't
expect regular updates or equivalents to this document, getting
the complete listing out of here is especially important because
the list in this document (as the probably-last RFC version) and
the IANA one will rapidly diverge (maybe even before this one is
published if draft-faltstrom-unicode11 represents the
publication version and Patrik updates the IANA tables forthwith
to reflect Unicode 12.0 as a recent note implied).

If we believe that registry and lookup implementations should be
able to pull a version of the IDNA tables that corresponds to
the version of Unicode they are using, then IANA should be
maintaining one table for each version of Unicode from 5.2
forward, not a single "current" version because there is
ultimately no such thing.  Conversely, if IANA is going to
maintain only "most recent", then this document needs a separate
appendix for each Unicode version starting immediately after
Unicode 6.2 and continuing through 11.0 (or 12.0), RFC 6452
should be updated to contain a table for Unicode 6.0, _and_ we
need to keep publishing update RFCs for each Unicode version
whether issues are identified or not.  My guess is we don't want
to go there, but others may disagree.

(2) Appendices A-E: The argument for retaining these into the
published RFC is stronger than for the main table represented by
Appendix F because each appendix identifies actual version to
version changes and those are not in the IANA database today (or
are at least not obviously available).  However, that is either
an argument for publishing a new RFC, with the appendices, for
each new Unicode version or it is an argument for getting these
lists of differences into the IANA database and keeping them
there.   Can't have it both ways unless we are convinced that
the comparison data are actually not important to anyone to have
in readily-available /archival form, in which case it should
still be safe to drop these appendices. 

Comment: In addition to the reasons for dropping these
appendices described above, 81 page, quarter-megabyte documents
discourage retrieval and reading.  It would be far better to
encourage people to read what is actually substantive (the body
of the documents) rather than either duplicating tables that
we've told people to find somewhere else and ones that we have
asserted are unnecessary if we decide against RFC publication of
future review reports.

Conclusion: Ask the RFC Editor to drop Appendix F, and, unless
someone has good reasons I haven't though of, Appendices A-E,
prior to publication.  Noting that the IANA Considerations
section (at least at draft-faltstrom-unicode11-08) doesn't
actually mention the appendices as a source for the tables IANA
is expected to store and that the body text of the I-D does not
appear to refer to the appendices at all, a one-line instruction
should be the only change necessary.  The absence of such
comments in the body text may be an added argument for dropping
them or, if they are not dropped, inserting text into the
document that explains what the appendices are for, and maybe
adding one or two.  Changing the document in that way would
clearly would require more work and time:-(

I would assume that Patrik and the IESG can negotiate with IANA
about version and data retention without requiring a formal
Standards Action.  If not, we should be able to quickly generate
an I-D to update the IANA Considerations section of RFC 5892 to
specify what is needed and get it processed without
draft-faltstrom-unicode11 having to wait on it.  We could also
slip such an update into draft-faltstrom-unicodeX, but that
would probably result in a small delay while the precise text
goes through obligatory nit-picking. 

best,
    john

p.s. This issue is one of the reasons why I thought it would be
helpful, perhaps critical, to understand our plan for this sort
of review going forward before we dug into the details of a
single report like draft-faltstrom-unicode11.  Had we done that,
it would have been possible to review the text of the present
document in the light of that plan (even if the plan was
tentative) and, if appropriate, to adjust the document to change
the IANA Considerations section of 5892 to reflect our
intentions.

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 13 Mar 2019 08:22:57 +0900

On 13 Mar 2019, at 5:18, John C Klensin wrote:

> Sorry this is so late, but it didn't occur to me until the last 24 hours or so that Patriks's plan was to not produce more
> summary report RFCs with new versions of Unicode after Unicode versions 11 (or 12.0).

Note that the policy I have described is from my perspective nothing new. I am of the view that I-D is only needed in the cases where incompatible changes are made.

> I note that RFC 6452 does not contain an appendix listing code points (or one listing differences, although that information is present in the text) so perhaps Patrik has planned all along for some or all of the appendices to be dropped before RFC
> should have been a topic which the community got to consider during IETF LC.

To be honest, I decided to add the code points as appendices to make it more clear what has happened (and not) between the various versions. Just like 5892 is including the list. Why I did not include it in 6452 I do not remember.

As you write below John, the tables are non-normative, and only there for explanation. So yes, F can safely be removed and possibly should not be there for reasons you explain, while the others might be more interesting.

   Patrik

> the corresponding ones for draft-faltstrom-unicode12 are left as a rather trivial exercise.
>
> (1) Appendix F: If we believe that an IANA copy of the derived property table is important in spite of not being normative, then let's be sure people go there rather than looking at an RFC and wondering which one is authoritative.   If we want to
> encourage people to look at the current IANA tables and we don't expect regular updates or equivalents to this document, getting the complete listing out of here is especially important because the list in this document (as the probably-last RFC version) and the IANA one will rapidly diverge (maybe even before this one is published if draft-faltstrom-unicode11 represents the
> publication version and Patrik updates the IANA tables forthwith to reflect Unicode 12.0 as a recent note implied).
>
> If we believe that registry and lookup implementations should be able to pull a version of the IDNA tables that corresponds to the version of Unicode they are using, then IANA should be
> maintain only "most recent", then this document needs a separate appendix for each Unicode version starting immediately after Unicode 6.2 and continuing through 11.0 (or 12.0), RFC 6452
> should be updated to contain a table for Unicode 6.0, _and_ we need to keep publishing update RFCs for each Unicode version whether issues are identified or not.  My guess is we don't want to go there, but others may disagree.
>
> (2) Appendices A-E: The argument for retaining these into the published RFC is stronger than for the main table represented by Appendix F because each appendix identifies actual version to version changes and those are not in the IANA database today (or are at least not obviously available).  However, that is either an argument for publishing a new RFC, with the appendices, for each new Unicode version or it is an argument for getting these lists of differences into the IANA database and keeping them there.   Can't have it both ways unless we are convinced that the comparison data are actually not important to anyone to have in readily-available /archival form, in which case it should still be safe to drop these appendices.
>
> appendices described above, 81 page, quarter-megabyte documents discourage retrieval and reading.  It would be far better to encourage people to read what is actually substantive (the body of the documents) rather than either duplicating tables that we've told people to find somewhere else and ones that we have asserted are unnecessary if we decide against RFC publication of future review reports.
>
> Conclusion: Ask the RFC Editor to drop Appendix F, and, unless someone has good reasons I haven't though of, Appendices A-E, prior to publication.  Noting that the IANA Considerations
> actually mention the appendices as a source for the tables IANA is expected to store and that the body text of the I-D does not appear to refer to the appendices at all, a one-line instruction should be the only change necessary.  The absence of such
> comments in the body text may be an added argument for dropping them or, if they are not dropped, inserting text into the
> document that explains what the appendices are for, and maybe adding one or two.  Changing the document in that way would
> I would assume that Patrik and the IESG can negotiate with IANA about version and data retention without requiring a formal
> Standards Action.  If not, we should be able to quickly generate an I-D to update the IANA Considerations section of RFC 5892 to specify what is needed and get it processed without
> draft-faltstrom-unicode11 having to wait on it.  We could also slip such an update into draft-faltstrom-unicodeX, but that
> would probably result in a small delay while the precise text goes through obligatory nit-picking.
>
> best,
>     john
>
> p.s. This issue is one of the reasons why I thought it would be helpful, perhaps critical, to understand our plan for this sort of review going forward before we dug into the details of a
> single report like draft-faltstrom-unicode11.  Had we done that, it would have been possible to review the text of the present document in the light of that plan (even if the plan was
> tentative) and, if appropriate, to adjust the document to change the IANA Considerations section of 5892 to reflect our
> intentions.

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 20:08:53 -0400

--On Wednesday, March 13, 2019 08:22 +0900 Patrik Fältström
<[email-masked]> wrote:

>> Sorry this is so late, but it didn't occur to me until the
>> last 24 hours or so that Patriks's plan was to not produce
>> more summary report RFCs with new versions of Unicode after
>> Unicode versions 11 (or 12.0).
> 
> Note that the policy I have described is from my perspective
> nothing new. I am of the view that I-D is only needed in the
> cases where incompatible changes are made.

Understood although I will continue to insist that you say
something more like "incompatible changes are made or other
problems are found", which is what 5892 says.   And 6452
carefully explains that it is because it was the first time,
rather than because this was expected every time.

But draft-faltstrom-unicode11 (at -07 or -08) creates a bit of a
dilemma wrt that position.   Unless I'm suffering from lack of
sleep, we have "incompatible changes" only with Unicode 11.0 and
"other problems" only with Unicode 7.0.  So, if no I-D is needed
for the other cases, it seems to me that there are several
choices:

(i) Produce separate I-Ds covering Unicode 7.0 and 11.0
respectively and either ignore everything in between or put a
sentence into the latter that says that nothing happened in
between that was worth reporting.

(ii) Produce a single I-D that covers those two versions and
that says, in prose, that none of the versions between those two
involved and issues or were reported.  I suggest that -07 may
have been closer to that approach than -08, so you have
presumably agreed to not take that option.

(iii) Stay with the structure of -08 but make it clear that, not
only does it not set a precedent for how future reviews and
decisions about them are going to be done but that it does not
set a precedent for the type of documentation that appears in
Section 4.2 being included in, or justifying, future I-Ds or
RFCs.

As I mentioned in an earlier note, absent evidence of an "other
problems" code point review, it is unclear whether specific code
points should have been called out for any of Unicode 8, 9, or
10 or, for that matter, for Unicode 11 in addition to the
changed code points identified in Section 4.3.  I don't want to
derail this train, which is one of the reasons I haven't either
tried to insist that Harald include more about the "other
problems" review or sent a note to the IETF and IESG lists
pointing out that the scope and conclusions of
draft-faltstrom-unicode11 are not compliant with the review
requirement (such as it is) for 5892, but it isn't reasonable
for us to continue sweeping that issue under the rug.

To save Asmus a note, while I'm still worried about long-term
maintenance and review of the troublesome list, it may well be
that we should conclude at this point that any future problems
with new code points are better handled by advice to registries
than by adding exceptions or otherwise modifying 5982.  But we
don't get there by sweeping that review requirement under the
rug and then ignoring whatever lumps it creates from down there
-- we do it by opening up the topic and changing/updating 5982
and, if necessary, 5984 to reflect what we have decided should
be done, getting IETF consensus on that decision.  Until then,
reviews are supposed to include _both_ a search for code points
for derived property values have changed and a pass through the
new code points to try to spot any that raise issues that have
not been considered earlier or that cause other problems.

>> I note that RFC 6452 does not contain an appendix listing
>> code points (or one listing differences, although that
>> information is present in the text) so perhaps Patrik has
>> planned all along for some or all of the appendices to be
>> dropped before RFC publication, but the I-D does not say that
>> and such a plan should have been a topic which the community
>> got to consider during IETF LC.
> 
> To be honest, I decided to add the code points as appendices
> to make it more clear what has happened (and not) between the
> various versions. Just like 5892 is including the list. Why I
> did not include it in 6452 I do not remember.
> 
> As you write below John, the tables are non-normative, and
> only there for explanation. So yes, F can safely be removed
> and possibly should not be there for reasons you explain,
> while the others might be more interesting.

I'm happy to either leave the others there or take them out, but
I think we need a bit of text to explain what we are doing and
why and to set expectations for future Unicode versions
appropriately.   My note assumes that we are going to follow
your recommendations/preference about that. but, as other notes
(by the two of us and others) point out, the directorate has yet
to make a recommendation to the IETF and the IETF has not
decided what to do.   I'm willing to agree that you have ability
to make that decision unilaterally, but, in the light of the
current text in -08 and the way in which decisions we are making
can be interpreted as precedents, I think doing it by your
making such a decision at this point would be ill-advised.

    john

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 13 Mar 2019 09:13:47 +0900

On 13 Mar 2019, at 9:08, John C Klensin wrote:

> Unless I'm suffering from lack of sleep, we have "incompatible changes" only with Unicode 11.0 and "other problems" only with Unicode 7.0.

This is why I split up the "steps" in:

1. To 7.0 (IAB statement)
2. To 10.0 (No change)
3. To 11.0 (One incompatible change)

And possibly:

4. To 12.0 (No change)

If that was confusing, my apologies.

   Patrik

---

From: Asmus Freytag <[email-masked]>
Date: Tue, 12 Mar 2019 20:00:28 -0700

> If that was confusing, my apologies.

Look, let's get this wrapped up.

I'm out of bandwidth following all the back and forth.

A./

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Mar 2019 23:11:33 -0400

(off list)

--On Wednesday, March 13, 2019 09:13 +0900 Patrik Fältström
<[email-masked]> wrote:

>> Unless I'm suffering from lack of sleep, we have
>> "incompatible changes" only with Unicode 11.0 and "other
>> problems" only with Unicode 7.0.
> 

It wasn't confusing at all.  I meant what I said, with a detail
I didn't mention: the combination of a few other things and this
(mostly this) have had me working 20+ hour days since Friday.
In addition to helping me figure out that I'm not 20 any more,
the combination of that and trying to looks at two drafts at the
same time (-07 which Harald was working from and -08 that
resolved many issues; 2 1/2 if the unicode12 piece is counted)
could easily have left me in a state in which I could have
missed something in your text or one of the many notes that has
gone by.  So I thought the precautionary comment was called for.
There are/were some things in the draft(s) that have confused
me, but you know what they were and that arrangement was not one
of them.

best,
   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 13 Mar 2019 12:14:45 +0900

On 13 Mar 2019, at 12:11, John C Klensin wrote:

> There are/were some things in the draft(s) that have confused me, but you know what they were and that arrangement was not one of them.

Thanks!

   Patrik