Subject: [I18nrp] The evolutionary future of IDNA (was: Re: Last Call: <draft-faltstrom-unicode11-05.txt> (IDNA2008 and Unicode 11.0.0) to Informational RFC)

From: John C Klensin <[email-masked]>
Date: Thu, 06 Dec 2018 11:01:08 -0500

Hi. 
Paul's note from Tuesday seems to raise some fundamental
questions about where we are taking IDNA.  They don't seem to me
to have much to do with the debate about directorates or
procedural correctness or even much to do with the argument for
tuning draft-faltstrom-unicode11 and pushing it forward at this
point unless the community concludes that advancing that
document without understanding where we are headed is a bad
idea.  I happen to believe that, but I think the key issues Paul
raises would still be key issues even if the community decides
it is ok to advance draft-faltstrom-unicode11 and then use it as
a new starting point.

So I want to make a start at reviewing some context and then
looking at those issues...

--On Tuesday, December 4, 2018 06:59 -0800 Paul Hoffman
<[email-masked]> wrote:

>...
>> First of all, this document evaluates the individual changes
>> made up  until and including Unicode 11. Sure, one could say
>> this has  implications on the IETF view of existence of
>> normalization rules (or  not) but that is not the intention
>> here. The result of this review  should neither be
>> extrapolated to future versions of Unicode nor to  future
>> evolutions of normalizations.
> 
> This is good to know. In that case, could you either remove
> the "One difference between these sequences..." sentences, or
> add a sentence in the Introduction section that says "The
> result of this review should neither be extrapolated to future
> versions of Unicode." Either action would clear up this
> confusion.

I think such a sentence in the introduction is needed even if
the "One difference' sentence is removed.
 
>>> =====
>> First of all, this document is (as it seems to me now) to be
>> Standards  Track. So that issue is taken care of.
> 
> Procedurally, it is not, I believe. A new draft needs to be
> issued, and the IETF Last Call has to start again.
> Fortunately, this IETF Last Call is only a few days old, so
> this should not delay anything much.

Noting that the Last Call has been withdrawn but other comments
have disagreed with that decision, I concur about the procedural
requirement.    

Another reason the Directorate should be able to form an opinion
and make a recommendation before the Last Call is restarted is
that, for this particular case, if Patrik wants to reference
other documents for details and explanations (which I agree is
the right thing to do), we'd better be sure that those documents
say what we want them to say and are at least reasonably stable.
There is no assurance at all of that with expired documents that
have had no meaningful discussion in the community.  Were the
Directorate to sort that out partway through the Last Call, we
would have the risk (I'd guess high odds) of having to revise
this document as well as some of those it references, meaning
that the Last Call would be about an obsolete and replaced I-D,
leaving the IESG to either try to guess at what the community
would have said about the new drafts or to repeat the Last Call.
Again.  If we are looking for things to complain about
procedurally, the IESG making a decision to advance a document
when the Last Call was about a version that was quite different
from the one being sent to the RFC Editor would be high on the
list.

So, again, Directorate first, then requests from the Directorate
to update and post relevant documents, then review by the
Directorate, then posted recommendations from the Directorate,
and only then an IETF Last Call on whatever document or
documents the Directorate recommends.

However, the important part of this note starts with Paul's
other (or perhaps main) concern.

>...
> This misses my concern. There is an active draft
> (draft-freytag-troublesome-characters) that seems to want to
> change the IANA registry. Your draft
> (draft-faltstrom-unicode11) also wants to change the registry,
> but in a different way. My question is whether we should be
> making the registry unstable in this way.

Possibly this has been adequately covered already but, if
draft-freytag-troublesome-characters is not clear about what I'm
about to say (which I think is consistent with Asmus's recent
postings) it needs to be fixed.  My understanding (as a
co-author) is that we expect to end up with two separate
registries (or one meta-registry with two or three subregistries
-- a detail that requires discussion with IANA).   Neither the
code point registry nor the proposed new one is normative in the
usual sense; the context rules registry is normative.  They
consist of:

(1) The registry of code points called for by IDNA2008
(including the Contextual Rules Registry) addressed by the
present I-D.  That registry should be stable except for
additions unless some new discovery or change in Unicode
requires reclassifying an existing code point or modifying or
writing new Contextual rules.  Changes due to the latter reasons
(i.e., other than adding new code points) should be rare and
made only after careful discussion, but are explicitly
contemplated as possible by IDNA2008.  It is perhaps worth
noting that this is, by the design of IDNA2008, an inclusion
registry: anything not explicitly permitted is forbidden.

(2) The registry of recommendations and advice for zone
administrators to consider in deciding what code points,
sequences, etc., to allow in their zones.
draft-freytag-troublesome-characters is intended to establish
and seed that registry.  It is not expected to be stable but,
instead, is expected to evolve to reflect new discoveries and
evolving knowledge.  

I hope this is clear from that draft, but there are three issues
with that draft and one more general issue that are almost
independent of its specific code point lists and tables and that
almost certainly require community discussion.  Personally, I
not only don't know the answers but am torn about the tradeoffs
I see so what follows is an attempt at a summary, not advice
about what to do about them.

(2.1) Whether having this sort of list as an official or
quasi-official IETF-provided list is a good idea at all.  As
clarified in draft-klensin-idna-rfc5891bis, IDNA2008 requires
that zone administrators [1] register only labels that contain
characters, and character sequences, drawn from scripts (and,
where relevant, associated languages) they fully understand and
which, of course, conform to the parts of IDNA2008 now under
discussion.   It has been widely observed that many or most
ICANN-Accredited Registrars and the Registries they support
ignore that rule.  The IETF can, at this point, go down either
of two paths (or can continue  to ignore the issue, which I'm
fairly sure would be irresponsible).   One is to reaffirm the
rule (in which case a global troublesome characters list may not
be needed or must be put in a different context).  Unless we
like making statements that are generally ignored, we would also
decide to use whatever formal and informal mechanisms we have to
convince ICANN (at least) to use whatever mechanisms they have
to hold registrars and registries who are blatantly ignoring
those rules accountable.    

The other is to conclude that registration of strings that the
registrar and registry may not understand or be willing to take
responsibility for is today's new normal and that providing
advice for those registrations who lack deep understanding is a
good idea (in which case draft-freytag-troublesome-characters is
extremely relevant and draft-klensin-idna-rfc5891bis should be
altered to explain the new reality and to update IDNA2008 (not
just 5891 - see discussion in the latter I-D) to adjust the
"don't register what you don't understand" rule to match the new
understanding.   If we go down that path, we probably need to
understand that those SLD registries whose label evaluation
model is essentially "we rely completely on the registrars to do
the right thing and will accept whatever they send us" and those
registrars whose model is close to "you pay for it and we
register it" (including combinations of the two that advertise
violations of the specs or good sense because they think they
can sell them) are unlikely to be affected by better guidance. 

A modification of those options would be to give advice along
the lines of "troublesome characters" to ICANN-accredited
Registrars and TLD Registries and/or Registrars and Registries
for so-called public domains but try to continue with the "don't
register what you don't understand" rule inside enterprise and
equivalent domains (most of whom, AFAICT, are mostly following
it just because the alternative makes little sense).   That
would probably require (or at least benefit from) modification
of both I-Ds.  It would also not be clear in that circumstance
whether the appropriate responsible party for the troublesome
character list and corresponding registry is the IETF, ICANN, or
someone else.  Probably unfortunately, there are candidates for
"someone else".
  
(2.2) Expanding from the above, there is a question about the
audience for any new efforts in this area, especially attempts
to give advice such as the "troublesome character" list.   If my
observations and a certain amount of logic are correct, most
administrators of domains that are used with relatively deep
structure intra-enterprise are already fairly careful about what
they register if only because reasonable rules fall out from the
way the enterprise (or equivalent organization) is organized and
from responsiveness to usability by people within that
enterprise.    If additional guidance is not needed at that
level, then the question becomes mostly about what can usefully
be done about SLDs (or, more generally, the domains at the
boundary between "public" and "private" domains).   To take an
extreme example, it seems to me that there is no chance that the
behavior of a registry that is happily making money registering
and delegating labels that are clearly invalid under the
IDAN2008 rules (with emoji as a prominent current examples), is
going to change its behavior if the IETF says that they should
only register strings that are conformant to IDNA2008 and that
they understand or if the IETF gives them a list of troublesome
characters that are PVALID under IDNA2008.   A significant
change in their behavior would almost certainly require forceful
action by ICANN or by some effective government with appropriate
jurisdiction.  There are questions whether the IETF should be
acting as an advocate for such actions.  Probably those
questions lie within the IAB's responsibility but, given
potentially far-reaching implications, IMO it would be really
unfortunate if they made such decisions without the consensus of
an informed IETF community.   This is not an easy problem.

(2.3) The IETF has traditionally been very reluctant to create
an IANA registry that requires maintenance but whose maintenance
depends entirely on a single person and his or her expertise.
We even have traditional jokes about those situations, e.g.,
about the consequences of "truck fade". IANA, too, has been
known to raise questions about the creation and management of
such registries.   At least IMO, the most recent published
version of draft-freytag-troublesome-characters does not address
the maintance issue in a definitive way.  I wouldn't expect the
Directorate to have the bandwidth and skill set (other than with
Asmus should he be appointed and want to sign up for that
maintenance role long-term and on personal title) to actively
maintain that registry.  AFAICT, the discussion during the IETF
102 BOF did not contemplate the Directorate taking on such a
role.

(3) Finally, Asmus seem to be arguing quite strongly (and
persuasively) that the contextual rules model of IDNA2008,
especially the CONTEXTO collection, is completely inadequate
and, in particular, that the rules specified in RFC 5892 do not
adequately reflect an understanding of what he calls "complex
scripts" (or don't reflect any understanding of complex script
issues at all).  Speaking as someone with considerable
responsibility for the development of that model and those
rules, he is almost certainly correct (although see below).
However, if he is, it seems to me that we need to reexamine that
part of IDNA2008 and decide whether to

(3.1) Drop the CONTEXTO category and rely on registries being
responsible [2], whether we provide more advice along the lines
of the troublesome character list or not.

(3.2) Review the CONTEXTO list and registry to be sure that the
boundary between rules in the protocol and advice is right,
modify the descriptive text as needed (that may affect other
IDNA2008 documents), and add, modify, or drop code points and
rules accordingly.

(3.3) Rethink the CONTEXTO rules and descriptions so that
important rules for complex scripts are adequately specified at
the protocol level.

It is worth noting that going very far down that path may
require reexamining the boundary between labels as "words" and
strings that may make mnemonic sense even if they make no
linguistic sense for a particular language or writing system at
all.   It was one interpretation of that boundary along with
experience with IDNA2003 (not just ignorance) that resulted in
the original decisions about what should and should not be
included in CONTEXTO.

One way or another, I don't think it is appropriate to avoid
asking whether Asmus's analysis makes a part of IDNA2008
technically defective and therefore obligates us to fix it.  If
the answer is that it is and we are, that may be the strongest
argument for not advancing draft-faltstrom-unicode11 at least
until we have the underlying issues in hand because the update
(or confirmation that no update is needed) it represents is
required to address contextual rules as well as code points.

best,
    john

[1] RFC 1591 effectively equates "registrar" with "whomever
decides what names to put in a zone", i.e., with "zone
administrator".  It is not clear whether those terms are
accepted as equivalent today or whether we should, e.g., be
using "zone administrator" to describe that function for all
zones and to reserve "registrar" for so-called public zones for
which they take money to register names.

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 6 Dec 2018 10:26:13 -0800

> required to address contextual rules as well as code points.

I come to the opposite conclusion.

Getting Patrik's draft out is urgent. Having the status of code points 
between
6.3 and 11.0 unresolved is untenable (and soon that will be 12.0 and 
13.0 and
whatever additions now come every year).

Any issues with complex scripts are not made worse by clarifying the 
status of
these new code points under existing RFC 5892.

Not updating the registry simply means implementers are using UTS#46
(option C3) to get the data.

Perennially yanking the emergency brake is really not the way to go 
about this;
it just leads to terminal paralysis and an impression that IETF is 
abandoning the
i18n issues, leaving them to other players.

A./

Moving forward, we can then focus on the remaining issues:

(1) in which form to present both guidance and expectations wrt to
       registry policies that conform to the spirit ("support only scripts
       languages that you fully understand").

       John has covered that in his previous post - there are different
       tacks we can take on this and we probably should discuss these
       options based on the drafts that he and I are coauthors on.

(2) whether it is advisable to augment the CONTEXTO registry and
        if so, how.

        Unicode has become more explicit in recent years in documenting
        complex scripts and in that process documenting that there are
        sequences that are deprecated. (They don't have a formal property
        unlike deprecated characters, but they are nevertheless listed as
         "do not use" which is equivalent to the recommendation Unicode
         makes for deprecated code points).

          These sequences could be listed using existing CONTEXTO pseudo
          code, but that, to me seems a terrible choice (on a practical 
level).

          It might be better to add a simple list of deprecated sequences.

  (3) whether it might not be useful to publish the CONTEXT information
         in machine-readable form (using the schema from RFC 7940).
         As of today, all existing context conditions can be modeled in the
         format described by RFC 7940.

  (4) whether there is a need/benefit to add CONTEXT restrictions that
         go beyond deprecated sequences and that model some generic
         behavior of code points for complex scripts.

         Tentatively, my gut-level reaction would be that this is going too
         far. While we should clearly express that all registry policies 
should
          contain such constraints suitable for the given script(s), 
this is one
         of the cases where one-size does not fit all: depending on what
         languages are supported, tighter or looser constraints are 
required.

          The existing rules on CONTEXTJ are about as far as generic rules
           can be written - in the Root Zone we simply disallowed Joiners
           so we didn't have to evaluate whether CONTEXTJ would have
           given the correct answer.

          In either case, it is clear that the implicit model in 
IDNA2008 which
          largely allows arbitrary sequences is not appropriate for all 
scripts.

  (5)    what form to give any guidance to registries. This is the question
           the "troublesome characters" draft attempts to answer. However,
           parallel to this, there is the ICANN effort to create "reference
           LGRs" that, instead of approaching the issue generically, intend
            to give specific "best practices" recommendations on the basis
           of individual scripts and languages.

            Would any activity by IETF merely be duplicative?

---

From: Jefsey <[email-masked]>
Date: Thu, 06 Dec 2018 22:56:57 +0100

John,
as a zone and DDDS  r&d manager I would:
1. favor "don't register what you don't understand"
2. and close shop as I don't understand anything 100% for sure.
My need as an Internet/IETF user would be the IETF to progressively 
cancel every RFCs; replacing them by a single maintained consistent 
internet user book. This would lead RFCs to include "IUB 
considerations" rather becoming a dumb 10,000 publications pile.
Who can read, understand and apply 10.000 RFCs of which many of them 
are obsolete but have left traces?
jfc

At 17:01 06/12/2018, John C Klensin wrote:
>_______________________________________________
>i18nRP mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/i18nrp

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 11 Dec 2018 07:55:53 +0000

Sorry to be late with this mail.

> whatever additions now come every year).

For the record, I come to virtually the same conclusion as Asmus.

While I wouldn't have used the word 'urgent' (it's kind of strange to 
call something that has been waiting around for four years 'urgent'), I 
very much agree with the sentiment.

I was greatly relieved when I saw the IAB statement at the start of this 
year, and was greatly relieved when I saw Patrik's draft for the first 
time. I saw some light at the end of the tunnel the IETF was in after 
having been frozen by a single new character.

I think it would be highly inappropriate to block Patrik's draft on any 
procedural ground. It would also be highly inappropriate to block 
Patrick's draft on the ground that there are larger technical issues 
that need to be addressed overall. Updates such as Patrik's were planned 
from the start when IDNA was developed. If there are larger issues, they 
should be addressed separately. Asmus gives some ideas about that below.

Regards,   Martin.