Subject: [I18nrp] Limits of IDNA2008 (was:Re: draft-faltstrom-unicode11-04.txt)

From: John C Klensin <[email-masked]>
Date: Wed, 10 Oct 2018 15:17:25 -0400

--On Wednesday, October 10, 2018 11:12 +0000 "Hollenbeck, Scott"
<shollenbeck=[email-masked]> wrote:

>...
>> The choices for IETF when things like this happens are:
>> 
>> 1. Keep IDNA2008 with no exceptions
>> 
>> 2. Keep IDNA2008 with exceptions
>> 
>> 3. Stop referring (directly) to Unicode as it is not stable
>> enough
>> 
>> Probably more choices than these...
>> 
>> My proposal is [1], together with a more forceful push to
>> strict IDNA2008 adoption. No IDNA2003, no UTS#46, no homebrew
>> mixes. Including that registries really do a careful
>> conservative selection of code points to be used in whatever
>> context it is to be used.
> 
> I tend to agree. It's more stable for registry operators.

Scott (and everyone else),

FWIW, I do too, but I'm a little concerned about about the
sleeping dragon (not merely a nice elephant [1]) in
this particular room.  When we designed IDNA2008 (and, to a
considerable degree IDNA2003, the JET variant model of RFC 3743,
and the "preferred syntax" rules of RFC 1034/1035) we more or
less assumed that almost everything could be handled by
character rules.  In other words, we would identify which
characters were ok, which ones were not, and which ones were
going to be treated as equivalent to which other ones.  RFC
1034/1035 also contained a few rules about positioning of
characters in labels: the "no leading digits" rule that was
later abandoned and at least a general assumption that hyphens
belonged in the middle of strings, not at either end (you will
recall that it didn't take long into the ICANN period before
registrants tested the latter).

With IDNA2008, we recognized that some rules were needed to
prevent real problems with multiple character sequences whole
label and reflected them in the CONTECTJ and CONTECTO rules, but
the standard is basically still about valid and disallowed
(invalid) characters for use in labels.  

What we didn't do was deal with a number of character-sequencing
issues that essentially would prohibit some labels even though
all of the characters (code points) in them are ok (PVALID or
conforming the the CONTEXTx rules) individually.  Most of the
issues are fundamental to the relevant writing systems, not
something that can be blamed on Unicode decisions.   We didn't
deal with them in the IDNA2008 rules and algorithm for at least
three reasons: (i) those of us who did the IDNA2008 design work
underestimated their importance and complexity and, as it turned
out, no one set us straight, (ii) we didn't know how to specify
appropriate rules, and (iii) we thought we had specified an
effective workaround.  

Well, we were wrong.  Our understanding of the effectiveness and
universality of the Unicode normalization rules was somewhere
between insufficient and just plain wrong.   We made some
assumptions about future (relative to circa version 3.2)
extensions to Unicode that were not quite adequate.  We did not
try to consider scripts that have rendering requirements that go
well beyond simply displaying a Unicode string in sequence, one
grapheme (treated atomically) at a time and the risks posed by
some systems trying to display things that way, others rendering
those strings correctly, and possible confusion between the two
groups.  We also did not consider special measures for complex
scripts in which certain sequences of characters just make no
sense (and cannot be rendered in any plausible way) even if the
equivalent sequence of code points can be formed into a string.
In addition, a great many of the discussions about IDNs in
recent years have focused on confusion among characters, and
IDNA2008 (deliberately) did not deal with that either.

It is clear (at least to me) that, if we decided we wanted to
change the IDNA2008 rules at categories to address those issues,
the solution would lie, not in an expanded exception list
(Patrik's #2) but in actually revisiting and adding to the
categories and rules themselves.  Whether that is feasible (even
if the IETF had more energy and expertise) is, to me at least,
an open question.

The IDNA2008 spec proposed to deal with that range of issues by
following and specifying a model that goes back at least to RFC
1591 and arguably earlier, a model that was reflected in the
earliest of ICANN IDN guidelines.  The model is that registries
were expected to exercise considerable responsibility to the
community and, in particular, that they not allow registration
of strings involving scripts that they didn't understand and for
which they were unwilling to be accountable [2].   That
provision may have been hopelessly naive from the beginning.
Certainly some registries have been better-behaved (by that
criterion) than others and there is a perception that those who
decided to behave well and carefully would put themselves at a
commercial disadvantage.  The possibility that it would have
provide a full employment situation for the relatively small
number of i18n experts out there may or may not be a
consideration.

However, whether that requirement for responsible,
community-serving, registry responsibility isn't up to the
pressures now being placed on IDNA or whether it was hopeless
from the beginning, it seems very clear that IDNA2008 without it
is not up to the job many people think it should do.   Even if
the only thing we should do is to adjust our expectations and be
clear about the modified versions, it seems to me that charging
ahead without reviewing those questions and whether the
standards need changing would border on irresponsible.

    john

[1] As another aspect of why true internationalization is
difficult, I note that I am contemplating a nasty-tempered,
fire-breathing, European dragon, not a kindly, luck-bringing
East Asian one :-)

[2] One of the documents in the i18n queue,
draft-klensin-idna-rfc5891bis, is about clarifying and
reinforcing that "registries must be responsible" requirement.
It is possible that a better alternative would be to face
reality and abandon it, but, if we do, it would be good to have
an alternative.

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 10 Oct 2018 18:17:32 -0700

> if the IETF had more energy and expertise) is, to me at least,

John (and all),

having deeply immersed myself for the last seven years in the problem of
drawing up conservative label generation rules for the root zone that
address all modern scripts (and languages), I feel I've reached an under-
standing that allows me to perhaps draw a few conclusions of my own.

First and foremost among them is the conclusion that the problem space
is too complex to yield to any attempts at creating a "one-size-fits-all"
approach. From different requirements of different user communities
to different behavior and internal constraints of different writing system,
a general solution will either have to be too restrictive (least common
denominator) or too permissive (allowing other, more focused solutions
to be built on it.

I believe that IDNA2008 is an example of the latter, and therefore must
be viewed as foundational; necessary to define the playing field, but not
sufficient to complete without other specifications built on top of it.

The JET specification for defining semantic variants or the RFC on Arabic
domain names are such specifications. As result of the additional research
and development work performed for the root zone LGRs, these RFCs
should probably be updated by the respective communities.

For Chinese, there's been a bit of a breakthrough in defining allocatable
code point variants in a way that restricts the number of allocatable
variant label to a reasonable number, even for long labels, avoiding
the risk of combinatorial explosion. The design should be codified in an
RFC so it can be more easily applied for other zones, if desired.

For Arabic, there's for the first time a full accounting of how to treat
*all* the orthographies, including African ones. Important conclusions
reached by the community might make for a useful updated RFC.

For the Indic and SEA languages, nobody had worked out an analysis
of what is actually required until now. (You find the current LGR drafts
from https://icann.org/idn - they make for interesting reading). The
community worked hard at creating a reasonably systematic approach
covering 10 or so scripts that share a common history and varying
degrees of a sometimes deep structural as well as surface similarity.

... (read on)

...

These community efforts have brought together a very healthy mix
of expertise from the linguistic or writing system end as well as domain
name expertise. This is not something that IETF has a business duplicating,
but perhaps, to the extent that the process resulted in some settled
conclusions, the same communities could be encouraged to capture
them also in an RFC.

Now that we have RFC7940 defining an unified mechanical representation
of label generation rules (including variants) these efforts, unlike earlier
RFC could focus on linguistic and typographic constraints and would
need to spend less effort on defining file formats or notations.

...

...

Finally, continuing with my understanding of IDNA2008 as both
foundational and (by necessity) having to be slightly permissive so as
to provide the base onto which all these efforts can be built, I want
to argue very strongly for separating all these questions:

(1) how to handle update to Unicode 11.0 (and beyond) in the existing
IDNA2008 framework

(2) how to handle any (minor) tweaking of IDNA2008 - as in clarifying
its role, setting expectations, and so on

(3) how to best capture the emerging "state of the art" in actually
designing robust label generation rulesets

and perhaps

(4) how and whether to address issues of confusing similarity; or
defining a framework for handling that issue, if that's more appropriate.

I'm very leery of creating unnecessary dependencies among all
of these. I think Patrik's got it right, that his suggested solution of (1)
is relatively cut&dried and can and should progress soonest (we are
not doing ourselves a favor by keeping everything added after Unicode
6.3.0 in some kind of limbo).

A wholesale replacement of IDNA2008 is likewise not on the table for
(2), I believe; no matter its intentions it would just muddle the waters.
That said, some tinkering in the margins, perhaps tightening some
requirements, setting expectations etc. may well be needed - John's
draft is the starting point of that discussion. In the unlikely event this
effort results in the conclusion that it is not possible to avoid some
breaking changes to IDNA2008, I'm rather of the opinion that they
would not be limited to the few issues covered in (1),  and we would
have deeper issues than the question of whether we should have
gone ahead and published (1).

(3) and (4) should build on top of IDNA2008 - as long as there are no
gravely incompatible changes introduced by (2), these efforts can
proceed at their own pace.

So, yes, my conclusion is, let's move ahead with (1) and get it out o
the way. Then tackle (2).

A./

> _______________________________________________
> i18nRP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18nrp
>