Subject: [I18ndir] Towards guidance on more restrictive LGRs

From: Asmus Freytag <[email-masked]>
Date: Mon, 25 Feb 2019 09:39:02 -0800

All,

some more detailed thoughts towards "guidance". Think of this as the 
structure behind "troublesome". It may not be necessary to maintain a 
separate database.

A./

General Principles

Unrestricted use of PVALID code points (and, in the case of some 
scripts, unrestricted permutation) leads to insecure identifiers. Users 
have difficulties recognizing rare, obsolete or out-of-context code 
points and will mentally substitute better known exemplars. They also 
have difficulties typing them. For some code points, presenting them out 
of order can result in unpredictable display. And yet other code points 
share meaning, sound value or appearance with other code points to such 
a degree that users will treat them as fully equivalent substitutes.

A more restricted set of allowed code points goes a long way towards 
making a zones identifiers more secure. Beyond that, restricting the 
context in which a code point may appear in a label helps make labels 
unambiguous. Finally, where neither technique is available, defining 
variants can prevent the registration of duplicate labels.

The set of PVALID code points can be subdivided into five categories 
which are partially overlapping. The membership of each category can be 
determined from Unicode properties or from external documentation.

RESTRICTION_RECOMMENDED:

    Code points that are not in use for an active orthography targeted
    by the zone.

    This category includes historic, archaic and otherwise limited use
    scripts, but also extensions for historic or obsolete characters,
    special use characters (such as phonetic notation) or liturgical and
    other non-secular uses.

    This category also includes most characters with the word
    "COMBINING" in their name as well as combining marks for the Arabic
    and Hebrew scripts. The latter are to some extent optional in their
    respective writing systems and their inclusion can compromise the
    uniqueness of identifiers.

    Not including these code points has little or no effect on available
    useful labels (those that users recognize and can type), but will
    eliminate the availability of unfamiliar, but confusingly similar
    code points that lend themselves mainly to malicious use.

    The "Maximal Starting Repertoire (MSR)" published by ICANN provides
    an upper bound of characters required for active orthographies in
    the 28 most widely used scripts. A tighter bound is provided by the
    "Root Zone LGR (RZ-LGR)", which further identifies code points not
    in modern use. (The RZ-LGR's coverage at this writing is 6 scripts,
    an extension to 15 scripts is imminent, and work on the remaining 13
    scripts is in progress).

    Note that neither set includes digits, so these have to be evaluated
    differently; for many scripts, native sets of digits exist but some
    may not be preferable for use in identifiers. Digits are defined by
    Unicode general category: Nd (decimal digit).

    The MSR also excludes PVALID code points that are deemed risky
    because they look like common punctuation marks (apostrophe,
    exclamation point, etc.), even though they are formally letters and
    are used in active orthographies. The reasoning is that the security
    risks outweigh the desire to fully express all possible words in
    such orthographies. The same reasoning leads to the exclusion of all
    CONTEXTO and CONTEXTJ code points.

ENUMERATED_CONTEXT:

    Several general combining marks as well as all Nuktas are used to
    extend the repertoire of letters for spelling certain sounds. For
    both cases, there is a limited set of combinations of base character
    and combining mark in actual use and recognized by the user population.

    For general combining marks, most of these combinations exist as
    precomposed characters, however a small number of accented letters
    in certain orthographies requires a combining sequence. Without
    exception the required combining marks also exist in decompositions
    of precomposed characters.

    This is by design: Unicode will encode new characters for which the
    glyph contains attached or overlaid marks, whenever such a character
    is found to be needed for some orthography. Existing combining marks
    for such attached or overlaid marks (as well as many free-standing
    marks containing letters, symbols and other features) are encoded
    for their use in notations, not orthographies.

    For characters in this category, it is recommended to enumerate the
    required combinations.

RESTRICTED_CONTEXT:

    In Neo-Brahmi and other South and South East Asian scripts, the unit
    of the writing system is the syllable, even if the encoded
    characters are only parts of such syllables. Users and rendering
    systems expect well-formed syllables; allowing defective sequences
    with characters out of position makes labels unpredictable. They may
    or may not be recognized by users and may or may not be rendered
    reliably or consistently.

    These code points therefore should be restricted to their expected
    contexts.

    The set includes all combining marks for these scripts. Consonants
    and (independent) vowels are not combining marks and do not require
    context.

    The "Root Zone LGR" published by ICANN gives examples of the kinds
    of contexts required. A given zone may be addressing a more specific
    user community, in which case context rules may be tailored; the
    Root Zone caters for all languages written in a given script for
    which a settled orthography exists and is in organized use (e.g. in
    an educational system).

    The modeling of  the native syllables doesn't need to be perfect
    from a linguistic perspective; usually some simplification of the
    rules is traded for a slight over- or under- production of
    well-formed labels.

VARIANTS_RECOMMENDED:

    To ordinary users, certain code points (or sequences) can be
    substituted for other code points (or sequences). The equivalence
    may be based on semantics, phonetics, or appearance. Examples
    include, but are not limited, traditional vs. simplified Han
    ideographs that have the same meaning, Ethiopic characters marking
    the same sound, or Arabic characters that may share the same form in
    certain positions in a label.

    There are also a number of characters that happen to be exact, or
    almost exact look-alikes of each other, so that even a careful user
    will accept a substitution. This is exacerbated by the tendency of
    modern user-interface fonts to align character designs, particularly
    across scripts.

    The majority of code point in this category exists across scripts,
    but some cases exist in the same script. More than one script has
    one or more native digits that are exact clones of one of the
    letters, for example.

    In these cases it is generally not possible to simply exclude one of
    the alternatives. Instead, they can be defined as variants, such
    that any attempt of delegating a variant label to an unrelated
    entity can be blocked. Sometimes, different user communities in the
    same script each prefer their own alternate. In that case, a proviso
    can be made to allow the same entity to allocate more than one variant.

    It is strongly recommended to limit the number of such "allocatable"
    variants to a minimum.

    Where a zone supports multiple scripts, such variants should be
    defined across scripts whenever it would be possible to create a
    label in one script that looks like a label in another script.

    The "Root Zone LGR" published by ICANN identifies code points and
    their variants for each of the covered scripts and for cross-script
    cases. If digits are included in a zone, these would need to be
    analyzed further, as some digits should be "blocked" variants of
    look-alike digit or letter code points.

    This category is not mutually exclusive to the others.

---

From: John C Klensin <[email-masked]>
Date: Mon, 25 Feb 2019 16:01:35 -0500

Asmus,

I think this is an interesting piece of work although I, at
least, am probably not going to be able to study it more
carefully and respond until we get past
draft-faltstrom-unicode11 and whatever lives (or hides) in its
penumbra.

However, before people spend too much time on this, I want to
pose a question about how useful it is likely to be in practice,
one that is closely connected to both my exchange of notes with
Scott and to some of the SSAC discussions.  

At one extreme, in my experience, registries associated with
enterprises (or most governmental or academic institutions)
allow and delegate names under rules that, in practice,
approximate "don't do it unless you understand what you are
doing".  They allocate and delegate names that are consistent
with enterprise organizational or institutional needs; an
internal registry operator who does stupid things often gets
fired or reassigned.  I've seen such subdomains try to do a lot
of things with synonyms (not just "variants" if we think that
term is bound to the appearance of graphemes or lexical
properties of labels) that may or may not be wise, but I don't
see that is a problem in our scope.

That leaves the ccTLDs and gTLDs that Scott was talking about
plus some other registries that are selling (or otherwise
supplying) names to unrelated parties.  I suggest that, in
practice, they fall into three categories:

(a) Registries that, for one reason or another, are at least as
conservative as IDNA2008 nominally expects.  That would result
from a "don't delegate strings you don't understand" rule,
requirements that restrict the number of names a given party can
hold and that link those names to business licenses or other
official proceedings, etc.  In a way, those rules are no
different from national lists of permitted names for children:
whatever one thinks of those rules and lists, they do not
require a collection of lexical rule about how names can be
constructed.  We don't hear a lot about such registries but it
is not because they don't exist; it is because they, and names
in them, don't cause problems or lead to noise.

(b) Registries who are anxious to sell whatever they find
profitable and who will push the boundaries of, or even attempt
to evade, any rules that would constrain them in that particular
way of seeking profit.  

(c) Registries who are trying to do only reasonable things, but
whose willingness (or even ability) to do such things depends on
"reasonable" being something that can be expressed in terms of
clear, and relatively uncomplicated, guidance.    

Now, my educated guess, based on experience that goes back long
before there was an ICANN, is that your categories and groupings
are going to be irrelevant to the first two groups -- the first
because they don't need them and the second because they don't
care.  For the third, I don't know but I'm guessing your list of
categories would prove unacceptably complex for many of those
domains in practice, even if you provided an algorithm or set of
regular expressions that would take a proposed label as input
and cause guidance to come out the other end.  If I'm right, I
think you are talking about a lot of work (both the first time
and in the maintenance activity with which we have had so much
trouble) with payoff for an extremely small number of zones.

And, independent of all of that, it remains unclear to me
whether such categories are an appropriate IETF problem
(especially if we are as short on resources as our handling if
I18n issues in recent years would seem to suggest) or whether it
belongs in ICANN or elsewhere.

best,
   john

--On Monday, February 25, 2019 09:39 -0800 Asmus Freytag
<[email-masked]> wrote:

> All,
> 
> some more detailed thoughts towards "guidance". Think of this
> as the structure behind "troublesome". It may not be necessary
> to maintain a separate database.
> 
> A./
> 
> 
> General Principles
> 
> Unrestricted use of PVALID code points (and, in the case of
> some scripts, unrestricted permutation) leads to insecure
> identifiers. Users have difficulties recognizing rare,
> obsolete or out-of-context code points and will mentally
> substitute better known exemplars. They also have difficulties
> typing them. For some code points, presenting them out of
> order can result in unpredictable display. And yet other code
> points share meaning, sound value or appearance with other
> code points to such a degree that users will treat them as
> fully equivalent substitutes.
> 
> A more restricted set of allowed code points goes a long way
> towards making a zones identifiers more secure. Beyond that,
> restricting the context in which a code point may appear in a
> label helps make labels unambiguous. Finally, where neither
> technique is available, defining variants can prevent the
> registration of duplicate labels.
> 
> The set of PVALID code points can be subdivided into five
> categories which are partially overlapping. The membership of
> each category can be determined from Unicode properties or
> from external documentation.
> 
> RESTRICTION_RECOMMENDED:
>

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Mon, 25 Feb 2019 14:08:57 -0800

> draft-faltstrom-unicode11 and whatever lives (or hides) in its

This is intended to flesh out the core pieces of guidance that 
eventually will have to be delivered. And intrinsically tries to answer 
Patrik's question whether this can be "property driven".

My tentative conclusion is that certain property combinations can be 
used to automatically "flag" troublesome characters other than those for 
which variants may be required. For those, there's no alternative to 
resorting to external work, be it Unicode's list of "identical by 
design" or the RZ-LGRs variant definitions. Flagged characters can then 
be dealt with by local Registry expertise, or lacking that, by cloning 
or slightly adapting some tables from the Root Zone (adaptation would be 
needed for digits, but that's not something that needs the same deep 
expertise).

I have no intention of launching this as a draft, but wanted to put it 
out here, so we can decide whether something like this should be 
developed later.

> Scott and to some of the SSAC discussions.

Not sure all of that discussion was copied, unless you are referring to 
a single pair of messages.

> doing".

Actually, any registry where requests *originate* from a single entity 
doesn't have the same issues with "malicious registrations".

Second: I doubt that any enterprise that enables complex scripts for 
regional offices/users really "knows what they are doing". My experience 
from the Root Zone does not make me optimistic about the robustness of 
solutions by IT support (even native). There seems to be quite a bit of 
learning curve.

However, they are likely to see more "well-formed" registration requests 
than an open registry. Users abiding by context restrictions since they 
are implicit in the language they use.

> They allocate and delegate names that are consistent

Variant labels stop short of full synonyms. An internal registry can 
address issues like "color" vs. "colour" but synonyms may be based on 
other internal criteria, so that would be out of scope here.

> in them, don't cause problems or lead to noise.

Thinking like a user of the Latin script.

All complex script require context rules to be robust, no matter how 
restrictive the "alphabet".

The "lets limit to the elementary school reader" restrictions on 
characters work well for almost all scripts. For CJK you need to go well 
beyond high school level . . .

Aside from that, it's nearly equivalent to what the MSR does and the 
RZ-LGR refines; except they do it for all active languages in a script.

Even well-behaved registry may want to be unbiased with a script or open 
to all users.

> to evade, any rules that would constrain them in that particular

(b) for "bad-actor" :)

I like it.

Nothing we do with RECOMMENDATIONS will affect these, other than if we 
give third parties a metric of how bad they are . . .

> clear, and relatively uncomplicated, guidance.

Most registries simply cut&paste idn tables.

It's as simple as that. Best we can do is give people some way to vet 
whether the tables they are copying are at least minimally effective.

A Devanagari table w/o context rules is not worth the disk space it 
resides on.

> care.

If the first only includes "language based" European registries plus 
Hangul-only or Japanese-only (but not Chinese) then I might buy that 
they may not need help. But already, if they start to run multiple 
language-based tables on the same zone, things can get interesting.

> For the third, I don't know but I'm guessing your list of

Well, you've just reasoned yourself into a corner. In that corner

(a) nothing we are doing in IETF makes any difference

(b) anything that's complex can't be done

If you (registry) are interested in doing a complex script, you simply 
cannot escape that complexity. However, at some point, there will be 
successful tables that you can clone.

The "guidance" I propose does not extend to creating LGRs automatically. 
It extends to letting you audit your LGRs for effectiveness.

If you are allowing a Nukta on every character in Devanagari, you'll be 
in a world of hurt. All I'm saying,

A./