Subject: [Lucid] [[email-masked]: Re: Non-normalizable diacritics - new property]

From: Andrew Sullivan <[email-masked]>
Date: Tue, 10 Mar 2015 21:33:47 -0400

Dear colleagues,

I hereby forward a message sent to a few people (including Asmus and
me) responding to the draft Asmus and I submitted yesterday.

I think this is a valuable response, though I am not sure I agree with
the approach it's taking.  Nevertheless, if we are to have any hope of
useful discussion in Dallas, it'd be good to have a look at the issues
in advance.

Best regards,

A

----- Forwarded message from Mark Davis ?️ <[email-masked]> -----

Date: Tue, 10 Mar 2015 16:53:33 +0100
From: Mark Davis ?️ <[email-masked]>
To: "Asmus Freytag (t)" <[email-masked]>
Cc: Roozbeh Pournader <[email-masked]>, Ken Whistler <[email-masked]>, Lisa
	Moore <[email-masked]>, Michel Suignard <[email-masked]>, Markus
	Scherer <[email-masked]>, Peter Constable <[email-masked]>,
	[email-masked]
Subject: Re: Non-normalizable diacritics - new property

+ Andrew

​> ​
In the meantime, the following has been released in preparation to the BOF
at the IETF meeting in Dallas.

​> ​
https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/
<https://datatracker.ietf.org/doc/draft-sullivan-lucid-prob-stmt/>

A well written report, and I think will help a great deal in resolving the
issues.

However, I'd strongly suggest a couple of changes to be more precise, and
thus avoid confusion (no pun intended). Although these may seem overly
formal, it is vital that the text be unambiguous, because the subject is so
very tricky, and it is thus so easy for people to be arguing based on
different interpretations of terms.

I will try to suggest some minimal changes, although it might be even
better with some more extensive rewording.

> I3 depends on the assumption that strings that will be used in
>   identifiers will not have any ambiguous matching to other strings.

When discussing strings, the term "ambiguous matching" is itself quite
ambiguous. I can have ambiguous matching with SJIS, for example, because
0x61 can be an 'a' or be part of another character; similarly there are
sequences of bytes ..XY.. where XY is a character or X is the end of one
character, and Y is the start of another. There are other forms of
ambiguity as well.

To eliminate this ambiguity, change every 8 other cases of "ambiguous/ity"
to prefix by "visually".

=>   identifiers will not be visually ambiguous with other strings used as
identifiers.
...

Look for other cases that could profit by that, such as like
"indistinguishable", and "matches".

See the later:

> Worse, identifiers, by their very nature, are things that must
>   provide reliable exact matches.  The whole point of an identifier is
>   that it provides a reliable way of uniquely naming the thing to be
>   identified.

In this section, to be precise, one would need to use "visual matches".
There are so, so many other ways strings can "match", like "have the same
code points".

However, the second sentence is still problematic. The characters in two
different strings that have similar appearances *are* different characters,
so the identifiers are "unique" in that sense. I guess what you meant to
say is something like:

=> it provides a reliable way of naming the thing to be identified, so as
to have a unique visual appearance.

Now, I think adding "visual" makes it clear that this paragraph needs some
work. As it stands, it would be too strong: it would call for eliminating
all homographs from all identifiers (eg disallowing "top" written in
Cyrillic characters.)

Later, there is a shift to confusability, but the text never relates that
directly to "confusability", which is what is elaborated upon.

(In general, many different phrases are (apparently) used to mean the same
thing: matching, indistinguishable, etc. While use of different terms makes
the text flow better, it has the big disadvantage that the reader never
knows whether phrase1 is meant to have a broader or narrower scope than
phrase2, or is meant to be identical. This would be a more extensive
change, however, to use more consistent defined language. Alternatively, at
the top you could say that when you use the terms "matches",
"indistinguishable", "ambiguous", etc. (you'll have to go through and find
them all) that what is meant is in terms of visual appearance.)

> (We use the term "homoglyph" strictly: code points that normally use the
same glyph when rendered.)

This needs a bit of work. First step would be:

Strings S1 and S2 are strict homoglyphs when they are different, yet
normally use the same glyph sequence when rendered.

Second would be to clarify "normally use". Does that mean "in the fonts
that most people use on most platforms"? There are often more significant
differences in serif fonts than non-serif, for example. Does that require
that the same font is used? There is huge variation among glyphs in fonts: ţ
 and ț may look the identical in font 1, and may be discernibly different
when both are in font2, but ţ in font1 may also look identical to ț in
font2. This also gets tricky because in many systems there is font
fallback. If a character is not in font1, then it might be displayed in a
fallback font 2.

Third would be to clarify "same glyph sequence". Does this mean pixel by
pixel? At all resolutions, or just some? Pixel by pixel is extremely strict.

One could try to bring in "intent", but that is very dicy. The intent is
from the Unicode encoding side may not be followed precisely by all or even
most font vendors. So it is of only theoretical value to point to intent.

The exact degree of strictness matters a great deal, because under a
sufficiently strict interpretation <ARABIC LETTER BEH (U+0628) + ARABIC
HAMZA ABOVE (U+0654)> is *not* a strict homograph for U+08A1, ARABIC LETTER
BEH WITH HAMZA ABOVE. That is, most current fonts do not display them as
pixel-for-pixel identical.

I know well that this is a continuum, but since you are focusing on strict
homoglyphs, you have to be clearer what you mean by that term; where you
draw the line. That also carries over into related terms elsewhere in the
document like "the same glyph"; does that mean in all fonts (or
some?/most?//common?), pixel for pixel or not, at all sizes or just
body-text sizes, etc.?

You might think about providing a definition for "the same glyph sequence",
then using that in the definition for homograph, and elsewhere in the text.

> Mitigation may be as simple as using a font
   designed to distinguish among different characters.

Should mention that in practice this is extremely difficult to enforce; it
only works in closed systems, those that have complete control over the
fonts used for display of the kinds of identifiers in question.

----- End forwarded message -----

-- 
Andrew Sullivan
[email-masked]

---

From: Ted Hardie <[email-masked]>
Date: Wed, 11 Mar 2015 10:58:15 -0700

Hi Andrew,

So, I'm far from an expert on this, but I have some concerns about the
proposal to use "visually" here in the way Mark proposes.  If I understand
things correctly, this problem will also occur in any system that reads a
sequence of characters aloud.  That is, there is no way to distinguish in
an audio system between U+08A1 and the combination of U+0628 and U+0654.  The
current draft talks about rendering the glyph, and I think that is closer
to correct--it doesn't matter whether the rendering is audio or graphical
for the problem to occur.  In the audio forms, the  quality of  running
text providing context vs. independent identifiers without it also seems
similar, at least to me.

I also don't believe that Mark's point about  parsing of byte sequences as
XY when X and Y or meant is quite correct; in string contexts that would be
a parsing error.  It might not be easily distinguished on the wire, but in
protocol slots, this seems to be an at best unrelated issue.  I may not be
familiar, though, with contexts in which this may occur.

regards,

Ted

On Tue, Mar 10, 2015 at 6:33 PM, Andrew Sullivan <[email-masked]>
wrote:

> _______________________________________________
> Lucid mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lucid
>

---

From: Ted Hardie <[email-masked]>
Date: Wed, 11 Mar 2015 12:15:39 -0700

On Wed, Mar 11, 2015 at 11:55 AM, Asmus Freytag (t) <[email-masked]

>  On 3/11/2015 10:58 AM, Ted Hardie wrote:
>
>  Hi Andrew,
>
> an audio system between U+08A1 and the combination of U+0628 and U+0654.
>
>
> Where do get that?
>
> U+0654 is used to represent a glottal stop, but my understanding is that
> U+08A1 was encoded partially because it represents a sound that does not
> have a glottal stop. If my understanding is correct, the distinction should
> be quite audible.
>

I asked a native Arabic speaker about the systems she used and she played
them to check.  Those systems may be non-representative, though, or simply
wrong.  Or, rather than wrong, they may be tuned to a case in which the
result isn't audible.

Are we confident that this difference is general for the other cases?  If
so, that would be a very powerful way to explain the problem--code points
which are visually indistinguishable but would render differently in
audio.  (If it is not general, we still may want be cautious about focusing
solely on visual renderings).

regards,

Ted

>   The current draft talks about rendering the glyph, and I think that is
> closer to correct--it doesn't matter whether the rendering is audio or
> graphical for the problem to occur.  In the audio forms, the  quality of
> running text providing context vs. independent identifiers without it also
> seems similar, at least to me.
>
>
> I think that the focus on visual as suggested by Mark ignores that there
> are identifiers for which the user community claims that certain visually
> distinct representations are considered "the same" by users, to the point
> that one cannot rely on users to faithfully retain which version of the
> identifier is the desired one.
>
> As the problem statement is concerned with general background, it is
> important to recognize such other levels of ambiguity exist.
>
> A./
>
>
>  I also don't believe that Mark's point about  parsing of byte sequences
> as XY when X and Y or meant is quite correct; in string contexts that would
> be a parsing error.  It might not be easily distinguished on the wire, but
> in protocol slots, this seems to be an at best unrelated issue.  I may not
> be familiar, though, with contexts in which this may occur.
>
>  regards,
>
>  Ted
>
> Lucid mailing [email-masked]://www.ietf.org/mailman/listinfo/lucid
>
>
>

---

From: Shawn Steele <[email-masked]>
Date: Wed, 11 Mar 2015 19:43:12 +0000

Re: audio vs visual rendering…  I pronounce my droid “L3-G0” as “El Three Gee Oh”.  Which leads to problems as the last character is a zero.

This document makes a hard line between “homoglyphs” and “visually similar”.  Specifically calling out to (paraphrasing) just use a font that makes characters look different.  I’d assert that there isn’t a hard line between these concepts and that just picking the right font glosses over the difficulties in the problem.

The ʻokina is a decent case where it looks a lot like another character, and often fonts may even use the same glyph, however sometimes font designers choose to make a distinction.  It’s nearly impossible to tell a developer to “use the right font”.

Additionally it continues to treat these newly noticed characters as a special case without considering the many existing problems.

I’m also confused by the document’s attention to the need for unique identifiers at the beginning, but then looks at the existing IDNA problem.  I don’t consider IDNA able to provide “secure” (meaning unconfusable) identifiers.

I’m definitely in the 4.1 camp “Just another species of Confusables”.  However I disagree with the final paragraph.

IMO the cost of any mitigation (and implied cost of having a firedrill the next time one of these happens) outweighs any possibility of reducing the confusable space.  We’re spending hundreds of hours and thousands of dollars on this topic, but I’d be really surprised if an attacker even bothered with this code point.  If they did, it’d be 0.1% of the attacks, with 90% of the attacks using far less esoteric methods.

IMO the reason to solve the problem with this character would be as part of a solution to solve confusability attacks in IDNA in general.  Yet this solution addresses only a few characters of what are probably billions of ways to make confusable attacks.

-Shawn

http://L3-G0.com

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 11 Mar 2015 16:09:41 -0400

On Wed, Mar 11, 2015 at 07:43:12PM +0000, Shawn Steele wrote:
> This document makes a hard line between “homoglyphs” and “visually similar”.

No, it does not.  It says right there in section 2.2.1:

       Any character that can be confused for
   another one can be called confusable, and confusability can be
   thought of as a spectrum with "visually similar" at one end, and
   "homoglyphs" at the other.  (We use the term "homoglyph" strictly:
   code points that normally use the same glyph when rendered.)

> The ʻokina is a decent case where it looks a lot like another character, and often fonts may even use the same glyph, however sometimes font designers choose to make a distinction.  It’s nearly imp…

It's funny that you should pick ʻokina, because you're sort of making
the point the draft is after.  In any properly-designed font, U+02BB
will be rendered as though it is a single opening curly-quote, in the
way English (not American) quotation marks were historically typeset.
But U+02BC (ʼ) is a letter, and it's actually called "APOSTROPHE", and
in any decent font will look like the single closing curly-quote,
which is what was historically used in English for the apostrophe as
well.  This is not U+0027 ('), of course.  So all three can be
distinguished in a proper font.  There's moreover an argument to be
made that U+0027 is close to U+02BC than it is to U+02BB, though of
course the context might make a difference.  (Also, of course, U+0027
has other different properties.)

None of this is like the case of, say, the precomposed e-with-acute
and the combining sequence, which should never even in principle show
a difference.  That particular case is solved by NFC, but it is
clearly different from the cases where a font should or could, in
principle, make them distinguishable.

I think we're all aware that you can't tell developers what font to
use, but there is clearly an in-principle difference between "could be
mitigated with font" and "cannot possibly be mitigated with font".
And as the draft notes, this is all a spectrum with many small
gradations.  The purpose of the discussion is to make distinctions
apparent where we can, so that we can talk sensibly about them.  So
trying to say they're all the same doesn't help make those
distinctions clear.

> Additionally it continues to treat these newly noticed characters as a special case without considering the many existing problems.

Where, please?

> I’m also confused by the document’s attention to the need for unique identifiers at the beginning, but then looks at the existing IDNA problem.  I don’t consider IDNA able to provide “secure” (mean…

IDNA is supposed to be providing unique identifiers.  It in fact does,
in the sense that when a series of U-labels or A-labels are put
together they (respectively) produce exactly one FQDN that can be
looked up.  That's why it's part of the problem.

> 
> IMO the reason to solve the problem with this character

Which character, exactly?

Best regards,

A
-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Mar 2015 17:15:27 -0400

Ted,

I won't claim to more than about 10% more expertise, but maybe I
can clarify some of the issues you raise (and with which I
generally agree)...

--On Wednesday, March 11, 2015 10:58 -0700 Ted Hardie
<[email-masked]> wrote:

> Hi Andrew,
> 
> So, I'm far from an expert on this, but I have some concerns
> about the proposal to use "visually" here in the way Mark
> proposes.  If I understand things correctly, this problem will
> also occur in any system that reads a sequence of characters
> aloud.

Or not.  To give more familiar example than the one that started
this (I'm assuming there are no first-language speakers of Fula
on this list and, if there are, that they probably don't
primarily write it in Arabic characters), text to speech
programs generally need to know the language involved and often
have language-specific dictionaries.  Anyone who speaks
competent French knows that English vowel phonemes and French
ones are different.  When one set is used to render the other
language, what first-language speakers of the second language
hear is "weird accent" at best and speech that is
incomprehensible at worst.   English is particularly bad in this
regard because it has a lot fewer vowel (and consonant, but the
vowels are more obvious) symbols than it has phonemes, while
many languages that use Latin Script "decorate" their characters
in various ways that provide much more written distinction among
those phonemes.

In written form, type style and rendering distinctions, which
are very much tied up with "visual", involve many of the same
issues because accurate rendering is often tied up with
language, rather than being intrinsic properties of the script.

Interestingly, as soon as a suggestion is made that takes us to
the "language" distinctions that are often key to both "visual"
and "spoken", it also brings up back to where the original
discussion about U+08A1 --the discussion that led to our
understanding of the extent and complexity of these problems--
started: if one knows the language contexts, the distinctions
that Unicode makes by not having some characters decompose
become obvious.  Conversely, if one it talking about the DNS or
other identifiers in which language information is inherently
unavailable or non-applicable, then statements and ideas that
depend on language just don't help.

There are a second set of distinctions that get involved here
and that do not depend on language -- a collection of combining
characters that (using a summary with which Unicode experts
might not agree) are not defined precisely enough that one can
predict how they will overlay on or combine with the base
characters.  The resulting precomposed characters (almost)never
decomponse.  This issue is discussed briefly in the forth
paragraph of Section 3.3.1 of
draft-klensin-idna-5892upd-unicode70-04 and the piece of the
Unicode Standard referenced there as "Unicod70-Overlay" (sic - I
hate it when a typo is spotted immediately after a document is
posted) so this isn't _just_ a language-distinction (or
phoneme-distinction) problem, but that is part of it.

>  That is, there is no way to distinguish in an audio
> system between U+08A1 and the combination of U+0628 and
> U+0654.

Actually, if we are willing to drag the discussion back into
into Hamza issues (one of the reasons why
draft-klensin-idna-5892upd-unicode70-04 is twice the length of
draft-sullivan-lucid-prob-stmt), there is a significant phonetic
difference.  See section 2.2.2 of the latter and a different
(and more comprehensive) version in 3.2.3 of the former and, as
needed, the Unicode explanations it references.  I prefer to
think of these distinctions as language ones rather than
phonetic ones because if they were the latter, no normal human
being would be able to tell which code point to use even though
reading (i.e., text to speech) systems would be happier.  If
both the combining sequence and the precomposed character could
reasonably appear with the same language and be treated as
distinct, there would be a big problem but, if that problem
exists, I haven't seen an example yet.

The other problem with "visual" is that it pulls us back into
subjective confusability and that is both a nightmare we would
prefer to avoid if possible and something that has been the
source of earlier claims that there is no solvable problem in
this area.

>  The current draft talks about rendering the glyph,
> and I think that is closer to correct--it doesn't matter
> whether the rendering is audio or graphical for the problem to
> occur.  In the audio forms, the  quality of  running text
> providing context vs. independent identifiers without it also

    john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 11 Mar 2015 21:58:26 +0000

Oops, I seem to have dropped the list.

From: Shawn Steele
Sent: wenSjaj, march 11, DIS 2015 tera' 14:23
To: 'Andrew Sullivan'
Subject: RE: [Lucid] [[email-masked]: Re: Non-normalizable diacritics - new property]

>   code points that normally use the same glyph when rendered.)

It says "there's a spectrum", but then "We use the term "homoglyph" strictly:..."  That seems like an attempt at a hard line, though it does say "normally"

> ʻokina

Hmm, I'd thought I'd seen it different, but maybe not, sorry if that was a bad example.  Actually, Wikipedia has a couple examples… http://en.wikipedia.org/wiki/%CA%BBOkina That’s part of my concern with saying “normally rendered the same”.  They don’t have to be, that’s the art of creating a font.

> I think we're all aware that you can't tell developers what font to use,

> but there is clearly an in-principle difference between "could be

> mitigated with font" and "cannot possibly be mitigated with font".

I suppose some characters are less likely to be able to be mitigated with a font, however I'm not sure I could easily find a font that mitigated a large set of characters.  I could perhaps suggest that font designers consider these things when adding characters or making a new font, but (as my okina choice demonstrates) I'm not sure which font(s) have what behavior.

Although I think it's fair for an RFC to indicate that some fonts can exacerbate the problem, I'm not sure that it's fair to state that parts of the problem could be "solved" by a font.  For example, sometimes the font choice may be under the control of the attacker.

> > Additionally it continues to treat these newly noticed characters as a

> > special case without considering the many existing problems.

> Where, please?

?? 2.2.2 explicitly emphasizes the Arabic Hamza Above case, though it does go on to mention that there are other characters.  However there are many confusable characters in IDNA.  Worse than just confusable glyphs perhaps are confusable concepts, as the discussion of "pronunciation" alludes.  'okina isn't confusable if you pronounce it 'okina and pronounce the quote as a left quote.  Yet it is visually confusable.

For example, this focuses on the examples illuminated by a very esoteric Arabic code point.  I’m digressing, but I never see any discussion of 拔 vs 拨 or  暖 vs 暧 (depending on font, YMMV).  However those can be very confusable to Chinese readers, particularly if the context leads one to expect one character or the other.

> IDNA is supposed to be providing unique identifiers.  It in fact does, in the

> sense that when a series of U-labels or A-labels are put together they (respectively)

> produce exactly one FQDN that can be looked up.  That's why it's part of the problem.

At what level?  Sure, they’re unique as a binary number sequence.  But clearly they aren’t visually (or even audibly) unique, UTS#46 was created for that.  Nor are they linguistically unique as any of the numerous …nyms could cause confusion.  Nor are they a unique resource identifier as multiple names could actually point to the same place.

Can they provide a unique FQDN that maps to a single server?  Sure, but they could do that if we’d just left it at “all of Unicode”.

At the machine level, yes, IDNA provides unique identifiers.

At the human level, it most certainly does not.  For it to be unique at a human level would mean that I’d have to be able to hand a written note to another human and they’d be able to key that in with 100% accuracy.  This document notes the problems with l and 1, and obviously 0 and O, let alone any number of other problems.

I created http://L3-G0.com  However despite that, my mnemonic has led me to type l3-go on more than one occasion.  It’s also led me to tell the name to others whom I’m fairly sure have then miskeyed it.

Despite all that, I believe that IDN suffices for the purposes of allowing users to find web servers & machines that they are interested in.

However IDN does not suffice to provide an unambiguous (to humans) identifier.  Addressing the class of characters being discussed hear provides no statistically meaningful reduction of the potential ambiguity of IDN, particularly by a malicious attacker.  It also provides no statistically interesting improvement in the ability for well-intentioned users to find the resource they’re looking for, as presumably the users interested in the variations discussed here will be fully capable of using the correct variation.

So I don’t know what our goals are here.  I can envision several possibilities:

A)     A machine identifier suitable for uniquely identifying resources.

a.      IMO, this is already solved.  No machine is going to confuse any of these, they’re just numbers.

b.      Alternatively machines could use actual numbers (or GUIDs, or a specific bit of ASCII, or any other of a number of ways of generating true unique identifiers).

B)     We want a human safe unique identifier.

a.      The problem described here has no meaningful impact on the scope of the problem.

b.      I’m not sure this is achievable.  It’ll be hard for sure.

c.      IMO such an identifier could not also be perfectly linguistic.

                                                    i.     We’d have to map things that could be confused together in a way that would ensure there was no ambiguity in the resulting identifier.

                                                   ii.     Eg: map Latin, Greek & Cyrillic as appropriate.

                                                  iii.     Clearly such an ID couldn’t round trip to a “pretty” human form.  Though we could ensure a pretty human form always mapped to the same ID

d.      This isn’t IDNA

C)     We want to provide “safe” browsing using IDNA.

a.      Again, the issues being discussed here have no meaningful impact on the scope of this problem

b.      That would need to consider some of the far grayer areas than are being discussed here.

c.      I think other technologies are appropriate here, eg: spam filters, blacklists, safety plugins, certificates, etc.

Clearly some people think that the class of the problem here is a real problem, and presumably that it is worth the effort to attempt to solve the perceived problem.  In other words, that solving this would provide some statistical and measurable improvement in the security of IDNA.  However, I’m not able to follow the logic.

Perhaps if the document could provide some sort of statement of the reduction of confusable characters that would help.  Eg:  “Addressing this issue will reduce 90% of the confusion of IDN labels”.

-Shawn

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 18 Mar 2015 21:40:19 -0400

> It says "there's a spectrum", but then "We use the term "homoglyph" strictly:..."  That seems like an attempt at a hard line, though it does say "normally"

As I said in an exchange off-list, the point here is to situate
"homoglyph" in a spectrum of different cases, all of which are more or
less confusable.  rn and m are plainly not homoglyphs, because if you
set them in even moderately large type with serifs you can immediately
see the difference.  Latin A and Cyrillic A are just homoglyphs: no
reasonable font even tries to differentiate them, partly because as a
matter of history they evolved from the same form and therefore it's
not surprising that they look the same now.  They're different
abstract characters because they are in different writing systems, but
they look identical.

> > ʻokina
> 
> 
> 
> Hmm, I'd thought I'd seen it different, but maybe not, sorry if that was a bad example.

It wasn't a bad example.  It was a good example, but of something
different than what you intended. :) The point is precisely that these
fine details are going to matter in this discussion, so we need to
attend to them.
 
> Although I think it's fair for an RFC to indicate that some fonts can exacerbate the problem, I'm not sure that it's fair to state that parts of the problem could be "solved" by a font.  For exampl…

I think the draft says "mitigate", not "solve".
 
> ?? 2.2.2 explicitly emphasizes the Arabic Hamza Above case, though it does go on to mention that there are other characters.

There's a _whole appendix_ that attempts to illustrate the range of
the issue, and the text goes out of its way to point the reader at
that and to avoid talking about specific characters except to explain
the history, because we're trying to attend to the general problem.

> For example, this focuses on the examples illuminated by a very esoteric Arabic code point.  I’m digressing, but I never see any discussion of 拔 vs 拨 or  暖 vs 暧 (depending on font, YMMV)

In fact there is a (different) Han character in the appendix.

> > IDNA is supposed to be providing unique identifiers.

> At what level?

At every level.  DNS names are exact-match.  Each identifier is
unique.

In the DNS, color.tld and colour.tld are not the same identifier.
Conceptually they are confusing for Canadians (and spelled wrong for
USians and English respectively), but they're different.  We want that
to be as reliable as possible, and we seem to have stumbled on a class
of cases for which we cannot make derivative-property rules.  I think
that might be a problem, which is why I wanted to get a discussion
going in order to understand.

> Can they provide a unique FQDN that maps to a single server?  Sure, but they could do that if we’d just left it at “all of Unicode”.

No, we would not have.  Are you quite sure you understand how the
matching rules work?  It was _never_ "all of Unicode", because that doesn't even catch the normalization cases.

> At the human level, it most certainly does not.

That's sort of the point, here, though.  But I think your description
of how this would work (elided) presents a false dichotomy: either
everyone can get this with 100% accuracy or it's not a unique
identifier.  By similar reasoning, if you cannot eliminate all crime
then you should have no laws.

> B)     We want a human safe unique identifier.

It seems to me that, more mildly, we could just say that we want one
as safe as possible, or whose "safety" we understand.  Remember, this
is a BoF and the point of the I-D was to get people to understand and
articulate problems.

> b.      I’m not sure this is achievable.  It’ll be hard for sure.

Yes.

> c.      IMO such an identifier could not also be perfectly linguistic.

Existing identifiers of all sorts are already not linguistic.  I
observe that "anvilwalrusden" is not, to my knowledge, a word in any
language.  Certainly "dyn" (my employer) is not -- the name is a
constant source of fun around the office, since virtually everybody
pronounces it "wrong" (i.e. correctly according to the
locally-prevailing norms of pronunciation).

>                                                    ii.     Eg: map Latin, Greek & Cyrillic as appropriate.

This might actually not be a bad idea.

> d.      This isn’t IDNA

This BoF is not only about IDNA.

> 
> Clearly some people think that the class of the problem here is a real problem, and presumably that it is worth the effort to attempt to solve the perceived problem.

I would be satisfied if, in Dallas, we came away with a clear
delineation of the problem, or even a clear idea of how we might
delineate the problem.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Shawn Steele <[email-masked]>
Date: Thu, 19 Mar 2015 02:11:56 +0000

> At every level.  DNS names are exact-match.  Each identifier is unique.

For the machine, sure.  But if you throw in font weirdness and stuff, they become non-unique (to humans, not to machines) even in ASCII.  To make them be more unique one has to impose more rules, like "use a decent font", "lowercase everything".  Etc. 

> That's sort of the point, here, though.  But I think your description of how this would work (elided) presents a false dichotomy: either everyone can get this with 100% accuracy or it's not a unique identifier.  By similar reasoning, if you cannot eliminate all crime then you should have no laws.

No, even all NFC or NFKC would be 100% unique to the machine, however IDN is stricter than that.  To humans it's impossible to be 100% unique (unless we went to only ASCII A-Z0-9 and threw out a few of those).  What I'm questioning is how unique is good enough?  By your analogy, adding a bunch of laws isn't necessarily going to stop crime.  There's a lot of debate about that, and we can see that different locales have different crime rates, some low crime and low laws, some high crime and lots of laws, and everywhere inbetween.  The challenge is to balance the results on crime with the overhead of the laws.

> It seems to me that, more mildly, we could just say that we want one as safe as possible, or whose "safety" we understand.  

I agree, with the caveat that we also balance the cost and the benefits/risk.  "As safe as possible" would argue against allowing both 0 and O or l and 1 to exist.  Almost nobody's argued that, so I don't think we want "As safe as possible".  

> Existing identifiers of all sorts are already not linguistic.  I observe that "anvilwalrusden" is not, to my knowledge, a word in any language.  

We do seem to have some desire to be linguistic.  Otherwise Sharp-S and Greek didn't need touched.  I think the opinions of the computer scientists and users in marketing departments may differ greatly here.  Many users treat domain names more as a natural linguistic thing than a mathematical identifier.  Enough to sue over names.

> This might actually not be a bad idea.

If we need more structure, that would be my preference (kinda).  We can't be perfect, but if we mapped most of the more obvious confusables together, then we'd avoid some of this problem.

> This BoF is not only about IDNA.

Which may be part of the problem.  IDNA is "permissive" and caters to linguistic representations of things.  Which IMO is hard to reconcile with a strict identifier.

-Shawn

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 18 Mar 2015 22:30:30 -0400

> For the machine, sure.  But if you throw in font weirdness and stuff, they become non-unique (to humans, not to machines) even in ASCII.  To make them be more unique one has to impose more rules, l…

Yes.  That's what we're trying to understand.
 
> No, even all NFC or NFKC would be 100% unique to the machine

This is either tautologically true, or false.  Certainly we learned
with IDNA2003 that NFKC doesn't work, because while it's good for
increasing match probability the identifiers aren't stable.  So when
they're handed around through different environments, stuff happens
that is bad.

>  What I'm questioning is how unique is good enough?

Surely that's part of what we're trying to understand?

> By your analogy, adding a bunch of laws isn't necessarily going to stop crime.

Similarity in one dimension of an analogy doesn't entail similarity in
all of them: if there were no differences, it wouldn't be an analogy.
In this case, I think you have made a syllogistic mistake of
conflating "all" and "some", but I suspect the analogy is just a
distraction.  The point is merely that you cannot get from "this won't
ever be pefect" to "therefore we shouldn't even try" without a bunch
more premises.

> I agree, with the caveat that we also balance the cost and the benefits/risk.  "As safe as possible" would argue against allowing both 0 and O or l and 1 to exist.  Almost nobody's argued that, so…

And hence, a BoF to try to understand these issues.

> 
> We do seem to have some desire to be linguistic.  Otherwise Sharp-S and Greek didn't need touched.

We have people who want identifiers that work as useful things in
their writing systems.  And sharp-s and sigma seemed to be cases where
people were quite surprised, which means that the identifers are less
useful (because if the identifier system furnishes you with surprises,
that's inconvenient).  This new set of cases is in fact another set of
lurking surprises, which is why some of us are concerned about these
cases.

> I think the opinions of the computer scientists and users in marketing departments may differ greatly here.  Many users treat domain names more as a natural linguistic thing than a mathematical identifier.  Enough to sue over names.
>

I think speculating about the anthropololical facts here is going to
lead us to grief.  Let's stick with a domain of discourse we know
well.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 18 Mar 2015 21:21:26 -0700

> they're handed around through different environments, stuff happens

As in, buggy implementations?
> Surely that's part of what we're trying to understand?

I understood the IETF concern this way:

"While we all know that human perception can be tricked, and poor 
rendering doesn't help, we are concerned about the case where careful, 
conscientious users cannot tell apart two identifiers that the protocol 
deems unique."

While this looks superficially like the normalization issue, it is not, 
but it may be (perhaps partially) something that can be addressed at the 
protocol level with reasonable cost. So why not deal with it there?

Yes, this doesn't address the rest of the human perception issues, but 
because the subset we identified doesn't really hinge on any human 
inadequacies, addressing the current issue should not interfere with 
other solutions that address these inadequacies.

> lurking surprises, which is why some of us are concerned about these
As  a btw:

I'm amazed at the near total lack of IDNs registered for the Latin 
script in the root. It seems that people like the "fall-back" nature of 
non-accented ASCII labels for anything that should be accessed 
universally (top level).

So, for that script at least, you could say that users don't like being 
surprised by a more linguistically accurate, but less universally 
accessible way of constructing identifiers.

Interesting....

A./

---

From: Shawn Steele <[email-masked]>
Date: Thu, 19 Mar 2015 04:31:23 +0000

> This is either tautologically true, or false.  Certainly we learned with IDNA2003 that NFKC doesn't work, because while it's good for increasing match probability the identifiers aren't stable.  So when they're handed around through different environments, stuff happens that is bad.

I said "to the machine".  They're "just numbers", and the NFC/NFKC rules are mathematical.  Yes, you do have to exclude unassigned code points as those don't have defined behavior, however "Assigned NFC or NFKC for defined code points" would be 100% unique to the machine.

>  The point is merely that you cannot get from "this won't ever be pefect" to "therefore we shouldn't even try" without a bunch more premises.

I'm not trying to go from "this can't be perfect" to "we shouldn't try".  I am trying to say that this is good enough.  The additional cost of trying doesn't add value.   

> I think speculating about the anthropololical facts here is going to lead us to grief.  Let's stick with a domain of discourse we know well.

IDN is sociological exercise.  If the need was purely scientific/mathematical, then we'd only need a bunch of numbers or opaque IDs.  In order to make much progress here I think we need to think about how people use them.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Mar 2015 04:07:15 -0400

--On Thursday, March 19, 2015 04:31 +0000 Shawn Steele
<[email-masked]> wrote:

>> This is either tautologically true, or false.  Certainly we
>> learned with IDNA2003 that NFKC doesn't work, because while
>> it's good for increasing match probability the identifiers
>> aren't stable.  So when they're handed around through
>> different environments, stuff happens that is bad.
> 
> I said "to the machine".  They're "just numbers", and the
> NFC/NFKC rules are mathematical.  Yes, you do have to exclude
> unassigned code points as those don't have defined behavior,
> however "Assigned NFC or NFKC for defined code points" would
> be 100% unique to the machine.

Shawn,

A few observations, although we may need to agree to disagree.

First, saying "NFC/NFKC rules" tends to obscure the issue here.
An important property of NFC/NFD is reversibility, i.e., a dual
relationship in the mathematical sense.  You may not be able to
recover whatever the original way, but you can go back and forth
between the two normalized forms without loss of information.
By contrast, NFKC (and NFKD) are potentially information-losing
and that is (although more of some sets of compatibility
equivalences than others) significant.

>...
> I'm not trying to go from "this can't be perfect" to "we
> shouldn't try".  I am trying to say that this is good enough.
> The additional cost of trying doesn't add value.   

Sorry.  I think you can argue that it doesn't add enough value
to be worth the trouble.  I disagree with that given an
appropriate definition of the problem and believe we don't agree
on that definition.   But "doesn't add value"... well, we
disagree.

>...
>> I think speculating about the anthropololical facts here is
>> going to lead us to grief.  Let's stick with a domain of
>> discourse we know well.
> 
> IDN is sociological exercise.  If the need was purely
> scientific/mathematical, then we'd only need a bunch of
> numbers or opaque IDs.  In order to make much progress here I
> think we need to think about how people use them.

At one level, I agree about "sociological exercise".  I would
normally characterize it a bit differently, but let's accept
that for the purposes of this note.  I would say much the same
thing, not just about IDNs, but about almost any human-readable
identifier, noting that, as soon as a "bunch of numbers" is
expressed as numerals in written forms, there are "sociological"
issues of number base and the set of numerals to be used (in
Unicode-land and modern writing systems bound to script for
everything but Arabic).  

However, maybe there is a different way to characterize this
issue that avoids the sociological exercise.  I continue to
accept that, at least in general, Unicode has a reasonable set
of coding principles and that, in general, a reasonable set of
decisions has been made given those principles.  I firmly
believe that a different coding system, with different
principles (or priorities among principles) would simply trade
one set of issues for another but might otherwise be equally
"good".  However, many of those decisions -- about coding, not
about characters -- do have alternatives.  For example, instead
of coding lower and upper case characters differently, one could
have dealt with case distinctions by assigning a single code
point to the case-insensitive abstract character and then using
a qualifier to designate case (either always for one case or
when anyone cared for both -- two different coding style
distinctions).  Similarly, ZWJ and ZWNJ are, in some sense,
indicators and artifacts of coding decisions, not "characters":
one could have avoiding them by assigning separate code points
to the characters they affect on rendering (note that there have
been passionate arguments that just that should have been done
with some Indic scripts).  

In a way, decisions about script boundaries are, themselves,
coding decisions rather than inherent character properties but,
having been vaguely involved with an attempt to define a
universal character set that did not depend on such boundaries,
that path seems to lead to madness.  Unicode still could have
defined its script boundaries differently, e.g., seeking a
higher level of integration (or "unification") across the board
but, while I think understand that as a choice is helpful,
pursuing it very far probably is not.

Modulo the script boundary issue, it is (approximately) possible
to see all of the "confusion" and "too-similar appearance"
problems as human perception issues involving the
recipient/viewer (not really "sociological", but the distinction
may not be important).   Seen that way, the issue here is not
about the viewer perception issue but about the coding one --
resolving differences in decisions about how something might
have been code (again and for convenience, within a script).  If
one goes back a few hundred years, the question might become,
not whether someone viewing the character would "see" the
difference but whether the calligrapher or perhaps even someone
trying to set a string in cold type would see an important
difference -- also, in a way, a coding decision problem.

Coming back to the issue that started this, had Unicode
(followed by the IETF) not deprecated embedded code points to
identify languages, U+08A1 could have been coded (using an odd
notation whose intent should be obvious) as 
    <lang:fula>U+0628 U+0654</lang>
without loss of any phonetic or semantic information and the
justification for adding the code point at all would disappear
(at least absent what we generically describe as "politics", in
which case the justification for its not decomposing would
disappear).

Similarly, a different CCS could have avoided at least the
portion of the "Mathematical" collection that are ultimately
Latin or Greek letters in special fonts by different coding
conventions that would use the base character plus qualifiers
for usage and/or type style.

Another example lies in the collection of combining characters
that can be used to for precomposed character that don't
decompose, not for, e.g., phonetic reasons but because the
definitions of those combining characters don't contain quite
enough information (see Section 3.3.2.3 of
draft-klensin-idna-5892upd-unicode70-04.txt and its citations).
At least in theory, Unicode could have chosen to assign code
points to more precisely-defined (as to how they affect base
characters) combining characters.  A coding system with
different principles might have used position and/or size
indicator coding with similar effect (an approach that will
probably be needed if Unicode ever takes on, e.g., Classic Mayan
script).

Now, from that perspective, this issue is about smoothing over
(by either some form of equivalence rules or exclusion (or
non-inclusion)) differences among character code sequences that
are the result of coding decisions (decisions that are at least
semi-arbitrary because others could have been made with no
"important" information loss -- and, yes, "important" is
sometimes debatable without yet other coding decisions).  For
sequences that compose and decompose symmetrically, NFC (or NFD)
normalization does the necessary job.  IDNA2008 disallows those
mathematical characters as a way to do a different part of the
job without making non-reversible compatibility equivalences
part of the standard.

As another coding decision matter, all of this would be
significantly easier were Unicode consistent about its coding
distinctions.   Such consistency is likely impossible, at least
given other decisions, but that doesn't mean it wouldn't be
helpful.   However, we have, instead, a combining sequence
preference (with exceptions) for Latin but a precomposed
character preference for Arabic.  We have all precombined Latin
characters where combining sequences decomposing, except for
some combining characters.  Most European scripts code the
abstract graphics in grapheme clusters but East and South Asian
ones use indicators like ZWJ and ZWNJ.  There is a strict rule
against assigning separate code points to typestyle distinctions
but an exception for some usage contexts such as mathematics and
phonetic description.  Unicode does not have indicator codes or
separate code points for layout or presentation (leaving that to
external markup), but such coding has proved necessary for
writing systems that are primarily right-to-left and in such
cases as non-breaking space.  There are no language-dependent or
pronunciation-dependent coding distinctions (see Section 2 of
draft-klensin-idna-5892upd-unicode70-04.txt and/or Chapter 2 of
Uniocde 7.0) except where there are. 

Again, I don't think any of those decisions are "wrong".  But
they are all problematic for the IETF's language-insensitive,
fairly context-free, identifier comparison purposes.  And they
are, at least IMO, worth some effort because (again, independent
of discussions about "confusion"), at least,

(i) We have already established the precedent of dealing with
all of the important groups of coding artifacts we knew about
when IDNA2008 was under consideration by adopting normalization
rules, DISALLOWing a lot of characters, and even developing
special context-dependent rules for some of them.

(ii) When different input methods, using data entry devices that
are indistinguishable to the user (e.g., the alphabetic key
layouts on a German keyboard for Windows, Linux, and the Mac are
the same) and will produce different output (stored) strings for
the same input, we are dealing with coding artifacts, not
"visual confusion".   Whether the difference in internal coding
is the decision of one system to prefer NFC and that of another
to prefer NFD or the result of one typist using an "ö" key and
another deciding to type an "o" and a "dead key" umlaut, we have
(and IMO, should have) comparison systems that eliminate those
coding differences.    This is, from that point of view, just a
new set of coding decision differences that neither Unicode nor
we arranged to compensate for earlier.

     john

One way to look at the issue involved here

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Mar 2015 04:47:58 -0400

--On Wednesday, March 18, 2015 21:21 -0700 Asmus Freytag
<[email-masked]> wrote:

> As  a btw:
> 
> I'm amazed at the near total lack of IDNs registered for the
> Latin script in the root. It seems that people like the
> "fall-back" nature of non-accented ASCII labels for anything
> that should be accessed universally (top level).
> 
> So, for that script at least, you could say that users don't
> like being surprised by a more linguistically accurate, but
> less universally accessible way of constructing identifiers.
> 
> Interesting....

Or it is a policy artifact.

Suppose I have a language (not, not just script) with
well-defined and often-used conventions for representing
characters in, e.g., a simplified or less decorated form.  I'm
almost certainly going to want that form in the DNS (not just
the root, but at other levels as well).  That is especially
likely to be true if people have been using those simplified
forms in the DNS for 20 or 25 years (as is the case for most
European use of Latin Script) so that "often-used" including
users getting used to those forms in Internet contexts.

Now we introduce IDNs, making "linguistically accurate" forms
possible.  In at least some domains, those IDN forms are
registered alongside the simplified (for Latin script, ASCII)
ones, either as privileged "variants" or because registration of
both forms separately is cheap, certain, and efficient.

Then ICANN come along with IDN rules for the root.   For, e.g.,
Chinese, delegation of both the Simplified and more decorated
Traditional form is straightforward and the marginal cost of
doing so (in dollars and/or aggravation) is almost zero.  For
Latin script, a decision was made (at least at one point) to ban
variants in the root, so the only way to get both the simplified
(i.e., ASCII) form and the decorated (and more correct) one
costs USD 168K in fees, plus whatever it costs to prepare the
application, plus an aggravating review process, plus the
possibility that some committee, process, or bureaucrat will
decide that the decorated form is confusingly similar to the
simplified one and reject the application causing that (probably
USD 200K plus) investment to disappear with no benefits to the
applicant.

The situation might be quite accidentally reinforced by the
observation that acronyms and abbreviations are much more common
and accepted (and likely to end up in the DNS) in many languages
that use Latin script than with some other languages.
Especially for the large subset of those languages for which
typical strings are mostly ASCII with a few "decorated" letters,
those abbreviations or acronyms are reasonably likely to be
ASCII in "linguistically accurate" form.

Under those conditions, would you expect a lot of Latin script,
"linguistically accurate", registrations?

It would be an interesting experiment to ask Latin script
registrants whose own languages make significant use of
non-ASCII characters or those whose chosen labels are an ACII
adaptation whether, if they were offered the option of
registering a "linguistically accurate" form at little or no
charge (and after a simple application) whether or not they
would do so.   Absent that experiment or something equivalent, I
think it is hard to infer anything reliably from your
observation about the number of such strings now registered.

best,
    john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 19 Mar 2015 10:20:44 +0000

> An important property of NFC/NFD is reversibility, i.e., a dual relationship in the mathematical sense....

I was trying to make an example.

Presuming I created a system where I mapped A-C to "1" and D-G to "2" and F-P to "3" and S-Z to "4", then I could still have a system that provided machine readable unique identifiers.  There's be more than one way to write them, but the mechanism is completely unambiguous.  Later if I decided to map !@#$%^ to "5" and &*()_+ to "6", I'd have more identifiers, but they'd still be unique to the machine.

We have a function where f(input) = X.  Every input will repeatedly and perfectly result in the same X.  The set of values that X may contain may change over time as more inputs are permitted, however every input will continue to be transformed by this function into the same X.  (Except for the change from IDNA 2003 to IDNA 2008, which broke this stability, but I'm hoping we don't do that again).

So we already have mathematically unambiguous identifiers - within the system that the machine uses.

> Sorry.  I think you can argue that it doesn't add enough value to be worth the trouble.  I disagree with that given an appropriate definition of the problem and believe we don't agree
> on that definition.   But "doesn't add value"... well, we disagree.

Disagreeing is OK :)

Mathematically, we have unique identifiers.  The problem is created when we introduce the human.

> However, many of those decisions -- about coding, not about characters -- do have alternatives.

Yup, many of those were attempted in code pages.  Often with side effects that were not very nice in hindsight.

>  "Unicode could have been designed differently" (paraphrasing).

I strongly suggest that if you are passionate about how characters are encoded, and feel that you could help encode new characters better, that you participate in Unicode.  I don't pretend to know everything about Unicode, but I fear you're seriously oversimplifying the complexity of the problem.  There are reasons, some good, some bad, for why stuff is the way it is Unicode.  I don't think that the IETF should be in the business of fixing or second guessing Unicode.  If people want a better Unicode, then participate.

> Again, I don't think any of those decisions are "wrong".  But they are all problematic for the IETF's language-insensitive, fairly context-free, identifier comparison purposes.  And they are, at least IMO, worth some effort because (again, independent of discussions about "confusion"), at least,

It's only problematic once you ask a human to interpret/transcribe/read the identifier.  There's no ambiguity at the machine level.

> (ii) When different input methods, using data entry devices that are indistinguishable to the user (e.g., the alphabetic key layouts on a German keyboard for Windows, Linux, and the Mac are the same) and will produce different output (stored) strings for the same input, we are dealing with coding artifacts, not
"visual confusion".   Whether the difference in internal coding
is the decision of one system to prefer NFC and that of another to prefer NFD or the result of one typist using an "ö" key and another deciding to type an "o" and a "dead key" umlaut, we have (and IMO, should have) comparison systems that eliminate those
coding differences.    This is, from that point of view, just a
new set of coding decision differences that neither Unicode nor we arranged to compensate for earlier.

That's handled in IDNA by using NFKC.  I haven't heard that there is any problem with using INDA for identifiers due to using ö or o + umlaut to represent German.

AFAIK if the human enters the correct characters using their keyboard in a normal fashion, using the correct script for their language, and barring any transcription errors, typos or homograph confusion, IDNA will consistently point to the same record regardless of the input method.  I don't recall any examples where use of an appropriate input method/keyboard would lead to ambiguous identifiers.  

-Shawn

---

From: Shawn Steele <[email-masked]>
Date: Thu, 19 Mar 2015 10:26:32 +0000

I agree with John on this one... there's a lot of momentum behind Latin, IDN is really new, and there are still systems that don't handle it (so I "need" to have a pure ASCII name anyway).  Businesses could register alternates, but I'm always amazed at the simple ASCII-only variations that they don't bother to register/protect/claim.

I think it'll take the non-Latin scripts really taking off to kick Latin into gear.  The other scripts have a much bigger incentive to make sure their systems work with IDN.  They'll drag the Latin crowd along, but it's going to take a bit.

-Shawn

-----Original Message-----
From: Lucid [mailto:[email-masked]] On Behalf Of John C Klensin
Sent: Thursday, March 19, 2015 1:48 AM
To: Asmus Freytag; [email-masked]
Subject: Re: [Lucid] FW: [[email-masked]: Re: Non-normalizable diacritics - new property]

--On Wednesday, March 18, 2015 21:21 -0700 Asmus Freytag <[email-masked]> wrote:

> script in the root. It seems that people like the "fall-back" nature 
> of non-accented ASCII labels for anything that should be accessed 
> universally (top level).
> 
> So, for that script at least, you could say that users don't like 
> being surprised by a more linguistically accurate, but less 
> universally accessible way of constructing identifiers.
> 
> Interesting....

Suppose I have a language (not, not just script) with well-defined and often-used conventions for representing characters in, e.g., a simplified or less decorated form.  I'm almost certainly going to want that form in the DNS (not just the root, but at other levels as well).  That is especially likely to be true if people have been using those simplified forms in the DNS for 20 or 25 years (as is the case for most European use of Latin Script) so that "often-used" including users getting used to those forms in Internet contexts.

Now we introduce IDNs, making "linguistically accurate" forms possible.  In at least some domains, those IDN forms are registered alongside the simplified (for Latin script, ASCII) ones, either as privileged "variants" or because registration of both forms separately is cheap, certain, and efficient.

Then ICANN come along with IDN rules for the root.   For, e.g.,
Chinese, delegation of both the Simplified and more decorated Traditional form is straightforward and the marginal cost of doing so (in dollars and/or aggravation) is almost zero.  For Latin script, a decision was made (at least at one point) to ban variants in the root, so the only way to get both the simplified (i.e., ASCII) form and the decorated (and more correct) one costs USD 168K in fees, plus whatever it costs to prepare the application, plus an aggravating review process, plus the possibility that some committee, process, or bureaucrat will decide that the decorated form is confusingly similar to the simplified one and reject the application causing that (probably USD 200K plus) investment to disappear with no benefits to the applicant.

The situation might be quite accidentally reinforced by the observation that acronyms and abbreviations are much more common and accepted (and likely to end up in the DNS) in many languages that use Latin script than with some other languages.
Especially for the large subset of those languages for which typical strings are mostly ASCII with a few "decorated" letters, those abbreviations or acronyms are reasonably likely to be ASCII in "linguistically accurate" form.

Under those conditions, would you expect a lot of Latin script, "linguistically accurate", registrations?

It would be an interesting experiment to ask Latin script registrants whose own languages make significant use of non-ASCII characters or those whose chosen labels are an ACII adaptation whether, if they were offered the option of registering a "linguistically accurate" form at little or no charge (and after a simple application) whether or not they
would do so.   Absent that experiment or something equivalent, I
think it is hard to infer anything reliably from your observation about the number of such strings now registered.

best,
    john

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 19 Mar 2015 10:50:40 -0700

> equivalences than others) significant.

Correct about the information loss, but, like case folding (which also loses
information that is significant to the human), NFKC does pick a unique
element out of each equivalence set, satisfying Shawn's statement about
"unique to the machine".

Any of these forms, if they can be guaranteed to be stable, will satisfy
that condition, but the point about the discussion is that identifiers
that are "reasonably mnemonic" as one could characterize IDNs, do
occupy a space between machine and human interaction with writing systems.
> Unicode-land and modern writing systems bound to script for

Actually, Arabic even has potential "variants" among the digits, so while
you can make the identifier be based on the numeric value of what the
human wrote in whatever writing system, you get into issues of recognition
of identifiers etc. once they are communicated in their human-readable
form.
> issue that avoids the sociological exercise.

It's not just the conventions of a group of users, but also human perception
issues that plague human-readable identifiers, and the "sociological" issue
of the need to counteract fraud.

>   I continue to
> "good".

Definitely agree with this take.

> However, many of those decisions -- about coding, not

Also definitely agree. The task of a universal character set is to make
available a set of what Unicode calls "code elements" that are sufficiently
fine-grained and have the correct properties so that you can use them to
represent all "text elements" needed in writing.

There is certainly some latitude in that analysis of the problem so that the
outcome could, in principle, be a different set of code elements.

Usually, competing solutions are not equal in their degree of efficiency or
usability, but, perhaps unfortunately, the metrics for that is not a single
scale: each type of operation on text, from input, to storage, search, sort
and finally output, could potentially benefit from a different way of
analyzing the fundamental text elements. Creating and processing identifiers
is, not surprisingly, no different.
> coding decisions rather than inherent character properties

Correct. You see that when Unicode is forced to duplicate letters because
they were borrowed from one script into another. (Like Latin Q and W
were borrowed into Cyrillic when it was used to write Kurdish)

> but,

In some sense one has to accept that writing is a historical process that
accumulates design decisions over time, and very conservatively sticks
to them for long time periods.

Not surprisingly, character encoding, though much younger than writing,
mimics that to some degree. For better or for worse, Unicode largely
inherited the analysis of many writing systems into code elements as
it had been present in existing precursors. And, yes, collectively, these
did include the duality of combining sequences and precomposed forms,
for example.

The value added proposition of a single, universal set was such that it
resulted in the critical mass needed to lead to first adoption and then
rapid replacement of the preceding encodings -- a process that's still
in the tail end of the curve.

While most likely many choices made by Unicode (where there was a
choice to be made that was not constrained) are not "perfect", it's
highly doubtful that they are collectively so bad that any alternative
would represent enough value added to offset the cost and disruption
of a wholesale replacement. Certainly not for some decades.

(I won't say never, because the same as languages go through wrenching
orthography reforms, nothing is truly static when it comes to writing
systems and the technology to implement them).

> which case the justification for its not decomposing would

It's not just that a language unaccountably prefers a different sequence,
but that the Hamza normally is used to indicate a glottal stop. It's a
separate letter, even though, because it graphically floats, Unicode
represents it as a combining character.  In the case of the Fula language,
the same shape is used as a decoration on a letter, making a new,
single entity, that just happens to look like a glottal stop was placed
after a beh.

There's no constraint, when it comes to Fula, for the writers and font
designers (over long periods) to maintain the precise shape of the
decoration on that letter. While they started out borrowing the shape
of a hamza, who knows where this letter shape will end up -- it's not
constrained, I would argue, because the decoration isn't a hamza
(glottal stop) any longer.

This, in addition to sorting, is one of the arguments for duplicating
letters across scripts, in case of borrowings. While this hasn't happened
on a large scale, glyph shapes and font designs definitely drift over
time and it's best to allow these borrowed letters to drift with the
new script they are now embedded in, and not link them to the
old script that they originated from.

While it's possible to convey all of these distinctions by invisible
codes (markup) they are not necessarily the most robust choice:
invisible codes and markup of any kind have a way of getting
separated from the letters they are supposed to affect.
> job without making non-reversible compatibility equivalences

Unfortunately, IDNA2008 lacks a tool that would make many of these issues
easy to manage: there is a subset of the problem where neither normalization
nor non-inclusion is the most adequate way of handling the issue.

Normalization presupposes that in each equivalence set, one an pick one
canonical element without prejudice. Where two different traditions of
writing each use a different element in the set, any canonical choice
in favor of one represents a prejudice against the other.

The same problem plagues non-inclusion. One would have to pick
favorites.

Outside the protocol, other means of addressing the problem exist, such
as mutual exclusion with first-come, first-serve. This requires a registry
with appropriate policies (resulting in a definition of a blocked variant).

The touchstone case for the current discussion, possibly falls into this
category, except that it seems that the use of the glottal stop in Arabic
in general is seen as less essential for identifier purposes (for example
the root zone would not include it).

Luckily, many of the other cases that have been identified following
the publication of the original paper, would be addressable by 
non-inclusion.
In those cases, Unicode has an operational principle of always encoding
any combinations needed for orthographic (as opposed to technical)
use, so that non-inclusion of certain combining marks would be feasible.
> character preference for Arabic.

It's actually more complex. There are systematic exceptions in Latin, based
on the nature of the mark.

The way Arabic ended up is perhaps more historical accident than perfect
design; few would argue that. But we are stuck with it, just as we are
stuck with other historical accidents.
>   We have all precombined Latin

An important issue is to remember that code elements aren't text elements,
they are merely guaranteed to be able to represent all needed text elements.

And, while the basic goal is to cover text elements needed to make the text
"readable" (for content) as opposed to final form (with style), some cases
exists where the borderline between form and content is drawn differently,
and in some cases, in order to process a text, more information about 
content
has to be supplied, such as using a no-break space between words that form
some sort of compound (e.g. title and name) without being written together.

Incidentally, not all of the invisible (format or control) characters 
have been
accepted by the user community (LS, PS, or NEL anyone?). But where they
have been, it's a sign that they were addressing a real need in a way that
added value.
> we arranged to compensate for earlier.

I think the process will bump up against he fundamental limitations that
I pointed out earlier, but, as long as IETF isn't trying to pretend 
those don't
exists, searching for a solution for the tractable cases (and there are 
many)
seems worthwhile to me.

I just don't think it adds value to get hung up over any intractable 
edge cases,
because there are whole classes of problems that are also not tractable on
the protocol level, and, perforce, these have to be handled outside of it.

That's where the recognition of the human perception issue as a big part
of the overall issue with identifiers is a necessary corrective to any 
attempts
to use only the tools available in the protocol.

A./

---

From: Shawn Steele <[email-masked]>
Date: Thu, 19 Mar 2015 18:29:06 +0000

> Any of these forms, if they can be guaranteed to be stable, will satisfy that condition, but the point about the discussion is that identifiers that are "reasonably mnemonic" as one could characterize IDNs, do occupy a space between machine and human interaction with writing systems.

This discussion seems to be about lots of things, but "reasonably mnemonic" would limit the problem hugely.  That implies that I can see it and remember what it was to type it later.  However that scenario doesn't hit any of the security concerns being raised as I'm going to type it using the appropriate keyboard for my language.  Sure, if the domain had a Cyrillic l may have trouble, however that's not mnemonic, that's someone being sneaky.  (See fraud below)

> Actually, Arabic even has potential "variants" among the digits, so while you can make the identifier be based on the numeric value of what the human wrote in whatever writing system, you get into issues of recognition of identifiers etc. once they are communicated in their human-readable form.

That's digressing from my point.  I was trying to say that, if we consider the problem without any human factors, we already have unique IDs.  There's no way that machines confuse any of these names, it's only the humans that do that.  (Yes, they're an important part). 

> It's not just the conventions of a group of users, but also human perception issues that plague human-readable identifiers, and the "sociological" issue of the need to counteract fraud.

That's an additional requirement to the "reasonably mnemonic" above.  However if this entire exercise is intended to counteract fraud, then we get my view that "trying to address these codepoints is way more costly than its benefit".  

The people in this room aren't going to click on "1stbank.trustme.com", but everyone else on the planet (or at least 90% of them) click that link.  So worrying about someone spelling it "lstbank.com" provides close to zero value.  At least in the right font 50% of the other humans might start suspecting that one.

Heck REAL BUSINESSES send me to crappy URLs like that all the time.  "We partnered with ABCsurvey.com to get your feedback, trust us".  "We're replying about your recent experience from [email-masked]".  "We see you like our museum, consider donating at fundraising.com/ourmuseum".  Even my BANK does that.  Heck, they even send me the "1stbank.trustme.com" links.  It irritates me to no end. 

(This total obliviousness to security/spoofing/phishing isn't even limited to DNS, I've received calls because of billing glitches and the bank says "you can pay now, give us your CC or check routing info".  That's normal.  They they ask my secret code or whatever to verify it's me, and I point out that I can't even verify that they're who they say they are and they're stumped.  Well, you can call us back at 1-800-123-4567.  Seriously!?!?!  Even my bank has no clue what security is?)

Fix that, and then we can talk about whether it's worth fixing a couple esoteric code points.  There are FAR easier exploits for any serious attacker to attack.  It's certainly not worth risking breaking real strings and real mnemonics that real users might want to use.  (IMO its not even worth prohibiting I♥NY.com)

At this point I'm well beyond the "identifying the problem statement", sorry.

However, that does bring me back to the "what are we trying to do this for?" question.  IMO there is no value in trying to secure DNS by prohibiting these.  However I'm open to the idea that there's an machine/computer science need for unique identifiers that apparently we want to be mnemonic.  I have no clue what those scenarios are.  I get wanting an IRI to identify a unique resource, but we already have that (presuming the machine does the interpretation there's no ambiguity).  I am clearly totally missing whatever scenario it is where a confusable type ID is needed for an identifier in a system where it could actually be confused.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Mar 2015 15:29:30 -0400

--On Thursday, March 19, 2015 18:29 +0000 Shawn Steele
<[email-masked]> wrote:

>> Any of these forms, if they can be guaranteed to be stable,
>> will satisfy that condition, but the point about the
>> discussion is that identifiers that are "reasonably mnemonic"
>> as one could characterize IDNs, do occupy a space between
>> machine and human interaction with writing systems.
> 
> This discussion seems to be about lots of things, but
> "reasonably mnemonic" would limit the problem hugely.  That
> implies that I can see it and remember what it was to type it
> later.  However that scenario doesn't hit any of the security
> concerns being raised as I'm going to type it using the
> appropriate keyboard for my language.  Sure, if the domain had
> a Cyrillic l may have trouble, however that's not mnemonic,
> that's someone being sneaky.  (See fraud below)

Shawn, for better or worse, "reasonably mnemonic" might be an
attractive way to think about things, but it is never been a
requirement the community has chosen to impose on the DNS (even
though ICANN has chosen to impose variations on it on domains it
controls).  See RFC 2181 for some rather strong statements to
the contrary.   "No mnemonic requirement" is actually important
outside the IDN space because there are a number of applications
that use strings  as owner names that, on inspection, we might
guess were hashes.  Lack of willingness to impose such a
restriction is also the main reasons why IDNA has never
contained a "no mixed scripts" rule (there is also a technical
reason, but we could have gotten around that one).

>> Actually, Arabic even has potential "variants" among the
>> digits, so while you can make the identifier be based on the
>> numeric value of what the human wrote in whatever writing
>> system, you get into issues of recognition of identifiers
>> etc. once they are communicated in their human-readable form.
> 
> That's digressing from my point.  I was trying to say that, if
> we consider the problem without any human factors, we already
> have unique IDs. 

We also, as you point out, have no justification for IDNs.  The
community has, for better or worse, thoroughly rejected that
position, so I hope we can move on.

> There's no way that machines confuse any of
> these names, it's only the humans that do that.

Quite the contrary.  If differences among coding sequences for
the same string, depending on human-invisible distinctions such
as choice of IME or host, can produce different results and then
a comparison procedure is used, then, in practical terms, it is
the machines that are confused.  That is precisely why we
normalize before comparing and one of the key reasons why
IDNA2008 DISALLOWs a lot of code points that might be
problematic for machine comparison purposes.   By contrast,
case-folding and DISALLOWing some other characters are about the
potential for human confusion because machines know perfectly
well that upper and lower case characters (no matter how coded
as long as they are coded distinctly) are different.   Probably
part of the present problem is that we may need to be a little
more precise about those distinctions than we have been in the
past but, while that would change our vocabulary, it would not
change the problem.

Some of the above might help with fraud detection or prevention,
but the requirements exist without any consideration of fraud.

Please read and try to understand the distinction I tried to
make between coding artifacts and character confusion in my
previous response to you and try to work your way through the
discussion of cases in draft-klensin-idna-5892upd-unicode70-04
-- they really are important to this.

>...
> That's an additional requirement to the "reasonably mnemonic"
> above.  However if this entire exercise is intended to
> counteract fraud, then we get my view that "trying to address
> these codepoints is way more costly than its benefit".  

It is not entirely or even largely to counteract fraud.  Things
like ICANN's Label Generation Rules might well be defined in
terms of fraud or at least confusion.  I encourage you to read
about them and comment.  But, these particular issues, as
described in draft-sullivan-lucid-prob-stmt and
draft-klensin-idna-5892upd-unicode70 (-04 and anything later),
are really about resolving different coding for the same
character, and not confusion or fraud except incidentally.
There is a big issue in that about what "same character" means,
but that still isn't about fraud or even about confusion between
pairs of characters that almost everyone agrees are different.

> The people in this room aren't going to click on
> "1stbank.trustme.com", but everyone else on the planet (or at
> least 90% of them) click that link.  So worrying about someone
> spelling it "lstbank.com" provides close to zero value.  At
> least in the right font 50% of the other humans might start
> suspecting that one.

And that, while interesting, seems to me completely irrelevant
to the current discussion.   If you want to make comments on the
fraud case, please explain how the "humans in this room" are
expected to distinguish between "nür" (with the middle
character coded as U+00FC) and "nür" (with the middle character
coded as U+0075 U+0308).  Applying NFC to both strings handles
that case perfectly well and that is why we do it.  Part of the
issue here is the cases that NFC does not handle as well.

>...
> (This total obliviousness to security/spoofing/phishing isn't
> even limited to DNS, I've received calls because of billing
> glitches and the bank says "you can pay now, give us your CC
> or check routing info".  That's normal.  They they ask my
> secret code or whatever to verify it's me, and I point out
> that I can't even verify that they're who they say they are
> and they're stumped.  Well, you can call us back at
> 1-800-123-4567.  Seriously!?!?!  Even my bank has no clue what
> security is?)

Of course.  And I've had endless arguments with various
financial institutions, both about their procedures and their
barriers to letting me use even rudimentary self-protective
measures.  But it has nothing to do with the present question
whether or not I carry out my fantasy of posting all of those
examples to rants-are-us.example or, better yet, the more
elegant (or perverse, depending one one's perspective)
rants-я-us.example.

> Fix that, and then we can talk about whether it's worth fixing
> a couple esoteric code points.  There are FAR easier exploits
> for any serious attacker to attack.  It's certainly not worth
> risking breaking real strings and real mnemonics that real
> users might want to use.  (IMO its not even worth prohibiting
> I♥NY.com)

Sorry, but the community believes you are in the rough.
Incidentally, one of the main reasons why the second character
in I♥NY is DISALLOWED (and hence why the string is invalid as
a U-label) also has nothing to do with fraud but is associated
with difficulties in precise enough description of symbols in
screen readers and other text-to-speech applications that they
can be typed back in.  That is another human consideration of
course... too bad about those humans, the world would be so much
neater without them.

I've probably said too much already -- please try to forgive my
frustration after struggling with the real for more months than
I want to think about and being regularly distracted by people
who want to talk about confusion.  Again, please read the two
I-Ds which, at least IMO, actually do give a decent description
of what the problem is thought to be or at least parts of that
puzzle.   IMO, if either uses words like "confusion" or "fraud",
they are historical errors grounded in ways of thinking about
other problems.  

If you want to talk about confusion and fraud, they are really
interesting discussions on which a lot of good people have spent
a lot of time, but they are really not what this is about.

 best,
    john

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 19 Mar 2015 15:31:59 -0400

On Thu, Mar 19, 2015 at 06:29:06PM +0000, Shawn Steele wrote:

> This discussion seems to be about lots of things, but "reasonably mnemonic" would limit the problem hugely.  That implies that I can see it and remember what it was to type it later.  However that scenario doesn't hit any of the security concerns being raised as I'm going to type it using the appropriate keyboard for my language.

No, because it's reasonably mnemonic _for someone_, not for everyone.
You're assuming that localization is the only relevant parameter, but
it's not: the problem with identifiers for Internet-wide use is that
they're supposed to be both internationalized and localized at the
same time. 

For instance, CAs are supposed to be doing validation on these names
and nailing them to organizations.  We've already had problems with
invalid issuance of certificates (as you may be aware, since your
apparent employer has been one of the targets), and this issue opens
that door more widely.  CAs are not going to be experts in every
writing system, and the same-script registration rules won't help
anything here because they're ineffective.

Similar issues might be expected for other kinds of identifiers in
Internet-scope use.

I think if you read the I-D carefully, you'll discover that this point
is made in section 3.1.

> That's digressing from my point.  I was trying to say that, if we consider the problem without any human factors, we already have unique IDs.

And as I said before, this is tautologically true or false, depending
on what you mean by "ID".  It seems you're devoted to "tautologically
true", but I fail to understand why you think that's a remotely
interesting point in this discussion.  Everyone already agrees that
there's a human factors issue here.

> Fix that, and then we can talk about whether it's worth fixing a couple esoteric code points.  There are FAR easier exploits for any serious attacker to attack.

"There is that problem over there, so this problem isn't worth
fixing."  I'm not suggesting that the other problem is not an issue,
but this one turns out to be an in-principle misunderstanding on the
part of the IETF about how it was going to construct internationalized
identifiers.  In principle misunderstandings are the sort of things
that one must address (even to conclude "not worth fixing") if one is
going to write protocols that are worth anything at all.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Mar 2015 16:36:39 -0400

--On Thursday, March 19, 2015 10:50 -0700 Asmus Freytag
<[email-masked]> wrote:

>> (at least absent what we generically describe as "politics",
>> in which case the justification for its not decomposing would
>> disappear).
> 
> but that the Hamza normally is used to indicate a glottal
> stop. 

"unaccountability" is a judgment that has nothing to do with
this.  I've talked about it and constructed an example in
language terms because that is the way this example was first
justified to us and I was trying to refer back to that example
and presentation.   Pursuing the same example, one could as
easily, in a hypothetical coding system different from Unicode,
but taking advantage of the same basic code space, write 
    U+0628 U+0654 <phoneme:U+0294/> 
which would use U+0654 to identify an abstraction for the Hamza
grapheme and qualify its phonetic use with a qualifier that
identifies the phoneme.  The same story could be applied to our
old example of "ö" with different phonetic and/or language
qualifiers for German and Swedish.

Again, I'm not advocating any of those alternatives, only trying
to reinforce my point that these results are the consequences of
decisions about coding (either due to a general, if complex, set
of rules or to per character decisions) and not intrinsic
properties of the grapheme clusters.

> It's a
> single entity, that just happens to look like a glottal stop
> was placed after a beh.

Given the above, and various other conversations, I believe the
above statement about "separate letter" is true, but that the
argument is ultimately circular.   In other words, Unicode has
adopted a set of rules about when characters are distinct (rules
that, sadly, are not completely described in the standard) and
them applied them.  Application of those rules (which I'm
prepared to believe are reasonable for some general case and
self-consistent) results in "combining character" in one place
and "separate character" in another.  Application of a
hypothetical different set of rules (including, purely as an
example, my reading of the [now obviously incomplete] set of
rules that actually appear in The Unicode Standards) might yield
a different result, such as a precomposed character that
decomposed or no precomposed character at all.

> constrained, I would argue, because the decoration isn't a
> hamza (glottal stop) any longer.
>...

And I suppose that, if we wait enough centuries and Fula evolves
backwards from being written primarily in Latin script (after
being written primarily in Arabic script until a few centuries
ago), we will find out.  But, if things a going to be coded
differently because the writing systems associated with
different languages might evolve differently (as I agree they
have over the centuries) and the desire is to future-proof
things), then we should really code "ö" differently for Swedish
than for German because, after all, Swedish writing might evolve
in the direction of Danish or Norwegian so that "ö" would be
replaced by the phonetically-similar "ø".     Or perhaps the
evolution would work the other way and we should therefore code
Norwegian and Danish separately lest "ø" in Norwegian evolve in
the direction of "ö" while Danish remains unchanged.  The
"borrowed letter" analogy applies there too because few, if any,
of those languages were originally written in Latin script
(whether we should permit, e.g., Runic in the DNS is of course a
separate issue as is the likelihood of, e.g., Norwegian evolving
back toward it).  I imagine those changes are probably unlikely,
but they are less drastic than the Fula example which would
require a complete reversal of the writing system's choice of
basic script.

Unless Unicode is going to go back and review the decisions
(which various stability rules now prohibit), I don't think this
helps except for understanding whether there are actually
predictable rules or whether code point allocation decisions are
ultimately made by a small cluster of people sitting more or
less in private and saying "that should be a separate,
non-decomposing, code point because it feels that way to us and
that other thing shouldn't because it doesn't" (or in the words
of a notorious US judge "I know it when I see it ").   I still
hope the situation is closer to the former and your explanations
have helped me a lot in understanding why that may be true.  

But our bottom line is still (or again) that we need some sort
of overlay or mechanism that will allow identifier comparison
among strings with absolutely no language, phonetic, or other
usage-context information (other than script) to work, present
tense, to a first order approximation of what those writing the
grapheme clusters or seeing them in isolation perceive.  How
symbols are pronounced or affect pronunciation, how those
symbols might evolve as the writing system evolves, whether they
are used as word-forming letters in a script or as, e.g.,
Mathematical notations, or what languages they are used with are
simply not relevant to the issue.  

Either enough of the alternate coding sequences have to be
disallowed to eliminate ambiguity because only one combination
is left, the alternatives have to somehow compare equal, or we
need to conclude that certain combinations are sufficiently
unlikely to justify not worrying about.  I imagine we might end
up with a combination of those options for different cases.   If
the choice rules can be turned into properties that help in that
process, that would be great.  If not, I think we may need to
lose interest in them other than accepting that they will
sometimes produce these distinctions in coding choices.

> separated from the letters they are supposed to affect.

Of course, that is an argument against direction indicators in
Bidi situations and a number of other things.  I trust we all
understand by now that this is all about tradeoffs, optimization
for particular classes of usage, good choices about what to
optimize for, and, as you have pointed out, sometimes the result
that optimizing a coding system for one group of things may
pessimize it for another.

best,
    john

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 19 Mar 2015 15:15:20 -0700

On 3/19/2015 1:36 PM, John C Klensin wrote:
> this.

I used that term to merely mean "apparently without cause".

>    I've talked about it and constructed an example in

Character encoding is not the process of numbering entities by shape.

It's the process of numbering entities by their underlying identity in
writing.

For 90% plus of all cases, the distinction seemingly doesn't matter,
because, especially in context, we experience the different entities
as having different shapes. We don't even realize that we abstract
away the shape, radically in many cases, and we treat unexpected
overlaps of the shapes associated with different characters as
accidents of fonts, low-resolution rendering, or what not.

But U+0041 really encodes the idea of an uppercase letter A.
That it is possible to render it in a Fraktur font, where it may well
look (to the uninitiated) as a form of triangular U, and that this
does not cause any issues (usually) is fully indicative of this:
to repeat, what is encoded is the (abstract) identity, not the concrete
shape.

This works really well for ordinary texts where context allows us
to effortlessly (more or less) read characters even if they are shown
in (artistically) distorted shapes, such as used in decorative fonts.

There are cases, mostly for symbols and punctuation, where there
is both less of a shared understanding of what the expected range
of appearance is for a given text element defined in the abstract.

Here, then, is the case where character encoders (including Unicode)
will decide to defer more to the appearance in marking the boundary
between different abstract text elements, and thus disunify them
in the encoding.

On top of that, frozen letter forms (selected, very specific renderings that
do no longer admit of the same range of glyph shapes as the original
did) have been borrowed into technical notation - most prominently
phonetic and mathematical. (Much phonetic notation has then given
rise to new alphabets, returning  the frozen form back into the
pool, but now as a new letter).

No wonder that Unicode doesn't have an "algorithm" to decide on
where the boundaries fall in identifying the boundary between
separate abstract entities to be encoded. In the end, more often
than not, this bottoms out in a pragmatic judgment of the "I know
it when I see it" variety.

It's a fundamental misunderstanding, then, of character encoding to
expect it to be glyph based, and an equally fundamental misunder-
standing to expect combining marks primarily as lego blocks for
glyph construction.

Combining marks represent, variously, separate letters that happen
to graphically "float" above other characters or result in merged
shapes (Arabic and Indic scripts have plenty of examples) or they
represent identifiable decorations of characters, best understood
by the combining arrow above to represent a vector in math.

The decoration, in this case has a very clear semantic component
of its own, hence is a separate character, but graphically exists
only as a decoration (requires a base).

Most diacritics (accent marks, etc.) at least in phonetics, also have
these separate identities.

True dual encoding entered the picture because of the conservative
nature of writing systems (and character encodings). The precursor
encodings had (largely, but not exclusively) chosen a simpler
analysis, and encoded precomposed entities. Unicode inherited that
model (but also the competing model). It turns out combinations could
not be rationally supported in a universal set, because to be universal
one would have to cater for the needs of scholars to apply these marks
to (practically) *any* other character. Incomplete drafts of lists of known
combinations, for Latin only, reached several thousand combinations,
before sanity finally prevailed, and the precomposed entities were
limited to the then existing set needed for legacy support.

So why does Unicode continue to encode "precomposed" Latin
letters? The reason is that for some decorations, it's not sufficient
to say "apply the decoration" -- you also have to say where and
how to apply it. And there are many possible options. Therefore,
Unicode has identified certain combinations that will always be
encoded whenever they are part of an alphabet -- leaving the
use of the combining mark for non-standard purposes, like a one-off
use in a phonetics paper.

The list of combining marks that are treated that way (for the purposes
of Latin and similar scripts) is not currently documented in machine
readable form, but Unicode Technical Committee experts are quite
willing to entertain the needs of the larger community to identify
them via code point properties, so that IETF can use that property
to DISALLOW the combining marks.

Other combining marks can be identified as never (realistically)
leading to potential use in letters that are part of conceivable alphabets;
these can be identified and be added to the set of combining marks
for which NFC already prevents additional precomposed entities.

Any discussion leading to the adoption of such properties by IETF
would receive the necessary support from Unicode, and would
allow to cut down the problem significantly in the future.

The hamza cases may be harder in some ways.

> properties of the grapheme clusters.

I digressed earlier away from a point by point reply. But the key item I
want to reiterate is that "identifying the abstract entity" that is to 
be encoded
is a matter that cannot be reduced to an algorithm; it cannot be reduced
to a pseudo algorithm (such as human-applied, but otherwise inflexible
principles). In its full generality (including all edge cases) it will 
always
boil down to judgment calls.
> them applied them.

See above about rules.

Principles exist, in the form of "guiding principles", but they cannot be
absolute, because the complexity of human writing system is such that
the only way you can make inflexible principles is retroactively -- and
not only is Unicode still a work in progress, writing systems themselves
are never fully stable.

> Application of those rules (which I'm

A good chunk of the problem is treatable "by rule" -- that is, Unicode
experts have agreed that it is reasonable to assign properties to some
subsets of combining marks that would identify their participation or
non-participation in forming sequences that are used as letters.
(non-participation means that any use of the combining mark today
is not considered a "letter" in the sense of "member of an alphabet").

There will be edge cases, hamza being one of them, where such simple
solutions appear not possible. Some code points, again hamza being an
example, could be explicitly given a property of having a problem in that
regard - if IETF would find that helpful.
> ago), we will find out.
I'm not proposing that we should wait --- the point is more about that
things that can evolve on different paths are probably distinct entities,
and as character encoding is not a cataloging of shapes, but of entities,
then this considerations enters in the decision where to draw the boundary
between entities.
>   But, if things a going to be coded

We have, in the history of Latin writing, nothing that might serve as a
strong precedent. Most orthographic reforms result in the adoption of
entirely new forms (like a-with-ring instead of aa in Danish).

The closest we come is a preference for different angles of
the accent when used in Polish vs. French.

The judgment call was to leave that to font designers.

A more problematic case is the distinction between comma below
and cedilla below on S and T, which are claimed (against available
evidence of actual use) to track with local preference.

The actual situation is that font designs are all over the map, which
is fine for the precomposed shapes. But for the combining sequence,
the two marks, comma below and cedilla below are distinct. That
presents a quandary, because now the name clearly indicates a
preferred shape and there is only one possible decomposition for the
precomposed shape. As a result, Unicode was forced, by the internal
logic of a universal set to provide the alternate shapes with alternate
decomposition - doing a huge disservice to users who now have
confusability and data migration issues (this went down before
NFC was frozen, so only the confusability issue remains, because
depending on the font, both comma below and cedilla below may
appear the same in the precomposed form).
> have helped me a lot in understanding why that may be true.

I think the bulk of all cases tends to fall in the predictable scenario, 
because
UTC strives quite hard, actually to follow precedent. There is an 
irreducible
set of cases where things come down to judgment. You see that more often
in code points in the symbols and punctuation area, because of the 
difficulty
of separating form and function there.

> simply not relevant to the issue.

See above.
> unlikely to justify not worrying about.
Yes, that too.
> I imagine we might end
> up with a combination of those options for different cases.

I agree - this will not be a "one size fits all".

>    If

Properties. Definitely.

> Bidi situations and a number of other things.

Correct -- and to the degree that they are not implemented, at some point
protocols can decide that they are safely prohibited. Unicode just recently
added a new set, one that could be used to isolate one bidi stretch from 
another.
Those look like they are useful ones for protocols to require when 
displaying
labels. Too early to see whether they will be widely adopted. (Looks 
like, though,
because they would allow plain text mapping of HTML directional markup).
>   I trust we all

Yep. As long as we understand that Unicode can't be optimized for IDNA,
we can then find ways to optimize IDNA for Unicode :)

A./