Subject: Re: [idn] Re: IDNA: is the specification proper, adequate, and complete?

From: Dan Oscarsson <[email-masked]>
Date: Tue, 18 Jun 2002 11:41:10 +0200

>
>I don't see where the problem lies.  Yes, U+00C5 and U+212B do look
>identical; the existence of pairs like this is the main reason why
>canonical equivalence tables exist. 

They look identical because it is the same character. As Dough said
they exist because a legacy encoding. Personally I think Unicode
should never have allowed them to have more than one code point.

While there is not doubt about the above, I am not sure that
the nameprep specification that 00DF (small letter sharp s) should
be matted to "ss". I am not sure how Germans handle this character.
Do they always replace double s with it? Or only on some special words?
If they do not generally do this, the mapping should not be done.
It is somewhat like the fact that the Greek version of latin A is
not mapped to the Roman version of latin A. Even though their origin
is the same latin A and look alike.
While "small letter sharp s" looks like "small letter beta", there are
no similarities between double s and "small letter sharp s". So why not
let "small letter sharp s" remain that and be a distinctive character
in a domain name. I assume Germans use it to write German words. Not
to replace double s in non-German names.

   Dan

---

From: "James Seng" <[email-masked]>
Date: Tue, 18 Jun 2002 21:22:06 +0800

Whether DF should be decompose to "SS" or otherwise is defined in NFKC.
Therefore, this question should be appropriately address in Unicode
Consortium.

-James Seng

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Jun 2002 10:12:59 -0400

--On Tuesday, 18 June, 2002 11:41 +0200 Dan Oscarsson
<[email-masked]> wrote:

> the nameprep specification that 00DF (small letter sharp s)
> should be matted to "ss". I am not sure how Germans handle
> this character. Do they always replace double s with it? Or
> only on some special words? If they do not generally do this,
> the mapping should not be done. It is somewhat like the fact
> that the Greek version of latin A is not mapped to the Roman
> version of latin A. Even though their origin is the same latin
> A and look alike.

This was discussed at length early in the history of the WG.  By
convention, it is always possible to replace Eszett with "ss",
and the upper-case form of Eszett is always "SS", but there are
many words in which "ss" appears for which a substitution back
to Eszett is not appropriate.  In other words, this is a
one-way, non-reversible (without word-context), mapping.  The
good news is that, stringprep gets it right (or at least
consistent with other WG decisions) given coding as 0x00DF,
which clearly identifies the character as Eszett.  

The bad news is that, if this character [form] is observed in a
written string, or a non-Unicode coding system that does not
distinguish between Eszett and "small letter [Greek] beta", one
really needs to have language context in order to determine
which Unicode character to map it to, especially because, given
stringprep, the consequences of getting it wrong are likely to
be very significant.

But this is not new news, or even a new example.  We may see it
differently as our sensitivities to the issues evolve, but the
bottom line is that Unicode is not especially well adapted to
coding of strings that appear without language, or even word,
contexts in non-Unicode form.  Whether that form is a
pre-existing coding system, or a sign on the side of a bus,
there are likely to be examples of problematic characters.
Unfortunately, there are no standardized alternatives for a UCS
with even near-global applicability.  And it appears obvious to
me that, while a hypothetical Unicode alternative could make
different choices, it wouldn't eliminate these problems, but
rather just create a different set of scary examples.

I think there are only two ways out of this, and neither
involves either changes in stringprep or more examples of this
type.  For the latter, I think almost everyone with a strong
desire to understand the problem has done so and that the odds
of convincing others are fairly low.

The alternatives are:

(1) To define the problem this working group is trying to solve
in a way that causes these problems to be non-issues.
Personally, every time I try to do that, I end up with what feel
to me to be silly states, but it is clear that I'm in the
minority of those speaking up in the WG.  For example, one could
say, and I think we essentially have, that the WG is solving the
problem of getting things into and out of the DNS given that the
Unicode coding form is accurately known.  This implies that any
applications which can't succeed in making the translations are
going to be in very bad trouble and that we offer them no help
or hope -- it isn't our problem.   Personally, I think that, if
the WG's position and recommendations are based on that model,
we should be obligated to write it down and make it explicit in
our documents before they go onto the standards track: we owe
that much to those who think we are solving any of a number of
more general internationalization problems.

(2) To just give it up.  The DNS effectively imposes "no
language information" and "no script information" restrictions
on us.  Its octet-based comparison rules effectively prevent us
from imposing  conventions that would permit guessing anything
from context, nor can we prohibit a string that contains 0x00DF
in the middle of a string of Greek, or for that matter Arabic,
characters.  In the Arabic case, it would at least stand out as
strange; in the Greek one, humans would almost certainly mistake
it, in displayed form, for lower-case beta.  Given the
restrictions, these presentation-relationship problems have no
in-DNS solution.

My own personal position on this is presumably well-known at
this point:  we have tried very hard to solve a "DNS
internationalizaiton" problem and have ended up with a number of
extremely convincing demonstrations (this example not least
among them) that it is overconstrained and can't be solved.  If
one accepts that, then there is a strong case for saying "sorry,
whatever it is people wish for, and can make work in restricted
contexts, DNS labels for common, existing, RRs and applications
are limited to ASCII-based protocol elements and we had best go
solve the real problem and requirement in some less-constrained
environment".  

While others clearly disagree (and are probably the majority), I
also don't believe it is appropriate for IETF to adopt a
protocol without clear evidence that it can be implemented and
used by at least a few of the applications that are anticipated
for it -- applications that need to deal with presentation
forms, operating systems, and other aspects of the real world.
But, again, if we are going to do it, or if we see some
restricted applications for which this _does_ provide a useful
part of a solution, I believe that we are obligated to explain
exactly what problem we are solving, so as to not inadvertently
surprise implementors and users with scenarios and problems that
we understand.  If we didn't know about these problems, it might
be different, but we certainly do.

But I do think those are the issues that the WG (and really, at
this stage, the IESG) should be discussing.  And, if there are
new issues, I, at least, would like to understand them.  But
more examples or transcription or transcoding difficulties
won't, I think, help: we already have enough examples to kill a
dozen protocols if the WGs considered them relevant.  More
examples, or repeating of older ones, would not persuade anyone
who is convinced that the ones documented so far are not
relevant that they are suddenly relevant. 

Just my opinion.

     john

      john

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 18 Jun 2002 08:05:26 -0700

> stringprep, the consequences of getting it wrong are likely to

A few items here.

1. While there are many Unicode characters with very similar shapes,
beta and eszed -- in normal fonts -- are really no more alike than y
and gamma (or, for that matter, capital I, lowercase L, and 1!!). Beta
typically has a descender, while eszed does not. See
http://www.macchiato.com/unicode/beta-eszed.htm for a GIF image from
three fonts.

2. There are no "non-Unicode coding systems" that unify beta and
eszed; the language issue is irrelevant.

3. As I pointed out some time ago on this list, it is *not* rocket
science to provide a user interface that makes it very clear to people
if there are mixed scripts in a domain name; and also simple to extend
it to other confusables. For a demo, see
http://www.macchiato.com/utc/show_script.html.

I have recommended and continue to recommend that the IDNA documents
contain some wording on this, something like:

"To help prevent confusion between characters that are visually
similar, it is recommended that implementations provide visual
indications where a domain name contains multiple scripts. Such
mechanisms can also be used to show when a name contains a mixture of
simplified and traditional characters, or to distinguish zero and one
from O and l."

Mark
__________
http://www.macchiato.com
◄  “Eppur si muove” ►

----- Original Message -----
From: "John C Klensin" <[email-masked]>
To: "Dan Oscarsson" <[email-masked]>; <[email-masked]>
Sent: Tuesday, June 18, 2002 07:12
Subject: Re: [idn] Re: IDNA: is the specification proper, adequate,
and complete?

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 18 Jun 2002 09:05:04 -0700

At 11:41 AM +0200 6/18/02, Dan Oscarsson wrote:
>They look identical because it is the same character.

That definition of "same" doesn't match the definition that the rest 
of us are using. They have different code points, they have different 
names, and they have different Unicode properties.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 18 Jun 2002 10:01:09 -0700

"Paul Hoffman / IMC" <phoffman at imc dot org> wrote:

>> They [U+00C5 and U+212B] look identical because it is the same
>> character.
>
> That definition of "same" doesn't match the definition that the
> rest of us are using. They have different code points, they have
> different names, and they have different Unicode properties.

Same Unicode properties -- "Lu", meaning "Letter, Uppercase."  Of
course, by itself that hardly makes them the same character -- "A" and
"B" are also classified as Lu.

I'm not sure how productive it's going to be to argue whether two
characters or sequences constitute "the same character."  You can go
either way in some cases (what about U+00C5 vs. U+0041 U+030A?).  Simon
Josefsson has been trying to show that, because U+00C5 and U+212B are
"different" characters but look the same, and because stringprep folds
them together, there is therefore "ambiguity" and security loopholes are
introduced.  (I still think it's the other way around; leaving them
separate is what would invite spoofing.)  Because of this line of
argument, some of us have stated that U+00C5 and U+212B aren't
completely different because of canonical equivalence.  But of course
they're not completely the same, either.  This might be sort of a
"dancing angels" debate that generates no light, but perhaps a lot of
heat.

In any case, hasn't the broad question of canonical folding been settled
for a long time now?  I'm really starting to sympathize with Georg
Ochsner, who asked when on earth this project would finally be
completed.  I have heard very few new arguments in the past six months,
but a great deal of rehashing of old issues (like this one) that I
thought had already been resolved.  Georg is right -- at this rate,
nobody will stick around to wait for the IETF solution, but will go with
whatever Verisign slaps together (probably with a lot less consensus
building that this WG) and that will become the de-facto standard.  Do
we want that?

-Doug Ewell
 Fullerton, California

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Jun 2002 13:51:33 -0400

--On Tuesday, 18 June, 2002 08:05 -0700 Mark Davis
<[email-masked]> wrote:

>> The bad news is that, if this character [form] is observed in
>> a written string, or a non-Unicode coding system that does not
>> distinguish between Eszett and "small letter [Greek] beta",
>> one really needs to have language context in order to
>> determine which Unicode character to map it to, especially
>> because, given stringprep, the consequences of getting it
>> wrong are likely to be very significant.
> 
> A few items here.
> 
> 1. While there are many Unicode characters with very similar
> shapes, beta and eszed -- in normal fonts -- are really no
> more alike than y and gamma (or, for that matter, capital I,
> lowercase L, and 1!!). Beta typically has a descender, while
> eszed does not. See
> http://www.macchiato.com/unicode/beta-eszed.htm for a GIF
> image from three fonts.

As you, and I, and others, have noted in the past, there are far
better examples than beta and eszett (or eszed if you prefer).
But there are certainly fonts (which I assume you would classify
as "not normal") -- designed in isolation for the two languages
-- whose interpretations of the two characters, if placed in
text of the other language, would not catch the eye of a casual
reader as being out of place.   And I would assume that, in the
long history of manual typesetting, there have been instances of
Eszett being substituted for beta (probably in German texts
containing mathematics because a lazy typesetter was disinclined
to walk across to a different font).   None of this, of course,
challenges your fundamental argument, with which I agree and
hope I was careful enough to write my note to not challenge.

> 2. There are no "non-Unicode coding systems" that unify beta
> and eszed; the language issue is irrelevant.

Sure there are.  We call some of them "books".  Transcription of
a language into printed form involves a coding system.  And I
have to assume, although I can claim no personal knowledge, that
German schoolchildren, brought up looking at Eszett, have to be
taught, when they encounter mathematical notation that uses
Greek characters (if not sooner), that it is important to notice
either the context or the descender -- that the two characters
are not the same.  These distinctions, including getting used to
the variations and similarities in different fonts of I-l-1, are
bits of pattern recogition that lay people --as distinct from
font or character set experts-- rapidly learn, within their own
language and script contexts, to distinguish from context or by
relatively subtle clues.  I can't even spell out Arabic or Thai
scripts because I don't have enough experience with the right
set of clues -- my loss, but these are learned skills.

But this isn't the point, so whether there are, or are not,
coded character sets that unify the two is not the point either
(I'll defer to your knowledge and experience on this subject,
since I haven't studied the question, but statements that sound
like universal negatives always scare me).  Your third comment
_is_ as key part of the point.

> 3. As I pointed out some time ago on this list, it is *not*
> rocket science to provide a user interface that makes it very
> clear to people if there are mixed scripts in a domain name;
> and also simple to extend it to other confusables. For a demo,
> see
> I have recommended and continue to recommend that the IDNA
> documents contain some wording on this, something like:
> 
> mechanisms can also be used to show when a name contains a
> mixture of simplified and traditional characters, or to
> distinguish zero and one from O and l."

Mark,

I ultimately have only three problems with IDNA and the IDN
proposals taken as a group.  The first is a technical one and
applies to IDNA specifically.  The others are problems that
would probably extend to any in-DNS substitute):

(i) It makes assertions about applicability that I believe are
over-broad and risky, to no good benefit.  I think we are
nearing consensus on that one and, in any event, that reviewing
it again wouldn't accomplish much.

(ii) It is addressed to, and solves, a very narrow problem.  We
(for some definition of "we") have not been explicit, in an
Internet context, as to what that problem is.  I believe that we
should be explicit.   Then, having carefully described that
problem, we then need to carefully evaluate the question of
whether the benefits of solving it outweigh the risks to the use
of the DNS in the Internet community that it might pose.  If we
conclude that we can't reasonably do that evaluation (e.g.,
because it isn't an IETF problem), then I think we are still
obligated to delineate the issues and risks to the best of our
ability -- at least to the extent of writing down the
implications of problems and issues we already know about.

(iii) A number of items of knowledge and recommendations have
surfaced in the working group -- of which your suggestion above
is an excellent example -- that could be used to reduce or
eliminate some of those risks to the DNS as a piece of usable
Internet infrastructure.  I think they need to be written down
as part of WG output, if only because "this risk can be
ameliorated if one does so-and-so" is a much more satisfactory
statement than "there is this horrible problem and we should
consider stopping progress until someone has a solution".

Of course, if "mixed scripts in domain names" are considered
good things, warning when they occur won't help much.   But
_that_ one, I would contend, is not an IETF problem although I
think it would be wise and responsible for us to point out that
mixed script labels pose challenges that homogeneous ones do not.

regards,
     john

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 18 Jun 2002 21:30:11 +0000

I don't understand why there is all fuss concerning the various folding
that Nameprep performs.

The Unicode standard says that two canonically equivalent strings
should be treated the same (either one should work in any place that
the other works).  IDNA is simply honoring that recommendation.  If it
didn't normalize strings before encoding them or comparing them, then
canonically equivalent strings wouldn't get treated the same.  Canonical
equivalence deals with things like "a with grave" versus "a" followed by
"combining grave".

The Unicode standard also defines compatible equivalence, and says that
compatibility characters are characters that didn't really deserve
their own code points, but were reluctantly given their own code points
for the sole purpose of preserving round-trip conversions to legacy
charsets.  Nameprep respects this reluctance by using NFKC rather
than NFC, so that the compatibility characters are folded into their
equivalents.

The Unicode standard also defines a locale-independent case folding
algorithm explicitly intended for doing case-insensitive comparisons,
which is exactly what IDNA needs it for.  German sharp s is folded to ss
because a case-insensitive comparison must match German sharp s with SS
(the latter is the normal uppercase form of the former), and must match
SS with ss (the latter is the normal lowercase form of the former), and
so by transitivity it must match German sharp s with ss.

John C Klensin <[email-masked]> wrote:

> For example, one could say, and I think we essentially have, that the
> WG is solving the problem of getting things into and out of the DNS
> given that the Unicode coding form is accurately known.

Yes (but not just DNS, all existing protocols that use domain names).

> if the WG's position and recommendations are based on that model,
> we should be obligated to write it down and make it explicit in our
> documents before they go onto the standards track: we owe that much
> to those who think we are solving any of a number of more general
> internationalization problems.

Suggested text addressing that concern was posted to the list a few
weeks ago, see message <[email-masked]>.

AMC

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 18 Jun 2002 15:02:19 -0700

> are not the same.

My point was that they probably have about the same level of confusion
that you had when you first saw a gamma as a schoolchild and confused
it with a y.

> These distinctions, including getting used to

True

> like universal negatives always scare me).

Let me qualify my negative: I have never heard of any that do, and of
the 750 odd code page mapping tables that we have collected on major
platforms (http://oss.software.ibm.com/cvs/icu/charset/data/xml/),
none of them do. Of course, if you go down to Arkansas Bob's "Bait,
Tackle, and Character Encodings Shack", he can whip up a nice
character set in no time flat that'll unify them. You should
understand my phrase 'no "non-Unicode coding systems"' as meaning 'no
"non-Unicode coding systems" of any importance or impact', and I'll
now understand your phrase "non-Unicode coding systems" as meaning
"books" ;-)

..
> consider stopping progress until someone has a solution".

I agree that in both of these cases capturing additional information
as guidelines for dealing with particular issues would be very
helpful.

> mixed script labels pose challenges that homogeneous ones do not.

I agree.  Such UI's would certainly alert people to something very
fishy going on in the case of "Intel.com" spelled with a Cyrillic 'e',
without preventing legitimate registrations such as
"ABC<alpha><beta><gamma>.com". For the latter, the visual indication
would be there, and people would be alerted to the multiple scripts,
but it would not prevent usage.

Regard,

Mark

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 18 Jun 2002 15:45:36 -0700

> eszed; the language issue is irrelevant.

MS-DOS code page 437 had a character at 0xE1 that was sometimes rendered
more like a sharp-s and sometimes more like a small beta, depending on
which screen font you were using.  In the standard 8x8 font and 8x14
fonts it was very definitely a beta, but in the 8x16 font it was a
sharp-s.

To make matters worse, the character was surrounded by small alpha at
0xE0 and capital gamma at 0xE2, implying a continuous sequence of Greek
letters.  But in code page 850, 0xE1 was unmistakably a sharp-s.

One might say that CP437 was inadequate to encode Greek anyway, but then
it was inadequate to encode plenty of Latin-based languages as well, so
that was not a great argument against the "beta" interpretation.

Of course, character tables hardcoded in the ROMs of PCs weren't
intended to be definitive.  Things were looser then.  Before Unicode
came around as sort of the "gold standard" of character encodings, most
people didn't worry about whether 0xE1 was a beta or a sharp-s.  It
could be either, depending on context.  Now our expectations are higher,
and we need to differentiate between the two.

-Doug Ewell
 Fullerton, California

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 21 Jun 2002 10:07:30 +0900

----- Original Message -----
From: "Doug Ewell" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "John C Klensin" <[email-masked]>; "Dan Oscarsson" <[email-masked]>;
<[email-masked]>
Sent: Wednesday, June 19, 2002 7:45 AM
Subject: Re: [idn] Re: IDNA: is the specification proper, adequate, and complete?

> fonts it was very definitely a beta, but in the 8x16 font it was a

One similar example in TC/SC:

  One of the most popular korean word processor "Arae-A Hangeul" (not unicode-based)
     provides with two chinese font sets, one for TC and the other for SC.
  If a Korean user wanna write a letter in SC that may contain  IRI or IDN,
   he should type  it in TC first and change the session font set into  given SC font.
  Then, most 1:1 TC letter is displayed/printed as its SC equivalent.

  Most Koreans are not familar to exotic Input Methods of Japanese and Chinese Simplified
  letters. Facilitating input of exotic chars by font substitutions has merits.  That remind me of
  the ASCII-mapped DINGBATS font sets.  I don't know well
  about whether or not   there are similar cases in japan , taiwan and china.

 Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 21 Jun 2002 11:13:02 +0900

Copy&Paste operations from  such legacy applications into other applications
may not preserve the intended character glyph informations  determined by  char sets+font sets.
That will pose another interoperablility problems to end users.

Some OS-addon/applications has it own proprietary legacy char set/locale, but share the same code space with ASCII
and use ASCII-mapped font-substitution to implement their legacy char sets, for example, in order to
implement CJK supports in English-only windows or linux. I found some other examples in Indic char sets.
 
In General, informations about assumed/system-wide font sets  
  are not available to applications  or not interpretable  with machine intelligence.

These kinds of font usages should be discouraged in IDN specifications ?

Soobok Lee

----- Original Message ----- 
From: "Soobok Lee" <[email-masked]>

---

From: tedd <[email-masked]>
Date: Fri, 21 Jun 2002 11:41:34 -0400

>That will pose another interoperablility problems to end users.

Soobok:

IMP, inoperable problems to end users will happen regardless of what 
the IDN does. Copy&Paste operations (legacy or otherwise) are not 
uniform across platforms and nothing this WG (or any WG) does will 
fix that.

However, Copy&Paste operations (at least in the Macintosh) are 
becoming much more complex than they were and what they would appear. 
While the specifics of what is evolving in the Copy&Paste world is 
beyond the interest of this WG, I foresee an end user OS software 
solution to your concerns. I do not expect this WG providing (or even 
having the means to provide) a solution to that problem.

tedd

-- 
http://sperling.com