Subject: Character equivalence mapping (was: Re: [idn] SLC minutes)

From: Kenneth Whistler <[email-masked]>
Date: Wed, 2 Jan 2002 17:51:24 -0800 (PST)

Edmon suggested:

> Character Equivalence mapping is to deal with this issue:
> 
> A registrant registers a domain <ALPHA><BETA>.example
> Advertises it to other people as their capital form AB.example
> An end user will not know whether it was Greek or English and attempts to
> access the site with ab.example and does not get to it.
> 
> With Character Equivalance mapping, this situation would not occur.  No
> matter how a domain name is represented, it is always unique.

I think this example nicely points up the contrary problem that
cross-script mapping has. If you start doing cross-script equivalence
mapping to eliminate differences between (to Latin-trained eyes) confusable
letters, you violate the integrity of other scripts and start mapping
the set of possible strings in those scripts even more confusably into
the already crowded domain namespace of Latin strings.

In this particular example, suppose I was a Greek and actually wanted
to register <ALPHA><BETA>.com, in addition to <ALPHA><BETA>.gr for
the <ALPHA><BETA> construction company in Athens. Whoops! I'd be
out of luck since ab.com already exists and is registered to
Allen-Bradley. (See www.ab.com ) Why should I, as a Greek, find my
own Greek namespace unpredictably polluted by some arbitrary list
of equivalences between Greek letters and Latin letters?

And exactly what equivalences would you suggest? Greek uppercase
eta is basically indistinguishable in shape from a Latin uppercase "H".
So do I equivalence map it to Latin "H", which would make no sense at
all for transliteration and serve only the purposes of dumb equations
for people who know nothing about Greek whatsoever? Or do I equivalence map it
to Latin "I", which is the normal transliteration for eta in Modern Greek?
Or do I equivalence map it to Latin "E", which is the normal transliteration
for eta in Ancient Greek?

So does: <ALPHA><BETA>.<OMICRON><MU><ETA><RHO><OMICRON><SIGMA>

equate to: ab.omhpo<sigma> or ab.omiro<sigma> or ab.omero<sigma> or
           ab.omhpos or ab.omiros or ab.omeros ?

By the way, the 5th example is how the Greeks themselves would Latinize
it. (see www.omiros.gr )

The problem of "AB.example" is generally dealt with by context. First
of all "example" would be in Greek if I was really dealing with Greek.
Second, if I wanted people to enter "ab.whatever" I'd be advertising
in *English* to set the expectations. If I wanted people to enter
"<alpha><beta>.....", I'd be advertising in *Greek* to set the expectations,
and people would be using Greek keyboards and expect to enter Greek.

Furthermore, visual confusability quickly runs off the road as the basis
for determining equivalence classes when you start to deal with scripts
that have more complicated rules for the presentation of glyphs than
is typical for the Latin script. Which of several possible forms is
the basis for the confusability used to determine the equivalence?
And this turns into an N-body problem, because you start having to
account for visual confusability between N different scripts -- not just
between N scripts and Latin characters. Where do you draw the line, in
principle? Or do we just end up arguing for the next decade about all
the edge cases?

> 
> Bear in mind that this need to happen only during matching of names within
> the DNS server.
> 
> A registrant can register <ALPHA><b>.example all they want.  This is the
> misconception that I wanted to point out.  Character Equivalence mapping
> does not prohibit mixed scripts.

But it does severe damage to the integrity of namespaces in other scripts.

This is Latin- and English-centric thinking, in my opinion, that would
damage the whole point of having IDN's by folding other scripts towards
Latin characters.

--Ken

> Edmon

---

From: John C Klensin <[email-masked]>
Date: Wed, 02 Jan 2002 21:40:46 -0500

Ken, 

We are in agreement that "equivalence matching" based on
character shapes is fairly hopeless.   But...

--On Wednesday, 02 January, 2002 17:51 -0800 Kenneth Whistler
<[email-masked]> wrote:

>...
> The problem of "AB.example" is generally dealt with by
> context. First of all "example" would be in Greek if I was
> really dealing with Greek. Second, if I wanted people to enter
> "ab.whatever" I'd be advertising in *English* to set the
> expectations. If I wanted people to enter
> "<alpha><beta>.....", I'd be advertising in *Greek* to set the
> expectations, and people would be using Greek keyboards and
> expect to enter Greek.

The problem is that, with an international Internet and (short)
DNS names, the context may not be there, or may be unreliable
enough that users, in practice, will need an opportunity to pick
among alternatives.  While I think Edmon's notions would make
that problem worse, rather than better (hence my attempt at a
reductio ad absurdum argument in my earlier note), I don't
believe that "resolve by context" is going to be realistic in
practice --again, in the DNS context-- either.

    john

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 2 Jan 2002 20:32:37 -0800

In addition,

1. This issue was debated at length some time ago. I suggest that the people
arguing for visual confusability as a criterion for matching look at that
discussion in detail before proceding.

2. Moreover, stop and think about the implications; using both case folding
and visual confusability would have some very unpleasant consequences. For
example, it would force the ASCII letters N and V to be in the same
equivalence class:

- N is in the same class as GREEK NU, by visual confusability.
- GREEK NU is in the same class as greek nu, by casefolding.
- greek nu is in the same class as v, by visual confusability.
- v is in the same class as V, by visual casefolding.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Kenneth Whistler" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, January 02, 2002 17:51
Subject: Character equivalence mapping (was: Re: [idn] SLC minutes)

> "<alpha><beta>.....", I'd be advertising in *Greek* to set the
expectations,

---

From: "Edmon" <[email-masked]>
Date: Thu, 3 Jan 2002 11:31:22 -0500

Hi John & Ken,

> > context.

But the context of domain names is Internet, and the Internet is
conceptually and practically becoming multilingual, which is why we are
discussing this issue today.  Therefore, the point is that if the context is
multilingual, AB.example needs to be unique regardless of what the
"intended" "language" is, because the "perceived" character is what becomes
the context in itself.  Therefore, the context of a character is the
character itself.  Whether by a Latin-trained-eye or by an
Arabic-trained-eye or a Chinese-trained-eye, some characters are
unmistakably identical.  That is the root of the problem at hand.

> > First of all "example" would be in Greek if I was

This is a presumption.  And "localization" by TLD is beyond the
technicalities of trying to make a solid IDN solution.  We should not resort
to the management of TLDs to finish our job..?..

Again, context should be irrelevant because context is not known.
For example (really just an example, there could be other cases)
I dont know greek, but I see an interesting ad that I think might be for a
Greek translation service.
I see the domain A2<beta>.example, (the actual domain was
<ALPHA>2<beta>.example)
I try it as I see it, it goes elsewhere.

> While I think Edmon's notions would make
> reductio ad absurdum argument in my earlier note),

My notion is to keep the DNS stupid and just map all Characters with
"perceived" equivalence during internal matching.  The domain names as they
conceptually exist however should remain user friendly.  In other words,
domain names for humans will contain context and meaning (an "ALPHA" will be
an "ALPHA" not "A"), but domain names as viewed in the DNS (machine) should
be devoid of context.  To bridge these two, a set of equivalent characters
are prepared as a table for the machine so that it can blindly treat them as
identical without contemplating its context.

> I don't

Very Agreeable.

Edmon

...Just as a final note here, I think Character Equivalence mapping could be
an optional "best practice" type of configuration for zone operators to
provide to their users, however, it would be a good idea to have a
technically sound approach, which is what I am trying to discuss.

---

From: John C Klensin <[email-masked]>
Date: Thu, 03 Jan 2002 11:45:15 -0500

--On Thursday, 03 January, 2002 11:31 -0500 Edmon
<[email-masked]> wrote:

>...
> The domain names as they
> conceptually exist however should remain user friendly.  In
> other words, domain names for humans will contain context and
> meaning (an "ALPHA" will be an "ALPHA" not "A"), but domain
> names as viewed in the DNS (machine) should be devoid of
> context.  To bridge these two, a set of equivalent characters
> are prepared as a table for the machine so that it can blindly
> treat them as identical without contemplating its context.

But, Edmon, we already have disproofs of this in trademark
registrations and the registrars claim (and I believe them) that
the companies involved are anxious to take advantage of
internationalization to register their trademarks more directly
in the DNS.  We have organizations who want to be 
  <alpha><roman-i><digits123>
  toys-<cyrillic-ya>-us
and so on, a list that will certainly get longer as more
characters are perceived as available.  

What you are trying to do also involves some rather complex
judgement calls which I don't know how to make.  As someone who
is not very familiar with either, I've seen a number of font
forms of Arabic and Thai that I'm not sure I could tell apart,
at least without considerable context.  I would assume that
daily users of the two scripts wouldn't have that problem, but
who is to make the decision about equivalence?

It seems to me that there, in practice, are only two ways to get
what you want without ambiguity or confusion.  One goes down the
path that Ken points out, which results, ultimately, in
"equivalencing" some very different things.  And the other is to
say "ok, we will add characters to the LDH list, but only those
characters that cannot be confused with anything else,
regardless of the case or font used".  Maybe, in the long term,
they turn out to be the same option.

It would have the advantage of adding many useful characters to
the DNS-application-permitted set (and Harald and Patrik could
stop worrying about the names of their respective sons :-)), but
it would do almost nothing for internationalization other than
throwing us back into strange transliterations or transpositions
for many scripts.   Put differently, we would end up with a DNS
character set that would probably support Latin-1 and Han
characters properly, but maybe nothing else.  I don't find that
very satisfying, much as I am concerned about accurate
transcription of labels from printed form into the DNS.

     john

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 4 Jan 2002 02:15:39 +0900

Hi, John and Edmon

I feel it is interesting to try to find some analogy for fun(!) between famous old Babel Tower story and modern Internet name system
history. How about replacing 'Shnar->Internet' and 'brick->computer' and 'bitumen -> network' and 'one language->ASCII domain' ?
Just enjoy reading the next excerpts from Genesis.

  "Now the whole earth had one language and few words. And as men migrated from the east, they found a plain in the land of Shinar
and settled there. And they said to one another, 'Come, let us make bricks, and burn them thoroughly.' And they had brick for stone,
and bitumen for mortar. Then they said, 'Come, let us build ourselves a city, and a tower with its top in the heavens, and let us
make a name for ourselves, lest we be scattered abroad upon the face of the whole earth.' And the LORD came down to see the city and
the tower, which the sons of men had built. And the LORD said, 'Behold, they are one people, and they have all one language; and
this is only the beginning of what they will do; and nothing that they propose to do will now be impossible for them. Come, let us
go down, and there confuse their language, that they may not understand one another's speech.' So the LORD scattered them abroad
from there over the face of all the earth, and they left off building the city.' Therefore its name was called Babel, because there
the LORD confused the language of all the earth; and from there the LORD scattered them abroad over the face of all the earth."
(Genesis 11:1-9)

  "Babel" is composed of two words, "baa" meaning "gate" and "el," "god." Hence, "the gate of god." A related word in Hebrew,
"balal" means "confusion."     (http://www.ldolphin.org/babel.html)

Will IDN  be "the gate of god" or "confusion" to the mankind ?

Soobok Lee

---

From: "Edmon" <[email-masked]>
Date: Thu, 3 Jan 2002 12:38:12 -0500

Hi John,

----- Original Message -----
> in the DNS.

While I dont think that trademark struggles should have too much (note that
I used not "too much" and not "without") bearing on the way we need to treat
the DNS.  Additionally I think it would actually be better for the situation
with Character Equivalence preparation.

Imagine this:

Company 1: AB holds the trademark for AB and the domain AB.com
Company 2: <ALPHA><BETA> holds the trademark for <ALPHA><BETA> and the
domain <ALPHA><BETA>.com

Can Company 2 advertise as <ALPHA><BETA>.com literally in its capitalized
form as AB.com?

> We have organizations who want to be

They can be.  That is the whole point of IDN, and I do not refute it.  But
we need to make sure that as we introduce these extended characters, we do
not cannibalize on what the DNS should be and has become, and that is to
allow a human-friendly name to be publicized without ambiguity and used to
access a resource uniquely over the Internet.  What Character Equivalence
preparation tries to achieve is to preserve the two keys for the domain
names to be able to be "publicized without ambiguity" and to "access
resource uniquely".

> who is to make the decision about equivalence?

This is undeniably a good question, but does it mean that we should not do
anything about what we know?  So long as we remain open and make the
necessary ammendments as we go along, I dont think we will be condemned.
But not doing anything about it, we will risk being accused.

> transcription of labels from printed form into the DNS.

That is precisely the misconception I want to point out.  Because the DNS
matching process treats characters as equal doesnt mean that they cannot be
conveyed within context.  On the contrary, it is designed to better support
the ability for domain names to continue to become an integral part as it
has now in the everyday context of the Internet and its content.

For example, the owner of <Alpha><Beta>.example will not have to worry about
how his design on his business card or advertisements or letterheads look
like even if his designer decides that it looks best when everything is
capitalized.  Without Character Equivalence preparation, the owner has to
worry about the confusion and make sure that in all cases, whether it comes
out in a magazine article or an editorial on the local newspaper to remind
them that they must use the lowercase form and never use the uppercase form!
Does this not actually result in the strangeness that one might thought one
was avoiding by not introducing Character Equivalence preparations?

Edmon

---

From: tedd <[email-masked]>
Date: Thu, 3 Jan 2002 12:35:25 -0500

>discussion in detail before proceding.

I'm not arguing (in this debate) the "look-a-like" position. In other 
words, it makes no difference to me if certain glyph's look identical 
in numerous char sets. I am arguing the opposite position -- the 
characters in my example don't look a like.

I am arguing the point that the decision "has been made" to map upper 
case Greek letters to lower case letters. For proof, look at the 
current version of nameprep ( http://www.imc.org/nameprep/  ) and try 
running code point 2126 (upper case omega) through it. You will find 
that it IS mapped to code point 03A9 (lower case omega).

My question is "Why?" What's the foundation for this determination? 
For what good reason is there to conclude that the upper case Omega 
should be mapped to a lower case omega?

I see no "A.com" to  "a.com" argument/problem here. Clearly, if 
someone registered ‡.com and someone else registered w.com there is 
significant difference in identification between the two names. Those 
two domain names can be completely unique domain names with no 
significant resultant problems. Whereas, in the Latin char set, I can 
see the reason for making W.com and w.com identical (i.e., mapping W 
to w) because there is an UC/LC consideration/distinction in the 
language. But, that's not a problem in the Greek char set -- is it... 
really?

>(i) From observation, when scripts have two cases, the
>upper-case form is more likely to be highly stylized, and hence
>differentiated from characters in other scripts, than the
>lower-case one.  Hence, if one is going to adopt
>stylization-based (glyph-distinction, if you prefer)
>canonicalization rules, one is better off treating upper case as
>the normal form, rather than lower case.

It looks to me as if someone has already made the determination to 
map other languages based upon the Latin char set UC/LC problem 
without concern that other languages may not have the UC/LC 
distinction and thus be absent of the UC/LC problem. I think the 
Greek example I gave above sufficiently demonstrates my observation.

tedd

-- 
http://sperling.com

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 3 Jan 2002 09:54:02 -0800

Other scripts do have upper/lowercase correspondences, just like the Latin
script does. Users of those scripts are just as likely to want caseless
matching as users of the Latin script (such as you).

For more information, see http://www.unicode.org/unicode/reports/tr21/.

Mark
—————

http://www.macchiato.com

----- Original Message -----
From: "tedd" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, January 03, 2002 09:35
Subject: Re: Character equivalence mapping (was: Re: [idn] SLC minutes)

someone registered ?.com and someone else registered w.com there is

tedd

--
http://sperling.com

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 3 Jan 2002 11:04:48 -0800

You do not seem to have read the material I suggested. There was quite a
discussion on this list on why it is hopeless AND counterproductive to try
identify all the characters whose glyphs could be visually indistinguishable
to a user, and create equivalence classes based upon that identification.

Hopeless: For this to be used in IDN, one would have to collect a massive
amount of data; and all at once -- the mappings can't simply change over
time. There are a huge number of questionable cases, where for accuracy one
would have to survey a substantial range of fonts in the common sizes used
on different platforms to determine visual distinguishability. Examples are
quotation dash and the CJK character for "one", the katakana KA and the CJK
character for power, etc. And in many of those cases, one would still have
to make a judgment call, since even if the pixels are always somewhat
different, users may perceive the glyphs as being the same.

Counterproductive: As I noted earlier, the N / V problem arises hundreds of
times -- a lowercase greek nu in common fonts (e.g. Arial Unicode MS) is
visually indistinguishable from a Latin v. That causes N, V, n, v, NU, nu to
be part of the same equivalence class, so a company could not register
NIA.com if VIA.com were already registered.

Although it is certainly "in character" for this list to endlessly (and
pointlessly) repeat earlier discussions, I'd suggest you look at the email
archives on this topic, since this subject has arisen many times. After you
have read them, if you have any new information it might be worth continuing
the discussion.

Mark

P.S. Unfortunately, there appears to be no web interface for viewing the
archive, so it is a bit clumsy to search for particular topics. I think this
topic was discussed at length sometime in 2000, with repeated comments
through 2001.

—————

http://www.macchiato.com

----- Original Message -----
Sent: Thursday, January 03, 2002 10:39

> QUIT
> QUIT
> Message-ID: <[email-masked]>
> From: "Edmon" <[email-masked]>
> To: "Mark Davis" <[email-masked]>
> References: <[email-masked]>
<008b01c1947d$03a4a830$08d8ea0c@c1340594a>
> Subject: Re:
> Date: Thu, 3 Jan 2002 13:48:13 -0500
> MIME-Version: 1.0
> Content-Type: text/plain;
> charset="utf-8"
> Content-Transfer-Encoding: 8bit
> X-Priority: 3
> X-MSMail-Priority: Normal
> X-Mailer: Microsoft Outlook Express 5.50.4522.1200
> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200
>
> Hi Mark,
>
> But I am not suggesting any transliteration or transformation.  I am only
> suggesting that some characters which might be "perceived" or "confused"
as
> equivalent/identical be collected together and regarded as "equivalent"
> during the DNS name matching process within the DNS server.  It should not
> hinder the ability to have different characters or representations send
> different data over the wire and to maintain the original information.
>
> Take for example the current DNS, neither uppercase or lowercase is the
> "primary" case, they are considered equal.  Which means that you can have
a
> domain that is DoMaIn.CoM, as well as a domain that is domain.COM, but
> during the matching process, the DNS will declare that they are the same.
> Not that it is mapped to (or declared to be "same as") domain.com or
> DOMAIN.COM, they are simply considered equivalent.  (at least it is what
its
> supposed to do I think)
>
> So, what is the complexity except for coming up with the exact list?  The
> "exact list" I believe should be a living document and be revised over
time,
> but we need to start somewhere.  It should be relatively uncomplicated to
> come up with a list for Latin-Greek-Cyrillic equivalent characters.  In
> fact, I think I dont mind working on a table that contains ones that could
> be "perceived" as equivalent to start with and allow other people to
> scrutinize on each's equivalence in form.  Or do you know if there is such
a
> list already?
> From: "Mark Davis" <[email-masked]>
> To: <[email-masked]>
> Cc: "Kenneth Whistler" <[email-masked]>
> Sent: Thursday, January 03, 2002 12:00 PM
> Subject: Re:
>
>
> > It is not that simple. I suggest you review the messages on this topic
in
> > the archive for this list, and also read both of the following:
> >
> > http://www.unicode.org/unicode/standard/where/
> > http://www.unicode.org/unicode/reports/tr17/
> >
> > Mark
> > —————
> >
> > Sent: Thursday, January 03, 2002 08:03
> >
> >
> > > QUIT
> > > QUIT
> > > Message-ID: <[email-masked]>
> > > From: "Edmon" <[email-masked]>
> > > To: "Mark Davis" <[email-masked]>
> > > References: <[email-masked]>
> > <00aa01c1940f$ac0fee30$08d8ea0c@c1340594a>
> > > Date: Thu, 3 Jan 2002 11:12:53 -0500
> > > MIME-Version: 1.0
> > > Content-Type: text/plain;
> > > charset="utf-8"
> > > Content-Transfer-Encoding: 7bit
> > > X-Priority: 3
> > > X-MSMail-Priority: Normal
> > > X-Mailer: Microsoft Outlook Express 5.50.4522.1200
> > > X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200
> > >
> > > Hi Mark
> > >
> > > From: "Mark Davis" <[email-masked]>
> > > > and visual confusability would have some very unpleasant
consequences.
> > For
> > > I will argue that <nu> and v have some subtle difference while <ALPHA>
> and
> > > A, <NU> and N,  are truly identical.  Perhaps character equivalence
was
> > not
> > > a good choice of word, lets try Character Identicality.  I think there
> are
> > > some characters that we can truly say that they are "identical" under
> > > scrutiny.  Do you think so?
> > >
> > > Edmon
> > >
> > >
> > >
> >
>
>
>

---

From: "Edmon" <[email-masked]>
Date: Thu, 3 Jan 2002 11:31:22 -0500

Hi John & Ken,

Very Agreeable.

Edmon

---

From: "Edmon" <[email-masked]>
Date: Thu, 3 Jan 2002 17:51:36 -0500

First of all, allow me to say that I am not proposing to have Character
Equivalence preparations in the "protocol".
In fact, I intend this topic to be more "operational", and I have mentioned
that this might not be the best list to discuss this, but I did want to
clarify what I said during the SLC meeting as was noted in the minutes
(which was what started this discussion).  Also, since the interested
parties on this topic is likely going to be on this list too.
I apologize for lingering on the discussion, but allow me to say the
following:

I do believe very much that since this issue sparks so much contention, we
should archive the problem and discussions better, perhaps into an
informational document, pointing out that there are a set of codepoints that
may be perceived as equivalent.  This in fact would include issues for
Latin-Greek-Cyrillic (LGC) characters as well as at least JPCHAR, HangulChar
and T/S Chinese.  Becuase we do not have much information on Arabic and the
other General Scripts, that is why I said that this document should be a
living document.  If it becomes apparent that some Character Equivalence
preparation is beneficial for these scripts, then we can ammend the
document.

The document will contain a discussion on the different issues surrounding
LGC and CJK characters in Unicode respectively, as well as a set of tables
that lists out the equivalent characters.

With this document, a zone operator (whether a local zone manager or a TLD
manager) can choose one of three approaches to deal with the issue (and can
choose to implment the mappings and which mappings to use for their own
zone):
1. do nothing
2. multiple registrations
3. consolidate characters before name matching within the name server

It will alarm me if anyone thinks that this is useless because they are real
issues that an implementor should be aware of.

Edmon

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Edmon" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, January 03, 2002 2:04 PM
Subject: Re: Character equivalence mapping (was: Re: [idn] SLC minutes)

> > References:
<[email-masked]>
> > > > > 2. Moreover, stop and think about the implications; using both
case
> > > > folding

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 4 Jan 2002 13:06:02 +0800

If you mean archive of the IDN list on the web, this is the URL :-)

http://www.imc.org/idn/mail-archive/maillist.html

-James Seng