Subject: Re: [idn] Determining equivalence in Unicode DNS names

From: Dave Crocker <[email-masked]>
Date: Mon, 14 Jan 2002 13:13:38 -0800

At 11:57 AM 1/14/2002 -0800, Stuart Cheshire wrote:
>It seems to me that one of the great problems of IDN is one that is
>fundamentally unsolvable: an attempt to determine, once and for all
time,
>a single global set of rules for deciding if two strings are "equal" or
>"equivalent".

Is "abc" = "abc"?

Is "aBc" = "AbC"?

Is "abeesee" = "abc"?

These have easy answers, and they depend on well-understood types of
rules
for comparing strings.

The current DNS has rules that make the answers yes, yes, and no,
respectively.  The actual characters used are currently ASCII.  Extend
the
characters to be Unicode, and the same sort of rules can still be used,
albeit with some tuning.

The danger is with trying to solve a broader set of problems, or with
doing
tuning that is more like a major overhaul.

In other words, the task of IDN is entirely solvable, as long as we do
not
define the task to be too broad.

That is, for example, why equivalence between traditional and simplified
Chinese is entirely out of scope for this working group, just as trying
to
define equivalence between arabic and hebrew characters is out of scope.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 17 Jan 2002 03:24:09 +0000

I tried sending the attached message on Monday, but it mysteriously
vanished.  Let's see if it gets through this time.

AMC

Subject: Re: [idn] Determining equivalence in Unicode DNS names
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Date: Mon, 14 Jan 2002 23:14:24 +0000

Stuart Cheshire <[email-masked]> wrote:

> time, a single global set of rules for deciding if two strings are
> "equal" or "equivalent".

You're right that we have no hope of defining a single global set of
rules for equivalence of *strings*.  But we don't need that for IDN.
We merely need a single global set of rules for equivalence of domain
labels.  IDNA defines those rules.

> The problem as I see it, right now, is that if a client asks for the
> address record for "www.pépsi.com." (with an accent), and it gets back
> a DNS reply with an answer giving the address for "www.pepsi.com."
> (without an accent), then the client will ignore the answer.

Indeed, because pépsi and pepsi are two distinct labels.  This is like
today, if a client asks for colour.com, then it will ignore a response
telling the address of color.com.  The server needs to answer the
question that was asked, not some other question that it considers
"close enough".

> If a client tries to look up "www.pépsi.com.", and the "com" name
> servers have been configured to treat "pépsi" as equivalent to
> "pepsi", then they return the answer for "pepsi.com.", and in the
> reply they also include a (programatically generated) DNAME record
> which *tells* the client and any intervening caching resolvers that
> these two names are equivalent:

Assuming DNAME is understood by the caches and clients, you could indeed
do that.  But this does not necessarily yield full equivalence between
pepsi.com and pépsi.com.  In the existing (pre-IDN) model, comparison
of two names for equivalence has never required a DNS lookup.  If
someone has visited pepsi.com and then visits a page containing links to
pépsi.com, will those links be colored correctly?  Probably not.  If an
SSL connection is used, and the certificate uses the name pepsi.com, but
the client used pépsi.com, will the SSL library believe that the server
is authentic?  I doubt it.

AMC

---

From: [email-masked]
Date: Thu, 17 Jan 2002 02:16:42 EST

> question that was asked, not some other question that it considers

For some people, this may not be all that obvious.  We have had discussions 
and read proposals in which it is stated that the relationship between 
"lookalike" characters, like U+0041 and U+0391 and U+0410, or between 
Simplified and Traditional Chinese characters with the same language-specific 
meaning, is no different from the relationship between Latin '"E" and "e".

Presumably there are those who would also see the relationship between "e" 
and "é" in the same way, and would therefore expect "pepsi.com" to be matched 
not only by "PEPSI.COM" but also by "pépsi.com".

There are good and valid reasons for not treating certain classes of 
characters the way Latin case pairs are treated, but ultimately there is 
bound to be a measure of arbitrariness in the system, and people will just 
have to accept the way things are defined.  After all, even traditional Latin 
case pairing gets complicated for languages such as German ("SS" -> "ß") and 
Turkish (where "I" and "i" are not a pair).

-Doug Ewell
 Fullerton, California

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 17 Jan 2002 03:07:36 -0800

--On 2002-01-14 11.57 -0800 Stuart Cheshire <[email-masked]> wrote:

> It seems to me that one of the great problems of IDN is one that is=20
time, =

> a single global set of rules for deciding if two strings are "equal"
or=20
> "equivalent".

In the DNS system, there is a general rule (as you point out) that the
same
matching rule is to be used for a label regardless of from where the
query
is sent, to what server, and in what domain the label exists.

Reason for this is to prevent confusion, and to secure the in DNS needed
global uniqueness.

Example: If a.com and b.com is considered equal, why should not a.se and
b.se be equal? If a.com and b.com is equal, why is not a.a.com and
b.a.com?

Basically, you ask for context specific matching rules, and that is
something which is impossible to handle in the DNS protocol, because the
client and server can not exchange context. Yes, you point out the only
context that is available, the name of the parent domain. But, how is
the
client able to know what rules exists in a specific domain? Further, is
not
the matching rule more depending on what language the registrant of the
domain and the one querying is using than what parent domain the domain
is
registered in? I would as a swedish-speaking person claim that 'a'
followed
by combining diaeresis be a different character than adiaeresis, while
someone using some other language might think they should be equal. The
same way, I would like having the 'o' with slash and the odiaeresis be
considered the same, while others disagree. All of this because of _my_
culture.

So, this wg decided that we will use one and only one matching rule,
just
like we decided to use only one character set. Both of these (the rule
and
the charset) are created in the Unicode Consortium.

You also talk about "evolving matching rules over time". That is a very
bad
thing.

Let's say that we have registered apple.com and =E4pple.com. They are
considered different. One day the equivalence rule change, and they are
considered being the same. Who is calling Steve Jobs saying that
apple.com
have to give back it's domain name and pick something else? You? ICANN?
The
registry for .com?

    paf

---

From: liana Ye <[email-masked]>
Date: Tue, 22 Jan 2002 02:34:21 -0800

On Mon, 21 Jan 2002 09:14:11 +0100 =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?=
<[email-masked]> writes:
> --On 2002-01-20 23.01 -0800 liana Ye <[email-masked]> wrote:
> 
> > The decision of use one and only one matching rule is 
> > at false, because of there is no ONE rule can deal with 
> > hundreds of different scripts no matter how strong it 
> > appears that you defend the stand.  
> 
> The DNS uses only one rule for matching. One charset and one 
> matching rule.
> 
> What you think about is something different than DNS, and that is 
> discussed
> in other forum than this working group.
> 
>    paf
> 

Which forum do you mean?  

The IDN currently is one rule - [nameprep].  One charset UCS.
The DNS is one matching rule and one charset ASCII.

Liana

---

From: liana Ye <[email-masked]>
Date: Tue, 22 Jan 2002 04:18:20 -0800

On Mon, 21 Jan 2002 22:55:34 -0500 John C Klensin <[email-masked]>
writes:
> Liana (and others),
> 
> Let's try to review how we have gotten here, since I partially
> disagree with Patrik (the disagreement leads, however, to the
> same conclusion, only more strongly).
> 
> --On Sunday, 20 January, 2002 23:01 -0800 liana Ye
> <[email-masked]> wrote:
> 
> >> So, this wg decided that we will use one and only one
> >> matching rule, just  like we decided to use only one
> >> character set. Both of these  (the rule and the charset) are
> >> created in the Unicode Consortium.
> 
> Actually, I don't believe this is correct --or that the language
> isn't precise enough-- so let me try to restate it.  The fact
> that there is only one matching rule is a consequence of the
> design of the DNS: largely because of the binary nature of the
> underlying  architecture, the only matching rule that is
> possible involves a bit-wise comparison on each octet in turn.
> Unless the labels are assumed to be "binary" (i.e., the
> case-mapping rule does not apply) that bit-wise comparison is
> made under a mask that collapses ASCII upper-case characters
> onto their lower-case counterparts.
> 

Agree, and this provides the stability of DNS system in the past
years.

> Now, we can map or transform all sorts of things together before
> they get injected into the actual DNS.  But, as far as the DNS
> is concerned, there is only one mapping rule, or two if "binary
> labels" are handled differently from "character labels", but
> that is it.
> 

I am not suggest anyother way, and the  "character labels" are
proposed to be handled the same way with current one 
matching rule in DNS, - NOT two rules.

> Are there ways around this?  Of course there are.  The WG has
> looked at many of them and given up on them long ago.  For
> example:
> 
> (i) One could introduce a completely new set of query types for
> labels that might contain non-ASCII strings, and apply a
> different matching rule for them.  Unfortunately, this would be
> hugely complex -- we are having quite enough problems with the
> three address-type records/queries we have today -- and might
> take forever to deploy.
> 
> (ii) One could introduce a single new label and query type whose
> sole purpose was to provide a new form of alias -- like CNAME
> only with Unicode (encoded somehow) in the label.  The WG looked
> at variants on that theme early on and decided to not pursue
> them.  In hindsight, the approach essentially implies "within
> DNS layering" and has all of the disadvantages of an "above DNS"
> layering scheme and few of the advantages.
> 
> (iii) One could adopt the "new class" model, using that class to
> impose a somewhat different set of matching rules by redefining
> the query types.  That was, you will recall, proposed and
> rejected (or, more accurately, ignored), I think mostly because
> people were concerned about the length of time it would take to
> deploy.
>
> (iv) One could try to use EDNS to specify alternate mapping
> rules or conventions.  Again, the problem is deployment,
> aggravated by some rather complex issues involving caching and
> secondary servers.  And, again, the WG considered some set of
> options in this group and couldn't get consensus around moving
> forward with them.
> 
> One thing that all of these --except, possibly, the last-- have
> in common is that, while they were permit a _different_ matching
> rule, none of them permit per-language or per-script rules.  You
> still get only one (or two).
> 
> The rule doesn't deal with scripts at all.  It doesn't deal with
> languages either.  It deals only with strings of bits and deals
> with them in a mathematically simplistic way.
> 
> > The charset appears in the form of a table, which gives 
> > us a better view of these chars and we have a base 
> > to work with. But when you try to use only ONE rule to 
> > deal with all of the chars, the rule has to be bias for 
> > only one type of script, currently Latin.
> 
> No.  The only "bias" is the ability to use a bit masking
> algorithm for ASCII upper-lower case matching.  There are severe
> and complex problems in dealing with the range of Latin scripts.
> 
> >  If we let this 
> > pass from this IDN wg, then all of us here are arguing
> > for nothing, as the "Prohibit CDN code points" is the 
> > only way out suggested by Kenny Huang.   
>  
> > Isn't the time for the WG to wake up from the "one rule 
> > for all" scope?
> > 
> > The only way out is to use multiple rules to treat multiple 
> > scripts, where each rule is identified by a tag. 
> 
> And where are you going to put the tags?  Again, the WG has been
> around this question many times.  There isn't a logical place to
> put the tags unless one further shortens the fraction of the
> label-length-space available for user-visible information.

The tag should be transparent to the user, and yes, should be
a part of the user-visible label.

> There are difficult problems in doing the tagging from a user
> interface basis.  Language (or even "script") tagging would
> prohibit many combinations of characters for which there is
> clear demand, and the WG has been reluctant or unwilling to
> start making the policy decisions that "one label, one script"
> implies.  
> 

Uncontroled mix of character scripts is dangerous as the 
discussion on the list already shown.  There must be some
types of control of which scripts can be combined which are 
not making sense. "one label, one script" is not accurate 
description of the tagging method.  I rather call it 
"one label, one language", ie. ja- means japanese using 
kana and Kanji two scripts; kr- means korean using 
Hangul and Hanja scripts; vi- means Vietnamese using 
Latin and Han scripts. If some group wants Latin and 
Greek mix, and there is such a language tag in [ISO639], 
then we can provide that.  Like Chinese, Latin or  Armenia 
can be  "one label, one script". Since Latin, Chinese and  
Arabic are used by multiple languages already. 

> The bottom line is that, if you really think you know how to do
> this, we are anxiously awaiting the Internet Draft.  But it
> needs to cover all of those complex cases with mixed-language
> labels and scripts that can't be neatly mapped onto languages
> and vice versa.

If there are enough interest in the idea, I will summit the next
version of idn-map today.

> My own belief is that many of these things require, not a
> complex set of binary matching rules, but enough of a notion of
> distance functions to talk, not about "matching" but about
> similarities and differences.  Cyrillic, Greek, and Latin "A"
> look similar, and are "sort of" the same letter (same origins,
> overlapping pronounciations), but don't "match".  Greek
> lower-case omega "looks somewhat like" Latin "w", but it isn't
> the same letter (even "sort of").  Really getting TC<->SC
> mappings right may require context, distance/probability
> functions, or both (the latter if there isn't enough context).
> We can posit doing those things in other sorts of systems, but
> not in the DNS, even with a broader set of matching rules.
> 
> Now, I know you don't like that answer, and that saddens me.
> But wishing, and even unhappiness, won't change the mathematics
> that underlie DNS matching.
> 
>      john

That is precise the reason we need multiple rules in IDN, so 
that we can keep one rule one charset in DNS.  

Liana