Subject: [idn] Problems in normalisation and matching

From: Dan Oscarsson <[email-masked]>
Date: Sun, 30 Jun 2002 14:49:10 +0200

After having made a close look at IDNA and stringprep
I see many problems in the handling of characters in domain names.

- In IDNA it says that in domain names more than one dot (full stop)
must be recognized as label separator.
While this might be a natural thing to do in some cases,
it is much cleaner if just U+002E is allowed. It simplifies
parsing a lot and is more like programmes are used to.
Also there are many more dots in UCS that could as well
be used.

In general I can see three basic contexts for domain names:
1) free text
2) standard form in protocols
3) comparing form.

In 1) free form you could write the text any way you like.

In 2) the form must be normalised.
Here only one "dot" should be allowed to separate labels
in a domain name so one one well defined character can
separate labels.
Stringprep/nameprep/idna defines one normalisation that
do not fit to use here. Its NFKC, lower casing and character
mapping does destroy to much of the original name.
NFKC has several mappings, those related to letters
are mostly well but those related to other types
of characters like symbols or accents are doubtfull.
If symbols and accents should be allowed in domain names,
they should only use NFC. NFKC is irregular in its handling
of accents, some are expanded from the compact form and some
are retaind in the compact form.
I would recommend NFC to be used, probably with some of
the elementary compatibility mappings for letters
added (like U+212B to U+00E5 and all fullwidth letters
to standard width).

In 3) I think Stringprep/nameprep/idna goes to far.
Domain names should be compared case insesitivly
using simple case folding instead of full+some Turkish
folding as of IDNA today. For example small letter sharp s
should not be folded into ss.
This would simplify domain name matching a lot and
make it quicker and easier to implement.
( though we need a way to do approximative matching
in programs interacting with the user. In this case
small letter sharp s could be matche to ss, and
other complex matching rules be used. But this
should not be the normal way for DNS ).
Is it good that IDNA goes to far and make people think
this is the way to go? It will make it difficult to
fix later on.

   Dan

---

From: "James Seng" <[email-masked]>
Date: Mon, 1 Jul 2002 08:56:34 +0800

Hi Dan,

I remember the "dot issues" was extensively discussed by the Nameprep Design
Team. It is decided that dots (other than U+002E) should be included because
there are IMEs which generate these dots in place of the normal dots (it
become a hassy to switch in and out of IME just for the dot). Now, some may
say IME is out of scope but on the other hand, we really dont need to rehash
a topic which have been concluded. Lets move forward.

If we can agreed on the above, then the "many problems" you point out are
really just misunderstanding of the Nameprep/IDNA relationships.

First, the Nameprep/Stringprep is designed to handle domain names on a _per
label_ basis. Before some IDNs going thru Nameprep, it is already broken up
into its individual labels so Nameprep arent the place to fixed.

The place where IDNs get broken down into label is in IDNA. What IDNA now
specify is that to break down IDNs into their label, you look for this set
of separators (U+002E, U+3002, U+FF0E, U+FF61). (See IDNA Requirement 1)

Comparison is also done on a per label basis. A IDN is considered equivalent
if and only if all their individual labels are equivalent. The separators
during comparison is also irrelevant. (See IDNA Requirement 4)

If the individual labels need to piece back together into a FQDN, then IDNA
have already clearly specified that U+002E should be used. (See IDNA
Requirement 2)

-James Seng

---

From: Dave Crocker <[email-masked]>
Date: Sun, 30 Jun 2002 19:09:49 -0700

At 08:56 AM 7/1/2002 +0800, James Seng wrote:
>become a hassy to switch in and out of IME just for the dot).

This is a confuses user interface issues with protocol issues.  The IETF 
tries to stay away from user interface standardization, even though domain 
names do have a human representation.

User interfaces must adapt to a wide range of usability issues.  Protocols 
are not supposed to suffer that burden.

It is the job of the user interface to map whatever typing codes it chooses 
to, into the constrained protocol codes.  The theory behind typical 
Internet protocols -- and most other modern protocol standards -- is that 
the world chooses ONE way to do a thing and everyone with other ways maps 
to that one way.

The concern for cut-and-paste is obviously valid, but it is not the job of 
the IETF protocol standards to operate well within a user cut-and-paste 
environment.

>  Now, some may

Introducing user interface issues into a protocol design is a good way to 
impair interoperability, because it adds variability.  That makes the 
protocol not work.

Moving forward is a good idea.  Except when it is moving backward.

>The place where IDNs get broken down into label is in IDNA.

James.  Forgive me, but I do not understand this statement.  IDNs are 
ALREADY series of separate labels.  IDNA does not "separate" a domain name 
into labels.

Note that IDN maintains the same kind of dot separater as the "unaware" 
legacy domain name world, even if it uses multiple choices for the dot 
character.

All IDNA does is to ENCODE those separate labels into a kind of UTF-7 (that 
is, ACE).

>during comparison is also irrelevant. (See IDNA Requirement 4)

One bit of confusion that I did not pursue with my suggested revisions is 
the idea that an IDN can only be compared it its IDNA form.

In terms of formal specification, this cannot be correct.  If there is 
another encoding for IDNs, then the IDNA specification is essentially 
saying that such encodings must be mapped to IDNA, first, in order to do 
comparisons.

If that is what the working group really does mean, it should be stated as 
part of an IDN specifications, separate from the IDNA specification, 
because it is another formal change to the DNS.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: Dave Crocker <[email-masked]>
Date: Sun, 30 Jun 2002 20:03:02 -0700

At 07:09 PM 6/30/2002 -0700, Dave Crocker wrote:
>This is a confuses user interface issues with protocol issues.

and it confuses my typing.  sorry.

The sentence was supposed to read:

         This confuses user interface issues with protocol issues.

d/

----------