Subject: [idn] The layers and character handling in DNS

From: Dan <[email-masked]>
Date: Sat, 3 Feb 2001 11:18:41 +0100 (MET)

There have been several comments on where to do nameprep and use of
different character sets. If I look at how I would like things to
work, if could be like this:

user enters name in application using LOCAL character set.
->
application calls routnes like: gethostbyname
->
gethostbyname converts form LOCAL character set to 
normalised (not namepreped) UTF-8 (or other format).
it may reject the query if the name contains forbidden characters.
and sends the query to DNS server.
->
DNS server receives the request, takes the UTF-8, compares it
using the namepreped form of the name to the namepreped form of
the names in the database. If it is a caching server if can do
the matching by binary matching.
->
The DNS server sends the response back to the client using
normalised (original form, not namepreped) UTF-8.
->
gethostbyname (or other routine) returns the answer with names
converted into LOCAL character set.
->
application presents answer using LOCAL chaarcter set to the user.

This way you get:
- applications can use local character set just like now.
  They do not have to know about format used for interoperability.
- conversion form local to protocol character set is done in
  resolver APIs.
- the complex name matching is done in authorative DNS servers.

Applications needing to compare to names themselves have to know
how to do name matching using nameprep (this could be implemented in
resolver libraries giving all applications a common API for doing this).
All the applications that do not need to compare names (there are a lot)
need not implement more the simple normalisation of UTF-8 (which need
to be done in all applications using UCS for interoperability, if they
want it to work).

If you want to support IDNs in non-upgraded DNS servers you could
require DNS queries to send the query name namepreped, but the answers
must be in original normalised form. This will unfortunately require
nameprep in all applications (or all resolver code) which is not needed 
otherwise.

   Dan

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 3 Feb 2001 13:27:28 -0800

I updated my online NamePrep/LACE converter to the latest version. It is in

http://www.macchiato.com/unicode/convert.html

Click on the "NP" checkbox to turn on NamePrep. It will fold, normalize, and
check for illegal characters.

I'd only recommend IE5.5 for trying it. On Windows, NN4.76 works but has a
wierd layout. NN6 is broken (but that's not new).

Mark

---

From: [email-masked] (David C Lawrence)
Date: Thu, 15 Feb 2001 04:08:54 -0500 (EST)

Dan writes:
> If I look at how I would like things to work, if could be like this:
> 

How does gethostbyname() know for sure what the character set of its
argument is?

---

From: 김_경석-GIM_Gyeongseog <[email-masked]>
Date: Thu, 15 Feb 2001 21:57:05 +0900 (KST)

> How does gethostbyname() know for sure what the character set of its

i guess that gethostbyname need be extended so that one additional
param indicates the local char set.

김 경석, 부산대 정보 컴퓨터 공학부; 
  KIM Kyongsok/GIM Gyeongseog, Busan National Univ.
  [email-masked], http://HANGEUL.cs.pusan.ac.kr/hangeul/
  Ph: (051) 510-2292, Fax: (051) 515-2208

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 15 Feb 2001 11:02:00 -0800

>param indicates the local char set.

And, with that information, gethostbyname then needs to translate 
completely correctly and authoritatively from that charset to UTF8 
(or at least to ISO/IEC 10646 code points). For this scheme to work, 
we need to have authoritative and non-changing mapping tables for 
every local charset. Even is this was possible (and it's not; see 
below), the software would be huge.

Those of us who have just the easy Latin characters in our names keep 
forgetting that many of the local charsets in the rest of the world 
don't have the mapping tables needed, and won't have them any time 
soon. There is wide dispute about what the mappings should be, and 
even the organizations that have spent the time doing mappings 
*still* are changing them. Part of this is due to the way that 
ISO/IEC 10646 is designed, so it is likely that this issue will not 
be resolved for decades, if ever.

Please, folks: this is a dead end. If you want to keep going down 
this path, maybe preface each message you write with "In an idealized 
universe where there was authoritative and non-changing mapping 
tables for every local charset, ...".

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Thu, 15 Feb 2001 14:31:03 -0500

> tables for every local charset, ...".

  Paul, I'm not sure I see this as anything other than the result of trying to
extend the codepage-based world beyond its useful boundaries.  Isn't this what
Unicode-ISO/IEC 10646-1 is all about?

  Using data in a codepage-based CES outside of that environment is going to
yield mixed results at best, and very poor results in some contexts.  This
should encourage users to migrate to Unicode-based systems and increase
pressure on software vendors to implement and improve such support in their
systems.

  Or am I being naive?

  -bws

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Feb 2001 14:52:31 -0500

so what you're saying is that, at least in some parts of the world,
folks won't be able to use IDNs until they upgrade their platforms
to support Unicode?

Keith

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 15 Feb 2001 11:56:13 -0800

>folks won't be able to use IDNs until they upgrade their platforms

No, not at all. "Platforms" don't need to support "Unicode"; 
applications need to support IDN. The IDN protocol we are working on 
has parts of The Unicode Standard in it.

--Paul Hoffman, Director

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Feb 2001 15:08:22 -0500

> has parts of The Unicode Standard in it.

I might believe that you can *display* IDNs independent of platform support
for Unicode, at least, if you have a pixel-addressable display of sufficient 
resolution, and sufficient storage for Unicode fonts.

whether you can effectively *input* IDNs without platform support for Unicode
is a different question.

Keith

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 15 Feb 2001 12:33:18 -0800

>whether you can effectively *input* IDNs without platform support for Unicode

Correct. Most "platforms" have support for some character sets, each 
of which is a subset of the Unicode (ISO/IEC 10646) character 
repertoire. Because of this, the input mechanism can be used with 
IDN. The "platform" decides the mapping between the character sets 
and Unicode.

The difference here and where this thread was going is that the 
mapping is done by the application (probably through "the platform"), 
*not* by the protocol itself.

--Paul Hoffman, Director

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Thu, 15 Feb 2001 17:26:57 -0500

Paul,

Is this a real requirement?

> every local charset

If real, is it strengthened or weakened by removing the word "mapping"?

I only ask because my experience at Sun and HP, and my peers at IBM and
elsewhere confirm, that historically vendors "cherry picked", going for
ROI (unit sales), prior to any code-set independent (or dependent if one
views the ASCII -> UTF8 substitution as just a code-set substitution)
i18n deployment. Vendor, Consortia, and National "authoritative" tables
were the norm, and were frequently modified.

I'm asking about cherry picking on tables.

Next, or to be confusing and go to your comment, with "mapping on",
previously, I only ask because the resolver supplied by the Chinese
Domain Name System (CDNS) discussed by Wenhui Zhang (let me know if
you haven't seen this mail, I thought it went to idn and provreg),
does convert GBK and Big5 encodings to UTF8. I assume something useful
(for some definition of utility) happens in some other CDNS supplied
service.

I'm asking about cherry picking on mappings.

If economically rational behavior is changed, then my prior experience
with cherry picking in the non-ASCII host system software market is a
limited guide.

Broadly, my question is "impossibility" relevant? Is the only alternative
to one path the one characterized as:

	"In an idealized universe where there was authoritative and
	 non-changing mapping tables for every local charset, ..."

I don't mind looking funny, idealized universii and formal reasoning are
run-of-the-mill for people who enjoy maths, but I'm puzzled by a course
so radically different from the historical provisioning of post-EBCDIC
writing system support, which has most of the CJ (I'm dead ignorant on K)
host market exchanging data like mad, with most traffic in a few encodings.

Incidently, until UCAS [1] went PDAM, about the time I started working on
i18n, my non-latin writing system didn't even have a prevailing encoding.

Thanks for your thoughts Paul,

kitakitamatsino (or U+15BD U+1455 U+15BD U+1455 U+14AA U+165F U+14C4)
Eric

[1] Dirk Vermeulen's work
http://www.egt.ie/standards/sl/n1441-en.html

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Feb 2001 09:09:44 -0500

Paul,

I think this leads to results that violate the law of least
astonishment.

Suppose I'm using local character code X to represent script Z
and so is Keith.  My local mapping is Unicode<-F1(X,Z) and
Keith's is Unicode<-F2(X,Z), and they are different.  Let's
assume that each is consistently reversible (although that is
another problem, both with this issue and, I suspect, with
Nameprep).
 
So I find an IDN domain on the network, and, by the time its name
emerges from my application, it is in X, having been passed
through F1'(Unicode,Z).  I pass X to Keith out of band (e.g., in
an email message with "text/plain; charset=X").  Now Keith looks
it up, applying F2(X,Z) before Nameprep.  And, behold, he either
gets 'nodomain' or, worse, the wrong stuff.

I think consistent behavior is important.  And I don't see how
one gets that without either native implementations of 10646
(i.e., no local character sets) or with standardized/ unambiguous
mappings between each local set and 10646.

Of course, we could try to make Nameprep complicated enough to
smooth over all such details, s.t., Nameprep(F1(X,Z)))=
Nameprep(F2(X,Z)) for all Fn.   Seems unlikely somehow.

     john

--On Thursday, 15 February, 2001 12:33 -0800 "Paul Hoffman / IMC"
<[email-masked]> wrote:

>> whether you can effectively *input* IDNs without platform
>> support for Unicode is a different question.
> 
> Correct. Most "platforms" have support for some character sets,
> each of which is a subset of the Unicode (ISO/IEC 10646)
> character repertoire. Because of this, the input mechanism can
> be used with IDN. The "platform" decides the mapping between
> the character sets and Unicode.
> 
> mapping is done by the application (probably through "the
> platform"), *not* by the protocol itself.
> 
> --Paul Hoffman, Director

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 18 Feb 2001 06:59:49 -0800

>another problem, both with this issue and, I suspect, with

Ok.

>an email message with "text/plain; charset=X").

I think you send "X" as data, where the charset is "Z".

>Now Keith looks

This might happen.

>Nameprep(F2(X,Z)) for all Fn.   Seems unlikely somehow.

The whole goal with the normalization which UTC has done is that the 
number of X and Fn which make Nameprep(F1(X,Z)) != Nameprep(Fn(X,Z)) 
is as small as possible. We can argue until we die how many of these 
X and n exists which break the pattern, and because of this, I think 
this discussion is mostly academic.

BUT: I think the key message John has, which the IDN group have to be 
aware of is that the IDN wg seems to have made a choice that Unicode 
with the normalization rules which are defined by UTC (not IETF) is 
the absolutely best functions that can be found. Further, the 
problems brought up _will_ happen, and we in the IETF can only rely 
on UTC being smart enough to develop normalization tables and do good 
marketing to see that the number of Fn is relatively small for each X.

Now, the big question which John doesn't ask explicitly is whether 
these mappings are good enough, and if the IDN wg is aware of the 
fact that it should have been able to say "no, too many Fn exists for 
too many X, so we don't belive these mappings are good enough".

Personally, I have been thinking of this a lot the last 6-8 months, 
and my conclusion is:

(1) The IETF is NOT a forum where we have the knowledge on making a 
decision whether a mapping function is good enough or not for 
characters. The only thing the IETF can do is to choose someone which 
works in this area and trust them doing an as good job as possible. 
UTC is doing a good job, but they have "bugs" and problems with their 
mapping tables (as anyone would have) and we in the IETF will inherit 
them -- for good and for bad.

(2) Because of the issues John list, or even more important the fact 
that we (amateurs) in the IETF do already know that two code points 
in Unicode which are not normalized to the same value still look the 
same exists, the only way of solving the problem is to (a) have 
special normalization rules (not the UTC ones) in the IETF or (b) 
regardless of IDN try to push all applications to be aware of this 
issue so they start looking at dictionary approaches in places of the 
user interface where misunderstanding can happen. And (a) is NOT a 
path to take due to the argument in (1) above.

So, consistent behaviour is something we will get only if F1 and Fn 
for the same X map to at least two code points which are normalized 
to the same in the nameprep phase -- and we in the IETF _have_ to 
rely on UTC for this. Eventual misunderstandings that can happen (or 
rather, WILL happen) because of non-normalization happening -- for 
example between latin, greek and cyrillic -- will not be solved by 
nameprep.

Or else we should not do this at all.

My personal opinion is that we SHOULD do this, but know about the 
limitations, and regardless of the IDN solution with nameprep 
understand that we need a dictionary _aswell_ because of the 
limitations that exists.

    paf

-- 
Patrik Fältström <[email-masked]>                         Cisco Systems
Consulting Engineer                                  Office of the CSO
Phone: (Stockholm) +46-8-4494212            (San Jose) +1-408-525-0940
        PGP: 2DFC AAF6 16F0 F276 7843  2DC1 BC79 51D9 7D25 B8DC

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Feb 2001 11:16:35 -0500

--On Sunday, 18 February, 2001 06:59 -0800 Patrik Fältström
<[email-masked]> wrote:

>> So I find an IDN domain on the network, and, by the time its
>> name emerges from my application, it is in X, having been
>> passed through F1'(Unicode,Z).  I pass X to Keith out of band
>> (e.g., in an email message with "text/plain; charset=X").

Yes, sorry -- typographical error.

> The whole goal with the normalization which UTC has done is
> that the number of X and Fn which make Nameprep(F1(X,Z)) !=
> Nameprep(Fn(X,Z)) is as small as possible. We can argue until
> we die how many of these X and n exists which break the
> pattern, and because of this, I think this discussion is mostly
> academic.

I wouldn't try to make a quantitative argument.  However, I
suggest that, if there are _any_ major "local" character sets
that have ambiguous and/or non-reversible mappings to 10646, then
we are in trouble with these sorts of algorithms.  I would define
"major" in terms of either the size or importance of the
potential user group: if we are in trouble with Chinese,
Japanese, Korean, or even with Latin-based alphabets with
different composition rules in different
interpretations/mappings, then I suggest that we are in big
trouble indeed.

> BUT: I think the key message John has, which the IDN group have
> to be aware of is that the IDN wg seems to have made a choice
> that Unicode with the normalization rules which are defined by
> UTC (not IETF) is the absolutely best functions that can be
> found. Further, the problems brought up _will_ happen, and we
> in the IETF can only rely on UTC being smart enough to develop
> normalization tables and do good marketing to see that the
> number of Fn is relatively small for each X.

I think this position works iff:

(i) There are _zero_ nameprep mapping rules or discounted
characters not specified in Unicode (and preferably ISO)
standards.   If we need a many-paged Nameprep document for any
purpose other than to provide a stable version of a UTC spec --I
consider it a problem if we need even that, but it is another,
and arguably separate, problem-- then the "hope UTC gets it
right" story fails.  A single IETF-specific rule, whether for
URIs, funny characters, or anything else, IMO kills this story
and turns it all into an IETF problem.

(ii) We have a really good versioning model so that someone
applying Nameprep can know what rules to use.  If
NamePrep2001(string) != NamePrep2002(string) and one can't tell
which one should be applied, I think that, long-term, the whole
thing deteriorates badly.   Given an obvious variation of the F1
!= F2 argument, I suspect (but haven't worked through the cases
to my satisfaction yet) that means that the relevant version
information needs to be encoded in the names themselves.  I.e.,
the real test is whether
  NamePrepN(F1(X,Z))=NamePrepM(F2(X,Z))   and whether
  M and N can be determined in all places in which it is relevant
(which would be especially important if the equality doesn't
apply --it won't, or one wouldn't need NamePrep versions-- and
one needs to know which version to apply.

> Now, the big question which John doesn't ask explicitly is
> whether these mappings are good enough, and if the IDN wg is
> aware of the fact that it should have been able to say "no, too
> many Fn exists for too many X, so we don't belive these
> mappings are good enough".

Since I don't believe that, if things don't work, users will
forgive us if we just say "don't blame us, it is UTC's problem" I
tend to ask a higher-level question.  And I'm getting
increasingly dissatisfied with the answer I get.

> Personally, I have been thinking of this a lot the last 6-8
> months, and my conclusion is:
> > (1) The IETF is NOT a forum where we have the knowledge on
> making a decision whether a mapping function is good enough or
> not for characters. The only thing the IETF can do is to choose
> someone which works in this area and trust them doing an as
> good job as possible. UTC is doing a good job, but they have
> "bugs" and problems with their mapping tables (as anyone would
> have) and we in the IETF will inherit them -- for good and for
> bad.

Agreed.  But that implies, as above, that we make _zero_ of our
own rules.  If the Nameprep document is more complex than "take
the UTC rules and apply them" (perhaps in some specific order
relative to other things that need doing), I think we are out of
this space and into applying our own judgement... we I think we
agree we should not do.

> (2) Because of the issues John list, or even more important the
> fact that we (amateurs) in the IETF do already know that two
> code points in Unicode which are not normalized to the same
> value still look the same exists, the only way of solving the
> problem is to (a) have special normalization rules (not the UTC
> ones) in the IETF or (b) regardless of IDN try to push all
> applications to be aware of this issue so they start looking at
> dictionary approaches in places of the user interface where
> misunderstanding can happen. And (a) is NOT a path to take due
> to the argument in (1) above.

Indeed, that is the conclusion I keep reaching.

> So, consistent behaviour is something we will get only if F1
> and Fn for the same X map to at least two code points which are
> normalized to the same in the nameprep phase -- and we in the
> IETF _have_ to rely on UTC for this. Eventual misunderstandings
> that can happen (or rather, WILL happen) because of
> non-normalization happening -- for example between latin, greek
> and cyrillic -- will not be solved by nameprep.

And the alternative, of course, is to move these issues into an
environment in which the ambiguities are tolerable or can be
passed back to the user.  That isn't the DNS.

> > My personal opinion is that we SHOULD do this, but know about
> the limitations, and regardless of the IDN solution with
> nameprep understand that we need a dictionary _aswell_ because
> of the limitations that exists.

And I've gotten a bit more pessimistic and --faced with
approximate agreement on the facts and first-level conclusions--
am getting close to concluding that

   * we should _not_ do this at all
      * we should press forward with a directory approach as the
   only one likely to solve the problem.
      * we should ask ourselves whether, if a directory approach is
   needed anyway, it makes sense to add this much complexity to
   the DNS when the real work will need to be done at another
   level.

And, coming back to the ongoing Last Call at a 10,000 meter
level, if the Requirements document doesn't draw out these types
of issues and at least point to the need for solutions or
explicit decisions, it isn't, IMO, ready for publication.

     john

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 18 Feb 2001 08:30:20 -0800

>this space and into applying our own judgement... we I think we

I agree with this in principle, BUT, we need still some extra text.

This because we might want to

- prohibit some code points as part of a label (compare with the 
problems with having no definition of what is a hostname compared 
with definition in DNS protocol to be able to handle decimal values 
of 0-255 (inclusive) as part of a label).

- specify what should happen with non-assigned characters (which 
solves the versioning problem).

- specify in what order case folding, normalization etc should happen.

    paf

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Feb 2001 12:09:27 -0500

--On Sunday, 18 February, 2001 08:30 -0800 Patrik Fältström
<[email-masked]> wrote:

>> relative to other things that need doing), I think we are out
>> of this space and into applying our own judgement... we I
>> think we agree we should not do.
> > - prohibit some code points as part of a label (compare with
> the problems with having no definition of what is a hostname
> compared with definition in DNS protocol to be able to handle
> decimal values of 0-255 (inclusive) as part of a label).

I think this takes us well out on a slippery slope.  And much
further out if the rules/ prohibitions are not _extremely_ short
and simple.

> - specify what should happen with non-assigned characters
> (which solves the versioning problem).

Solves the versioning problem iff you believe that we will get it
exactly right, for all currently-assigned characters, the first
time.   Watching the Nameprep discussions over the last few
months does not lead me to be optimistic about this.

> - specify in what order case folding, normalization etc should

Yes, this I agree with.  But this one can be done as a few simple
rules or not at all.  And we had better be able to enumerate
"etc" and not need to add anything to that list in the future, or
we will have another versioning problem.

    john

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 18 Feb 2001 21:28:18 -0800

>further out if the rules/ prohibitions are not _extremely_ short

Yes, it is a slippery slope, but we already do the same thing when we 
use only ASCII in protocols. Now when having many more characters to 
pick from, we have to do the same kind of selection. I don't see how 
we suddenly can _not_ do it, and for example allow '/' (ascii slash) 
as a character which is allowed. I.e. we have to at least have the 
same ascii characters still being forbidden in textual based 
protocols in applications as we have today.

>months does not lead me to be optimistic about this.

Two different issues:

(a) UTC claims that assigned code points will not change. So, we 
either have to trust them or not. Yes, this is strong wording, but it 
was intentional.

(b) People in IETF don't understand that the IETF do _NOT_ do 
normalization. My job as liason (and some others) is to see that the 
IETF does not do anything which where we instead reference UTC.

So, some people bring up "bugs"/"issues" or whatever we should call 
what is brought up on the mailing list.

I will strongly oppose personally to any such mappings/fixes ending 
in any document from the IDN wg.

>we will have another versioning problem.

Yes. I agree.

   paf

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 Feb 2001 17:08:16 +0100

>one needs to know which version to apply.

actually versioning nameprep does not work, for any case but the one where 
a string illegal in NameprepM is now legal in NameprepN (M < N).

for this case, one can assume that 1) the name is installed using NameprepN 
(since M would disallow it) and that 2) the owner doesn't care that he is 
not reachable under NameprepM (or he would have chosen another).

For the case of NameprepM <> NameprepN, we can store the info on which name
to use:
- in the human-readable string. Will be pelted with rotten bananas.
- in the DNS entry for the name. But since we can't get to the DNS-stored 
entry before we have done nameprep, the chicken-and-egg conundrum strikes 
again.
- in the global /hosts.txt file?????

For any existing name, Nameprep can't change.

--
Harald Tveit Alvestrand, [email-masked]
+47 41 44 29 94
Personal email: [email-masked]

---

From: John C Klensin <[email-masked]>
Date: Mon, 19 Feb 2001 18:28:39 -0500

--On Monday, 19 February, 2001 17:08 +0100 Harald Alvestrand
<[email-masked]> wrote:

> actually versioning nameprep does not work, for any case but
> the one where a string illegal in NameprepM is now legal in
> NameprepN (M < N).

That has been my impression.  But, if it is the case, it implies
that the first, and every subsequent version, of nameprep must be
perfectly correct for any unbanned characters.  Sorry, but being
certain about that...   Well, the WG just doesn't appear have the
level of divine guidance needed to be that confident; we need to
design things on the assumption that we will need versions
because we got something wrong and it turned out to be important.
If we can't, then the system fails.

> - in the human-readable string. Will be pelted with rotten
> bananas. - in the DNS entry for the name. But since we can't
> get to the DNS-stored entry before we have done nameprep, the
> chicken-and-egg conundrum strikes again. - in the global
> /hosts.txt file?????
> 

Again, requires perfect wisdom.  Since we don't have that, I
think you are adding to my argument that the current approach
just cannot work.

    john

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 19 Feb 2001 17:16:41 -0800

>think you are adding to my argument that the current approach

We don't do the algorithms in nameprep. UTC does.

I claim they are better than we (as you say we don't have knowledge 
about these things) and that we don't have any other choice than 
pointing at them. Have anyone done a better job than UTC regarding 
normalization and case folding and creation of one charset? If so, 
the discussion is interesting whether we should go with UTC or 
someone else, depending on how many bugs there is in their tables.

NOT doing this solution and only a dictionary solution (something I 
think we need regardless) means staying with the subset of ascii 
accepted as "hostnames" for a very very long time. I don't see the 
market wanting to wait for that -- which includes changing all 
protocols, and all applications.

    paf