Subject: host names and nameprep (was: Re: [idn] IRIs ought to use internationalized *host* names)

From: Martin Duerst <[email-masked]>
Date: Thu, 04 Apr 2002 16:52:13 +0900

Hello Adam,

Sorry for the delay.
I'm splitting my answer into two. This one is on the
host name vs. domain name question.

At 03:30 02/03/27 +0000, Adam M. Costello wrote:
>James Seng/Personal <[email-masked]> wrote:
>
> > The discussion of the how URL is to be encoded and how Host: field are
> > to be handled is probably more relevant so lets get back to that.

Just to make sure that I don't get something wrong:

- Domain names are whatever can be used on the lookup side
   of a dns query. This includes all kinds of current and potential
   uses besides the core use that people are usually equating with
   the DNS.

- Host names are the names of machines. They are a subset of
   domain names, used in certain queries/records (e.g. A record).

>Okay.  Eventually this message will arrive at the following proposal:
>
>     Proposed repertoire for internationalized *host* labels:  All
>     characters in classes L (letter), M (mark), and N (number) are
>     allowed, and U+002D (hyphen-minus) is also allowed.  Everything else
>     is forbidden.

This is a very good first shot. There are some things that have
to be carefully checked, e.g. do some M (marks) have to be excluded,
or should some signs corresponding to the hyphen-minus be allowed.
Two examples I know would be the zero-width space which could be
desirable for Farsi, and the (idographic) middle dot, for which
several people in Japan have complained that it's not available
in XML names.

>Which characters should be allowed in internationalized host labels?
>This is an interesting question in its own right, and it's possible that
>the IESG will demand an answer.

>Notice that there is no conflict with Nameprep, because Nameprep does
>not prohibit any characters in classes L, M, or N.

I guess that if there were a conflict, the host names would
just have to satisfy conditions on both sides.

>If we were to adopt this definition of internationalized host name, it
>would best be understood as an amendment of ToASCII step 3 (which checks
>host name restrictions if applicable), tightening substep 3a from:
>
>          (a) Verify the absence of non-LDH ASCII code points; that is,
>              the absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.
>
>to:
>
>          (a) Verify that the sequence contains only host code points;
>              that is, U+002D (hyphen-minus) and code points classified
>              as L (letter), M (mark), or N (number).  See appendix ? for
>              an enumeration of host code points.
>
>Or maybe the enumeration would go in Nameprep, or in a separate document
>that defines internationalized host names.

Looking back on when working on nameprep as a member of the design
team, I think the distinction between host names and domain names
wasn't clear, at least to me, and probably to several other
participants. At some point, I started to worry that having all
the symbols allowed might not have been the best choice. Of course,
if it's for domain names, then that's a bit different.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 04 Apr 2002 17:14:13 +0900

Hello Adam,

Sorry for the delay.
I'm splitting my answer into two. This one is about what
the IRI spec should say.

At 03:30 02/03/27 +0000, Adam M. Costello wrote:

>The IRI proposal (draft-masinter-url-i18n-08) calls for the host labels
>to be ASCII LDH only, just like in URIs.
>
>When converting an IRI to a URI, you have to convert the path components
>from the local charset to Unicode, then do Unicode normalization, UTF-8
>encoding, and %-escaping.  But you don't do anything to the host labels
>because they're already LDH.
>
>I suspect that the reason the IRI proponents don't internationalize the
>host field is that they don't yet have an official IDN spec to point at.
>When they do, I suspect they'll want to revise their proposal so that
>the host field can use the local charset.

This is very close, but it's actually a tiny little bit more
complicated.

The IRI proposal is based on defining various important aspects
of IRIs in terms of their mapping to URIs. This way, the IRI
proposal doesn't have to deal with questions such as "what's a resource".
The mapping is completely uniform, based on UTF-8 and %HH.
There are strong reasons for keeping it uniform, because you
can't really look into URIs (and therefore IRIs) in the general
case.

So there are three pieces:

1) Extending URIs to use non-ASCII characters (i.e. IRIs)

2) Extending the host name part of URIs to use %HH
    (in some browsers, that already works, in others, it doesn't,
     for ASCII; try e.g. http://www.w%33.org in a few browsers)

3) Extending IRIs to use non-ASCII characters in hostnames

The way the drafts are currently structured, it's:

draft-masinter-url-i18n-08: 1)

draft-ietf-idn-uri-01: 2) and 3)

It would be more straightforward to have it as follows:

draft-ietf-idn-uri: 2)

draft-masinter-url-i18n: 1) and 3)

The main advantage would be that people implementing IRIs
find everything to start in the same place, and don't have
to do the implementation in two stages.

>This raises the question of what characters should be allowed in host
>labels.  Since URIs do not allow arbitrary ASCII labels, only host
>labels restricted to LDH characters, one would expect, analogously,
>that IRIs would not allow arbitrary IDNs containing the exotic symbols
>and punctuation allowed by Nameprep, but would allow only host labels
>restricted to a selected set of characters,

There are two questions:

a) What are the allowed host names (see my other mail)
b) What should the URI/IRI specs say about it

The URI/IRI spec should not be the place where the syntax of allowed
host names is defined. The generic URI syntax currently has a very
careful definition of the domain name syntax, but as far as I am
aware, it is just a copy from somewhere else. And I think it was
just done that carefully because some of it was needed and it was
then easier to just do it all. I'm coping the [email-masked] list to
get information on this point.

With IRIs, whatever the syntax of internationalized host names, it
will be much larger, and too big to fully copy it to the IRI spec.
Also, because new characters will be added to Unicode, it's a moving
target, and it's better to put it just in a single spec that can
then be updated.

draft-masinter-url-i18n-08 currently has a general clause to make
clear that even if the IRI syntax by itself allows quite a lot,

2.3 Mapping of IRIs to URIs

...

This mapping has two purposes:

   a) Syntactical: Many URI schemes and components define additional
      syntactical restrictions not captured in Section 2.2. Such
      restrictions can be applied to IRIs by noting that IRIs are only
      valid if they map to syntactically valid URIs. This means that
      such syntactical restrictions do not have to be defined again
      on the IRI level.

Of course, if there is something specific to point to for
host names, we would be very glad to actually do that.

>Getting back to IRIs and URIs:  I propose that conversion of an IRI
>to URI involve applying ToASCII to each host label.  This would allow
>conversion of any IRI to a URI without changing the syntax of URIs.  In
>contrast, the method proposed in draft-ietf-idn-uri-01 would change the
>URI syntax.

As said in another thread, this is not easy at all (i.e. impossible).
But let's continue the discussion of that aspect in the other thread.

Regards,   Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: Martin Duerst <[email-masked]>
Date: Thu, 04 Apr 2002 18:53:51 +0900

Some more explanations:

Please note that the goal of draft-ietf-idn-uri-01
(and of draft-masinter-url-i18n-08) is not to have
more and more URIs with lots of %HH in the host name
part. Indeed, ideally none would ever show up anywhere.

The idea is to have as much as possible IRIs only.
When they are resolved, they should be resolved directly,
i.e. an IRI resolver wouldn't need to produce the %HH
URI representation, it should do toASCII directly as
soon as it has identified a host name.

In the current drafts, that's not yet clear, but I plan
to make this clear in the next round.

Of course, that can't be realized overnight. So I think
it's perfectly acceptable for people to choose to put
an ACEd hostname into an IRI slot as long as they don't
want to risk the chance that it doesn't get resolved
otherwise.

So the work on IRIs and URIs with these two drafts is
just part of what many people have been talking about,
namely that we have to look at every IETF spec and see
how it can be upgraded to deal with IDNs. Every upgrade
will take some time. But if we don't write a spec,
receiver-side implementers won't know what to prepare for,
and therefore, the sender side will never be able to
move forward.

Regards,  Martin.

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 04 Apr 2002 10:23:27 -0600

Martin Duerst wrote:

> participants.

That has been my view from the outside as well, and it has resulted in a
split-the-baby approach which serves neither usage. The most likely
outcome IMO is that nameprep will be the first document in the series to
get replaced.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 9 Apr 2002 06:05:32 +0000

Martin Duerst <[email-masked]> wrote:

> - Domain names are whatever can be used on the lookup side of a dns
>   query.  This includes all kinds of current and potential uses
>   besides the core use that people are usually equating with the DNS.

Right.

> - Host names are the names of machines.  They are a subset of domain
>   names, used in certain queries/records (e.g. A record).

Basically, yes, although there is at least one instance where host names
are not names of machines:  RFC 821 (SMTP) uses host names (not general
domain names) in mail addresses, even though the host names might refer
only to MX records, not A records.

So I would say a "host name" is any name conforming to a particular
syntax whose primary purpose is to name machines on the internet, but
which can also be used for naming other things.

> >    allowed, and U+002D (hyphen-minus) is also allowed.  Everything
> >    else is forbidden.
>
> This is a very good first shot. There are some things that have to
> be carefully checked, e.g. do some M (marks) have to be excluded, or
> should some signs corresponding to the hyphen-minus be allowed.  Two
> examples I know would be the zero-width space which could be desirable
> for Farsi,

But that would mean allowing a white-space character in host names.
There are *lots* of contexts where white-space is a delimiter.  Is there
any non-space character that could serve this purpose for Farsi?

> and the (idiographic) middle dot, for which several people in Japan
> have complained that it's not available in XML names.

While I have no doubt that the katakana middle dot would get used in
programming language identifiers in much the same way as underscore,
I suspect that in domain names it would be even less popular than
hyphen-minus.  In Latin domain names, hyphen-minus is rarely used, and
Japanese readers are much more accustomed to the lack of word boundaries
than readers of Latin-based languages.

One of the nice things about host names is that they're often easy to
remember and guess, because they're so constrained.  Adding more choices
for punctuation characters would erode that advantage.

AMC

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 9 Apr 2002 06:38:34 +0000

Martin Duerst <[email-masked]> wrote:

> The IRI proposal is based on defining various important aspects of
> IRIs in terms of their mapping to URIs.

A very good approach.

> The mapping is completely uniform, based on UTF-8 and %HH.  There are
> strong reasons for keeping it uniform, because you can't really look
> into URIs (and therefore IRIs) in the general case.

Okay, but you can easily distinguish generic-syntax [UI]RIs (which
always have a slash just after the scheme:) from all other [UI]RIs
(which never have a slash just after the scheme:).  And you can find the
host field of any generic-syntax [UI]RI.

Let me refine my proposal:  To convert IRIs to URIs, you are welcome to
use UTF8-NFC-%HH wherever %HH is allowed (which is almost everywhere).
But in the host field of generic-syntax URIs, where %HH is not allowed,
use ToASCII.  Optionally, if you know the syntax of the IRI scheme and
you know where the domain names are, you can use ToASCII instead of
UTF8-NFC-%HH to increase the chance that the recipient will not choke on
the name.

Notice that unlike http://www.w%33.org/, mailto:user@dom%61in is valid,
and any browser that doesn't already understand it is broken (which is
probably a lot of them, unfortunately).

>     for ASCII; try e.g. http://www.w%33.org in a few browsers)

www.w%33.org does not work in Netscape 4.77 for Linux, nor in Mozilla
0.9.8 for Linux, nor in w3m 0.3 for Linux, nor in lynx 2.8.4rel.1 for
Linux.  It does work in Konqueror 2.2.2 for Linux.

The idea of having two concepts, IRIs and URIs, with every IRI being
equivalent to some URI, is a good idea, because it allows us to add
internationalization without changing the URI syntax and without
breaking any existing software that continues to use that syntax.  It's
the exact same technique being employed by IDNA.

So it would be a shame to change the URI syntax unnecessarily.

AMC

---

From: Roozbeh Pournader <[email-masked]>
Date: Tue, 9 Apr 2002 16:21:29 +0430 (IRST)

> any non-space character that could serve this purpose for Farsi?

I had a lot of debates with Persian experts here in Iran about ZWNJ in
domain names. There are four possible scenarios:

1. Forbid ZWNJ: This means you can't write many words in a readable way,
the best example being the word "daanesh-aamooz" (student), written "Dal,
Alef, Noon, Sheen, ZWNJ, Alef with Madda Above, Meem, Waw, Zain".

2. Respect ZWNJ: Which makes two domain names that look equal in every way
resolve to different hosts. For example, my first name, written "Reh, Waw,
Zain, Beh, Heh", will be displayed exactly the same way if we insert a
ZWNJ between the first two letters, according to the definition of ZWNJ in
Unicode.

3. Ignore ZWNJ: Sadly, you should then treat names with ZWNJ the same as
those without it. There exist examples which will have different meanings
with or without ZWNJ, or with changing the place of ZWNJ. One such example
is "Noon, Alef, Meem, Heh, Alef, Farsi Yeh". Without a ZWNJ, it translates
to "names"; with a ZWNJ between Heh and Alef, it means "a letter".

4. Treat ZWNJ in an intelligent way: The implementation should ignore ZWNJ
if it doesn't make a visual difference. So it is ignored between two
letters that don't join, but respected in other places. This will require 
the domain resolver to be able to do some Arabic contextual shaping.

My personal preference is 4, 3, 2, 1. Latest nameprep does 3.

I'm reading this out of context, so just ask for any specific details,

roozbeh

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 9 Apr 2002 19:59:43 +0000

Roozbeh Pournader <[email-masked]> wrote:

> domain names.

Oh, I thought we were talking about zero-width space.  Zero-width
non-joiner is not a space character (class Zs), but a formatting
character (class Cf).  Although ASCII host names do not allow characters
of broad-class C, the only such characters that exist in ASCII are
control characters (class Cc).  So it's not obvious whether any/some/all
characters of class Cf should be allowed in host labels.

Let's see what nameprep does with characters of class Cf...

prohibited:

     070F        SYRIAC ABBREVIATION MARK
     180E        MONGOLIAN VOWEL SEPARATOR
     200E..200F  [BIDI FORMATTING]
     202A..202E  [BIDI FORMATTING]
     206A..206F  [SWAPPING/SHAPING]
     FFF9..FFFB  [INTERLINEAR ANNOTATIONS]
    1D173..1D17A [MUSICAL SYMBOLS]
    E0001        LANGUAGE TAG
    E0020..E007F [TAGGING CHARACTERS]

mapped out:

     200C        ZERO WIDTH NON-JOINER
     200D        ZERO WIDTH JOINER
     FEFF        ZERO WIDTH NO-BREAK SPACE

unassigned:

     2060        WORD JOINER
     2061        FUNCTION APPLICATION
     2062        INVISIBLE TIMES
     2063        INVISIBLE SEPARATOR

kept:

     06DD        ARABIC END OF AYA

What makes 06DD special?  It was class Me in Unicode 3.1, and became
class Cf in Unicode 3.2, but nameprep is based on Unicode 3.1.

Anything prohibited by nameprep should also be prohibited in host
labels.  The question is which, if any, of the Cf characters allowed
by nameprep (U+06DD, U+200C, U+200D, U+FEFF) should be allowed in host
labels.

AMC