Subject: Re: [idn] Fast nameprep vs. slow nameprep

From: Harald Alvestrand <[email-masked]>
Date: Thu, 01 Feb 2001 02:14:33 -0800

At 09:22 30/01/2001 +0000, D. J. Bernstein wrote:
>There are five possible approaches under discussion:
>
>    (1) UTF-8 with fast nameprep. Names in configuration files are good:
>        KC-normalized, no confusing characters, no uppercase characters.
>        (Except uppercase ASCII, for compatibility.)
>
>        The keyboard interface has to help users type good names. No
>        changes are needed in qmail, djbdns, and many other programs. A
>        few programs with 8-bit bugs, notably Sendmail, need to be fixed.
>
>        Patrik and Keith refuse to acknowledge this approach.

I think you mean by "fast" what I mean when I say "early", and by "slow" 
you mean what I mean when I say "late".
I also think you mean by "refuse to acknowledge" what I mean when I say "do 
not believe that it is workable".

And when you say "Patrik and Keith", you mean the people who constituted 
the rough consensus at the San Jose IETF, and those who share that rough 
consensus on the list.

I continue to not believe that just-send-UTF-8 is workable.

--
Harald Tveit Alvestrand, [email-masked]
+47 41 44 29 94
Personal email: [email-masked]

---

From: Dave Crocker <[email-masked]>
Date: Fri, 02 Feb 2001 22:35:37 -0800

>I continue to not believe that just-send-UTF-8 is workable.

It is approximately the 10th anniversary of the beginning of the effort to 
make Internet mail support "international character-sets".

So I suppose it is fitting that we get to be re-introduced to the "just 
send the raw data" argument that was held so fondly by some.

Happily, they did not win the debate then, but it is curious that the 
lessons from that earlier exercise seems to be unlearned by some.

d/

---

From: "D. J. Bernstein" <[email-masked]>
Date: 3 Feb 2001 09:00:05 -0000

> make Internet mail support "international character-sets".

It has been nearly twenty years since people realized how stupid it was
to mangle 8-bit mail. But there is still no IETF specification saying
that it's bad for Sendmail to destroy bytes 128 through 159 in the
header of a message.

In fact, in the world according to Crocker, it's perfectly acceptable
for a new MTA to prohibit 8-bit text in the _body_ of a message!

Of course, such an MTA will fail to interoperate with hundreds of
thousands of machines running qmail, Exim, Sendmail with the 8 flag,
etc. But Crocker says he's right and they're all wrong.

What I find most amazing is that Crocker keeps claiming success. His
own MTA (MMDF) has withered and practically died, along with all its
commercial spinoffs; his rules are widely disobeyed by popular MTAs, as
demanded by the users and system administrators; and he still doesn't
get the message. He is out of touch with reality.

Back to 8-bit headers: We wouldn't be faced with Sendmail's 8-bit bugs
today if the IETF had required 8-bit-clean software ten years ago. But
the IESG followed Crocker's approach. Now he wants us to continue down
the same path, extending this disaster for another ten years. And _he_
says that _I_ am not learning the lessons of history!

---Dan

---

From: [email-masked]
Date: Sat, 3 Feb 2001 05:05:29 -0500 (EST)

> that it's bad for Sendmail to destroy bytes 128 through 159 in the

Ok, given that:
  IPv6 will requires changes at all of the same layers as IDN from
  application to network

    http://127.0.0.1/
    http://127.1/
    http://0:0:0:0:0:0:0:1/
    http://::1/

  I know of no production IPv6 networks
    http://www.ipv6.org/v6-www.html

  Production use of IPv6 will take another 5-20 years
    Ask your ISP if you can connect to them via IPv6
    I doubt they even have IPv6 address space.

Because of this I propose the following idea:

IDN for IPv6 should be implemented in a raw unicode character set aka
just do 8-bit. Until then *ACE or some other solution can be
designed and implemented. 

-Bill

---

From: vint cerf <[email-masked]>
Date: Sat, 03 Feb 2001 09:04:02 -0500

worldcom is happy to provide IPv6 production service
on its vBNS+ network. We do have IPv6 address allocations.

vint cerf

---

From: Rick H Wesson <[email-masked]>
Date: Sat, 3 Feb 2001 12:19:39 -0800 (PST)

Dan,

your conversation about 8bit SMTP is completely out of scope for this
list/wg.

please, keep on topic and focused on idn issues.

-rick

---

From: "D. J. Bernstein" <[email-masked]>
Date: 4 Feb 2001 05:54:34 -0000

> your conversation about 8bit SMTP is completely out of scope for this

If we have UTF-8 IDNs, then they will be transmitted in mail messages,
both in the header (e.g., From lines) and in the body (e.g., responses
from automated mailing-list managers). Many clients send 8-bit text as
is, labelled as 8-bit MIME or not at all. Many MTAs do just-send-8.

It is clearly within the scope of IDN to consider the interoperability
issues here---as illustrated by the Sendmail entry in my list of
upgrades necessary for UTF-8. It is also sensible for us to review the
Quoted-Printable disaster and evaluate its similarity to ACE.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Sun, 04 Feb 2001 02:11:01 -0500

> is, labelled as 8-bit MIME or not at all. Many MTAs do just-send-8.

yes, some leakage will occur.  and many UAs and MTAs will mung these in 
such a way that it becomes impossible to reply to such messages.

> upgrades necessary for UTF-8. 

of course.  after all, it's this very set of interoperability isssues
that makes ACE solutions so compelling.  

> It is also sensible for us to review the

I suspect you're confusing quoted-printable with RFC 2047 and its
predecessors.

but if one is going to "review" the experience from that decision,
(which I agree is relevant, and we've already done some of that)
then one also needs to compare it with the experience of just-send-8.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 04 Feb 2001 11:35:03 -0800

> > It is clearly within the scope of IDN to consider the
> > interoperability issues here---as illustrated by the Sendmail
> > entry in my list of upgrades necessary for UTF-8.
> 

I'm not taking sides in the debate but I would like to point out that by
not making forward compatibility an explicit extension, the use of ACE and
a legacy-friendly message practically guarantees that problems with mixed
encodings will go up. Legacy apps and resolvers that encounter UTF8 names
in documents and links will use the UTF8 domain names even though they
should not do so. As IDNs are promulgated, this problem will get worse,
not better.

Just as a statement of fact, it really would be *easier* (not better, but
easier) to embrace UTF8 directly since there is nothing that prevents UTF8
from being used in queries today except the hostname limitations. Just as
a simple measurement of where implementations are today, UTF8 mostly works
while ACE does not work at all. This means that we are *closer* to UTF8
everywhere than we are to ACE everywhere. This is a simple statement of
fact without examining the technical merits of the solutions, I am not
taking sides in the debate but instead am trying to make simple
observations plain.

In short, ACE actually makes it easier for bad things to happen, since it
does not mandate new behavior. In the end, going with ACE translates
directly into more apps trying to use UTF8, not less. This is not a
compelling (in a good way) feature.

FWIW, these same problems exist if just-send-UTF8 is used. Both ACE and
just-send-UTF8 suffer the same inheritance problems with legacy apps and
resolvers. For example, UTF8-encoded names don't necessarily signify that
the name has been nameprep'd, so it is just as easy for a UTF8 lookup to
send bad data as it is for an ACE system. As such, they both have
tremendous problems.

[Personally, if there were a vote tomorrow I would go for the EDNS
approach, for a variety of reasons, but primarily because it requires
explicit forward compatibility and solves a bunch of other problems along
the way.]

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Keith Moore <[email-masked]>
Date: Sun, 04 Feb 2001 20:21:21 -0500

> should not do so. As IDNs are promulgated, this problem will get worse,

I am not sure I agree with the last sentence, but it's clear to me that
even if we adopt ACE

(a) leakage of non-ACEd UTF-8 IDNs will occur
    (as well as leakage of IDNs in other charsets), and
(b) it's desirable for applications protocols to define what should 
    happen when they do encounter such names.

And while I could argue that applications that detect such names should 
signal these as error conditions (on the grounds that the earlier 
in the signal path such errors are detected, the more likely they are
to be corrected), realistically I suspect that there will be tremendous
pressure on most applications to "fix things up later".  

At least UTF-8 stings of more than a few characters can generally be 
detected even without a label, so there's less likely to be a 
misinterpretation of a UTF-8 IDN than of an IDN in some other 
charset without a label.

So for example in email, even though ACE might be required in SMTP
and in message headers, we'll see UTF-8 in both places, and we'll
be better off if MTAs and UAs that do try to handle these things 
do so in a uniform way.  

> from being used in queries today except the hostname limitations. 

this would be true only if 

a) existing DNS servers could reliably compare UTF8 IDNs without nameprep
b) DNS queries were the major area of concern

> Just as
> while ACE does not work at all. 

for DNS servers, perhaps.  for applications that transmit DNS names as
protocol elements, the opposite is closer to the truth.  

> does not mandate new behavior. 

on the contrary, ACE does mandate new behavior in any application that
needs to do more with an IDN than query its DNS records, pass it to 
another application, or compare one IDN with another.  any application
that inputs or displays IDNs will have to behave differently in order
to support ACE; but any application that inputs or compares IDNs would 
have to behave  differently to support UTF-8 IDNs)

> In the end, going with ACE translates

any IDN scheme (whether ACE or not) will cause more leakage of 
non-ASCII names into applications that are expecting ASCII names.
the very fact that we are creating a standard mechanism to do
this, combined with the growng number of non-English speakers on
the net, ensures this.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 04 Feb 2001 18:55:47 -0800

Keith, let me redefine the point:

One of the great things about the EDNS proposal is that it provides long
labels and names. Since this would be a favorite feature of a lot of the
non-English locales, it would be widely-adopted. In turn, its very usage
would break legacy apps and resolvers that tried to resolve the long
labels/names. That keeps UTF8 names out of the legacy lookup path, while
also encouraging upgrades to applications, resolvers and servers that
support the necessary EDNS extensions. None of the other proposals have
the ability to preserve legacy lookups while simultaneously promoting the
adoption of the new service.

There are numerous other benefits -- any application that does the EDNS
formatted lookup is (very) likely to have done nameprep on the name in the
application beforehand, it uses the same technology as A6 RRs so it can
ride the same adoption wave, and many others -- but the key point is that
the EDNS approach lets the domain names (and just the names) break the
apps and resolvers. Very little additional effort is required except for
the labels/names that do not exceed 63/255 bytes. That scenario is very
likely yes but it will only happen in a percentage of the cases instead of
all of them.

> be better off if MTAs and UAs that do try to handle these things

In a common scenario, the relay would write out the HELO name, the IP
address, and the PTR data. Setting aside HELO names for a moment, if the
PTR data included EDNS data which the relay or its resolver did not
understand, then the relay should leave the PTR string empty. If the app
is able to extract the EDNS name then there are ugly but feasible encoding
methods to keep it from destroying downstream mailers. As for HELO names,
there are also some potential problems there, but they don't look like
show stoppers (I don't know of any restrictions on length so encoding
appears to be the only issue). What this all means is that mail servers
aren't broken during a transition period, so no problem with EDNS-encoded
names (regardless of the length issues). (Yes I know that some anal sites
will break if they can't match HELO with PTR but that is an operational
concern which is not covered by protocol).

As for outbound mail, there might be problems if MX RRs mix legacy and
modern names. Probably not if the netadmins design their forwarding
through multi-homed and multi-named relays, and maybe no problems at all
if the mailer can't read the complete list anyway. I mean, if the mailer
only sees MX 100 on an ASCII name and never sees MX 50 with an IDN, then
SMTP will still work if the mailer only sends to 100. It will break in
situations where intermediary relays aren't capable of handling delivery
to downstreams which have IDNs but that again is an operational issue
which can be overcome. Worth exploration and testing but I don't see any
problems that are insurmountable.

> > a simple measurement of where implementations are today, UTF8 mostly
> > works while ACE does not work at all.
> 

I'm not arguing for or against ACE or just-use-UTF8. The point of the
response was to point out that ACE and just-use-UTF8 both introduce major
issues with backwards compatibility. If we really want to avoid the
problems of bad names in legacy lookups then they should both be avoided.

Yes I also realize it's too late for this argument and that ACE is the
interim solution of choice. My bad for showing up late. Still the opinion
is valid.

> another application, or compare one IDN with another.

None of the simple stuff above is reliable with ACE or just-use-UTF8 using
legacy apps or resolvers.

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 5 Feb 2001 11:25:19 +0800

Hi Eric,

This pros and cons discussion have been done before and been considered
by the Protocol Design Team. Check out their reports on
http://www.i-d-n.net/ietf49/idn-sandiego-protocol-design-term-report.ppt

While you are there http://www.i-d-n.net/, you may also want to read a
few I-Ds, including

draft-idn-idne-01.txt, on how to use EDNS for IDN

but more importantly, draft-klensin-dns-role-00.txt which discuss why
above may not be right solution either.

And for a really long term solution (which may be totally out of the
scope of this WG) due to the radical approach proposed,
draft-klensin-dnsclass0e-00.txt. :-)

-James Seng

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 04 Feb 2001 19:42:27 -0800

I've read the material there, almost all of it anyway.

> draft-klensin-dnsclass0e-00.txt. :-)

Doesn't work to extend the label space.

The question section has a fixed structure, so even with a new class the
Qname is still limited to 63 byte labels and 255 byte octets. You'd have
to do some ugly hacks to extend it (or use EDNS).

There are lots of other problems with new classes. CNAMEs across classes
probably won't work for example. All of the IN specific RRs (of which
there are many) would have to be reinvented/redefined. Etc.

Alternative class is creative but it brings too much baggage. EDNS is the
most effective proposal currently on the table for many reasons.

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Mon, 5 Feb 2001 10:31:16 -0500

| One of the great things about the EDNS proposal is that it provides long
| labels and names. Since this would be a favorite feature of a lot of the
| non-English locales, it would be widely-adopted.

  I don't find this to be a convincing statement.  My impression was that
the conventional wisdom with respect to CJK was that names would generally
be shorter.  Host labels are used by humans to identify and reach hosts and
network services, not to compose short novellas. :-)

  Verisign folks (or anyone who might know), from the ML testbed, do we find
that the CJK labels in terms of number of codepoints are longer, shorter, or
about the same as the non-ML ones?

  If one is really worried about the 'long label problem', what data do we
have on whether ACE or UTF-8 is better overall?  Some ACEs are better at
compression than others of course.

  All of this has been discussed in the past, but it doesn't hurt to re-open
this issue if for no other reason than to facilitate consensus.

  I still think Klensin's point is right on:  there are things we're doing
with the DNS today that don't fit well with its purpose.  We can either
choose to change the DNS or provide a new service layer that is designed to
meet those needs.

  -bws