Subject: Re: [idn] URL encoding in html page

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Mon, 1 Apr 2002 01:25:38 -0500

> > ACE will work with "EXISTING" systems but ALL client end(browser, MUA,
etc)
> > and SOME server(Apache, etc) end software actually needs to upgrade in
order
> > to be able to recognize IDN and convert it to ACE back and forth, unless
we
> > want to say that www.ietf.org that we are using nowadays is IDN already
: )
> >
> > On the other hand, we see a lot of applications is moving towards the
> > support of UTF8/Unicode on the client side, and this is going to be the
> > future anyways... so why not just upgrade the server software, with one
> > server software upgraded it will serve many many users... I dont
understand
> > why no one see this simple math : )
>
> David,
>
> You seem to be assuming that there would be only two entities involved
> in the application protocol between the endpoints. The world isn't that
simple.
> As an example, the mail I'm responding to contained 8 Received lines when
I
> got it. Thus, assuming the return path have the same number of SMTP hops,
if
> you had nameprepped-UTF-8 in your domain name in the from: line, if any of
> those 8 relays didn't handle UTF-8 properly, this mail would not have
reached
> you. Thus it isn't just a question of upgrading two pieces of software.

There are more than 2 entities involves in all protocol, but as you see in
my other mail that i posted, my thought is, for example(in mail header) the
FROM: will have ACE and have new header say X-UTF8FROM: to have UTF8, so
when all those clients and servers that use the protocol are ready to handle
the new commands or headers, the transition will be seemless... and old
software that doesnt support these new commands and headers and/or syntax
will simply just ignore them, and the everything just appears as the same as
today, is this thinking got anything wrong?

> Likewise for http. Upgrading my browser wouldn't do much good when
> there are 2 caching proxies plus a socks host inside sun that look
> at the http headers, and some number of load balancers (global and local),
> ssl proxies, and akamai caches, that sit in front of many web servers.

Same as what i said above for SMTP, these entities use HTTP, so maybe for
HTTP we can add a new header to have IDNHOST as the UTF8 host and keep the
HOST line to have the ACE, so what's wrong again with this, and when doing
the initial HTTP session it will denotes the version as well, so when during
the process, no matter how many entites the request has been passed thru,
any compatible "new entities"(no matter client or server softwares), it will
maintain the new added command/header/syntax and for any entity thru out the
path that is not capable would be fallback automatically by the capable
entity into the ACE format again...

> Dealing with this type of reality is part of the challenge in the
transition
> to a scheme which uses nameprepped-UTF-8 in application protocols.

Since i am facing the reality that "WE" should look for a solution that can
fallback and also step forward(expandable), and not limit ourselves to a
solution that only fallbacks.

---

From: Kenneth Whistler <[email-masked]>
Date: Tue, 2 Apr 2002 10:07:55 -0800 (PST)

Dan asked:

> Kenneth Whistler writes:
> > The *real* problem is guaranteeing interoperability for UTF-8, UTF-16,
> > and UTF-32, which are the three sanctioned encoding forms of Unicode
> 
> The obvious choice for Internet protocols is UTF-8. See RFC 2277.

No argument there.

> Systems that use 16-bit encodings internally, such as Windows, handle
> UTF-8 conversions at the boundary between the system and the network.
> 
> What's the problem?

What I am referring to is code point range interoperability for the 3 encoding
forms of Unicode. The Unicode Standard is tied to ISO/IEC 10646, which
is architecturally a 31-bit character encoding standard, by which I mean
that its nominal code space is 0..0x7FFFFFFF.

The Unicode Standard formally limits that code space, however, to
a 21-bit range, namely 0..0x10FFFF. The reason for that is that
UTF-16 can only address that range. In the Unicode Standard, then,
UTF-32 is also constrainted to 0..0x10FFFF, and UTF-8 is constrained
to four-byte forms up to <F4 8F BF BF> (i.e. U+10FFFF). *That* is the
guarantee of interoperability, since it means that any valid value
in UTF-8 can be accurately converted to either of the two other
forms, and vice versa.

*If* 10646 were ever to encode a character at a code point beyond
0x10FFFF, *then* there would be an interoperability problem. And
that is why 10646 has been amended recently to retrofit the same
constraints on allowable ranges for encoding as specified in the
Unicode Standard. That is everyone's guarantee that neither SC2/WG2
nor the Unicode Consortium is going to encode a character that
breaks encoding form interoperability, no matter which of the three
forms (or combinations thereof) you are using for an implementation.

The reason I brought this up at all was to head off the zany
garden-path discussions about "UTF-128" and extending UTF-8
because putatively there might not be enough code points at
some unspecified time in the future. Let the character encoding
committees deal with that issue. In the meantime, the IETF (and
this IDN WG) have the Unicode Standard and the IS 10646, with
their standard encoding forms -- just use them, with the guarantee
of interoperability that the relevant committees are providing,
and don't hare off into discussions about their supposed inadequacy
or limitations.

> Converting between UTF-8 and UTF-16 and UTF-32 doesn't cause IDNA-style
> interoperability failures.

Correct. Because the UTC has limited the code space range to ensure
that interoperability.

> Of course, Windows still has all sorts of problems related to its old
> ``code pages,'' and there are many similar problems with old character
> encodings under UNIX. The use of more than one 8-bit ASCII extension
> provides ample opportunity for IDNA-style interoperability failures.
> UTF-8 is a way out of this mess.

Actually, Unicode is a way out of this mess. And then what particular
encoding form(s) you choose depend on the requirements of the protocol
or application you are designing and implementing.

--Ken