Subject: Re: [idn] URL encoding in html page

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Mon, 1 Apr 2002 01:12:31 -0500

> > Agrees, the design of how UTF-8 encodes and decodes can be expanded to
> > support more bits... so that's why UTF8 should be reasonable to be the
> long
> > term solution for designing i18n applications, including IDN...
>
> Please, everybody, read the definition of UTF-8 before making these
> claims that it supports an encoding structure wider than 31 bits.  (And
> please read the statements of WG2 and Unicode concerning expansion
> beyond U+10FFFD.)
>
> UTF-8 seems to have taken on a mystical quality whereby it can now
> provide food for the starving, shelter for the homeless, etc.

The word "EXPAND" here means it can be extended or modify to cope with
future requirements over the limits now... i didn't say that it is "READY"
to support more... any one that reads carefully show see the words "can be
expanded" and not "can be able" : )

---

From: Martin Duerst <[email-masked]>
Date: Thu, 04 Apr 2002 18:37:15 +0900

Hello Adam,

At 23:52 02/03/28 +0000, Adam M. Costello wrote:
>Martin Duerst <[email-masked]> wrote:

> > You seem to imply that the HTML editor is checking or should check
> > the URI syntax, and that they could be upgraded to add a legacy
> > encoding->Unicode->ACE conversion.
>
>Regardless of whether today's HTML editors already check URI syntax,
>they could be upgraded to convert IRIs input by the user into URIs in
>the document, and that conversion could involve ToASCII.

In some cases, that might work, but in the general case, it won't.

> > If you enter hppt:something, should the editor tell you this is an
> > error?
>
>No, because it's not an error.  A warning might be helpful, or might be
>annoying, just like spell-checking.

So if you enter hppt:KANJI.KANJI.jp, what should the HTML editor do?
How does it know whether KANJI.KANJI.jp is a domain name or not?

> > Similarly, even if the HTML editor does the very limited checks that
> > RFC 2396 allows, this doesn't get it very far. For example, while
> > http: uses generic URI syntax, mailto: uses opaque syntax, so there is
> > no general way to know where in an URI there is a domain name. This
> > includes the case that a domain name is sent as a parameter.
>
>True, the editor can't be expected to know the syntax for every URI
>scheme.  For the ones it does know, it can fully check the syntax.  For
>any URI that begins with / or scheme:/, it can check the generic URI
>syntax (which is quite detailed).

This would mean that the URI syntax cannot be updated in any way.
There are some advantages in careful checking at some places,
and there are some disadvantages. For URI syntax, the general
tendency is to not try to check too much. The generic syntax
is there to allow things like relative URI resolution,...

>For opaque URIs with unknown schemes,
>it can at least check for characters disallowed in all URIs.
>
>I think that gets you pretty far in practice.  The vast majority of
>opaque URIs could also be handled if the editor knows just a few schemes
>(mailto, and maybe news and telnet).

In practice, an HTML editor could indeed do such things.
And for domain names, it may indeed be reasonable to
have the editor use toASCII when possible, at least for
a certain time.

> > HTML 4 already says what a browser should do if it finds a
> > non-ASCII character in an URI.  Please see
> > http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.2.1
> > Appendix B.2.1: Non-ASCII characters in URI attribute values
> > (this is supported by all major browsers in newer versions)
>
>It says:

...

>     This procedure results in a syntactically legal URI (as defined in
>     [RFC1738], section 2.2 or [RFC2141], section 2) that is independent
>     of the character encoding to which the HTML document carrying the
>     URI may have been transcoded.

>And RFC-2396, which is a normative reference
>of the HTML spec and which updates RFC-1738, is unambiguous in its
>prohibition of %HH escapes in the host part.
>
>Therefore, it is not true that "this procedure results in a
>syntactically legal URI".

Well, indeed this sentence is a bit too general. For example,
a space in a src attribute would not lead to a legal URI.

>I suggest that the procedure be ammended so
>that it does result in a syntactically legal URI.

How do you want to do that. Just saying 'convert host names
with toASCII' doesn't do the job.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 04 Apr 2002 18:10:59 +0900

A small point, and just because it hasn't been mentioned
yet (as far as I have seen):

I guess that the main reason that RFC 2277,... point to UTF-8
is that both the Internet protocols (v4 and v6) and most
hardware work with 8-bit bytes, and there is absolutely
no indication that this will change soon.

Regards,  Martin.

At 11:15 02/03/29 +0100, Keld J$B�S(Bn Simonsen wrote:
>On Fri, Mar 29, 2002 at 12:40:41PM +0900, Bruce Thomson wrote:
> > > The question is really why 8/16/32 bit Unicode is better than 5bit (ACE)?
> >
> > ACE and UTF-8 are just compression algorithms that squeeze larger
> > Unicode characters. ACE is more efficient than UTF, although more
> > complex.
> >
> > But the claim to fame that UTF-8 has is that it is a standard that
> > idn can reference, and it is coming into widespread use elsewhere.
> >
> > So moving to UTF-8 long term seems like such an obvious choice
> > is surprises me that it even gets debated.
>
>I think the UTF-8 is the way to go forward too, and I actuallt think
>this is also IESG policy, viz RFC 2277 and RFC 2130. The UTF-8 RFC 2279 is
>the only standards track RFC on charsets for the same reason. Citing
>from RFC 2277, the IESG policy on charactyer sets and language:
>
>    "Protocols MUST be able to use the UTF-8 charset, which consists of
>    the ISO 10646 coded character set combined with the UTF-8 character
>    encoding scheme, as defined in [10646] Annex R (published in
>    Amendment 2), for all text."
>
>I dont mind having an ASCII fallback as we also have it in email, but
>going into the other encoding forms of ISO 10646 is discouraged by
>IESG, as they do not want to see a lot of encodings for the same
>character set, with the possible problems of interoperability.
>So UCS-2, UCS-4, UTF-16, UTF-16-LE, UTF-16-BE etc are discouraged.
>
>Best regards
>Keld Simonsen