Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Mon, 17 Sep 2007 06:31:57 -0700

Martin Duerst <duerst at it dot aoyama dot ac dot jp> wrote:

> I think one very specific aspect of our datafile is that it's 
> extremely large and contains almost no actual UTF-8 data, and that 
> data appears rather late.

The first occurrence of a hex NCR in the current Registry, and therefore 
the first occurrence of a UTF-8 character, is at decimal offset 11,049. 
This is roughly one-eighth of the way into the 88,259-byte file.  I'm 
not sure how far auto-detection tools typically look for the first 
non-ASCII character; at least 16 kiB would sound reasonable to me.

> One proposal to improve this situation and potentially help some tools 
> would be to add a few UTF-8 heavy comments at the start of the file. 
> This might look something like
>
> File-Date: 2029-09-09
> Comments: >>>>lots of UTF-8 here<<<
> Comments: >>>>lots of UTF-8 here<<<
> Comments: >>>>lots of UTF-8 here<<<
> Comments: >>>>lots of UTF-8 here<<<
> Comments: >>>>lots of UTF-8 here<<<
> %%

This would help only if the problem is late detection of UTF-8, not an 
inability to grok UTF-8 at all.

> I'm not sure whether this is possible with the current file format,

It's not; the File-Date line is extraordinary, and we don't currently 
allow Comments fields to be attached to it.  Of course this could be 
changed.

> or how much backwards-incompatibility we would incur. Because UTF-8 
> itself is something new, some might argue that the above is good 
> because it provides an upfront warning about the change, and others 
> might argue that it's too much of a change.

I suppose I could point out that up-front identification of UTF-8 is one 
of the advantages of using a BOM.  I'd better not, because I make myself 
only slightly less unpopular defending the UTF-8 BOM than I do defending 
Plane 14 tags.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru