Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Fri, 14 Sep 2007 09:47:51 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> I've now updated 4645bis.awk to convert NCRs to UTF-8, see 
> http://xyzzy.webhop.info/home/ltru/4645bis.awk
>
> The script can extract an UTF-8 language subtag registry from Doug's 
> drafts, and it can also convert 4646 registries to UTF-8.
>
> As John said: _doom_

Doom?  Because 22 characters, about 0.025% of the Registry, don't 
display correctly by default in your Web browser?

> When I try to open a plain text UTF-8 file with Firefox 2 under Win XP 
> I've to set the encoding manually (unless I add a BOM).

The UTF-8 version at 
http://www.langtag.net/registries/language-subtag-registry-utf8 looks 
fine to me under Firefox 2.0.0.6, IE 6, and IE 7, all under Windows XP 
SP2.  Locally saved files may not behave the same under all browsers.

> When I convert an UTF-8 registry to XML and use the "file upload" 
> interface of the W3C validator I run into several fatal errors at the 
> W3C validator.

XMLSpy 2006 Home Edition thought it was fine.

> Unless I use the "hard" character encoding override option at the 
> validator.  Arguably it's the fault of Firefox 2 under Win XP to use 
> Content-Tye: text/xml without charset in its file upload header for 
> 4645bisU.xml.  But in practice it's as it is, both FF2 and the W3C 
> validator ignore encoding="UTF-8" in the 1st line <?xml version="1.0" 
> encoding="UTF-8" standalone="yes" ?>

Not arguably; you are *definitely* supposed to be able to use UTF-8 in 
properly marked XML documents.

But in any case, this is an XML issue and not an issue with the Registry 
itself, which is in record-jar format.  If you have problems with UTF-8 
in XML, then perhaps your process that converts record-jar to XML also 
needs to convert UTF-8 to hex NCRs.

> I'm not yet sure that I got the NCR to UTF-8 conversions right.

Then we know nothing, really.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 15 Sep 2007 19:22:51 +0200

Doug Ewell wrote:
 
> fine to me

Stephane's server "knows" that it's UTF-8 and sends a corresponding
header field Content-Type: text/plain; charset=UTF-8

It's an ordinary Apache server, therefore it "knows" how to get this
right because Stephane said so manually in a configuration file.

BTW, this version uses LF instead of CrLf for lineends, I get garbage
when I look at it with the admittedly simple MS editor.  When I use
Wordpad to look at a local copy the lineends are fine, but the UTF-8
is ignored.  The first test entry is the description for "nb", that
should be Bokmål, not BokmÃ¥l.  

> Locally saved files may not behave the same under all browsers.

Or text editors.  Or Web servers.  Not all users know how to set the
character encoding manually, or they can't because it's not their own
server.

With a BOM the effects might be somewhat more predictable for simple
Windows tools, under W2K it helps with the MS editor but not Wordpad.

>> interface of the W3C validator I run into several fatal errors at
>> the W3C validator.

>From my POV it's the same issue with the "URL" interface of the W3C
vaidator, because the used Web server sends the XML file as text/xml
without charset, example (1175 KB):
http://xyzzy.webhop.info/home/ltru/4645bisU.xml  

>> Arguably it's the fault of Firefox 2 under Win XP to use 
>> validator ignore encoding="UTF-8" in the 1st line
>> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

> Not arguably; you are *definitely* supposed to be able to use UTF-8
> in properly marked XML documents.

The W3C validator is tuned to get the encoding right for HTML 4 or
XHTML documents, long story, short summary:  For text/xml it assumes
ASCII (so far that's correct) and ignores the encoding in the XML
declaration (and that's an oddity).

> this is an XML issue and not an issue with the Registry itself,
> which is in record-jar format.

No, it's an issue with the source format.  The US-ASCII XML based on
a record-jar registry with hex. NCRs works as is.  The UTF-8 XML
based on an UTF-8 record-jar registry does not work as is.
 
> Then we know nothing, really.

FWIW when I compare Stephane's language-subtag-regitry-utf8 with
the output of gawk -f 4645bis.awk lstr709.txt > lstr709U.txt they
are identical (modulo lineends).  I also know nothing about the
conversion from the ISO 639-3 UTF-8 input to record-jar with NCRs,
do you have a test suite ?  
 
Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 16 Sep 2007 21:27:18 +0200

On Sat, Sep 15, 2007 at 07:22:51PM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 76 lines which said:

> When I use Wordpad to look at a local copy the lineends are fine,
> but the UTF-8 is ignored.  The first test entry is the description
> for "nb", that should be Bokmål, not BokmÃ¥l.

It seems there is nothing new here. Without meta-data such as HTTP's
Content-type, an editor has no way to know if a file is in ISO 8859-1,
ISO 8859-2, UTF-8, etc.

There are tricks like Emacs variables at the end of the file, or some
heuristics can be used but there is no good way to be sure.

It is certainly a big issue for all text formats where there is no
structure to indicate the encoding but it is not a new issue. XML does
not have this problem because there is a standard way to specify the
encoding in the file itself.

> declaration (and that's an oddity).

If it's true, it is a major bug, which should be reported ASAP.

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 16 Sep 2007 23:19:38 +0200

> Content-type, an editor has no way to know if a file is in ISO 8859-1,

Yes.  The OS/2 HPFS filesystem offers "extended attributes".  With W2K
and XP I'm apparently limited to "file extension" plus/minus BOM.  On a
personal note, my OS/2 box crashed three months ago, and that severly
limited my online activities.  OTOH it forced me to work with W2K, XP,
and (relatively) modern tools, and I'm rather annoyed that the UTF-8 situation is still so bad even with current tools.

 [W3C validator] 
>> For text/xml it assumes ASCII (so far that's correct) and ignores
>> the encoding in the XML declaration (and that's an oddity).

> If it's true, it is a major bug, which should be reported ASAP.

On the validator list Olivier convinced me that is actually correct,
compare <http://permalink.gmane.org/gmane.org.w3c.validator/9912>

For text/xml without charset RFC 3023 says "MUST assume US ASCII",
and XML 1.0 (4.3.3) says that it's a "fatal error" if a document
contains octets in conflict with the externally declared charset.

In theory the major bug is a Firefox 2.0.0.6 issue, it shouldn't
upload *.xml as text/xml without charset.  I can't tell why it does
not use application/xml for *.xml.  But it's not limited to FF2, the
Web servers of my pages also use text/xml without charset, and don't
allow me to create my own .ht-thingy files to specify what I want.

With googlepages it's even odder, sometimes it sends *.xml as some
kind of XML, at other times it claims application/octet-stream.  So
far for the wonders of UTF-8 XML, sigh.

Frank

---

From: John Cowan <[email-masked]>
Date: Sun, 16 Sep 2007 17:20:58 -0400

> encoding in the file itself.

True.  However, XML has its own problem, namely that people tend to
give it a media type of text/xml, and text/xml *means* that the file
is in US-ASCII, no matter what the encoding declaration says.  If
the type is application/xml, then the encoding declaration is believed.

-- 
What asininity could I have uttered     John Cowan <[email-masked]>
that they applaud me thus?              http://www.ccil.org/~cowan
        --Phocion, Greek orator

---

From: Martin Duerst <[email-masked]>
Date: Mon, 17 Sep 2007 13:11:32 +0900

At 06:20 07/09/17, John Cowan wrote:

>is in US-ASCII,

Yes, but >>>text/xml; charset=utf-8<<< would mean utf-8.

>no matter what the encoding declaration says.  If

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Mon, 17 Sep 2007 13:53:07 +0900

>Content-type, an editor has no way to know if a file is in ISO 8859-1,

I think one very specific aspect of our datafile is that it's extremely
large and contains almost no actual UTF-8 data, and that data appears
rather late.

One proposal to improve this situation and potentially help some tools
would be to add a few UTF-8 heavy comments at the start of the file.
This might look something like

File-Date: 2029-09-09
Comments: >>>>lots of UTF-8 here<<<
Comments: >>>>lots of UTF-8 here<<<
Comments: >>>>lots of UTF-8 here<<<
Comments: >>>>lots of UTF-8 here<<<
Comments: >>>>lots of UTF-8 here<<<
%%

I'm not sure whether this is possible with the current file format,
or how much backwards-incompatibility we would incur. Because
UTF-8 itself is something new, some might argue that the above
is good because it provides an upfront warning about the change,
and others might argue that it's too much of a change.

Regards,   Martin.

---

From: GerardM <[email-masked]>
Date: Mon, 17 Sep 2007 11:04:46 +0200

Hoi,
One issue seems to be if it is good enough when meta data indicates that
content is in a particular encoding. I do not understand this as an issue as
the work that we do is to allow language information to be part of the meta
data of a document.

When it is not acceptable that in the meta data it is indicated what
encoding is used, is it acceptable to have codes that indicate languages in
Meta data? When UTF-8 is indicated, it is UTF-8. When this is not good
enough I think we are talking for the talking.

Thanks,
    Gerard

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 17 Sep 2007 11:16:21 +0200

On Mon, Sep 17, 2007 at 11:04:46AM +0200,
 GerardM <[email-masked]> wrote 
 a message of 49 lines which said:

> encoding is used, is it acceptable to have codes that indicate
> languages in Meta data?

We are talking about the format of the *registry*, not about the
format of the resources that will be tagged with language tags.

The registry is not tagged with a language tag. It is plain text and
provides no easy way to indicate the encoding in the file itself. The
only meta-data currently allowed in the registry is File-Date.

---

From: Addison Phillips <[email-masked]>
Date: Mon, 17 Sep 2007 02:53:46 -0700

We could follow draft-phillips-record-jar and include an encoding 
signature in the file:

[bom]%%encoding=utf-8
File-Date: 2007-09-17
Comments: Some UTF-8 identifying gunk here, such as the file description
    using zenkaku Latin characters and/or the BOM as UTF-8.

To follow Martin's comment, the only changes we'd have to make would be 
to permit the Comments field in the File-Date record and inclusion of 
the encodingSig production in the ABNF.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 17 Sep 2007 12:53:09 +0200

>    using zenkaku Latin characters and/or the BOM as UTF-8.

While I don't like UTF-8 for the registry I think that adding a BOM helps.

Adding %%encoding=utf-8 makes sense if you intend to publish your
record-jar I-D as RFC in the foreseeable future. 

I'm not aware of any tools  where adding dummy content with non-ASCII
UTF-8 near the begin helps, let alone in addition to a BOM.

> To follow Martin's comment, the only changes we'd have to make would
> be to permit the Comments field in the File-Date record and inclusion of 

The minimal change for UTF-8 would be to introduce a BOM and to get rid
of the hex. NCRs.   

Adding comments would be a major change for say XML converters if
they try to preserve these comments.

Frank

---

From: Addison Phillips <[email-masked]>
Date: Tue, 18 Sep 2007 05:09:40 -0700

> record-jar I-D as RFC in the foreseeable future. 

I intend to wrap it up as quickly as possible. Since the status will be 
Informational, I don't foresee vast delays.

I note that having an "obvious" encoding declaration on the very first 
line of the file might help those who forgot about encodings when they 
opened the file.

> UTF-8 near the begin helps, let alone in addition to a BOM.

Browsers look at the first few K of the file when doing auto-detection, 
although that is usually for HTML rather than text/plain.

> they try to preserve these comments.

BOM is much more effective, albeit slightly weird for a UTF-8 file. At 
least Notepad would read the file as UTF-8, though. Less weird than 
putting random UTF-8 gunk near the top of the file "just to have some 
UTF-8 there".

Addison

---

From: Martin Duerst <[email-masked]>
Date: Wed, 19 Sep 2007 11:37:30 +0900

At 21:09 07/09/18, Addison Phillips wrote:
>Frank Ellermann wrote:
>> 
>>> We could follow draft-phillips-record-jar and include an encoding signature in the file:
>>>
>>> [bom]%%encoding=utf-8
>>> File-Date: 2007-09-17
>> Adding %%encoy noticed the discrepancy because I 
happened to be browsing through the modification list.

> But it's already obvious that "stable forever" macrolanguage relations 
> in the form of "extlang + prefix" won't fly, they split, merge, add, 
> and remove sets far too often for this approach.

I don't think anything's obvious any more about macrolanguages and 
extlangs, at least as far as LTRU is concerned.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 19 Sep 2007 11:37:30 +0900

At 21:09 07/09/18, Addison Phillips wrote:
>I intend to wrap it up as quickly as possible. Since the status will be Informational, I don't foresee vast delays.
>
>I note that having an "obvious" encoding declaration on the very first line of the file might help those who forgot about encodings when they opened the file.

Yes indeed. One problem with it is that some people
might think that the registry can come in versions
with different encodings. I wouldn't want to give
them such an impression.

>Browsers look at the first few K of the file when doing auto-detection, although that is usually for HTML rather than text/plain.

Many Japanese text editors do. But this has been the case historically,
even before UTF-8.

>>> be to permit the Comments field in the File-Date record and inclusion of the encodingSig production in the ABNF.
>BOM is much more effective, albeit slightly weird for a UTF-8 file. At least Notepad would read the file as UTF-8, though. Less weird than putting random UTF-8 gunk near the top of the file "just to have some UTF-8 there".

Random gunk is definitely a bad idea.
It would have to be part of the official syntax.
A BOM is problematic because it inherently changes parsing.
Rather than just parse US-ASCII and be 8-bit-clean for data
values, some very special parsing has to be done.

My overall personal preferences would probably be as follows:

1) Go for UTF-8, to help end users.
2) Okay to add some comment fields at the start, but I don't
   think they are really necessary
3) Marginally okay to add some 'encoding-declaration'-like
   field at the top.
4) Against the BOM, but okay if everybody else really wants it.

Regards,   Martin.

ding=utf-8 makes sense if you intend to publish your
>I intend to wrap it up as quickly as possible. Since the status will be Informational, I don't foresee vast delays.
>
>I note that having an "obvious" encoding declaration on the very first line of the file might help those who forgot about encodings when they opened the file.

Yes indeed. One problem with it is that some people
might think that the registry can come in versions
with different encodings. I wouldn't want to give
them such an impression.

>Browsers look at the first few K of the file when doing auto-detection, although that is usually for HTML rather than text/plain.

Many Japanese text editors do. But this has been the case historically,
even before UTF-8.

>>> be to permit the Comments field in the File-Date record and inclusion of the encodingSig production in the ABNF.
>BOM is much more effective, albeit slightly weird for a UTF-8 file. At least Notepad would read the file as UTF-8, though. Less weird than putting random UTF-8 gunk near the top of the file "just to have some UTF-8 there".

Random gunk is definitely a bad idea.
It would have to be part of the official syntax.
A BOM is problematic because it inherently changes parsing.
Rather than just parse US-ASCII and be 8-bit-clean for data
values, some very special parsing has to be done.

My overall personal preferences would probably be as follows:

1) Go for UTF-8, to help end users.
2) Okay to add some comment fields at the start, but I don't
   think they are really necessary
3) Marginally okay to add some 'encoding-declaration'-like
   field at the top.
4) Against the BOM, but okay if everybody else really wants it.

Regards,   Martin.

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 19 Sep 2007 08:55:16 +0200

>From my POV an initial line [BOM]encoding=utf-8 would work,
I'd ignore it triggering UTF-8 mode, where an XML converter
has to escape "&" in addition to "<" and ">".  A BOM also
helps with some Windows tools.  Without this line the same
converter could assume 4646 "NCR-mode" keeping all "&" as is
and using encoding="US-ASCII" in the XML declaration.

Frank