Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Sat, 15 Sep 2007 16:31:02 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> Stephane's server "knows" that it's UTF-8 and sends a corresponding 
> header field Content-Type: text/plain; charset=UTF-8
>
> It's an ordinary Apache server, therefore it "knows" how to get this 
> right because Stephane said so manually in a configuration file.

Which proves that some tools that are designed and configured to work 
with UTF-8 can do so.

> BTW, this version uses LF instead of CrLf for lineends, I get garbage 
> when I look at it with the admittedly simple MS editor.

Section 3.1 of RFC 4646 specifies that records in the Registry are 
supposed to end with CRLF, not LF or anything else.  Of course, we know 
that some data-transfer protocols arrogantly convert CRLF to LF (or vice 
versa) anyway, and there's really not much we can do to change that.

> When I use Wordpad to look at a local copy the lineends are fine, but 
> the UTF-8 is ignored.  The first test entry is the description for 
> "nb", that should be BokmÃl, not BokmÃÂl.

Which proves that some tools that are *not* designed to work with UTF-8 
*cannot* do so (including, evidently, not only WordPad but also the Web 
interface to the LTRU mail archive).

>> Locally saved files may not behave the same under all browsers.
>
> Or text editors.  Or Web servers.  Not all users know how to set the 
> character encoding manually, or they can't because it's not their own 
> server.
>
> With a BOM the effects might be somewhat more predictable for simple 
> Windows tools, under W2K it helps with the MS editor but not Wordpad.

UTF-8 support isn't magic.  Programs and protocols have to be designed 
to support UTF-8, or they can't be expected to do so.

But I still assert that the number of tools that can interpret and 
display characters encoded as UTF-8 correctly, without manual 
intervention, in a non-XML, non-HTML text file, is greater than the 
number of tools that will do the same for characters encoded as hex 
NCRs.

> From my POV it's the same issue with the "URL" interface of the W3C 
> vaidator, because the used Web server sends the XML file as text/xml 
> without charset, example (1175 KB):
> http://xyzzy.webhop.info/home/ltru/4645bisU.xml

As soon as you convert the Registry from its native record-jar format to 
XML or some other format, you assume responsibility for making the XML 
data work with existing XML tools.  If you wanted to convert it to CSV 
for import into Excel, you would have to make sure items with embedded 
commas were suitably quoted.  If you wanted to convert it to Access, you 
would have to deal with whatever Access makes you deal with.  And so 
forth.

Converting the Registry from record-jar to something else either (a) is 
our problem, in which case we need to specify rules and guidelines for 
making it work, or (b) is not, in which case it is not.

> The W3C validator is tuned to get the encoding right for HTML 4 or 
> XHTML documents, long story, short summary:  For text/xml it assumes 
> ASCII (so far that's correct) and ignores the encoding in the XML 
> declaration (and that's an oddity).

I'm sorry that you are experiencing browser and/or protocol problems 
getting an XML document encoded in UTF-8 to display or validate 
correctly.  I do not see what this has to do the record-jar Registry.

>> this is an XML issue and not an issue with the Registry itself, which 
>> is in record-jar format.
>
> No, it's an issue with the source format.  The US-ASCII XML based on a 
> record-jar registry with hex. NCRs works as is.  The UTF-8 XML based 
> on an UTF-8 record-jar registry does not work as is.

Then you need to change the conversion process.  Our purpose in using 
record-jar is not solely to provide a format that is directly 
convertible to XML.  If it were, we'd be better off just using XML in 
the first place, and we've already had that debate many times.

> FWIW when I compare Stephane's language-subtag-regitry-utf8 with the 
> output of gawk -f 4645bis.awk lstr709.txt > lstr709U.txt they are 
> identical (modulo lineends).

Actually, I'm sure your NCR-to-UTF-8 conversions are fine, although I 
haven't checked them.

> I also know nothing about the conversion from the ISO 639-3 UTF-8 
> input to record-jar with NCRs, do you have a test suite ?

I wrote a program to convert the 639-3 data to Registry format, 
including the conversion from UTF-8 to hex NCRs for insertion into the 
I-D (the opposite of the transformation that I'm suggesting you 
perform).  I haven't written a test suite or back-conversion programs to 
see if I did the job right.  That's why I asked the list last week to 
review the 4645bis draft and send comments.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 16 Sep 2007 05:22:05 +0200

Doug Ewell wrote:
 
> some tools that are designed and configured to work with UTF-8
> can do so.

No surprise here, I worry about tools which are not designed or
not configured to work with UTF-8.  The hex. NCRs are robuster
with arbitrary tools.

> some tools that are *not* designed to work with UTF-8 *cannot*
> do so (including, evidently, not only WordPad but also the Web 

Ugh... http://article.gmane.org/gmane.ietf.ltru/8492/raw is what
I sent, QP UTF-8 =C3=A5 for aring http://purl.net/net/ucode/e5

It's still okay in the current mbox file of the IETF archive,
=C3=A5 in ftp://ftp.ietf.org/ietf-mail-archive/ltru/current

In the corresponding Web server document it's shown as &#xC3;
http://www1.ietf.org/mail-archive/web/ltru/current/msg08509.html
Apparently a bug near the Web interface to the list archive (?)
 
> I still assert that the number of tools that can interpret and 
> intervention, in a non-XML, non-HTML text file, is greater than
> the number of tools that will do the same for characters 
> encoded as hex NCRs.

There should be no tool at all displaying a plain text file with
hex. NCRs as Unicode without manual intervention.  And that's
the point, the hex. NCRs can't be interpreted or misinterpreted
without some obvious and manual intervention of registry users.

But it's very easy to run into obscure bugs with 3rd party tools
for a plain text UTF-8 file without manual intervention.  So
far we have identified an IETF list interface, the W3C validator 
(for UTF-8 XML), modern browsers like FF2, and some Win editors,
where plain text UTF-8 doesn't work without manual intervention.

> I'm sorry that you are experiencing browser and/or protocol
> problems getting an XML document encoded in UTF-8 to display
> or validate correctly.  I do not see what this has to do the
> record-jar Registry.

I don't have these problems with encoding="US-ASCII" based on
the robust 4646 record-jar input.  I run into these problems
using a hypothetical 4646bis UTF-8 registry and precisely the
same tool (after s/US-ASCII/UTF-8/ in its XML declaration).

Of course I could convert 4646bis UTF-8 to 4646 hex. NCRs, and
then create encoding="US-ASCII" XML, but that's an odd approach.
Something like http://purl.net/xyzzy/src/ncrify.cmd isn't what
you find everywhere.

> Our purpose in using record-jar is not solely to provide a 
> format that is directly convertible to XML.  If it were, we'd
> be better off just using XML in the first place, and we've 
> already had that debate many times.

I'm on record with saying that UTF-8 *_requires_* XML for some
time now.  The purpose of record-jar with hex. NCRs was that
it works on all platforms with all tools as expected and as is.

It's even possible to display the RFC 4646 record-jar format
correctly with modern browsers after adding only two lines.

> I haven't written a test suite or back-conversion programs to 
> see if I did the job right.  That's why I asked the list last
> week to review the 4645bis draft and send comments.

The references are as always okay, the input is apparently the
2007-07-17 639-3 version (I checked some new + retired subtags),
and the NCRs are in the range upto u+10FFFF, otherwise my script
would throw an error.  If there's a problem in the UTF-8 input
it's no LTRU problem.  If you have an UTF-8 proto-registry we
could compare it with my gawk -f 4645bis output, and if that's
identical your NCR encoder and my NCR decoder agree.  I get an
UTF-8 registry with filedate 2029-09-09, file size 698043, and
MD5sum 43443a34cbb996a3e5a42ed551ef803f

No special observation in the prose of your I-D.  For the source
(the 639-3 Web site) I'd wish that they show the outcome of the
modification requests (approved or not) directly in their list,
not only in the "more" details per request.

There are quite a lot of modifications, almost 400 in two years,
128 resolved (= reflected in your I-D).  If that continues with
a bulk update once per year the language subtag reviewer will be
rather busy at this time of the year.

Frank