Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Tue, 18 Sep 2007 20:26:02 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> [Conversions]
>> I would like the chairs to rule on whether we should add "rules and 
>> guidelines for converting the record-jar Registry to other formats" 
>> to our list of deliverables, before we spend any more time debating 
>> this.
>
> IMO <http://www.langtag.net> covers this already.

That was what I thought too, and in that case I'd prefer to spend less 
WG time focusing on XML and how easy or hard it is to convert the 
Registry to XML.  We have settled on record-jar.

> One point might be relevant in this debate:  I'm pretty sure that XML 
> will continue to support hex. NCRs "forever".  IMHO we're not obliged 
> to switch to UTF-8 *now* because we'd have to do it anyway *later*.

I don't see any evidence that the official Registry will inevitably be 
converted to XML.  If we were so sure of that, we probably would have 
done it now, considering all the other scope creep that has gone on.

There is also no guarantee that there will be a "later," i.e. an LTRU 
3.0.  Many people don't seem to feel that collection codes are 
particularly valuable in tags, so it's not completely obvious that we'll 
do another revision to incorporate ISO 639-5.  And I know there were 
some who disliked the additional precision that 639-6 codes are expected 
to bring.

> [639-3 pages]
>> The outcome of the modification requests is reflected in the latest 
>> data files, though, so strictly speaking this information isn't 
>> necessary to review the I-D, which uses the data files and not the 
>> requests.
>
> It could help later if an approved modification somehow gets lost on 
> its way from these pages via the review list to the registry.

Actually the bits were dropped somewhere else: an approved modification, 
or part of one, got lost on its way to the ISO 639-3 download files, 
which I picked up faithfully.  I onlFrom [email-masked] Tue Sep 18 23:27:04 2007
From: "Doug Ewell" <[email-masked]>
To: "LTRU Working Group" <[email-masked]>
Date: Tue, 18 Sep 2007 20:26:02 -0700
Subject: [Ltru] Re: Wrapping up the UTF-8 debate

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> [Conversions]
>> I would like the chairs to rule on whether we should add "rules and 
>> guidelines for converting the record-jar Registry to other formats" 
>> to our list of deliverables, before we spend any more time debating 
>> this.
>
> IMO <http://www.langtag.net> covers this already.

That was what I thought too, and in that case I'd prefer to spend less 
WG time focusing on XML and how easy or hard it is to convert the 
Registry to XML.  We have settled on record-jar.

> One point might be relevant in this debate:  I'm pretty sure that XML 
> will continue to support hex. NCRs "forever".  IMHO we're not obliged 
> to switch to UTF-8 *now* because we'd have to do it anyway *later*.

I don't see any evidence that the official Registry will inevitably be 
converted to XML.  If we were so sure of that, we probably would have 
done it now, considering all the other scope creep that has gone on.

There is also no guarantee that there will be a "later," i.e. an LTRU 
3.0.  Many people don't seem to feel that collection codes are 
particularly valuable in tags, so it's not completely obvious that we'll 
do another revision to incorporate ISO 639-5.  And I know there were 
some who disliked the additional precision that 639-6 codes are expected 
to bring.

> [639-3 pages]
>> The outcome of the modification requests is reflected in the latest 
>> data files, though, so strictly speaking this information isn't 
>> necessary to review the I-D, which uses the data files and not the 
>> requests.
>
> It could help later if an approved modification somehow gets lost on 
> its way from these pages via the review list to the registry.

Actually the bits were dropped somewhere else: an approved modification, 
or part of one, got lost on its way to the ISO 639-3 download files, 
which I picked up faithfully.  I only noticed the discrepancy because I 
happened to be browsing through the modification list.

> But it's already obvious that "stable forever" macrolanguage relations 
> in the form of "extlang + prefix" won't fly, they split, merge, add, 
> and remove sets far too often for this approach.

I don't think anything's obvious any more about macrolanguages and 
extlangs, at least as far as LTRU is concerned.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 18 Sep 2007 20:40:33 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> BOM is much more effective, albeit slightly weird for a UTF-8 file.

I still don't understand why a U+FEFF signature in a UTF-8 file is 
considered so "weird."  (Notice I did not use the words "byte order 
mark.")  The role of U+FEFF as a signature to identify the encoding was 
introduced as early as 1992 (TUS 1.0, Volume 2, p. 7), before there was 
a UTF-8.  TUS 5.0 describes the appropriate use of UTF-8 signatures in 
sections 2.13 and 16.8 (although Section 3.10 provides a mixed signal: 
"neither required nor recommended"), and RFC 3629 (STD 63) also 
describes conditions where the three-byte signature might be appropriate 
(Second 6, third bullet point).

---

From: Martin Hosken <[email-masked]>
Date: Wed, 19 Sep 2007 10:56:51 +0100

Dear Doug,

> mark.")

Because to a non-Unicode 8-bit or ASCII based process, U+FEFF in UTF8 is
just so much upper ASCII non-whitespace line noise just at the start of
the file. So the file starts with junk and the process will fall over.

Yours,
Martin

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 19 Sep 2007 12:05:00 +0200

On Wed, Sep 19, 2007 at 10:56:51AM +0100,
 Martin Hosken <[email-masked]> wrote 
 a message of 22 lines which said:

> Because to a non-Unicode 8-bit or ASCII based process, U+FEFF in
> UTF8 is just so much upper ASCII non-whitespace line noise just at
> the start of the file. So the file starts with junk and the process
> will fall over.

+1 

Just say no to BOM in UTF-8 files

http://unicode.org/faq/utf_bom.html#29

Q: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)?
If yes, then can I still assume the remaining UTF-8 bytes are in
big-endian order?

A: Yes, UTF-8 can contain a BOM. However, it makes no difference as to
the endianness of the byte stream. UTF-8 always has the same byte
order. An initial BOM is only used as a signature -- an indication
that an otherwise unmarked text file is in UTF-8. Note that some
recipients of UTF-8 encoded data do not expect a BOM. Where UTF-8 is
used transparently in 8-bit environments, the use of a BOM will
interfere with any protocol or file format that expects specific ASCII
characters at the beginning, such as the use of "#!" of at the
beginning of Unix shell scripts. [AF] & [MD]

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 19 Sep 2007 07:05:54 -0700

Martin Hosken <martin underscore hosken at sil dot org> wrote:

> Because to a non-Unicode 8-bit or ASCII based process, U+FEFF in UTF8 
> is just so much upper ASCII non-whitespace line noise just at the 
> start of the file. So the file starts with junk and the process will 
> fall over.

That's fine for UTF-8 files that need to be understood by 
non-UTF-8-aware programs.  I thought the problem here was UTF-8-aware 
programs that weren't being adequately informed that the file was UTF-8 
so they could switch into the right mode.  Non-UTF-8-aware programs are 
going to interpret high-bit characters in the Registry as junk anyway.

I agree with Addison: a gratuitous Comments field with one or more 
gratuitous non-ASCII characters is sillier than using U+FEFF.

If our goal is to stay 100% compatible with ASCII-only readers, and I 
know some people think that should be our goal, then the solution is to 
stick with hex NCRs.  But then we will continue to endure the complaints 
of other people that strings like Birl&#x4D9;&#x15F;dirilmi&#x15F; are 
not readable.  And I am categorically opposed to filling the Registry 
with "transcriptions" from accented Latin to unaccented Latin as a 
kludge to get around our character-encoding problems.

---

From: Addison Phillips <[email-masked]>
Date: Wed, 19 Sep 2007 07:28:08 -0700

>> start of the file. So the file starts with junk and the process will 

Not very many well-written programs "fall over" when they see non-ASCII 
byte sequences in a file. They might display them as mojibake garbage, 
but then that also applies to any non-ASCII that appears in the file.

However, having the BOM appear on the same line as the start of the 
File-Date field in the first record *would* represent a problem, since a 
naive processor might well consider the sequence 0xEF.BB.BF followed by 
"File-Date" not to match the string "File-Date". Hence, my suggestion 
that we include it on a line announcing the encoding in plain text.

> kludge to get around our character-encoding problems.

Our goal at the moment is to make a plain-text UTF-8 encoded file. The 
question is: do we announce the encoding or just rely on having users 
RTFRFC (read the fine RFC)?

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.