Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Sun, 16 Sep 2007 21:06:43 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> No surprise here, I worry about tools which are not designed or not 
> configured to work with UTF-8.  The hex. NCRs are robuster with 
> arbitrary tools.

Of course you will be able to read the 11-character string "Bokm&#xE5;l" 
on just about any computer configuration build in the past 20 years. 
It's pure ASCII.  The question is whether the robust hex NCRs are 
meaningful to consumers of the Registry.

What we really have here is not so much a technical argument, but a 
philosophical argument with technical elements.  Someone please correct 
me if I'm wrong, but:

* The UTF-8 supporters believe it is most important to represent the 
correct characters of the Registry using a direct encoding, even if not 
all computers are able to render it, and even if extra work is needed to 
convert it to some other format.

* The hex-NCR supporters believe it is most important to be able to 
reconstruct the correct characters of the Registry unambiguously, even 
if they do not display correctly on any computer without being 
converted.

Which side holds greater sway for you probably determines whether you 
prefer UTF-8 or hex NCRs for the Registry.

> Ugh... http://article.gmane.org/gmane.ietf.ltru/8492/raw is what I 
> sent, QP UTF-8 =C3=A5 for aring http://purl.net/net/ucode/e5
>
> It's still okay in the current mbox file of the IETF archive, =C3=A5 
> in ftp://ftp.ietf.org/ietf-mail-archive/ltru/current
>
> In the corresponding Web server document it's shown as &#xC3; 
> http://www1.ietf.org/mail-archive/web/ltru/current/msg08509.html
> Apparently a bug near the Web interface to the list archive (?)

Not a single one of those displayed lowercase a-with-ring as I would 
expect.  I don't dispute that e-mail systems and mail archives have 
problems displaying UTF-8 correctly, nor that different systems have 
different problems.  We already discussed that from the perspective of 
sending updates to IANA via e-mail.

> But it's very easy to run into obscure bugs with 3rd party tools for a 
> plain text UTF-8 file without manual intervention.  So far we have 
> identified an IETF list interface, the W3C validator (for UTF-8 XML), 
> modern browsers like FF2, and some Win editors, where plain text UTF-8 
> doesn't work without manual intervention.

We agree.  I think I view this position as, it's better to have hex NCRs 
which are never scrutable without intervention than UTF-8 which is 
sometimes scrutable without intervention, because at least with hex NCRs 
you know.

>> ... I do not see what this has to do the record-jar Registry.
>
> I don't have these problems with encoding="US-ASCII" based on the 
> robust 4646 record-jar input.  I run into these problems using a 
> hypothetical 4646bis UTF-8 registry and precisely the same tool (after 
> s/US-ASCII/UTF-8/ in its XML declaration).

Pure ASCII is usable on more systems than any other encoding.  Nobody 
disputes that, and it will probably always be true.

> Of course I could convert 4646bis UTF-8 to 4646 hex. NCRs, and then 
> create encoding="US-ASCII" XML, but that's an odd approach. Something 
> like http://purl.net/xyzzy/src/ncrify.cmd isn't what you find 
> everywhere.

I would like the chairs to rule on whether we should add "rules and 
guidelines for converting the record-jar Registry to other formats" to 
our list of deliverables, before we spend any more time debating this.

> The references are as always okay, the input is apparently the 
> 2007-07-17 639-3 version (I checked some new + retired subtags),

The input should have been the 2007-08-01 and 2007-08-14 downloadable 
files, as stated in Section 7.1.  If you found some content that is not 
consistent with that, please let me know so I can make corrections.

> If you have an UTF-8 proto-registry we could compare it with my 
> gawk -f 4645bis output, and if that's identical your NCR encoder and 
> my NCR decoder agree.  I get an UTF-8 registry with filedate 
> 2029-09-09, file size 698043, and MD5sum 
> 43443a34cbb996a3e5a42ed551ef803f

I usually use BabelPad or SC UniPad to convert between UTF-8 and hex 
NCRs.  I could write my own tool rather easily, but it's even easier to 
use editors I already have.  I can check to see if my size and MD5 match 
yours.

> No special observation in the prose of your I-D.  For the source (the 
> 639-3 Web site) I'd wish that they show the outcome of the 
> modification requests (approved or not) directly in their list, not 
> only in the "more" details per request.

That might be helpful.  I think for every standard-code-list Web site 
I've seen, I can think of at least some improvement that would make the 
list easier to use.  The outcome of the modification requests is 
reflected in the latest data files, though, so strictly speaking this 
information isn't necessary to review the I-D, which uses the data files 
and not the requests.

> There are quite a lot of modifications, almost 400 in two years, 128 
> resolved (= reflected in your I-D).  If that continues with a bulk 
> update once per year the language subtag reviewer will be rather busy 
> at this time of the year.

It looks like it will be twice per year.  Every time ISO 639-3 announces 
a round of changes, and they say this will happen in January 2008, the 
changes will need to be reflected in the Registry.

It is my hope that the rate of change slows down and ISO 639-3 becomes 
more stable over time, but unfortunately this is the price we have to 
pay if we want to expand BCP 47 to allow ISO 639-3 code elements.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 17 Sep 2007 19:10:33 +0200

Doug Ewell wrote:
 
> What we really have here is not so much a technical argument,
> but a philosophical argument with technical elements.

Yes.

 [obscure aring problem] 
> Not a single one of those displayed lowercase a-with-ring as I
> would expect.

The "raw" GMaNe link was supposed to show the raw quoted printable
mail as I sent it, =C3=A5.  The normal link for this article is
http://article.gmane.org/gmane.ietf.ltru/8492 (without /raw).

And the current LTRU archive on the IETF ftp server is an ordinary
ASCII mbox file, also with quoted printable =C3=A5.  Any UA or tool
supporting MIME would decode the transfer encoding and arrive at
the intended UTF-8 encoding of u+00E5.  Apparently the Web interface
of the IETF archive gets this wrong, it shows &#xC3; for the =C3=A5.

> We already discussed that from the perspective of sending updates
> to IANA via e-mail.

Yes, and I'm sure that the subtag reviewer, his appointed record
editor, and IANA will get this right.  IANA will also manage to
configure its Web servers correctly.  UTF-8 is no rocket science,
at some point in time it will be a kind of new universal charset.

[...]
> I think I view this position as, it's better to have hex NCRs 
> which are never scrutable without intervention than UTF-8 which
> is sometimes scrutable without intervention, because at least
> with hex NCRs you know.

+1, that's how I feel about it.  It's also clear that a majority
prefers UTF-8 (including Michael, although he doesn't post here).

 [Conversions] 
> I would like the chairs to rule on whether we should add "rules
> and guidelines for converting the record-jar Registry to other
> formats" to our list of deliverables, before we spend any more
> time debating this.

IMO <http://www.langtag.net> covers this already.

One point might be relevant in this debate:  I'm pretty sure that
XML will continue to support hex. NCRs "forever".  IMHO we're not
obliged to switch to UTF-8 *now* because we'd have to do it anyway
*later*.

  [hex. NCR en- and decoders]
>> I get an UTF-8 registry with filedate 2029-09-09, file size 
>> 698043, and MD5sum 43443a34cbb996a3e5a42ed551ef803f
[...]
> I can check to see if my size and MD5 match yours.

Okay.  If you use established encoders (unlike my ad hoc decoder)
the output is likely okay, but I'd feel better if we know that my
side matches your input.  
 
 [639-3 pages]
> The outcome of the modification requests is reflected in the 
> latest data files, though, so strictly speaking this information
> isn't necessary to review the I-D, which uses the data files 
> and not the requests.

It could help later if an approved modification somehow gets lost
on its way from these pages via the review list to the registry.

 [almost 400 modification requests in two years]  
> It is my hope that the rate of change slows down and ISO 639-3
> becomes more stable over time

+1.  But it's already obvious that "stable forever" macrolanguage
relations in the form of "extlang + prefix" won't fly, they split,
merge, add, and remove sets far too often for this approach.

In another article you wrote:

| I could point out that up-front identification of UTF-8 is one 
| of the advantages of using a BOM.  I'd better not, because I make
| myself only slightly less unpopular defending the UTF-8 BOM than
| I do defending Plane 14 tags.

For the awk scripts I'll have to figure out how to match it if the
BOM is immediately followed by the file-date.  If it's on a separate
"encoding" line before the file-date line (Addison's proposal) there
won't be a problem, I'd just ignore the encoding line assuming UTF-8.

Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 17 Sep 2007 13:20:04 -0400

> merge, add, and remove sets far too often for this approach.

Red herring.  I'm proposing extlang subtags as a one-time permanent
shim for 639-2 to 639-3 bridging and nothing else, in order to
make a smooth on-ramp for using the new tags without breaking old
searches.

-- 
John Cowan   [email-masked]
    "Mr. Lane, if you ever wish anything that I can do, all you will have
        to do will be to send me a telegram asking and it will be done."
    "Mr. Hearst, if you ever get a telegram from me asking you to do
        anything, you can put the telegram down as a forgery."

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 17 Sep 2007 20:37:21 +0200

John Cowan wrote:

>> it's already obvious that "stable forever" macrolanguage relations
>> in the form of "extlang + prefix" won't fly, they split, merge, 
>> add, and remove sets far too often for this approach.
 
> make a smooth on-ramp for using the new tags without breaking old

I was talking about the almost 400 resolved and pending modification
requests wrt the extlang concept in draft-ietf-ltru-4646bis-08.txt
with up to three extlang subtags behind a macrolanguage subtag, all
of this "stable forever" (modulo deprecation).

That "old" concept in the current draft won't fly, the macrolanguage
sets are far too volatile at the moment.  I wasn't talking about your
proposal, actuallly I'm not completely sure what your proposal is:

Which macrolanguages in addition to ar, zh, and the special sgn case
would you adopt for the initial 4645bis registry ?

Are later additions to the existing initial sets allowed ?

What happens if 639-3 removes a language from one of the existing
sets (e.g. ar-xxx becomes xxx, would we deprecate extlang xxx in
favour of a new language xxx) ?

I think your proposal would allow to replace "up to three" by "at
most one" extlang in the 4646bis syntax for a fixed initial set of 
macrolanguages, is that correct ?

Can we enumerate this fixed initial set directly in the syntax ?

Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 17 Sep 2007 18:03:50 -0400

> would you adopt for the initial 4645bis registry ?

All of them present in 639-3 as of Date C.

> Are later additions to the existing initial sets allowed ?

No (with the possible exception of sgn).

> favour of a new language xxx) ?

No.

Yes.

> Can we enumerate this fixed initial set directly in the syntax ?

In principle, but that would make the syntax unreadable.

This is my current position, which differs from earlier positions of mine.

Gadshill:  Stand!
Falstaff:  So I do, against my will.
	--Shakespeare, Henry IV Part I

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Objective consideration of contemporary phenomena compel the conclusion
that optimum or inadequate performance in the trend of competitive
activities exhibits no tendency to be commensurate with innate capacity,
but that a considerable element of the unpredictable must invariably be
taken into account. --Ecclesiastes 9:11, Orwell/Brown version

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 21 Sep 2007 18:04:53 +0200

> All of them present in 639-3 as of Date C.

That's at the moment 354 extlang-s:  ´

30 ar, 2 ay, 2 az, 3 bal, 5 bik, 3 bua, 2 chm, 6 cr, 
2 del, 2 den, 5 din, 2 doi, 2 fa, 9 ff, 5 gba, 5 gn, 2 gon, 
5 grb, 2 hai, 21 hmn, 2 ik, 2 iu, 5 jrb, 3 kg, 2 kok,
2 kpe, 3 kr, 3 ku, 2 kv, 8 lah, 7 man, 10 mg, 2 mn, 13 ms,
6 mwr, 7 oj, 4 om, 3 ps, 44 qu, 6 raj, 7 rom, 4 sc, 4 sq,
2 sw, 2 syr, 4 tmh, 2 uz, 2 yi, 6 za, 45 zap, 11 zh, and
2 zza.

In other words 52 sets.  Sets that will be likely modified
in future revisions of 639-3, while the 4646bis registry
would be condemned to reflect an obsolete date C state
wrt to the extlang + prefix properties.

Your concept doesn't fix this.  Some of these 354 extlangs
might make it into 639-2, and users will wish to use 639-2
subtags as is, without prefix, let alone historical date C
prefixes not more reflecting the then actual Macrolanguage.

> No (with the possible exception of sgn).

Won't help, we could end up with a new language subtag uzw
for "Western Uzbek" plus old constructs uz-uzs and uz-uzn:

Some uzw folks would try to use uz-uzw, and some uz-uzs or
uz-uzn users would try to omit the prefix.  Your idea "but
it helps with 4647 matching" has merit, but I think the
potential confusion for taggers is worse than the potential
advantage for 4647 applications.

>> What happens if 639-3 removes a language from one of the
>> existing sets (e.g. ar-xxx becomes xxx, would we deprecate
>> extlang xxx in favour of a new language xxx) ?
 
> No.

Also bad, if users happen to know a 639-2 language they should
be able to use it in language tags without need to first check
if that used to be a 639-3 extlang at date C.

>> I think your proposal would allow to replace "up to three" ´
>> by "at most one" extlang in the 4646bis syntax for a fixed
>> initial set of macrolanguages, is that correct ?
 
> Yes.

Good, I like this part.

 Frank

---

From: John Cowan <[email-masked]>
Date: Fri, 21 Sep 2007 12:13:40 -0400

> prefixes not more reflecting the then actual Macrolanguage.

The whole point of the registry is that it makes it unnecessary -- indeed,
incorrect -- to look at 639-[123], 3166-1, and 19542 to determine the
referents of codes.  (It may still be necessary to look at supplementary
information to find out what "Ainu (China)" means, or what Gurmukhi
script is, but that's a different level of hermeneutics.)

> [W]e could end up with a new language subtag uzw

Hmm.  You may be right that it's good to be able to add new extlang
subtags to new or existing sets.

> if that used to be a 639-3 extlang at date C.

639-2 is no longer authoritative for RFC 4646 language tags;
only the LSR is.

-- 
Do what you will,                       John Cowan
   this Life's a Fiction                [email-masked]
And is made up of                       http://www.ccil.org/~cowan
   Contradiction.  --William Blake

---

From: Addison Phillips <[email-masked]>
Date: Fri, 21 Sep 2007 09:15:28 -0700

>>> initial set of macrolanguages, is that correct ?

Would we actually alter the ABNF? Or keep the extra extlangs in the ABNF 
for "future use" and/or as a monument to complexity?

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 21 Sep 2007 20:27:46 +0200

> Would we actually alter the ABNF? Or keep the extra extlangs in
> the ABNF for "future use" and/or as a monument to complexity?

Shorter tags are better for RFC 2231 encoded words (that's 2047
plus language tags).  At the moment we have 42 as recommended
minimal length.  Without extlang2 and extlang3 we'd get 34.

For a realistic example Subject: =?utf-8*xxx?q?yyy?= with the
2822upd-03 RECOMMENDED line length limit 78 that's 22 (or 21)
fixed characters for "Subject: =?utf-8*" + "?q?" + "?=", and
66 for the language tag xxx plus the encoded content yyy.

With 66-42 that's only 3*8 for yyy, or 8 octets for the UTF-8
content.  Splitting UTF-8 subjects into pieces of 2 or 3 Unicode
points per line for some cases would be a pain.  For a folded
line without the "subject:" we get 74-42 now, or 74-34=40 with
at most one extlang.  3*13 for 13 UTF-8 octets is much better.

For an oddity like "pc-multilingual-850+euro" (which isn't too
odd if you run CHCP on a command line under Windows, any "850"
is actually 850+euro) you'd get 78-41=37.  That never worked
in the first "subject:" line, 37 is smaller than 42.

In a folded line 45-42=3 just barely worked to encode a single
850+euro character per line.  With 37-34 or 45-34 we could do
three 850+euro characters per line.  Actually the 37-34 case
is _required_ for NetNews articles where it's not allowed to
fold a header field before the first non-white-space character
of the header field body.  Of course these theoretically worst
cases won't happen in practice, but I'd feel better if 4646bis
limits work also in theory (RFC 2231 encoded word in the first
line of a Netnews subject for "pc-multilingual-850+euro" and
the minimally supported language tag length).

Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 22 Sep 2007 00:03:22 +0200

John Cowan wrote:
 
> The whole point of the registry is that it makes it
> unnecessary -- indeed, incorrect -- to look at 639-[123],
> 3166-1, and 19542 to determine the referents of codes.
> (It may still be necessary to look at supplementary
> information to find out what "Ainu (China)" means, or
> what Gurmukhi script is, but that's a different level of
> hermeneutics.)

It's not more necessary, and in rare cases related to 
future "recycled" 3166-1 codes it could be even invalid.

But generally a valid 639-[12] (in that order) language
code is a valid 4646 language subtag, a valid 3166-1
code is a valid 4646 region subtag, and a valid 19542
code is a valid 4646 script subtag.

Minus scripts that's as it also was before RFC 4646, and
the registries only offered additional tags or subtags.

In essence RFC 4646 introduced a process to construct
tags from the various sources, solved potential conflicts
like the famous CS case, and added scripts + variants +
+ UN M49 continents + (in theory) extensions.  It fixed
some RFC 3066 issues guaranteeing stability.  

RFC 4646 didn't remove the general assumptions about valid
639-[12] language codes.

RFC 4646bis might do this, if some 639-3 language codes
will be registered as extlang subtags needing a prefix,
while other 639-3 codes will be registered as language
subtags.

For various reasons I think that's a bad idea, we should
not "freeze" info in the registry unless it's strictly
necessary to guarantee stability.  The "macrolanguage"
concept is more flexible than extlang-prefix relations
"frozen" at date C and repeated forever in 4646bis tags.

Admittedly the prefix-extlang notation could help with
matching near date C, and likely it's stable for many
years after date C and most affected tags, but at some
point in time there will be confusing differences between
"real" 639-3 macrolanguage sets and the then "obsolete"
prefix-extlang tags.
 
> 639-2 is no longer authoritative for RFC 4646 language 
> tags; only the LSR is.

I'm not aware of any 639-[12] code (in that order if both
alpha-2 and alpha-3 codes exist) that's not also a valid
RFC 4646 language subtag.  AFAIK the only way to create a
permanent difference is that the ISO 639 folks break the
promise published in RFC 3066 and 4646, and the language
subtag review process refuses to follow suite.

 Frank

---

From: John Cowan <[email-masked]>
Date: Fri, 21 Sep 2007 18:18:11 -0400

Frank Ellermann scripsit:

> RFC 4646 language subtag.  

I made an intensional claim and you replied with an extensional one.
All those who have hearts have livers and vice versa, but "having a heart"
and "having a liver" are not the same concept.

-- 
So that's the tune they play on                 John Cowan
their fascist banjos, is it?                    [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan