Subject: [Ltru] Re: John Cowan throws in the towel on extlangs

From: "Doug Ewell" <[email-masked]>
Date: Sat, 1 Dec 2007 09:16:29 -0800

John Cowan <cowan at ccil dot org> wrote:

>> 3) The audio content we tag with the "zh" tag is sometimes Cantonese.
>
> No problem.  When you know it is Cantonese, you'd be encouraged to 
> switch to "yue".   When you know it is Mandarin, you may use "cmn", 
> but that might cause interop problems with applications that don't 
> know that tag.

Isn't that equally true for "yue"?

>> 4)  We need a tag for written Cantonese used in subtitles, and today 
>> we use "zh-yue."
>
> That will still be valid, but deprecated; you'd be encouraged to 
> switch to "yue".

Then if they currently use "zh-cmn" for some Mandarin purpose, wouldn't 
they be equally encouraged to switch to "cmn"?

"zh" has historically been used to tag much more Mandarin than Cantonese 
or any other flavor of Chinese, but "zh" does not specifically mean 
Mandarin.  I was sure you and I agreed on that.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://home.roadrunner.com/~dewell
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 1 Dec 2007 09:45:48 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> Cross-references like that are generally considered a good thing, 
>> aren't they?  They reduce the chances of introducing a typo that 
>> changes the ABNF, and serve to discourage each application from 
>> adopting their own, slightly different syntax.
>
> Depends.  For 2616bis they could use a simplified syntax in the 
> direction of  tag = primary *( "-" auxiliary )  directly expanded in 
> tag = 1*8( ALPHA / DIGIT ) *( "-" 1*8( ALPHA / DIGIT ))  with a note 
> in prose that tags are actually supposed to be BCP 47 tags.

We tried that with the "grandfathered" production in 4646, and we got 
complaints from people unhappy with the loose syntax that appeared, 
taken out of context, to allow "en-a-b-c-d-e".  Having the real 
definition in prose didn't seem to help.  Now we have a separate 
"irregular" production that enumerates all the grandfathered tags that 
don't fit the rest of the syntax.

(BTW, are we going to remove the extlang production from the syntax and 
add a lot more items to the "irregular" list, or are we going to leave 
the production in place and cause people to scratch their heads 
wondering what it's there for?)

>> "Self-sufficiency" should be a complete non-goal.
>
> As proposed above, where is the problem ?

If their home-grown syntax doesn't agree with BCP 47, it's not clear 
which one people will believe.  Remember how many people thought RFC 
1766 and 3066 promised only "xx" and "xx-XX".

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 1 Dec 2007 09:49:19 -0800

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> Thanks, with Debbie's help we managed to avoid the ugliest traps and 
> pitfalls for RFC 4646.  For the DG+EA+IC+... zoo I hope that it won't 
> pass an IAB review.  The IETF isn't supposed to overrule RFC 1591 
> without first publishing a 1591bis (IMO, of couse, and obviously IANA 
> already managed this stunt with its odd ccTLD EU).

Don't forget their use of "UK" as a ccTLD instead of the ISO 3166 
standard "GB", and their use of other exceptionally reserved code 
elements such as "AC".

---

From: Peter Constable <[email-masked]>
Date: Sat, 1 Dec 2007 10:37:32 -0800

> From: Doug Ewell [mailto:[email-masked]]

> they be equally encouraged to switch to "cmn"?

There was a time when Traditional Chinese content was often tagged zh-TW and Simplified Chinese content was tagged zh-CN. There's been some transition in industry here, though I don't know how much inappropriate use of zh-TW or zh-CN still lingers. (Of course, in terms of this issue, zh-CN and zh-TW may still be appropriate for locale identification.)

Recommending a transition from zh to cmn, yue etc. would have some merits, but making that transition would be much more of an uphill climb.

> Mandarin.  I was sure you and I agreed on that.

And I'm quite certain John has not changed his thoughts in that regard.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sat, 1 Dec 2007 10:44:07 -0800

> wondering what it's there for?)

Well, that's up to this list to decide. It was what those of us in the teleconf had in mind. It looks to me like there hasn't been a clear statement of recommendations coming out of that teleconf, so I'll write separately on that.

> 1766 and 3066 promised only "xx" and "xx-XX".

+1

Peter

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 1 Dec 2007 23:23:12 +0100

Doug Ewell wrote:

>> For 2616bis they could use a simplified syntax in the direction
>> of  tag = primary *( "-" auxiliary )  directly expanded
>> in  tag = 1*8( ALPHA / DIGIT ) *( "-" 1*8( ALPHA / DIGIT ))  with
>> a note in prose that tags are actually supposed to be BCP 47 tags.
 
> We tried that with the "grandfathered" production in 4646, and
> we got complaints from people unhappy with the loose syntax that 
> appeared, taken out of context, to allow "en-a-b-c-d-e".

2616bis (http) mainly needs to *_transport_* tags, interpreting them
is IMO not the same layer.  We likely agree that 2616bis "MUST NOT" 
copy 4646 syntax, especially not while 4646bis rewrites major parts
of it.  They could try  tag = <as specified in BCP 47>  adding some
valid examples based on the current registry.  It's already clear
that they get rid of any 1*8( ALPHA ), and i-cherokee or en-cockney
in the latest 2616bis draft were also bad ideas.

> Now we have a separate "irregular" production that enumerates all
> the grandfathered tags that don't fit the rest of the syntax.

That list could be trimmed by regulating i-whatever instead of the
zero registered languages matching  5*8( ALPHA )  for i-uniling or
i-liaden.  If a registered language ever needs scripts, or other
subtags, then it most likely already qualifies for a proper 639-3
code.  My crystal ball says, hard to judge based on no real case.

> are we going to remove the extlang production from the syntax
> and add a lot more items to the "irregular" list

I'd hope that we do this.  We could mitigate it by building some
sublists, e.g. sgn-irregular / zh-irregular / other-irregular

> If their home-grown syntax doesn't agree with BCP 47, it's not
> clear which one people will believe.

Well, the generic URI syntax in STD 66 doesn't explain details
for various specific URI schemes, including schemes specified 
in STD 66.  Nothing's wrong with that, just a different level
of abstraction.  If you want to know how many servers handle the
name=value details of queries you're forced to test it with real
servers, it's not specified in STD 66.  If you want to know how
mailto URIs work (if at all) you have to look in Martin's draft,
and after that you've to judge how far away the I-D still is 
from anything related to real mailto-URIs.  But real mailto-URIs
not matching the generic STD 66 syntax are broken, if they work
or not, so it's not quite hopeless... :-)

> Remember how many people thought RFC 1766 and 3066 promised 
> only "xx" and "xx-XX".

Yes, but it wasn't too hard to learn that this wasn't the case,
with beautiful examples such as i-default or en-GB-oed.  If an
implementor doesn't bother to read the spec. trying to improve
the spec. might be a waste of time (unless it's an improvement
towards readability and KISS)

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 1 Dec 2007 23:55:25 +0100

Doug Ewell wrote:

> Don't forget their use of "UK" as a ccTLD instead of the 
> ISO 3166 standard "GB", and their use of other exceptionally
> reserved code elements such as "AC".

RFC 1591 was published 1994, I'm almost sure that these codes
are older, and that the author of RFC 1591 published it as a 
kind of "lessons learned" (among other reasons).  Jon Postel
was "IANA" for many years, see RFC 2468.

 Frank

---

From: John Cowan <[email-masked]>
Date: Sun, 2 Dec 2007 14:14:04 -0500

> they be equally encouraged to switch to "cmn"?

Yes.  However, people using "zh" to tag Mandarin shouldn't be encouraged
to switch to "cmn".

> "zh" has historically been used to tag much more Mandarin than Cantonese 

Partly because there *is* much more Mandarin (text) than Cantonese.

> but "zh" does not specifically mean Mandarin.  I was sure you and I
> agreed on that.

Yes, we do.

-- 
Here lies the Christian,                        John Cowan
        judge, and poet Peter,                  http://www.ccil.org/~cowan
Who broke the laws of God                       [email-masked]
        and man and metre.

---

From: John Cowan <[email-masked]>
Date: Sun, 2 Dec 2007 14:15:36 -0500

> standard "GB", and their use of other exceptionally reserved code 

The existence of the ".uk" domain (or more precisely the "uk." domain,
since JANET domain names were big-endian) predates the application
of 3166 to gTLDs.

-- 
Even the best of friends cannot                 John Cowan
attend each others' funeral.                    [email-masked]
        --Kehlog Albran, The Profit             http://www.ccil.org/~cowan

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 2 Dec 2007 21:58:17 -0800

John Cowan <cowan at ccil dot org> wrote:

>> Then if they currently use "zh-cmn" for some Mandarin purpose, 
>> wouldn't they be equally encouraged to switch to "cmn"?
>
> Yes.  However, people using "zh" to tag Mandarin shouldn't be 
> encouraged to switch to "cmn".

To me, and perhaps only to me, saying that people tagging Cantonese as 
"zh[-yue]" should be encouraged to switch to "yue", but people tagging 
Mandarin as "zh[-cmn]" should not be encouraged to switch to "cmn", is 
tantamount to saying that "zh" is a synonym for "cmn".  If it is not, 
I'd like to know how it is not.

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 3 Dec 2007 10:37:20 -0000

Doug wrote:

> Don't forget their use of "UK" as a ccTLD instead of the ISO
> 3166 standard "GB", and their use of other exceptionally

UK - exceptionally reserved - was used for the ccTLD because that code more
clearly marks the country (UK, not GB).  We are also assigned GB and that is
still in use by a legacy application in MoD.  There is no intention to give
this code up, but neither is there any intention to use it further.

Best

Debbie

> -----Original Message-----
> Sent: 01 December 2007 17:49
> To: LTRU Working Group
> Subject: [Ltru] RFC 1591 (was: Re: draft-ietf-ltru-4646bis-09.txt)
>
> Don't forget their use of "UK" as a ccTLD instead of the ISO
> 3166 standard "GB", and their use of other exceptionally
> Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *
> UTN #14 http://home.roadrunner.com/~dewell
> http://www.alvestrand.no/mailman/listinfo/ietf-languages  ^
>
>
>

---

From: John Cowan <[email-masked]>
Date: Thu, 13 Dec 2007 23:42:13 -0500

> I'd like to know how it is not.

No, it's saying that as a pragmatic matter, most "cmn" documents are tagged
"zh" for historical reasons, and that tagging your document "cmn" may make
it difficult to find.

-- 
Samuel Johnson on playing the violin:           John Cowan
"Difficult do you call it, Sir?                 [email-masked]
 I wish it were impossible."                    http://www.ccil.org/~cowan

---

From: [email-masked]
Date: Fri, 14 Dec 2007 09:40:29 -0800

Is the situation any different for Cantonese language?  Most Cantonese 
documents and speech are tagged zh or zh-yue. If you deprecate "zh", it 
might not be helpful for written language, but it's a necessity to have 
two separate tags for tagged spoken language. With the explosion of online 
video, tagged audio content is increasing.

Regards,

Karen Broome 

John Cowan <[email-masked]> 
12/13/2007 08:42 PM

To
Doug Ewell <[email-masked]>
cc
LTRU Working Group <[email-masked]>
Subject
[Ltru] Re: John Cowan throws in the towel on extlangs

> I'd like to know how it is not.

No, it's saying that as a pragmatic matter, most "cmn" documents are 
tagged
"zh" for historical reasons, and that tagging your document "cmn" may make
it difficult to find.

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Dec 2007 15:10:58 -0500

> video, tagged audio content is increasing.

1) zh is not being deprecated.

2) zh-yue is being deprecated in favor of plain yue.

3) zh-cmn is being deprecated in favor of cmn.

4) People should think twice about using cmn rather than plain zh.

I think that's a fair summary.

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.ccil.org/~cowan
Men sleep without watch all night?

---

From: Peter Constable <[email-masked]>
Date: Fri, 14 Dec 2007 14:01:39 -0800

> From: John Cowan [mailto:[email-masked]]

> 4) People should think twice about using cmn rather than plain zh.

I mentioned on 12/8 that there are other cases that also need consideration. Here's what I wrote then:

<quote>
There is a deeper problem we should be considering: when does it make sense for people to use the macro-language or the individual-language IDs in cases like Swahili, Malay, etc.? In fact, when does it make sense to use which in cases for any macrolanguage ID in widespread use when there is a dominant encompassed language? This is an issue in these cases:

ar/ara vs. arb (Arabic)
kok vs. Knn (Konkani)
ms/msa vs. mly (Malay)
sw/swa vs. swh (Swahili)
uz/uzb vs. uzn (Uzbek)
zh/zho vs. cmn (Chinese)

These are all the cases in category 2 of 'Macrolanguage analysis.txt' which I sent as an attachment on 11/29. I didn't include Dogri as I had no indication that doi is widely used, but we can certainly include it for consideration of this issue.
</quote>

>From what you say in (4), it seems your suggestion would be that people should think twice about using the individual-language ID rather than the macrolanguage ID in *any* of these cases.

Peter