Subject: [Ltru] Re: Suggested text for future compatibility of registry processors.

From: "Doug Ewell" <[email-masked]>
Date: Sat, 14 Jul 2007 16:18:08 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> While catching up on my backlog I notice the following note from Mark.
>
> I would note that we already have a shorter equivalent of this text at 
> the end of 3.1.2, where it says:
>
> --
> Future versions of this document might add additional fields to the 
> registry, so implementations SHOULD ignore fields found in the 
> registry that are not defined in this document.
> --
>
> This text was apparently added back in draft-02. Any reason to edit it 
> some more?

This is fine, and the right thing to do for 4646bis, but the point still 
remains that this wording isn't in place for the current spec, RFC 4646, 
and so adding new fields will break strictly 4646-conformant processors 
in the same way that converting the Registry to UTF-8 will.  Note that I 
didn't say either change is a bad thing, or unjustified.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Sat, 14 Jul 2007 19:32:55 -0400

> and so adding new fields will break strictly 4646-conformant processors 

I don't think so.  There is no claim in 4646 that there are no other
fields.  "Be liberal in what you accept" is a general IETF principle.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 14 Jul 2007 23:02:19 -0700

> fields.  "Be liberal in what you accept" is a general IETF principle.

I would have sworn there was such a claim in 4646, but all I can find 
now is inclusionary wording in Section 3.1:

Each record MUST contain the following fields:
   o  'Type'
   o  Either 'Subtag' or 'Tag'
   o  Description
   o  Added
Each record MAY also contain the following fields:
   o  Preferred-Value
   o  Deprecated
   o  Prefix
   o  Comments
   o  Suppress-Script

implying (at least to me) that each record MAY NOT contain any other 
fields.  This may not be as strict as I interpret it, but in any case 
there is no explicit wording (unlike in draft-4646bis) that other fields 
may be added in the future and should be ignored by processors that 
don't recognize them.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 14 Jul 2007 23:21:09 -0700

Mark Davis <mark dot davis at icu dash project dot org> wrote:

> Addision and I have discussed the issue of extlang and Macrolanguages 
> and are proposing the following text replacing the use of extlang.

I won't object to this proposed solution.  It is certainly not as bad as 
some of the alternatives that were mentioned, such as an "Encompasses:" 
field with 30 or 40 comma-separated language subtags.  We have to decide 
on something and move along.

It still floors me that we expect any parsers at all to be able to match 
"yue" with "zh" but not to be able to pick "en-US" out of "en-Latn-US". 
And of course all of the grandfathered tags of the form "zh-yue" will 
now have to be deprecated in favor of "yue", instead of being made 
redundant.

---

From: John Cowan <[email-masked]>
Date: Sun, 15 Jul 2007 11:23:01 -0400

Doug Ewell scripsit:

> I won't object to this proposed solution.  

I don't object to having a Macrolanguage header as such, and even to
making it mutable (and therefore informative).

I do most strenuously object to upsetting the extlang applecart.  Extlangs
are a sensible and necessary shim between the 639-2-only world of 4646
and the mixed 639-2/3 world of 4646bis, and I have seen no compelling
reasons why they should be discarded.  We have been planning them for
years, and we should stick to our plan unless there is some hard reason
to change it.

> now have to be deprecated in favor of "yue", instead of being made 

Exactly.

-- 
Possession is said to be nine points of the law,                John Cowan
but that's not saying how many points the law might have.       [email-masked]
        --Thomas A. Cowan (law professor and my father)

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 15 Jul 2007 15:20:28 -0700

The proposed text makes no special arrangement for sign languages, but 
ties the Macrolanguage field to ISO 639-3 assignments.  Shall I assume, 
using American Sign Language as an example, that the plan is to 
deprecate "sgn-US" and assign it a Preferred-Value of "ase", but not to 
give "ase" a Macrolanguage field?

%%
Type: language
Subtag: ase
Description: American Sign Language
Added: 20xx-xx-xx
%%
Type: redundant
Tag: sgn-US
Description: American Sign Language
Added: 2001-03-02
Preferred-Value: ase
Deprecated: 20xx-xx-xx
%%

How should people then interpret the existing language subtag "sgn"?

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 15 Jul 2007 16:12:40 -0700

> years, and we should stick to our plan unless there is some hard reason

The main argument I've hear for extlang is behind-the-scenes-inertia. While
we made provision in 4646 for possibly accepting them in the future, it was
by no means a done-deal.

The only reason I've heard advocated for them is that it makes matching
easier. But in practice, we have simply not found that to be true. If it is
indeed  worthwhile to add this mechanism to 4646, a good case needs to be
made for it; and inertia isn't a good case.

Mark

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 15 Jul 2007 16:33:14 -0700

Mark Davis wrote:

> The main argument I've hear for extlang is behind-the-scenes-inertia. 
> While we made provision in 4646 for possibly accepting them in the 
> future, it was by no means a done-deal.

The main argument that has been offered is that it makes matching 
easier.  Whether that argument was heard is a different question.

> The only reason I've heard advocated for them is that it makes 
> matching easier. But in practice, we have simply not found that to be 
> true. If it is indeed  worthwhile to add this mechanism to 4646, a 
> good case needs to be made for it; and inertia isn't a good case.

Here is the argument restated.  It is based not on behind-the-scenes 
inertia, but on backward compatibility, the exact same issue that caused 
us to adopt Suppress-Script.

Existing Cantonese text has been tagged as "zh", the basic ISO 
639-1-based tag, or as "zh-yue", the tag that was registered for this 
purpose back in 1999.

The extlang mechanism would have established "yue" as an extlang under 
"zh", so the proper tagging of Cantonese would continue to be "zh" (more 
general) or "zh-yue" (more specific).  Matching engines would continue 
to operate as they do now.

The proposed mechanism establishes "yue" as a primary language subtag, 
so the proper tagging of Cantonese becomes "yue".  Matching engines must 
be upgraded to RFC 4646bis in order to have any chance at finding a 
match.  The much-beloved and much-catered-to RFC 3066 remove-from-right 
"fallback" algorithm will NEVER find a match between "yue" and "zh", 
regardless of whether script and/or region subtags are involved.

Put another way: Extlangs may not make matching easier, but *not* having 
extlangs will make matching harder.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 16 Jul 2007 11:31:15 +0200

On Sat, Jul 14, 2007 at 07:32:55PM -0400,
 John Cowan <[email-masked]> wrote 
 a message of 19 lines which said:

> "Be liberal in what you accept" is a general IETF principle.

And not always a good one, specially in the case of the LSR where
there is just one supplier of data. Being liberal when you accept
stuff from many unknown clients is one thing, when there is only one,
the IANA, I doubt it is a good idea.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 16 Jul 2007 11:33:43 +0200

On Sat, Jul 14, 2007 at 11:21:09PM -0700,
 Doug Ewell <[email-masked]> wrote 
 a message of 28 lines which said:

> It is certainly not as bad as some of the alternatives that were
> mentioned, such as an "Encompasses:" field with 30 or 40
> comma-separated language subtags.  We have to decide on something
> and move along.

Speaking as someone who likes databases, I can say it seems better to
have the information in the "encompassed language" record rather than
in the "macrolanguage" record. That way, adding a record will not
require to modify *another* record.

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Jul 2007 08:52:21 -0700

It's not clear to me why matching

(A) zh-Hans-CN with zh-yue-Hans-CN

is much harder than matching

(B) zh-Hans-CN with yue-Hans-CN.

In the (B) case, you need to treat "zh" and "yue" as a match at some level. In the (A) case, you need to treat "zh" and "zh-yue" as a match at that same level. The only added work for (A) is that you need to recognize "zh-yue" as the entity to be compared. But then, for (B) you still need to recognize that "yue" may match something other than "yue". There is some extra work either way.

The RFC 1766/3066 remove-from-right algorithm won't get a match in case (A), but neither will it do so in case (B). At least with the approach used in (A), it will match zh and zh-yue, whereas it won't match zh and yue, as Doug pointed out.

On the other hand...

It's not clear how important the zh/zh-yue versus zh/yue example is for Chinese: it makes an valid argument against (B) if existing content is tagged "zh-yue"/"zh-yue-Han?" and a language range "zh" is used, but says nothing about cases of content tagged "zh"/"zh-Han?"/"zh-??" and a language range "zh-yue". If we're concerned about legacy content created before the extlang "yue" was introduced, that would be the latter.

Mark's and Addison's proposal does have a couple of points in its favour -- these may or may not be significant:

- Whereas using an extlang in a case like "zh-yue" was easy to consider, not all other cases will necessarily be as easy. (Clearly we don't want to introduce macrolanguage/extlang pairs for Norwegian or Serbo-Croatian, but I think we've assumed all along these can be treated as grandfathered cases.) Mark's proposal means we're never required to consider whether "xxx-yyy" is really helpful: we just always use "yyy".

- If there's ever a case in which ISO 639 has related languages "xxx" and "yyy" and a new macrolanguage "mmm" encompassing the two is later added, we don't need to retag existing content as "mmm-xxx" and "mmm-yyy", nor do we need to do any special-case processing with the registry to keep "mmm-xxx" and "mmm-yyy" from being used. And neither do implementers need to introduce new mechanisms to get appropriate matches between "mmm" and "xxx"/"yyy" since, under M&A's proposal, they already would have a general mechanism in their matching process for this; all they need is to incorporate the new data.

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 17 Jul 2007 12:19:46 -0700

I just find your position hard to understand, perhaps you can help me. Maybe
some scenarios would help.

Suppose that:

   - My site has support for zh, zh-Hans, and zh-Hant. zh = Mandarin
   since that is what everyone means by "zh" currently. As is customary in
   fallback, the content of zh is the predominent form (zh-Hans) [this is just
   for example; if a TW site has a different convention alternate examples can
   be given.
   - A user comes in with different requests, listed below.

Scenario 1. The user's browser has the proposed "zh-yue-Hant-US". My lookup
falls back to zh, so I serve it up to the user. So even if the target of the
match (zh) is not Cantonese, you want a fallback to zh. I'm guessing that
you see this as better than if we defined the tag as "yue-Hant-US", since it
gets to some fallback that the user is likely to understand. But I don't see
this as much different than if we had fr-br-BE (meaning Breton, but fall
back to French), or ro-mo (meaning Moldavian, but falling back to Romanian).
And note that in the fallback, the script and region are completely lost.

Scenario 2. The user's browser has zh-cmn-Hant-US. In matching, we fall back
to zh. Note than in the fallback, the script and region are completely lost.
We have essentially just introduced a synonym for zh which causes fallback
to lose information, for no good reason.

The problem with extlang is that the fallback from encompassed language to
macrolanguage is fundamentally different in kind than a fallback from region
to script to base language. In the case of script, like uz-Arab and uz-Latn,
or en-US vs en-GB, we really have variations on the same language, and
fallback makes sense. We ordered the subtags so that it works optimally
overall.

The encompassed languages, on the other hand, are not just dialects, not
just variants. They are languages in their own right. Trying to insert them
into the fallback process just screws things up, because they need a
"sideways" matching not just simple truncation fallback. If you want to do
any fallback with extlang, it would be to fall back from zh-yue-<other
stuff> to zh-<other stuff>. That means that in order to do reasonable
fallback, you can't just use truncation fallback anyway. So I see the
situation this way:

   1. The only reason for adding the complication of the extlang
   mechanism is to make truncation fallback work better.
   2. Truncation fallback with extlang doesn't work better.
   3. So there is no need to make encompassed languages be "secondary"
   languages by making them be "secondary" subtags.

The goals of extlang are good, to make matching work better, but in practice
it just makes things worse. [Speaking to those familiar with C++, it feels a
bit like the default assignment operator in C++. Nice in theory, but in
practice it gums things up more than it fixes, since once you are beyond
very simple (toy) classes, the default is almost always wrong -- but because
it is supplied behind your back you don't realize it.]

So instead of adding the extlang mechanism to RFC 4646, what we really need
to do is to point people to how to handle yue and other encompassed
languages along with mo/ro, tl/fil, and other edge cases in a reasonable
way, by augmenting matching.

Mark

On 7/15/07, Doug Ewell <[email-masked]> wrote:
> http://users.adelphia.net/~dewell/ <http://users.adelphia.net/%7Edewell/>

---

From: Addison Phillips <[email-masked]>
Date: Tue, 17 Jul 2007 12:34:16 -0700

So far I've let Mark take this thread on, but now I have a few minutes 
to chime in.

> making it mutable (and therefore informative).

Good, although I note that it can't be (as) informative if we have 
extlangs. I mean, we could allow extlangs to lose their encompassed 
status, but not their Prefix. This is a problem we don't have to have if 
we don't have extlangs.

> years, and we should stick to our plan unless there is some hard reason

Up until now I've been pushing the extlang applecart (as it were), so 
Mark's message marks a departure for me. The problem is whether the shim 
is actually necessary.

With extlangs we get the benefit of the subtag hierarchy being visible. 
Users are required to retag data to get full benefit from the new 
subtags, but they aren't *required* to retag their data. Neither are 
they required to retag their data if we just include all of the 639-3 
codes as primary language subtags and forego extlangs.

For most of the affected languages, extlangs are nothing but a hassle. 
Rather limited amounts of content are tagged with the macrolanguage, let 
alone with the enclosed sub language. Users can choose between primary 
language subtags on their own and need not use complex (and possibly not 
well-understood) subtag sequences. The various Hmongs, Quechuas, and 
Zapotecs fall into this category.

The languages which are really affected are Arabic and Chinese. For 
Arabic, I think we could deprecate (or just frown at) the use of 'arb' 
(Standard Arabic) and let people choose regional Arabic dialects--or 
just use 'ar-*'.

So we're left with Chinese, which is a huge sticky wicket. It's quite 
clear that there is a *lot* of content tagged with 'zh-*'. It's also 
clear that, for written content, we can maintain the "fiction" of 
Chinese and basically ignore regional languages/dialects for most 
tagging purposes. Users who really need the different Sinitic languages 
can tag them using the appropriate primary language subtag.

The people who are hurt by this are those with non-written content, such 
as audio or video recordings, or similar kinds of applications (such as 
text-to-speech). For these sorts of applications, the users will have to 
retag their data. Either they use extlangs or they just use primary 
langauge subtags. It does mean that I might get a DVD with:

   <subtitles xml:lang="zh-Hant-HK" />
   <audio xml:lang="yue-HK" />

This situation isn't that different, I might as well note, as the one 
for Norwegian (no/nn/nb) and some other languages. The range of tag 
choices is appalling at times, but not intractable.

Either of these solutions would be acceptable to me and each involves 
some level of compromise. However, with only primary language subtags, 
we don't have to invent any silly rules or procedures for stability, 
maintenance, or choosing subtag types. We don't have to cherry-pick and 
we aren't reliant on ISO 639 being picture perfect with their 
macrolanguage definitions.

And I am concerned that users have a hard enough time figuring out 
language tags as it is. Extlang is another level of complexity that 
users will not fully understand. So that's what starts me leaning away 
from extlangs.

In addition, my guess is that the world mostly won't notice. Chinese 
will mostly be tagged as "zh-*". Some Cantonese, Min-Nan, Hakka, Xiang, 
etc. users will use the available subtags, but mostly in a recognizable 
context. No matter which solution we choose, we'll have to write 
extensive documents called, roughly, "HOW TO TAG YOUR CHINESE" :-).

>> now have to be deprecated in favor of "yue", instead of being made 

And this is bad because...?

The truth is, "yue" (or, more importantly, "cmn") and "zh" don't match. 
And neither do "no" and "nb". Which is semantically silly, but not 
fatal. For that matter, if 'cmn' or 'yue' mark an actual linguistic 
distinction, either "zh" != "zh-cmn" or "zh" matches both "zh-yue" and 
"zh-cmn". Any change to tagging yields new matching problems. If, for 
the most part, people do not need to change their tagging behavior, then 
that's best.

Note that the proposed text probably isn't strong enough in this area. 
It should probably say something like:

--
In most cases, use the Macrolanguage to form the language tag in 
preference to the encompassed language. Only use the encompassed 
language if it adds useful distinguishing information to the tag within 
your application.
--

Note that one can (and should) as easily s/use the encompassed 
language/use the extlang subtag/

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 17 Jul 2007 13:23:58 -0700

Hi -

(as a technical contributor)

Perhaps it would be helpful to take a step back and consider
some goals (and non-goals) here.  For example:

Goal 1a: do not trigger retagging of existing ar-* and zh-* materials.
Goal 1b: do not trigger retagging of existing tagged material in other languages
Goal 2: avoid introducing "synonym" tags - there should be a single tag for a given language
Goal 3: minimize changes to matching specification

Non-goal 1: using the registry to solve the general problem of fallback
Non-goal 2: identification of all language varieties in a single subtag

I also think it should be a non-goal that the tag for a language being introduced to
the registry for the first time would have to look like existing tags.  Since
by definition there's no legacy of existing tagged data to worry about (goals
1a and 1b don't apply) we should strive to uphold goal 1c in whatever scheme
we decide to use.

I think the key issue underneath this is whether there are any cases similar to
ar-* or zh-* in the data to be added, and whether supporting those cases in
a similar manner is worthwhile.  Rather than a theoretical discussion of fallbacks
in general, I'd like to see identification of the specific worthwhile cases among
the languages slated for addition.  If there are such beasts and they are deemed
worthwhile, we should make provision in the registry format and procedures to
handle them.  If not, then we should go with the simplest approach that will
let us attain our other goals.

Randy

---

From: John Cowan <[email-masked]>
Date: Wed, 18 Jul 2007 01:14:26 -0400

> status, but not their Prefix. This is a problem we don't have to have if 

Not necessarily.  For example, 639-3/RA might decide to group the
four Min Chinese languages (cdo, czo, mnp, nan) as a macrolanguage
within the general zh macrolanguage.  In that case, we would change the
Macrolanguage: field for these four languages from zh to whatever the
new code is, but we would leave the Prefix: field at zh.  So Prefix:
will remain normative and Macrolanguage: informative.

> codes as primary language subtags and forego extlangs.

Indeed, without extlangs it is a disadvantage to tag your data correctly,
and that's the main argument for having them.  Suppose that you have
a written document in Cantonese.  The difference between that and the
equivalent Mandarin document is not zero, but it will be plausible
that someone who can read Mandarin will be able to make sense of it.
If you tag it zh, on the one hand, such people will discover it; if
you tag it yue, on the other, they will not discover it unless they are
using advanced matchers.  But on the gripping hand, if you use zh-yue
(which you may, because that is a grandfathered tag), you combine the
advantages of discoverability and correctness.  Allowing extlangs extends
the advantage from zh-yue to the other zh languages.  Currently they
are not often written, but "not often" is not "never".

> Zapotecs fall into this category.

Actually, you don't know that.  There are a ferocious number of books
in the world, and 639-2 began as a bibliographic standard.  The whole
point of tags like "hmn", "qu", and "zap" is that bibliographers often
don't know exactly which language they are dealing with, but a partial
tag is much better than nothing.

It's important not to take a parochial view of language tagging, as if
only electronic resources count.

> (Standard Arabic) and let people choose regional Arabic dialects--or 

Standard Arabic is the most common kind of written Arabic, but it
is rather rare as spoken Arabic goes.  My guess would be that
a plurality of recorded spoken Arabic is Egyptian Arabic.

> for Norwegian (no/nn/nb) and some other languages. 

Eh?  The distinction between nn and nb is strictly written.  Spoken
Norwegian is a congeries of dialects with none really dominant,
much like spoken American English.

> macrolanguage definitions.

You overestimate the problem.  Once we get past the initial 4645bis load, we
never have to do an extlang again, except in the unlikely case
(and I would be OK with ignoring this) that a new macrolanguage
is created encompassing a bunch of languages created at the same
time; e.g. 639/RA adds the macrolanguage Foovian encompassing
Barvian and Bazvian.

> users will not fully understand. So that's what starts me leaning away 

This is also an overestimation.  There will be an entry for Cree
and another for Plains Cree: if you know you have Cree, use
cre; if you further know you have Plains Creee, use cre-crk.
That's not much more complicated than saying "use crk".

> extensive documents called, roughly, "HOW TO TAG YOUR CHINESE" :-).

Agreed.

I'm happy with the "use the extlang subtag" version of this.

-- 
Your worships will perhaps be thinking          John Cowan
that it is an easy thing to blow up a dog?      http://www.ccil.org/~cowan
[Or] to write a book?
    --Don Quixote, Introduction                 [email-masked]

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 17 Jul 2007 22:41:18 -0700

Mark Davis wrote:

> zh = Mandarin since that is what everyone means by "zh" currently.

Hold that thought.

> Scenario 1. The user's browser has the proposed "zh-yue-Hant-US". My 
> lookup falls back to zh, so I serve it up to the user. So even if the 
> target of the match (zh) is not Cantonese, you want a fallback to zh. 
> I'm guessing that you see this as better than if we defined the tag as 
> "yue-Hant-US", since it gets to some fallback that the user is likely 
> to understand. But I don't see this as much different than if we had 
> fr-br-BE (meaning Breton, but fall back to French), or ro-mo (meaning 
> Moldavian, but falling back to Romanian).

You're right: providing a fallback from encompassed languages back to 
their macrolanguages (as defined by Ethnologue) doesn't extend to all 
other imaginable fallback scenarios.  I'm pretty sure nobody ever 
claimed that it would.

> Scenario 2. The user's browser has zh-cmn-Hant-US. In matching, we 
> fall back to zh. Note than in the fallback, the script and region are 
> completely lost.

Both of these scenarios assume that script and/or region subtags are 
likely to be present.  You haven't addressed my scenario, where they are 
not.

> We have essentially just introduced a synonym for zh which causes 
> fallback to lose information, for no good reason.

But according to your earlier statement, "cmn" is already effectively a 
synonym for "zh".

> The problem with extlang is that the fallback from encompassed 
> language to macrolanguage is fundamentally different in kind than a 
> fallback from region to script to base language. In the case of 
> script, like uz-Arab and uz-Latn, or en-US vs en-GB, we really have 
> variations on the same language, and fallback makes sense. We ordered 
> the subtags so that it works optimally overall.

Agreed.

> The encompassed languages, on the other hand, are not just dialects, 
> not just variants. They are languages in their own right.

Agreed to a certain point.  The whole idea of encompassed languages is 
that people sometimes consider them to be languages in their own right, 
and sometimes as "dialects" or "variants" of a macrolanguage.  I hate to 
keep harping on Chinese, but try searching on "Chinese dialects" and 
"Chinese languages" and see which search is more likely to tell you more 
about Mandarin vs. Cantonese vs. Wu vs. Hakka.  There are a *lot* of 
people who think of these as variants of a single language.  And the 
same is likely to be true for Standard Arabic vs. Algerian Arabic vs. 
Libyan Arabic vs. Uzbeki Arabic.

> Trying to insert them into the fallback process just screws things up, 
> because they need a "sideways" matching not just simple truncation 
> fallback. If you want to do any fallback with extlang, it would be to 
> fall back from zh-yue-<other stuff> to zh-<other stuff>. That means 
> that in order to do reasonable fallback, you can't just use truncation 
> fallback anyway.

Agreed.  Then again, I'm not the biggest fan of truncation fallback.

> So I see the situation this way:
>
> * The only reason for adding the complication of the extlang mechanism 
> is to make truncation fallback work better.
>
> * Truncation fallback with extlang doesn't work better.

I claim it does work better if there are no script or region or variant 
subtags, and is no worse even if there are.

> So there is no need to make encompassed languages be "secondary" 

I do agree with the possible public perception that extended languages 
are second-class in some way, although of course that is not the intent.

> So instead of adding the extlang mechanism to RFC 4646, what we really 
> need to do is to point people to how to handle yue and other 
> encompassed languages along with mo/ro, tl/fil, and other edge cases 
> in a reasonable way, by augmenting matching.

Well, you'll never get me to disagree that matching should be more 
sophisticated than RFR truncation.  But I hope you're not suggesting 
that we add an RFC 4647bis to our plate at this late date.  Look how 
much time we burned on RFC 4647, and look at the end result: we're still 
designing everything under the assumption that matching engines will be 
limited to RFR.

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 19 Jul 2007 21:47:45 -0700

Here are some comments on Mark and Addison's proposed Macrolanguage 
section.  All of this is presuming that the Macrolanguage field is in 
and the extlang mechanism is out; none of these comments is meant as an 
argument against this course of action.

> The Macrolanguage field contains a primary language subtag that 
> *encompasses* this subtag. That is, this language is a dialect or 
> sub-language of the Macrolanguage, and is called an *encompassed* 
> subtag.

I think we need to adhere very closely to the ISO 639-3 wording rather 
than coming up with our own definition of macrolanguage.  And the ISO 
639-3 wording is fairly clear that encompassed languages are not 
dialects:

"The linguistic varieties denoted by each of the identifiers in this 
part of ISO 639 are assumed to be distinct languages and not dialects of 
other languages, even though for some purposes some users may consider a 
variety listed in this part of ISO 639 to be a "dialect" rather than a 
"language"....  The dialects of a language are included within the 
denotation represented by the identifier for that language. Thus, each 
language identifier represents the complete range of all the spoken or 
written varieties of that language, including any standardized form."

The ISO 639-3 explanation of macrolanguage is at 
http://www.sil.org/iso639-3/scope.asp#M.  Basically the determining 
factor is that the encompassed languages are (a) considered to be 
different languages in some contexts, or by some people, and (b) 
considered to be a single language in (or by) others.  That is the 
concept we need to emphasize, not the standard vs. regional 
relationship.  This is important to help guide users toward the correct 
understanding and choice of macrolanguages and encompassed languages.

> Only values assigned by ISO 639-3 will be considered for inclusion.

I note again that this precludes assigning a macrolanguage of "sgn" to 
any of the 124 sign languages that have ISO 639-3 identifiers, and even 
though it's often a good idea to avoid special exceptions, this 
particular case worries me.

> For example, the language subtags 'nb' (Norwegian Bokmal) and 'nn' 
> (Norwegian Nynorsk) has a Macrolanguage entry of 'no' (Norwegian).

"subtags... have"

> Nor does it define how the encompassed languages are related to 
> one-another.

Remove hyphen.

> In some cases, the Macrolanguage has a standard form as well as a 
> variety of less-common dialects.

"varieties"

> In other cases there is no particular standard form and the 
> encompassed subtags describe specific variations within the parent 
> language.

But again, there must always be the condition that the variations are 
sometimes considered to be a single language.  That's what makes it a 
macrolangauge relationship, and not a collection or something we should 
be using variants for.

> Care in selecting which subtags are used is crucial to 
> interoperability. In general, use the most specific tag. However, 
> where the standard written form of an encompassed language is captured 
> by the Macrolanguage, the Macrolanguage should still be used for 
> written material.

I feel this is too concrete; it almost feels like "ALWAYS use the 
specific tag, except when you MUST NOT."  John pointed out that taggers 
who use the more specific "yue" instead of the more general "zh" might 
be putting themselves at a disadvantage, considering that we expect few 
matching engines to understand how to use the Macrolanguage field (at 
least initially).  This is true even for spoken material, not just 
written.

> In particular, chinese language(s) and dialects call for special 
> consideration.

Uppercase "Chinese," and get rid of "dialects."  ISO 639-3 considers 
them languages, and we need to adhere to their model.

> ... languages such as 'yue' (Cantonese) have usually used tags 
> beginning with the subtag 'zh'. This past practice of tagging...

It is still the current practice.  I assume this wording is meant for an 
envisioned future when everyone knows which subtag to use: "Remember way 
back in 2007 when we HAD to tag Cantonese as 'zh'?"  But I don't see 
this tagging practice going away any time soon, and so I suggest the 
word "past" be removed.  This will drive the point home even further 
that filtering and lookup engines need to understand the Macrolanguage 
field and do the right thing with it.

> For example, the information that 'yue' has a macrolangauge of 'zh' 
> could be used in the Lookup algorithm to fallback from a request for 
> "yue-Hans-CN" to "zh-Hans-CN"

"Fall back" is two words when used as a verb.

> For example, in a given application the best fallback for "be" 
> (Breton), may be "fr" (French) -- rather than the more closely related 
> "cy" (Welsh) -- because...

For a smoother read IMHO, remove the intrusive second comma and then 
convert the dashes to commas:

"For example, in a given application the best fallback for "be" (Breton) 
may be 'fr' (French), rather than the more closely related 'cy' (Welsh), 
because..."

> ... Breton readers are far more likely to be able to read French than 
> Welsh.

"Far" sounds hyperbolic and presumptive, even if it is provably true. 
Remove it; the sentence still carries its full impact without it.

---

From: John Cowan <[email-masked]>
Date: Fri, 20 Jul 2007 01:00:58 -0400

> argument against this course of action.

+1 except as noted.

> macrolangauge relationship, and not a collection or something we should 

These revisions don't go nearly far enough.  I think we'd need
something like:

	Sometimes one language encompassed by the macrolanguage
	is sociologically dominant over the others, perhaps because
	it has more speakers or is closest to the standard written
	form that represents the macrolanguage as a whole.
	In other cases this is not so.

I continue to support the use of extlangs.

-- 
What asininity could I have uttered     John Cowan <[email-masked]>
that they applaud me thus?              http://www.ccil.org/~cowan
        --Phocion, Greek orator

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 20 Jul 2007 12:20:26 +0200

> > (Norwegian Nynorsk) has a Macrolanguage entry of 'no' (Norwegian).

Should that not be "each of ... subtags ... has"?

With this particular example it should be noted that both
the spoken and the written languages are different. (So 'no'
would be better as a (rather small) collection than a
macrolanguage... Note that for Frisian, as similar "fix"
was made, but the old code was reassigned to one of the
languages.)

> > variety of less-common dialects.

They weren't regarded as dialects, were they. See Doug's earlier
and later remarks on that (not quoted here).

	/kent k

---

From: John Cowan <[email-masked]>
Date: Fri, 20 Jul 2007 09:40:19 -0400

> Should that not be "each of ... subtags ... has"?

That adds emphasis, but Doug's version is not incorrect.

> macrolanguage... 

Not so, for we speak of the Norwegian language, not the Norwegian
languages; and yet Bokmaal and Nynorsk are separate (written)
languages.  This is the very essence of what it is to be a macrolanguage.

-- 
He made the Legislature meet at one-horse       John Cowan
tank-towns out in the alfalfa belt, so that     [email-masked]
hardly nobody could get there and most of       http://www.ccil.org/~cowan
the leaders would stay home and let him go      --H.L. Mencken's
to work and do things as he pleased.              Declaration of Independence