Subject: [Ltru] A radical proposal (was: Re: extlang & deprecation)

From: "Doug Ewell" <[email-masked]>
Date: Tue, 1 Jul 2008 23:34:00 -0600

What about this:
(Warning: some of these suggestions are radical, but meant to reach a 
workable accord)

1.  Provide an explicit definition for the word "deprecated" in BCP 47, 
something on the order of "There is another tag or subtag that means the 
same as this one, but is preferred, and validating processors will 
translate this subtag into the preferred equivalent.  But nobody will 
beat you up or make you register with the police if you use the 
non-preferred one instead."  Make sure other text in the draft does not 
contradict this.

2.  Define all of the Y forms (e.g. "cmn") in the Registry as Type: 
extlang, none as Type: language.  Bear with me here.

3.  Change the prose (and ABNF if necessary) to specify that an 
"extlang" can be used either alone, as "cmn" (Y), or together with its 
Prefix, as "zh-cmn" (X-Y).  In other words, an "extlang" is a special 
type of "language" that has an optional Prefix.  Specify that the Y form 
is "preferred" and the X-Y form is "deprecated," using the clearly 
specified, watered-down definition of "deprecated" from item 1.

4.  Apply this logic to all encompassed languages, not just those 
belonging to a cherry-picked set of macrolanguages plus "sgn".  For 
extra credit, figure out how to make this work with changing ISO 639-3 
groupings.

This would allow both extlangistas and non-extlangistas to tag as needed 
to achieve the desired matching and fallback behavior, and would solve 
Addison's concern about being able to normalize without the Registry, 
since all valid tags of form XX(X)-YYY could be normalized to YYY 
blindly.  Determining validity in the first place still requires the 
Registry, but there is no way around that anyway.

Part 3 prevents needless and confusing duplication in the Registry, a 
problem which would be made worse by Part 4.

Part 4 removes arbitrariness, partly to help this document survive IETF 
LC, but also for its own sake.  I am sensitive to Addison's point that 
we should have as few duplicate tags as possible, but I still feel this 
is an adequate compromise, in which the extlangistas concede the formal 
"preferability" of the non-extlangy forms but reserve the right to use 
the easily normalized, deprecated forms that they prefer without being 
branded with a scarlet D.

Parts 2 and 3 must be implemented together or not at all.  Part 1 is 
severable from the rest, not all that radical, and probably a good idea 
no matter what else we do.  Part 4 is also severable from the rest.

I've put on my asbestos tactical vest and am now ready for WG feedback. 
Please avoid personal attacks; they don't solve technical problems.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 2 Jul 2008 10:18:06 +0200

Doug Ewell wrote:
> type of "language" that has an optional Prefix. [...]

I don't think the -16 ABNF needs to be changed at all for that,
just the prose.

	/kent k

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 2 Jul 2008 08:31:20 -0700

It sounds reasonable to me.

Mark

On Tue, Jul 1, 2008 at 10:34 PM, Doug Ewell <[email-masked]> wrote:

> same as this one, but is preferred, and validating processors will translate
> this subtag into the preferred equivalent.  But nobody will beat you up or
> make you register with the police if you use the non-preferred one instead."
>  Make sure other text in the draft does not contradict this.
>
> 3.  Change the prose (and ABNF if necessary) to specify that an "extlang"
> can be used either alone, as "cmn" (Y), or together with its Prefix, as
> "zh-cmn" (X-Y).  In other words, an "extlang" is a special type of
> "language" that has an optional Prefix.  Specify that the Y form is
> "preferred" and the X-Y form is "deprecated," using the clearly specified,
> watered-down definition of "deprecated" from item 1.
>
> 4.  Apply this logic to all encompassed languages, not just those belonging
> to a cherry-picked set of macrolanguages plus "sgn".  For extra credit,
> figure out how to make this work with changing ISO 639-3 groupings.
>
> This would allow both extlangistas and non-extlangistas to tag as needed to
> achieve the desired matching and fallback behavior, and would solve
> Addison's concern about being able to normalize without the Registry, since
> all valid tags of form XX(X)-YYY could be normalized to YYY blindly.
>  Determining validity in the first place still requires the Registry, but
> there is no way around that anyway.
>
> Part 4 removes arbitrariness, partly to help this document survive IETF LC,
> but also for its own sake.  I am sensitive to Addison's point that we should
> have as few duplicate tags as possible, but I still feel this is an adequate
> compromise, in which the extlangistas concede the formal "preferability" of
> the non-extlangy forms but reserve the right to use the easily normalized,
> deprecated forms that they prefer without being branded with a scarlet D.
>
> severable from the rest, not all that radical, and probably a good idea no
> matter what else we do.  Part 4 is also severable from the rest.
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 2 Jul 2008 09:45:37 -0700

It *almost* sounds reasonable to me. Some comments follow.

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, July 02, 2008 8:31 AM
To: Doug Ewell
Cc: LTRU Working Group
Subject: Re: [Ltru] A radical proposal (was: Re: extlang & deprecation)

Mark
On Tue, Jul 1, 2008 at 10:34 PM, Doug Ewell <[email-masked]> wrote:
What about this:
(Warning: some of these suggestions are radical, but meant to reach a workable accord)

1.  Provide an explicit definition for the word "deprecated" in BCP 47, something on the order of "There is another tag or subtag that means the same as this one, but is preferred, and validating processors will translate this subtag into the preferred equivalent.  But nobody will beat you up or make you register with the police if you use the non-preferred one instead."  Make sure other text in the draft does not contradict this.

<AP> -1 I don’t think we need to define ‘deprecated’, the current thread notwithstanding (more below). </AP>

2.  Define all of the Y forms (e.g. "cmn") in the Registry as Type: extlang, none as Type: language.  Bear with me here.

<AP> This introduces the problem of defining rules for when an ISO 639 code is registered as type extlang and when as type language. I think it would be better if the proposal were that all language subtags have type 'language'. Those with a prefix field MAY be used as "extended language subtags" with their prefix. The (future) decision to include a prefix would rest with ietf-languages. </AP>

3.  Change the prose (and ABNF if necessary) to specify that an "extlang" can be used either alone, as "cmn" (Y), or together with its Prefix, as "zh-cmn" (X-Y).  In other words, an "extlang" is a special type of "language" that has an optional Prefix.  Specify that the Y form is "preferred" and the X-Y form is "deprecated," using the clearly specified, watered-down definition of "deprecated" from item 1.

<AP> The ABNF would be tricky to do. As I say, it would be easier to make everything a language.

I think it a mistake to water down "deprecated". The problem we have with deprecation at present is the use of the Preferred-Value field in 'extlang' records requires deprecation. If we don't want to deprecate the extlang form, then we don't deprecate it. The question is whether a tag of form "X-Y" is in its canonical form or not. That is, is "zh-cmn-Hans-CN" canonical? Or is its canonical form "cmn-Hans-CN"? It isn't necessarily true that non-canonical forms are deprecated. For example, the tag "en-b-moo-a-foo" is not in canonical form, but isn't necessarily deprecated.</AP>

4.  Apply this logic to all encompassed languages, not just those belonging to a cherry-picked set of macrolanguages plus "sgn".  For extra credit, figure out how to make this work with changing ISO 639-3 groupings.

<AP> Sure: do as we've suggested and ban all such future registrations. Or we can just set the rules to track ISO 639-3, with the proviso that prefixes may only be added or broadened, never removed. </AP>

This would allow both extlangistas and non-extlangistas to tag as needed to achieve the desired matching and fallback behavior, and would solve Addison's concern about being able to normalize without the Registry, since all valid tags of form XX(X)-YYY could be normalized to YYY blindly.

<AP> Canonicalization is not the same thing as permission to normalize. Permission to normalize is much weaker. If we say that the 'Y' form is canonical, it will save us having to modify aspects of matching (we may still have to say something about matching). </AP>

Part 4 removes arbitrariness, partly to help this document survive IETF LC, but also for its own sake.  I am sensitive to Addison's point that we should have as few duplicate tags as possible, but I still feel this is an adequate compromise, in which the extlangistas concede the formal "preferability" of the non-extlangy forms but reserve the right to use the easily normalized, deprecated forms that they prefer without being branded with a scarlet D.

<AP> It doesn't remove the arbitrariness. It is just a different set of arbitrariness. Personally, I prefer the smaller set of languages we have now. But we could solve this by only initially registering a few and allowing ietf-languages to register others. Or just register them all. </AP>

---

From: Shawn Steele <[email-masked]>
Date: Wed, 2 Jul 2008 10:14:42 -0700

Re: Deprecated.

I happen to work for a large company that regularly gets "beat up" in the media for not using the preferred forms of standards.  Those doing the ranting usually don't care if there is a "good reason" or not for using the non-preferred form.  They also aren't going to bother with minor details like reading the definition of a well-known term like "deprecated".

Since its unclear which form will end up being preferred or not, I don't really know whether we'd end up using the preferred or form or not, but I don't think its appropriate to stick other people in that position whether or not the standard works for us.

- Shawn

---

From: John Cowan <[email-masked]>
Date: Wed, 2 Jul 2008 13:23:57 -0400

> non-preferred one instead."  Make sure other text in the draft does not 

I strongly support this idea, and I think it should appear early in the
document, for clarity.  At the same time we should get rid of the use of
"supersede(d)" in favor of "deprecate(d)" everywhere.

> extlang, none as Type: language.  Bear with me here.

Presumably this is meant to exclude the ones which are already in as
languages?  I don't think we want to allow sh-bos, for example.

> specified, watered-down definition of "deprecated" from item 1.

This breaks the invariant that subtags are deprecated iff they have a
"Deprecated:" field, but that's tolerable (-16 makes all the extlangs
deprecated with a Preferred-Value of themselves, interpreted as a full
tag rather than as a subtag).

More seriously, though, it breaks the invariant that the syntactic slot
"language" is filled only with "language" subtags.  Currently this
invariant applies to all four kinds of subtags, and I think it's an
important part of Getting It Right for people who haven't read all the
fine print: once you clear away the grandfathered rubbish, you can just
take the Cartesian product of the four lists of subtags as a rough guide
to validity.

I would rather keep -16's scheme than this, frankly.

> extra credit, figure out how to make this work with changing ISO 639-3 

This is where we go from Grotty Compromise to Seriously Nasty.  639-3 is
free to change a language from independent to encompassed or vice versa
at any time, and then the carefully crafted scheme goes down the toilet.

I continue to think that the best defense against cherry-picking
complaints is to say we are doing this for backward compatibility,
and we need backward compatibility only for "zh" and "sgn", period.
(In this scheme I think it might make sense to restore the table from
-14 that explains that the macrolanguage tag is preferred for Konkani,
Malay, Swahili, and Uzbek.)

> Please avoid personal attacks; they don't solve technical problems.

So here's my take:  +1 on #1, -1 on the other ideas.  Also, cherry-pick
zh and sgn only (-16 isn't specific, as this is an RFC 4645bis issue),
restore the -14 table of special-case macrolanguages.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If he has seen farther than others,
        it is because he is standing on a stack of dwarves.
                --Mike Champion, describing Tim Berners-Lee (adapted)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 2 Jul 2008 10:59:46 -0700

> languages?  I don't think we want to allow sh-bos, for example.

We would have to do this. Further, we'd need to make stability rules related to it and these would likely be somewhat complicated. You wouldn't want only some of the enclosed languages of a macrolanguage to be included, for example. This is, basically, a lot of the messy logic from draft-14 that we scrapped to get the current compromise position. Only more so.

> tag rather than as a subtag).

We have two semantics of "deprecated" at work here: we allow subtags to be formally deprecated with a  Preferred-Value (that's the current setup). But Doug is talking about deprecating specific tag forms (which the registry would have a hard time indicating). Hence all my blather about canonical in my last email.

> take the Cartesian product of the four lists of subtags as a rough

This is certainly a Good Thing about the current design. In fact, my current implementation relies on it rather heavily and I think is somewhat elegant because of it. All this extra mappy cruft from the registry makes for fairly ugly complications.

> at any time, and then the carefully crafted scheme goes down the

Hence: stability rules. The problem being that these are likely to be complicated. Something like:

- if a language is assigned and it has a macrolanguage and that macrolanguage's other encompassed languages are extlangs, then so is this one.
- if a language is changed to be encompassed and the macrolanguage's other encompassed languages are extlangs, then so is this one.
- if a language is changed from encompassed to individual and it is an extlang, it remains an extlang (old tags still valid), but the extlang form is deprecated (somehow?)
- if a language was at some point encompassed by one language and later by another language, file a complaint retag your data as "und".

> and we need backward compatibility only for "zh" and "sgn", period.

+1

> Malay, Swahili, and Uzbek.)

+1

Addison

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 2 Jul 2008 12:35:01 -0700

Hi -

As a technical contributor...

> From: "John Cowan" <[email-masked]>
> To: "Doug Ewell" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Wednesday, July 02, 2008 10:23 AM
...
> I would rather keep -16's scheme than this, frankly.

Though I think I see where the logic is going, I have the same trouble
with this part of the proposal.

> and we need backward compatibility only for "zh" and "sgn", period.

I'm sympathetic to this line of reasoning, but I find the analogy for things
like ar-arb too persuasive to fully agree to the limitation.

> Malay, Swahili, and Uzbek.)

I think this would be a step in the wrong direction.  The macrolanguage
tag by itself should always have the meaning "this is some kind of X,
but we're not able / willing to say precisely which kind."

Randy

---

From: John Cowan <[email-masked]>
Date: Wed, 2 Jul 2008 17:45:04 -0400

> like ar-arb too persuasive to fully agree to the limitation.

Fine, but where do we stop?  The only valid tag for Malay right now is
'ms', so that's what people use.  Almost all Malay written content is
Standard Malay, a language that didn't even have its own code element
until the last round of 639-3 changes, where it was split off from "Malay
(individual language)" which now means the varieties of non-standard Malay
varieties that are close enough to Standard Malay not to be classified as
separate Malayan languages.  (At the code element level, the older code
element for "Malay (individual language) was retired to avoid narrowing.
This is very much like Arabic and Chinese.

So as you see, we are cherry-picking already.

-- 
Unless it was by accident that I had            John Cowan
offended someone, I never apologized.           [email-masked]
        --Quentin Crisp                         http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Thu, 3 Jul 2008 15:43:25 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Cowan
> Sent: Wednesday, July 02, 2008 2:45 PM

> 'ms', so that's what people use...

There is a slight difference between the zh case and cases like ms and ar in that for zh we have known, explicit usage of that single ID for what are widely recognized to be distinct languages with separate business relevance. Currently, that's not the case (that I know of) for ar or ms. (Of course, that could change.)

Peter

---

From: Peter Constable <[email-masked]>
Date: Mon, 7 Jul 2008 07:38:28 -0700

Well, in the case of zh, we have no choice about "zh=yue" etc.: those tags are out there in the wild already. But no such extlang-form tags exist yet for ar, nor has that been used explicitly to mean distinct languages, making it pre-conditioned, so to speak, for extlang use: it is still an option to leave "ar" to legacy usage, and to tag the more granular Arabic distinctions "arb", "aao", etc. without the extlang form.

(A small correction wrt ms: MARC explicitly describes the corresponding alpha-3 ID "may" as being used for "Pelambang Malay" as well as for "Malay". It's not clear to me exactly what is meant by "Pelambang Malay" or how it relates to existing entries in ISO 639-3.)

Peter

> From: Martin Duerst [mailto:[email-masked]]
> Sent: Friday, July 04, 2008 12:29 AM
> To: Peter Constable; LTRU Working Group
> Peter - Just a clarifying question:
>
> Are you saying that zh has been used for both Mandarin and Cantonese
> (and others), whereas ar has only been used for Standard Arabic, and
> ms has only been used for Standard Malay, at least as far as business-
> relevant languages?
>
> In my eyes, that would make a stronger suggestion for keeping ms and
> ar (i.e. going the extlang way) than for keeping zh.
>
> Can you confirm or explain further?
>
> Regards,    Martin.
>
> At 07:43 08/07/04, Peter Constable wrote:
> >> > I'm sympathetic to this line of reasoning, but I find the analogy
> for
> >> things
> >There is a slight difference between the zh case and cases like ms and
> ar
> >in that for zh we have known, explicit usage of that single ID for
> what are
> >widely recognized to be distinct languages with separate business
> >relevance. Currently, that's not the case (that I know of) for ar or
> ms.
> >(Of course, that could change.)
> >
> >
> >Peter
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
> #-#-#  http://www.sw.it.aoyama.ac.jp
> mailto:[email-masked]
>