Subject: [Ltru] Review of 4646bis-10, macrolanguages in section 4.1

From: John Cowan <[email-masked]>
Date: Fri, 7 Dec 2007 16:55:57 -0500

Because of the thicket of rewordings in this part, I'm just presenting
my suggested revised text here.  It's very important to make sure
that we don't talk about "dialects" or "sub-languages" here.  Also,
I've used "Macrolanguage" for the header only, but "macrolanguage"
for the languages.

The affected text begins "Languages with a Macrolanguage field" and ends
"did not specify zh-Hans-CN in their request.)".

        Some of the languages in the registry are labeled
        "macrolanguages" by ISO 639-3, which defines the term as
        "clusters of closely-related language varieties that [...] can
        be considered distinct individual languages, yet in certain
        usage contexts a single language identity for all is needed".
        These correspond to codes registered in ISO 639-2 as single
        languages that were found to correspond to more than one language
        in ISO 639-3.  The languages encompassed by a macrolanguage
        contain a Macrolanguage header in the registry; the macrolanguages
        themselves are not specially marked.

        It is always permitted, and sometimes useful, to tag an
        encompassed language using the subtag for its macrolanguage.
        However, the Macrolanguage field doesn't define what the
        relationship is between the encompassed language and its
        macrolanguage, nor does it define how languages encompassed by the
        same macrolanguage are related to each other.  In some cases, In
        some cases, one of the encompassed languages serves as a standard
        form for the entire macrolanguage and is frequently identified
        with it; in other cases there is no dominant language, and the
        macrolanguage simply serves as a cover term for the entire group.

        Applications MAY use macrolanguage information to improve matching
        or language negotiation.  For example, the information that 'sr'
        (Serbian) and 'hr' (Croatian) share a macrolanguage expresses
        a closer relation between those languages than between, say,
        'sr' (Serbian) and 'ma' (Macedonian).  It is valid to use the
        subtag of the encompassed language or of the macrolanguage to
        form language tags.  However, many matching applications will
        not be aware of the relationship between the languages.  Care in
        selecting which subtags are used is crucial to interoperability.

        In general, use the most specific tag.  However, where the
        macrolanguage tag has been historically used to denote a dominant
        encompassed language, it SHOULD be used in place of the subtag
        specific to that encompassed language unless it is necessary
        to clearly distinguish the macrolanguage as a whole from the
        dominant language variety.

	In particular, the Chinese family of languages call for special
	consideration.	Because the written form is very similar for most
	languages having 'zh' as a macrolanguage (and because historically
	subtags for the various encompassed languages were not available),
	languages such as 'yue' (Cantonese) have historically used
	either 'zh' or a tag (now grandfathered) beginning with 'zh'.
	This means that macrolanguage information can be usefully
	applied when searching for content or when providing fallbacks
	in language negotiation.  For example, the information that 'yue'
	has a macrolangauge of 'zh' could be used in the Lookup algorithm
	to fallback from a request for "yue-Hans-CN" to "zh-Hans-CN"
	without losing the script and region information (even though
	the user did not specify "zh-Hans-CN" in their request).

--
John Cowan              [email-masked]          http://www.ccil.org/~cowan
Any day you get all five woodpeckers is a good day.  --Elliotte Rusty Harold

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 7 Dec 2007 14:54:19 -0800

LGTM

On Dec 7, 2007 1:55 PM, John Cowan <[email-masked]> wrote:

> John Cowan              [email-masked]          http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>

-- 
Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 07 Dec 2007 15:18:13 -0800

Firstly, let me express my Deep Annoyance at getting emails with 
proposed text with line breaks and big wads of whitespace in front of 
each line. Yes, a regular expression will tame the beast, but I'd prefer 
to cut-and-paste a bit more :-)

I incorporated the below with only very minor tidying.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: John Cowan <[email-masked]>
Date: Sat, 8 Dec 2007 00:26:11 -0500

> to cut-and-paste a bit more :-)

I format my emails as God intended, for an 80-column fixed-width text
screen (which I still use), and I indent quotations.  If you don't like
it, "fmt -20000" is your friend.

-- 
What is the sound of Perl?  Is it not the       John Cowan
sound of a [Ww]all that people have stopped     [email-masked]
banging their head against?  --Larry            http://www.ccil.org/~cowan

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 8 Dec 2007 16:53:16 +0100

> encompassed language using the subtag for its macrolanguage.

s/always permitted, and//  It is sometimes useful, enough said,
the rest of your proposed text is IMO unnecessary, "let the
market decide" (not exactly my favourite option, but if you 
want to go into details you need examples)

> Applications MAY use macrolanguage information to improve 
> matching or language negotiation.

+1 (no Michael here, I can say +1 ;-)

> For example, the information that 'sr' (Serbian) and 'hr' 
> (Croatian) share a macrolanguage expresses a closer relation
> between those languages than between, say, 'sr' (Serbian)
> and 'ma' (Macedonian).

Dubious example, what if ISO 639-3 adds 'ma' to the 'sh' set ?

> It is valid to use the subtag of the encompassed language or
> of the macrolanguage to form language tags.

At a price, 'sh' is deprecated, and 'sh' has no Suppress-Script.
You picked certainly an interesting example for this business.

> macrolanguage tag has been historically used to denote a
> dominant encompassed language, it SHOULD be used in place

-1  It is messy, don't make it worse, "let the market decide".

 Frank

---

From: John Cowan <[email-masked]>
Date: Sun, 9 Dec 2007 00:51:12 -0500

> Dubious example, what if ISO 639-3 adds 'ma' to the 'sh' set ?

Not gonna happen.  Macedonian is bogus Bulgarian, not split-off Serbo-Croat.

> You picked certainly an interesting example for this business.

Yes, 'sh' isn't a good example.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
In the sciences, we are now uniquely privileged to sit side by side
with the giants on whose shoulders we stand.
        --Gerald Holton

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 9 Dec 2007 10:39:29 +0100

John Cowan wrote:

>> what if ISO 639-3 adds 'ma' to the 'sh' set ?
 
> not split-off Serbo-Croat.

Ugh, sorry, I confused "ma" with Montenegrin.

 Frank

---

From: Peter Constable <[email-masked]>
Date: Sun, 9 Dec 2007 04:58:29 -0800

> From: Frank Ellermann [mailto:[email-masked]]

> Ugh, sorry, I confused "ma" with Montenegrin.

Coding Montenegrin as a distinct individual language would be a bit messy. Right now, sh/hbs encompasses three individual-language entries, Bosnian (bs/bos), Croatian (hr/hrv/scr) and Serbian (sr/srp/scc). (Note: 639-2 coded separate T and B IDs for Croatian and Serbian.) If Montenegrin were coded, we'd be looking at a split of the existing Serbian. Therefore, we'd have to code *two* new languages:

'xxx' Serbian (Serbia, 2007)
'yyy' Montenegrin

(Some reference name for the new Serbian would have to be coined to differentiate it from the old Serbian, and/or maybe the name for the old Serbian could be changed. I haven't a clue what names to use.)

Now there are is the issue of how these relate to the existing entries. There are two options; I'll describe these in terms of 639-3 IDs, "MM" = macrolanguage mapping:

A)
        ID      Scope   MM      Status
        hbs     M       -
        bos     I       hbs
        hrv     I       hbs
        srp     I       hbs     deprecated (use xxx or yyy)
        xxx     I       hbs
        yyy     I       hbs

B)
        ID      Scope   MM      Status
        hbs     M       -
        bos     I       hbs
        hrv     I       hbs
        srp     M       hbs
        xxx     I       srp
        yyy     I       srp

With option B, we would also face the problem of what best practice should be for tagging new "Serbian" content: use srp or xxx? (Software platforms would also face the same problem in relation to locales.) Option A assumes that there would be no new use of srp for tagging content. (Software platforms would also need new xxx locales.)

I think it's fairly obvious that either of these options would present several problems in implementation.

Btw, the linguistic reality is that a distinct language did not suddenly pop into existence. Hence, if the split were coded, it's quite likely that lots of content / language resources would be created once for both. With either option A or B, we're likely to end up with lots of inconsistency in tagging.

Also btw, there has been no formal request submitted to the JAC to code Montenegrin. The question of adding Montenegrin was raised with the JAC, but given the many issues that would create, the JAC has chosen to take a wait-and-see stance.

Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 21 Dec 2007 16:25:53 -0500

Here's some more wording changes for 4.1, embedded in the change I
suggested before.

>         macrolanguage simply serves as a cover term for the entire group.

Remove one instance of "In some cases,", obviously.  My bad.

>         dominant language variety.

Add:

        Macrolanguages and encompassed languages affected by this issue
        include:

        Arabic 'ar'                     Standard Arabic 'arb'
        Konkani (macrolanguage) 'kok'   Konkani (individual language) 'knn'
        Malay (macrolanguage) 'ms'      Malay (individual language) 'mly'
        Swahili (macrolanguage) 'sw'    Swahili (individual language) 'swh'
        Uzbek 'uz'                      Northern Uzbek 'uzn'
        Chinese 'zh'                    Mandarin Chinese 'cmn'

-- 
There are three kinds of people in the world:   John Cowan
those who can count,                            [email-masked]
and those who can't.

---

From: Addison Phillips <[email-masked]>
Date: Fri, 21 Dec 2007 13:54:07 -0800

> Remove one instance of "In some cases,", obviously.  My bad.

Already done in the editor's copy.

>         Macrolanguages and encompassed languages affected by this issue

I made it say "The pairs of macro and encompassed languages affected by 
this issue when this document was published were:"

Saying it this way inoculates us against future Macrolanguage 
assignments, even though none may be forthcoming that fall into this 
category.

>         Chinese 'zh'                    Mandarin Chinese 'cmn'

Added this table.

thanks,

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 21 Dec 2007 14:46:20 -0800

LGTM.

==========

BTW, In John's signature there was:

There are three kinds of people in the world:
those who can count, <[email-masked]>
and those who can't.

My other favorites are:

   - There are 10 kinds of people: those who understand binary and those
   who don't.
   - There are 2 kinds of people: those who divide the world into 2 kinds
   of people, and those who don't.

Mark

---

From: John Cowan <[email-masked]>
Date: Fri, 21 Dec 2007 17:52:45 -0500

> those who can count, <[email-masked]>

That's just because you are viewing my .sig in HTML, with spaces collapsed
and with a variable-width font, instead of monowidth as I (and God) intended.

>    of people, and those who don't.

This I think is the original version of the snowclone (see
http://snowclones.org ).

-- 
John Cowan      <[email-masked]>       http://www.ccil.org/~cowan
                Charles li reis, nostre emperesdre magnes,
                Set anz totz pleinz ad ested in Espagnes.

---

From: Peter Constable <[email-masked]>
Date: Sun, 23 Dec 2007 14:49:26 -0800

> From: John Cowan [mailto:[email-masked]]

>         Chinese 'zh'                    Mandarin Chinese 'cmn'

Thanks for taking note of this. I trust this reflects that you reviewed the analysis I sent on 11/29 and agree that this is the right set of cases to mention in this regard.

Peter

---

From: John Cowan <[email-masked]>
Date: Sun, 23 Dec 2007 18:48:02 -0500

Peter Constable scripsit:

> Thanks for taking note of this. I trust this reflects that you reviewed
> the analysis I sent on 11/29 and agree that this is the right set of
> cases to mention in this regard.

Indeed.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
I come from under the hill, and under the hills and over the hills my paths
led. And through the air. I am he that walks unseen.  I am the clue-finder,
the web-cutter, the stinging fly. I was chosen for the lucky number.  --Bilbo