Subject: [Ltru] Macrolanguage and extlang

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Jul 2007 18:06:03 -0700

Addision and I have discussed the issue of extlang and Macrolanguages and
are proposing the following text replacing the use of extlang.

*[A new section called Macrolanguages: ]*

The Macrolanguage field contains a primary language subtag that *encompasses
* this subtag. That is, this language is a dialect or sub-language of the
Macrolanguage, and is called an *encompassed* subtag. The Macrolanguage
value is defined by ISO 639-3. The field can be useful to applications or
users when selecting language tags or as additional metadata useful in
matching. The Macrolanguage field can only occur in records of type
'language'. Only values assigned by ISO 639-3 will be considered for
inclusion. Macrolanguage fields MAY be added via the normal registration
process whenever ISO 639-3 defines new values. Macrolanguages are
informational, and MAY be removed or changed if ISO 639-3 changes the
values.
For example, the language subtags 'nb' (Norwegian Bokmal) and 'nn'
(Norwegian Nynorsk) has a Macrolanguage entry of 'no' (Norwegian). For more
information see [Choice].

*[A new section in tag choice (section 4.1), referenced from the above] *

Languages with a Macrolanguage field in the registry sometimes can be
usefully referenced using their Macrolanguage. However, the Macrolanguage
field doesn't define what the relationship is between the language subtag
whose record it appears in and its encompassed language or languages. Nor
does it define how the encompassed languages are related to one-another. In
some cases, the Macrolanguage has a standard form as well as a variety of
less-common dialects. For example, the Macrolanguage 'ar' (Arabic) and the
subtag 'arb' (Standard Arabic) generally describe the same language, with
other subtags describing less-common local variations. In other cases there
is no particular standard form and the encompassed subtags describe specific
variations within the parent language.

Applications MAY use Macrolanguage information to improve matching or
language negotiation. For example, the information that 'sr' and 'hr' share
a Macrolanguage expresses a closer relation between those languages than
between, say, "sr" and "ma" (Macedonian). It is valid to use either the
encompassed language or its Macrolanguage to form language tags. However,
many matching applications will not be aware of the relationship between the
languages. Care in selecting which subtags are used is crucial to
interoperability. In general, use the most specific tag. However, where the
standard written form of an encompassed language is captured by the
Macrolanguage, the Macrolanguage should still be used for written material.

In particular, chinese language(s) and dialects call for special
consideration. Because the written form is very similar for most languages
having 'zh' as a Macrolanguage (and because historically subtags for the
various sub-languages and dialects were not available), languages such as
'yue' (Cantonese) have usually used tags beginning with the subtag 'zh'.
This past practice of tagging means that Macrolanguage information is
encouraged when searching for content or when providing fallbacks in
language negotiation. For example, the information that 'yue' has a
macrolangauge of 'zh' could be used in the Lookup algorithm to fallback from
a request for "yue-Hans-CN" to "zh-Hans-CN" without losing the script and
region information (even though the user did not specify "zh-Hans-CN" in
their language priority list).

However, the Macrolanguage is only one of many additional pieces of
information  that can be used in matching languages. There are many other
circumstances where the "best fit" information is not contained in the
language registry. For example, the languages "ro" (Romanian) and "mo"
(Moldavian) are very closely related, and so for searching it is often best
to treat them as being the same. In other cases, the best fallback for a
requested language may be a completely unrelated language, but one that a
majority of speakers in the requested language may understand. For example,
in a given application the best fallback for "be" (Breton), may be "fr"
(French) -- rather than the more closely related "cy" (Welsh) -- because
Breton readers are far more likely to be able to read French than Welsh.

For more information on matching, see [RFC 4647].

 *[In the section talking about updates]*

The Macrolanguage field is added whenever a language has a corresponding
Macrolanguage in [ISO 639-3]. For example, 'sr' (Serbian) will have the
Macrolanguage value 'sh' (Serbo-Croatian).

*[Other changes]*

[Search for instances of "Suppress-Script" (just as a place to find where
field descriptions are) and make an addition of "Macrolanguage" if
appropriate, eg in the "LANGUAGE SUBTAG REGISTRATION FORM"]

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Don Osborn" <[email-masked]>
Date: Sat, 14 Jul 2007 08:26:21 -0400

Mark, Thanks for this update. In reading this over (and trying to see between the lines) with an eye to implications for many African languages and macrolanguages, the mention of Romanian/Moldavian seems particularly relevant, as an example of cases  "where the 'best fit' information is not contained in the language registry." This is an area to which I hope that experts on African languages who have a familiarity with tagging isues can be organized to propose amendments to the system (i.e., ISO 639, which I realize is not the purview of this list, and the current RFCs).

There are also cases where macrolanguages are defined, but a still somewhat fluid situation wrt standardization makes defining their use problematic (I've posted previously on some of the issues as I see them, both on this list and on ietf-languages - such as cases where the language tag may be less appropriate than the macrolanguage tag).  Here too there seems to be a need for input by experts on African languages in discussions of tagging as well as in language planning.

In the meantime, I hope that the new wording can accommodate all such situations, especially for languages with less resources and emerging standards.

The fallback language issue (mentioned in the example re Breton) raises another question: can there be more than one fallback language? In the case of many of the crossborder languages in Africa (such as Hausa, Swahili, Wolof, Fula, Tsonga, Oshiwambo, etc.) this would be helpful.

Don

From: Mark Davis [mailto:[email-masked]] 
Sent: Friday, July 13, 2007 9:06 PM
To: LTRU Working Group
Subject: [Ltru] Macrolanguage and extlang

Addision and I have discussed the issue of extlang and Macrolanguages and are proposing the following text replacing the use of extlang.

[A new section called Macrolanguages: ]

The Macrolanguage field contains a primary language subtag that encompasses this subtag. That is, this language is a dialect or sub-language of the Macrolanguage, and is called an encompassed subtag. The Macrolanguage value is defined by ISO 639-3. The field can be useful to applications or users when selecting language tags or as additional metadata useful in matching. The Macrolanguage field can only occur in records of type 'language'. Only values assigned by ISO 639-3 will be considered for inclusion. Macrolanguage fields MAY be added via the normal registration process whenever ISO 639-3 defines new values. Macrolanguages are informational, and MAY be removed or changed if ISO 639-3 changes the values.

For example, the language subtags 'nb' (Norwegian Bokmal) and 'nn' (Norwegian Nynorsk) has a Macrolanguage entry of 'no' (Norwegian). For more information see [Choice]. 

[A new section in tag choice (section 4.1), referenced from the above] 

Languages with a Macrolanguage field in the registry sometimes can be usefully referenced using their Macrolanguage. However, the Macrolanguage field doesn't define what the relationship is between the language subtag whose record it appears in and its encompassed language or languages. Nor does it define how the encompassed languages are related to one-another. In some cases, the Macrolanguage has a standard form as well as a variety of less-common dialects. For example, the Macrolanguage 'ar' (Arabic) and the subtag 'arb' (Standard Arabic) generally describe the same language, with other subtags describing less-common local variations. In other cases there is no particular standard form and the encompassed subtags describe specific variations within the parent language. 

Applications MAY use Macrolanguage information to improve matching or language negotiation. For example, the information that 'sr' and 'hr' share a Macrolanguage expresses a closer relation between those languages than between, say, "sr" and "ma" (Macedonian). It is valid to use either the encompassed language or its Macrolanguage to form language tags. However, many matching applications will not be aware of the relationship between the languages. Care in selecting which subtags are used is crucial to interoperability. In general, use the most specific tag. However, where the standard written form of an encompassed language is captured by the Macrolanguage, the Macrolanguage should still be used for written material.

In particular, chinese language(s) and dialects call for special consideration. Because the written form is very similar for most languages having 'zh' as a Macrolanguage (and because historically subtags for the various sub-languages and dialects were not available), languages such as 'yue' (Cantonese) have usually used tags beginning with the subtag 'zh'. This past practice of tagging means that Macrolanguage information is encouraged when searching for content or when providing fallbacks in language negotiation. For example, the information that 'yue' has a macrolangauge of 'zh' could be used in the Lookup algorithm to fallback from a request for "yue-Hans-CN" to "zh-Hans-CN" without losing the script and region information (even though the user did not specify "zh-Hans-CN" in their language priority list). 

However, the Macrolanguage is only one of many additional pieces of information  that can be used in matching languages. There are many other circumstances where the "best fit" information is not contained in the language registry. For example, the languages "ro" (Romanian) and "mo" (Moldavian) are very closely related, and so for searching it is often best to treat them as being the same. In other cases, the best fallback for a requested language may be a completely unrelated language, but one that a majority of speakers in the requested language may understand. For example, in a given application the best fallback for "be" (Breton), may be "fr" (French) -- rather than the more closely related "cy" (Welsh) -- because Breton readers are far more likely to be able to read French than Welsh.  

[In the section talking about updates]

The Macrolanguage field is added whenever a language has a corresponding Macrolanguage in [ISO 639-3]. For example, 'sr' (Serbian) will have the Macrolanguage value 'sh' (Serbo-Croatian).

[Other changes]

[Search for instances of "Suppress-Script" (just as a place to find where field descriptions are) and make an addition of "Macrolanguage" if appropriate, eg in the "LANGUAGE SUBTAG REGISTRATION FORM"]

-- 
Mark

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 14 Jul 2007 09:32:22 -0700

Thanks.

I think one of the things that we realized when looking at how this would
work in practice is that we are better off if we treat macrolanguage as a
piece of very useful information for matching, but one that can be enhanced
(that is, changed) over time as more information becomes available.
Hard-coding it into extlang doesn't serve that purpose, and causes other
problems, notably that the other fields are lost in fallback: if we had
zh-yue-Hant, then by the time we get to zh in fallback, we've lost the Hant.

So that is the origin of the text we are proposing.

There are a number of edge cases such as deprecated codes, closely related
languages, or practical fallbacks (eg if someone speaks X they are likely to
speak Y, even if X and Y are not linguistically related) that are simply
unsuited to hard-coding in the tag. If I hit a code like "iw-PL", I want to
match that with "he-PL", not depend on some kind of fallback between "iw"
and "he"; otherwise it loses information (the PL). We do provide that kind
of information in the deprecated field, and with the macrolanguage field we
would provide more (for example, it would make clear the relation between
no, nb, and nn and how that could be used in matching). Other useful
information is the scripts used with a language in practice; suppress script
supplies just a little information, but doesn't tell me that Uzbek is
customarily written with Arabic, Latin, or Cyrillic, but not with (say)
Tagalog.

The more information there is available, whether it be in the language
subtag registry or somewhere else, the better a job people can do in dealing
with some of the edge cases that turn up in matching.

Mark

On 7/14/07, Don Osborn <[email-masked]> wrote:
>
>  Mark, Thanks for this update. In reading this over (and trying to see
> between the lines) with an eye to implications for many African languages
> and macrolanguages, the mention of Romanian/Moldavian seems particularly
> relevant, as an example of cases  "where the 'best fit' information is not
> contained in the language registry." This is an area to which I hope that
> experts on African languages who have a familiarity with tagging isues can
> be organized to propose amendments to the system (i.e., ISO 639, which I
> realize is not the purview of this list, and the current RFCs).
>
>
>
> There are also cases where macrolanguages are defined, but a still
> somewhat fluid situation wrt standardization makes defining their use
> problematic (I've posted previously on some of the issues as I see them,
> both on this list and on ietf-languages - such as cases where the language
> tag may be less appropriate than the macrolanguage tag).  Here too there
> seems to be a need for input by experts on African languages in discussions
> of tagging as well as in language planning.
>
>
>
> In the meantime, I hope that the new wording can accommodate all such
> situations, especially for languages with less resources and emerging
> standards.
>
>
>
> The fallback language issue (mentioned in the example re Breton) raises
> another question: can there be more than one fallback language? In the case
> of many of the crossborder languages in Africa (such as Hausa, Swahili,
> Wolof, Fula, Tsonga, Oshiwambo, etc.) this would be helpful.
>
>
>
> Don
>
>
>
>
>
>
>
> *From:* Mark Davis [mailto:[email-masked]]
> *Sent:* Friday, July 13, 2007 9:06 PM
> *To:* LTRU Working Group
> *Subject:* [Ltru] Macrolanguage and extlang
>
>
>
> The Macrolanguage field contains a primary language subtag that *
> encompasses* this subtag. That is, this language is a dialect or
> sub-language of the Macrolanguage, and is called an *encompassed* subtag.
> The Macrolanguage value is defined by ISO 639-3. The field can be useful to
> applications or users when selecting language tags or as additional metadata
> useful in matching. The Macrolanguage field can only occur in records of
> type 'language'. Only values assigned by ISO 639-3 will be considered for
> a request for "yue-Hans-CN" to "zh-Hans-CN" *without losing the script and
> region information* (even though the user did not specify "zh-Hans-CN" in

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 16 Jul 2007 11:20:42 +0200

On Fri, Jul 13, 2007 at 06:06:03PM -0700,
 Mark Davis <[email-masked]> wrote 
 a message of 172 lines which said:

> For example, in a given application the best fallback for "be"
> (Breton), may be "fr" (French) -- rather than the more closely
> related "cy" (Welsh) -- because Breton readers are far more likely
> to be able to read French than Welsh.

Good example but Breton is "br"...

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 16 Jul 2007 11:25:16 +0200

On Sat, Jul 14, 2007 at 08:26:21AM -0400,
 Don Osborn <[email-masked]> wrote 
 a message of 524 lines which said:

> The fallback language issue (mentioned in the example re Breton)
> raises another question: can there be more than one fallback
> language?  In the case of many of the crossborder languages in
> Africa (such as Hausa, Swahili, Wolof, Fula, Tsonga, Oshiwambo,
> etc.) this would be helpful.

Breton was just an example and I do not think that anyone suggested to
add such information in the registry. It would require a lot of work,
anyway. Case in point: you're in Helsinski (Finland) and you want to
ask someone something. You do not speak Finnish. What do you try?

* Swedish (administrative best fit, since it is the second official
language in Finland),

* Hungarian (linguistic best fit, since it is the closest language),

* English (practical best fit, since it is probably the best
fallback, in practice).

Trying to make a registry of such "best fits" is a work of a size
comparable to ISO 639.

---

From: John Cowan <[email-masked]>
Date: Mon, 16 Jul 2007 13:40:28 -0400

> * Hungarian (linguistic best fit, since it is the closest language),

In fact the closest relatives of Finnish are Estonian and various
minority languages of Russia, Latvia, Norway, and Sweden, comprising
the Finnic branch of the Uralic language family.  Hungarian belongs to
the Ugric branch, which is maximally distant from the Finnic branch.
So this is like comparing English to Hindi and skipping over Scots,
Frisian, and Dutch.

See http://www.ethnologue.com/show_family.asp?subid=90210 .

-- 
Overhead, without any fuss, the stars were going out.
        --Arthur C. Clarke, "The Nine Billion Names of God"
                John Cowan <[email-masked]>

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 17 Jul 2007 20:00:52 +0200

> See http://www.ethnologue.com/show_family.asp?subid=90210 .

I would give more credibility to http://en.wikipedia.org/wiki/Finno-ugric.

Sami languages are closely related to Finnish, which is not
reflected in the Ethnologue page but is reflected in the
Wikipedia page), Hungarian is quite distant from the Finnic
languages; English/Hindi is a good comparison. See the second
paragraph of that Wikipedia page.

		/kent k

---

From: John Cowan <[email-masked]>
Date: Tue, 17 Jul 2007 17:04:13 -0400

> I would give more credibility to http://en.wikipedia.org/wiki/Finno-ugric.

The only point of disagreement is how the various branches are classified
into superbranches.  The Wikipedia page gives the traditional view,
as is stated there; the Ethnologue takes the more modern viewpoint,
which abandons any attempt to make superbranches and lists each branch
as directly from Uralic.  The truth is doubtless more complex than either.

-- 
John Cowan      [email-masked]        http://www.ccil.org/~cowan
        Is it not written, "That which is written, is written"?