Subject: [Ltru] 4.1: 639-2 special cases

From: Peter Constable <[email-masked]>
Date: Tue, 8 Jul 2008 08:56:53 -0700

In Section 4.1, numbered item 4 discusses entries in 639-2 that require special care. The text then goes on to discuss the various cases individually in numbered paragraphs. A few comments:

1. Wrt formatting, the second-level numbering uses the same format as the top level numbering. It’s not clear to me why the second-level items need to be numbered, but if they are then formatting should be consistent with formatting practice elsewhere in the doc. E.g. compare 3.4 which uses numbers, then letters.

2. The first itemized paragraph discusses collections:

----------------
Use specific language subtags or subtag sequences in preference to subtags for language collections. A "language collection" is a subtag derived from one of the [ISO639‑5] codes that represents multiple related languages. These codes are included as primary language subtags in the registry. For example...
----------------

I find the wording creates a feel of discontinuity from the intro paragraph as it's not obvious how this relates to 639-2, particularly given the reference to 639-5.

Suggested revision:
----------------
ISO 639-2 includes codes for language collections, as does [ISO 639-5]. A "language collection" code represents multiple related languages. These codes are included as primary language subtags in the registry. However, specific language subtags or subtag sequences SHOULD be used in preference to subtags for language collections. For example...
----------------

3. The second itemized paragraph has the following:

----------------
The 'mul' (Multiple) primary language subtag identifies content in multiple languages. This subtag SHOULD NOT be used when a list of languages (such as Content-Language)...
----------------

Please give a context and reference when introducing external concepts such as Content-Language. In this case, you might say "such as MIME Content-Language headers [RFC3282]...", "such as [Content-Language] headers", or something similar.

4. Finally, the last itemized paragraph in this section discusses "i-default", which has not connection to ISO 639-2. This should not be a sub-item under item 4 on ISO 639-2, but should be a top-level item, numbered 5.

Peter
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 10:23:22 -0700

(editor hat)

~Addison
>
> In Section 4.1, numbered item 4 discusses entries in 639-2 that
> require special care. The text then goes on to discuss the various
> cases individually in numbered paragraphs. A few comments:
>
> 1. Wrt formatting, the second-level numbering uses the same format
> as the top level numbering. It’s not clear to me why the second-
> level items need to be numbered, but if they are then formatting
> should be consistent with formatting practice elsewhere in the doc.
> E.g. compare 3.4 which uses numbers, then letters.

This is an idiosyncrasy of the XML: I didn't specify a format on the list and the default is "numbers". I changed it to "symbols" (bullets, essentially) to be consistent with the other items in Section 4.1.

> Use specific language subtags or subtag sequences in preference to
> subtags for language collections. A "language collection" is a
> subtag derived from one of the [ISO639‑5] codes that represents
> multiple related languages. These codes are included as primary
> language subtags in the registry. For example...
> ----------------
>
> I find the wording creates a feel of discontinuity from the intro
> paragraph as it's not obvious how this relates to 639-2,
> particularly given the reference to 639-5.
>
> Suggested revision:
> ----------------
> ISO 639-2 includes codes for language collections, as does [ISO
> 639-5]. A "language collection" code represents multiple related
> languages. These codes are included as primary language subtags in
> the registry. However, specific language subtags or subtag
> sequences SHOULD be used in preference to subtags for language
> collections. For example...
> ----------------

I think the first sentence is somewhat artificial (as is the original para): you can't tell the source from the registry anyway.

Your suggested revision doesn't follow the pattern for other items here, which all start with specific advice. Thus I modified the para to read (note that I quote the whole thing):

<t>Use a specific language subtags or sequences of subtags in preference to subtags for language collections. A "language collection" is a subtag that represents multiple related languages. These codes, which are the subset of <xref target="ISO639-2"></xref> that is also defined in <xref target="ISO639-5"></xref>, are included as primary language subtags in the registry. The 'Description' field of a collection typically includes the word "languages" to indicate that it represents more than one language. For example, these subtags include "Chamic languages" ('cmc'), "Algonquin languages" ('alg'), and "Germanic languages" ('gem').  Each collection is also represented by subtags for the individual languages. In the case of 'cmc', the registry also contains the values for each of the approximately ten individual languages represented by this collective code. The subtag 'gem' helps illustrate this further: since it is interpreted inclusively, content tagged with "en" (English), "de" (German), or "gsw" (Swiss German, Alemannic) could also (but SHOULD NOT) be tagged with "gem" (Germanic languages). Obviously, the languages in a collection are frequently not mutually intelligible, as this example demonstrates. Subtags derived from collection codes MAY be used when more specific language information is not available. Note that matching implementations generally do not understand the relationship between the collection and its encompassed languages, and so users ought not assume a subtag based on a language collection is a useful means for selecting content in its encompassed languages.</t>

> The 'mul' (Multiple) primary language subtag identifies content in
> multiple languages. This subtag SHOULD NOT be used when a list of
> languages (such as Content-Language)...
> ----------------
>
> Please give a context and reference when introducing external
> concepts such as Content-Language. In this case, you might say
> "such as MIME Content-Language headers [RFC3282]...", "such as
> [Content-Language] headers", or something similar.

DONE. New text reads:

<t>The 'mul' (Multiple) primary language subtag identifies content in multiple languages. This subtag SHOULD NOT be used when a list of languages or individual tags for each content element can be used instead. For example, the 'Content-Language' header (<xref target="RFC3282"></xref>) allows a list of languages to be used, not just a single language tag.</t>

>
>
>
> 4. Finally, the last itemized paragraph in this section discusses
> "i-default", which has not connection to ISO 639-2. This should not
> be a sub-item under item 4 on ISO 639-2, but should be a top-level
> item, numbered 5.

DONE.

---

From: Peter Constable <[email-masked]>
Date: Wed, 9 Jul 2008 00:19:28 -0700

> From: Phillips, Addison [mailto:[email-masked]]

> > 2. The first itemized paragraph discusses collections...

> I think the first sentence is somewhat artificial (as is the original
> para): you can't tell the source from the registry anyway.
>
> Your suggested revision doesn't follow the pattern for other items here,
> which all start with specific advice. Thus I modified the para...

You had some follow-on comments, and in the end there's just a reference to ISO 639-5, which still leaves a discontinuity from the intro paragraph and its reference to 639-2. One more suggested revision:

----------------
Use a specific language subtags or sequences of subtags in preference to subtags for language collections. A "language collection" is a subtag that represents multiple related languages. These codes comprise the set of [ISO639‑5] codes, and many are also included in ISO 639-2; all are included as primary language subtags in the registry. The 'Scope' field for a collection subtag has the value 'collection'. ...
----------------

I'm satisfied with the resolution of other items I raised in this thread.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 10:40:05 -0700

(editor hat)

DONE, but with Yet Another Set of Edits.

<t>Use a specific language subtags or sequences of subtags in preference to subtags for language collections. A "language collection" is a subtag that represents multiple related languages. These subtags, which are comprised of the set of <xref target="ISO639-5"></xref> codes, are included as primary language subtags in the registry. (Some of the ISO 639-5 codes are also defined as collections in ISO 639-2.) Language collections in the registry have a 'Scope' field with a value of 'collection'. A subtag representing a language collection MAY be used when more specific language information is not available, but most tag processes and users do not know there is a relationship between the collection and its encompassed languages. In addition, in many cases the encompassed languages are not mutually intelligible or closely related enough for the common identifier to be useful in a language tagging context. Since the subtags are different, the tags are treated as being different languages: a request for the collection will only produce items tagged with the collection code, not with the individual languages it contains. For example, for the collection 'cmc' (Chamic languages), the registry contains the values for each of its approximately ten individual languages. Collections are interpreted inclusively, so the subtag 'gem' (Germanic langauges) could (but SHOULD NOT) be used with content that would be better tagged with "en" (English), "de" (German), or "gsw" (Swiss German, Alemannic). While 'gem' collects all of these (and other) languages under a single identifier, no reasonable user or process would prefer the collection to the more accurate individual language subtag.</t>

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: Peter Constable [mailto:[email-masked]]
> Sent: Wednesday, July 09, 2008 12:19 AM
> To: Phillips, Addison; LTRU Working Group
> Subject: RE: 4.1: 639-2 special cases
>
> You had some follow-on comments, and in the end there's just a
> reference to ISO 639-5, which still leaves a discontinuity from the
> intro paragraph and its reference to 639-2. One more suggested
> revision:
>
> ----------------
> Use a specific language subtags or sequences of subtags in
> preference to subtags for language collections. A "language
> collection" is a subtag that represents multiple related languages.
> These codes comprise the set of [ISO639‑5] codes, and many are also
> included in ISO 639-2; all are included as primary language subtags
> in the registry. The 'Scope' field for a collection subtag has the
> value 'collection'. ...
> ----------------
>
>

---

From: "Broome, Karen" <[email-masked]>
Date: Fri, 11 Jul 2008 10:47:47 -0700

I would delete the last sentence. "gem" is preferable to "und" -- so if one knows that one has a Germanic language, but does not know which one, "gem" is a reasonable tag to use. It would aid future processes to more correctly determine the variant to collect "Germanic" variants under this tag in an asset management system.

Regards,

Karen Broome

>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Phillips, Addison
>Sent: Friday, July 11, 2008 10:40 AM
>To: Peter Constable; LTRU Working Group
>Subject: Re: [Ltru] 4.1: 639-2 special cases
>
>(editor hat)
>
><t>Use a specific language subtags or sequences of subtags in
>These subtags, which are comprised of the set of <xref
>target="ISO639-5"></xref> codes, are included as primary language
>subtags in the registry. (Some of the ISO 639-5 codes are also
>defined as collections in ISO 639-2.) Language collections in the
>registry have a 'Scope' field with a value of 'collection'. A subtag
>representing a language collection MAY be used when more specific
>language information is not available, but most tag processes and
>users do not know there is a relationship between the collection and
>its encompassed languages. In addition, in many cases the
>encompassed languages are not mutually intelligible or closely
>related enough for the common identifier to be useful in a language
>tagging context. Since the subtags are different, the tags are
>treated as being different languages: a request for the collection
>will only produce items tagged with the collection code, not with
>the individual languages it contains. For example, for the
>collection 'cmc' (Chamic languages), the registry contains the
>values for each of its approximately ten individual languages.
>Collections are interpreted inclusively, so the subtag 'gem'
>(Germanic langauges) could (but SHOULD NOT) be used with content
>that would be better tagged with "en" (English), "de" (German), or
>"gsw" (Swiss German, Alemannic). While 'gem' collects all of these
>(and other) languages under a single identifier, no reasonable user
>or process would prefer the collection to the more accurate
>individual language subtag.</t>
>
>
>
>Addison Phillips
>> > > languages. These codes are included as primary language
>subtags
>> in

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 11 Jul 2008 10:52:54 -0700

almost +1 (see below)

Mark

On Fri, Jul 11, 2008 at 10:40 AM, Phillips, Addison <[email-masked]>
wrote:

> <t>Use a specific language subtags or sequences of subtags in preference to
> subtags for language collections. A "language collection" is a subtag that
> represents multiple related languages. These subtags, which are comprised of
> the set of <xref target="ISO639-5"></xref> codes, are included as primary
> language subtags in the registry. (Some of the ISO 639-5 codes are also
> defined as collections in ISO 639-2.) Language collections in the registry
> have a 'Scope' field with a value of 'collection'. A subtag representing a
> language collection MAY be used when more specific language information is
> not available, but most tag processes and users do not know there is a
> relationship between the collection and its encompassed languages. In
> addition, in many cases the encompassed languages are not mutually
> intelligible or closely related enough for the

in many cases => in most cases

> common identifier to be useful in a language tagging context. Since the
> subtags are different, the tags are treated as being different languages: a
> request for the collection will only produce items tagged with the
> collection code, not with the individual languages it contains. For example,
> for the collection 'cmc' (Chamic languages), the registry contains the
> values for each of its approximately ten individual languages. Collections
> are interpreted inclusively, so the subtag 'gem' (Germanic langauges) could
> (but SHOULD NOT) be used with content that would be better tagged with "en"
> (English), "de" (German), or "gsw" (Swiss German, Alemannic). While 'gem'
> collects all of these (and other) languages under a single identifier, no
> reasonable user or process would prefer the collection to the more accurate

no reasonable user or process would prefer the collection to the more
accurate individual language subtag.
=> most implementations will not match 'gem' to the encompassed languages;
thus it will not have the desired result.

[no reasonable is a bit too strong - better to point out the practical
results]

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 10:57:53 -0700

Hm... but that's not what the last sentence says. It says:

> no reasonable user or process would prefer the collection to the more accurate

Earlier in the paragraph we give permission to the use you describe:

> >A subtag
> >language information is not available...

Perhaps we should add:

"A language collection subtag, when it can be determined, is always preferred to less specific alternatives such as 'und' (Undetermined) or 'mul' (Multiple languages)."

Addison

Addison Phillips

> From: Broome, Karen [mailto:[email-masked]]
> Sent: Friday, July 11, 2008 10:48 AM
> I would delete the last sentence. "gem" is preferable to "und" --
> so if one knows that one has a Germanic language, but does not know
> which one, "gem" is a reasonable tag to use. It would aid future
> processes to more correctly determine the variant to collect
> "Germanic" variants under this tag in an asset management system.
>
> Regards,
>
> Karen Broome
>
> >> I'm satisfied with the resolution of other items I raised in
> this
> >> thread.
> >>
> >>
> >>
> >> Peter
> >

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 11 Jul 2008 11:04:34 -0700

Ah, if that's what you meant by that last sentence, then your suggested text
below is better.

Mark

On Fri, Jul 11, 2008 at 10:57 AM, Phillips, Addison <[email-masked]>
wrote:

> accurate
> "A language collection subtag, when it can be determined, is always
> preferred to less specific alternatives such as 'und' (Undetermined) or
> 'mul' (Multiple languages)."
>
> Addison
>
>
>
> Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 11:07:52 -0700

Karen and Mark both commented. I have tried to incorporate all of our individual thoughts into a collection:

<t>Use a specific language subtags or sequences of subtags in preference to subtags for language collections. A "language collection" is a subtag that represents multiple related languages. These subtags, which are comprised of the set of <xref target="ISO639-5"></xref> codes, are included as primary language subtags in the registry. (Some of the ISO 639-5 codes are also defined as collections in ISO 639-2.) Language collections in the registry have a 'Scope' field with a value of 'collection'. A language collection subtag is always preferred to less-specific alternatives such as 'mul' and 'und' (see below) and a subtag representing a language collection MAY be used when more specific language information is not available. However, most users and implementations do not know there is a relationship between the collection and its encompassed languages. In addition, the relationship between the individual languages in the collection is not well defined; the languages might not be mutually intelligible or the relationship might prove unsuitable for a given application. Since the subtags are different, the tags are treated as being different languages: a request for the collection will only produce items tagged with the collection code, not with the individual languages it contains. Collections are interpreted inclusively, so, for example, the subtag 'gem' (Germanic langauges) could, but SHOULD NOT, be used with content that would be better tagged with "en" (English), "de" (German), or "gsw" (Swiss German, Alemannic). While 'gem' collects all of these (and other) languages under a single identifier, most implementations will not match 'gem' to the encompassed languages; thus using the subtag will not produce the desired result.</t>

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, July 11, 2008 11:05 AM
To: Phillips, Addison
Cc: Broome, Karen; LTRU Working Group
Subject: Re: [Ltru] 4.1: 639-2 special cases

Ah, if that's what you meant by that last sentence, then your suggested text below is better.

Mark
On Fri, Jul 11, 2008 at 10:57 AM, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:

Perhaps we should add:

Addison

Addison Phillips

> From: Broome, Karen [mailto:[email-masked]<mailto:[email-masked]>]
> >From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On
> Behalf
> >Of Phillips, Addison
> >> From: Peter Constable [mailto:[email-masked]<mailto:[email-masked]>]
> >> > From: Phillips, Addison [mailto:[email-masked]<mailto:[email-masked]>]
> >>
> >[email-masked]<mailto:[email-masked]>
_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 11 Jul 2008 20:38:10 +0200

> preference to subtags for language collections. A "language 

"Use a specific language subtags"...

Apart from the singular/plural combo, should this not be
"individual language" rather than "specific language".

	/kent k

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 11:41:01 -0700

> Apart from the singular/plural combo, should this not be

Now fixed.

> "individual language" rather than "specific language".

Probably. I've snuck up on using individual language consistently in the paragraph over the last couple of revisions. This makes the sentence read:

--
Use subtags or sequences of subtags for individual languages in preference to subtags for language collections.
--