Subject: Re: [Ltru] 4.1: 639-2 special cases

From: "Doug Ewell" <[email-masked]>
Date: Tue, 8 Jul 2008 19:40:18 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

> <t>Use a specific language subtags or sequences of subtags in 
> preference to subtags for language collections. A "language 
> collection" is a subtag that represents multiple related languages. 
> These codes, which are the subset of <xref target="ISO639-2"></xref> 
> that is also defined in <xref target="ISO639-5"></xref>,

This is incorrect on two counts.  First, the set of collection codes 
isn't the subset of -2 codes that are also in -5, because that implies 
that the -5 codes that aren't in -2 aren't collection codes.  Of course, 
everything in -5 is a collection code.

Second, the collection codes in -2 aren't a subset of -5 anyway, because 
of 'bih' and 'day' and 'him' which are in -2 but not -5.  Yes, I know --  
this is supposed to be just a one-time synchronization error, an 
isolated anomaly, won't happen again, 639/JAC will be fixing it any day 
now.  Fine, I just know that they haven't fixed it yet, a month after it 
was reported, and if it happened once it can happen again, and the LSR 
doesn't have the luxury of waiting indefinitely for ISO to react.  He 
has to make a decision in 2 weeks, or explicitly extend the "review 
period" every 2 weeks; in this case we would be into our third 
fortnight.

> are included as primary language subtags in the registry. The 
> 'Description' field of a collection typically includes the word 
> "languages" to indicate that it represents more than one language.

Why don't we simply direct the user to look for the magic field "Scope: 
collection" in the Registry, instead of telling her to parse Description 
fields for the word "languages" (which will miss Bihari and Himachali 
anyway) or worrying about whether the Registry gives the original source 
of subtags?

I thought the main, overriding reason for adding the Scope field was so 
users wouldn't have to go through all this.

> For example, these subtags include "Chamic languages" ('cmc'), 
> "Algonquin languages" ('alg'), and "Germanic languages" ('gem').  Each 
> collection is also represented by subtags for the individual 
> languages. In the case of 'cmc', the registry also contains the values 
> for each of the approximately ten individual languages represented by 
> this collective code.

Should be "In the case of 'cmc', for example, the registry also 
contains...."  With the current wording, it looks as though only 'cmc' 
is special in this regard.  Maybe I just read the passage too fast, but 
when the draft is 80+ pages long, it's safe to say I won't be the last 
to do so.

> The subtag 'gem' helps illustrate this further: since it is 
> interpreted inclusively, content tagged with "en" (English), "de" 
> (German), or "gsw" (Swiss German, Alemannic) could also (but SHOULD 
> NOT) be tagged with "gem" (Germanic languages). Obviously, the 
> languages in a collection are frequently not mutually intelligible, as 
> this example demonstrates.

Should be "may not be" or "are not necessarily" mutually intelligible. 
This one selected example doesn't demonstrate anything about whether 
languages in a collection are "frequently" not mutually intelligible, 
although we here on the list know that to be the case.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 8 Jul 2008 20:47:26 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

> I removed the overall uniqueness requirement, narrowing it to within a 
> record and focusing on formatting variations.
> ...
> Each 'Description' field SHOULD be unique within the record in which 
> it appears and formatting variations of the same description SHOULD 
> NOT occur in that specific record. For example, while the ISO 639-1 
> code 'fy' contains both the descriptions "Western Frisian" and 
> "Frisian, Western", only one of these descriptions appears in the 
> registry.

I read this as going beyond the existing ban on comma-inverted 
variations, and implying that other types of minor formatting variations 
should also be collapsed into one Description field.  This would be VERY 
GOOD in my opinion.  It would mean that hyphenated versus non-hyphenated 
variations ("Macedo Romanian") would be collapsed, as would variations 
that differ only in the type of apostrophe used.  Is that correct?

If so, then we have to decide *which* variations to put in the Registry 
and which to leave out:

(language)
Description: Macedo Romanian
Description: Macedo-Romanian

(scripts)
Description: Ge&#x2BB;ez
Description: Ge'ez

Description: Hangul
Description: Hang&#x16D;l

Description: Hanunoo
Description: Hanun&#xF3;o

I might actually argue in favor of keeping the accented and unaccented 
forms of Hangul and Hanunoo; at least they are meant to be visually 
different.  The first two examples are a matter of individual taste in 
formatting; I would be surprised if any was expected to have normative 
staying power.

How about pairs of descriptions like the following, which exist because 
639-3 has to distinguish between similarly named items where 639-2 (with 
its smaller repertoire) does not, and which we've kept because of the 
desire for perfect traceability:

Description: Swahili (macrolanguage)
Description: Swahili

Description: Ainu (Japan)
Description: Ainu

There are lots of these.

This is a matter for LTRU, not ietf-languages, because we are the ones 
putting together draft-4645bis.  It would, however, fall to 
ietf-languages to make similar decisions during the maintenance phase.

---

From: Peter Constable <[email-masked]>
Date: Tue, 8 Jul 2008 20:19:00 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell

> now.  Fine, I just know that they haven't fixed it yet, a month after
> it was reported

Sigh... At least this week LoC acknowledged that JAC made a decision wrt day to make it a collection and that there was an oversight regarding that in the preparation of 639-5.

> collection" in the Registry, instead of telling her to parse
> Description fields for the word "languages"

What a concept! (+1)

> is special in this regard.

+1

> although we here on the list know that to be the case.

In and of itself, the wording "are frequently not" is OK. I agree, though, that the gsw/de example does not demonstrate *that* claim, whereas Doug's wording, making a lesser claim, would be demonstrated by the example.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 22:13:20 -0700

> everything in -5 is a collection code.

I removed the reference to -2, since, as you point out, it is a subset of the overall list of collections (and a strict subset at that).

> now.

I'm temporarily pretending that this glitch is a one-time issue that will disappear pre-publication. But we can take that up in draft-17.

> Description

Argh. Good point.

+1

> Should be "may not be" or "are not necessarily" mutually

Well, maybe not "obviously".

> intelligible.
An extensive edit produces:

<t>Use a specific language subtags or sequences of subtags in preference to subtags for language collections. A "language collection" is a subtag that represents multiple related languages. These codes, which comprise the set of <xref target="ISO639-5"></xref> codes, are included as primary language subtags in the registry. The 'Scope' field for a collection has the value 'collection'. Two illustrative examples of collections as language subtags are "Chamic languages" ('cmc') and "Germanic languages" ('gem').  Each of these collections is also represented by subtags for the individual languages. In the case of 'cmc', for example, the registry also contains the values for each of the approximately ten individual languages represented by this collective code. Collections are interpreted inclusively, so content tagged with "en" (English), "de" (German), or "gsw" (Swiss German, Alemannic) could (but SHOULD NOT) be tagged with "gem" (Germanic languages), which includes all of these (and other) languages. Languages in a collection are often not mutually intelligible and, while subtags derived from collection codes MAY be used when more specific language information is not available, most tag processes and users do not understand the relationship between the collection and its encompassed languages. Thus, users ought not assume a subtag based on a language collection is a useful means for selecting or identifying content in its encompassed languages.</t>

Addison

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 22:15:24 -0700

I specifically left it in the hands of the LSR and cohorts. That seems like a reasonable way to get out of spelling out every possible Bad Variation.

> ietf-languages to make similar decisions during the maintenance

Descriptions can be registered if LTRU gets it "wrong" (or at least less desirable) somehow. We should allow ietf-languages to do something useful when we're done.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: Tuesday, July 08, 2008 7:47 PM
> To: LTRU Working Group
> Subject: Re: [Ltru] 3.1.4: description field uniqueness
>

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 8 Jul 2008 23:18:21 -0600

+1

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 8 Jul 2008 23:25:39 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

> I specifically left it in the hands of the LSR and cohorts. That seems 
> like a reasonable way to get out of spelling out every possible Bad 
> Variation.
>>
>> This is a matter for LTRU, not ietf-languages, because we are the 
>> ones putting together draft-4645bis.  It would, however, fall to 
> Descriptions can be registered if LTRU gets it "wrong" (or at least 
> less desirable) somehow. We should allow ietf-languages to do 
> something useful when we're done.

Are you proposing that the Reviewer and cohorts be allowed to review the 
4645bis Registry for this type of duplication, and resolve each case 
unilaterally?

(rubs hands gleefully)

At what point can we establish that the WG has agreed to this, so we can 
get started?

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 22:50:51 -0700

> 4645bis Registry for this type of duplication, and resolve each

Not exactly: I'm proposing that the text in the various drafts that allow the Description fields to be modified using the registration process means that the Reviewer and cohorts be allowed to review requests for changes (which they can initiate) and resolve each case "unilaterally", provided "unilaterally" is understood to mean "a decision taken by the LSR subject to rough consensus of ietf-languages and subject to the usual appeals".

The text also says that the Reviewer (and cohorts) have broad editorial powers when registering changes by the ISO folks at the time a record addition/modification is *proposed*. The results, of course, are subject to the definition of "unilaterally" given above :-D.

> At what point can we establish that the WG has agreed to this, so

Now is good.

Addison

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 10:33:54 -0400

Phillips, Addison scripsit:

> <t>Use a specific language subtags or sequences of subtags in preference

Drop "a".

> to subtags for language collections. A "language collection" is
> a subtag that represents multiple related languages.

Metaphysical confusion here: a collection is not a subtag.  "A language
collection is a group of languages that descend from a common ancestor,
are spoken in the same geographical area, or are otherwise related.
Certain language collections are assigned codes by ISO 639-5."

> These codes,
> which comprise the set of <xref target="ISO639-5"></xref> codes,

Drop this line.

> are included as primary language subtags in the registry. The 'Scope'
> field for a collection has the value 'collection'. Two illustrative
> examples of collections as language subtags are "Chamic languages"

Again, collections are not subtags.  Read "collections with language subtags".

> ('cmc') and "Germanic languages" ('gem').  Each of these collections
> is also represented by subtags for the individual languages.

Collections are not represented by subtags for individual languages.
"Each of the languages in these collections are represented by primary
language subtags."

> In the
> case of 'cmc', for example, the registry also contains the values
> for each of the approximately ten individual languages represented
> by this collective code.

Collections don't represent languages, subtags represent languages.
"In the case of the Chamic languages, for example, the registry
contains not only 'cmc' but also subtags for each of the approximately
ten separate Chamic languages."

> Collections are interpreted inclusively,
> so content tagged with "en" (English), "de" (German), or "gsw"
> (Swiss German, Alemannic) could (but SHOULD NOT) be tagged with "gem"
> (Germanic languages), which includes all of these
> (and other) languages. Languages in a collection are often not mutually
> intelligible

"Often not" seems too weak:  "almost never" is more like it.

> and, while subtags derived from collection codes MAY be
> used when more specific language information is not available, most
> tag processes and users do not understand the relationship between
> the collection and its encompassed languages. Thus, users ought not

"Encompassed languages" is specific to macrolanguages.  Say
"the languages it contains" or just "its languages".

> assume a subtag based on a language collection is a useful means for
> selecting or identifying content in its encompassed languages.</t>

Same comment.

-- 
John Cowan              [email-masked]          http://www.ccil.org/~cowan
C'est la` pourtant que se livre le sens du dire, de ce que, s'y conjuguant
le nyania qui bruit des sexes en compagnie, il supplee a ce qu'entre eux,
de rapport nyait pas.               --Jacques Lacan, "L'Etourdit"
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 11:25:10 -0700

And, having revised the paragraph deeply, I find John's comments next in my backlog. Argh.

> preference

... usw.

I incorporated John's comments as appropriate. The paragraph now reads:

<t>Use specific language subtags or sequences of subtags in preference to subtags for language collections. A "language
collection" is a group of languages that are descended from a common ancestor,
are spoken in the same geographical area, or are otherwise related.
Certain language collections are assigned codes by <xref target="ISO639-5"></xref> (and some of these ISO 639-5 codes are also defined as collections in ISO 639-2). These codes are included as primary language subtags in the registry. Subtags for a language collection in the registry have a 'Scope' field with a value of 'collection'. A subtag for a language collection is always preferred to less-specific alternatives such as 'mul' and 'und' (see below) and a subtag representing a language collection MAY be used when more specific language information is not available. However, most users and implementations do not know there is a relationship between the collection and its individual languages. In addition, the relationship between the individual languages in the collection is not well defined; in particular, the languages are usually not mutually intelligible. Since the subtags are different, the tags are treated as being different languages: a request for the collection will only produce items tagged with the collection's subtag, not items tagged with subtags for the individual languages contained in the collection. Collections are interpreted inclusively, so, for example, the subtag 'gem' (Germanic langauges) could, but SHOULD NOT, be used with content that would be better tagged with "en" (English), "de" (German), or "gsw" (Swiss German, Alemannic). While 'gem' collects all of these (and other) languages, most implementations will not match 'gem' to the individual languages; thus using the subtag will not produce the desired result.</t>

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 11 Jul 2008 11:32:06 -0700

almost there
Mark

On Fri, Jul 11, 2008 at 11:25 AM, Phillips, Addison <[email-masked]>
wrote:

> <t>Use specific language subtags or sequences of subtags in preference to
> subtags for language collections. A "language
> Certain language collections are assigned codes by <xref
> target="ISO639-5"></xref> (and some of these ISO 639-5 codes are also
> defined as collections in ISO 639-2). These codes are included as primary
> language subtags in the registry. Subtags for a language collection in the
> registry have a 'Scope' field with a value of 'collection'. A subtag for a
> language collection is always preferred to less-specific alternatives such
> as 'mul' and 'und' (see below) and a subtag representing a language
> collection MAY be used when more specific language information is not
> available. However, most users and implementations do not know there is a
> relationship between the collection and its individual languages. In
> addition, the relationship between the individual languages in the
> collection is not well defined; in particular, the languages are usually not
> mutually intelligible. Since the subtags are different, the tags are treated
> as being different languages: a request for the collection wil
>  l only produce items tagged with the collection's subtag, not items tagged
> with subtags for the

...will typically only produce ...
[we can only say typically, since people may have more or less sophisticated
matching]

individual languages contained in the collection. Collections are
> interpreted inclusively, so, for example, the subtag 'gem' (Germanic
> langauges) could, but SHOULD NOT, be used with content that would be better
> tagged with "en" (English), "de" (German), or "gsw" (Swiss German,
> Alemannic). While 'gem' collects all of these (and other) languages, most
> implementations will not match 'gem' to the individual languages; thus using
> the subtag will not produce the desired result.</t>
>
>
> Addison

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 11 Jul 2008 11:38:23 -0700

Since the subtags are different, the tags are treated as being different languages: a request for the collection wil
 l only produce items tagged with the collection's subtag, not items tagged with subtags for the

...will typically only produce ...
[we can only say typically, since people may have more or less sophisticated matching]

DONE, as:

Since the subtags are different, a request for the collection will typically only produce items tagged with the collection's subtag, not items tagged with subtags for the individual languages contained in the collection.
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru