Subject: [Ltru] 4.5: canonicalization and subtags mapped to Preferred-Value

From: Peter Constable <[email-masked]>
Date: Tue, 8 Jul 2008 17:56:40 -0700

In the current draft, Section 4.5 lists requirements for canonical form. In those requirements, point 2 states the need to map Region subtags to their preferred value, and point 3 states the need to map Grandfathered and Redundant tags to their preferred value. Then, point 4 states the following:

------------
Other subtags that have a Preferred-Value mapping in the IANA registry (see Section 3.1 (Format of the IANA Language Subtag Registry)) MUST be replaced with their mapped value. These items consist entirely of clerical corrections to ISO 639-1 in which the deprecated subtags have been maintained for compatibility purposes.
------------

"Other subtags" here would include any language, extlang, script or variant subtags. Section 3.1.2 indicates that any of these types of subtags potentially can have Preferred-Value mappings. Yet the text I've quoted states that the set of all these cases is limited to clerical corrections in 639-1 (the Javanese and Hebrew cases). That's definitely not the case (e.g. every extlang must have a Preferred-Value field.) Something's not right.

(I tried to check if something has changed here since previous drafts, but the Inter-Locale site appears to be down.)

Peter

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 21:57:56 -0700

Server at inter-locale should now be up again.

This section, as noted, is a mess. I've altered it significantly just now in an attempt to fix Peter's issues, plus others.

I quote the text here for archival purposes, but probably you'll want to see it on the document in a few minutes.

--
<section title="Canonicalization of Language Tags" anchor="canonical">
                                <t>Since a particular language tag is sometimes used by many processes, language tags SHOULD always be created or generated in a canonical form.</t>
                                <t>A language tag is in canonical form when:
   <list style="numbers">
                                                <t>The tag is well-formed according the rules in <xref target="syntax"/> and
      <xref target="sources"/>.</t>
                                                <t>Redundant or grandfathered tags that have a Preferred-Value mapping
      in the IANA registry (see <xref target="ianaformat"/>) MUST
      be replaced with their mapped value. These items either are
      deprecated mappings created before the adoption of this document
      (such as the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh")
      or are the result of later registrations or additions to this
      document (for example, "zh-hakka" was deprecated in favor of the
      ISO 639-3 code 'hak' when this document was adopted). These mappings SHOULD be done before additional processing, since there can be additional changes to subtag values. These field-body of the Preferred-Value for grandfathered and redundant tags is an "extended language range" (<xref target="RFC4647"></xref>) and might consist of more than one subtag.</t>

                                                <t>Subtags of type 'extlang' SHOULD be mapped to their Preferred-Value. The field-body of the Preferred-Value for extlangs is an "extended language range" and typically maps to a primary language subtag. For example, the subtag sequence "zh-hak" (Chinese, Hakka) would be replaced with the tag "hak" (Hakka).</t><t>Other subtags that have a Preferred-Value field
      in the IANA registry (see <xref target="ianaformat"/>) MUST be
      replaced with their mapped value. Most of these are either Region subtags where the country name or designation has changed or clerical corrections to ISO 639-1.</t>
                                                <t>If more than one extension subtag sequence exists, the extension
      sequences are ordered into case-insensitive ASCII order by singleton
      subtag.</t>
                                        </list>
                                </t>
                                <t>Example: The language tag "en-a-aaa-b-ccc-bbb-x-xyz" is in canonical form,
while "en-b-ccc-bbb-a-aaa-X-xyz" is well-formed and potentially valid (extensions 'a' and 'b' are not defined as of the publication of this document) but not in canonical form (the extensions are not in alphabetical order).</t>
                                <t>Example: The language tag "en-BU" (English as used in Burma) is
not canonical because the 'BU' subtag has a canonical mapping to 'MM' (Myanmar), although the tag "en-BU" maintains its validity.</t>
                                <t>Canonicalization of language tags does not imply anything about the use of upper or lowercase letters when processing or comparing subtags (and as described in <xref target="syntax"/>). All comparisons MUST be performed in a case-insensitive manner.</t>
                                <t>When performing canonicalization of language tags, processors MAY regularize the case of the subtags (that is, this process is OPTIONAL), following the case used in the registry (see <xref target="casing"></xref>). </t>

                                <t>Note: if the field 'Deprecated' appears in a registry record without an accompanying 'Preferred-Value' field, then that tag or subtag is deprecated without a replacement. Validating processors SHOULD NOT generate tags that include these values, although the values are canonical when they appear in a language tag.</t>
                                <t>An extension MUST define any relationships that exist between the
various subtags in the extension and thus MAY define an alternate
canonicalization scheme for the extension's subtags. Extensions MAY
define how the order of the
extension's subtags are interpreted. For example, an extension could
define that its subtags are in canonical order when the subtags are placed
into ASCII order: that is, "en-a-aaa-bbb-ccc" instead of
"en-a-ccc-bbb-aaa". Another
extension might define that the order of the subtags influences their
semantic meaning (so that "en-b-ccc-bbb-aaa" has a different value from
"en-b-aaa-bbb-ccc"). However, extension specifications SHOULD be designed so that they are tolerant of the typical processes described in <xref target="extensions"/>.</t>
                        </section>
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Peter Constable
> Sent: Tuesday, July 08, 2008 5:57 PM
> To: LTRU Working Group
> Subject: [Ltru] 4.5: canonicalization and subtags mapped to
> Preferred-Value
>
> In the current draft, Section 4.5 lists requirements for canonical
> form. In those requirements, point 2 states the need to map Region
> subtags to their preferred value, and point 3 states the need to
> map Grandfathered and Redundant tags to their preferred value. Then,
> point 4 states the following:
>
> ------------
> Other subtags that have a Preferred-Value mapping in the IANA
> registry (see Section 3.1 (Format of the IANA Language Subtag
> Registry)) MUST be replaced with their mapped value. These items
> consist entirely of clerical corrections to ISO 639-1 in which the
> deprecated subtags have been maintained for compatibility purposes.
> ------------
>
> "Other subtags" here would include any language, extlang, script or
> variant subtags. Section 3.1.2 indicates that any of these types of
> subtags potentially can have Preferred-Value mappings. Yet the text
> I've quoted states that the set of all these cases is limited to
> clerical corrections in 639-1 (the Javanese and Hebrew cases).
> That's definitely not the case (e.g. every extlang must have a
> Preferred-Value field.) Something's not right.
>
> (I tried to check if something has changed here since previous
> drafts, but the Inter-Locale site appears to be down.)
>
>
> Peter
>

---

From: Martin Hosken <[email-masked]>
Date: Wed, 9 Jul 2008 12:33:44 +0700

Dear Anderson,

I would propose adding (to draft-15):

6. Redundant script tags which are the same as the suppressed script for a language are not present.

In addition, I would like to ask about fonipa and fonupa. Currently in 4645 they have no prefix, which makes sense since they can be used with any language. But they can only be used with Latn script. In fact, they imply Latn script. So what should the canonical form of Eastern Bru in IPA be?

bru-Latn-fonipa

or can we in some way indicate the implication of Latn by fonipa and just use:

bru-fonipa

I would suggest that if we want to do this, then specifying a prefix in the record for fonipa, of Latn would be sufficient although we may want some wording somewhere along the lines of:

Where the prefix tag for a variant consists of a script, that variant implies the presence of that prefix and the prefix is required not to exist in the canonical form of the tag.

Yours,
Martin Hosken

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 8 Jul 2008 22:46:57 -0700

> script for a language are not present.

(technical hat)

We can't do that. There are valid reasons why someone might use a script subtag in this way (we actually spell some of them out). Canonicalization that removes real subtag distinctions would be bad, even if we suggest that such a tag choice is not wise in the vast preponderance of cases.

>
> In addition, I would like to ask about fonipa and fonupa. Currently
> in 4645 they have no prefix, which makes sense since they can be
> used with any language. But they can only be used with Latn script.
> In fact, they imply Latn script. So what should the canonical form
> of Eastern Bru in IPA be?
>
> bru-Latn-fonipa
>

Not in the registry, but not everything can or should be done in the registry. If you use 'fonipa' you most certainly understand that the text is in some inscrutable phonetic alphabet. A script subtag is probably not a good tagging choice at all in this case.

>
> I would suggest that if we want to do this, then specifying a
> prefix in the record for fonipa, of Latn would be sufficient
> although we may want some wording somewhere along the lines of:
>
> Where the prefix tag for a variant consists of a script, that
> variant implies the presence of that prefix and the prefix is
> required not to exist in the canonical form of the tag.

Prefixes never consist of "a script". They consist of language ranges. We could achieve the effect you want by changing the range to an extended language range. The record would then look something like:

Subtag: fonipa
Prefix: *-Latn

But this would have the negative effect of disallowing a tag such as "bru-fonipa", which I think is a perfectly fine tag: it would require the form "*-Latn-*-fonipa". Furthermore, 'fonipa' doesn't really imply "Latin script". It implies characters from the latin script, certainly. But this is not the same thing. Especially since some of the IPA characters aren't Latin script at all!

Best Regards,

Addison

---

From: Peter Constable <[email-masked]>
Date: Tue, 8 Jul 2008 23:11:27 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Martin Hosken

> I would propose adding (to draft-15):

Presumably this would be in 16.

> for a language are not present.

IIUC, you're saying that a canonical tag won't have a script subtag if that subtag is given in the Suppress-Script field in the subtag record for the language subtag used in that tag. (E.g., suppose language subtag 'aa' has Suppress-Script: 'Bbbb'; then tags of the form "aa-(xxx-)Bbbb..." are not canonical.) Is that right?

Two problems: first, you don't suggest how to produce a canonical equivalent for any given tag. Secondly, while "en-Latn" is not generally recommended, there may be application scenarios in which it is appropriate, perhaps used in contrast to something like "en-Hang", in which case the canonical form should include the normally-suppressed script subtag.

> In addition, I would like to ask about fonipa and fonupa. Currently in
> 4645 they have no prefix, which makes sense since they can be used with
> any language. But they can only be used with Latn script. In fact, they
> imply Latn script. So what should the canonical form of Eastern Bru in
> IPA be?
>
> bru-Latn-fonipa
>

By that line of reasoning, the canonical form for "fr-1694acad" could be simply "1694acad" -- except that that's not even well formed. You could remove script or region subtags in deriving a canonical tag if they were specified as the prefix for a variant, but they would have to be the *only* prefix for that variant (else the canonical form wouldn't be deterministic), and you'd end up with a canonical tag that is not valid, and not likely the same as what is used anywhere else. Comparisons would not be any easier, and we couldn't suggest that these canonical forms actually be what gets generated and used in interchange.

That is, if these got a Prefix field and you were eliminating the prefix. Alternatively, I suppose you could allow Suppress-Script or Suppress-Region fields for variant subtags and use that in deriving canonical tags, but that just means adding more machinery.

Either way, I think this would just get complicated for no particular gain. My inclination would be to say that 'fonipa' should require a prefix of 'Latn' (probably specified as an extended language range "*-Latn"), and always require that script tag be included, even if that seems redundant. It's clear that, in many cases such as "fr-1694acad" or "sl-nedis", we already end up requiring some degree of redundancy in the tag; I don't see a problem doing that in these cases.

> I would suggest that if we want to do this, then specifying a prefix in
> the record for fonipa, of Latn would be sufficient although we may want
> some wording somewhere along the lines of:
>
> Where the prefix tag for a variant consists of a script, that variant
> implies the presence of that prefix and the prefix is required not to
> exist in the canonical form of the tag.

But there's the rub: we'd be saying in one place that prefix SHOULD be included in tags with the given variant, but in another place that they not be used in the canonical form, and that the caonical form SHOULD be used. Not very self-consistent.

Peter

---

From: Peter Constable <[email-masked]>
Date: Tue, 8 Jul 2008 23:21:55 -0700

> Phillips, Addison

> Furthermore, 'fonipa' doesn't really imply
> "Latin script". It implies characters from the latin script, certainly.
> But this is not the same thing. Especially since some of the IPA
> characters aren't Latin script at all!

This is really more appropriate for the other list, but...

>From Section 4.2: "[T]he use of a script subtag such as 'Arab' (Arabic script) does not mean that the content contains only Arabic characters. It does mean that the language involved is predominantly in the Arabic script."

IMO, it would be entirely appropriate to say that IPA phonetic transcription is always "in" the Latin script and to give 'fonipa' a Prefix field of *-Latn-*.

Peter

---

From: Martin Hosken <[email-masked]>
Date: Wed, 9 Jul 2008 13:37:13 +0700

Dear Peter and also Addison,

> IIUC, you're saying that a canonical tag won't have a script subtag if that subtag is given in the Suppress-Script field in the subtag record for the language subtag used in that tag. (E.g., suppos…

Yes, as per 3.1.8 which says that tags SHOULD NOT specify the script if it is the same as a suppressed script.

> Two problems: first, you don't suggest how to produce a canonical equivalent for any given tag. 

How about: 6. A script subtag that is the same as the suppress-script script subtag for the language MUST be removed.

>Secondly, while "en-Latn" is not generally recommended, there may be application scenarios in which it is appropriate, perhaps used in contrast to something like "en-Hang", in which case the canonical form should include the normally-suppressed script subtag.

? What's wrong with "en-Hang" and "en" in that case? Are we saying that something that one part of the standard says you SHOULD NOT do is still considered appropriate for a canonical tag? If you want to be very explicit nobody is saying you CANNOT use "en-Latn" just that it isn't canonical.

> By that line of reasoning, the canonical form for "fr-1694acad" could be simply "1694acad" -- except that that's not even well formed. You could remove script or region subtags in deriving a canoni…

Yes I'm rapidly going off prefix here :)

> That is, if these got a Prefix field and you were eliminating the prefix. Alternatively, I suppose you could allow Suppress-Script or Suppress-Region fields for variant subtags and use that in deri…

Well it depends whether you want to have redundant information in your tags or a more complex analyser. There's plenty enough machinery in an analyser already and I would suggest that much of it is just as spurious. So what's a little more? :)

> Either way, I think this would just get complicated for no particular gain. My inclination would be to say that 'fonipa' should require a prefix of 'Latn' (probably specified as an extended languag…

I don't have a problem with this, although it's different from what I have been told regarding fonipa before that said that the Latn can be implied.

So which is canonically correct: en-fonipa (Latn removed by suppress script on en) or en-Latn-fonipa (since *-Latn is the prefix for fonipa)?

> But there's the rub: we'd be saying in one place that prefix SHOULD be included in tags with the given variant, but in another place that they not be used in the canonical form, and that the caonic…

I agree, and withdraw the Prefix hack suggestion.

So there are choices before us:

1. Add Prefix: *-Latn and probably require en-Latn-fonipa
2. Add Suppress Script: Latn to a variant and increase complexity slightly but I think, that would solve the whole issue relatively simply.
3. Do nothing and allow en-fonipa but bru-Latn-fonipa

Yours,
Martin

---

From: Peter Constable <[email-masked]>
Date: Wed, 9 Jul 2008 00:47:01 -0700

> From: Phillips, Addison [mailto:[email-masked]]
> Sent: Tuesday, July 08, 2008 9:58 PM

> This section, as noted, is a mess. I've altered it significantly...

A further problem I note is that the portion starting with "A language tag is in canonical form when..." is inconsistent: parts of it describe conditions satisfied by canonical form (as expected by that into sentence), and parts describe instead what changes to make to map a tag to a canonical equivalent. Something like the following seems like it might be better:

---------------
A language tag is in canonical form when:

1. The tag is well-formed according the rules in Section 2.1 (Syntax) and Section 2.2 (Language Subtag Sources and Interpretation).

2. The tag is not a redundant or grandfathered tag that has a Preferred-Value mapping in the IANA registry.

3. The tag does not contain language, script, region or variant subtags that have a Preferred-Value mapping in the IANA registry.

4. Any extensions are ordered into case-insensitive ASCII order by singleton subtag.

For a given tag, an equivalent tag in canonical form can be derived by applying the following changes:

1. Redundant or grandfathered tags that have a Preferred-Value mapping in the IANA registry MUST be replaced with their mapped value.

2. Subtags of type 'extlang' SHOULD be mapped to their Preferred-Value.

3. Other subtags that have a Preferred-Value field in the IANA registry MUST be replaced with their mapped value.

4. If more than one extension subtag sequence exists, the extension sequences MUST be re-ordered into case-insensitive ASCII order by singleton subtag.
---------------

I've left out some of the substantive wording that is in the current draft (e.g. explaining the origin of the grandfathered and reduandant tags); if this suggestion is accepted in principle, you might want to figure out appropriate ways to work that content back in.

There's another concern raised by the changes you made in that the canonical derivation for tags is vague wrt extlang. I suggest we leave that for a separate thread.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 07:40:28 -0700

>
> IMO, it would be entirely appropriate to say that IPA phonetic
> transcription is always "in" the Latin script and to give 'fonipa'
> a Prefix field of *-Latn-*.
>

That would have the effect of requiring the Latn subtag with 'fonipa', which I don't think is necessarily appropriate. What you really kind of mean is "fonipa has a Suppress-Script of *"... in the same way that a sign language subtag doesn't have much meaning with a script (sign writing aside).

Addison

---

From: Martin Hosken <[email-masked]>
Date: Wed, 9 Jul 2008 21:49:41 +0700

Dear Addison,

> That would have the effect of requiring the Latn subtag with 'fonipa', which I don't think is necessarily appropriate. What you really kind of mean is "fonipa has a Suppress-Script of *"... in the…

Not really. The 15924 JAC decided that IPA is a subset of Latn. So saying bru-Thai-fonipa is semantically meaningless. fonipa only makes sense in conjunction with Latn and therefore can imply it.

Yours,
Martin

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 07:56:21 -0700

>
> Not really. The 15924 JAC decided that IPA is a subset of Latn. So
> saying bru-Thai-fonipa is semantically meaningless. fonipa only
> makes sense in conjunction with Latn and therefore can imply it.
>

It doesn't necessarily make sense to REQUIRE a tag such as "en-Latn-fonipa", though. The 'Latn' subtag is superfluous and adds no additional semantic value. In fact, no language tag adds semantic value by including a script in the case of fonipa. For example, "zh-yue-Latn-HK-fonipa" is also unnecessarily long.

There are other similar examples. 'baku1926' implies Latn too. 'monoton' implies 'Grek'.

Addison

---

From: Martin Hosken <[email-masked]>
Date: Thu, 10 Jul 2008 12:12:29 +0700

Dear Addison,

> There are other similar examples. 'baku1926' implies Latn too. 'monoton' implies 'Grek'.

I agree entirely. But there is nothing in the registry or standard that says this. So I think we need a suppress script for these variants.

Yours,
Martin

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 22:17:00 -0700

No. We don't.

We have rule #1: Tag Content Wisely, complete with several pages of text saying this already. We could go on adding more registry cruft until the crack of doom. At some point, we have to let people tag stuff.

Addison

Addison Phillips

> Behalf Of Martin Hosken
> Sent: Wednesday, July 09, 2008 10:12 PM
> Cc: [email-masked]
> Subject: Re: [Ltru] Latn and fonipa
>
> Dear Addison,
>
> > It doesn't necessarily make sense to REQUIRE a tag such as "en-
> Latn-fonipa", though. The 'Latn' subtag is superfluous and adds no
> additional semantic value. In fact, no language tag adds semantic
> value by including a script in the case of fonipa. For example,
> "zh-yue-Latn-HK-fonipa" is also unnecessarily long.
> >
> I agree entirely. But there is nothing in the registry or standard
> that says this. So I think we need a suppress script for these
> variants.
>
> Yours,
> Martin

---

From: Martin Hosken <[email-masked]>
Date: Thu, 10 Jul 2008 14:36:29 +0700

Dear Addison,

> We have rule #1: Tag Content Wisely, complete with several pages of text saying this already. We could go on adding more registry cruft until the crack of doom. At some point, we have to let people…

The question is how do we turn 'tag wisely' into a set of canonicalisation rules. If we want wisdom in how canonical tags are made, then we have to codify that wisdom somehow. The point of canonicalisation is to take out as much ambiguity as we can. You say: -fonipa implies Latn so don't include it in your tag if you are being wise. What is the canonicalisation rule for that?

Yours,
Martin

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 10 Jul 2008 10:31:46 -0700

I disagree with your characterization of canonicalization. It does not "eliminate ambiguity". What it does is eliminate otherwise-unimportant differences between two semantically equivalent values.

We allow many unwise tags. For example, "tlh-Cyrl-AQ-fonupa" is well-formed, valid, and canonical. It's also laughable, bringing to mind, as it does, freezing mythical Star Trek aliens typing on cast off Soviet typewriters.

Canonicalization says how to take a tag and make it consistent in terms of capitalization/format and in terms of modern subtag choice. However, as a process, it does not disturb or destroy the original meaning of the tag. We cannot know why the user chose a specific set of subtags---they may have meaning in her particular context.

In the post I was responding to, you suggested we add even more information about tag choice to the registry--in particular, you suggested that we expand Suppress-Script to other subtag types (which, please note, is NOT considered as part of canonicalization today). I can't really support doing this or adding some other mechanism such as a Disallow-Prefix or some-such. It adds to the overall edifice of what implementers must deal with---at a time when most implementations are of the "stick 2*8alpha subtags together with hyphens" variety. The more arcane we require software to implement, the fewer conforming applications we will have.

Given that variants are rare and tend to be very highly specialized, supplying lots of formal requirements for their use and implementation will make them even less appealing.

Addison Phillips

> Sent: Thursday, July 10, 2008 12:36 AM
> Cc: [email-masked]
> > We have rule #1: Tag Content Wisely, complete with several pages
> of text saying this already. We could go on adding more registry
> cruft until the crack of doom. At some point, we have to let people
> tag stuff.
>
> The question is how do we turn 'tag wisely' into a set of
> canonicalisation rules. If we want wisdom in how canonical tags are
> made, then we have to codify that wisdom somehow. The point of
> canonicalisation is to take out as much ambiguity as we can. You
> say: -fonipa implies Latn so don't include it in your tag if you
> are being wise. What is the canonicalisation rule for that?
>
> Yours,
> Martin
>
> >
> > Addison
> >
> > Addison Phillips
> > > > > Not really. The 15924 JAC decided that IPA is a subset of
> Latn.
> > > So

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 10 Jul 2008 11:10:46 -0700

+1

Mark

On Thu, Jul 10, 2008 at 10:31 AM, Phillips, Addison <[email-masked]>
wrote:

> I disagree with your characterization of canonicalization. It does not
> "eliminate ambiguity". What it does is eliminate otherwise-unimportant
> differences between two semantically equivalent values.
>
> We allow many unwise tags. For example, "tlh-Cyrl-AQ-fonupa" is
> well-formed, valid, and canonical. It's also laughable, bringing to mind, as
> it does, freezing mythical Star Trek aliens typing on cast off Soviet
> typewriters.
>
> Canonicalization says how to take a tag and make it consistent in terms of
> capitalization/format and in terms of modern subtag choice. However, as a
> process, it does not disturb or destroy the original meaning of the tag. We
> cannot know why the user chose a specific set of subtags---they may have
> meaning in her particular context.
>
> In the post I was responding to, you suggested we add even more information
> about tag choice to the registry--in particular, you suggested that we
> expand Suppress-Script to other subtag types (which, please note, is NOT
> considered as part of canonicalization today). I can't really support doing
> this or adding some other mechanism such as a Disallow-Prefix or some-such.
> It adds to the overall edifice of what implementers must deal with---at a
> time when most implementations are of the "stick 2*8alpha subtags together
> with hyphens" variety. The more arcane we require software to implement, the
> fewer conforming applications we will have.
>
> Given that variants are rare and tend to be very highly specialized,
> supplying lots of formal requirements for their use and implementation will
> make them even less appealing.
>
> Addison Phillips