Subject: [Ietf-languages] Adding prefixes with dialect variants to Occitan orthographic variants

From: "Doug Ewell" <[email-masked]>
Date: Sat, 17 Apr 2021 18:45:00 -0600

I've been asked about the following case, broadly paraphrased:

Suppose one wants to create a tag for Occitan that specifically calls out both (a) the Gascon dialect and (b) the Mistralian orthography. This is not the most commonly encountered combination of dialect and orthography, but seems to exist in the wild, which is all the more reason why one might want to tag it.

The logical choice of tag would seem to be "oc-gascon-grmistr". However, all 13 of the variant subtags for Occitan -- those for dialects as well as those for orthographies -- have only one Prefix value:

Prefix: oc

That means, while it is technically permissible to create the tag "oc-gascon-grmistr" (just as any combination of discrete variant subtags is technically permissible), BCP 47 does not recommend it, and tools that create or validate tags will usually flag such a tag with a warning.

The approach recommended in BCP 47, and the one we actually use in practice, is to add additional Prefix values:

Prefix: oc
Prefix: oc-gascon
Prefix: oc-nicard
Prefix: oc-provenc
etc.

This has the added advantage of indicating the recommended order of variants, so that one will be encouraged to write, for example, "oc-gascon-grmistr" and not "oc -grmistr-gascon".

I'd like to get a feel for the Registrar's and other list members' inclination toward augmenting the Occitan orthographic variant subtags by adding Prefix values that include some or all of the dialect variants, as illustrated above. I've also cc'd stakeholders on this message, the requester, David Mediavilla, as well as Lo Congrès Permanent de la Langue Occitane and its representative who originally proposed the 13 variants in 2018.

If it is not expected that all combinations of 10 dialect variants and 3 orthographic variants might reasonably be used together, then we would need a list of which combinations do need to be added.

If we do not decide to add these Prefix values, then we should probably provide some sort of advice to the stakeholders how they should tag such a combination, in light of what BCP 47 says.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org

---

From: "Phillips, Addison" <[email-masked]>
Date: Sun, 18 Apr 2021 16:27:58 +0000

I think you're right that some additional prefixes should be added to clarify the relationship of dialectical and orthographic variant subtags in order to allow them to be used together. This is clearly described in the RFC (using the Slovenian dialects as examples) and seems like an oversight in the previous registrations.

Addison

Addison Phillips
Sr. Principal SDE – I18N (Amazon)
Chair I18N WG, Amazon ACRep (W3C)

Internationalization is not a feature.
It is an architecture.

-----Original Message-----
From: Ietf-languages <[email-masked]> On Behalf Of Doug Ewell
Sent: Saturday, April 17, 2021 5:45 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; 'David Mediavilla' <[email-masked]>
Subject: [EXTERNAL] [Ietf-languages] Adding prefixes with dialect variants to Occitan orthographic variants

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Prefix: oc

Prefix: oc
Prefix: oc-gascon
Prefix: oc-nicard
Prefix: oc-provenc
etc.

_______________________________________________
Ietf-languages mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf-languages

---

From: John Cowan <[email-masked]>
Date: Sun, 18 Apr 2021 19:43:09 -0400

>From what I understand, the classical orthography is suitable (or
unsuitable) for any dialect, the Mistralian orthography for Provencal only,
and the Italian orthography for Aranese only.

What concerns me now is which order the tags should go in.  The BCP 47
principle is that tags that create a greater barrier to mutual
intelligibility should come first.  But I am not sure if it is easier to
read text in the "wrong" orthography or in the "wrong" dialect from what
you are most familiar with.  That is a question for an Occitan-speaker or a
Occitanist.

On Sat, Apr 17, 2021 at 8:55 PM Doug Ewell <[email-masked]> wrote:

The logical choice of tag would seem to be "oc-gascon-grmistr". However,
> all 13 of the variant subtags for Occitan -- those for dialects as well as
> those for orthographies -- have only one Prefix value:
>
> Prefix: oc
>
> That means, while it is technically permissible to create the tag
> "oc-gascon-grmistr" (just as any combination of discrete variant subtags is
> technically permissible), BCP 47 does not recommend it, and tools that
> create or validate tags will usually flag such a tag with a warning.
>
> The approach recommended in BCP 47, and the one we actually use in
> practice, is to add additional Prefix values:
>
> Prefix: oc
> Prefix: oc-gascon
> Prefix: oc-nicard
> Prefix: oc-provenc
> etc.
>
> This has the added advantage of indicating the recommended order of
> variants, so that one will be encouraged to write, for example,
> "oc-gascon-grmistr" and not "oc -grmistr-gascon".
>
> I'd like to get a feel for the Registrar's and other list members'
> inclination toward augmenting the Occitan orthographic variant subtags by
> adding Prefix values that include some or all of the dialect variants, as
> illustrated above. I've also cc'd stakeholders on this message, the
> requester, David Mediavilla, as well as Lo Congrès Permanent de la Langue
> Occitane and its representative who originally proposed the 13 variants in
> 2018.
>
> If it is not expected that all combinations of 10 dialect variants and 3
> orthographic variants might reasonably be used together, then we would need
> a list of which combinations do need to be added.
>
> If we do not decide to add these Prefix values, then we should probably
> provide some sort of advice to the stakeholders how they should tag such a
> combination, in light of what BCP 47 says.
>
> --

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 18 Apr 2021 20:22:07 -0600

John Cowan wrote:

> unsuitable) for any dialect, the Mistralian orthography for Provencal
> only, and the Italian orthography for Aranese only.

The very use case from which this discussion arose was that the individual "was trying to tag Gascon text with Mistralian orthography." He cited the following image as evidence, although the inscription at the bottom of the statue is admittedly a very small, almost trivial, text sample:

https://commons.wikimedia.org/wiki/File:France-002076_-_Our_Lady_of_Lourdes_(15771785681).jpg

> intelligibility should come first.  But I am not sure if it is easier
> to read text in the "wrong" orthography or in the "wrong" dialect from
> what you are most familiar with.  That is a question for an Occitan-
> speaker or a Occitanist.

As neither the Registrar, an Occitan speaker, nor a resident of Occitania, I personally would think of this situation as "language X, dialect Y, written in the Z orthography" and not "language X, written in the Y orthography, dialect Z." I think "dialect" binds more closely to "language" than "orthography" does.

The only precedent we have for this in the LSR is for Resian, for which the orthographic variant '1994' always comes at the end, even when the sub-sub-dialect variants are included (e.g. "sl-rozaj-biske-1994").

As a coding specialist, I would hate to think that the canonical order for such combinations in BCP 47 should have to be decided on a per-language, or even per-dialect or per-orthography, basis.

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 18 Apr 2021 20:35:51 -0600

Addison Phillips wrote:

> I think you're right that some additional prefixes should be added to
> clarify the relationship of dialectical and orthographic variant
> subtags in order to allow them to be used together. This is clearly
> described in the RFC (using the Slovenian dialects as examples) and
> seems like an oversight in the previous registrations.

I'm counting on some designated representative of Lo Congrès to weigh in on which combinations might be valid.

At one extreme, if John Cowan's analysis is correct, we would only need to (1) add "oc-provenc" to 'grclass' and 'grmistr', and (2) add "oc-aranes" to 'grclass' and 'grital'. But then that would not satisfy the original requester's need.

At the other extreme, if any of the 3 orthographies could reasonably be used with any of the 10 dialects, we would need to add them all explicitly.

For what little it's worth, the current record for most Prefix values is 10 for 'baku1926'.

---

From: Mark Davis ☕️ <[email-masked]>
Date: Sun, 18 Apr 2021 20:41:01 -0700

When we originally worked on BCP47, we were worried about applications
truncating the variants to fit into a smaller footprint. And for such
cases, it makes sense to have the "most important" first.

However, since that time, truncation is far less of a concern, and the
bigger issue is that BCP47 doesn't provide a well-defined canonical order
of all prefixes. Without that, comparison of language tags is fundamentally
flawed. So CLDR extends BCP47 to simply order the variants alphabetically.
Any application that cares about variants (and there are very few) can
treat those variants they care about simply as a set. Thus both
oc-gascon-phonipa and oc-phonipa-gascon are both interpreted by CLDR as
[language=oc, variantSet={gascon, phonipa}].

> on which combinations might be valid

I know what you mean there, but this would be better phrased as "on which
combinations are *useful*". *All* combinations of registered variant
subtags are *valid*; it is just that some of them are pointless. This is
completely analogous to a language tag such as "yev-Cher-AQ"; it is
completely valid — syntactically and semantically —
signifying "Yapunda written in the Cherokee script as used in Antarctica".
But outside of fabricated samples, no text matches "yev-Cher-AQ".

Mark

On Sun, Apr 18, 2021 at 7:36 PM Doug Ewell <[email-masked]> wrote:

> At one extreme, if John Cowan's analysis is correct, we would only need to
> (1) add "oc-provenc" to 'grclass' and 'grmistr', and (2) add "oc-aranes" to
> 'grclass' and 'grital'. But then that would not satisfy the original
> requester's need.
>
> At the other extreme, if any of the 3 orthographies could reasonably be
> used with any of the 10 dialects, we would need to add them all explicitly.
>

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 19 Apr 2021 13:49:39 -0600

Mark Davis wrote:

> bigger issue is that BCP47 doesn't provide a well-defined canonical
> order of all prefixes. Without that, comparison of language tags is
> fundamentally flawed.

I agree with Addison that an ordering mechanism does exist, between the text of BCP 47 (Section 4.1, item 6) and the Prefix values in the Registry. There may be some gaps, making the mechanism less than perfect mathematically.

> So CLDR extends BCP47 to simply order the variants alphabetically. Any
> application that cares about variants (and there are very few) can
> oc-gascon-phonipa and oc-phonipa-gascon are both interpreted by CLDR
> as [language=oc, variantSet={gascon, phonipa}].

However, processes that use BCP 47 but not CLDR need to work with what is in BCP 47, and that includes adding Prefix values to guide the use of multiple variants.

> I know what you mean there, but this would be better phrased as "on
> which combinations are useful". All combinations of registered variant
> subtags are valid; it is just that some of them are pointless.

Yes, I was imprecise in my use of "valid", which is a term of art and needs to be used precisely. I meant combinations that would be "meaningful" in Occitan or have a snowball's chance of existing in non-artificial text samples, not "allowable" by the rules of BCP 47 or any other specification.

---

From: Mark Davis ☕️ <[email-masked]>
Date: Mon, 19 Apr 2021 14:15:49 -0700

See message in other thread, also

Mark

On Mon, Apr 19, 2021 at 12:49 PM Doug Ewell <[email-masked]> wrote:

> I agree with Addison that an ordering mechanism does exist, between the
> text of BCP 47 (Section 4.1, item 6) and the Prefix values in the Registry.
> There may be some gaps, making the mechanism less than perfect
> mathematically.
>

Sorting algorithms, for example, kinda depend on mathematical correctness.

> However, processes that use BCP 47 but not CLDR need to work with what is
> in BCP 47, and that includes adding Prefix values to guide the use of
> multiple variants.
>

A general-purpose implementation that depends on an ordering of variants
that is not *required* by BCP47 is a fragile implementation. And note that
the canonicalization process in BCP47 does not put the variant tags in a
particular order. The incoming language tags sl-rozaj-biske and
sl-biske-rozaj are both equally valid.

> Yes, I was imprecise in my use of "valid", which is a term of art and
> needs to be used precisely. I meant combinations that would be "meaningful"
> in Occitan or have a snowball's chance of existing in non-artificial text
> samples, not "allowable" by the rules of BCP 47 or any other specification.
>
> --