Subject: Re: [Ietf-languages] Adding prefixes with dialect variants to Occitan orthographic variants

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 19 Apr 2021 14:53:12 +0000

Mark,

The prefix system *is* a canonical ordering mechanism---for those variant subtags that use it. As you note, that doesn’t constitute a *comprehensive* well-defined canonical ordering scheme, in part due to the existence of “generic” variants. CLDR attempts to skirt this with the alphabetic ordering of variants, but that ignores the “remove from right” and “prefix matching” semantics that language tags use matching, fallback, and selection outside the locale realm. It makes sense to provide an indication of which (and a canonical order for) subtags that are meant to be used together in these cases I think.

Addison

Addison Phillips
Sr. Principal SDE – I18N (Amazon)
Chair I18N WG, Amazon ACRep (W3C)

Internationalization is not a feature.
It is an architecture.

From: Ietf-languages <[email-masked]> On Behalf Of Mark Davis ??
Sent: Sunday, April 18, 2021 8:41 PM
To: Doug Ewell <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Phillips, Addison <addison=[email-masked]>; David Mediavilla <[email-masked]>
Subject: RE: [EXTERNAL] [Ietf-languages] Adding prefixes with dialect variants to Occitan orthographic variants

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

When we originally worked on BCP47, we were worried about applications truncating the variants to fit into a smaller footprint. And for such cases, it makes sense to have the "most important" first.

However, since that time, truncation is far less of a concern, and the bigger issue is that BCP47 doesn't provide a well-defined canonical order of all prefixes. Without that, comparison of language tags is fundamentally flawed. So CLDR extends BCP47 to simply order the variants alphabetically. Any application that cares about variants (and there are very few) can treat those variants they care about simply as a set. Thus both oc-gascon-phonipa and oc-phonipa-gascon are both interpreted by CLDR as [language=oc, variantSet={gascon, phonipa}].

> on which combinations might be valid

I know what you mean there, but this would be better phrased as "on which combinations are useful". All combinations of registered variant subtags are valid; it is just that some of them are pointless. This is completely analogous to a language tag such as "yev-Cher-AQ"; it is completely valid — syntactically and semantically — signifying "Yapunda written in the Cherokee script as used in Antarctica". But outside of fabricated samples, no text matches "yev-Cher-AQ".

Mark

On Sun, Apr 18, 2021 at 7:36 PM Doug Ewell <[email-masked]<mailto:[email-masked]>> wrote:
Addison Phillips wrote:

> I think you're right that some additional prefixes should be added to
> clarify the relationship of dialectical and orthographic variant
> subtags in order to allow them to be used together. This is clearly
> described in the RFC (using the Slovenian dialects as examples) and
> seems like an oversight in the previous registrations.

I'm counting on some designated representative of Lo Congrès to weigh in on which combinations might be valid.

At one extreme, if John Cowan's analysis is correct, we would only need to (1) add "oc-provenc" to 'grclass' and 'grmistr', and (2) add "oc-aranes" to 'grclass' and 'grital'. But then that would not satisfy the original requester's need.

At the other extreme, if any of the 3 orthographies could reasonably be used with any of the 10 dialects, we would need to add them all explicitly.

For what little it's worth, the current record for most Prefix values is 10 for 'baku1926'.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org<http://ewellic.org>

_______________________________________________
Ietf-languages mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ietf-languages

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 19 Apr 2021 13:51:21 -0600

Thanks to Lo Congrès for providing the expertise to answer this question. I’ll post paperwork to add these prefixes later today.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org

From: Lo Congrès <[email-masked]> 
Sent: Monday, April 19, 2021 11:34
To: Phillips, Addison <[email-masked]>; Mark Davis ☕ <[email-masked]>; Doug Ewell <[email-masked]>
Cc: [email-masked]; [email-masked]; Phillips, Addison <addison=[email-masked]>; David Mediavilla <[email-masked]>
Subject: Re: [Ietf-languages] Adding prefixes with dialect variants to Occitan orthographic variants

Hi,
Thanks for informing us about this issue. We didn't anticipate this when we submitted the subtags and, indeed, the prefix values should be added.
To answer your question about the combinations between varieties and scripts :
- "grclass" and "grmistr" tags could be useful with all varieties : auvern, gascon, lemosin, lengadoc, provenc, vivaraup, creiss, aranes, cisaup, nicard. And with the "oc" tag alone (when a word is common to all occitan varieties).
- The "grital" tag should normally be used only with the "nicard" and "provenc" subtags, and with the "oc" tag alone. I'm not sure about the "cisaup" tag, maybe we should add it as a precaution.
If you have any other questions, we would be glad to answer it.
Regards,
Aure Séguier
Lo Congrès permanent de la lenga occitant

---

From: Mark Davis ☕️ <[email-masked]>
Date: Mon, 19 Apr 2021 14:00:55 -0700

The ordering established by Prefix values is a SHOULD, not a MUST, so what
CLDR does with variants is completely conformant to BCP47.

   If more than one variant appears within a tag, processors MAY reorder
   the variants to obtain better matching behavior or more consistent
   presentation.  Reordering of the variants SHOULD follow the
   recommendations for variant ordering in Section 4.1
<https://tools.ietf.org/html/bcp47#section-4.1>.

It is unfortunate that the working group was not able to settle on a
complete canonicalization of variants, since a partial order is mostly
useless. Some of us pushed for that, but it was not to be.

No CLDR-conformant implementation is stopped from reordering variants as it
chooses, for operations using 'the “remove from right” and “prefix
matching” semantics that language tags *may* use *for* matching, fallback,
and selection outside the locale realm'. I add the two extra words {may,
for}, for while BCP47 describes algorithms for performing these operations,
BCP47 does not *require* those algorithms. The reasons for not requiring
those algorithms are clear, and were known at the time BCP47 was
written: implementations quickly discover that those algorithms are pretty
basic, and there are better ways to handle the operations of matching,
fallback, and selection.

Mark

On Mon, Apr 19, 2021 at 7:53 AM Phillips, Addison <[email-masked]>
wrote:

> Mark,
>
>
>
> The prefix system **is** a canonical ordering mechanism---for those
> variant subtags that use it. As you note, that doesn’t constitute a *
> *comprehensive** well-defined canonical ordering scheme, in part due to
> the existence of “generic” variants. CLDR attempts to skirt this with the
> alphabetic ordering of variants, but that ignores the “remove from right”
> and “prefix matching” semantics that language tags use matching, fallback,
> and selection outside the locale realm. It makes sense to provide an
> indication of which (and a canonical order for) subtags that are meant to
> be used together in these cases I think.
>
>
>
> Addison
>
>
>
> Addison Phillips
>
> *From:* Ietf-languages <[email-masked]> *On Behalf Of *Mark
> Davis ??
> *Sent:* Sunday, April 18, 2021 8:41 PM
> *To:* Doug Ewell <[email-masked]>
> *Cc:* [email-masked]; [email-masked]; [email-masked];
> Phillips, Addison <addison=[email-masked]>; David Mediavilla
> <[email-masked]>
> *Subject:* RE: [EXTERNAL] [Ietf-languages] Adding prefixes with dialect
> variants to Occitan orthographic variants
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> When we originally worked on BCP47, we were worried about applications
> truncating the variants to fit into a smaller footprint. And for such
> cases, it makes sense to have the "most important" first.
>
>
>
> However, since that time, truncation is far less of a concern, and the
> bigger issue is that BCP47 doesn't provide a well-defined canonical order
> of all prefixes. Without that, comparison of language tags is fundamentally
> flawed. So CLDR extends BCP47 to simply order the variants alphabetically.
> Any application that cares about variants (and there are very few) can
> treat those variants they care about simply as a set. Thus both
> oc-gascon-phonipa and oc-phonipa-gascon are both interpreted by CLDR as
> [language=oc, variantSet={gascon, phonipa}].
>
>
>
> I know what you mean there, but this would be better phrased as "on which
> combinations are *useful*". *All* combinations of registered variant
> subtags are *valid*; it is just that some of them are pointless. This is
> completely analogous to a language tag such as "yev-Cher-AQ"; it is
> completely valid — syntactically and semantically —
> signifying "Yapunda written in the Cherokee script as used in Antarctica".
> But outside of fabricated samples, no text matches "yev-Cher-AQ".
>
>
>
> Mark
>
>
>
>
>
> On Sun, Apr 18, 2021 at 7:36 PM Doug Ewell <[email-masked]> wrote:
>
> Addison Phillips wrote:
>
> At one extreme, if John Cowan's analysis is correct, we would only need to
> (1) add "oc-provenc" to 'grclass' and 'grmistr', and (2) add "oc-aranes" to
> 'grclass' and 'grital'. But then that would not satisfy the original
> requester's need.
>
> At the other extreme, if any of the 3 orthographies could reasonably be
> used with any of the 10 dialects, we would need to add them all explicitly.
>
> [email-masked]