Subject: Re: [Ietf-languages] ISO TR 21636 (was: Language subtag registration form)

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 27 Nov 2020 23:49:38 +0000

I agree with Mark’s previous observations, as well as with Peter’s below.

While I can’t speculate on the utility of the conceptual framework (or of the coding system that might therefore be derived from it) without knowing more, I will observe that this isn’t the first attempt at a some type of universal language classification system. The primary concern that I had about previous efforts of this sort was: what problem is it solving? What applications would such a system enable (thereby sparking interest in widespread implementation and adoption)?

I’ll be curious to see materials as they develop.

Addison

Addison Phillips
Sr. Principal SDE – I18N (Amazon)
Chair I18N WG (W3C)

Internationalization is not a feature.
It is an architecture.

From: Ietf-languages [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Wednesday, November 25, 2020 11:06 PM
To: Mark Davis ☕️ <[email-masked]>; Doug Ewell <[email-masked]>
Cc: [email-masked]; Sebastian Drude <[email-masked]>
Subject: RE: [EXTERNAL] [Ietf-languages] Language subtag registration form

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Right now, the draft for TR 21636 doesn’t provide any _list(s)_, other than the list of eight dimensions of variation. It does define a coding system; it defines a conceptual framework. A coding system could be developed based on the framework, but I think it would perhaps be a bit premature for it to do so in its initial publication (for which reason, a TR is more appropriate than an IS).

> the allowable values for each of the eight dimensions may not be open-ended; they must be specified concretely somewhere

Some of the dimensions would probably be amenable to defining an enumeration of values from the outset. E.g., maybe that would be possible for modality/medium; or perhaps a preliminary set of values could be defined for which there’s good confidence that they would be widely used, with other values added over time as there is more clarity.

But coded values for some dimensions really would need to be long-term extensible. Just like the list of variant subtags in BCP 47.

Of course, neither Mark nor Doug has suggested that lists of coded values would need to be defined up-front before a coding system could be useful. Only that the code sets would need to be publicly and freely documented. The Language Subtag Registry itself is a good model for that, as well as ISO 639-3.

In principle, one might expect the registries for the “t” and “u” extensions to be good examples, though they aren’t as good as the above examples.

-        The documentation of the “u” extension is OK: they’re defined in a specific section of the LDML spec, and it’s not too hard to discover (UTS #35: Unicode Locale Data Markup Language<http://www.unicode.org/reports/tr35/#Locale_Extension_Key_and_Type_Data>); but if you want a machine-readable data file, you’ll have to figure out how to navigate all of the data contained in a CLDR release—which is not going to be easy for a newcomer. Of course, the “u” extension is primarily (solely?) used in connection with CLDR, so that’s not surprising. But it still means it’s not the best model for something based on 21636 to follow.
-        The “t” extension really is not a good example to follow: assignment of ‘t’ subtags is done as part of the Unicode CLDR project. When RFC 6497 was written, it defined certain values, and it allows for future assignments to be made in the LDML spec; but it’s harder to discover the documentation since there’s no direct trail from the RFC or from the ‘t’ extension registration form to a specific section of the LDML spec; so the reader has to figure out it’s somewhere in the LDML spec and go navigate that doc to find it. Manageable if you’re somewhat familiar, but probably not otherwise. And, like ‘u’, any machine-readable data is buried in the CLDR data. And that’s unfortunate, really, since the ‘t’ extension is potentially useful outside of the CLDR context, unlike the ‘u’ extension.

Peter

From: Ietf-languages <[email-masked]<mailto:[email-masked]>> On Behalf Of Mark Davis ??
Sent: Wednesday, November 25, 2020 9:26 AM
To: Doug Ewell <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Sebastian Drude <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Ietf-languages] Language subtag registration form

> they must be specified concretely somewhere

I'd like to emphasize that the list must be freely available (not behind the common ISO paywall), versioned, and stable (in the sense that new items can be added, and existing items can be deprecated, but nothing can be removed). That is, similar to what we see in https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Flanguage-subtag-registry%2Flanguage-subtag-registry&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639376226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=G%2Fq%2BK%2BUuipRvaHKNHxEURA%2Fk%2FLP7GctcsZCzFs6OxcM%3D&reserved=0> and in https://iso639-3.sil.org/code_tables/download_tables<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiso639-3.sil.org%2Fcode_tables%2Fdownload_tables&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639376226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VfrQzsT10TuSSqRkEPgt86Fa0BDqtIg9n8a9sRU%2Bayk%3D&reserved=0> for ISO 639-3.

Mark

On Tue, Nov 24, 2020 at 4:21 PM Doug Ewell <[email-masked]<mailto:[email-masked]>> wrote:
I'd be willing to work with the ISO TC 21636 folks, as a specialist in encoding things rather than in languages or linguistics per se.

Keep in mind that in order to make 21636 work with BCP 47, it really has to be framed as an extension, and the allowable values for each of the eight dimensions may not be open-ended; they must be specified concretely somewhere. See https://tools.ietf.org/html/rfc5646#section-2.2.6<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5646%23section-2.2.6&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wXO%2B%2FbVB7Q%2FmvNU0tr0qcBCwIA0JuvpLzI8lismCTQw%3D&reserved=0> for more information about BCP 47 extensions. Changing the syntax of BCP 47 tags to add new subtag types is almost certainly a non-starter.

Regarding "sociolect," Ben may be open to changing that one instance of the word. Alternatively, it could be argued that users of Arcaicam Esperantom do indeed belong to a particular social group: that subset of Esperanto users who wish to explore a posited "early form."

--
Doug Ewell, CC, ALB | Thornton, CO, US | ewellic.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fewellic.org%2F&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5sYvwsPCUZv8rG%2FASYvqF716EEBzBvqYpjne3CblLlA%3D&reserved=0>

From: Ietf-languages <[email-masked]<mailto:[email-masked]>> On Behalf Of Sebastian Drude
Sent: Tuesday, November 24, 2020 16:36
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [Ietf-languages] Language subtag registration form

Hi everyone,
this is the first time I reply directly to such a request, please forgive me if I do mistakes or do not know the correct procedures.
I agree with the proposal; it makes sense to me to have a subtag for this variety of Esperanto.
I only do not agree with the characterization as a "Sociolect" -- a variety of a language which is characteristic for a certain (social) group within the whole speaker community.  As the applicant explains, the same speakers/writers may use it in the same text, so this is (according to the upcoming ISO TC 21636) clearly some kind of register, not a sociolect.
Which brings us back to the question of how in the future the 8 dimensions identified in ISO TC 21636 can be reflected/applied in the current frameworks, in particular BCP 47).  Are there members of this group who would like to participate with me in a working group that would work out proposals for that?
(Now I see that in point (4) of the proporal 'Arcaicam Esperantom' is also correctly identified as such, so the characterization as sociolect in (3) is probably a mistake or misunderstanding.)

_______________________________________________
Ietf-languages mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ietf-languages<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fietf-languages&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xmeWTKgk9DAu7TGtK4AhEqvnf0AEKseyjWSq5Gvz0Ts%3D&reserved=0>

---

From: Sebastian Drude <[email-masked]>
Date: Fri, 27 Nov 2020 22:53:59 -0300

Dear Addison,

I agree that it remains to be seen which dimensions will actually be 
used in language resource tagging/description.  Some, as time and 
person, are usually covered already in other metadata, others may have 
very few applications, and for most the default values will hold in 99% 
of the cases (communicative functioning, for instance, or proficiency).

But one can foresee that in a world in which we increasingly communicate 
via and with machines, and increasingly orally, labeling or recognition 
of the linguistic variation of a given utterance or resource may indeed 
be increasingly relevant.  We know that dialects and some salient 
sociolects (such as BAE) are already being tagged to adjust speech 
recognition systems.

So, as ISO was, some 10 years ago, planning to implement ISO 639-6, for 
linguistic varieties, the first attempt propsed a very unfortunate 
one-dimensional classification system that mixed, at least, the medium, 
space and time dimensions.  It was rejected.
It became clear that, in order to avoid that each company builds their 
own system, where the same variety may be inconsistently described or 
classified, we proposed to first devise a conceptual framework, where 
then the varieties that one in the future would need to tag at least 
have a proper place, recognizing that any utterance ALWAYS is located in 
EACH of a number of independent dimensions, and so coherence is achieved 
much more easily (we hope).  That is why work on the new working item 
(aiming at a technical report) 21636 was started.  (As my linguistic 
training had a strong theoretical emphasis on linguistic variation, it 
was easy for me to draft the first versions, and we took it from there.)

I hope I will be able to share the draft TR document soon with this 
group.  Be aware, however, that after around 5 years of work and intense 
discussion in ISO TC37/SC2/WG1, the framework as such will not be 
fundamentally or drastically changed.  Individual terms, definitions, 
examples or use cases can, however, be added or changed.

Best wishes,

Sebastian

-- 

Museu P.E. Goeldi, CCH, Linguistica ▪ Av. Perimetral, 1901
Terra Firme, CEP: 66077-530 ▪ Belém do Pará – PA ▪ Brazil
[email-masked] ▪ +55 (91) 3217 6024 ▪ +55 (91) 983733319
Priv: Tv. Juvenal Cordeiro, 184, Apt 104 ▪ 66070-300 Belém

On 27/11/2020 20:49, Phillips, Addison wrote:
> While I can’t speculate on the utility of the conceptual framework (or 
> of the coding system that might therefore be derived from it) without 
> knowing more, I will observe that this isn’t the first attempt at a 
> some type of universal language classification system. The primary 
> concern that I had about previous efforts of this sort was: what 
> problem is it solving? What applications would such a system enable 
> (thereby sparking interest in widespread implementation and adoption)?
>
> *From:* Ietf-languages [mailto:[email-masked]] *On 
> Behalf Of *Peter Constable
> *Sent:* Wednesday, November 25, 2020 11:06 PM
> *To:* Mark Davis ☕️ <[email-masked]>; Doug Ewell <[email-masked]>
> *Cc:* [email-masked]; Sebastian Drude <[email-masked]>
> *Subject:* RE: [EXTERNAL] [Ietf-languages] Language subtag 
> registration form
>
> *CAUTION*: This email originated from outside of the organization. Do 
> not click links or open attachments unless you can confirm the sender 
> and know the content is safe.
>
> Right now, the draft for TR 21636 doesn’t provide any _/list(s)/_, 
> other than the list of eight dimensions of variation. It does define a 
> coding system; it defines a conceptual framework. A coding system 
> could be developed based on the framework, but I think it would 
> perhaps be a bit premature for it to do so in its initial publication 
> (for which reason, a TR is more appropriate than an IS).
>
> > the allowable values for each of the eight dimensions may not be 
> open-ended; they must be specified concretely somewhere
>
> Some of the dimensions would probably be amenable to defining an 
> enumeration of values from the outset. E.g., maybe that would be 
> possible for modality/medium; or perhaps a preliminary set of values 
> could be defined for which there’s good confidence that they would be 
> widely used, with other values added over time as there is more clarity.
>
> But coded values for some dimensions really would need to be long-term 
> extensible. Just like the list of variant subtags in BCP 47.
>
> Of course, neither Mark nor Doug has suggested that lists of coded 
> values would need to be defined up-front before a coding system could 
> be useful. Only that the code sets would need to be publicly and 
> freely documented. The Language Subtag Registry itself is a good model 
> for that, as well as ISO 639-3.
>
> In principle, one might expect the registries for the “t” and “u” 
> extensions to be good examples, though they aren’t as good as the 
> above examples.
>
> -The documentation of the “u” extension is OK: they’re defined in a 
> specific section of the LDML spec, and it’s not too hard to discover 
> (UTS #35: Unicode Locale Data Markup Language 
> <http://www.unicode.org/reports/tr35/#Locale_Extension_Key_and_Type_Data>); 
> but if you want a machine-readable data file, you’ll have to figure 
> out how to navigate all of the data contained in a CLDR release—which 
> is not going to be easy for a newcomer. Of course, the “u” extension 
> is primarily (solely?) used in connection with CLDR, so that’s not 
> surprising. But it still means it’s not the best model for something 
> based on 21636 to follow.
>
> -The “t” extension really is not a good example to follow: assignment 
> of ‘t’ subtags is done as part of the Unicode CLDR project. When RFC 
> 6497 was written, it defined certain values, and it allows for future 
> assignments to be made in the LDML spec; but it’s harder to discover 
> the documentation since there’s no direct trail from the RFC or from 
> the ‘t’ extension registration form to a specific section of the LDML 
> spec; so the reader has to figure out it’s somewhere in the LDML spec 
> and go navigate that doc to find it. Manageable if you’re somewhat 
> familiar, but probably not otherwise. And, like ‘u’, any 
> machine-readable data is buried in the CLDR data. And that’s 
> unfortunate, really, since the ‘t’ extension is potentially useful 
> outside of the CLDR context, unlike the ‘u’ extension.
>
> Peter
>
> *From:* Ietf-languages <[email-masked] 
> <mailto:[email-masked]>> *On Behalf Of *Mark Davis ??
> *Sent:* Wednesday, November 25, 2020 9:26 AM
> *To:* Doug Ewell <[email-masked] <mailto:[email-masked]>>
> *Cc:* [email-masked] <mailto:[email-masked]>; 
> Sebastian Drude <[email-masked] <mailto:[email-masked]>>
> *Subject:* Re: [Ietf-languages] Language subtag registration form
>
> I'd like to emphasize that the list must be freely available (not 
> behind the common ISO paywall), versioned, and stable (in the sense 
> that new items can be added, and existing items can be deprecated, but 
> nothing can be removed). That is, similar to what we see in 
> https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry 
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Flanguage-subtag-registry%2Flanguage-subtag-registry&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639376226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=G%2Fq%2BK%2BUuipRvaHKNHxEURA%2Fk%2FLP7GctcsZCzFs6OxcM%3D&reserved=0> and 
> in https://iso639-3.sil.org/code_tables/download_tables 
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiso639-3.sil.org%2Fcode_tables%2Fdownload_tables&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639376226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VfrQzsT10TuSSqRkEPgt86Fa0BDqtIg9n8a9sRU%2Bayk%3D&reserved=0> 
> for ISO 639-3.
>
> Mark
>
>     I'd be willing to work with the ISO TC 21636 folks, as a
>     specialist in encoding things rather than in languages or
>     linguistics per se.
>
>     Keep in mind that in order to make 21636 work with BCP 47, it
>     really has to be framed as an extension, and the allowable values
>     for each of the eight dimensions may not be open-ended; they must
>     be specified concretely somewhere. See
>     https://tools.ietf.org/html/rfc5646#section-2.2.6
>     <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5646%23section-2.2.6&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wXO%2B%2FbVB7Q%2FmvNU0tr0qcBCwIA0JuvpLzI8lismCTQw%3D&reserved=0>
>     for more information about BCP 47 extensions. Changing the syntax
>     of BCP 47 tags to add new subtag types is almost certainly a
>     non-starter.
>
>     Regarding "sociolect," Ben may be open to changing that one
>     instance of the word. Alternatively, it could be argued that users
>     of Arcaicam Esperantom do indeed belong to a particular social
>     group: that subset of Esperanto users who wish to explore a
>     posited "early form."
>
>     --
>     Doug Ewell, CC, ALB | Thornton, CO, US | ewellic.org
>     <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fewellic.org%2F&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5sYvwsPCUZv8rG%2FASYvqF716EEBzBvqYpjne3CblLlA%3D&reserved=0>
>
>
>     From: Ietf-languages <[email-masked]
>     <mailto:[email-masked]>> On Behalf Of Sebastian Drude
>     To: [email-masked] <mailto:[email-masked]>
>     this is the first time I reply directly to such a request, please
>     forgive me if I do mistakes or do not know the correct procedures.
>     I only do not agree with the characterization as a "Sociolect" --
>     a variety of a language which is characteristic for a certain
>     (social) group within the whole speaker community.  As the
>     applicant explains, the same speakers/writers may use it in the
>     same text, so this is (according to the upcoming ISO TC 21636)
>     clearly some kind of register, not a sociolect.
>     Which brings us back to the question of how in the future the 8
>     dimensions identified in ISO TC 21636 can be reflected/applied in
>     the current frameworks, in particular BCP 47).  Are there members
>     of this group who would like to participate with me in a working
>     group that would work out proposals for that?
>     (Now I see that in point (4) of the proporal 'Arcaicam Esperantom'
>     is also correctly identified as such, so the characterization as
>     sociolect in (3) is probably a mistake or misunderstanding.)
>
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/ietf-languages
>     <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fietf-languages&data=04%7C01%7C%7Cc49d24d160c5470eac7a08d891676aa3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637419220639386221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xmeWTKgk9DAu7TGtK4AhEqvnf0AEKseyjWSq5Gvz0Ts%3D&reserved=0>
>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-languages