Subject: [Ietf-languages] First cut at a BCP 47 extension structure for ISO TR 21636 (was: Language subtag registration form)

From: "Doug Ewell" <[email-masked]>
Date: Fri, 27 Nov 2020 16:01:23 -0700

Sebastian Drude wrote:

> I am here participating in this group exactly to touch base and see
> how we can implement this in a way that it is usable and compatible
> with BCP 47, BEFORE a list is created which may not be compatible in
> some way to IETF/Languages' approach.

Here's one possible way the structure of ISO TR 21636 could be made to fit into a BCP 47 extension. This shows the type of groundwork that should, and really must, be done up front if it is a major goal to make 21636 fit into BCP 47, without requiring all or even most of the values or code elements to be known.

As a partial reference, I'm using a document called "NP-21636," provided by Sebastian and dated 2019-09-30. It's up to Sebastian whether I may redistribute that document to anyone.

First, an extension singleton must be chosen. Here I suggest 'v' for "varieties," as the title of the TR is "Identification and description of language varieties." The alphabetically consecutive nature of the three extensions, 't' and 'u' and now 'v', may or may not be unfortunate.

Next, the TR identifies eight "dimensions" which are independent of each other to a greater or lesser extent. It is clearly desirable to be able to specify more than one dimension in a single tag, up to all eight, but not necessarily so. For this I will suggest a key-value mechanism like that devised for the 'u' extension, so that each dimension is indicated by a two-character key, derived from the names Sebastian provided on the 24th:

sp	Space
ti	Time
so	Social group
me	Medium
si	Situation
pe	Person
pr	Proficiency
co	Communicative functioning

(The NP-21636 document used the term "performance" instead of "communicative functioning," which would have caused a minor inconvenience by overloading 'pe'.)

Then there would be "value" identifiers of 3 to 8 characters each, providing values for each of the dimensions. The lengths of each of these subtags, 1 for the extension singleton and 2 for the dimensions and 3 to 8 for the values, are a critical part of BCP 47 syntax, permitting humans and processes to parse the tag and figure out what modifies what. 

So you might have "pt-BR-v-so-business-pr-fullpro" as one example.

A finite number of values for each dimension would need to be identified so they could be encoded, keeping uniqueness and syntactical constraints in mind. This is the process of building a code list, or registry. The set of values would be extensible; no one and no group could be expected to get this perfect the first time. But stability rules would also come into play here.

Note that the values might not have to be unique across dimensions; you could have "en-v-so-foo-pr-foo" if the string 'foo' represented a social group value and coincidentally also a proficiency value. Too much of this could lead to human confusion, though.

I have no idea what to do about things like "certainty status," but some syntax along these lines could be devised. If this is an "adjective" modifying dimensional values, rather than a value itself, that should be syntactically clear.

The dimension that troubles me the most, of course, is "person." The document makes it clear in several passages that Sebastian has his own personal variety (times the number of languages in which he can communicate), I have my own, Michael and John and Leon and Peter have their own, and so forth. Obviously neither we at ietf-languages, nor anyone else, are going to encode identifiers for each of 7.7 billion living people, plus those deceased, plus those not yet born. It may be that the only possible way to encode this dimension will be with a private-use subtag at (by definition) the end of the tag.

This list may or may not be the proper venue to continue this discussion.

--
Doug Ewell, CC, ALB | Thornton, CO, US | ewellic.org

---

From: Sebastian Drude <[email-masked]>
Date: Fri, 27 Nov 2020 20:43:46 -0300

Dear Doug,

thank you so much for this.  It goes EXACTLY along the lines of what I 
have imagined could be achieved, as much as I have understood of BCP 47.

For me, all what you have proposed could be taken over to be the seed of 
a future regulation.

I also agree that repeating values in different dimensions should be 
avoided, even if they are syntactically possible.

I believe we should not worry about the personal varieties at this 
point.  I am quite confident that whatever framework of metadata (which 
is what BCP 47 is all about, or is always in a context of) will 
certainly already provide ways of indicating the 
author/speaker/signer/... (one -- or several, in a resource with a 
dialogue, for instance), so that it would be a redundant feature for the 
language tag anyways.  Also, this dimension would be relevant only in 
very special applications, if ever.

Probably we should form a small committee to continue this discussion, 
instead of involving (and spamming, at this point) the whole list.  I am 
certainly willing to be instrumental in (contributing to) forming and 
driving such a group, in whichever role I can contribute best.

Points that I can see now that would need to be discussed, besides 
flashing out what you have begun:

-- interaction with existing subtags, in particular dialects -- probably 
we want to avoid synonym language tags composed according to different 
frameworks

-- can more than one value in the SAME dimension be indicated? (I would 
argue yes, if that is syntactically okay)

-- the 'certainty' and similar "adjectives" (yes, that is how I would 
see them; -- e.g. primary vs. secondary modality, genuine vs. imitated, ...)

-- default values

-- requirements for a registry, and its feasability, and finally 
implementation

One question I have: need the values in the key-value-pairs be unique 
over different languages?  As an example, can two dialects of different 
languages share the same string xyz as a designation used as value in 
...v-...-sp-xyz?
Alternatively/additionally, we could use the glottocodes for the major 
dialects already included in the Glottolog (they are a string of 4 
letters and 4 digits), and come to an agreement with the Glottolog folks 
to extend their list of dialects.
(It would be excellent to be in touch with them anyways, also for cases 
where ISO is less accurate than the Glottologue.)

Again, thanks so much.  I hope we can build on this initial proposal!

Sebastian

-- 

Museu P.E. Goeldi, CCH, Linguistica ▪ Av. Perimetral, 1901
Terra Firme, CEP: 66077-530 ▪ Belém do Pará – PA ▪ Brazil
[email-masked] ▪ +55 (91) 3217 6024 ▪ +55 (91) 983733319
Priv: Tv. Juvenal Cordeiro, 184, Apt 104 ▪ 66070-300 Belém

---

From: Hugh Paterson III <[email-masked]>
Date: Sat, 28 Nov 2020 01:54:08 +0100

Sebastian,

This is all very interesting, I am particularly interested by the comment
related to the Glottolog.
1. You refer to their "list of dialects" but are you certain that some of
these sub-varieties are not actually socio-lects, and therefore should not
be placed in the space dimension? It would be problematic if one were to
assume that the Glottolog's list of ways to subcategorize languages were
all within one dimension. Having said that, my understanding, having talked
with one of their editors is that they don't create their own data but
rather they aggregate other people's data. But perhaps the editors do have
first hand or documentary evidence for their sub-categorization as you seem
to suggest.
2. My understanding is that people can already use glottocodes (even now)
with BCP47 with the -x- extension in the private use area.
3. Having communicated with a different editor of the Glottolog, my
understanding from him was that the seed data for the Glottolog came from
the Ethnologue directly and indirectly via Multi-tree which the indirect
method included the sub-categorization. So, maybe it would be equally good
to consult that reference work.
4. Please correct me if I am wrong, but my impression after having read
some of the Glottolog's about pages was that they do occasionally obliviate
their ID codes that they use, which is a different process than the status
that the ISO 639-3 gives to retired language codes which get a special
status when they enter retirement. see:
https://github.com/glottolog/glottolog/issues/243

all the best,
- Hugh

On Sat, Nov 28, 2020 at 12:44 AM Sebastian Drude <[email-masked]> wrote:

> > Here's one possible way the structure of ISO TR 21636 could be made to
> fit into a BCP 47 extension. This shows the type of groundwork that should,
> and really must, be done up front if it is a major goal to make 21636 fit
> into BCP 47, without requiring all or even most of the values or code
> elements to be known.
> >
> > As a partial reference, I'm using a document called "NP-21636," provided
> by Sebastian and dated 2019-09-30. It's up to Sebastian whether I may
> redistribute that document to anyone.
> >
> > First, an extension singleton must be chosen. Here I suggest 'v' for
> "varieties," as the title of the TR is "Identification and description of
> language varieties." The alphabetically consecutive nature of the three
> extensions, 't' and 'u' and now 'v', may or may not be unfortunate.
> >
> > Next, the TR identifies eight "dimensions" which are independent of each
> other to a greater or lesser extent. It is clearly desirable to be able to
> specify more than one dimension in a single tag, up to all eight, but not
> necessarily so. For this I will suggest a key-value mechanism like that
> devised for the 'u' extension, so that each dimension is indicated by a
> two-character key, derived from the names Sebastian provided on the 24th:
> >
> > sp    Space
> > ti    Time
> > so    Social group
> > me    Medium
> > si    Situation
> > pe    Person
> > pr    Proficiency
> > (The NP-21636 document used the term "performance" instead of
> "communicative functioning," which would have caused a minor inconvenience
> by overloading 'pe'.)
> >
> > Then there would be "value" identifiers of 3 to 8 characters each,
> providing values for each of the dimensions. The lengths of each of these
> subtags, 1 for the extension singleton and 2 for the dimensions and 3 to 8
> for the values, are a critical part of BCP 47 syntax, permitting humans and
> processes to parse the tag and figure out what modifies what.
> >
> > A finite number of values for each dimension would need to be identified
> so they could be encoded, keeping uniqueness and syntactical constraints in
> mind. This is the process of building a code list, or registry. The set of
> values would be extensible; no one and no group could be expected to get
> this perfect the first time. But stability rules would also come into play
> here.
> >
> > Note that the values might not have to be unique across dimensions; you
> could have "en-v-so-foo-pr-foo" if the string 'foo' represented a social
> group value and coincidentally also a proficiency value. Too much of this
> could lead to human confusion, though.
> >
> > I have no idea what to do about things like "certainty status," but some
> syntax along these lines could be devised. If this is an "adjective"
> modifying dimensional values, rather than a value itself, that should be
> syntactically clear.
> >
> > The dimension that troubles me the most, of course, is "person." The
> document makes it clear in several passages that Sebastian has his own
> personal variety (times the number of languages in which he can
> communicate), I have my own, Michael and John and Leon and Peter have their
> own, and so forth. Obviously neither we at ietf-languages, nor anyone else,
> are going to encode identifiers for each of 7.7 billion living people, plus
> those deceased, plus those not yet born. It may be that the only possible
> way to encode this dimension will be with a private-use subtag at (by
> definition) the end of the tag.
> >
> _______________________________________________
> Ietf-languages mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-languages
>

---

From: Sebastian Drude <[email-masked]>
Date: Fri, 27 Nov 2020 23:22:06 -0300

Thanks again, Hugh.

The interaction with or building on Glottolog would certainly have to be 
thought through thoroughly, and I have not even started to do so.  But 
they have a good list of major language varieties, especially if these 
have already received some kind of description.

More comments below.

Best,

Sebastian

-- 

On 27/11/2020 21:54, Hugh Paterson III wrote:
> Sebastian,
>
> This is all very interesting, I am particularly interested by the 
> comment related to the Glottolog.
> 1. You refer to their "list of dialects" but are you certain that some 
> of these sub-varieties are not actually socio-lects, and therefore 
> should not be placed in the space dimension? It would be problematic 
> if one were to assume that the Glottolog's list of ways to 
> subcategorize languages were all within one dimension.
Very true.  One more resource that was started out without paying 
attention to the different dimensions of inner-language linguistic 
variation.

> Having said that, my understanding, having talked with one of their 
> editors is that they don't create their own data but rather they 
> aggregate other people's data.
Indeed, but they (mainly Harald Hammarström) review the literature 
carefully and come to an informed conclusion, for instance regarding the 
genealogical trees.

> But perhaps the editors do have first hand or documentary evidence for 
> their sub-categorization as you seem to suggest.
No, I did not intend to suggest that.  I know they work with many 
specialists for local/regional language families and groups to ensure 
maximum accuracy, and being done by the same circle of few people, the 
whole work has much consistency in the criteria used for distinguishing 
languages, dialects etc.

> 2. My understanding is that people can already use glottocodes (even 
> now) with BCP47 with the -x- extension in the private use area.
Sure, but is it not true that in that area you can do anything? However, 
differently from when BCP 47 was created, Glottolog now is an impressive 
and very complete and accurate work.  Many scholars combine it, or even 
prefer it over ISO 639, it is used in WIkipedia, etc., so perhaps it is 
the time for rethinking the relationship between BCP and Glottolog, 
especially for the cases of languages that are missing in the Ethnologue 
/ ISO 639.

> understanding from him was that the seed data for the Glottolog came 
> from the Ethnologue directly and indirectly via Multi-tree which the 
> indirect method included the sub-categorization. So, maybe it would be 
> equally good to consult that reference work.
The Ethnologue and ISO 639 are twins maintained by SIL International.  
Multi-Tree is exactly that: MULTI.  So, for many major language 
families, there are a large number of trees, compiled from decades of 
scholarly work, and unfortuantely it is not ensured at all that the same 
terms for languages, dialects and (sub)families refer to the same 
entities in the different trees, so that is quite a mess to start with.  
Good for historical comparative linguists who want to compare 
hypotheses, but not very good for standardization. True, they have a 
"composite" or compromise tree for many or all families, but in my 
experience it is either very close to the Ethnologue, or nevertheless 
inferior in quality to the Glottolog. So I believe, in particular for 
language classification (which is not at all relevant for BCP 47, as far 
as I understand), Glottolog is the best resource.

> 4. Please correct me if I am wrong, but my impression after having 
> read some of the Glottolog's about pages was that they do occasionally 
> obliviate their ID codes that they use, which is a different process 
> than the status that the ISO 639-3 gives to retired language codes 
> which get a special status when they enter retirement. see: 

Glottolog (and even the Glottocodes) was not made for standardization or 
tagging, but then, that was true for the Ethnologue 20 years ago, too.  
I understand Glottolog also forbids recycling of Glottocodes, so 
somewhere there must be a list of once valid 
retired/withdrawn/deprecated codes.  Perhaps we can come to agreements 
with them.  It would indeed be good if Glottolog also recognized the 
multi-dimensionality of inner-language linguistic variation.

>
> all the best,
> - Hugh
>
> On Sat, Nov 28, 2020 at 12:44 AM Sebastian Drude <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Dear Doug,
>
>     [email-masked] <mailto:[email-masked]> ▪ +55 (91) 3217 6024 ▪
>     +55 (91) 983733319
>     > Here's one possible way the structure of ISO TR 21636 could be
>     made to fit into a BCP 47 extension. This shows the type of
>     groundwork that should, and really must, be done up front if it is
>     a major goal to make 21636 fit into BCP 47, without requiring all
>     or even most of the values or code elements to be known.
>     >
>     > As a partial reference, I'm using a document called "NP-21636,"
>     provided by Sebastian and dated 2019-09-30. It's up to Sebastian
>     whether I may redistribute that document to anyone.
>     >
>     > First, an extension singleton must be chosen. Here I suggest 'v'
>     for "varieties," as the title of the TR is "Identification and
>     description of language varieties." The alphabetically consecutive
>     nature of the three extensions, 't' and 'u' and now 'v', may or
>     may not be unfortunate.
>     >
>     > Next, the TR identifies eight "dimensions" which are independent
>     of each other to a greater or lesser extent. It is clearly
>     desirable to be able to specify more than one dimension in a
>     single tag, up to all eight, but not necessarily so. For this I
>     will suggest a key-value mechanism like that devised for the 'u'
>     extension, so that each dimension is indicated by a two-character
>     key, derived from the names Sebastian provided on the 24th:
>     >
>     > sp    Space
>     > ti    Time
>     > so    Social group
>     > me    Medium
>     > si    Situation
>     > pe    Person
>     > pr    Proficiency
>     "communicative functioning," which would have caused a minor
>     inconvenience by overloading 'pe'.)
>     >
>     > Then there would be "value" identifiers of 3 to 8 characters
>     each, providing values for each of the dimensions. The lengths of
>     each of these subtags, 1 for the extension singleton and 2 for the
>     dimensions and 3 to 8 for the values, are a critical part of BCP
>     47 syntax, permitting humans and processes to parse the tag and
>     figure out what modifies what.
>     >
>     > A finite number of values for each dimension would need to be
>     identified so they could be encoded, keeping uniqueness and
>     syntactical constraints in mind. This is the process of building a
>     code list, or registry. The set of values would be extensible; no
>     one and no group could be expected to get this perfect the first
>     time. But stability rules would also come into play here.
>     >
>     > Note that the values might not have to be unique across
>     dimensions; you could have "en-v-so-foo-pr-foo" if the string
>     'foo' represented a social group value and coincidentally also a
>     proficiency value. Too much of this could lead to human confusion,
>     though.
>     >
>     > I have no idea what to do about things like "certainty status,"
>     but some syntax along these lines could be devised. If this is an
>     "adjective" modifying dimensional values, rather than a value
>     itself, that should be syntactically clear.
>     >
>     > The dimension that troubles me the most, of course, is "person."
>     The document makes it clear in several passages that Sebastian has
>     his own personal variety (times the number of languages in which
>     he can communicate), I have my own, Michael and John and Leon and
>     Peter have their own, and so forth. Obviously neither we at
>     ietf-languages, nor anyone else, are going to encode identifiers
>     for each of 7.7 billion living people, plus those deceased, plus
>     those not yet born. It may be that the only possible way to encode
>     this dimension will be with a private-use subtag at (by
>     <http://ewellic.org>
>     [email-masked] <mailto:[email-masked]>

---

From: Peter Constable <[email-masked]>
Date: Sat, 28 Nov 2020 03:04:57 +0000

From: Ietf-languages <[email-masked]> On Behalf Of Doug Ewell
Sent: Friday, November 27, 2020 3:01 PM

>The dimension that troubles me the most, of course, is "person." ... Obviously neither we at ietf-languages, nor anyone else, are going to encode identifiers for each of 7.7 billion living people, plus those deceased, plus those not yet born. 

In case anyone's wondering, using subtags of length 3 to 8, up to 217.18 billion individuals could be accommodated.

Peter