Subject: Re: [Ltru] ISO 639-6 (was: Geocoordinates)

From: "Doug Ewell" <[email-masked]>
Date: Tue, 10 Mar 2009 22:10:07 -0600

John Cowan <cowan at ccil dot org> wrote:

> (a) codes for language collections that aren't in 639-5, which could 
> be added to BCP 47 as 4-letter primary language subtags (that size is 
> currently reserved);

*If* we are to support these code elements, I think this is the right 
place to do so.  The 4-letter language subtag was reserved in BCP 47 for 
exactly this purpose, and if anyone, such as (ahem) Peter, sees a 
conflict with this due to their "derivative use of the language tag 
syntax," I'm sorry to say it, but that is a problem of their own making.

> (b) codes for variants other than script and national variants, which 
> could be added to BCP 47 as 5-character variant subtags by adding a 
> 5th character (I propose prefixing '6').

(and later)

> I wasn't proposing a singleton, but a convention: variant code abcd 
> would come into BCP 47 as variant subtag 6abcd, to avoid confusing it 
> with script subtags.

I continue to be strongly opposed to adding hundreds or even thousands 
of variant subtags, using the same mechanism we currently use to 
register variants one by one.  The boonts and fonipas and njivas would 
be completely overwhelmed by the oceans of 6xxxx subtags.

If these don't qualify as language subtags, for whatever reason, then 
they should be encoded using the extension mechanism described in 
Section 3.7 of RFC 4646, preferably with an appropriate singleton such 
as '6'.  This would be orders of magnitude more appropriate than 
variants.

--
Doug Ewell  *  Thornton, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

---

From: Peter Constable <[email-masked]>
Date: Wed, 11 Mar 2009 08:07:16 -0700

From: [email-masked] [mailto:[email-masked]] On Behalf Of Doug Ewell

> syntax," I'm sorry to say it, but that is a problem of their own making.

Just because we reserved something for some future possibility -- a decision taken without any consideration whatsoever of the merits of that possibility -- that doesn't mean that that possibility does have merits. For my part, I have always doubted the merits. On that basis, and assuming that nothing was likely to come of that possibility, I chose to pursue a derivative use in a particular application context. If it turns out that a future version of BCP47 does make use of those currently-reserved alpha-4 subtags, then indeed the problem will be mine (and I'm sure I'll survive). But I can tell you up front that I will most likely push back on a revision to make use of the reserved alpha-4 subtags not only because it's inconvenient for my derivative application but also because I remain very skeptical that there will be benefits (let alone ROI, considering the costs).

> register variants one by one.

Do you mean, "opposed to adding ... thousands of variant subtags _in mass_, and prefer to continue ... register[ing] variants one by one"?

If so, I agree.

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 11 Mar 2009 11:26:05 -0400

> be completely overwhelmed by the oceans of 6xxxx subtags.

And a Good Thing Too.  Boontling is a variety of English, and ought
to have a 639-6 code element.  Fonipa is a kludge to compensate for a
limitation in ISO 15924.  With 639-6 we don't need to register variants,
we make people go to 639-6/RA instead.

> as '6'.  This would be orders of magnitude more appropriate than 

Most of the 639-6 codes correspond semantically to variant subtags.
Consider 'yoli' (or '6yoli' under my proposal): it's the Yo-Li variety of
the Xinan dialect of Mandarin.  We know this because of the 639-6 parent
data, which shows the following chain:  yoli > xuuk (Xinan dialect,
spoken) > xghu (Xinan dialect) > cmn (Mandarin proper) > micr (Mandarin
cluster) > mycx (Mandarin-Yue complex) > zho (Chinese) > ssic (Sinitic)
> snot (Sino-Tibetan).

This is not, of course, the One True Tree; what's more, it's preliminary
data and subject to change.  But just the fact that 'yoli' is subordinate
to 'cmn', a known individual language code (from the 639-3 data), and
that it isn't cross-referenced with a 3166 or 15924 code, tells us this
is a variant.  The code 'biar' (Bai cluster) > ssic > snot, on the other
hand, is not subordinate to any 639-3 individual language and therefore
is a collection.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 11 Mar 2009 19:02:26 -0600

Peter Constable <petercon at microsoft dot com> wrote:

> Just because we reserved something for some future possibility -- a 
> decision taken without any consideration whatsoever of the merits of 
> that possibility -- that doesn't mean that that possibility does have 
> merits. For my part, I have always doubted the merits. On that basis, 
> and assuming that nothing was likely to come of that possibility, I 
> chose to pursue a derivative use in a particular application context. 
> If it turns out that a future version of BCP47 does make use of those 
> currently-reserved alpha-4 subtags, then indeed the problem will be 
> mine (and I'm sure I'll survive). But I can tell you up front that I 
> will most likely push back on a revision to make use of the reserved 
> alpha-4 subtags not only because it's inconvenient for my derivative 
> application but also because I remain very skeptical that there will 
> be benefits (let alone ROI, considering the costs).

We're talking about two separate issues here:

1. Whether the RFC 4646 decision to reserve 4-letter language subtags 
"for future use," widely assumed to be ISO 639-6, was a good one, and 
specifically whether ISO 639-6-based subtags would add sufficient value 
to BCP 47.

2. Whether a future decision on creating 4-letter language subtags, for 
whatever reason, should be influenced by the existence of one or more 
derivative applications that are non-conformant to BCP 47.

Regarding issue 1, I'm reserving judgment until all of the 639-6 data 
becomes publicly available, and until LTRU and/or ietf-languages have 
had a chance to look at it and start discussing it.  I also agree with 
Martin that some working experience with the 7,500 language subtags 
available in the 4646bis era would be helpful.  As you can tell, I do 
have opinions about how they should be added *if* they are to be added.

Regarding issue 2, I'm very much against letting this be a contributing 
factor.

>> I continue to be strongly opposed to adding hundreds or even 
>> thousands of variant subtags, using the same mechanism we currently 
>> use to register variants one by one.
>
> Do you mean, "opposed to adding ... thousands of variant subtags _in 
> mass_, and prefer to continue ... register[ing] variants one by one"?

I mean I'm opposed to adding thousands of variant subtags, either in 
mass *or* one by one, using a formulaic process such as prepending '6' 
to thousands of ISO 639-6 code elements.  Variants are supposed to be 
proposed, discussed, and approved (or rejected) on a case-by-case basis. 
They are not supposed to be a means of shoehorning in another external 
standard.  That is what the other two solutions -- 4-letter language 
subtags and extensions -- are for.

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 11 Mar 2009 19:15:38 -0600

John Cowan <cowan at ccil dot org> wrote:

>> ... The boonts and fonipas and njivas would be completely overwhelmed 
>> by the oceans of 6xxxx subtags.
>
> And a Good Thing Too.  Boontling is a variety of English, and ought to 
> have a 639-6 code element.  Fonipa is a kludge to compensate for a 
> limitation in ISO 15924.  With 639-6 we don't need to register 
> variants, we make people go to 639-6/RA instead.

Then if you like, view the "legacy" variants as warts on the side of a 
collection of '6' variants.  Doesn't matter; the warts are not going 
away.

Despite its size and ambitious goals, and the expertise of those 
preparing it, I would be very surprised if 639-6 encompassed a proper 
superset of everything that has ever been (a) registered in the Registry 
or (b) taken seriously but not registered in the Registry, going back to 
the whole tags registered under RFC 1766/3066.  That is one of the major 
problems with claiming that 639-6 will enable ietf-languages to be shut 
down.

The other, of course, is that we have 14 years of history with a 
Reviewer and a discussion list that offers him advice, and RFC 4646 and 
draft-4646bis have expanded their role to include vetting of new and 
changed subtags stemming not only from user requests, but from changes 
to the core ISO and UN standards.  We have six such requests pending 
right now.

---

From: Mark Davis <[email-masked]>
Date: Wed, 11 Mar 2009 18:17:39 -0700

I agree with Doug that we'd want to tread very carefully with regards to
adding ISO 639-6. The rollout over a couple of years of the changes that we
are currently making will teach us a lot.

Mark

On Wed, Mar 11, 2009 at 18:02, Doug Ewell <[email-masked]> wrote:

>> decision taken without any consideration whatsoever of the merits of that
>> possibility -- that doesn't mean that that possibility does have merits. For
>> my part, I have always doubted the merits. On that basis, and assuming that
>> nothing was likely to come of that possibility, I chose to pursue a
>> derivative use in a particular application context. If it turns out that a
>> future version of BCP47 does make use of those currently-reserved alpha-4
>> subtags, then indeed the problem will be mine (and I'm sure I'll survive).
>> But I can tell you up front that I will most likely push back on a revision
>> to make use of the reserved alpha-4 subtags not only because it's
>> inconvenient for my derivative application but also because I remain very
>> skeptical that there will be benefits (let alone ROI, considering the
>> costs).
>>
>
> 1. Whether the RFC 4646 decision to reserve 4-letter language subtags "for
> future use," widely assumed to be ISO 639-6, was a good one, and
> specifically whether ISO 639-6-based subtags would add sufficient value to
> BCP 47.
>
> becomes publicly available, and until LTRU and/or ietf-languages have had a
> chance to look at it and start discussing it.  I also agree with Martin that
> some working experience with the 7,500 language subtags available in the
> 4646bis era would be helpful.  As you can tell, I do have opinions about how
> they should be added *if* they are to be added.
>
>  I continue to be strongly opposed to adding hundreds or even thousands of
>>> variant subtags, using the same mechanism we currently use to register
>>> variants one by one.
>>>
>>
> I mean I'm opposed to adding thousands of variant subtags, either in mass
> *or* one by one, using a formulaic process such as prepending '6' to
> thousands of ISO 639-6 code elements.  Variants are supposed to be proposed,
> discussed, and approved (or rejected) on a case-by-case basis. They are not
> supposed to be a means of shoehorning in another external standard.  That is
> what the other two solutions -- 4-letter language subtags and extensions --
> are for.
>
>  If so, I agree.
>>
>
> --
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru
>

---

From: Peter Constable <[email-masked]>
Date: Wed, 11 Mar 2009 21:55:16 -0700

From: Doug Ewell [mailto:[email-masked]] 

> proposed, discussed, and approved (or rejected) on a case-by-case basis. 

I can certainly go along with that last statement.

Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Mar 2009 17:53:19 -0400

> collection of '6' variants.  Doesn't matter; the warts are not going 

True, but then the irregular tags aren't going away either: nonetheless,
most of them can be easily deprecated.

> problems with claiming that 639-6 will enable ietf-languages to be shut 

Well, it may not contain them now, but if something is a language variant,
it ought to have a 639-6 tag, because 639-6 is deliberately unselective.

-- 
MEET US AT POINT ORANGE AT MIDNIGHT BRING YOUR DUCK OR PREPARE TO FACE WUGGUMS
John Cowan      [email-masked]      http://www.ccil.org/~cowan

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 13 Mar 2009 23:39:10 -0700

Hi -

As co-chair...

I don't see much progress in this discussion.  I suggest that those
wishing to see work in ltru on the use of ISO 639-6 as a source of
subtags should submit a specific charter update proposal to this
mailing list for discussion.  Please refrain from further comment
on this topic until someone provides a specific (i.e. containing dates
and delivarables) charter update proposal.

Randy

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Mar 2009 22:10:51 -0700

Due to off-line mail I received in relation to some of my comments in this thread, I feel I need to make one additional comment:

The opinions I have expressed are my own and not those of my employer or any other party with whom I may be affiliated.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Randy Presuhn
Sent: Friday, March 13, 2009 11:39 PM
To: LTRU Working Group
Subject: Re: [Ltru] ISO 639-6

Hi -

As co-chair...

Randy

---

From: Gerard Meijssen <[email-masked]>
Date: Tue, 17 Mar 2009 09:16:48 +0100

Hoi,
I found the reply that Peter made in his reply on the thread "Re: [Ltru] ISO
639-6 (was: Geocoordinates)" problematic. In it he said: "But I can tell you
up front that I will most likely push back on a revision to make use of the
reserved alpha-4 subtags not only because it's inconvenient for my
derivative application but also because I remain very skeptical that there
will be benefits (let alone ROI, considering the costs).". I find this
problematic because it puts commercial interests clearly ahead of a
structured approach that comes with the ISO-639-6.

Now that Peter states that he is expressing his own opinions and not the
opinion of Microsoft, I find the statement quoted above unacceptable. It is
unacceptable because he is spouting FUD on a standard that intends to
reconcile the different standards in one body, a standard that intends to
finally give a place to the different orthographies. When Peter speaks for
Microsoft about ROI and costs, I find it problematic because I wonder how he
reconciles this with being a member of the community, when Peter "only"
expresses his own opinion it becomes unacceptable to me because he clearly
presents himself as a Microsoft employee. ROI and costs are business
related  considerations and making these "personal opinion" is disingenuous
probably dishonest and in my opinion unacceptable.

There are clear benefits to a standard that reconciles the current mess of
ISO-639 standards that are problematic to reconcile and have made the
current RFC process a waiting game. At this time there is no software that
provides the "most basic" language support. The most basic language support
is recognising that people want to write in their own language and enabling
them to do so. Enabling this is nothing more then setting the appropriate
meta data and allowing people to express themselves. The best argument
against providing a "most basic" language support has been the standards do
not support this and with "personal opinions" expressed in this way it is
less likely when we will see the day that people can express themselves in
their language.
Thanks,
        Gerard

2009/3/17 Peter Constable <[email-masked]>

> Due to off-line mail I received in relation to some of my comments in this
> thread, I feel I need to make one additional comment:
>
> The opinions I have expressed are my own and not those of my employer or
> any other party with whom I may be affiliated.
>
>
> Peter
>

---

From: Peter Constable <[email-masked]>
Date: Tue, 17 Mar 2009 05:18:56 -0700

I’ll provide this reply and then drop the discussion.

Gerard appears to suppose that, since I refer to “ROI”, I must be speaking on behalf of Microsoft. That is simply not the case. I was speaking as a language technologist expressing my thoughts on what benefits, in practical terms, I think would accrue to language communities around the world. Benefits are not realized simply by having a tagging system that comes with tens of thousands of already-sanctioned elements. Rather, benefits are realized by having a tagging system that accommodates their needs *and* that works in practice and is widely supported. I am strongly influenced by those practical considerations from time I spent in a not-for-profit organization, prior to working for Microsoft, trying to find technology solutions for thousands of language workers working in divers language communities throughout the world, with sizeable investments made in solutions that were *not* widely adopted in industry. Such solutions did not scale and fostered dependence on a small support infrastructure rather than giving long-term empowerment.

In fact, the opinions I was expressing were in line with an opinion I formed some years before I started to work at Microsoft: that sub-language variations are not appropriate objects for a comprehensive encoding because there’s no clear way to define dialects. I first discussed this in 2000, but the first time I published may have been in a paper I presented to a Unicode conference in 2002. The full paper is available here:

http://www.sil.org/silewp/2002/SILEWP2002-003.pdf

See section 4.6, p. 29f:

“It was
claimed earlier that it is possible to enumerate a comprehensive list of individual languages based
on a particular operational definition—to “tile the plane” of languages. In contrast, this is, in
principle, impossible for dialects. The reason has to do with operational definitions….”

I simply hold the same views that I had then.

I’m completely in favour of doing what is good for goodness sake. That is what has motivated my work over the past many years in character encoding and to develop ISO 639-3 and get it incorporated into BCP 47 – I think those have been steps toward goodness from where we were before. (Gerard may think ISO 639 is a mess, but IMO it’s less of a mess and a more well-defined one than it used to be.)

I remain unconvinced that ISO 639-6 will, on the balance, provide overall goodness to BCP 47. Even so, I haven’t done anything to stand in the way of ISO 639-6 as an ISO standard, and I wouldn’t necessarily oppose a revision to BCP 47 to allow ISO 639-6 subtags – depending on how it is handled. All that I said in this thread I would not support is putting these in the reserved alpha-4 subtags. I have various reasons for that, but I’ll refrain from going further in respect of Randy’s request.

Peter

From: Gerard Meijssen [mailto:[email-masked]]
Sent: Tuesday, March 17, 2009 1:17 AM
To: Peter Constable
Cc: LTRU Working Group
Subject: Re: [Ltru] ISO 639-6

Hoi,
I found the reply that Peter made in his reply on the thread "Re: [Ltru] ISO 639-6 (was: Geocoordinates)" problematic. In it he said: "But I can tell you up front that I will most likely push back on a revision to make use of the reserved alpha-4 subtags not only because it's inconvenient for my derivative application but also because I remain very skeptical that there will be benefits (let alone ROI, considering the costs).". I find this problematic because it puts commercial interests clearly ahead of a structured approach that comes with the ISO-639-6.

Now that Peter states that he is expressing his own opinions and not the opinion of Microsoft, I find the statement quoted above unacceptable. It is unacceptable because he is spouting FUD on a standard that intends to reconcile the different standards in one body, a standard that intends to finally give a place to the different orthographies. When Peter speaks for Microsoft about ROI and costs, I find it problematic because I wonder how he reconciles this with being a member of the community, when Peter "only" expresses his own opinion it becomes unacceptable to me because he clearly presents himself as a Microsoft employee. ROI and costs are business related  considerations and making these "personal opinion" is disingenuous probably dishonest and in my opinion unacceptable.

There are clear benefits to a standard that reconciles the current mess of ISO-639 standards that are problematic to reconcile and have made the current RFC process a waiting game. At this time there is no software that provides the "most basic" language support. The most basic language support is recognising that people want to write in their own language and enabling them to do so. Enabling this is nothing more then setting the appropriate meta data and allowing people to express themselves. The best argument against providing a "most basic" language support has been the standards do not support this and with "personal opinions" expressed in this way it is less likely when we will see the day that people can express themselves in their language.
Thanks,
        Gerard
2009/3/17 Peter Constable <[email-masked]<mailto:[email-masked]>>
Due to off-line mail I received in relation to some of my comments in this thread, I feel I need to make one additional comment:

Peter

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Randy Presuhn
Sent: Friday, March 13, 2009 11:39 PM
To: LTRU Working Group
Subject: Re: [Ltru] ISO 639-6

Hi -

As co-chair...

Randy

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 17 Mar 2009 12:08:46 -0700

Hi Gerard -

As Co-chair:

> From: "Gerard Meijssen" <[email-masked]>
> To: "Peter Constable" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, March 17, 2009 1:16 AM
...
> It is unacceptable because he is spouting FUD
...
> ROI and costs are business

We *all* participate in these discussions as individuals.
Accusations like these are highly inappropriate on an IETF
mailing list.  Stop it.  NOW.

Randy