Subject: [Ltru] Abolishing registered language subtags (was: Re: 3 extlang to 1 extlang?)

From: "Doug Ewell" <[email-masked]>
Date: Sun, 23 Sep 2007 23:03:34 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> While we're at it I'd like to get rid of another "anomaly" in the 
> ABNF, the "registered language subtag" 5*8ALPHA.

I don't agree with this.  It's our only chance to register a language, 
most likely a constructed one, that doesn't meet the criteria of ISO 639 
JAC but which qualifies as a language under the operational guidelines 
of ietf-languages.

> Three reasons:
>
> 1 - Length, at the moment we have 3+3*(1+3)=15 as worst case for a 
> language subtag with three extlangs.  My calculations claimed that we 
> could reduce this by 8 to 3+(1+3)=7.  Actually that's incorrect, we'd 
> arrive at 8 instead of 7 for the 8ALPHA case.

This hardly seems like a compelling justification to remove a 
potentially useful option.

> 2 - Complexity, there are long explanations in RFC 4646 why this 
> "registered language subtag" construct SHOULD NOT be used, it is in 
> essence an emergency exit if ISO 639-2 refuses to accept a proposed 
> language code following their rules.  IMHO we don't need this anymore 
> for 4646bis, because ISO 639-3 will deal with any proposed language 
> passing a giggle test.  And where that's a matter of timing 4646bis 
> offers variants to fill the gaps.

http://en.wikibooks.org/wiki/Unilingua

"Giggle test" is in the eye of the beholder, and clearly not everything 
is a variant.

> 3 - Identification of isolated subtags, quoting the "known issues" 
> paragraph of <http://purl.net/xyzzy/home/ltru/ltru2xml.awk>:
> |                If the language subtag review list introduces a
> # language subtag with 5-8 characters which clashes with a variant
> # subtag these subtags would get the same xml:id resulting in an
> # XML syntax error.  In practice it's unlikely that there ever
> # will be any IANA language subtag not derived from ISO 936, let
> # alone using the same string also for a variant subtag.

A 5-letter language subtag that is the same as a 5-letter variant subtag 
is not an error and is not a problem, any more than it is an error or 
problem to have a 2-letter language subtag 'af' meaning "Afrikaans" and 
a 2-letter region subtag 'AF' meaning "Afghanistan."

> Of course I could also fix this script by using different IDs for 
> language tags and the corresponding IDrefs in prefixes.  But that 
> would be rather complex, because a Macrolanguage field references 
> either a language or an extlang.

There's no consensus yet that we will have a macrolanguage field.  In 
any case, this is a question of conflict within one implementation 
written against a draft, not a question of ambiguity or confusion within 
the draft.

> Likely I'd end up with tons of dead code relevant only for 5*8ALPHA 
> "registered language subtags", never used, never tested, and muddying 
> the water.

Tons?  Anyway, this seems like a better description of extensions than 
of registered language subtags.  Extensions have become one of those 
awful "deprecated at birth" inventions; it is clear there will never be 
a registered extension as long as some ietf-languages contributors have 
anything to say about it.

> Another approach could be a decree that any given 5*8ALPHA string MUST 
> NOT be used to denote a variant and a language in the LSTR, but then 
> we'd get even more "dead prose" in 4646bis about this construct.

See above.  An implementation that has a problem with registered 
language 'foobly' and variant 'foobly' also has a problem with language 
'af' and region 'AF', since all subtags are treated as case-insensitive.

--
Doug Ewell * Fullerton, California, USA * RFC 4645 * UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Mon, 24 Sep 2007 18:14:58 +0900

As a technical contributor, I agree with Doug.
I don't think theoretical maximal length is that important,
in particular if the main target seems to be RFC 2231,
which I haven't seen implemented (let along used) anywhere.
(If somebody has, I'd be glad to see some pointers.)

Regards,     Martin.

At 15:03 07/09/24, Doug Ewell wrote:
>> While we're at it I'd like to get rid of another "anomaly" in the ABNF, the "registered language subtag" 5*8ALPHA.
>
>I don't agree with this.  It's our only chance to register a language, most likely a constructed one, that doesn't meet the criteria of ISO 639 JAC but which qualifies as a language under the operational guidelines of ietf-languages.
>
>> Three reasons:
>>
>> 1 - Length, at the moment we have 3+3*(1+3)=15 as worst case for a language subtag with three extlangs.  My calculations claimed that we could reduce this by 8 to 3+(1+3)=7.  Actually that's incorrect, we'd arrive at 8 instead of 7 for the 8ALPHA case.
>
>This hardly seems like a compelling justification to remove a potentially useful option.
>
>> 2 - Complexity, there are long explanations in RFC 4646 why this "registered language subtag" construct SHOULD NOT be used, it is in essence an emergency exit if ISO 639-2 refuses to accept a proposed language code following their rules.  IMHO we don't need this anymore for 4646bis, because ISO 639-3 will deal with any proposed language passing a giggle test.  And where that's a matter of timing 4646bis offers variants to fill the gaps.
>
>"Giggle test" is in the eye of the beholder, and clearly not everything is a variant.
>
>> 3 - Identification of isolated subtags, quoting the "known issues" paragraph of <http://purl.net/xyzzy/home/ltru/ltru2xml.awk>:
>A 5-letter language subtag that is the same as a 5-letter variant subtag is not an error and is not a problem, any more than it is an error or problem to have a 2-letter language subtag 'af' meaning "Afrikaans" and a 2-letter region subtag 'AF' meaning "Afghanistan."
>
>> Of course I could also fix this script by using different IDs for language tags and the corresponding IDrefs in prefixes.  But that would be rather complex, because a Macrolanguage field references either a language or an extlang.
>
>There's no consensus yet that we will have a macrolanguage field.  In any case, this is a question of conflict within one implementation written against a draft, not a question of ambiguity or confusion within the draft.
>
>> Likely I'd end up with tons of dead code relevant only for 5*8ALPHA "registered language subtags", never used, never tested, and muddying the water.
>
>Tons?  Anyway, this seems like a better description of extensions than of registered language subtags.  Extensions have become one of those awful "deprecated at birth" inventions; it is clear there will never be a registered extension as long as some ietf-languages contributors have anything to say about it.
>
>> Another approach could be a decree that any given 5*8ALPHA string MUST NOT be used to denote a variant and a language in the LSTR, but then we'd get even more "dead prose" in 4646bis about this construct.
>
>See above.  An implementation that has a problem with registered language 'foobly' and variant 'foobly' also has a problem with language 'af' and region 'AF', since all subtags are treated as case-insensitive.
>
>--

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 24 Sep 2007 17:58:00 +0200

Doug Ewell wrote:

  [Length]
>> we'd arrive at 8 instead of 7 for the 8ALPHA case.
 
> remove a potentially useful option.

Of course that alone isn't compelling to junk 5*8ALPHA.
We have to note this case for the worst case deduction
in chapter 4.3.1.
  
> "Giggle test" is in the eye of the beholder

Sure, and I propose that the ISO 639-3 folks can decide
this.  If I seriously need say "liaden" I could simply
use x-liaden or maybe art-x-liaden.

> A 5-letter language subtag that is the same as a 5-letter
> variant subtag is not an error and is not a problem

Well, I've just shown that it in fact *IS* a problem for 
some implementors if they look at subtags in isolation:

For most cases it works:
- 3DIGIT is always an UN M.49 region (if registered)
- 4DIGIT is always a variant if registered
- 4ALPHA is always a script in 4646 and 4646bis,
  4ALPHA as a language isn't used in 4646(bis).
- 5*8alphanum is _in practice_ always a variant.
- 3ALPHA is always a language in 4646, starting
  with 4646bis it can be also an extlang.

> any more than it is an error or problem to have a 
> 2-letter language subtag 'af' meaning "Afrikaans" and 

Oh yes, this *IS* a problem, but it's relatively simple
to solve it:  

a) For an "XML-converter" needing IDREFS an xml:id is 
   case sensitive, therefore af and AF can be used as
   is simultaneously.

   In other words the converter can pick any referenced
   subtag or prefix, replace "-" by space, add "_" to 
   subtags starting with a digit, and be done with it.

   There are only two exceptions in the 4646 registry,
   yi-latn and zh-hakka, and both will be fixed in the
   4645bis registry.

b) For a script working with "real" (cases insensitive)
   tags it's also simple, your af vs. AF and similar
   cases are the only difficulty.  The script can note
   the position of the subtag, 2*8ALPHA languages are
   always at the begin of prefix tags.  Other subtags
   including 2ALPHA can be noted as e.g. -AF, and then
   "AF" != "-AF".  

   Scripts woking with the registry as their input can
   use a similar case insensitive logic, I've done this
   in REXX before I started the awk-XML converter.

But as soon as 4*8ALPHA language subtags are used, and
simultaneously a 3ALPHA Macrolanguage can be either a
language or an extlang, these simple solutions won't
work anymore.

> There's no consensus yet that we will have a 
> macrolanguage field.

It's in the last 4645bis proto-registry, from my POV
it's fine.  It would break my REXX script using the
"AF" != "-AF" trick, for a macrolanguage "xxx" I can't
tell if that's a reference to a language "XXX" or to
an extlang "-XXX".

But it's no issue for my case-sensitive awk-script as
long as all 5*8ALPHA are variants.  Or rather as long
as any given 5*8ALPHA is either language or variant,
the recipe (a) only breaks if it's both.

> In any case, this is a question of conflict within
> one implementation written against a draft, not a 
> question of ambiguity or confusion within the draft.

No, it's a design issue exhibited by "running code":

If you can decide what the type an isolated subtag is,
either by noting its position (b) or by noting it in
its case-sensitive "pretty" form (a), then that's an
important feature of the design.

Syntacticaly we'd be better off if we use i-abcdefgh
instead of abcdefgh for "registerd language subtags".

We could then create i-uniling or i-liaden without
identifcation issues for variants uniling or liaden.
[Disclamer:  That's an example, and "Liaden" might 
 be a registered trademark in some parts of Terra.]

> Extensions have become one of those awful "deprecated
> at birth" inventions; it is clear there will never be 
> a registered extension as long as some ietf-languages
> contributors have anything to say about it.

Anybody (including posters on the ietf-languages list) 
is free to write a draft proposing an extension, and
if "the IETF" likes it it will be an RFC.  But let's
please discuss the prohibition of "generic variants"
like fon?pa in another thread.

 Frank