Subject: Re: [Ltru] Test suite for language tags?

From: "Doug Ewell" <[email-masked]>
Date: Thu, 3 Aug 2006 22:51:27 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> Your best bet is probably to generate subtag sequences based on the 
> ABNF. Some particular problem cases to check would be:

Just a few minor nits here.  This is slightly more complex than meets 
the eye, unfortunately.

> - singletons in the first position (except for 'x' and the 
> grandfathered list)

Sadly, the existence of the grandfathered list means that all 
well-formed processors must also do a limited amount of validity 
checking.  I don't question the importance of maintaining support for 
the grandfathered tags, but this is a side effect.

> - misplaced extlang (3ALPHA in the third or later position following 
> any of these: 4ALPHA, 2ALPHA, 3DIGIT, 5*8alphanum, DIGIT 3alpha)[note: 
> stop at singleton]

Repalce "following" by "immediately following."  2ALPHA followed by 
3ALPHA followed by 3ALPHA is fine.

"DIGIT 3alpha" here should be "DIGIT 3alphanum".

> - misplaced script (4ALPHA following any of these: 2ALPHA, 3DIGIT, 
> 5*8alphanum, DIGIT 3alphanum)[note: stop at singleton]

Needs another qualifier.  2ALPHA followed by 4ALPHA is fine: "zh-Hans".

> - missing subtag ("--")
> - a dangling hyphen ("foo-bar-baz-") or initial hyphen 
> ("-foo-bar-baz")

The second is really just a special case of the first: a missing subtag 
at the end or beginning, respectively.  One thing I found useful, when 
building my validator, was to parse out the subtags first and check them 
for validity afterward, so the hyphens never become part of the validity 
checking per se.

> Thus, these are all errors:
>
> "a-foo"
> "abcdefghi-012345678"
> "ab-abc-abc-abc-abc"
> "ab-abcd-abc"
> "ab-ab-abc"

Why pick on Abkhasian so much?  :-)

> "ab-x-abc-x-abc" // anything goes after x

Not quite anything, of course:  1*("-" (1*8alphanum))

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Fri, 04 Aug 2006 08:19:52 -0700

> Just a few minor nits here.  This is slightly more complex than meets 

Yep.
> the grandfathered tags, but this is a side effect.

Agreed: all RFC 3066bis processors have to have the list of 
grandfathered tags baked in.
> 
>> - missing subtag ("--")
>> - a dangling hyphen ("foo-bar-baz-") or initial hyphen ("-foo-bar-baz")
> 

I did the same thing. However, one must check the hyphens. Tokenizers 
sometimes do not return "empty" tokens and can miss these cases.
> Not quite anything, of course:  1*("-" (1*8alphanum))

Yes. At least one alphanumeric subtag must follow 'x' and it cannot 
exceed eight characters in length.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 29 Aug 2006 11:22:57 +0200

On Fri, Aug 04, 2006 at 08:19:52AM -0700,
 Addison Phillips <[email-masked]> wrote 

> grandfathered tags baked in.

Does it mean that a processor should (ordinary "should" not RFC
"SHOULD") define the grammar production "grandfathered" as a list of
possible grandfathered tags, rather than the grammar rules expressed
in draft-ietf-ltru-registry-14?

I ask the question because these rules accept things like "fr-Lat"
which is not a well-formed tag. So, I was planning to replace them by
a hardwired list of GF tags.

---

From: John Cowan <[email-masked]>
Date: Tue, 29 Aug 2006 09:33:01 -0400

> in draft-ietf-ltru-registry-14?

Yes, absolutely.  MUST would be the appropriate modal verb, in fact.
The grammar rules are meant to cover the possibilities, not to serve as
a test for recognizing grandfathered tags.

At the present time there are exactly 34 grandfathered tags, of which
8 are deprecated with replacements (but must still be recognized).
Two more will probably be changed to redundant status before the adoption
of 3066ter.

At the adoption of 3066ter, six more tags will be transferred to
redundant status.  The only tags that will remain both grandfathered and
not deprecated are cel-gaulish (unless ISO 639-3 is extended to include
it), en-GB-oed, i-default, i-enochian (unless ISO 639-3 is extended to
include it), and the rather useless zh-min.  However, there will be no
harm in continuing to bake the full list of 34 into validators.

This suggests that in 3066ter we should specify the ABNF definition of
"grandfathered" explicitly as a choice of tags.

-- 
John Cowan    http://ccil.org/~cowan    [email-masked]
Mr. Henry James writes fiction as if it were a painful duty.  --Oscar Wilde

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 29 Aug 2006 07:40:07 -0700

> a hardwired list of GF tags.

To add to what John wrote:

Since you said [1] you were building a non-validating processor, this is 
all you need.  Grandfathered tags can be reclassified as redundant, if 
all the component subtags become subtags in their own right, but they 
could remain in your hard-wired list with no ill effect.

If you were building a validating processor, you would need access to 
the full Registry (possibly in a baked form) and would not need to 
hard-wire any tags or subtags.

[1] http://www1.ietf.org/mail-archive/web/ltru/current/msg05157.html

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 10:09:00 -0700

> Yes, absolutely.  MUST would be the appropriate modal verb, in fact.

No, actually not.

There are two types of grandfathered tag, actually. There are those tags 
which are "well-formed" but for which some of the subtags are not 
registered (let us call these "regular" tags). And there are those tags 
which do not follow the grammar of RFC 3066bis (let us call these 
"irregular" tags).

A well-formed processor needs only a list of the irregular tags, but it 
MUST have such a list or it will malfunction.

A validating processing, by definition, needs the complete list.

Note that the list of irregular tags is permanent and immutable.

It should be noted that Stephane's example of "fr-Lat" passing is not an 
error. A three-letter subtag in the second position is an extlang. The 
tag is not currently valid, but it is well-formed. A validating 
processor, of course, will catch this.

> a test for recognizing grandfathered tags.

Yes. And the fact is that we can only list the irregular tags in RFC, 
since the regular grandfathered tags can become redundant.
> Two more will probably be changed to redundant status before the adoption

Which ones did you have in mind here, John? I don't think I see any that 
will become redundant at present (unless you have designs on en-scouse 
or en-boont). Deprecated, certainly, but not redundant, I think.

> The only tags that will remain both grandfathered and

Indeed, it remains important to have the full list of grandfathered tags 
in validating processors. Well-formed processors will continue to need 
the full list of irregular tags.

Note that "cel-gaulish" is a regular tag: it still has potential of 
becoming redundant. Although "zh-min" is regular, the code "min" in ISO 
639-3 (at present) represents a wholly different language (Minangkabau) 
that will be a primary language subtag (and thus can't be an extlang).

> "grandfathered" explicitly as a choice of tags.

If we can dispose, somehow, of all the regular tags, then I would 
support such a scheme. If there are unresolved regular tags (whose 
stripes could change to redundant), I think it would be more of a problem.

Addison

---

From: John Cowan <[email-masked]>
Date: Tue, 29 Aug 2006 15:28:41 -0400

> Note that the list of irregular tags is permanent and immutable.

+1

> processor, of course, will catch this.

My point is that fr-Lat is not a *grandfathered* tag.  Nothing can ever
be a grandfathered tag except the 34 existing examples.  The current ABNF
for a grandfathered tag, 1*3ALPHA 1*2("-" (2*8alphanum)), therefore
recognizes too much.

> since the regular grandfathered tags can become redundant.

Listing them would do no harm.

> or en-boont). Deprecated, certainly, but not redundant, I think.

See my last two ietf-languages postings.

> that will be a primary language subtag (and thus can't be an extlang).

Correct; these are regular in form but effectively irregular (Gaulish
isn't a variant; if anything, it's a macrolanguage, but not treated
as such in 639-3.)

> stripes could change to redundant), I think it would be more of a problem.

I think there will be none:  cel-gaulish and zh-min are regular but
cannot be made redundant; the other remaining tags will be redundant,
deprecated, or both.

-- 
MEET US AT POINT ORANGE AT MIDNIGHT BRING YOUR DUCK OR PREPARE TO FACE WUGGUMS
John Cowan      [email-masked]      http://www.ccil.org/~cowan

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 12:36:34 -0700

All of the sgn-xx-yy tags (3 of these), all of the i-xxxx tags (13 of 
them), plus the Oxford tag.

Addison

Frank Ellermann wrote:
> Addison Phillips wrote:
> 
>> there are those tags which do not follow the grammar of RFC
>> 3066bis (let us call these "irregular" tags).
> 
> Actually only en-GB-oed, or did I miss other "irregular" tags ?
> 
> Frank
> 
> 
>

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 12:45:11 -0700

> for a grandfathered tag, 1*3ALPHA 1*2("-" (2*8alphanum)), therefore

Yes, that's correct. However, his example is well-formed according to 
the regular ABNF. You are entirely correct that implementing the ABNF 
for grandfathered is the wrong thing to do. Grandfathered checking of 
any stripe requires *some* list of tags.

> Listing them would do no harm.

If the list changes, that would be bad. Ideally, with a couple of 
registrations and some deprecations, the list will entirely be formed of 
tags that have no future, allowing a list to be, actually, a Good Thing 
(not just "here they are", but also "don't use these any more, even 
though they remain valid").

> isn't a variant; if anything, it's a macrolanguage, but not treated

Regular tags don't concern well-formed processors, except that they may 
mis-identify the purpose of a grandfathered subtag ("min" is NOT an 
extlang, for example).

> cannot be made redundant; the other remaining tags will be redundant,

Both cel-gaulish and zh-min could be deprecated, though, leaving only 
dead tags behind.

Addison

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 29 Aug 2006 22:00:41 +0200

On Tue, Aug 29, 2006 at 10:09:00AM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 76 lines which said:

> And there are those tags which do not follow the grammar of RFC

I may seem pedantic but even such tags follow the grammar, no? You
mention in another message the i-xxxx tags and they are described by
the grammar:

   grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum))
                   ; grandfathered registration
                   ; Note: i is the only singleton
                   ; that starts a grandfathered tag

Or do you mean, instead of "the grammar of RFC 3066bis", "the grammar
of the 'langtag' production"?

> It should be noted that Stephane's example of "fr-Lat" passing is
> not an error. A three-letter subtag in the second position is an
> extlang.

Yes, correct. Sorry.

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 13:05:30 -0700

> of the 'langtag' production"?

Yes, exactly so.
>> not an error. A three-letter subtag in the second position is an

Nothing to be sorry about. It's just not an "edge case" example. A 
better example might be "f-Latn" :-).

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 29 Aug 2006 22:02:09 +0200

On Tue, Aug 29, 2006 at 12:45:11PM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 60 lines which said:

> You are entirely correct that implementing the ABNF for
> grandfathered is the wrong thing to do.

I discovered this the hard way, when testing my parser. Shouldn't we
add a warning to implementors in RFC 3066ter?

---

From: John Cowan <[email-masked]>
Date: Tue, 29 Aug 2006 16:40:05 -0400

> though they remain valid").

Here's my conclusions about each of the 34:

Tag                     Type            Action                  Preferred value
===                     ====            ======                  ===============
art-lojban              Broken          Already deprecated      jbo
cel-gaulish             Broken          Deprecated in 3066ter
en-boont                Regular         Redundant in 3066bis
en-GB-oed               Irregular       Preserved
en-scouse               Regular         Redundant in 3066bis
i-ami                   Irregular       Deprecated in 3066ter   ami
i-bnn                   Irregular       Deprecated in 3066ter   bnn
i-default               Irregular       Preserved
i-enochian              Irregular       Deprecated in 3066ter   not known
i-hak                   Irregular       Already deprecated      zh-hakka
i-klingon               Irregular       Already deprecated      tlh
i-lux                   Irregular       Already deprecated      lb
i-mingo                 Irregular       Deprecated in 3066ter   not known
i-navajo                Irregular       Already deprecated      nv
i-pwn                   Irregular       Deprecated in 3066ter   pwn
i-tao                   Irregular       Deprecated in 3066ter   tao
i-tay                   Irregular       Deprecated in 3066ter   tal
i-tsu                   Irregular       Deprecated in 3066ter   tsu
no-bok                  Broken          Already deprecated      nb
no-nyn                  Broken          Already deprecated      nn
sgn-BE-fr               Irregular       Deprecated in 3066ter   not known
sgn-BE-nl               Irregular       Deprecated in 3066ter   not known
sgn-CH-de               Irregular       Deprecated in 3066ter   sgn-sgg
zh-cmn                  Regular         Redundant in 3066ter
zh-cmn-Hans             Regular         Redundant in 3066ter
zh-cmn-Hant             Regular         Redundant in 3066ter
zh-gan                  Regular         Redundant in 3066ter
zh-guoyu                Irregular       Already deprecated      zh-cmn
zh-hakka                Irregular       Deprecated in 3066ter   zh-hak
zh-min                  Broken          Deprecated in 3066ter
zh-min-nan              Irregular       Deprecated in 3066ter   zh-nan
zh-wuu                  Regular         Redundant in 3066ter
zh-xiang                Irregular       Deprecated in 3066ter   zh-hsn
zh-yue                  Regular         Redundant in 3066ter

A Regular tag is one that is both syntactically and semantically regular;
a Broken tag is one that is syntactically regular but semantically
irregular (it does not mean what it appears to mean); an Irregular tag
is one that is syntactically irregular.

So 8 tags will be changed to redundant in the RFC 3066ter regime; of
the remaining 24, all will be deprecated except i-default and en-GB-oed.

Possible discrepancies:

a) Nothing in 639-3 seems to correspond directly to i-mingo or i-enochian.

b) We may decide not to deprecate cel-gaulish or zh-min, which are sort
of mini-macrolanguages.

c) The split between sgn-BE-fr and sgn-BE-nl seems to be finer than
639-3 recognizes; we may need variant subtags to handle these.

-- 
As you read this, I don't want you to feel      John Cowan
sorry for me, because, I believe everyone       [email-masked]
will die someday.                               http://www.ccil.org/~cowan
        --From a Nigerian-type scam spam

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 29 Aug 2006 22:02:06 +0200

John Cowan wrote:
 
> My point is that fr-Lat is not a *grandfathered* tag.
> Nothing can ever be a grandfathered tag except the 34
> existing examples.  The current ABNF for a grandfathered
> tag, 1*3ALPHA 1*2("-" (2*8alphanum)), therefore recognizes
> too much.

Okay, now I got it.  We could create more elaborated rules
with less mismatches, something in the direction of "if it
doesn't start with art / cel / en / i / no / sgn / zh it's
certainly not grandfathered".  And that won't match fr-Lat.

Or we split it into i / sgn / zh, adding the rest as direct
enumeration (1*art, 1*cel, 3*en, 2*no = 5 exceptions).  But
then we'd be in a situation if boont / gaulish / scouse are
registered as ordinary variants.

We could also decree that boont / gaulish / lobjan / scouse
are dummy + deprecated variants, this would move 4 of the 34
to redundant.  Maybe also guoyu / hakka / xiang for 7 of 34.

We had reasons why we didn't do this for 3066bis, but if you
have a better plan that could be changed in 3066ter.  There's
no chance to get rid of en-GB-oed / no-bok / no-nyn.

> Listing them would do no harm.

Listed they are, you propose to copy the list into the ABNF.
For en / no / sgn / zh I don't see a perfect syntax solution.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 29 Aug 2006 22:13:16 +0200

Addison Phillips wrote:

> All of the sgn-xx-yy tags (3 of these), all of the i-xxxx
> tags (13 of them), plus the Oxford tag.

Yes, sgn-BE-fr, sgn-BE-nl, sgn-CH-de, en-GB-oed.  For i-xxxx
a parser should have no problem, a special case like x-yyyy,
but not really "irregular".  Depends on how you implement it.

Frank

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 14:33:08 -0700

> but not really "irregular".  Depends on how you implement it.

Anything starting with a singleton (except "x-" and the grandfathered 
"i-*" list of tags) is illegal. A parser that claims to be either 
well-formed or validating should yack on such a tag.

Addison

---

From: Addison Phillips <[email-masked]>
Date: Tue, 29 Aug 2006 13:16:11 -0700

> add a warning to implementors in RFC 3066ter?

Yes. It should be more explicit.

Addison

---

From: John Cowan <[email-masked]>
Date: Tue, 29 Aug 2006 16:46:51 -0400

> is one that is syntactically irregular.

In hindsight I should have used "Misleading" instead of "Broken".

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
Female celebrity stalker, on a hot morning in Cairo:
"Imagine, Colonel Lawrence, ninety-two already!"
El Auruns's reply:  "Many happy returns of the day!"

---

From: Martin Duerst <[email-masked]>
Date: Wed, 30 Aug 2006 16:06:57 +0900

>See my last two ietf-languages postings.

[chair hat on]

This is very difficult for people to follow, in particular for
those who are reading postings in the archives. Everybody,
please be more explicit, in particular for stuff from other lists.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 31 Aug 2006 11:21:18 +0200

Martin Duerst wrote:
 
> please be more explicit

He tries to register variants "boont" and "scouse" with
prefix "en".  This would move two "grandfathered" tags to
"redundant", reducing the number of "special cases" for a
test suite.

Relevant for any en-* (excl. en-GB-oed) if 3066ter goes
for a "variants MUST have a prefix" instead of the SHOULD
in 3066bis.

Frank