Subject: RE: [Ltru] Great Script Debate Part II: Formats...

From: "Addison Phillips" <[email-masked]>
Date: Mon, 11 Apr 2005 14:15:23 -0700

Hi Debbie,

I don't think you understood the ramifications of the last paragraphs.

Extlang has exactly the same problems that script does in the second position. Moving alpha4 script codes would permit an extlang that is an alpha4, but then you have the same problem script codes have: you fill the same subtag position in the tag that some "smart" processor expects (stupidly) to be region.

More to the point, I thought that ISO 639-6 was supposed to supplant ISO 639-1/-2 codes in the primary language tag (something that can happen in the registry as designed).

With regard to "forever": maybe so. I don't think I want to scrawl my name under Ozymandias's?:-). But, at the same time, it makes a LOT of sense to define a fairly rigid grammar for language tags with the intention of "forever". This allows us to create a more complex series of subtags without the danger of future incompatibility. We enumerate exactly how tags can change in the future, so that implementations can be built to deal with those changes now. Otherwise your future lang(3066ter) tag is just a big opaque piece of junk I don't know how to deal with (and I have to upgrade to a 3066ter capable piece of code to deal with your 3066ter content).

In my 3066bis implementation, even though I don't know what an extension means, I can correctly process one every time. Ditto with extlangs.

RFC 3066bis, in fact, makes use of the fact that it narrows the range of subtags (and ONLY narrows it) to be compatible with 3066 implementations.

> Central Siberian Yupik = ypkw (written)
> Central Siberian Yupik = ypkc (written Cyrl script)
> Central Siberian Yupik = ypkl (written Latn script)

And how do I select "all documents written in Central Siberian Yupik"?

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: Debbie Garside [mailto:[email-masked]]
> Sent: lundi 11 avril 2005 13:10
> To: Addison Phillips; 'LTRU Working Group'
> Subject: RE: [Ltru] Great Script Debate Part II: Formats...
> 
> Addison
> 
> Option 5 has my vote as it may leave room for an *ext_langtag (4 character
> alphanumeric) for the proposed ISO 639-6 in what I think is a logical
> place
> e.g.
> 
> ll-lll-llll-CC-S-Ssss
> 
> There MAY (something that will be discussed at a meeting of the 639-6 Task
> Group in June 05) also be a possibility that 639-6 will offer script and
> language in the same tag e.g.
> 
> Central Siberian Yupik = ypkw (written)
> Central Siberian Yupik = ypkc (written Cyrl script)
> Central Siberian Yupik = ypkl (written Latn script)
> 
> As mentioned before, I also think that the tag scheme should be referenced
> in the langtag itself...
> 
> e.g. Lang(scheme_3066bis)= etc.... or it could be the BCP reference...
> 
> Nothing is forever.
> 
> It is also MHO that it SHOULD be a requirement to tag content wisely...
> for
> me wisely means put as much information in as possible... if the
> information
> is there and known to you... use it... the return for the time will be a
> much improved, user-friendly, informative WWW.
> 
> This is just my opinion.
> 
> Kind regards
> 
> Debbie
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Addison Phillips
> Sent: 11 April 2005 19:46
> To: LTRU Working Group
> Subject: [Ltru] Great Script Debate Part II: Formats...
> 
> Having established that there exist some RFC 3066 tag processors in broad
> distribution whose results are impacted by the lang-script-region format,
> let's skip over the question of the level of harm and look at alternative
> formats next. I promise (JFC) to come back and visit the relative harm
> question.
> 
> In this email, I'll use the subtags in the following 3066bis-style tag for
> demonstration purposes:
> 
> de-Latn-CH-1901-x-gleep
> 
> Let's consider the different ways that we can include script in a tag. For
> each option, I'll write out a tag with a script and a tag without a script.
> 
> 1. Script first.
> 
> With: Latn-de-CH-1901-x-gleep
> None: de-CH-1901-x-gleep
> 
> This tag format has little to recommend it. Some folks may find that the
> script-first position seems more suited to a hierarchy of tags (many
> languages are written in one script), but this position doesn't add
> appreciably to the information conveyed by a *language* tag. It has the
> same
> problems as 3066bis's design for "smart" 3066 processors. In fact, it is
> worse, since it also pushes the language out of position.
> 
> 2. Script combines with language.
> 
> deLatn-CH-1901-x-gleep
> de-CH-1901-x-gleep
> 
> This tag format fixes the region code problem. It also installs a "unique
> new format" that some have asked for. The problem here is that the two
> tags
> above really should refer to the same language and 3066 processors of all
> types (RFR, smart) will never get that.
> 
> 3. Script after language
> 
> de-Latn-CH-1901-x-gleep
> de-CH-1901-x-gleep
> 
> This is RFC 3066bis, of course. The issue here is the script vs. region
> code
> that we're discussing. The advantage to this design is that script is at
> the
> right level in the tag in most cases. RFR processors can remove the script
> (unlike in 2) during matching. RFR matching works as long as users are
> consistent about using or not using script for a particular prefix.
> "Smart"
> 3066 matching can't find the region code with this design. (Smart 3066bis
> processing finds all subtags)
> 
> 4. Script after region and before variant
> 
> de-CH-Latn-1901-x-gleep
> de-CH-1901-x-gleep
> de-Latn-1901-x-gleep // lookout...
> 
> This doesn't harm 3066 processors in most cases, but harms users of these
> implementations who need to find specific script versions of language.
> That
> is, RFR processors can't find matching scripts. "Smart" 3066 processors
> (such as the J2EE demo), get language and region into the right slots.
> 
> 4a. Script as a variant.
> 
> This is the same as (4), only the script is just another variant. Too
> boring
> to discuss.
> 
> 5. Script as an extension
> 
> de-CH-1901-s-Latn-x-gleep
> de-CH-1901-x-gleep
> de-1901-s-Latn-x-gleep // my previous example of a tag that harms smart
> processors
> 
> This has the advantage of allowing 3066bis "smart" processors to put the
> script back between "de" and "CH" if it wants to internally, while not
> interfering with existing tags (note emphasis here: *tags*). RFR
> processors
> remove the script even before variant.
> 
> 6. Script as an attribute
> 
> de-CH-1901-x-gleep; script=Latn; q=1.0
> de-CH-1901-x-gleep; q=1.0
> 
> This has the advantage of not mucking with the tags at all, but can't be
> used in contexts such as XML :-(. Who knows how matching works.
> 
> 7. Script inferred from the tag
> 
> de-CH-1901-x-gleep
> 
> This is what we have today. If you see "zh-TW", that must mean Hant. Good
> luck if you live in HK.
> 
> 
> Any I missed?
> 
> 
> Now, before you choose one of these, don't' forget that you can do
> s/script/extlang/ on the above text. We don't just have one interpolated
> subtag, we have two.
> 
> Best Regards,
> 
> Addison
> 
> Addison P. Phillips
> Globalization Architect, Quest Software
> http://www.quest.com
> 
> Chair, W3C Internationalization Core Working Group
> http://www.w3.org/International
> 
> Internationalization is not a feature.
> It is an architecture.
> 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 11 Apr 2005 23:28:53 +0100

Hi Addison

See inline comments

Debbie

> From: Addison Phillips [mailto:[email-masked]]
> Sent: 11 April 2005 22:15
> To: Debbie Garside; LTRU Working Group

You could be right :-)
> 
> Extlang has exactly the same problems that script does in the second
> position. Moving alpha4 script codes would permit an extlang that is an
> alpha4, but then you have the same problem script codes have: you fill the
> same subtag position in the tag that some "smart" processor expects
> (stupidly) to be region.

OK...
> 
> More to the point, I thought that ISO 639-6 was supposed to supplant ISO
> 639-1/-2 codes in the primary language tag (something that can happen in
> the registry as designed).

Yes... they can be used in this way, if required, or as an extension to
other 639 tags. -6 is not designed to supplant anything... it is part of the
639 family... and they are all required for differing reasons... but yes...
-6 (as it stands) could be used on its own.

When I look at your draft it says Lang = 2*3 Alpha... -6 is CURRENTLY
alpha4.  Have I missed something?  The last I heard was that -6 (if it
becomes standard) would have a singleton placed in front of it and be
treated as a variant... has this changed?  Perhaps you could clarify exactly
how you would deal with -6 in its present form - which includes language
families, groups, languages, dialects, written, spoken and signed.  

OAFN, it MAY be possible to use the Alpha4 langtag in -6 to denote writing
system rather than script - something, as you are aware, I feel is more
relevant and useful. 

Are you saying that, some time in the future, you could possibly foresee the
proposed 639-6 Alpha4 as the primary language tag?  

> 
> With regard to "forever": maybe so. I don't think I want to scrawl my name
> under Ozymandias's?:-). But, at the same time, it makes a LOT of sense to
> define a fairly rigid grammar for language tags with the intention of
> "forever". This allows us to create a more complex series of subtags
> without the danger of future incompatibility. We enumerate exactly how
> tags can change in the future, so that implementations can be built to
> deal with those changes now. Otherwise your future lang(3066ter) tag is
> just a big opaque piece of junk I don't know how to deal with (and I have
> to upgrade to a 3066ter capable piece of code to deal with your 3066ter
> content).
> 
Just MHO... but what happens if 4 years down the line it isn't "forever" and
some, as yet unseen, circumstance arises that means the whole system needs
changing... who knows... but I do accept your point.  I am looking at those
industries that are going to use this RFC's rules to tag content for archive
purposes... archives that may not be touched for 10 years. I hope you are
right :-)

> In my 3066bis implementation, even though I don't know what an extension
> means, I can correctly process one every time. Ditto with extlangs.
> 
> RFC 3066bis, in fact, makes use of the fact that it narrows the range of
> subtags (and ONLY narrows it) to be compatible with 3066 implementations.
> 

In this instance... query "ypk*" but, preferably you would reference the
hierarchical registry, as proposed for -6, that includes the "Mother" tag...
just go up one LEVEL and stop at the associated "Written Object".  If you
want everything... written, spoken, signed... follow the "Mother Tags" until
you get to the "Language Level" (in this instance up one level again).

Best regards

Debbie

---

From: John Cowan <[email-masked]>
Date: Mon, 11 Apr 2005 18:44:57 -0400

> proposed 639-6 Alpha4 as the primary language tag?  

Until we know more about 639-6, nobody can say.  From what I understand,
639-6 4-alpha tags encode language varieties, and one must make use of
tables to learn how the language varieties are interrelated.  I don't
know whether these external tables are a hierarchy or what.

Any information you can reveal would be extremely useful.

-- 
                Si hoc legere scis, nimium eruditionis habes.

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 12 Apr 2005 00:16:38 +0100

Hi John

> know whether these external tables are a hierarchy or what.

The data that -6 is currently based on is contained within a hierarchical
system which relates language families, language groups, languages and
varieties (I use that term loosely).

> Any information you can reveal would be extremely useful.

At present, other than the information that I gave last year outlining the
system (which basically stands), it would not be a good idea for me to
reveal too much (perhaps my ISO/BSI colleagues currently monitoring this
list can help here... I don't know the ISO protocol on this).  

The draft I (and my colleagues in the BSI WG) are working on will be
discussed in Paris in June (1st) by the ISO 639-6 Task Group and at the ISO
639 WG in Poland in August.  I will endeavour to have the document
distributed before end of July and at that time I hope to be able to
distribute for "expert" opinion.

Kind regards

Debbie

---

From: John Cowan <[email-masked]>
Date: Mon, 11 Apr 2005 22:52:35 -0400

> list can help here... I don't know the ISO protocol on this).  

I think that the most important point for this list is: is there going
to be a normative mapping between -6 tags and -1/-2 tags, and if so,
will -3 and -5 tags be mapped when those parts are available?  Or is
the basis for -6 irreconcilable (not in principle but in detail) with
the unified model of -1/-2/-3/-4/-5?

-- 
You let them out again, Old Man Willow!                 John Cowan
What you be a-thinking of?  You should not be waking!   [email-masked]
Eat earth!  Dig deep!  Drink water!  Go to sleep!       www.reutershealth.com
Bombadil is talking.                                    www.ccil.org/~cowan

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 12 Apr 2005 09:20:19 +0100

Hi John

In answer to your question... yes, -6 is currently being mapped to ISO 639
1; 2T/B;and the proposed 3 and 5 as well as 15924.  

Debbie

> From: John Cowan [mailto:[email-masked]]
> Sent: 12 April 2005 03:53
> To: Debbie Garside
> Cc: [email-masked]
> Subject: Re: [Ltru] Great Script Debate Part II: Formats...
> 
> Debbie Garside scripsit:
>

---

From: John Cowan <[email-masked]>
Date: Tue, 12 Apr 2005 09:38:27 -0400

> 1; 2T/B;and the proposed 3 and 5 as well as 15924.  

Excellent.  It's a design point of the RFC 3066 system that no two tags
have the same denotation (except where forced on us by ISO 3166), so if
we decide to include 639-6 tags, we would want to bar those which are
equivalent to existing -1, -2, or -3 tags (just as, e.g.  we bar "eng"
from 639-2 in favor of "en" from 639-1).

Another question: Is it the intention of the future 639-6/RA to maintain
a strict hierarchy moving forward?  Or would it be in order, for example,
to request a code for "Balkan Sprachbund languages" (roughly, Bulgarian,
Romanian, Serbian, and Greek) even though that cuts across the standard
genetic grouping for Indo-European?  If that can be done, what about a
code for "languages spoken in Nigeria" (mostly Benue-Congo languages,
but also other Niger-Congo languages as well as Chadic from Afroasiatic
plus English and Shuwa Arabic)?  This question is also relevant to ISO
639-5, though not directly to RFC 3066-style language tags.

-- 
Do I contradict myself?                         John Cowan
Very well then, I contradict myself.            [email-masked]
I am large, I contain multitudes.               http://www.ccil.org/~cowan
        --Walt Whitman, Leaves of Grass         http://www.reutershealth.com