Subject: [Ltru] Great Script Debate Part II: Formats...

From: "Addison Phillips" <[email-masked]>
Date: Mon, 11 Apr 2005 11:46:02 -0700

Having established that there exist some RFC 3066 tag processors in broad distribution whose results are impacted by the lang-script-region format, let's skip over the question of the level of harm and look at alternative formats next. I promise (JFC) to come back and visit the relative harm question.

In this email, I'll use the subtags in the following 3066bis-style tag for demonstration purposes:

de-Latn-CH-1901-x-gleep

Let's consider the different ways that we can include script in a tag. For each option, I'll write out a tag with a script and a tag without a script.

1. Script first.

With: Latn-de-CH-1901-x-gleep
None: de-CH-1901-x-gleep

This tag format has little to recommend it. Some folks may find that the script-first position seems more suited to a hierarchy of tags (many languages are written in one script), but this position doesn't add appreciably to the information conveyed by a *language* tag. It has the same problems as 3066bis's design for "smart" 3066 processors. In fact, it is worse, since it also pushes the language out of position.

2. Script combines with language.

deLatn-CH-1901-x-gleep
de-CH-1901-x-gleep

This tag format fixes the region code problem. It also installs a "unique new format" that some have asked for. The problem here is that the two tags above really should refer to the same language and 3066 processors of all types (RFR, smart) will never get that.

3. Script after language

de-Latn-CH-1901-x-gleep
de-CH-1901-x-gleep

This is RFC 3066bis, of course. The issue here is the script vs. region code that we're discussing. The advantage to this design is that script is at the right level in the tag in most cases. RFR processors can remove the script (unlike in 2) during matching. RFR matching works as long as users are consistent about using or not using script for a particular prefix. "Smart" 3066 matching can't find the region code with this design. (Smart 3066bis processing finds all subtags)

4. Script after region and before variant

de-CH-Latn-1901-x-gleep
de-CH-1901-x-gleep
de-Latn-1901-x-gleep // lookout...

This doesn't harm 3066 processors in most cases, but harms users of these implementations who need to find specific script versions of language. That is, RFR processors can't find matching scripts. "Smart" 3066 processors (such as the J2EE demo), get language and region into the right slots.

4a. Script as a variant.

This is the same as (4), only the script is just another variant. Too boring to discuss.

5. Script as an extension

de-CH-1901-s-Latn-x-gleep
de-CH-1901-x-gleep
de-1901-s-Latn-x-gleep // my previous example of a tag that harms smart processors

This has the advantage of allowing 3066bis "smart" processors to put the script back between "de" and "CH" if it wants to internally, while not interfering with existing tags (note emphasis here: *tags*). RFR processors remove the script even before variant.

6. Script as an attribute

de-CH-1901-x-gleep; script=Latn; q=1.0
de-CH-1901-x-gleep; q=1.0

This has the advantage of not mucking with the tags at all, but can't be used in contexts such as XML :-(. Who knows how matching works.

7. Script inferred from the tag

de-CH-1901-x-gleep

This is what we have today. If you see "zh-TW", that must mean Hant. Good luck if you live in HK.

Any I missed?

Now, before you choose one of these, don’t' forget that you can do s/script/extlang/ on the above text. We don't just have one interpolated subtag, we have two.

Best Regards,

Addison

Addison P. Phillips
Globalization Architect, Quest Software
http://www.quest.com

Chair, W3C Internationalization Core Working Group
http://www.w3.org/International

Internationalization is not a feature.
It is an architecture. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 11 Apr 2005 21:32:12 +0200

> Any I missed?

Add script to the charset ;-)  Joke off, you probably wanted
to demonstrate that de-CH-s-Latn is worse than de-Latn-CH, but
I don't see it immediately.
                            Bye, Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 11 Apr 2005 21:10:29 +0100

Addison

Option 5 has my vote as it may leave room for an *ext_langtag (4 character
alphanumeric) for the proposed ISO 639-6 in what I think is a logical place
e.g. 

ll-lll-llll-CC-S-Ssss

There MAY (something that will be discussed at a meeting of the 639-6 Task
Group in June 05) also be a possibility that 639-6 will offer script and
language in the same tag e.g.

Central Siberian Yupik = ypkw (written)
Central Siberian Yupik = ypkc (written Cyrl script) 
Central Siberian Yupik = ypkl (written Latn script)

As mentioned before, I also think that the tag scheme should be referenced
in the langtag itself... 

e.g. Lang(scheme_3066bis)= etc.... or it could be the BCP reference...

Nothing is forever.

It is also MHO that it SHOULD be a requirement to tag content wisely... for
me wisely means put as much information in as possible... if the information
is there and known to you... use it... the return for the time will be a
much improved, user-friendly, informative WWW.

This is just my opinion.

Kind regards

Debbie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Addison Phillips
Sent: 11 April 2005 19:46
To: LTRU Working Group
Subject: [Ltru] Great Script Debate Part II: Formats...

Having established that there exist some RFC 3066 tag processors in broad
distribution whose results are impacted by the lang-script-region format,
let's skip over the question of the level of harm and look at alternative
formats next. I promise (JFC) to come back and visit the relative harm
question.

In this email, I'll use the subtags in the following 3066bis-style tag for
demonstration purposes:

de-Latn-CH-1901-x-gleep

Let's consider the different ways that we can include script in a tag. For
each option, I'll write out a tag with a script and a tag without a script.

1. Script first.

This tag format has little to recommend it. Some folks may find that the
script-first position seems more suited to a hierarchy of tags (many
languages are written in one script), but this position doesn't add
appreciably to the information conveyed by a *language* tag. It has the same
problems as 3066bis's design for "smart" 3066 processors. In fact, it is
worse, since it also pushes the language out of position.

deLatn-CH-1901-x-gleep
de-CH-1901-x-gleep

This tag format fixes the region code problem. It also installs a "unique
new format" that some have asked for. The problem here is that the two tags
above really should refer to the same language and 3066 processors of all
types (RFR, smart) will never get that.

3. Script after language

de-Latn-CH-1901-x-gleep
de-CH-1901-x-gleep

This is RFC 3066bis, of course. The issue here is the script vs. region code
that we're discussing. The advantage to this design is that script is at the
right level in the tag in most cases. RFR processors can remove the script
(unlike in 2) during matching. RFR matching works as long as users are
consistent about using or not using script for a particular prefix. "Smart"
3066 matching can't find the region code with this design. (Smart 3066bis
processing finds all subtags)

de-CH-Latn-1901-x-gleep
de-CH-1901-x-gleep

This doesn't harm 3066 processors in most cases, but harms users of these
implementations who need to find specific script versions of language. That
is, RFR processors can't find matching scripts. "Smart" 3066 processors
(such as the J2EE demo), get language and region into the right slots.

4a. Script as a variant.

This is the same as (4), only the script is just another variant. Too boring
to discuss.

de-CH-1901-s-Latn-x-gleep
de-CH-1901-x-gleep
de-1901-s-Latn-x-gleep // my previous example of a tag that harms smart
processors

This has the advantage of allowing 3066bis "smart" processors to put the
script back between "de" and "CH" if it wants to internally, while not
interfering with existing tags (note emphasis here: *tags*). RFR processors
remove the script even before variant.

This has the advantage of not mucking with the tags at all, but can't be
used in contexts such as XML :-(. Who knows how matching works.

de-CH-1901-x-gleep

This is what we have today. If you see "zh-TW", that must mean Hant. Good
luck if you live in HK.

Any I missed?

Now, before you choose one of these, don't' forget that you can do
s/script/extlang/ on the above text. We don't just have one interpolated
subtag, we have two.

Best Regards,

Addison

Addison P. Phillips

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 12 Apr 2005 01:33:04 +0200

>In this email, I'll use the subtags in the following 3066bis-style tag for 

Dear Addison,
Thank you for this. I will comment from my point of view which is 
orthogonality between format and content, calling for the format to be 
adapted to the application rather than the content to be adapted to the 
format. But I also understand your historical constraints (I did not though 
they were so bad). So, do not take my comments as critics, but as pure 
comment/explanation/question to be sure I follow your point.

Then I consider if there is a way out of all this.

>appreciably to the information conveyed by a *language* tag. It has the 
>same problems as 3066bis's design for "smart" 3066 processors. In fact, it 
>is worse, since it also pushes the language out of position.

This might interest in several cases:
1. when you already know the script, this format permits to forget the 
script - and stay compatible with former formats? (LDAP?)
2. when you want to access the information in using a system in its own 
language, you must first know the script. But access (DNS, database, 
matrix) is related to the query order, not to the sequence; so we agree.

>new format" that some have asked for. The problem here is that the two 
>tags above really should refer to the same language and 3066 processors of 
>all types (RFR, smart) will never get that.

The unique format does not match the requirement of the Charter to be able 
to identify the subtags. The other problem is that this format follows the 
ISO tables and is therefore not independent from the ISO evolution. It adds 
both problems.

However, this negative aspects should not hide a totally different avenue 
which is aliasing (name or number or IPv6 address). The main problem in our 
discussion is still to define the framework - charter has not been 
discussed. What do you think a language is, what is the purpose of the tag, 
what is its application process, what is its use, what is the IANA 
implication.

We can discuss endlessly if the different associated decision parameters 
are not defined. You decided that the solution was to be in a way you 
analyzed and you know it is not perfect. And you want to be helped in 
having it the less imperfect. I am not really able to help because I will 
have probably very different responses to the Charter "thorny" questions. 
But where I may be really able to help is that these responses - which will 
necessarily be those of the market at some stage - could help you to 
consider the problem differently and may be better solve it.

>3. Script after language
>
>de-Latn-CH-1901-x-gleep
>de-CH-1901-x-gleep
>
>This is RFC 3066bis, of course. The issue here is the script vs. region 
>code that we're discussing. The advantage to this design is that script is 
>at the right level in the tag in most cases. RFR processors can remove the 
>script (unlike in 2) during matching. RFR matching works as long as users 
>are consistent about using or not using script for a particular prefix. 
>"Smart" 3066 matching can't find the region code with this design. (Smart 
>3066bis processing finds all subtags)

I obviously have no objection to that. Except that format should not be 
limited to 2 letters or 3 digits. I understand the sense it makes for XML 
processors. It makes no sense for processors of mine. The real point is: 
does it make sense to people and web services? You responded - if I am 
correct - that formats are orthogonal to web services and you did not 
respond yet about word processors etc. So the only real issue is with users.

>implementations who need to find specific script versions of language. 
>That is, RFR processors can't find matching scripts. "Smart" 3066 
>processors (such as the J2EE demo), get language and region into the right 
>slots.

OK. This is again a problem internal to "your" W3C applications. Nothing 
against this patching of the past, as long as we can proceed with some 
innovative thinking for the future. Randy proposes a one tag per subtag 
approach. I think it may not be the proper way to do that as tags, but that 
this is definitely a serious step ahead. I am open to every flexibility in 
calling on a stable external reference.

>This is the same as (4), only the script is just another variant. Too 

Accepted.

>interfering with existing tags (note emphasis here: *tags*). RFR 
>processors remove the script even before variant.

Would Randy's proposition not be supported as a complete extension of that? 
I say that 5 descriptors are necessary. I have nothing against the idea 
that all are supported this way. However ugly a format it would make.

>used in contexts such as XML :-(. Who knows how matching works.

gee! it seems that you (W3C) created yourself quite a problem. I should 
reread the architectural framework you published some times ago. But I feel 
you violated there some of your extensibility, etc. most basic rules?

>This is what we have today. If you see "zh-TW", that must mean Hant. Good 

Frankly, is this a real problem? I mean status quo with the system you 
specified, got developed and deployed. Why not to think about a new 
generation? I know this means a big problem. But would that problem not be 
smaller that the addition of all the problems you are going to create 
yourself and ourselves? You are thinking about it for 2 years... makes a 
lot of pages in the meanwhile. BCP 047 will not be updated before some 
time. And then? All this will lead to will be ugly patches you do not want 
to match our own parameters when we go through.

I do not know XML but I know the DNS. We were explained by many people all 
the problems of the DNS if this, if that. Due to that they forced the IDNA. 
My relations here were "nice" when compared with the WH-IDN. Except that 
they talked privately and every of them except one (not an author) accepted 
the Draft was far from perfect (including conceptually). Anyway at the end 
of the day we had IDNA to please ICANN (I would say as to please W3C) and 
to please China. There were after that a lot of fuss on the way to apply 
IDNA and there is quite no more use except may be in the Latn countries.

What I fear with your approach is that you take the same concept as IDNA: 
to try to compromise impossible needs, between an ASCII and a Multilingal 
Internets. And to be tied by your own format rigidity? Same problem with 
the IP¨v6 format rigidity ...

>Any I missed?

Yes, mine :-)

Again, your proposition is OK for identification (with additional 
referent/style element if more than a menu) because it is likely that the 
current ISO tables will not change much before a entire revision of the 
matter has been carried. But, we must give the direction of the transition 
and of the future.

The problem is not your proposition. But the way you want to present and 
introduce it within the IETF framework .... as BCP 047.

>Now, before you choose one of these, donâ€™t' forget that you can do 

General comment.

Let be candid. There is no real good solution to your need today. And your 
need is part of the far more general answer to the need of a new generation 
network IETF awaits for 13 years (since they found they missed IP 
addresses) and I observed 21 years ago (when I interconnected the DoD, etc. 
to the public international networks and saw we missed some of the hooks 
others networks had). The question is simple: is a patch worth the try now, 
or would it not be better to reshape the whole network (I talk of its real 
architecture, not of NGN)?

My feeling is that the reshape is necessary but I do not believe IETF is 
ready for it. So the patch is advisable. But the patch must not try to take 
the place of the reshape while the reshape is going to occur as a 
grassroots process. Instead they should cooperate.

This is why I advocate documents to be split a little bit differently that 
you did.

1. a standard virtual tags framework, describing the registry, the 
procedures, the applications, the obligations of the formats to be BCP 047 
compliant. This will address ISO tables inclusion (and possibly other 
tables) and IANA registry, as well as IANA data dissemination and 
possible  filtering guidelines as part of the registration process. Access 
to pertinence is not an easy issue to address.

2. the format/filtering system for XML, HTML, CLDR, LDAP, OPES, etc. May be 
one document by application.

The interest is that the first document will be a Multilingual Internet 
building block. With a universal scope. It should help your "local" W3C 
needs to be put in a wider perspective and help their evolution. Its 
interest is also that incidentally it respects the spirit of the Charter 
better.

Thank you.
jfc