Subject: [Ltru] seeking resolution of the Great Script Debate

From: "Addison Phillips" <[email-masked]>
Date: Sat, 9 Apr 2005 15:20:14 -0700

It seems to me that in order to achieve a resolution of this issue, we need to take a step back and look concretely at the problem.

This message is going to be somewhat long, since I'm going to look at both sides of the problem in detail.

Let's start with the compatibility problem. The claim of Ned, Ira, and others is that 3066bis "breaks" existing implementations. I think that word "breaks" is problematic, because it does not describe accurately what happens. No implementations actually crash when you send them tags that they don't recognize (those that do are beyond consideration here). What we mean is that the results produced are either not what the user expected or are different than what the user previously received for the same request.

There are two kinds of implementation that we've been holding up for examination. I'll take them in turns.

First are pure RFC 2616 matching implementations which don't care about the contents of a particular subtag. This is equivalent to what 3066bis calls a "well-formed" processor. Common examples of these include xml:lang, CSS 2.1, Apache's language negotiation mechanism and Kurt's LDAP RFC. In these implementations, it doesn't matter if you send them a tag like "foo-bar-baz-gleep" or "zh-Hant-CN". All that matters is the matching of each token or subtag in order.

For these implementations, script subtags pose a problem if they are inconsistently used for a given language prefix. That is, if I sometimes request "xx-Latn-CC" and sometimes request "xx-CC" or if I sometimes tag content using "xx-Latn-CC" and sometimes as "xx-CC", then I will experience problems with matching the "xx-Latn" prefix to "xx-CC" and vice versa. The approach 3066bis takes here is to suggest that content authors and requesters be systematic in using or not using script subtags. 

Let us pause and recognize that there are many implementations of this nature and also recognize that this is not by any means "all" implementations either.

Second are the kinds of implementations Ned Freed has described, which DO care about the contents of specific subtags. In an RFC 3066bis implementation, these are validating processors. A well-known example that I'll use as a proof-of-concept is the ServletRequest.getLocale method of J2EE. This method takes Accept-language tags in the HTTP header and attempts to match them to java.util.Locale objects predefined in the Java runtime environment. There are roughly 150 of these in a JRE (less, usually, and the number varies by JRE) in the form language_region_variant.

There are four ways that these implementations may react to a tag in the form "xx-Latn-CC". First, they may find the language and region code and ignore interstitial subtags (producing xx-CC in our example). Second, they may find as many subtags in order until the meet one they don't recognize (producing xx in our example). Third, they may reject the whole tag as unrecognized. Fourth, they may assign wrong values to the wrong fields (which produces the same results in many ways as option the second).

Any of these is a valid reaction.

It is certainly possible in the J2EE case for a user to set up all of their content and resource bundles to work "correctly" with script subtags, but it is significant work to do so. In addition, Locale objects created with a script in either the region or variant slot will not work correctly and probably return the default language for the particular configuration or JRE. 

This is not an insignificant problem.

In fact, ironically, I have a demo of this on my own website here:

http://www.inter-locale.com/LocalesDemo.jsp

Only... I wrote the AcceptLanguageBean class that powers this demo around my RFC 3066bis implementation and it gets the right answer for "zh-Hant-TW" (try it yourself).

So I wrote one that uses pure J2EE here:

http://www.inter-locale.com/bis.jsp

It produces:

Lang = zh
Region = Hant
Country = TW

This is essentially the same thing as saying "zh" in terms of results.

There isn't much that we can tell users to do with their content tags or requests that will ameliorate this mismatch, since the implementation is using values in the subtags to perform some kind of mapping or processing. 

This problem (in this example, but not, please note, all possible examples) could be addressed by moving the script subtag down in the order.

The problem is the mismatch between RFC 2616 matching's needs (see Misha's email) and this J2EE-style matching's needs. 

Addison

Addison P. Phillips
Globalization Architect, Quest Software
http://www.quest.com

Chair, W3C Internationalization Core Working Group
http://www.w3.org/International

Internationalization is not a feature.
It is an architecture. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sun, 10 Apr 2005 02:07:29 +0200

Addison,
I may be dumb stupid. But I see that you honnestly document a problem which 
results from a change in the format of the entry in a function. I have some 
difficulty understanding why do you want to do that? What you document is 
that the current functions work well with the current tags. But if you 
change them to work well with the new tags, they will not to work well 
anymore with old tags what seems to make sense.

Why not just to do as Jesus said: to have new functions for new tags and to 
keep old functions for old tags? Is this a very big problem for W3C to say 
that xml:lang will keep using the old tags format and that xml:nlang will 
use a new format built from experience? Since you say libraries will have 
to be updated and no one is using yet your new format, the update would 
only to add nlangtag support to existing langtag support? By the same token 
you could also announce the support of xml:xlang to support the extended 
language tag we need.

Question (I am not at all an XML person):
- would this be a big problem if the xml:*lang tag was an URL?
- would this be a big problem if the language parameters were documented in 
another more extended way?
jfc

On 00:20 10/04/2005, Addison Phillips said:
>content-class: urn:content-classes:message
>Content-Type: text/plain;
>         charset="utf-8"
>
>It seems to me that in order to achieve a resolution of this issue, we 
>need to take a step back and look concretely at the problem.
>
>Let's start with the compatibility problem. The claim of Ned, Ira, and 
>others is that 3066bis "breaks" existing implementations. I think that 
>word "breaks" is problematic, because it does not describe accurately what 
>happens. No implementations actually crash when you send them tags that 
>they don't recognize (those that do are beyond consideration here). What 
>we mean is that the results produced are either not what the user expected 
>or are different than what the user previously received for the same request.
>
>First are pure RFC 2616 matching implementations which don't care about 
>the contents of a particular subtag. This is equivalent to what 3066bis 
>calls a "well-formed" processor. Common examples of these include 
>xml:lang, CSS 2.1, Apache's language negotiation mechanism and Kurt's LDAP 
>RFC. In these implementations, it doesn't matter if you send them a tag 
>like "foo-bar-baz-gleep" or "zh-Hant-CN". All that matters is the matching 
>of each token or subtag in order.
>
>For these implementations, script subtags pose a problem if they are 
>inconsistently used for a given language prefix. That is, if I sometimes 
>request "xx-Latn-CC" and sometimes request "xx-CC" or if I sometimes tag 
>content using "xx-Latn-CC" and sometimes as "xx-CC", then I will 
>experience problems with matching the "xx-Latn" prefix to "xx-CC" and vice 
>versa. The approach 3066bis takes here is to suggest that content authors 
>and requesters be systematic in using or not using script subtags.
>
>Let us pause and recognize that there are many implementations of this 
>nature and also recognize that this is not by any means "all" 
>implementations either.
>
>Second are the kinds of implementations Ned Freed has described, which DO 
>care about the contents of specific subtags. In an RFC 3066bis 
>implementation, these are validating processors. A well-known example that 
>I'll use as a proof-of-concept is the ServletRequest.getLocale method of 
>J2EE. This method takes Accept-language tags in the HTTP header and 
>attempts to match them to java.util.Locale objects predefined in the Java 
>runtime environment. There are roughly 150 of these in a JRE (less, 
>usually, and the number varies by JRE) in the form language_region_variant.
>
>There are four ways that these implementations may react to a tag in the 
>form "xx-Latn-CC". First, they may find the language and region code and 
>ignore interstitial subtags (producing xx-CC in our example). Second, they 
>may find as many subtags in order until the meet one they don't recognize 
>(producing xx in our example). Third, they may reject the whole tag as 
>unrecognized. Fourth, they may assign wrong values to the wrong fields 
>(which produces the same results in many ways as option the second).
>
>It is certainly possible in the J2EE case for a user to set up all of 
>their content and resource bundles to work "correctly" with script 
>subtags, but it is significant work to do so. In addition, Locale objects 
>created with a script in either the region or variant slot will not work 
>correctly and probably return the default language for the particular 
>configuration or JRE.
>
>Only... I wrote the AcceptLanguageBean class that powers this demo around 
>my RFC 3066bis implementation and it gets the right answer for 
>"zh-Hant-TW" (try it yourself).
>
>There isn't much that we can tell users to do with their content tags or 
>requests that will ameliorate this mismatch, since the implementation is 
>using values in the subtags to perform some kind of mapping or processing.
>
>This problem (in this example, but not, please note, all possible 
>examples) could be addressed by moving the script subtag down in the order.
>
>The problem is the mismatch between RFC 2616 matching's needs (see Misha's 
>email) and this J2EE-style matching's needs.
>
>Addison
>
>Addison P. Phillips

---

From: John Cowan <[email-masked]>
Date: Sun, 10 Apr 2005 03:35:17 -0400

JFC (Jefsey) Morfin scripsit:

> But if you change them to work well with the new tags, they will not to 
> work well anymore with old tags what seems to make sense.

This is simply incorrect.

-- 
You escaped them by the will-death              John Cowan
and the Way of the Black Wheel.                 [email-masked]
I could not.  --Great-Souled Sam                http://www.ccil.org/~cowan