Subject: RE: [Ltru] Re: Compatibility with existing use (LDAP)

From: "Addison Phillips" <[email-masked]>
Date: Thu, 7 Apr 2005 10:04:49 -0700

Interlinear below, elision as necessary.

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of McDonald, Ira
> 
> 
> I don't believe that this quiet insertion of script between
> language and region is harmless to RFC 3066 implementations.
[Addison Phillips] 

It is harmless so long as users and content authors understand the ramifications of strict RFC 2616 prefix matching. Content authors who start to use script subtags will have to deal with the possibility that certain implementations will not match e.g. "en-Latn-US" to e.g. the range "en-US".
> 
> It's true that such tags were allowed to be registered by
> RFC 3066.  But there are literally millions of documents
> currently language tagged on the Internet in the style of
> 'language-REGION-whatever' and lots of naive software thinks
> that's the only valid form of a language tag.
[Addison Phillips] 

Naïve implementations generally assume 'language-REGION' and nothing more--and no registrations. These will have the troubles they have.

Most naïve implementations actually assume like RFC 3688 does: a sequence of subtags in a strict order. See comment above for why those implementations can be defended as valid and are not broken by 3066bis as long as users and content authors agree to either use or not use the script subtags for a particular language.

> 
> The linguistic purity of infixing 'script' (between language
> and region) is unwise.  The recent (all unsuccessful) numeric
> weighting algorithms for 'smart matching' make this point
> eloquently.
[Addison Phillips] 

It isn't "linguistic purity". No one has responded to the point that putting script last DOESN'T FIX ANYTHING:

Tag:  "en-US-Latn"
Range: "en-Latn"

Uh oh.

Tag: "zh-CN-Hans"
Range: "zh-Hans"

Uh oh.

It doesn't matter what order you pick, some users will get results that don't work out using strict prefix matching as long as you allow subtags to be omitted (as we do).

Users and content authors that use script will have to decide. Using extended language ranges will help by making the matching work in more cases.

> 
> > Ira McDonald had written:
> >
> > > Also, LTRU is developing an IANA registry of EVERY
> > > possible valid language tag.
> >
> > We're not doing that either.  Important distinction: we are
> > developing a
> > registry of every possible valid *subtag*, not tag.  The
> > former adds up
> > to just under a thousand items; the latter (including all the
> > ridiculous
> > ones) would be about 15 million.
> >
> 
> Oops!  My silly mistake.  Tired fingers.
> 
> > -Doug Ewell
> >  Fullerton, California
> >  http://users.adelphia.net/~dewell/
> >
> >
> >
> > _______________________________________________
> > Ltru mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ltru
> >
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Misha Wolf <[email-masked]>
Date: Thu, 07 Apr 2005 19:11:19 +0100

Consider the "zh" case.  Existing practice is a mess:
   zh-cn
   zh-hk
   zh-mo
   zh-sg
   zh-tw
   zh
[from IE 6]

It's up to each user to guess what the tagger meant by each of these 
tags.

If a user wants just Simplified Chinese what should s/he request?  
If s/he wants just Traditional Chinese what should s/he request?

If we replace the above mess with zh-Hans and zh-Hant, the situation 
becomes much clearer.

As far as legacy tags matched by legacy applications are concerned, 
in both cases, if the user requests "zh" they will get the lot and 
it's up to them to select.  It's true that such an application won't 
match zh-Hant with zh-TW, but that seems like a price worth paying.

Misha

-----------------------------------------------------------------
        Visit our Internet site at http://www.reuters.com

To find out more about Reuters Products and Services visit http://www.reuters.com/productinfo 

Any views expressed in this message are those of  the  individual
sender,  except  where  the sender specifically states them to be
the views of Reuters Ltd.

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 7 Apr 2005 13:30:55 -0700

Yes, exactly so.

For the Brai case, I don't think that's necessary. A script without a Recommended Prefix can be used to form any subtag with no constraint on validating processors.

Addison

Addison P. Phillips

> From: John Cowan [mailto:[email-masked]]
> Sent: jeudi 7 avril 2005 13:23
> To: Addison Phillips
> Cc: [email-masked]
> Subject: Re: [Ltru] Re: Compatibility with existing use (LDAP)
> 
> Addison Phillips scripsit:
> 
> > What if we put Recommended Prefix fields onto script subtags in the
> > registry?
> >
> > This solves the default script problem because we are recommending when
> > to use the subtag (when your language is in the list for that subtag).
> 
> Excellent!  So Latn would have a Recommended Prefix of az (among others)
> but
> not en.
> 
> We might want Brai to have a recommended prefix of *, since brai could be
> attached to almost any language, or at least any language that is written.
> 
> --
> Mark Twain on Cecil Rhodes:                     John Cowan
> I admire him, I freely admit it,               http://www.ccil.org/~cowan
> and when his time comes I shall
> http://www.reutershealth.com
> buy a piece of the rope for a keepsake.        [email-masked]

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 7 Apr 2005 16:06:36 -0700

> Not completely.  For something like "fy" implementations still
> don't know what the undocumented default (Latn in this case)
> is.  

Implementations do not care what the default is. Just like they don't care what the "default region" is for a language code. Because it doesn't matter. 

What matters is whether users adopt script codes consistently. Inconsistency breeds problems for RFR matching schemes (requiring that implementations "care" about default values). All of which is amply demonstrated on this long thread. 

Your concern was that we provide no guidance to users on whether or not to use the new subtags. RFR works as long the number of subtags being matched is consistent within a given language range. Here is a way to provide exactly that guidance without burdening language tag applications with having to know about it. Default scripts require smarts inside applications (to expand the script subtag at appropriate times). This merely requires "smarts" inside users (or implementers), which is a far more likely place to find intelligence.

Users are still free to ignore the advice in the registry if that is their whim (or requirement), but they're adequately warned about the matching issues.

> And you'd get a rather long list of prefixes for scripts
> like Latn or Cyrl (all languages using both), don't you ?

Certain scripts will have a long-ish list of languages while most scripts will languish unadorned. This is a problem because...?

In practice, while the list of multi-script languages may turn out to be impressively long, it probably isn't that long. And users are really looking to see if they should tag their language with a script or not. If implementations follow the registry, most users will pick from a list of appropriately formatted tags and this won't be a problem.

I think we've also shown that this problem cannot be 100% solved for all cases of all tags, no matter what their formation. I think this could be good enough because it helps guide users into making good choices that mostly avoids the problem.

Addison

Addison P. Phillips

> Behalf Of Frank Ellermann
> Sent: jeudi 7 avril 2005 15:15
> To: [email-masked]
> Subject: [Ltru] Re: Compatibility with existing use (LDAP)
> 
> Addison Phillips wrote:
> 
> > This solves the default script problem because we are
> > recommending when to use the subtag (when your language
> > is in the list for that subtag).
> 
> Not completely.  For something like "fy" implementations still
> don't know what the undocumented default (Latn in this case)
> is.  And you'd get a rather long list of prefixes for scripts
> like Latn or Cyrl (all languages using both), don't you ?  Bye.
> 
> 
>

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 7 Apr 2005 23:11:05 -0700

John Cowan <jcowan at reutershealth dot com> wrote:

>> This solves the default script problem because we are recommending
>> when to use the subtag (when your language is in the list for that
>> subtag).
>
> Excellent!  So Latn would have a Recommended Prefix of az (among
> others) but not en.

I still oppose the premise of "default script" itself, regardless of the
mechanism used to implement it.  If there is a single piece of content
labeled "en-Cyrl" or "en-Arab" or "en-Dsrt", someone may have a need to
request it.  Similarly, someone else may have a need to request
"en-Latn" and, by doing so, explicitly exclude whatever Cyrillic or
Aaraib or Deseret content might otherwise be returned.  I do not believe
we are in a position to "recommend" these on anyone's behalf.

This is not at all the same as the "recommended prefix" associated with
variant subtags.  We are able to recommend certain prefixes for use with
certain variants, because other combinations would be absurd.  It is
absurd to speak of Korean written according to the 1996 Rechtschreibung,
or the Natisone dialect of Xhosa.  It is not absurd to speak of Korean
written in Cyrillic, or Xhosa written in Yi, no matter how uncommon
those combinations may be.

-Doug Ewell
 Fullerton, California

---

From: "Addison Phillips" <[email-masked]>
Date: Fri, 8 Apr 2005 10:29:07 -0700

> mechanism used to implement it.  

I'm not wild about default script, but I think this is qualitatively different. This is a suggestion of which languages would be most appropriate to use a script with. This list is not closed and is non-exclusive. We could create Yet Another Field Type to do this specifically for scripts. In any case, this is informational rather than normative content.

> If there is a single piece of content
> request it. 

This proposal doesn't prevent these cases, nor would it prevent the widespread adoption of script subtags for any specific language, no matter how "default" the script might be. It merely addresses the comment that there is no guidance on when to use the script subtags.

> Similarly, someone else may have a need to request

We aren't. We are recommending that specific languages always be used with a script subtag (and we don't define which one per-se) and are suggesting that other languages not use them unless there is some need.

> variant subtags.  

Not quite the same, no. Variants SHOULD NOT be used with other prefixes. But note that scripts SHOULD NOT be used with languages for which they add no distinguishing information (Section 2.3, rule 2) and recommended prefix provides a mechanism to achieve exactly that.

Note also that one could write that sentence as "Scripts MAY be used with languages for which they add no distinguishing information." :-)

The draft doesn't, therefore, claim omniscience regarding what a good language tag choice is for your particular application, it merely suggests, based on past experience, which script subtags make good combinations. Default scripts, by contrast, do make such a claim of omniscience.

> We are able to recommend certain prefixes for use with

Yes, I agree. But the question isn't one of tagging unusual content. It is a question of how to tag run-of-the-mill content. '-Hang' adds no distinguishing information to the language subtag 'ko'. I can certainly imagine tagging content "ko-Cyrl" to distinguish it properly, though.

Anyway, it is just an idea for how to break the deadlock on the only real substantive objection to the draft as a whole at this date. Once the general design reaches full consensus, then all of the remaining discussion is about text in the draft, which is fighting over the table scraps as far as I'm concerned.

Addison

Addison P. Phillips

> Behalf Of Doug Ewell
> Sent: jeudi 7 avril 2005 23:11
> To: LTRU Working Group

---

From: "McDonald, Ira" <[email-masked]>
Date: Fri, 8 Apr 2005 12:35:31 -0700

Hi Addison,

I'm lost (hundreds of LTRU messages polluted my inbox today).

Does your suggestion about "recommended prefix" include moving 
the country codes firmly back to second position?

If yes, then I agree.  If no, then nothing substantive is
accomplished.

The idea that compatible software that supports comparison of
language tags in the RFC3066bis era MUST have a copy of the
registry is a lousy idea.

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

> From: [email-masked] 
> [mailto:[email-masked]]On
> Behalf Of Addison Phillips
> Sent: Friday, April 08, 2005 1:31 PM
> To: John Cowan; [email-masked]
> Cc: [email-masked]
> Subject: RE: [Ltru] Re: Compatibility with existing use (LDAP)
> 
> 
> To back up for a second: does my proposal about recommended 
> prefix (I think you indicated initial enthusiasm, Ned) solve 
> this problem sufficiently to move forward with this design?
> 
> Addison
> 
> Addison P. Phillips
> > Behalf Of John Cowan
> > Sent: vendredi 8 avril 2005 10:18
> > To: [email-masked]
> > Cc: [email-masked]
> > [email-masked] scripsit:
> > 
> > > A compliant RFC 3066 implementation can (and I know for a 
> fact has) been
> > > written that handles these exceptions through an 
> exception table, but
> > still
> > > operates under the assumption that a two letter code in the second
> > position is
> > > a country code but a two letter code elsewhere is not.
> > 
> > An assumption quite without warrant in the text of RFC 3066.
> > 
> > > Like it or not, there are lots of application domains where script
> > information
> > > is much less readily available than country information.
> > 
> > The intention (as Addison has developed in his messages on 
> the subject) is
> > to use script information only for those languages where it 
> is commonly
> > required.
> > 
> > > Not true. The question is what was or was not 
> standardized in the past.
> > The
> > > rules for country code positioning were clearly written 
> and clearly
> > normative.
> > 
> > RFC 3066 states that a 2-letter subtag in the second position is
> > normatively
> > a country code.  It nowhere states the converse, and goes 
> out of its way
> > to say that there are no rules for positions past the second.
> > 
> > --
> > De plichten van een docent zijn divers,         John Cowan
> > die van het gehoor ook.                         
> [email-masked]
> >       --Edsger Dijkstra                         
http://www.ccil.org/~cowan

---

From: "Peter Constable" <[email-masked]>
Date: Sat, 9 Apr 2005 05:04:39 -0700

> that's the only valid form of a language tag.

Where it has made sense to tag documents using language-REGION-whatever,
that will likely continue to be the case, and the RFC as proposed *will*
be harmless wrt those millions of documents. It is only for specific
cases, such as zh-TW that might be affected, but for which zh-TW has
generally be problematic in the past and for which zh-Hant or zh-Hant-TW
really solves problems.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Sat, 9 Apr 2005 05:04:38 -0700

> This is a suggestion of which languages would be most
> appropriate to use a script with.

Wouldn't it be more useful to state in which cases a script does not
normally need to be specified for a given language? E.g. we'd be saying
for en that it's not normally recommended to include "Latn", that it's
not normally recommended to include "Cyrl" for ru, etc.

This is not the same as listing a default script for languages: that was
problematic due to a sense of expectation that some default should be
identified for every language, which has many vagueries; and because
they imply (as you state) a certain omniscience. For the alternative I'm
suggesting here, we'd only be saying something for known cases in which
it was applicable (which would require a lot less space to state than
the other, btw), and we'd only stating it as a recommendation for
general use without assuming what may be needed in all scenarios.

Both may be able to accomplish the same result, but what I'm suggesting
states the recommendation we want users to go away with directly, rather
than them having to infer if by analyzing lists of what languages use
what scripts (and understanding what's implied by how many scripts are
associated with a given language).

> We aren't. We are recommending that specific languages always be used
with
> a script subtag (and we don't define which one per-se) and are
suggesting
> that other languages not use them unless there is some need.

The way you proposed doing this -- listing languages in the Recommended
Prefix field for a script subtag -- does this in an indirect way that
would not be transparent to users, IMO.

Peter Constable

---

From: "Addison Phillips" <[email-masked]>
Date: Sat, 9 Apr 2005 14:27:01 -0700

> Much earlier in this thread, it was suggested to Kurt Zeilenga
> that an LDAP server be updated to infix script tags in EVERY
> record containing language tags (e.g., 'en-GB' --> 'en-Latn-GB').
> If that's legal, then some misguided person may actually write
> a tool to do just that.  Breaking all access by existing clients.

That is a valiant mis-reading of my previous email, at best, if you are referring to my email to Kurt.

I suggested that users who wished to use script subtags would, by necessity, need to use them consistently for some particular language prefix. Either all in or all out, as it were.

This does not suggest that someone write an LDAP implementation to perform the expansion. Only that users choose tags wisely.

Addison

Addison P. Phillips

> Sent: samedi 9 avril 2005 10:46
> To: [email-masked]'; Peter Constable
> Cc: [email-masked]
> > Behalf Of [email-masked]
> > Sent: Saturday, April 09, 2005 11:14 AM
> > To: Peter Constable
> > Cc: [email-masked]
> > > This whole argument is a strawman. The number of cases in
> > which tags of
> > > the form ll-Ssss-CC will be used are limited.
> >
> > You have absolutely no way of knowing that that will be the
> > case. As I pointed
> > out in other mail, script information is something that often
> > can be derived
> > from other data. Country information is often something you
> > have available as
> > well. This makes it entirely possible that such tags will be
> > created in an
> > attempt to make things more informative.
> 
> Since the inclusion or omission of script tags in future language
> tags would at best be a SHOULD in RFC3066bis, no definitive
> statement that the use of script tags will be rare is accurate.
> 
> Much earlier in this thread, it was suggested to Kurt Zeilenga
> that an LDAP server be updated to infix script tags in EVERY
> record containing language tags (e.g., 'en-GB' --> 'en-Latn-GB').
> If that's legal, then some misguided person may actually write
> a tool to do just that.  Breaking all access by existing clients.
> 
> John Cowan gave an example (level 2 Braille) where the script
> tag definitely should be sorted AFTER the region tag.  In order
> to clean up what someday may well be called "The Chinese Mess"
> and provide for the use of script tags elsewhere, you folks are
> maintaining that it's harmless to reorder the subtags.  It's not!
> 
> If script tags SHOULD be rare in use, then it's no burden for
> them to occupy the third position, rather than the second.
> 
> Semi-OT - the inclusion of script tags in language tags breaks
> utterly the utility of presenting language choices to users in
> GUIs as country-flag icons - the script differences would be
> invisible.
> 
> Cheers,
> - Ira
>

---

From: "L.Gillam" <[email-masked]>
Date: Mon, 11 Apr 2005 10:08:13 +0100

> FYI the logical network multilingual layers come in the 
> following order :
> 
> - script so you can write each other strings
> - language so you can read each other commands

Commands? Do you mean e.g. SMTP commands? That's hardly a
"natural" language.

> - geolegal so you know the applying rules ... this permits 
> interoperationality

Laws apply to my use of language? Beyond libel, what is there?

> - referent so you can understand each other ... this permits 
> interintellegibility

Unclear what this "referent" is.

> - the vernacular style/practice so you can work together ... 
> this permits 
> interusability.

Again unclear.

> If you really want to make it a formated langtag it should be 
> script-language-country (in your parlance  [4-]3-2 character 
> strings, with 
> 2 characters being acceptable in 1st position and script 
> preposition being 
> optional. This is what we should have in the IDNA and the big 
> mistake was 
> to put it in a different order, not understanding that the 
> Internet is 
> actually the ASCII class of the Multilingual Internet system.
> 
> jfc

Until this message gets registered, "Multilingual Internet system" is
a one-hit-wonder (google).

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 11 Apr 2005 09:36:54 -0700

> From: [email-masked] [mailto:[email-masked]]

> > It is a completely invalid to say that "backward compatibility" is
being
> > broken.
> 
> On the contrary, it is completely demonstrable that that is what is
happening.
> 
> > That assumes that backward compatibility is defined in terms of
> > certain expectations that have never been guaranteed: that if a
second
> > subtag exists, it would be a country ID.
> 
> This isn't the expectation.
> 
> > The appropriate way to present
> > an argument on this issue is to say that some specific functionality
> > would not exist, and then to evaluate in relation to realistic
scenarios
> > just how great a loss that would be.
> 
> I disagree. It's a big world out there, and it is next to impossible
to figure
> out what people have done or will do.

Ned, you *must* define a specific functionality that isn't working
rather than simply declare "back compat is broken": back compat can be
defined in different ways, and someone else can (and some have) declared
that backward compatibility is maintained with equal validity. If you
insist on making blanket statements like this, you will IMO be blocking
progress.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 11 Apr 2005 09:49:53 -0700

> From: McDonald, Ira [mailto:[email-masked]]

> GUIs as country-flag icons - the script differences would be

There's not much utility in that UI device when a script distinction
exists and trumps any country distinction. It's not a very useful device
in general, anyway. What language is an icon of the Swiss flag supposed
to represent?

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 11 Apr 2005 13:32:19 -0700

> From: [email-masked] [mailto:[email-masked]]
On Behalf Of
> Frank Ellermann

> > quite different from the unqualified statement "backwards
> > compatibility is broken".
> 
> It's related...

You miss my point. Arguing "the current proposal breaks backward
compatibility" is not constructive as the terms are wide open to
different interpretations. The discussion had turned to considering a
specific issue -- the impact of inserting a script ID in ll-Ssss-CC in
cases for which ll-CC is commonly used -- on which we can make progress
because the specific functional behaviours and expectations are clear.

Peter Constable

---

From: "Peter Constable" <[email-masked]>
Date: Mon, 11 Apr 2005 15:31:03 -0700

On Behalf Of
> Randy Presuhn

> > But then you have a
> > significant usability problem: should my current en-GB content be
tagged
> 
> (alternatives elided)
> 
> > This scheme requires far more user guidance than what is currently
> > proposed, and more than is feasible: it requires telling them how to
> > judge e.g. whether a country distinction reflects a distinction in
> > language variety, a distinction in written form, or both.
> ...
> 
> True, but these are distinctions that have been causing us grief.

IMO, the distinctions we'd have to explain in an RFC would cause much
more grief than the relatively minor grief we've been debating.

Peter Constable

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 12 Apr 2005 08:34:06 -0700

Randy Presuhn <randy underscore presuhn at mindspring dot com> wrote:

> For example, "Microsoft are on their best behaviour" shows en-GB
> language (the use of "are" and "their" for a collective noun, which
> would be ungrammatical in en-US) and en-GB orthography (the
> spelling of "behaviour" rather than "behavior"), all in Latin script.

My favorite example was in a TV recap of an English soccer game:  "But
on forty-three minutes, United were level."  I tried to count the number
of differences in this short en-GB example from how an en-US speaker
would say it, and ended up with five.

The sports world often supplies jargon that is highly region-specific.
Phillips flew out to Orange County, but Bonds flied out to left field.
And no, I don't propose tagging to differentiate these.

-Doug Ewell
 Fullerton, California