Subject: RE: [Ltru] Moving Forward

From: "Peter Constable" <[email-masked]>
Date: Tue, 12 Apr 2005 14:00:13 -0700

> From: [email-masked] [mailto:[email-masked]]
On Behalf Of
> Mark Davis

> Coming back to this subject after being gone for a week, it appears to
me
> that there is only one really substantial issue remaining, and that is
the
> issue of what to do with the script. We have gone back and forth with
this
> issue, but it appears like we are spinning our wheels a bit. Addison
and I
> discussed this, and here's what we propose:
> 
> 1. We add one field to the language records:
> 
> Associated_Script: ...

> 2. We leave the script in the current position...

I think this is a good solution to the concerns raised. I would suggest
one minor change: since this field is to be populated only when there
are two or more scripts commonly used, change the label for this field
to "Associated_Scripts".

Peter Constable

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Misha Wolf <[email-masked]>
Date: Wed, 13 Apr 2005 16:46:13 +0100

Peter Constable wrote:

> The one concern for our plan would be what to do if some 
> apparently stable mono-script situation suddenly changes. If we 
> tag "got" today, with Latin script assumed, and "got-Goth" for the 
> unusual scenario, and then next year a reform gets underway with 
> the result that in five years the situation has reversed, then 
> what do we do? 
> 
> Of course, I'm describing another scenario like Turkish, and those 
> aren't all that likely. But a scenario such as going from only 
> Cyrillic today to Cyrillic and Latin next year might not be so far 
> fetched.

Indeed.  At that point we would change the record for the affected 
language to have an Expected_Script_Subtags field.  Old tags would 
be "ll", while new tags would be "ll-Cyrl" or "ll-Latn".  Search for 
"ll" would work fine (ie return all stuff tagged with "ll*"), as 
would searches for "ll-Cyrl" or "ll-Latn".

Misha

-----------------------------------------------------------------
        Visit our Internet site at http://www.reuters.com

To find out more about Reuters Products and Services visit http://www.reuters.com/productinfo 

Any views expressed in this message are those of  the  individual
sender,  except  where  the sender specifically states them to be
the views of Reuters Ltd.

---

From: "M.T. Carrasco Benitez" <[email-masked]>
Date: Wed, 13 Apr 2005 18:06:32 +0100 (BST)

> [Wolf]
> > [Constable]
> [Wolf]

In the very unlikely event that a language goes back to a previous
script and the script subtag does not work anymore, one can handle
this with:

 - New variant
 - New language code

For example, with Turkish:

 - Code before 1500 ?
 - 1500-1928: tr-Arab (I assume); ISO-639-2 "ota" turc ottoman
 - Present: tr (in Latn)
 - Future back to Arab: tr-Arab, hence clash with 1500-1928.

 - New variant: tr-foo
 - New language code: foo

The approach of 3066bis should be sufficient. Otherwise,
we could generalize the Special David-Wolf Proposition.

Regards
Tomas

Send instant messages to your online friends http://uk.messenger.yahoo.com

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 13 Apr 2005 10:54:14 -0700

So which of the alternate designs do you think is the right one? (I'm not asking antagonistically, but because I want to see this item, the last really major one, resolved in *some* way)

I note that every alternative has a downside too. IMO, script is in the best (most logical) position between language and country. 

Moving it into the variant slot (by merging with variant or between region and variant) produces content selection problems (see Misha's message) and fallback problems. And you still have the problem of omitted fields (my German examples, examples with script but no region, etc.). 

Using an extension is appealing in some ways, but would require many currently stupid processors (XML, CSS, etc.) to become at least a little bit smarter about handling subtags (something I'm not too keen on, but maybe it could work).

C'mon folks, we're offering different ways of addressing objections to our design, but no one has offered a contrasting position around which to build consensus. 

I'm open-minded enough to consider any solution and I will abide by whatever consensus emerges which results in us having the ability to distinguish script in a language tag. But please don't just say "don't put it there". Tell us which alternative and why you favor it. Or tell us exactly how our design would have to be modified to be at least minimally acceptable. Then Randy counts noses to see if one has reached critical mass.

That is what we've tried to do with our script proposals: work on overcoming specific individual's most pressing objections in hopes of building (at worst) a grudging harmony.

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of McDonald, Ira
> Sent: mercredi 13 avril 2005 10:39
> To: 'Peter Constable'; [email-masked]
> Subject: RE: [Ltru] Re: Moving Forward
> 
> Hi,
> 
> A good deal of digital ink is being expended on a problem
> (Default/Associated/Expected Script) that _only_ occurs
> because RFC3066bis is infixing script between language
> and region.
> 
> And there remains every possibility that a software vendor
> who sells a widely used application for mounting content
> on Web servers will ignore that SHOULD NOT about using tags
> like 'en-Latn-GB' and wreak a fair amount of havoc.
> 
> Cheers,
> - Ira
> 
> Ira McDonald (Musician / Software Architect)
> Blue Roof Music / High North Inc
> PO Box 221  Grand Marais, MI  49839
> phone: +1-906-494-2434
> email: [email-masked]
> 
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]]On
> > Behalf Of Peter Constable
> > Sent: Wednesday, April 13, 2005 11:23 AM
> > To: [email-masked]
> > Subject: RE: [Ltru] Re: Moving Forward
> >
> >
> > On
> > > Behalf Of John Cowan
> > > Note that the purpose of Associated_Script is
> > > not to tell an implementer the usual script used with a
> > language, but
> > > simply to serve notice which script *subtags*, if any,
> > should normally
> > > be used with this language subtag.  This difference is fundamental.
> >
> > We could make it more obvious, though. What about changing
> > the label for
> > the field to Expected_Script_Subtags? (I.e. 'it is expected
> > that one of
> > the following script subtags will also be used'.)
> >
> >
> > > The specific answer is that for centuries Gothic has been written
> > solely
> > > in the Latin script.  (This may start to change with the presence of
> > > Gothic in Unicode and the ready availability of Gothic fonts, or it
> > > may not.)  Therefore, no Associated_Script header should be present,
> > > and the tag got-Latn should be avoided (though got-Goth would be in
> > > order as an unusual script).
> >
> > The one concern for our plan would be what to do if some apparently
> > stable mono-script situation suddenly changes. If we tag "got" today,
> > with Latin script assumed, and "got-Goth" for the unusual
> > scenario, and
> > then next year a reform gets underway with the result that in
> > five years
> > the situation has reversed, then what do we do?
> >
> > aren't all that likely. But a scenario such as going from
> > only Cyrillic
> > today to Cyrillic and Latin next year might not be so far fetched.
> >
> >
> >
> > Peter Constable
> >

---

From: "Peter Constable" <[email-masked]>
Date: Wed, 13 Apr 2005 11:53:33 -0700

> From: McDonald, Ira [mailto:[email-masked]]

> like 'en-Latn-GB' and wreak a fair amount of havoc.

I find these kinds of statements (just like "breaks back compat") to be
so vague as to be unconstructive. This is particularly concerning
because the statements are also alarmist in tone. I have no qualms with
identifying serious problems with a proposal if it can be clearly shown
what the problem is, that the problems are valid and have a reasonable
likelihood of occurrence.

Could we please qualify and quantify "a fair amount of havoc"? For a
worst case scenario, what might the negative impact be to the vendor, to
customers using that application to mount content on Web servers, and to
end users interacting with those Web sights?

For instance, I can easily imagine cases in which none of the parties
involved notice that there has been some loss of functionality; and
other scenarios in which loss of functionality is immediately noticed,
prompting immediate corrective measures, during the course of which the
vendor takes time to read the RFC and implement correctly, as they
should have done in the first place.

Peter Constable

---

From: "Addison Phillips" <[email-masked]>
Date: Wed, 13 Apr 2005 15:02:41 -0700

The selection pairing:

Range= en-Latn
Content= en-US-Latn

results in no matching in LDAP and other RFR/RFC 2616 matching scenarios, unless you change the matching algorithm (at which point you could have put script second...) to deal with the interpolated subtag "US".

I think your point is that the user explicitly did not select a region in my example above, which is true. The problem is that the cases where script is important, the region code tends to be something one would want to strip first. Consider:

Range= zh-Hant
Content= zh-HK-Hant
Content2= zh-TW-Hant

RFR matching only works if the same fields are either all full or all empty for a given subtag. Neither script in second or script in third fixes all cases, only certain sets of cases.

Notice that the fallback from range "en-Latn-US" for "en-US" tagged content is "en" (for fallback type matching as in Apache or in my J2EE example). This is also true for range "en-US" and "en-Latn-US" tagged content. The work around is the same in both cases: to create content on the "mezzanine" level of "en". (Notice that this *is* a problem for languages that have multiple scripts like Chinese and which really should use a script subtag: the vector really should go through the script before falling to a bare language subtag because the language content could be in either script and realistically is only in one of the two or more scripts.)

For Web applications, I think that we can assume that users will not go modify their language preferences (Accept-Language) just because a specific page returns the "wrong" language for their specified range or the empty set of content for a given URI, just as you say. They will just curse the 404 and move on. Most implementations do not look at the second and subsequence Accept-Language value, so setting both "en-Latn" and "en" doesn't do any good. It is important to harmonize the values that content and the associated ranges use. And the subtags (given that we use a script subtag) will produce erroneous behavior in both script-second and script-third cases (but in different circumstances for each).

Hence our attempts to regularize those languages that need script or allow script only in languages that need it or some similar formulation. I think this would still be necessary with a script-comes-third approach anyway. If we do a good enough job of this, then we can choose the most logical design (which for me is script-second).

Does this explanation (and those of others) sway you in the direction of (accepting) script second?

Addison

Addison P. Phillips

> Sent: mercredi 13 avril 2005 12:50
> To: Addison Phillips; McDonald, Ira; Peter Constable; [email-masked]
> With no disrespect meant, I'll ignore those blithe responses from
> Mark and Peter who seem to think the problem (loss of region utility)
> will never happen or will always get quickly fixed by some vendor.
> 
> [The 'havoc' I meant was the complete loss of lookup capability for
> a deployed HTTP client application that has a configured 'en-GB' tag
> and tries to read Web content from an 'en-Latn-GB' tagged HTTP server.
> And the weeks of problems before "support" fudges some fix - since
> "support" didn't put the 'en-Latn-GB' tags there - the crummy content
> distribution application did.]
> 
> To Addison's question I'll answer more seriously:
> 
> Make script _third_ and allow (as always) the omission of the _second_
> position region.  That format has the desirable property that 'en-Latn'
> means "English in Latin script, don't care about region" (because
> region was explicitly omitted).  That can never cause breakage in
> existing LDAP and other applications because of displacement of
> region and still gets you exactly what you want for script primacy.
> 
> Cheers,
> - Ira
> 
> > From: Addison Phillips [mailto:[email-masked]]
> > Sent: Wednesday, April 13, 2005 1:54 PM
> > To: McDonald, Ira; Peter Constable; [email-masked]
> > So which of the alternate designs do you think is the right
> > one? (I'm not asking antagonistically, but because I want to
> > see this item, the last really major one, resolved in *some* way)
> >
> > I note that every alternative has a downside too. IMO, script
> > is in the best (most logical) position between language and country.
> >
> > Moving it into the variant slot (by merging with variant or
> > between region and variant) produces content selection
> > problems (see Misha's message) and fallback problems. And you
> > still have the problem of omitted fields (my German examples,
> > examples with script but no region, etc.).
> >
> > Using an extension is appealing in some ways, but would
> > require many currently stupid processors (XML, CSS, etc.) to
> > become at least a little bit smarter about handling subtags
> > (something I'm not too keen on, but maybe it could work).
> >
> > C'mon folks, we're offering different ways of addressing
> > objections to our design, but no one has offered a
> > contrasting position around which to build consensus.
> >
> > I'm open-minded enough to consider any solution and I will
> > abide by whatever consensus emerges which results in us
> > having the ability to distinguish script in a language tag.
> > But please don't just say "don't put it there". Tell us which
> > alternative and why you favor it. Or tell us exactly how our
> > design would have to be modified to be at least minimally
> > acceptable. Then Randy counts noses to see if one has reached
> > critical mass.
> >
> > That is what we've tried to do with our script proposals:
> > work on overcoming specific individual's most pressing
> > objections in hopes of building (at worst) a grudging harmony.
> >
> > Addison
> >
> > Addison P. Phillips
> > [mailto:[email-masked]] On
> > > Behalf Of McDonald, Ira
> > apparently

---

From: "McDonald, Ira" <[email-masked]>
Date: Thu, 14 Apr 2005 08:05:43 -0700

Hi,

Peter - your proposal appears to be wobbling between
(1) One new field 
    - ExpectedScript (complete list)
(2) Two new fields 
    - DefaultScript (do NOT use a script subtag)
    - ExpectedScript (not including DefaultScript value??)

If this registry (and RFC3066bis) don't make VERY clear
when script subtags MUST NOT be used, then we're making
no headway.

Cheers,
- Ira         

> Sent: Thursday, April 14, 2005 10:20 AM
> To: LTRU Working Group
> > Behalf Of Doug Ewell
> > I do think that anyone proposing that the initial registry contain
> > default-script information for all 479 ISO 639-1 and -2 languages
> 
> That is not what Frank or I have suggested. A default-script 
> field only
> needs values for languages like English for which we don't want people
> to use script subtags in general.
> 
> 
> 
> Peter Constable
>

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 14 Apr 2005 08:15:22 -0700

All,

I'm working on a prototype revision to section 2.3 (Choice of Language Tag) that encapsulates this thread. I'll post it later this morning on this list.

I think it addresses your concerns Ira without being too onerous to implement in the registry.

Addison

Addison P. Phillips

> Sent: jeudi 14 avril 2005 08:06
> To: 'Peter Constable'; LTRU Working Group

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 14 Apr 2005 11:12:10 -0700

> From: John Cowan [mailto:[email-masked]]

> Since almost all 639-2 languages are written, and since the great
> preponderance
> of those are written in just one script (things like Braille and IPA
> aside),
> that means that, indeed, almost all languages will require the
default-
> script
> field to be populated.
> 
> Addison's proposal involves populating only those languages
customarily
> written in two or more scripts.  This is much easier to deal with
> logistically.

Logistically, you're talking about a one-time process of putting in
values. It is the process of developers looking at the registry and
trying to glean from it what they should do that will be repeated many
many times and is where errors may occur.

Peter Constable

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 14 Apr 2005 15:29:21 -0700

You don't, but you don't need to.

The rule for that language is "don't use a script subtag" (unless it adds something valuable to the tag). Whether there is a default script or the language just isn't customarily written or not is not important in that context.

The only place where default script *is* important is in situations in which a range such as "ysr-RU" is being matched to content that uses a script subtag (let's say it's "ysr-Cyrl-RU"). But extended language ranges ("ysr-*-RU") handle that case just as well. The benefit to a Default_Script could be that implemented in an API of course (for tag "ysr-RU", what is the result of getScript() in my little Java program? Now it is "", but it could be "Cyrl" I suppose). 

Notice that the only processors (as opposed to people) that can gain any advantage from these fields are also implementations of RFC 3066bis. RFC 3066 processors only benefit from humans choosing to include or not include the subtag consistently. As a result, the processors that gain advantage from the field already know what the fields values mean (by their length) and thus aren't benefiting all that much from the information about script use anyway. John's example of warning users not to use a particular subtag, though, is a good one.

In fact, I would say that once you have some content in a language--even one that is "strongly scripted", such as English or French--that uses a script subtag, all of your content should start to use script subtags for consistency of processing *in that context*. The script problem we're trying to deal with usually occurs when unequal fields are being compared ("en-US" and "en-Brai-US").

Addison

Addison P. Phillips

> Behalf Of Debbie Garside
> Sent: jeudi 14 avril 2005 14:53
> To: 'Mark Davis'; 'John Cowan'
> Cc: [email-masked]
> Mark Davis wrote:
> > Suppose that I am querying a database of phrase books for Thai. They
> > contain
> > a mixture of those using Latin and those using Thai script. I can issue
> 3
> > queries:
> >
> > 1. th-Thai - give me only the ones in Thai, and written in the Thai
> script
> > 2. th-Latn - give me only the ones in Thai, using a Latin transcription
> > 3. th - give me anything.
> >
> > This is a perfectly sensible thing to do. But if #1 is illegal, it does
> > not
> > let users make a perfectly useful discrimination between #1 and #3.
> >
> > The right thing to do is to indicate, as others have indicated, is that
> in
> > the normal case (NC) for stuff like webpage selection, you should not
> use
> > the script unless there are two scripts listed in the registry.
> 
> I agree with this, it seems perfectly logical... but...
> 
> If the tag for Sirenik is ysr-RU how do you know what the default script
> is?
> 
> Have I missed something here?
> 
> Debbie
> 
> 
> > Behalf Of Mark Davis
> > Sent: 14 April 2005 20:47
> > To: John Cowan
> > Cc: [email-masked]
> > Subject: Re: [Ltru] Re: Moving Forward
> >
> > 1. I don't see that separating out D from C adds anything. But if you
> have
> > a
> > use-case, that would be interesting.
> >
> > > From what I can understand, which is by no means sure, the advocates
> > > for B information want it so that they can vet tags like en-Latn(-GB)
> > But for that, for the NC, they don't need the information that the
> default
> > script is Latn. All they need to know is that if there are two scripts
> in
> > the registry, then in that situation put no script in the tag.
> >
> > > I believe the desire of this part of the WG is to reject en-Latn
> > > absolutely (because it is the default) while still retaining the
> > > possibility of ko-Latn.
> > *Requiring* the default to be rejected is absolutely the wrong thing to
> > do.
> > Suppose that I am querying a database of phrase books for Thai. They
> > contain
> > a mixture of those using Latin and those using Thai script. I can issue
> 3
> > queries:
> >
> > 1. th-Thai - give me only the ones in Thai, and written in the Thai
> script
> > 2. th-Latn - give me only the ones in Thai, using a Latin transcription
> > 3. th - give me anything.
> >
> > This is a perfectly sensible thing to do. But if #1 is illegal, it does
> > not
> > let users make a perfectly useful discrimination between #1 and #3.
> >
> > The right thing to do is to indicate, as others have indicated, is that
> in
> > the normal case (NC) for stuff like webpage selection, you should not
> use
> > the script unless there are two scripts listed in the registry.
> >
> >
> > ‎Mark
> >
> > ----- Original Message -----
> > From: "John Cowan" <[email-masked]>
> > To: "Mark Davis" <[email-masked]>
> > Cc: <[email-masked]>
> > Sent: Thursday, April 14, 2005 11:34
> > Subject: Re: [Ltru] Re: Moving Forward
> >
> >
> > > Mark Davis scripsit:
> > >
> > > > A. We know the language is customarily used with more than one
> script
> > > > B. We know the language is customarily used with one script
> > > > C. The information for A or B is not available in the registry:
> either
> > just
> > > > not entered yet, or hard to find out.
> > >
> > > You omit:
> > >
> > >   D. The language is not customarily used with any script.  (Any
> > language
> > may
> > >   be transcribed in writing, as any language may be spoken over the
> > radio;
> > >   but just the same, Chinese is written and Burushaski is not, as
> Navajo
> > is
> > >   regularly broadcast and Akkadian is not.)  This does not apply to
> > current
> > >   639-2 languages but will apply to 639-3 languages.
> > >
> > > > As I said before, I have not yet seen an implementation scenario
> where
> > we
> > > > need the infomation in B, and both the strategies that use it
> require
> > > > entering in much more data than the ZZ scenario. So I'd like to see
> a
> > > > scenario that drives B clearly stated.
> > >
> > > From what I can understand, which is by no means sure, the advocates
> > > for B information want it so that they can vet tags like en-Latn(-GB)
> > > and warn against or reject them.  Without B information we cannot
> > > distinguish between en-Brai (anomalous, so correct) and en-Latn
> > (undesirable).
> > >
> > > > 2. As Peter says, we mustn't say MUST NOT, however: there are
> > scenarios
> > in
> > > > which it may be appropriate to use en-Latn or ko-Latn.
> > >
> > > I believe the desire of this part of the WG is to reject en-Latn
> > > absolutely (because it is the default) while still retaining the
> > > possibility of ko-Latn.
> > >
> > > --
> > > All Gaul is divided into three parts: the part          John Cowan
> > > that cooks with lard and goose fat, the part
> > www.ccil.org/~cowan
> > > that cooks with olive oil, and the part that
> > www.reutershealth.com
> > > cooks with butter. -- David Chessler
> > [email-masked]
> > >
> > >
> >
> >
> >