Subject: RE: [Ltru] Re: Compatibility with existing use (LDAP)

From: "McDonald, Ira" <[email-masked]>
Date: Thu, 7 Apr 2005 09:44:26 -0700

Hi,

Buggy and nonconformant?  Most matching software dates to
RFC 1766 days and the 'language-REGION' form was clearly
the intended/expected style.

I don't buy the importance of 'en-Brai' versus 'en-US-Brai'.
Very few existing Internet resources are tagged with script.  
Compatibility really is important. 

Human users currently know that 'en-GB' _means_ 'en-Latn-GB',
unless they've changed their browser options.

Cheers,
- Ira

PS - I know I'm going to lose this argument, but I feel
honor bound to point out the pain these script tags in
the 'right' place are going to cause.  All old matching
software won't know what to do with them.  And upgrading
both servers and clients simultaneously (e.g, for LDAP
directories in enterprises) will simply never happen.

Ira McDonald (Musician / Software Architect)
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

> -----Original Message-----
> From: John Cowan [mailto:[email-masked]]
> Sent: Thursday, April 07, 2005 12:32 PM
> To: McDonald, Ira
> Cc: [email-masked]
> Subject: Re: [Ltru] Re: Compatibility with existing use (LDAP)
> 
> 
> McDonald, Ira scripsit:
> 
> > I don't believe that this quiet insertion of script between
> > language and region is harmless to RFC 3066 implementations.
> > 
> > It's true that such tags were allowed to be registered by
> > RFC 3066.  But there are literally millions of documents 
> > currently language tagged on the Internet in the style of 
> > 'language-REGION-whatever' and lots of naive software thinks
> > that's the only valid form of a language tag.
> 
> Buggy and nonconformant software, I think you mean.  In any case, the
> expectation is that such tags will be relatively rare, and only used
> where needed.
> 
> > The linguistic purity of infixing 'script' (between language
> > and region) is unwise.  The recent (all unsuccessful) numeric 
> > weighting algorithms for 'smart matching' make this point 
> > eloquently.
> 
> It's not linguistic purity, it's practicality.  A language in 
> a script you
> can't read, like en-Brai, is much worse than one from a 
> different region.
> 
> -- 
> Híggledy-pìggledy / XML programmers            John Cowan
> Try to escape those / I-eighteen-N woes;        
http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      http://www.reutershealth.com
Unicode weenies and / François Yergeaus.        [email-masked]

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 7 Apr 2005 13:02:09 -0400

> the intended/expected style.

Even in RFC 1766, language-tag processors had to be able to accept
registered tags.  The main difference between 1766 and 3066 is that
3066 allows 3-alpha ISO 639-2 subtags.

Note that the differences can be huge: if you request zh content, you
typically only want Mandarin, and if you get zh-yue instead, you will
be in the same position as someone requesting fr and getting ro.

> Compatibility really is important. 

IIRC, Braille level 2 is actually different between en-US and en-GB in ways
that interfere with intelligibility.  (Level 1 Braille, just the letters, is
compatible.)

However, the intended application of script tags is in languages that
are routinely written in more than one script.  So far, this has either
been pushed under the rug or handled (in the case of zh) by abusing
the region code.  Separating out script from region is essential for
reliable discrimination in these cases.  The vast majority of content
will probably never use a tag that has a script subtag, and the overuse
of subtags is specifically discouraged.

> unless they've changed their browser options.

Absolutely.  It's the people who are trying to work in Azerbaijani that need
to care about this.

> directories in enterprises) will simply never happen.

Most of the time it shouldn't need to happen.

-- 
John Cowan  [email-masked]  www.reutershealth.com  ccil.org/~cowan
Dievas dave dantis; Dievas duos duonos          --Lithuanian proverb
Deus dedit dentes; deus dabit panem             --Latin version thereof
Deity donated dentition;
  deity'll donate doughnuts                     --English version by Muke Tever
God gave gums; God'll give granary              --Version by Mat McVeagh

---

From: [email-masked]
Date: Thu, 07 Apr 2005 10:22:35 -0700 (PDT)

> directories in enterprises) will simply never happen.
 
As you might expect, I am in complete agreement with Ira here. This break with
backwards compatibility is going to cause a bunch of problems. I also find the
assertion that implementations done in good faith to what was specified in RFC
1766 are "buggy and nonconformant" to be a bit disturbing.

Speaking as a coauthor of MIME, I know better than most what sorts of
compromises are required to make things backwards compatible. But having
watched various attempts to do multimedia email prior to MIME fail due in large
part to their lack of backwards compatibility, I also know what can happen if
you don't hit the sweet spot on these matters. 

Balkanization of how language tags are used is the last thing we want, and I
think this is where we're going to end up if we break compatibility.

				Ned

---

From: John Cowan <[email-masked]>
Date: Thu, 7 Apr 2005 13:52:16 -0400

> 1766 are "buggy and nonconformant" to be a bit disturbing.

Implementing RFC 1766 while ignoring IANA-registered tags is about
like implementing C while ignoring the "goto" statement.  Most programs
compile.

> you don't hit the sweet spot on these matters. 

The only way to ensure 100% backward compatibility is to change
nothing.  Putting the script subtag after the region subtag exchanges
one incompatibility for another, as Doug points out.  What is more, we
already have some language-script-region tags registered under RFC 3066
(IANA hasn't posted them yet), so it's a fait accompli.

> think this is where we're going to end up if we break compatibility.

We are not breaking compatibility.

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.reutershealth.com
And now here I was, in a country where a right to say how the country should
be governed was restricted to six persons in each thousand of its population.
For the nine hundred and ninety-four to express dissatisfaction with the
regnant system and propose to change it, would have made the whole six
shudder as one man, it would have been so disloyal, so dishonorable, such
putrid black treason.  --Mark Twain's Connecticut Yankee

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 07 Apr 2005 21:13:53 +0200

John Cowan wrote:

> The only way to ensure 100% backward compatibility is to
> change nothing.

IMHO that's wrong, MIME is ingenious (excl. 2231).

> Putting the script subtag after the region subtag exchanges
> one incompatibility for another, as Doug points out.

If it makes no big difference for 3066bis implementations and
helps 3066 implementations I'd like it.

Or if it's better for the interoperability of "old" (all) and
"new" (future) implementations.

> What is more, we already have some language-script-region
> tags registered under RFC 3066 (IANA hasn't posted them yet),
> so it's a fait accompli.

That's no argument, "not-yet-posted" grandfathered tags could
be deprecated and replaced by a better solution, there won't
be millions of say iu-Cans pages.

> We are not breaking compatibility.

Maybe we're not breaking it radically enough - pretending that
all is fine where in fact the new scheme has problems with the
old scheme.  I'd be interested to see different schemes like
Debbie's 639-4.

Maybe it's better than this obscure "don't use script tags
unless you must, but we don't tell you when you must and when
you shouldn't" (= no default script) rule.

                             Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Thu, 7 Apr 2005 15:35:21 -0400

> IMHO that's wrong, MIME is ingenious (excl. 2231).

How so?  Pre-MIME MUAs will display rubbish when sent MIME messages, and
nothing can stop it.

> helps 3066 implementations I'd like it.

It does make a big difference:  incompatible script is much worse than
incompatible region, as you yourself pointed out in a different connection.
3066 implementations will not be helped, because it's already the case
under existing 3066 tags that a request for az-Cyrl will not match az-AZ.

> be millions of say iu-Cans pages.

There won't be millions of az-Arab or az-Latn pages either.

> old scheme.  I'd be interested to see different schemes like

639-6, you mean.  No fallback at all: each language variety has a
four-letter tag, and you *have* to have the fallback tables to interpret
one that you don't know directly.

> you shouldn't" (= no default script) rule.

Not the same thing.  Perhaps we should indeed add a non-normative
Recommended_Script field to those few languages that will normally use
script tags.

-- 
Long-short-short, long-short-short / Dactyls in dimeter,
Verse form with choriambs / (Masculine rhyme):  [email-masked]
One sentence (two stanzas) / Hexasyllabically   http://www.reutershealth.com
Challenges poets who / Don't have the time.     --robison who's at texas dot net

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 07 Apr 2005 23:04:39 +0200

> How so?  Pre-MIME MUAs will display rubbish when sent MIME
> messages, and nothing can stop it.

No, they will display plain text US-ASCII, that's not rubbish.

And pre-MIME MTAs can completely ignore the "rubbish" like the
weird Subject: in this article (intentionally something your
MUA most probably doesn't know, for a pre-MIME experience).

MIME is really ingenious, only RfC 2231 has some dark corners.

> incompatible script is much worse than incompatible region

Sure, but only "new" implementations support it at all, and so
they can do the right thing no matter where the script tag is.

>> different schemes like Debbie's 639-4.
> 639-6, you mean.

Yes.  I've no clear idea what it's about, but there can't be
compatibility issues with 3066 if it's radically incompatible.

                          Bye, Frank

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Fri, 08 Apr 2005 01:41:44 +0200

>think this is where we're going to end up if we break compatibility.

Ned,
full agreement. I delay the Draft I intended to present, ir order to follow 
this. Our approach is to say that BCP 047 should provide a framework to 
support application related formats. The different formats would then be 
easily supported. RFC 1776/3066 would continue to be accepted and the new 
format proposed by APMD could be recommended for HTML/XML/CLDR 
applications. This would not affect existing applications. This would also 
support any new need, without a risk of balkanization or even of 
fragmentation.
jfc

---

From: [email-masked]
Date: Thu, 07 Apr 2005 14:31:43 -0700 (PDT)

> like implementing C while ignoring the "goto" statement.  Most programs

The analogy is a stretch in and of itself, but it is also irrelevant since the
basis of comparison is a strawman. We're not talking about incompliant
implementations here.

A compliant RFC 3066 implementation can (and I know for a fact has) been
written that handles these exceptions through an exception table, but still
operates under the assumption that a two letter code in the second position is
a country code but a two letter code elsewhere is not.

Like it or not, there are lots of application domains where script information
is much less readily available than country information. As such, it is
entirely reasonable for these implementations to rely on extraction of country
code information for the purposes of making finer grained matches. And this
proposal breaks this.

> nothing.

Not true. The question is what was or was not standardized in the past. The
rules for country code positioning were clearly written and clearly normative.

The prefix matching rule, OTOH, was specifically called out as NOT necessarily
being generally applicable in RFC 3066.

> one incompatibility for another, as Doug points out.

I'm sorry, but I don't have the time to dig out what Doug said somewhere else.
I assume he was referring to the fact that putting script codes after the
country code breaks the short-test-prefix-match scheme in cases where both
script and country code information is available and the script is more
relevant.

The difference is that there was never any guarantee given that the
shortest-prefix-match approach would work well in all cases.

> What is more, we

Actually, sgn- created an exception situation long ago. (I suppose a fair
number of implementations could safely ignore sgn- given the limited realm of
applicability of sign languages, but others cetainly cannot.)

> We are not breaking compatibility.

You're effectively moving country codes to a new position in the string. That's
breaking compatibility in my book. And since you referred to this previously as
an incompatibility, I guess you agree...

In any case, like Ira, I don't expect to win this argument. Nor do I have the
time to argue the point at the level of frequency that is clearly needed on
this list to make any sort of difference.

				Ned

---

From: John Cowan <[email-masked]>
Date: Fri, 8 Apr 2005 13:17:35 -0400

> a country code but a two letter code elsewhere is not.

An assumption quite without warrant in the text of RFC 3066.

> is much less readily available than country information. 

The intention (as Addison has developed in his messages on the subject) is
to use script information only for those languages where it is commonly
required.

> rules for country code positioning were clearly written and clearly normative.

RFC 3066 states that a 2-letter subtag in the second position is normatively
a country code.  It nowhere states the converse, and goes out of its way
to say that there are no rules for positions past the second.

-- 
De plichten van een docent zijn divers,         John Cowan
die van het gehoor ook.                         [email-masked]
      --Edsger Dijkstra                         http://www.ccil.org/~cowan

---

From: [email-masked]
Date: Fri, 08 Apr 2005 10:34:07 -0700 (PDT)

> An assumption quite without warrant in the text of RFC 3066.

On the contrary, I believe the RFC is very clear on this . It says:

 The following rules apply to the second subtag:

   - All 2-letter subtags are interpreted as ISO 3166 alpha-2 country
     codes from [ISO 3166], or subsequently assigned by the ISO 3166
     maintenance agency or governing standardization bodies, denoting
     the area to which this language variant relates.

It also says that country codes can appear in other places:

 The information in a subtag may for instance be:

   - Country identification, such as en-US (this usage is described in
     ISO 639)

Given this text, I believe an implementation seeking to perform matches
based on country codes is compliant if:

(1) It treats a two letter code in the second position as a country code.
(2) It uses an extensible exception table to enumerate cases where
    country codes appear in other places that should be matched.

Please explain why such an implementation is incompliant.

> to use script information only for those languages where it is commonly

This has never been made clear, and as I have said elsewhere, simple directives
along these lines are often ignored.

> to say that there are no rules for positions past the second.

Which, as I have stated over and over and over, can be readily handled with
a table.

				Ned