Subject: RE: [Ltru] Re: Registry in record-jar format

From: "Addison Phillips" <[email-masked]>
Date: Mon, 4 Apr 2005 12:32:17 -0700

> Yes, and it's impossible to come to a final conclusion "now"
> (like say today), because there is no "smart matching draft".

That's not really accurate.

We can debate the matching problem for 3066bis language tags without having the matching I-D finished. The basic questions are:

1. Does the insertion of script subtags between language and region subtags constitute a problem for matching user requests to existing (remove-from-right [RFR]) RFC 3066 implementations?

Answer: Yes, it is a problem, since users might have to specify two ranges or may get results inconsistent with their expectations. For example, the range "zh-TW" does not match the tag "zh-Hant-TW" using strict 3066 matching rules.

2. Does placing the script after the region solve the problem?

Answer: No, because variants also pose a positioning problem (of lower frequency) and, more importantly, because users who need to select content where script is more important have to know every potential region subtag they are interested in. For example, the range "zh-Hant" doesn't match the tag "zh-TW-Hant" using strict 3066 matching rules.

Once we admit more than two subtags, you need at least extended range matching or some requests will produce results inconsistent with user expectations.

In fact, RFC 3066 already has the subtag ordering problem. It is just that there are relatively few registered tags that display it. Notably, the German orthographic variations produce this exact pattern of problem.

3. Is there a proposal to solve this?

Answer: Yes. There are several.

i. "Extended Range Matching" is an extension of RFR in which "missing" subtags are considered to be (or are expanded to be) wildcards. So "de-1901" is expanded to "de-*-*-1901" and matches "de-Latn-1901" and "de-AT-1901", not to mention "de-Latg-NA-1901".

NB> Probably we need to extend this scheme to allow users to specify that they do NOT want a specific field to be filled in. For example "de-!-1901" would match only "de-1901" and not a tag such as "de-Latn-1901".

ii. "Lookup" is the opposite of matching, in which subtags are removed from the tag rather than the range (i.e. locale-style) to fill in all slots in a dataset. This matches a very different content set.

iii. "Scored Matching" was proposed by John Cowan and produces content with a range of scores, allowing the content to be filtered by choosing a "quality level" (my words) for the match. The user can set a threshold for matching, presumably.

The problem here is that matching cannot be as simple as it was under RFC 3066 and still capture all possible cases.

The alternatives as I see it would be:

a. Force script to appear at the end of the subtag sequence, but write new matching rules that allow processors to consider script in its more appropriate position (after language). 

I think this creates enormous processing issues for very little gain: RFR matching works for existing requests, but not for new content. Processing in cases such as CSS 2.1 or XML is greatly increased for no apparent benefit and at the cost of more obscure tag syntax.

b. Make RFC 3066bis into an extension itself (for example "de-1901-i-Latn").

This is just "re-spelling" the problem in (a). It fixes #1 above, but not #2 for RFR processors.

c. Go with RFC 3066bis's design and Tag Content Wisely.

This basically says that scripts will only be used when they contribute something of value in the tag (or that they will be used more-or-less universally). Users who don't adapt to this for languages (such as Chinese or Serbian) where content will be semi-universally tagged with scripts get inconsistent results until they adopt implementations that use extended range or some other form of matching. Extended range matching or scored matching may help this transition along, but isn't a pre-requisite for forming tags. RFR works correctly for new-style ranges and produces inconsistent results in some cases. This is the price of transition.

Again, note that any 3066bis tag can be registered under RFC 3066. When more tags with scripts come into existence, the matching problems will be present under RFC 3066 and with no way out for the end user.

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Frank Ellermann
> Sent: lundi 4 avril 2005 10:49
> To: [email-masked]
> Subject: [Ltru] Re: Registry in record-jar format
> 
> Doug Ewell wrote:
> 
> > The thread titled "Preparations for smart matching" was
> > started on March 19 by Frank Ellermann, to introduce the
> > idea of adding "default script."
> 
> Yes, and it's impossible to come to a final conclusion "now"
> (like say today), because there is no "smart matching draft".
> 
> At the moment I only _think_ that default scripts are a MUST
> HAVE, because otherwise en-Latn-GB-scouse, en-GB-scouse,
> en-Latn-scouse, and en-scouse could run into trouble if they
> don't match each other.
> 
> Maybe I'm wrong and this only breaks RTL matching.  Maybe it
> causes major havoc for all Web servers and browsers.  I don't
> know before I see the matching draft.
> 
> > A better approach is to determine first what the entity
> > should do, and then name it accordingly.
> 
> I want it to be used for the default script, because otherwise
> we might create a "language tag soup" and no way to match them
> except from a few trivial cases, where RfC 3066 is good enough
> or even better than RfC 3066bis.
> 
> A viable alternative is a RECOMMENDATION to never omit the
> script or the region in tags with a variant.  I hope that the
> matching draft proposes a better way, but at the moment some
> kind of "default script" is the only solution I see, and the
> "prefix" field of languages is the only place to add this info.
> 
> > In other words, we should (or should not) add the "default
> > script" mappings based on an agreement that it is (or is
> > not) a good thing, not because the name of the field appears
> > to permit it.
> 
> You can't decide this before the real issue (= "matching") is
> clear.  That's like discussing cut-off dates together with the
> language tags (where it's mostly irrelevant) before the strange
> side-effect for region codes are visible.
> 
> Let's just see a new state of the art.  There have been so many
> changes from the "ABNF" over "record-jar" to the "normative
> references" that I lost track.  It's only an I-D, not the last
> chance to get it right.
>                         Bye, Frank
> 
> 
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Mon, 4 Apr 2005 16:03:07 -0400

Addison Phillips scripsit:

> Answer: Yes, it is a problem, since users might have to specify two
> ranges or may get results inconsistent with their expectations. For
> example, the range "zh-TW" does not match the tag "zh-Hant-TW" using
> strict 3066 matching rules.

Just a terminological note: I think these should be called HTTP or RFC 2616
matching rules, since RFC 2616 is the true and authoritative source for them.

> i. "Extended Range Matching" is an extension of RFR in which "missing"
> subtags are considered to be (or are expanded to be) wildcards. So
> "de-1901" is expanded to "de-*-*-1901" and matches "de-Latn-1901" and
> "de-AT-1901", not to mention "de-Latg-NA-1901".

I think this should be mentioned in the matching I-D.

> NB> Probably we need to extend this scheme to allow users to specify
> that they do NOT want a specific field to be filled in. For example
> "de-!-1901" would match only "de-1901" and not a tag such as
> "de-Latn-1901".

I'd like to see a use case for this (not involving boont).

> ii. "Lookup" is the opposite of matching, in which subtags are removed
> from the tag rather than the range (i.e. locale-style) to fill in all
> slots in a dataset. This matches a very different content set.

This too should be mentioned.

> iii. "Scored Matching" was proposed by John Cowan and produces content
> with a range of scores, allowing the content to be filtered by choosing
> a "quality level" (my words) for the match. The user can set a threshold
> for matching, presumably.

As should this.

> The problem here is that matching cannot be as simple as it was under
> RFC 3066 and still capture all possible cases.

This is badly stated.  Only matching of predefined tags in RFC 3066 captures
all cases.

-- 
Deshil Holles eamus.  Deshil Holles eamus.  Deshil Holles eamus.
Send us, bright one, light one, Horhorn, quickening, and wombfruit. (3x)
Hoopsa, boyaboy, hoopsa!  Hoopsa, boyaboy, hoopsa!  Hoopsa, boyaboy, hoopsa!
  -- Joyce, Ulysses, "Oxen of the Sun"       [email-masked]