Subject: [Ltru] Re: Registry in record-jar format

From: "Doug Ewell" <[email-masked]>
Date: Mon, 4 Apr 2005 08:40:57 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> If default script is going to be in the registry, its
>> behavior needs to be specified in the draft and in the
>> matching draft
>
> Yes.  But so far there is no matching draft, and the subject of
> the corresponding thread was "preparations for smart matching".

The thread titled "Preparations for smart matching" was started on March
19 by Frank Ellermann, to introduce the idea of adding "default script."

> OTOH changing all "Prefix" to "Related" later (if necessary)
> would be only a minor point before the "last call" of the draft
> registry and registry draft.  Only changing this later would be
> a bad idea.

Changing the name of something *so that* it can be used for a different,
or expanded, purpose is not a good way to design standards or protocols.
We could call the field "Frank" and it would have no impact on the
things we could do with that field.

A better approach is to determine first what the entity should do, and
then name it accordingly.  In other words, we should (or should not) add
the "default script" mappings based on an agreement that it is (or is
not) a good thing, not because the name of the field appears to permit
it.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 04 Apr 2005 19:49:01 +0200

Doug Ewell wrote:

> The thread titled "Preparations for smart matching" was
> started on March 19 by Frank Ellermann, to introduce the
> idea of adding "default script."

Yes, and it's impossible to come to a final conclusion "now"
(like say today), because there is no "smart matching draft".

At the moment I only _think_ that default scripts are a MUST
HAVE, because otherwise en-Latn-GB-scouse, en-GB-scouse,
en-Latn-scouse, and en-scouse could run into trouble if they
don't match each other.

Maybe I'm wrong and this only breaks RTL matching.  Maybe it
causes major havoc for all Web servers and browsers.  I don't
know before I see the matching draft.

> A better approach is to determine first what the entity
> should do, and then name it accordingly.

I want it to be used for the default script, because otherwise
we might create a "language tag soup" and no way to match them
except from a few trivial cases, where RfC 3066 is good enough
or even better than RfC 3066bis.

A viable alternative is a RECOMMENDATION to never omit the
script or the region in tags with a variant.  I hope that the
matching draft proposes a better way, but at the moment some
kind of "default script" is the only solution I see, and the
"prefix" field of languages is the only place to add this info.

> In other words, we should (or should not) add the "default
> script" mappings based on an agreement that it is (or is
> not) a good thing, not because the name of the field appears
> to permit it.

You can't decide this before the real issue (= "matching") is
clear.  That's like discussing cut-off dates together with the
language tags (where it's mostly irrelevant) before the strange
side-effect for region codes are visible.

Let's just see a new state of the art.  There have been so many
changes from the "ABNF" over "record-jar" to the "normative
references" that I lost track.  It's only an I-D, not the last
chance to get it right.
                        Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 4 Apr 2005 14:49:06 -0400

> know before I see the matching draft.

I have yet to receive any comments on my proposed alternative
(not replacement) matching algorithm.  People seemed to get lost in
the details of Date A and Date B.  To recap in a way that doesn't
reawaken that:

1) Normalize the tag.

2) Create a quadruple (lang, script, region, variant), filling in missing
   components with *s.

3) Subtags match if they are the same or if either is "*".

4) Add {8, 4, 2, 1} to the figure of merit if the {lang, script, region,
   variant} matches.

-- 
Using RELAX NG compact syntax to        John Cowan
develop schemas is one of the simple    http://www.reutershealth.com
pleasures in life....                   http://www.ccil.org/~cowan
        --Jeni Tennison                 <[email-masked]>

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 4 Apr 2005 21:02:51 +0100

Doug Ewell wrote:

> At the moment I only _think_ that default scripts are a MUST HAVE,

What do you -think- you would apply to the proposed 639-3 tags like "ess"
Central Siberian Yupik which is written in Cyrillic script in Siberia and
Latin script in Alaska?

Debbie Garside

www.linguasphere.com 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Frank Ellermann
Sent: 04 April 2005 18:49
To: [email-masked]
Subject: [Ltru] Re: Registry in record-jar format

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 04 Apr 2005 22:59:37 +0200

John Cowan wrote:
 
> 1) Normalize the tag.
> 2) Create a quadruple (lang, script, region, variant),
>    filling in missing components with *s.
> 4) Add {8, 4, 2, 1} to the figure of merit if the {lang,
>    script, region, variant} matches.

Nice.  I'd prefer {4, 3, 1, 2}.  If somebody goes to the
trouble of specifying a variant it could be more important
than the region (2 > 1).  4+3+2+1=10 because I like REXX ;-)

                          Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 04 Apr 2005 23:06:33 +0200

Debbie Garside wrote:

> What do you -think- you would apply to the proposed 639-3
> tags like "ess" Central Siberian Yupik which is written in
> Cyrillic script in Siberia and Latin script in Alaska?

No default when it's unclear.  This case sounds like the now
deprecated sh (sr + hr).  Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Mon, 4 Apr 2005 18:58:07 -0400

> than the region (2 > 1).  4+3+2+1=10 because I like REXX ;-)

I didn't choose powers of 2 capriciously.  Your algorithm would prefer
"ru-latn-1901" to "de" when matching against "de-Latn-1901".
(We don't have enough variants yet to make a really compelling
counterexample, but that indicates the general feel of it.)

-- 
If you understand,                      John Cowan
   things are just as they are;         http://www.ccil.org/~cowan
if you do not understand,               http://www.reutershealth.com
   things are just as they are.         [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 05 Apr 2005 19:03:19 +0200

John Cowan wrote:

> I didn't choose powers of 2 capriciously.

Yes, we have to test the effects of different metrics.  I've
created a very quick and very dirty script for this purpose,
adding other metrics or more test cases would be easy, please
tell me if you have some good candidates.

At the moment I check four + five against four tags:

column 1 is en-scouse
column 2 is en-GB-scouse
column 3 is en-Latn-GB-scouse
column 4 is en-Latn-scouse

Here's a result for your metrics:

en-scouse          9  9  9  9
en-GB-scouse       9 11 11  9
en-Latn-GB-scouse  9 11 15 13
en-Latn-scouse     9  9 13 13

xx-Latn-GB-scouse  1  3  7  5
en-Latn-GB         8 10 14 12
en-Latn-US         8  8 12 12
en-Latn-US-scouse  9  9 13 13
en-Brai-GB-scouse  9 11 11  9

I've added 0 for '*' = '*' and no match.  Otherwise 8/4/2/1,
is this what you wanted ?  Your metrics has apparently a
problem with en-Latn-US-scouse:

If one side wants en-GB-scouse, and the other side offers
en-Latn-US-scouse (9) or en-Latn-GB (10), and it also has
en-Brai-GB-scouse (11), then en-Brai-GB-scouse "wins".  All
in the 2nd column for en-GB-scouse.

In the 3rd column for en-Latn-GB-scouse you see the expected
but odd effect, where en-GB-Latn (14) would match before an
also available en-Latn-US-scouse (13).

> Your algorithm would prefer "ru-latn-1901" to "de" when
> matching against "de-Latn-1901".

Not okay, but not completely unintentional, for some languages
I can guess what the text is about, as long as it's Latn:  The
combined power of forgotten school Latin plus miserable French
sometimes helps with es or pt.  But with ru I'd be lost - with
luck I can decode some Cyrl.  For fy my chances are lousy, for
dk or nl it's better than zero.

Here's the result with my metrics (as you have shown it's bad):

en-scouse          5  5  5  5
en-GB-scouse       5  7  7  5
en-Latn-GB-scouse  5  7 10  8
en-Latn-scouse     5  5  8  8

xx-Latn-GB-scouse  1  3  6  4
en-Latn-GB         4  6  9  7
en-Latn-US         4  4  7  7
en-Latn-US-scouse  5  5  8  8
en-Brai-GB-scouse  5  7  7  5

One effect you see with both metrics;  If one side wants
en-scouse, and the other side has only en-Latn-US-scouse and
en-Brai-GB-scouse, you get a draw.  Apparently your algorithm
cannot completely replace the "default script" approach.

                          Bye, Frank

---

From: John Cowan <[email-masked]>
Date: Tue, 5 Apr 2005 22:04:33 -0400

> problem with en-Latn-US-scouse:

I'm not clear on whether * = * should count as a match or a no-match.
Originally I thought it should count as a match, but perhaps not.

> in the 2nd column for en-GB-scouse.

Fortunately en-US-scouse doesn't exist.

> dk or nl it's better than zero.

Fair enough.

> cannot completely replace the "default script" approach.

Well, that problem applies at all levels: if you ask for en-AU,
then no algorithm can choose between the offered en-GB and en-US
(except RFC 2616, which will simply fail).

For that matter, if you ask for de and nn and nb are all that's
available, the matching algorithm won't help then either.

-- 
Is a chair finely made tragic or comic? Is the          John Cowan
portrait of Mona Lisa good if I desire to see           [email-masked]
it? Is the bust of Sir Philip Crampton lyrical,         www.ccil.org/~cowan
epical or dramatic?  If a man hacking in fury           www.reutershealth.com
at a block of wood make there an image of a cow,
is that image a work of art? If not, why not?               --Stephen Dedalus