Subject: [Ltru] Canonical variants

From: Martin Hosken <[email-masked]>
Date: Wed, 9 Jul 2008 14:01:34 +0700

Dear All,

I may have missed something in my reading but strong things are written about extension authorities having to provide canonicalizations for their subvariants, but I can find no canonicalization criteria for registered variants.

e.g. is it en-boont-fonipa, en-fonipa-boont, en-Latn-boont-fonipa, en-Latn-fonipa-boont

Yours,
Martin Hosken
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 9 Jul 2008 01:10:31 -0700

I'm sure we've discussed this at some point in the past...

Peter

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Martin Hosken
> Sent: Wednesday, July 09, 2008 12:02 AM
> To: [email-masked]
> Subject: [Ltru] Canonical variants
>
> Dear All,
>
> I may have missed something in my reading but strong things are written
> about extension authorities having to provide canonicalizations for
> their subvariants, but I can find no canonicalization criteria for
> registered variants.
>

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 10:34:59 -0400

> their subvariants, but I can find no canonicalization criteria for

You are confusing variants with extensions.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have not seen as far as others, it is because giants were standing
on my shoulders.
        --Hal Abelson
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Martin Hosken <[email-masked]>
Date: Wed, 9 Jul 2008 21:48:15 +0700

Dear John,

> You are confusing variants with extensions.

In what way? If we consider variants to be elements of an unmarked extension, then just as an extension must provide a canonical order for its elements, so should we for the variants. Otherwise what is the relative canonical order for two variants?

Yours,
Martin

---

From: Peter Constable <[email-masked]>
Date: Wed, 9 Jul 2008 08:55:15 -0700

> In what way? If we consider variants to be elements of an unmarked
> extension

Waffling with terminology doesn't necessarily help get people on the same page. The key point you want to raise is whether a canonical order for variants should be defined. Variants are not extensions, so nothing is necessarily entailed.

The key question, I think, is to ask what canonicalization is for. I think you're wondering about a canonical order for variants because you're thinking about tag comparison: e.g. is aa-variant1-variant2 "the same" as aa-variant2-variant1 or not -- or should they be considered a match or not.

If that's all that canonicalization were about, I think it would be covered in 4647, not 4646(bis). Rather, I think the intent of canonicalization in 4646(bis) is to direct users toward preferred, best-practice tags.

If we could assume that variant subtags do not semantically interact, then we could define a normalization process for purposes of matching, such as putting them in alphabetical order. But that would be distinct from specifying a preferred order that should be used.

Given the way some matching algorithms work, the order of variants can impact behaviours, even if not declared semantics. And given that tags with multiple variant subtags are probably not going to be all that likely, I'm not inclined to adding new machinery into 4646bis at this time -- let's not get into too much feature creep.

Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 11:56:12 -0400

> In what way? If we consider variants to be elements of an unmarked

If.

(See http://en.wikipedia.org/wiki/Laconic_phrase .)

-- 
Well, I have news for our current leaders       John Cowan
and the leaders of tomorrow: the Bill of        [email-masked]
Rights is not a frivolous luxury, in force      http://www.ccil.org/~cowan
only during times of peace and prosperity.
We don't just push it to the side when the going gets tough.  --Molly Ivins
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 9 Jul 2008 11:36:41 -0700

I think Martin has something here. The overriding goal of canonicalization
*is* comparison: that if any two tags are known to have the same semantics,
then they will have the same canonical form. Otherwise there is little
purpose to it. I was going to speak up about this earlier: if we want to
state that xx-yyy and yyy have the same semantics, then they have to have
the same canonical form.

I think what you are talking about is something different: "preferred
forms". For that, one could have multiple options, but then different tags
could have the same semantics.

What I would prefer is that the canonical form of variants be as
alphabetical as possible:

   1. alphabetize the variants
   2. order the variants for prefixes by iterating as follows:

   - if the variant V is valid (given the earlier subtags at that point)
      then proceed to the next variant (or exit if done)
      - otherwise find the first subsequent prefix P that would make P
      valid, move it to be immediately in front of V, and loop (so we'll
      reconsider P)

(multiple variants are rare, so this should be rarely invoked)
Mark

On Wed, Jul 9, 2008 at 8:55 AM, Peter Constable <[email-masked]>
wrote:

> Waffling with terminology doesn't necessarily help get people on the same
> page. The key point you want to raise is whether a canonical order for
> variants should be defined. Variants are not extensions, so nothing is
> necessarily entailed.
>
> The key question, I think, is to ask what canonicalization is for. I think
> you're wondering about a canonical order for variants because you're
> thinking about tag comparison: e.g. is aa-variant1-variant2 "the same" as
> aa-variant2-variant1 or not -- or should they be considered a match or not.
>
> If that's all that canonicalization were about, I think it would be covered
> in 4647, not 4646(bis). Rather, I think the intent of canonicalization in
> 4646(bis) is to direct users toward preferred, best-practice tags.
>
> If we could assume that variant subtags do not semantically interact, then
> we could define a normalization process for purposes of matching, such as
> putting them in alphabetical order. But that would be distinct from
> specifying a preferred order that should be used.
>
> Given the way some matching algorithms work, the order of variants can
> impact behaviours, even if not declared semantics. And given that tags with
> multiple variant subtags are probably not going to be all that likely, I'm
> not inclined to adding new machinery into 4646bis at this time -- let's not
> get into too much feature creep.
>
>
> Peter

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 15:05:25 -0400

>    1. alphabetize the variants

That assumes more than we know.  The order of variants may or may not
be meaningful.

-- 
It was impossible to inveigle           John Cowan <[email-masked]>
Georg Wilhelm Friedrich Hegel           http://www.ccil.org/~cowan
Into offering the slightest apology
For his Phenomenology.                      --W. H. Auden, from "People" (1953)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 9 Jul 2008 13:06:31 -0700

You make a good point. It is a reasonable position to say that the order is
meaningful, and that thus canonicalization cannot disturb the order. But
we're the ones who define what the variants mean. If the order *is*
meaningful, we have to say so clearly in the spec, and say what difference
the ordering is supposed to make.

Otherwise someone could easily assume that the ordering is not meaningful
(except for prefixing).

So

Mark

On Wed, Jul 9, 2008 at 12:05 PM, John Cowan <[email-masked]> wrote:

> Georg Wilhelm Friedrich Hegel           http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 13:13:59 -0700

The order is meaningful to several matching schemes (else we would have put script at the end). However, I don’t think we can definitively say what the meaning of the order is, as it will vary by variant. We stop short of saying this in Section 2.2.5, where we say:

--
A variant that can meaningfully be used in combination with another variant SHOULD include a 'Prefix' field in its registry record that lists that other variant. For example, if another German variant 'example' were created that made sense to use with '1996', then 'example' should include two Prefix fields: "de" and "de-1996".
--

To this we could add:

--
If a variant (such as ‘example’ above) has another variant as its prefix, then the variant that includes the other as a prefix SHOULD appear after the other variant when they are used together to ensure consistent matching. Some variants (such as ‘fonipa’) do not include any Prefix: these subtags MAY be used with any preceding sequence of subtags, including other variants. Typically these “general purpose” variants SHOULD appear [before|after] any variants that include a Prefix.
--

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, July 09, 2008 1:07 PM
To: John Cowan
Cc: [email-masked]
Subject: Re: [Ltru] Canonical variants

You make a good point. It is a reasonable position to say that the order is meaningful, and that thus canonicalization cannot disturb the order. But we're the ones who define what the variants mean. If the order *is* meaningful, we have to say so clearly in the spec, and say what difference the ordering is supposed to make.

Otherwise someone could easily assume that the ordering is not meaningful (except for prefixing).

So

Mark
On Wed, Jul 9, 2008 at 12:05 PM, John Cowan <[email-masked]<mailto:[email-masked]>> wrote:

--
It was impossible to inveigle           John Cowan <[email-masked]<mailto:[email-masked]>>
Georg Wilhelm Friedrich Hegel           http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>
Into offering the slightest apology
For his Phenomenology.                      --W. H. Auden, from "People" (1953)

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 16:47:51 -0400

Phillips, Addison scripsit:

> If a variant (such as ‘example’ above) has another variant as its
> prefix, then the variant that includes the other as a prefix SHOULD
> appear after the other variant when they are used together to ensure
> consistent matching. Some variants (such as ‘fonipa’) do not include
> any Prefix: these subtags MAY be used with any preceding sequence
> of subtags, including other variants. Typically these “general
> purpose” variants SHOULD appear [before|after] any variants that
> include a Prefix.

+1

-- 
Cash registers don't really add and subtract;           John Cowan
        they only grind their gears.                    [email-masked]
But then they don't really grind their gears, either;   
        they only obey the laws of physics.  --Unknown
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 9 Jul 2008 13:50:24 -0700

We can only say SHOULD for a reordering of variants that reordering does not
make a semantic difference. And if the reordering *doesn't* make a semantic
difference, then that needs to be a MUST in the canonicalization.

So with the existing text in 2.5, and your new text, we would need to add
something like the following to what you MUST do in canonicalization. But
unless there is some demonstrable semantic that we want to attach to the
ordering of variants, we should also alphabetize. So here would be suggested
text:

   1. Separate the variant tags into two lists those requiring prefixes and
   those not. Alphabetize each list (case-insensitive ASCII order). Put all of
   the subtags that don't need prefixes first, and those not needing prefixes
   second.
   2. Iterate through the variant subtags, and if any subtag needs a prefix
   and doesn't have one, look through the remaining variants to see if such a
   prefix is there. If so, move that prefix to just before the subtag. (Because
   cycles are not allowed, this will terminate.)

Example:

   - sl-lipaw-fonipa-rozaj-1996 // This is for illustration: 1996 should
   have a prefix of de
   - => sl-fonipa-1996-lipaw-rozaj // applying #1 to {fonipa}, {1996, lipaw,
   rozaj}
   - => sl-fonipa-1996-rozaj-lipaw // applying #2 to lipaw

Mark

On Wed, Jul 9, 2008 at 1:13 PM, Phillips, Addison <[email-masked]>
wrote:

>  The order is meaningful to several matching schemes (else we would have
> put script at the end). However, I don't think we can definitively say what
> the meaning of the order is, as it will vary by variant. We stop short of
> saying this in Section 2.2.5, where we say:
>
>
>
> --
>
> A variant that can meaningfully be used in combination with another variant
> SHOULD include a 'Prefix' field in its registry record that lists that other
> variant. For example, if another German variant 'example' were created that
> made sense to use with '1996', then 'example' should include two Prefix
> fields: "de" and "de-1996".
>
> --
>
>
>
> To this we could add:
>
>
>
> --
>
> If a variant (such as 'example' above) has another variant as its prefix,
> then the variant that includes the other as a prefix SHOULD appear after the
> other variant when they are used together to ensure consistent matching.
> Some variants (such as 'fonipa') do not include any Prefix: these subtags
> MAY be used with any preceding sequence of subtags, including other
> variants. Typically these "general purpose" variants SHOULD appear
> [before|after] any variants that include a Prefix.
>
> --
>
>
>
> Addison Phillips
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Mark Davis
> *Sent:* Wednesday, July 09, 2008 1:07 PM
> *To:* John Cowan
> *Cc:* [email-masked]
> *Subject:* Re: [Ltru] Canonical variants
>
>
>

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 17:06:21 -0400

> difference, then that needs to be a MUST in the canonicalization.

It would be a Good Thing if we could canonicalize away all syntactic
differences that make no semantic difference, but [insert CGJ here] it
doesn't work out.  In this case, we simply can't say, considering an
arbitrary number of variants, whether or not two given variants
aaaaa and bbbbb interact semantically such that foo-aaaaa-bbbbb means
something different from foo-bbbbb-aaaaa.

-- 
When I'm stuck in something boring              John Cowan
where reading would be impossible or            (who loves Asimov too)
rude, I often set up math problems for          [email-masked]
myself and solve them as a way to pass          http://www.ccil.org/~cowan
the time.      --John Jenkins
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 14:29:19 -0700

Especially so since both 'bbbbb' and 'aaaaa' tend to be modifying 'foo', not each other. For example, 'fonipa' and 'boont' have different effects on subtag 'en'. (you can s/boont/Scotland/ or s/boont/scouse/ if you prefer).

Addison Phillips

> From: John Cowan [mailto:[email-masked]]
> Sent: Wednesday, July 09, 2008 2:06 PM
> To: Mark Davis
> Cc: Phillips, Addison; John Cowan; [email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 9 Jul 2008 15:18:44 -0700

As I said, we are the ones who are determining the meaning of these things.
If the meaning of xx-fonipa-boont is different than the meaning of
xx-boont-fonipa, then it is up to us to say what the difference is. If you
can state what the difference is, then we should put that in the spec, and
canonicalization should not reorder.

Mark

On Wed, Jul 9, 2008 at 2:06 PM, John Cowan <[email-masked]> wrote:

> myself and solve them as a way to pass          http://www.ccil.org/~cowan<http://www.ccil.org/%7Ecowan>
> the time.      --John Jenkins
>

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Jul 2008 18:32:04 -0400

> canonicalization should not reorder.

Those are two different we's:  it is ietf-languages who determines
the meaning of variant subtags, but LTRU that determines the
canonicalization rules.  LTRU is temporary, but ietf-languages
is permanent, and it is not reasonable for LTRU to make a rule
in this respect that binds ietf-languages into the indefinite
future.

Furthermore, using the Prefix field means that if bbbbb is to be
ordered after aaaaa, it cannot be used without aaaaa (or some
alternative prefix).  That may well be too much information for
a particular language tag.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
"The exception proves the rule."  Dimbulbs think: "Your counterexample proves
my theory."  Latin students think "'Probat' means 'tests': the exception puts
the rule to the proof."  But legal historians know it means "Evidence for an
exception is evidence of the existence of a rule in cases not excepted from."
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 9 Jul 2008 15:44:26 -0700

> alternative prefix).

It can, with "some alternative prefix" as you put it. In fact, we already have examples of this and text saying that this is the case. See, for example:

%%
Type: variant
Subtag: 1994
Description: Standardized Resian orthography
Added: 2007-07-28
Prefix: sl-rozaj
Prefix: sl-rozaj-biske
Prefix: sl-rozaj-njiva
Prefix: sl-rozaj-osojs
Prefix: sl-rozaj-solba
Comments: For standardized Resian an orthography was published in 1994.

1994 is ordered after 'biske', 'njiva', 'osojs', and 'solba'. Note that it is not recommended with 'lipaw' (which shares the sl-rozaj prefix). In cases where all of the variants have prefixes, one can determine the canonical order (provided one does not use prefixes that are mutually exclusive together).

What we don't say is where "no-prefix" variants go.

Addison

---

From: Peter Constable <[email-masked]>
Date: Thu, 10 Jul 2008 08:41:48 -0700

If the main intent of canonicalization is comparison, then I’m just curious as to how it ended up being part of 4646 rather than 4647, and why we bother to say that tags SHOULD always be in canonical form – pushing canonical form everywhere, not just into comparison.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, July 09, 2008 11:37 AM
To: Peter Constable
Cc: Martin Hosken; [email-masked]
Subject: Re: [Ltru] Canonical variants

I think Martin has something here. The overriding goal of canonicalization *is* comparison: that if any two tags are known to have the same semantics, then they will have the same canonical form. Otherwise there is little purpose to it. I was going to speak up about this earlier: if we want to state that xx-yyy and yyy have the same semantics, then they have to have the same canonical form.

I think what you are talking about is something different: "preferred forms". For that, one could have multiple options, but then different tags could have the same semantics.

What I would prefer is that the canonical form of variants be as alphabetical as possible:

    *   if the variant V is valid (given the earlier subtags at that point) then proceed to the next variant (or exit if done)
    *   otherwise find the first subsequent prefix P that would make P valid, move it to be immediately in front of V, and loop (so we'll reconsider P)
(multiple variants are rare, so this should be rarely invoked)
Mark
On Wed, Jul 9, 2008 at 8:55 AM, Peter Constable <[email-masked]<mailto:[email-masked]>> wrote:
> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of
> Martin Hosken

Peter
_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Thu, 10 Jul 2008 09:02:22 -0700

On reflection, I don’t think relative order can be semantically significant. These tags represent nothing more than semantic meaning components; unlike encoded characters, their encoded order does not correspond to any visual ordering. For the order to be semantically significant, we would have to explicitly devise some semantic metaphor to which ordering corresponds.

Now, we have, in fact, defined a metaphor somewhat related to position of subtags: it is that the addition of a subtag onto the end of a tag should have the effect of narrowing the denotation, making it more specific. In effect, subtags combine logicially in a conjunctive manner – criterion a AND criterion b AND … But note that logical conjunction is a commutative operation, and so two different criteria applied to a filter lead to the same result whether the filter is expressed as a AND b or b AND a.

In terms of tags as elements of declarative meanings, the only issue in ordering of subtags has to do with required prefixes: for any pair of subtags, they are either fully independent, or strictly dependent, and in the latter case we may specify a prefix.

On the other hand, in terms of tags as constructs that are processed in particular ways in matching algorithms, we know that order can matter, particularly in relation to right-side-truncation processing. That has led us to put subtags for which the meaning can have greater impact on user experience toward the start of a tag; in particular, that is what led us to put the script subtag before subtags that would be used for dialectal variations, region and variant. For the same reasons, I could see potential motivation to canonicalize so that variant subtags related to written form are before variant subtags related to dialectal variation. But down at the level of variant, I don’t see enough benefit: we would need to add subtag attributes for variants to differentiate those that pertain to written form from those do not, and there aren’t clear scenarios where the need for the effects could provide.

So, it seems to me that defining a canonicalization for variant subtags that considers only prefix and some arbitrary basis for ordering (such as alphabetic order) would be adequate.

Peter

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Wednesday, July 09, 2008 3:19 PM
To: John Cowan
Cc: [email-masked]
Subject: Re: [Ltru] Canonical variants

As I said, we are the ones who are determining the meaning of these things. If the meaning of xx-fonipa-boont is different than the meaning of xx-boont-fonipa, then it is up to us to say what the difference is. If you can state what the difference is, then we should put that in the spec, and canonicalization should not reorder.

Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 10 Jul 2008 10:31:25 -0700

I tend to agree with what you wrote below, at least generally.

Tag canonicalization includes a number of operations, the purpose of which is mainly to make matching/selection operations work reliably. That is, when tags are semantically equivalent, a request for one form of the tag produces all of its semantic matches. In programming, we call this “do what I mean”.

We’ve defined a number of operations to help this:

-          We normalize case, even though we shout repeatedly that case is not important in processing tags. This is partly for human legibility purposes, but it certainly helps naïve string matching

-          We map to preferred-values. This makes a request for things like region “BU” include results with region “MM”, which is what the user probably intends.

-          We reorder extensions. This makes extension processing easier to implement (well, only a little).

Variant ordering seems to fit the above. However, unlike other subtag types, variants can convey a variety of classes of meaning. We have already registered a variety of transcriptions, orthographies, regional or specialized dialects/jargons/etc., spelling variations (“oed”), and so forth. The interplay between these is complex and the order may or may not have meaning---either in terms of the semantic meaning of the tag or in terms of matching.

Since we did not specify variant ordering in 4646, adding it now could (but is extremely unlikely to) break situations in which people have relied on variant order for some kind of semantic score keeping. Personally, the less we require implementations to look inside the tags and dink around with the values provided, the better. Most implementations today are way more naïve than what we’re specifying.

And given that variants should always be rare, specifying any canonical order seems like overkill. Tag Content Wisely…. Adding a health warning stating that variant order is “significant to some matching operations, so try to be consistent” seems to me a better course of action.

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
Sent: Thursday, July 10, 2008 9:02 AM
To: [email-masked]
Subject: Re: [Ltru] Canonical variants

Peter

Mark

---

From: Peter Constable <[email-masked]>
Date: Thu, 10 Jul 2008 10:48:51 -0700

I don’t have any problem not making changes to deal with canonicalization of variants.

Peter

From: Phillips, Addison [mailto:[email-masked]]
Sent: Thursday, July 10, 2008 10:31 AM
To: Peter Constable; [email-masked]
Subject: RE: [Ltru] Canonical variants

Addison Phillips

Peter

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 11 Jul 2008 10:47:29 -0700

I agree that we can revisit the canonicalization in a future version;
however, we have to make sure that we are clear that ordering doesn't
matter. It is implicit currently, because we don't specify any semantics for
the ordering of variants, but we need to make it explicit. Here is suggested
text.

2.2.5.  Variant Subtags6. The order of variant subtags does not carry any
semantic meaning.

Mark

On Thu, Jul 10, 2008 at 10:48 AM, Peter Constable <[email-masked]>
wrote:

>  I don't have any problem not making changes to deal with canonicalization
> of variants.
>
>
>
> Peter
>
>
>
> *From:* Phillips, Addison [mailto:[email-masked]]
> *Sent:* Thursday, July 10, 2008 10:31 AM
> *To:* Peter Constable; [email-masked]
> *Subject:* RE: [Ltru] Canonical variants
>
>
>
> Tag canonicalization includes a number of operations, the purpose of which
> is mainly to make matching/selection operations work reliably. That is, when
> tags are semantically equivalent, a request for one form of the tag produces
> all of its semantic matches. In programming, we call this "do what I mean".
>
>
>
> We've defined a number of operations to help this:
>
>
>
> -          We normalize case, even though we shout repeatedly that case is
> not important in processing tags. This is partly for human legibility
> purposes, but it certainly helps naïve string matching
>
> -          We map to preferred-values. This makes a request for things
> like region "BU" include results with region "MM", which is what the user
> probably intends.
>
> Variant ordering seems to fit the above. However, unlike other subtag
> types, variants can convey a variety of classes of meaning. We have already
> registered a variety of transcriptions, orthographies, regional or
> specialized dialects/jargons/etc., spelling variations ("oed"), and so
> forth. The interplay between these is complex and the order may or may not
> have meaning---either in terms of the semantic meaning of the tag or in
> terms of matching.
>
>
>
> Since we did not specify variant ordering in 4646, adding it now could (but
> is extremely unlikely to) break situations in which people have relied on
> variant order for some kind of semantic score keeping. Personally, the less
> we require implementations to look inside the tags and dink around with the
> values provided, the better. Most implementations today are way more naïve
> than what we're specifying.
>
>
>
> And given that variants should always be rare, specifying any canonical
> order seems like overkill. Tag Content Wisely…. Adding a health warning
> stating that variant order is "significant to some matching operations, so
> try to be consistent" seems to me a better course of action.
>
>
>
> Addison Phillips
>
> *Peter Constable
> *Sent:* Thursday, July 10, 2008 9:02 AM
> *To:* [email-masked]
> On reflection, I don't think relative order can be semantically
> significant. These tags represent nothing more than semantic meaning
> components; unlike encoded characters, their encoded order does not
> correspond to any visual ordering. For the order to be semantically
> significant, we would have to explicitly devise some semantic metaphor to
> which ordering corresponds.
>
>
>
> Now, we have, in fact, defined a metaphor somewhat related to position of
> subtags: it is that the addition of a subtag onto the end of a tag should
> have the effect of narrowing the denotation, making it more specific. In
> effect, subtags combine logicially in a conjunctive manner – criterion a AND
> criterion b AND … But note that logical conjunction is a commutative
> operation, and so two different criteria applied to a filter lead to the
> same result whether the filter is expressed as a AND b or b AND a.
>
>
>
> In terms of tags as elements of declarative meanings, the only issue in
> ordering of subtags has to do with required prefixes: for any pair of
> subtags, they are either fully independent, or strictly dependent, and in
> the latter case we may specify a prefix.
>
>
>
> On the other hand, in terms of tags as constructs that are processed in
> particular ways in matching algorithms, we know that order can matter,
> particularly in relation to right-side-truncation processing. That has led
> us to put subtags for which the meaning can have greater impact on user
> experience toward the start of a tag; in particular, that is what led us to
> put the script subtag before subtags that would be used for dialectal
> variations, region and variant. For the same reasons, I could see potential
> motivation to canonicalize so that variant subtags related to written form
> are before variant subtags related to dialectal variation. But down at the
> level of variant, I don't see enough benefit: we would need to add subtag
> attributes for variants to differentiate those that pertain to written form
> from those do not, and there aren't clear scenarios where the need for the
> effects could provide.
>
>
>
> So, it seems to me that defining a canonicalization for variant subtags
> that considers only prefix and some arbitrary basis for ordering (such as
> alphabetic order) would be adequate.
>
>
>
>
>
> Peter
>
>
>
> *Sent:* Wednesday, July 09, 2008 3:19 PM
> *To:* John Cowan
> *Cc:* [email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Sat, 12 Jul 2008 17:26:41 +0900

At 02:47 08/07/12, Mark Davis wrote:
>I agree that we can revisit the canonicalization in a future version; however, we have to make sure that we are clear that ordering doesn't matter. It is implicit currently, because we don't specify any semantics for the ordering of variants, but we need to make it explicit. Here is suggested text.

I tend to disagree. Because we don't say anything doesn't mean
that it's one way or the other. It's very easy to argue that our
matching procedures assume a fixed ordering, or in other words
that there is a difference between different orderings.

Regards,   Martin.

>
><http://2.2.5.>2.2.5.  Variant Subtags
>
>
>
>6. The order of variant subtags does not carry any semantic meaning.
>
>
>Mark 
>
>On Thu, Jul 10, 2008 at 10:48 AM, Peter Constable <<mailto:[email-masked]>[email-masked]> wrote:
>>
>>I don't have any problem not making changes to deal with canonicalization of variants.
>>
>> 
>>
>>Peter
>>
>> 
>>
>>To: Peter Constable; <mailto:[email-masked]>[email-masked]
>>Tag canonicalization includes a number of operations, the purpose of which is mainly to make matching/selection operations work reliably. That is, when tags are semantically equivalent, a request for one form of the tag produces all of its semantic matches. In programming, we call this "do what I mean". 
>>
>> 
>>
>>-          We normalize case, even though we shout repeatedly that case is not important in processing tags. This is partly for human legibility purposes, but it certainly helps naテッve string matching
>>
>>-          We map to preferred-values. This makes a request for things like region "BU" include results with region "MM", which is what the user probably intends.
>>
>>Variant ordering seems to fit the above. However, unlike other subtag types, variants can convey a variety of classes of meaning. We have already registered a variety of transcriptions, orthographies, regional or specialized dialects/jargons/etc., spelling variations ("oed"), and so forth. The interplay between these is complex and the order may or may not have meaning---either in terms of the semantic meaning of the tag or in terms of matching.
>>
>> 
>>
>>Since we did not specify variant ordering in 4646, adding it now could (but is extremely unlikely to) break situations in which people have relied on variant order for some kind of semantic score keeping. Personally, the less we require implementations to look inside the tags and dink around with the values provided, the better. Most implementations today are way more naテッve than what we're specifying.
>>
>> 
>>
>>And given that variants should always be rare, specifying any canonical order seems like overkill. Tag Content Wisely窶ヲ. Adding a health warning stating that variant order is "significant to some matching operations, so try to be consistent" seems to me a better course of action.
>>
>> 
>>
>>Addison Phillips
>>
>>From: <mailto:[email-masked]>[email-masked] [mailto:[email-masked]] On Behalf Of Peter Constable
>>To: <mailto:[email-masked]>[email-masked]
>>On reflection, I don't think relative order can be semantically significant. These tags represent nothing more than semantic meaning components; unlike encoded characters, their encoded order does not correspond to any visual ordering. For the order to be semantically significant, we would have to explicitly devise some semantic metaphor to which ordering corresponds. 
>>
>> 
>>
>>Now, we have, in fact, defined a metaphor somewhat related to position of subtags: it is that the addition of a subtag onto the end of a tag should have the effect of narrowing the denotation, making it more specific. In effect, subtags combine logicially in a conjunctive manner 窶�criterion a AND criterion b AND 窶ヲ But note that logical conjunction is a commutative operation, and so two different criteria applied to a filter lead to the same result whether the filter is expressed as a AND b or b AND a.
>>
>> 
>>
>>On the other hand, in terms of tags as constructs that are processed in particular ways in matching algorithms, we know that order can matter, particularly in relation to right-side-truncation processing. That has led us to put subtags for which the meaning can have greater impact on user experience toward the start of a tag; in particular, that is what led us to put the script subtag before subtags that would be used for dialectal variations, region and variant. For the same reasons, I could see potential motivation to canonicalize so that variant subtags related to written form are before variant subtags related to dialectal variation. But down at the level of variant, I don't see enough benefit: we would need to add subtag attributes for variants to differentiate those that pertain to written form from those do not, and there aren't clear scenarios where the need for the effects could provide.
>>
>> 
>>
>>From: <mailto:[email-masked]>[email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
>>Cc: <mailto:[email-masked]>[email-masked]
>><mailto:[email-masked]>[email-masked]

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 12 Jul 2008 09:12:21 -0700

Martin wrote:
> >I agree that we can revisit the canonicalization in a future
> version; however, we have to make sure that we are clear that
> ordering doesn't matter. It is implicit currently, because we don't
> specify any semantics for the ordering of variants, but we need to
> make it explicit. Here is suggested text.
>

I agree with Martin. There is a difference between differently ordered variants. Indeed, this is sometimes significant, which you can see in the schedule of prefixes for Slovenian variants. That is, some variants modify the meaning of other variants that precede them.

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 12 Jul 2008 13:30:58 -0700

I don't understand this point of view. WE (in this specification) are
establishing the meaning of the variant subtags. And that includes either
stating that the order doesn't matter, or that it does (and how it
matters). Now, we certainly do have a preferred order when there are
prefixes, but that only partially determines the order of variants; there is
not a determinate order.
We certainly don't want Joe thinking there is no difference in ordering and
sending V2-V1 to Sue, who thinks that V2-V1 means something different than
V2-V1. We don't want a Google saying that alphabetical order means the
modern form and inverse alphabetic order means a non-modern form. And so on.

This also makes a difference for GUIs. When we let people compose a tag, we
have to be able to let them know when something makes a difference, and how
to explain that difference if there is one. (It is far easier to just let
them pick subtags, and then compose a tag by inserting any necessary
prefixes and putting it in a preferred order.)

When we parse a tag, also we have to know whether the order makes a
difference or not. And if it does make a difference, what is that difference
supposed to be?

So what it comes down to is:

   1. If you think that there is a semantic difference between
   de-fonipa-1996 and de-1996-fonipa, what is that difference supposed to be?
   2. And if there is no difference, what is the preferred order, so that we
   promote interoperability?
   3. Or do you just think that variants are so unimportant that it doesn't
   matter?

Mark

BTW, I think the simplest thing would be to say that variant order has no
semantic meaning, but that the preferred order is alphabetical + adjusted to
make prefixes first (see more details earlier).

2008/7/12 Martin Duerst <[email-masked]>:

> >On Thu, Jul 10, 2008 at 10:48 AM, Peter Constable <<mailto:
> [email-masked]>[email-masked]> wrote:
> >>
> >>Tag canonicalization includes a number of operations, the purpose of
> which is mainly to make matching/selection operations work reliably. That
> is, when tags are semantically equivalent, a request for one form of the tag
> produces all of its semantic matches. In programming, we call this "do what
> I mean".
> >>
> >>
> >>
> >>-          We normalize case, even though we shout repeatedly that case
> is not important in processing tags. This is partly for human legibility
> purposes, but it certainly helps naテッve string matching
> >>
> >>Since we did not specify variant ordering in 4646, adding it now could
> (but is extremely unlikely to) break situations in which people have relied
> on variant order for some kind of semantic score keeping. Personally, the
> less we require implementations to look inside the tags and dink around with
> the values provided, the better. Most implementations today are way more
> naテッve than what we're specifying.
> >>
> >>
> >>
> order seems like overkill. Tag Content Wisely窶ヲ. Adding a health warning
> >>From: <mailto:[email-masked]>[email-masked] [mailto:
> [email-masked]] On Behalf Of Peter Constable
> effect, subtags combine logicially in a conjunctive manner 窶�criterion a AND
> criterion b AND 窶ヲ But note that logical conjunction is a commutative
> >>From: <mailto:[email-masked]>[email-masked] [mailto:
> [email-masked]] On Behalf Of Mark Davis
> >>As I said, we are the ones who are determining the meaning of these
> things. If the meaning of xx-fonipa-boont is different than the meaning of

---

From: Peter Constable <[email-masked]>
Date: Sat, 12 Jul 2008 15:35:57 -0700

> I agree with Martin. There is a difference between differently ordered
> variants. Indeed, this is sometimes significant, which you can see in
> the schedule of prefixes for Slovenian variants. That is, some variants
> modify the meaning of other variants that precede them.

See my earlier mail about this. It is not the ordering per se that causes one variant to modify the meaning of the other; we simply have a syntactic-validation convention about ordering in those cases, using Prefix to declare in the registry that co-occurrence restriction. And in those cases, for which a Prefix is specified for the more-specific variant, only one order is expected: we don't *require* that order, but I wouldn't expect anybody to use such a pair of subtags in the opposite order. And, for instance, even if someone *did* tag content as "sl-biske-rozaj", while that could impact matching behaviour in some context, there would not be any conventional semantic distinction between that tag and "sl-rozaj-biske". It would even seem reasonable to me to have an analog to Unicode's conformance clause C6:

C6 A process shall not assume that the interpretations of two canonical-equivalent character
sequences are distinct.

I.e., that we declare explicitly that "sl-rozaj-biske" shall not be assumed to have a distinct interpretation from "sl-biske-rozaj" -- after all, unless we explicitly declare some conventional semantic associated with that ordering distinction, there *isn't any* conventional semantic associated with it.

An alternative to such a semantic constraint would be to impose a syntactic constraint along these lines: if subtag X is part of the Prefix value of subtag Y, then tags of the form ...Y...X... MUST NOT be used / are not valid.

Peter

---

From: Peter Constable <[email-masked]>
Date: Sat, 12 Jul 2008 15:37:44 -0700

+1

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Saturday, July 12, 2008 1:31 PM
To: Martin Duerst
Cc: Peter Constable; [email-masked]
Subject: Re: [Ltru] Canonical variants

I don't understand this point of view. WE (in this specification) are establishing the meaning of the variant subtags. And that includes either stating that the order doesn't matter, or that it does (and how it matters). Now, we certainly do have a preferred order when there are prefixes, but that only partially determines the order of variants; there is not a determinate order.

We certainly don't want Joe thinking there is no difference in ordering and sending V2-V1 to Sue, who thinks that V2-V1 means something different than V2-V1. We don't want a Google saying that alphabetical order means the modern form and inverse alphabetic order means a non-modern form. And so on.

This also makes a difference for GUIs. When we let people compose a tag, we have to be able to let them know when something makes a difference, and how to explain that difference if there is one. (It is far easier to just let them pick subtags, and then compose a tag by inserting any necessary prefixes and putting it in a preferred order.)

When we parse a tag, also we have to know whether the order makes a difference or not. And if it does make a difference, what is that difference supposed to be?

 1.  If you think that there is a semantic difference between de-fonipa-1996 and de-1996-fonipa, what is that difference supposed to be?
 2.  And if there is no difference, what is the preferred order, so that we promote interoperability?
 3.  Or do you just think that variants are so unimportant that it doesn't matter?
Mark

BTW, I think the simplest thing would be to say that variant order has no semantic meaning, but that the preferred order is alphabetical + adjusted to make prefixes first (see more details earlier).

2008/7/12 Martin Duerst <[email-masked]<mailto:[email-masked]>>:

Regards,   Martin.

>
><http://2.2.5.>2.2.5.<http://2.2.5.>  Variant Subtags
>
>
>
>On Thu, Jul 10, 2008 at 10:48 AM, Peter Constable <<mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]>> wrote:
>>
>>From: Phillips, Addison [mailto:[email-masked]<mailto:[email-masked]>]
>>To: Peter Constable; <mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]>
>>From: <mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Peter Constable
>>To: <mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]>
>>From: <mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mark Davis
>>Cc: <mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]>
>><mailto:[email-masked]<mailto:[email-masked]>>[email-masked]<mailto:[email-masked]>

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]<mailto:[email-masked]>

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 12 Jul 2008 21:13:25 -0700

I would be happy with the equivalent of C6.
Mark

On Sat, Jul 12, 2008 at 3:35 PM, Peter Constable <[email-masked]>
wrote:

> See my earlier mail about this. It is not the ordering per se that causes
> one variant to modify the meaning of the other; we simply have a
> syntactic-validation convention about ordering in those cases, using Prefix
> to declare in the registry that co-occurrence restriction. And in those
> cases, for which a Prefix is specified for the more-specific variant, only
> one order is expected: we don't *require* that order, but I wouldn't expect
> anybody to use such a pair of subtags in the opposite order. And, for
> instance, even if someone *did* tag content as "sl-biske-rozaj", while that
> could impact matching behaviour in some context, there would not be any
> conventional semantic distinction between that tag and "sl-rozaj-biske". It
> would even seem reasonable to me to have an analog to Unicode's conformance
> clause C6:
>
> I.e., that we declare explicitly that "sl-rozaj-biske" shall not be assumed
> to have a distinct interpretation from "sl-biske-rozaj" -- after all, unless
> we explicitly declare some conventional semantic associated with that
> ordering distinction, there *isn't any* conventional semantic associated
> with it.
>
> An alternative to such a semantic constraint would be to impose a syntactic
> constraint along these lines: if subtag X is part of the Prefix value of
> subtag Y, then tags of the form ...Y...X... MUST NOT be used / are not
> valid.
>
>
>
> Peter

---

From: Martin Duerst <[email-masked]>
Date: Tue, 15 Jul 2008 18:55:57 +0900

>canonical-equivalent character

The main processes that are relevant to our work are defined
in RFC 4647. They wouldn't pass muster, they assume (implicitly)
that the order of subtags matters.

I hope you don't propose that we rewrite RFC 4647 (and all the
implementations of the various things described therein).

Regards,   Martin.

>to have a distinct interpretation from "sl-biske-rozaj" -- after all, 
>unless we explicitly declare some conventional semantic associated with 
>that ordering distinction, there *isn't any* conventional semantic 
>associated with it.
>
>subtag Y, then tags of the form ...Y...X... MUST NOT be used / are not valid.
>
>
>
>Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 07:43:47 -0700

That is why I've been saying we need a canonical order. You still haven't
addressed the questions I had for you:

   1. If you think that there is a semantic difference between
      de-fonipa-1996 and de-1996-fonipa, what is that difference supposed to be?
      2. And if there is no difference, what is the preferred order, so that
      we promote interoperability?
      3. Or do you just think that variants are so unimportant that it
      doesn't matter?

Mark

---

From: John Cowan <[email-masked]>
Date: Tue, 15 Jul 2008 11:19:52 -0400

>       de-fonipa-1996 and de-1996-fonipa, what is that difference

The burden of persuasion runs the other way.  In order to permit
mechanical rearrangement of variant subtags without semantic loss,
it's necessary to show not that a particular case of rearrangement
changes nothing, but that *every* case of rearrangement changes nothing.
And since prediction is very difficult, especially about the future,
that's not easy to show.

As we both know, it was this sort of lack of forethought (probably
unavoidable at the time, to be sure) that caused Unicode canonical
order to wind up as a purely conventional ordering no longer tied to
either presentational utility or semantic meaningfulness, contrary to
its original intention.  I wouldn't like to see that happen to BCP 47
canonicalization.

(What is more, a tag with both 'fonipa' and '1996' is self-contradictory,
since German cannot be expressed at one and the same time in its 1996
conventional orthography and in IPA.  Nonsense remains nonsense no matter
how much you shake it up, and reasoning from a nonsensical case isn't
going to lead to a sensible result.)

-- 
Schlingt dreifach einen Kreis vom dies!    John Cowan <[email-masked]>
Schliesst euer Aug vor heiliger Schau,     http://www.ccil.org/~cowan
Denn er genoss vom Honig-Tau,
Und trank die Milch vom Paradies.            --Coleridge (tr. Politzer)
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 08:43:05 -0700

I'm just confused by this. If order is supposed to be semantically relevant,
*what* is it exactly that establishes what each order means (V1-V2 vs
V2-V1), if not BCP47? Google? Microsoft? the DOD?

Unicode ordering is not necessarily a good model, since it applies to
characters that reflect real usage, whereas BCP47 is completely responsible
for the meaning and syntax of these subtags.

It sounds to me like what people are saying is that multiple variants are
not important enough to worry about -- if so, then I'm fine with that. My
main concern is whether to have an API that returns a set or a list for
variants; if BCP47 leaves it up in the air, then we can just use a set,
since the implementation is simpler, and we won't then return false
inequalities.

Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 15 Jul 2008 08:48:47 -0700

Martin suggested:

And Mark responded:
--
That is why I've been saying we need a canonical order. You still haven't addressed the questions I had for you:

I think:

1. There isn't a semantic difference between these two tags, but there is a practical/procedural difference as Martin points out. I might as well point out that '1996' is mainly a spelling reform, so using it with 'fonipa' is kind of silly. The real issue is what we think a general-purpose variant modifies:

 i. the whole tag's meaning
 ii. the meaning of the preceding variant subtag(s)
 iii. the meaning of the primary language subtag
 iv. the meaning of some other subtag(s)

We have different examples of some of these cases, out of a limited inventory of variants to consider.

For (i), the order doesn't matter (consider 'fonipa'). For (ii) the order matters (consider 'biske'). For (iii) the order doesn't matter (consider 'boont'). For (iv) the order doesn't matter (consider 'monoton' as modifying 'Grek'??).

That is: order usually doesn't matter *semantically*, except when it does.

2. I tend to think that subtags with a prefix stick closer to the tag than subtags without a prefix. That is, in your example, 'fonipa' modifies the tag "de-1996". A better example would be "en-boont-fonipa". So I would tend to recommend that you arrange the prefixed items first, according to their Prefix values ('rozaj' always comes before 'biske', both always come before '1994') and then arrange any general purpose variants after that ('fonipa' comes after the sequence '-rozaj-biske-1994').

3. I tend to think that variants are relatively unimportant and that this matter probably isn't going to prove that problematic in the real world. In the real world, variants are rare and multiple variants exceedingly so. That doesn't mean we shouldn't handle the problem. Only that a very small minority of us will ever be bothered by it.

Thus, I propose that in section 4.1 (Choice) we add this item:

--
6. Use variants sparingly and in the correct order. Most variant subtags have a Prefix field or fields that express the list of subtags that they are appropriate with. Variants SHOULD only be used with subtags that appear in one of these Prefix fields. If a variant lists another variant in one of its Prefix fields, it SHOULD appear after that variant in the language tag (where both occur). General purpose variants (those with no Prefix) SHOULD appear after any other variant subtags. Because variants are very specialized, using many of them together generally makes the tag so narrow as to override the additional precision gained.

For example: the Slovenian variant 'rozaj' (Resian dialect) has a Prefix of 'sl'. Other Slovenian variants include 'biske' and '1994'. The subtag 'biske' lists "sl-rozaj" as a Prefix. The subtag '1994' lists both "sl-rozaj" and "sl-rozaj-biske" as a Prefix. Thus, the proper form of a tag that uses all three is "sl-rozaj-biske-1994". Putting the subtags into another order would interfere with interoperability, as well as the overall interpretation of the tag.
--

And then in canonicalization say:
--
6. If more than one variant appears, order the variants according to their Prefix fields in the registry. In this order, if a variant appears in one of a variant's Prefix fields, it MUST appear in the tag preceding that variant. If none of the other variants appear in the tag, order it after any variants that have a prefix that matches the tag, otherwise preserving the original order selected by the user. For example, the tag "sl-monoton-fonipa-1994-biske-rozaj" would be reordered as "sl-rozaj-biske-1994-monoton-fonipa" because 'biske' lists 'rozaj' in its prefixes, but not '1994' and '1994' lists both 'biske' and 'rozaj'; 'monoton' is supposed to be used with 'el' (Greek) and 'fonipa' has no Prefix. Their original order is preserved. Note that this tag, considered as a whole, has no practical meaning.
--

Comments?

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 08:56:08 -0700

That language looks good to me, except for one thing:"otherwise preserving
the original order selected by the user" "
=>
"otherwise use alphabetical order".

Mark

On Tue, Jul 15, 2008 at 8:48 AM, Phillips, Addison <[email-masked]>
wrote:

> 1. If you think that there is a semantic difference between de-fonipa-1996
> and de-1996-fonipa, what is that difference supposed to be?
> 1. There isn't a semantic difference between these two tags, but there is a
> practical/procedural difference as Martin points out. I might as well point
> out that '1996' is mainly a spelling reform, so using it with 'fonipa' is
> kind of silly. The real issue is what we think a general-purpose variant
> modifies:
>
> For (i), the order doesn't matter (consider 'fonipa'). For (ii) the order
> matters (consider 'biske'). For (iii) the order doesn't matter (consider
> 'boont'). For (iv) the order doesn't matter (consider 'monoton' as modifying
> 'Grek'??).
>
> 2. I tend to think that subtags with a prefix stick closer to the tag than
> subtags without a prefix. That is, in your example, 'fonipa' modifies the
> tag "de-1996". A better example would be "en-boont-fonipa". So I would tend
> to recommend that you arrange the prefixed items first, according to their
> Prefix values ('rozaj' always comes before 'biske', both always come before
> '1994') and then arrange any general purpose variants after that ('fonipa'
> comes after the sequence '-rozaj-biske-1994').
>
> 3. I tend to think that variants are relatively unimportant and that this
> matter probably isn't going to prove that problematic in the real world. In
> the real world, variants are rare and multiple variants exceedingly so. That
> doesn't mean we shouldn't handle the problem. Only that a very small
> minority of us will ever be bothered by it.
>
> 6. Use variants sparingly and in the correct order. Most variant subtags
> have a Prefix field or fields that express the list of subtags that they are
> appropriate with. Variants SHOULD only be used with subtags that appear in
> one of these Prefix fields. If a variant lists another variant in one of its
> Prefix fields, it SHOULD appear after that variant in the language tag
> (where both occur). General purpose variants (those with no Prefix) SHOULD
> appear after any other variant subtags. Because variants are very
> specialized, using many of them together generally makes the tag so narrow
> as to override the additional precision gained.
>
> For example: the Slovenian variant 'rozaj' (Resian dialect) has a Prefix of
> 'sl'. Other Slovenian variants include 'biske' and '1994'. The subtag
> 'biske' lists "sl-rozaj" as a Prefix. The subtag '1994' lists both
> "sl-rozaj" and "sl-rozaj-biske" as a Prefix. Thus, the proper form of a tag
> that uses all three is "sl-rozaj-biske-1994". Putting the subtags into
> another order would interfere with interoperability, as well as the overall
> interpretation of the tag.
> --
>
> 6. If more than one variant appears, order the variants according to their
> Prefix fields in the registry. In this order, if a variant appears in one of
> a variant's Prefix fields, it MUST appear in the tag preceding that variant.
> If none of the other variants appear in the tag, order it after any variants
> that have a prefix that matches the tag, otherwise preserving the original
> order selected by the user. For example, the tag
> "sl-monoton-fonipa-1994-biske-rozaj" would be reordered as
> "sl-rozaj-biske-1994-monoton-fonipa" because 'biske' lists 'rozaj' in its
> prefixes, but not '1994' and '1994' lists both 'biske' and 'rozaj';
> 'monoton' is supposed to be used with 'el' (Greek) and 'fonipa' has no
> Prefix. Their original order is preserved. Note that this tag, considered as
> a whole, has no practical meaning.
> --
>
>
> Comments?
>
> Addison
>

---

From: John Cowan <[email-masked]>
Date: Tue, 15 Jul 2008 12:06:58 -0400

> I'm just confused by this. If order is supposed to be semantically

It's not that it's necessarily semantically relevant.  It's that it's a
bad idea, considering the state of our ignorance, to *assume* it's going
to be semantically irrelevant in all possible future cases (modulo the
ones involving prefixes that Addison discussed).

It's like asking "Is the order of XML child elements semantically
relevant"?  Sometimes it is, sometimes it isn't.  Some schemas for
documents where the order is irrelevant prescribe a fixed order so as
to prevent user-specified channels from becoming a covert information
channel; others don't prescribe a fixed order.  Not having a general
scheme for canonicalizing child-element order costs something, but the
gain in flexibility is generally considered to outweigh it.

> V2-V1), if not BCP47? Google? Microsoft? the DOD?

Ietf-languages, and after them, the community of usage.

> Unicode ordering is not necessarily a good model, since it applies
> to characters that reflect real usage, whereas BCP47 is completely
> responsible for the meaning and syntax of these subtags.

But it is not responsible for the fact that '1996' and 'fonipa' are
self-contradictory: it's real-word facts that determine that.

> It sounds to me like what people are saying is that multiple variants
> are not important enough to worry about -- if so, then I'm fine with
> that. My main concern is whether to have an API that returns a set
> or a list for variants; if BCP47 leaves it up in the air, then we can
> just use a set, since the implementation is simpler, and we won't then
> return false inequalities.

Preserve the order; that way you may lose some rare opportunities for
canonicalization, but you don't throw away information that's potentially
valuable.  LinkedHashSet is your friend.

-- 
A witness cannot give evidence of his           John Cowan
age unless he can remember being born.          [email-masked]
  --Judge Blagden                               http://www.ccil.org/~cowan
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 15 Jul 2008 09:26:17 -0700

+1

Note that my proposal, even if we were to do the "alphabetization thing" Mark proposes, is not a full reordering. There are several unaddressed edge cases in it. More-or-less, I did that on purpose. The subtag order has meaning (in a matching context). Excepting certain cases where we know what the meaning is (because it is registered), the only indication of the user's intent is the order we receive the subtags in.

This plays well, please note, with existing matching implementations. For this to work reliably, you would have to revise the matching schemes to do the variant reordering. Readers of 4647 today expect that the basic range "de-zzzzz" does not select the tag "de-aaaaa-zzzzz".

Addison

---

From: Peter Constable <[email-masked]>
Date: Tue, 15 Jul 2008 09:53:56 -0700

> Unicode ordering is not necessarily a good model, since
> it applies to characters that reflect real usage, whereas
> BCP47 is completely responsible for the meaning and syntax
> of these subtags.

The biggest way that Unicode breaks down as an analogy is that combining marks can interact graphically in terms of positional ordering, which gives a very transparent metaphor for encoded ordering, whereas for our subtags there is no obvious metaphor for what subtag ordering might mean other than increasing degrees of specificity -- and that is typically relevant in cases where a prefix is given, and always irrelevant if the subtags are fully independent (e.g. rozaj and fonipa).

Peter

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 10:01:23 -0700

Mark

> gain in flexibility is generally considered to outweigh it.

It's bad to leave it open, because then we have no idea whether
rearrangement is supposed to determine meaning or not.

> Ietf-languages, and after them, the community of usage.

But then what you are calling for is a mechanism we don't have. Remember,
what we are talking about is the ordering of variant subtags that do
nothave a prefix relationship to one another. (If they do, then we
have a
canonical order.)

Do you really expect Ietf-languages to specify that V1 means X if it is
before V2, and means Y if it is after? Or that V1 needs to come before V2
although V1 is not a prefix of V2? What would be an example of what you
think that ietf-languages would do? Would this be in the description?

Or the "community of users"? How am I on the parsing end supposed to know
that some nebulous "community of users" says that the ordering of V1 and V2
means Y. And frankly, the ordering of subtags is, in the vast majority of
cases due to some GUI -- and the way in which the GUI presents how to build
the tags has more to do with the ordering than what the users choose. And
what is that GUI developer supposed to do?

My take-away from all this is that multiple variants without prefix
orderings are nearly so nebulous as to be not worth supporting until some
later version clarifies them.

> valuable.  LinkedHashSet is your friend.

Of course one can use something like LinkedHashSet; the issue is whether one
is preserving a material ordering or not.

---

From: John Cowan <[email-masked]>
Date: Tue, 15 Jul 2008 13:07:20 -0400

Peter Constable scripsit:

> The biggest way that Unicode breaks down as an analogy is that combining
> marks can interact graphically in terms of positional ordering, which
> gives a very transparent metaphor for encoded ordering, 

Alas, some combining marks that were thought to be in irrelevant positions
turned out not to be so.

The analogy isn't about the specifics: it's about assuming that something
is meaningless that later on turns out to be meaningful.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
        --Julius Caesar
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Tue, 15 Jul 2008 10:20:22 -0700

> Alas, some combining marks that were thought to be in irrelevant

This response to the point I made is ignoring the point I was making: that those particular aspects of Unicode are not analogous: encoding order in Unicode is a transparent metaphor for positional ordering, whereas there isn't a metaphor for ordering of our subtags.

> is meaningless that later on turns out to be meaningful.

But the analogy breaks down for the reasons I've stated: encoding order has no obvious metaphor for our subtags, and so unless *we decide* to define some conventional semantic, there is no conventional semantic.

Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 15 Jul 2008 11:07:58 -0700

There is no conventional semantic. However, users might sometimes supply one. Generally this isn't a huge issue because true variants are rare in the wild and of limited practicality. While that could change, in practice most general variants are going to be nonsense together. At present, none of the general purpose variants make sense with one-another. While that will change over time, the utility of more than a couple together is going to always remain limited.

When implementing language tags, I have always made the assumption that the subtag's order had some meaning to the user, even though my implementation could not know what it was. All of the other subtags are canonically ordered in a tag, after all---extlang, private use, and extension sequences have to remain in the same order. Why would variants be any different? And somehow I don't think it is a good thing that canonicalizing processors produce a vastly different result from non-canonicalizing ones when that result is not measurably better (it is measurably better to match a request for en-BU also to the tag en-MM, etc.)

I've had the opportunity to recommend various of the matching schemes to different working groups. RFC 4647's compatibility with RFC 3066 has been a tremendous boon here. However, if some implementations start reordering subtags, they'll produce different results than those that are strictly of the "Tag Content Wisely" flavor (i.e. "you give me subtags and I'll remove them, one-by-one, until I find something"). The less I need the registry, the happier I am overall because that hews closer to the original language tag implementation requirements. So it makes me feel more comfortable saying that matching implementations don't need to worry about subtag (re-)ordering.

Addison

Addison Phillips

> Sent: Tuesday, July 15, 2008 10:20 AM
> To: [email-masked]
> This response to the point I made is ignoring the point I was
> making: that those particular aspects of Unicode are not analogous:
> encoding order in Unicode is a transparent metaphor for positional
> ordering, whereas there isn't a metaphor for ordering of our
> subtags.
>
>
> But the analogy breaks down for the reasons I've stated: encoding
> order has no obvious metaphor for our subtags, and so unless *we
> decide* to define some conventional semantic, there is no
> conventional semantic.
>
>
>
> Peter
>

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 11:38:01 -0700

> However, users might sometimes supply one.

It is this statement that gives me hiccoughs. If John Smith gives meaning to
V1-V2 vs V2-V1, how is anyone to know what meaning he gives them? Suppose
Joe has V1-V2 mean "as used before the Rapture" and V2-V1 mean "as used
after the Rapture". Who's to know? We have to have something concrete, or it
is not interoperable.

However, what you and others seem to be implicitly assuming is the ordering
"most significant subtag first". So just to get out of this issue, we could
use that as a guiding principle. That is, we give the advice to users (in
the "tag wisely section") for variant subtag ordering as follows:

- put prefix variant subtags before their suffixes (that is, if V1 occurs in
a Prefix of V2, V1 should go first)
- otherwise, put "more significant" variant subtags before others
- otherwise -- or in case there is any doubt as to relative significance --
put variant subtags in alphabetical order.

Mark

On Tue, Jul 15, 2008 at 11:07 AM, Phillips, Addison <[email-masked]>
wrote:

> There is no conventional semantic. However, users might sometimes supply
> one. Generally this isn't a huge issue because true variants are rare in the
> wild and of limited practicality. While that could change, in practice most
> general variants are going to be nonsense together. At present, none of the
> general purpose variants make sense with one-another. While that will change
> over time, the utility of more than a couple together is going to always
> remain limited.
>
> When implementing language tags, I have always made the assumption that the
> subtag's order had some meaning to the user, even though my implementation
> could not know what it was. All of the other subtags are canonically ordered
> in a tag, after all---extlang, private use, and extension sequences have to
> remain in the same order. Why would variants be any different? And somehow I
> don't think it is a good thing that canonicalizing processors produce a
> vastly different result from non-canonicalizing ones when that result is not
> measurably better (it is measurably better to match a request for en-BU also
> to the tag en-MM, etc.)
>
> I've had the opportunity to recommend various of the matching schemes to
> different working groups. RFC 4647's compatibility with RFC 3066 has been a
> tremendous boon here. However, if some implementations start reordering
> subtags, they'll produce different results than those that are strictly of
> the "Tag Content Wisely" flavor (i.e. "you give me subtags and I'll remove
> them, one-by-one, until I find something"). The less I need the registry,
> the happier I am overall because that hews closer to the original language
> tag implementation requirements. So it makes me feel more comfortable saying
> that matching implementations don't need to worry about subtag
> (re-)ordering.
>
> Addison
>
> Addison Phillips

---

From: John Cowan <[email-masked]>
Date: Tue, 15 Jul 2008 14:41:11 -0400

Mark Davis scripsit:

> However, what you and others seem to be implicitly assuming is the
> ordering "most significant subtag first". So just to get out of this
> issue, we could use that as a guiding principle. That is, we give
> the advice to users (in the "tag wisely section") for variant subtag
> ordering as follows:
> 
> - put prefix variant subtags before their suffixes (that is, if V1
>   occurs in a Prefix of V2, V1 should go first)

I'm good with that, since it's advisory and decoupled from
canonicalization.

-- 
But that, he realized, was a foolish            John Cowan
thought; as no one knew better than he          [email-masked]
that the Wall had no other side.                http://www.ccil.org/~cowan
        --Arthur C. Clarke, "The Wall of Darkness"
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 15 Jul 2008 22:21:49 +0200

Mark Davis wrote:

- put prefix variant subtags before their suffixes (that is, if V1 occurs in a Prefix of V2, V1 should go first) 

Hmm, shouldn't variant subtags that have declared prefixes in the registry that do not come *directly* after one of those prefixes in the tag make the tag invalid? That should be the case also for extlangs, otherwise things like zh-cmn-nan would be valid tags, and I don't think they should be.
 
And for variant subtags that don't have a prefix declared in the registry, multiple occurrences of the *same* variant subtag in a tag should also render the tag invalid.
 
    /kent k

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 15 Jul 2008 13:37:03 -0700

Prefixes have never required being "directly after"; and having the prefix
is a SHOULD not a must. (This was all discussed some time ago).

Multiple occurrences are invalid: see 2.2.5 "The same variant subtag MUST
NOT be used more than once within a language tag."

Variant subtags have different requirements than extlang. Extlangs can occur
either 0 or 1 times, no more. see 2.2.2 "Although the ABNF production
'extlang' permits up to three extended language tags in the language tag,
extended language subtags MUST NOT include another extended language subtag
in their Prefix. That is, the second and third extended language subtag
positions in a language tag are permanently reserved and tags that include
subtags in that position are invalid.

Mark

On Tue, Jul 15, 2008 at 1:21 PM, Kent Karlsson <[email-masked]>
wrote:

> Hmm, shouldn't variant subtags that have declared prefixes in the registry
> that do not come **directly** after one of those prefixes in the tag make
> the tag invalid? That should be the case also for extlangs, otherwise things
> like *zh-cmn-nan* would be valid tags, and I don't think they should be.
>
> And for variant subtags that don't have a prefix declared in the registry,
> multiple occurrences of the *same* variant subtag in a tag should also
> render the tag invalid.
>
>     /kent k
>
>

---

From: Peter Constable <[email-masked]>
Date: Tue, 15 Jul 2008 14:50:14 -0700

> Phillips, Addison

> 6. If more than one variant appears, order the variants according to
> their Prefix fields in the registry. In this order, if a variant
> appears in one of a variant's Prefix fields, it MUST appear in the tag
> preceding that variant. If none of the other variants appear in the tag,
> order it after any variants that have a prefix that matches the tag,
> otherwise preserving the original order selected by the user.

> Comments?

Only that the 2nd and 3rd sentences need to be reworded -- though it's not exactly easy to come up with straightforward wording for this. He's alternate wording -- not sure that it's much better, though:

6. If more than one variant appears, order the variants according to their Prefix fields in the registry. In this order, all variant subtags within a tag that have a Prefix-field relationship with another variant subtag also within that tag MUST occur before variant subtags that do not have such a relationship, and they must be ordered such that if variant1 is listed as a Prefix for variant2 then variant1 MSUT occur in a tag before variant2.

Peter

---

From: Martin Duerst <[email-masked]>
Date: Wed, 16 Jul 2008 10:28:44 +0900

At 23:43 08/07/15, Mark Davis wrote:
>If you think that there is a semantic difference between de-fonipa-1996 and de-1996-fonipa, what is that difference supposed to be? 
>And if there is no difference, what is the preferred order, so that we promote interoperability? 
>Or do you just think that variants are so unimportant that it doesn't matter?
[after reading many more posts in this thread]

Yes, I think that variants are relatively unimportant, in particular
combinations of non-prefixed variants. As Addison has said, we currently
don't have a single example that would actually make sense.
I'd prefer we could move on with our main work of finishing the
spec rather than to work out details of nonexisting cases.

Making decisions based on 0 examples is a bad idea. In some cases,
it may be necessary, but especially the IETF tries to avoid this.

The fact that right-to-left fallback is the most important process for
language tags suggests to me that the best way to order any potential
tags is so that fallback works best.

So, assume two extensions, 'fofoo' and 'babar'. Then assume a specific
language 'xyz'. Actual users might agree that there is no noticable
"semantic" difference between xyz-fofoo-babar and xyz-babar-fofoo,
in the sense that users who see these tags easily explain them the
same way, imagine the same content, and so on.

However, the above two tags behave differently with respect to fallback.
So there may be a clear and obvious reason to everybody in the community
to prefer one order over the other. And because these are general
tags, that order may be different for different languages, i.e.
xyz-fofoo-babar is preferred over xyz-babar-fofoo, while
zyx-babar-fofoo is preferred over zyx-fofoo-babar, for reasons
absolutely obvious for those knowing those specific language variants.

If we specify "alphabetic order" or some such, the chance we get this
right is 50% for two subtags. If we leave it to users, we give them a
100% chance (of course some of them will mess up).

So I strongly suggest:
- We say something about the order of variants with prefixes.
- For variants without prefixes, we don't say anything, or
  - We say that order matters (to RFC 4647, that is)
  - We say 'tag wisely'
  - We make clear that there should be no reordering, that order
    is to be preserved (i.e. applications should use an array,
    not a set)

In any case, I'm against specifying alphabetic ordering just
because we don't know anything better.

Regards,    Martin.

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 16 Jul 2008 12:53:31 -0700

I tried to fit your rewording into the draft, but it really wasn't quite right either. I ended up making a list of rules, as follows:

<t>If more than one variant appears:

  <list style="letters">

  <t>Order those variants related to the "base tag" via a 'Prefix' ahead of those variants that lack the relationship. For example, the tag "en-scottish-fonipa" (English, Scottish dialect, IPA phonetic transcription) is in canonical order because 'scottish' has a 'Prefix' of "en", while 'fonipa' has no 'Prefix' field.</t>

  <t>Order any variant that includes another variant or list of variants in one of its 'Prefix' fields directly following that variant (or list of variants). For example, the tag "sl-1994-biske-rozaj" would be canonicalized as "sl-rozaj-biske-1994".  This is because 'rozaj' lists "sl" as its sole 'Prefix'; 'biske' lists "sl-rozaj" as its sole Prefix. The subtag '1994' has several prefixes, including "sl-rozaj". However, it follows both subtags because one of its 'Prefix' fields is "sl-rozaj-biske".</t>

  <t>Keep any remaining variant subtags in their original order. For example, both the tags "tlh-boont-fonipa" (Klingon, Boontling, IPA phonetic transcription) and "tlh-fonipa-boont" are considered "canonical". The 'boont' subtag has a 'Prefix' of "en" (it really shouldn't appear in this tag), while 'fonipa' has no 'Prefix'. The reason for the selection and ordering of these subtags cannot be determined from the registry, so the original order is maintained.</t>

</list>
</t>

Comments?

Addison

Addison Phillips

> Sent: Tuesday, July 15, 2008 2:50 PM
> To: [email-masked]
> Only that the 2nd and 3rd sentences need to be reworded -- though
> it's not exactly easy to come up with straightforward wording for
> this. He's alternate wording -- not sure that it's much better,
> though:
>
> 6. If more than one variant appears, order the variants according
> to their Prefix fields in the registry. In this order, all variant
> subtags within a tag that have a Prefix-field relationship with
> another variant subtag also within that tag MUST occur before
> variant subtags that do not have such a relationship, and they must
> be ordered such that if variant1 is listed as a Prefix for variant2
> then variant1 MSUT occur in a tag before variant2.
>
>
>
> Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 16 Jul 2008 13:01:03 -0700

The proposed paragraph for 'Choice' is also a bit different that originally proposed. There were no substantive changes made (only editorial ones). For the record:

<t>Use variant subtags sparingly and in the correct order. Most variant subtags have a 'Prefix' field or fields that express the list of subtags that they are appropriate with. Variants SHOULD only be used with subtags that appear in one of these 'Prefix' fields. If a variant lists another variant in one of its 'Prefix' fields, it SHOULD appear directly after that variant in any language tag where both occur. General purpose variants (those with no 'Prefix' fields at all) SHOULD appear after any other variant subtags. Because variants are very specialized, using many of them together generally makes the tag so narrow as to override the additional precision gained.

For example: the Slovenian variant 'rozaj' (Resian dialect) has a Prefix of "sl". Other Slovenian variants include 'biske' and '1994'. The subtag 'biske' lists "sl-rozaj" as a 'Prefix'. The subtag '1994' lists both "sl-rozaj" and "sl-rozaj-biske" as a 'Prefix'. Thus, the proper form of a tag that uses all three is "sl-rozaj-biske-1994". Putting the subtags into another order would interfere with interoperability, as well as the overall interpretation of the tag.</t>

Addison Phillips

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Phillips, Addison
> Sent: Wednesday, July 16, 2008 12:54 PM
> I tried to fit your rewording into the draft, but it really wasn't
> quite right either. I ended up making a list of rules, as follows:
>
>   <t>Order those variants related to the "base tag" via a 'Prefix'
> ahead of those variants that lack the relationship. For example,
> the tag "en-scottish-fonipa" (English, Scottish dialect, IPA
> phonetic transcription) is in canonical order because 'scottish'
> has a 'Prefix' of "en", while 'fonipa' has no 'Prefix' field.</t>
>
>   <t>Order any variant that includes another variant or list of
> variants in one of its 'Prefix' fields directly following that
> variant (or list of variants). For example, the tag "sl-1994-biske-
> rozaj" would be canonicalized as "sl-rozaj-biske-1994".  This is
> because 'rozaj' lists "sl" as its sole 'Prefix'; 'biske' lists "sl-
> rozaj" as its sole Prefix. The subtag '1994' has several prefixes,
> including "sl-rozaj". However, it follows both subtags because one
> of its 'Prefix' fields is "sl-rozaj-biske".</t>
>
>   <t>Keep any remaining variant subtags in their original order.
> For example, both the tags "tlh-boont-fonipa" (Klingon, Boontling,
> IPA phonetic transcription) and "tlh-fonipa-boont" are considered
> "canonical". The 'boont' subtag has a 'Prefix' of "en" (it really
> shouldn't appear in this tag), while 'fonipa' has no 'Prefix'. The
> reason for the selection and ordering of these subtags cannot be
> determined from the registry, so the original order is
> maintained.</t>
>
> </list>
> </t>
>
> Comments?
>
> Addison
>
> Addison Phillips
> > to

---

From: Peter Constable <[email-masked]>
Date: Wed, 16 Jul 2008 22:14:03 -0700

> variants in one of its 'Prefix' fields directly following that variant
> (or list of variants).

For this, maybe wording along the lines of the following would be useful:

"For a given variant subtag within a given tag, examine the 'Prefix' fields for that subtag and determine the longest 'Prefix' that is also part of that tag. If there is such a prefix for that variant subtag, then the variant subtag must directly following that prefix."

Just an idea...
Peter

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 17 Jul 2008 14:17:31 -0700

Hmm... yes, this is a complicated thing to explain. I started with your text and arrived at:

--
<t>If more than one variant appears within a tag:

<list style="letters">

  <t>Order any variant related to the other subtags via a 'Prefix' ahead of those variants that lack any relationship to other subtags but following any subtags that appear in any of its 'Prefix' fields. For example, the tag "en-scottish-fonipa" (English, Scottish dialect, IPA phonetic transcription) is in canonical order because 'scottish' has a 'Prefix' of "en", while 'fonipa' has no 'Prefix' field. Another example would be the tag "sl-IT-rozaj-biske-1994". This tag is in canonical form: 'rozaj' lists "sl" as its sole 'Prefix'; 'biske' lists "sl-rozaj" as its sole Prefix. The subtag '1994' has several prefixes, including "sl-rozaj". However, it follows both 'rozaj' and 'biske' because one of its 'Prefix' fields is "sl-rozaj-biske".</t>

</list>
</t>
--

Addison Phillips

> From: Peter Constable [mailto:[email-masked]]
> Sent: Wednesday, July 16, 2008 10:14 PM
> To: Phillips, Addison; [email-masked]
> variant
> "For a given variant subtag within a given tag, examine the
> 'Prefix' fields for that subtag and determine the longest 'Prefix'
> that is also part of that tag. If there is such a prefix for that
> variant subtag, then the variant subtag must directly following
> that prefix."
>
>
> Just an idea...
> Peter

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 17 Jul 2008 15:13:05 -0700

In my view, your text is now going in simply the wrong direction. I had
proposed the following, which John seconded and you and Peter ignored.

At this point, I think we are better reverting to the text of 9 days ago on
this topic, and just leaving this issue open for the future. We can then get
back to the main issue, of seeing whether we are sufficiently done to do a
real last call.

Mark

On Thu, Jul 17, 2008 at 2:17 PM, Phillips, Addison <[email-masked]>
wrote:

>  <t>Order any variant related to the other subtags via a 'Prefix' ahead of
> those variants that lack any relationship to other subtags but following any
> subtags that appear in any of its 'Prefix' fields. For example, the tag
> "en-scottish-fonipa" (English, Scottish dialect, IPA phonetic transcription)
> is in canonical order because 'scottish' has a 'Prefix' of "en", while
> 'fonipa' has no 'Prefix' field. Another example would be the tag
> "sl-IT-rozaj-biske-1994". This tag is in canonical form: 'rozaj' lists "sl"
> as its sole 'Prefix'; 'biske' lists "sl-rozaj" as its sole Prefix. The
> subtag '1994' has several prefixes, including "sl-rozaj". However, it
> follows both 'rozaj' and 'biske' because one of its 'Prefix' fields is
> "sl-rozaj-biske".</t>
>
>  <t>Keep any remaining variant subtags in their original order. For
> example, both the tags "tlh-boont-fonipa" (Klingon, Boontling, IPA phonetic
> transcription) and "tlh-fonipa-boont" are considered "canonical". The
> 'boont' subtag has a 'Prefix' of "en" (it really shouldn't appear in this
> tag), while 'fonipa' has no 'Prefix'. The reason for the selection and
> ordering of these subtags cannot be determined from the registry, so the
> original order is maintained.</t>
>
> </list>
> </t>
> --
>
> Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 17 Jul 2008 16:20:41 -0700

In my view, your text is now going in simply the wrong direction. I had proposed the following, which John seconded and you and Peter ignored.
I didn’t ignore it: I believe I disagreed with it, or at least aspects of it.
However, what you and others seem to be implicitly assuming is the ordering "most significant subtag first". So just to get out of this issue, we could use that as a guiding principle. That is, we give the advice to users (in the "tag wisely section") for variant subtag ordering as follows:
I agree that “most significant first” is the idea—for cases where the order is known. What my proposed text does is have tags that are related to one-another appear together. The Slovenian subtags are a good example here. This is on the theory, perhaps half-baked, that ‘mumble’ dialect of ‘xx’ language is being modified by general purpose variants, rather than the other way around.
- put prefix variant subtags before their suffixes (that is, if V1 occurs in a Prefix of V2, V1 should go first)
- otherwise, put "more significant" variant subtags before others
- otherwise -- or in case there is any doubt as to relative significance -- put variant subtags in alphabetical order.

I don’t like the alphabetical order thing, since I don’t much care for reordering subtags at all---it’s a lot of work to code and test it, for no practical (and modest theoretical) benefit. This may reflect a difference in implementation choice :-). I’ve always assumed that the order was important and used Vector or some sort of List, rather than Set. In part that’s because order *is* important up to the variant position.

At this point, I think we are better reverting to the text of 9 days ago on this topic, and just leaving this issue open for the future. We can then get back to the main issue, of seeing whether we are sufficiently done to do a real last call.
I tend to agree, although the text previously proposed on choice seems useful.

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 17 Jul 2008 16:39:48 -0700

Remember, this is advice in the 'tag wisely' section, not in
canonicalization. Canonicalization is a dead end, because there is no
agreement on how to do it. It sounds like what you want is something like
the following:

- put prefix variant subtags before their suffixes (that is, if V1 occurs in
a Prefix of V2, V1 should go first)
- otherwise, where possible, keep prefixes and suffixes together
- otherwise, put "more significant" variant subtags before others
- otherwise -- or in case there is any doubt as to relative significance --
put variant subtags in random order, up to your whim of the moment ;-)

Personally, for the last phrase in clause 4, I'd advise people to use
alphabetical if one of the first three conditions don't apply, because at
least then we have some concrete advice for a determinate order. If you
leave clause 4 off, then random is what you are asking for...

Mark

On Thu, Jul 17, 2008 at 4:20 PM, Phillips, Addison <[email-masked]>
wrote:

> I didn't ignore it: I believe I disagreed with it, or at least aspects of
> it.
>
> I agree that "most significant first" is the idea—for cases where the order
> is known. What my proposed text does is have tags that are related to
> one-another appear together. The Slovenian subtags are a good example here.
> This is on the theory, perhaps half-baked, that 'mumble' dialect of 'xx'
> language is being modified by general purpose variants, rather than the
> other way around.
>
> I don't like the alphabetical order thing, since I don't much care for
> reordering subtags at all---it's a lot of work to code and test it, for no
> practical (and modest theoretical) benefit. This may reflect a difference in
> implementation choice :-). I've always assumed that the order was important
> and used Vector or some sort of List, rather than Set. In part that's
> because order **is** important up to the variant position.
>
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 17 Jul 2008 16:47:11 -0700

Hmm… okay, so in my last proposal, which you agreed to except for our differing between my “keep in the same order” and your “alphabetize”, we modified by the choice and canonicalization section. The text we’re discussing below fell into canonicalization. We can remove that and just make it part of choice.

When I get a moment, I’ll update Inter-Locale (which is back up after a hiatus) with some proposed text. I can see recommending alpha order for cases in which the user doesn’t have another order.

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Thursday, July 17, 2008 4:40 PM
To: Phillips, Addison
Cc: Peter Constable; [email-masked]
Subject: Re: [Ltru] Canonical variants

Remember, this is advice in the 'tag wisely' section, not in canonicalization. Canonicalization is a dead end, because there is no agreement on how to do it. It sounds like what you want is something like the following:

- put prefix variant subtags before their suffixes (that is, if V1 occurs in a Prefix of V2, V1 should go first)
- otherwise, where possible, keep prefixes and suffixes together
- otherwise, put "more significant" variant subtags before others
- otherwise -- or in case there is any doubt as to relative significance -- put variant subtags in random order, up to your whim of the moment ;-)

Personally, for the last phrase in clause 4, I'd advise people to use alphabetical if one of the first three conditions don't apply, because at least then we have some concrete advice for a determinate order. If you leave clause 4 off, then random is what you are asking for...

Mark
On Thu, Jul 17, 2008 at 4:20 PM, Phillips, Addison <[email-masked]<mailto:[email-masked]>> wrote:

I didn't ignore it: I believe I disagreed with it, or at least aspects of it.

I agree that "most significant first" is the idea—for cases where the order is known. What my proposed text does is have tags that are related to one-another appear together. The Slovenian subtags are a good example here. This is on the theory, perhaps half-baked, that 'mumble' dialect of 'xx' language is being modified by general purpose variants, rather than the other way around.

I don't like the alphabetical order thing, since I don't much care for reordering subtags at all---it's a lot of work to code and test it, for no practical (and modest theoretical) benefit. This may reflect a difference in implementation choice :-). I've always assumed that the order was important and used Vector or some sort of List, rather than Set. In part that's because order *is* important up to the variant position.

Addison

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 17 Jul 2008 16:51:58 -0700

There are two parts that I think you don't have yet in the choice; order by
prefix relations (you have) then most significant tag first, then
alphabetic.

Then I'm happy. (Well, reasonably. There is that whole war in Iraq, and
melting economy, and moron president thing...)

Mark

On Thu, Jul 17, 2008 at 4:47 PM, Phillips, Addison <[email-masked]>
wrote:

>  Hmm… okay, so in my last proposal, which you agreed to except for our
> differing between my "keep in the same order" and your "alphabetize", we
> modified by the choice and canonicalization section. The text we're
> discussing below fell into canonicalization. We can remove that and just
> make it part of choice.
>
>
>
> When I get a moment, I'll update Inter-Locale (which is back up after a
> hiatus) with some proposed text. I can see recommending alpha order for
> cases in which the user doesn't have another order.
>
>
>
> Addison
>
>
>
> Addison Phillips
>
> *From:* [email-masked] [mailto:[email-masked]] *On
> Behalf Of *Mark Davis
> *Sent:* Thursday, July 17, 2008 4:40 PM
> *To:* Phillips, Addison
> *Cc:* Peter Constable; [email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Fri, 18 Jul 2008 10:42:56 +0900

At 08:51 08/07/18, Mark Davis wrote:
>There are two parts that I think you don't have yet in the choice; order by prefix relations (you have) then most significant tag first, then alphabetic.

[as a technical contributor]

If 'alphabetic' comes after most significant subtag first, I'm able
to tolerate it in a section on how to create tags. When it comes to
any potential change of tags, I think the best advice we can give
is 'leave as is', and I very much do not want to see 'alphabetical'
in there, because the "most important" may be a judgement by the
creator, which should be respected. 

Regards,   Martin.

>Then I'm happy. (Well, reasonably. There is that whole war in Iraq, and melting economy, and moron president thing...)

[chair hat on: let's concentrate on the work of this WG :-]

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 18 Jul 2008 08:33:51 -0700

Mark

On Thu, Jul 17, 2008 at 6:42 PM, Martin Duerst <[email-masked]>
wrote:

> >There are two parts that I think you don't have yet in the choice; order
> by prefix relations (you have) then most significant tag first, then
> alphabetic.
>
> to tolerate it in a section on how to create tags.

good

> When it comes to

I'm ok with leaving that out -- it sounds like we may have sufficient
consensus to move forward on this, putting in the tag wisely.

(On the side, since I really don't want to stir this up again: my view is if
I pass in a tag xx-variant1-variant2 and some collection of data
has xx-variant2-variant1, as far as I'm concerned that should return as a
perfect match: as likely as not, any difference in the ordering is purely
arbitrary on the part of either the user making the first tag or the user
making the second tag. And if they are a perfect match, that implies that
the canonical forms should be the same. But there is clearly no consensus on
that so the important thing is to close this and move on.)

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 18 Jul 2008 11:22:44 -0700

Okay, here’s what I did.

First, in “choice” I made changes to the text to recommend alphabetization following “most significant order” thus:

--
<t>Use variant subtags sparingly and in the correct order. Most variant subtags have a 'Prefix' field or fields that express the list of subtags that they are appropriate with. Variants SHOULD only be used with subtags that appear in one of these 'Prefix' fields. If a variant lists another variant in one of its 'Prefix' fields, it SHOULD appear directly after that variant in any language tag where both occur. General purpose variants (those with no 'Prefix' fields at all) SHOULD appear after any other variant subtags. Order any remaining variants by placing the most significant subtag first. If none of the subtags is more significant or no relationship can be determined, alphabetize the subtags. Because variants are very specialized, using many of them together generally makes the tag so narrow as to override the additional precision gained. Putting the subtags into another order interferes with interoperability, as well as the overall interpretation of the tag.

<list style="letters">

<t>For example, the tag "en-scottish-fonipa" (English, Scottish dialect, IPA phonetic transcription) is correctly ordered because 'scottish' has a 'Prefix' of "en", while 'fonipa' has no 'Prefix' field.</t>

<t>For example, the tag "sl-IT-rozaj-biske-1994" is correctly ordered: 'rozaj' lists "sl" as its sole 'Prefix'; 'biske' lists "sl-rozaj" as its sole Prefix. The subtag '1994' has several prefixes, including "sl-rozaj". However, it follows both 'rozaj' and 'biske' because one of its 'Prefix' fields is "sl-rozaj-biske".</t>

</list>
</t>
--

Then, in canonicalization I removed the text from the required canonicalizations. However, I think it makes sense to mention it there and permit it. So I included this text:

<t>If more than one variant appears within a tag, processors MAY reorder the variants to obtain better matching behavior or more consistent presentation. Reordering of the variants SHOULD  follow the recommendations for variant ordering in <xref target="choice"></xref>.</t>

Does that work for everyone?

Addison

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: Friday, July 18, 2008 8:34 AM
To: Martin Duerst
Cc: Phillips, Addison; [email-masked]
Subject: Re: [Ltru] Canonical variants

Mark
On Thu, Jul 17, 2008 at 6:42 PM, Martin Duerst <[email-masked]<mailto:[email-masked]>> wrote:

good

When it comes to

I'm ok with leaving that out -- it sounds like we may have sufficient consensus to move forward on this, putting in the tag wisely.

(On the side, since I really don't want to stir this up again: my view is if I pass in a tag xx-variant1-variant2 and some collection of data has xx-variant2-variant1, as far as I'm concerned that should return as a perfect match: as likely as not, any difference in the ordering is purely arbitrary on the part of either the user making the first tag or the user making the second tag. And if they are a perfect match, that implies that the canonical forms should be the same. But there is clearly no consensus on that so the important thing is to close this and move on.)

Regards,   Martin.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 18 Jul 2008 11:29:12 -0700

Works for me.

Mark

On Fri, Jul 18, 2008 at 11:22 AM, Phillips, Addison <[email-masked]>
wrote:

>  Okay, here's what I did.
>
>
>
> First, in "choice" I made changes to the text to recommend alphabetization
> following "most significant order" thus:
>
>
>
> --
>
> <t>Use variant subtags sparingly and in the correct order. Most variant
> subtags have a 'Prefix' field or fields that express the list of subtags
> that they are appropriate with. Variants SHOULD only be used with subtags
> that appear in one of these 'Prefix' fields. If a variant lists another
> variant in one of its 'Prefix' fields, it SHOULD appear directly after that
> variant in any language tag where both occur. General purpose variants
> (those with no 'Prefix' fields at all) SHOULD appear after any other variant
> subtags. Order any remaining variants by placing the most significant subtag
> first. If none of the subtags is more significant or no relationship can be
> determined, alphabetize the subtags. Because variants are very specialized,
> using many of them together generally makes the tag so narrow as to override
> the additional precision gained. Putting the subtags into another order
> interferes with interoperability, as well as the overall interpretation of
> the tag.
>
>
>
> <list style="letters">
>
>
>
> <t>For example, the tag "en-scottish-fonipa" (English, Scottish dialect,
> IPA phonetic transcription) is correctly ordered because 'scottish' has a
> 'Prefix' of "en", while 'fonipa' has no 'Prefix' field.</t>
>
>
>
> <t>For example, the tag "sl-IT-rozaj-biske-1994" is correctly ordered:
> 'rozaj' lists "sl" as its sole 'Prefix'; 'biske' lists "sl-rozaj" as its
> sole Prefix. The subtag '1994' has several prefixes, including "sl-rozaj".
> However, it follows both 'rozaj' and 'biske' because one of its 'Prefix'
> fields is "sl-rozaj-biske".</t>
>
>
>
> </list>
>
> </t>
>
> --
>
>
>
> Then, in canonicalization I removed the text from the required
> canonicalizations. However, I think it makes sense to mention it there and
> permit it. So I included this text:
>
>
>
> <t>If more than one variant appears within a tag, processors MAY reorder
> the variants to obtain better matching behavior or more consistent
> presentation. Reordering of the variants SHOULD  follow the recommendations
> for variant ordering in <xref target="choice"></xref>.</t>
>
>
>
>
>
> *Sent:* Friday, July 18, 2008 8:34 AM
> *To:* Martin Duerst
> *Cc:* Phillips, Addison; [email-masked]
> (On the side, since I *really* don't want to stir this up again: my view
> is if I pass in a tag xx-variant1-variant2 and some collection of data

---

From: Martin Duerst <[email-masked]>
Date: Sun, 20 Jul 2008 11:16:36 +0900

I'm happy with Addison's latest text, too.

At 00:33 08/07/19, Mark Davis wrote:

>(On the side, since I really don't want to stir this up again: my view is if I pass in a tag xx-variant1-variant2 and some collection of data has xx-variant2-variant1, as far as I'm concerned that should return as a perfect match: as likely as not, any difference in the ordering is purely arbitrary on the part of either the user making the first tag or the user making the second tag. And if they are a perfect match, that implies that the canonical forms should be the same. ...)

[On the side, too: If the extension subtags are independent, then yes,
ideally, they should match as you describe. But that's not what RFC 4647
says, nor is it what implementations do. So assume, for lack of better
examples, that we have two independent subtags "northern" and "western"
(I know these are bad examples, because such general variant subtags have
explicitly been rejected). Now assume that in language xyz, whether
a variant is western or not is much more important than whether it's
northern or not, so the tag order would in general be xyz-western-northern,
to allow the best possible fallback. In another language, say
zyx, it could be that northern is more important, so everybody
would use zyx-northern-western. In both cases, it would be rather
clear what to use to actual users, but just using alphabetic
order would be counter-productive, at least for zyx.

Regards,   Martin.