Subject: [Ltru] draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU

From: "Tex Texin" <[email-masked]>
Date: Mon, 2 Mar 2009 01:05:18 -0800

With respect to the proposed update to the Language Subtag Registry draft-ietf-ltru-4645bis-10:

I would like to lodge an objection to the deletion of the Preferred-Value for language subtag YU.

This change breaks the equivalence class relation between YU and CS. It detrimentally changes the behavior of existing implementations.

The loss of the relationship between YU and CS makes documents that were believed to be tagged equivalently, to no longer be equivalent.

There is also no benefit to this change.

To be concrete, assume a user attempts to find documents for languages from Yugoslavia.

Using the then current registry data, a query tool noting the preferred value relationship, matches either xx-YU and xx-CS.

Another user searches for documents for Serbia.

A query tool using the current registry data noting the preferred value relationship, matches either xx-YU and xx-CS.

The results are in some sense accurate and complete, given the history of the subtag.

After this change in the preferred value relationship, the query tool does not know to search for both xx-YU and xx-CS, since the registry does not indicate a relationship. Only one or the other subtag is used for each query. However, the query results are now incomplete since some documents for xx-YU have been tagged with the one-time preferred tag of xx-CS.

Comments in the registry are not a solution. Comments are a good thing for recording rationale and tangential history. However, implementers are not going to go thru and read the comments on any or all tags in order to make a correct implementation. They are going to implement based on the schema and operate with the data values.

The registry should stay as it is with respect to YU and retain CS as the preferred value.

As CS is now being used as a preferred value, deprecated or not, there isn't a compelling motivation to remove the preferred value for YU.

Please eliminate this needless change that breaks applications relying on the relationship between YU and CS.

tex

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 2 Mar 2009 09:20:07 -0000

I fully support this objection.

Best regards

Debbie

   _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tex
Texin
Sent: 02 March 2009 09:05
To: [email-masked]; [email-masked]
Subject: draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU

With respect to the proposed update to the Language Subtag Registry
draft-ietf-ltru-4645bis-10:

I would like to lodge an objection to the deletion of the Preferred-Value
for language subtag YU.

This change breaks the equivalence class relation between YU and CS. It
detrimentally changes the behavior of existing implementations.

The loss of the relationship between YU and CS makes documents that were
believed to be tagged equivalently, to no longer be equivalent.

To be concrete, assume a user attempts to find documents for languages from
Yugoslavia.

Using the then current registry data, a query tool noting the preferred
value relationship, matches either xx-YU and xx-CS.

A query tool using the current registry data noting the preferred value
relationship, matches either xx-YU and xx-CS.

The results are in some sense accurate and complete, given the history of
the subtag.

After this change in the preferred value relationship, the query tool does
not know to search for both xx-YU and xx-CS, since the registry does not
indicate a relationship. Only one or the other subtag is used for each
query. However, the query results are now incomplete since some documents
for xx-YU have been tagged with the one-time preferred tag of xx-CS.

Comments in the registry are not a solution. Comments are a good thing for
recording rationale and tangential history. However, implementers are not
going to go thru and read the comments on any or all tags in order to make a
correct implementation. They are going to implement based on the schema and
operate with the data values.

The registry should stay as it is with respect to YU and retain CS as the
preferred value.

As CS is now being used as a preferred value, deprecated or not, there isn't
a compelling motivation to remove the preferred value for YU.

Please eliminate this needless change that breaks applications relying on
the relationship between YU and CS.

tex

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.557 / Virus Database: 270.11.5/1979 - Release Date: 01/03/2009
17:46

No virus found in this outgoing message.
Checked by AVG. 
Version: 7.5.557 / Virus Database: 270.11.5/1979 - Release Date: 01/03/2009
17:46

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 2 Mar 2009 10:00:33 -0800

Hi -

> From: "Tex Texin" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>
> Sent: Monday, March 02, 2009 1:05 AM
> Subject: [Ltru] draft-ietf-ltru-4645bis-10.txt issue with preferred valuefor YU
>

As ltru co-chair: it's exceedingly late for such an objection - the issue was
discussed at length in the working group over a year ago.  A recent
revisiting of the question arrived at the same conclusion.

> This change breaks the equivalence class relation between YU and CS.
> It detrimentally changes the behavior of existing implementations.

As a technical contributor:

The main reason that CS does not make sense as a preferred value for
YU is that there is *not* an "equivalence class relation" between them.
There are pieces of what was YU that are not covered by CS.  To treat
them as an "equivalence class" ignores linguistic, geographic, and
political reality.

> believed to be tagged equivalently, to no longer be equivalent.

In my opinion, regarding them as equivalent is an error, since
CS and YU don't encompass the same regions.

> There is also no benefit to this change.

I disagree.  The change removes an error.

> To be concrete, assume a user attempts to find documents for languages from Yugoslavia.

Language tags do *not* pretend to be able to answer this sort of query.

Using a region subtag (e.g. 'CS') says that the data subtag uses a specific
variety of the primary language, and that the party tagging the data believes
that this distinction is useful.  For example, I could tag this paragraph with
'en' or with 'en-US'.  Is that extra distinction necessary or useful?  In this
case, no.  Consequently, the "retrieving documents by region subtag" use case,
although technically permitted by RFC 4647, is not realistic, and in many
ways contrary to the basic "tag wisely" principle.

> The results are in some sense accurate and complete, given the history of the subtag.

No, they are not.
  (1) there is no requirement, much less a guarantee, that the data will
      bear a region subtag at all
  (2) there are many bits and pieces of YU not covered by CS - 
      even if data always bore a region subtag, the YU->CS mapping
      would miss all the other territory that once belonged to YU.
  (3) blindly replacing all YU subtags with CS subtags would in fact
      falsify some data, since the language could well be of a variety
      covered by YU but not by CS.

> After this change in the preferred value relationship, the query
> tool does not know to search for both xx-YU and xx-CS, since the
> registry does not indicate a relationship. Only one or the other
> subtag is used for each query. However, the query results are now
> incomplete since some documents for xx-YU have been tagged with
> the one-time preferred tag of xx-CS.

The relationship cannot be adequately automated with a simple
one-way pointer like "preferred-value".  The former YU also
encompassed BA, HR, ME, MK, RS, and SI.

> Comments in the registry are not a solution. Comments are a good
> thing for recording rationale and tangential history. However,
> implementers are not going to go thru and read the comments on any
> or all tags in order to make a correct implementation. They are going
> to implement based on the schema and operate with the data values.

If someone (or something) is applying region subtags, they'd better
have sufficient knowledge of the language varieties to do so meaningfully.
This effectively requires *understanding* those comments and much more.
The Language Subtag Registry does *not* attempt to record all the
information needed to recognize language varieties.  Rather, once
someone (or something) has made a distinction, the LSR provides
the bits needed to encode a tag for that language variety.

In the particular case of the languages of the former YU, the
region subtags now available (such as BA, HR, ME, MK, RS, and SI)
are arguably far more useful, if someone needs to distinguish
regional variations in their Croatian-language data, than just YU.
(It's unclear to me whether YU would ever have been terribly useful,
since it would allow the distinction of Croatian as spoken there
from Croatian spoken somewhere (where?) else.)

> As CS is now being used as a preferred value, deprecated or not,
> there isn't a compelling motivation to remove the preferred value for YU.

Please, let's look at the actual tagged language data.
What corpora out there have employed YU (correctly) as a subtag?
To what extent would replacing that subtag with 'CS' (rather than
with BA, HR, ME, MK, RS, or SI) be correct, for Serbian, Croatian,
or any of the other languages of that region?

> Please eliminate this needless change that breaks applications
> relying on the relationship between YU and CS.

I would argue rather than an application that relies on an
equivalence relation between YU and CS is already in some sense
broken, in the same way as one assuming that Russia and the
Soviet Union are somehow equivalent.
 
> tex

Randy

---

From: "Tex Texin" <[email-masked]>
Date: Tue, 3 Mar 2009 13:57:26 -0800

Randy, et al.

1) I agree the equivalence is faulty. However, it has been established and
in existence for some time.
Correcting the error in the manner proposed does a greater damage.

2) If comments were a suitable solution, then it could be argued that adding
a comment to CS that despite it being a preferred value of YU it might not
be the best choice. At least then the equivalence would not be lost.

3) I agree with you and retract the statement that "there is no benefit" to
the proposed change. I instead claim the benefit does not outweigh the harm
caused.

4) I am surprised by your remarks with respect to en vs en-US. Of course the
distinction could be significant. If I want to find documents about a
"jumper", en-US vs en-GB will be significant. Or if the count is a
"billion". The fact that it might not be relevant to the one paragraph you
wrote does not mean it is irrelevant in all other statements.

Frankly it doesn't make sense to argue that the regional subtag is not
significant, but it is different and important enough to warrant breaking
the equivalence.

5) The search query was an example to demonstrate the significance of
breaking the equivalence. The problem will show itself in other linguistic
applications where the differences are significant.

6) I agree YU=> CS is an unfortunate occurrence in the registry. I disagree
we should fix it in the manner proposed.
It is one thing to optimize preferred values. It is another thing entirely
to have them changing in a way that breaks or redefines existing tags. 

7) The registry was created to ensure language tag stability. If we treat
preferred values as proposed, we lose stability.
We would be better off without preferred value relationships at all if we
can't ensure stability.

8) I do not want to undermine the standard moving forward. I do think this
particular issue is bad policy and precedent. I am not intimately familiar
with the way that IETF treats these drafts and the last call process etc.
What are the options here for addressing this without derailing the
standard? 

tex
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Randy Presuhn
Sent: Monday, March 02, 2009 10:01 AM
To: [email-masked]; [email-masked]
Subject: Re: [Ltru] draft-ietf-ltru-4645bis-10.txt issue with preferred
valuefor YU

Hi -

for language subtag YU.

As ltru co-chair: it's exceedingly late for such an objection - the issue
was
discussed at length in the working group over a year ago.  A recent
revisiting of the question arrived at the same conclusion.

Using a region subtag (e.g. 'CS') says that the data subtag uses a specific
variety of the primary language, and that the party tagging the data
believes
that this distinction is useful.  For example, I could tag this paragraph
with
'en' or with 'en-US'.  Is that extra distinction necessary or useful?  In
this
case, no.  Consequently, the "retrieving documents by region subtag" use
case,
although technically permitted by RFC 4647, is not realistic, and in many
ways contrary to the basic "tag wisely" principle.

the subtag.

No, they are not.

 

Randy

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 3 Mar 2009 14:28:30 -0800

> Correcting the error in the manner proposed does a greater damage.

Please note that Preferred-Value is not equivalence.

> to have them changing in a way that breaks or redefines existing

You know... we have this process, documented in the RFC, for resolving differences of opinion like this. It's called the registration process. I actually think that having tags canonicalize to point to 'CS' (hey, didn't that code used to be Czechoslovakia??) is less good than fixing it. But that's my opinion.

> 
> 7) The registry was created to ensure language tag stability. If we
> treat preferred values as proposed, we lose stability.

Actually, we don't. The tags themselves remain stable and valid. Canonicalization is never required. The most prevalent implementations don't canonicalize today.

> We would be better off without preferred value relationships at all
> if we can't ensure stability.

I think our definition of "stability" differs.

> What are the options here for addressing this without derailing the

It isn't a precedent, really. It's one entry (out of literally thousands) in the registry. There is a process for registering information that *is* precedent setting and rules for what is and is not permitted. I have already pointed out that you may use this process to achieve what you want. What's in 4645bis is only a template for the actual, operating registry. Once the registry is in operation, you can have a Preferred-Value request for YU and there will be a consensus and a reviewer decision and all that.

Frankly, I think that the doomsday scenarios you paint for this record are a bit alarmist. Countries have and will continue to change identity, border, form, etc. and this will impact any tagging scheme that uses region codes. The best language tag for Serbian language documents is very unlikely to be the same as it was 10 years ago and it may very well change again in the future. However, ten years from now all of the tags will still be valid and have the same meaning that they had. That is stability. Whether we can automatically equate tags such as "sh-YU" and "sr-RS" is a separate issue (and there is a long thread about language defaulting and such that is useful reading on this very topic).

If you haven't already done so, I'd urge you to read 4646bis and participate in the Last Call as necessary.

Addison

---

From: "Tex Texin" <[email-masked]>
Date: Tue, 3 Mar 2009 15:04:02 -0800

Addison,

1) You say preferred-value is not equivalence. You then refer to it as canonicalization.

I am confused. How is it not an equivalence?
If you mean that it is one directional (all dogs are mammals, all mammals are not dogs) ok. For the purposes of things tagged -YU, -CS is an equivalent.

2) Tags are not stable if one day CS is a valid substitute for YU and the next day it is not.
The meaning of CS has changed if there is an implied distinction between the two.
This is exactly what people are arguing. That YU and CS do not mean the same things and therefore YU=> CS is incorrect.
When the preferred value relationship is removed the meaning of CS is changed.

My point is that the meaning of the tags should be stable and this change does not maintain the existing meaning.

3) I don't know how many implementations canonicalize. It is a bad idea for standards to change based on "prevalent usage" unless it is known to be close to 100%. Otherwise every change will be for the benefit of the prevalent users, and over time every user gets damaged by one or another change. Standards should protect the investment made in them.

4) I have read 4646bis and considered objecting to its current treatment of preferred value. It does not offer a meaningful guideline for the circumstances by which they can be changed, it just offers that they can be modified. I was hoping that the discussion of YU would shed some light on how these issues would be resolved in practice. I would prefer not to go that route, but can do.

-----Original Message-----
From: Phillips, Addison [mailto:[email-masked]] 
Sent: Tuesday, March 03, 2009 2:29 PM
To: Tex Texin; 'Randy Presuhn'; [email-masked]
Subject: RE: [Ltru] draft-ietf-ltru-4645bis-10.txt issue with preferred valuefor YU

Addison

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 3 Mar 2009 16:48:05 -0800

Hi -
 
> To: "'Randy Presuhn'" <[email-masked]>; <[email-masked]>
> Sent: Tuesday, March 03, 2009 1:57 PM

As a technical contributor...
To which corpora?  Unless there is a substantial amount of data out there
that has meaningful YU region subtags, this would seem to be mere arm-waving.

> be the best choice. At least then the equivalence would not be lost.

As a technical contributor...
An issue for [email-masked] if someone files a registration
update request.  I would hope that such a discussion would be backed up
by concrete data about the state of Serbian, Croatian, or other relevant
languages.

> the proposed change. I instead claim the benefit does not outweigh the harm

As a technical contributor...
I would argue that there is no harm to eliminating the Preferred-Value,
while retaining it causes incorrect results for data with tags including
a YU region subtag where treating it as equivalent to a CS region subtag
would be inappropriate.

> wrote does not mean it is irrelevant in all other statements.

As a technical contributor...
The point is that there is abolutely no requirement whatsoever that all
data should bear a region subtag. It's up to the party tagging the data
to decide whether a region subtag would be useful for the purposes
for which the data is being tagged.  Consequently, once cannot use
region subtags to find all data using the language variants associated
with a particular region.  Though that might be an interesting thing to
do, language tags are not designed to support that application.

> significant, but it is different and important enough to warrant breaking

I'm not arguing that region subtags aren't meaningful or significant, I'm
simply stating the fact that there is absolutely no guarantee that they
will be present.

> applications where the differences are significant.

Doesn't matter.  If the tagger is applying coarser-grained "tag wisely"
principles than the user, this kind of application will produce misleading
results.

> to have them changing in a way that breaks or redefines existing tags. 

What tags will be broken?
What tags will be redefined?  The only "redfinition" is if some hypothetical
data bore a "YU" region subtag, removing the preferred-value prevents
it from being incorrectly redefined in terms of "CS".

> We would be better off without preferred value relationships at all if we

As co-chair: this was debated at length in the WG.  The current text
reflects of the rough consensus of the WG.  Keep in mind that
the use of Preferred-Value is RECOMMENDED, not REQUIRED, so
don't read too much into it.

> What are the options here for addressing this without derailing the

As co-chair:
I see four possible paths:
   (1) Appeal the determination of WG consensus by the co-chairs
        on procedural grounds.  I don't think this would get far, but
        if it succeeded, we'd end up re-opening the issues under (2).
   (2) Appeal the WG consensus on technical grounds.  The
         underlying issues, if I understand your argument correctly, are
         that:
           (a) the Preferred-Value field mapping behaviour should be
                 REQUIRED rather than merely recommended
           (b) the Preferred-Value field needs a replacement whose
                syntax and semantcs can cope with region splits and
                mergers
           (c) the results of applying a canoncalization algorithm
                must be stable over time
        Note that *all* of these questions have been discussed
        at length in the WG.  A re-work, changing the WG consensus
        on all three of these points, would probably be lengthy.
   (3) Raise the issue during IETF last call.  If the IETF consensus
         was that the WG got it wrong, we'd effectively travel the same
         path as (2).
   (4) Submit a subtag registration update request on [email-masked]
         after IANA has updated the registry, and try to convince the
         language subtag reviewer that a preferred-value for YU
         should be added.

Randy