Subject: RE: [Ltru] conformance requirements for matching

From: "Addison Phillips" <[email-masked]>
Date: Sat, 1 Apr 2006 10:33:29 -0800

> I disagree with the wholesale removal. I think it is perfectly
> reasonable for someone to claim conformance with additional
> qualifications. That is done in quite a few other specifications.

I agree (although maybe we were discussing different text?!?), but think the problem is trying to rework the text as it stands rather than spell out implementation-chosen detail. The problem with the current text is that it makes interoperability more difficult. The text currently says:

--
The types of matching in this document are designed so that implementations are not required to validate or understand any of the semantics of the language tags or ranges or of the subtags in them. None of them require access to the IANA Language Subtag Registry (see Section 3 in [RFC3066bis]). This simplifies implementation. Applications, protocols, or specifications MUST NOT depend on the language ranges being "well-formed" or "valid" (see [RFC3066bis], Section 2.2.9).

Regardless of the matching scheme chosen, protocols and implementations MAY canonicalize language tags and ranges by mapping grandfathered and obsolete tags or subtags into modern equivalents. If an implementation canonicalizes either ranges or tags, then the implementation will require the IANA Language Subtag Registry information for that purpose. Implementations MAY also use semantic information external to the registry when matching tags. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an implementation might infer that content labeled "zh-Hans" (Chinese as written in the Simplified script) is more likely to match the range "zh-CN" (Chinese as used in China, where the Simplified script is predominant) than equivalent content labeled "zh-TW" (Chinese as used in Taiwan, where the Traditional script is predominant).
--

Instead of this, I propose:

--
<t>The matching schemes in this document do not require an implementation to validate or understand any of the semantics of the language tags or ranges or of the subtags in them, nor do they require access to the IANA Language Subtag Registry (see Section 3 in <xref target="RFC3066bis"></xref>). This simplifies implementation.</t>

<t>However, applications, protocols, or specifications are encouraged to use information in the IANA Language Subtag Registry to canonicalize language tags and ranges, mapping grandfathered and obsolete tags or subtags into modern equivalents. Such an implementation MAY also use semantic information contained in the tags to augment matching.</t>

<t>For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an implementation might infer that content labeled "zh-Hans" (Chinese as written in the Simplified script) is more likely to match the range "zh-CN" (Chinese as used in China, where the Simplified script is predominant) than equivalent content labeled "zh-TW" (Chinese as used in Taiwan, where the Traditional script is predominant).</t>
--

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature. 
> -----Original Message-----
> From: Mark Davis [mailto:[email-masked]]
> Sent: 2006年3月31日 19:16
> To: Addison Phillips
> Cc: 'Randy Presuhn'; 'LTRU Working Group'
> Subject: Re: [Ltru] conformance requirements for matching
> 
> I disagree with the wholesale removal. I think it is perfectly
> reasonable for someone to claim conformance with additional
> qualifications. That is done in quite a few other specifications.
> 
> For example, I think it is perfectly reasonable to state the following.
> 
> The implementation is conformant to RFC xxx Lookup, with the following
> extensions:
> - default content matches the default locale as specified in
> setDefaultLocale()
> - closely related variants, such as no, nn, nb, fallback to each other
> first, before resorting to the default content. The exact mapping is
> found with getVariantFallbacks().
> 
> Mark
> 
> Addison Phillips wrote:
> >> This would be absurd, and this text should be removed.
> >>
> >
> > I have no problem with removing that text.
> >
> >
> >> In its current form, it's not even clear how a protocol specification
> >> referencing this document is supposed to identify the portions of the
> >> matching specification it employs.
> >>
> >
> > This I don't understand. There are three mechanisms. They are clearly
> named and identified. What would you suggest we do differently in the text
> to make implementation decisions clear?
> >
> > There are clearly a large number of potential applications that can use
> these schemes (ranging from language negotiation of web pages to applying
> styles to paragraphs to selecting content in a database). The actual
> protocol or application has to define how the scheme chosen works in the
> context of this higher level application of matching. Specifically, I'd
> refer you to Harald's comment:
> >
> >    http://article.gmane.org/gmane.ietf.ltru/4180/match=protocol
> >
> > (The text suggested in this message appears in Section 3.1)
> >
> > Can you identify examples of what the problem is? I have no idea how we
> would address this.
> >
> >
> >> As a WG chair and as an I-D author / editor I know how frustrating
> >> very general comments can be
> >>
> >
> > Yeah. And you saved one for the last day of the Last Call too :-).
> >
> > Addison Phillips
> > Internationalization Architect - Yahoo! Inc.
> >
> > Internationalization is an architecture.
> > It is not a feature.
> >
> >
> >> -----Original Message-----
> >> From: Randy Presuhn [mailto:[email-masked]]
> >> Sent: 2006年3月30日 13:41
> >> To: LTRU Working Group
> >> Subject: [Ltru] conformance requirements for matching
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Ltru mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ltru
> >
> >
> >

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 1 Apr 2006 16:57:24 -0800

Hi -

> From: "Mark Davis" <[email-masked]>
> To: "Addison Phillips" <[email-masked]>
> Cc: "'Randy Presuhn'" <[email-masked]>; "'LTRU Working Group'" <[email-masked]>
> Sent: Friday, March 31, 2006 7:16 PM
...

But the current text isn't just about extensions. Section three says:

   There are several types of matching scheme.  This document presents
   two types: those that produce zero or more information items (called
   "filtering") and those that produce a single information item for a
   given request (called "lookup").

   Implementations or protocols MAY use different matching schemes from
   the ones described in this document, as long as those mechanisms are
   clearly specified.

It doesn't talk about extensions or augmentations or
"additional qualifications." 

It says "different".

That is *far* too loose.

Furthermore, even if we do permit conformant implementations to have
extensions, there needs to be some way of ensuring that they don't
affect interoperability.  If they do, then they defeat the whole
point of claiming conformance to a specification.

I don't care whether implementations support other algorithms, but when
they claim to support one from this document, the observable behaviour
must be exactly what is specified here, within the "wiggle room" of
the defined options.  Implementations might do other things as well,
but those are outside the scope of this specification, and we really
don't need to talk about them.

Randy

---

From: "Addison Phillips" <[email-masked]>
Date: Mon, 3 Apr 2006 13:17:09 -0700

Okay... in the interest of expediency, I have implemented a bunch of edits in the Editor's copy of draft-12 in an attempt to bring both sides of this discussion together. I tried to quote them all in an email, but the rewrites involved a lot of paragraphs of text. It is easier to view the diff and see what I changed.

http://www.inter-locale.com/ID/matching-diff-11-12.html

The notable differences are:

1. Insertion of the "seems to imply" text into Section 2.1

2. A complete rewrite of the text describing the mapping of extended language ranges to basic ones in Section 2.2

3. Upgrading of SHOULD to MUST in conformance requirements and elimination of the text allowing unspecified algorithms to be used in Section 3.0

4. A total rewrite of the material about canonicalization of tags/ranges and understanding the semantics of language tags in Section 3.1

5. A full rewrite of Section 3.2.2 (Extended Filtering) to include a step-by-step algorithm.

6. Insertion of Martin's text at the end of 3.2.2

7. A number of changes to Section 3.3 (Lookup) to make the return of default content a requirement and to be specific that implementations specify what the content is and how it is arrived at.

8. A rewrite of the text on extensions in Section 3.3 to make it more specific.

9. Removed the SHOULDs in the paragraph about the range "*" in Section 3.3 (this makes the specified behavior obligatory).

10. Added Martin's texts to Section 4.1 on subtag selection.

11. Removed the application specific directions in Section 4.1. One of these makes a modified reappearance in Section 3.2.2 as part of the text about removing extensions.

Randy, Mark (and anyone else who cares): you comments on the relevant changes would be most appreciated.

Obviously the diff shows everything. Since substantive changes were made, does this mean another trek through WG Last Call? Also, should I publish this as draft-12? Martin: what is the process from here?

Addison

Addison Phillips

> Sent: 2006年4月1日 16:57

---

From: John Cowan <[email-masked]>
Date: Mon, 3 Apr 2006 16:26:03 -0400

Addison Phillips scripsit:

> Okay... in the interest of expediency, I have implemented a bunch of
> edits in the Editor's copy of draft-12 in an attempt to bring both
> sides of this discussion together. I tried to quote them all in an
> email, but the rewrites involved a lot of paragraphs of text. It is
> easier to view the diff and see what I changed.

+1.  I have no further comments.

-- 
The experiences of the past show                John Cowan
that there has always been a discrepancy        [email-masked]
between plans and performance.                  http://www.ap.org
        --Emperor Hirohito, August 1945         http://www.ccil.org/~cowan

---

From: Mark Davis <[email-masked]>
Date: Mon, 03 Apr 2006 15:22:48 -0700

This might be an artifact of the diff, but looks like a typo.

 > (see: ).

Other than that, I like the rewrite. +1
Mark

Addison Phillips wrote:
>>> Cc: "'Randy Presuhn'" <[email-masked]>; "'LTRU Working
>>>       
>> Group'" <[email-masked]>
>>

---

From: "Addison Phillips" <[email-masked]>
Date: Mon, 3 Apr 2006 16:08:59 -0700

> This might be an artifact of the diff, but looks like a typo.

Good catch.

It's a typo: the format attribute of the xref is set to "none" instead of "default" or "title". I've fixed it in my local copy and will post it when I get a chance later tonight.

Addison

Addison Phillips

> Sent: 2006年4月3日 15:23
> edits in the Editor's copy of draft-12 in an attempt to bring both sides
> of this discussion together. I tried to quote them all in an email, but
> the rewrites involved a lot of paragraphs of text. It is easier to view
> the diff and see what I changed.
> >
> > 2. A complete rewrite of the text describing the mapping of extended
> language ranges to basic ones in Section 2.2
> >
> > 3. Upgrading of SHOULD to MUST in conformance requirements and
> elimination of the text allowing unspecified algorithms to be used in
> Section 3.0
> >
> > 4. A total rewrite of the material about canonicalization of tags/ranges
> and understanding the semantics of language tags in Section 3.1
> >
> > 7. A number of changes to Section 3.3 (Lookup) to make the return of
> default content a requirement and to be specific that implementations
> specify what the content is and how it is arrived at.
> >
> > 9. Removed the SHOULDs in the paragraph about the range "*" in Section
> 3.3 (this makes the specified behavior obligatory).
> >
> > 11. Removed the application specific directions in Section 4.1. One of
> these makes a modified reappearance in Section 3.2.2 as part of the text
> about removing extensions.
> >
> > Obviously the diff shows everything. Since substantive changes were made,
> does this mean another trek through WG Last Call? Also, should I publish
> this as draft-12? Martin: what is the process from here?
> >
> > Addison
> >
> > Addison Phillips

---

From: Martin Duerst <[email-masked]>
Date: Tue, 04 Apr 2006 15:23:06 +0900

[co-chair hat off]

I was glad we cut back the spec to elimitate the too many options
we gave users. Now Randy is saying that there are still too many
options.

 >> found with getVariantFallbacks().

Very good. If we were not already at this stage, and maybe even
at this stage, I'd propose that we work out a few more such examples
of how to specify specifics (using specification rather than implementation
in some examples). This is a very useful exercise, and might also make
a nice addition to our document (e.g. in an Appendix).

Mark, you seem to be good at these things, any other proposals?
Anybody else?

 >That is *far* too loose.

I think this language got here in part due to the fact that we are
(part of a) BCP, and in part because some of the WG members
(including at least me) didn't think that a sentence in a spec
along the lines of "hey, there may be other ways to do what we
do here, this may not be the only one" would be interpreted as
"I can claim that I'm conforming even if I don't do anything of
what this spec says.".

 >point of claiming conformance to a specification.

Again, we are (part of) a BCP. In that sense, I see it as the
document's main job to say "given the language tags in RFC 3066bis,
here is we think how it makes most sense to match them for
filtering and lookup".

I do not currently see much of an interoperability problem.
First, we have an additional layer of actual specifications.
It is ultimately not our document, but specs such as HTTP,
SMTP, POP, IMAP, CSS, XSLT, whatever,... that define what
implementations have to do. If these specs actually choose
the options we give them, they should lead to interoperable
implementations (for each spec).

We could try to achieve interoperability on a wider level.
Our goal could be to make sure that users have portable
(not only among implementations, but also among specs/protocols)
language preferences, and content can be portably (same as above)
used. But as I have explained in my last call comments, for a
case such as zh/zh-Hans/zh-Hant, we already have blown that
as soon as we both offer filtering and lookup.

Also, regarding options, I'm not affraid to leave options open
for specs to fix. The main criterion I personally care about is
not whether we are totally precise, but to what extent giving
options to other spec writers will help them get the best
solution for their case, and to what extent it will get them
into troubles because they have no clue which option to choose.

Regards,   Martin.

---

From: Mark Davis <[email-masked]>
Date: Tue, 04 Apr 2006 10:09:49 -0700

Addison's revised text (see from yesterday) is fine by me.

Mark

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 4 Apr 2006 12:21:13 -0700

Hi -

As a technical contributor...

> From: "Martin Duerst" <[email-masked]>
> To: "Randy Presuhn" <[email-masked]>; "LTRU Working Group" <[email-masked]>
> Sent: Monday, April 03, 2006 11:23 PM
...
...

I'm not arguing against options per se, though I'd be happy to do so.

My point is we should provide a clear means for specifications using
the BCP to identify the options used for each matching algorithm;
and that we should make it clear which options must be nailed down by
specifications using the BCP, and which options may be left entirely
to implementations.

For example, in the editor's copy (I'd rather reference the i-d, but in the
interest of moving forward quickly I'll use the text at
http://www.inter-locale.com/ID/matching-diff-11-12.html) it says:

| Applications, protocols, or specifications that use basic ranges
| might sometimes receive extended language ranges instead.  An
| application, protocol, or specification MUST choose to a) map
| extended language ranges to basic ranges using the algorithm below,
 | b) skip over (or reject) any extended language ranges in the language
| priority list that are not valid basic language ranges, or c) treat
| each extended language range as if it were a basic language range
| (these ranges will match *no* content).

I think (b) as currently formulated overlaps (c).  I'd suggest a slight
reformulation as:
  b) reject any extended language ranges in the language
  priority list that are not valid basic language ranges, or c) treat
  each extended language range as if it were a basic language range,
  which will have the same result as ignoring them, since these ranges
  will match *no* content.

In any event the text is an improvement over the previous version
(though I'd strike the anthropomorphic "choose to"), but doesn't give
a concise name or other way to refer to this particular set of options.
I suggest:
(1) that the paragraph cited above be made part of a new subsection,
"2.2.1 Extended Language Range Compatibility Modes".
(2) that the following sentence be added to the end of this paragraph:
  This choice of compatibility mode is to be documented as "map",
  "reject", and "ignore".

Randy

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 4 Apr 2006 14:16:57 -0700

Comments follow.

Addison

Addison Phillips

> Sent: 2006?4?4? 12:21

If you do, try to be specific, please :-).
> 
> For example, in the editor's copy (I'd rather reference the i-d, but in
> the interest of moving forward quickly I'll use the text at

I don't quite see that. Skipping a range means not returning any content
that might match that range. Rejecting a range might potentially be an error
condition. Treating an extended language range as if it were a basic
language range, by contrast, will cause the matching algorithm to attempt to
match the content. Although the results should be the same, they might
potentially not be.

Nonetheless, your text is clearer than mine. I'll use yours.

> "2.2.1 Extended Language Range Compatibility Modes".

That doesn't flow very nicely. Instead of that, why don't we put it where it
belongs, as a new subsection following 3.1 and called "Implementation
Considerations". This could also, then, include the last four paragraphs in
the current S3.1.

>   This choice of compatibility mode is to be documented as "map",

That's kind of icky. Let me play with it a bit.