Subject: Re: [Ltru] Proposed resolution for Issue 13 (language tags)

From: Martin Duerst <[email-masked]>
Date: Tue, 15 Apr 2008 11:47:19 +0900

I have copied the IETF LTRU (Language Tag Registry Update) WG
to make sure they can help get this right.

LTRU WG - this is a copy of a reply to a mail on the HTTP WG
mailing list about language tags. The spec being updated is
http://www.ietf.org/rfc/rfc2616.txt. See in particular
sections 3.10, Language Tags, 14.4, Accept-Language, and
14.12, Content-Language.

Please stop cross-posting for issues that don't affect both groups!
Please change the Subject if you change the subject!

At 01:02 08/04/15, Julian Reschke wrote:
>
>Hi,
>
>(see also <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/13>).
>
>So far we have delayed the resolution 13 until we're done with the ABNF conversion, but since then we have made other changes to the ABNF anyway. Therefore, I'd like to get this one resolved; there's doesn't seem to be anything holding it up...:
>
>Section 3.5., paragraph 2:
>OLD:
>
>    The syntax and registry of HTTP language tags is the same as that
>    defined by [RFC1766].  In summary, a language tag is composed of 1 or
>    more parts: A primary language tag and a possibly empty series of
>    subtags:
>
>NEW:
>
>    The syntax and registry of HTTP language tags is the same as that
>    defined by [RFC4646].  In summary, a language tag is composed of one
>    or more parts: A primary language tag and a possibly empty series of
>    subtags,

The above text gives the impression that there is a separate
concept of a "HTTP language tag". Why not just say something
like "HTTP uses language tags as defined in ...".

Also, with RFC 4646, any further (currently being worked on by the LTRU WG)
extensions (not in syntax, but in the number of languages covered) might
be excluded. People have been wondering e.g. whether they can use
RFC 3066 or RFC 4646 language tags with RFC 2616, we don't want that
to happen again. RFC 4646 (and RFC 4647, which defines matching) can
be referenced as BCP 47, which doesn't have to be updated even if
a new RFC makes more language tags available. The basic syntax
is still the same. So I strongly suggest you reference BCP 47
rather than a specific RFC.

>Section 3.5., paragraph 3:
>OLD:
>
>      language-tag  = primary-tag *( "-" subtag )
>      primary-tag   = 1*8ALPHA
>      subtag        = 1*8ALPHA
>
>NEW:
>
>      language-tag  = <Language-Tag, defined in [RFC4646], Section 2.1>

See above.

>Section 3.5., paragraph 4:
>OLD:
>
>    White space is not allowed within the tag and all tags are case-
>    insensitive.  The name space of language tags is administered by the
>    IANA.  Example tags include:
>
>NEW:
>
>    White space is not allowed within the tag and all tags are case-
>    insensitive.  The name space of language tags is administered by the
>    IANA (<http://www.iana.org/assignments/language-tags>).

As you can see on that page, the registry of full language tags is
obsolete. It has been replaced by the language subtag registry, at
http://www.iana.org/assignments/language-subtag-registry.

>    Example tags include:
>
>
>Section 3.5., paragraph 6:
>OLD:
>
>    where any two-letter primary-tag is an ISO-639 language abbreviation
>    and any two-letter initial subtag is an ISO-3166 country code.  (The
>    last three tags above are not registered tags; all but the last are
>    examples of tags which could be registered in future.)
>
>NEW:
>
>    (The last three tags above are not registered tags; all but the last
>    are examples of tags which could be registered in future.)

This has to be reworded. en-US is a tag allowed based on the current
subtag registrations. I'm not totally sure about en-cockney and i-cherokee.
The LTRU WG can provide more or different examples.

>    See RFC 4646 for further information.

Again, better use BCP 47.

For 14.4, Accept-Language, please note that BCP 47 (RFC 4647 currently)
also defines a language-range, probably the same as you have, so you
should reference that. There are also various variants for matching
predefined; you should be able to choose the one that fits your needs
best and then only have to define a few details.

>...feedback appreciated.

Great. I'm sure that with the cross-posting, you'll get some more.

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Apr 2008 13:33:08 +0200

OK,

thanks for all the feedback so far. I (hopefully) have addressed many of 
the issues; here's the new proposed text for 3.5:

------
3.5.  Language Tags

    A language tag, as defined in [RFC4646], identifies a natural
    language spoken, written, or otherwise conveyed by human beings for
    communication of information to other human beings.  Computer
    languages are explicitly excluded.  HTTP uses language tags within
    the Accept-Language and Content-Language fields.

    In summary, a language tag is composed of one or more parts: A
    primary language tag and a possibly empty series of subtags:

    White space is not allowed within the tag and all tags are case-
    insensitive.  The name space of language subtags is administered by
    the IANA (see
    <http://www.iana.org/assignments/language-subtag-registry>).

    Example tags include:

      en, en-US, es-419, az-Arab, x-pig-latin, man-Nkoo-GN

------

(see also <http://www.tools.ietf.org/wg/httpbis/trac/ticket/13>).

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Apr 2008 13:25:36 +0200

> like "HTTP uses language tags as defined in ...".

Ack.

> rather than a specific RFC.

As we're including the Language-Tag ABNF production by reference, I'd 
prefer to stick with a fixed reference.

> http://www.iana.org/assignments/language-subtag-registry.

ACK.

> The LTRU WG can provide more or different examples.

I think the simplest fix is just to remove the statement.

> best and then only have to define a few details.

Good catch; I'd prefer to deal with this separately.

> ...

BR, Julian

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 15 Apr 2008 10:26:32 -0700

Hi,

I have a number of thoughts on the proposed changes, which appear below my signature.

I would suggest that HTTPbis wait in order to reference 4646bis, which is in WG last call, if at all possible. I mention this for two reasons:

1. It would be better to reference the update than the soon-to-be-obsolete version, even though the differences are minor.

2. The new version includes an ABNF production of value to HTTPbis (obs-language).

Best Regards,

Addison

Addison Phillips
Globalization Architect -- Lab126 (Amazon)
Chair -- W3C Internationalization Core WG

Internationalization is not a feature.
It is an architecture.

>
> Martin Dürst wrote comments on Julian's email which said in part:
> >

> like "HTTP uses language tags as defined in ...".

I agree with Martin here. However, it may be useful to reference the RFC 3066 Language-Tag production in Section 2.2.9, for compatibility with existing RFC 2616 implementations, and to specify "well-formed" conformance.
So I strongly suggest you reference BCP 47
> rather than a specific RFC.

+1

> >      language-tag  = <Language-Tag, defined in [RFC4646], Section

It may be better to reference Language-Tag as defined in 2.2.9 for compatibility. While it would be good to adopt the modern language tag ABNF, that would suggest that receivers reject tags that were well-formed but no longer are.

> subtag registrations. I'm not totally sure about en-cockney and i-

"en-cockney" is not valid, nor is "i-cherokee". "i-cherokee" can NEVER be registered. The "cockney" subtag could be registered. However, there is no need to use artificial examples (as there was when RFC 2616 was written). It would be better to use examples from RFC 4646/4646bis.

Note: Cherokee is in the registry with the subtag 'chr'.

> best and then only have to define a few details.

Actually, Accept-Language is an example of a "language priority list" in RFC 4647. However, RFC 4647 doesn't define the exact syntax for a language priority list. The "language-range" production *is* defined by RFC 4647, as a "Basic Language Range".

The text about matching of language ranges and tags in RFC 2616 is the same as the Basic Filtering algorithm--in fact, the latter was designed to be identical. However, I think that choosing this algorithm explicitly would be a Bad Thing. Sometimes the Lookup algorithm is the right choice for an application (as when selecting user interface language for an application), while at other times one of the Filtering algorithms makes more sense (as when selecting search results).

Note that the text in RFC 2616 should have been superseded by RFC 3282. Ultimately, I think that 2616bis should define Accept-Language using the existing ABNF but adopting the terminology in RFC 4647. It should then guide users to BCP 47 (RFC 4647) regarding the selection of matching algorithms.

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 15 Apr 2008 10:29:38 -0700

"No sooner do I push send..."

This text would work for me.

Addison

Addison Phillips

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Julian Reschke
> Sent: mardi 15 avril 2008 04:33
> To: HTTP Working Group; LTRU Working Group
> Subject: Re: [Ltru] Proposed resolution for Issue 13 (language tags)
>
> OK,
>

---

From: Martin Duerst <[email-masked]>
Date: Wed, 16 Apr 2008 16:01:32 +0900

At 20:33 08/04/15, Julian Reschke wrote:
>
>OK,
>
>thanks for all the feedback so far. I (hopefully) have addressed many of the issues; here's the new proposed text for 3.5:
>
>------
>3.5.  Language Tags
>

This is going very much in the right direction, but there
is some confusion about what a subtag means. Where it says
"A primary language tag and a possibly empty series of subtags",
the first tag (e.g. 'en' in 'en-US') seems to not be a subtag,
but the subtag registry clearly also includes that as a subtag,
and RFC 4646 is using terminology consistent with that.

Regards,   Martin.

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 16 Apr 2008 10:20:34 +0100

I would agree with Martin and propose the following changes:

From:
---
> >    primary language tag and a possibly empty series of subtags:

To:
---

In summary, a language tag is composed of one or more parts: A
primary language subtag followed by a, possibly empty, series of subtags:

---

best regards

Debbie Garside

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Martin Duerst
> Sent: 16 April 2008 08:02
> To: Julian Reschke; HTTP Working Group; LTRU Working Group
> >thanks for all the feedback so far. I (hopefully) have
> addressed many of the issues; here's the new proposed text for 3.5:
> >
> >------
> >3.5.  Language Tags
> >
> This is going very much in the right direction, but there is
> some confusion about what a subtag means. Where it says "A
> primary language tag and a possibly empty series of subtags",
> the first tag (e.g. 'en' in 'en-US') seems to not be a
> subtag, but the subtag registry clearly also includes that as
> a subtag, and RFC 4646 is using terminology consistent with that.
>
> Regards,   Martin.
>
> >    Example tags include:
> >

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 16 Apr 2008 06:46:09 -0700

Agreed. Even that is incorrect, since there may not be a primary language
subtag, such as in the case of "x-duerst". So I'd recommend something more

>    primary language tag and a possibly empty series of subtags:
=>
In summary, a language tag is composed of one or more parts: typically a
primary language subtag followed optionally by a series of other subtags
such as script and region.

-- 
Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 16 Apr 2008 07:59:40 -0700

Julian replied:

> > It may be better to reference Language-Tag as defined in 2.2.9 for
> compatibility. While it would be good to adopt the modern language tag
> ABNF, that would suggest that receivers reject tags that were well-
> formed but no longer are.
>
> For the record: that's in RFC4646bis, but not in RFC4646, right?

Yes, exactly so.

> Now that would be a good reason.

I'm of two minds here:

I would prefer not to reference the 3066-compatible syntax. Language tags which match that production but not 4646/4646bis's have never been valid and the majority of "bad tags" in the wild turn out to match the newer ABNF anyway.

However, some bad tags in the wild might not. Perhaps reference the Section 2.1 ABNF but with a cautionary note about the change?

>
> Are you saying that HTTP should use that production (obs-language,
> <http://www.tools.ietf.org/html/draft-ietf-ltru-4646bis-12#section-
> 2.2.9>),

(wearing editor hat) I erred in referencing the production name here. In draft-12 it is (sadly) called "Language-Tag" too. I will change the production name to make it distinct in the next-and-hopefully-last version.

Best Regards,

Addison

Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 16 Apr 2008 08:00:46 -0700

Mark wrote:
--
In summary, a language tag is composed of one or more parts: typically a primary language subtag followed optionally by a series of other subtags such as script and region.
--
I think that is either too specific or not specific enough ☺

Perhaps:

In summary, a language tag is composed of one or more subtags separated by hyphens. Typically there will be a primary language subtag to which other subtags are sometimes appended to indicate linguistic variations such as script or region.

Addison Phillips

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: mercredi 16 avril 2008 06:46
To: Martin Duerst
Cc: Julian Reschke; LTRU Working Group; HTTP Working Group
Subject: Re: [Ltru] Proposed resolution for Issue 13 (language tags)

>    primary language tag and a possibly empty series of subtags:
=>
In summary, a language tag is composed of one or more parts: typically a primary language subtag followed optionally by a series of other subtags such as script and region.
On Wed, Apr 16, 2008 at 12:01 AM, Martin Duerst <[email-masked]<mailto:[email-masked]>> wrote:

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]<mailto:[email-masked]>

_______________________________________________
Ltru mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 16 Apr 2008 08:58:34 -0700

That works for me.

On Wed, Apr 16, 2008 at 8:00 AM, Phillips, Addison <[email-masked]>
wrote:

> I think that is either too specific or not specific enough J
>
>
>
> Perhaps:
>
>
>
> In summary, a language tag is composed of one or more subtags separated by
> hyphens. Typically there will be a primary language subtag to which other
> subtags are sometimes appended to indicate linguistic variations such as
> script or region.
>
>
>
> Addison Phillips
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Mark Davis
> *Sent:* mercredi 16 avril 2008 06:46
> *To:* Martin Duerst
> *Cc:* Julian Reschke; LTRU Working Group; HTTP Working Group
> *Subject:* Re: [Ltru] Proposed resolution for Issue 13 (language tags)
>
>
>

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 16 Apr 2008 17:30:27 +0100

Rather than re-inventing the wheel, why not just go with what is already
written within RFC4646bis-12 
----
   The language tag is composed of one or more parts, known as

   "subtags".  Each subtag consists of a sequence of alphanumeric

   characters.  Subtags are distinguished and separated from one another

   by a hyphen ("-", ABNF [RFC5234] %x2D).  A language tag consists of a

   "primary language" subtag and a (possibly empty) series of subsequent

   subtags, each of which refines or narrows the range of languages

   identified by the overall tag.
 
----
By adding the information with regard to script and region you may encourage
people to go ahead and create their own tags as opposed to reading
RFC4646bis.  
There are many other issues to be considered e.g. suppress script.
Best regards
Debbie Garside

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark
Davis
Sent: 16 April 2008 16:59
To: Phillips, Addison
Cc: Julian Reschke; LTRU Working Group; HTTP Working Group
Subject: Re: [Ltru] Proposed resolution for Issue 13 (language tags)

That works for me.

Mark wrote:

--

--

Perhaps:

Addison Phillips

It is an architecture.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark
Davis
Sent: mercredi 16 avril 2008 06:46
To: Martin Duerst
Cc: Julian Reschke; LTRU Working Group; HTTP Working Group 

 

=>

Regards,   Martin.

Ltru mailing list
[email-masked]

-- 
Mark 

-- 
Mark

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Apr 2008 15:01:16 +0200

> primary language subtag followed by a, possibly empty, series of subtags:

Hi Debbie,

thanks for the proposed text (which I'll include).

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Apr 2008 15:11:14 +0200

Are you saying that HTTP should use that production (obs-language, 
<http://www.tools.ietf.org/html/draft-ietf-ltru-4646bis-12#section-2.2.9>), 
instead of Language-Tag 
(<http://www.tools.ietf.org/html/draft-ietf-ltru-4646bis-12#section-2.1>)?

I'm no expert on that matter... do we have consensus on this among those 
who are?

> I agree with Martin here. However, it may be useful to reference the RFC 3066 Language-Tag production in Section 2.2.9, for compatibility with existing RFC 2616 implementations, and to specify "wel…

In which case it may be simpler to just align the definition in HTTPbis 
with that production (instead of referring directly to RFC3066).

>> rather than a specific RFC.

I don't see how, as long as we want to include a specific ABNF production.

BR, Julian