Subject: [Ltru] Tag perversity text revised... (#944)

From: "Addison Phillips" <[email-masked]>
Date: Tue, 7 Jun 2005 11:26:25 -0700

All,

Some time has passed and many emails have flowed about the most perverse tag, its relation to RFC 2047, and so forth.

Based on the all of this thread, I have prepared a substantially revised text for section 2.1.1. This incorporates the 33-character "most perverse generative tag" and my suggested text about handling user choice and language tag truncation and a bunch of other suggestions (Peter, John, Frank, etc), combined with a through-going rewrite. This version is posted in HTML and text formats at the following URIs:

http://www.inter-locale.com/ID/draft-ietf-ltru-registry-04a.html
http://www.inter-locale.com/ID/draft-ietf-ltru-registry-04a.txt

Basically the remaining issue I see (apart from fixing any text that I created) is deciding which lower-limit to endorse and whether to use SHOULD or MUST to endorse said limit. I currently show the proposed options as:

  - 33 (in this text): longest 3066ter generative
  - 42: longest perverse tag, no private use
  - 62: longest perverse tag, 2 private use
  - 26: longest 3066bis generative
  - none: who are we to propose limits?

Here is the text of the revised section:

--
2.1.1  Length Considerations

   RFC 3066 [23] did not provide for an upper limit on the size of
   language tags.  While the largest tag that could be generated from
   language and region codes defined under RFC 3066 could not exceed six
   characters in length, much larger registered tags were not only
   possible but were actually registered.

   The ABNF and other guidelines in this document also do not impose a
   fixed upper limit on the number of subtags in a Language Tag (and
   thus the upper bound on the size of a tag) and it is possible to
   envision quite long and complex subtag sequences.  The upper bound on
   generative subtag combinations has expanded and, depending on the
   specific language, may require more characters to form a complete
   tag.

   In practice, most tags will not require substantially more
   characters.  This is partly because additional granularity in tags
   seldom adds useful distinguishing information and because longer,
   more granular tags interfere with the meaning, understanding, and
   processing of language tags.  Guidelines in the registry (see
   Section 3.1) and in Choice of Language Tag also limit the way in
   which subtags may be combined to form meaningful, valid tags.

   For example, variant subtags SHOULD be used only with the prefix
   specified in the registry.  This limits most tags to a maximum of
   four subtags, and thus a maximum length of 26 characters (excluding
   any extensions or private use sequences).  A future revision of this
   document may provide for the registration of extended language
   subtags (see Section 2.2.2): useful tags that include extended
   language subtags extend this limit to 33 characters, due to the
   limitations on subtag length and combinations imposed by this
   document.

   Some applications and protocols must allocate fixed buffer sizes or
   otherwise limit the length of a language tag in a particular
   application.  A conformant implementation or specification MAY refuse
   to support the storage of language tags which exceed a specified
   length.  For an example, see [RFC 2231] [22].  Any such limitation
   SHOULD be clearly documented, and such documentation SHOULD include
   the disposition of any longer tags (for example, whether an error
   value is generated or the language tag is truncated).  This limit
   SHOULD be at least 33 characters in length.  If truncation is
   permitted it MUST NOT permit a subtag to be divided.  Implementations
   that restrict storage should consider removing extensions before
   processing.  A protocol that allows tags to be truncated at an
   arbitrary limit, without giving any indication of what that limit is,
   has the potential for causing harm by changing the meaning of tags in
   substantial ways.  Here is how the 33-character length of the longest
   generated tag is derived:

   language      = 3
   extlang1      = 4 (each subsequent subtag includes '-')
   extlang2      = 4 (unlikely: needs prefix="language-extlang1")
   extlang3      = 4 (extremely unlikely)
   script        = 5 (must not be suppressed)
   region        = 4 (UN M.49; ISO 3166 requires 3)
   variant1      = 9 (must have language as a prefix)

   total         = 33 characters

             Figure 2: Derivation of the Longest Generated Tag

   Since language tags longer than 33 characters may be truncated by an
   application or protocol that limit tag sizes, when choosing language
   tags users and applications SHOULD avoid adding subtags that add no
   distinguishing value and SHOULD follow fields in the registry (such
   as Prefix and Suppress-Script) that limit subtag composition.  See
   Choice of Language Tag.

   Applications or protocols that must truncate a tag should first
   remove all extension subtags.  If the tag is still too long, the tag
   should then be truncated by progressively removing subtags from the
   right of the tag.  Note that private use subtags are removed first.
   When removing private use subtags, the singleton '-x-' should be
   removed along with the last remaining private use subtag.

   Note: The longest tag that might be generated based on the
   grandfathered tags (see Section 2.2.8) that contain subtags not
   otherwise defined by this document (and in a form that could be
   registered as a variant) requires 20 characters (an example of this
   tag might be "cel-Latn-155-gaulish").

   The longest possible tags, excluding extensions, that might have some
   marginal utility could have a length of up to 62 characters, as shown
   below.  The longest such tag without private use subtags requires 42
   characters.

   Here is how the 62-character length of the longest practical tag
   (excluding extensions) is derived:

   language      = 3
   extlang1      = 4 (each subsequent subtag includes '-')
   extlang2      = 4 (unlikely: needs prefix="language-extlang1")
   extlang3      = 4 (extremely unlikely)
   script        = 5 (must not be suppressed)
   region        = 4 (UN M.49; ISO 3166 requires 3)
   variant1      = 9
   variant2      = 9 (unlikely: needs prefix="language-variant1")
   private use 1 = 11 ("-x-" + subtag)
   private use 2 = 9
   total         = 62 characters

                  Figure 3: Derivation of the Longest Tag
--

Addison P. Phillips
Globalization Architect, Quest Software
http://www.quest.com

Chair, W3C Internationalization Core Working Group
http://www.w3.org/International

Internationalization is not a feature.
It is an architecture. 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "John.Cowan" <[email-masked]>
Date: Tue, 7 Jun 2005 14:55:17 -0400

> http://www.inter-locale.com/ID/draft-ietf-ltru-registry-04a.txt

Note that these, unlike the copy below, still contain the typo "requries".

>   - 33 (in this text): longest 3066ter generative

I propose that all implementations that limit the length of language tags
MUST support at least 33-character tags.  Saying SHOULD is too weak: it
allows silly 2-character or 5-character limits to still claim conformance.

>    as Prefix and Suppress-Script) that limit subtag composition.  See

The whole point of Prefix and Suppress-Script is to avoid the creation of
problematic tags.  I'd like the second SHOULD here to be MUST.

>    removed along with the last remaining private use subtag.

This is confusingly non-iconic.  I propose this text:

     Applications or protocols that must truncate a tag should first
     remove all private use subtags including the '-x-'.  If the tag is
     still too long, the tag should then be truncated by progressively
     removing extension subtags including the singleton from the right
     of the tag.

If the hard lower limit of 33 is accepted, no further truncation will be
required.  Otherwise, add:

     If the tag is still too long, the tag should then be truncated by
     progressively removing variant, region, and script subtags from
     the right of the tag.

>    tag might be "cel-Latn-155-gaulish").

I think this graf should be deleted, as it's productive of nothing but confusion.
'gaulish' is not a plausible candidate for registration as a variant.

>                   Figure 3: Derivation of the Longest Tag

I'd leave this out altogether.

-- 
John Cowan <[email-masked]>     http://www.reutershealth.com
I amar prestar aen, han mathon ne nen,    http://www.ccil.org/~cowan
han mathon ne chae, a han noston ne 'wilith.  --Galadriel, LOTR:FOTR

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 7 Jun 2005 12:18:31 -0700

Hi -

(as a technical contributor)

> From: "John.Cowan" <[email-masked]>
> To: "Addison Phillips" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Tuesday, June 07, 2005 11:55 AM
> Subject: Re: [Ltru] Tag perversity text revised... (#944)
...
> allows silly 2-character or 5-character limits to still claim conformance.

Agreed.

> >    Since language tags longer than 33 characters may be truncated by an

may -> MAY

> >    application or protocol that limit tag sizes, when choosing language

limit -> limits

> problematic tags.  I'd like the second SHOULD here to be MUST.

Agreed.

>      removing extension subtags including the singleton from the right

This is an improvement, but leaves me with two questions:
    1) is the intent that all private use subtags are to be removed in
        one fell swoop, or is it that they are to be removed progressively
        until the truncated tag falls within the implementations length limitations?
    2) are the two "shoulds" intended to be MUSTs?  if we want interoperability
         of implementations that truncate, I think we do.

...
> I'd leave this out altogether.
...

This is one of those cases where I think it's helpful to the reader of a
specification for the design rationale to be left in.  Our experience in
the SNMP community was that insufficiently explained numbers (like
484 in RFC 1157) keep coming back again and again to haunt whoever
is maintaining the specification.  Also, if it somehow turns out to be
incorrect, we have a ready answer to the question: "What were you
thinking?"  :-)

Randy

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 7 Jun 2005 12:16:53 -0700

> Implementations
   processing.
...
>   Applications or protocols that must truncate a tag should first
   remove all extension subtags.

I disagree strongly with this language, as I stated before. It appears that
you didn't see my message of Monday, June 06, 2005 13:49. Please look at
that.

>This limit SHOULD be at least 33 characters in length.

I haven't heard anyone say why we shouldn't use 63 instead. (here and in
other locations)

>Here is how the 33-character length of the longest generated tag is
derived:

This is incorrect.
en-US-x-abcdefgh-abcdefgh-abcdefgh-abcdefgh-abcdefgh-abcdefgh-abcdefgh
is a generated tag, and it is longer.

Moreover, I don't see the purpose of having this analysis be present in the
specification itself. We should delete "Here is...Figure 2: Derivation of
the Longest Generated Tag"

And should delete also "Note: The longest tag...Figure 3: Derivation of the
Longest Tag". I disagree that this is the longest practical tag -- what is
"practical" is an extremely gray area -- and there is no reason for us to go
into the analysis.

‎Mark

----- Original Message ----- 
From: "Addison Phillips" <[email-masked]>
To: "LTRU Working Group" <[email-masked]>
Sent: Tuesday, June 07, 2005 11:26
Subject: [Ltru] Tag perversity text revised... (#944)

> All,
>
> Some time has passed and many emails have flowed about the most perverse
tag, its relation to RFC 2047, and so forth.
>
> Based on the all of this thread, I have prepared a substantially revised
text for section 2.1.1. This incorporates the 33-character "most perverse
generative tag" and my suggested text about handling user choice and
language tag truncation and a bunch of other suggestions (Peter, John,
Frank, etc), combined with a through-going rewrite. This version is posted
in HTML and text formats at the following URIs:
> Basically the remaining issue I see (apart from fixing any text that I
created) is deciding which lower-limit to endorse and whether to use SHOULD
> Here is the text of the revised section:

----------------------------------------------------------------------------
----

---

From: "John.Cowan" <[email-masked]>
Date: Tue, 7 Jun 2005 15:53:39 -0400

Randy Presuhn scripsit:

> specification for the design rationale to be left in.  

If we accept the lower-level hard limit of 33 characters, though, then
a justification for 62 characters is not necessary.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is 
dramatically overdescribed.  Still other languages are simultaneously 
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 7 Jun 2005 13:09:50 -0700

‎Mark

----- Original Message ----- 
From: "John.Cowan" <[email-masked]>
To: "Addison Phillips" <[email-masked]>
Cc: "LTRU Working Group" <[email-masked]>
Sent: Tuesday, June 07, 2005 11:55
Subject: Re: [Ltru] Tag perversity text revised... (#944)

> allows silly 2-character or 5-character limits to still claim conformance.

I am happy with a should of 63. I'm unhappy with a MUST of 33, since that
may not be backwards compatible.

> problematic tags.  I'd like the second SHOULD here to be MUST.

I strongly disagree. As discussed many times, there are legitimate cases
where an application may reasonably need to have the script in queries even
if  Suppress-Script is present. A SHOULD is what we have now, and that
should remain.

>      removing extension subtags including the singleton from the right

While this is better, it still unnecessarily distinguishes between private
use and extensions. If it is ok to truncate

en-US-a-abcdefg-abcdefg
to
en-US-a-abcdefg

then it makes just as much sense to truncate

en-US-x-abcdefg-abcdefg
to
en-US-x-abcdefg

There is no reason for the x subtags to be all or nothing, and the extension
subtags not to be.

All of the trunctation discussion can be much simplified by saying:

When truncation to a specific length is required:
1. If the tag doesn't fit, remove the rightmost subtag and immediately
preceeding '-'
2. Repeat #1 until it does fit.
3. If the rightmost subtag is a singleton, remove it and the preceeding '-'.

There is actually no need to do any analysis of what subtag means what in
truncation; and the simpler it is, the easier it is for people to get it
right.

(In practice, of course, it is even simpler, since any legit truncation
point is followed by a '-', and preceeded by two letters. Here is some
quick&dirty code:
 static String truncateValidLanguageTag(String tag, int limit) {
  if (tag.length() <= limit) return tag;
  do {
   if (tag.charAt(limit) == '-' && tag.charAt(limit-1) != '-' &&
tag.charAt(limit-2) != '-') break;
  } while (--limit > 2);
  return tag.substring(0,limit);
 }
)

> 'gaulish' is not a plausible candidate for registration as a variant.

I agree; we should nuke this note.

> I'd leave this out altogether.
Agree here also.