Subject: RE: [Ltru] Tag perversity text revised... (#944)

From: "Addison Phillips" <[email-masked]>
Date: Tue, 7 Jun 2005 12:48:51 -0700

Comments follow...

Addison

Addison P. Phillips
Globalization Architect, Quest Software
Chair, W3C Internationalization Core Working Group

Internationalization is not a feature.
It is an architecture. 

> -----Original Message-----
> From: John.Cowan [mailto:[email-masked]]
> Sent: 2005?6?7? 11:55
> To: Addison Phillips
> Cc: LTRU Working Group
> Subject: Re: [Ltru] Tag perversity text revised... (#944)
> 
> Addison Phillips scripsit:
> 
> > http://www.inter-locale.com/ID/draft-ietf-ltru-registry-04a.html
> > http://www.inter-locale.com/ID/draft-ietf-ltru-registry-04a.txt
> 
> Note that these, unlike the copy below, still contain the typo "requries".
[Addison Phillips] 

I fixed all the typos in my XML source document when I fixed them in the email. Thanks.
> 
> >   - 33 (in this text): longest 3066ter generative
> 
> I propose that all implementations that limit the length of language tags
> MUST support at least 33-character tags.  Saying SHOULD is too weak: it
> allows silly 2-character or 5-character limits to still claim conformance.
[Addison Phillips] 

Okay.

> 
> >    Since language tags longer than 33 characters may be truncated by an
> >    application or protocol that limit tag sizes, when choosing language
> >    tags users and applications SHOULD avoid adding subtags that add no
> >    distinguishing value and SHOULD follow fields in the registry (such
> >    as Prefix and Suppress-Script) that limit subtag composition.  See
> >    Choice of Language Tag.
> 
> The whole point of Prefix and Suppress-Script is to avoid the creation of
> problematic tags.  I'd like the second SHOULD here to be MUST.
[Addison Phillips] 

Agree.
> 
> >    Applications or protocols that must truncate a tag should first
> >    remove all extension subtags.  If the tag is still too long, the tag
> >    should then be truncated by progressively removing subtags from the
> >    right of the tag.  Note that private use subtags are removed first.
> >    When removing private use subtags, the singleton '-x-' should be
> >    removed along with the last remaining private use subtag.
> 
> This is confusingly non-iconic.  I propose this text:
> 
>      Applications or protocols that must truncate a tag should first
>      remove all private use subtags including the '-x-'.  If the tag is
>      still too long, the tag should then be truncated by progressively
>      removing extension subtags including the singleton from the right
>      of the tag.
[Addison Phillips] 

Agree.
> 
> If the hard lower limit of 33 is accepted, no further truncation will be
> required.  Otherwise, add:
> 
>      If the tag is still too long, the tag should then be truncated by
>      progressively removing variant, region, and script subtags from
>      the right of the tag.
> 
> >    Note: The longest tag that might be generated based on the
> >    grandfathered tags (see Section 2.2.8) that contain subtags not
> >    otherwise defined by this document (and in a form that could be
> >    registered as a variant) requires 20 characters (an example of this
> >    tag might be "cel-Latn-155-gaulish").
> 
> I think this graf should be deleted, as it's productive of nothing but
> confusion.
> 'gaulish' is not a plausible candidate for registration as a variant.
[Addison Phillips] 

It is the longest remaining grandfathered subtag. The next longest is 'scouse', followed by 'xiang'/'boont', etc.

This is more like special pleading for later (e.g. the limit is reasonable because even long-ish tags that are reasonable don't exceed it).

> 
> >    The longest possible tags, excluding extensions, that might have some
> >    marginal utility could have a length of up to 62 characters, as shown
> >    below.  The longest such tag without private use subtags requires 42
> >    characters.
> >
> >    Here is how the 62-character length of the longest practical tag
> >    (excluding extensions) is derived:
> >
> >    language      = 3
> >    extlang1      = 4 (each subsequent subtag includes '-')
> >    extlang2      = 4 (unlikely: needs prefix="language-extlang1")
> >    extlang3      = 4 (extremely unlikely)
> >    script        = 5 (must not be suppressed)
> >    region        = 4 (UN M.49; ISO 3166 requires 3)
> >    variant1      = 9
> >    variant2      = 9 (unlikely: needs prefix="language-variant1")
> >    private use 1 = 11 ("-x-" + subtag)
> >    private use 2 = 9
> >    total         = 62 characters
> >
> >                   Figure 3: Derivation of the Longest Tag
> 
> I'd leave this out altogether.
[Addison Phillips] 

The 62/42 limit paragraph? The derivation I can see, but the longer "minimum" is useful, especially if the lower limit is a mere 33. Moreover, the 42 limit seems like a more responsible number than the 33 limit if we trim extensions and private use, as your text elegantly provides.

Addison

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 7 Jun 2005 12:56:36 -0700

Some notes below.

Addison

Addison P. Phillips

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Randy Presuhn
> ...
[Addison Phillips] 

So far 33 seems most popular. Note that 42 is probably a better number in retrospect, especially if we go with John Cowan's text that shows trimming off private and extension subtags. Two variants and three extlangs is more of a boundary condition.

> may -> MAY
[Addison Phillips] 

Are you sure this makes sense? This use of "may" is an observation of something previously made normative.
> limit -> limits
[Addison Phillips] 

Good catch.
> Agreed.
[Addison Phillips] 

I agree.
> This is an improvement, but leaves me with two questions:
>     1) is the intent that all private use subtags are to be removed in
>         one fell swoop, or is it that they are to be removed progressively
>         until the truncated tag falls within the implementations length
> limitations?
[Addison Phillips] 

Progressively.

>     2) are the two "shoulds" intended to be MUSTs?  if we want
> interoperability
>          of implementations that truncate, I think we do.
[Addison Phillips] 

That seems reasonable.
> ...
> ...
> This is one of those cases where I think it's helpful to the reader of a
> specification for the design rationale to be left in.  Our experience in
> the SNMP community was that insufficiently explained numbers (like
> 484 in RFC 1157) keep coming back again and again to haunt whoever
> is maintaining the specification.  Also, if it somehow turns out to be
> incorrect, we have a ready answer to the question: "What were you
> thinking?"  :-)
> 
That is a useful point. Magic numbers (especially weird ones like "33" or "62") can be a lot of work for non-initiates to understand.

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 7 Jun 2005 12:57:38 -0700

Yes, although then the illustration of the magic number 33 (or 42 or...) becomes interesting.

Addison

Addison P. Phillips

> Behalf Of John.Cowan
> Sent: 2005?6?7? 12:54
> To: Randy Presuhn
> Cc: [email-masked]
> Randy Presuhn scripsit:
> > specification for the design rationale to be left in.
> 
> If we accept the lower-level hard limit of 33 characters, though, then
> a justification for 62 characters is not necessary.
> 
> --
> John Cowan   [email-masked]  http://www.ccil.org/~cowan
> Most languages are dramatically underdescribed, and at least one is
> dramatically overdescribed.  Still other languages are simultaneously
> overdescribed and underdescribed.  Welsh pertains to the third category.
>         --Alan King
>

---

From: "Addison Phillips" <[email-masked]>
Date: Tue, 7 Jun 2005 16:02:08 -0700

Yes, although (Section 4.3) says:

--
An extension MUST define any relationships that may exist between the various subtags in the extension and thus MAY define an alternate canonicalization scheme for the extension's subtags. Extensions MAY define how the order of the extension's subtags are interpreted. For example, an extension could define that its subtags are in canonical order when the subtags are placed into ASCII order: that is, "en-a-aaa-bbb-ccc" instead of "en-a-ccc-bbb-aaa". Another extension might define that the order of the subtags influences their semantic meaning (so that "en-b-ccc-bbb-aaa" has a different value from "en-b-aaa-bbb-ccc"). However, extension specifications SHOULD be designed so that they are tolerant of the typical processes described in Section 3.6 (Extensions and Extensions Namespace).
--

And Section 3.6 in turn says:

--
Extension authors are strongly cautioned that many (including most well-formed) processors will be unaware of any special relationships or meaning inherent in the order of extension subtags. Extension authors SHOULD avoid subtag relationships or canonicalization mechanisms that interfere with matching or with length restrictions that may exist in common protocols where the extension is used. In particular, applications may truncate the subtags in doing matching or in fitting into limited lengths, so it is RECOMMENDED that the most significant information be in the most significant (left-most) subtags, and that the specification gracefully handle truncated subtags.

When a language tag is to be used in a specific, known, protocol, it is RECOMMENDED that that the language tag not contain extensions not supported by that protocol. In addition, it should be noted that some protocols may impose upper limits on the length of the strings used to store or transport the language tag.
--

Removing the entire extension or all extensions at first seems appealing, although it requires that software be rather "smarter" than with pure RFR truncation.

Addison

Addison P. Phillips

> Behalf Of Peter Constable
> Sent: 2005?6?7? 15:31
> To: LTRU Working Group
> Subject: RE: [Ltru] Tag perversity text revised... (#944)
> 
> > From: [email-masked] [mailto:[email-masked]]
> On Behalf Of
> > Mark Davis
> 
> 
> > While this is better, it still unnecessarily distinguishes between
> private
> > use and extensions. If it is ok to truncate
> >
> > en-US-a-abcdefg-abcdefg
> > to
> > en-US-a-abcdefg
> >
> > then it makes just as much sense to truncate
> >
> > en-US-x-abcdefg-abcdefg
> > to
> > en-US-x-abcdefg
> >
> > There is no reason for the x subtags to be all or nothing, and the
> extension
> > subtags not to be.
> 
> At least for extensions, since the meaning of extension subtags is
> defined elsewhere, we have no idea if changing
> 
> en-US-x-abcdefg-abcdefg
> to
> en-US-x-abcdefg
> 
> has had a neutral effect, a mildly negative effect, or a disastrous
> effect. Since *this* spec and protocols that conform to it are
> responsible only for interpretation of stuff *other than* the extension,
> I would have thought the safest this to do if any part of an extension
> needs to be removed is to remove the whole extension.
> 
> I think I'd rather be responsible for some downstream process getting no
> info when some was expected than for it to get info that implied
> something significantly different than what was intended by the original
> sender.
> 
> 
> 
> Peter Constable
>

---

From: "Peter Constable" <[email-masked]>
Date: Tue, 7 Jun 2005 16:52:27 -0700

I had forgotten about that text. (I think I may have even suggested some
of it.) I guess with that in mind, RFR is tolerable -- we've warned
authors of extension specs that they should anticipate the possibility
of such truncation.

Peter Constable

> From: Addison Phillips [mailto:[email-masked]]
> Sent: Tuesday, June 07, 2005 4:02 PM
> To: Peter Constable; LTRU Working Group
> An extension MUST define any relationships that may exist between the
various
> subtags in the extension and thus MAY define an alternate
canonicalization scheme for
> the extension's subtags. Extensions MAY define how the order of the
extension's
> subtags are interpreted. For example, an extension could define that
its subtags are in
> canonical order when the subtags are placed into ASCII order: that is,
"en-a-aaa-bbb-
> ccc" instead of "en-a-ccc-bbb-aaa". Another extension might define
that the order of
> the subtags influences their semantic meaning (so that
"en-b-ccc-bbb-aaa" has a
> different value from "en-b-aaa-bbb-ccc"). However, extension
specifications SHOULD
> be designed so that they are tolerant of the typical processes
described in Section 3.6
> (Extensions and Extensions Namespace).
> --
> 
> Extension authors are strongly cautioned that many (including most
well-formed)
> processors will be unaware of any special relationships or meaning
inherent in the
> order of extension subtags. Extension authors SHOULD avoid subtag
relationships or
> canonicalization mechanisms that interfere with matching or with
length restrictions
> that may exist in common protocols where the extension is used. In
particular,
> applications may truncate the subtags in doing matching or in fitting
into limited
> lengths, so it is RECOMMENDED that the most significant information be
in the most
> significant (left-most) subtags, and that the specification gracefully
handle truncated
> subtags.
> 
> When a language tag is to be used in a specific, known, protocol, it
is RECOMMENDED
> that that the language tag not contain extensions not supported by
that protocol. In
> addition, it should be noted that some protocols may impose upper
limits on the
> length of the strings used to store or transport the language tag.
> --
> 
> Removing the entire extension or all extensions at first seems
appealing, although it
> requires that software be rather "smarter" than with pure RFR
truncation.

---

From: "McDonald, Ira" <[email-masked]>
Date: Tue, 7 Jun 2005 20:53:50 -0700

+1

Despite Mark's protests, lower bound for conformance to _this_ spec
has to be a MUST for length (and Frank's right that 33 is the magic
number - breaking all email and MIME bodies is not acceptable).

And including the rationale for SHOULD of 63 (safest lower bound)
is a good idea.  One of the best ISO specs is any later version
of the C Language with the clause-by-clause companion rationale
(also done in later POSIX specs), because the _meaning_ of those
apparently arbitrary numbers and constraints is manifest.

Leaving out rationale is throwing away the value of a working group.

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

> From: [email-masked] 
> [mailto:[email-masked]]On
> Behalf Of John.Cowan
> Sent: Tuesday, June 07, 2005 5:46 PM
> To: Randy Presuhn
> Cc: [email-masked]
> > >    value is generated or the language tag is truncated).  
> This limit
> > >    SHOULD be at least 42 characters in length.  If truncation is
> > ...
> > 
> > SHOULD -> MUST, otherwise there is no lower bound to what can claim
> > to be a conformant implementation.
> 
> +1.  Otherwise I'm happy with Addison's latest.
> 
> -- 
> Do NOT stray from the path!             John Cowan 
> <[email-masked]>
>         --Gandalf                       http://www.ccil.org/~cowan
>