Subject: [Ltru] Review of 4646bis-10, sections 1 to 3.4

From: John Cowan <[email-masked]>
Date: Thu, 6 Dec 2007 11:37:55 -0500

Yet another freakin' de novo review of 4646bis draft 10, sections 1
through 3.4.  More review will follow.  The following two changes are
substantive:

1) I suggest adding this to 2.2.6 (extensions) after the first sentence:

        They are intended to identify information which is commonly used
        in association with language tags but is not part of language
        identification.

2) In 3.1.2, I'd prefer to say that implementations MUST rather than
SHOULD ignore undefined fields in the registry.  There is no reason why
implementations should not be required to be liberal in what they accept.

The remaining changes are all editorial:

In 2.2, the term "code" is defined to refer to a value defined in an
external standard.  We should, pervasively throughout the document,
change this to "code element", which is proper ISO terminology.  (For ISO,
"code" means the whole list.)

The final graf in 2.2 should be bulleted like its predecessors.

In 2.2.x, there are numbered points like "X subtags MUST follow any
... subtags and MUST precede any ... subtags".  The numbers used are #2 in
2.2.3, #1 in 2.2.4, #2 in 2.2.5, #9 in 2.2.6, #3 in 2.2.7.  These should
all be moved to #1 in each list, and minor wording variations removed.

2.2.4(3)(C) uses the phrase "with ambiguous ISO 3166 alpha-2 codes".
This is not very clear to those who don't know the history, and should
be expanded to something like "whose alpha-2 code was formerly (since
<Date B>) associated with a different country".  Someone will have to
look up what the cutoff date was -- I forget.

2.2.5:  s/suitable to form/suitable for forming/

2.2.8: s/current registered/currently registered/

3.1.1: change the definition of folding to "Folding is always done on
Unicode default grapheme boundaries".  That says what the current text
says, and also prohibits folding in the middle of a Hangul syllable
written as separate jamo.

3.1.2:  In the explanation of "Preferred-Value", the first two bullet
points say "Preferred-Value contains" but the third does not.

3.1.3:  "The 'Subtag' field MUST use lowercase letters" says too much;
it implicitly forbids digits.  What is meant is "The 'Subtag' field
MUST NOT use uppercase letters".  Adding "all" before "uppercase" in
the sentence about region subtags will clarify it.

3.1.4: s/For records taken/For subtags taken/.

3.1.4:  s/English localized names/localized English names/.

3.1.7:  s/a prefix a prefix/a prefix/.

3.1.4: it's not about countries changing their official names, as official
names aren't even in the registry.  (Hands up all those who can say the
official name of the U.S.  without looking it up.)  Remove "official".

3.4(16): s/should it be possible/should it become possible/.

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
Today an interactive brochure website, tomorrow a global content
management system that leverages collective synergy to drive "outside of
the box" thinking and formulate key objectives into a win-win game plan
with a quality-driven approach that focuses on empowering key players
to drive-up their core competencies and increase expectations with an
all-around initiative to drive up the bottom-line. --Alex Papadimoulis

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 6 Dec 2007 17:44:35 +0100

On Thu, Dec 06, 2007 at 11:37:55AM -0500,
 John Cowan <[email-masked]> wrote 
 a message of 78 lines which said:

> SHOULD ignore undefined fields in the registry.  There is no reason
> why implementations should not be required to be liberal in what
> they accept.

I believe it would forbid a registry-checking application (like the
one IANA should run after any change). So, it seems a bad idea.

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Dec 2007 12:02:52 -0500

> one IANA should run after any change). So, it seems a bad idea.

Fair enough.  Concedo.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 6 Dec 2007 09:38:51 -0800

With that change, I think all of John's suggestions look good.

-- 
Mark

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 7 Dec 2007 21:08:13 +0100

John Cowan wrote:

> I suggest adding this to 2.2.6 (extensions) after the first sentence:
>| They are intended to identify information which is commonly used
>| in association with language tags but is not part of language
>| identification.

The intro says that language tags are about identifying languages, I
don't see why extensions should be limited to "non-identification".

> change this to "code element", which is proper ISO terminology.

It used to be "code" already in RFC 1766.  I see no reason to change
this to "proper ISO terminology" when the shorthand is unambiguous.
  
> For ISO, "code" means the whole list.

For the IETF it doesn't necessarily, so far.    

> look up what the cutoff date was -- I forget.

2005-10-16 is the Filedate in the last 4645 draft, was that "date B" ?

I think 2.2.4(3)(C) is about codes deprecated after "date A" and later
reused for a *different* country.  Maybe without "different" qualifier,
as Doug read it.

BTW, 2.2.4(3)(E) talks about 4645 instead of 4645bis, and I think it's
not more needed after Debbie registered GG and JE.

> 3.1.1: change the definition of folding to "Folding is always
> done on Unicode default grapheme boundaries".

AFAIK the normal (2822upd) definition of folding is "at places
where WSP is allowed", and unfolding means "replace FWS by SP".

No grapheme boundaries involved.  If 4646bis tries to invent
its own folding rules this should be better explained.  4646bis
shouldn't expose such Unicode oddities, checking my own code:

| #### unfold field body ############################################
| /^[\t ]/ { BODY = BODY " " STRIP( $0 )
|            next
|          }

That replaces FWS by one SP, I'm almost certain that I don't want 
an SP between the "graphemes" of a "folded word".

> also prohibits folding in the middle of a Hangul syllable

Do they have "WSP separated words", or is there a serious
chance of more than 72-1 adjacent bytes belonging to one
or more graphemes ?  Talking about "bytes" in conjunction
with UTF-8 makes me nervous.

> Hands up all those who can say the official name of the
> U.S.  without looking it up.

The CIA world fact book claims I hit the "conventional long
form" - maybe "official" isn't the same as "conventional".

 Frank

---

From: Addison Phillips <[email-masked]>
Date: Fri, 07 Dec 2007 12:54:01 -0800

Notes follow indicating incorporation into draft-11 editor's copy. After 
I figure out the insta-submit process and do other housekeeping, look 
for it in draft-11. My disposition of these edits as follows.

>         in association with language tags but is not part of language

Done, with a minor edit. Paragraph reads:

--
Extensions provide a mechanism for extending language tags for use in
various applications. They are intended to identify information which is
commonly used in association with languages or language tags, but which
are not part of language identification. See <xref
target="extensions"></xref>. The following rules apply to extensions:
--

> implementations should not be required to be liberal in what they accept.

While I agree with the sentiment, you and I accept Stephane's reading 
later in this thread. Not incorporated.

> "code" means the whole list.)

Argh. I tried to do this, but:

1. The names of the ISO standards use the word "code" or "codes"
2. The quote from ISO 639RA/JAC uses the word "code"
3. Using the term "code element" was really REALLY ugly... it made the 
prose even more stiff and pedantic.

So I backed the change out. Not incorporated. If you feel strongly about 
this, let's discuss on the list.

> The final graf in 2.2 should be bulleted like its predecessors.

Done.
> all be moved to #1 in each list, and minor wording variations removed.

Done. I used the form:

	Region subtags MUST follow any language or script subtags
	and MUST precede any other type of subtag.

(where the name of the subtag is changed along with the list of 
preceding subtags)

> look up what the cutoff date was -- I forget.

The is actually the Date C rule. That is, it covers UN M.49 codes for 
countries whose code has been reused by ISO 3166. At present, there are 
none of these.

So...

I reworked the graf to:

<t>UN numeric codes for countries or areas which are assigned ISO 3166 
alpha2 codes already present in the registry, MUST be defined according 
to the rules in <xref target="ianastability"/> and MUST be used to form 
language tags that represent the country or region for which they are 
defined. This happens when ISO 3166 reassigns a code formerly used for 
one country to another.</t>

> 2.2.5:  s/suitable to form/suitable for forming/

1 occurrence replaced.
> 2.2.8: s/current registered/currently registered/

ditto
> written as separate jamo.

:: shiver ::

Yes, I know. (laughing) I hoped to avoid implementing it in record-jar 
though. But you're right.

The sentence was rewritten as follows, to include the example:

Folding is always done on Unicode default grapheme boundaries (that is, 
never in the middle of a multibyte UTF-8 sequence nor in the middle of a 
combining character sequence).

> points say "Preferred-Value contains" but the third does not.

Done.

> the sentence about region subtags will clarify it.

Done.

> 3.1.4: s/For records taken/For subtags taken/.

Done.
> 3.1.4:  s/English localized names/localized English names/.

Done.

> 3.1.7:  s/a prefix a prefix/a prefix/.

Done.
> official name of the U.S.  without looking it up.)  Remove "official".

Done.

> 3.4(16): s/should it be possible/should it become possible/.

Done.

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Addison Phillips <[email-masked]>
Date: Fri, 07 Dec 2007 13:30:14 -0800

> an SP between the "graphemes" of a "folded word".

Except... our record-jar format (and this document) doesn't follow that 
rule. In particular, if we permit the folding (which is done by people 
and not machines, in the case of the registry) to occur mid-grapheme, it 
will break the character.

For example, if you were to put U+0065 U+0300 as a sequence and the fold 
occurred between those two characters, the result would be U+0065 U+0020 
U+0300, or e ̀ (e followed by accent) rather than è.

IOW> the registry is supposed to be viewable and readable as a 
plain-text (UTF-8) file.

And note that my record-jar draft goes further and suppresses the 
addition of the space on unfolding (at the cost of complicating the 
syntax slightly with a line-continuation character).

> or more graphemes ? 

They don't have WSP separated words in Korean, typically. This also 
applies to other languages, such as Thai, that also use combining 
sequences. There is no real (only theoretical) danger of 72-1 adjacent 
bytes belonging on ONLY one grapheme.

> Talking about "bytes" in conjunction

Why so? UTF-8 is a multibyte character encoding. So it's code unit is 
the byte. When we speak of counting things, it is good to know what is 
being counted. In this case, we count bytes, but ensure that breaks 
occur between characters, indeed, with John's suggestion, between 
*visual* "characters", aka graphemes.

Addison

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 8 Dec 2007 16:34:59 +0100

> Folding is always done on Unicode default grapheme boundaries (that
> is, never in the middle of a multibyte UTF-8 sequence nor in the
> middle of a combining character sequence).

Do they offer a list of combining characters somewhere, or is that
a case of "grep 996 KB list for non-zero combining class" ?

You need a note that "folding" is supposed to be replaced by NO
space in your definition.  In other words it MUST NOT occur where
WSP is allowed (replacing a real WSP by folding, which is later
unfolded into nothing, joins words):

1 - input was Example:<SP>fold<SP>me<CRLF>
2 - folded is Example:<SP>fold<CRLF><SP>me<CRLF>
3 - output is Example:<SP>foldme<CRLF>

Your "folding" lost the space here, I see no way to protect it.  
The folks who created STD 11 etc. knew what they were up to,
and the whole world knows how STD 11 folding works.

 [In another message =========================================]
> IOW the registry is supposed to be viewable and readable as
> a plain-text (UTF-8) file.

The folding business should be similar with NCRs for graphemes,
not folding "within" a NCR is obvious, like not folding within
an UTF-8 code point.  Record-jar is rather pointless for UTF-8,
I always wanted "UTF-8 => XML", how about simply deleting the
line length limit 72 ?  "72 bytes" is a pointless concept for
UTF-8, and 72 "graphemes" don't help with half- vs. full width.
(Actually I've no clue, maybe NFC gets rid of the width hurdle)

> They don't have WSP separated words in Korean, typically

Ugh.  Transform the 72 MUST in a SHOULD, and allow folding only
at 1*WSP, allowing folding within a word can't work as expected.
Violating the SHOULD for longer words is perfectly fine.  We're
not interested in 2047 / 2231 encoded "words" for the registry

[ MIME has smart rules about SP between encoded words, but not
 trivial, this killed several EAI downgrade drafts from my POV ]

>> Talking about "bytes" in conjunction with UTF-8 makes me nervous.

> Why so? UTF-8 is a multibyte character encoding. So it's code
> unit is the byte.

Sure, but what are "72 bytes" supposed to do, if that could be 18
to 72 code points, and hopefully more than zero graphemes per line.
The 72 only made sense for the NCR registry when viewed as ASCII.

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 8 Dec 2007 23:23:45 +0100

On Sat, Dec 08, 2007 at 04:34:59PM +0100,
 Frank Ellermann <[email-masked]> wrote 
 a message of 73 lines which said:

> Do they offer a list of combining characters somewhere, or is that a
> case of "grep 996 KB list for non-zero combining class" ?

I believe that combining characters are simply those who are in the
categories "Mark, Nonspacing", "Mark, Spacing Combining" and "Mark,
Enclosing", no? If so, it is quite simple to extract them:

ucd=> SELECT To_U(codepoint) AS U_Codepoint, name FROM Characters WHERE category = 'Mn' OR category = 'Mc' OR category = 'Me' ORDER BY codepoint;
 u_codepoint |                           name                            
-------------+-----------------------------------------------------------
 U+300       | COMBINING GRAVE ACCENT
 U+301       | COMBINING ACUTE ACCENT
 U+302       | COMBINING CIRCUMFLEX ACCENT
...
(1065 rows)

---

From: John Cowan <[email-masked]>
Date: Sun, 9 Dec 2007 00:49:39 -0500

> and the whole world knows how STD 11 folding works.

This is quite compelling, and I think we should consider changing it.
There's no backward-compatibility hit for the registry, but there
is a hit for existing parsing code.

-- 
John Cowan              [email-masked]          http://www.ccil.org/~cowan
Any day you get all five woodpeckers is a good day.  --Elliotte Rusty Harold

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Dec 2007 15:59:55 -0500

>         in association with language tags but is not part of language

"which is ... but which are ..." should be "which is ... which is ...";
the implicit subject in either case is "information".

> written as separate jamo.

I think we should reconsider this as Frank suggests, and fold only
at whitespace, the same rule that RFC 2822 headers and the like use.
As things stand, we cannot fold at whitespace except by placing the
break after the space, making it a trailing space.

-- 
In politics, obedience and support      John Cowan <[email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan

---

From: Addison Phillips <[email-masked]>
Date: Fri, 14 Dec 2007 13:47:05 -0800

> the implicit subject in either case is "information".

<editor hat="on">Done</editor>

> break after the space, making it a trailing space.

<technicalContributor>

I dunno, John. The problem here is messy, since, if we fold on 
whitespace, we preclude comments in a native language that doesn't use 
whitespace or we at least forcibly break it in a weird way (unfolding 
the text results in a spurious space being created). Although currently 
all comments are in English, I can certainly imagine including a quote 
or descriptive text in-language.

Note that any trailing space is eaten by the unfolding rules. In 
record-jar's draft, I solved this by including a continuation character.

I'm not suggesting that we include the continuation character and I'm 
not quite against your/Frank's suggested change (since I suspect we are 
engaged in a bit of over-engineering, as there isn't evidence of our 
needing this feature just yet). I just want to be a bit careful here.

So... I guess in the course of the paragraph above I've talked myself 
into making the change.....

</technicalContributor>