Subject: [Ltru] Re: Wrapping up the UTF-8 debate

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Jul 2007 18:28:49 -0700

Randy Presuhn <randy underscore presuhn at mindspring dot com> wrote:

> 1)  Are we willing to use a representation (for the discussion
>     of changes on the [email-masked] list and communication
>     with IANA) which is different (perhaps only mechanically/trivially 
> so)
>     from what is actually published in the registry files?

Yes.  We can use hex NCRs on the list and ask IANA to convert them, 
assuming David says this is OK and within their capabilities.

> 2)  The registry file itself currently uses something which is similar
>     to an NCR.

How is it not *exactly* like an NCR?  The XML Fourth Edition 
specification (2006-08-16) defines has this to say:

<quote>
4.1 Character and Entity References

[Definition: A character reference refers to a specific character in the
ISO/IEC 10646 character set, for example one not directly accessible
from available input devices.]

Character Reference

[66]   CharRef   ::=   '&#' [0-9]+ ';'
                       | '&#x' [0-9a-fA-F]+ ';' [WFC: Legal Character]

Well-formedness constraint: Legal Character

Characters referred to using character references MUST match the
production for Char.

If the character reference begins with "&#x", the digits and letters up
to the terminating ; provide a hexadecimal representation of the
character's code point in ISO/IEC 10646. If it begins just with "&#",
the digits up to the terminating ; provide a decimal representation of
the character's code point.
</quote>

Reference: http://www.w3.org/TR/2006/REC-xml-20060816/#sec-references

Our production is:
   UNICHAR    = "&#x" 2*6HEXDIG ";"

In practical terms, how does this differ fro the XML production?

>     Are we willing to change the registry format to
>        a) use actual NCRs for non-ASCII code points, making conversion
>           to XML even more trivial than it already is, while still
>           giving some fallback to folks inspecting the data for errors
>           or looking at it through ASCII windows
>        b) embed the actual (UTF-8 encoded) characters into the file
>        c) something else?

B.

> 3)  Are we going to instruct IANA to maintain a "pure" ASCII version
>     of the registry, in which everything not ASCII will have been 
> flattened
>     or translitered?

I don't support asking IANA to do this, because of the minimal pairs. 
Stéphane can create such a file if he likes.  If we were talking about 
maintaining a Latin-1 version, as ISO 639-3 does, I might support this.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Jul 2007 18:30:42 -0700

Peter Constable <petercon at microsoft dot com> wrote:

> Another possible issue (assuming consensus on a file with actual 
> UTF-8-encoded characters): does the UTF-8 file begin with a UTF 
> BOM/encoding signature?

Most people seem to hate the BOM and feel it interferes with text 
processing in one way or another, so I won't fight for it.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 21 Jul 2007 18:35:22 -0700

David Conrad <david dot conrad at icann dot org> wrote:

> As long as the registry maintenance can be automated such that 
> non-technical IANA staff can do what needs to be done, we'll not raise 
> any objection.
>
> P.S. If somebody wants to help and provide code, we'd be more than 
> happy to accept... :-)

I can supply a Windows executable and/or source code that can convert 
between hex NCRs and UTF-8.  (Not right away, though; I have to write it 
first.  :)

Alternatively, you could use an existing editor that can perform this 
conversion, such as BabelPad 
(http://www.babelstone.co.uk/Software/BabelPad.html) or SC UniPad 
(http://www.unipad.org/main/).  I'm sure there are others.

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 21 Jul 2007 21:02:20 -0700

Hi -

> From: "Doug Ewell" <[email-masked]>
> To: "LTRU Working Group" <[email-masked]>
> Sent: Saturday, July 21, 2007 6:28 PM
> Subject: [Ltru] Re: Wrapping up the UTF-8 debate
...
> specification (2006-08-16) defines has this to say:
...

You're right.  All the file needs is some constant
HTML mumble up to a <PRE> at the beginning,
and a </PRE> and some trailing constant mumble
at the end, and the conversion (for browser display
purposes only) is done.  :-) * 0.5

Randy

---

From: Martin Duerst <[email-masked]>
Date: Sun, 22 Jul 2007 14:18:51 +0900

At 13:02 07/07/22, Randy Presuhn wrote:
>> How is it not *exactly* like an NCR?

Well, just for the record, it's not exactly like an NCR,
because we only allow a subset of NCRs (e.g. only hex
rather than also decimal,...).

Regards,    Martin.s

>>The XML Fourth Edition 

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]