Subject: New draft (Was: I-D ACTION:draft-klensin-unicode-escapes-00.txt

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jan 2007 08:33:35 -0500

Hi.

We have reached the point at which several people are repeating
essentially the same suggestions.  That probably represents
progress, but it also indicates to me that the -00 version of
the draft has outlived its usefulness.

I've just submitted draft-klensin-unicode-escapes-01.txt and
assume it will show up in the posting directory today or
tomorrow.  

Highlights (if I got things right):
	
	* All of the typographic editorial errors that have been
	identified have been fixed.
	
	* The preference for \u / \U has been removed and the
	document restructured to express no preference among the
	several ways to denote Unicode code points.  However,
	the preference of many for paired delimiters has been
	made explicit, as has a prohibition on surrogates.
	
	* The issue of escaping escapes has been avoided,
	leaving that problem to the applications and protocols
	that use the escapes.  That may not be good enough, but
	it at least moves us forward and should facilitate a
	focused discussion.

There are two sets of suggestions I haven't followed (yet):

	* Frank made a strong recommendation that we make an
	explicit and normative reference to W3C's CharMod spec
	and call out several of its explicit rules.  There is a
	tradeoff with length, additional need to read other
	documents, and general document and reading complexity.
	It may no longer be needed.  Opinions and comments
	welcome.
	
	* I have not touched the ABNF associated with the \u /
	\U case.  I have inserted an explicit placeholder but,
	as discussed on this list, I think we need to figure out
	what we want to do and then go back and adjust the
	metalanguage productions.   In particular, there has
	been one strong suggestion, with which I agree, that we
	not take the obvious approach of substituting %x5C.75
	for "\u", since the intent is a character string
	abstraction (independent of the implementation character
	set) rather than specific octets.

thanks,
    john

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jan 2007 19:53:56 -0500

--On Monday, 29 January, 2007 22:32 +0100 Bjoern Hoehrmann
<[email-masked]> wrote:

>...
> I think better discussion of this case, and also escaping
> characters with some pre-defined meaning inside a protocol
> element that accepts UTF-8, is necessary. E.g., in section 1.1
> it is said "This recommen- dation is not applicable to
> protocols that already accept native UTF-8 or some other
> encoding of Unicode." I am not sure how to read that in such a
> case.

If the protocol can actually accept UTF-8 (or UTF-16 or UTF-32),
this recommendation is not applicable at all.   If it is somehow
necessary to escape Unicode characters that have special
interpretations in those environments, that is the problem of
the relevant protocols.

Beyond that, I'm always open to suggested text.

> Some nits:
> 
> In section 3 I am not sure how to read "U+NNN[N[N]]". Perhaps
> there is an "N" missing?

That is correct.   And <obscenity>, I thought I had caught all
of those.  Fixed in -02.  Sorry.

> I think this should say instead in
> prose that four to six digits should be used. Section 1.1 uses
> "U+NNNN[N[N]]", there I'd prefer to simply use U+NNNN or
> U+NNNNNN.

But, as long as five characters is permissible (and it is a
frequent case), either of those two cases would be wrong.  I've
added prose in the working draft for -02 to the first case.

> In section 5.2 it is said HTML uses the &#xNNNN; form and that
> this form has a clear terminator. This is not really false but
> HTML allows to omit the terminator if it is not needed, for
> example <p>Bj&#xf6rn</p> is also valid. I would suggest to
> mention only XML or note that HTML's mechanism is similar to
> that of XML.

Good.  Will fix.

thanks,
   john

---

From: "Tim Bray" <[email-masked]>
Date: Tue, 30 Jan 2007 16:45:52 -0800

Pardon me for being late to this party, I was on vacation in
Australia.  I think this is a positive contribution.

First, a detail point:  In section 5.4, it's probably relevant that
per the Java Language Specification
(http://java.sun.com/docs/books/jls/third_edition/html/lexical.html#95413p)
it's clear that a Java character literal or variable represents, not a
Unicode character, but a UTF-16 code point.   I guess the conclusion
is that it may be OK in certain circumstances to use \uNNNN, but it's
not OK to explain that by calling out to Java.

Second: I think that the discussion shows that the syntax problems
around representing Unicode characters in ASCII and other
Unicode-oblivious texts are tricky; witness the issues with delimiters
and ABNF/case.  This is further evidence, were any needed, that IETF
Working Groups SHOULD NOT specify Internet protocols which may be used
to transfer text but are not capable of representing the Unicode
character set, either by specifying the use of either hard-wired UTF-8
or alternatively XML, both of which have cracked this nut.

So here's a proposed recasting of second para of 1.1:

  When one moves to Unicode [Unicode] [ISO10646], where characters
   occupy two or more octets and may be coded in several different
   forms, the question of escapes becomes even more complicated.  In
   particular, we have seen fairly extensive use of both hexadecimal
   representations of the UTF-8 encoding [RFC3629] of a character and
   variations on the U+NNNN[N[N]] notation commonly used in conjunction
   with the Unicode Standard.

  New protocols that are required to carry textual content SHOULD be designed
  in such a way that the full repertoire of Unicode characters may be
represented
  in that text; UTF-8 and XML are both good options.

  This document proposes that existing protocols being internationalized SHOULD
   use some contextually-appropriate variation of the U+NNNN[N[N]]
notation unless
   other considerations outweigh those described here.