Subject: Next step (Was: I-D ACTION:draft-klensin-unicode-escapes-00.txt

From: John C Klensin <[email-masked]>
Date: Wed, 24 Jan 2007 09:14:21 -0500

Hi.

I've gotten a very large amount of mail about this draft,
actually more addressed to me personally than on the list.

The mail seems to me to fall into three categories:

(1) Evidence that I still cannot do ABNF properly, plus a few
other editorial issues.  The most important of the ABNF issues
is that "u" and "U" cannot be specified as different without
escaping both into their character code positions (since
character-literals in ABNF are case-insensitive).  The person
who pointed that out suggested that this is yet more evidence
that \u and \U are a bad choice, but that question is part of
(3).  All of these editorial and ABNF problems are easily fixed.

(2) Concerns about "escaping the escape", with some differences
in opinion about what the escape-escape should be.

(3) Distaste for the particular \uNNNN and \UNNNNNNNN mechanisms
chosen, with several different preferences for what should be
chosen instead.

I have seen nothing that expresses a strong preference for the
"encode UTF-8 octets" forms instead of the "represent Unicode
code points" forms.  The preference for the latter was the mail
purpose of this document/ effort, so we are making progress.

Let's assume that the issues in (1) and (2) can be fairly easily
addressed, (1) more easily than (2), obviously.    If it is
helpful, I can immediately issue a new draft to correct the
editorial and ABNF issues that have been identified, clearing up
(1). 

But I'd like to focus on (3), which is where the showstoppers
presumably lie.  

I don't see how to get agreement on a single form: almost
everyone who doesn't like \u / \U likes something different...
there is no evidence in the notes I have received that indicates
one uniform strong preference in the community as to what the
syntax should be. One possibility is that I can revise the
document to indicate that there are many syntax possibilities
and that decisions should be made on a protocol-by-protocol
basis, being sure to note how to escape the escape for whatever
is chosen.    Or someone can suggest a way for us to make a
choice.

Thoughts?

      john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 24 Jan 2007 16:31:26 +0100

On Wed, Jan 24, 2007 at 09:14:21AM -0500,
 John C Klensin <[email-masked]> wrote 
 a message of 52 lines which said:

> I don't see how to get agreement on a single form: almost everyone
> who doesn't like \u / \U likes something different...  there is no
> evidence in the notes I have received that indicates one uniform
> strong preference in the community as to what the syntax should be.

But there is (may be) a rough consensus that every scheme with
explicit delimiters (&#xNNNN; or \u{NNNNN}) is better than any scheme
without them? If so, it would be a progress.

---

From: der Mouse <[email-masked]>
Date: Wed, 24 Jan 2007 11:03:10 -0500 (EST)

> (1) Evidence that I still cannot do ABNF properly, plus a few other
> editorial issues.  The most important of the ABNF issues is that "u"
> and "U" cannot be specified as different without escaping both into
> their character code positions (since character-literals in ABNF are
> case-insensitive).

This says to me that ABNF is the wrong tool for the job here.  It seems
to me that we really want to specify "backslash" "lowercase u" and
"backslash" "uppercase U" rather than octet sequences, so that the
result is not tied to ASCII (while it's unlikely that anyone wants to
inject Unicode into EBCDIC text, I just don't like seeing a problem
"fixed" by introducing other problems in a workaround instead of
addressing the real problem).

If ABNF can't do what we want, I'd say, use something better-suited to
the need.  Perhaps it would be enough to say that the spec uses ABNF
except that the character after the backslash is case-sensitive?

That said, I don't consider this a showstopper - not that my opinion of
what is or isn't a showstopper necessarily matters....

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: John C Klensin <[email-masked]>
Date: Wed, 24 Jan 2007 15:42:37 -0500

--On Wednesday, 24 January, 2007 16:31 +0100 Stephane Bortzmeyer
<[email-masked]> wrote:

>> everyone who doesn't like \u / \U likes something
>> different...  there is no evidence in the notes I have
>> received that indicates one uniform strong preference in the
>> community as to what the syntax should be.
> 
> explicit delimiters (&#xNNNN; or \u{NNNNN}) is better than any
> scheme without them? If so, it would be a progress.

Much as I would personally prefer that answer, I haven't seen
such a consensus emerge yet.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 25 Jan 2007 01:05:35 +0100

> actually more addressed to me personally than on the list.

Hi, I hope the issue with the list "eating" my mails is now
fixed, and try it again (minus typos, but keeping the ABNF
issue because "8 vs. 6 digits" might be still unclear):

~~~ I-D ~~~
[U+NNNN]
                               This document proposes that a specific
   variation on the latter SHOULD be used in protocols unless other
   considerations apply and explains that choice.

I disagree with that proposal, more below.

-  BMP-form =  "\u" Hex-quad
-  Full-form =  "\U" 2*2 Hex-quad
+  BMP-form =  %x5C.75 Hex-quad         ; starting with lower case "\u"
+  Full-form = %x5C.55 2*2 Hex-quad     ; starting with upper case "\U"

You fixed something there already resulting in either four or six
digits, but for a case-sensitive u vs. U you can't use "u" or "U"
in ABNF.  Sometimes this ABNF feature is annnoying.  Looking at your
fix, is that still either four or *_eight_* instead of six digits ?

   (e.g., in RFCs) although the U+NNNN form MAY be used when Unicode
   character encoding is clearly expected.

It SHOULD be used for the purpose of talking _about_ Unicode points
in the prose of Internet drafts and RFCs, but it SHOULD NOT be used
to encode only the non-ASCII characters in Unicode strings.  It MAY
be used to encode complete obviously delimited Unicode strings.

-                This specification recommends that, in the absence of
-  compelling reasons to do otherwise, the Unicode code point forms be
-  used rather than the UTF-8 ones.  There are several reasons for this,
-  including:
+                This specification recommends that, in the absence of
+  compelling reasons to do otherwise, the Unicode code point forms
+  SHOULD be used rather than the UTF-8 ones.  There are several
+  reasons for this, including:

Adding a SHOULD to 3.1, otherwise folks won't believe it.

   o  Perl uses the form \x(NNN...).  The advantage of this form is that
      there are explicit delimiters

Indeed, hitting the important C044 point in [CharMod].

   o  Java uses the form \uNNNN, but can represent characters outside
      Plane 0 (i.e., above U+FFFF) only by the use of surrogate pairs.

One of the reasons why anything with \u or \U is a non-starter, there
are too many incompatible conventions in use.

                                                   Codings that depend
      on surrogates SHOULD NOT be used.

Strong ACK.

   o  HTML and XML use the form &#xNNNN;.  Like the Perl form, this form
      has a clear terminator, reducing ambiguity.  However, it is
      generally considered ugly and awkward outside of its native HTML,
      XML, and similar contexts.

IMO it is THE encoding.  It's also trivial to convert files using this
technique into XML.   For the RFC 4646 language subtag registry I use
a simple gawk script.

   There is one significant disadvantage of the recommended form.  The

No, there are more, folks will assume that it's a convention they know
or a variant of U+NNNN[N[N]] with an arbitrary number of leading 0s.
Nobody will use \U012345 when they can hope to get away with \U12345.

   should not introduce any security issues that are not present as a

My objections are also security considerations, because folks will
screw up with this encoding it could cause havoc.

6.1.  Normative References

There should be a normative reference to the [CharMod] bible, and
especially to its conformance criteria C042 up to C048 starting at
http://www.w3.org/TR/charmod/#C042

In theory your proposal is compatible with C044, but in practice I
fear that it won't work as you expect it.  I could live with e.g.
"authors SHOULD either pick hex. NCRs as in XML or" (your proposal),
but in fact I think that the XML-notation is much better.

Frank

---

From: John C Klensin <[email-masked]>
Date: Wed, 24 Jan 2007 14:30:06 -0500

--On Wednesday, 24 January, 2007 11:03 -0500 der Mouse
<[email-masked]> wrote:

>> other editorial issues.  The most important of the ABNF
>> issues is that "u" and "U" cannot be specified as different
>> without escaping both into their character code positions
>> (since character-literals in ABNF are case-insensitive).
 
> This says to me that ABNF is the wrong tool for the job here.
> It seems to me that we really want to specify "backslash"
> "lowercase u" and "backslash" "uppercase U" rather than octet
> sequences, so that the result is not tied to ASCII (while it's
> unlikely that anyone wants to inject Unicode into EBCDIC text,
> I just don't like seeing a problem "fixed" by introducing
> other problems in a workaround instead of addressing the real
> problem).

It is clear to me that, if we can decide what we are trying to
do, we can invent ABNF to do it -- without introducing the
constraint about which you are concerned.   It may be a bit
clumsy, but it is feasible.   

The main problem here is a personal one: for historical reasons
going back long before there was an IETF, I've never been a big
ABNF fan.  I'm even less of a fan of some of the extensions in
RFC 2234/ 4234 which I consider fussy, too extensive, and
unnecessary. While I have accepted the IETF consensus on ABNF
(with the extensions), that distaste, aggravated by the onset of
old age, has impeded my learning it well and preventing my
remembering all of the details that I have learned from one time
to the next.  The IETF should certainly not make decisions on
the basis of my problems with ABNF; even I won't make decisions
on that basis.  I very much appreciate the efforts of those who
have learned that they have to check my work and their patience
in correcting it.  But let's make the right decisions and not
get hung up on my idiosyncrasies. 
 
> If ABNF can't do what we want, I'd say, use something
> better-suited to the need.  Perhaps it would be enough to say
> that the spec uses ABNF except that the character after the
> backslash is case-sensitive?

As I said, I'm confident we can figure something out.  That
would certainly be among the alternatives although my instinct
would be to hunt up Dave and Paul and ask them for advice, at
least as a first step.

> That said, I don't consider this a showstopper - not that my
> opinion of what is or isn't a showstopper necessarily
> matters....

Nor mine.

     john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 25 Jan 2007 01:05:35 +0100

~~~ I-D ~~~
[U+NNNN]

                                                   Codings that depend

Strong ACK.

Frank

---

From: John C Klensin <[email-masked]>
Date: Wed, 24 Jan 2007 14:30:06 -0500

Nor mine.

     john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 25 Jan 2007 08:01:18 +0000

> such a consensus emerge yet.

Well, let's start by seeing if there is.

I've seen three people here in favour of it. I've seen you saying that one
variety (the XML one) is ugly, but I don't know if you think that that
outweighs the benefits of explicit delimiters.

So: is there anybody here - including John - who thinks that the chosen
format SHOULD NOT use explicit delimiters?

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 25 Jan 2007 08:05:05 +0000

> The mail seems to me to fall into three categories:
[...]
> purpose of this document/ effort, so we are making progress.

Good.

I presume that we can also throw out any suggestion of using surrogates
for code points above U+FFFF.

> addressed,

Agreed - they're nitpicks rather than core issues. Once we've solved the
core points, they'll fall down easily.

> If it is

I would suggest we solve those first, then worry about a new draft.

> One possibility is that I can revise the
> is chosen.

I don't think that's beneficial at this point. Let's explore the delimited
versus undelimited question first.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 25 Jan 2007 08:06:06 +0000

> would be to hunt up Dave and Paul and ask them for advice, at

Paul works in the same building as me, so I'll happily deal with that once
we know what we're trying to do.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 25 Jan 2007 08:11:15 +0000

> are too many incompatible conventions in use.

In particular, C uses \U with 8 hex digits because, at the time, there was
a serious possibility that ISO 10646 would still allow all 32 bits (or at
least 31) to be used.

Had it been a few years later, we would probably have make \U indicate 6
hex digits rather than 8.

> Nobody will use \U012345 when they can hope to get away with \U12345.

+1

> screw up with this encoding it could cause havoc.

+2

If UTF-8 can make a security issue out of having more than one way to
encode a character, so can we.

[Which reminds me: being able to encode ASCII characters in this form might
be a security issue as well, or it might be a useful benefit.]

> but in fact I think that the XML-notation is much better.

My only, small, discomfort is that people will expect all protocols to
accept both hex (&#x1234;) and decimal (&#1234;).

---

From: John C Klensin <[email-masked]>
Date: Thu, 25 Jan 2007 06:58:40 -0500

--On Thursday, 25 January, 2007 08:01 +0000 "Clive D.W. Feather"
<[email-masked]> wrote:

> John C Klensin said:
>>> But there is (may be) a rough consensus that every scheme
>>> with explicit delimiters (&#xNNNN; or \u{NNNNN}) is better
>>> than any scheme without them? If so, it would be a progress.
> I've seen three people here in favour of it. I've seen you
> saying that one variety (the XML one) is ugly, but I don't
> know if you think that that outweighs the benefits of explicit
> delimiters.
> 
> So: is there anybody here - including John - who thinks that
> the chosen format SHOULD NOT use explicit delimiters?

I personally favor explicit delimiters, I have all along.  My
definition of "ugly" involves two things.  The first may be
unavoidable, the second is not:

(i) Use of a sequence of special characters, none of which may
be completely familiar to those who don't regularly use a fairly
full set of Roman characters and both of which, IIR, are
assigned to "national use" positions in ISO 646 BV, meaning that
they may show up in even more different ways in some
presentations.

(ii) If we are going to use explicit delimiters, my aesthetic
and historical sense, and the ability to get help from common
programming-oriented editors and syntax-checkers, favors paired
delimiters over subjectively unmatched ones, e.g.,
   X(nnnn...)
or
   U'nnnn...'
in preference to
   &#nnnn...;

These are, however, just matters of personal taste.

     john

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 22:55:12 +0100

> My only, small, discomfort is that people will expect all protocols
> to accept both hex (&#x1234;) and decimal (&#1234;).

Then they are at odds with <http://www.w3.org/TR/charmod/#C043> and
<http://www.w3.org/TR/charmod/#C045>, and missed the most important
W3C last call in this millennium...  slighty exaggerated. <g>

Seriously, that's something where a MUST might do what we both want.

RFC 4646 chapter 3.1 simply states this as fact, and it uses MUSTard
elsewhere rather frankly.  Decimal NCRs are horrible, and the last
browser worldwide not supporting hex. NCRs in text/html is under my
firm control... :-)

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 23:34:46 +0100

John C Klensin wrote:

 [&#x102345; might be ugly]
> Use of a sequence of special characters, none of which may
> be completely familiar to those who don't regularly use a
> fairly full set of Roman characters and both of which, IIR,
> are assigned to "national use" positions in ISO 646 BV

No C trigraph for "&", but "#" has "??=".

IMO that's a historic foot note, nobody used weird "national
variants" of ASCII for at least a decade.  And we're talking
about US ASCII anyway.

> the ability to get help from common programming-oriented 
> editors and syntax-checkers, favors paired delimiters over
> subjectively unmatched ones

Is that about protocols needing a new escape mechanism, or
about the prose in RFCs ?  In the former case some piece of
software will do it, in the latter case an editor macro can
take care of the raw UTF-8 or whatever it is.  An example is
<http://purl.net/xyzzy/kex/x-wiki.kex> for an XEDIT-clone.

Frank