Subject: Escaping Unicode characters

From: John C Klensin <[email-masked]>
Date: Thu, 09 Nov 2006 18:04:08 -0500

Hi.

In looking through protocols approved in the IETF or being 
worked on, we find two different methods of encoding/ escaping 
Unicode characters.  One, used in URIs and the IRI->URI 
transition, among others, uses a %NN-style mechanism on a UTF-8 
string.   The other, used in DHCP and elsewhere, uses a 
\uNNNN-style mechanism to identify the Unicode character, rather 
than capturing its encoding.

In most cases that we have kept protocols and protocol elements 
in ASCII, rather than going to some "binary" form (such as 
ASN.1), a significant part of our motivation was to keep things 
readable and understandable by people.   If that is our intent, 
then the \uNNNN form, or its close relative, U+NNNN, are far 
preferable to %-forms on the UTF-8.  \uNNNN can be easily looked 
up in tables; encoded UTF-8 requires either a computer program 
(however trivial) or some fussing around to decode, after which 
one needs to get to U+nnnn form to do the lookup.

But, in any event, we are in need of some architectural guidance 
as to when to use which one -- arbitrary choices on a 
per-protocol basis do no one any good.

    john

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 9 Nov 2006 15:33:09 -0800

At 6:04 PM -0500 11/9/06, John C Klensin wrote:
>Hi.
>
>In looking through protocols approved in the IETF or being worked 
>on, we find two different methods of encoding/ escaping Unicode 
>characters.  One, used in URIs and the IRI->URI transition, among 
>others, uses a %NN-style mechanism on a UTF-8 string.   The other, 
>used in DHCP and elsewhere, uses a \uNNNN-style mechanism to 
>identify the Unicode character, rather than capturing its encoding.
>
>In most cases that we have kept protocols and protocol elements in 
>ASCII, rather than going to some "binary" form (such as ASN.1), a 
>significant part of our motivation was to keep things readable and 
>understandable by people.   If that is our intent, then the \uNNNN 
>form, or its close relative, U+NNNN, are far preferable to %-forms 
>on the UTF-8.  \uNNNN can be easily looked up in tables; encoded 
>UTF-8 requires either a computer program (however trivial) or some 
>fussing around to decode, after which one needs to get to U+nnnn 
>form to do the lookup.
>
>But, in any event, we are in need of some architectural guidance as 
>to when to use which one -- arbitrary choices on a per-protocol 
>basis do no one any good.

+1 to U+nnnn

---

From: Ned Freed <[email-masked]>
Date: Thu, 09 Nov 2006 15:47:55 -0800 (PST)

> +1 to U+nnnn

+1

				Ned

---

From: Lisa Dusseault <[email-masked]>
Date: Thu, 9 Nov 2006 15:55:29 -0800

Sounds like a 1 page RFC to me...

Lisa

---

From: Tony Hansen <[email-masked]>
Date: Thu, 09 Nov 2006 19:03:23 -0500

with 3 pages of boiler plate :-)

---

From: Chris Newman <[email-masked]>
Date: Thu, 09 Nov 2006 16:14:37 -0800

Does the proposed more readable syntax have a model for characters outside 
Plane 0?

U+NNNNNN?

Regardless, I agree we should think about this issue and have guidance.

                - Chris

John C Klensin wrote on 11/9/06 18:04 -0500:

> Hi.
>
> In looking through protocols approved in the IETF or being worked on, we find
> two different methods of encoding/ escaping Unicode characters.  One, used in
> URIs and the IRI->URI transition, among others, uses a %NN-style mechanism on
> a UTF-8 string.   The other, used in DHCP and elsewhere, uses a \uNNNN-style
> mechanism to identify the Unicode character, rather than capturing its
> encoding.
>
> In most cases that we have kept protocols and protocol elements in ASCII,
> rather than going to some "binary" form (such as ASN.1), a significant part
> of our motivation was to keep things readable and understandable by people.
> If that is our intent, then the \uNNNN form, or its close relative, U+NNNN,
> are far preferable to %-forms on the UTF-8.  \uNNNN can be easily looked up
> in tables; encoded UTF-8 requires either a computer program (however trivial)
> or some fussing around to decode, after which one needs to get to U+nnnn form
> to do the lookup.
>
> But, in any event, we are in need of some architectural guidance as to when
> to use which one -- arbitrary choices on a per-protocol basis do no one any
> good.
>
>     john
>
>
>

---

From: John C Klensin <[email-masked]>
Date: Thu, 09 Nov 2006 20:41:16 -0500

--On Thursday, November 09, 2006 16:14 -0800 Chris Newman 
<[email-masked]> wrote:

> Does the proposed more readable syntax have a model for
> characters outside Plane 0?
>
> U+NNNNNN?

yes.  and yes.   And, yes, that is what is already specified in 
a few protocols.

> Regardless, I agree we should think about this issue and have

thanks
    john

---

From: John C Klensin <[email-masked]>
Date: Thu, 09 Nov 2006 20:43:50 -0500

> Sounds like a 1 page RFC to me...

If the responses continue in today's vein, I'll take the action 
to write it.

thanks,
   john

>>> > But, in any event, we are in need of some architectural
>>> > guidance as to when to use which one -- arbitrary choices
>>> > on a per-protocol basis do no one any good.
>>
>>> +1 to U+nnnn
>>
>> +1
>>
>> 				Ned
>>
>

---

From: Martin Duerst <[email-masked]>
Date: Fri, 10 Nov 2006 14:08:26 +0900

At 08:04 06/11/10, John C Klensin wrote:
>Hi.
>
>In looking through protocols approved in the IETF or being worked on, we find two different methods of encoding/ escaping Unicode characters.  One, used in URIs and the IRI->URI transition, among others, uses a %NN-style mechanism on a UTF-8 string.   The other, used in DHCP and elsewhere, uses a \uNNNN-style mechanism to identify the Unicode character, rather than capturing its encoding.

Hello John, others,

In general, I agree with \uHHHH being better than %hh. The choice of
UTF-8 based %hh in URIs/IRIs (I don't know of any other place it is
used) was not because this is considered the a-priori best
option, but because it was the best way to make sure we would be
backwards-compatible.

I strongly suggest everybody have a look at
http://www.w3.org/TR/charmod/, and for the issue of escaping in
particular http://www.w3.org/TR/charmod/#sec-Escaping.

Of special importance, or maybe potentially controversial in an
IETF context, in that section are C044 and C046
(http://www.w3.org/TR/charmod/#C044 and http://www.w3.org/TR/charmod/#C046).

C044 says:  Escape syntax SHOULD require either explicit end delimiters or
a fixed number of characters in each character escape. Escape syntaxes where
the end is determined by any character outside the set of characters
admissible in the character escape itself SHOULD be avoided.

Thus having \uHHHH (\u followed by four hex characters) and
\Uhhhhhh (\U followed by six hex characters) is okay, but
allowing \u followed by a variable number of characters
(similar to e.g. what CSS does) is a bad idea.

C046 says: Escaped characters SHOULD be acceptable wherever their unescaped
forms are; this does not preclude that syntax-significant characters, when
escaped, lose their significance in the syntax. In particular, if a character
is acceptable in identifiers and comments, then its escaped form should also
be acceptable.

The main reason for this is that in XML, it doesn't work that way.
You can write <name>D&#xFC;rst</name>, but not <D&#xFC;rst/>
or <!-- D&#xFC; --> (well, the later is possible, but in there,
'&#xFC;' doesn't mean u-umlaut, it just means this sequence of characters.

This means that when converting something from one encoding to
another, you sometimes escape and sometimes have to fail,
and you need to parse the content to know when to do what.
For IETF protocols, that's maybe not that much of an issue,
because you usually don't have identifiers with non-ASCII
characters. But in case you do have that, and for comments,
it's important.

>In most cases that we have kept protocols and protocol elements in ASCII, rather than going to some "binary" form (such as ASN.1), a significant part of our motivation was to keep things readable and understandable by people.   If that is our intent, then the \uNNNN form, or its close relative, U+NNNN, are far preferable to %-forms on the UTF-8.  \uNNNN can be easily looked up in tables; encoded UTF-8 requires either a computer program (however trivial) or some fussing around to decode, after which one needs to get to U+nnnn form to do the lookup.

Yes indeed.

>But, in any event, we are in need of some architectural guidance as to when to use which one -- arbitrary choices on a per-protocol basis do no one any good.

I think guidance would be a good thing. But I'm not sure a one-page
(with three pages of boilerplate) RFC is the right thing. Not that
this topic needs more space, but the problem is that if we write
a one-page RFC on escaping, then a one-page RFC e.g. on private
use codepoints, and so on, we will end up with an awfully long list
of RFCs that poor spec writers have to go through and reed in
order to do the right thing (or just only in order to find out
that they are not affected). So at least in the long term, we
should aim for longer documents cosolidating related topics.

Also, while I think very much about character encoding is very
fundamental and architectural, the choice between %hh and \uhhhh,
while being important and being helped by guidance, is not at
all something architectural. The way John has explained it,
and the way I see it, it's mostly a usability issue for human
users.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 10 Nov 2006 07:01:51 +0000

>> +1 to U+nnnn

-1

What's the delimiter with U+NNNN? How do I distinguish that from those six
characters in plain text? How many hex digits are expected? This sort of
notation works in running text read by humans, but not machine-readable
text.

\u is at least a clear escape code, though it still has the length problem.
ISO C and C++ use \uNNNN and \UNNNNNNNN, but I've never been that happy
with the only distinction being case. Particularly where the next character
is also a hex digit.

Human readability of long strings of \uNNNN leaves something to be desired.
It strikes me (and has struck me in the past in other contexts) that a
notation that allows multiple characters to be included would be
worthwhile. So how about:

    \u+(NNNN,NNNN,NNNNNNNN,NN)

e.g.

    \u+(00C9,2144,1FF21,B7)

(possibly using [] or {} or <> instead of () as delimiters)?

This also leaves room for extensions such as:

   \u+('e acute')

though I wouldn't want to require implementations to accept that.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: John C Klensin <[email-masked]>
Date: Fri, 10 Nov 2006 10:07:58 -0500

--On Friday, November 10, 2006 07:01 +0000 "Clive D.W. Feather" 
<[email-masked]> wrote:

> What's the delimiter with U+NNNN? How do I distinguish that
> from those six characters in plain text? How many hex digits
> are expected? This sort of notation works in running text read
> by humans, but not machine-readable text.
>
> \u is at least a clear escape code, though it still has the
> length problem. ISO C and C++ use \uNNNN and \UNNNNNNNN, but
> I've never been that happy with the only distinction being
> case. Particularly where the next character is also a hex
> digit.
>
> Human readability of long strings of \uNNNN leaves something
> to be desired. It strikes me (and has struck me in the past in
> other contexts) that a notation that allows multiple
> characters to be included would be worthwhile. So how about:
>

Clive, I think there are really two questions here.   I didn't 
separate them in my initial note because I didn't want to make 
it a lot longer and didn't really want to have this discussion 
until the first question was settled.

Question 1: Do we want to prefer
  (a) Some escaped form of the UTF-8 string  or
  (b) Some representation of the Unicode table location/ 
character number.
   And, if not universally, under which circumstances do we 
prefer which one?
This question is really philosophical, even if not deeply 
architectural, and relates to general principles of protocol 
design (albeit not profound ones).

Question 2: Given the answer to Question 1, what is the best 
syntax for presentation and input?  This is a human interface 
question and is the one addressed in your note.  I have opinions 
about it (like you, I prefer explicit starting and ending 
delimiters of some sort), but I don't think the question is even 
interesting until we arrive at an answer to Question 1.

Please see Martin Duerst's note, with which I am in complete 
agreement, for more on this topic.

    john

p.s. IAB: One of the reasons you were copied on this is that, 
while I expect to produce Lisa's 'One page' RFC, it will update 
RFC2277 which, in turn, was effectively an output of RFC2130. 
I increasingly believe that the issues addressed in 2130 --well 
over a decade ago in an area where I think we have learned a 
great deal, mostly the hard way-- need to be reopened and 
reviewed.