Subject: FWD: I-D ACTION:draft-klensin-unicode-escapes-00.txt

From: John C Klensin <[email-masked]>
Date: Fri, 19 Jan 2007 10:34:27 -0500

Hi.

At the Apps Area meeting at the last IETF, there was a
discussion of ways of referring to or escaping a Unicode
character in an ASCII protocol or context.  I took an action to
write a short I-D that explicitly recommended the use of form
that reference Unicode code points directly rather than, e.g.,
encoding the octets of UTF-8.

That document has been posted and, with the concurrence of the
ADs, discussion has been directed to the [email-masked]
list.

If you have any interest in internationalization issues, a
careful reading of, and comments on, this proposal would be
greatly appreciated.  

In the absence of comments, I will be pushing for Last Call
fairly soon.

thanks,
     john

Subject: I-D ACTION:draft-klensin-unicode-escapes-00.txt
From: [email-masked]
To: [email-masked]
Date: Thu, 18 Jan 2007 15:50:02 -0500
List-Id: i-d-announce.ietf.org

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.

	Title		: ASCII Escaping of Unicode Characters
	Author(s)	: J. Klensin
	Filename	: draft-klensin-unicode-escapes-00.txt
	Pages		: 8
	Date		: 2007-1-18
	
   There are a number of circumstances in which an escape mechanism is
   needed in conjunction with a protocol to encode characters that
   cannot by represented or transmitted directly.  With ASCII coding the
   traditional escape has been either the decimal or hexadecimal offset
   of the character, written in a variety of different ways.  The move
   to Unicode, where characters occupy two or more octets and may be
   coded in several different forms, has further complicated the
   question of escapes.  This document discusses some the options now in
   use and makes a proposal for general use in IETF protocols.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-klensin-unicode-escapes-00.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-klensin-unicode-escapes-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-klensin-unicode-escapes-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/i-d-announce

---

From: Keith Moore <[email-masked]>
Date: Fri, 19 Jan 2007 11:35:47 -0500

based on a quick scan, it mostly looks fine to me, with a few caveats:

- it should be clear that this is for newly-designed protocols only.  it 
shouldn't be interpreted as a request to change existing protocols 
(including deployed and nonstandard protocols being standardized by 
IETF), as this would generally break backward compatibility by changing 
the meaning of '\'

- it should be clear that this is for occasional use of non-ASCII 
characters within a protocol field that is constrained to contain only 
ASCII characters (or a subset), rather than a recommendation for how to 
represent non-ASCII characters in a protocol field that is capable of 
carrying, say, UTF-8.

if the text already makes this clear, I apologize for not reading the 
document more carefully.

it might also be worthwhile to recommend this notation for use in 
describing character literals in RFCs, in such a way that it could be 
referenced by such RFCs.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 19 Jan 2007 11:43:34 -0500

one more caveat: protocol specifications need to specify this notation 
explicitly (either directly or by reference to the published RFC) if 
they are going to use it. conversely, this notation SHOULD NOT (maybe 
MUST NOT) be used unless it is part of the protocol specification.

one problem I see with introducing any new notation for characters is 
that it does create normalization issues by introducing additional ways 
to say the same thing.  e.g. after introducing this notation, "A" could 
also be expressed as \u0041 or \U00000041.  and it then becomes 
necessary to manage this conversion when copying fields from one 
protocol that supports the new notation (or in which it is benign) to 
another protocol that does not support the notation.

as an example of a potential source of problems, I'd hate to see this 
notation end up in X.509 certs.

---

From: John C Klensin <[email-masked]>
Date: Fri, 19 Jan 2007 12:40:46 -0500

--On Friday, 19 January, 2007 11:35 -0500 Keith Moore
<[email-masked]> wrote:

> - it should be clear that this is for newly-designed protocols
> only.  it shouldn't be interpreted as a request to change
> existing protocols (including deployed and nonstandard
> protocols being standardized by IETF), as this would generally
> break backward compatibility by changing the meaning of '\'

That was intended to be clear already.  If it is not
sufficiently so, suggested text, or at least a place to put it,
would be welcome.

More generally, I'm not completely wild about the \u and \U
conventions, as you can probably deduce from the text.  It
seemed like the best choice on balance, but this is an area in
which, were a consensus to emerge around something else, I would
enthusiastically agree.

> - it should be clear that this is for occasional use of
> non-ASCII characters within a protocol field that is
> constrained to contain only ASCII characters (or a subset),
> rather than a recommendation for how to represent non-ASCII
> characters in a protocol field that is capable of carrying,
> say, UTF-8.
> 
> if the text already makes this clear, I apologize for not
> reading the document more carefully.

I don't know if it is clear enough or not.   At some level, if
you didn't conclude that it was clear on reading the draft, then
that is evidence that it isn't clear enough... but I don't know
how carefully you read it.  Certainly I agree with both points
and would welcome suggestions as to how they can be clarified if
they are not sufficiently clear.

> it might also be worthwhile to recommend this notation for use
> in describing character literals in RFCs, in such a way that
> it could be referenced by such RFCs.

Yeah.  The document does address this, but (deliberately) more
or less in passing.   Again, I'd welcome specific community
input and suggestions about this.  I've looked at several RFCs
and U+NNNN seems to be the preferred format for character
literals and, more commonly, for identifying the code point
associated with a named character.  It is also, fwiw, the one I
prefer for that purpose.  But it is fairly poor for inline use
in a protocol.  The authoritative definition and reference for
that form is the "Code Points" section of "Appendix A:
Notational Conventions" of Unicode 5.0 (the reference to the
book is the I-D).  I don't believe that we add value by
repeating that definition in an RFC, but could be easily talked
out of that position.

>From your followup note...

> one more caveat: protocol specifications need to specify this
> notation explicitly (either directly or by reference to the
> published RFC) if they are going to use it. conversely, this
> notation SHOULD NOT (maybe MUST NOT) be used unless it is part
> of the protocol specification.

Please suggest text for specifying those rules.  I constructed
this rather more as advice to protocol designers and, to a
lesser extent, to document authors, rather than a base for
notational definitions to be included by reference.  That could
be changed, but I'd welcome textual suggestions.

> one problem I see with introducing any new notation for
> characters is that it does create normalization issues by
> introducing additional ways to say the same thing.  e.g. after
> introducing this notation, "A" could also be expressed as
> \u0041 or \U00000041.  and it then becomes necessary to manage
> this conversion when copying fields from one protocol that
> supports the new notation (or in which it is benign) to

We are, unfortunately, already _deep_ into that problem.  Part
of what motivated this draft was to prevent it from getting
worse.  And, using the example above, the conversion between
\u0041 and \U00000041 is obvious.  You can even do it by eye,
and the relationship to %41 is equally clear.  However, consider
that "A" with a Diaeresis (U+00C4 -- note the use of the "code
point" notation here).  We then have \u00C4 and \U000000C4,
which are easily mapped visually to each other and to the code
point form, which can easily be looked up in a table if you
don't know what it represents.  But it is also, if I have done
the calculation correctly, %C3%83 and that form (used in URIs
and IRIs) is seriously non-intuitive and certainly can't be
converted visually.

> as an example of a potential source of problems, I'd hate to
> see this notation end up in X.509 certs.

To which I'd have to say "what would you like to see?".  My
personal answer is that I'd prefer to see those certs contain
UTF-8 encoding with escapes used only to discuss or present the
certs in contexts in which the UTF-8 and native characters are
not available (and maybe if it is, for explanation and clarity).
But I don't think this spec has much impact on that problem.  If
it should, I'd welcome text suggestions.

General observation: I generated this document because it became
clear that someone needed to something.  The alternative was
that we continue to drift toward a "every protocol makes its own
choices, ignoring all others" situation, with many of them
favoring  escaped UTF-8 (which I consider to be nearly the worst
possible choice for most circumstances... for reasons that
should be clear even from the trivial examples above).  But I
have no particularly strong commitment to any particular
recommendation as long as we establish a recommendation.   From
this point forward, I'm more or less acting as secretary/editor
for whatever comments and consensus emerge rather than trying to
advocate a particular solution or specification.   One
implication of that is that if anyone believes that additional
text, clarification, or specification belongs in the document, I
would really appreciate very specific textual suggestions.

     john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 21 Jan 2007 22:06:42 +0100

On Fri, Jan 19, 2007 at 10:34:27AM -0500,
 John C Klensin <[email-masked]> wrote 
 a message of 179 lines which said:

> In the absence of comments, I will be pushing for Last Call fairly
> soon.

I may be blind but I do not see a discussion about how to escape the
escape sequence. For instance, when discussing the XML standard
(&#nnnn;), there is no mention that it requires a way to write a bare
& (&amp;).

In the same way, if you use the \unnnn standard, how do you ensure
that you can still write a real \u string if you want it?

It seems this is important enough to postpone the Last Call.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 22 Jan 2007 08:48:27 +0000

> careful reading of, and comments on, this proposal would be

Please remove the term "extended ASCII" from 1.1 - there is no such thing.
If you want to talk about an 8 bit character set, then "the ISO 8859
family" would do.

At the end of 3.1, you can't say "the same considerations" apply to
Punycode and such encodings, because they *are* more compact (your third
and textually closest point).

I disagree with the claim that the \u notation is easier to read and less
"ugly and awkward" than the HTML one. Is:

    Ank\uabcdef

really easier to parse than:

    Ank&#xabcd;ef

? I think - particularly in contexts using alphabetic text - that having a
clear and obvious delimiter is much more preferable and will produce far
less mistakes.

I can accept that the specific HTML notation is a bit clumsy, and in
particular that the use of 'x' implies other bases are available. Therefore
I would prefer something like \u(xxxx).

The text in 3.2 implies that \U is followed by 6 digits, not 8. That risk
of confusion (particularly since Unicode ends at U+10FFFF) is itself
reason for concern with any undelimited option.

I haven't seen a "consensus" that there isn't a problem with being
case-sensitive; the fact that you messed up the ABNF on this particular
point is evidence that this is unwise. I know C does it that way (and in
retrospect, I would have fought harder for doing it differently), but C has
always been case sensitive in such matters. IETF protocols mostly aren't.

Reference [ISO-C-Chars] is wrong - the \u and \U notation was added to
ISO 9899 in the 1999 edition, not in that TR. They aren't extensions; they
are part of the core language.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 22 Jan 2007 09:02:30 +0000

> sufficiently so, suggested text, or at least a place to put it,

How about adding "new" before "protocols" in the middle paragraph of 1.1
and the abstract?

> how carefully you read it.

I don't think it would hurt to add something in 1.1. I'm not sure how to
word it, but something about "Some protocols already accept native UTF-8 or
some other encoding of Unicode, and this recommendation does not apply to
such protocols.".

> I've looked at several RFCs
> book is the I-D).

I don't have that book. The online version 4.1 suggests the notation
<U+0061, U+0300>, which can be abbreviated to <0061, 0030>. This would
still need some kind of introductory indicator (like \u) to show that it's
a Unicode escape.

> be changed, but I'd welcome textual suggestions.

"This specification is a recommendation to protocol designers and document
authors. A protocol or other specification MUST NOT be interpreted as
using it unless it explicitly copies this syntax or refers to this RFC
as normative."

> But it is also, if I have done

I certainly agree that encoding of UTF-8 sequences is the wrong thing to
do.

Oh: you should explicitly forbid the use of surrogates to encode characters
above U+FFFF.

> But I
> recommendation as long as we establish a recommendation.

(1) I agree that anything is better than nothing.

(2) While \uXXXX is better than encoded UTF-8, it's far worse than
something explicitly delimited.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 22 Jan 2007 09:03:45 +0000

> that you can still write a real \u string if you want it?

To write the string "\u1234", you write "\u005Cu1234". Isn't that obvious
enough not to need stating?

---

From: Julian Reschke <[email-masked]>
Date: Mon, 22 Jan 2007 10:25:39 +0100

> enough not to need stating?

I would understand "obvious" as "the first thing I can think of". Under 
that assumption, I would find it obvious if "\\u" would do the trick, 
thus any backslash needs to be escaped.

Best regards, Julian

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 22 Jan 2007 10:27:11 +0100

On Mon, Jan 22, 2007 at 09:02:30AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 
 a message of 87 lines which said:

> I certainly agree that encoding of UTF-8 sequences is the wrong
> thing to do.

++++1
 
> (1) I agree that anything is better than nothing.

+0.5
 
> something explicitly delimited.

+1 (parsing programs author's POV)

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 22 Jan 2007 10:28:28 +0100

On Mon, Jan 22, 2007 at 09:03:45AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 
 a message of 17 lines which said:

> To write the string "\u1234", you write "\u005Cu1234". Isn't that
> obvious enough not to need stating?

It was not obvious to me, I did not even think of it (\\u1234 seemed
more reasonable).

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 22 Jan 2007 09:31:37 +0000

> It was not obvious to me, I did not even think of it (\\u1234 seemed

Sorry, I was a bit terse there.

You can always do things by escaping the \ using the same notation; that
seemed obvious enough not to need stating.

I agree that \\ is easier to read, but it might not be compatible with
other uses of \ in the protocol this is going into. As such, it might be
seen as inappropriate to require it. However, we could add it as a MAY.