Subject: RE: UTF-8

From: "John Brezak" <[email-masked]>
Date: Sun, 11 Nov 2001 21:14:52 -0800

Realistically, you cannot limit Kerberos names (including realms) to
just the 7-bit ascii character set - even in the short term.

Whatever is done needs to remain consistent with internationalization of
DNS names since Kerberos realm names can be represented by DNS names.

Is there a standard mechanism that can use the Unicode character set in
GeneralStrings?

-----Original Message-----
From: Russ Allbery [mailto:[email-masked]] 
Sent: Saturday, November 10, 2001 3:44 PM
To: [email-masked]
Subject: Re: UTF-8

Jeffrey Altman <[email-masked]> writes:

> For those who are not familiar with the details behind the creation of

> Unicode, unlike other ISO character sets or OEM Code Pages, Unicode is

> not simply a large table of characters.  In order to reduce the number

> of code points that would be needed, Unicode allows characters to be 
> described as a base character and set of modifiers.  Of course, a 
> character that has more then one modifier can be represented in 
> 2^(n-1) ways where 'n' is the number of modifiers.  But that is not 
> the worst part.  Due to socio-political concerns many characters which

> could be represented as a base character plus modifier are also 
> present as a combined character.

Unicode however also defines a set of normalization rules that are
intended to deal with this situation.  For the purposes of Kerberos, I
expect one would want to use the most aggressive canonicalization rule
(compatibility decomposition).  The choice of NFKD or NFKC is largely
dependent on the needs of the application; I'm not sure which would be
more appropriate for Kerberos.  (The difference lies in whether
characters are left in a fully decomposed state or not.)

See <http://www.unicode.org/unicode/reports/tr15/> for the
specifications of Unicode normalization.  There are available libraries
for doing Unicode normalization.

> Not to mention all of the characters are visibly alike but are 
> semanticly different.

Does this matter?  If people use those characters as part of their
identity, they may need to deal with that issue, but I don't see why the
protocol needs to beyond using Unicode normalization.  Note that Unicode
compatibility decomposition already deals with some of this, as I
recall.

If Kerberos *really* needs more than that, the DNS working group is
working on more aggressive canonicalization methods to deal with the
fact that hostnames are frequently mentioned in print and then retyped.
But Kerberos strikes me as a much simpler case that doesn't have to be
as robust against mis-entered names.

> Then you can throw in the fact that Unicode 1.0, 1.1, and 2.0 are not 
> upwardly compatible.  The sorting rules have changed with each Unicode

> release.

I'm not sure why Kerberos would care about sorting rules, but I may be
missing something important.

The normalization rules are guaranteed to be upwardly compatible to all
future versions of Unicode.

> The issues are not simply technical ones.  There are some major 
> socio-political issues here as well.  Unicode is a 16-bit 
> character-set.

No, it's not, and hasn't been since Unicode 3.0.  See, for example,
<http://www.unicode.org/unicode/faq/basic_q.html#19>.  Unicode defines a
32-bit character standard.

> ISO-10646 is the superset of Unicode which is a 32-bit space.

No, it's not.  See <http://www.unicode.org/unicode/faq/unicode_iso.html>
for an explanation of the relationship between ISO 10646 and Unicode.
Unicode is actually a superset of ISO 10646, sharing the same character
encoding and adding a great deal of additional useful information and
standardization to make it easier to use the character set.

There is no need to choose one or the other; they define precisely the
same set of characters, although Unicode releases standards faster (so
there are characters that have been agreed upon by both the ISO working
group and the Unicode Consortium which are in Unicode 3.1 but which have
not yet been released as an ISO standard, although they will be).

> The choice of UTF-8 is also troublesome.  UTF-8 discrimates in favor 
> of Western European languages and against those of Asia.  This is 
> because a US-ASCII character takes up one byte in UTF-8 whereas a CJK 
> character can take four bytes to represent.  Whereas using straight 
> Unicode requires two bytes for all characters.

There is no such thing as "straight Unicode" in the context in which
you're using it, as Unicode is not a character encoding.  Unicode is a
set of characters, and you have to pick an encoding.

It sounds like you're talking about UTF-16 without surrogates in the
last sentence above, which is obviously a bad encoding to choose for any
new application for the reasons that you state.  The IETF is
standardizing on UTF-8; the other reasonable alternative is UTF-32.
UTF-16 even with surrogates should be avoided since it combines all of
the problems of both UTF-8 and UTF-32 for no real gains.

When evaluating whether UTF-8 "discriminates" (which I think is a very
poor way of phrasing the problem, although a common one -- the problem
isn't that the creators of Unicode didn't care as much about Asian
languages, but that Asian languages have a *lot* of characters and
therefore by necessity require more bytes to encode), it's worth bearing
in mind that most *native* Asian encodings use at least two bytes per
character.  The UTF-8 expansion over the native encodings is not
actually that bad (and I don't think that we need to be so parsiminous
with bytes at this level; in fact, using UTF-32 may well be the best
choice).

> (If we decide on ISO-10646 things get even worse with characters in 
> higher planes taking up to eight bytes to represent.)

Those are incredibly rare characters that are very unlikely to be seen
within a Kerberos application; in fact, such characters are currently
all reserved for future assignment.  I don't think this is a problem.

> In the meantime, I am going to write based upon my comments above, 
> Tom's comments in revisions, and Jeff's summary of a few days ago to 
> this list an Informational I-D describing why we as a Working Group 
> strongly recommend that all current implementations and deployments of

> Kerberos 5 restrict themselves to US-ASCII.

<http://www.unicode.org/> has a lot of good information that you may
want to review before writing that I-D.  Quite a lot has changed since
Unicode 2.0.

-- 
Russ Allbery ([email-masked])
<http://www.eyrie.org/~eagle/>

---

From: "Douglas E. Engert" <[email-masked]>
Date: Mon, 12 Nov 2001 08:41:49 -0600

To help keep the WG focused, there really needs to be two discussions with 
regards to UTF-8.

The first is how to get the revisions draft moving, with adding just enough
to it to allow for a future solution without causing backward compatibility
problems. (Jeff Altman is preparing a ID about the problems, and current 
practices.)

The second is to solve the problem but in a separate document. 

So focusing on the first point, we have now seen two proposals to allow 
for expansion. The CHOICE, and today a proposal for :

PrincipalName ::= SEQUENCE {
        name-type[0]            INTEGER,
        name-string[1]          SEQUENCE OF GeneralString,
        name-string-utf8[2]     SEQUENCE OF UTF8String OPTIONAL

Are there any others?

-- 

 Douglas E. Engert  <[email-masked]>
 Argonne National Laboratory
 9700 South Cass Avenue
 Argonne, Illinois  60439 
 (630) 252-5444

---

From: Tom Yu <[email-masked]>
Date: 12 Nov 2001 23:56:16 -0500

>>>>> "deengert" == Douglas E Engert <[email-masked]> writes:

deengert> So focusing on the first point, we have now seen two
deengert> proposals to allow for expansion. The CHOICE, and today a
deengert> proposal for :

deengert> PrincipalName ::= SEQUENCE {
deengert>         name-type[0]            INTEGER,
deengert>         name-string[1]          SEQUENCE OF GeneralString,
deengert>         name-string-utf8[2]     SEQUENCE OF UTF8String OPTIONAL

I would much prefer that the above solution not be utilized.  I
believe it is a step in the wrong direction with respect to
extensibility and clarity.  Making a KerberosString type that is an
extensible CHOICE will add a useful abstraction in the correct place,
the sort of thing that is sorely lacking in the RFC1510 protocol.

My proposal is roughly to use the following:

    KerberosString ::= CHOICE {
        general GeneralString (VisibleString),
        ...
    }

which we could then update (when we figure out how to do UTF-8
properly) into something like:

    KerberosString ::= CHOICE {
        general GeneralString (VisibleString),
        ...,
        utf8    UTF8String
    }

We would replace most of the existing occurrences of GeneralString
with KerberosString.  The DER encoding would not change as a
consequence of this replacement, provided that we use the
GeneralString alternative of the CHOICE.

The constraint to VisibleString would be only enforced for the
purposes of permitting the widest compatibility.  Implementations
should be free to accept other character encodings (perhaps
erroneously) tagged as GeneralString in order to maintain
interoperability, but implementors should be aware that emitting any
encoding beyond 7-bit US-ASCII will lead to problems.

Incidentally, one could argue that VisibleString is too narrow a
constraint, and that we should permit the C0 control characters of
US-ASCII as well.  If we were to do that, we would have to be careful
to prevent implementors from emitting escape sequences and the like,
somehow.

---Tom

---

From: "Douglas E. Engert" <[email-masked]>
Date: Tue, 13 Nov 2001 08:57:21 -0600

I personally like the CHOICE as Tom has described below, and would like
to see this in the next draft of the revisions. 
Any other comments?

--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 13 Nov 2001 10:37:34 -0500

IMO:

a) breaking wire compatibility now and reducing the pain of future
   changes beats adding bloat on the wire. As long as old clients,
   services and KDCs continue working fine, with restrictions wrt their
   referencing non-ASCII names(*).

b) clients should be able to detect whether a KDC supports the new
   protocol changes -- I'm not sure how though; the technical details
   of how clients would detect KDC support for these changes have not
   been spoken to. New clients should cache KDC support for these
   changes on a per-realm basis.

c) all KDCs in a given realm should support the new proto changes or
   none of the KDCs in that realm should. This way new clients can cache
   KDC support for these changes status on a per-realm basis.

d) life is tough -- old clients(*) won't be able to speak to services
   that have non-ASCII names or which reside in realms with non-ASCII
   names(**).

e) life is tough -- users with non-ASCII names will not be able to login
   on old clients(*).

(*) Excepting clients that support treating GeneralString as UTF-8;
    e.g., MS clients.

(**) Unless princ name canonicalization is used to provide ASCII
     "canonical aliases" for principals with non-ASCII canon names (ugh).

Nico

> to see this in the next draft of the revisions. 

[...]

--
-DISCLAIMER: an automatically appended disclaimer may follow. By posting-
-to a public e-mail mailing list I hereby grant permission to distribute-
-and copy this message.-

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

---

From: [email-masked] (Johan Danielsson)
Date: 14 Nov 2001 12:03:30 +0100

Tom Yu <[email-masked]> writes:

> I would much prefer that the above solution not be utilized. 

I can't say I'm very fond of it myself, but I would pefer to keep the
protocol compatible. I don't know how difficult that is though.

/Johan

---

From: Tom Yu <[email-masked]>
Date: 14 Nov 2001 11:30:48 -0500

>>>>> "joda" == Johan Danielsson <[email-masked]> writes:

>> I would much prefer that the above solution not be utilized. 

joda> I can't say I'm very fond of it myself, but I would pefer to keep the
joda> protocol compatible. I don't know how difficult that is though.

Whatever happens, the protocol will change.  However this change
happens, I will have a stragety for backwards compatibility.

---Tom