Subject: Re: UTF-8

From: "Douglas E. Engert" <[email-masked]>
Date: Fri, 09 Nov 2001 10:43:19 -0600

Jeff Altman,
I like the idea of an informational RFC. Why don't you go ahead and write
it. I see other members have been giving good feed back as well.

But the revisions still needs some statements as to how we intend to handle
this in the future, Tom, I believe you where going to write this?

I would also like to move this discussion to the list. 

Jeffrey Altman wrote:
> 
> At the end of the last meeting Nico and I were having a heated
> discussion about this issue.  I was told to drop it until such time as
> jhutz was able to publish the notes on the subject.  What I was
> calling for was a statement from the working group saying:
> 
>   to ensure future interoperability between existing deployments
>   of Kerberos and future standards efforts the Kerberos Working
>   Group strongly recommends that users of Kerberos do not deploy
>   Kerberos principal or service names that include characters not
>   included in the 7-bit US-ASCII character set.
> 
> Is there any reason why this recommendation cannot be issued today as
> an Informational RFC?
> 
> Assuming that is not objectionable.  I am pretty sure that we already
> know what we want the character set support to look like.  Its just
> a question of what the ASN.1 encoding is going to look like.
> 
> Clearly we have two needs:
> 
>  . assuming the above recommendation is adhered to, we want existing
>    principal and service names to just work
> 
>  . for all new implementations we want to use UTF-8 as the encoding
>    with an appropriate set of rules for determining a canonical form
>    so that clients and servers can be independently written.
> 
> Now as far as the revisions document goes, all that is required is
> to specify how the current GeneralString and the new UTF-8 string
> can be distinguished.
> 
> I don't think there is anyway that we can provide a backward
> compatibility mode to allow the GeneralString contents to be treated
> as something else.  If there are any deployments of Kerberos 5 using
> Big-5, JIS, or some other character encoding for principal names
> they are going to have to perform a conversion to the new UTF8String
> in order to maintain interoperability.
> 
> Microsoft is most likely going to need to hack support for both the
> new UTF8String and if not that the old GeneralString as UTF8 as they
> already do.  In fact, assuming that we do issue the above
> recommendation as an Informational RFC it would be perfectly safe for
> implementations that support UTF8String to use the GeneralString as
> UTF8 to provide compatibility with Microsoft.  A non-UTF8 string sent
> in the GeneralString would cause the UTF8 parser to generate an
> invalid sequence error.
> 
> Tom, you mentioned some point back that we could support an either or
> behavior with a CHOICE field.  What are the negatives associated with
> implementing this in revisions?
> 
> As for the canonical forms.  There are two different canonical forms
> at issue:
> 
>  . canonical UTF8 encoding
> 
>  . canonical Unicode character set usage
> 
> Rules for both of these issues are available as part of the Unicode
> 3.1 specification.  Unfortunately, the rules for canonical usage of
> the Unicode character set require access to a fairly large and
> constantly changing database.  Each revision of Unicode has altered
> the new database as characters have been removed and others added.
> The Unicode Consortium has promised to never remove characters again
> but you never know what might happen in the future.  It is very
> important that the final specification indicate which revision of the
> Unicode Standard is applicable to our RFCs.
> 
> Unless I am forgetting something I think we should be able to make a
> decision on this in very short order.
> 
> I was unable to attend the last two meetings due to budgetary issues.
> I am hoping to attend the Utah meeting if only for a day or two.
> 
> - Jeff
> 
> > On 7 Nov 2001, Ken Raeburn wrote:
> >
> > > Jeffrey Hutzelman <[email-masked]> writes:
> > > >     - Lie through our teeth -- send UTF-8 strings encoded as GeneralString.
> > > >       Doesn't work because new implementations won't be able to tell the
> > > >       difference between an UTF-8 string and a locale-specific string
> > > >       encoded by an old just-send-8 implementation.
> > >
> > > Is it any worse than the existing case of two deployments using
> > > different locales?  In either case, it's the transition or
> > > interoperability where things break; each deployment can be internally
> > > self-consistent.
> >
> > Yes; it's different in an important way.  There's nothing we can do for
> > existing implmentations; they're already deployed in an environment where
> > the string usage is underspecified or incorrectly specified.  However, it
> > is critical that once a standard is set, conforming implementations be
> > able to distinguish between messages that follow the new standard and
> > messages that do not.
> 
> > > >     - May delay kerberos-revisions by up to another year
> > >
> > > JIS has indicated that he is Not Happy with the delays in
> > > kerberos-revisions.  UTF-8 is important, but time consuming.
> >
> > Agreed.
> >
> 
>  Jeffrey Altman * Sr.Software Designer      C-Kermit 8.0 Beta available
>  The Kermit Project @ Columbia University   includes Secure Telnet and FTP
>  http://www.kermit-project.org/             using Kerberos, SRP, and
>  [email-masked]          OpenSSL.  SSH soon to follow.

-- 

 Douglas E. Engert  <[email-masked]>
 Argonne National Laboratory
 9700 South Cass Avenue
 Argonne, Illinois  60439 
 (630) 252-5444

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 9 Nov 2001 13:18:43 -0500 (EST)

> I would also like to move this discussion to the list. 

... and you have.  For those just tuning in, I'll try to provide a bit of
information about what's going on.

At the working group meeting in London, a few of us agreed to talk to the
IESG and try to get a sense for what would be acceptable ways for Kerberos
to handle non-ASCII character sets.  Later in the week, we met with
Applications AD Patrik Faltstrom, whose with with the idn group has made
him something of an expert on the sort of problems we're facing.  The rest
of this message is an attempt to describe what happened at that meeting,
and what direction we believe the working group should take. 

First, a quick overview of the situation.  This is substantially what we
told Patrik, and while some of you are already familiar with these issues,
I know that some are not.

In the following discussion, I will use the phrase "new implementation" to
describe an implementation which supports the UTF-8 extensions (whatever
those turn out to be); the phrase "old implementation" describes one that
does not.

Kerberos currently makes extensive use of the GeneralString type.  The
details about what you can and cannot do with this type are rather
complicated and a bit unclear, but for our purposes we need to know the
following: 

  + In the ASN.1 DER, the encoding format used for GeneralStrings is
    capable of representing an arbitrary sequence of 8-bit octets. 
    However, not all such sequences are _permissible_ -- the set of what
    you're allowed to have in a GeneralString is limited, and of those,
    some may not be encodable in DER.

  + By tradition, Kerberos has permitted only US-ASCII characters in
    those fields which are represented as GeneralStrings.  Printable
    US-ASCII strings are clearly among those which may be legally
    encoded as GeneralStrings.  However, this traditional restriction
    has never been written down, and there are not implementations and
    installations which do not follow it.

  + There are some existing implementations (Microsoft) in which all
    of these fields are tagged as GeneralStrings but actually contain
    unescaped UTF-8.  These implementations interoperate with traditional
    implementations as long as only US-ASCII characters are used.

  + There are some existing implementations in which all of these
    fields are tagged as GeneralStrings but actually contain unescaped
    8-bit strings using whatever is the prevailing local character set.
    When the local character set is a superset of US-ASCII, such as
    any of the ISO 8859-* character sets, these implementations will
    interoperate with traditional implementations as long as only
    US-ASCII characters are used.

For various reasons, a transition to using UTF-8 encoding is clearly
desirable.  First, there is a mandate from the IESG to support
international character sets generally, and UTF-8 specifically.  Also, the
fact that there are existing installations violating the US-ASCII
tradition and accepting all the pain that goes with that indicates that
there is a clear need to support alternate character sets in places like
princpal names and passwords.

At the same time, backward compatibility with the existing installed base
is also important.  Few site administrators have the luxury of declaring a
flash cut-over of all users, applications, servers, etc to an incompatible
protocol -- many have non-local users over whom they have little or no
control.  To this end, it is important for new implementations to be able
to tell whether a particular non-US-ASCII string was encoded as UTF-8 by a
new implementation, or as something else by an old implementation.  In the
latter case, it is of course impossible to know what the "something else" 
is without being told in advance.

I have seen three proposals for how to handle the fields currently encoded
as GeneralStrings in order to accomplish such a transition:

  (1) Lie.  Start using UTF-8, but continue to encode all of these
      fields as GeneralStrings.  To my knowledge, this is what Microsoft
      is doing today.  This approach is attractive because it requires
      no changes to the message format specification, but it has several
      key problems.  First, it does not allow a new implementation to
      tell whether a particular string was encoded as UTF-8 by another
      new implementation or as some 8-bit local character set by an old
      implementation.  Also, there are potential problems with encoding
      arbitrary 8-bit strings, particularly for those who are using
      off-the-shelf ASN.1 compilers.  Finally, violating the ASN.1 spec
      in this manner would be unpopular with the ITU; my understanding
      is that this is a real issue.

  (2) Don't lie.  Start using UTF-8, encoded in GeneralStrings with the
      appropriate escapes.  While this has the appearance of following
      the spec, it still has the problem that UTF-8 cannot be legally
      encoded in a GeneralString.  This again creates problems for
      implementors using off-the-shelf ASN.1 compilers as well as
      political issues with the ITU.

  (3) Don't use GeneralString.  In all the places where we currently
      use GeneralString, begin using a new "KerberosString" type instead.
      This type would be defined as an ASN.1 choice, with GeneralString
      and some form of UTF-8 strings as alternatives.  The selection of
      which alternative to use would be based on whether one was talking
      to an old implementation or a new one.  This approach does involve
      changing the message format _specifications_, but as long as the
      GeneralString choice is used, the actual encoding does not change.
      There is a transition issue in that replacing a type with a choice
      containing that type is not always a legitimate thing to do, but
      as long as DER are used (which is always the case in Kerberos), it
      does work correctly.

It is our belief that option (3) is the correct direction for the working
group to take.  However, this is still up for discussion, and in any case
is only one of the problems that must be addressed in order to fully
specify UTF-8 support.  There are actually several other problems to be
solved, which means that specifying UTF-8 support for Kerberos will take
some time, and getting it implemented and tested will take even more time.

If done now, this is IMHO likely to delay kerberos-revisions for at least
another year.  I don't feel that is an acceptable delay -- this has
already been going on for too long, and there are implementors, users, and
other protocol documents which need new features from kerberos-revisions.

OK; so much for the overview.  We asked Patrik a number of specific
questions about what direction we should be taking and what the IESG would
be likly to support:

  - How important is it to the IESG that Kerberos support UTF-8 in the
    short term, and how quickly do we need to address this issue?
  - Must kerberos-revisions address UTF-8 now, or can we put out this
    document without UTF-8 specification, and then add it in the next
    (hopefully much quicker) round of revisions?
  - If kerberos-revisions must include UTF-8 support, can we specify
    it but make it optional-to-implement, in order to avoid causing
    problems for implementors who are trying to get something out soon?

  The outcome of our discussion was that it would be best to put out
  a document soon which does _not_ specify how to handle UTF-8, but
  does explicitly state that the semantics are well-defined only when
  all GeneralStrings are limited to US-ASCII.  This makes it clear that
  those who are using extended character sets are likely to have problems
  in the future, while not actually prohibiting such usage.

We also asked an additional question:

  - Are the proposals (described above) for backward compatibility
    reasonable?  In particular, can we allow new implementations to
    send old possibly illegal encodings (e.g. unescaped UTF-8 or
    locale-specific strings encoded as GeneralStrings) when talking to
    old implementations?

  Patrik indicated that it would be OK to do this for backward
  compatibility, but that the Kerberos specification should leave the
  precise semantics of GeneralStrings (or at least, GeneralStrings
  containing other than US-ASCII characters) unspecified, as they are
  today.  It would be good to note that there are old implementations
  which use various other semantics for those strings, but we should
  not document in kerberos-revisions what those implementations are
  or what they do.

After that, the conversation turned to a technical discussion of what some
of the details of the UTF-8 specification might be.  I'm not going to
detail that discussion here, since it was largely of a preliminary nature
and not really ready for general discussion.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: Matt Crawford <[email-masked]>
Date: Fri, 09 Nov 2001 14:52:20 -0600

> It is our belief that option (3) is the correct direction for the
> working group to take.

Then you'd often have two ways to write the same string.  Better
place a restriction on the choice or say a few words about the
equivalence test between two strings.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 9 Nov 2001 16:05:12 -0500 (EST)

> equivalence test between two strings.

Not really.  Strings that are composed entirely of US-ASCII characters can
be tagged as either GeneralString or UTF-8, but the encoding will be the
same.  While obviously all the details haven't been worked out yet, I
think the intent is that new implementations will always use UTF-8 strings
when talking to each other, and will use the GeneralString choice only
when talking to old implementations.

It is certainly possible that strings containing characters not in
US-ASCII may be encoded one way by a new implementation (using UTF-8) and
in any number of other ways by old implementations (using GeneralString).
There's not a whole lot that can be done about that, since the existing
deployed base is not going to magically change.

This is why kerberos-revisions should explicitly state that the semantics
of non-US-ASCII GeneralStrings are undefined, even if UTF-8 support does
not make it into this document.  That way, implementors are free to handle
such strings in whatever way they want, such as by returning an error or
by interpreting them based on realm-specific knowledge about nonstandard
principal names.  However, we must be careful to indicate that such
nonstandard semantics can be used only for backward compatibility, and are
not intended to serve as a loophole for nonstandard extensions.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 9 Nov 2001 16:51:14 -0500

> when talking to old implementations.

I find (3) to be quite appealing. It allows the KDC to distinguish
clients that support the new approach from clients that don't. The KDC
will have to keep track of which service principals can handle this the
new approach though.

Clients that support (3) should always try KerberosString first, then
GeneralString if the KDCs respond with an error.

Has any testing been done to see how existing KDC implementations react
when presented with requests whose strings are not GeneralString?

Cheers,

Nico
--
-DISCLAIMER: an automatically appended disclaimer may follow. By posting-
-to a public e-mail mailing list I hereby grant permission to distribute-
-and copy this message.-

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

---

From: Tom Yu <[email-masked]>
Date: 09 Nov 2001 17:21:19 -0500

>>>>> "Nicolas" == Nicolas Williams <[email-masked]> writes:

Nicolas> Has any testing been done to see how existing KDC
Nicolas> implementations react when presented with requests whose
Nicolas> strings are not GeneralString?

What exactly do you mean by this?  Do you mean requests containing
strings tagged as GeneralString but that are something else, or do you
mean requests containing strings that are tagged as something not
GeneralString, such as UTF8String?

---Tom

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 9 Nov 2001 17:24:22 -0500 (EST)

> will have to keep track of which service principals can handle this the

Yes, pretty much.  By having new implementations always encode messages
using UTF-8 strings when they know the recipient will be able to decode
them, you can build a chain that propagates that information through the
entire authentication process, where the only a priori knowledge required
is that of the KDC regarding which services can support UTF-8.

> GeneralString if the KDCs respond with an error.

Actually, probably not, because there are cases where existing
implementations will do bad things if given data they don't support.  I
don't know offhand precisely how bad it is, but it turns out that it's
important in some cases for new implementations to avoid sending
new-format messages to old implementations.  Perhaps someone has done some
testing and will speak up.

In any event, while I absolutely want to continue discussion on how best
to accomplish UTF-8 support, I also want t come to a concensus on whether
we should try to do it in kerberos-revisions or hold off until the next
document.  In case it's not already clear, it is my position that we
should _not_ delay kerberos-revisions for this, but that we should get
working on it as soon as possible after publishing revisions as an RFC.

-- Jeff

---

From: Tom Yu <[email-masked]>
Date: 09 Nov 2001 17:42:14 -0500

>>>>> "jhutz" == Jeffrey Hutzelman <[email-masked]> writes:

>> will have to keep track of which service principals can handle this the

jhutz> Yes, pretty much.  By having new implementations always encode
jhutz> messages using UTF-8 strings when they know the recipient will
jhutz> be able to decode them, you can build a chain that propagates
jhutz> that information through the entire authentication process,
jhutz> where the only a priori knowledge required is that of the KDC
jhutz> regarding which services can support UTF-8.

Ultimately, there is the problem of handling AS-REQ when the client
doesn't know whether the KDC supports new-style messages or not.  It
is, however, solvable.  Since the KDC will need to keep track of which
service principals support UTF-8, it would be significantly
advantageous if we could bundle together all forseeable changes that
might require the KDC to keep track of such capabilities.

>> GeneralString if the KDCs respond with an error.

jhutz> Actually, probably not, because there are cases where existing
jhutz> implementations will do bad things if given data they don't
jhutz> support.  I don't know offhand precisely how bad it is, but it
jhutz> turns out that it's important in some cases for new
jhutz> implementations to avoid sending new-format messages to old
jhutz> implementations.  Perhaps someone has done some testing and
jhutz> will speak up.

It is very important to not send new-format messages to old
implementations.  We have never specified a framework for changes to
the protocol, and an implementation could quite legitimately ignore a
message it doesn't recognize.  The KDC probably *should* drop messages
it doesn't recognize (our implementation does), since a
denial-of-service attack may be more likely then.

jhutz> In any event, while I absolutely want to continue discussion on
jhutz> how best to accomplish UTF-8 support, I also want t come to a
jhutz> concensus on whether we should try to do it in
jhutz> kerberos-revisions or hold off until the next document.  In
jhutz> case it's not already clear, it is my position that we should
jhutz> _not_ delay kerberos-revisions for this, but that we should get
jhutz> working on it as soon as possible after publishing revisions as
jhutz> an RFC.

I believe we should add a hole for handling future UTF-8 support.  The
message format changes shouldn't be too difficult.  Things such as
canonical encodings, normalization, etc. are going to be quite
difficult, but we can solve them later.

---Tom

---

From: "Charles Stephens" <[email-masked]>
Date: Fri, 9 Nov 2001 14:52:01 -0800

I know this assumption is dangerous, but if everything was a 7-bit
US-ASCII string encoded in a 8-bit octet, it would seem to me that most
implementations would use a simple comparison routine (like strcmp in
POSIX).  This routine should work for basic UTF-8 strings as well.  I
know there are edge cases where two glyphs can have two different
encodings, but I would think that is rare.  The UNICODE standard gives
advice on how to best deal with this.

cfs

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Tom Yu
> Sent: Friday, 09 November 2001 14:42
> To: Jeffrey Hutzelman
> Cc: Nicolas Williams; Matt Crawford; [email-masked]
> Subject: Re: UTF-8
> 
> 
> >> clients that support the new approach from clients that don't. The 
> >> KDC will have to keep track of which service principals can handle 
> >> this the new approach though.
> 
> jhutz> messages using UTF-8 strings when they know the 
> recipient will be 
> jhutz> able to decode them, you can build a chain that 
> propagates that 
> jhutz> information through the entire authentication process, 
> where the 
> jhutz> only a priori knowledge required is that of the KDC regarding 
> jhutz> which services can support UTF-8.
> 
> Ultimately, there is the problem of handling AS-REQ when the 
> client doesn't know whether the KDC supports new-style 
> messages or not.  It is, however, solvable.  Since the KDC 
> will need to keep track of which service principals support 
> UTF-8, it would be significantly advantageous if we could 
> bundle together all forseeable changes that might require the 
> KDC to keep track of such capabilities.
> 
> jhutz> implementations.  Perhaps someone has done some 
> testing and will 
> jhutz> speak up.
> 
> implementations.  We have never specified a framework for 
> changes to the protocol, and an implementation could quite 
> legitimately ignore a message it doesn't recognize.  The KDC 
> probably *should* drop messages it doesn't recognize (our 
> implementation does), since a denial-of-service attack may be 
> more likely then.
> 
> kerberos-revisions 
> jhutz> or hold off until the next document.  In case it's not already 
> jhutz> clear, it is my position that we should _not_ delay 
> jhutz> kerberos-revisions for this, but that we should get 
> working on it 
> jhutz> as soon as possible after publishing revisions as an RFC.
> 
> I believe we should add a hole for handling future UTF-8 
> support.  The message format changes shouldn't be too 
> difficult.  Things such as canonical encodings, 
> normalization, etc. are going to be quite difficult, but we 
> can solve them later.
> 
> ---Tom
>

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 9 Nov 2001 18:02:01 -0500

> GeneralString, such as UTF8String?

The latter.

Nico

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 9 Nov 2001 18:08:48 -0500

> denial-of-service attack may be more likely then.

Could a flag be added to ask the KDC if it supports the new
KerberosString, and have KDCs that support it include some token in the
responses indicating as much? Then the clients could cache whether the
KDCs support the new KerberosString and start sending requests
accordingly.

> difficult, but we can solve them later.

As long as there's a way to get UTF-8 in without breaking any old clients
and servers and KDCs and if consensus on that issue can be obtained
quickly, with tests to prove backwards compatibility, then it should go
in now, or in parallel (but soon). No?

Nico

---

From: Matt Crawford <[email-masked]>
Date: Fri, 09 Nov 2001 17:32:52 -0600

Flame gently if I'm wrong, but my vague grasp of DER tells me that

  if KerberosString is a choice of either GeneralString or
  SomeOtherString, and if a "new" client happens to encode the
  GeneralString choice, the result looks just like it would if
  KerberosString had never been invented.

Am I right?

In any case, I'm not crazy about feature tests that add more round
trips.  I have plenty of users at non-trivial RTTs from the KDCs.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 9 Nov 2001 18:44:52 -0500

> trips.  I have plenty of users at non-trivial RTTs from the KDCs.

I think round-trips can be avoided by: a) client-side caching, b)
configuration hints (all the KDCs in a given realm must support the new
KerberosString or no clients should use it for that realm).

Going to UTF-8 complicates life...

Nico

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 9 Nov 2001 19:56:47 -0500 (EST)

>   KerberosString had never been invented.

Yes, that is correct.  That property is critical to making this work in a
backwards-compatible way, as well as to handling several protocol
extensions not related to UTF-8 (which have been discussed before on this
mailing list and in face-to-face meetings).

---

From: Tom Yu <[email-masked]>
Date: 09 Nov 2001 19:56:15 -0500

>>>>> "crawdad" == Matt Crawford <[email-masked]> writes:

crawdad> Flame gently if I'm wrong, but my vague grasp of DER tells me that
crawdad>   if KerberosString is a choice of either GeneralString or
crawdad>   SomeOtherString, and if a "new" client happens to encode the
crawdad>   GeneralString choice, the result looks just like it would if
crawdad>   KerberosString had never been invented.

You are correct.  Changing occurrences of GeneralString to syntax such
as

KerberosString ::= CHOICE { general GeneralString, utf8 UTF8String }

or

KerberosString ::= CHOICE { general GeneralString, ... }

where the second is an extensible ChoiceType, will not change the
encoding under DER.  Essentially, if we have an type named
OriginalType, and we replace uses of it with

NewType ::= CHOICE { old OriginalType, new RevisedType }

as long as we choose the first alternative of NewType, the
encodings will be identical in DER.

Use of other encoding rules, e.g. PER, will cause the encoding to
change in this case, so the change we're proposing probably isn't
completely valid in the ASN.1 model of type extensibility.  It's
probably less of a perversion of the ASN.1 spec than some other things
that have been suggested, though.

---Tom

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Fri, 9 Nov 2001 20:05:46 -0500 (EST)

> KDCs support the new KerberosString and start sending requests

Well, the KDC can and must know whether each service can handle the new
type, since there is no other way it can find out.  And yes, once a client
discovers whether a particular KDC supports new messages, it can cache
that information.  However, it turns out that you only need such cached
information when you are sending an AS_REQ.  Once you have a ticket, you
know based on the types used in that ticket whether the KDC and/or service
support the new type.

> in now, or in parallel (but soon). No?

Soon, yes, definitely.  But not now.  As I said in a previous message,
attempting to get UTF-8 into kerberos revisions will delay the document
for quite some time -- while it looks easy on the surface, there are lots
of issues that need to be nailed down, and IMHO we simply cannot make
kerberos-revisions wait longer while we get it right.

---

From: Lawrence Greenfield <[email-masked]>
Date: Sat, 10 Nov 2001 09:38:48 -0500

Date: Fri, 9 Nov 2001 20:05:46 -0500 (EST)
   From: Jeffrey Hutzelman <[email-masked]>
   [...]
   Soon, yes, definitely.  But not now.  As I said in a previous message,
   attempting to get UTF-8 into kerberos revisions will delay the document
   for quite some time -- while it looks easy on the surface, there are lots
   of issues that need to be nailed down, and IMHO we simply cannot make
   kerberos-revisions wait longer while we get it right.

You've said this several times; can you list out the issues so we can
make our own judgement on how long they'll take?

It's fairly worthless to application protocols to accept UTF-8
usernames if the authentication system can't prove the identity of any
such users, therefore this is an important aspect.

Larry

---

From: Nicolas Williams <[email-masked]>
Date: Sat, 10 Nov 2001 09:59:58 -0500

> know based on the types used in that ticket whether the KDC and/or service

Yes. Also, there should be a rule that either all of the KDCs for a
given realm support the new KerberosString or none of them do. This way
clients need only cache per-realm realm-supports-new-KerberosString
state.

Also, SPNs acting as clients will know whether their realm's KDCs
support the new KerberosString a priori, much the same way as the KDCs
will know which SPNs support the new KerberosString. UPNs acting as
services... that's another matter... for that it might be more useful if
the KDCs tracked new KerberosString support on a per host or HostAddress
basis, rather than per-SPN...

Another interesting thing: whether or not old clients might run into
services or realms that have non-ASCII names will depend on the name
services and resolvers, and the client's ability to handle non-ASCII
input at those points. Of course, MS clients will be expecting
unescaped-UTF-8-in-GeneralString...

> kerberos-revisions wait longer while we get it right.

Fair enough.

Cheers,

Nico

---

From: [email-masked] (Johan Danielsson)
Date: 12 Nov 2001 11:32:45 +0100

Has there been any thought about adding new fields for this purpose?
For instance changing the PrincipalName (which is where utf8 is most
important) to:

PrincipalName ::= SEQUENCE {
        name-type[0]            INTEGER,
        name-string[1]          SEQUENCE OF GeneralString,
        name-string-utf8[2]     SEQUENCE OF UTF8String OPTIONAL
}

and then require that the old name-string still be something that is
valid ascii. Under the assumption that you drop extra unknown fields,
this change would be transparent to old clients, and have the benefit
that utf8-challenged terminals still can output something that can
make some kind of sense. Don't know about any security implications.

/Johan

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 12 Nov 2001 05:59:42 -0500

Interesting.

However, if I read the recent posts correctly, there's a desire to fix
other aspects of the protocol, such as enlarging the nonces.

At some point a non-backwards compatible change will be inevitable.

It would be nice if there were an easier way to introduce such changes.

It would be nice if new clients could send "version == 6" to version 5
KDCs and get an error answer back.

If there's any major protocol revision then I hope this version revving
problem is addressed. All that's needed is a: a statement that all
future Kerberos protocol versions' messages will start, as currently,
with a protocol version number AND a definition of an error message
corresponding to "requested protocol version not supported".

Nico

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 12 Nov 2001 10:11:36 -0500

> KDCs and get an error answer back.

Never mind. The thought occurs that an older KDC wouldn't know how to
respond securely to a newer request, so this would open up a DoS...

Nico

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 12 Nov 2001 11:36:07 -0500 (EST)

> make some kind of sense. Don't know about any security implications.

First, there are a few other places, most notably passwords, where UTF-8
support is important and unaddressed.  While passwords don't appear on the
wire, they are used to construct keys, so defining their representation is
important.  So, addressing the issue only for principal names would be
insufficient.

Second, the extension you propose won't work, because it's based on the
assumption that you "drop extra unknown fields".  Unfortunately, that's
not what the ASN.1 spec says.  In the case of an extensible sequence, you
generally _keep_ extra members even though you don't understand their
meaning, so that they can be preserved if the data is reencoded.  However,
PrincipalName is _not_ an extensible sequence.  The ASN.1 spec is a little
unclear on what to do when you see extra stuff in a non-extensible
sequence, but one valid (and common) behaviour is to fail the entire parse
and drop the message on the floor.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 12 Nov 2001 11:50:43 -0500 (EST)

> It would be nice if there were an easier way to introduce such changes.

Actually, Tom Yu has been working quite hard to write extensibility into
the new message formats, so that once we've changed message formats once
we won't have to do it again.  And several of us have been working on
coming up with ways to make it possible to do that first change in a
backward-compatible way.  Some of that work has been recapped lately in
this thread, in the form of discussions on how to determine if someone
supports UTF-8.  Most of the same problems and techniques for solving them
apply to the general problem as well.  But using them is going to
necessarily introduce some complexity. 

> KDCs and get an error answer back.

You've already touched on one reason this won't work -- the possibility
for an attacker to send a false error.  There is also another reason,
which has been discussed at pase IETF's both in and out of working group
meetings, but I'll recap i here.  You can't change the protocol version. 
It's buried too deeply in the protocol messages, such that in order to
parse enough of the message to get the version, you have to already know
which message format to parse (and thus what the version is).  This was a
bad design decision made in the early days of Kerberos 5 by someone who
didn't understand ASN.1 enough to see the implications.  Unfortunately, we
are stuck with it.  Changing the protocol version number is an
incompatible change.

---

From: [email-masked] (Johan Danielsson)
Date: 14 Nov 2001 12:00:29 +0100

Jeffrey Hutzelman <[email-masked]> writes:

> First, there are a few other places, most notably passwords, where
> UTF-8 support is important and unaddressed.

Yes, but this is mostly a user-interface issue. The KDC doesn't know
anything about passwords, as you said.

> Second, the extension you propose won't work, because it's based on
> the assumption that you "drop extra unknown fields".

Drop as in ignore, as in don't return an error if they're included.

I agree that this isn't a good solution, but it should be compatible
with old software, which anything using UTF8String direct is not. If
we really want to break the protocol, we should think more about
future extensibility.

/Johan

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 14 Nov 2001 11:49:20 -0500 (EST)

> anything about passwords, as you said.

Yes, but the Kerberos protocol must specify the encoding to be used for
password strings which are used as input to the string-to-key operation.
And it's actually _worse_ here than with principal names, because if the
wrong encoding is used, there is no way for the other party to reencode
the string.

> Drop as in ignore, as in don't return an error if they're included.

But that's not what happens.  Old software doesn't drop unknown fields
without returning an error.  In general, old software drops the entire PDU
if it contains unexpected fields.  ASN.1 sequences are not C structures;
you can't just add stuff to the end and expect existing parsers to ignore
it.  It's wrong, and it also doesn't work -- there are existing Kerberos
KDC implementations which WILL DROP such a message.

> we really want to break the protocol, we should think more about

Actually, we've thought a lot about future extensibility.  Tom's CHOICE
proposal allows complete interoperability with existing clients without
changing the encoding for old implementations.  New implementations will
be able to talk to each other using UTF-8 strings, once each makes the
determination that the other is new enough.  The means for making this
determination vary depending on the protocol message involved, but in most
cases it is possible to determie unambiguously whether a given party
supports UTF-8 by looking either at a message from that party or at a
message encoded or that party by someone who already knows whether they
support UTF-8 (for example, a Kerberos client can determine whether an
application server supports UTF-8 by examining the ticket, which was
encoded by the KDC using a priori knowledge of the application server's
capabilities).

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 14 Nov 2001 13:35:51 -0500

> wrong encoding is used, there is no way for the other party to reencode

Perhaps something like ETYPE_INFO could be added so an AS can tell a
client what encoding was used for the cleartext password of a long-term
key... That would be useful only during the transition to UTF-8 until
all password-derived-keys in a realm are changed to UTF-8 encoded
passwords. So such an extension probably wouldn't be very useful...

[...]

> encoded by the KDC using a priori knowledge of the application server's

The way a client determines whether a TGS or any application service
supports the new KerberosString is clear.

How does a client determine if an AS supports the new KerberosString?
This is unclear. I assume the client must either: a) know a priori, or
b) try an AS-REQ using the old KerberosString and then expect something
in the response which indicates that the KDC supports the new
KerberosString; what? A ticket extension, perhaps?

It has already been agreed that a client couldn't probe an AS using a
UTF-8String and expect an error back from the AS.

Cheers,

Nico