Subject: Extensibility and UTF-8: What about Interoperability?

From: Jeffrey Altman <[email-masked]>
Date: Sat, 8 Dec 2001 15:47:55 EST

Extensibility and UTF-8

We have a clear consensus from the working group that the current
revisions document is not going to specify how the future UTF-8
encoding will be defined.  This work is going to wait until after
revisions has gone to last call.  For the present time we need to
decide on the extensibility mechanism that will be used to support
UTF-8 in the future.  At some point in the future we will have defined
the UTF-8 encoding rules.  This implies that we have three different
phases of implementation:

  phase 1: rfc 1510 - knows nothing about extensibility.  GeneralString
  is being abused as octet string.  It is impossible to know which 
  character encoding is being used.

  phase 2: revisions - knows about the extensibility mechanism.  However,
  the official UTF-8 encoding has not yet been defined.  It is assumed
  the encoding mechanism used in Phase 1 continues to be used in Phase 2.

  phase 3: utf8  - knows about the extensibility mechanism and the official
  utf8 encoding.

Before going ahead and presenting possible implementations it is
important that we come to a consensus on two extremely important
questions:

1) Do we want interoperability at phase 2 between implementations that
   are using different encodings?  (For example, Windows with its UTF8
   encoding and MIT on Unix with a default locale of ISO Latin-1.)

2) When providing interoperability, do we want more round trips or fewer 
   round trips.

It is understood that it is impossible for any two phase 1
implementations to ensure interoperability.  This is because it is
impossible to determine which character encoding is being used within
the supposed GeneralString fields.

When the UTF8 encoding is defined (phase 3) we will have
interoperability as everyone will be using the same mechansim.  But
what about phase 2? Is this important to the working group?  As
currently discussed the replacement of GeneralString by KerberosString
where KerberosString is nothing more than a backward compatible CHOICE
between an abused GeneralString and an as yet undefined UTF8String
provides a means for avoiding the ASN.1 violations sometime in the
future but does nothing to resolve them for the immediate
implementation of Revisions.

   KerberosString       ::= CHOICE {
        GeneralString (IA5String),
        UTF8String
   }

Implementors of Revisions would not be able to use the UTF8String
because the appropriate encoding rules will not have been agreed to
yet.  Therefore, they must continue to abuse the backward compatible
and highly non-interoperable GeneralString.

The definition of KerberosString presents us with an opportunity.  We
can add a level of character-set interoperability to the
implementations of Revisions.

 There are many ways this can be done:

1) We can use error codes to indicate the encoding scheme that is understood
   by the receiver and expect the sender to perform the necessary conversions
   and retry

2) We can use the CHOICE to define the known character-sets and thereby allow
   the sender (when communicating with another Revisions implementation) to
   specify the encoding being used.  This allows the receiver to perform 
   the necessary conversions (if necessary) and reply with the appropriate
   character-set.

3) We can use SEQUENCES of character-set tagged OctetStrings to provide 
   alternatives to the receiver so that conversions do not need to be 
   performed.  

Which direction we choose is dependent upon the performance
characteristics we require for Revisions and extensibility.  Do we
prefer to provide extensibility at the cost of round trips or message
size?  Round trips can produce unnecessary delays and increase the
complexity of the information caching that must be performed.
Increased message size can result in packet fragmentation issues which
could lead to an inability to continue to use UDP with Kerberos.

I believe there is a strong consensus among the working group
participants that we must provide interoperability between
implementations.  That is why we are focusing on extensibility
mechanisms that allow the detection of the extensibility support to be
backward compatible with RFC1510.  However, it is unclear whether the
participants care or not about interoperability issues related to
character-set encoding.

I believe that if it is possible for us to provide character-set
interoperability we need to define it.  Both for Phase 2
implementations and between Phase 2 and Phase 3 implementations.
Revisions is going to legitimize

the use of character sets other than IA5.  Microsoft will continue to
use their UTF8 encoding of Unicode for their KDC; Western Europe will
continue to use ISO Latin-1 and Latin-9 (Euro aware) when running KDCs
on Unix; Eastern Europe will use ISO Latin-2 on its KDCs; and Windows
clients running MIT Kerberos for Windows will use Code Page 1252 or
1250 depending on the locale; Macintosh OS clients will use Macintosh
Latin on release 9 and earlier, and ISO Latin 1 on OS X.  I do not
know what the DCE implementations use for character-set encoding.
(Could someone please post that to the list?)

With all of these different character-sets and the reality that a
working group accepted UTF-8 encoding is at least a year away with
implementations supporting it even further away than that, it seems
clear to me that the interoperability requirements for character-sets
are something we need to deal with now.  When implementations that
support Revisions are released we need to be able to tell which
character-sets they are using in the messages.  Not just for
interoperability between Phase 2 implementations but so that Phase 3
implementations which will be Unicode (UTF8) based will understand how
to convert their strings when communicating with a Phase 2 peer.

Of course it is possible that the working group does not consider this
type of interoperability to be important.  In that case, a deployment
using non-IA5 strings will simply fail if the two peers are not using
the same character sets in their implementations and environment.
While this might be acceptable for Phase 2 to Phase 2 operations I do
not believe it is acceptable for Phase 2 to Phase 3 communications.

I would like us to come to consensus on this point.  If we can do that
in the next couple of days I will publish a follow-up letter
describing some possible ASN.1 representations of KerberosString which
will meet the level of interoperability we agree is required.

 Jeffrey Altman * Sr.Software Designer      C-Kermit 8.0 Beta available
 The Kermit Project @ Columbia University   includes Secure Telnet and FTP
 http://www.kermit-project.org/             using Kerberos, SRP, and 
 [email-masked]          OpenSSL.  SSH soon to follow.