Subject: RE: UTF-8, ASN.1, and our past sins

From: "Paul Leach" <[email-masked]>
Date: Tue, 20 Nov 2001 13:21:33 -0800

> -----Original Message-----
> From: Tom Yu [mailto:[email-masked]] 
> Sent: Tuesday, November 20, 2001 11:13 AM
> To: Paul Leach
> Cc: Nicolas Williams; Jeffrey Hutzelman; Johan Danielsson; 
> [email-masked]
> Subject: Re: UTF-8, ASN.1, and our past sins
> 
> 
> >>>>> "Paul" == Paul Leach <[email-masked]> writes:
> 
> Paul> However, if appears to me that some parts of the latest 
> section 5 
> Paul> change or come very close to changing the semantics of 
> ASN.1. If 
> Paul> existing implementations really obeyed 100% pure ASN.1, 
> would we 
> Paul> need all of these?
> 
> Existing implementations violate ASN.1 in various ways.  We 
> are changing the protocol specification's use of ASN.1 in 
> small and mostly compatible ways to ensure that these 
> implementations will no longer be in violation.

I'm afraid I don't understand this. If existing correct implementations
of Kerberos violate ASN.1, and they implement the Kerberos protocol
correctly, it would seem to follow that the existing protocol can't be
described in correct ASN.1. And I don't see how changes to the spec
could fix existing implementations to cause them to not be in violation
of ASN.1. Are you saying that they are compatible with _some_ protocol,
different than RFC 1510, that can be described in correct ASN.1, and
that this protocol is what is being defined in 1510bis? In that case,
though, wouldn't be appropriate to say that the existing implementations
violate ASN.1, but rather that they violated RFC 1510? And if so, then
they wouldn't be correct implementations, would they? But assumption was
that they were correct; proof by contradiction.

> We are also 
> adding reasonable constraints on things such as integer 
> values so that implementations will not be required to, for 
> instance, handle arbitrarily large integers.

To me, that seems to say that Kerberos changes the definition of
"Integer" in ASN.1, but in such way that still lets them use ASN.1
parsers. Admittedly, that's perhaps just a small change, but I think its
still a change. Ditto on the tag numbers. RFC 1510 implicitly (as
defined by correct implementations) uses a variant of ASN.1 where tags
can't be greater than 30.

Paul

---

From: Tom Yu <[email-masked]>
Date: 20 Nov 2001 17:19:34 -0500

>> implementations will no longer be in violation.

Paul> I'm afraid I don't understand this. If existing correct
Paul> implementations of Kerberos violate ASN.1, and they implement
Paul> the Kerberos protocol correctly, it would seem to follow that
Paul> the existing protocol can't be described in correct ASN.1.

The main example of this would be the BIT STRING usage for various
flag words.  Using NamedBits notation for the flags requires that when
encoding using DER, we truncate trailing zero bits.  Implementations
made the (quite understandable) mistake of unconditionally sending 32
bits.

Paul> And I don't see how changes to the spec could fix existing
Paul> implementations to cause them to not be in violation of ASN.1.

The BIT STRING change fixes the kerberos spec so that existing
implementations are no longer in violation of the combination of ASN.1
and the krb-revisions document.

Paul> Are you saying that they are compatible with _some_ protocol,
Paul> different than RFC 1510, that can be described in correct ASN.1,
Paul> and that this protocol is what is being defined in 1510bis?

The protocol they implement is different from what RFC 1510 specifies,
but if you ask whether the implementation emits a correct DER ASN.1
encoding of RFC 1510, it does not.  The resultant encoding can be
defined using correct ASN.1, but it is not the exactly the same
protocol as RFC 1510.  You could also take the view that the
implementation emits a correct DER ASN.1 encoding of a protocol that
is slightly different from RFC 1510.

Paul> In that case, though, wouldn't be appropriate to say that the
Paul> existing implementations violate ASN.1, but rather that they
Paul> violated RFC 1510?

No.  They in fact violated ASN.1, in particular DER, by not truncating
trailing zero bits from BIT STRING encodings that were specified using
NamedBits notation.  The changes in krb-revisions to remove NamedBits
notation and unconditionally send at least 32 bits of flags mean that
the existing implementations will no longer violate DER.

Paul> And if so, then they wouldn't be correct implementations, would
Paul> they? But assumption was that they were correct; proof by
Paul> contradiction.

I do not recall assuming that existing implementations of RFC 1510
were strictly correct with regards to ASN.1 encoding.  It is entirely
possible that I said it in error, or that somehow you interpreted
something I said as implying that.

>> instance, handle arbitrarily large integers.

Paul> To me, that seems to say that Kerberos changes the definition of
Paul> "Integer" in ASN.1, but in such way that still lets them use
Paul> ASN.1 parsers.

I challenge anyone to show me an off-the-shelf ASN.1 implementation
that allows for arbitrarily large encodings of the ASN.1 INTEGER type
(particularly those large than native to the host CPU), and actually
makes use of it whenever an unconstrained INTEGER appears in an ASN.1
type.  RFC 1510 did not constrain uses of INTEGER, so implementations
in theory would have been required to accept arbitrarily large encoded
integers.  (Note, though, that DER adds some effective limits on the
size of an integer encoding, though they're so large as to be mostly
irrelevant.)

Paul> Admittedly, that's perhaps just a small change, but I think its
Paul> still a change.

I agree that it is a change from RFC 1510 to krb-revisions.  It is a
change in the relevant types that use it.  It's a low-risk change,
since I believe that very few implementations of RFC 1510 would
actually emit encodings of integers larger than 32 bits.

Paul> Ditto on the tag numbers. RFC 1510 implicitly (as defined by
Paul> correct implementations) uses a variant of ASN.1 where tags
Paul> can't be greater than 30.

RFC 1510 did not specify exception behavior at all.  The text I wrote
in section 5 merely cautions implementors to gracefully handle
(probably rejecting) tags greater than 30.  This is merely
implementation guidance, not a modification of the ASN.1 standard.

In any case, I believe that we can produce a specification that:

* is correct ASN.1 syntax

* does not require deviations from correct DER

* is compatible with existing implementations of RFC 1510

---Tom