Subject: ASN.1 and XER

From: Keith Burdis <[email-masked]>
Date: Fri, 10 Dec 1999 00:21:51 +0200

Hi there

  While reading the archives of this mailing list I found the following
  messages dicussing problems with ASN.1:

    "The worst thing about ASN.1 is that it's extremely hard to figure
    out what it means and where the hell it is actually documented.
    From what I gathered during a 2-hour WWW-odyssey is that there are
    different versions of the specification and that for each version
    there are several $$$-documents that comprise the specification.
    And it looks like one either needs a $$$-ASN.1 compiler or quite
    a lot of time to learn and implement a parser ..."
     
       -- Martin Rex - IETF CAT WG mailing list - Fri, 5 Mar 1999

    "True, but the point Martin raises is that it may be easier to get
    running code if we avoid the use of ASN.1 and use a raw form.
    Certainly our experience with Kerberos is that there were a lot of
    interoperability problems caused by the use of people misunderstanding
    ASN.1, and if the protocol data units are simple enough, it's not
    clear that ASN.1 adds sufficient value to be worth its cost in
    implementation complexity.  If we were doing Kerberos V5 all over
    again, we would *not* have used ASN.1.  It has caused us far more
    problems than it has ever saved us, and it is continueing to cause us
    problems even today as we discuss RFC 1510bis.
     
    I believe it would be appropriate to discuss whether we should take
    steps to make the standard more easily implementable, and eschewing
    the use of ASN.1 in favor of explicitly designed packet forms is one
    of the ways that we might make the standard much more successful."
      
       -- Theodore Y. Ts'o - IETF CAT WG mailing list - Fri, 5 Mar 1999

  I was glad to see that I'm not the only one who has had problems with
  ASN.1 :-) I too spent a marathon searching session trying to find out info
  on ASN.1. Thankfully I found RSA's Layman's guide which helped.
  Unfortunately this discussion appears to have stopped after these two
  messages.

  The main problem I've had with ASN.1 itself is understanding the difference
  between IMPLICIT and EXPILICIT tags. According to [1] the 1994 version of
  ASN.1 added something called "AUTOMATIC TAGS" to get rid of this confusion.
  Does anyone know anything about this and how it works?

  I think that the big hassle is with encoding and decoding the ASN.1 using
  BER and DER. This appears to be quite complex and you need special
  libraries to help you do this. Thankfully there are ASN.1 compilers around
  that help with this task by generating source code to do this, though some
  have restrictive licensing and others are incomplete. It's also hard to
  verify that something has been properly encoded since you just get binary
  output.

  Recently I came across the XML Encoding Rules (XER). A conceptual overview
  is given in [2], which summarises XER as:

    "The basic idea of XER (XML Encoding Rules) is to standardize rules that
    would allow for information described in ASN.1 (Abstract Syntax Notation)
    to be carried in XML (eXtensible Markup Language). An XER definition
    specifies the equivalence and necessary conversion between appropriate
    ASN.1 encoded data structures and XML encoded data structures."

  The output of a XER encoding is human-readable text which makes it easy to
  see and verify the data that was encoded. In addition XML tools can be used
  to process this data. [2] has this to say:

    "For developer communities making significant investments in XML, XER may
    leverage investments already made in ASN.1-based applications and thereby
    preserve the hard-won interoperability already achieved among such
    applications.

    For developer communities currently using ASN.1, XER may leverage
    investments being made in XML-based applications that are becoming
    widespread on the Internet."

  Since many of this working group's specifications use ASN.1 I'd like to see
  dicussion on the merits of using XER, or why it would be a bad idea.

  -------------------------------------------------------------------

  A good starting point for XER info is:

    http://asf.gils.net/xer/

  There's a document by Hoylen Sue entitled "XER - A Bridge between ASN.1 and
  XML" which discussed DSTC's implementation at:

    http://www.dstc.edu.au/Research/Projects/xer/xer-2.html

  -------------------------------------------------------------------

  [1] "XER Decisions and proposals", URL:
  http://asf.gils.net/xer/decisions.html

  [2] "XER (XML Encoding Rules) Concept", URL:
  http://asf.gils.net/xer/concept.html

  -------------------------------------------------------------------

    - Keith
-- 
Keith Burdis - MSc (Computer Science) - Rhodes University, South Africa
IRC: Panthras					JAPH 	QEFH
---
==========================================================================
This message was posted through the Stanford campus mailing list
server.  If you wish to unsubscribe from this mailing list, send the
message body of "unsubscribe ietf-cat-wg" to [email-masked]

---

From: Mike Eisler <[email-masked]>
Date: Thu, 9 Dec 1999 15:15:31 -0800 (PST)

Folks might also want to look at http://www.valicert.com/download/ and look
for the ASN.1 parser link. From what I gather (please correct me if I'm wrong
Jack), but Jack Kabat has written a parser that generates C code that invokes
SSLeay ASN.1 primitives to encode and decoe ASN.1. This should, in theory,
take much of the pain out of dealing with ASN.1 in C. 

>        -- Theodore Y. Ts'o - IETF CAT WG mailing list - Fri, 5 Mar 1999

A lot of the public key technology is specified with ASN.1, X.509 certificates
for example. GSS-API itself has tiny bit of ASN.1 compared to the ASN.1 that
many GSS-API plug ins might need. Thus I'm not sure if making GSS-API
eschew ASN.1 is a big savings.

Instead, if the combination of OpenSSL (www.openssl.org has SSLeay code) +
the ValiCert parser doesn't yield a workable approach, then I suggest that
interested folks find a way to fund a body like the Internet Software
Consortium to produce tools to make dealing with ASN.1 easier. 

	-mre

---

From: Tom Yu <[email-masked]>
Date: 09 Dec 1999 19:18:34 -0500

Oh no here we go again...

>>>>> "KB" == Keith Burdis <[email-masked]> writes:

KB> Hi there
KB>   While reading the archives of this mailing list I found the following
KB>   messages dicussing problems with ASN.1:

[snip]

KB>   I was glad to see that I'm not the only one who has had problems
KB>   with ASN.1 :-) I too spent a marathon searching session trying
KB>   to find out info on ASN.1. Thankfully I found RSA's Layman's
KB>   guide which helped.  Unfortunately this discussion appears to
KB>   have stopped after these two messages.

The RSA Layman's guide is a good start, but in my opinion glosses over
a number possible gotchas.  There is available for download a complete
book about ASN.1 by one of the major players in the ASN.1
standardization arena, John Larmouth:

http://www.nokalva.com/asn1/bookreg.html

I find it to be significantly more useful than the Layman's Guide, but
it is still no substitute for the actual standards documents.

Anyway, the problems I see with the ASN.1 standards in terms of
someone not connected to the ISO/ITU-T standards process are largely
due to the inaccessibility of authoritative copies of the
specification, including:

* The ASN.1 specification is a joint ITU-T/ISO effort, thus
  multiplying by two the number of documents out there.

* There have been multiple revisions of the documents, some of which
  are not completely compatible with each other, but these
  incompatibilities largely revolve around the deprecation of the
  macro notation in 1994 or so and replacing it with the "information
  object" specification.

* The ISO and ITU-T versions of the same documents have different
  "publication" dates due to procedural differences between the ISO
  and ITU-T.

* Not the least, both the ISO and the ITU-T want serious $$$ for
  copies of the spec.

KB>   The main problem I've had with ASN.1 itself is understanding the
KB>   difference between IMPLICIT and EXPILICIT tags. According to [1]
KB>   the 1994 version of ASN.1 added something called "AUTOMATIC
KB>   TAGS" to get rid of this confusion.  Does anyone know anything
KB>   about this and how it works?

The difference between IMPLICIT and EXPLICIT tags largely affects BER
based encodings.  If you use an IMPLICIT tag on a field of a
structure, you end up losing the intrinsic tag of the type that you're
tagging.

i.e. if you have a

    foo ::= SEQUENCE {
        bar [0] EXPLICIT INTEGER,
        baz [1] IMPLICIT INTEGER
    }

(using an explicit EXPLICIT to make things more clear) the first
(tagged with [0]) field will be encoded with the context-specific tag
0 followed by the complete encoding of an integer, including the tag
for the integer, while the second (tagged with [1]) field will be
encoded with the context-specific tag 1 followed by the encoding of
the integer without the tag for the integer.  I'm glossing over the
details of the encoding; I can expand upon this if people really want
to know the BER encoding rules of a TaggedType in ASN.1.

In any case, using EXPLICIT tags almost always bulks up the encoding
needlessly in BER and DER for very little actual gain in
functionality, with the possible exception of being able to use a
generic ASN.1 decoder to decode your protocol stream.

AUTOMATIC tagging, on the other hand, is a new (1994) invention to
make it easier to write ASN.1 specifications.  The upshot is that any
structure will get IMPLICIT tags inserted for each field by your ASN.1
compiler.  This results in somewhat less clutter in the description,
but may result in trouble if you're going to modify the fields in a
structure.  Nevermind that this will cause serious compatibility
problems unless you carefully design your protocol for version
changes, which should be done anyway.

KB>   I think that the big hassle is with encoding and decoding the
KB>   ASN.1 using BER and DER. This appears to be quite complex and
KB>   you need special libraries to help you do this. Thankfully there
KB>   are ASN.1 compilers around that help with this task by
KB>   generating source code to do this, though some have restrictive
KB>   licensing and others are incomplete. It's also hard to verify
KB>   that something has been properly encoded since you just get
KB>   binary output.

The complexity is there in DER, but it goes up by an order of
magnitude when you try to write an encoder that will handle all the
cases permitted by full-blown BER.  Unfortunately, there are existing
implementations of Kerberos 5 that use full-blown BER for their
encodings, and dealing with these have resulted in many compatibility
problems.  These compatibility problems largely stem from the
"constructed indefinite" encoding of structures that is permitted by
BER but forbidden by DER.

There are (somewhat) general ASN.1 decoders out there... I don't have
the details at the moment, but there is one included with the MIT krb5
source distribution.

I will look at the XER stuff and possibly comment on it later.

---Tom

---

From: Keith Burdis <[email-masked]>
Date: Fri, 10 Dec 1999 03:46:54 +0200

> take much of the pain out of dealing with ASN.1 in C. 

There are two Java ASN.1/DER compilers that I've used.

  Cryptix ASN.1 - http://www.cryptix.org/products/asn1/

  ARC - http://www.forge.com.au/

There's a also SNACC which generates C or C++ code to encode or decode BER:

  http://www.fokus.gmd.de/ovma/freeware/snacc/entry.html

Development seems to have stopped in 1997 though, and it doesn't appear to do
do DER.

> Consortium to produce tools to make dealing with ASN.1 easier. 

One of the advantages of using XER is that we can make use of XML tools to
help deal with ASN.1.

Libraries and tools are already available to handle XML, and more seem to be
being developed every day. A search on freshmeat for XML tools and libraries
found the libxml, LTXML, RXP, Cocoon, EZ/X, KBML, OpenXML, The Igor Project,
XML Parser for Java, GSLgen code generator and XML Security Suite. Some of
these are general enough to be used with XER if necessary.

Contrast this with the few tools that are available for working with BER or
DER.

  - Keith

---

From: Keith Burdis <[email-masked]>
Date: Fri, 10 Dec 1999 04:05:32 +0200

> Oh no here we go again...

I take it I missed a previous thread on this issue :-)
How far back in the archives do I have to go to find it?
 
> it is still no substitute for the actual standards documents.

Thanks. I'll take a look.

> structure, you end up losing the intrinsic tag of the type that you're

[SNIP explanation of difference between IMPLICIT and EXPILICIT tags]

Thanks. That makes a little more sense.
 
> changes, which should be done anyway.

So, would it break anything to change:

  DEFINITIONS EXPLICIT TAGS ::=
  BEGIN
      
    InitialContextToken ::= [APPLICATION 0] IMPLICIT SEQUENCE {
      thisMech OBJECT IDENTIFIER,
      innerContextToken ANY DEFINED BY thisMech
    }

  END

to:

  DEFINITIONS AUTOMATIC TAGS ::=
  BEGIN

    InitialContextToken ::= [APPLICATION 0] SEQUENCE {
      thisMech OBJECT IDENTIFIER,
      innerContextToken ANY DEFINED BY thisMech
    }

  END

using DER encoding?

> I will look at the XER stuff and possibly comment on it later.

Please do.

> ---Tom

Thanks.

  - Keith

---

From: Tom Yu <[email-masked]>
Date: 09 Dec 1999 23:33:31 -0500

>> Oh no here we go again...

KB> I take it I missed a previous thread on this issue :-)
KB> How far back in the archives do I have to go to find it?
 
Not sure.  It may be that I'm recalling lots of in-person flamage
about this subject.

KB> So, would it break anything to change:

KB>   DEFINITIONS EXPLICIT TAGS ::=
KB>   BEGIN
      
KB>     InitialContextToken ::= [APPLICATION 0] IMPLICIT SEQUENCE {
KB>       thisMech OBJECT IDENTIFIER,
KB>       innerContextToken ANY DEFINED BY thisMech
KB>     }

KB>   END

KB> to:

KB>   DEFINITIONS AUTOMATIC TAGS ::=
KB>   BEGIN

KB>     InitialContextToken ::= [APPLICATION 0] SEQUENCE {
KB>       thisMech OBJECT IDENTIFIER,
KB>       innerContextToken ANY DEFINED BY thisMech
KB>     }

KB>   END

KB> using DER encoding?

Yes, this will break things.  The "EXPLICIT TAGS" just means that a
type of the form "[0] INTEGER" will become "[0] EXPLICIT INTEGER",
which is actually the default if you don't specify the tagging for the
module.  What you've written will cause the second definition to
become:

    InitialContextToken ::= [APPLICATION 0] IMPLICIT SEQUENCE {
        thisMech [0] IMPLICIT OBJECT IDENTIFIER,
        innerContextToken [1] IMPLICIT ANY DEFINED BY thisMech
    }

Which has problems because:

* it's different from the first definition in terms of tagging

* contains an implicit tag on an open type, which is disallowed

The 1994 and 1997 revisions of the ASN.1 specification contain better
ways of dealing with open types, but they may not be compatible with
what we have already.  Basically the "ANY" and "ANY DEFINED BY" types
are deprecated...

---Tom

---

From: Keith Burdis <[email-masked]>
Date: Fri, 10 Dec 1999 13:29:00 +0200

> Not sure.  It may be that I'm recalling lots of in-person flamage

Ok, I'll don my flame-proof suit then ;-)

[snip]
> KB> using DER encoding?
[SNIP]
> what we have already.  Basically the "ANY" and "ANY DEFINED BY" types

Would it not be better to move towards using the latest version of the ASN.1
standard? These later revisions seem to have definite advantages. For
interests sake, what would the InitialContextToken definition look like in
1997 ASN.1?

You noted that protocols should be designed to handle version changes. Are
CAT protocols like Kerberos and SPKM so designed? (I know SPKM has a protocol
version number. Does this count?).
How do you design your ASN.1 so that it is future-resistant?

Note that XER does not distinguish between implicit and explicit tagging. It
treats everything as IMPLICIT ala AUTOMATIC TAGS.

Thanks.

  - Keith

---

From: [email-masked]
Date: Fri, 10 Dec 1999 14:41:34 -0500

Date: Fri, 10 Dec 1999 13:29:00 +0200
   From: Keith Burdis <[email-masked]>

   You noted that protocols should be designed to handle version
   changes. Are CAT protocols like Kerberos and SPKM so designed? (I
   know SPKM has a protocol version number. Does this count?).  

When Kerberos was designed, the people who were doing the design didn't
understand some of the subtle nuances of how to use ASN.1; which
basically goes to the heart of why I dislike ASN.1.  ASN.1 has a lot of
complexity to it, and so it gives you plenty of rope with which to hang
yourself (and other hapless users and implementors of your protocol)
with.  

The flameage of whether or not ASN.1 is a good thing or not basically
revolves around quantifying the costs and the benefits.  Advocates of
ASN.1 think the benefits are great, and downplay the costs, saying that
"someone who knows what they're doing won't have problems with ASN.1"
and "that's what ASN.1 compilers are for".   People who don't like ASN.1
counter that the costs are a lot higher (because arguably many people
don't know what they are doing, and because compilers often emit C
structures which no sane C programmer would be willing to use).  

Fortunately for Kerberos, a number of the extensibility traps were
avoided by using explicit tags everywhere; the main cost paid for that
V5 tickets are much, much bigger than V4 packets.

A generic "how not to get screwed by ASN.1" would be a wonderful
Informational RFC; if people must use ASN.1, warning them about some of
the traps is a good thing.  

						- Ted

---

From: Tom Yu <[email-masked]>
Date: 10 Dec 1999 21:56:43 -0500

>>>>> "KB" == Keith Burdis <[email-masked]> writes:

KB> Would it not be better to move towards using the latest version of
KB> the ASN.1 standard? These later revisions seem to have definite
KB> advantages. For interests sake, what would the InitialContextToken
KB> definition look like in 1997 ASN.1?

I'm not certain that you can properly define the RFC 2078
InitialContextToken compatibly with 1997 ASN.1, but here's a stab at
something close to it, using the "information object" notation, which
is what Appendix A of ITU-T X.680:1997 suggests as a replacement for
the ANY DEFINED BY type:

    GSS-API DEFINITIONS ::=
    BEGIN

    INITIAL-TOKEN-CLASS ::= CLASS {
        &mechType OBJECT IDENTIFIER UNIQUE
        &Type
    } WITH SYNTAX {&Type IDENTIFIED BY &mechType}

    -- This defines an object class INITIAL-TOKEN-CLASS

    InitialContextToken {INITIAL-TOKEN-CLASS:MECH-TOKEN-CLASS} ::=
        [APPLICATION 0] IMPLICIT SEQUENCE {
        thisMech MECH-TOKEN-CLASS.&mechType,
        innerContextToken MECH-TOKEN-CLASS.&Type
    }

    -- This makes use of the object class to define a parameterized
    -- type, with the dummy parameter MECH-TOKEN-CLASS.

    END

You might then define a mechanism as follows:

    Mech1 DEFINITIONS ::=
    BEGIN

    IMPORTS
    INITIAL-TOKEN-CLASS, InitialContextToken FROM GSS-API

    mech1ObjectIdentifier OBJECT IDENTIFIER ::= { -- OID here -- }

    Mech1InnerContextToken ::= SEQUENCE { -- etc. -- }

    mech1InitialTokenClass GSS-API.INITIAL-TOKEN-CLASS ::= {
        Mech1InnerContextToken IDENTIFIED BY mech1ObjectIdentifier
    }

    Mech1InitialContextToken ::=
        GSS-API.InitialContextToken { mech1InitialTokenClass }

    END

This is what I've come up with on a brief look through the ASN.1
specs; it may want more refinement, so nobody should go off and copy
it directly into the next revision of the GSS-API spec! :-) You
*could* use the TYPE-IDENTIFIER object class and the INSTANCE OF type
to define an InitialContextToken, but there's an extraneous "[0]
EXPLICIT" in that which we may not necessarily want.  I don't know
offhand why that extra tag is there in the definition of an
InstanceOfType... I should try to find out.

Note that many of the free ASN.1 tools haven't yet been updated to
handle 1994 or 1997 ASN.1 extensions such as those I've just used
above.  Also, the syntax that I've written above *does* have the
limitation that innerContextToken must be an ASN.1 type.  This means
that the current Kerberos mechanism can't be defined with the object
class stuff, as its innerContextToken isn't an ASN.1 type.  Of course
if you really want to use a non-ASN.1 type for your innerContextToken
you can just wrap it with something like an ExternalType.

KB> You noted that protocols should be designed to handle version
KB> changes. Are CAT protocols like Kerberos and SPKM so designed? (I
KB> know SPKM has a protocol version number. Does this count?).  How
KB> do you design your ASN.1 so that it is future-resistant?

Kerberos certainly isn't well designed with respect to extensibility,
largely because there are multiple types of exchanges or dialogs
within the Kerberos protocol, which makes version identification
rather tricky.  Also, there's the issue that there's no specification
of what to do when a party in a Kerberos exchange encounters something
it doesn't know how to deal with.

I will have to look at SPKM to determine if it actually is
"future-resistant".  Keep in mind that designing protocols to handle
version changes is very hard to do right.

---Tom

---

From: Tom Yu <[email-masked]>
Date: 10 Dec 1999 22:25:41 -0500

>>>>> "tytso" == tytso  <[email-masked]> writes:

tytso> When Kerberos was designed, the people who were doing the
tytso> design didn't understand some of the subtle nuances of how to
tytso> use ASN.1; which basically goes to the heart of why I dislike
tytso> ASN.1.  ASN.1 has a lot of complexity to it, and so it gives
tytso> you plenty of rope with which to hang yourself (and other
tytso> hapless users and implementors of your protocol) with.

I wonder whether the people doing the design of the Kerberos protocol
had access to a copy of the actual ASN.1 specs.  I suspect that they
did not, and relied on things such as the "Layman's Guide".  True,
there is a lot of complexity to ASN.1, as it is very obviously the
product of a design by committee.  Consider the dozen or so character
string types, all different and mostly all useless.  Many protocols
may benefit from design based on XDR or similar things if they do not
need the complexity that ASN.1 offers.

I'm currently neutral on whether ASN.1 was the right protocol
description language for Kerberos.  ASN.1 might be the correct
language to describe something such as SNMP though, especially with
the rich parameterization features that the 1994/1997 spec offer.

The "plenty of rope" largely arises from the effort to separate
abstract syntax from actual encoding.  This is very likely a big
mistake.  If protocol designers end up designing protocols that they
have no clue of the encodings of, it is quite possible that only at
the implementation stage will problems with the protocol be
discovered.  This is a way in which needless abstraction can hurt
badly.

tytso> The flameage of whether or not ASN.1 is a good thing or not
tytso> basically revolves around quantifying the costs and the
tytso> benefits.  Advocates of ASN.1 think the benefits are great, and
tytso> downplay the costs, saying that "someone who knows what they're
tytso> doing won't have problems with ASN.1" and "that's what ASN.1
tytso> compilers are for".  People who don't like ASN.1 counter that
tytso> the costs are a lot higher (because arguably many people don't
tytso> know what they are doing, and because compilers often emit C
tytso> structures which no sane C programmer would be willing to use).

I suspect that the expense of obtaining an authoritative copy of the
spec is doing a lot of damage.  How many IETF standards depend on ISO
or ITU-T standards that are not freely available?  I'd be interested
to find out.  It would be really nice if in the new spirit of
cooperation between the ITU-T and the IETF, that there would be some
provision to make available, free of charge, authoritative copies of
relevant ITU-T/ISO standards.  Of course, the ITU-T is quite unlikely
to go along with this idea...

Another problem with ASN.1 is the outright *lie* that you can write an
abstract syntax for the protocol without regard to how it will be
encoded.  That might be a nice theoretical dream, but actual
implementations as well as the design of the encoding rules basically
ensure that you *cannot* write an effective abstract syntax for a
protocol without careful consideration of how it will be encoded.

tytso> Fortunately for Kerberos, a number of the extensibility traps
tytso> were avoided by using explicit tags everywhere; the main cost
tytso> paid for that V5 tickets are much, much bigger than V4 packets.

I'm not convinced that explicit tagging helps extensibility all that
much if you don't have a well-defined set of behaviors for the parties
in a Kerberos exchange when they understand different revisions of the
protocol.  There are *no* provisions for negotiating certain aspects
fo the protocol.

tytso> A generic "how not to get screwed by ASN.1" would be a
tytso> wonderful Informational RFC; if people must use ASN.1, warning
tytso> them about some of the traps is a good thing.

I'm still contemplating writing such a thing in my Copious Free Time.
There's also Peter Gutmann's rant about how not to use ASN.1 in a
crypto protocol, which I don't have a current bookmark handy for.

---Tom