Subject: Re: APC-1711, 1712 review

From: John Segers <[email-masked]>
Date: Sat, 16 Oct 1999 14:33:10 -0400

All,

Last night I tried to post a reply to Abdallah Rayhan's message with
comments on APC-1711 and APC-1712, but it seems that it didn't get
through.  Here's a second try.

John
--
John Segers                                  email: [email-masked]
Lucent Technologies                                        Room HE 306
Dept. Forward Looking Work                      phone: +31 35 687 4724
P.O. Box 18, 1270 AA  Huizen                      fax: +31 35 687 5954

Subject: Re: APC-1711, 1712 review
From: John Segers <[email-masked]>
To: Abdallah Rayhan <[email-masked]>, [email-masked], [email-masked]
Date: Fri, 15 Oct 1999 20:19:39 -0400

Abdallah,

It's good to see that someone had a good look at APC-1711 and 1712.
Below you find some responses to your observations.

I'm x-posting to SG16 list, as the contributions will be discussed in
the Red Bank meeting.

John

Abdallah Rayhan wrote:
>
>
> I took a pass through APC-1711 and APC-1712 and I would like to post
> my analysis
> on the mailing list. I don't claim my analysis to be complete, however
> I tried to point
> out the main problems I came across while reading APC-1711 ABNF. My
> primary
> goal is the final ABNF and not the ASN.1 for two reasons. First, I am
> not an ASN.1
> expert, so I would leave critiquing it to the ASN.1 experts. Second,
> it would
> be a big distraction to the IETF having the MEGACO ABNF at an advanced
>
> stage with a set of testbenches (ANNEX G). Hence, I will try to focus
> on APC-1711 ABNF
> which I call ABNF1711 and MEGACO/H.248 ABNF which I call ABNF248.
>
> To start with I have noted that the ABNF1711 does not take the ABNF248
> as is
> but modifies it to suite ASN.1 encoding ( is ASN.1 so restrictive that
> it does not
> allow ABNF248 syntax ? )

In developing an ASN.1 to ABNF translation mechanism, you have to take
into account that the ABNF that is generated must be unambiguous.  One
of the dangers of omitting tags (field names) is introducing ambiguity,
for instance because a string value may happen to be the same as a tag.
If you do an ABNF spec by hand, you can verify that this does not lead
to ambiguity _for that ABNF specification_ but in a mechanical
translation process this is much harder.  That's why the mechanism
proposed in APC-1712 puts restrictions on the use of "AS ?"

>                          These discrepancies are found in: (I started
> noting
> the different  PRs but then found myself marking almost every one.)

Well, one grammar can be specified by different sets of PRs.  So
different sets of PRs I don't find worrying.  Differences in the grammar
is what's important.

>
> - Tagging every value.
> ABNF1711:
> PkgdName = LBRKT  PkgItemID [ PkgName ]  RBRKT
> PkgName = LBRKT  "/"  ( ( "PE" EQUAL Name )  / "*" ) RBRKT
> PkgItemID = ( ( "PI" EQUAL Name )  /   "*"  )
>
> ABNF248:
> pkgdName  = [ (PackageName | "*")  SLASH ] (ItemID | "*" )
> PackageName  = NAME
> ItemID = NAME
>
> You need "PE=" before package name and "PI=" before ItemID.
> These differences appear in many production rules.

Yes, this is true.  But I don't think it is a major change because the
structure isn't changed very much.

>
> - Extra brackets as illustrated in the previous example. In ABNF1711,
> there is a curly bracket before and after PkgdName while ABNF248 does
> not
> have that. This is all over ABNF1711.

There are indeed many instances where the generated ABNF has more
braces.  This is because braces are used to delimit sequences of
parameters, consistently throughout the entire syntax.  A packaged name
is a sequence of two parameters: package name and item name.  Hence it
is surrounded by braces.

>
> - Extensions do not appear any where in ABNF1711 while they are a
> major
> feature of ABNF248.

The ASN.1 has extension markers (the ellipses).  These are not
translated into ABNF, because the indicate that optional parameters may
be added.  If it is important, every extension marker in the ASN.1 of
APC-1711 can be preceded by a sequence to produce the extension fields
in the ABNF.

>
> -ABNF1711 changes the syntax of the MEGACO constructs.
>
> ABNF1711:
>  RequestedActions = LBRKT RequestedAction [ "E" EQUAL
> SecondEventsDescriptor ]
>     [ "SG" EQUAL SignalsDescriptor ]  [ "IC" ]   RBRKT
>
> ABNF248:
> requestedActions     = requestedAction LWSP [COLON LWSP
> embeddedSignalEvents ] [COLON LWSP InterceptToken ]
> embeddedSignalEvents = firstEmbedding [COMMA firstEmbedding]
>
> as you can see the first level of embedding (firstEmbedding) is
> chopped in ABNF1711.
> As a matter of fact, it changes the syntax into something different.

There's no more production for firstEmbedding, but in ABNF248, the
syntax continues:
firstEmbedding = (secondEvent | signalsDescriptor)
secondEvent = EventsDescriptorToken EQUAL [...]
signalsDescriptor = SignalsDescriptorToken [...]

Noting that the EventsDescriptorToken is an "E" and that the
SignalsDescriptorToken is the string "SG", you see that the differences
are very minor.  ABNF1711 does not use colons for separators, but I
think that's not important.

>
> -ABNF1711 enforces ordering of descriptors and parameters while the
> ABNF248
> does not. This issue was raised before and the resolution was made to
> enforce no order
> since there is no relationship between descriptors, e.g., modems or
> muxes, whichever comes
> first does not affect the other.

True.  This is not a problem of the approach per se, but my personal
stubborness.  ABNF1711 enforces ordering, but the syntax specifies that
every descriptor can only occurs once in a list, while ABNF248 puts that
in a comment (i.e., specifies it as a semantic constraint).  It's a
small operation to write ASN.1 that will produce ABNF that contains
"descriptor *(COMMA descriptor)" constructs instead of the
"[descr1][descr2][descr3]" constructs.

>
> - ABNF248 uses COMMA as a separator for commands, descriptors
> and parameters. This concept is gone in ABNF1711.

If we rewrite as explained above, you'll have your commas back.

>
> - ABNF1711 changes the syntax and the semantics of MEGACO, e.g.,
> ABNF1711:
> NotifyReply = LBRKT TerminationID  [ "ER" EQUAL ErrorDescriptor ]
> RBRKT
> You always have TerminationID regardless of whether it is NotifyReply
> error or not,
> and ErrorDescriptor is tagged.
>
> ABNF248
> notifyReply       = NotifyToken ( commandError | [EQUAL
> TerminationID])
> commandError = EQUAL TerminationID LBRKT errorDescriptor RBRKT
> TerminationID is optional in pure notify reply and mandatory in notify
> error.
> ErrorDescriptor is not tagged.

I explained about the tagging above, so let that go.  The optional
TerminationID presented a bit of a problem for the ASN.1.  Because I
couldn't for the life of me remember why it's important to be able to
leave TerminationIDs out in NotifyReply, I made them mandatory.

>
> Also, ABNF1711:
> SecurityParmIndex = ( 1( BASE64 ) BASE64END )
> SequenceNum = ( 1( BASE64 ) BASE64END )
> AuthData = ( 5( BASE64 ) BASE64END )
> BASE64    = 4( BASE64CHAR )
> BASE64END  = 4( BASE64CHAR ) / ( 3( BASE64CHAR ) "?" ) / ( 2(
> BASE64CHAR ) "??" )
> Where  did "?" and "??" come from ?

They ensure that the number of characters in the BASE64 encoded octet
string is a multiple of 4.  I was told that BASE64 works that way.

>
> ABNF248:
> SecurityParmIndex    = 8(HEXDIG)
> SequenceNum          = 8(HEXDIG)
> AuthData             = 32*64(HEXDIG)
>
> These are some of the differences. If we review each PR, no PR in
> ABNF1711
> will be saved. So I will stop here.

Once again, the PRs aren't the most important, the differences in
grammar are.

>
> I dont know if the kind of constraints ABNF1711 enforces on the ABNF
> are due to
> to the ASN.1 encoding constraints or because of ASN.12ABNF.
> Whatever the reasoning is, APC1711 should focus on completing the
> ASN.1
> description and generate similar testbenches as in the MEGACO I-D
> either
> binary or text, or both. The binary will be definitely different than
> the text because of
> tagging, however the text should be same as the test benches written
> so far.
> In theory, bin and text should reflect the same protocol and should
> introduce the least
> entropy into the syntax.
>
> My recommendation is that rather than writing ASN.12ABNF tool, APC1711
>
> should write examples that use the ASN.1 and bin2text and text2bin
> tools in order to
> test the ASN.1 encoding. Once that is out of the way and we know that
> the ASN.1 is ok,
> we could look into tools like ASN.12ABNF presuming that it would
> generate ABNF
> that is equivalent to ABNF248 syntactical and semantical wise.
> Otherwise, ABNF1711
> approach is unproductive and waste of effort trying to verify every PR
> of ABNF1711.

I agree that examples are useful, but this comment is counter
productive.

>
> In addition to that, if we assume that the ASN.1 is ok and the
> ASN.12ABNF is
> ok, from what I saw both are not,

Well, if you define ABNF248 as being right and anything that's different
as being wrong, then obviously your right, right?  Otherwise, your're
wrong, right?

As far as I can tell, the ASN.1 is ok, but it definitely needs review.
I'm glad you took the effort to be the first to comment. And I'm glad
that you didn't find any semantic errors (barring the mandatory
TerminationID, which I don't consider an error but a difference).

>                                   is there any guarantee that when
> they change
> the ASN.1, that will not introduce problems into the ABNF ? My feeling
> is that
> it would be a source of confusion and we would need to review the ABNF
> and
> probably hand craft some PRs to make it consistent and ambiguous free.
>
> If that is that is the case, why should we throw out ABNF248 which
> we have tested and have a high degree of confidence in. We are very
> familiar
> with ABNF248; we know the ins and outs of it; and we know how to fix
> any problem
> without disturbing the rest of the grammar. How would ABNF1711 do that
> ?

Using the mechanical translation mechanism gives me a higher degree of
confidence.  I don't have to worry anymore about having accidentally
omitted a closing brace, or being inconsistent with the use of
separators.

>
> As for ASN.1, it is not compatible with the standard. This means that
> ASN.1 compilers
> can not produce a correct code.

APC-1712 addresses this.  The extensions that were added to deal with
the different types of separators and brackets in ABNF248 can be put
into special comments in the ASN.1.  They take the form "--& <extension>
--"  A tool that does the text encoding as described, recognizes these
comments, a normal ASN.1 compiler sees them as comments, ignores them
and produces code (correctness is up to the compiler writer :-).

> regards
> Abdallah Rayhan

--
John Segers                                  email: [email-masked]
Lucent Technologies                                        Room HE 306
Dept. Forward Looking Work                      phone: +31 35 687 4724
P.O. Box 18, 1270 AA  Huizen                      fax: +31 35 687 5954

---

From: Pete Cordell <[email-masked]>
Date: Sun, 17 Oct 1999 10:30:48 +0100

I've attached a few comments below to clarify some of the issues raised by Abdallah...
 
Pete

=============================================
Pete Cordell
[email-masked]
=============================================

    -----Original Message-----
    From: Abdallah Rayhan <[email-masked]>
    To: [email-masked] <[email-masked]>
    Date: 14 October 1999 16:05
    Subject: APC-1711, 1712 review

    I took a pass through APC-1711 and APC-1712 and I would like to post my analysis 
    on the mailing list. I don't claim my analysis to be complete, however I tried to point 
    out the main problems I came across while reading APC-1711 ABNF. My primary 
    goal is the final ABNF and not the ASN.1 for two reasons. First, I am not an ASN.1 
    expert, so I would leave critiquing it to the ASN.1 experts. Second, it would 
    be a big distraction to the IETF having the MEGACO ABNF at an advanced 
    stage with a set of testbenches (ANNEX G). Hence, I will try to focus on APC-1711 ABNF 
    which I call ABNF1711 and MEGACO/H.248 ABNF which I call ABNF248. 
    Just to reassure you, the level of ASN.1 understanding required to work with the definition in 1711 is fairly easy to acquire.  I would say slightly easier then ABNF, but then we're talking about fairly straight forward things in both cases so the differences are not really significant.  I've attempted to put a bit of a tutorial about ASN.1 in APC-1712.  I'm happy to expand this if it is felt to be desirable (I didn't want to go too far straight away in case the editors jumped on me!).  In short, if you know C, SEQUENCE == struct, CHOICE == union (except type in union is also stored), and each parameter is "name type" rather than "type name".  Each parameter can also be optional, in which case OPTIONAL appears at the end, and/or can appear multiple times, in which case the format is "name SEQUENCE OF type".  Knowing that you should be able to pick up the rest by looking at 1711.  (I for one learnt ASN.1 just by looking at examples of it.) 
    To start with I have noted that the ABNF1711 does not take the ABNF248 as is 
    but modifies it to suite ASN.1 encoding ( is ASN.1 so restrictive that it does not 
    allow ABNF248 syntax ? ) These discrepancies are found in: (I started noting 
    the different  PRs but then found myself marking almost every one.) 

    - Tagging every value. 
    ABNF1711: 

    This is one construct the ASN.1 definition has a lot of trouble with.  One proposal is just to make this an IA5String (in ASN.1 speak).  It can then be coded in the same way as ABNF1711.

    ABNF248: 

    A similar comment applies to this also, i.e. making IA5String removes the problem.

    - Extra brackets as illustrated in the previous example. In ABNF1711, 
    there is a curly bracket before and after PkgdName while ABNF248 does not 
    have that. This is all over ABNF1711. 

    The IA5String would also remove the need for the curly-wurlies around this.

    - Extensions do not appear any where in ABNF1711 while they are a major 
    feature of ABNF248. 

    Right, extensions to the ASN.1 are implicit.  By that I mean that, every SEQUENCE and CHOICE can be extended.  That's way there are a few more braces and there are restrictions on what is untagged.  The encoding rules philosophy is that it should be possible to skip any parameter, and then re-start parsing the next parameter without losing parameters, and without interpreting tags as values and so on.

    ABNF1711: 
     RequestedActions = LBRKT RequestedAction [ "E" EQUAL SecondEventsDescriptor ] 
        [ "SG" EQUAL SignalsDescriptor ]  [ "IC" ]   RBRKT 

    ABNF248: 
    requestedActions     = requestedAction LWSP [COLON LWSP  embeddedSignalEvents ] [COLON LWSP InterceptToken ] 
    embeddedSignalEvents = firstEmbedding [COMMA firstEmbedding] 

    as you can see the first level of embedding (firstEmbedding) is chopped in ABNF1711. 
    As a matter of fact, it changes the syntax into something different. 

    -ABNF1711 enforces ordering of descriptors and parameters while the ABNF248 
    does not. This issue was raised before and the resolution was made to enforce no order 
    since there is no relationship between descriptors, e.g., modems or muxes, whichever comes 
    first does not affect the other. 

    I think I know what your referring to here, but I can't find it now that I look for it.  But basically, the order of tagged parameters after the last untagged parameter is not important.  It's a limitation of my ABNF skills that prevented me from being able to express both the multiplicity (if that's the right word for how many times a thing appears) and also say that it could appear in any order.

    - ABNF1711 changes the syntax and the semantics of MEGACO, e.g., 
    ABNF1711: 
    NotifyReply = LBRKT TerminationID  [ "ER" EQUAL ErrorDescriptor ]  RBRKT 
    You always have TerminationID regardless of whether it is NotifyReply error or not, 
    and ErrorDescriptor is tagged. 

    ABNF248 
    notifyReply       = NotifyToken ( commandError | [EQUAL TerminationID]) 
    commandError = EQUAL TerminationID LBRKT errorDescriptor RBRKT 
    TerminationID is optional in pure notify reply and mandatory in notify error. 
    ErrorDescriptor is not tagged. 

    Also, ABNF1711: 
    BASE64END  = 4( BASE64CHAR ) / ( 3( BASE64CHAR ) "?" ) / ( 2( BASE64CHAR ) "??" ) 

    John has put the wrong modifier type for this OCTET STRING.  It should be HEX OCTET STRING, which would result in the use of the two hex values per octet representation.  

    I don't think base64 is used in megaco, but the reason for changing the base64 padding character is using = is ambiguous.  It's not clear in the case when parsing parameters that have been added after the definition that the parser knows about whether the = are separating a tag from a value, or part of the value.  A simple way to resolve this is to change the padding character.  I think this is been done in some other protocol (can't remember which) and it's not difficult to do.  Even if you have a black box base64 encoder you can always fix it up later by doing if( last-char == '=' ) last-char = '?' and same for the next to last.

    ABNF248: 

    If the correct modifier is used as described above, the revised form of this becomes 

    SP = 4 OCTET

    SN = 4 OCTET

    AD = 16*32( OCTET )

    OCTET = ( HEXDIG HEXDIG )

    These are some of the differences. If we review each PR, no PR in ABNF1711 
    will be saved. So I will stop here. 

    I don't know if the kind of constraints ABNF1711 enforces on the ABNF are due to 
    to the ASN.1 encoding constraints or because of ASN.12ABNF. 
    Whatever the reasoning is, APC1711 should focus on completing the ASN.1 
    description and generate similar testbenches as in the MEGACO I-D either 
    binary or text, or both. The binary will be definitely different than the text because of 
    tagging, however the text should be same as the test benches written so far. 
    In theory, bin and text should reflect the same protocol and should introduce the least 
    entropy into the syntax. 

    I agree with you that at the moment we have a serious issue that megaco might become two protocols; one binary, one text.  I can also see that they may well diverge if we are not careful.  We probably differ on the best way to prevent that though :).

    My recommendation is that rather than writing ASN.12ABNF tool, APC1711 
    should write examples that use the ASN.1 and bin2text and text2bin tools in order to 
    test the ASN.1 encoding. Once that is out of the way and we know that the ASN.1 is ok, 
    we could look into tools like ASN.12ABNF presuming that it would generate ABNF 
    that is equivalent to ABNF248 syntactical and semantical wise. Otherwise, ABNF1711 
    approach is unproductive and waste of effort trying to verify every PR of ABNF1711. 

    I'm not sure how you see the bin2text and text2bin working.  Perhaps you could explain this further.  

    In addition to that, if we assume that the ASN.1 is ok and the ASN.12ABNF is 
    ok, from what I saw both are not, is there any guarantee that when they change 
    the ASN.1, that will not introduce problems into the ABNF ? My feeling is that 
    it would be a source of confusion and we would need to review the ABNF and 
    probably hand craft some PRs to make it consistent and ambiguous free. 
    If that is that is the case, why should we throw out ABNF248 which 
    we have tested and have a high degree of confidence in. We are very familiar 
    with ABNF248; we know the ins and outs of it; and we know how to fix any problem 
    without disturbing the rest of the grammar. How would ABNF1711 do that ? 

    One reason the ASN.1 definition is having a tough time matching the ABNF is because its fundamental to the way it works to be consistent and avoid ambiguity.  By defining how parameters are encoded in a consistent way, and ensuring that that encoding allows them to be readily skipped when they are not understood means there's less (hopefully no) chance that you end up with something that's ambiguous, especially when a v1 implementation parses a v2 message etc. Fortunately, ABNF248 is already pretty consistent which is way its not an impossible task for the ASN.1.

    As for ASN.1, it is not compatible with the standard. This means that ASN.1 compilers 
    can not produce a correct code. 

    regards 
    Abdallah Rayhan 

    My over all feeling is that a lot of effort has been put in to the ASN.1 close to the ABNF.  I'll admit it doesn't result in exactly the same ABNF as 248, but I hope that we can still be part of the consensus forming process that defines the ABNF if the functionality is not changed.  Or put another way, I hope we are not precluded from being able to suggest changes to the ABNF because we are also trying to describe it in ASN.1.  I'm also aware that this approach was adopted to get us out of the text vs. binary black hole.  If this initiative fails then we are back to that, something that I would expect that none of us would want to go back to.  So let's work together and try and sort this out once and for all.