Subject: [abnf-discuss] constrained-01 - advantage?

From: Doug Royer <[email-masked]>
Date: Sun, 13 Nov 2016 14:38:10 -0700

It has been over 30 years since I have taken any math classes. The 
introduction sections were a bit difficult to follow. I would suggest 
simplification.

   ---------------------------------------------------------

How is:

  resent-field    = "Resent-" field-name ":" unstructured CRLF

  resent-date   ^ resent-field = "Resent-Date:" date-time CRLF

  resent-from   ^ resent-field = "Resent-From:" mailbox-list CRLF

  resent-sender ^ resent-field = "Resent-Sender:" mailbox CRLF

  ...

Simpler or more of an advantage than:

  resent-field  = resent-unstructured / resent-date / resent-from / 
resent-sender / ...

  resent-unstructured = "Resent-" field-name ":" unstructured CRLF

  resent-date         = "Resent-Date:" date-time CRLF

  resent-from         = "Resent-From:" mailbox-list CRLF

  resent-sender       = "Resent-Sender:" mailbox CRLF

  ...

-- 

Doug Royer - (http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Sean Leonard <[email-masked]>
Date: Mon, 14 Nov 2016 08:21:08 +0900

> On Nov 14, 2016, at 6:38 AM, Doug Royer <[email-masked]> wrote:
> 
> 
> It has been over 30 years since I have taken any math classes. The introduction sections were a bit difficult to follow. I would suggest simplification.

Okay; thanks for reading! Yes, there is some parsing / graph theory in there...

> resent-field  = resent-unstructured / resent-date / resent-from / resent-sender / ...
> 
> …

Is simpler than:

> resent-field  = resent-unstructured / resent-date / resent-from / resent-sender / ...

First of all, it looks simpler, because it is.

Second of all, <resent-date>, <resent-from>, <resent-sender> etc. are all “subsumed” in <resent-unstructured>, leading to an intentionally ambiguous grammar. The problem with the existing way of writing things out, is that the former are all superfluous when the latter is present. Another way to write it tends to be:

 field = date / from / sender / … / extension-field

 extension-field = field-name ":" unstructured CRLF

Well it makes perfect sense when you read it, of course, but when a computer / parser parses productions, every <date> is going to match with <extension-field> as well. The point being <extension-field> is not really doing what you think it’s doing in ABNF: it’s really <any-and-every-field>.

There is nothing “wrong” with creating an ambiguous grammar, but it can lead to subtle parsing errors if you are not careful. Usually ambiguous grammars are undesirable because you want your code to go down one, and only one, path. An automated ABNF validator should be able to detect ambiguous grammars and inform you about that.

When you actually go to program what the ABNF represents, most people are not going to program:

if (chars[0] = ‘d’ && chars[1] = ‘a’ & chars[2] = ’t’ & chars[3] = ‘e’ & chars[4] = ‘:’) {
  // parse as date
} else if (chars[0] = ‘f’ …) {
  // parse as from
} else if (…) {
  // ...
} else {
  field-name = everything before ‘:’;
  unstructured = everything between ‘:’ and CRLF;
}

Instead, most people are going to want a table-based parsing strategy, namely:

field-name = everything before ‘:’;
unstructured = everything between ‘:’ and CRLF;

switch (field-name) {
case ‘from’:
  // parse unstructured as mailbox-list
case ‘date’:
  // parse unstructured as date-time
…
default:
  // store and propagate, don’t do special things
}

Where the switch is generally based on some kind of efficient table, appropriate to the programming language.

And speaking of tables, usually these sorts of things are backed by a registry, like an IANA registry, of enumerated values.

Using Constrained ABNF helps to generate the said table, and validate that said table matches the registered set of enumerated values.

Third of all, writing:

> resent-field  = resent-unstructured / resent-date / resent-from / resent-sender / ...

Makes it look like certain productions (initial fields) are the “important” ones, while subsequent fields are “not as important”. But subsequent fields can be just as important. The point is that they are enumerated elsewhere in the spec, or (more likely) in subsequent RFCs.

The relationship between a generic production and its specific instances could be notated in many other ways, such as:

resent-field    = "Resent-" {{field-name ":"}} unstructured CRLF
                            + "Date:" date-time CRLF
                            + "From:" mailbox-list CRLF
                            + "Sender:" mailbox CRLF
                            ...

but when we chose the proposed constraint syntax in the draft, we tried to be as conservative as possible with introducing new symbols and positional elements. (I have to dig out old e-mails, but we did take a look at a variety of syntaxes.)

Best regards,

Sean

---

From: Doug Royer <[email-masked]>
Date: Sun, 13 Nov 2016 20:26:33 -0700

On 11/13/2016 04:21 PM, Sean Leonard wrote:

> Well it makes perfect sense when you read it, of course,
 > but when a computer / parser parses productions, every <date> is going
 > to match with <extension-field> as well. The point being
 > <extension-field> is not really doing what you think it’s doing in ABNF:
 > it’s really <any-and-every-field>.
 > ...
> There is nothing “wrong” with creating an ambiguous grammar,
 > but it can lead to subtle parsing errors if you are not careful.
 > Usually ambiguous grammars are undesirable because you want your
 > code to go down one, and only one, path. An automated ABNF validator
 > should be able to detect ambiguous grammars and inform you about that.

If your talking about replacing ABNF with new syntax. then I would 
agree. To a person reading the ABNF, they mean the same thing.
At no point is one path more important than another.

Unless you deprecate the old method, its not going to help.
Because people will use which ever makes sense to them.

Is the goal of this draft to make all future ABNF in RFC's parsable by a 
computer program? To mandate it?  I did not understand that after 
reading the draft.

If this keeps going, it will be a new ASN.1

--

---

From: Julian Reschke <[email-masked]>
Date: Mon, 14 Nov 2016 04:39:32 +0100

>   unstructured = everything between ‘:’ and CRLF;

If this is the only use case (is it?), then I'd say that you're dealing 
with a problem you could avoid in the first place.

See how RFC 7230 distinguishes between parsing the HTTP message info 
fields and field values, and then has completely *separate* ABNFs for 
each field value.

Best regards, Julian

---

From: Sean Leonard <[email-masked]>
Date: Mon, 14 Nov 2016 13:45:44 +0900

> On Nov 14, 2016, at 12:26 PM, Doug Royer <[email-masked]> wrote:
> 
> If your talking about replacing ABNF with new syntax. then I would agree. To a person reading the ABNF, they mean the same thing.

I suppose that deprecating the old method would be something on the table to discuss. I believe it is desirable to avoid ambiguous grammars. At the same time, a sufficiently sophisticated tool could be written to identify these sorts of cases (of intentionally ambiguous grammars for parameterized names and values).

> Is the goal of this draft to make all future ABNF in RFC's parsable by a computer program? To mandate it?  I did not understand that after reading the draft.
> 

We don’t want that. But the idea of “Table Constraints” is an ASN.1 term. (It is also a SQL concept, because, surprise, it’s about constraining data to tables of values.)

One thing is that none of the drafts that are being proposed change the context-free nature of ABNF.

I would say that a goal is that ABNF in RFCs should be easy to validate, and this would include by means of computer programs. This increases document quality. I recognize that there is a spectrum of views around here about how “formal” ABNF ought to be.

Regards,

Sean

---

From: Sean Leonard <[email-masked]>
Date: Mon, 14 Nov 2016 13:55:32 +0900

> On Nov 14, 2016, at 12:39 PM, Julian Reschke <[email-masked]> wrote:
> 
> If this is the only use case (is it?), then I'd say that you're dealing with a problem you could avoid in the first place.
> 
> See how RFC 7230 distinguishes between parsing the HTTP message info fields and field values, and then has completely *separate* ABNFs for each field value.

Yes, I saw that. It is a difference approach to the “mail standards” approach, which uses a lot of / and =/ and extension-header syntaxes.

A disadvantage of the RFC 7230 approach is that the relationship between the generic header production and specific headers is not formalized. For example, 7230 says:

   HTTP-message = start-line *( header-field CRLF ) CRLF [ message-body
    ]

   header-field = field-name ":" OWS field-value OWS

   Via = *( "," OWS ) ( received-protocol RWS received-by [ RWS comment
    ] ) *( OWS "," [ OWS ( received-protocol RWS received-by [ RWS
    comment ] ) ] )

All that is very interesting, but if you want to verify that the ABNF is correct (or if you want to verify that sample HTTP messages conform to the ABNF), you have to do extra steps to extract each header-field and match them to the particular field values.

It is desirable to express the relationship between the <Via> production, and the <field-name> “Via”. Specifically, when <field-name> is “Via”, then <field-value> is <Via>.

A recent place where this leads to a protocol problem is the PKCS #11 URI scheme. RFC 7512 says that “|” can appear in <pk11-query-res-avail>, as a delimiter to some command-path. But “|” is not a part of URIs under [RFC3986]. Therefore, any [RFC3986] conforming URI parser is going to reject what RFC 7512 says are valid pkcs11: URIs. If the relationship between URI@[RFC3986] and pk11-URI@[RFC7512] could have been expressed formally (namely that pk11-URI is a subset of URI), then a validator could have easily flagged that problem during the editorial process.

Regards,

Sean

---

From: Julian Reschke <[email-masked]>
Date: Mon, 14 Nov 2016 06:12:09 +0100

> Yes, I saw that. It is a difference approach to the “mail standards” approach, which uses a lot of / and =/ and extension-header syntaxes.

And RFC 2616 did that as well - it's just that we (that is the HTTPbis 
WG) decided that this use of ABNF isn't helpful -- IMHO the structure of 
the ABNF should be aligned with how messages are parsed; so this change 
in RFC 723x reflects that reality.

>     ] ) *( OWS "," [ OWS ( received-protocol RWS received-by [ RWS

Can you clarify why that is a problem?

> All that is very interesting, but if you want to verify that the ABNF is correct (or if you want to verify that sample HTTP messages conform to the ABNF), you have to do extra steps to extract each…

There are two questions here.

- if you want the validity of the message itself, you check the message ABNF

- if you want to check the validity of a given field value, you check 
the field value against the field's ABNF

> It is desirable to express the relationship between the <Via> production, and the <field-name> “Via”. Specifically, when <field-name> is “Via”, then <field-value> is <Via>.

Yes, I wouldn't be opposed to a mechanism that just deals with that.

> A recent place where this leads to a protocol problem is the PKCS #11 URI scheme. RFC 7512 says that “|” can appear in <pk11-query-res-avail>, as a delimiter to some command-path. But “|” is not a…

I believe that's a much better example to use.

Best regards, Julian

---

From: "Jonathan Hansford" <[email-masked]>
Date: Mon, 14 Nov 2016 08:45:42 -0000

Re the formality of ABNF, we use it to help enforce naming conventions for system management (which, with multiple naming conventions from various vendors and standards impacting on a single name, is why this is of interest). The less ambiguity the better, otherwise we will have to develop our own "ABNF" to more formally capture what is presented in RFCs.

Jonathan

> -----Original Message-----
> From: abnf-discuss [mailto:[email-masked]] On Behalf Of
> Sean Leonard
> Sent: 14 November 2016 04:46
> To: Doug Royer <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [abnf-discuss] constrained-01 - advantage?
> 
> 
> >> Second of all, <resent-date>, <resent-from>, <resent-sender> etc. are all
> “subsumed” in <resent-unstructured>, leading to an intentionally ambiguous
> grammar. The problem with the existing way of writing things out, is that the
> former are all superfluous when the latter is present. Another way to write it
> tends to be:
> >>
> > > but when a computer / parser parses productions, every <date> is
> > > going to match with <extension-field> as well. The point being
> > If your talking about replacing ABNF with new syntax. then I would agree.
> To a person reading the ABNF, they mean the same thing.
> I suppose that deprecating the old method would be something on the table
> to discuss. I believe it is desirable to avoid ambiguous grammars. At the same
> time, a sufficiently sophisticated tool could be written to identify these sorts
> of cases (of intentionally ambiguous grammars for parameterized names and
> values).
> 
> computer program? To mandate it?  I did not understand that after reading
> the draft.
> >
> We don’t want that. But the idea of “Table Constraints” is an ASN.1 term. (It
> is also a SQL concept, because, surprise, it’s about constraining data to tables
> of values.)
> 
> I would say that a goal is that ABNF in RFCs should be easy to validate, and
> this would include by means of computer programs. This increases document
> quality. I recognize that there is a spectrum of views around here about how
> “formal” ABNF ought to be.
> 
> Regards,
> 
> Sean
> 
> _______________________________________________
> abnf-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/abnf-discuss

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 14 Nov 2016 10:23:03 -0500

On 11/13/16 4:38 PM, Doug Royer wrote:
>
> It has been over 30 years since I have taken any math classes.

I have you beat - 50 years for me.

> The

The latter is what is commonly used now for extensible fields. When 
extensions are done in separate drafts it becomes:

    resent-field = <defined in RFCnnn>
    resent-field =/ "Resent-Flag:" flag CRLF

With extensions allowed in separate drafts there is usually a rule that 
anything matching resent-unstructured and not one of the other 
alternatives is to be ignored. (This makes extensibility with backward 
compatibility work.)

But this only works if every extension matches resent-unstructured. But 
when using this form there is no validation that they do. I have 
reviewed extensions of this sort that I discovered did not match in this 
way. And often the ABNF isn't reviewed carefully enough to guarantee 
that such things will be noticed.

What the proposal does is ensure that the compatibility is verified by 
the ABNF parser.

	Thanks,
	Paul

---

From: Sean Leonard <[email-masked]>
Date: Tue, 15 Nov 2016 13:51:26 +0900

Can you elaborate on your request or concern? Is this about ABNF rule names, or something else?

Thanks!

Sean

> On Nov 14, 2016, at 5:45 PM, Jonathan Hansford <[email-masked]> wrote:
>

---

From: Sean Leonard <[email-masked]>
Date: Tue, 15 Nov 2016 13:51:31 +0900

> On Nov 14, 2016, at 2:12 PM, Julian Reschke <[email-masked]> wrote:
> 
> And RFC 2616 did that as well - it's just that we (that is the HTTPbis WG) decided that this use of ABNF isn't helpful -- IMHO the structure of the ABNF should be aligned with how messages are parsed; so this change in RFC 723x reflects that reality.

Okay.

> I believe that's a much better example to use.

Sounds like we have a plan for progress then.

> Can you clarify why that is a problem?

In some scenarios (namely if the author wants), it is nice to determine that all rules in a specification are reachable from some starting rule. This can help to identify and eliminate orphan rules that are no longer used by the spec. Of course, that is just if the author wants.

> - if you want to check the validity of a given field value, you check the field value against the field's ABNF
> 

Yes. In this example, I would say that the “message ABNF” can have a header-line that conforms with the generic syntax, but does not conform with the particular field-name -> field-value production. This should not be a fatal error, in the sense that the (HTTP) message is unusable; but merely that particular field cannot be used. (The actual behavior is up to whatever spec.) I think we are on the same page here...

Overall, for document authors, it helps for a tool to check that the subordinate rule (constrained rule) is purely a subset of the generic rule. For development/debugging/testing/analysis of data on-the-wire, it helps for a single ABNF spec to be used to check the generic data (generic rule) as well as the specific data (constrained rule) in one swoop, without much additional coding to glue the two together.

Regards,

Sean

---

From: "Jonathan Hansford" <[email-masked]>
Date: Fri, 18 Nov 2016 15:48:57 -0000

We use ABNF to define naming conventions for devices, URIs, etc. within our managed system. For example, the system may require a device's identify (as defined by its own naming convention) to be a host name. Its identity may also need to be displayed on a second device that has naming conventions of its own. Clearly if we can define each naming convention (including that for a host name) unambiguously in ABNF we can provide guidance to the system manager as to whether any entered name is valid across all applicable naming conventions. This would apply to all objects that need to be identified within the system and for all protocols, etc. we need to support. If there is ambiguity in the ABNF within the relevant RFCs, etc. then we have to remove that ambiguity to the extent we need to provide support. If there is no ambiguity, full support is available "out of the box".

Hope that helps,

Jonathan

> From: Sean Leonard [mailto:[email-masked]]
> Sent: 15 November 2016 04:51
> To: Jonathan Hansford <[email-masked]>
> Cc: Doug Royer <[email-masked]>; [email-masked]
> > Re the formality of ABNF, we use it to help enforce naming conventions for
> system management (which, with multiple naming conventions from various
> vendors and standards impacting on a single name, is why this is of interest).
> The less ambiguity the better, otherwise we will have to develop our own
> "ABNF" to more formally capture what is presented in RFCs.
> >
> > Jonathan
> >
> >> From: abnf-discuss [mailto:[email-masked]] On Behalf
> >> Of Sean Leonard
> >>>>> resent-
> >> sender / ...
> >> “subsumed” in <resent-unstructured>, leading to an intentionally
> >> ambiguous grammar. The problem with the existing way of writing
> >> things out, is that the former are all superfluous when the latter is
> >> present. Another way to write it tends to be:
> >>>>
> >>>> Well it makes perfect sense when you read it, of course, but when a
> >>>> computer / parser parses productions, every <date> is going to
> >>>> match with <extension-field> as well. The point being
> >>>> There is nothing “wrong” with creating an ambiguous grammar, but it
> >>>> can lead to subtle parsing errors if you are not careful.
> >>>> code to go down one, and only one, path. An automated ABNF
> >>>> validator should be able to detect ambiguous grammars and inform
> >>>> you about
> >> that.
> >>>
> agree.
> >> I suppose that deprecating the old method would be something on the
> >> table to discuss. I believe it is desirable to avoid ambiguous
> >> grammars. At the same time, a sufficiently sophisticated tool could
> >> be written to identify these sorts of cases (of intentionally
> >> ambiguous grammars for parameterized names and values).
> >>
> >> We don’t want that. But the idea of “Table Constraints” is an ASN.1
> >> term. (It is also a SQL concept, because, surprise, it’s about
> >> constraining data to tables of values.)
> >>
> >> I would say that a goal is that ABNF in RFCs should be easy to
> >> validate, and this would include by means of computer programs. This
> >> increases document quality. I recognize that there is a spectrum of
> >> views around here about how “formal” ABNF ought to be.
> >>
> >> Regards,
> >>
> >> Sean
> >>

---

From: Doug Royer <[email-masked]>
Date: Fri, 18 Nov 2016 10:50:47 -0700

I do not have a problem with making ABNF machine parsable.

This draft makes it possible to make ABNF that is still not machine 
parsable. Because the old difficult to parse methods are still valid.

In my opinion, if this draft is going to go forward, this draft will 
need to go much further, because its going to make a confusing mess of 
mixed ABNF usage in future drafts.

And much further, its going to be more controversial.

--

---

From: Sean Leonard <[email-masked]>
Date: Sat, 19 Nov 2016 04:55:49 +0900

Hello Doug:

Thank you...I think?

ABNF is already “machine-parsable”. A couple of months ago, I published (ss)reaex, which is a machine parser (extractor, specifically). Other people have written full parser-compilers (apg, bap). The fact that a compiler “can” be written for ABNF has always been there, although perhaps it was not a design goal of RFC x234.

I suppose that a goal of draft-seantek-constrained-abnf is to help authors make ABNF more machine-parsable, but, that is not the only goal. Another goal is to assist specification writers with making their ABNF clearer for (human) readers, which means introducing a minimum amount of new syntax for specification writers to learn. There are other IETFers who do not feel that machine parsing is a (strong) design goal. To that end, another design criteria is that ABNF still meets all the criteria of BNF: a context-free grammar notation.

There are other enhancements to RFC 5234 that have been proposed, which do take the language further:
RFC 7405
draft-seantek-unicode-in-abnf
draft-seantek-abnf-more-core-rules

Can you describe what you mean by “needing to go much further”?

Regards,

Sean

> On Nov 19, 2016, at 2:50 AM, Doug Royer <[email-masked]> wrote:
> 
> 
> This draft makes it possible to make ABNF that is still not machine parsable. Because the old difficult to parse methods are still valid.
> 
> In my opinion, if this draft is going to go forward, this draft will need to go much further, because its going to make a confusing mess of mixed ABNF usage in future drafts.
>

---

From: "Jonathan Hansford" <[email-masked]>
Date: Sat, 19 Nov 2016 08:15:04 -0000

With the work we are doing on naming conventions within managed systems, RFC7405, draft-seantek-abnf-more-core-rules and draft-seantek-constrained-abnf all take ABNF closer to what we need. I would value an RFC that added all their additional features to RFC5234 and an equivalent ABNF parser.

Doug,

Are you talking about preventing ABNF such as the following:

yang-version-arg-str = < a string that matches the rule >
                                              < yang-version-arg >

Jonathan

> Sent: 18 November 2016 19:56
> ABNF is already “machine-parsable”. A couple of months ago, I published
> (ss)reaex, which is a machine parser (extractor, specifically). Other people
> have written full parser-compilers (apg, bap). The fact that a compiler “can”
> be written for ABNF has always been there, although perhaps it was not a
> design goal of RFC x234.
> 
> I suppose that a goal of draft-seantek-constrained-abnf is to help authors
> make ABNF more machine-parsable, but, that is not the only goal. Another
> goal is to assist specification writers with making their ABNF clearer for
> (human) readers, which means introducing a minimum amount of new
> syntax for specification writers to learn. There are other IETFers who do not
> feel that machine parsing is a (strong) design goal. To that end, another
> design criteria is that ABNF still meets all the criteria of BNF: a context-free
> grammar notation.
> 
> > This draft makes it possible to make ABNF that is still not machine parsable.
> Because the old difficult to parse methods are still valid.
> >
> > In my opinion, if this draft is going to go forward, this draft will need to go
> much further, because its going to make a confusing mess of mixed ABNF
> usage in future drafts.
> >