Subject: [abnf-discuss] ABNF List Extension: #rule

From: Paul Kyzivat <[email-masked]>
Date: Tue, 02 Jun 2015 10:11:24 -0400

I just became aware of an ABNF extension that is embedded in RFC7230 
(HTTP/1.1 Message Syntax and Routing):

https://tools.ietf.org/html/rfc7230#section-7

    A construct "#" is defined, similar to "*", for defining
    comma-delimited lists of elements.  The full form is "<n>#<m>element"
    indicating at least <n> and at most <m> elements, each separated by a
    single comma (",") and optional whitespace (OWS).

I find it annoying/disturbing that such an extension was done that way - 
embedded in an RFC that uses it.

Do others share that opinion? What can be done about it?

I do think there could be some value in enhancements to ABNF to improve 
the ability to specify lists. But I personally would not be in favor of 
"wasting" one of the few remaining unused likely candidate operator 
characters on such a specialized extension, tied to a particular 
separator syntax.

There are other issues with specifying lists that are IMO more 
important. In particular, there is no practical way to define a list 
consisting of a combination of elements drawn from a set of alternatives.

	Thanks,
	Paul

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Jun 2015 16:23:24 +0200

> Do others share that opinion? What can be done about it?

I disagree. It's an extension that's totally specific to HTTP header fields.

> characters on such a specialized extension, tied to a particular

See? That's why it's specific to HTTP.

> consisting of a combination of elements drawn from a set of alternatives.

Example?

Best regards, Julian

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 02 Jun 2015 10:33:18 -0400

Julian,

The main problem I see in defining this as an HTTP-specific extension to 
ABNF is that it invalidates all general ABNF tools - checkers, 
compilers, etc.

When this RFC was reviewed the syntax was supposed to be verified. If I 
had been doing that, I would have wanted to use a tool like bap to check 
the syntax. But because of this extension that wouldn't be possible. 
(And in the future, I hope that it will become possible to embed such 
checking into standard submission tools like IdNits.)

And it could become more of a problem in the future, if some other 
extension to ABNF decides to use # for some other purpose.

	Thanks,
	Paul

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Jun 2015 16:48:33 +0200

> ABNF is that it invalidates all general ABNF tools - checkers,

That's why the spec says how it translates into generic ABNF *and* 
includes these expansions.

> When this RFC was reviewed the syntax was supposed to be verified. If I

It was as part of our document build process.

> the syntax. But because of this extension that wouldn't be possible.

Yes, it is. See above. It requires preprocessing.

> extension to ABNF decides to use # for some other purpose.

We inherited it from RFC 2068 and 2616, and we wouldn't recommend it for 
anything other than HTTP header field definitions.

Best regards, Julian

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 02 Jun 2015 12:07:09 -0400

> includes these expansions.

Swell. Then to verify, one must apply the expansion to every use. That 
introduces work for the verifier, and the potential for screwing it up.

Note the confusion that occurred here when a reviewer didn't realize 
that the "ABNF" being used required a special extension.

The syntax of ABNF (a character for every operator) doesn't lend itself 
to lots of special purpose extensions. Maybe we should define a 
function-with-arguments syntax for extensions, or a macro preprocessor 
for ABNF. E.g., maybe we could have:

   #HTML-list(1,*,foo-parameter)

as a function/macro call, with HTML-list defined somewhere/somehow.

But it will be tricky to find a syntax that is harmonious with ABNF.

	Thanks,
	Paul

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Jun 2015 18:31:18 +0200

> introduces work for the verifier, and the potential for screwing it up.

Or the verifier could instead verify that the expansions were done 
consistently with a tool that was explicitly created for this purpose 
(here: a modified version of bap).

> that the "ABNF" being used required a special extension.

Well, the spec says that an extension is being used.

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Tue, 02 Jun 2015 22:13:55 -0700 (PDT)

>     single comma (",") and optional whitespace (OWS).

This concept did not originate with RFC 7230; it's present in the ABNF 
specified in RFC 822:

     2.7.  #RULE:  LISTS
 
          A construct "#" is defined, similar to "*", as follows:
 
                              <l>#<m>element
 
     indicating at least <l> and at most <m> elements, each  separated
     by  one  or more commas (","). This makes the usual form of lists
     very easy; a rule such as '(element *("," element))' can be shown
     as  "1#element".   Wherever this construct is used, null elements
     are allowed, but do not  contribute  to  the  count  of  elements
     present.   That  is,  "(element),,(element)"  is  permitted,  but
     counts as only two elements.  Therefore, where at least one  ele-
     ment  is required, at least one non-null element must be present.
     Default values are 0 and infinity so that "#(element)" allows any
     number,  including  zero;  "1#element" requires at least one; and
     "1#2element" allows one or two.

> Do others share that opinion?

I don't. RFC 7230 is a revision of much older RFCs, which used RFC 822 ABNF,
it's actually more of a change to try and implement this syntax using
current ABNF facilities.

But even if this was an extension that originated in this RFC, as long
as its use is justified and clearly documented I wouldn't have a problem
with it.

> What can be done about it?

I don't think anything can or should be done.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Wed, 03 Jun 2015 06:38:16 -0700

> it's actually more of a change to try and implement this syntax using

I've always liked the construct (duh).

We took it out, when RFC 822 was being revised, because it wasn't
popular enough and Internet technical work back then tended to be driven
by making the core specification be the smaller set of essential bits,
with extensions (standardized or private) be (initially) of interest in
more specialized groups.  This makes the core simpler. Hence, it was
taken out of the core, but adopted by a group.

That is, the core should be what /will/ be widely used, not what /might/
be useful to some.

Also note that ABNF is about extensibility.  So cramming everything into
the core is inherently counter-design.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 03 Jun 2015 11:00:01 -0400

On 6/3/15 9:38 AM, Dave Crocker wrote:
> We took it out, when RFC 822 was being revised,

Ah. I didn't know it had history that far back! Interesting.

> because it wasn't

I understand that, especially for this particular feature. It is quite 
specialized, since it only works with lists that use comma with 
whitespace as the separator. There are lots of places where such a 
construct could be useful but where a different separator is called for. 
That made sense when ABNF itself was being defined for one particular 
usage. But not so much when it was being defined as a general purpose 
language.

> the core is inherently counter-design.

*use* of ABNF can be a useful tool for providing extensibility in 
protocols. But IMO ABNF itself isn't well suited for extension. The 
design employs special characters as operators, and there are only so 
many of those available. In this way it differs from languages that 
allow keywords as operators, or named macros.

Also, making extensions to ABNF cuts you off from the tools built for 
it, unless you extend the tools too. We don't have a good way of 
publishing tools extensions the same way we publish RFC extensions.

People don't commonly extend C, C++, or Java directly. They extend it be 
adding libraries, or macros. That way they can still use all the tools 
for those languages.

This of course isn't a problem if you don't use tools - if you simply 
use ABNF as a way to share syntax between people, and "implement" it 
manually without specific tools.

	Thanks,
	Paul

---

From: Tony Finch <[email-masked]>
Date: Wed, 3 Jun 2015 16:08:08 +0100

> Ah. I didn't know it had history that far back! Interesting.

RFC 733 also has it. (RFC 724 uses a less flavoursome BNF.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Irish Sea: Southwest backing south 4 or 5, occasionally 6 at first. Slight or
moderate. Fair. Good.

---

From: Paul Overell <[email-masked]>
Date: Thu, 04 Jun 2015 10:50:17 +0100

> taken out of the core, but adopted by a group.

A further historical note on the reasons we dropped #.

An important difference between RFC822-ABNF and RFC5234-ABNF is in the
handling of white space.  RFC822 has a two level scheme 1) parsing
characters into tokens, ignoring all white space between tokens, 2)
parsing tokens.  This freed RFC822 syntax from any need to deal with
white space, it was all implicit.  In RFC5234 there is only one level,
parsing characters, so all white space must be explicitly handled in the
ABNF.  This was a deliberate decision.  However it gave us a problem
with # operator as it has embedded implicit rules for handling white
space around the ",".  This was inconsistent with our decision to make
white space handling explicit.   As well as the reasons Dave has given,
removing # solved this inconsistency.

> the core is inherently counter-design.

+1

Regards

-- 
Paul Overell

---

From: Dave Crocker <[email-masked]>
Date: Thu, 04 Jun 2015 03:46:51 -0700

On 6/3/2015 8:08 AM, Tony Finch wrote:
> RFC 733 also has it.

As I said, I liked the construct...

d/