Subject: [abnf-discuss] Comprehensive Core Rules for ABNF (draft-seantek-abnf-more-core-rules-00.txt)

From: Sean Leonard <[email-masked]>
Date: Mon, 9 Mar 2015 16:21:14 -0700

Hello ABNF-Discuss World:

Please take a look at my Internet-Draft, draft-seantek-abnf-more-core-rules-00.txt. This work is based on the discussions in January-February that originally took place on the rfc-interest mailing list. The draft is intentionally just a short 5 pages and I would like to keep it that way.

(I am not cross-posting at this time, but depending on the volume on this list, I might notify the rfc-interest list too after a few days.) 

Thank you,

Sean

Begin forwarded message:

> From: [email-masked]
> Subject: New Version Notification for draft-seantek-abnf-more-core-rules-00.txt
> Date: March 9, 2015 at 4:06:03 PM PDT

A new version of I-D, draft-seantek-abnf-more-core-rules-00.txt
has been successfully submitted by Sean Leonard and posted to the
IETF repository.

Name:		draft-seantek-abnf-more-core-rules
Revision:	00
Title:		Comprehensive Core Rules for ABNF
Document date:	2015-03-09
Group:		Individual Submission
Pages:		5
URL:            http://www.ietf.org/internet-drafts/draft-seantek-abnf-more-core-rules-00.txt
Status:         https://datatracker.ietf.org/doc/draft-seantek-abnf-more-core-rules/
Htmlized:       http://tools.ietf.org/html/draft-seantek-abnf-more-core-rules-00

Abstract:
  This document extends the base definition of ABNF (Augmented Backus-
  Naur Form) to include comprehensive support for certain symbols,
  namely those in the US-ASCII standard.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Paul Overell <[email-masked]>
Date: Thu, 12 Mar 2015 20:03:50 +0000

Hi,

> Please take a look at my Internet-Draft, draft-seantek-abnf-more-core-rules-00.txt. This work is based on the discussions in January-February that originally took place on the rfc-interest mailing…

I'm not convinced that rules for US-ASCII control characters should
count as core rules.  The core rules were intended to be a small set of
commonly used rules.  Are US-ASCII control codes commonly used in
grammars specified by ABNF?  DC1, GS?  Is it worth breaking the long
standing stability of the core rules for this?  Although I can see that
if more than one grammar needs rules for US-ASCII then having an RFC for
them could be a good idea.

i think it would be better if this draft confined itself to just
specifying the US-ASCII control characters, and not as a "drop-in
replacement for Appendix B.1".  With, perhaps, an appropriate name
change such as "ABNF rules for US-ASCII control characters" or some
such.  It could then be cited, where needed, in conjunction with an
unmodified RFC5234.

It would also show how in future other sets of rules could be published
as standalone RFCs.  I think this is a more scalable solution for how to
add more rules, does not run any risk of breaking backwards
compatibility, and avoids any need to worry about if a rule is "core" or
not.

Regards

-- 
Paul Overell (joint author RFC5234)

---

From: Sean Leonard <[email-masked]>
Date: Sun, 22 Mar 2015 15:47:26 -0500

> compatibility, and avoids any need to worry about if a rule is "core" or

Several thoughts (sorry I did not see this before today):

You asked, “are US-ASCII control does commonly used in grammars specified by ABNF? DC1, GS?”

Yes, more often than you think. For example, Nico Williams’ JSON text sequence document (now RFC 7464) uses, and defines, RS and LF. RS is a critical part of the protocol. There are other examples if you look in the RFC series.

The counter-question is, “are certain US-ASCII control codes so much more important that they should be defined by a core set of rules, while others should be ignored?” I submit that they are not. All control code points are more-or-less the same for ABNF purposes. Put another way, why would anyone want to define DC1 as anything other than %d17? If a draft or RFC is defining DC1 as something other than %d17, it is confusing and will lead to implementation errors.

A case in point is ESC, which is %d27 (0x1B). LDAP (RFC 4512) has the following strange rules, among others:

 ESC = %x5C ; backslash (“\”)
 SPACE = %x20 ; space (“ ”)
 SP = 1*SPACE ; one or more “ ”

Saying SP is multiple space characters and ESC is BACKSLASH is confusing. LDAP ought to be excused since RFC 451x predate RFC 5234. The point however is that going forward, we ought to be consistent. There are many examples of pre-5234 ABNF that deviates significantly form RFC 5234, that do not undermine the standardization effort that RFC 5234 represents, or RFCs post-5234.

This draft is a milestone based on a conversation about this as well as a) other core rule updates, and b) other extensions to ABNF, such as “ABNF modules”. Should this document progress, the purpose was to see if others can justify other core rule updates beyond what I have proposed. I haven’t heard of anything yet.

The broader editorial issue is how should improvements to RFC 5234 be handled. For my part, I request that this update be treated similarly to RFC 7405 (Case-Sensitive String Support in ABNF). Other than that I do not have a strong preference.

A second counter-question is: “if RFC 5234bis is made, will RFC 5234bis incorporate RFC 7405 or leave it as an optional extension to the grammar?”

Furthermore:
“Is it necessary for a document that uses case-sensitive string syntax to make a normative reference to RFC 7405, or is RFC 5234 sufficient?”

“After RFC 5234bis, should future documents that use case-sensitive string syntax make a normative reference to RFC 7405 [or bis], or is RFC 5234bis sufficient?”

Best regards,

Sean

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 23 Mar 2015 09:16:58 -0400

> The broader editorial issue is how should improvements to RFC 5234 be handled. For my part, I request that this update be treated similarly to RFC 7405 (Case-Sensitive String Support in ABNF). Othe…

I think there is a big difference between extensions to the *syntax* of 
ABNF, and extensions that simply define sets of rules. For those 
(including this proposal for defining control characters), I would like 
to see an explicit formal mechanism for including externally defined 
(and standardized) sets of rules into an ABNF-based grammar. And that 
ought to be used for the core rules as well.

> A second counter-question is: “if RFC 5234bis is made, will RFC 5234bis incorporate RFC 7405 or leave it as an optional extension to the grammar?”

In that case I would hope that 7405 would be merged in.

> “Is it necessary for a document that uses case-sensitive string syntax to make a normative reference to RFC 7405, or is RFC 5234 sufficient?”

For the moment I am assuming it is necessary to reference 7405.

> “After RFC 5234bis, should future documents that use case-sensitive string syntax make a normative reference to RFC 7405 [or bis], or is RFC 5234bis sufficient?”

Assuming all of the above, then just a reference to the bis.

However I do get the impression that some have a different opinion - 
that there should be a core ABNF specification and a variety of optional 
extensions.

IMO *that* would make sense if the extensions were "domain specific". 
But I don't think it is the best way forward for "generic" extensions. 
And I'm dubious that it would make sense to define domain specific 
extensions to ABNF *syntax*.

	Thanks,
	Paul

---

From: Sean Leonard <[email-masked]>
Date: Mon, 23 Mar 2015 10:08:04 -0500

Thanks for the comments…

On Mar 23, 2015, at 8:16 AM, Paul Kyzivat <[email-masked]> wrote:

> I think there is a big difference between extensions to the *syntax* of ABNF, and extensions that simply define sets of rules. For those (including this proposal for defining control characters), I would like to see an explicit formal mechanism for including externally defined (and standardized) sets of rules into an ABNF-based grammar. And that ought to be used for the core rules as well.

As I think I have said elsewhere, for those who want an explicit formal mechanism (aka “modules and imports”, or as M&I), please write an Internet-Draft about it. However it should not affect progress on this proposal.

> In that case I would hope that 7405 would be merged in.

In agreement.

> For the moment I am assuming it is necessary to reference 7405.

In agreement.

> Assuming all of the above, then just a reference to the bis.

In agreement.

The way I see case-sensitive string syntax is that it is a normative (if not currently frequently used) part of IETF/RFC 5234 ABNF. I see these Core Rules as the same.

> 
> However I do get the impression that some have a different opinion - that there should be a core ABNF specification and a variety of optional extensions.

I think we are in agreement but our agreement is leading us to different places.

The problems with optional extensions include:
#1 ABNF grammar parser implementers have to implement switches or controls to turn optional extensions on or off, complicating the implementation;
#2 ABNF readers (i.e., human beings) need to parse the prose text surrounding the ABNF to see what features are on or off.

The tradeoffs of optional extensions outweigh the benefits.

I do not see Core Rules to be different or separable from the utility of post-5234 ABNF (including its syntax). From a programming perspective, the analogy is whether ABNF should be more like C (libs optional) or Java (always-present libs). With C, libraries are optional so it is possible to compile a program with no external dependencies whatsoever, not even <stdio.h>. Everything must be included. In contrast, with Java, all programs can assume the existence of certain standard libraries, some of which are necessary to access really core features (Object, etc.). In the middle we have something like C++, where libs are supposedly optional (like C) but there are things like std::bad_cast that are generated from the language itself. Then we have things like JavaScript, where some things are global properties (Infinity) that really functionally behave like keywords, and the only reason why someone would attempt to change them is to do something unexpected or malicious. Later JS specifications have since “calcified” such global properties so that they are no longer subject to change.

In this case in the ABNF I believe that we are closer to the always-present libs end of the spectrum, or the JS view, which is that maybe the definitions are (or were) malleable but you shouldn’t do that unless you are intentionally trying to generate unexpected behavior. This comes partially from RFC 5234 itself, and partially from the way that the rules are referenced.

RFC 5234 says “Appendix B supplies rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications.” (In fairness, it also says that Appendix B is provided as a convenience and is separable from the syntax.) However, the rules appear in the same document, and when other RFCs reference RFC 5234, they basically assume the existence of these Core Rules. Except for backwards compatibility, new RFCs that need to use SP, DQUOTE, etc. should just refer to those definitions in RFC 5234.

In sum, the reason why this Internet-Draft is proposing a “drop-in replacement” for RFC 5234 Appendix B is because this is the same effective approach as used before and does not re-litigate prior issues.

> 
> IMO *that* would make sense if the extensions were "domain specific". But I don't think it is the best way forward for "generic" extensions. And I'm dubious that it would make sense to define domain specific extensions to ABNF *syntax*.

I have not seen examples of domain-specific extensions to ABNF in those ways.

In this case, we are talking about US-ASCII symbols that have been around literally since 1968. I would say this is about as generic as one can get.

Cheers,

Sean

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 23 Mar 2015 12:04:41 -0400

> As I think I have said elsewhere, for those who want an explicit formal mechanism (aka “modules and imports”, or as M&I), please write an Internet-Draft about it. However it should not affect progr…

There has been some discussion of this, but nothing definite.
I'll take the flip side of this: let us first establish that such a 
mechanism is a good thing that we would like to see, and nail down what 
properties we would like to have, before worrying about the details.
(I don't think anyone will argue that defining such a mechanism is hard.)

> I think we are in agreement but our agreement is leading us to different places.

Perhaps.

> #2 ABNF readers (i.e., human beings) need to parse the prose text surrounding the ABNF to see what features are on or off.

This is true of *syntax* extensions to ABNF. Not so much for core rules 
and their extensions.

> I do not see Core Rules to be different or separable from the utility of post-5234 ABNF (including its syntax). From a programming perspective, the analogy is whether ABNF should be more like C (libs optional) or Java (always-present libs). With C, libraries are optional so it is possible to compile a program with no external dependencies whatsoever, not even <stdio.h>. Everything must be included. In contrast, with Java, all programs can assume the existence of certain standard libraries, some of which are necessary to access really core features (Object, etc.). In the middle we have something like C++, where libs are supposedly optional (like C) but there are things like std::bad_cast that are generated from the language itself. Then we have things like JavaScript, where some things are global properties (Infinity) that really functionally behave like keywords, and the only reason why someone would attempt to change them is to do something unexpected or malicious. Later JS specif
 i
cations have since “calcified” such global properties so that they are no longer subject to change.
> RFC 5234 says “Appendix B supplies rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications.” (In fairness, it also says that Appendix B is pr…

My interpretation is different from yours.
In my experience, those specs that use the core rules say *something* 
indicating explicitly that they are importing either the complete set of 
core rules, or more often that they are importing certain specific rules 
from the core rules. They don't assume that all the core rules are 
available by default to all who use ABNF. (In fact, I and others often 
give review comments requesting that such statements be added if they 
are missing.)

But there isn't a lot of consistency in *how* such imports are 
specified. Current usage is not something that can be implemented in 
code. Rather it requires that someone who wants to syntax check or 
compile a set ABNF rules from a draft must manually augment those by 
cutting and pasting from the 5234 core rules and other sources.

> In sum, the reason why this Internet-Draft is proposing a “drop-in replacement” for RFC 5234 Appendix B is because this is the same effective approach as used before and does not re-litigate prior…

I agree this is a pragmatic choice given that there is currently no 
import mechanism. But I don't think doing it means that ABNF parsers 
will automatically recognize all those rules.

> I have not seen examples of domain-specific extensions to ABNF in those ways.

Me either.

> In this case, we are talking about US-ASCII symbols that have been around literally since 1968. I would say this is about as generic as one can get.

Agreed.

However, if this was done, and if ABNF parsers automatically define all 
the core rules, then this change would present backward compatibility 
issues for existing ABNF syntaxes, such as the ones you mention above 
that define ESC and SP differently than your proposal does.

	Thanks,
	Paul