Subject: [rfc-i] Table of conformance requirements.

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 17 Jun 2020 23:49:10 -0400

All,

I would like to propose an indexed table of all MUST/SHOULD/MAY normative
language for RFCs with labels for each requirement.

I already have code to do this but I have realized I need to go further to
really bind specification and implementation together properly.

Each MUST in a specification should have at least one corresponding unit
test to check compliance. And there should be a mechanism which allows the
test set to be audited against the specification. Which means that I need
to be able to label each requirement.

So in the text the requirement would be 'clients MUST send round pegs' and
in the list this would be something like

MUST-2: Round Pegs

And this needs to be expressible somehow in XML.

Comments?
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 18 Jun 2020 09:08:08 +0200

There is a whole industry around requirements tracking.
I’m sure the Ribose people can tell us more about that.

> Each MUST in a specification should have at least one corresponding unit test to check compliance.

Yeah sure.

RFC 4120:
   *  Principals MUST keep their secret keys secret.

I’d love to see that unit test :-)

The other problem is that we simply don’t state all requirements in BCP14 language.
Very often, most requirements are stated in describing an architecture or a protocol; they are phrased as statements of fact.  Extracting and labeling these requirements for requirements tracking is an art form.

> And this needs to be expressible somehow in XML. 

I played around some with using kramdown’s auto-indexing feature for BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14 usage.  Not exactly what you want, but just a few lines of kramdown markdown.

I never used the results of these experiments in a document, mainly because the lack of BCP14 language on so many of the actual requirements made the result too sketchy.

Grüße, Carsten

---

From: Robert Sparks <[email-masked]>
Date: Thu, 18 Jun 2020 08:37:58 -0500

I have a script that builds such lists, and have used it for 20-ish 
years at this point in various contexts.

Some example output:

https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-enumreqs.txt

https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt

For the most part, its best use has been to help identify when there are 
meaningless, duplicate, or contradictory uses of the BCP14 keywords.

In interop testing (primarily the SIPit), having these requirements 
enumerated this way didn't help drive test behavior or the creation of 
test suites. It turned out that far more context was needed as a 
predicate to "I'm going to check this thing" than the isolated sentence 
containing the requirement. And agreeing on that context often turned 
out to be a subjective exercise (when that happened, I'd help put 
pressure on changing the spec to reduce the probability of different 
interpretations).

So, it's been somewhat helpful in my experience, but not helpful enough 
to try to build extra support for it into, say, the v3 grammar.

RjS

---

From: Robert Sparks <[email-masked]>
Date: Thu, 18 Jun 2020 11:31:46 -0500

On 6/18/20 11:29 AM, Larry Masinter wrote:
> A previous proposal
> https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00
> and experiment
> https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00
Yeah, well. Fwiw: https://datatracker.ietf.org/doc/rfc5657/
>
>
> --
> https://LarryMasinter.net https://going-remote.info
>
>> -----Original Message-----
>> From: rfc-interest <[email-masked]> On Behalf Of
>> Robert Sparks
>> Sent: Thursday, June 18, 2020 6:38 AM
>> To: [email-masked]
>> Subject: Re: [rfc-i] Table of conformance requirements.
>>
>> I have a script that builds such lists, and have used it for 20-ish years at this
>> point in various contexts.
>>
>> Some example output:
>>
>> enumerated this way didn't help drive test behavior or the creation of test
>> suites. It turned out that far more context was needed as a predicate to "I'm
>> going to check this thing" than the isolated sentence containing the
>> requirement. And agreeing on that context often turned out to be a
>> subjective exercise (when that happened, I'd help put pressure on changing
>> the spec to reduce the probability of different interpretations).
>>
>> So, it's been somewhat helpful in my experience, but not helpful enough to
>> try to build extra support for it into, say, the v3 grammar.
>>
>> RjS
>>
>> test to check compliance.
>>> Yeah sure.
>>>
>>> RFC 4120:
>>> Very often, most requirements are stated in describing an architecture or a
>> protocol; they are phrased as statements of fact.  Extracting and labeling
>> these requirements for requirements tracking is an art form.
>>> I played around some with using kramdown’s auto-indexing feature for
>> BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14
>> usage.  Not exactly what you want, but just a few lines of kramdown
>> markdown.
>>> I never used the results of these experiments in a document, mainly
>> because the lack of BCP14 language on so many of the actual requirements
>> made the result too sketchy.
>>> Grüße, Carsten
>>>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 18 Jun 2020 09:41:43 -0700

I think Robert's example shows the limit of the exercise. Take this
requirement from the list:

  MUST-3: There are certain operations that an application **MUST** be able to perform when interacting with QUIC streams.

How are you going to test that snippet of text? In the original
document, what follows is the list of these operations that the
application is supposed to perform. But if you just take out of context
the snippet of text surrounding the MUST statement, you get something
meaningless.

-- Christian Huitema

On 6/18/2020 9:31 AM, Robert Sparks wrote:
>>>> The other problem is that we simply don’t state all requirements in
>>>> BCP14
>>> language.

---

From: Larry Masinter <[email-masked]>
Date: Thu, 18 Jun 2020 09:29:19 -0700

A previous proposal

---

From: Larry Masinter <[email-masked]>
Date: Thu, 18 Jun 2020 10:16:27 -0700

At the time, and also now, being clear enough about conformance requirements would be a requirement for the STD series, not the RFC series.

Given that you can’t just concatenate XML of separate RFCs to make a STD which has more than one component, there is a need for some other kind of way of specifying Internet Standards.

I imagine  Internet Standards which are a document series which might start out as pointers to the relevant RFCs (and errata) but which could be updated as the standard evolved.

--

 <https://larrymasinter.net/> https://LarryMasinter.net  <https://going-remote.info/> https://going-remote.info

From: rfc-interest <[email-masked]> On Behalf Of Christian Huitema
Sent: Thursday, June 18, 2020 9:42 AM
To: Robert Sparks <[email-masked]>; Larry Masinter <[email-masked]>; [email-masked]
Subject: Re: [rfc-i] Table of conformance requirements.

I think Robert's example shows the limit of the exercise. Take this requirement from the list:

How are you going to test that snippet of text? In the original document, what follows is the list of these operations that the application is supposed to perform. But if you just take out of context the snippet of text surrounding the MUST statement, you get something meaningless.

-- Christian Huitema

A previous proposal 

-----Original Message----- 
From: rfc-interest  <mailto:[email-masked]> <[email-masked]> On Behalf Of 
Robert Sparks 
Sent: Thursday, June 18, 2020 6:38 AM 
To: [email-masked] <mailto:[email-masked]>  
Subject: Re: [rfc-i] Table of conformance requirements. 

Some example output: 

https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29- 
enumreqs.txt 

RjS 

Yeah sure. 

RFC 4120: 

The other problem is that we simply don’t state all requirements in BCP14 

language. 

Grüße, Carsten 

_______________________________________________ 
rfc-interest mailing list 
[email-masked] <mailto:[email-masked]>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest 

_______________________________________________ 
rfc-interest mailing list 
[email-masked] <mailto:[email-masked]>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest 

_______________________________________________ 
rfc-interest mailing list 
[email-masked] <mailto:[email-masked]>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Joseph Touch <[email-masked]>
Date: Thu, 18 Jun 2020 10:30:40 -0700

> On Jun 18, 2020, at 10:16 AM, Larry Masinter <[email-masked]> wrote:
> 

Even the STD series has lots of requirements that cannot be tested, e.g., when the MUST is about users or deployments.

And a SHOULD or MAY is degenerately tested as “true” for all implementations because there’s no strict requirement to objectively state conditions under which exceptions hold.

Finally, there are lots of cases where requirements are nested in ways that we don’t have normative language for, e.g., MUST do A or B, unless it does C.

Joe

---

From: Larry Masinter <[email-masked]>
Date: Thu, 18 Jun 2020 11:01:58 -0700

The idea of formalizing interoperability reports was not to take every MUST and SHOULD out of the RFCs involved, but rather to get consensus at the time the Standard is approved about what the testable features are.

*	Even the STD series has lots of requirements that cannot be tested, e.g., when the MUST is about users or deployments.
*	And a SHOULD or MAY is degenerately tested as “true” for all implementations because there’s no strict requirement to objectively state conditions under which exceptions hold.

*	Finally, there are lots of cases where requirements are nested in ways that we don’t have normative language for, e.g., MUST do A or B, unless it does C.

There needs to be some way of deciding whether to report there are multiple interoperable implementations of every feature, whether or not that decision can be automated.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 18 Jun 2020 20:26:43 +0200

> Finally, there are lots of cases where requirements are nested in ways that we don’t have normative language for, e.g., MUST do A or B, unless it does C.

Enter Formal Description Techniques (➔ [email-masked])…

https://mailarchive.ietf.org/arch/msg/ietf/WklzIVbUutbBFyWxkSfON8qnVsI/

Not a panacea, but we can still try to get better in using them properly.

Grüße, Carsten

---

From: Joseph Touch <[email-masked]>
Date: Thu, 18 Jun 2020 13:42:10 -0700

> On Jun 18, 2020, at 11:01 AM, Larry Masinter <[email-masked]> wrote:
> 

Needs? Why?

The IETF has no compliance authority anyway. Sure, as an exercise in “yet another doc that might or might not be followed”, but otherwise, to what end?

Joe

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 18 Jun 2020 23:20:55 +0200

> On 2020-06-18, at 22:42, Joseph Touch <[email-masked]> wrote:
> 
> 
> 

Such a need was created in RFC 2026 (Section 4.1.2).

RFC 6410 (which updates RFC 2026), Section 2.2, fixes this to:

   (3) There are no unused features in the specification that greatly
       increase implementation complexity.

That was only 8.5 years ago, so many people still think in RFC 2026 categories here.

Grüße, Carsten

---

From: Joseph Touch <[email-masked]>
Date: Thu, 18 Jun 2020 19:42:52 -0700

> On Jun 18, 2020, at 2:20 PM, Carsten Bormann <[email-masked]> wrote:
> 
> 
> 

Features. The terms MUST, SHOULD, and MAY are not used in the reserved sense in RFC 6140.

And “greatly increase implementation complexity” is vague and impossible to objectively evaluate. There’s no utility in automating the incomplete input to a vague evaluation.

Joe