Subject: RE: [newtrk] Key words to Indicate Requirement Levels

From: "Lars-Erik Jonsson (LU/EAB)" <[email-masked]>
Date: Fri, 30 Apr 2004 13:18:16 +0200

I tend to agree with Brian here, we should not go into
the compliance rat-hole. For our specifications, 2119
is good enough, at least its intentions. However, the
question is how to better make use of 2119, and whether
we should better clarify when and how to use it. Both
the non-specification usage and over-usage of it within
technical specifications makes this tool less valuable.

When used in e.g. requirements documents, I have often
a hard time guessing what is really meant (as a MUST
usage can have some tricky implications), and when
seeing upper-case keywords used in standards-track
RFC's for things that are clearly not required, one
will wonder whether any of these words can be trusted
in the spec.

The fact that 2119 is not very clear about how to use
the terms, versus how to not use the terms, makes it
a less useful tool. And I am of the same opinion as
Harald that we should provide useful tools to document
writers.

/L-E

> -----Original Message-----
> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: den 29 april 2004 20:53
> To: Alex Rousskov
> Cc: Lars-Erik Jonsson (LU/EAB); New Track
> Subject: Re: [newtrk] Key words to Indicate Requirement Levels
> 
> 
> The IETF has always got into endless loops when attempting
> to define compliance. In general, "good enough" is good
> enough. So I don't think we should go one millimetre in
> this direction. 2119 is good enough.
> 
>     Brian
> 
> Alex Rousskov wrote:
> > On Thu, 29 Apr 2004, Lars-Erik Jonsson (LU/EAB) wrote:
> > 
> > 
> >>Alex, could you elaborate your thoughts, what "wider range of
> >>documents" do you mean,
> > 
> > 
> > - Documents that do not talk about specifications for 
> software agents.
> >   For example, a document like RFC 2119 itself that essentially
> >   specifies "behavior" of other specifications. While RFC 2119
> >   definitions do not say "agent" explicitly, the words
> >   "IETF", "specification", "vendor" and a couple of 
> examples let many
> >   people believe that the keywords are only useful for agents. Such
> >   people reinvent the wheel with their own set of keywords or
> >   numbers... Ideally, the scope of the keywords should be explicit
> >   and not limited to agents.
> > 
> > - Non-IETF documents. The RFC says "interpreted in IETF documents".
> >   It should say something like "interpreted in documents claiming
> >   compliance with this specification". This is less important.
> > 
> > 
> >>and in what way do you think the keyword definitions are not good
> >>enough for the tech. spec. purpose?
> > 
> > 
> > They are probably "good enough" in some sense, but I do not want to
> > use that vague metric when it comes to core IETF documents. I would
> > rewrite the definitions so that they are tied to compliance, which
> > should be defined. What happens now is that we put MUST/SHOULD/MAYs
> > and then either say nothing more (most of the time) or invent a
> > compliance model that essentially redefines 
> MUST/SHOULD/MAYs meaning.
> > 
> > Take HTTP, for example. RFC 2616 defines two levels of compliance.
> > Full compliance means all MUSTs and SHOULDs are "satisfied".
> > Conditional compliance means all MUSTs are "satisfied". 
> Thus, one can
> > ignore all SHOULDs and still be compliant. How does one satisfy a
> > SHOULD? Unknown. Note that SHOULD is defined as "there may 
> exist valid
> > reasons", implying that there may not exist any reason and leaving
> > "valid" undefined.
> > 
> > I would argue for formal keyword definitions tied to conformance,
> > something along these lines:
> > 
> > 	MUST -   to claim conformance, the conformance subject
> >                  has to act as prescribed by the requirement
> > 
> >         SHOULD - to claim conformance, the conformance subject
> >                  MUST act as prescribed by the requirement or
> > 	         MUST provide a reason not to act as prescribed
> > 		 by the requirement
> > 
> > 	MAY -    to claim conformance, the conformance subject
> > 	         MUST act as prescribed by the requirement or
> > 		 MUST NOT act as prescribed by the requirement
> > 
> > Note that both SHOULD and MAY are defined in terms of MUST. 
> The above
> > definitions must be polished, of course, but I hope you get 
> the idea.
> > 
> > The document will also contain a section explaining how to claim
> > conformance with it (MUST site a formal "The key words MUST ..."
> > paragraph and SHOULD use capitalized keywords). IESG and RFC Editor
> > can then demand that whenever capitalized keywords are used, the
> > conformance is claimed, and check for conformance when it 
> is claimed.
> > 
> > Alex.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 30 Apr 2004 09:50:52 -0600 (MDT)

On Fri, 30 Apr 2004, Lars-Erik Jonsson (LU/EAB) wrote:

> I tend to agree with Brian here, we should not go into the
> compliance rat-hole.

Any pointers where that rat-hole was identified and discussed?

Note that I am _not_ arguing that IETF should enforce compliance; I am
saying that most IETF standards would be better off if they define
compliance and do so in a uniform way were possible. Currently, most
standards do not define compliance, and those that do make mistakes
while reinventing the wheel.

Moreover, compliance-based definitions may be the only meaningful way
to define 2119 keywords. For example, currently, we define MUST as an
"absolute requirement of the specification with a force relative to
the requirement level of the specification". IMO, that's not even
"good enough" and immediately causes problems with MUST interpretation
in non-trivial contexts.

> For our specifications, 2119 is good enough,

Not sure what you mean by "our". It was not good enough for IETF
specifications I helped to author or had to use. YMMV.

> at least its intentions.

Intentions are irrelevant when it comes to specs. I have no doubts
that RFC 2119 and the vast majority of all other RFCs were written
with good intentions.

> However, the question is how to better make use of 2119, and whether
> we should better clarify when and how to use it. Both the
> non-specification usage and over-usage of it within technical
> specifications makes this tool less valuable.

And the tool itself can be blamed for part of those problems.
Improving the tool will reduce the number of misuses and enforcing the
new tool (where applicable) can virtually eliminate them.

> When used in e.g. requirements documents, I have often a hard time
> guessing what is really meant (as a MUST usage can have some tricky
> implications),

It does not have to be that way. For example, there is nothing tricky
in compliance-based definitions I posted (which should be polished and
accompanied by usage rules, of course).

> and when seeing upper-case keywords used in standards-track RFC's
> for things that are clearly not required, one will wonder whether
> any of these words can be trusted in the spec.

Overspecification is a different problem that cannot be solved by
improving 2119. We already have other tools to solve it, like IESG
review.

> The fact that 2119 is not very clear about how to use the terms,
> versus how to not use the terms, makes it a less useful tool. And I
> am of the same opinion as Harald that we should provide useful tools
> to document writers.

I agree that besides fixing keyword definitions, a few simple but
essential usage rules must be added (and, hopefully, enforced).

For example, a specification MUST use verb keywords with compliance
subjects only (e.g., "server MUST validate X" not "X MUST be valid")
and MUST provide an explicit list of compliance subjects. Good
specification would minimize the number of subjects, of course.

For me, 2119 quality is directly related to IETF spec quality issue. I
hope that there are enough IETF folks whose quality goals are higher
than "good enough", even if published RFCs imply otherwise. We are
producing specifications that are often nearly impossible to comply
with and then are complaining about buggy implementations. In most
cases, it is relatively easy to get the code "running". To get out of
the current buggy dumpster, we should aim higher.

Alex.
.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 30 Apr 2004 09:39:32 -0700

For keywords in technical specifications:

If we are still going to require "multiple independent
interoperable implementations of every feature", one way
to count features is to count each occurrence of
RFC 2119's MUST, MAY, and SHOULD. 

I would suggest we could add a guideline to authors
and reviewers to examine each use of MUST, MAY and SHOULD
in a technical specification, and decide whether the use
could appear, as a feature, in an interoperability report.

This might focus our discussion of RFC 2119 keywords,
too: what else, in the IETF process, could they be used
for, anyway?

Would it be unreasonable to add a NEW set of keywords,
for use in requirements documents (which lay out constraints
on technical specifications)? 

 CRITICAL there is general agreement that this is a critical
          characteristic of any definition ...
IMPORTANT most believe that this is an important characteristic ...
   USEFUL there is general belief that this is a useful feature...
          but that other factors might override;
          a definition that does not provide this element is
          acceptable.

In some cases, these could combine with RFC 2119, e.g.,
"It is IMPORANT that the protocol require that all
 implementations MUST implement TLS for the control link ..."

Larry
-- 
http://larry.masinter.net

.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 30 Apr 2004 11:02:49 -0600 (MDT)

On Fri, 30 Apr 2004, Larry Masinter wrote:

> If we are still going to require "multiple independent interoperable
> implementations of every feature", one way to count features is to
> count each occurrence of RFC 2119's MUST, MAY, and SHOULD.

This would only be remotely possible (in a sane manner) if keywords
are used with compliance subjects or equivalent. For example, counting
"server MUST do X" makes sense but counting "message MUST have X" does
not because the former is a "feature" and the latter is not.

Note that instead of demanding "interoperable implementations" we
should be demanding compliance since interoperability often has
nothing to do with the protocol (early implementations often
misinterpret or underutilize the protocol in the same way). For
example, most HTTP servers cannot handle chunked requests, yet
interoperate with most clients just fine.

> This might focus our discussion of RFC 2119 keywords, too: what
> else, in the IETF process, could they be used for, anyway?

Any normative expression of any requirements in a document. That's
what 2119 scope should be.

> Would it be unreasonable to add a NEW set of keywords, for use in
> requirements documents (which lay out constraints on technical
> specifications)?
>

Why not use MUST/SHOULD/MAY instead? If properly defined, their
meaning in a requirements document would match the above.

>  implementations MUST implement TLS for the control link ..."

The above can already be expressed using existing keywords:

  The protocol SHOULD require on a MUST-level that implementations
  support TLS for the control link

If writing "require on a MUST-level that" is bothersome, it can be
replaced with equivalent formal verbs such as REQUIRE (which would
only be defined, in a separate RFC, for requirements documents):

  The protocol SHOULD REQUIRE implementations to
  support TLS for the control link

Alex.

.