Subject: [bmwg] Murray Kucherawy's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

From: Murray Kucherawy via Datatracker <[email-masked]>
Date: Wed, 02 Feb 2022 21:24:26 -0800

Murray Kucherawy has entered the following ballot position for
draft-ietf-bmwg-ngfw-performance-13: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
for more information about how to handle DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I may be wandering into unfamiliar territory here, i.e., how benchmarking specs
are typically written, but this is sufficiently confusing that I'd like to
discuss it.

I note that RFC 3511, which this document obsoletes, didn't cite RFC 2119 (BCP
14) but rather defined those same key words on its own.  Then it used SHOULD
rather liberally, in a way that seems kind of peculiar to me (especially
compared to the text of Section 6 of RFC 2119).  Do any of them matter to the
outcome of the benchmark being constructed or executed?  If so and they would
spoil the test, shouldn't they be MUSTs?  If not, why include them?  Or in the
alternative, why might I, as someone setting up a test, legitimately do
something contrary to the SHOULD in each case (which "SHOULD" expressly
permits)?

This document does cite BCP 14 directly, and then seems to take that curious
pattern to the next level.  Among the 130+ SHOULDs in here, I'm particularly
confused by stuff like this in Section 4.3.1:

   This section specifies which parameters SHOULD be considered while
   configuring clients using test equipment.

I have no idea what this means to the test.  If I've simply thought about these
parameters, have I met the burden here?

This in Section 4.3.1.1 ("TCP Stack Attributes") seems an odd thing to have to
stipulate:

  The client SHOULD initiate and close TCP connections.

Then Section 7.1.3, which contains subsections about each of the test
parameters for the benchmark described in Section 7.1, consists of this text:

   In this section, the benchmarking test specific parameters SHOULD be
   defined.

As I read it, this is a self-referential SHOULD about this document!  I'm very
confused.  This happens again in Section 7.2.3, 7.3.3, etc., up to 7.9.3, and
even Appendix A.3.  I think in each case you just want:

   This section defines test-specific parameters for this benchmark.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Nits not yet mentioned by others:

Section 4.2:

* "... users SHOULD configure their device ..." -- s/device/devices/ (unless
all users share one device)

Section 6.3:

* "The value SHOULD be expressed in millisecond." -- s/millisecond/milliseconds/

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Thu, 10 Feb 2022 23:52:06 +0100

Dear Murray,

Apologies for the late response to your review.

Upon re-reading the draft, we agree that the number of SHOULD statements 
is indeed quite high.  In our defense :-), we wanted to create 
well-defined benchmarking methodology that does not leave uncertainty 
(or allow compliant but unrealistic behavior that would give the user an 
unfair advantage in benchmarking comparisons).  Anyway, throughout the 
discussions in 20 drafts over three years, some of these statements got 
changed in a way that their intended meaning got distorted.

With help from Al Morton, we concluded that the SHOULD statements in the 
document seem to fall into a couple of categories:

*Category*
	*Explanation*
	*Suggested Authors' Action*
Conditional
	a specific way is recommended, but other more or less harmful ways 
would also be possible
	Review text to ensure that alternatives and their consequences are 
described explicitly
Aspirational
	recommending that the industry will adopt a specific behavior or 
methodology
	Keep
Pleonastic
	statements that are without viable technical alternative
	Eliminate and transfer into indicative text form
Hidden MUSTs
	editors frowned upon the use of MUST and chose SHOULD instead
	Convert into explicit MUST?

Q1) Do you agree with this classification and the suggested actions?

Q2) Do you recommend that we go through each of the SHOULDs in the whole 
document and modify if needed, or focus only on the explicit occurrences 
that you quoted in your review?

Q3) Should we state explicit "MUST" in category 4 cases -- do you think 
this is permissible for an informational RFC -- or do you have any other 
suggested action?

Best regards, Carsten

Am 2/3/2022 um 6:24 AM schrieb Murray Kucherawy via Datatracker:
> Please refer tohttps://www.ietf.org/blog/handling-iesg-ballot-positions/
-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked],https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 10 Feb 2022 22:03:53 -0800

> Apologies for the late response to your review.

Not at all, I appreciate the attention and thought you're giving to it.

> is indeed quite high.  In our defense :-), we wanted to create well-defined
> benchmarking methodology that does not leave uncertainty (or allow
> compliant but unrealistic behavior that would give the user an unfair
> advantage in benchmarking comparisons).  Anyway, throughout the discussions
> in 20 drafts over three years, some of these statements got changed in a
> way that their intended meaning got distorted.
>

I am by no means an expert on crafting test documents.  I am instead going
by the text of RFC 2119 which guides our use of "SHOULD" and friends to
describe requirements for protocol interoperability.  As applied to a test
description, I have to extrapolate a bit, but I would imagine use of them
as follows:

"SHALL" or "MUST" means if you don't do exactly what this says when
constructing your test or test framework, the test is ruined and its
results cannot be meaningfully applied.

"RECOMMENDED" or "SHOULD" means you really ought to do this (perhaps
because it will yield a simulation closer to reality), but if you don't,
the test is still considered viable.  And oh, by the way, here are some
reasons why you might want or need to deviate.

"OPTIONAL" or "MAY" means the test constructor has full discretion here;
whatever you choose is very unlikely to affect the meaningful outcome of
the test.  (The same would go for any aspect or parameter of the test not
specifically addressed by the document.). One thing to consider is whether
such a parameter needs to be described at all.  Maybe you want to
explicitly say "We thought about this, and yes, it really is optional" as
opposed to omitting it, where the reader might wonder if this aspect of the
test got any consideration.

With so many SHOULDs in there, however, I begin to wonder about just how
much variability this test framework has.

Ergo:

> a specific way is recommended, but other more or less harmful ways would
> also be possible
> First, nice work on such classifications.

> Q1) Do you agree with this classification and the suggested actions?
For "Conditional", yes.

For "Aspirational", I suggest not using the key words at all, but instead
including a section (perhaps early in the document) addressed to the
industry advising the methodology you think it should adopt (and why), an
example of which is the test framework described by this document.

For "Pleonastic", if by that explanation you mean they really ought to be
MUSTs, then I would make that conversion.

For "Hidden MUST", I would consider why there was the preference for the
weaker assertion.  Some of them might deserve MUST, some might deserve
something else, some might rightly remain SHOULD.  But I think given your
chart above, you're on the right track to thinking about these already.

> that you quoted in your review?
I would recommend going through them all.  I found them by typing SHOULD
into the search tool of my browser and then just selecting them in
sequence.  There are indeed a lot, but you could perhaps divide and
conquer.  The good news is that there are a lot of them that are just
fine.  :-)

The ones I cited in my DISCUSS stood out to me as thoroughly confusing and
I would definitely reconsider them.  But I can give you some other examples
where I paused while reviewing:

* the first SHOULD in Section 4.2 -- why isn't that a MUST?  is the test
still viable if I don't do that?  if it stays SHOULD, what deviations might
be acceptable?
* the SHOULD in Section 4.2.1 refers to some minima (500 CVEs in the last
10 years), but why those numbers?  what if I only do 100?  what about one
or none?  what about 499?
* in 4.3.1.2 you have a SHOULD covering a bullet list, and a SHOULD on the
first bullet; this second SHOULD is thus redundant to the first

On the other hand, for example, all but the last SHOULD in 4.3.1.1, and all
of the ones in 4.3.1.3, look perfectly reasonable to me (modulo your action
under "Conditional" above).

> this is permissible for an informational RFC -- or do you have any other
I think since you've cited BCP 14 (RFC 2119 and RFC 8174), you may as well
use them.  It is a little weird for an Informational document to read like
a protocol specification, but this is hardly the first document we've seen
like this.

For "Hidden MUSTs", I think the action is to figure out why the editors
balked at MUST, re-evaluate, and commit by either making it a MUST, or
leaving it SHOULD but treating it as you say you will for "Conditional".

You bring up an interesting point though.  One of my mentors demonstrated
to me that it's possible to write normative text without using our
MUST/SHOULD/MAY key words.  For example, here are two instances of
normative text, arguably of equal force, copied from your Section 4:

OLD:

   Test setup defined in this document applies to all benchmarking tests
   described in Section 7
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-13#section-7>.
The test setup MUST be contained within an
   Isolated Test Environment (see Section 3 of [RFC6815]
<https://datatracker.ietf.org/doc/html/rfc6815#section-3>).

NEW:

   Test setup defined in this document applies to all benchmarking tests
   described in Section 7
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-13#section-7>.
A test setup compliant with this specification
   is contained within an Isolated Test Environment (see Section 3 of
[RFC6815] <https://datatracker.ietf.org/doc/html/rfc6815#section-3>).

So if you want to avoid having to pick one of them and slot it within the
definitions we use for those key words, you definitely have some latitude
with your prose without losing the force of the message.

I hope this has been helpful.  Please let me know if I can provide further
guidance.

-MSK

---

From: Scott Bradner <[email-masked]>
Date: Fri, 11 Feb 2022 07:08:23 -0500

speaking as the editor of RFC 2119

nice chart of how to misuse SHOULD but - imo - SHOULD is a MUST with an escape clause

I do not think it is useful to use SHOULD without specifically saying what the escape clause is - 
i.e. specifically say when its OK to not act as if it was a MUST

so, if you cannot say when the SHOULD should not be a MUST then it should be a MUST or a MAY

Scott

(if I had to do it all over again I would not have included SHOULD/SHOULD NOT)

> On Feb 11, 2022, at 1:03 AM, Murray S. Kucherawy <[email-masked]> wrote:
> 
> Upon re-reading the draft, we agree that the number of SHOULD statements is indeed quite high.  In our defense :-), we wanted to create well-defined benchmarking methodology that does not leave uncertainty (or allow compliant but unrealistic behavior that would give the user an unfair advantage in benchmarking comparisons).  Anyway, throughout the discussions in 20 drafts over three years, some of these statements got changed in a way that their intended meaning got distorted. 
> 
> 
> I am by no means an expert on crafting test documents.  I am instead going by the text of RFC 2119 which guides our use of "SHOULD" and friends to describe requirements for protocol interoperability.  As applied to a test description, I have to extrapolate a bit, but I would imagine use of them as follows:
> 
> "SHALL" or "MUST" means if you don't do exactly what this says when constructing your test or test framework, the test is ruined and its results cannot be meaningfully applied.
> 
> "RECOMMENDED" or "SHOULD" means you really ought to do this (perhaps because it will yield a simulation closer to reality), but if you don't, the test is still considered viable.  And oh, by the way, here are some reasons why you might want or need to deviate.
> 
> "OPTIONAL" or "MAY" means the test constructor has full discretion here; whatever you choose is very unlikely to affect the meaningful outcome of the test.  (The same would go for any aspect or parameter of the test not specifically addressed by the document.). One thing to consider is whether such a parameter needs to be described at all.  Maybe you want to explicitly say "We thought about this, and yes, it really is optional" as opposed to omitting it, where the reader might wonder if this aspect of the test got any consideration.
> 
> With so many SHOULDs in there, however, I begin to wonder about just how much variability this test framework has.
> 
> Ergo:
> 
> 
> With help from Al Morton, we concluded that the SHOULD statements in the document seem to fall into a couple of categories: 
> 
> Category
> Explanation
> Suggested Authors' Action
> Conditional
> a specific way is recommended, but other more or less harmful ways would also be possible
> Review text to ensure that alternatives and their consequences are described explicitly
> Aspirational
> recommending that the industry will adopt a specific behavior or methodology
> Keep
> Pleonastic
> For "Aspirational", I suggest not using the key words at all, but instead including a section (perhaps early in the document) addressed to the industry advising the methodology you think it should adopt (and why), an example of which is the test framework described by this document.
> 
> For "Pleonastic", if by that explanation you mean they really ought to be MUSTs, then I would make that conversion.
> 
> For "Hidden MUST", I would consider why there was the preference for the weaker assertion.  Some of them might deserve MUST, some might deserve something else, some might rightly remain SHOULD.  But I think given your chart above, you're on the right track to thinking about these already.
> Q2) Do you recommend that we go through each of the SHOULDs in the whole document and modify if needed, or focus only on the explicit occurrences that you quoted in your review?
> 
> I would recommend going through them all.  I found them by typing SHOULD into the search tool of my browser and then just selecting them in sequence.  There are indeed a lot, but you could perhaps divide and conquer.  The good news is that there are a lot of them that are just fine.  :-)
> 
> The ones I cited in my DISCUSS stood out to me as thoroughly confusing and I would definitely reconsider them.  But I can give you some other examples where I paused while reviewing:
> 
> * the first SHOULD in Section 4.2 -- why isn't that a MUST?  is the test still viable if I don't do that?  if it stays SHOULD, what deviations might be acceptable?
> * the SHOULD in Section 4.2.1 refers to some minima (500 CVEs in the last 10 years), but why those numbers?  what if I only do 100?  what about one or none?  what about 499?
> * in 4.3.1.2 you have a SHOULD covering a bullet list, and a SHOULD on the first bullet; this second SHOULD is thus redundant to the first
> 
> On the other hand, for example, all but the last SHOULD in 4.3.1.1, and all of the ones in 4.3.1.3, look perfectly reasonable to me (modulo your action under "Conditional" above).
> Q3) Should we state explicit "MUST" in category 4 cases -- do you think this is permissible for an informational RFC -- or do you have any other suggested action?
> 
> I think since you've cited BCP 14 (RFC 2119 and RFC 8174), you may as well use them.  It is a little weird for an Informational document to read like a protocol specification, but this is hardly the first document we've seen like this.
> 
> For "Hidden MUSTs", I think the action is to figure out why the editors balked at MUST, re-evaluate, and commit by either making it a MUST, or leaving it SHOULD but treating it as you say you will for "Conditional".
> 
> You bring up an interesting point though.  One of my mentors demonstrated to me that it's possible to write normative text without using our MUST/SHOULD/MAY key words.  For example, here are two instances of normative text, arguably of equal force, copied from your Section 4:
> 
> OLD:
>    described in 
> Section 7
> .  The test setup MUST be contained within an
>    Isolated Test Environment (see 
> Section 3 of [RFC6815]).
> NEW:
>    described in 
> Section 7.  A test setup compliant with this specification
>    is contained within an Isolated Test Environment (see Section 3 of [RFC6815]).
> So if you want to avoid having to pick one of them and slot it within the definitions we use for those key words, you definitely have some latitude with your prose without losing the force of the message.
> 
> I hope this has been helpful.  Please let me know if I can provide further guidance.
> 
> -MSK
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 11 Feb 2022 12:22:09 +0000

> On 11 Feb 2022, at 12:08, Scott Bradner <[email-masked]> wrote:
> 

Hi Scott

The problem with that is it requires perfect foresight on the exceptions.

I have always taken SHOULD to mean that you MUST do this unless you have a good reason not to do this and understand the consequences for your system in its deployment environment.

In other words I think we need to trust the engineer implementing the system to apply good judgement when they ignore the MUST element of the SHOULD.

- Stewart

---

From: Scott Bradner <[email-masked]>
Date: Fri, 11 Feb 2022 08:44:13 -0500

if every implementor were so disciplined it would not be so bad but, I fear, far too often
an implementor views a SHOULD as something that is not important enough to be a MUST
so I do not have to worry about it

at least the document author should say what could go wrong if its not done 

Scott

ps document authors should take to heart the following from 2119

6. Guidance in the use of these Imperatives

   Imperatives of the type defined in this memo must be used with care
   and sparingly.  In particular, they MUST only be used where it is
   actually required for interoperation or to limit behavior which has
   potential for causing harm (e.g., limiting retransmisssions)  For
   example, they must not be used to try to impose a particular method
   on implementors where the method is not required for
   interoperability.

---

From: Michael Richardson <[email-masked]>
Date: Fri, 11 Feb 2022 09:06:06 -0500

> i.e. specifically say when its OK to not act as if it was a MUST

I've gotten caught by this regularly.

I don't want to write MUST because if feels rude to be so demanding, and I
can envision *some* escape clause, I'm just not yet sure what it is yet.
That's wrong on my part.

    > (if I had to do it all over again I would not have included SHOULD/SHOULD NOT)

It's hard to imagine some other word that would allow for the escape clause.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: John C Klensin <[email-masked]>
Date: Fri, 11 Feb 2022 11:13:10 -0500

--On Friday, February 11, 2022 12:22 +0000 Stewart Bryant
<[email-masked]> wrote:

>> I do not think it is useful to use SHOULD without
>> specifically saying what the escape clause is -  i.e.
>> specifically say when its OK to not act as if it was a MUST
> 
> Hi Scott
> 
> I have always taken SHOULD to mean that you MUST do this
> unless you have a good reason not to do this and understand
> the consequences for your system in its deployment environment.
> 
> In other words I think we need to trust the engineer
> implementing the system to apply good judgement when they
> ignore the MUST element of the SHOULD.

Stewart,

As Scott probably remembers, I've advocated several times  --
with the IESG, IAB, and with the RFC Editor -- for a rule that
is a little more flexible than the one Scott suggests above but
that treats "good judgment" differently.  That suggestion has
been something like the following:

If SHOULD is used, then it must be accompanied by at least one
of:
 (1) A general description of the character of the
	exceptions and/or in what areas exceptions are likely to
	arise.  Examples are fine but, except in plausible and
	rare cases, not enumerated lists. 
 (2) A statement about what should be done, or what the
	considerations are, if the "SHOULD" requirement is not
	met. 
 (3) A statement about why it is not a MUST.

The first option was not just because of your "perfect
foresight" comment above but because attempts to enumerate all
cases can easily lead into "what is not explicitly forbidden is
permitted" thinking which is never good for standards.  And,
especially if there is any ambiguity, all three need to be read
through the lens of the original intended interpretation of the
Robustness Principle.

I have no opinion about what should be done for the case of this
particular specification but, because this is not the first time
a problem like this has arisen _and_ because there have
occasionally been objections from within the IESG to documents
that discuss what to do if an implementation does not follow a
given specification, I would encourage the IESG and the
community to think about the issue more broadly.

   john

---

From: <[email-masked]>
Date: Fri, 11 Feb 2022 11:15:31 -0500

Thanks everyone for your input and comments. They are helpful to the
authours.

Brian

-----Original Message-----
From: John C Klensin <[email-masked]> 
Sent: Friday, February 11, 2022 11:13 AM
To: Stewart Bryant <[email-masked]>; Scott Bradner <[email-masked]>
Cc: [email-masked]; IETF <[email-masked]>; The IESG <[email-masked]>;
[email-masked]; [email-masked]
Subject: Re: [bmwg] Murray Kucherawy's Discuss on
draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

--On Friday, February 11, 2022 12:22 +0000 Stewart Bryant
<[email-masked]> wrote:

>> I do not think it is useful to use SHOULD without specifically saying 
>> what the escape clause is -  i.e.
> I have always taken SHOULD to mean that you MUST do this unless you 
> have a good reason not to do this and understand the consequences for 
> your system in its deployment environment.
> 
> In other words I think we need to trust the engineer implementing the 
> system to apply good judgement when they ignore the MUST element of 
> the SHOULD.

Stewart,

As Scott probably remembers, I've advocated several times  -- with the IESG,
IAB, and with the RFC Editor -- for a rule that is a little more flexible
than the one Scott suggests above but that treats "good judgment"
differently.  That suggestion has been something like the following:

The first option was not just because of your "perfect foresight" comment
above but because attempts to enumerate all cases can easily lead into "what
is not explicitly forbidden is permitted" thinking which is never good for
standards.  And, especially if there is any ambiguity, all three need to be
read through the lens of the original intended interpretation of the
Robustness Principle.

I have no opinion about what should be done for the case of this particular
specification but, because this is not the first time a problem like this
has arisen _and_ because there have occasionally been objections from within
the IESG to documents that discuss what to do if an implementation does not
follow a given specification, I would encourage the IESG and the community
to think about the issue more broadly.

   john

---

From: Scott Bradner <[email-masked]>
Date: Fri, 11 Feb 2022 11:32:12 -0500

I like John's suggestion - it gets to the issue I am worried about

Scott

> On Feb 11, 2022, at 11:13 AM, John C Klensin <[email-masked]> wrote:
> 
> 
>

---

From: Barry Leiba <[email-masked]>
Date: Fri, 11 Feb 2022 11:34:41 -0500

I agree with what Scott, Michael, and John have said, and agree with
Scott that we should not have "SHOULD", but it's too late to change
that.  The sense that "SHOULD" is really there for would be adequately
covered by "MUST" with an explicit escape clause documented: "You MUST
do this unless <escape explicitly specified>."

For example, "You MUST include the exact length value unless computing
it is so time-consuming that it would be likely to exceed client
timeout periods."  (In contrast to "You MUST include the exact length
value even if it is time-consuming to compute."  And even better, "You
MUST include the exact length value unless computing it is so
time-consuming that it would be likely to exceed client timeout
periods.  Clients rely on the length value <for purpose>, and
inaccurate length values can cause <these types of problems>."

Our specs would be much better if we made every effort to specify
SHOULD conditions that way instead.

Barry

---

From: Michael Richardson <[email-masked]>
Date: Fri, 11 Feb 2022 13:45:16 -0500

> on implementors where the method is not required for

My impression from the ~10 RFCs I've gotten out in the past three years
is that IESG of recent has been pushing for all SHOULDs that do not have
clear escape clauses to be MUSTs.
I guess if there is no interoperation issue at all, then they become MAYs.
(Or even, why is it even there?)

That's not consistent with the above text.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: John C Klensin <[email-masked]>
Date: Fri, 11 Feb 2022 18:56:20 -0500

Barry, 

While I don't think our position are far apart, I'm not sure I
agree, partially because of the concern I (and Stewart) tried to
express earlier about perfectd foresight and enumerating all
possible cases.   Now, if every statement of the sort you
suggest came with an additional clause equivalent to "unless you
run into a case that we didn't anticipate at all", it would end
up fairly close to what I suggested.   More generally, I think
the IETF and its predecessors have always -- both before and
after we started using the term "standard" regularly -- been a
bit ambivalent and conflicted about clear conformance
statements.  Some of that ambivalence, IMO, comes from the
differences between trying to write specs for implementers who
want to work together in good faith to produce interoperable
implementations, using those specs for guidance versus writing
specs for regulators and contract-enforcers who are concerned
about specifications that can identify implementations that try
to cut corners or offer variations to gain, e.g., competitive
advantage.  

I remember some discussions very early on, long before OSI
turned into a poster child for the other point of view, about
how it was better to reach agreements on a design or protocol,
document it mostly for the reference of those who agreed and
those who might follow in their footsteps, and then move on
rather than spending weeks, months, or longer trying to pin down
every possible detail.  I vaguely even remember a Padlipsky rant
or two on the subject.  To some extent, the Robustness Principle
came out of that thinking too: we might more precisely restate
at least part of the intent today, in far more words, as "if you
are not very confident you know what the spec intended and that
others will read it the same way you do, then be conservative in
what you send..."

So, I think that, unless we have reached the stage where we want
and expect our specifications to cover every detail and be exact
about it, SHOULD serves real value... but only if we require
explanations of why it is present rather than using it as "MUST,
but maybe not" or "MUST unless you don't feel like it".

best,
  john

--On Friday, February 11, 2022 11:34 -0500 Barry Leiba
<[email-masked]> wrote:

> I agree with what Scott, Michael, and John have said, and
> agree with Scott that we should not have "SHOULD", but it's
> too late to change that.  The sense that "SHOULD" is really
> there for would be adequately covered by "MUST" with an
> explicit escape clause documented: "You MUST do this unless
> <escape explicitly specified>."
> 
> For example, "You MUST include the exact length value unless
> computing it is so time-consuming that it would be likely to
> exceed client timeout periods."  (In contrast to "You MUST
> include the exact length value even if it is time-consuming to
> compute."  And even better, "You MUST include the exact length
> value unless computing it is so time-consuming that it would
> be likely to exceed client timeout periods.  Clients rely on
> the length value <for purpose>, and inaccurate length values
> can cause <these types of problems>."
> 
> Our specs would be much better if we made every effort to
> specify SHOULD conditions that way instead.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 15 Feb 2022 19:56:58 -0600

I also agree with Barry's note further down in this thread, but wanted to
talk about one of John's points here.

>  (3) A statement about why it is not a MUST.

I never understood how a receiver of  a packet that did not satisfy what I
might call a "naked SHOULD" (with none of these coverings) was supposed to
know what to do next.

Since the receiver can't count on the SHOULD requirement being satisfied,
it seems like the least a specification could do, was to provide guidance
about what to do next, as in (2).

So I think (2) is even more helpful than the helpful (1) and (3).

Best,

Spencer

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Feb 2022 00:16:01 -0500

--On Tuesday, February 15, 2022 19:56 -0600 Spencer Dawkins at
IETF <[email-masked]> wrote:

> I also agree with Barry's note further down in this thread,
> but wanted to talk about one of John's points here.
> 
>>         exceptions and/or in what areas exceptions are likely
>>         to arise.  Examples are fine but, except in plausible
>>         and rare cases, not enumerated lists.
> I never understood how a receiver of  a packet that did not
> satisfy what I might call a "naked SHOULD" (with none of these
> coverings) was supposed to know what to do next.
> 
> Since the receiver can't count on the SHOULD requirement being
> satisfied, it seems like the least a specification could do,
> was to provide guidance about what to do next, as in (2).
> 

I think it depends on the protocol.  For network  protocols near
the bottom of the stack, it will often be the case that things
either work or they don't and it seems to me that Scott's
observation -- which I might restate as "SHOULD is almost never
helpful" -- is correct or nearly so and the right answer might
be closer to (3) or maybe (1).  Closer to the application layer,
where interoperation with non-Internet systems may be important,
and maybe for Security and other issues, edge cases where things
usually work may be more common, it may be less possible to
precisely spell out all of the possibilities, and SHOULD (and/or
the Robustness Principle) become more useful and possibly
necessary, at least if we wish to avoid very long and complex
standards.

And that said, when I think of the places where I have used
SHOULD, far more of them would be best associated with the type
of explanation associated with (2) than with the other two cases.

best,
   john