Subject: [Last-Call] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03

From: Mališa Vučinić via Datatracker <[email-masked]>
Date: Tue, 15 Dec 2020 03:29:40 -0800

Reviewer: Mališa Vučinić
Review result: Ready

I reviewed this document as part of the Security Directorate's ongoing effort
to review all IETF documents being processed by the IESG. These comments were
written primarily for the benefit of the Security Area Directors. Document
authors, document editors, and WG chairs should treat these comments just like
any other IETF Last Call comments.

Thank you for this well-written document, it was a pleasure to read and I think
it is ready to proceed. Since the document updates RFC2544 benchmarking
procedure for estimating the buffer time of a Device Under Test (DUT), it does
not raise any security issues. Security Considerations section is quite clear
and it stresses that these tests are performed in a lab environment.

I do have a question regarding the last paragraph of the Security
Considerations on special capabilities of DUTs for benchmarking purposes.
Currently, the sentence reads: "Special capabilities SHOULD NOT exist in the
DUT/SUT specifically for benchmarking purposes." Why is this a SHOULD NOT and
not a MUST NOT? Could you give an example when such special capabilities in a
DUT are appropriate?

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 15 Dec 2020 13:45:16 +0000

Hi Mališa, 
thanks for your review, please see below for one reply to your question (acm].
Al

> -----Original Message-----
> From: bmwg [mailto:[email-masked]] On Behalf Of Mališa Vucinic via
> Datatracker
> Sent: Tuesday, December 15, 2020 6:30 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; draft-ietf-bmwg-b2b-
> [email-masked]
> Subject: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03
> 
> Reviewer: Mališa Vučinić
> Review result: Ready
> 
[acm] 
We can only make a strong recommendation in this area. As testers/benchmarkers are often independent from the DUT developers and conduct testing external to the DUT, we assume honesty among other parties but we cannot require it. If someone constructed a DUT that recognized test conditions and operated differently to perform better somehow, our tests would measure the intended "better" performance. It takes a special/additional test effort to prove that a DUT has "designed to the test" (consider Volkswagen and fuel efficiency testing [0]).

We simply do not have any authority in this matter, but we can let all parties know that gaming the test can be discovered and reported (albeit with more testing that we do not describe).

[0] https://www.consumerreports.org/fuel-economy-efficiency/volkswagen-used-special-software-to-exaggerate-fuel-economy/
 
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
> !BhdT!1JFeLsENzMU-ew89jxmJKxfp4wj5Zo3AZ6V8iULU3hWAentH1dymqJmDOvw7$

---

From: Mališa Vučinić <[email-masked]>
Date: Tue, 15 Dec 2020 15:20:59 +0100

Hi Al,

Thanks, that is clear. I think that discussing the assumption of honesty among the parties involved in benchmarking  would be a useful addition to the Security Considerations section in the draft.

Mališa

On 15/12/2020 14:45, "MORTON, ALFRED C (AL)" <[email-masked]> wrote:

    Hi Mališa, 
    
    [acm]

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 15 Dec 2020 19:22:19 +0000

Hi Mališa,
please see below...

> From: Mališa Vučinić [mailto:[email-masked]]
> Sent: Tuesday, December 15, 2020 9:21 AM
> To: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
> Subject: Re: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-frame-03
> 
> Hi Al,
> 
> Thanks, that is clear. I think that discussing the assumption of honesty
> among the parties involved in benchmarking  would be a useful addition to
> the Security Considerations section in the draft.
[acm] 

I don't mind explaining the requirement using the term "honesty", but I can only imagine raised eyebrows and subsequent DISCUSS/comments if we try to assert a need for/assumption of honesty anywhere in the memo.

Do you have suggested wording?

Do others have opinions whether or not this is needed?

thanks,
Al

>     > I reviewed this document as part of the Security Directorate's
> ongoing
>     > effort
>     > to review all IETF documents being processed by the IESG. These
> comments
>     > were
>     > authors, document editors, and WG chairs should treat these comments
> just
>     > like
>     > Thank you for this well-written document, it was a pleasure to read
> and I
>     > think
>     > procedure for estimating the buffer time of a Device Under Test
> (DUT), it
>     > does
>     > not raise any security issues. Security Considerations section is
> quite
>     > clear
>     > Currently, the sentence reads: "Special capabilities SHOULD NOT
> exist in
>     > the
>     > DUT/SUT specifically for benchmarking purposes." Why is this a
> SHOULD NOT
>     > and
>     > not a MUST NOT? Could you give an example when such special
> capabilities
>     > in a
>     > DUT are appropriate?
>     [acm]
>     We can only make a strong recommendation in this area. As
> testers/benchmarkers are often independent from the DUT developers and
> conduct testing external to the DUT, we assume honesty among other parties
> but we cannot require it. If someone constructed a DUT that recognized
> test conditions and operated differently to perform better somehow, our
> tests would measure the intended "better" performance. It takes a
> special/additional test effort to prove that a DUT has "designed to the
> test" (consider Volkswagen and fuel efficiency testing [0]).
> 
>     We simply do not have any authority in this matter, but we can let all
> parties know that gaming the test can be discovered and reported (albeit
> with more testing that we do not describe).
> 
>     [0] https://urldefense.com/v3/__https://www.consumerreports.org/fuel-
> economy-efficiency/volkswagen-used-special-software-to-exaggerate-fuel-
> economy/__;!!BhdT!0KS_VCF5ZQfIGkVyPLoJXuAxdcoS3-
> xJTE0LoKZPWuSiHjQZM1u0H9M36YXByCk$
> 
>     >
>     >
>     >

---

From: Mališa Vučinić <[email-masked]>
Date: Wed, 16 Dec 2020 13:21:40 +0100

Al,

I don't have a strong opinion on using the term "honesty" here. How about this phrasing, just before the last paragraph in Security Considerations:

The DUT developers are commonly independent from the personnel and institutions conducting the benchmarking.
The DUT developers might have incentives to alter the performance of the DUT if the test conditions are detected.
Procedures described in this document are not designed to detect such activity.
Additional testing, outside of the scope of this document, is needed and has been successfully used in the past to discover such malpractices.

Mališa

On 15/12/2020 20:22, "MORTON, ALFRED C (AL)" <[email-masked]> wrote:

    Hi Mališa,
    please see below...
    
    [acm] 

    
    
    thanks,
    Al

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 16 Dec 2020 17:44:04 +0000

Hi Mališa,

Thanks for your proposed wording, it seems sufficiently neutral and with a few small tweaks, WFM.

I see that Roman's COMMENT also supports this additional text.

So, consider it part of the next version, and thanks for your help!
Al

> Sent: Wednesday, December 16, 2020 7:22 AM
> I don't have a strong opinion on using the term "honesty" here. How about
> this phrasing, just before the last paragraph in Security Considerations:
> 
> The DUT developers are commonly independent from the personnel and
> institutions conducting the benchmarking.
> Additional testing, outside of the scope of this document, is needed and
> has been successfully used in the past to discover such malpractices.
> 
> Mališa
> 
>     I don't mind explaining the requirement using the term "honesty", but
> I can only imagine raised eyebrows and subsequent DISCUSS/comments if we
> try to assert a need for/assumption of honesty anywhere in the memo.
> 
>     >     > Subject: [bmwg] Secdir telechat review of draft-ietf-bmwg-b2b-
> frame-
>     > 03
>     >     >
>     >     > Reviewer: Mališa Vučinić
>     >     > Review result: Ready
>     >     >
>     >     > written primarily for the benefit of the Security Area
> Directors.
>     > Document
>     >     > Considerations on special capabilities of DUTs for
> benchmarking
>     > purposes.
>     >     [0]
> https://urldefense.com/v3/__https://www.consumerreports.org/fuel-
>     >     > !BhdT!1JFeLsENzMU-
> ew89jxmJKxfp4wj5Zo3AZ6V8iULU3hWAentH1dymqJmDOvw7$
>     >
>     >
> 
> 
>

---

From: Mališa Vučinić <[email-masked]>
Date: Wed, 16 Dec 2020 18:58:57 +0100

Sounds good, thanks!

Mališa

On 16/12/2020 18:44, "MORTON, ALFRED C (AL)" <[email-masked]> wrote:

    Hi Mališa,