Subject: [bmwg] Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

From: Adam Roach <[email-masked]>
Date: Wed, 07 Jun 2017 15:35:35 -0700

Adam Roach has entered the following ballot position for
draft-ietf-bmwg-vswitch-opnfv-03: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

I think this is a variation on the issue that Alissa calls out, but I'm having
a hard time reconciling:

   It's unlikely that the virtual switch will be the only application
   running on the System Under Test (SUT), so CPU utilization, Cache
   utilization, and Memory footprint should also be recorded for the
   virtual implementations of internetworking functions.

...with...

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT/SUT.

Please add text that clarifies how the metrics that 3.1 says should be recorded
in section 3.1 relate to benchmarking.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 7 Jun 2017 23:20:53 +0000

Hi Adam,
Thanks for your review, please see reply below.
Al (for the co-authors)

> -----Original Message-----
> From: Adam Roach [mailto:[email-masked]]
> Sent: Wednesday, June 07, 2017 6:36 PM
> To: The IESG
> Cc: [email-masked]; Sarah Banks; bmwg-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03:
> (with COMMENT)
> 
...
> I think this is a variation on the issue that Alissa calls out, but I'm
> having a hard time reconciling:
> 
[ACM] 
Ok, this is a somewhat different issue.
There are metrics like CPU, Cache, and Memory Utilization
which are reported by measurement processes running on the 
Device Under Test. These are useful for the user/operations community
to know, and may also be useful if there is a problem encountered 
during testing. However, these metrics cannot be promoted to
"Benchmark" status, because we cannot measure any of them 
external to the DUT or SUT.  In fact, only the most important 
externally observable metrics are designated as Benchmarks
in our consensus process, and BMWG refers to internally measured
metrics as "white-box metrics" to clearly distinguish them
from the others.

>    virtual implementations of internetworking functions.
+    However, internally-measured metrics such as these are not
+    benchmarks; they should be useful for the audience (operations)
+    to know, and may also be useful if there is a problem encountered 
+    during testing.

---

From: Adam Roach <[email-masked]>
Date: Thu, 8 Jun 2017 08:32:19 -0500

Thanks. Your proposed clarification looks good to me. 

/a

> On Jun 7, 2017, at 18:20, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Hi Adam,