Subject: [bmwg] Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03: (with DISCUSS and COMMENT)

From: Alissa Cooper <[email-masked]>
Date: Wed, 07 Jun 2017 14:52:42 -0700

Alissa Cooper has entered the following ballot position for
draft-ietf-bmwg-vswitch-opnfv-03: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Picking up on the thread that started with the Gen-ART review, I'm not clear on
the position of this document when it comes to repeatability. If all of the
parameters listed in Section 3.3 (assuming they apply) are configured and
documented, is it assumed that the benchmarks will be repeatable and comparable
with hardware implementation benchmarks? Or is the position of the document
that the benchmarks are not likely to be repeatable/comparable in some (many?)
cases, given the increase in complexity? Or that more work needs to be done
(outside the specification process) to achieve repeatability? I think this
needs to be more clear in the document.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

I understand the debate about the publication of this document as an IETF
consensus RFC. To me it seems valuable to publish and that this item is within
the WG's charter. I think adding the updates necessary to make the document
up-to-date with the Danube 3.0 release as suggested in the thread with Alvaro
would be valuable.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 7 Jun 2017 22:57:00 +0000

Hi Alissa,
Thanks for your review, please see reply below.
Al (for the co-authors)

> -----Original Message-----
> From: Alissa Cooper [mailto:[email-masked]]
> Sent: Wednesday, June 07, 2017 5:53 PM
> To: The IESG
> Cc: [email-masked]; Sarah Banks; bmwg-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03:
> (with DISCUSS and COMMENT)
...
> with hardware implementation benchmarks?
[ACM] 
Of course, it has been the goal of the project to produce 
repeatable results, and a large set of the parameters
believed to be critical was provided so that the benchmarking
community could better appreciate the increase in configuration
complexity inherent in this work. So, yes, this set was assumed 
sufficient for the infrastructure in use by the VSPERF project
to obtain repeatable results from test-to-test. 
>>> We could add summary above as a new paragraph in section 3.3.

Benchmark Comparability between virtual and physical/hardware 
implementations of equivalent functions will likely place more
detailed and exact requirements on the *testing systems*
(in terms of stream generation, algorithms to search 
for max values, and their configurations of course). 
This is another area for standardization to appreciate,
now that we have a few years testing experience.
However, the is a topic for a future draft.

> Or is the position of the document
> (outside the specification process) to achieve repeatability?
[ACM] 
Thanks to some server re-arrangements in the lab that hosts
the VSPERF project, we are able to evaluate our long-term
result repeatability and repeatability across two similar
hardware platforms in different buildings. I'm still processing
some of the results we will present next week at the OPNFV Summit,
but the cross hardware platform results look quite consistent,
even with a benchmark where Scott Bradner experienced some
repeatability problems in his lab back in the 90’s.

> I think this needs to be more clear in the document.
[ACM] 
If the >>>paragraph<<< accomplishes that for you,
with some minor tweaks to word it less like a reply, let me know.

[ACM] 
Great, I've located the list of tests added in Colorado and Danube,
so I'll add them in section 5, and update the release info accordingly.

---

From: Alissa Cooper <[email-masked]>
Date: Thu, 8 Jun 2017 07:36:50 -0400

Hi Al,

> On Jun 7, 2017, at 6:57 PM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Hi Alissa,

I think both of your first two paragraphs above make useful points for inclusion in the draft, so if you’re able to write up some text that incorporates them, that would resolve my issue.

Thanks,
Alissa

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 8 Jun 2017 12:19:09 +0000

> Sent: Thursday, June 08, 2017 7:37 AM
> To: MORTON, ALFRED C (AL)
> 
> Hi Al,
> 
...
> I think both of your first two paragraphs above make useful points for
> inclusion in the draft, so if you’re able to write up some text that
> incorporates them, that would resolve my issue.
[ACM] 
OK, the paragraph beginning 
"Benchmark Comparability between virtual and physical/hardware..."
seems to fit best in section 3.1 (Comparison with Physical Network Functions).

Al

...

---

From: Warren Kumari <[email-masked]>
Date: Thu, 8 Jun 2017 14:55:27 -0400

Hi there,

The authors have just integrated these changes. I think that they
address your concern, and so I asked them to post a new version -
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

Please let me / them know if they correctly address your DISCUSS.
W

On Thu, Jun 8, 2017 at 7:36 AM, Alissa Cooper <[email-masked]> wrote:
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Benoit Claise <[email-masked]>
Date: Fri, 9 Jun 2017 09:59:12 +0200

Dear authors,

The new version reads way better.

Thank you.

Regards, B.