Subject: [bmwg] Benoit Claise's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

From: Benoit Claise <[email-masked]>
Date: Thu, 08 Jun 2017 05:09:16 -0700

Benoit Claise has entered the following ballot position for
draft-ietf-bmwg-vswitch-opnfv-03: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

For sure, the topic of benchmarking of benchmarking virtual switches is
important in the industry today.

Technical Summary: This draft summarizes the progress of the OPNFV project on
virtual switch performance characterization, and notes that the OPNFV work
builds upon the IETF work, particularly the BMWG work within IETF, with an
emphasis on RFC2544.

I like the cross SDO aspects of this draft, as mentioned in the shepherd
writeup: The document itself was reviewed in ETSI NFV, and used as a reference
by at least one of the TST WG and IFA WF specifications (2 references in
total). The document has also been reviewed by several individuals from the
OPNFV community, and NFVRG (IRTF) is aware of the draft.

I would have preferred to see a document that specifies "benchmarking virtual
switches" instead of "benchmarking virtual switches in OPNFV". In other words,
just looking at the abstract, this is completely different to write:

   This memo describes the way to benchmark virtual switch performance.
   The specifications are based on the Open Platform for NFV (OPNFV) project on
   virtual switch performance "VSPERF".

As opposed to:

   This memo describes the progress of the Open Platform for NFV (OPNFV)
   project on virtual switch performance "VSPERF".

Regarding the "describes the progress of", sentences such as ...
    - This project intends to build ...
    - Therefore, this memo begins to describe the additional considerations ...
    - This memo summarizes the progress of the Open Platform for NFV (OPNFV)
    project on virtual switch performance characterization, "VSPERF", through
    the Brahmaputra (second) release [BrahRel].
... lead me to think that this is unfinished work, or that work will anyway
have to be updated.

Again, looking at the scope:

   The primary purpose and scope of the memo is to inform the industry
   of work-in-progress that builds on the body of extensive BMWG
   literature and experience, and describe the extensions needed for
   benchmarking virtual switches.

I was hoping that the primary purpose and scope of the memo would be the
specifications of the virtual switch benchmarking, not to inform about the
work-in-progress.

Reading further, I see text such as:

    Specifications to be included in future updates of the LTD include:

       o  [RFC3918] Methodology for IP Multicast Benchmarking

       o  [RFC4737] Packet Reordering Metrics

or

   When considering characteristics important to "telco" network
   functions, we must begin to consider additional performance metrics.

or

   Tests have been (or will be) designed to collect the metrics below:

On one side, I appreciate that you're publishing what you learned, which should
lead up to revisions in the future. On the other side, I've not been used to
that approach in BMWG. I'll trust the responsible AD regarding the publication
of this document as a kind of intermediary document.

I believe the document would be a better flow (and that some concerns would
disappear) if it would have the following structure
    - benchmarking virtual switches is key in the industry today
    - there are the virtual switches benchmarking specification today
    - we worked on this with OPNFV (VSPERF). Great collaboration.
    - this is a complex and evolving field:
      compared to RFC 2544, we learn that ...
      we realize that this is work in progress ...
    - future plan: we will need to integrate more "stuff" in the benchmark,
    such as RFC3918, RFC4737, additional perf metrics, you-name-it - we'll work
    with OPNFV (and other groups) on this future plan.

Editorial
- Section 3.4. Flow timeout.
You mean the active timeout, section 2.2.23 in RFC 6645?
Should this section refer to RFC6645, or RFC 5470 (section 5.1.1)?

-

   In addition to this, the LTD also re-uses the terminology defined by:

   o  [RFC2285] Benchmarking Terminology for LAN Switching Devices

   o  [RFC5481] Packet Delay Variation Applicability Statement

RFC5481 is already mentioned in the previous bullet list (The base performance
tests in the LTD are based on the pre-existing specifications developed by BMWG
to test the performance of physical switches. These specifications include). So
hopefully, it re-uses the terminology, right?

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 8 Jun 2017 13:45:21 +0000

Hi Benoit,
Thanks for your review. please see replies below.
I really appreciate the specific pointers you've
provided for revisions to make this a more 
appropriate and valuable RFC.

Al (for the co-authors)

> -----Original Message-----
> From: Benoit Claise [mailto:[email-masked]]
> Sent: Thursday, June 08, 2017 8:09 AM
> To: The IESG
> Cc: [email-masked]; Sarah Banks; bmwg-
> [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Benoit Claise's No Objection on draft-ietf-bmwg-vswitch-opnfv-
> 03: (with COMMENT)
> 
...
[ACM] 
Yes.  We should have made some wording changes when this draft
transitioned from the Individual draft to WG item.
It would be great to be able to write your sentence, 
but the WG is not ready to do that.

However, we could say this instead:

This memo describes the contributions of the Open Platform 
for NFV (OPNFV) project on virtual switch performance "VSPERF", 
particularly in the areas of test set-ups and configuration 
parameters for the system under test.

>     - This project intends to build ...
[ACM] 
This project has extended ...

>     - Therefore, this memo begins to describe the additional
[ACM] 
Therefore, this memo describes the additional

>     - This memo summarizes the progress of the Open Platform for NFV (OPNFV)
[ACM] 
This memo describes the contributions of the Open Platform...

>     the Brahmaputra (second) release [BrahRel].
[ACM] 
... through the Danube 3.0 (fourth) release to the chartered work 
of the BMWG (with stable references to their test descriptions). 

> ... lead me to think that this is unfinished work, or that work will anyway
[ACM] 
As I've mentioned in other comments, the initial rush of
test development has transitioned to inclusion of more
test devices, including open source testing software.

But I think with the emphasis on the specific contributions
and stable references, this memo is taking a more solid stand
(as much as anything is in our world where what we know changes fast).
 
> specifications of the virtual switch benchmarking, not to inform about the
[ACM] 
Yes, so let's say this instead:

The primary purpose and scope of the memo is to describe key 
aspects of vswitch benchmarking, particularly in the areas of test 
set-ups and configuration parameters for the system under test, 
and build on the body of extensive BMWG literature and experience.

>        o  [RFC4737] Packet Reordering Metrics
[ACM] turning this into a recommendation:

>    functions, we must begin to consider additional performance metrics.
[ACM] 
When considering characteristics important to "telco" network 
functions, additional performance metrics are needed.
>    Tests have been (or will be) designed to collect the metrics below:
[ACM] (these are all included now):

> of this document as a kind of intermediary document.
[ACM] 
Thank you. We know we have a lot more to learn before we 
update truly fundamental RFCs like 2544 and 2889.
That would be a big step, so we need to be very sure.

>     with OPNFV (and other groups) on this future plan.
[ACM] 
Hopefully, the suggested changes above will accomplish that,
but some of your ideas can be easily added to the Introduction.
Please trust me for that text...
> Should this section refer to RFC6645, or RFC 5470 (section 5.1.1)?
[ACM] 
Ben raised a similar comment.  
I'll add the best reference for BMWG docs, which is

For example, RFC 2889 says:
   " ... The DUT/SUT address aging time SHOULD be configured to be greater than
   the period of the learning phase of the test plus the trial duration
   plus any configuration time required by the testing device."
See:
https://tools.ietf.org/html/rfc2889#section-3
The learning phase is benchmarked separately.

> hopefully, it re-uses the terminology, right?
[ACM] 
Right, 5481 removed.

Except for the Intro paragraph I mentioned, 
the working text is revised as described.