Subject: [bmwg] Benoit Claise's No Objection on draft-ietf-bmwg-dcbench-methodology-15: (with COMMENT)

From: Benoit Claise <[email-masked]>
Date: Wed, 21 Jun 2017 08:49:53 -0700

Benoit Claise has entered the following ballot position for
draft-ietf-bmwg-dcbench-methodology-15: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-methodology/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

At some point in time, I was wondering if this document was about
(benchmarking) terminology or about (benchmarking) performance metric
definition? And I was wondering what the connection was with the RFC 6390 PMOL
template and with
https://tools.ietf.org/html/draft-ietf-ippm-metric-registry-11? When I look at
the section 3 "jitter", there are obvious elements from the PMOL template.
Thinking about it, whether a performance metric is used for benchmark or not,
we need a formal definition. Is it time for: 1. IPPM to finish up
draft-ietf-ippm-metric-registry 2. start populating the registry 3. BMWG to
start using/specifying those performance metrics

Thoughts?

Below is Tim Chown's OPS DIR review:
verall, the document is well written, and I believe it to be Ready with minor
nits.

General comment:

It would be interesting to see an Appendix with an example of a recorded test
using the language defined here.

Specific comments:

Section 1:

“- Low amount of buffer (in the Mb range)”
Change to MB?  (given later you refer to KB/MB/GB as the measurement unit for
buffers)

Section 2.1

Expand DUT on first use.

Section 3.1

Perhaps clarify relationship of Delay and Latency, since you focus on Latency
in the document and not Delay?

Last para, you say “If” here, but for Latency the FILO timestamp was a MUST in
Section 2?  This doesn’t seem consistent?

Expand PDV on first use.

Section 6.1.1

“1518 bytes frames” -> “1518 byte frames”

Section 7.1,

Why is ‘and’ in quotes here?  Not sure you can say the balance is defined by
goodput?  Do you mean that goodput is an indication of the balance?  For
standard TCP, a very small loss can have a dramatic effect on application
throughput.

The second para should follow the first, change “[RFC2647].  Goodput…” to
“[RFC2647], i.e., goodput…”

Section 7.3

I don’t understand how the example given correlates to G = S/Ft ?

There’s a few typos in this section; please re-check.

---

From: Lucien <[email-masked]>
Date: Wed, 21 Jun 2017 10:17:23 -0700

Hi Benoit,

Thank you for the comment! ( I did address also Tim's comment by updating
the draft and answering his email)

Please see inline for your comments:

> https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-methodology/

Initially when we started this over 4 years ago i wanted to do 1 document
for both terminology and method of benchmarking. It became a very long and
intense publication, and I believe back then when you were also our AD, the
feedback we kept getting was to make this easier and break it down in two
publications wich we did. Now looking back, I am glad we did because it
provides a much better structure and it's more digestable and succinct.

The document terminology is about defining the terms specific for data
center benchmarking. It's not about the performance per say. But it's
related to the implication of performance testing.

The document methodology (on the same last call), is about how and what to
benchmark (using the terminologies defined in the other document).

I think it would be a great idea to harmonize all performance metrics
accross WGs, but that's really beyond the scope of what our goal is in
those two publications. I am happy to help participate in such incentive /
project if there is critical mass willing to do it.