Subject: [bmwg] draft-ietf-bmwg-dcbench-methodology-07

From: Scott Bradner <[email-masked]>
Date: Fri, 16 Jun 2017 22:27:59 -0400

IJ have been asked to do a OPS-DIR review of draft-ietf-bmwg-dcbench-methodology-07

I quickly For all tests, the percentage of traffic per port capacity sent MUST

   be 99.98% at most, with no PPM adjustment to ensure stressing the DUT

   in worst case conditions. Tests results at a lower rate MAY be

   provided for better understanding of performance increase in terms of

   latency and jitter when the rate is lower than 99.98%. The receiving

   rate of the traffic SHOULD be captured during this test in % of line

   rate.

---

From: Scott Bradner <[email-masked]>
Date: Fri, 16 Jun 2017 22:33:07 -0400

try again - my new mac book often sends mail before I want it to

I have been asked to do a OPS-DIR review of draft-ietf-bmwg-dcbench-methodology-07

I quickly became confused - a paragraph in section 2.2 reads

For all tests, the percentage of traffic per port capacity sent MUST

Thus reads to me like a conformance test not a performance test (saying what is “passing”)  unless
BMWG has morphed a lot since I was last involved conformance and pass/fail are not things that BMWG
should be doing - 

what am I missing?

Scott

---

From: Lucien Avramov <[email-masked]>
Date: Sat, 17 Jun 2017 10:49:04 -0700

Hi Scott!

   - The draft you are reviewing is the ''*methodology*'' draft which
   explains the methodology for line rate performance testing. The exact
   reason onto why it is 99.98% with 0 PPM adjustment is covered extensively
   in the *terminology draft*, section 5: https://tools.ietf.org/html
   /draft-ietf-bmwg-dcbench-terminology-12#section-5
   <https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-12#section-5>.
   That draft dependency is covered in the introduction and also in the
   reference lists and is referenced throughout the document
   - There is a clock drift difference between tester and DUT explained in
   section 5.2 of the *terminology draft*

In our view BMWG has not morphed, our goal has been to be very precise and
accurate with performance testing, and this is not about conformance
testing at all. These definitions that you see, have not changed in 4 years and
have been proven themselves in a very large amount of testing done since we
started publishing.

As you may remember, we have been working on this draft actively for over 4
years now. We've incorporated numerous of your feedbacks on these drafts
over the IETF sessions (since IETF 86 - captured in the meeting minutes of
the IETF sessions).  On April 2016, we had your blessing on the list also
regarding our drafts.

We appreciate as always your feedback, Scott!

Lucien

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 18 Jun 2017 05:03:58 +0000

Hi Lucien, Scott, and all,

When I read this material, I assumed the goal was to
limit the maximum sending rate of the test traffic generator
in accordance with the Ethernet clock tolerances,
effectively choosing the low-end of clock tolerance
as the maximum sending rate with a factor of 2 margin.
99.98% of the nominal line rate would be -200 ppm,
and any conforming interface clock should work fine.

If I understood your goal correctly,
I wonder if a small clarification will help future
readers.

When section 2.2 says:
   For all tests, the percentage of traffic per port capacity sent MUST
                                                ^^^^^^^^^^^^^^^^^^
   be 99.98% at most, with no PPM adjustment to ensure stressing the DUT
      +++++++++++++++
   in worst case conditions.

You’ve defined the concept of Port Capacity in terminology Section 5,
under the Line Rate definition, as

   The term "port capacity" defines the maximum speed capability for the
   given port; for example 1GE, 10GE, 40GE, 100GE etc.

Actually, the port capacity examples are the *nominal value of Line Rate*,
before the clock tolerances are considered.

Here’s a suggestion for Section 2.2 of the meth:

   For all tests, the test traffic generator sending rate MUST be
   less than or equal to 99.98% of the nominal value of Line Rate
   (with no further PPM adjustment to account for interface clock
   tolerances), to ensure stressing the DUT in reasonable worst case
   conditions.

Changing
s/port capacity/nominal value of Line Rate/
here, requires an additional substitution in section 2.3 of the meth draft,
and a single substitution in section 5.1 of the term draft.

Does this help?
Al

From: Lucien Avramov [mailto:[email-masked]]
Sent: Saturday, June 17, 2017 1:49 PM
To: Scott Bradner
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: draft-ietf-bmwg-dcbench-methodology-07

Hi Scott!
·         The draft you are reviewing is the ''methodology'' draft which explains the methodology for line rate performance testing. The exact reason onto why it is 99.98% with 0 PPM adjustment is covered extensively in the terminology draft, section 5: https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-12#section-5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-2D12-23section-2D5&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=XTSYKMpGlG-drxu-CVqDhTNx8YcVWdMmypOAISknLvc&s=UkXcmruEslwhl7CcJhOFERz_27CZSfAfR-0WHqZl13g&e=>. That draft dependency is covered in the introduction and also in the reference lists and is referenced throughout the document
·         There is a clock drift difference between tester and DUT explained in section 5.2 of the terminology draft
In our view BMWG has not morphed, our goal has been to be very precise and accurate with performance testing, and this is not about conformance testing at all. These definitions that you see, have not changed in 4 years and have been proven themselves in a very large amount of testing done since we started publishing.

As you may remember, we have been working on this draft actively for over 4 years now. We've incorporated numerous of your feedbacks on these drafts over the IETF sessions (since IETF 86 - captured in the meeting minutes of the IETF sessions).  On April 2016, we had your blessing on the list also regarding our drafts.

Lucien

On Fri, Jun 16, 2017 at 7:33 PM, Scott Bradner <[email-masked]<mailto:[email-masked]>> wrote:
try again - my new mac book often sends mail before I want it to

what am I missing?

Scott

---

From: Scott Bradner <[email-masked]>
Date: Sun, 18 Jun 2017 09:52:48 -0400

my issue was that I read the current text as talking about the result of the testing not the offered load

so I was just going to suggest coming similar to what Al suggested 

but I’d add specific pointers to the explanation in the terminology document as to why

e.g.

   For all tests, the test traffic generator sending rate MUST be
   less than or equal to 99.98% of the nominal value of Line Rate
   (with no further PPM adjustment to account for interface clock
   tolerances), to ensure stressing the DUT in reasonable worst case
   conditions. (see RFC XXX section ??? - note to RFC editor replace XXX
   with number of terminology RFC)

I have not reviewed the rest of the ID to see if similar misreading is possible so that
a similar change would be useful

Scott

> On Jun 18, 2017, at 1:03 AM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> ·         The draft you are reviewing is the ''methodology'' draft which explains the methodology for line rate performance testing. The exact reason onto why it is 99.98% with 0 PPM adjustment is covered extensively in the terminology draft, section 5: https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-12#section-5. That draft dependency is covered in the introduction and also in the reference lists and is referenced throughout the document
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 18 Jun 2017 16:51:29 +0000

adding OPS-DIR,
> -----Original Message-----
> From: bmwg [mailto:[email-masked]] On Behalf Of Scott Bradner
> Sent: Sunday, June 18, 2017 9:53 AM
> To: MORTON, ALFRED C (AL)
> Cc: [email-masked]; bmwg-
> [email-masked]; [email-masked]
> Subject: Re: [bmwg] draft-ietf-bmwg-dcbench-methodology-07
> 
[ACM] 
good addition, Scott.

> a similar change would be useful
[ACM] 
Let's assume for now that other Area reviewers and IESG review will
catch some more of these, if needed. 

thanks,
Al

> > ·         The draft you are reviewing is the ''methodology'' draft
> which explains the methodology for line rate performance testing. The
> exact reason onto why it is 99.98% with 0 PPM adjustment is covered
> extensively in the terminology draft, section 5:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-
> 2D12-23section-2D5&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=cY2eIWfAaujzPr7A49lWCjuJ_Vx
> 0ywhEDDmsWPBcTJA&s=96T3Kqxhm3c-gASDa1w_XDRAcXYh30m7fyjPEm28HCE&e= . That
> draft dependency is covered in the introduction and also in the
> > ·         There is a clock drift difference between tester and DUT
> explained in section 5.2 of the terminology draft
> >
> > In our view BMWG has not morphed, our goal has been to be very precise
> and accurate with performance testing, and this is not about conformance
> testing at all. These definitions that you see, have not changed in 4
> years and have been proven themselves in a very large amount of testing
> done since we started publishing.
> >
> >
> >
> > As you may remember, we have been working on this draft actively for
> over 4 years now. We've incorporated numerous of your feedbacks on these
> drafts over the IETF sessions (since IETF 86 - captured in the meeting
> minutes of the IETF sessions).  On April 2016, we had your blessing on
> the list also regarding our drafts.
> >
> >
> >
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=cY2eIWfAaujzPr7A49lWCjuJ_Vx
> 0ywhEDDmsWPBcTJA&s=NwNczwTUN_yQqjl4Aw4jBTqk3iMbJpTrJMDiuFT8g20&e=

---

From: Lucien Avramov <[email-masked]>
Date: Sun, 18 Jun 2017 10:00:40 -0700

Hi Al, Scott

thanks for the comments, updated both drafts just now with all comments!

Thanks,
Lucien

---

From: Lucien Avramov <[email-masked]>
Date: Tue, 20 Jun 2017 08:55:23 -0700

Hi Scott, Al, IESG reviewers, BMWG!

After a good amount of work and feedback from Scott yesterday, I am pleased
to report that we have posted the final versions now.

The changes to look for in the methodology draft are about:

1) clarifying terms and pointing to the terminology draft when appropriate
when new terms were defined in the terminology draft (such as for example
intensity )
2) clarifying the iteration process for all tests, explaining briefly how
to read the test sequence each time
3) clarifying when TCP type of testing and UDP type of testing is needed
(for goodput type of testing)

Many thanks for all the work from so many folks getting us together to
mature documents.

Regards,
Lucien