Subject: Re: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 13 Feb 2019 18:21:41 +0000

Hi Yoshiaki-san and BMWG,

I have revised the draft (04) to include the points
we agreed, below (at the end of section 5):

https://datatracker.ietf.org/doc/draft-morton-bmwg-b2b-frame/

If there are other clarifications needed, we can discuss
further.

thanks and regards,
Al
(as a participant)

From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Tuesday, December 18, 2018 11:40 AM
To: Yoshiaki Itou <[email-masked]>; Sarah B <[email-masked]>; [email-masked]
Subject: Re: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
Hi Yoshiaki-san,

Thank you again for your support, and for illustrating
your understanding of the calculations in the b2b-frame draft.

I think you have uncovered two details that I didn't describe
in the Corrected DUT Buffer Time Calculation, which are:

1.  The "Measured Throughput" is the RFC2544 Throughput Benchmark

for the frame size tested, and MUST be expressed in

Frames per second in this equation.

2.  The "Max Theoretical Frame Rate" is a calculated value for the

interface speed and link layer technology used, and MUST be

expressed in Frames per second in this equation.

These details would change the right-side of the slide
you shared with us on bmwg-list.  For example, the RFC 2544
Throughput Benchmark is based on the offered load on Ingress
over a limited trial duration. The frame rate is not measured on egress,
the receiver only evaluates if all frames sent to the ingress
were received (no loss), after allowing the internal buffers
to empty.

Also, my simplified model of the DUT includes a packet header
processing function with limited rate of operation (not shown in
your slide):
                     |------------ DUT --------|
Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver

So, in the back2back frame testing,

1.  the Ingress burst arrives at Max Theoretical Frame Rate,

and initially the frames are buffered

2.  a packet header processing function operates at

approximately the "Measured Throughput", removing frames

from the buffer.

3.  Frames that have been processed are clearly not in the buffer,

so the Corrected DUT buffer time equation estimates and removes

the frames that the DUT forwarded on Egress during the busrt.

There are some example calculations here:
https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_vsperf_Traffic-2BGenerator-2BTesting-23TrafficGeneratorTesting-2DAppendixB-3ABack2BackTestingTimeSeries-28fromCI-29&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=0OU4C4AqvLg_D7DELrCSk_F5msO3yBHvQJ_sXHbiArw&s=DAkdeAhmHqU_aSUZQvXoF3iKwtSZd0pjMkkXpobvhcc&e=>

I propose to add the listed points to the explanation in the draft,
as shown below:

Implied DUT Buffer Time =

      Average num of Back-to-back Frames / Max Theoretical Frame Rate

   The formula above is simply expressing the Burst of Frames in units
   of time.

   Corrected DUT Buffer Time =

                                      Measured Throughput
        = Implied DUT Buffer Time * --------------------------
                                    Max Theoretical Frame Rate

|  where:

1.  The "Measured Throughput" is the RFC2544 Throughput Benchmark

for the frame size tested, and MUST be expressed in

Frames per second in this equation.

2.  The "Max Theoretical Frame Rate" is a calculated value for the

interface speed and link layer technology used, and MUST be

expressed in Frames per second in this equation.
|

   The term on the far right in the formula for Corrected DUT Buffer
   Time accounts for all the frames in the Burst that were transmitted
  by the DUT *while the Burst of frames were sent in*..

(I'll add some other clarifications, too)
regards,
Al

From: Yoshiaki Itou [mailto:[email-masked]]
Sent: Tuesday, December 18, 2018 6:13 AM
To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>; Sarah B <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

Hello Morton-san,

Thank you for your detailed explanation.

I support your draft in the pre-requisites of the  draft.
May I understand  "Corrected DUT Buffer Time" as attached ppt.

Best Regards,
Yoshiaki Itou

From: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Sent: Monday, December 17, 2018 2:37 AM
To: Yoshiaki Itou <[email-masked]<mailto:[email-masked]>>; Sarah B <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

Hi Yoshiaki Itoh,

Thank you for your past reviews and most recent comments.

The main innovations of this draft:

                  https://datatracker.ietf.org/doc/draft-morton-bmwg-b2b-frame/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmorton-2Dbmwg-2Db2b-2Dframe_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=HS7Ed6lrlBb60NZsCWT7WGKCJJIeGwqkssw7Jo0fsCk&s=Oqg5iKz3YLnA2seSHeJEgwf-Ga7nAK8GargCnf52jKM&e=>
(which is currently considered for WG adoption in BWMG,
and comments on this call are appreciated)

involve the pre-requisite testing for RFC 2544 Throughput
that results in:

  1.  development of a correction factor for the calculation

to remove the frames that were forwarded from the frames

that were buffered, and give an accurate estimate of

buffer size for the single port-pair case.

  1.  reduction of the back-to-back frame tests needed,

where only packet rates that induce buffering would

be tested.

I shared test results that illustrate the value of these two
innovations:
https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_vsperf_Traffic-2BGenerator-2BTesting-23TrafficGeneratorTesting-2DAppendixB-3ABack2BackTestingTimeSeries-28fromCI-29&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=0OU4C4AqvLg_D7DELrCSk_F5msO3yBHvQJ_sXHbiArw&s=DAkdeAhmHqU_aSUZQvXoF3iKwtSZd0pjMkkXpobvhcc&e=>
and
https://datatracker.ietf.org/meeting/100/materials/slides-100-bmwg-back-to-back-frame-benchmark-01<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_meeting_100_materials_slides-2D100-2Dbmwg-2Dback-2Dto-2Dback-2Dframe-2Dbenchmark-2D01&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=0OU4C4AqvLg_D7DELrCSk_F5msO3yBHvQJ_sXHbiArw&s=qzkaoZhPVRVlXREOzTML9dE6oU3H0ghVitwb7ItuTLI&e=>

RFC2544 does not discuss either of these points, so
this draft would update and expand the current test
in the simplest way possible. However, the current draft
benefits from your previous comments, specifically that
we distinguish the scope of the back2back test from the
RFC8239 multi-port testing and cases where line-rate
switching is possible at the smallest frame sizes
(both are out of scope, but there are still many places
where the current back2back frame tests will be useful).

The Pause method you are proposing has some interesting
possibilities for new test methods, in the same scope of the multi-port
scenarios addressed in RFC 8239 (Datacenter).  We discussed this
during our IETF-103 session. I think the WG really appreciates
that you have shared your test results. There were several issues
raised during the session, and further discussion would
benefit from development of a procedure and other details
usually found in an Internet Draft, such as new test equipment
capabilities (ability to send "Pause"). I can help you with
this documentation aspect, if you would like.

It seems clear that we need a set of buffer measurement solutions
that depend on the specific circumstances and scenarios to
measure.

Thanks again for reviewing the back-to-back frame draft,
and for your contributions to the many WG discussions on
buffer measurement. I hope you'll support my draft's first steps
to better measurement, with many steps to follow.

best regards,
Al
(as a participant/author)

From: bmwg [mailto:[email-masked]] On Behalf Of Yoshiaki Itou
Sent: Friday, December 14, 2018 1:05 AM
To: Sarah B <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

Hello BMWG,

I would like to comment on the definition of back-to-back Frame value.

>5.2.  Test for a Single Frame Size
>
>The Back-to-back Frame value is the longest burst of frames that the
>DUT can successfully process and buffer without frame loss,

The description of this back-to-back Frame value is exact.
To explain more strictly:
The Back-to-back Frame value is the longest burst of frames that the
DUT can successfully process(forward not buffered) and buffer without frame loss,

"Pause" method is a measuring the number of buffered frames only.

https://datatracker.ietf.org/doc/minutes-103-bmwg/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_minutes-2D103-2Dbmwg_&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=HS7Ed6lrlBb60NZsCWT7WGKCJJIeGwqkssw7Jo0fsCk&s=MjJaOUmICdbkd2ls5SZN47QrTvYrHR76s8UZzrPbm4I&e=>

8. New Buffer assessment method for RFC 8239 Data Center Benchmarking
   Author: Yoshiaki Itou
   Mail List References:
   https://mailarchive.ietf.org/arch/msg/bmwg/lKiImpq8RlNapD8CVRG1dRZlMZ8<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_bmwg_lKiImpq8RlNapD8CVRG1dRZlMZ8&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=HS7Ed6lrlBb60NZsCWT7WGKCJJIeGwqkssw7Jo0fsCk&s=sloQPaNgjEx7RZCURpl7ReYqeT2eytL6Rr1xQdyq_50&e=>
   https://mailarchive.ietf.org/arch/msg/bmwg/elCgGpsB-TH1zCwaRhzM7B2mW4g<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_bmwg_elCgGpsB-2DTH1zCwaRhzM7B2mW4g&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=HS7Ed6lrlBb60NZsCWT7WGKCJJIeGwqkssw7Jo0fsCk&s=EsDXYSsfsoIkrcNSC0fp9qYEcx9kJXOq7pTu2Irbs7M&e=>

Best Regards,
Yoshiaki Itou
From: bmwg <[email-masked]<mailto:[email-masked]>> On Behalf Of Sarah B
Sent: Saturday, December 08, 2018 2:57 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

Hello BMWG,
                  At IETF103 in Bangkok Al Morton, author of the "back to back framework" draft, presented an update.

                  https://datatracker.ietf.org/doc/draft-morton-bmwg-b2b-frame/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmorton-2Dbmwg-2Db2b-2Dframe_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=HS7Ed6lrlBb60NZsCWT7WGKCJJIeGwqkssw7Jo0fsCk&s=Oqg5iKz3YLnA2seSHeJEgwf-Ga7nAK8GargCnf52jKM&e=>

                  There has been sufficient interest, review, and comments on the draft, and I'd like to call for adoption of this draft by the WG. Please reply to this email before December 21, 2018 and indicate:

(1) whether you support addition of the following milestone to
  the BMWG charter:

date TBD: Updates for the Back-to-back Frame Benchmark in RFC 2544

(2) whether you support the adoption of draft-morton-bmwg-b2b-frame-03
      as the basis document for this milestone

(3) whether you commit to reviewing the WG document if adopted

Thanks for your consideration of the draft!

Kind regards,
Sarah
bmwg co-chair

---

From: Yoshiaki Itou <[email-masked]>
Date: Sat, 16 Feb 2019 05:38:02 +0000

Hello Morton-san,

Thank you for your update.

You already explained the model.
Moreover, it is easy to understand if you add the following figure.

Model: single ingress port and sinle egress port

Best Regards,
Yoshiaki Itou

From: MORTON, ALFRED C (AL) <[email-masked]>
Sent: Thursday, February 14, 2019 3:22 AM
To: Yoshiaki Itou <[email-masked]>; Sarah B <[email-masked]>; [email-masked]
Subject: RE: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

thanks and regards,
Al
(as a participant)

  1.  The "Max Theoretical Frame Rate" is a calculated value for the

  1.  a packet header processing function operates at

from the buffer.

  1.  Frames that have been processed are clearly not in the buffer,

                                      Measured Throughput

|  where:

  1.  The "Max Theoretical Frame Rate" is a calculated value for the

Hello Morton-san,

Best Regards,
Yoshiaki Itou

Hi Yoshiaki Itoh,

be tested.

best regards,
Al

Hello BMWG,

Best Regards,
Yoshiaki Itou

Hello BMWG,

Kind regards,
Sarah
bmwg co-chair

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 17 Feb 2019 13:53:30 +0000

Thanks for your suggestion, Yoshiaki-san.
I will include the figure describing the
simple model in the next revision (before
the draft-submission deadline).

Other comments are welcome,
Al
(as a participant)

From: Yoshiaki Itou [mailto:[email-masked]]
Sent: Saturday, February 16, 2019 12:38 AM
To: MORTON, ALFRED C (AL) <[email-masked]>; Sarah B <[email-masked]>; [email-masked]
Subject: RE: [bmwg] Call for Adoption: draft-morton-bmwg-b2b-frame

Hello Morton-san,

Best Regards,
Yoshiaki Itou

https://datatracker.ietf.org/doc/draft-morton-bmwg-b2b-frame/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmorton-2Dbmwg-2Db2b-2Dframe_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=byrxlswTYlaKHQkij2bJqkxq0Pl07P8ShV2qaAktbiY&s=8ygiXGRLm9ko__KOQu6LimncVYS6wIqApJYYJXd1EYs&e=>

thanks and regards,
Al
(as a participant)

from the buffer.

                                      Measured Throughput

|  where:

Hello Morton-san,

Best Regards,
Yoshiaki Itou

Hi Yoshiaki Itoh,

be tested.

best regards,
Al

Hello BMWG,

Best Regards,
Yoshiaki Itou

Hello BMWG,

Kind regards,
Sarah
bmwg co-chair