Subject: [bmwg] Martin Duke's Discuss on draft-ietf-bmwg-b2b-frame-03: (with DISCUSS and COMMENT)

From: Martin Duke via Datatracker <[email-masked]>
Date: Wed, 09 Dec 2020 15:03:59 -0800

Martin Duke has entered the following ballot position for
draft-ietf-bmwg-b2b-frame-03: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-b2b-frame/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Thank you for engaging with the TSVART review. Despite the wordsmithing that
has gone on, I am not sure that we have captured the correct text.

The proposed change is:
> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers
to > deplete. > > and I'll add: > The upper search limit for the time to send
each burst MUST be configurable as > high as 30 seconds (buffer time results
reported at the configured upper limit are > likely invalid, and the test MUST
be repeated with a higher search limit).

But IIUC it's the additional time that needs to scale up. A layman's reading of
the document, IMO, suggests that the burst length has a binary search but the 2
seconds of waiting can be fixed.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Other than that, this a well-written document. Thanks!

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 12 Dec 2020 15:18:26 +0000

Hi Martin, thanks for your review and comment,
please see my reply, [acm] below,
Al

> -----Original Message-----
...
> > The duration of the trial MUST include at least 2 seconds in addition to the time
> > required to send and receive each burst of frames, to ensure that DUT buffers to deplete. 
> > and I'll add: 
> > The upper search limit for the time to send each burst MUST be configurable as 
> > high as 30 seconds (buffer time results
> > reported at the configured upper limit are likely invalid, and the test MUST
> But IIUC it's the additional time that needs to scale up. 
[acm] 

In the revised text where David and I reached agreement, we identified 3 time components of the trial duration, making the duration variable: no longer static and at "at least 2 seconds".

1. the time to send the burst of frames (at the back-to-back rate), determined by the search algorithm
2. the time to receive the transferred burst of frames (at the RFC2544 Throughput rate), possibly truncated by buffer overflow, but certainly including the latency of the DUT with or without buffer-bloat
3. at least 2 seconds in addition to the time to receive the burst (2.), to ensure that DUT buffers have depleted.

So, both components 1. and 2. are variables, and the burst receive time component (2.) compensates for large buffers, non-back-to-back burst egress, and anything else that contributes to DUT latency. The final "at least 2 seconds" is simply about making sure the trial is really over before moving on in an automated test - we won't make an error if frames trickle-out very late for some unfortunate reason.

> A layman's reading of
[acm] 
Yes, that's right, plus all the other factors above.

So, let's try this, but I'm trying not to extend or complicate the buffer time << 2 seconds testing for the sake of the buffer-bloat case:

-=-=-=-=-=-=-

The duration of the trial includes three REQUIRED components:

1. the time to send the burst of frames (at the back-to-back rate), determined by the search algorithm
2. the time to receive the transferred burst of frames (at the RFC2544 Throughput rate), possibly truncated by buffer overflow, and certainly including the latency of the DUT 
3. at least 2 seconds not overlapping the time to receive the burst (2.), to ensure that DUT buffers have depleted.

The upper search limit for the time to send each burst MUST be configurable as high as 30 seconds (buffer time results reported at or near the configured upper limit are likely invalid, and the test MUST be repeated with a higher search limit).

-=-=-=-=-=-=-=-=-

Does that wording do it?

[acm] 
Thank you!

---

From: "Scott O. Bradner" <[email-masked]>
Date: Sat, 12 Dec 2020 17:18:29 -0500

this would seem to work if 2 seconds is significantly longer than it takes to send the burst - but if it takes 2 second to send the burst 
then 2 seconds extra buffer could easily lose packets - seems to me that he extra time should be related to the time it takes to send the burst

e.g 50% of the burst time but not less than 2 seconds

Scott

> On Dec 12, 2020, at 10:18 AM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 15 Dec 2020 00:24:51 +0000

Hi Scott, thanks for helping with this discussion.

I'm trying to formulate adaptive extra time based on the time it takes to *receive* the burst, with the additional "at least 2 seconds"  waiting time to be sure we received all the packets that might arrive.  Let me try drawing the timeline that's in my mind, and I'll use a buffer-bloat case example of a 1 second buffer (which dominates all other buffers in the DUT). 

One of the key contributions of this memo is recognizing that the buffer is being emptied while the burst of back-to-back frames is simultaneously trying to fill the buffer.  

Assume that the RFC 2544 Throughput is only half of the back-to-back frame rate for the frame size used. 

>From the draft:
   4.  A helpful concept is the buffer filling rate, which is the
       difference between the Max Theoretical Frame Rate (ingress) and
       the Measured Throughput (HeaderProc on egress).  If the actual
       buffer size in frames was known, the time to fill the buffer
       during a measurement can be calculated using the filling rate as
       a check on measurements.  However, the Buffer in the model
       represents many buffers of different sizes in the DUT data path.

So (danger: calculating while typing and drawing!), a 1 second burst of B2B frames only raises the occupation buffer to 50%, and another second of transmission is needed before reaching 100% occupation.

Trial
Time, sec: 0          1          2          3         4          5          6

Sender:    |==========|==========|
Receiver:  |= = = = = |= = = = = |= = = = = |= = = = =|
Waiting Time                                          |          |          |
                                                                            Trial
                                                                            Ends

In the ideal example timeline above, the back-to-back burst stopped exactly when the buffer reached capacity, so there is no loss. The buffer fill rate is half the back-to-back rate. Also, it takes 2 seconds to deplete the buffer and for frames to stop arriving at the receiver. Only then do we start the 2 second waiting time to ensure no more frames will arrive!

While we're here, let's look at a calculation from the memo:

   Corrected DUT Buffer Time =
                          /                                         \
           Implied DUT    |Implied DUT       Measured Throughput    |
        =  Buffer Time -  |Buffer Time * -------------------------- |
                          |              Max Theoretical Frame Rate |
                          \                                         /
        =  2 - [ 2 * 0.5 ] seconds
        =  1 second

and we avoid the error of calculating buffer time based on the sender's burst duration alone.

hope this helps,
Al

> From: Scott O. Bradner [mailto:[email-masked]]
> Sent: Saturday, December 12, 2020 5:18 PM
> To: MORTON, ALFRED C (AL) <[email-masked]>
> Cc: Martin Duke <[email-masked]>; The IESG <[email-masked]>;
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [bmwg] Martin Duke's Discuss on draft-ietf-bmwg-b2b-frame-03:
> (with DISCUSS and COMMENT)
> 
> this would seem to work if 2 seconds is significantly longer than it takes
> to send the burst - but if it takes 2 second to send the burst
> then 2 seconds extra buffer could easily lose packets - seems to me that
> he extra time should be related to the time it takes to send the burst
> 
> > In the revised text where David and I reached agreement, we identified 3
> time components of the trial duration, making the duration variable: no
> longer static and at "at least 2 seconds".
> >
> > 2. the time to receive the transferred burst of frames (at the RFC2544
> Throughput rate), possibly truncated by buffer overflow, but certainly
> including the latency of the DUT with or without buffer-bloat
> > 3. at least 2 seconds in addition to the time to receive the burst (2.),
> to ensure that DUT buffers have depleted.
> >
> > So, both components 1. and 2. are variables, and the burst receive time
> component (2.) compensates for large buffers, non-back-to-back burst
> egress, and anything else that contributes to DUT latency. The final "at
> least 2 seconds" is simply about making sure the trial is really over
> before moving on in an automated test - we won't make an error if frames
> trickle-out very late for some unfortunate reason.
> >
> >> A layman's reading of
> > So, let's try this, but I'm trying not to extend or complicate the
> buffer time << 2 seconds testing for the sake of the buffer-bloat case:
> >
> > -=-=-=-=-=-=-
> >
> > 2. the time to receive the transferred burst of frames (at the RFC2544
> Throughput rate), possibly truncated by buffer overflow, and certainly
> including the latency of the DUT
> > 3. at least 2 seconds not overlapping the time to receive the burst
> (2.), to ensure that DUT buffers have depleted.
> >
> > The upper search limit for the time to send each burst MUST be
> configurable as high as 30 seconds (buffer time results reported at or
> near the configured upper limit are likely invalid, and the test MUST be
> repeated with a higher search limit).
> >
> > -=-=-=-=-=-=-=-=-
> >
> > Does that wording do it?
> >
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
> !BhdT!1uRJDJBUadSunB4ZCkgOTzg3ZssPtiufcyrsTcxEc1F67df5q4YNUa9IYHacnsA$

---

From: "Scott O. Bradner" <[email-masked]>
Date: Tue, 15 Dec 2020 09:06:06 -0500

I basically understood that but it seemed to me that using a fixed (2 second) extra time, which is unrelated  
to whatever time that the burst might have taken to be sent seemed risky since I could
imagine cases where the play out speed was less than the receive speed 

but if you are convinced that the 2 seconds extra time would cover all possible cases the go to it

Scott

> On Dec 14, 2020, at 7:24 PM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
>> configurable as

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 15 Dec 2020 22:04:38 +0000

Hi Scott,

Please see my replies below, marked [acm], with a couple of questions.
I hope I'm not missing something obvious, so trying to be very clear in all replies!
But I could be overlooking something, and if so I will learn something very soon...

> Sent: Tuesday, December 15, 2020 9:06 AM

[acm] 
I guess I don't understand your example, where (buffer?) play-out speed plays a role in the results, and how play-out speed could be less than the receive speed in the multi-second time scale of the buffer-bloat example. I think (buffer) play-out speed and receive speed should be nominally the same.

Although RFC 2544 Throughput definition is based on offered load delivered loss-free to the receiver, we use it here as the best approximation available for packet header processing rate (equal to playout rate from the buffer?), egress from the DUT, and the speed at which the test system receives packets. 

So, in our diagram from the memo:

                        |------------ DUT --------|
   Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver

Is your play-out speed the HeaderProc speed, or Egress speed?

And how can the (buffer) play-out speed be less than the speed at a subsequent interface (for very long)?

help me understand the mechanics I'm overlooking, my friend!

> 
> but if you are convinced that the 2 seconds extra time would cover all
> possible cases then go to it
[acm] 

Well, we say "at least 2 seconds" and allow for customization if necessary.

As you know, I've conducted LOTS of production network testing, where we have used static waiting times to distinguish packet loss from long delay, and prescribed the same in IPPM RFCs, etc. A static waiting time "Tmax" has served us well.

Here, we have the added stability of the Isolated Test Environment (ITE, as Kevin Dubray called it), and the three time-component definition of trial duration, where we wait 2 seconds after the last packet on seen egress (it is more like a cool-down interval between trials). I think all the adaptation we need comes from explicit recognition that the time for the Test Receiver to receive the entire burst depends on the buffer size, the DUT header processing rate, the actual interface speed, etc. IOW, all the unknown variables.

Thanks again for your time, Scott!
Al

> > I'm trying to formulate adaptive extra time based on the time it takes
> to *receive* the burst, with the additional "at least 2 seconds"  waiting
> time to be sure we received all the packets that might arrive.  Let me try
> drawing the timeline that's in my mind, and I'll use a buffer-bloat case
> example of a 1 second buffer (which dominates all other buffers in the
> DUT).
> >
> > One of the key contributions of this memo is recognizing that the buffer
> is being emptied while the burst of back-to-back frames is simultaneously
> trying to fill the buffer.
> >
> > So (danger: calculating while typing and drawing!), a 1 second burst of
> B2B frames only raises the occupation buffer to 50%, and another second of
> transmission is needed before reaching 100% occupation.
> >
> > Trial
> > Time, sec: 0          1          2          3         4          5
> 6
> >
> > Waiting Time                                          |          |
> |
> > In the ideal example timeline above, the back-to-back burst stopped
> exactly when the buffer reached capacity, so there is no loss. The buffer
> fill rate is half the back-to-back rate. Also, it takes 2 seconds to
> deplete the buffer and for frames to stop arriving at the receiver. Only
> then do we start the 2 second waiting time to ensure no more frames will
> arrive!
> >
> >>>> Thank you for engaging with the TSVART review. Despite the
> wordsmithing
> >> that
> >>>>> The duration of the trial MUST include at least 2 seconds in
> addition
> >> to the time
> >>>>> required to send and receive each burst of frames, to ensure that
> DUT
> >> buffers to deplete.
> >>>>> and I'll add:

---

From: "Scott O. Bradner" <[email-masked]>
Date: Wed, 16 Dec 2020 06:43:48 -0500

> On Dec 15, 2020, at 5:04 PM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Hi Scott,
> 

I expect that generally they would be about the same but its all software and different routines would 
handle input & output so one can not be sure - in addition the system could be adding keep-alive packets, 
routing updates etc to the output stream (not that they would take much time to send)
> And how can the (buffer) play-out speed be less than the speed at a subsequent interface (for very long)?

"very long" is a relative term :-)

I agree that there should not be any issue if 2 seconds is long relative to the burst length but
maybe not so if the burst length is long relative to 2 seconds (e.f. it takes a minute or two
to fill the buffer)

Scott

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 16 Dec 2020 18:31:43 +0000

Hi Scott,

I appreciate your practical insights shared below, as always!

Let me propose some text, expressing both the concerns you and Martin raised, and the caution that's in my mind (buffer-bloat sizes change the time scale of testing, but the problem we currently face with the "latest technology" is a very small buffer <<1 sec and difficulty increasing the size through configuration changes = we need to re-run tests often to determine implementation success/failure).

The duration of the trial includes three REQUIRED components:
...
3. At least 2 seconds not overlapping the time to receive the burst (2.), to ensure that DUT buffers have depleted. Longer times MUST be used when conditions warrant, such as when buffer times >2 seconds are measured or when burst sending times are >2 seconds, but care is needed since this time component directly increases trial duration and many trials and tests comprise a complete benchmarking study.

hope this works,
Al

> Sent: Wednesday, December 16, 2020 6:44 AM
> >> to whatever time that the burst might have taken to be sent seemed
> risky
> >> since I could
> > I guess I don't understand your example, where (buffer?) play-out speed
> plays a role in the results, and how play-out speed could be less than the
> receive speed in the multi-second time scale of the buffer-bloat example.
> I think (buffer) play-out speed and receive speed should be nominally the
> same.
> 
> 
> > Although RFC 2544 Throughput definition is based on offered load
> delivered loss-free to the receiver, we use it here as the best
> approximation available for packet header processing rate (equal to
> playout rate from the buffer?), egress from the DUT, and the speed at
> which the test system receives packets.
> >
> > As you know, I've conducted LOTS of production network testing, where we
> have used static waiting times to distinguish packet loss from long delay,
> and prescribed the same in IPPM RFCs, etc. A static waiting time "Tmax"
> has served us well.
> >
> > Here, we have the added stability of the Isolated Test Environment (ITE,
> as Kevin Dubray called it), and the three time-component definition of
> trial duration, where we wait 2 seconds after the last packet on seen
> egress (it is more like a cool-down interval between trials). I think all
> the adaptation we need comes from explicit recognition that the time for
> the Test Receiver to receive the entire burst depends on the buffer size,
> the DUT header processing rate, the actual interface speed, etc. IOW, all
> the unknown variables.
> >
> >>> and we avoid the error of calculating buffer time based on the
> sender's
> >> burst duration alone.
> >>>
> >>> hope this helps,
> >>> Al
> >>>
> >>>
> >>>> Subject: Re: [bmwg] Martin Duke's Discuss on draft-ietf-bmwg-b2b-
> frame-
> >> 03:
> >>>>>> -------------------------------------------------------------------
> --
> >> -
> >>>>>> DISCUSS:
> >>>>>> -------------------------------------------------------------------
> --
> >> -
> >>>>>>
> >>>>> In the revised text where David and I reached agreement, we
> identified
> >> 3
> >>>>>> the document, IMO, suggests that the burst length has a binary
> search
> >>>> but the 2
> >>>>>> -------------------------------------------------------------------
> --
> >> -
> >>>>>> COMMENT:
> >>>>>> -------------------------------------------------------------------
> --
> >> -
> >>>>>>

---

From: Martin Duke <[email-masked]>
Date: Wed, 16 Dec 2020 10:52:45 -0800

Looks good to me.

On Wed, Dec 16, 2020 at 10:32 AM MORTON, ALFRED C (AL) <[email-masked]>
wrote:

> Let me propose some text, expressing both the concerns you and Martin
> raised, and the caution that's in my mind (buffer-bloat sizes change the
> time scale of testing, but the problem we currently face with the "latest
> technology" is a very small buffer <<1 sec and difficulty increasing the
> size through configuration changes = we need to re-run tests often to
> determine implementation success/failure).
>
> 3. At least 2 seconds not overlapping the time to receive the burst (2.),
> to ensure that DUT buffers have depleted. Longer times MUST be used when
> conditions warrant, such as when buffer times >2 seconds are measured or
> when burst sending times are >2 seconds, but care is needed since this time
> component directly increases trial duration and many trials and tests
> comprise a complete benchmarking study.
>
> hope this works,
> Al
>
>
> > >> Subject: Re: [bmwg] Martin Duke's Discuss on
> draft-ietf-bmwg-b2b-frame-
> > 03:
> > >> B2B frames only raises the occupation buffer to 50%, and another
> second
> > of
> > >>>> Cc: Martin Duke <[email-masked]>; The IESG <[email-masked]
> >;
> > >>>>>>> reported at the configured upper limit are likely invalid, and
> the
> > >>>> test MUST
> > >>>> near the configured upper limit are likely invalid, and the test
> MUST
> > >> be