Subject: [Tsv-art] Tsvart last call review of draft-ietf-bmwg-b2b-frame-03

From: David Black via Datatracker <[email-masked]>
Date: Tue, 24 Nov 2020 07:26:37 -0800

Reviewer: David Black
Review result: Ready with Issues

This document has been reviewed as part of the transport area review team's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors and WG to allow them to address any issues raised and also to the IETF
discussion list for information.

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC
[email-masked] if you reply to or forward this review.

This draft updates the back-to-back frame testing procedure in RFC 2544 to take
account of experience.

The draft is in good shape, with one notable exception in Section 5.2:

   The duration of the trial MUST be at least 2 seconds, to allow DUT
   buffers to deplete.

That duration of 2 seconds has been carried forward from RFC 2544 without
change.  A 2 second duration may have been sufficient to deplete buffers in
1999, but that is no longer reliably the case.  For example, on-site
measurement of the network for the 2020 Linux Conference in Australia indicated
a  at least 1.6 seconds of buffering, as indicated by Figure 1 at
https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/,
which is slide 6 in from the complete slide deck at:
https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/
.  Experience with bufferbloat suggests that one network device was primarily
responsible.  Also, see slide 14 in that slide deck.

That 1.6 seconds measured on an actual network is entirely too close to 2
seconds for confidence that buffers will be depleted in any tested device. 
Hence, the 2 second minimum duration ought to be increased by at least a factor
of 10.  I'd suggest changing it to 30 seconds or 60 seconds as convenient round
numbers, and providing the rationale that increased buffering in WiFi devices,
e.g., home "routers," as indicated by experience with bufferbloat measurements,
is the reason for the increased duration.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 24 Nov 2020 16:10:52 +0000

Hi David, Thanks for your review and comment!

Please see a proposed resolution below, [acm]
Al

> -----Original Message-----
> From: David Black via Datatracker [mailto:[email-masked]]
> Sent: Tuesday, November 24, 2020 10:27 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; last-
> [email-masked]
> Subject: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
> 
> Reviewer: David Black
...
> https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-
> may-be-solved-but-its-not-over-
> yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-
> yK2DOVVn0ecoYjf7mFEABLmlwDk$ ,
> https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-
> may-be-solved-but-its-not-over-
> yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-
> yK2DOVVn0ecoYjf7mFEABLmlwDk$
[acm] 
[acm] I agree with your observation that there are cases where trial duration should be increased to accommodate the encountered in the DUT, but not as a mandate for all testing. I have four factors in mind:
1. Some of the virtual network DUTs we are testing now have very small buffers, and the B2B stream of frames is quite short -- less than 2000 frames@10GE in some cases -- so 2 seconds fully sufficient.
2. The trial duration is a factor in total test duration, where each trial is one step in the Binary Search. We need to manage the tension between the time needed to reach a search result and confidence that we have depleted the queues.
3. The RFC 2544 Latency benchmark will tell us if bufferbloat is present.
4. The current text says "at least 2 seconds".

So I suggest adding the following text:

    The duration of the trial MUST be at least 2 seconds, to allow DUT
    buffers to deplete. When RFC2544 Latency measurements indicate that
    large buffers are present in the DUT, the trial duration SHOULD be 
    increased to ensure that buffer depletion takes place, without unduly 
    extending the total test time.

I hope this suggestion resolves your issue; thanks for highlighting it!
Al

---

From: "Black, David" <[email-masked]>
Date: Tue, 24 Nov 2020 20:01:35 +0000

Al,

> [acm] I agree with your observation that there are cases where trial duration
> should be increased to accommodate the encountered in the DUT, but not as a
> mandate for all testing. I have four factors in mind:
> 1. Some of the virtual network DUTs we are testing now have very small buffers,
> and the B2B stream of frames is quite short -- less than 2000 frames@10GE in
> some cases -- so 2 seconds fully sufficient.
> 2. The trial duration is a factor in total test duration, where each trial is one step in
> the Binary Search. We need to manage the tension between the time needed to
> reach a search result and confidence that we have depleted the queues.

The overall approach of collecting evidence that there is a problem before increasing the 2 second minimum duration is fine, but the details appear to need more attention:

1. Obtaining the RFC 2544 Latency measurements would need to be added to Section 4 (Prerequisites) of this draft to ensure that the buffer size information is available.

2. I did not see any requirements in the RFC 2544 Latency test (Section 26.2) to deplete buffers.  Did I miss something?

3. I would think that the trial duration MUST be increased, not just SHOULD be increased if there is evidence of large buffer size, as buffer depletion appears to be a necessary characteristic of this B2B measurement.

It also looks like the link to the entire slide deck didn't make it into my original review correctly - that slide deck is at: http://www.taht.net/~d/lca_tcp3.odp .  In addition to slide 6 from this slide deck (Figure 1 in the APNIC blog), slide 14 is also relevant to this discussion.

Thanks, --David

> From: MORTON, ALFRED C (AL) <[email-masked]>
> Sent: Tuesday, November 24, 2020 11:11 AM
> To: Black, David; [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03
> 
> 
> [EXTERNAL EMAIL]
> 
> Please see a proposed resolution below, [acm] Al
> 
> > This draft updates the back-to-back frame testing procedure in RFC
> > 2544 to take account of experience.
> >
> > That duration of 2 seconds has been carried forward from RFC 2544
> > without change.  A 2 second duration may have been sufficient to
> > deplete buffers in 1999, but that is no longer reliably the case.  For
> > example, on-site measurement of the network for the 2020 Linux
> > Conference in Australia indicated a at least 1.6 seconds of buffering,
> > as indicated by Figure 1 at
> > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-yK2DOVVn0ecoYjf7mFEABLmlwDk$ ,
> > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-yK2DOVVn0ecoYjf7mFEABLmlwDk$
> > .  Experience with bufferbloat suggests that one network device was
> > primarily responsible.  Also, see slide 14 in that slide deck.
> >
> > That 1.6 seconds measured on an actual network is entirely too close
> > to 2 seconds for confidence that buffers will be depleted in any tested device.
> > Hence, the 2 second minimum duration ought to be increased by at least
> > a factor of 10.  I'd suggest changing it to 30 seconds or 60 seconds
> > as convenient round numbers, and providing the rationale that
> > increased buffering in WiFi devices, e.g., home "routers," as
> > indicated by experience with bufferbloat measurements, is the reason for the
> increased duration.
> >
> [acm]
> [acm] I agree with your observation that there are cases where trial duration
> should be increased to accommodate the encountered in the DUT, but not as a
> mandate for all testing. I have four factors in mind:
> 1. Some of the virtual network DUTs we are testing now have very small buffers,
> and the B2B stream of frames is quite short -- less than 2000 frames@10GE in
> some cases -- so 2 seconds fully sufficient.
> 2. The trial duration is a factor in total test duration, where each trial is one step in
> the Binary Search. We need to manage the tension between the time needed to
> reach a search result and confidence that we have depleted the queues.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 25 Nov 2020 05:00:12 +0000

Hi David, some more clarity below.
Al

> From: Black, David [mailto:[email-masked]]
> Sent: Tuesday, November 24, 2020 3:02 PM
> To: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
> [email-masked]; Black, David <[email-masked]>
> The overall approach of collecting evidence that there is a problem before
> increasing the 2 second minimum duration is fine, but the details appear
> to need more attention:
> 
> 1. Obtaining the RFC 2544 Latency measurements would need to be added to
> Section 4 (Prerequisites) of this draft to ensure that the buffer size
> information is available.
[acm] 
Section 26.2 Latency is a very primitive procedure by today's standards. It is common for testers to measure delay on every packet, and report a true average. This is enough to alert the operator to the presence of bufferbloat, but there's no assurance that the Latency benchmark measures the extent of the buffer time.

> 
> 2. I did not see any requirements in the RFC 2544 Latency test (Section
> 26.2) to deplete buffers.  Did I miss something?
[acm] 
It's an overall requirement in Section 23, after running the trial:

   d) Wait for two seconds for any residual frames to be received.

> 
> 3. I would think that the trial duration MUST be increased, not just
> SHOULD be increased if there is evidence of large buffer size, as buffer
> depletion appears to be a necessary characteristic of this B2B
> measurement.
[acm] 

We need to keep in mind that the sentence(s) in this discussion pertain to the trials of the B2B Frame Benchmark testing, in the section that begins:

  Each trial in the test requires the tester to send a burst of frames (after idle time) with the minimum inter-frame gap, and to count the corresponding frames forwarded by the DUT.

Let's say we have a DUT that offers 1.6 seconds of buffering. The Binary Search will increase the length of the burst until the buffer drops frames, and then try to find the longest burst where frame loss is zero. This simple procedure always waits the minimum trial duration for frames to exit the DUT.

A burst of 1.7 seconds falls well within the coverage of a 2 second trial to count the frames and determine that loss has occurred. We don't need a big margin time-wise. Also, the trial duration increases with the search parameters: IOW, if the tester is still sending frames after 2 seconds, then obviously we can't allow the trial to terminate prematurely.

So, I suggest we use this wording instead:

       The duration of the trial MUST be at least 2 seconds in addition to 
       the time to send each burst of frames, to allow DUT buffers to deplete.

and we establish an adaptive waiting time for extreme cases, consistent with RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the Trial duration would be 3.6 seconds.

BTW, one tester issue we are trying to fix here is the case when the frame header processing rate is sufficiently high that no buffer limit is measureable at the large frame sizes, and we implore people not to waste time with the B2B Frame Benchmark at those frame sizes. One commercial tester sent bursts up to 30 seconds in length, and then happily reported the impossible result of 30 second buffers. 

I want to avoid over-reacting to the buffer bloat case. We all agree that bloated buffers are bad, and knowing a DUT has buffering >=1 second may be enough. I certainly do not think that trial durations for the B2B Frame benchmark need to be as long as you originally suggested, where the overwhelming number of test conditions can use a <<1 second burst and 2 second buffer depletion time:

> > > a factor of 10.  I'd suggest changing it to 30 seconds or 60 seconds...

I hope the approach of the single sentence above works for you.

> 
> It also looks like the link to the entire slide deck didn't make it into
> my original review correctly - that slide deck is at:
> https://urldefense.com/v3/__http://www.taht.net/*d/lca_tcp3.odp__;fg!!BhdT
> !0oWabwvUrtVlGQY7ZKnFnMa-un_e95tfCJcRKIWG7wECiJkVISb4sjd9Salwehw$  .  In
> addition to slide 6 from this slide deck (Figure 1 in the APNIC blog),
> slide 14 is also relevant to this discussion.
> 
> Thanks, --David
>

---

From: "Black, David" <[email-masked]>
Date: Wed, 25 Nov 2020 14:34:32 +0000

Al,

We appear to not be communicating on a key aspect of the concern:

> A burst of 1.7 seconds falls well within the coverage of a 2 second trial to count the
> frames and determine that loss has occurred. We don't need a big margin time-
> wise. Also, the trial duration increases with the search parameters: IOW, if the
> tester is still sending frames after 2 seconds, then obviously we can't allow the trial
> to terminate prematurely.

The 1.6 seconds value is *not* an upper bound - it was measured in practice on an operational conference network as part of what appears to have been a just-in-time slide preparation exercise.  Hence that 1.6 seconds of buffering ought to be viewed as being somewhere in the midst of the expected distribution of buffer times.  For that reason, the upper bound on test duration needs to be set with a safety factor to ensure that even if a bizarre DUT has 10 seconds of buffering, the B2B test depletes that buffering so that the B2B test is testing the forwarding path as intended, not the buffering capacity.

> I want to avoid over-reacting to the buffer bloat case. We all agree that bloated
> buffers are bad, and knowing a DUT has buffering >=1 second may be enough. I
> certainly do not think that trial durations for the B2B Frame benchmark need to be
> as long as you originally suggested, where the overwhelming number of test
> conditions can use a <<1 second burst and 2 second buffer depletion time:

I agree with this approach provided that there is a test procedure (in the B2B test and/or its prerequisites) that will discover the behavior of the bizarre DUT with 10 seconds of buffering and adjust the B2B test time to ensure that this excessive buffering is depleted.

> I hope the approach of the single sentence above works for you.

Based on this discussion, in addition to that sentence (which I think has to contain a "MUST" for situations in which excessive buffering is found), I would like to see a brief explanation of how the procedures in the draft, along with additional test procedures pulled in from RFC 2544 or elsewhere, avoid a 2 second test duration on a DUT with 10 seconds of buffering.

Thanks, --David

> Sent: Wednesday, November 25, 2020 12:00 AM
> > > should be increased to accommodate the encountered in the DUT, but
> > > not
> > as a
> > > 1. Some of the virtual network DUTs we are testing now have very
> > > small
> > buffers,
> > > the Binary Search. We need to manage the tension between the time
> > > needed
> > to
> > The overall approach of collecting evidence that there is a problem
> > before increasing the 2 second minimum duration is fine, but the
> > details appear to need more attention:
> >
> > 1. Obtaining the RFC 2544 Latency measurements would need to be added
> > to Section 4 (Prerequisites) of this draft to ensure that the buffer
> > size information is available.
> [acm]
> Section 26.2 Latency is a very primitive procedure by today's standards. It is
> common for testers to measure delay on every packet, and report a true average.
> This is enough to alert the operator to the presence of bufferbloat, but there's no
> assurance that the Latency benchmark measures the extent of the buffer time.
> 
> >
> > SHOULD be increased if there is evidence of large buffer size, as
> > buffer depletion appears to be a necessary characteristic of this B2B
> > measurement.
> [acm]
> 
> We need to keep in mind that the sentence(s) in this discussion pertain to the trials
> of the B2B Frame Benchmark testing, in the section that begins:
> 
>   Each trial in the test requires the tester to send a burst of frames (after idle time)
> with the minimum inter-frame gap, and to count the corresponding frames
> forwarded by the DUT.
> 
> Let's say we have a DUT that offers 1.6 seconds of buffering. The Binary Search will
> increase the length of the burst until the buffer drops frames, and then try to find
> the longest burst where frame loss is zero. This simple procedure always waits the
> minimum trial duration for frames to exit the DUT.
> 
> A burst of 1.7 seconds falls well within the coverage of a 2 second trial to count the
> frames and determine that loss has occurred. We don't need a big margin time-
> wise. Also, the trial duration increases with the search parameters: IOW, if the
> tester is still sending frames after 2 seconds, then obviously we can't allow the trial
> to terminate prematurely.
> 
> and we establish an adaptive waiting time for extreme cases, consistent with
> RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the Trial duration
> would be 3.6 seconds.
> 
> BTW, one tester issue we are trying to fix here is the case when the frame header
> processing rate is sufficiently high that no buffer limit is measureable at the large
> frame sizes, and we implore people not to waste time with the B2B Frame
> Benchmark at those frame sizes. One commercial tester sent bursts up to 30
> seconds in length, and then happily reported the impossible result of 30 second
> buffers.
> 
> I want to avoid over-reacting to the buffer bloat case. We all agree that bloated
> buffers are bad, and knowing a DUT has buffering >=1 second may be enough. I
> certainly do not think that trial durations for the B2B Frame benchmark need to be
> as long as you originally suggested, where the overwhelming number of test
> conditions can use a <<1 second burst and 2 second buffer depletion time:
> 
> > > > Hence, the 2 second minimum duration ought to be increased by at
> > > > least a factor of 10.  I'd suggest changing it to 30 seconds or 60 seconds...
> 
> > It also looks like the link to the entire slide deck didn't make it
> > into my original review correctly - that slide deck is at:
> > https://urldefense.com/v3/__http://www.taht.net/*d/lca_tcp3.odp__;fg!!
> > BhdT !0oWabwvUrtVlGQY7ZKnFnMa-
> un_e95tfCJcRKIWG7wECiJkVISb4sjd9Salwehw$
> > .  In addition to slide 6 from this slide deck (Figure 1 in the APNIC
> > blog), slide 14 is also relevant to this discussion.
> >
> > Thanks, --David
> >
> > > > deplete buffers in 1999, but that is no longer reliably the case.
> > > > For example, on-site measurement of the network for the 2020 Linux
> > > > Conference in Australia indicated a at least 1.6 seconds of
> > > > buffering, as indicated by Figure 1 at
> > > >
> > > > .  Experience with bufferbloat suggests that one network device
> > > > was primarily responsible.  Also, see slide 14 in that slide deck.
> > > >
> > > > That 1.6 seconds measured on an actual network is entirely too
> > > > close to 2 seconds for confidence that buffers will be depleted in
> > > > any
> > tested device.
> > > > Hence, the 2 second minimum duration ought to be increased by at
> > > > least a factor of 10.  I'd suggest changing it to 30 seconds or 60
> > > > seconds as convenient round numbers, and providing the rationale
> > > > that increased buffering in WiFi devices, e.g., home "routers," as
> > > > indicated by experience with bufferbloat measurements, is the
> > > > reason
> > for the
> > > increased duration.
> > > >
> > > [acm]
> > > should be increased to accommodate the encountered in the DUT, but
> > > not
> > as a
> > > 1. Some of the virtual network DUTs we are testing now have very
> > > small
> > buffers,
> > > the Binary Search. We need to manage the tension between the time
> > > needed
> > to

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 25 Nov 2020 18:19:50 +0000

David,

Allow me to  points more clearly: 

- I'm relying on the Binary Search aspect of the procedure to measure unexpected large buffer times; searches are an integral part of the procedure.

- These are benchmarking measurements in an isolated test environment, where we will control every source of variability possible to foster repeatability.

- The trial duration has two components: time to conduct sending + receiving, plus additional waiting time for buffer depletion. The test equipment orchestrates this easily, because it is a single device.

I clarified:
The duration of the trial MUST include least 2 seconds in addition to
the time required to send and receive each burst of frames, to ensure that DUT buffers to deplete.

and I'll add:
The upper search limit for the time to send each burst MUST be configurable as high as 30 seconds (buffer time results reported at the configured upper limit are likely invalid, and the test MUST be repeated with a higher search limit).

Al

> Sent: Wednesday, November 25, 2020 9:35 AM
> The 1.6 seconds value is *not* an upper bound - it was measured in
> practice on an operational conference network as part of what appears to
> have been a just-in-time slide preparation exercise.  Hence that 1.6
> seconds of buffering ought to be viewed as being somewhere in the midst of
> the expected distribution of buffer times.  For that reason, the upper
> bound on test duration needs to be set with a safety factor to ensure that
> even if a bizarre DUT has 10 seconds of buffering, the B2B test depletes
> that buffering so that the B2B test is testing the forwarding path as
> intended, not the buffering capacity.
> 
> I agree with this approach provided that there is a test procedure (in the
> B2B test and/or its prerequisites) that will discover the behavior of the
> bizarre DUT with 10 seconds of buffering and adjust the B2B test time to
> ensure that this excessive buffering is depleted.
> 
> Based on this discussion, in addition to that sentence (which I think has
> to contain a "MUST" for situations in which excessive buffering is found),
> I would like to see a brief explanation of how the procedures in the
> draft, along with additional test procedures pulled in from RFC 2544 or
> elsewhere, avoid a 2 second test duration on a DUT with 10 seconds of
> buffering.
> 
> Thanks, --David
> 
> seconds...

---

From: "Black, David" <[email-masked]>
Date: Wed, 25 Nov 2020 21:27:42 +0000

Al,

> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers to
> deplete.
> 
> and I'll add:
> The upper search limit for the time to send each burst MUST be configurable as
> high as 30 seconds (buffer time results reported at the configured upper limit are
> likely invalid, and the test MUST be repeated with a higher search limit).

I think this will work - please make sure to address the scenario in which the search algorithm hits the upper search limit without causing any frame loss (e.g., upper search limit configured to 2 seconds for a bizarre DUT that has 10 seconds of buffering).

Thanks, --David

> Sent: Wednesday, November 25, 2020 1:20 PM
> - I'm relying on the Binary Search aspect of the procedure to measure unexpected
> large buffer times; searches are an integral part of the procedure.
> 
> - These are benchmarking measurements in an isolated test environment, where
> we will control every source of variability possible to foster repeatability.
> 
> - The trial duration has two components: time to conduct sending + receiving, plus
> additional waiting time for buffer depletion. The test equipment orchestrates this
> easily, because it is a single device.
> 
> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers to
> deplete.
> 
> and I'll add:
> The upper search limit for the time to send each burst MUST be configurable as
> high as 30 seconds (buffer time results reported at the configured upper limit are
> likely invalid, and the test MUST be repeated with a higher search limit).
> 
> Al
> 
> 
> > > A burst of 1.7 seconds falls well within the coverage of a 2 second
> > > trial to count the frames and determine that loss has occurred. We
> > > don't need a big margin time- wise. Also, the trial duration
> > > increases with the search parameters: IOW, if the tester is still
> > > sending frames after 2 seconds, then obviously we can't allow the trial to
> terminate prematurely.
> >
> > practice on an operational conference network as part of what appears
> > to have been a just-in-time slide preparation exercise.  Hence that
> > 1.6 seconds of buffering ought to be viewed as being somewhere in the
> > midst of the expected distribution of buffer times.  For that reason,
> > the upper bound on test duration needs to be set with a safety factor
> > to ensure that even if a bizarre DUT has 10 seconds of buffering, the
> > B2B test depletes that buffering so that the B2B test is testing the
> > forwarding path as intended, not the buffering capacity.
> >
> > > I want to avoid over-reacting to the buffer bloat case. We all agree
> > > that bloated buffers are bad, and knowing a DUT has buffering >=1
> > > second may be enough. I certainly do not think that trial durations
> > > for the B2B Frame benchmark need to be as long as you originally
> > > suggested, where the overwhelming number of test conditions can use a <<1
> second burst and 2 second buffer depletion time:
> >
> > I agree with this approach provided that there is a test procedure (in
> > the B2B test and/or its prerequisites) that will discover the behavior
> > of the bizarre DUT with 10 seconds of buffering and adjust the B2B
> > test time to ensure that this excessive buffering is depleted.
> >
> > Based on this discussion, in addition to that sentence (which I think
> > has to contain a "MUST" for situations in which excessive buffering is
> > found), I would like to see a brief explanation of how the procedures
> > in the draft, along with additional test procedures pulled in from RFC
> > 2544 or elsewhere, avoid a 2 second test duration on a DUT with 10
> > seconds of buffering.
> >
> > Thanks, --David
> >
> > > > > [acm] I agree with your observation that there are cases where
> > > > > trial
> > > > duration
> > > > > should be increased to accommodate the encountered in the DUT,
> > > > > but not
> > > > as a
> > > > > 2. The trial duration is a factor in total test duration, where
> > > > > each
> > > > trial is one step in
> > > > > the Binary Search. We need to manage the tension between the
> > > > > time needed
> > > > to
> > > > >     The duration of the trial MUST be at least 2 seconds, to
> > > > > allow
> > DUT
> > > > >     buffers to deplete. When RFC2544 Latency measurements
> > > > > indicate
> > that
> > > > >     large buffers are present in the DUT, the trial duration
> > > > > SHOULD
> > be
> > > > >     increased to ensure that buffer depletion takes place,
> > > > > without
> > unduly
> > > > The overall approach of collecting evidence that there is a
> > > > problem before increasing the 2 second minimum duration is fine,
> > > > but the details appear to need more attention:
> > > >
> > > > 1. Obtaining the RFC 2544 Latency measurements would need to be
> > > > added to Section 4 (Prerequisites) of this draft to ensure that
> > > > the buffer size information is available.
> > > [acm]
> > > common for testers to measure delay on every packet, and report a
> > > true
> > average.
> > > This is enough to alert the operator to the presence of bufferbloat,
> > > but
> > there's no
> > > assurance that the Latency benchmark measures the extent of the
> > > buffer
> > time.
> > >
> > > >
> > > > 3. I would think that the trial duration MUST be increased, not
> > > > just SHOULD be increased if there is evidence of large buffer
> > > > size, as buffer depletion appears to be a necessary characteristic
> > > > of this B2B measurement.
> > > [acm]
> > >
> > > We need to keep in mind that the sentence(s) in this discussion
> > > pertain
> > to the trials
> > >   Each trial in the test requires the tester to send a burst of
> > > frames
> > (after idle time)
> > > with the minimum inter-frame gap, and to count the corresponding
> > > frames forwarded by the DUT.
> > >
> > > Let's say we have a DUT that offers 1.6 seconds of buffering. The
> > > Binary
> > Search will
> > > increase the length of the burst until the buffer drops frames, and
> > > then
> > try to find
> > > the longest burst where frame loss is zero. This simple procedure
> > > always
> > waits the
> > > frames and determine that loss has occurred. We don't need a big
> > > margin
> > time-
> > > tester is still sending frames after 2 seconds, then obviously we
> > > can't
> > allow the trial
> > >        The duration of the trial MUST be at least 2 seconds in
> > > addition
> > to
> > >        the time to send each burst of frames, to allow DUT buffers
> > > to
> > deplete.
> > >
> > > and we establish an adaptive waiting time for extreme cases,
> > > consistent
> > with
> > > RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the
> > > Trial
> > duration
> > > would be 3.6 seconds.
> > >
> > > processing rate is sufficiently high that no buffer limit is
> > > measureable
> > at the large
> > > frame sizes, and we implore people not to waste time with the B2B
> > > Frame Benchmark at those frame sizes. One commercial tester sent
> > > bursts up to
> > 30
> > > seconds in length, and then happily reported the impossible result
> > > of 30
> > second
> > > buffers.
> > >
> > > certainly do not think that trial durations for the B2B Frame
> > > benchmark
> > need to be
> > > as long as you originally suggested, where the overwhelming number
> > > of
> > test
> > > > > > Hence, the 2 second minimum duration ought to be increased by
> > > > > > at least a factor of 10.  I'd suggest changing it to 30
> > > > > > seconds or 60
> > seconds...
> > >
> > > > It also looks like the link to the entire slide deck didn't make
> > > > it into my original review correctly - that slide deck is at:
> > > > .  In addition to slide 6 from this slide deck (Figure 1 in the
> > > > APNIC blog), slide 14 is also relevant to this discussion.
> > > >
> > > > Thanks, --David
> > > >
> > > > > > Cc: [email-masked]; [email-masked];
> > > > > > last- [email-masked]
> > > > > > The draft is in good shape, with one notable exception in
> > > > > > Section
> > 5.2:
> > > > > >
> > > > > >    The duration of the trial MUST be at least 2 seconds, to
> > > > > > allow
> > DUT
> > > > > >    buffers to deplete.
> > > > > >
> > > > > > That duration of 2 seconds has been carried forward from RFC
> > > > > > 2544 without change.  A 2 second duration may have been
> > > > > > sufficient to deplete buffers in 1999, but that is no longer reliably the case.
> > > > > > For example, on-site measurement of the network for the 2020
> > > > > > Linux Conference in Australia indicated a at least 1.6 seconds
> > > > > > of buffering, as indicated by Figure 1 at
> > > > > >
> > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff
> > > > erbl
> > > > oat-
> > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff
> > > > erbl
> > > > oat-
> > > > > > .  Experience with bufferbloat suggests that one network
> > > > > > device was primarily responsible.  Also, see slide 14 in that slide deck.
> > > > > >
> > > > > > close to 2 seconds for confidence that buffers will be
> > > > > > depleted in any
> > > > tested device.
> > > > > > Hence, the 2 second minimum duration ought to be increased by
> > > > > > at least a factor of 10.  I'd suggest changing it to 30
> > > > > > seconds or 60 seconds as convenient round numbers, and
> > > > > > providing the rationale that increased buffering in WiFi
> > > > > > devices, e.g., home "routers," as indicated by experience with
> > > > > > bufferbloat measurements, is the reason
> > > > for the
> > > > > increased duration.
> > > > > >
> > > > > [acm]
> > > > > [acm] I agree with your observation that there are cases where
> > > > > trial
> > > > duration
> > > > > should be increased to accommodate the encountered in the DUT,
> > > > > but not
> > > > as a
> > > > > 2. The trial duration is a factor in total test duration, where
> > > > > each
> > > > trial is one step in
> > > > > the Binary Search. We need to manage the tension between the
> > > > > time needed
> > > > to
> > > > >     The duration of the trial MUST be at least 2 seconds, to
> > > > > allow
> > DUT
> > > > >     buffers to deplete. When RFC2544 Latency measurements
> > > > > indicate
> > that
> > > > >     large buffers are present in the DUT, the trial duration
> > > > > SHOULD
> > be
> > > > >     increased to ensure that buffer depletion takes place,
> > > > > without
> > > > unduly
> > > > > I hope this suggestion resolves your issue; thanks for
> > > > > highlighting
> > it!
> > > > > Al