Subject: Re: [bmwg] draft-morton-bmwg-b2b-frame-05

From: "Maciek Konstantynowicz (mkonstan)" <[email-masked]>
Date: Mon, 24 Jun 2019 11:32:04 +0000

Hi Al,

I finally had time to properly review this draft.

It is a very well written benchmarking specification that does address
an important area of measuring ingress packet buffer capacity used for
adapting packet arrival rate to not-always-steady packet processing rate
capability in any packet processing system.

It does take into consideration specifics of NFV systems (i.e.
vswitches, other VNFs) where this capability is of elevated importance
based on experience from OPNFV VSPERF benchmarking (backed up by a
number of measurement references).

The draft rightly notes that the size of this buffer matters as it
enables storing packets disruptions in the software packet processor
operation due to some external system interference. This a very
important aspect of NFV and as such I would suggest to make this aspect
more prominent in the draft, instead of burying it at the end of section
3, as it is a case now.

Two other general observations:

- The goal is measuring (ingress) buffer size in front of HeaderProc,
   per DUT definition in section 3.
   - This works if there is no other buffering in the system under test.
   - Suggest to add a paragraph dictating the setup where no egress
     queue build up is possible.

- Proposed methodology works for setups where the DUT's (composed here
   of "Buffer" and "HeaderProc" functions) behaviour can be measured
   with external tester:
   - This requires that any "noise" impacting DUT's behaviour is
     identified and isolated.
   - Potential sources of "noise":
     - In-path active components, other than DUT, noted in draft as
       "Ingress", "Egress".
     - Operating system environment interrupting DUT operation.
     - Shared resource(s) access collisions between DUT and some
       off-path component(s), impacting DUT's behaviour, a.k.a. "noisy
       neighbour" problem.
   - To deal with this e.g. for NFV DUT, the draft suggests to use
     enhanced Binary Search with Loss Verification as specified in
     [TST009], sec. 5.2. Plus repeating the test N times, sec. 5.3.
     - Agree this is the right way to isolate the DUT behaviour.
   - But I am puzzled when it comes to proposed calculation of
     "Corrected DUT Buffer Time", sec. 5.4.
     - There "Measured Throughput" is measured as per RFC2544, instead
       of referring to Binary Search with Loss Verification [TST009], or
       possibly using MLRsearch [draft-vpolak-mkonstan-bmwg-mlrsearch]
       to find NDR (non drop rate) and PDR (partial drop rate) and use
       those to calculate the actual DUT buffer time.
     - Another potential candidate for the "Measured Throughput" is the
       maximum measured throughput regardless of loss, metric popular
       with academics dealing with software network processing, also
       defined in FD.io CSIT as Maximum Receive Rate.

Hope this makes sense.

Cheers,
-Maciek