Subject: Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

From: Lencse Gábor <[email-masked]>
Date: Mon, 1 Jun 2020 11:43:50 +0200

Dear Al,

Thank you very much for your reply. Please see my answers inline. (I 
keep only those parts of the text that I reply to.)

2020.05.23. 19:40 keltezéssel, MORTON, ALFRED C (AL) írta:
>
>> As for the non-overlapping areas of our draft and the documents you
>> cited, I have not found anything in them about our suggestion for
>> "Improved Throughput and Frame Loss Rate Measurement Procedures using
>> Individual Frame Timeout".
>>
>> If you think this one joins well into your efforts to update RFC 2544,
>> then we could focus on this one first, and deal with some others later
>> one by one in (in different documents).
>>
>> What do you think of it?
> [acm]
> Using a constant frame timeout for declaration of Loss (and delay) is
> much like the IPPM WG metrics and methods (see RFC 7679 and RFC 7680)
> for the production network measurements (Tmax). A constant waiting time for
> frames to arrive at the receiver simply excludes frames on an on-going
> basis. RFC 2544 has a waiting time at the end of the trial, where the
> tester must wait for 2 seconds for buffers to clear but the first frame
> sent has the entire trial duration+2sec to arrive.
>

Yes, this is why I feel that some of the frames we count as "received" 
may be completely useless for the applications.

> Perhaps the biggest potential change to the Throughput definition is
> whether or not we demand frame correspondence between sender and receiver,
> so that we can calculate one-way delay. RFC 2544 and ETSI NFV TST009 only
> require equal send and receive frame counts to satisfy the zero loss
> criteria in the Throughput benchmark definition. But in ETSI NFV TST009,
> the Capacity at X% Loss Ratio metric-variant definition begins to infer
> frame correspondence. Later, the definitions of (one-way) Latency and
> Delay Variation and Loss allow a Sample Filter, which could be a constant
> maximum time-out for individual frames, as you suggest. Note that a Sample
> Filter could be applied in post-measurement processing, assuming all the
> delays are available.

I have implemented siitperf-pdv (part of siitpef, available from: 
https://github.com/lencsegabor/siitperf ) in a way that depending on the 
value of its last parameter (*frame timeout*), it can do the 
post-processing in two different ways:
- If the value of *frame timeout* is 0, then proper PDV measurement is 
done.
- If the value of *frame timeout* is higher than zero, then rather a 
special throughput (or frame loss rate) measurement is performed, where 
the tester checks *frame timeout* for each frame individually: if the 
measured delay of a frame is longer than the timeout, then the frame is 
reclassified as lost.

> But the real question in my mind now, after looking into the possibilities
> to take frame correspondence with a fixed timeout into account, is whether
> we can find some examples where adding timeout criteria makes a significant
> difference for Throughput measurements that would not be accounted for by
> revised/modernized Latency and new Delay Variation Benchmarks and
> metric variants?

With other words (as my co-author, Keiichi Shima expressed):

"Can defining a per-frame timeout test provide more sense than the delay 
variation test?"

IMHO, we can answer "yes".

Let us consider the following example. A delay sensitive application can 
tolerate at most 100ms delay and 0.01% frame loss. (The "lost" frames 
include also frames with higher than 100ms delay.)

Section 7.3.1. of RFC 8219 ( 
https://tools.ietf.org/html/rfc8219#section-7.3.1) defines PDV as follows:

    PDV = D99.9thPercentile - Dmin

    Where:

    o  D99.9thPercentile = the 99.9th percentile (as described in
       [RFC5481  <https://tools.ietf.org/html/rfc5481>]) of the one-way delay for the stream

    o  Dmin = the minimum one-way delay in the stream

There can be two problems with using PDV:
- As  PDV uses 99.9th percentile, and even if we measure PDV as 99ms, it 
gives a guarantee for 99.9% of the frames and 0.1% of them may have 
higher than 100ms latency, whereas our system tolerates only 0.01% frame 
loss (including frames with higher latency than 100ms)
- Dmin is subtracted from the 99.9th percentile, thus the final result 
of the PDV measurement is not exactly, what we need.

IMHO, our suggested method can provide a better solution:
- The *frame timeout* parameter of siitperf-pdv should be set to 100ms
- The bash shell script that performs the binary search (and executes 
siitperf-pdv in every single step) should allow 0.01% frame loss.
Thus the measurement can be easily performed.

What do you think of it?

Best regards,

Gábor

---

From: Gábor LENCSE <[email-masked]>
Date: Sun, 14 Jun 2020 20:08:44 +0200

Dear Al Morton,

I would like to ask your advice regarding the usage of different source 
and destination port numbers.

On 5/23/2020 19:40, MORTON, ALFRED C (AL) wrote:
>
>> Thank you very much for your reply and for all the references! I was not
>> aware of them and now I see that they have already solved several of the
>> issues we wanted to handle. (Even some issues, which we did not mention
>> in the draft, but I had in my mind, like using a high number of
>> different source port numbers for the tests.)
> [acm]
> Yes, starting out with 1 flow will probably reap the best performance,
> but this can be a distinguishing factor, as seen with virtual switches
> (OVS, OVS+DPDK and VPP).

RFC 4814 requires that (emphasis by bold font is from me):

    Test instruments have the capability of generating packets with
    random TCP and UDP source and destination port numbers.  Known
    destination port numbers are often required for testing application-
    layer devices.  However, unless known port numbers are specifically
    required for a test, it is RECOMMENDED to use*pseudorandom and uniformly distributed values for both source and 
destination port numbers*.

    In addition, it may be desirable to pick pseudorandom values from a
    selected pool of numbers.  Many services identify themselves through
    use of reserved destination port numbers between*1 and 49151*
    inclusive.  Unless specific port numbers are required, it is
    RECOMMENDED to pick randomly distributed destination port numbers
    between these lower and upper boundaries.

    Similarly, clients typically choose source port numbers in the space
    between*1024 and 65535*  inclusive.  Unless specific port numbers are
    required, it is RECOMMENDED to pick randomly distributed source port
    numbers between these lower and upper boundaries.

Source: https://tools.ietf.org/html/rfc4814#section-4.5

I want to implement this feature into siitperf, my RFC 8129 compliant 
SIIT tester: https://github.com/lencsegabor/siitperf

My question is, what parameters and parameter combinations can be 
meaningful?

For example:

random_src_port yes|no
if yes, then: first_src_port, last_src_port

random_dst_port yes|no
if yes, then: first_dst_port, last_dst_port

RFC 2544 requires bidirectional traffic. It also requires testing with a 
single source and destination address pair first, and the using 256 
destination networks. In siitperf, I implemented it in a way that the 
number or destination networks can be set from 1 to 255 *per direction*.

My question is if it is worth giving an opportunity to set the above 
parameters *per direction*?

Thank you very much for your advice in advance!

Best regards,

Gábor Lencse

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 20 Jun 2020 18:47:56 +0000

Hi Gábor,

Please see my reply below, and sorry for the latency!

BMWG,
even though some messages to the list might be addressed to me, feel free to chime-in with your opinions and experiences too!

Al
bmwg co-chair

From: bmwg [mailto:[email-masked]] On Behalf Of Gábor LENCSE
Sent: Sunday, June 14, 2020 2:09 PM
To: [email-masked]
Subject: [bmwg] Question regarding source and destination port numbers -- Re: An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

Dear Al Morton,

I would like to ask your advice regarding the usage of different source and destination port numbers.
On 5/23/2020 19:40, MORTON, ALFRED C (AL) wrote:

[acm]

(OVS, OVS+DPDK and VPP).

   required for a test, it is RECOMMENDED to use pseudorandom and

   uniformly distributed values for both source and destination port

   numbers.

   use of reserved destination port numbers between 1 and 49151

   between 1024 and 65535 inclusive.  Unless specific port numbers are

Source: https://tools.ietf.org/html/rfc4814#section-4.5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc4814-23section-2D4.5&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=d2qGiVfbiAiPe3jfEeeeSe99GibSyq-T5I8dk2A8y6U&s=M88c6YG_1os47Hvh6hSUxp67F3fd1QDVPcidQLqFTzA&e=>

I want to implement this feature into siitperf, my RFC 8129 compliant SIIT tester: https://github.com/lencsegabor/siitperf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lencsegabor_siitperf&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=d2qGiVfbiAiPe3jfEeeeSe99GibSyq-T5I8dk2A8y6U&s=6qeskk8jE5s4LsOS7G66yJIM64xxdZR40ujPT4jRHhU&e=>

My question is, what parameters and parameter combinations can be meaningful?

For example:

random_src_port yes|no

random_dst_port yes|no

RFC 2544 requires bidirectional traffic. It also requires testing with a single source and destination address pair first, and the using 256 destination networks. In siitperf, I implemented it in a way that the number or destination networks can be set from 1 to 255 per direction.

My question is if it is worth giving an opportunity to set the above parameters per direction?

[acm] I think so, you would then have full control over the 4-tuple of traffic in each direction.

I would like to hear form the Next-Gen Firewall authors and advocates on this topic, too.

Al (as a participant)

Best regards,

Gábor Lencse

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 20 Jun 2020 19:41:18 +0000

Hi Gábor,

Please see long-delayed replies below: Sorry !
I hope the replies are still useful!

Al

From: bmwg [mailto:[email-masked]] On Behalf Of Lencse Gábor
Sent: Monday, June 1, 2020 5:44 AM
To: [email-masked]
Subject: Re: [bmwg] An Upgrade to Benchmarking Methodology for Network Interconnect Devices -- Fwd: New Version Notification for draft-lencse-bmwg-rfc2544-bis-00.txt

Dear Al,

Thank you very much for your reply. Please see my answers inline. (I keep only those parts of the text that I reply to.)
2020.05.23. 19:40 keltezéssel, MORTON, ALFRED C (AL) írta:

What do you think of it?

[acm]

Yes, this is why I feel that some of the frames we count as "received" may be completely useless for the applications.
[acm]
Since we are testing a small part of the end-2-end path between users and their applications (a single DUT in the network, or a few DUTs in a SUT), we have to be careful when setting thresholds for useless frames, both loss and delay.

delays are available.

I have implemented siitperf-pdv (part of siitpef, available from: https://github.com/lencsegabor/siitperf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lencsegabor_siitperf&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=i-yejWHFzdhCqrAxYUSaq-iSfAYq4sR8cvP-uc5Yjco&s=ywCqqfJV6g6Mf2GypEyUWw3GF4NrykY1O2EMRKoIktY&e=> ) in a way that depending on the value of its last parameter (frame timeout), it can do the post-processing in two different ways:
- If the value of frame timeout is 0, then proper PDV measurement is done.
- If the value of frame timeout is higher than zero, then rather a special throughput (or frame loss rate) measurement is performed, where the tester checks frame timeout for each frame individually: if the measured delay of a frame is longer than the timeout, then the frame is reclassified as lost.

metric variants?

"Can defining a per-frame timeout test provide more sense than the delay variation test?"

Let us consider the following example. A delay sensitive application can tolerate at most 100ms delay and 0.01% frame loss. (The "lost" frames include also frames with higher than 100ms delay.)

Section 7.3.1. of RFC 8219 ( https://tools.ietf.org/html/rfc8219#section-7.3.1<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc8219-23section-2D7.3.1&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=i-yejWHFzdhCqrAxYUSaq-iSfAYq4sR8cvP-uc5Yjco&s=eF00ug97jcGIrilQQom5RBsDOfrRiTj4bPNkEKrREGw&e=>) defines PDV as follows:

   Where:

      [RFC5481<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc5481&d=DwMDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=i-yejWHFzdhCqrAxYUSaq-iSfAYq4sR8cvP-uc5Yjco&s=CAU9on15yDMbAzOQUl8zVEwPzoZMwnEXL0lUNCFdElU&e=>]) of the one-way delay for the stream

There can be two problems with using PDV:
- As  PDV uses 99.9th percentile, and even if we measure PDV as 99ms, it gives a guarantee for 99.9% of the frames and 0.1% of them may have higher than 100ms latency, whereas our system tolerates only 0.01% frame loss (including frames with higher latency than 100ms)
- Dmin is subtracted from the 99.9th percentile, thus the final result of the PDV measurement is not exactly, what we need.
[acm]
I think that you need to evaluate the one-way delay distribution w.r.t. the application threshold of 100ms in your example. This means that any packet with one-way delay >100ms would be declared lost by enforcing the fixed time-out.  We don’t need to know the PDV to perform these steps, if just a simple evaluation of one-way delay against a threshold.

IMHO, our suggested method can provide a better solution:
- The frame timeout parameter of siitperf-pdv should be set to 100ms
- The bash shell script that performs the binary search (and executes siitperf-pdv in every single step) should allow 0.01% frame loss.
Thus the measurement can be easily performed.

What do you think of it?
[acm]
So, as I alluded-to up-front, it is very hard to set application-specific thresholds in a useful way when benchmarking a single DUT.  We don’t know what the rest of the network will look like, and therefore how much additional delay and loss will be contributed by the “rest of the network”.
I think this is why we have always chosen Simple justifications for Benchmark thresholds, like zero loss, but it’s ok to investigate around these thresholds, and it sounds as though you will add that capability to your test tool

Here’s another point: If you are testing a virtualized network/DUT, you may see substantial benefit from the more advanced Binary Search with Loss Verification in ETSI NFV TST009. The justification appears in the text (clause 12.3.3) and the associated Annexes B and C. See [0]

hope this helps,
Al (participant)

[0] https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST009ed341/NFV-TST009ed341v341_001.docx

Best regards,

Gábor