Subject: [bmwg] My review of draft-ietf-bmwg-mlrsearch-01

From: Gabor LENCSE <[email-masked]>
Date: Mon, 4 Oct 2021 22:33:42 +0200

Dear Authors and BMWG members,

I have reviewed 
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-mlrsearch-01

General comments
-------------------------

- I have found only one significant issue, and the majority of my 
findings are only nits (usually typos).

- I think it was a good idea to give first a general overview of the 
algorithm, then present the details (in a simplified way), then give 
some more details, and finally include an example with numbers. I humbly 
admit that I still had parts that I did not understand for the first 
reading. I wonder, if there is some more formal method that could help. 
(I am sorry, I cannot propose one.)

Discuss:
----------

Issue #1 -- The measurement result depends of the performance of the Tester

Page 17:

           +  The final bounds describe the performance of not just SUT,
              but of the whole system, including the traffic generator.

I believe that final result of benchmarking measurements SHOULD depend 
only on the performance of the DUT/SUT and it SHOULD NOT depend on the 
performance of the Tester.

I am aware that this situation is the consequence of the condition 
described on page 16:

    5.  Duration stretching.

        *  In some cases, traffic generator may get overloaded, causing
           it to take significantly longer (than duration) to send all
           packets.

Yes, it may happen, especially, if we use software based traffic 
generators and the performance of the hardware is not enough for a given 
rate. However, it can be (and I think it should be) handled before the 
measurements. A possible solution can be to use a self-test of the 
Tester. E.g. if one would like to use the Tester let us say up to 5 
million frames per second rate, then first the Tester should be looped 
back and be tested at 5.5M fps rate, if it can surely transmit and 
receive the frames within the required duration. (The 10% is the 
performance reserve. It is a high handed choice, I usually use this value.)

We recommended this solution for DNS64 benchmarking measurement in RFC 
8219: https://datatracker.ietf.org/doc/html/rfc8219#section-9.2.1

All further things are much less significant, they are rather questions.

Issue # 2 -- well behaving and bad behaving systems

As far as I understood, there are some well behaving systems: their 
throughput is nearly the same with different trial duration lengths. 
Whereas this property is highly utilized to reduce the overall execution 
time, the correctness of the results is guaranteed by using the proper 
trial duration length at the final measurement series.

I am not sure, but perhaps it would be worth mentioning some potential 
root causes of the "bad behavior" of systems. Such a thing(s) can be the 
buffering of the SUT and the 2s timeout Required by Section 23 of  RFC 2544:

    d) Wait for two seconds for any residual frames to be received.

If the trial is only 1s long and the SUT can forward packets and 1Mfps, 
but it can store 2M frames in its buffers, then its throughput may be 
(wrongfully) measured as 3Mfps.

(We have described this issue in the case of DNS64 testing in this 
paper: 
http://www.hit.bme.hu/~lencse/publications/ECC-2017-B-M-DNS64-revised.pdf 
See page 8 and search for the word "gaming".)

Perhaps the issue could be mitigated by significantly decreasing the 
timeout from 2s for the measurements using shorter test duration than 
the final one.

Issue #3

I wonder about the correctness of the following constants:

page 17

5.1.1.  FD.io CSIT Input Parameters

    1.  *max_rate* - Typical values: 2 * 14.88 Mpps for 64B 10GE link
        rate, 2 * 18.75 Mpps for 64B 40GE NIC (specific model).

Using 64-byte frames, 14.88Mpps is correct for 10GE, but 18.75 Mpps for 
40GE sounds strange to me.

    2.  *min_rate* - Value: 2 * 9001 pps (we reserve 9000 pps for latency
        measurements).

I do not understand the comment in parenthesis. AFAIK, latency 
measurements are to be performed at the rate determined by the 
throughput measurement.

    3.  *final_trial_duration* - Value: 30.0 seconds.

RFC 2455 REQUIRES 60.0 seconds.

    4.  *initial_trial_duration* - Value: 1.0 second.

And the timeout is still 2 seconds as required by RFC 2455? Then one 
trial will still last at least 1+2=3 seconds.

Nits (usually typos)
-------------------------

page 2: (as well as on all the following pages)

Internet-DraMultiple Loss Ratio Search for Packet Throughput July 2021

The title of the draft is too long to be displayed correctly in the page 
header. In this case, you can specify an abbreviated title in the XML 
file. It may look like this:

     <title abbrev="MLRsearch">Multiple Loss Ratio Search for Packet 
Throughput (MLRsearch)</title>

page 3:

imput --> input

page 4:

tighterst --> tightest

bandwith --> bandwidth (appears twice)

page 5:

A comma is missing:

In practice two rates --> In practice, two rates

page 8:

A comma is missing:

In general MLRsearch --> In general, MLRsearch

page 13:

overal --> overall

page 16

byt --> ???

overriden --> overridden

page 17:

offes --> offers

start --> starts

page 18:

immediatelly --> immediately

page 20:

purposes.Any --> purposes. Any

page 21:

    Maciek Konstantynowicz (editor)

    Vratko Polak (editor)

It may be correct, I just wonder, who is the author, if both authors are 
(only) editors. :-)

That's all.

Best regards,

Gábor