Subject: [bmwg] Agenda for BMWG session at IETF-109

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 7 Nov 2020 21:47:28 +0000

BMWG:

The first draft agenda is available [0] for bashing.

I constructed this agenda based on updated drafts, messages to the list, and some messages to the co-chairs.  IF I OVERLOOKED an item, please let me know! 

There are lots of drafts to read to prepare for a productive session at IETF-109! 

Note the time of the session: 12:00-14:00 UTC+7 on Thursday, November 19, 2020. 

Please check your local time! [1]
In some time zones, like US PST, this meeting will start and end on Wednesday evening. 
The US PST session is 2100 - 2300, on November 18 !

Authors, if you are not planning to attend this session, please let me know!

Al
bmwg co-chair

[0] https://www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00

[1] https://www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224

---

From: Lencse Gábor <[email-masked]>
Date: Sun, 8 Nov 2020 19:21:34 +0100

Dear Al,

Thank you very much for including our draft, too:

  - An Upgrade to Benchmarking Methodology for Network Interconnect Devices
    https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00
    status:
    - no updated text, but two exchanges on the list:
    >>>>> add <<<<<<

Unfortunately, we did not have time to progress with the draft. So now 
we do not have anything to present regarding the draft. But we do not 
want to abandon it. I hope that we'll be able to progress with it before 
IETF-110.

    - Several discussion threads related to this draft on list:
        Many other RFC2544 updates:
    https://mailarchive.ietf.org/arch/msg/bmwg/yEzFCign03ZveEUkvXsEevm8dDU/
        strict packet time-outs for PDV and the siitperf implementation/tool
    https://mailarchive.ietf.org/arch/msg/bmwg/50qoL0gxTEKGU6CkUwPIf8FO-hc/
        different source and destination port numbers
    https://mailarchive.ietf.org/arch/msg/bmwg/Y3XIteCBVMScSolsagPH_FNoi_E/

However, I have results regarding siitperf, my RFC 8219 compliant SIIT 
tester. (The usage of random port numbers is working well. I'm currently 
working on calibrating it with a legacy RFC 2544 tester by benhmarking 
the same DUT with the two devices and comparing their results.) If there 
is interest in the WG for it, I would be happy to present. But I do not 
want to push it.

So, what do you think?

Best regards,

Gábor

07/11/2020 22:47 keltezéssel, MORTON, ALFRED C (AL) írta:
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: Vladimir Vassilev <[email-masked]>
Date: Wed, 11 Nov 2020 09:43:32 +0100

Hi Gábor,

On 08/11/2020 19.21, Lencse Gábor wrote:
> tester. (The usage of random port numbers is working well. I'm 
> currently working on calibrating it with a legacy RFC 2544 tester by 
> benhmarking the same DUT with the two devices and comparing their 
> results.) If there is interest in the WG for it, I would be happy to 
> present. But I do not want to push it.
>
> So, what do you think?

I looked into the implementation of siitperf. It is a well designed 
reference implementation for RFC 8219 and a useful DPDK API usecase and 
example in general.

Maybe it is a long shot but I was wondering if you are participating in 
the ongoing Hackathon? We are working on a relevant project that 
implements a general network interconnect tester based on a YANG model 
[1]. Instead of a single executable implementing a benchmark test (e.g. 
RFC2544, RFC8219 etc.) one can split the design into control plane 
(search logic, reconfiguration of the generator for different trials, 
reading status) and data plane (the timing critical frame input/output 
operations and timestamping). The control plane is managed with a python 
script over NETCONF [2] and the data plane workload is processed by 
software (native code in C) [3] or hardware logic (Verilog) [4] that is 
started/enabled by the NETCONF server for each interface when traffic 
generator configuration is committed.  The command line looks like this 
(based on the YANG model for a generator with single stream):

$ ./traffic-generator --interface=eth0 --frame-size=64 
--interframe-gap=20 --interburst-gap=124999852 \
    --frames-per-burst=2 --total-frames=4 
--realtime-epoch="2020-11-10T13:00:00.000000000Z" 
--interface-speed=1000000000
--frame-data="123456789ABCDEF01234567808004500002E000000000A112CBCC0000201C0000202C0200007001A0000000102030405060708090A0B0C0D0E0F10119CD50E0F"

We have an alternative traffic-generator-gmii tool [4] that instead of 
using software to generate the packets writes to the registers of a 
hardware core we have designed in Verilog. So replacing the command name 
results in deterministic generation for the systems that have the core 
available.

We were thinking of a third option based on DPDK 
(traffic-generator-dpdk) could be a compromise that is more 
deterministic then the default (Unix userspace) option and would not 
require programmable logic like the hardware dependent one.

We could augment the YANG model adding choice and an alternative to the 
static frame-data specifying the range and algorithm for generation of a 
pseudorandom port numbers in the specified range as a second reference 
implementation and do some testing of common DUTs and compare the 
results produced by the different implementations.

References:

[1] 
https://datatracker.ietf.org/doc/draft-vassilev-bmwg-network-interconnect-tester/

[2] 
https://github.com/vlvassilev/litenc/blob/master/tntapi/example/ietf-network-interconnect-tester/test-rfc2544-throughput.py 
(seems the script diverged from the initial  throughput sec. 26.1 target)

[3] 
https://github.com/vlvassilev/yuma123/blob/master/example-modules/ietf-traffic-generator/traffic-generator.c

[4] 
https://github.com/vlvassilev/network-interconnect-tester-cores/blob/master/lib/hw/lsi/cores/traffic_generator_gmii/hdl/traffic_generator_gmii.v

/Vladimir

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 13 Nov 2020 18:29:43 +0000

Hi Gábor,

sorry for the delay replying - another very busy week here!

We can try to fit in your proposed short presentation on:
results regarding siitperf, my RFC 8219 compliant SIIT tester. (The usage of random port numbers is working well. I'm currently working on calibrating it with a legacy RFC 2544 tester by benchmarking the same DUT with the two devices and comparing their results.)

maybe title and 2 slides (setup and results)?  at the end of the meeting.  we might not have time in which case I hope you understand and we will try to cover the topic at another opportunity. But BMWG'ers will have your slides to review and discuss on the mailing list, in any case.

Al
bmwg co-chair

From: bmwg [mailto:[email-masked]] On Behalf Of Lencse Gábor
Sent: Sunday, November 8, 2020 1:22 PM
To: [email-masked]
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Dear Al,

   https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zQwjDDPN$>

   status:

   >>>>> add <<<<<<

Unfortunately, we did not have time to progress with the draft. So now we do not have anything to present regarding the draft. But we do not want to abandon it. I hope that we'll be able to progress with it before IETF-110.

   https://mailarchive.ietf.org/arch/msg/bmwg/yEzFCign03ZveEUkvXsEevm8dDU/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/yEzFCign03ZveEUkvXsEevm8dDU/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zWLvhnqp$>

   https://mailarchive.ietf.org/arch/msg/bmwg/50qoL0gxTEKGU6CkUwPIf8FO-hc/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/50qoL0gxTEKGU6CkUwPIf8FO-hc/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zSKCWhzH$>

   https://mailarchive.ietf.org/arch/msg/bmwg/Y3XIteCBVMScSolsagPH_FNoi_E/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/Y3XIteCBVMScSolsagPH_FNoi_E/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zchbk9Re$>
However, I have results regarding siitperf, my RFC 8219 compliant SIIT tester. (The usage of random port numbers is working well. I'm currently working on calibrating it with a legacy RFC 2544 tester by benhmarking the same DUT with the two devices and comparing their results.) If there is interest in the WG for it, I would be happy to present. But I do not want to push it.

So, what do you think?

Best regards,

Gábor

BMWG:

Al

bmwg co-chair

[0] https://www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00<https://urldefense.com/v3/__https:/www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zR12tFsp$>

[1] https://www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224<https://urldefense.com/v3/__https:/www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zQr1Tl1V$>

bmwg mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zU14IPVJ$>

---

From: Lencse Gábor <[email-masked]>
Date: Sat, 14 Nov 2020 12:09:18 +0100

Dear Vladimir,

Thank you very much for your e-mail. Your general network interconnect 
tester based on a YANG model sounds interesting to me. (I was not aware 
of your draft [1] before.) As far as I now understand it, on the one 
hand, our projects are similar, and on the other hand, they are 
complementary.

Similarity:
Both your YANG model based tester and my siitperf uses a split design. I 
think your concept with data plane and control plane is more clear, but 
I also used a similar idea: my C++ code is responsible for the data 
plane and my bash shell scripts are responsible for the control plane.

Differences:
- Your project implements a general tester, whereas siitperf is very 
specific (designed for SIIT, including IPv4 or IPv6 routing) and it uses 
different executables for different measurements like throughput, 
latency, packet-delay variation.
- You used the two "extreme" solutions in my eyes: hardware and socket 
interface, whereas siitperf uses DPDK, which is in between of the two.

I think it can be a good idea for you to include also a DPDK version as 
a balanced compromise. Let me give you some performance data to support 
my suggestion:

On the one hand, I could test authoritative DNS servers by dns64perf++, 
which uses TCP/IP socket interface programming, by sending and receiving 
3.3 million queries per second using a DELL PowerEdge R430 server: I 
used 16 cores for sending and 16 cores for receiving. Please refer to my 
open access paper for the details: G. Lencse, "Benchmarking 
Authoritative DNS Servers", /IEEE Access/, vol. 8. pp. 130224130238, 
July 2020. DOI: https://doi.org/10.1109/ACCESS.2020.3009141

On the other hand, the DPDK-based siitperf could send and receive more 
than 7 million frames per second _per direction_ using the same type of 
server and only a single core for sending and another for receiving. (In 
a bidirectional test, 4 cores in all, but then siitperf sent and 
received twice 7 million frames per second.) Please refer to my open 
access paper for the details: G. Lencse, "Design and Implementation of a 
Software Tester for Benchmarking Stateless NAT64 Gateways", /IEICE 
Transactions on Communications/, DOI: 
https://doi.org/10.1587/transcom.2019EBN0010

I think the numbers are convincing. :-)

I admit that DPDK has some requirements for the CPUs and especially for 
the NICs, but there are a lot of supported hardware: 
https://core.dpdk.org/supported/

Thank you very much for inviting me into your project at Hackathon. 
Unfortunately, I am too busy to take part in it. However, I am happy to 
share my experience with you.

I have documented the design of siitperf in my above mentioned paper and 
the source code is also available for you as a free software. I would be 
happy if your team could reuse some part of it. The implementation of 
the RFC 4814 random port feature is currently in the "varport" branch, 
which I plan to merge into the master branch later on. As for random 
number generation, 64-bit Mersenne Twister (std::mt19937_64) is fast enough.

Currently the accuracy of frame timing is the greatest challenge for me. 
The inter-sending time of software testers is usually rather inaccurate 
at higher (e.g. several million
frames per second) rates. I have found a good paper about it, which 
compares some software packet generators. The authors did measurements 
with a NetFPGA device. If you plan to use DPDK, I highly recommend you 
to read their paper: P. Emmerich, S. Gallenmüller, G Antichi, A. W. 
Moore, G. Carle, "Mind the gap - A comparison of software packet 
generators", in: /2017 ACM/IEEE Symposium on Architectures for 
Networking and Communications Systems (ANCS)/, Beijing, China, 2017, 
doi: https://doi.org/10.1109/ANCS.2017.32

I have just finished the first round of testing the accuracy of siitperf 
in the sense, how the inaccurate timing may influence its results. I 
plan to finish my paper about it soon and it will be available from my 
list of publications: http://www.hit.bme.hu/~lencse/publications/

Perhaps I can also include some of its results into the slides of my 
presentation about siitperf at the BMWG session of IETF 109.

Just one advice concerning DPDK. Please be careful, when choosing its 
version to use. I used version 16.11.9-1+deb9u1 distributed in Debian 
9.9. However, my code does not compile with some newer DPDK versions.

I want to emphasize that I am not an expert of DPDK, but I am happy to 
help you, if I can. :-)

And let me ask you some questions about how your general tester could be 
used for my purposes. Siitperf is only the first step in the series of 
the RFC 8219 compliant testers necessary for benchmarking the five most 
important IPv4aaS technologies (464XLAT, DS-Lite, MAP-T, MAP-E, lw4o6), 
which we plan to use for the comparison of their performance as a part 
of our draft: 
https://tools.ietf.org/html/draft-lmhp-v6ops-transition-comparison

Perhaps it is not so difficult to use your tester for benchmarking SIIT 
implementations. (Even if I have siitperf, it could be interesting to 
compare its results with that of your tester!) However, I cannot see, if 
it is possible to benchmark the B4 or AFTR devices of DS-Lite (RFC 6333) 
with your general tester. Is it possible to send encapsulated packets 
and/or to decapsulate them with your tester?

If you allow me some critical comments regarding your tester, let me 
mention two things.

1. It seems from your example that your tester sends bursts of frames. I 
understand that you can achieve higher frame rates in this way, however 
using a burst size higher than 1 makes your tests non compliant of RFC 
2544 / RFC 5180 / RFC 8219 as they require a constant frame rate.

2. It seems that you use integers to express the delay between frames 
(either within a burst or between bursts). Originally, I used the same 
solution to avoid a fixed point division: I used cpf (clock-ticks per 
frame), but it limited the resolution of the frame rates. (Just a 
numeric example: you have a 2GHz CPU and you want a 4 million frames per 
second rate. It means 2,000,000,000Hz/4,000,000fps=500cpf. The first 
possible highest frame rate is: 2,000,000,000Hz/499cpf=4,008,016fps.) I 
do not state that it is very bad, but I did not like it and I rather 
specify the required frame rate. (It also means that I use a fix point 
division in every single sending cycle. This is the cost of accuracy.)

Best regards,

Gábor

11/11/2020 09:43 keltezéssel, Vladimir Vassilev írta:
>> However, I have results regarding siitperf, my RFC 8219 compliant 
>> SIIT tester. (The usage of random port numbers is working well. I'm 
> reference implementation for RFC 8219 and a useful DPDK API usecase 
> and example in general.
>
> Maybe it is a long shot but I was wondering if you are participating 
> in the ongoing Hackathon? We are working on a relevant project that 
> [1]. Instead of a single executable implementing a benchmark test 
> (e.g. RFC2544, RFC8219 etc.) one can split the design into control 
> plane (search logic, reconfiguration of the generator for different 
> trials, reading status) and data plane (the timing critical frame 
> input/output operations and timestamping). The control plane is 
> managed with a python script over NETCONF [2] and the data plane 
> workload is processed by software (native code in C) [3] or hardware 
> logic (Verilog) [4] that is started/enabled by the NETCONF server for 
> each interface when traffic generator configuration is committed.  The 
> command line looks like this (based on the YANG model for a generator 
> with single stream):
>
> hardware core we have designed in Verilog. So replacing the command 
> name results in deterministic generation for the systems that have the 
> core available.
>
> We could augment the YANG model adding choice and an alternative to 
> the static frame-data specifying the range and algorithm for 
> generation of a pseudorandom port numbers in the specified range as a 
> second reference implementation and do some testing of common DUTs and 
> compare the results produced by the different implementations.
>
> References:
>
> [1]

---

From: Lencse Gábor <[email-masked]>
Date: Sat, 14 Nov 2020 13:26:10 +0100

Dear Al,

Thank you very much for this opportunity! I understand and accept the 
risk, that we may run out of time and then I will not be able to present 
my slides.

I would like to send my slides to you for a review before the session. 
What is your deadline?

Best regards,

Gábor

13/11/2020 19:29 keltezéssel, MORTON, ALFRED C (AL) írta:
>
> Hi Gábor,
>
> sorry for the delay replying – another very busy week here!
>
> results regarding siitperf, my RFC 8219 compliant SIIT tester. (The 
> usage of random port numbers is working well. I'm currently working on 
> calibrating it with a legacy RFC 2544 tester by benchmarking the same 
> DUT with the two devices and comparing their results.)
>
> maybe title and 2 slides (setup and results)?  at the end of the 
> meeting.  we might not have time in which case I hope you understand 
> and we will try to cover the topic at another opportunity. But 
> BMWG’ers will have your slides to review and discuss on the mailing 
> list, in any case.
>
> Al
>
> bmwg co-chair
>
> *From:*bmwg [mailto:[email-masked]] *On Behalf Of *Lencse Gábor
> *Sent:* Sunday, November 8, 2020 1:22 PM
> *To:* [email-masked]
> *Subject:* Re: [bmwg] Agenda for BMWG session at IETF-109
>
> Dear Al,
>
>     https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zQwjDDPN$>
>     status:
>     >>>>> add <<<<<<
> want to abandon it. I hope that we'll be able to progress with it 
> before IETF-110.
>
>     https://mailarchive.ietf.org/arch/msg/bmwg/yEzFCign03ZveEUkvXsEevm8dDU/  <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/yEzFCign03ZveEUkvXsEevm8dDU/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zWLvhnqp$>
>     https://mailarchive.ietf.org/arch/msg/bmwg/50qoL0gxTEKGU6CkUwPIf8FO-hc/  <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/50qoL0gxTEKGU6CkUwPIf8FO-hc/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zSKCWhzH$>
>     https://mailarchive.ietf.org/arch/msg/bmwg/Y3XIteCBVMScSolsagPH_FNoi_E/  <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/bmwg/Y3XIteCBVMScSolsagPH_FNoi_E/__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zchbk9Re$>
>
>     [0]https://www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00  <https://urldefense.com/v3/__https:/www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zR12tFsp$>
>
>     [1]https://www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224  <https://urldefense.com/v3/__https:/www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zQr1Tl1V$>
>
>     [email-masked]  <mailto:[email-masked]>
>
>     https://www.ietf.org/mailman/listinfo/bmwg  <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!x5bCGyOmAgXSJkn_g2pPkZsqSPUm5qg8v2yyK3_FZdZ27NBdhsj8zU14IPVJ$>
>

---

From: Vladimir Vassilev <[email-masked]>
Date: Sun, 15 Nov 2020 16:53:40 +0100

Hi Gábor,

Thank you for the reply.

On 14/11/2020 12.09, Lencse Gábor wrote:
> tester based on a YANG model sounds interesting to me. (I was not 
> aware of your draft [1] before.) As far as I now understand it, on the 
> one hand, our projects are similar, and on the other hand, they are 
> complementary.
>
> Similarity:
> Both your YANG model based tester and my siitperf uses a split design. 
> I think your concept with data plane and control plane is more clear, 
> but I also used a similar idea: my C++ code is responsible for the 
> data plane and my bash shell scripts are responsible for the control 
> plane.
>
> Differences:
> specific (designed for SIIT, including IPv4 or IPv6 routing) and it 
> uses different executables for different measurements like throughput, 

Yes

The Unix raw socket solution [3] is just a very simple way to document 
the design pretending it will operate in an ideal world.

While the Verilog core [4] requires specialized hardware with 
programmable logic available but provides full determinism (runs at the 
GMII interface) which we needed to calibrate and evaluate other 
implementations.

>
> I think it can be a good idea for you to include also a DPDK version 
> as a balanced compromise. Let me give you some performance data to 
> support my suggestion:
>
> On the one hand, I could test authoritative DNS servers by 
> dns64perf++, which uses TCP/IP socket interface programming, by 
> sending and receiving 3.3 million queries per second using a DELL 
> PowerEdge R430 server: I used 16 cores for sending and 16 cores for 
> receiving. Please refer to my open access paper for the details: G. 
> Lencse, "Benchmarking Authoritative DNS Servers", /IEEE Access/, vol. 
> 8. pp. 130224–130238, July 2020. DOI: 
> https://doi.org/10.1109/ACCESS.2020.3009141
>
> than 7 million frames per second _per direction_ using the same type 
> of server and only a single core for sending and another for 
> receiving. (In a bidirectional test, 4 cores in all, but then siitperf 
> sent and received twice 7 million frames per second.) Please refer to 
> my open access paper for the details: G. Lencse, "Design and 
> Implementation of a Software Tester for Benchmarking Stateless NAT64 
> Gateways", /IEICE Transactions on Communications/, DOI: 
> I admit that DPDK has some requirements for the CPUs and especially 
> for the NICs, but there are a lot of supported hardware: 

I agree with that.

> Unfortunately, I am too busy to take part in it. However, I am happy 
> to share my experience with you.
>
> I have documented the design of siitperf in my above mentioned paper 
> and the source code is also available for you as a free software. I 
> would be happy if your team could reuse some part of it. The 
> implementation of the RFC 4814 random port feature is currently in the 
> "varport" branch, which I plan to merge into the master branch later 
> on. As for random number generation, 64-bit Mersenne Twister 
> (std::mt19937_64) is fast enough.
>
> Currently the accuracy of frame timing is the greatest challenge for 
> me. The inter-sending time of software testers is usually rather 
> inaccurate at higher (e.g. several million

One of the requirements for the model design was that it can be used 
both with not very deterministic software testers as well as with 
deterministic hardware testers. The main benefit being that replacing a 
tester can change the precision without requiring alternative test to be 
coded. Currently existing hardware testers can mostly implement the base 
model with simple agent translating NETCONF transactions to the native 
interfaces they have. (SCPI etc.)

>
> I have just finished the first round of testing the accuracy of 
> siitperf in the sense, how the inaccurate timing may influence its 
> results. I plan to finish my paper about it soon and it will be 
> available from my list of publications: 
> http://www.hit.bme.hu/~lencse/publications/
>

Based on the siitperf code I went through I think you are being too 
humble. I will ask for help when we get there.

>
> And let me ask you some questions about how your general tester could 
> be used for my purposes. Siitperf is only the first step in the series 
> of the RFC 8219 compliant testers necessary for benchmarking the five 
> most important IPv4aaS technologies (464XLAT, DS-Lite, MAP-T, MAP-E, 
> lw4o6), which we plan to use for the comparison of their performance 
> as a part of our draft: 
> Perhaps it is not so difficult to use your tester for benchmarking 
> SIIT implementations. (Even if I have siitperf, it could be 
> interesting to compare its results with that of your tester!) However, 
> I cannot see, if it is possible to benchmark the B4 or AFTR devices of 
> DS-Lite (RFC 6333) with your general tester. Is it possible to send 
> encapsulated packets and/or to decapsulate them with your tester?

The concept of allowing external technology specific YANG modules to 
augment the base module (ref. ietf-interfaces from RFC8343 etc.) is 
available to add support for this kind of dynamic transformations of the 
stream frame.

One example would be a protocol field like the ipv4 udp destination port 
to increment (or variate pseudo randomly) in certain range. To address 
that requirement one would need to specify additional YANG module and 
implement the functionality. Another example would be dynamic frame-size 
(rfc6985) within the stream (not multi-stream which is supported but 
requires a stream for each alternative frame). I intend to add some 
examples of common trivial cases of dynamic transformations and 
hopefully the experts in different areas can build on top of that.

> 1. It seems from your example that your tester sends bursts of frames. 
> I understand that you can achieve higher frame rates in this way, 
> however using a burst size higher than 1 makes your tests non 
> compliant of RFC 2544 / RFC 5180 / RFC 8219 as they require a constant 
> frame rate.

Since bursts are optional in the model (when 'interburst-gap' and 
'frames-per-burst' parameters are not present) one can use them only 
when needed (not use them in RFC2544 tests e.g. specify only 
'interframe-gap').

> numeric example: you have a 2GHz CPU and you want a 4 million frames 
> per second rate. It means 2,000,000,000Hz/4,000,000fps=500cpf. The 
> first possible highest frame rate is: 
> 2,000,000,000Hz/499cpf=4,008,016fps.) I do not state that it is very 
> bad, but I did not like it and I rather specify the required frame 
> rate. (It also means that I use a fix point division in every single 
> sending cycle. This is the cost of accuracy.)

Clearly the supported frames per second (fps) with a CONSTANT interframe 
gap per stream is a discrete set of real numbers - for a 64 octet frame 
and 1Gb ethernet interface (1000000000/8)/(64+8+12+i) => 
{1488095.238095238, 1470588.235294118, 1453488.372093023 ...}. However 
the generation is specified in a 100% deterministic way for that set 
when one uses the model with integer frame-size/gap. There is a way to 
support fps outside of that set if one specifies a VARIABLE 
interframe-gap augmentation (e.g.  in case you need 12.5 interframe gap 
to achieve certain fps you need to iterate between 12 and 13 octets) or 
one can add a second stream for every second to add a slightly longer 
interframe gap at the end of each second (or other period) to get a 
integer number of frames in that period. If one does that as part of the 
model we will not loose the determinism in order to gain flexibility at 
some higher level abstraction.

That said a higher-level interface can still use alternative metrics but 
it should be able to map those to the deterministic model so that it can 
execute on a hardware tester.

Let me know if you plan some open test event we can join. Perhaps the 
next IETF hackathon?

/Vladimir

>> implements a general network interconnect tester based on a YANG 
>> model [1]. Instead of a single executable implementing a benchmark 
>> test (e.g. RFC2544, RFC8219 etc.) one can split the design into 
>> control plane (search logic, reconfiguration of the generator for 
>> different trials, reading status) and data plane (the timing critical 
>> frame input/output operations and timestamping). The control plane is 
>> each interface when traffic generator configuration is committed.  
>> The command line looks like this (based on the YANG model for a 
>> generator with single stream):
>>
>> We have an alternative traffic-generator-gmii tool [4] that instead 
>> of using software to generate the packets writes to the registers of 
>> a hardware core we have designed in Verilog. So replacing the command 
>> name results in deterministic generation for the systems that have 
>> the core available.
>>
>> second reference implementation and do some testing of common DUTs 
>> and compare the results produced by the different implementations.
>>
>> References:
>>
>> [1]

---

From: 선경재 <[email-masked]>
Date: Mon, 16 Nov 2020 15:54:13 +0900

Dear Al,

Thank you to put our draft in the agenda. I can present at the BMWG session.

In this IETF, our team worked at the hackathon for testing network performance tests depending on hugepage setting with SR-IOV/DPDK environment. So, I can share our last experiment with our new experiment together.

Please let me know deadline for sending PPT material to you

Best regards,

KJ

Kyoungjae Sun / K. J. SUN
----------------------------------------------------------------------------
Researcher
Distributed Cloud and Network Laboratory (DCN Lab)
Internet Infrastructure System Technology Research Center(IISTRC)
369 Sangdo-ro, Dongjak-gu, Seoul.
----------------------------------------------------------------------------
Tel. +82-2-814-0151
Mobile +82-10-3643-5627
Email [email-masked]

> 2020. 11. 8. 오전 6:47, MORTON, ALFRED C (AL) <[email-masked]> 작성:
> 
> BMWG:
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 16 Nov 2020 14:01:22 +0000

Hi KJ, and Gabor, and others who need to know,

The deadline for slides is 23:59 UTC on Tuesday, which gives a day to post & review them.

Thanks,
Al

From: 선경재 [mailto:[email-masked]]
Sent: Monday, November 16, 2020 1:54 AM
To: MORTON, ALFRED C (AL) <[email-masked]>
Cc: [email-masked]
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Dear Al,

Best regards,

KJ

Email [email-masked]<mailto:[email-masked]>

2020. 11. 8. 오전 6:47, MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> 작성:

BMWG:

Al
bmwg co-chair

[0] https://www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00<https://urldefense.com/v3/__https:/www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLYzUAjRo$>

[1] https://www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224<https://urldefense.com/v3/__https:/www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLXEmiHGk$>

_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLRBFVO98$>

---

From: "Maciek Konstantynowicz (mkonstan)" <[email-masked]>
Date: Mon, 16 Nov 2020 14:56:39 +0000

Thanks Al et al! Vratko and me will oblige :)
Cheers,
Maciek

On 16 Nov 2020, at 14:01, MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> wrote:

Thanks,
Al

From: 선경재 [mailto:[email-masked]]
Sent: Monday, November 16, 2020 1:54 AM
To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Dear Al,

Best regards,

KJ

BMWG:

Al
bmwg co-chair

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 18 Nov 2020 00:00:07 +0000

Hi BMWG,

I updated our draft agenda, and uploaded slides received. I’m missing slides for 3 agenda items at present.

Check-out the HTML Agenda [2] for session materials as you prepare for our meeting,
Al

[2] https://datatracker.ietf.org/meeting/109/agenda

From: MORTON, ALFRED C (AL)
Sent: Monday, November 16, 2020 9:01 AM
To: '선경재' <[email-masked]>
Cc: [email-masked]
Subject: RE: [bmwg] Agenda for BMWG session at IETF-109

Thanks,
Al

Dear Al,

Best regards,

KJ

BMWG:

Al
bmwg co-chair

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 19 Nov 2020 04:57:26 +0000

Meeting starting now!

From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Tuesday, November 17, 2020 7:00 PM
To: [email-masked]
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Hi BMWG,

[2] https://datatracker.ietf.org/meeting/109/agenda<https://urldefense.com/v3/__https:/datatracker.ietf.org/meeting/109/agenda__;!!BhdT!3sfhW9HGabLYQcuhALe4Y_5g8w_D53pbf1EW8j9_w-VXK2iXBfrL52GI5wPP$>

To: '선경재' <[email-masked]<mailto:[email-masked]>>

Thanks,
Al

Dear Al,

Best regards,

KJ

BMWG:

Al
bmwg co-chair

---

From: "Maciek Konstantynowicz (mkonstan)" <[email-masked]>
Date: Thu, 19 Nov 2020 10:48:59 +0000

Oups, looks like we missed it, apologies!

For some reason we had it in our calendars as 12:00-14:00 UTC today, probably because everything we do is referenced to UTC.
But I guess living in the UTC TZ (UK GMT = UTC) is not a sufficient excuse, so apologies again.

We will follow up on our 3 open drafts on the mailing list per emails sent last night (UTC time), with the following subject lines:

1. "Call for views: draft-vpolak-mkonstan-bmwg-mlrsearch-03” [1]
2. "Call for views: draft-vpolak-bmwg-plrsearch-03” [2]
3. “Call for feedback: draft-mkonstan-nf-service-density-01” [3]

Hope the meeting went well. Will parse the notes and follow up.
Looking forward to continuous discussion on the BMWG mailing list.

Cheers,
Maciek

[1] https://mailarchive.ietf.org/arch/msg/bmwg/_oZJOWMXT0BpSO7pMqg8KvEcrTs/
[2] https://mailarchive.ietf.org/arch/msg/bmwg/2sjQVT9TEapPM7WBLaBryedLAgg/
[3] https://mailarchive.ietf.org/arch/msg/bmwg/6hZzRD54zRvfbAxcF1-Q4w5u2aU/

On 19 Nov 2020, at 04:57, MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> wrote:

Meeting starting now!

From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Tuesday, November 17, 2020 7:00 PM
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Hi BMWG,

Thanks,
Al

Dear Al,

Best regards,

KJ

BMWG:

Al
bmwg co-chair

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Thu, 19 Nov 2020 11:58:31 +0100

Hi Maciek,

Yes, that is a bug in the IETF calendar I believe.   I switched to "UTC 
Calendar" as well, but after having filtered for a subset of working 
groups, the times were shown in Singapore local timezone despite the 
"UTC Calendar" setting still on.

Best regards, Carsten

On 2020-11-19 11:48 AM, Maciek Konstantynowicz (mkonstan) wrote:
> For some reason we had it in our calendars as 12:00-14:00 UTC today, 
> probably because everything we do is referenced to UTC.
> We will follow up on our 3 open drafts on the mailing list per emails 
> sent last night (UTC time), with the following subject lines:
>
> https://mailarchive.ietf.org/arch/msg/bmwg/_oZJOWMXT0BpSO7pMqg8KvEcrTs/ 
> <https://mailarchive.ietf.org/arch/msg/bmwg/_oZJOWMXT0BpSO7pMqg8KvEcrTs/>
> [2] 
> https://mailarchive.ietf.org/arch/msg/bmwg/2sjQVT9TEapPM7WBLaBryedLAgg/ 
> <https://mailarchive.ietf.org/arch/msg/bmwg/2sjQVT9TEapPM7WBLaBryedLAgg/>
> [3] 
> https://mailarchive.ietf.org/arch/msg/bmwg/6hZzRD54zRvfbAxcF1-Q4w5u2aU/ 
> <https://mailarchive.ietf.org/arch/msg/bmwg/6hZzRD54zRvfbAxcF1-Q4w5u2aU/>
>
>> *From:*bmwg [mailto:[email-masked] 
>> <mailto:[email-masked]>]*On Behalf Of*MORTON, ALFRED C (AL)
>> *Sent:*Tuesday, November 17, 2020 7:00 PM
>> *To:[email-masked] <mailto:[email-masked]>
>> *Subject:*Re: [bmwg] Agenda for BMWG session at IETF-109
>> Hi BMWG,
>> [2]https://datatracker.ietf.org/meeting/109/agenda 
>> <https://urldefense.com/v3/__https:/datatracker.ietf.org/meeting/109/agenda__;!!BhdT!3sfhW9HGabLYQcuhALe4Y_5g8w_D53pbf1EW8j9_w-VXK2iXBfrL52GI5wPP$>
>> *From:*MORTON, ALFRED C (AL)
>> *Sent:*Monday, November 16, 2020 9:01 AM
>> *To:*'선경재' <[email-masked] <mailto:[email-masked]>>
>> *Cc:[email-masked] <mailto:[email-masked]>
>> *Subject:*RE: [bmwg] Agenda for BMWG session at IETF-109
>> *From:*선경재[mailto:[email-masked] <mailto:[email-masked]>]
>> *Sent:*Monday, November 16, 2020 1:54 AM
>> *To:*MORTON, ALFRED C (AL) <[email-masked] 
>> <mailto:[email-masked]>>
>> *Cc:[email-masked] <mailto:[email-masked]>
>> *Subject:*Re: [bmwg] Agenda for BMWG session at IETF-109
>> Dear Al,
>> In this IETF, our team worked at the hackathon for testing network 
>> performance tests depending on hugepage setting with SR-IOV/DPDK 
>> environment. So, I can share our last experiment with our new 
>> experiment together.

>> Email [email-masked] <mailto:[email-masked]>
>>
>>     2020. 11. 8.오전6:47, MORTON, ALFRED C (AL) <[email-masked]
>>     <mailto:[email-masked]>>작성:
>>     BMWG:
>>
>>     I constructed this agenda based on updated drafts, messages to
>>     the list, and some messages to the co-chairs.  IF I OVERLOOKED an
>>     item, please let me know!
>>
>>     [0]https://www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00
>>     <https://urldefense.com/v3/__https:/www.ietf.org/proceedings/109/agenda/agenda-109-bmwg-00__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLYzUAjRo$>
>>
>>     [1]https://www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224
>>     <https://urldefense.com/v3/__https:/www.timeanddate.com/worldclock/meetingtime.html?iso=20201119&p1=28&p2=37&p3=224__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLXEmiHGk$>
>>
>>
>>     <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!zLyZrpuWBilBSPOSeQnnSqDiMAErahA1LrSoa3IrPLNSmO2lIwhxQ6dLRBFVO98$>
>>
>> <https://www.ietf.org/mailman/listinfo/bmwg>
>
>

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked], http://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 19 Nov 2020 14:29:19 +0000

Sorry for the mix-up (on behalf of IETF).

I have done my best to avoid these issues, including the actual times in the agenda:

BMWG Session (IETF-109)
Thursday, November 19, 2020 (UTC+07)
12:00-14:00     Thursday Session I  (Room 4)
https://datatracker.ietf.org/meeting/109/agenda/

and as below, a start-time e-mail reminder!

Al
bmwg co-chair

From: bmwg [mailto:[email-masked]] On Behalf Of Carsten Rossenhoevel
Sent: Thursday, November 19, 2020 5:59 AM
To: [email-masked]
Subject: Re: [bmwg] Agenda for BMWG session at IETF-109

Hi Maciek,

Yes, that is a bug in the IETF calendar I believe.   I switched to "UTC Calendar" as well, but after having filtered for a subset of working groups, the times were shown in Singapore local timezone despite the "UTC Calendar" setting still on.

Best regards, Carsten

Cheers,
Maciek

Meeting starting now!

Hi BMWG,

Thanks,
Al

Dear Al,

Best regards,

KJ

BMWG:

Al
bmwg co-chair

bmwg mailing list

--

Carsten Rossenhövel

[email-masked]<mailto:[email-masked]>, http://www.eantc.de

EU VAT No: DE812824025