Subject: Re: [bmwg] An improved version of our I-D: draft-lencse-bmwg-benchmarking-stateful-02

From: "MORTON JR., AL" <[email-masked]>
Date: Sat, 6 Nov 2021 20:10:11 +0000

Hi Gábor,

Thanks for your recent updates to the “stateful” GW benchmarking draft, and for sharing your test results with BMWG and v6OPS mailing lists.

I have a few comments, as a participant, and we can discuss during the IETF-112 session:

Sec4.2
It seems that the preliminary test phase establishes the size of the state table in the DUT.
Q: (note to check – is the state table capacity mentioned as a benchmark, later in the memo?
    No, it is not.  Then)
Q: Is the state table capacity measurable? It seems useful to know during testing, at least.

Also:
   *  The connections do not time out in the DUT even during the
      beginning of the real test phase.
comment: this time-out might be a parameter to configure, or to know and ensure that the time between test phases does not exceed. It’s expressed as a statement of fact above, but it’s really a requirement (to configure) for testing.

Sec4.3
   *  setting the UDP timeout of the NATxy gateway to a value higher
      than the length of the preliminary phase.
It seems that this is the minimum timeout, and is only safe if traffic on the first connection established during the preliminary phase appears immediately when the “real” test phase begins.
The “safer” specification of the UDP timeout seems to be:
   *  setting the UDP timeout of the NATxy gateway to a value higher
      than the length of the preliminary phase and the real phase combined.

(I don’t think Sec4.7 offers the above as a solution, just cautions and the loss problem.)

regards,
Al
(as a participant)

From: bmwg <[email-masked]> On Behalf Of Gábor LENCSE
Sent: Sunday, October 10, 2021 2:11 PM
To: [email-masked]
Subject: [bmwg] An improved version of our I-D: draft-lencse-bmwg-benchmarking-stateful-02

Dear BMWG List Members,

We have gained some practical experience with the methodology for stateful NATxy gateways by measuring the scale-up of iptables. There were two similar measurement series:

1. Examining how the performance of iptables scales up with the number of CPU cores. The results are available from: https://mailarchive.ietf.org/arch/msg/v6ops/zwJJGAFG3BRtuO74IyoocP6kAL0/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/v6ops/zwJJGAFG3BRtuO74IyoocP6kAL0/__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6UacMMsa$>

2. Examining how the performance of iptables scales (degrades) with very high number of connections (up to 800 million). The results are available from: https://mailarchive.ietf.org/arch/msg/v6ops/axKSI8hnEbQPzPZGxPoS7KaLmXM/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/v6ops/axKSI8hnEbQPzPZGxPoS7KaLmXM/__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6Vvz4x9u$>

Based on the experience of the about tests, we have refined the methodology. The novel thing is how to ensure repeatable results by the proper setting of the timeout values. Now we can guarantee that the connection tracking table of the DUT always contains the required number of elements, if its size is large enough and no connection tear down happens due to timeout during the measurements.

Section 4.3 was completely rewritten, and Section 4.4 was also adjusted as needed. Please see the details here: https://www.ietf.org/rfcdiff?url2=draft-lencse-bmwg-benchmarking-stateful-02<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-lencse-bmwg-benchmarking-stateful-02__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6bwkaC-q$>

We welcome all comments!

Best regards,

Gábor

-------- Továbbított üzenet --------
Tárgy:
New Version Notification for draft-lencse-bmwg-benchmarking-stateful-02.txt
Dátum:
Sun, 10 Oct 2021 09:20:10 -0700
Feladó:
[email-masked]<mailto:[email-masked]>
Címzett:
Gabor Lencse <[email-masked]><mailto:[email-masked]>, Keiichi Shima <[email-masked]><mailto:[email-masked]>

A new version of I-D, draft-lencse-bmwg-benchmarking-stateful-02.txt
has been successfully submitted by Gabor Lencse and posted to the
IETF repository.

Name: draft-lencse-bmwg-benchmarking-stateful
Revision: 02
Title: Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers
Document date: 2021-10-10
Group: Individual Submission
Pages: 15
URL: https://www.ietf.org/archive/id/draft-lencse-bmwg-benchmarking-stateful-02.txt<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-lencse-bmwg-benchmarking-stateful-02.txt__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6U32CwIs$>
Status: https://datatracker.ietf.org/doc/draft-lencse-bmwg-benchmarking-stateful/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-lencse-bmwg-benchmarking-stateful/__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6ak4IPmN$>
Htmlized: https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6d5lG967$>
Diff: https://www.ietf.org/rfcdiff?url2=draft-lencse-bmwg-benchmarking-stateful-02<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-lencse-bmwg-benchmarking-stateful-02__;!!BhdT!wvq0sF1VVXjuutBemxKTAyWCl84R9cyNEELJaFNwuzJRHPb85vsR6bwkaC-q$>

Abstract:
RFC 2544 has defined a benchmarking methodology for network
interconnect devices. RFC 5180 addressed IPv6 specificities and it
also provided a technology update, but excluded IPv6 transition
technologies. RFC 8219 addressed IPv6 transition technologies,
including stateful NAT64. However, none of them discussed how to
apply RFC 4814 pseudorandom port numbers to any stateful NAT (NAT44,
NAT64, NAT66) technologies. We discuss why using pseudorandom port
numbers with stateful NAT gateways is a hard problem and recommend a
solution.

The IETF Secretariat

---

From: Gábor LENCSE <[email-masked]>
Date: Sun, 7 Nov 2021 22:39:09 +0100

Dear Al,

Thank you very much for your comments! And special thanks for sending 
them to me in advance, thus giving me time to think about them. :-)

First of all, let me mention that at the time of writing of the "02" 
version of the draft I had experience only with iptables (stateful 
NAT44). Since then, I have some experience with Jool (stateful NAT64).

11/6/2021 9:10 PM keltezéssel, MORTON JR., AL írta:
>
> Hi Gábor,
>
> Thanks for your recent updates to the “stateful” GW benchmarking 
> draft, and for sharing your test results with BMWG and v6OPS mailing 
> lists.
>

It is not mentioned as a benchmark. As for iptables, I can tune the size 
of the connection tracking table nearly arbitrarily, it seems that the 
memory capacity is the only limit. (Thus I did not feel it necessary at 
the time of writing version "02".)

Details:

I can tune two values:
nf_conntrack_max-- It specifies the maximum of the number of entries in 
the state table.
hashsize-- It specifies the number of entries of the hash table. (I far 
as I remember, all of them are linked list heads. The elements of the 
lists store the parameters of the connections.)

This source https://ixnfo.com/en/tuning-nf_conntrack.html recommends 
hashsize=nf_conntrack_max/8, (what would result in linked lists of 8 
elements on average) and my experience shows that larger hashsize 
ensures higher performance. (When I could, I used 
hashsize=nf_conntrack_max.) The value of hashsize is limited by the size 
of the memory. E.g. having 384GB of RAM, I could set it to 2^28, but I 
could not set it to 2^29. (I got an error message.) With hashsize=2^28, 
I used nf_conntrack_max=2^30, and the memory usage was somewhat below 
300GB. (Using another computer, I once accidentally exhausted its 256GB 
of RAM.)

> No, it is not.Then)

As we did not find any parameters, how to tune that of Jool, it may be 
useful to measure it. However, it does not seem to be simple. Why?

When I measured the maximum connection establishment rate, I set both 
the size of the connection tracking table and timeout so that the only 
reason for failure of the test could be that the rate of sending the 
preliminary tests frames (resulting in new connections) was too high.

If I do not know the size of the connection tracking table and I would 
like to measure it, then I would do a binary search as follows:

Send N number of preliminary tests frames with all different source port 
number destination number combinations in the preliminary phase.
Send N number of tests frames in the real test phase only in the 
"reverse" direction (from the Responder to the Initiator) using up all 
the element of the state table of the Tester.

If the frame rate is "not too high" both in the preliminary test phase 
and in the real test phase, and the timeout is large enough, then the 
failure of the test indicates the exhaustion of the connection tracking 
table of the DUT. Thus its size may be determined by a binary search.
(Note: It is not trivial how to ensure the frame rate conditions without 
measuring the maximum connection establishment rate and throughput, 
because their measurement would require the knowledge of the size of the 
connection tracking table.)

> Also:
>
> *The connections do not time out in the DUT even during the
>

In fact this statement just "remained" at the end of Section 4.2 from 
version "00" in version "02", but more specific criteria are given in 
Section 4.3. It either should be deleted or rewritten as:

*The connections do not time out in the DUT even during the

the real test phase, please see the details of the timeout

settings in Section 4.3.

> comment: this time-out might be a parameter to configure, or to know 
> and ensure that the time between test phases does not exceed. It’s 
> expressed as a statement of fact above, but it’s really a requirement 
> (to configure) for testing.
>

Yes, it can be ensured as stated in Section 4.3.

> Sec4.3
>
> *setting the UDP timeout of the NATxy gateway to a value higher
>

This is for achieving the "first extreme situation", which is used for 
measuring the maximum connection establishment rate. (In this case there 
is no real test phase.)

> It seems that this is the minimum timeout, and is only safe if traffic 
> on the first connection established during the preliminary phase 
> appears immediately when the “real” test phase begins.
>
> *setting the UDP timeout of the NATxy gateway to a value higher
>

Something similar is written about how to achieve the "second extreme 
situation", which is used for all measurement in the real test phase:

    *  setting the UDP timeout of the NATxy gateway to a value higher
       than the length of the preliminary phase plus the gap between the
       two phases plus the length of the real test phase.

For various reasons, there is a gap between the two phases. (There is a 
timeout at the end of the preliminary phase, plus the transmitters and 
receivers have to be stopped after the end of the preliminary phase and 
started before the real test phase, and that requires nonzero time, 
especially in the case of software testers.)

> (I don’t think Sec4.7 offers the above as a solution, just cautions 

At the time of re-writing of Section 4.3 for version 02, we did not 
check the consistency with Section 4.7.1. Unfortunately Section 4.7.1. 
still reflects the old measurement method. (It used lower timeout value 
than now specified in Section 4.3 for the real test phase.) The first 
paragraph of Section 4.7.1. can be surely deleted due to the solution 
provided in Section 4.3. (And I will think about the rest of Section 
4.7.1, too.)

Even the loss problem presented in Section 4.7.2 is somewhat mitigated 
by the new method using higher timeout. Of course, zero loss is a MUST 
in the preliminary phase, but non-zero loss may be tolerated in the real 
test phase, as now the timeout is set to a high enough value that 
refreshing is not needed during the real test phase.

Lesson learnt: I should have read the entire draft after modifying 
section 4.3 to recognize the inconsistency with the other parts! We will 
correct all these things in the next version.

Thank you very much for your thorough reading!

Best regards,

Gábor