Subject: Re: [bmwg] Scalability measurement results of the Jool stateful NAT64 solution

From: "MORTON JR., AL" <[email-masked]>
Date: Thu, 3 Mar 2022 15:20:02 +0000

Hi Gábor,

I read through your measurements this morning, thanks for sharing them.

Some small suggestions:

In the tables of results, I thought it might be useful to have a table that explains the abbreviations you used in the left-hand column. For example, the abbreviation for connections per second, cps, doesn’t appear in the text, and I had to look-around to find that throughput is expressed in units of packets per second. But otherwise the tables of results are very useful. You expressed the variation of repeated measurements, etc.

Also, it would be good to use network addresses from the benchmarking v4 and v6 address space. The nits check is flagging the current addresses, which may have been the actual addresses used in your experiment (and that’s normal practice, but nits will continue to flag them unless you substitute...). See below.

Al

  Checking nits according to https://www.ietf.org/id-info/checklist :
  ----------------------------------------------------------------------------

  == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses
     in the document.  If these are example addresses, they should be changed.

  == There are 2 instances of lines with private range IPv4 addresses in the
     document.  If these are generic example addresses, they should be changed
     to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x,
     198.51.100.x or 203.0.113.x.

  == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses
     in the document.  If these are example addresses, they should be changed.

From: bmwg <[email-masked]> On Behalf Of Gábor LENCSE
Sent: Monday, February 21, 2022 4:01 PM
To: [email-masked]
Subject: [bmwg] Scalability measurement results of the Jool stateful NAT64 solution

Dear All,

As an application of you draft "Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers", I have measured the peformance of the Jool stateful NAT64 implementation. I have examined, how its performance metrics (maximum connection establisment rate and throughput) scale up with the number of CPU cores, and how the number of connections degrade its performance. (Jool scaled up much less well than iptables.)

Taday I have uploaded the updated version of my v6ops I-D about the results. You can find the new results in its Section 3: https://datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability-01#section-3<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability-01*section-3__;Iw!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7g-9L0jU$>

Any comments, recommendations, questions, etc. are welcome!

We plan to update our BMWG draft about the methodology in 1-2 weeks (by adding the method for measuring connection tear down rate).

Best regards,

Gábor

-------- Továbbított üzenet --------
Tárgy:
New Version Notification for draft-lencse-v6ops-transition-scalability-01.txt
Dátum:
Mon, 21 Feb 2022 05:13:54 -0800
Feladó:
[email-masked]<mailto:[email-masked]>
Címzett:
Gabor Lencse <[email-masked]><mailto:[email-masked]>

A new version of I-D, draft-lencse-v6ops-transition-scalability-01.txt
has been successfully submitted by Gabor Lencse and posted to the
IETF repository.

Name: draft-lencse-v6ops-transition-scalability
Revision: 01
Title: Scalability of IPv6 Transition Technologies for IPv4aaS
Document date: 2022-02-21
Group: Individual Submission
Pages: 12
URL: https://www.ietf.org/archive/id/draft-lencse-v6ops-transition-scalability-01.txt<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-lencse-v6ops-transition-scalability-01.txt__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7ifFqxTI$>
Status: https://datatracker.ietf.org/doc/draft-lencse-v6ops-transition-scalability/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-lencse-v6ops-transition-scalability/__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7si8Vb2k$>
Htmlized: https://datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7MZBB174$>
Diff: https://www.ietf.org/rfcdiff?url2=draft-lencse-v6ops-transition-scalability-01<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-lencse-v6ops-transition-scalability-01__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7ABCA1pM$>

Abstract:
Several IPv6 transition technologies have been developed to provide
customers with IPv4-as-a-Service (IPv4aaS) for ISPs with an IPv6-only
access and/or core network. All these technologies have their
advantages and disadvantages, and depending on existing topology,
skills, strategy and other preferences, one of these technologies may
be the most appropriate solution for a network operator.

This document examines the scalability of the five most prominent
IPv4aaS technologies (464XLAT, Dual Stack Lite, Lightweight 4over6,
MAP-E, MAP-T) considering two aspects: (1) how their performance
scales up with the number of CPU cores, (2) how their performance
degrades, when the number of concurrent sessions is increased until
hardware limit is reached.

The IETF Secretariat

---

From: Gábor LENCSE <[email-masked]>
Date: Mon, 7 Mar 2022 22:14:23 +0100

Dear Al,

> I read through your measurements this morning, thanks for sharing them.

Thank you very much for being interested in them!

> Some small suggestions:
>
> In the tables of results, I thought it might be useful to have a table 
> that explains the abbreviations you used in the left-hand column. For 
> example, the abbreviation for connections per second, cps, doesn’t 
> appear in the text, and I had to look-around to find that throughput 
> is expressed in units of packets per second. But otherwise the tables 
> of results are very useful. You expressed the variation of repeated 
> measurements, etc.
>

Yes, it was really missing. Did you think something like this one?

    abbreviation          explanation
    ------------          -----------
    num. CPU cores        number of CPU cores
    src ports             size of the source port number range
    dst ports             size of the destination port number range
    num. conn.            number of connections = src ports * dst ports
    conntrack t. s.       size of the connection tracking table of the
                          DUT
    hash table size       size of the hash table of the DUT
    c.t.s/num.conn.       conntrack table size / number of connections
    num. experiments      number of experiments
    error                 the difference between the upper and the lower
                          bound of the binary search when it stops
    cps (median/min/max)  maximum connection establishment rate
                          (median, minimum, maximum)
    cps rel. scale up     the relative scale up of the maximum connection
                          establishment rate against the number of CPU
                          cores
    tp. rel. scale up     the relative scale up of the throughput
    throughput ratio (%)  the ratio of the throughput of iptables and the
                          throughput of IPv4 packet forwarding

        Figure 3: Explanation of the abbreviations for the scale up of
                   iptables against the number of CPU cores

I prepared it only as a sample. I plan to prepare similar ones for the 
other tables in the next version.

> Also, it would be good to use network addresses from the benchmarking 
> v4 and v6 address space. The nits check is flagging the current 
> addresses, which may have been the actual addresses used in your 
> experiment (and that’s normal practice, but nits will continue to flag 
> them unless you substitute...). See below.
>

In theory, I agree with your suggestion, however, I think that the usage 
of the documentation IP addresses would be inappropriate in this special 
case.
Why?
- I used the 10.0.0.1 and 10.0.0.2 private IPv4 addresses to express 
that "this is the private side of the NAT44 box".
- I used the 198.19.0.1 and 198.19.0.2 IPv4 addresses as well as the 
2001:2::1 and 2001:2::1 IPv6 addresses to express that "this drawing 
depicts a benchmarking setup".

What do you think of it?

Best regards,

Gábor

> in the document.If these are example addresses, they should be changed.
>
> document.If these are generic example addresses, they should be changed
>
> in the document.If these are example addresses, they should be changed.
>
> *From:*bmwg <[email-masked]> *On Behalf Of *Gábor LENCSE
> *Sent:* Monday, February 21, 2022 4:01 PM
> *To:* [email-masked]
> *Subject:* [bmwg] Scalability measurement results of the Jool stateful 
> NAT64 solution
>
> Dear All,
>
>
> As an application of you draft "Benchmarking Methodology for Stateful 
> NATxy Gateways using RFC 4814 Pseudorandom Port Numbers", I have 
> measured the peformance of the Jool stateful NAT64 implementation. I 
> have examined, how its performance metrics (maximum connection 
> establisment rate and throughput) scale up with the number of CPU 
> cores, and how the number of connections degrade its performance. 
> (Jool scaled up much less well than iptables.)
>
> Taday I have uploaded the updated version of my v6ops I-D about the 
> results. You can find the new results in its Section 3: 
> https://datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability-01#section-3 
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability-01*section-3__;Iw!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7g-9L0jU$>
>
>
> We plan to update our BMWG draft about the methodology in 1-2 weeks 
> (by adding the method for measuring connection tear down rate).
>
> Best regards,
>
> Gábor
>
> *Tárgy: *
> New Version Notification for 
> draft-lencse-v6ops-transition-scalability-01.txt
>
> *Dátum: *
>
> 	
>
> *Feladó: *
> [email-masked]
> *Címzett: *
> Gabor Lencse <[email-masked]> <mailto:[email-masked]>
>
>
>
>
> URL: 
> https://www.ietf.org/archive/id/draft-lencse-v6ops-transition-scalability-01.txt 
> <https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-lencse-v6ops-transition-scalability-01.txt__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7ifFqxTI$>
> Status: 
> https://datatracker.ietf.org/doc/draft-lencse-v6ops-transition-scalability/ 
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-lencse-v6ops-transition-scalability/__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7si8Vb2k$>
> Htmlized: 
> https://datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability 
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-lencse-v6ops-transition-scalability__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7MZBB174$>
> Diff: 
> https://www.ietf.org/rfcdiff?url2=draft-lencse-v6ops-transition-scalability-01 
> <https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-lencse-v6ops-transition-scalability-01__;!!BhdT!kifrO2k7qtNtx_d0VHH5xBEEO8NQ3xajmXH03ZaWpd7lqGLB0DjiL7-PUqRHWKSsdaUgIac7ABCA1pM$>
>
> Abstract:

---

From: "MORTON JR., AL" <[email-masked]>
Date: Fri, 18 Mar 2022 22:42:37 +0000

Hi Gábor, please see below,
Al

From: Gábor LENCSE <[email-masked]>
Sent: Monday, March 7, 2022 4:14 PM
To: MORTON JR., AL <[email-masked]>; [email-masked]
Subject: Re: [bmwg] Scalability measurement results of the Jool stateful NAT64 solution

Dear Al,

Some small suggestions:

   abbreviation          explanation

   ------------          -----------

                         DUT

                         cores

I prepared it only as a sample. I plan to prepare similar ones for the other tables in the next version.
[acm]
That’s great. I’m sure that if I didn’t ask, someone else who was interested in your work would have...  They will appreciate your extra effort.

In theory, I agree with your suggestion, however, I think that the usage of the documentation IP addresses would be inappropriate in this special case.
Why?
- I used the 10.0.0.1 and 10.0.0.2 private IPv4 addresses to express that "this is the private side of the NAT44 box".
- I used the 198.19.0.1 and 198.19.0.2 IPv4 addresses as well as the 2001:2::1 and 2001:2::1 IPv6 addresses to express that "this drawing depicts a benchmarking setup".

What do you think of it?
[acm]
I think that the nits-checker doesn’t recognize the BMWG address space as “ok for drafts”, so that’s fine.
You might add the sentence about Net10 private network addresses in your draft, and see how far that will go...

thanks for your reply!

Best regards,

Gábor

Al

  Checking nits according to https://www.ietf.org/id-info/checklist<https://urldefense.com/v3/__https:/www.ietf.org/id-info/checklist__;!!BhdT!iCKjq5hOhZCwjYwjkgNuvukUrmo0e-HSynwO4-Hr21cqJU-5MxYgjbZyk2kNzkwqTybBYFxsB9N0Yrs$> :
  ----------------------------------------------------------------------------

From: bmwg <[email-masked]><mailto:[email-masked]> On Behalf Of Gábor LENCSE
Sent: Monday, February 21, 2022 4:01 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [bmwg] Scalability measurement results of the Jool stateful NAT64 solution

Dear All,

Best regards,

Gábor

-------- Továbbított üzenet --------
Tárgy:

Abstract:

The IETF Secretariat