Subject: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

From: "MORTON JR., AL" <[email-masked]>
Date: Mon, 3 May 2021 17:36:39 +0000

BMWG,

After extensive discussion at previous sessions, a revised draft (08) is available for
another Working Group Last Call.

Please review and express your support and/or comments on the bmwg-list by

     23:59 UTC on May 17, 2021

Thanks to the author/editors for their efforts to bring us to this point, and to everyone
who has participated in previous reviews!

see you on the list,
Al
bmwg co-chair and doc shepherd

From: bmwg <[email-masked]> On Behalf Of [email-masked]
Sent: Thursday, April 22, 2021 1:28 PM
To: [email-masked]
Subject: [bmwg] New version of draft-ietf-bmwg-ngfw-performance

Folks,

The latest draft has been posted and is ready for review. The Diff between version 7 and version 8 can be found here:

https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt<https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDylH-nbgBFbOTdtctPaeeez96m7LPH8uNNegRqDFu$>

and here:

https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt<https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDylH-nbgBFbOTdtctPaeeez96m7LPH8uNNf1_GGQU$>

Please review and provide comments. I believe Al Morton is going to call for last call.

Brian

---

From: Gabor LENCSE <[email-masked]>
Date: Mon, 10 May 2021 17:08:42 +0200

Dear Al, Authors and BMWG members,

I have reviewed the first 22 pages of the draft.

As far as I could read, I have found the draft reasonable and well 
written. I have found some minor issues, which I classified as 
Discuss/Bugs/Nits:

Discuss:
--------

page 13-14

    (Option 2)  DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP
                (e.g.  50,000-100,000 IPs per 10Gbit/s throughput)

[...]

    (Option 1)  100 % IPv4, no IPv6

[...]

    Note: The IANA has assigned IP address range for the testing purpose
    as described in Section 8.

However, from the 198.18.0.0/15 range, only 198.18.0.0/16 can be used on 
the one side, 198.19.0.0/16 is for the other side of the DUT.
A /16 has only 64k number of IP addresses, thus 100,000 IPs can not be 
ensured for the 10Gbit/s throughput. (And of course, there are even 
faster links, e.g. 25/40/100Gbit/s)

As a possible resolution, perhaps the usage of the private use 
10.0.0.0/8 range could be allowed?

------------------

page 14

    For HTTP traffic emulation, the emulated browser MUST negotiate HTTP
    1.1.  HTTP persistence MAY be enabled depending on the test scenario.
    The browser MAY open multiple TCP connections per Server endpoint IP
    at any time depending on how many sequential transactions are needed
    to be processed.  Within the TCP connection multiple transactions MAY
    be processed if the emulated browser has available connections.  The
    browser SHOULD advertise a User-Agent header.  Headers MUST be sent
    uncompressed.  The browser SHOULD enforce content length validation.

I highly recommend to include HTTP/2 (defined by RFC 7540 in 2015), too.

Rationale:

"The standardization effort was supported by Chrome, Opera, Firefox,[11] 
Internet Explorer 11, Safari, Amazon Silk, and Edge browsers.[12] Most 
major browsers had added HTTP/2 support by the end of 2015.[13] About 
97% of web browsers used have the capability,[14] while according to 
W3Techs, as of April 2021, 50% of the top 10 million websites supported 
HTTP/2.[15]"
Source: https://en.wikipedia.org/wiki/HTTP/2

As I expect that HTTP/2 will dominate withing a few years, I recommend 
to include HTTP/2 with a SHOULD and include HTTP/1.1 either also with a 
SHOULD or with a MAY.

What do others think?

------------------

page 16

    The server pool for HTTP SHOULD listen on TCP port 80 and emulate
    HTTP version 1.1 with persistence.

As for the HTTP version, please see my comments regarding the client 
side text on page 14.

------------------

page 16

    [...]  The behavior of the
    client is to sweep through the given server IP space, sequentially
    generating a recognizable service by the DUT.

I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. 
To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different 
maximum connection establishment rate, when I used pseudorandom port 
numbers or when I enumerated the port numbers in increasing order. 
(Details available upon request.)

------------------

Bugs:
-----

Page 9 / Table 3:

DLP ??? -- Previously, in Table 1 (on page 7)  DLP was used for Data 
Loss Protection

I do not understand this definition:

+------------------+------------------------------------------------+
    | DLP              | DUT/SUT detects and blocks the transmission    |
    |                  | of Personally Identifiable Information (PII)   |
    |                  | and specific files across the monitored network|
+------------------+------------------------------------------------+

To me it sounds like preventing the leaking of sensitive information.
If it is so, then I would call DLP as Data *Leak* Protection in Table 1.

------------------

page 13

    o  The IPv4 Type of Service (ToS) byte or IPv6 traffic class should
       be set to '00' or '000000' respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 16

    o  A default gateway is permitted.  The IPv4 ToS byte and IPv6
       traffic class bytes should be set to '00' and '000000'
       respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 18

    Test bed reference pre-tests help to ensure that the maximum desired
    traffic generator aspects such as throughput, transaction per second,
    connection per second, concurrent connection, and latency.

I do not understand this sentence. I feel that something is missing from 
the end of the sentence (e.g. "are satisfied").

Nits:
-----

page 5

    In some deployment scenarios, the network security devices (Device
    Under Test/System Under Test) are connected to routers and switches
    which will reduce the number of entries in MAC or ARP tables of the
    Device Under Test/System Under Test (DUT/SUT).

I THINK it is non-defining:

swithes which
-->
swithes, which

------------------

Page 9 / Table 3:

+------------------+------------------------------------------------+
    | Logging and      | DUT/SUT logs and reports all traffic at the    |
    | Reporting        | flow level across the monitored.               |
+------------------+------------------------------------------------+

-->

+------------------+------------------------------------------------+
    | Logging and      | DUT/SUT logs and reports all traffic at the    |
    | Reporting        | flow level across the monitored network.       |
+------------------+------------------------------------------------+

------------------

    o  All RECOMMENDED security inspection enabled
-->
    o  All RECOMMENDED security inspections enabled

I think here "inspection" is used in a countable sense (various types of 
inspections), thus it has a plural, please refer to: 
https://www.wordhippo.com/what-is/the-plural-of/inspection.html

------------------

page 13

    SYN/ACK, ACK).  and it MUST be closed via either a TCP three way
-->
    SYN/ACK, ACK), and it MUST be closed via either a TCP three way

------------------

page 14

    profile,the
-->
    profile, the

------------------

page 17

use a set approximate number of
-->
use a set of approximate number of

(If I understood the sentence correctly.)

------------------

page 17

traffic SHOULD ramp from zero
-->
traffic SHOULD ramp up from zero

(perhaps)

------------------

page 19

    2.  Summary of test bed software and Hardware details
-->
    2.  Summary of test bed software and hardware details

(I felt it inconsistent to capitalize "software" and "hardware" 
differently.)

page 22

HTTP Transaction Latency (Section 7.8)
-->
HTTPS Transaction Latency (Section 7.8)

That's all for today. (Now I had only so much time.)

If you find my comments useful, I would be happy to read the rest of the 
document, however, I cannot promise to keep the below deadline due to my 
other workload.

Best regards,

Gábor

On 5/3/2021 7:36 PM, MORTON JR., AL wrote:
> bmwgco-chair and doc shepherd
>
> *From:*bmwg <[email-masked]> *On Behalf Of [email-masked]
> *Sent:* Thursday, April 22, 2021 1:28 PM
> *To:* [email-masked]
> *Subject:* [bmwg] New version of draft-ietf-bmwg-ngfw-performance
>
> Folks,
>
> The latest draft has been posted and is ready for review. The Diff 
> between version 7 and version 8 can be found here:
>
> https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt 
> <https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDylH-nbgBFbOTdtctPaeeez96m7LPH8uNNegRqDFu$> 
>
>
> and here:
>
> https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt 
> <https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDylH-nbgBFbOTdtctPaeeez96m7LPH8uNNf1_GGQU$>
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: <[email-masked]>
Date: Mon, 10 May 2021 12:45:27 -0400

Hi Gabor,

Thank you very much for your comments. On initial review they all look
reasonable to us. How long do you think it will take you to complete your
review of the rest of the document?

Brian

From: bmwg <[email-masked]> On Behalf Of Gabor LENCSE
Sent: Monday, May 10, 2021 11:09 AM
To: [email-masked]
Subject: Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

As far as I could read, I have found the draft reasonable and well written.
I have found some minor issues, which I classified as Discuss/Bugs/Nits:

Discuss:
--------

page 13-14

[...]

[...]

However, from the 198.18.0.0/15 range, only 198.18.0.0/16 can be used on the
one side, 198.19.0.0/16 is for the other side of the DUT.
A /16 has only 64k number of IP addresses, thus 100,000 IPs can not be
ensured for the 10Gbit/s throughput. (And of course, there are even faster
links, e.g. 25/40/100Gbit/s)

As a possible resolution, perhaps the usage of the private use 10.0.0.0/8
range could be allowed?

------------------

page 14

Rationale:

"The standardization effort was supported by Chrome, Opera, Firefox,[11]
Internet Explorer 11, Safari, Amazon Silk, and Edge browsers.[12] Most major
browsers had added HTTP/2 support by the end of 2015.[13] About 97% of web
browsers used have the capability,[14] while according to W3Techs, as of
April 2021, 50% of the top 10 million websites supported HTTP/2.[15]"
Source: https://en.wikipedia.org/wiki/HTTP/2

As I expect that HTTP/2 will dominate withing a few years, I recommend to
include HTTP/2 with a SHOULD and include HTTP/1.1 either also with a SHOULD
or with a MAY.

What do others think?

------------------

page 16

As for the HTTP version, please see my comments regarding the client side
text on page 14.

------------------

page 16

I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To
surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different
maximum connection establishment rate, when I used pseudorandom port numbers
or when I enumerated the port numbers in increasing order. (Details
available upon request.)

------------------

Bugs:
-----

Page 9 / Table 3:

DLP ??? -- Previously, in Table 1 (on page 7)  DLP was used for Data Loss
Protection

------------------

page 13

------------------

page 16

------------------

page 18

I do not understand this sentence. I feel that something is missing from the
end of the sentence (e.g. "are satisfied").

Nits:
-----

page 5

swithes which
--> 
swithes, which

------------------

Page 9 / Table 3:

-->

------------------

------------------

page 13

------------------

page 14

   profile,the 
-->
   profile, the

------------------

page 17

------------------

page 17

(perhaps)

------------------

page 19

page 22

Best regards,

Gábor

BMWG,

After extensive discussion at previous sessions, a revised draft (08) is
available for 

Thanks to the author/editors for their efforts to bring us to this point,
and to everyone

see you on the list,

Al

 

From: bmwg  <mailto:[email-masked]> <[email-masked]> On Behalf
Of [email-masked] <mailto:[email-masked]> 
Sent: Thursday, April 22, 2021 1:28 PM
To: [email-masked] <mailto:[email-masked]> 
Subject: [bmwg] New version of draft-ietf-bmwg-ngfw-performance

Folks,

The latest draft has been posted and is ready for review. The Diff between
version 7 and version 8 can be found here:

https://tools.ietf.org/rfcdiff?difftype=--hwdiff
<https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?difftype=--hwdiff
&url2=draft-ietf-bmwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDy
lH-nbgBFbOTdtctPaeeez96m7LPH8uNNegRqDFu$>
&url2=draft-ietf-bmwg-ngfw-performance-08.txt 

and here:

https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt
<https://urldefense.com/v3/__https:/tools.ietf.org/rfcdiff?url2=draft-ietf-b
mwg-ngfw-performance-08.txt__;!!BhdT!3RQm3tbMKyQIuhNFujUDylH-nbgBFbOTdtctPae
eez96m7LPH8uNNf1_GGQU$> 

Please review and provide comments. I believe Al Morton is going to call for
last call.

Brian

_______________________________________________
bmwg mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Gabor LENCSE <[email-masked]>
Date: Mon, 10 May 2021 18:58:17 +0200

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete 
it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [email-masked] wrote:
> reasonable to us. How long do you think it will take you to complete 
> your review of the rest of the document?
>
> Brian
>
>

---

From: <[email-masked]>
Date: Mon, 10 May 2021 13:01:59 -0400

Thanks Gabor.

Al, could you extend the deadline for WGLC to May 21st from May 17th?

Brian

From: Gabor LENCSE <[email-masked]> 
Sent: Monday, May 10, 2021 12:58 PM
To: [email-masked]
Cc: 'Bala Balarajah' <[email-masked]>; 'Bala Balarajah'
<[email-masked]>; [email-masked]; 'MORTON, ALFRED C (AL)'
<[email-masked]>; 'Sarah Banks' <[email-masked]>
Subject: Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it
next week. 

Best regards,

Gábor

On 5/10/2021 6:45 PM, [email-masked]
<mailto:[email-masked]>  wrote:

Hi Gabor,

Brian

---

From: "MORTON JR., AL" <[email-masked]>
Date: Mon, 10 May 2021 17:34:24 +0000

of course Brian and Gábor, I can use the extra time myself.

BMWG,
We will extend the deadline for WGLC to May 21, as requested.
Al
bmwg co-chair

From: [email-masked] <[email-masked]>
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[email-masked]>
Cc: 'Bala Balarajah' <[email-masked]>; 'Bala Balarajah' <[email-masked]>; [email-masked]; 'MORTON, ALFRED C (AL)' <[email-masked]>; 'Sarah Banks' <[email-masked]>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Thanks Gabor.

Brian

From: Gabor LENCSE <[email-masked]<mailto:[email-masked]>>
Sent: Monday, May 10, 2021 12:58 PM
To: [email-masked]<mailto:[email-masked]>
Cc: 'Bala Balarajah' <[email-masked]<mailto:[email-masked]>>; 'Bala Balarajah' <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; 'MORTON, ALFRED C (AL)' <[email-masked]<mailto:[email-masked]>>; 'Sarah Banks' <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor
On 5/10/2021 6:45 PM, [email-masked]<mailto:[email-masked]> wrote:
Hi Gabor,

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

Brian

---

From: Gabor LENCSE <[email-masked]>
Date: Fri, 14 May 2021 18:56:29 +0200

Dear Al, Authors and BMWG Members,

Al, thank you very much for extending the deadline. However, today I 
managed to finish the review of the draft.

I think this draft is matured enough. I could find only minor formal 
issues in the text about the benchmarking procedures. (Although I kept 
the categories of Discuss/Bugs/Nits, I did not find any significant new 
issue. My entries in the "Discuss" category are either reiterations from 
my previous review, or rather editorial ones.)

Discuss:
--------

page 25
7.1.4.2.

    [...] The test equipment SHOULD start to measure and record
    all specified KPIs and the frequency of measurements SHOULD be less
    than 2 seconds.

What does the "frequency of measurements" mean?
I would expect that the KPIs are mausered continously.
Perhaps it is the "frequency of results produced", that is, the "time 
window for collecting the results and calculating their average" should 
be less than 2 seconds?

------------------

page 26

    The client SHOULD negotiate HTTP 1.1 and close the connection with
    FIN immediately after completion of one transaction.  In each test
    iteration, client MUST send GET command requesting a fixed HTTP
    response object size.

As I mentioned before (I mean: in my previous review), I recommend the 
inclusion of HTTP/2.

------------------

page 27

    c.  During the sustain phase, traffic should be forwarded at a
        constant rate.

- How is "constant rate" defined?
   E.g. the maximum deviation SHOULD be less than 10%, as required at "d."?
- Why "should" is not capitalized here?

------------------

page 28

    The test equipment SHOULD start to measure and record all specified
    KPIs and the frequency of measurements SHOULD be less than 2 seconds.
    Continue the test until all traffic profile phases are completed.

The same comment, as for the "frequency of measurements" on page 25.

------------------

page 30

    b.  Traffic should be forwarded constantly.

The same comments as regarding the "constant rate" on page 27.
 From now on, I will not report it again...

------------------

pages 28-32

Unlike in other cases, HTTP version was not mentioned at all in Section 7.3.
However, I think it is as important here, as in other cases.

------------------

page 31

    [...] The test
    equipment SHOULD start to measure and record all specified KPIs and
    the frequency of measurements SHOULD be less than 2 seconds.

The same comment, as for the "frequency of measurements" on page 25.

 From now on, I will not report it again...

------------------

page 32

    For consistency both the single and multiple transaction test MUST be
    configured with HTTP 1.1.

As I mentioned before, I recommend the inclusion of HTTP/2.
HTTP 1.1 is mentioned again it the next two paragraphs and later many 
times. I will not report it again...

------------------------------------------------------

Bugs:
-----

page 30

                           Table 4: Mixed Objects

There is a "Table 4" on page 11.
(Should be renumbered. Please take care for their citations in the text, 
too!)

------------------

page 31

    The measured KPIs during the sustain phase MUST meet the test results
    validation criteria "a" defined in Section 7.3.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 34

    results validation criteria a, b, c, d, e and f defined in
    Section 7.4.3.3.

Criterion "f" is undefined, as the last one is "e".

------------------

page 37

    During the sustain phase, the DUT/SUT SHOULD reach the "Initial
    concurrent TCP connections".  The measured KPIs during the sustain
    phase MUST meet the test results validation criteria "a" and "b"
    defined in Section 7.5.3.3.

Why only "a" and "b" counts?
There is "c" too.

------------------

page 44

    The measured KPIs during the sustain phase MUST meet the test results
    validation criteria "a" defined in Section 7.7.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 47

    results validation criteria a, b, c, d, e and f defined in
    Section 7.8.3.3.

Criterion "f" is undefined, as the last one is "e".

------------------

page 50

    During the sustain phase, the DUT/SUT SHOULD reach the "Initial
    concurrent TCP connections".  The measured KPIs during the sustain
    phase MUST meet the test results validation criteria "a" and "b"
    defined in Section 7.9.3.3.

Why only "a" and "b" counts?
There is "c" too.

------------------------------------------------------

Nits:
-----

page 27

    "initial connections per second" as defined in the parameters
-->
    "Initial connections per second" as defined in the parameters

------------------

page 41

    iteration can be interrupted and the result for 64 KByte SHOULD NOT
    be reported).
-->
    iteration MAY be interrupted and the result for 64 KByte SHOULD NOT
    be reported).

(It is so in Section 7.2.4.2, thus it would be consistent to do so here, 
too)

------------------

Page 43:

Table 5 seems to have the same content as Table 4.
If it is so, it is worth including it twice?

------------------

page 44

    "initial inspected throughput" as defined in the parameters
-->
    "Initial inspected throughput" as defined in the parameters

------------------

page 53

    mode and all detected attack traffic MUST be dropped and the session
    Should be reset
-->
    mode and all detected attack traffic MUST be dropped and the session
    SHOULD be reset.

(perhaps)

------------------

    This document attempts to classify the DUT/SUT in four different four
    different categories based on its maximum supported firewall
-->
    This document attempts to classify the DUT/SUT in four different
    categories based on its maximum supported firewall

("four different" was repeated)

------------------

page 56

    Extra Small (XS) - supported throughput less than 1Gbit/s

    Small (S) - supported throughput less than 5Gbit/s

    Medium (M) - supported throughput greater than 5Gbit/s and less than
    10Gbit/s

    Large (L) - supported throughput greater than 10Gbit/s

Mathematicians could ask: and what if the supported throughput is 
exactly 10Gbit/s? ;-)

That's all.

Best regards,

Gábor

On 5/10/2021 7:34 PM, MORTON JR., AL wrote:
> bmwgco-chair

---

From: <[email-masked]>
Date: Tue, 18 May 2021 16:28:51 -0400

Hi Gabor,

Apologies for the tardy reply. All of the comments/suggestions/nits you
submitted look fine, with one exception. In your original e-mail you said:

“page 16

“I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To
surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different
maximum connection establishment rate, when I used pseudorandom port numbers
or when I enumerated the port numbers in increasing order. (Details
available upon request.)”

We feel that in order to meet the goal of reproducible results that
introducing randomness is not something we want to do.  While we do
understand your position, we hope you can support our desire to no make this
change.

Brian

From: Gabor LENCSE <[email-masked]> 
Sent: Friday, May 14, 2021 12:56 PM
To: MORTON JR., AL <[email-masked]>; [email-masked]
Cc: 'Bala Balarajah' <[email-masked]>; 'Bala Balarajah'
<[email-masked]>; [email-masked]; 'MORTON, ALFRED C (AL)'
<[email-masked]>; 'Sarah Banks' <[email-masked]>; Gábor LENCSE
<[email-masked]>
Subject: Second part -- Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Al, thank you very much for extending the deadline. However, today I managed
to finish the review of the draft. 

I think this draft is matured enough. I could find only minor formal issues
in the text about the benchmarking procedures. (Although I kept the
categories of Discuss/Bugs/Nits, I did not find any significant new issue.
My entries in the "Discuss" category are either reiterations from my
previous review, or rather editorial ones.)

Discuss:
--------

page 25
7.1.4.2.

What does the "frequency of measurements" mean?
I would expect that the KPIs are mausered continously. 
Perhaps it is the "frequency of results produced", that is, the "time window
for collecting the results and calculating their average" should be less
than 2 seconds?

------------------

page 26

------------------

page 27

------------------

page 28

------------------

page 30

------------------

pages 28-32

------------------

page 31

   [...] The test

------------------

page 32

As I mentioned before, I recommend the inclusion of HTTP/2.
HTTP 1.1 is mentioned again it the next two paragraphs and later many times.
I will not report it again...

Bugs:
-----

page 30

                          Table 4: Mixed Objects

------------------

page 31

Why only "a" counts? 

------------------

page 34

------------------

page 37

------------------

page 44

Why only "a" counts? 

------------------

page 47

------------------

page 50

Nits:
-----

page 27

------------------

page 41

------------------

Page 43:

------------------

page 44

------------------

page 53

(perhaps)

------------------

------------------

page 56

Mathematicians could ask: and what if the supported throughput is exactly
10Gbit/s? ;-)

That's all.

Best regards,

Gábor

 

BMWG,

Al

bmwg co-chair

---

From: Gábor LENCSE <[email-masked]>
Date: Wed, 19 May 2021 09:49:20 +0200

Dear Brian,

I think that you know firewall testing better then me, thus you can make 
a better decision than me, and I will accept your decision anything it 
will be.

However, I would like to support you decision by sharing my experience 
and results with you. Full disclosure: I find the question of sequential 
vs. random very interesting and important regarding my own research. I 
hope that our discussion may be beneficial for both of us.

STATELESS TESTING

1. DNS64 and authoritative DNS server benchmarking

I used sequential source port numbers in dns64perf++ (implemented by my 
BSc student Dániel Bakai) only to support RSS (Receive-Side Scaling). It 
made me possible to perform authoritative DNS server performance 
measurements over 3 million queries per second rate. (The details were 
published in my open access paper: G. Lencse, "Benchmarking 
Authoritative DNS Servers", /IEEE Access/, vol. 8. pp. 130224-130238, 
July 2020. DOI: 10.1109/ACCESS.2020.3009141)

2. SIIT (also called stateless NAT64) benchmarking

In the original version of my DPDK-based RFC 8219 compliant SIIT tester 
called siitperf, I implemented the possibility of using random 
destination networks to comply with the requirement of RFC 2544. But I 
used fixed source and destination port numbers by literally following 
the Test Frame Format supplied in the appendix of RFC 2544. Then Al 
Morton has called my attention to RFC 4814. Then I added the feature of 
pseudorandom port numbers to comply with RFC 4814. Besides pseudorandom 
port numbers, I also added the possibility of increasing (or decreasing) 
port numbers. (I have seen this feature also in commercial RFC 2544 
testers.)

I have very interesting measurement results with using pseudorandom vs. 
sequential port numbers in IPv4 router benchmarking. (IPv4 routers were 
chosen to be able to achieve high throughput.) Please see Table I. in my 
open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to 
Siitperf: Design, Implementation and Performance Estimation", 
/International Journal of Advances in Telecommunications, 
Electrotechnics, Signals and Systems/, vol 9, no 3, pp. 18-26, 2020, 
DOI: 10.11601/ijates.v9i3.291 available also from: 
http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port 
numbers result in somewhat higher throughput than random ones. (It is 
also interesting that varying source port numbers cause higher 
throughput than varying destination port numbers. I attribute this 
difference to an asymmetry in the hash function: varying source port 
numbers can distribute the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps 
throughput measured using pseudorandom source and destination port 
numbers to comply with RFC 4814, is very important for me: using 
sequential source (or destination, but not both) port numbers can be a 
computationally cheaper alternative to using pseudorandom source and 
destination port numbers.

STATEFUL TESTING

This is my very recent research area, and my surprising results made me 
recommending you to consider pseudorandom port numbers.

When I designed the stateful extension of siitperf, my original idea was 
to use sequential sweep in the port number range, which I called as 
"port number enumeration", in the preliminary phase of the stateful 
testing to load the (source IP, source port, destination IP, destination 
port) four tuples into both the connection tracking table of the DUT and 
the state table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum 
connection establishment rate of iptables!
Parameters of the tests:
source port range: 10,000 - 49,999
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. 
The results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible 
port number combinations (some combinations appeared twice or even more 
times). Therefore, I have repeated the measurement with 40,000,000 
preliminary frames using random port numbers. The result was 
1,271,023fps, which still about the double of the result with enumerated 
port numbers.

You can find all the details in my paper currently under review: 
http://www.hit.bme.hu/~lencse/publications/SFNAT64-tester-for-review.pdf

Regarding the possible root cause of the phenomenon, I have my own 
explanation, but I do not want influence you.

What do you think of it?

This experience was the reason, why we wrote in our I-D that:

    Important warning: in normal router testing, the port number
    selection algorithm (whether it is pseudo-random or enumerated) does
    not affect final results.  However, our experience with iptables
    shows that if the connection tracking table is filled using port
    number enumeration in increasing order, then the maximum connection
    establishment rate of iptables degrades significantly compared to its
    performance using pseudorandom port numbers [LEN2021  <https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful#ref-LEN2021>].

Source: 
https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful

APPLICATION TO FIREWALL TESTING

Of course, I am aware that your case may be different in several aspects 
than yours.

For example, I willfully increased the size of the connection tracking 
table of iptables from its default value to a value recommended for a 
high-loaded NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking 
table of the DUT!

These parameters likely do not apply to firewalls preforming deep packet 
inspection. :-)

My aim was to test the abilities of my stateful Tester in a demanding 
situation. I used NAT44 instead of NAT64 to be able to achieve higher 
rates. NAT44 is much-much simpler than the tasks performed by your 
firewalls, thus the lookup of the connections may dominate the execution 
time in my case, but it may be marginal in your case.

And there may be several other differences, which I do not even think 
of. This is why wrote that you need to decide, what is suitable for you. 
The most I can do is to give ideas to consider. :-)

Best regards,

Gábor

18/05/2021 22:28 keltezéssel, [email-masked] írta:
>
> Hi Gabor,
>
> Apologies for the tardy reply. All of the comments/suggestions/nits 
> you submitted look fine, with one exception. In your original e-mail 
> you said:
>
> “page 16
>
> - I feel that the spirit of RFC 4814 would rather suggest 
> pseudorandom. To surely include all of them, I recommend random 
> permutation.
> - According to my experience, iptables has shown significantly 
> different maximum connection establishment rate, when I used 
> pseudorandom port numbers or when I enumerated the port numbers in 
> increasing order. (Details available upon request.)”
>
> understand your position, we hope you can support our desire to no 
> make this change.
>
> Brian
>

---

From: "MORTON JR., AL" <[email-masked]>
Date: Thu, 20 May 2021 02:01:05 +0000

Thanks to the authors/editor for continuing to chase-down comments and resolving them.
I know very well how much work this is.

I'm trying to finish-up my review tonight (as a participant):

   o  Inspected Throughput
      The number of bits per second of allowed traffic a network
      security device is able to transmit to the correct destination
      interface(s) in response to a specified offered load.  The
      throughput benchmarking tests defined in Section 7 SHOULD measure
      the average OSI model Layer 2 throughput value.  This document
      recommends presenting the throughput value in Gbit/s rounded to
      two places of precision with a more specific Kbit/s in
      parenthesis.

Let me suggest a couple of tweaks on this definition:

It took me several minutes to remember that "inspected" invokes the role of security device, for some reason.

The OSI model doesn't have a good reputation here at IETF, and the Internet was not built based on ISO's model.

So I'll try a little editing:

   o  Inspected Throughput
      The number of bits per second of examined and allowed traffic a network
      security device is able to transmit to the correct destination
      interface(s) in response to a specified offered load.  The
      throughput benchmarking tests defined in Section 7 SHOULD measure
      the average Layer 2 throughput value when the DUT is "inspecting" traffic.  This document
      recommends presenting the inspected throughput value in Gbit/s rounded to
      two places of precision with a more specific Kbit/s in
      parenthesis.

I also checked the nits, which revealed use of 2 incorrect addresses as examples.
BMWG has it's own IPv4 and v6 address space assigned. Please use it where appropriate.
I'm not sure where the offending addresses are, sorry!

Checking nits according to https://www.ietf.org/id-info/checklist :

  ----------------------------------------------------------------------------

  == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses

     in the document.  If these are example addresses, they should be changed.

  == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses

     in the document.  If these are example addresses, they should be changed.

The tables below are from RFC 6890:

                    +----------------------+---------------+

                    | Attribute            | Value         |

                    +----------------------+---------------+

                    | Address Block        | 198.18.0.0/15 |

                    | Name                 | Benchmarking  |

                    | RFC                  | [RFC2544<https://datatracker.ietf.org/doc/html/rfc2544>]     |

                    | Allocation Date      | March 1999    |

                    | Termination Date     | N/A           |

                    | Source               | True          |

                    | Destination          | True          |

                    | Forwardable          | True          |

                    | Global               | False         |

                    | Reserved-by-Protocol | False         |

                    +----------------------+---------------+

          Table 12: Network Interconnect Device Benchmark Testing

and

                    +----------------------+----------------+

                    | Attribute            | Value          |

                    +----------------------+----------------+

                    | Address Block        | 2001:2::/48    |

                    | Name                 | Benchmarking   |

                    | RFC                  | [RFC5180<https://datatracker.ietf.org/doc/html/rfc5180>]      |

                    | Allocation Date      | April 2008     |

                    | Termination Date     | N/A            |

                    | Source               | True           |

                    | Destination          | True           |

                    | Forwardable          | True           |

                    | Global               | False          |

                    | Reserved-by-Protocol | False          |

                    +----------------------+----------------+

                          Table 24: Benchmarking

Also:

  Miscellaneous warnings:

  ----------------------------------------------------------------------------

  == The document seems to lack the recommended RFC 2119 boilerplate, even if

     it appears to use RFC 2119 keywords.

     (The document does seem to have the reference to RFC 2119 which the

     ID-Checklist requires).

The section seems to exist, but may need slight re-wording,
to match RFC 8174, or this version of the nits-checker is out of date
(it doesn't mention rfc 8174?):

2<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-08#section-2>.  Requirements

   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",

   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and

   "OPTIONAL" in this document are to be interpreted as described in BCP<https://datatracker.ietf.org/doc/html/bcp14>

   14<https://datatracker.ietf.org/doc/html/bcp14> [RFC2119<https://datatracker.ietf.org/doc/html/rfc2119>], [RFC8174<https://datatracker.ietf.org/doc/html/rfc8174>] when, and only when, they appear in all

   capitals, as shown here.

thanks,
Al

From: bmwg <[email-masked]> On Behalf Of MORTON JR., AL
Sent: Monday, May 10, 2021 1:34 PM
To: [email-masked]; 'Gabor LENCSE' <[email-masked]>
Cc: 'MORTON, ALFRED C (AL)' <[email-masked]>; [email-masked]; 'Bala Balarajah' <[email-masked]>; 'Bala Balarajah' <[email-masked]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
of course Brian and Gábor, I can use the extra time myself.

BMWG,

From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[email-masked]<mailto:[email-masked]>>
Cc: 'Bala Balarajah' <[email-masked]<mailto:[email-masked]>>; 'Bala Balarajah' <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; 'MORTON, ALFRED C (AL)' <[email-masked]<mailto:[email-masked]>>; 'Sarah Banks' <[email-masked]<mailto:[email-masked]>>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Thanks Gabor.

Brian

Hi Brian,

Best regards,

Gábor

Brian

---

From: <[email-masked]>
Date: Thu, 20 May 2021 10:45:03 -0400

Hi Al,

Thanks for sending this over. Our comments are embedded below, prefaced by
[bpm].

Brian

From: MORTON JR., AL <[email-masked]> 
Sent: Wednesday, May 19, 2021 10:01 PM
To: [email-masked]; 'Gabor LENCSE' <[email-masked]>
Cc: 'MORTON, ALFRED C (AL)' <[email-masked]>; [email-masked]; 'Bala
Balarajah' <[email-masked]>; 'Bala Balarajah' <[email-masked]>
Subject: RE: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Thanks to the authors/editor for continuing to chase-down comments and
resolving them.

I’m trying to finish-up my review tonight (as a participant):

   o  Inspected Throughput

      parenthesis.

It took me several minutes to remember that “inspected” invokes the role of
security device, for some reason.

The OSI model doesn’t have a good reputation here at IETF, and the Internet
was not built based on ISO’s model.

So I’ll try a little editing:

   o  Inspected Throughput

      The number of bits per second of examined and allowed traffic a
network

      the average Layer 2 throughput value when the DUT is “inspecting”
traffic.  This document

      recommends presenting the inspected throughput value in Gbit/s rounded
to

      parenthesis.

[bpm] Good suggestion. We will change the wording.

I also checked the nits, which revealed use of 2 incorrect addresses as
examples.

BMWG has it’s own IPv4 and v6 address space assigned. Please use it where
appropriate.

I’m not sure where the offending addresses are, sorry!

 
  == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses
     in the document.  If these are example addresses, they should be
changed.
 
  == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses
     in the document.  If these are example addresses, they should be
changed.

                    | RFC                  | [RFC2544
<https://datatracker.ietf.org/doc/html/rfc2544> ]     |

and 

                    | RFC                  | [RFC5180
<https://datatracker.ietf.org/doc/html/rfc5180> ]      |
 
                          Table 24: Benchmarking

[bpm] We have looked through the document multiple times and in multiple
ways and cannot find anything that might trigger this. However, we did catch
a format error in the IPv6 address in section 8 – IANA considerations. So,
we are updating that section. 

Also:

  Miscellaneous warnings:

  == The document seems to lack the recommended RFC 2119 boilerplate, even
if
     it appears to use RFC 2119 keywords. 

(it doesn’t mention rfc 8174?):

2
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-08#s
ection-2> .  Requirements
 
   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
<https://datatracker.ietf.org/doc/html/bcp14> 
   14 <https://datatracker.ietf.org/doc/html/bcp14>  [RFC2119
<https://datatracker.ietf.org/doc/html/rfc2119> ], [RFC8174
<https://datatracker.ietf.org/doc/html/rfc8174> ] when, and only when, they
appear in all
   capitals, as shown here.

[bpm] I went and looked at the wording RFC 8174 and it is identical to the
wording above.

thanks,

Al

From: bmwg <[email-masked] <mailto:[email-masked]> > On Behalf
Of MORTON JR., AL
Sent: Monday, May 10, 2021 1:34 PM
To: [email-masked] <mailto:[email-masked]> ; 'Gabor
LENCSE' <[email-masked] <mailto:[email-masked]> >
Cc: 'MORTON, ALFRED C (AL)' <[email-masked]
<mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> ;
'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >; 'Bala
Balarajah' <[email-masked] <mailto:[email-masked]> >
Subject: Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

 

BMWG,

Al

bmwg co-chair

From: [email-masked] <mailto:[email-masked]>
<[email-masked] <mailto:[email-masked]> > 
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[email-masked] <mailto:[email-masked]> >
Cc: 'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >;
'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >;
[email-masked] <mailto:[email-masked]> ; 'MORTON, ALFRED C (AL)'
<[email-masked] <mailto:[email-masked]> >; 'Sarah Banks'
<[email-masked] <mailto:[email-masked]> >
Subject: RE: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Thanks Gabor.

Brian

From: Gabor LENCSE <[email-masked] <mailto:[email-masked]> > 
Sent: Monday, May 10, 2021 12:58 PM
To: [email-masked] <mailto:[email-masked]> 
Cc: 'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >;
'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >;
[email-masked] <mailto:[email-masked]> ; 'MORTON, ALFRED C (AL)'
<[email-masked] <mailto:[email-masked]> >; 'Sarah Banks'
<[email-masked] <mailto:[email-masked]> >
Subject: Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Hi Brian,

Best regards,

Gábor

Hi Gabor,

Brian

---

From: <[email-masked]>
Date: Thu, 20 May 2021 10:49:40 -0400

HI Gabor,

We will update page 16 of the draft to allow for either sequential or
pseudorandom port numbers. 

Thanks for your detailed explanation.

Brian 

From: Gábor LENCSE <[email-masked]> 
Sent: Wednesday, May 19, 2021 3:49 AM
To: [email-masked]; 'MORTON JR., AL' <[email-masked]>
Cc: 'Bala Balarajah' <[email-masked]>; 'Bala Balarajah'
<[email-masked]>; [email-masked]; 'MORTON, ALFRED C (AL)'
<[email-masked]>; 'Sarah Banks' <[email-masked]>; Dr. Lencse
Gábor <[email-masked]>
Subject: Sequential vs. random -- Re: [bmwg] FW: WGLC on New version of
draft-ietf-bmwg-ngfw-performance

Dear Brian,

I think that you know firewall testing better then me, thus you can make a
better decision than me, and I will accept your decision anything it will
be.

However, I would like to support you decision by sharing my experience and
results with you. Full disclosure: I find the question of sequential vs.
random very interesting and important regarding my own research. I hope that
our discussion may be beneficial for both of us. 

STATELESS TESTING

I used sequential source port numbers in dns64perf++ (implemented by my BSc
student Dániel Bakai) only to support RSS (Receive-Side Scaling). It made me
possible to perform authoritative DNS server performance measurements over 3
million queries per second rate. (The details were published in my open
access paper: G. Lencse, "Benchmarking Authoritative DNS Servers", IEEE
Access, vol. 8. pp. 130224-130238, July 2020. DOI:
10.1109/ACCESS.2020.3009141) 

In the original version of my DPDK-based RFC 8219 compliant SIIT tester
called siitperf, I implemented the possibility of using random destination
networks to comply with the requirement of RFC 2544. But I used fixed source
and destination port numbers by literally following the Test Frame Format
supplied in the appendix of RFC 2544. Then Al Morton has called my attention
to RFC 4814. Then I added the feature of pseudorandom port numbers to comply
with RFC 4814. Besides pseudorandom port numbers, I also added the
possibility of increasing (or decreasing) port numbers. (I have seen this
feature also in commercial RFC 2544 testers.)

I have very interesting measurement results with using pseudorandom vs.
sequential port numbers in IPv4 router benchmarking. (IPv4 routers were
chosen to be able to achieve high throughput.) Please see Table I. in my
open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to
Siitperf: Design, Implementation and Performance Estimation", International
Journal of Advances in Telecommunications, Electrotechnics, Signals and
Systems, vol 9, no 3, pp. 18-26, 2020, DOI: 10.11601/ijates.v9i3.291
available also from:
http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port
numbers result in somewhat higher throughput than random ones. (It is also
interesting that varying source port numbers cause higher throughput than
varying destination port numbers. I attribute this difference to an
asymmetry in the hash function: varying source port numbers can distribute
the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps
throughput measured using pseudorandom source and destination port numbers
to comply with RFC 4814, is very important for me: using sequential source
(or destination, but not both) port numbers can be a computationally cheaper
alternative to using pseudorandom source and destination port numbers.

STATEFUL TESTING

When I designed the stateful extension of siitperf, my original idea was to
use sequential sweep in the port number range, which I called as "port
number enumeration", in the preliminary phase of the stateful testing to
load the (source IP, source port, destination IP, destination port) four
tuples into both the connection tracking table of the DUT and the state
table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum connection
establishment rate of iptables! 
Parameters of the tests:
source port range: 10,000 - 49,999 
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. The
results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible port
number combinations (some combinations appeared twice or even more times).
Therefore, I have repeated the measurement with 40,000,000 preliminary
frames using random port numbers. The result was 1,271,023fps, which still
about the double of the result with enumerated port numbers.

What do you think of it?

   Important warning: in normal router testing, the port number
   selection algorithm (whether it is pseudo-random or enumerated) does
   not affect final results.  However, our experience with iptables
   shows that if the connection tracking table is filled using port
   number enumeration in increasing order, then the maximum connection
   establishment rate of iptables degrades significantly compared to its
   performance using pseudorandom port numbers [LEN2021
<https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-statef
ul#ref-LEN2021> ].

Source:
https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-statefu
l

For example, I willfully increased the size of the connection tracking table
of iptables from its default value to a value recommended for a high-loaded
NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking
table of the DUT!

My aim was to test the abilities of my stateful Tester in a demanding
situation. I used NAT44 instead of NAT64 to be able to achieve higher rates.
NAT44 is much-much simpler than the tasks performed by your firewalls, thus
the lookup of the connections may dominate the execution time in my case,
but it may be marginal in your case.

And there may be several other differences, which I do not even think of.
This is why wrote that you need to decide, what is suitable for you. The
most I can do is to give ideas to consider. :-)

Best regards,

Gábor

18/05/2021 22:28 keltezéssel, [email-masked]
<mailto:[email-masked]>  írta:

Hi Gabor,

“page 16

Brian

---

From: Sarah Banks <[email-masked]>
Date: Thu, 20 May 2021 13:34:48 -0700

Hi,
    Sharing feedback with my participant hat on. I do apologize for sending it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next gen security devices" - and they're not clearly defined, either. For the three that are covered, I think there's a lumping of the IPS and IDS that would be better served separated. From my perspective, an IPS *is* a bump in the wire, and does make "blocking" decisions, as called out in this document. An IDS however, does not - it tells you it happened after the fact, and it's a passive device. This draft is really keyed around "active inline" functions - ie/ a bump in the wire - and I think keeping the focus there makes sense, and lumping the IDS in does more harm than good. 

In short, I don't support this draft moving forward with the inclusion of IDS - it doesn't make sense to me. One way forward might be to remove the IDS as a device from scope (as a suggestion).

- I've been testing for a long time, but I find this draft ... uncomfortable to approach. I expected, and would have preferred, something that walks me through a specific set of recommended scenarios with specifics on what to configure for repeatable tests that I could compare results with, but there are a lot of moving parts here and a lot of assertions that have to be made for the test beds as a whole where I think the test results could vary wildly when the same topology is handed to Test Person A and Test Person B. 

- Most of the feedback below omits the IDS pieces, because so much of it didn't apply. 

Thanks,
Sarah

- The draft aims to replace RFC3511, but expands scope past Firewalls, to "next generation security devices". I'm not finding a definition of what a "next generation security device is", nor an exhaustive list of the devices covered in this draft. A list that includes is nice, but IMO not enough to cover what would be benchmarked here - I'd prefer to see a definition and an exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here, we're putting active inline security devices in the same category as passive devices. On one hand, I'm not sure I'd lump these three together in the first place; on the other, active inline devices typically include additional functions to allow administrators to control what happens to packets in the case of failure, and I don't see those test cases included here.
- Section 4.1 - it reads as if ANY device in the test setup cannot contribute to network latency or throughput issues, including the DUTs - is that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed. There are options for multiple emulated routers. As a tester, I expect to see a specific, proscribed test bed that I should configure and test against. 
- Follow on: I'm curious as to the choice of emulated routers here. The previous test suggests you avoid routers and switches in the topo, but then there are emulated ones here. I'm curious as to what advantages you think these bring over the real deal, and, why they aren't subject to the same limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed, which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of additional physical gear in Option 2 - it's not clear why that's needed, or why the tester can't simply directly connect the test equipment to the DUT and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the difference is between RECOMMENDED and OPTIONAL? (I realize that you might be saying that RECOMMENDED is the "must have enabled" features, where as optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at least more than 1, feels like a strange choice here - I'd have expected tests cases that either test the specific features one at a time, or suggest several combinations, but that ultimately, we'd tell the tester to document WHICH features were enabled, to make the test cases repeatable? This allows the tester to apply a same set of apples to apples configurations to different vendor gear, and omit the 1 feature that doesn't exist on a different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the features under "recommended", I disagree with this list. For example, some customers break and inspect at the tap/agg layer of the network - in this case, the feed into the NGIDS might be decrypted, and there's no need to enable SSL inspection, for example. 
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates". 
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" - what's meant here specifically, and why do they have to be logged? (Logging, particularly under high loads, will impact it's own performance marks, and colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one suggested set of ACLs. If you're using them, great, reference that, but otherwise, make note of the ACLs you use, and use the same ones for repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no discussion around why they're chosen - that color could be useful to the reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and" that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment manufacturers, I can't possibly begin to guess which ones of them would conform to this - or even if they'd answer these questions. How helpful is this section to the non test houses? I suggest expansion here, ideally with either covering the scope of what you expect to cover, or hopefully which (open source/generally available) test tools or emulators could be considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a distinct phase before moving on to the next. You might clarify why this is a requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as to how one confirmed that the ancillary switching or routing functions didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the assertion?
- 6.1 - I would suggest that the test report include the configuration of ancillary devices on both client/server side as well 
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the "initial inspected throughput"? Maybe I missed that in the document overall, but it's not clear to me where these KPIs are collected? I suggest this be called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b) require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside the scope of this document?

---

From: <[email-masked]>
Date: Fri, 21 May 2021 14:37:18 -0400

Sarah,

Thanks for your comments. Given when it was received with respect to the
WGLC timeframe your suggestions will be considered for the next draft.
(Which is being posted shortly.)

We will get back to you in due course.

Brian

-----Original Message-----
From: Sarah Banks <[email-masked]> 
Sent: Thursday, May 20, 2021 4:35 PM
To: ALFRED MORTON <[email-masked]>
Cc: [email-masked]; Gabor LENCSE <[email-masked]>; Bala
Balarajah <[email-masked]>; Bala Balarajah <[email-masked]>;
[email-masked]; MORTON, ALFRED C (AL) <[email-masked]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi,
    Sharing feedback with my participant hat on. I do apologize for sending
it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next
gen security devices" - and they're not clearly defined, either. For the
three that are covered, I think there's a lumping of the IPS and IDS that
would be better served separated. From my perspective, an IPS *is* a bump in
the wire, and does make "blocking" decisions, as called out in this
document. An IDS however, does not - it tells you it happened after the
fact, and it's a passive device. This draft is really keyed around "active
inline" functions - ie/ a bump in the wire - and I think keeping the focus
there makes sense, and lumping the IDS in does more harm than good. 

In short, I don't support this draft moving forward with the inclusion of
IDS - it doesn't make sense to me. One way forward might be to remove the
IDS as a device from scope (as a suggestion).

- I've been testing for a long time, but I find this draft ... uncomfortable
to approach. I expected, and would have preferred, something that walks me
through a specific set of recommended scenarios with specifics on what to
configure for repeatable tests that I could compare results with, but there
are a lot of moving parts here and a lot of assertions that have to be made
for the test beds as a whole where I think the test results could vary
wildly when the same topology is handed to Test Person A and Test Person B. 

- Most of the feedback below omits the IDS pieces, because so much of it
didn't apply. 

Thanks,
Sarah

- The draft aims to replace RFC3511, but expands scope past Firewalls, to
"next generation security devices". I'm not finding a definition of what a
"next generation security device is", nor an exhaustive list of the devices
covered in this draft. A list that includes is nice, but IMO not enough to
cover what would be benchmarked here - I'd prefer to see a definition and an
exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing
to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here,
we're putting active inline security devices in the same category as passive
devices. On one hand, I'm not sure I'd lump these three together in the
first place; on the other, active inline devices typically include
additional functions to allow administrators to control what happens to
packets in the case of failure, and I don't see those test cases included
here.
- Section 4.1 - it reads as if ANY device in the test setup cannot
contribute to network latency or throughput issues, including the DUTs - is
that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed.
There are options for multiple emulated routers. As a tester, I expect to
see a specific, proscribed test bed that I should configure and test
against. 
- Follow on: I'm curious as to the choice of emulated routers here. The
previous test suggests you avoid routers and switches in the topo, but then
there are emulated ones here. I'm curious as to what advantages you think
these bring over the real deal, and, why they aren't subject to the same
limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed,
which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of
additional physical gear in Option 2 - it's not clear why that's needed, or
why the tester can't simply directly connect the test equipment to the DUT
and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the
difference is between RECOMMENDED and OPTIONAL? (I realize that you might be
saying that RECOMMENDED is the "must have enabled" features, where as
optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at
least more than 1, feels like a strange choice here - I'd have expected
tests cases that either test the specific features one at a time, or suggest
several combinations, but that ultimately, we'd tell the tester to document
WHICH features were enabled, to make the test cases repeatable? This allows
the tester to apply a same set of apples to apples configurations to
different vendor gear, and omit the 1 feature that doesn't exist on a
different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the
features under "recommended", I disagree with this list. For example, some
customers break and inspect at the tap/agg layer of the network - in this
case, the feed into the NGIDS might be decrypted, and there's no need to
enable SSL inspection, for example. 
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This
behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the
wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking
anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates". 
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" -
what's meant here specifically, and why do they have to be logged? (Logging,
particularly under high loads, will impact it's own performance marks, and
colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one
suggested set of ACLs. If you're using them, great, reference that, but
otherwise, make note of the ACLs you use, and use the same ones for
repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no
discussion around why they're chosen - that color could be useful to the
reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment
manufacturers, I can't possibly begin to guess which ones of them would
conform to this - or even if they'd answer these questions. How helpful is
this section to the non test houses? I suggest expansion here, ideally with
either covering the scope of what you expect to cover, or hopefully which
(open source/generally available) test tools or emulators could be
considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing
the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option
here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a
distinct phase before moving on to the next. You might clarify why this is a
requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as
to how one confirmed that the ancillary switching or routing functions
didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the
assertion?
- 6.1 - I would suggest that the test report include the configuration of
ancillary devices on both client/server side as well 
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the
"initial inspected throughput"? Maybe I missed that in the document overall,
but it's not clear to me where these KPIs are collected? I suggest this be
called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b)
require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside
the scope of this document?

---

From: <[email-masked]>
Date: Wed, 26 May 2021 15:33:35 -0400

Hi Sarah,

Just wanted to let you know that members of the team working on the draft
just met. We will be sending a more detailed response soon. But in the
meantime, We thought we would share a couple of decisions/comments.

First, your suggestion that IDS be removed from the document is reasonable.
Thank you for your input. IDS will be removed from the draft. As such, we
will ignore any item you raised that was related directly to IDS.

Your comment, "As a tester with long time experience with major test
equipment manufacturers, I can't possibly begin to guess which ones of them
would conform to this - or even if they'd answer these questions" warrants
an immediate comment.  Both Ixia and Spirent have been closely involved with
the drafting of this document, have implemented the tests as documented and
have been extraordinarily willing to answer any questions related to the
requirements outlined within. The creation of this draft has been actively
supported by every member of our org. That is one of the reasons it has
taken us this long to reach where we are today. 

Our goal from day one was to produce guidance on how to test a network
security product in a manner that would provide realistic results based on
real world deployment needs in a manner that can be reproduced and would
provide results that can be comparable regardless of the test tool or test
house used. Additionally, we feel strongly that the resulting guidelines
should become part of the public domain. We believed, and still believe, the
IETF is the right place. We have been very pleased with the input received
and the spirit it was provided. It appears that everyone who has commented
wants this work to be a useful addition to the knowledgebase of BMWG test
specs that have come before.

We believe the differences that are evidenced by your comments/questions
result from philosophical differences between the approach to testing
documented within the draft and your approach. It is indeed possible we can
both be right. I hope we can find compromise. 

We will get the responses to the rest of your items posted as soon as
possible. Thank you for your patience.

Brian Monkman on behalf of....

Alex Samonte (Fortinet), Amritam Putatunda (Ixia/Keysight), Bala Balarajah
(NetSecOPEN), Carsten Rossenhoevel (EANTC), Chris Brown (UNH-IOL), Mike Jack
(Spirent), Ryan Liles (Cisco), Tim Carlin (UNH-IOL), Tim Otto (Juniper)

-----Original Message-----
From: Sarah Banks <[email-masked]> 
Sent: Thursday, May 20, 2021 4:35 PM
To: ALFRED MORTON <[email-masked]>
Cc: [email-masked]; Gabor LENCSE <[email-masked]>; Bala

Hi,

Thanks,
Sarah