Subject: [bmwg] [iotdir] telechat Review for draft-ietf-bmwg-ngfw-performance-13

From: [email-masked]
Date: Sun, 30 Jan 2022 13:16:29 +0100

Reviewer: toerless eckert
Review result: On the right track

Summary:
Thanks a lot for this work. Its an immensibley complex and important problem to
tackle. I have in my time only measured router traffic performance and that already was an
infinite matrix. This looks to me like some order of infinite bigger a problem.

Meaning: however questioning my reviews feedback may be wrt. nitpicking about the document,
i think that the document in its existing form is already a great advancement to measure
performance for these security devices, and in doubt should be progressed rather faster than
slower especially because in my (limited) market understanding, many security device vendors will
only provide actual feedback once it is an RFC (community i think overall more conservative in
adopting IETF work, most not proactively engaging during draft stage).

But of course: feel free to improve the document with any of the feedback/suggestions
in my review that you feel are useful.

Maybe high level, i would suggest most importantly to add more explanations, especially in
an appropriate section about those aspects known NOT to be considered (but potentially
important) so that the applicability of the tests that are described are better put into
perspective by adopters of the draft to their real-world situations.

Favorite pet topic: Add req. to measure the DUT through a power meter and report consumption
so we can start making sure products with lower power consumptions will see sales benefits
when reporting numbers from this document (see details inline).

Formal:
I choose to keep the whole document inline to make it easier for readers to vet
my comments without having to open in parallel a copy of the whole document.

Rest inline - email ends with string EOF (i have seen some email truncation happening).

Thanks!
    Toerless

---
Please fix the following nits - from https://www.ietf.org/tools/idnits
idnits 2.17.00 (12 Aug 2021)

> /tmp/idnits29639/draft-ietf-bmwg-ngfw-performance-13.txt:
> ... 
> 
>   Checking nits according to https://www.ietf.org/id-info/checklist :
>   ----------------------------------------------------------------------------
> 
>   ** The abstract seems to contain references ([RFC3511]), which it
>      shouldn't.  Please replace those with straight textual mentions of the
>      documents in question.
> 
>   == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses
>      in the document.  If these are example addresses, they should be changed.
> 
>   == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses
>      in the document.  If these are example addresses, they should be changed.
> 
>   -- The draft header indicates that this document obsoletes RFC3511, but the
>      abstract doesn't seem to directly say this.  It does mention RFC3511
>      though, so this could be OK.
> 
> 
>   Miscellaneous warnings:
>   ----------------------------------------------------------------------------
> 
>   == The document seems to lack the recommended RFC 2119 boilerplate, even if
>      it appears to use RFC 2119 keywords. 
> 
>      (The document does seem to have the reference to RFC 2119 which the
>      ID-Checklist requires).
> 
> 

The lines in the following commented copy of the document are from idnits too

When a comment/question is preceeded with "Nit:", then it indicates, that
it seems to me the best answer would be modified draft text.

When a comment/question is preceeded with "Q:", then i am actually not so
sure what the outcome could be, so an answer in mail would be a start.

2	Benchmarking Methodology Working Group                      B. Balarajah
3	Internet-Draft
4	Obsoletes: 3511 (if approved)                            C. Rossenhoevel
5	Intended status: Informational                                  EANTC AG
6	Expires: 16 July 2022                                         B. Monkman
7	                                                              NetSecOPEN
8	                                                            January 2022

10	    Benchmarking Methodology for Network Security Device Performance
11	                  draft-ietf-bmwg-ngfw-performance-13

13	Abstract

15	   This document provides benchmarking terminology and methodology for
16	   next-generation network security devices including next-generation
17	   firewalls (NGFW), next-generation intrusion prevention systems
18	   (NGIPS), and unified threat management (UTM) implementations.  The

Nit: Why does it have to be next-generation for all example type
of devices except for UTMs, and what does next-generation mean.
Would suggest to rewrite text so reader does not ask herself these
questions.

18	   (NGIPS), and unified threat management (UTM) implementations.  The
19	   main areas covered in this document are test terminology, test
20	   configuration parameters, and benchmarking methodology for NGFW and
21	   NGIPS.  This document aims to improve the applicability,

I don't live and breathe the security device TLA space, but i start to
suspect a UTM is some platform on which FW and IPS could run as software
modules, and because its only software you assume the UTM does not have
to be next-gen ? I wonder how much of this guesswork/thought process you
want the reader to have or if you want to avoid that by being somehawt
clearer...

21	   NGIPS.  This document aims to improve the applicability,
22	   reproducibility, and transparency of benchmarks and to align the test
23	   methodology with today's increasingly complex layer 7 security
24	   centric network application use cases.  As a result, this document
25	   makes [RFC3511] obsolete.

[minor] I kinda wonder if / how obsoleting RFC3511 could/should work.
I understand when we do a bis of a standard protocol and really don't
want anyone to implement the older version. But unless there is a
similar IETF mandate going along with this draft that says
non-NG FW and non-NG IPS are hereby obsoleted by the IETF, i can not
see how this draft can obsolete RFC3511 because it simply applies
to a different type of benchmarked entities. And RFC3511 would stay
on forever for whatever we call non-NG.

[minor] At least i think that is the case Unless this document actually does apply also
to non-NG FW/IPS and can therefore superceed RFC3511 and actually obsolete it. But the
text so far does say the opposite.

[mayor] I observe that RFC3511 asks to measure and report goodput (5.6.5.2), and this document 
does not mention the term, and if at all, the loss in performance of client/server TCP
or QUIC connections through behavior of the DUT (such as proxying) is at best
covered indirectly by mentioning parameters such as less than 5% reduction in
throughput. If this document is superceeding rfc3511 i think it should have a very
explicit section discussing goodput - and maybe expanding on it.

consider for example the impact of TCP connection throughput and goodput.
Very likely  DUT proxying TCP connections will have quite a different performance/goodput
impact for a calssical web-page vs. video streaming. Therefore i am also worried
about sending only average bitrates per session as opposed to some sessions going
up to e.g. 500Mbps for a video streaming connection (example best commercial available
UHD video streaming today). Those type of sessions might incur a lot of goodput loss
with bad DUTs, but if i understand the test profile, then the per-TCP connection througput
of the test profiles will be much less than 100Mbps. If such range in client session
bitrates is not meant to be tested, it might at least be useful to add a section listing
candidate gaps like this. Another one for example is the impact of higher RTT especially
between DUT and server in the Internet. This mostly challenges TCP window size
operation on DUT operating as TCP hosts and also their ability to buffer for retransmissions.
Test Equipment IMHO may/should be able to emulate such long RTT. But this is not included
in this document (RTT not mentioned).

Beside goodput related issues, there are a couple other points in this review that may be too 
difficult to fix this late in the development of the document, but maybe for any of those
considered to be useful input maybe add them to a section "out-of-scope (for future versions)
considerations" or the like to capture them.

27	Status of This Memo

29	   This Internet-Draft is submitted in full conformance with the
30	   provisions of BCP 78 and BCP 79.

32	   Internet-Drafts are working documents of the Internet Engineering
33	   Task Force (IETF).  Note that other groups may also distribute
34	   working documents as Internet-Drafts.  The list of current Internet-
35	   Drafts is at https://datatracker.ietf.org/drafts/current/.

37	   Internet-Drafts are draft documents valid for a maximum of six months
38	   and may be updated, replaced, or obsoleted by other documents at any
39	   time.  It is inappropriate to use Internet-Drafts as reference
40	   material or to cite them other than as "work in progress."

42	   This Internet-Draft will expire on 5 July 2022.

44	Copyright Notice

46	   Copyright (c) 2022 IETF Trust and the persons identified as the
47	   document authors.  All rights reserved.

49	   This document is subject to BCP 78 and the IETF Trust's Legal
50	   Provisions Relating to IETF Documents (https://trustee.ietf.org/
51	   license-info) in effect on the date of publication of this document.
52	   Please review these documents carefully, as they describe your rights
53	   and restrictions with respect to this document.  Code Components
54	   extracted from this document must include Revised BSD License text as
55	   described in Section 4.e of the Trust Legal Provisions and are
56	   provided without warranty as described in the Revised BSD License.

58	Table of Contents

60	   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   4
61	   2.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   4
62	   3.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
63	   4.  Test Setup  . . . . . . . . . . . . . . . . . . . . . . . . .   4
64	     4.1.  Testbed Configuration . . . . . . . . . . . . . . . . . .   5
65	     4.2.  DUT/SUT Configuration . . . . . . . . . . . . . . . . . .   6
66	       4.2.1.  Security Effectiveness Configuration  . . . . . . . .  12
67	     4.3.  Test Equipment Configuration  . . . . . . . . . . . . . .  12
68	       4.3.1.  Client Configuration  . . . . . . . . . . . . . . . .  12
69	       4.3.2.  Backend Server Configuration  . . . . . . . . . . . .  15
70	       4.3.3.  Traffic Flow Definition . . . . . . . . . . . . . . .  17
71	       4.3.4.  Traffic Load Profile  . . . . . . . . . . . . . . . .  17
72	   5.  Testbed Considerations  . . . . . . . . . . . . . . . . . . .  18
73	   6.  Reporting . . . . . . . . . . . . . . . . . . . . . . . . . .  19
74	     6.1.  Introduction  . . . . . . . . . . . . . . . . . . . . . .  19
75	     6.2.  Detailed Test Results . . . . . . . . . . . . . . . . . .  21
76	     6.3.  Benchmarks and Key Performance Indicators . . . . . . . .  21
77	   7.  Benchmarking Tests  . . . . . . . . . . . . . . . . . . . . .  23
78	     7.1.  Throughput Performance with Application Traffic Mix . . .  23
79	       7.1.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  23
80	       7.1.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  23
81	       7.1.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  23
82	       7.1.4.  Test Procedures and Expected Results  . . . . . . . .  25
83	     7.2.  TCP/HTTP Connections Per Second . . . . . . . . . . . . .  26
84	       7.2.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  26
85	       7.2.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  27
86	       7.2.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  27
87	       7.2.4.  Test Procedures and Expected Results  . . . . . . . .  28
88	     7.3.  HTTP Throughput . . . . . . . . . . . . . . . . . . . . .  30
89	       7.3.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  30
90	       7.3.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  30
91	       7.3.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  30
92	       7.3.4.  Test Procedures and Expected Results  . . . . . . . .  32
93	     7.4.  HTTP Transaction Latency  . . . . . . . . . . . . . . . .  33
94	       7.4.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  33
95	       7.4.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  33
96	       7.4.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  34
97	       7.4.4.  Test Procedures and Expected Results  . . . . . . . .  35
98	     7.5.  Concurrent TCP/HTTP Connection Capacity . . . . . . . . .  36
99	       7.5.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  36
100	       7.5.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  36
101	       7.5.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  37
102	       7.5.4.  Test Procedures and Expected Results  . . . . . . . .  38
103	     7.6.  TCP/HTTPS Connections per Second  . . . . . . . . . . . .  39
104	       7.6.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  40
105	       7.6.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  40
106	       7.6.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  40
107	       7.6.4.  Test Procedures and Expected Results  . . . . . . . .  42
108	     7.7.  HTTPS Throughput  . . . . . . . . . . . . . . . . . . . .  43
109	       7.7.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  43
110	       7.7.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  43
111	       7.7.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  43
112	       7.7.4.  Test Procedures and Expected Results  . . . . . . . .  45
113	     7.8.  HTTPS Transaction Latency . . . . . . . . . . . . . . . .  46
114	       7.8.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  46
115	       7.8.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  46
116	       7.8.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  46
117	       7.8.4.  Test Procedures and Expected Results  . . . . . . . .  48
118	     7.9.  Concurrent TCP/HTTPS Connection Capacity  . . . . . . . .  49
119	       7.9.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  49
120	       7.9.2.  Test Setup  . . . . . . . . . . . . . . . . . . . . .  49
121	       7.9.3.  Test Parameters . . . . . . . . . . . . . . . . . . .  49
122	       7.9.4.  Test Procedures and Expected Results  . . . . . . . .  51
123	   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  52
124	   9.  Security Considerations . . . . . . . . . . . . . . . . . . .  53
125	   10. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  53
126	   11. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  53
127	   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  53
128	     12.1.  Normative References . . . . . . . . . . . . . . . . . .  53
129	     12.2.  Informative References . . . . . . . . . . . . . . . . .  53
130	   Appendix A.  Test Methodology - Security Effectiveness
131	           Evaluation  . . . . . . . . . . . . . . . . . . . . . . .  54
132	     A.1.  Test Objective  . . . . . . . . . . . . . . . . . . . . .  55
133	     A.2.  Testbed Setup . . . . . . . . . . . . . . . . . . . . . .  55
134	     A.3.  Test Parameters . . . . . . . . . . . . . . . . . . . . .  55
135	       A.3.1.  DUT/SUT Configuration Parameters  . . . . . . . . . .  55
136	       A.3.2.  Test Equipment Configuration Parameters . . . . . . .  55
137	     A.4.  Test Results Validation Criteria  . . . . . . . . . . . .  56
138	     A.5.  Measurement . . . . . . . . . . . . . . . . . . . . . . .  56
139	     A.6.  Test Procedures and Expected Results  . . . . . . . . . .  57
140	       A.6.1.  Step 1: Background Traffic  . . . . . . . . . . . . .  57
141	       A.6.2.  Step 2: CVE Emulation . . . . . . . . . . . . . . . .  58
142	   Appendix B.  DUT/SUT Classification . . . . . . . . . . . . . . .  58
143	   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  58

145	1.  Introduction

147	   18 years have passed since IETF recommended test methodology and
148	   terminology for firewalls initially ([RFC3511]).  The requirements
149	   for network security element performance and effectiveness have
150	   increased tremendously since then.  In the eighteen years since

[nit] What is a network security element ? Please provide reference or define.
If we are talking about them in this doc why are they not mentioned in the
abstract ?

150	   increased tremendously since then.  In the eighteen years since
151	   [RFC3511] was published, recommending test methodology and
152	   terminology for firewalls, requirements and expectations for network
153	   security elements has increased tremendously.  Security function

[nit] This does not parse as correct english to me "recommending test methodology ...
has increased tremendously". It would, if you mean that more and more
test methodologies where recommended, but not if there is an outstanding
need to do so (which this document intends to fill). 

[nit] Why does the recommending part apply only to firewalls and the requirements
and expectations only to security elements ? 

153	   security elements has increased tremendously.  Security function

[nit] What is a security function ? (i know, but i don't know if the reader is
supposed to know). Aka: provide reference, add terminology section or define.
Maybe easiest to restructure this intro paragraph to start with the
explanation of the evolution from firewalls to network security elements
which support one or more securit functions including firewall, intrusion
detection etc. pp - and then conclude easily how this means that this
requires this document to define all the good BMWG stuff it hopefully does.

Although a terminology section is never a bad thing either ;-)

154	   implementations have evolved to more advanced areas and have
155	   diversified into intrusion detection and prevention, threat
156	   management, analysis of encrypted traffic, etc.  In an industry of
157	   growing importance, well-defined, and reproducible key performance
158	   indicators (KPIs) are increasingly needed to enable fair and
159	   reasonable comparison of network security functions.  All these

[nit] maybe add what to compare - performance, functionality, scale,
flexibility, adjustability - or if you knowingly only discuss subsets
of these aspects, then maybe still list all the aspects you are aware
of to be of interest to likely readers of this document and summarize
those that you will and those that you won't cover in this document, so
that the readers don't have to continue reading the document hoping to find them described.

160	   reasons have led to the creation of a new next-generation network
161	   security device benchmarking document, which makes [RFC3511]
162	   obsolete.

[nit] as mentioned above, whether or not the obsolete is true is
not clear to me yet.

164	2.  Requirements

166	   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
167	   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
168	   "OPTIONAL" in this document are to be interpreted as described in BCP
169	   14 [RFC2119], [RFC8174] when, and only when, they appear in all
170	   capitals, as shown here.

172	3.  Scope

174	   This document provides testing terminology and testing methodology
175	   for modern and next-generation network security devices that are
176	   configured in Active ("Inline", see Figure 1 and Figure 2) mode.  It

[nit] The word Active does not again happen in the document, instead, the
description on line 261 defines Inline mode as "active", which in my book
makes 176+261 a perfect circular definition. I would suggest to have a 
terminology section that define "Inline", for example by also adding one
most likely possible alternative mode description.

177	   covers the validation of security effectiveness configurations of

[nit] security configuration effectiveness ?

179	   network security devices, followed by performance benchmark testing.
179	   This document focuses on advanced, realistic, and reproducible
180	   testing methods.  Additionally, it describes testbed environments,

[nit] are you sure advanced and realistic are meant to characterize the
testing method or the scenario that is being tested ? "reroducible
testing methods for advanced real world scenarios" ?

181	   test tool requirements, and test result formats.

183	4.  Test Setup

185	   Test setup defined in this document applies to all benchmarking tests

[nit] "/Test setup defined/The test setup defined/

186	   described in Section 7.  The test setup MUST be contained within an
187	   Isolated Test Environment (see Section 3 of [RFC6815]).

189	4.1.  Testbed Configuration

191	   Testbed configuration MUST ensure that any performance implications
192	   that are discovered during the benchmark testing aren't due to the

[nit] /aren't/are not/

193	   inherent physical network limitations such as the number of physical
194	   links and forwarding performance capabilities (throughput and
195	   latency) of the network devices in the testbed.  For this reason,
196	   this document recommends avoiding external devices such as switches
197	   and routers in the testbed wherever possible.

199	   In some deployment scenarios, the network security devices (Device
200	   Under Test/System Under Test) are connected to routers and switches,
201	   which will reduce the number of entries in MAC or ARP tables of the
202	   Device Under Test/System Under Test (DUT/SUT).  If MAC or ARP tables
203	   have many entries, this may impact the actual DUT/SUT performance due
204	   to MAC and ARP/ND (Neighbor Discovery) table lookup processes.  This
205	   document also recommends using test equipment with the capability of

[nit] /also/therefore/

206	   emulating layer 3 routing functionality instead of adding external
207	   routers in the testbed.

209	   The testbed setup Option 1 (Figure 1) is the RECOMMENDED testbed
210	   setup for the benchmarking test.

212	   +-----------------------+                   +-----------------------+
213	   | +-------------------+ |   +-----------+   | +-------------------+ |
214	   | | Emulated Router(s)| |   |           |   | | Emulated Router(s)| |
215	   | |    (Optional)     | +----- DUT/SUT  +-----+    (Optional)     | |
216	   | +-------------------+ |   |           |   | +-------------------+ |
217	   | +-------------------+ |   +-----------+   | +-------------------+ |
218	   | |     Clients       | |                   | |      Servers      | |
219	   | +-------------------+ |                   | +-------------------+ |
220	   |                       |                   |                       |
221	   |   Test Equipment      |                   |   Test Equipment      |
222	   +-----------------------+                   +-----------------------+

224	                     Figure 1: Testbed Setup - Option 1

226	   If the test equipment used is not capable of emulating layer 3
227	   routing functionality or if the number of used ports is mismatched
228	   between test equipment and the DUT/SUT (need for test equipment port
229	   aggregation), the test setup can be configured as shown in Figure 2.

231	    +-------------------+      +-----------+      +--------------------+
232	    |Aggregation Switch/|      |           |      | Aggregation Switch/|
233	    | Router            +------+  DUT/SUT  +------+ Router             |
234	    |                   |      |           |      |                    |
235	    +----------+--------+      +-----------+      +--------+-----------+
236	               |                                           |
237	               |                                           |
238	   +-----------+-----------+                   +-----------+-----------+
239	   |                       |                   |                       |
240	   | +-------------------+ |                   | +-------------------+ |
241	   | | Emulated Router(s)| |                   | | Emulated Router(s)| |
242	   | |     (Optional)    | |                   | |     (Optional)    | |
243	   | +-------------------+ |                   | +-------------------+ |
244	   | +-------------------+ |                   | +-------------------+ |
245	   | |      Clients      | |                   | |      Servers      | |
246	   | +-------------------+ |                   | +-------------------+ |
247	   |                       |                   |                       |
248	   |    Test Equipment     |                   |    Test Equipment     |
249	   +-----------------------+                   +-----------------------+

251	                     Figure 2: Testbed Setup - Option 2

[nit] Please elaborate on the "number of used ports", and if possible show in
Figure 2 by drawing multiple links. I guess that in a common case, the test
equipment might provide few, but fast ports, whereas the DUT/SU might provide
more slower ports, and one would there use external switches as port multiplexer ?
Or vice-versa ? Butif such adaptation is performed, i wonder how different
setup might impact the measurements. So for example let's say the Test Equipment
(TE) has a 100Gbps port, and the DUT has 4 * 10Gbps port, so you need on each
side a switch with 100Gbps and 2 * 10 Gbps. Would you try to use VLANs into the
TE, or would you just build a single LAN. Any recommendations for the switch
config, and why.

[mayor] The fact that the left side says only client and the right side says only
server is worth some more discussion. Especially because the Filtering in
Figure 3 also lets me wonder in which direction traffic is meant to be filtered/inspected.
Are you considering the case that clients are responders to (TCP/QUIC/UDP) connections ?
For example left side is "inside", DUT is a site firewall to the Internet (right side),
and there is some server on the left side (e.g.: SMTP). How about that you do
have on the right an Internet and a separate site DMZ interface and then of course
traffic not only between left and right, but between those interfaces on the right ?

More broadly applicable, dynamic port discovery for ICE/STUN, where you want to permit
inside to outside connections (to the STUN server) to permit new connections from other
external nodes to go back inside). E.g.: would be good to have some elaboration about the
rype of connections covered by this document. If its only initiators on the left and
responders on the right, that is fine, but it should be said so and maybe point to
those above cases (DMZ, inside servers, STUN/ICE) not covered by this document.

253	4.2.  DUT/SUT Configuration

255	   A unique DUT/SUT configuration MUST be used for all benchmarking
256	   tests described in Section 7.  Since each DUT/SUT will have its own
257	   unique configuration, users SHOULD configure their device with the
258	   same parameters and security features that would be used in the
259	   actual deployment of the device or a typical deployment in order to
260	   achieve maximum network security coverage.  The DUT/SUT MUST be

[nit] What is a "unique configuration" ? It could be different configurations
across two different DUT but both achieving the same service/filtering, just
difference in syntax, or it could be difference in functional outcome. Would
be good to be more precise what is meant.

[nit] Why would a user choose an actual deployment vs. a typical deployment ?
I am imagining that a user would choose an actual deployment to measure performance
specifically for that deployment but a typical deployment when the DUT would
need to be deployed in different setups but not each of those can be measured
individually, or because the results are meant to be comparable with other
users who may have taken performance numbers. WOuld be good to elaborate a bit
more so readers have a clearer understanding what "actual deployment" and
"typical deployment" means and how/why to pick one over the other.

[nit] I do not understand how the text up to "in order to" justifies that it will
achieve the maximum network security coverage. I also do not know what
"maximum network security coverage" means. If there is a definition, please
provide it. Else introduce it.

260	   achieve maximum network security coverage.  The DUT/SUT MUST be
261	   configured in "Inline" mode so that the traffic is actively inspected
262	   by the DUT/SUT.  Also "Fail-Open" behavior MUST be disabled on the
263	   DUT/SUT.

265	   Table 1 and Table 2 below describe the RECOMMENDED and OPTIONAL sets
266	   of network security feature list for NGFW and NGIPS respectively.
267	   The selected security features SHOULD be consistently enabled on the
268	   DUT/SUT for all benchmarking tests described in Section 7.

270	   To improve repeatability, a summary of the DUT/SUT configuration
271	   including a description of all enabled DUT/SUT features MUST be
272	   published with the benchmarking results.

274	          +============================+=============+==========+
275	          | DUT/SUT (NGFW) Features    | RECOMMENDED | OPTIONAL |
276	          +============================+=============+==========+
277	          | SSL Inspection             |      x      |          |
278	          +----------------------------+-------------+----------+
279	          | IDS/IPS                    |      x      |          |
280	          +----------------------------+-------------+----------+
281	          | Anti-Spyware               |      x      |          |
282	          +----------------------------+-------------+----------+
283	          | Anti-Virus                 |      x      |          |
284	          +----------------------------+-------------+----------+
285	          | Anti-Botnet                |      x      |          |
286	          +----------------------------+-------------+----------+
287	          | Web Filtering              |             |    x     |
288	          +----------------------------+-------------+----------+
289	          | Data Loss Protection (DLP) |             |    x     |
290	          +----------------------------+-------------+----------+
291	          | DDoS                       |             |    x     |
292	          +----------------------------+-------------+----------+
293	          | Certificate Validation     |             |    x     |
294	          +----------------------------+-------------+----------+

[mayor] This may be bogs because i don't know well enough how for the purpose
of this document security devices are expected to inspect HTTP connections
from client to server. Maybe this is a sane approach where the security device
operates as a client trusted HTTPs proxy, maybe its one of the more hacky approaches
(faked server certs). But however it works, i think that a security device can not
get away from validating the certificate of the server in a connection. Else
it shouldn't be called a security DUT.

But i am not sure if that validation is what you call "Certificate Validation".

294	          +----------------------------+-------------+----------+
295	          | Logging and Reporting      |      x      |          |
296	          +----------------------------+-------------+----------+
297	          | Application Identification |      x      |          |
298	          +----------------------------+-------------+----------+

300	                      Table 1: NGFW Security Features

[nit] Why are "Web Filtering"..."Certificate Validation" only MAY ?
Please point to a place in the document (or elsewhere) that rationales
the SHOULD/MAY recommendations. Same applies to Table 2.

[nit] 

302	          +============================+=============+==========+
303	          | DUT/SUT (NGIPS) Features   | RECOMMENDED | OPTIONAL |
304	          +============================+=============+==========+
305	          | SSL Inspection             |      x      |          |
306	          +----------------------------+-------------+----------+
307	          | Anti-Malware               |      x      |          |
308	          +----------------------------+-------------+----------+
309	          | Anti-Spyware               |      x      |          |
310	          +----------------------------+-------------+----------+
311	          | Anti-Botnet                |      x      |          |
312	          +----------------------------+-------------+----------+
313	          | Logging and Reporting      |      x      |          |
314	          +----------------------------+-------------+----------+
315	          | Application Identification |      x      |          |
316	          +----------------------------+-------------+----------+
317	          | Deep Packet Inspection     |      x      |          |
318	          +----------------------------+-------------+----------+
319	          | Anti-Evasion               |      x      |          |
320	          +----------------------------+-------------+----------+

322	                      Table 2: NGIPS Security Features

[nit] I ended up scrolling up and down to compare the tables.
It might be useful for other readers like me to merge the tables,
aka: put the columns for NGFW and NGIPS into one table.

[nit] Please start with Table 3 as it introduces the security features,
else the two above tables introduce a lot of features without defining them.

324	   The following table provides a brief description of the security
325	   features.

327	    +================+================================================+
328	    | DUT/SUT        | Description                                    |
329	    | Features       |                                                |
330	    +================+================================================+
331	    | SSL Inspection | DUT/SUT intercepts and decrypts inbound HTTPS  |
332	    |                | traffic between servers and clients.  Once the |
333	    |                | content inspection has been completed, DUT/SUT |
334	    |                | encrypts the HTTPS traffic with ciphers and    |
335	    |                | keys used by the clients and servers.          |
336	    +----------------+------------------------------------------------+
337	    | IDS/IPS        | DUT/SUT detects and blocks exploits targeting  |
338	    |                | known and unknown vulnerabilities across the   |
339	    |                | monitored network.                             |
340	    +----------------+------------------------------------------------+
341	    | Anti-Malware   | DUT/SUT detects and prevents the transmission  |
342	    |                | of malicious executable code and any           |
343	    |                | associated communications across the monitored |
344	    |                | network.  This includes data exfiltration as   |
345	    |                | well as command and control channels.          |
346	    +----------------+------------------------------------------------+
347	    | Anti-Spyware   | Anti-Spyware is a subcategory of Anti Malware. |
348	    |                | Spyware transmits information without the      |
349	    |                | user's knowledge or permission.  DUT/SUT       |
350	    |                | detects and block initial infection or         |
351	    |                | transmission of data.                          |
352	    +----------------+------------------------------------------------+
353	    | Anti-Botnet    | DUT/SUT detects traffic to or from botnets.    |
354	    +----------------+------------------------------------------------+
355	    | Anti-Evasion   | DUT/SUT detects and mitigates attacks that     |
356	    |                | have been obfuscated in some manner.           |
357	    +----------------+------------------------------------------------+
358	    | Web Filtering  | DUT/SUT detects and blocks malicious website   |
359	    |                | including defined classifications of website   |
360	    |                | across the monitored network.                  |
361	    +----------------+------------------------------------------------+
362	    | DLP            | DUT/SUT detects and prevents data breaches and |
363	    |                | data exfiltration, or it detects and blocks    |
364	    |                | the transmission of sensitive data across the  |
365	    |                | monitored network.                             |
366	    +----------------+------------------------------------------------+
367	    | Certificate    | DUT/SUT validates certificates used in         |
368	    | Validation     | encrypted communications across the monitored  |
369	    |                | network.                                       |
370	    +----------------+------------------------------------------------+
371	    | Logging and    | DUT/SUT logs and reports all traffic at the    |
372	    | Reporting      | flow level across the monitored network.       |
373	    +----------------+------------------------------------------------+
374	    | Application    | DUT/SUT detects known applications as defined  |
375	    | Identification | within the traffic mix selected across the     |
376	    |                | monitored network.                             |
377	    +----------------+------------------------------------------------+

379	                   Table 3: Security Feature Description

[nit] Why is DDoS and DPI not listed in this table ? I just randomnly stumbled across
that one, but maybe there are more mismatches between Table 1 and 2. Pls. make
sure all Table 1/2  Features are mentioned.

[nit] I have a bout 1000 questions and concerns about this stuff: Are there
actually IETF specifications for how any of these features on the DUT do work or
should work, or is this all vendor proprietary functionality ? For anything that
is vendor / market proprietary specification, how would the TE (Test Equipment)
know what the DUT does, so that it can effectively test it ? I imagine that
if there is a difference in how a particular feature functions across different
vendor DUTs, that the same is true for TE, so some TE would have more functional
overlap with DUT than others. ?

[nit (continued] E.g.: lets say some DUT1 feature , e.g.: DLP is really simple
and therefore ot very secure. But that makes it a lot faster than a DUT2 DLP
feature which is a lot more secure. Maybe there is a metric for this security,
like if i rememver correctly from the past, the number of signatures in virus
detection or the like... How would such differences be taken into account in
measurement? 

381	   Below is a summary of the DUT/SUT configuration:

383	   *  DUT/SUT MUST be configured in "inline" mode.

385	   *  "Fail-Open" behavior MUST be disabled.

387	   *  All RECOMMENDED security features are enabled.

389	   *  Logging SHOULD be enabled.  DUT/SUT SHOULD log all traffic at the
390	      flow level - Logging to an external device is permissible.

[nit] Does that mean logging of ALL flows or only of flows that trigger some
security issue ? Logging of ALL flows seems like a big performance hog and
may be something infeasible in fast deployments and may need to be tested as
a separate case by itself. (but my concern may be outdated).

[nit] If logging is to an external device, it may be useful to indicate in
Figure 1/2 such a logging receiver, and ideally have it operate via a link from the DUT that
does not pass test traffic so that it does not interfere.

392	   *  Geographical location filtering, and Application Identification
393	      and Control SHOULD be configured to trigger based on a site or
394	      application from the defined traffic mix.

[nit] Geographic location filtering does not sound like a generically necessary
or applicable security feature. If you are for example a high-tech manufacturer
that sells all over the world, you may appreciate customers visiting your
webserver from countries that happen to also host a lot of botnets. Or is this
document focussed on a more narrower set of use-cases ? E.g.: DUT only to filter
anything that could can not put into the cloud (such as web services) ? E.g.:
would be good to write up some justification for the GeoLoc SHOULD that would
then help readers to better understand when/how to conffigure and and when/how not.

396	   In addition, a realistic number of access control rules (ACL) SHOULD
397	   be configured on the DUT/SUT where ACLs are configurable and
398	   reasonable based on the deployment scenario.  This document
399	   determines the number of access policy rules for four different
400	   classes of DUT/SUT: Extra Small (XS), Small (S), Medium (M), and
401	   Large (L).  A sample DUT/SUT classification is described in
402	   Appendix B.

[mayor] IMHO, you can not put numbers such as those in Figure 3 into the main
text of the document, but the speed definitions of the four classes into an
Appendix B. It seems clear to me that the numbers in Figure 3 (and probably elsewhere) where
derived from the assumptions that the four speed classes are defined as in
Appendix B. Suggestion: inline the text of Appendix B here and mention that numbers
such as in Figure 3 are derived from the assumption of those XS/S/M/L numbers,
Add (if necessary, else not) that it may be appropriate to choose other numbers for
XS/S/M/L, but if one does that, then the dependent numbers (such as those from Figure 3)
may also need to be re-evaluated.

404	   The Access Control Rules (ACL) defined in Figure 3 MUST be configured
405	   from top to bottom in the correct order as shown in the table.  This
406	   is due to ACL types listed in specificity decreasing order, with
407	   "block" first, followed by "allow", representing a typical ACL based
408	   security policy.  The ACL entries SHOULD be configured with routable
409	   IP subnets by the DUT/SUT.  (Note: There will be differences between
410	   how security vendors implement ACL decision making.)  The configured

[nit] /security vendors/DUT/

[nit] I don't understand what i am supposed to learn from the (Note: ...) sentence.
Rephrase ? or remove.

410	   how security vendors implement ACL decision making.)  The configured
411	   ACL MUST NOT block the security and measurement traffic used for the
412	   benchmarking tests.

[nit] what is "security traffic" ? what is "measurement traffic" ?  Don't see these
terms defined before. Those two terms do not immediately click to me. I guess
measured user/client-server traffic vs. test-setup management traffic (including logging) ??
In any case introduce the terms, define them and use them consistently. Whatever they are.

414	                                                       +---------------+
415	                                                       | DUT/SUT       |
416	                                                       | Classification|
417	                                                       | # Rules       |
418	   +-----------+-----------+--------------------+------+---+---+---+---+
419	   |           | Match     |                    |      |   |   |   |   |
420	   | Rules Type| Criteria  |   Description      |Action| XS| S | M | L |
421	   +-------------------------------------------------------------------+
422	   |Application|Application| Any application    | block| 5 | 10| 20| 50|
423	   |layer      |           | not included in    |      |   |   |   |   |
424	   |           |           | the measurement    |      |   |   |   |   |
425	   |           |           | traffic            |      |   |   |   |   |
426	   +-------------------------------------------------------------------+
427	   |Transport  |SRC IP and | Any SRC IP subnet  | block| 25| 50|100|250|
428	   |layer      |TCP/UDP    | used and any DST   |      |   |   |   |   |
429	   |           |DST ports  | ports not used in  |      |   |   |   |   |
430	   |           |           | the measurement    |      |   |   |   |   |
431	   |           |           | traffic            |      |   |   |   |   |
432	   +-------------------------------------------------------------------+
433	   |IP layer   |SRC/DST IP | Any SRC/DST IP     | block| 25| 50|100|250|
434	   |           |           | subnet not used    |      |   |   |   |   |
435	   |           |           | in the measurement |      |   |   |   |   |
436	   |           |           | traffic            |      |   |   |   |   |
437	   +-------------------------------------------------------------------+

[nit] WOuld suggest to remove the word "Any" to minimize misinterpretation.

[nit] These three blocks seem to never get exercised by the actual measurement
traffic, right ? So the purpose would then be to simply load up the DUT with
them in case the DUT implementation is stupid enough to have these cause relevant
performance impacts even when not exercised by traffic. Would be good to write
this down as a rationale after the table.  Especially because the "Any" had me
confused first that in a real-world deployment you would of course not include
250 individual application/port/prefixes, but you just have some simple block-all.

[nit] Even 27 years ago i've seen routers acting as firewalls for universities
that had thousands of such ACL entries. Aka: i think these numbers are way too low.

438	   |Application|Application| Half of the        | allow| 10| 10| 10| 10|
439	   |layer      |           | applications       |      |   |   |   |   |
440	   |           |           | included in the    |      |   |   |   |   |
441	   |           |           | measurement traffic|      |   |   |   |   |
442	   |           |           |(see the note below)|      |   |   |   |   |
443	   +-------------------------------------------------------------------+
444	   |Transport  |SRC IP and | Half of the SRC    | allow| >1| >1| >1| >1|
445	   |layer      |TCP/UDP    | IPs used and any   |      |   |   |   |   |
446	   |           |DST ports  | DST ports used in  |      |   |   |   |   |
447	   |           |           | the measurement    |      |   |   |   |   |
448	   |           |           | traffic            |      |   |   |   |   |
449	   |           |           | (one rule per      |      |   |   |   |   |
450	   |           |           | subnet)            |      |   |   |   |   |
451	   +-------------------------------------------------------------------+
452	   |IP layer   |SRC IP     | The rest of the    | allow| >1| >1| >1| >1|
453	   |           |           | SRC IP subnet      |      |   |   |   |   |
454	   |           |           | range used in the  |      |   |   |   |   |
455	   |           |           | measurement        |      |   |   |   |   |
456	   |           |           | traffic            |      |   |   |   |   |
457	   |           |           | (one rule per      |      |   |   |   |   |
458	   |           |           | subnet)            |      |   |   |   |   |
459	   +-----------+-----------+--------------------+------+---+---+---+---+

[mayor] There should be an explanation of how this is supposed to work, and
it seems there are rules missing:

      rule on row 438 explicitly permits half the traffic sent by the test
      equiment. So supposedly only the other half has to be checked by rule on row 444.
      So when 444 says "Half of the SRC...", is that half of the total ? Would that
      have to be set up so that after 444 we now have 75% of the measurement
      traffic going through ?  Likewise then rule 452 does it bring the total amount
      of permitted traffic to 87.5% ?.

[nit] Ultimately, we only have "allows" here.
      Is there an assumption that after row 459 there is an implicit deny-anything-else ?
      I guess so, but it should be written out explicitly in the table.

461	                       Figure 3: DUT/SUT Access List

463	   Note: If half of the applications included in the measurement traffic
464	   is less than 10, the missing number of ACL entries (dummy rules) can
465	   be configured for any application traffic not included in the
466	   measurement traffic.

468	4.2.1.  Security Effectiveness Configuration

470	   The Security features (defined in Table 1 and Table 2) of the DUT/SUT
471	   MUST be configured effectively to detect, prevent, and report the
472	   defined security vulnerability sets.  This section defines the
473	   selection of the security vulnerability sets from Common

[nit] "from the CVE" ?!

474	   vulnerabilities and Exposures (CVE) list for the testing.  The

[nit] Add reference for CVE. (Not sure whats best spec, or wikipedia or cve.org,...)

475	   vulnerability set SHOULD reflect a minimum of 500 CVEs from no older
476	   than 10 calendar years to the current year.  These CVEs SHOULD be
477	   selected with a focus on in-use software commonly found in business
478	   applications, with a Common vulnerability Scoring System (CVSS)
479	   Severity of High (7-10).

481	   This document is primarily focused on performance benchmarking.
482	   However, it is RECOMMENDED to validate the security features
483	   configuration of the DUT/SUT by evaluating the security effectiveness
484	   as a prerequisite for performance benchmarking tests defined in the

[nit]  /in the/in/

485	   section 7.  In case the benchmarking tests are performed without
486	   evaluating security effectiveness, the test report MUST explain the
487	   implications of this.  The methodology for evaluating security
488	   effectiveness is defined in Appendix A.

490	4.3.  Test Equipment Configuration

492	   In general, test equipment allows configuring parameters in different
493	   protocol layers.  These parameters thereby influence the traffic
494	   flows which will be offered and impact performance measurements.

496	   This section specifies common test equipment configuration parameters
497	   applicable for all benchmarking tests defined in Section 7.  Any
498	   benchmarking test specific parameters are described under the test
499	   setup section of each benchmarking test individually.

501	4.3.1.  Client Configuration

503	   This section specifies which parameters SHOULD be considered while
504	   configuring clients using test equipment.  Also, this section
505	   specifies the RECOMMENDED values for certain parameters.  The values
506	   are the defaults used in most of the client operating systems
507	   currently.

509	4.3.1.1.  TCP Stack Attributes

511	   The TCP stack SHOULD use a congestion control algorithm at client and
512	   server endpoints.  The IPv4 and IPv6 Maximum Segment Size (MSS)
513	   SHOULD be set to 1460 bytes and 1440 bytes respectively and a TX and
514	   RX initial receive windows of 64 KByte.  Client initial congestion
515	   window SHOULD NOT exceed 10 times the MSS.  Delayed ACKs are
516	   permitted and the maximum client delayed ACK SHOULD NOT exceed 10
517	   times the MSS before a forced ACK.  Up to three retries SHOULD be
518	   allowed before a timeout event is declared.  All traffic MUST set the
519	   TCP PSH flag to high.  The source port range SHOULD be in the range
520	   of 1024 - 65535.  Internal timeout SHOULD be dynamically scalable per
521	   RFC 793.  The client SHOULD initiate and close TCP connections.  The
522	   TCP connection MUST be initiated via a TCP three-way handshake (SYN,
523	   SYN/ACK, ACK), and it MUST be closed via either a TCP three-way close
524	   (FIN, FIN/ACK, ACK), or a TCP four-way close (FIN, ACK, FIN, ACK).

[nit] Would be nice to have reference to where/how these parameters are determined.
Would be nice to mention why these parameters are choosen. Probably to reflect the
most common current TCP behavior that achieves best performance ?

[minor] The document mentions QUIC in three places, but has no equivalent
section for QUIC here as it has for TCP. I would suggest to add a section here,
even if it can just say "Due to the absence of suficient experience, QUIC parameters
are unspecified. Similarily to TCP, parameters should be choosen that best reflect
state-of-the art performance results for QUIC client/server traffic".

526	4.3.1.2.  Client IP Address Space

528	   The sum of the client IP space SHOULD contain the following
529	   attributes.

531	   *  The IP blocks SHOULD consist of multiple unique, discontinuous
532	      static address blocks.

534	   *  A default gateway is permitted.

[comment] How is this relevant, what do you expect it to do ? What would happen
if you just removed it ?

536	   *  The DSCP (differentiated services code point) marking is set to DF
537	      (Default Forwarding) '000000' on IPv4 Type of Service (ToS) field
538	      and IPv6 traffic class field.

540	   The following equation can be used to define the total number of
541	   client IP addresses that will be configured on the test equipment.

543	   Desired total number of client IP = Target throughput [Mbit/s] /
544	   Average throughput per IP address [Mbit/s]

546	   As shown in the example list below, the value for "Average throughput
547	   per IP address" can be varied depending on the deployment and use
548	   case scenario.

550	   (Option 1)  DUT/SUT deployment scenario 1 : 6-7 Mbit/s per IP (e.g.
551	               1,400-1,700 IPs per 10Gbit/s throughput)

553	   (Option 2)  DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP
554	               (e.g.  50,000-100,000 IPs per 10Gbit/s throughput)

556	   Based on deployment and use case scenario, client IP addresses SHOULD
557	   be distributed between IPv4 and IPv6.  The following options MAY be
558	   considered for a selection of traffic mix ratio.

560	   (Option 1)  100 % IPv4, no IPv6

562	   (Option 2)  80 % IPv4, 20% IPv6

564	   (Option 3)  50 % IPv4, 50% IPv6

566	   (Option 4)  20 % IPv4, 80% IPv6

568	   (Option 5)  no IPv4, 100% IPv6

[minor] This guidance is IMHO not very helpfull. It seems to me the first guidance
seems to be that the percentage of IPv4 vs. IPv6 addresses should be based on the
relevant ratio of IPv4 vs. IPv6 traffic in the target deployment because the way
the test setup is done, some N% IPv4 addresses will also roughly result in N% IPv4 traffic
in the test.

That type of explanation might be very helpfull, because the risk is that readers may
think they can derive the percentage of test IPv4/IPv6 addresses from the ratio 
of IPv4/IPv6 addresses in the target deployment, but that very often will not work:

For example in the common dual-stack deployment, every client has an IPv4 and an IPv6 address,
so its 50% IPv4, but the actual percentage of IPv4 traffic will very much depend on the
application scenario. Some enterprises may go up to 90% or more IPv6 traffic if the main
traffic is all newer cloud services traffic. An vice versa, it could be as little as 10% IPv6
if all the cloud services are legacy apps in the cloud not supporting IPv6.

570	   Note: The IANA has assigned IP address range for the testing purpose
571	   as described in Section 8.  If the test scenario requires more IP
572	   addresses or subnets than the IANA assigned, this document recommends
573	   using non routable Private IPv4 address ranges or Unique Local
574	   Address (ULA) IPv6 address ranges for the testing.

[minor] See comments in Section 8. It might be useful to merge the text of this
paragraph with the one in Section 8, else the addressing recommendations are
somewhat split in the middle.

[minor] It would be prudent to add a disclaimer that this document does not consider
to determine whether DUT may emobdy optimizations in performance behavior for known testing
address ranges. Such a disclaimer may be more general and go on the end of the
document, e.g.: before IANA section - no considerations against DUT optimizations of known
test scenarios including addressing ranges or other test profile specific parameters.

576	4.3.1.3.  Emulated Web Browser Attributes

578	   The client emulated web browser (emulated browser) contains
579	   attributes that will materially affect how traffic is loaded.  The

[nit] what does "how traffic is loaded" mean ? Rephrase.

580	   objective is to emulate modern, typical browser attributes to improve
581	   realism of the result set.

[nit] /result set/resulting traffic/ ?

583	   For HTTP traffic emulation, the emulated browser MUST negotiate HTTP
584	   version 1.1 or higher.  Depending on test scenarios and chosen HTTP
585	   version, the emulated browser MAY open multiple TCP connections per
586	   Server endpoint IP at any time depending on how many sequential
587	   transactions need to be processed.  For HTTP/2 or HTTP/3, the
588	   emulated browser MAY open multiple concurrent streams per connection
589	   (multiplexing).  HTTP/3 emulated browser uses QUIC ([RFC9000]) as
590	   transport protocol.  HTTP settings such as number of connection per
591	   server IP, number of requests per connection, and number of streams
592	   per connection MUST be documented.  This document refers to [RFC8446]
593	   for HTTP/2.  The emulated browser SHOULD advertise a User-Agent
594	   header.  The emulated browser SHOULD enforce content length
595	   validation.  Depending on test scenarios and selected HTTP version,
596	   HTTP header compression MAY be set to enable or disable.  This
597	   setting (compression enabled or disabled) MUST be documented in the
598	   report.

600	   For encrypted traffic, the following attributes SHALL define the
601	   negotiated encryption parameters.  The test clients MUST use TLS
602	   version 1.2 or higher.  TLS record size MAY be optimized for the

[minor] I would bet SEC review will challenge you to comment on TLS 1.3.
Would make sense to add a sentence stating that the ratio of TLS 1.2 vs TLS 1.3
traffic should be choosen based on expected target deployment and may range
from 100% TLS 1.2 to 100% TLS 1.3. In the absence of known ratios, a 50/50%
ratio is RECOMMENDED.

602	   version 1.2 or higher.  TLS record size MAY be optimized for the
603	   HTTPS response object size up to a record size of 16 KByte.  If
604	   Server Name Indication (SNI) is required in the traffic mix profile,
605	   the client endpoint MUST send TLS extension Server Name Indication
606	   (SNI) information when opening a security tunnel.  Each client

[minor] SNI is pretty standard today. I would remove the "if" and make the
whole sentence a MUST.

606	   (SNI) information when opening a security tunnel.  Each client
607	   connection MUST perform a full handshake with server certificate and
608	   MUST NOT use session reuse or resumption.

610	   The following TLS 1.2 supported ciphers and keys are RECOMMENDED to
611	   use for HTTPS based benchmarking tests defined in Section 7.

613	   1.  ECDHE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash
614	       Algorithm: ecdsa_secp256r1_sha256 and Supported group: secp256r1)

616	   2.  ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash
617	       Algorithm: rsa_pkcs1_sha256 and Supported group: secp256r1)

619	   3.  ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash
620	       Algorithm: ecdsa_secp384r1_sha384 and Supported group: secp521r1)

622	   4.  ECDHE-RSA-AES256-GCM-SHA384 with RSA 4096 (Signature Hash
623	       Algorithm: rsa_pkcs1_sha384 and Supported group: secp256r1)

625	   Note: The above ciphers and keys were those commonly used enterprise
626	   grade encryption cipher suites for TLS 1.2.  It is recognized that
627	   these will evolve over time.  Individual certification bodies SHOULD
628	   use ciphers and keys that reflect evolving use cases.  These choices
629	   MUST be documented in the resulting test reports with detailed
630	   information on the ciphers and keys used along with reasons for the
631	   choices.

633	   [RFC8446] defines the following cipher suites for use with TLS 1.3.

635	   1.  TLS_AES_128_GCM_SHA256

637	   2.  TLS_AES_256_GCM_SHA384

639	   3.  TLS_CHACHA20_POLY1305_SHA256

641	   4.  TLS_AES_128_CCM_SHA256

643	   5.  TLS_AES_128_CCM_8_SHA256

645	4.3.2.  Backend Server Configuration

647	   This section specifies which parameters should be considered while
648	   configuring emulated backend servers using test equipment.

650	4.3.2.1.  TCP Stack Attributes

652	   The TCP stack on the server side SHOULD be configured similar to the
653	   client side configuration described in Section 4.3.1.1.  In addition,
654	   server initial congestion window MUST NOT exceed 10 times the MSS.
655	   Delayed ACKs are permitted and the maximum server delayed ACK MUST
656	   NOT exceed 10 times the MSS before a forced ACK.

658	4.3.2.2.  Server Endpoint IP Addressing

660	   The sum of the server IP space SHOULD contain the following
661	   attributes.

663	   *  The server IP blocks SHOULD consist of unique, discontinuous
664	      static address blocks with one IP per server Fully Qualified
665	      Domain Name (FQDN) endpoint per test port.

[minor] The "per FQDN per test port" is likely underspecified/confusing.
How would you recommend to configure the testbed if the same FQDN may be reachable
across more than one DUT server port and the DUT is doing load balancing ?
If that is not supposed to be considered, then it seems as if every FQDN is
supposed to be reachable across only one DUT port, but then the sentence
ikely should just say "per FQDN" (without the "per test port qualification").
Not 100% sure...

[minor] Especially for IPv4, there is obviously a big trend in DC to save
IPv4 address space by using SNI. Therefore a realistic scanerio would be
to have more than one FQDN per IPv4 address. Maybe as high as 10:1 (guesswork).
In any case i think it is prudent to include testing of such SNI overload
of IP addresses because it likely can impact performance (demux of processing
state not solely based on 5-tuple).

667	   *  A default gateway is permitted.  The DSCP (differentiated services

[minor] Again wondering why default gateway adds value to the doc.

667	   *  A default gateway is permitted.  The DSCP (differentiated services
668	      code point) marking is set to DF (Default Forwarding) '000000' on
669	      IPv4 Type of Service (ToS) field and IPv6 traffic class field.

671	   *  The server IP addresses SHOULD be distributed between IPv4 and
672	      IPv6 with a ratio identical to the clients distribution ratio.

674	   Note: The IANA has assigned IP address range for the testing purpose
675	   as described in Section 8.  If the test scenario requires more IP
676	   addresses or subnets than the IANA assigned, this document recommends
677	   using non routable Private IPv4 address ranges or Unique Local
678	   Address (ULA) IPv6 address ranges for the testing.

[minor] same note about moving these addressing recommendations out as in the
client section.

680	4.3.2.3.  HTTP / HTTPS Server Pool Endpoint Attributes

682	   The server pool for HTTP SHOULD listen on TCP port 80 and emulate the
683	   same HTTP version (HTTP 1.1 or HTTP/2 or HTTP/3) and settings chosen
684	   by the client (emulated web browser).  The Server MUST advertise
685	   server type in the Server response header [RFC7230].  For HTTPS
686	   server, TLS 1.2 or higher MUST be used with a maximum record size of
687	   16 KByte and MUST NOT use ticket resumption or session ID reuse.  The
688	   server SHOULD listen on TCP port 443 for HTTP version 1.1 and 2.  For
689	   HTTP/3 (HTTP over QUIC) the server SHOULD listen on UDP 443.  The
690	   server SHALL serve a certificate to the client.  The HTTPS server
691	   MUST check host SNI information with the FQDN if SNI is in use.
692	   Cipher suite and key size on the server side MUST be configured
693	   similar to the client side configuration described in
694	   Section 4.3.1.3.

696	4.3.3.  Traffic Flow Definition

698	   This section describes the traffic pattern between client and server
699	   endpoints.  At the beginning of the test, the server endpoint
700	   initializes and will be ready to accept connection states including
701	   initialization of the TCP stack as well as bound HTTP and HTTPS
702	   servers.  When a client endpoint is needed, it will initialize and be
703	   given attributes such as a MAC and IP address.  The behavior of the
704	   client is to sweep through the given server IP space, generating a
705	   recognizable service by the DUT.  Sequential and pseudorandom sweep
706	   methods are acceptable.  The method used MUST be stated in the final
707	   report.  Thus, a balanced mesh between client endpoints and server
708	   endpoints will be generated in a client IP and port to server IP and
709	   port combination.  Each client endpoint performs the same actions as
710	   other endpoints, with the difference being the source IP of the
711	   client endpoint and the target server IP pool.  The client MUST use
712	   the server IP address or FQDN in the host header [RFC7230].

[minor] given the prevalence of SNI centric server selection, i would suggest
to change server IP to server FQDN and note that server IP is simply derived from
server FQDN. Likewise server port is dervice from server protocol, which seems
to be just HTTP or HTTPs, so its unclear to me where we would get ports different
from 80 and 443 (maybe thats mentioned later). Aka: server Port may not be
relevant to mention.

714	4.3.3.1.  Description of Intra-Client Behavior

716	   Client endpoints are independent of other clients that are
717	   concurrently executing.  When a client endpoint initiates traffic,
718	   this section describes how the client steps through different
719	   services.  Once the test is initialized, the client endpoints
720	   randomly hold (perform no operation) for a few milliseconds for
721	   better randomization of the start of client traffic.  Each client
722	   will either open a new TCP connection or connect to a TCP persistence
723	   stack still open to that specific server.  At any point that the
724	   traffic profile may require encryption, a TLS encryption tunnel will
725	   form presenting the URL or IP address request to the server.  If
726	   using SNI, the server MUST then perform an SNI name check with the
727	   proposed FQDN compared to the domain embedded in the certificate.
728	   Only when correct, will the server process the HTTPS response object.
729	   The initial response object to the server is based on benchmarking
730	   tests described in Section 7.  Multiple additional sub-URLs (response
731	   objects on the service page) MAY be requested simultaneously.  This
732	   MAY be to the same server IP as the initial URL.  Each sub-object
733	   will also use a canonical FQDN and URL path, as observed in the
734	   traffic mix used.

[minor] This may be necessary to keep the configuration complexity at bay,
but in practice each particular IP client will likely exhibit quite different
traffic profiles. One may continuously request HTTP video segments when
streaming video. Another one may continuously do WebRTC (zoom), and the like.
BY having every client randomnly do all the services (this is what i figure from
above description), you forego the important performance aspect of "worst hit
client" if the DUT exhibits specific issues with specific services (false
filtering, performance degradation etc..). IMHO it would be great if test
equipment could create different client traffic profiles by segmentation of
the possible application space into groups and then assign new clients
randomnly to groups. Beside being able to easier find performance issues,
it is also resulting in more real-world performance, which might be higher.
For example in a multi-core CPU based DUT, there may be heuristics of
assigning different clients traffic to different CPU cores, so that L1..L3
cache of the CPU core can be better kept focussed on the codespace for
a particular type of client inspection. (just guessing).

736	4.3.4.  Traffic Load Profile

738	   The loading of traffic is described in this section.  The loading of
739	   a traffic load profile has five phases: Init, ramp up, sustain, ramp
740	   down, and collection.

742	   1.  Init phase: Testbed devices including the client and server
743	       endpoints should negotiate layer 2-3 connectivity such as MAC
744	       learning and ARP.  Only after successful MAC learning or ARP/ND
745	       resolution SHALL the test iteration move to the next phase.  No
746	       measurements are made in this phase.  The minimum RECOMMENDED
747	       time for Init phase is 5 seconds.  During this phase, the
748	       emulated clients SHOULD NOT initiate any sessions with the DUT/
749	       SUT, in contrast, the emulated servers should be ready to accept
750	       requests from DUT/SUT or from emulated clients.

752	   2.  Ramp up phase: The test equipment SHOULD start to generate the
753	       test traffic.  It SHOULD use a set of the approximate number of
754	       unique client IP addresses to generate traffic.  The traffic
755	       SHOULD ramp up from zero to desired target objective.  The target
756	       objective is defined for each benchmarking test.  The duration
757	       for the ramp up phase MUST be configured long enough that the
758	       test equipment does not overwhelm the DUT/SUTs stated performance
759	       metrics defined in Section 6.3 namely, TCP Connections Per
760	       Second, Inspected Throughput, Concurrent TCP Connections, and
761	       Application Transactions Per Second.  No measurements are made in
762	       this phase.

764	   3.  Sustain phase: Starts when all required clients are active and
765	       operating at their desired load condition.  In the sustain phase,
766	       the test equipment SHOULD continue generating traffic to constant
767	       target value for a constant number of active clients.  The
768	       minimum RECOMMENDED time duration for sustain phase is 300
769	       seconds.  This is the phase where measurements occur.  The test
770	       equipment SHOULD measure and record statistics continuously.  The
771	       sampling interval for collecting the raw results and calculating
772	       the statistics SHOULD be less than 2 seconds.

774	   4.  Ramp down phase: No new connections are established, and no
775	       measurements are made.  The time duration for ramp up and ramp
776	       down phase SHOULD be the same.

778	   5.  Collection phase: The last phase is administrative and will occur
779	       when the test equipment merges and collates the report data.

781	5.  Testbed Considerations

783	   This section describes steps for a reference test (pre-test) that
784	   control the test environment including test equipment, focusing on
785	   physical and virtualized environments and as well as test equipments.
786	   Below are the RECOMMENDED steps for the reference test.

788	   1.  Perform the reference test either by configuring the DUT/SUT in
789	       the most trivial setup (fast forwarding) or without presence of

[nit] Define/explain or provide reference for "fast forwarding".

790	       the DUT/SUT.

[minor] Is the DUT/SUT assumed to operate as a router or transparent L2 switch ?
Asking because "or without presence" should be amended (IMHO) with mentioning
that instead of the DUT one would put a router or switch in its place that
is pre-loaded with a config equivalent to that of the DUT but without any
seurity functions, just passing traffic at rates to bring the TE to its limits.

792	   2.  Generate traffic from traffic generator.  Choose a traffic
793	       profile used for HTTP or HTTPS throughput performance test with
794	       smallest object size.

796	   3.  Ensure that any ancillary switching or routing functions added in
797	       the test equipment does not limit the performance by introducing
798	       network metrics such as packet loss and latency.  This is
799	       specifically important for virtualized components (e.g.,
800	       vSwitches, vRouters).

802	   4.  Verify that the generated traffic (performance) of the test
803	       equipment matches and reasonably exceeds the expected maximum
804	       performance of the DUT/SUT.

806	   5.  Record the network performance metrics packet loss latency
807	       introduced by the test environment (without DUT/SUT).

809	   6.  Assert that the testbed characteristics are stable during the
810	       entire test session.  Several factors might influence stability
811	       specifically, for virtualized testbeds.  For example, additional
812	       workloads in a virtualized system, load balancing, and movement
813	       of virtual machines during the test, or simple issues such as
814	       additional heat created by high workloads leading to an emergency
815	       CPU performance reduction.

[minor] Add something to test the performance of the logging system. Without
DUT actually generating logging, this will so far not have been validated.
Maybe TE can generate logging records ? Especially burst logging from DUT
without loss is important to verify (no packet loss of logged events).

817	   The reference test SHOULD be performed before the benchmarking tests
818	   (described in section 7) start.

820	6.  Reporting

[minor] I would swap section 6 and 7, because it is problematic to read what's
to be reported without knowing whats to be measured first. For example, when i
read 6. first it was not clear to me if/how you would test the performance limits,
so the report data had a lot of questions for me.

Of course when you do run the testbed you first should have read both sections first.

822	   This section describes how the benchmarking test report should be
823	   formatted and presented.  It is RECOMMENDED to include two main
824	   sections in the report, namely the introduction and the detailed test
825	   results sections.

827	6.1.  Introduction

829	   The following attributes SHOULD be present in the introduction
830	   section of the test report.

[minor] I'd suggest to say here that the test report needs to include all information
sufficient for independent third-party reproduction of the test setup to permit
third party falsification of the test results. This includes but may not be limited
to the following...

832	   1.  The time and date of the execution of the tests

834	   2.  Summary of testbed software and hardware details
835	       a.  DUT/SUT hardware/virtual configuration

837	           *  This section SHOULD clearly identify the make and model of
838	              the DUT/SUT

840	           *  The port interfaces, including speed and link information

842	           *  If the DUT/SUT is a Virtual Network Function (VNF), host
843	              (server) hardware and software details, interface
844	              acceleration type such as DPDK and SR-IOV, used CPU cores,
845	              used RAM, resource sharing (e.g.  Pinning details and NUMA
846	              Node) configuration details, hypervisor version, virtual
847	              switch version

849	           *  details of any additional hardware relevant to the DUT/SUT
850	              such as controllers

852	       b.  DUT/SUT software

854	           *  Operating system name

856	           *  Version

858	           *  Specific configuration details (if any)

[minor] Any software details necessary and sufficient to preproduce the software setup of DUT/SUT.

860	       c.  DUT/SUT enabled features

862	           *  Configured DUT/SUT features (see Table 1 and Table 2)

864	           *  Attributes of the above-mentioned features

866	           *  Any additional relevant information about the features

868	       d.  Test equipment hardware and software

870	           *  Test equipment vendor name

872	           *  Hardware details including model number, interface type

874	           *  Test equipment firmware and test application software
875	              version

877	       e.  Key test parameters

879	           *  Used cipher suites and keys

881	           *  IPv4 and IPv6 traffic distribution
882	           *  Number of configured ACL

884	       f.  Details of application traffic mix used in the benchmarking
885	           test "Throughput Performance with Application Traffic Mix"
886	           (Section 7.1)

888	           *  Name of applications and layer 7 protocols

890	           *  Percentage of emulated traffic for each application and
891	              layer 7 protocols

893	           *  Percentage of encrypted traffic and used cipher suites and
894	              keys (The RECOMMENDED ciphers and keys are defined in
895	              Section 4.3.1.3)

897	           *  Used object sizes for each application and layer 7
898	              protocols

900	   3.  Results Summary / Executive Summary

902	       a.  Results SHOULD resemble a pyramid in how it is reported, with
903	           the introduction section documenting the summary of results
904	           in a prominent, easy to read block.

906	6.2.  Detailed Test Results

908	   In the result section of the test report, the following attributes
909	   SHOULD be present for each benchmarking test.

911	   a.  KPIs MUST be documented separately for each benchmarking test.
912	       The format of the KPI metrics SHOULD be presented as described in
913	       Section 6.3.

915	   b.  The next level of details SHOULD be graphs showing each of these
916	       metrics over the duration (sustain phase) of the test.  This
917	       allows the user to see the measured performance stability changes
918	       over time.

920	6.3.  Benchmarks and Key Performance Indicators

922	   This section lists key performance indicators (KPIs) for overall
923	   benchmarking tests.  All KPIs MUST be measured during the sustain
924	   phase of the traffic load profile described in Section 4.3.4.  All
925	   KPIs MUST be measured from the result output of test equipment.

[minor] At some other place of the document i think to remember observing of
DUT self-reporting. Shouldn't then the self-reporting of the DUT be vetted as well,
e.g.: compared against the TE report data ?

927	   *  Concurrent TCP Connections
928	      The aggregate number of simultaneous connections between hosts
929	      across the DUT/SUT, or between hosts and the DUT/SUT (defined in
930	      [RFC2647]).

[minor] Add reference to section in rfc2647 where this is defined. Also: If you
refer but not reproduce

932	   *  TCP Connections Per Second

934	      The average number of successfully established TCP connections per
935	      second between hosts across the DUT/SUT, or between hosts and the
936	      DUT/SUT.  The TCP connection MUST be initiated via a TCP three-way
937	      handshake (SYN, SYN/ACK, ACK).  Then the TCP session data is sent.
938	      The TCP session MUST be closed via either a TCP three-way close
939	      (FIN, FIN/ACK, ACK), or a TCP four-way close (FIN, ACK, FIN, ACK),
940	      and MUST NOT by RST.

942	   *  Application Transactions Per Second

944	      The average number of successfully completed transactions per
945	      second.  For a particular transaction to be considered successful,
946	      all data MUST have been transferred in its entirety.  In case of
947	      HTTP(S) transactions, it MUST have a valid status code (200 OK),
948	      and the appropriate FIN, FIN/ACK sequence MUST have been
949	      completed.

951	   *  TLS Handshake Rate

953	      The average number of successfully established TLS connections per
954	      second between hosts across the DUT/SUT, or between hosts and the
955	      DUT/SUT.

957	   *  Inspected Throughput

959	      The number of bits per second of examined and allowed traffic a
960	      network security device is able to transmit to the correct
961	      destination interface(s) in response to a specified offered load.
962	      The throughput benchmarking tests defined in Section 7 SHOULD
963	      measure the average Layer 2 throughput value when the DUT/SUT is
964	      "inspecting" traffic.  This document recommends presenting the
965	      inspected throughput value in Gbit/s rounded to two places of
966	      precision with a more specific Kbit/s in parenthesis.

968	   *  Time to First Byte (TTFB)

970	      TTFB is the elapsed time between the start of sending the TCP SYN
971	      packet from the client and the client receiving the first packet
972	      of application data from the server or DUT/SUT.  The benchmarking
973	      tests HTTP Transaction Latency (Section 7.4) and HTTPS Transaction
974	      Latency (Section 7.8) measure the minimum, average and maximum
975	      TTFB.  The value SHOULD be expressed in milliseconds.

977	   *  URL Response time / Time to Last Byte (TTLB)

979	      URL Response time / TTLB is the elapsed time between the start of
980	      sending the TCP SYN packet from the client and the client
981	      receiving the last packet of application data from the server or
982	      DUT/SUT.  The benchmarking tests HTTP Transaction Latency
983	      (Section 7.4) and HTTPS Transaction Latency (Section 7.8) measure
984	      the minimum, average and maximum TTLB.  The value SHOULD be
985	      expressed in millisecond.

[minor] Up to this point i don't think the report would include comparison for these
KPI between no-DUT-present vs. DUT-present. Is that true ? How then is the reaader of
the report meant to be able to vet the relative impact of the DUT for all these metrics 
vs. DUT not being present ?

987	7.  Benchmarking Tests

[minor] I think it would be good to insert here some descriptive and comparative overview
of the tests from the different 7.x sections.

For example, i guess (but don't know from the test), that the 7.1 test should ?
perform throughput test for non-http/https applications, or else if all the applications
in 7.1 would be http/https, then it would duplicate the results of 7.3 and 7.7, right ?
Not sure though if/where it is written out that you therefore want a traffic mix of
only non-HTTP/HTTPS application traffic for 7.1.

If instead the customer relevant application mix (7.1.1) does include some percentage
of HTTP/HTTP applications, then shouldn't all the tests, even those focussing on the
HTTP/HTTPs characteristic also always include the non-HTTP/HTTPs application flows as
kind of "background" traffic, even if not measured in the tests of particular 7.x sub-section ?

[minor] Section 7. is a lot of work to get right. I observe that there is a lot of
procedural replication across the steps. It would be easier to read if all that
duplication was removed and described once - such as the initial/max/iterative step
description. But i can understand how much work this might be, to then especially
extraxct only the differences for each 7.x and only describe those 7.x differences there.

989	7.1.  Throughput Performance with Application Traffic Mix

991	7.1.1.  Objective

993	   Using a relevant application traffic mix, determine the sustainable
994	   inspected throughput supported by the DUT/SUT.

996	   Based on the test customer's specific use case, testers can choose
997	   the relevant application traffic mix for this test.  The details
998	   about the traffic mix MUST be documented in the report.  At least the
999	   following traffic mix details MUST be documented and reported
1000	   together with the test results:

1002	      Name of applications and layer 7 protocols

1004	      Percentage of emulated traffic for each application and layer 7
1005	      protocol

1007	      Percentage of encrypted traffic and used cipher suites and keys
1008	      (The RECOMMENDED ciphers and keys are defined in Section 4.3.1.3.)

1010	      Used object sizes for each application and layer 7 protocols

1012	7.1.2.  Test Setup

1014	   Testbed setup MUST be configured as defined in Section 4.  Any
1015	   benchmarking test specific testbed configuration changes MUST be
1016	   documented.

1018	7.1.3.  Test Parameters

1020	   In this section, the benchmarking test specific parameters SHOULD be
1021	   defined.

1023	7.1.3.1.  DUT/SUT Configuration Parameters

1025	   DUT/SUT parameters MUST conform to the requirements defined in
1026	   Section 4.2.  Any configuration changes for this specific
1027	   benchmarking test MUST be documented.  In case the DUT/SUT is
1028	   configured without SSL inspection, the test report MUST explain the
1029	   implications of this to the relevant application traffic mix
1030	   encrypted traffic.

[nit] /SSL inspection/SSL Inspection/ - capitalized in all other places in the doc.

[minor] I am not quite familiar with the details, so i hope a ready knows what
the "MUST explain the implication" means.

[minor] What is the equivalent for TLS (inspection), and why is it not equally mentioned ?

1032	7.1.3.2.  Test Equipment Configuration Parameters

1034	   Test equipment configuration parameters MUST conform to the
1035	   requirements defined in Section 4.3.  The following parameters MUST
1036	   be documented for this benchmarking test:

1038	      Client IP address range defined in Section 4.3.1.2

1040	      Server IP address range defined in Section 4.3.2.2

1042	      Traffic distribution ratio between IPv4 and IPv6 defined in
1043	      Section 4.3.1.2

1045	      Target inspected throughput: Aggregated line rate of interface(s)
1046	      used in the DUT/SUT or the value defined based on requirement for
1047	      a specific deployment scenario

[minor] maybe add: or based on DUT specified performance limits (DUT may not always
provide "linerate" throughput, so the ultimate test would be to see if/how much of
the vendor promised performance is reachable.

1049	      Initial throughput: 10% of the "Target inspected throughput" Note:
1050	      Initial throughput is not a KPI to report.  This value is
1051	      configured on the traffic generator and used to perform Step 1:
1052	      "Test Initialization and Qualification" described under the
1053	      Section 7.1.4.

1055	      One of the ciphers and keys defined in Section 4.3.1.3 are
1056	      RECOMMENDED to use for this benchmarking test.

1058	7.1.3.3.  Traffic Profile

1060	   Traffic profile: This test MUST be run with a relevant application
1061	   traffic mix profile.

1063	7.1.3.4.  Test Results Validation Criteria

1065	   The following criteria are the test results validation criteria.  The
1066	   test results validation criteria MUST be monitored during the whole
1067	   sustain phase of the traffic load profile.

1069	   a.  Number of failed application transactions (receiving any HTTP
1070	       response code other than 200 OK) MUST be less than 0.001% (1 out
1071	       of 100,000 transactions) of total attempted transactions.

[minor] So this is the right number, as opposed to the 0.01% in A.4...
If you don't intend to fix A.4 (requested there), pls. explain the reason for the
difference.

1073	   b.  Number of Terminated TCP connections due to unexpected TCP RST
1074	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
1075	       connections) of total initiated TCP connections.

1077	7.1.3.5.  Measurement

1079	   Following KPI metrics MUST be reported for this benchmarking test:

1081	   Mandatory KPIs (benchmarks): Inspected Throughput, TTFB (minimum,
1082	   average, and maximum), TTLB (minimum, average, and maximum) and
1083	   Application Transactions Per Second

1085	   Note: TTLB MUST be reported along with the object size used in the
1086	   traffic profile.

1088	   Optional KPIs: TCP Connections Per Second and TLS Handshake Rate

[minor] I would prefer for TCP connections to be mandatory too. Makes it easier to
communicate test data with lower layer folks. FOr example, network layer equipment
often has per 5-tuple flow state also with build/churn-rate limits, so to match
a security SUT with the other networking equipment this TCP connection rate rate
is quite important.

1090	7.1.4.  Test Procedures and Expected Results

1092	   The test procedures are designed to measure the inspected throughput
1093	   performance of the DUT/SUT at the sustaining period of traffic load
1094	   profile.  The test procedure consists of three major steps: Step 1
1095	   ensures the DUT/SUT is able to reach the performance value (initial
1096	   throughput) and meets the test results validation criteria when it
1097	   was very minimally utilized.  Step 2 determines the DUT/SUT is able
1098	   to reach the target performance value within the test results
1099	   validation criteria.  Step 3 determines the maximum achievable
1100	   performance value within the test results validation criteria.

1102	   This test procedure MAY be repeated multiple times with different IP
1103	   types: IPv4 only, IPv6 only, and IPv4 and IPv6 mixed traffic
1104	   distribution.

1106	7.1.4.1.  Step 1: Test Initialization and Qualification

1108	   Verify the link status of all connected physical interfaces.  All
1109	   interfaces are expected to be in "UP" status.

1111	   Configure traffic load profile of the test equipment to generate test
1112	   traffic at the "Initial throughput" rate as described in
1113	   Section 7.1.3.2.  The test equipment SHOULD follow the traffic load
1114	   profile definition as described in Section 4.3.4.  The DUT/SUT SHOULD
1115	   reach the "Initial throughput" during the sustain phase.  Measure all
1116	   KPI as defined in Section 7.1.3.5.  The measured KPIs during the
1117	   sustain phase MUST meet all the test results validation criteria
1118	   defined in Section 7.1.3.4.

1120	   If the KPI metrics do not meet the test results validation criteria,
1121	   the test procedure MUST NOT be continued to step 2.

1123	7.1.4.2.  Step 2: Test Run with Target Objective

1125	   Configure test equipment to generate traffic at the "Target inspected
1126	   throughput" rate defined in Section 7.1.3.2.  The test equipment
1127	   SHOULD follow the traffic load profile definition as described in
1128	   Section 4.3.4.  The test equipment SHOULD start to measure and record
1129	   all specified KPIs.  Continue the test until all traffic profile
1130	   phases are completed.

1132	   Within the test results validation criteria, the DUT/SUT is expected
1133	   to reach the desired value of the target objective ("Target inspected
1134	   throughput") in the sustain phase.  Follow step 3, if the measured
1135	   value does not meet the target value or does not fulfill the test
1136	   results validation criteria.

1138	7.1.4.3.  Step 3: Test Iteration

1140	   Determine the achievable average inspected throughput within the test
1141	   results validation criteria.  Final test iteration MUST be performed
1142	   for the test duration defined in Section 4.3.4.

1144	7.2.  TCP/HTTP Connections Per Second

1146	7.2.1.  Objective

1148	   Using HTTP traffic, determine the sustainable TCP connection
1149	   establishment rate supported by the DUT/SUT under different
1150	   throughput load conditions.

1152	   To measure connections per second, test iterations MUST use different
1153	   fixed HTTP response object sizes (the different load conditions)
1154	   defined in Section 7.2.3.2.

1156	7.2.2.  Test Setup

1158	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1159	   specific testbed configuration changes (number of interfaces and
1160	   interface type, etc.)  MUST be documented.

1162	7.2.3.  Test Parameters

1164	   In this section, benchmarking test specific parameters SHOULD be
1165	   defined.

1167	7.2.3.1.  DUT/SUT Configuration Parameters

1169	   DUT/SUT parameters MUST conform to the requirements defined in
1170	   Section 4.2.  Any configuration changes for this specific
1171	   benchmarking test MUST be documented.

1173	7.2.3.2.  Test Equipment Configuration Parameters

1175	   Test equipment configuration parameters MUST conform to the
1176	   requirements defined in Section 4.3.  The following parameters MUST
1177	   be documented for this benchmarking test:

1179	   Client IP address range defined in Section 4.3.1.2

1181	   Server IP address range defined in Section 4.3.2.2

1183	   Traffic distribution ratio between IPv4 and IPv6 defined in
1184	   Section 4.3.1.2

1186	   Target connections per second: Initial value from product datasheet
1187	   or the value defined based on requirement for a specific deployment
1188	   scenario

1190	   Initial connections per second: 10% of "Target connections per
1191	   second" (Note: Initial connections per second is not a KPI to report.
1192	   This value is configured on the traffic generator and used to perform
1193	   the Step1: "Test Initialization and Qualification" described under
1194	   the Section 7.2.4.

1196	   The client SHOULD negotiate HTTP and close the connection with FIN
1197	   immediately after completion of one transaction.  In each test
1198	   iteration, client MUST send GET request requesting a fixed HTTP
1199	   response object size.

1201	   The RECOMMENDED response object sizes are 1, 2, 4, 16, and 64 KByte.

1203	7.2.3.3.  Test Results Validation Criteria

1205	   The following criteria are the test results validation criteria.  The
1206	   Test results validation criteria MUST be monitored during the whole
1207	   sustain phase of the traffic load profile.

1209	   a.  Number of failed application transactions (receiving any HTTP
1210	       response code other than 200 OK) MUST be less than 0.001% (1 out
1211	       of 100,000 transactions) of total attempted transactions.

1213	   b.  Number of terminated TCP connections due to unexpected TCP RST
1214	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
1215	       connections) of total initiated TCP connections.

1217	   c.  During the sustain phase, traffic SHOULD be forwarded at a
1218	       constant rate (considered as a constant rate if any deviation of
1219	       traffic forwarding rate is less than 5%).

1221	   d.  Concurrent TCP connections MUST be constant during steady state
1222	       and any deviation of concurrent TCP connections SHOULD be less
1223	       than 10%. This confirms the DUT opens and closes TCP connections
1224	       at approximately the same rate.

1226	7.2.3.4.  Measurement

1228	   TCP Connections Per Second MUST be reported for each test iteration
1229	   (for each object size).

[minor] Add Variance or min/max rates to report in case above point d (line 1221) problem does
exist ?

1231	7.2.4.  Test Procedures and Expected Results

1233	   The test procedure is designed to measure the TCP connections per
1234	   second rate of the DUT/SUT at the sustaining period of the traffic
1235	   load profile.  The test procedure consists of three major steps: Step
1236	   1 ensures the DUT/SUT is able to reach the performance value (Initial
1237	   connections per second) and meets the test results validation
1238	   criteria when it was very minimally utilized.  Step 2 determines the
1239	   DUT/SUT is able to reach the target performance value within the test
1240	   results validation criteria.  Step 3 determines the maximum
1241	   achievable performance value within the test results validation
1242	   criteria.

1244	   This test procedure MAY be repeated multiple times with different IP
1245	   types: IPv4 only, IPv6 only, and IPv4 and IPv6 mixed traffic
1246	   distribution.

1248	7.2.4.1.  Step 1: Test Initialization and Qualification

1250	   Verify the link status of all connected physical interfaces.  All
1251	   interfaces are expected to be in "UP" status.

1253	   Configure the traffic load profile of the test equipment to establish
1254	   "Initial connections per second" as defined in Section 7.2.3.2.  The
1255	   traffic load profile SHOULD be defined as described in Section 4.3.4.

1257	   The DUT/SUT SHOULD reach the "Initial connections per second" before
1258	   the sustain phase.  The measured KPIs during the sustain phase MUST
1259	   meet all the test results validation criteria defined in
1260	   Section 7.2.3.3.

1262	   If the KPI metrics do not meet the test results validation criteria,
1263	   the test procedure MUST NOT continue to "Step 2".

1265	7.2.4.2.  Step 2: Test Run with Target Objective

1267	   Configure test equipment to establish the target objective ("Target
1268	   connections per second") defined in Section 7.2.3.2.  The test
1269	   equipment SHOULD follow the traffic load profile definition as
1270	   described in Section 4.3.4.

1272	   During the ramp up and sustain phase of each test iteration, other
1273	   KPIs such as inspected throughput, concurrent TCP connections and
1274	   application transactions per second MUST NOT reach the maximum value
1275	   the DUT/SUT can support.  The test results for specific test
1276	   iterations SHOULD NOT be reported, if the above-mentioned KPI
1277	   (especially inspected throughput) reaches the maximum value.
1278	   (Example: If the test iteration with 64 KByte of HTTP response object
1279	   size reached the maximum inspected throughput limitation of the DUT/
1280	   SUT, the test iteration MAY be interrupted and the result for 64
1281	   KByte SHOULD NOT be reported.)

1283	   The test equipment SHOULD start to measure and record all specified
1284	   KPIs.  Continue the test until all traffic profile phases are
1285	   completed.

1287	   Within the test results validation criteria, the DUT/SUT is expected
1288	   to reach the desired value of the target objective ("Target
1289	   connections per second") in the sustain phase.  Follow step 3, if the
1290	   measured value does not meet the target value or does not fulfill the
1291	   test results validation criteria.

1293	7.2.4.3.  Step 3: Test Iteration

1295	   Determine the achievable TCP connections per second within the test
1296	   results validation criteria.

1298	7.3.  HTTP Throughput

1300	7.3.1.  Objective

1302	   Determine the sustainable inspected throughput of the DUT/SUT for
1303	   HTTP transactions varying the HTTP response object size.

[nit] High level, what is the difference between 7.2 and 7.3 ? Some more explanation
would be useful. One interpretation i came up with is that 7.2 measures performane
of e.g.: HTTP connections where each connection performs a single GET, and 7.3
measures long-lived HTTP connections in which a high rate of HTTP GET is performed
(so as to differentiate transactions at TCP+HTTP level (7.2) from those only happening
at HTTP level (7.3). If that is a lucky guess it might help other similarily guessing
readers to write this out more explicitly.

1305	7.3.2.  Test Setup

1307	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1308	   specific testbed configuration changes (number of interfaces and
1309	   interface type, etc.)  MUST be documented.

1311	7.3.3.  Test Parameters

1313	   In this section, benchmarking test specific parameters SHOULD be
1314	   defined.

1316	7.3.3.1.  DUT/SUT Configuration Parameters

1318	   DUT/SUT parameters MUST conform to the requirements defined in
1319	   Section 4.2.  Any configuration changes for this specific
1320	   benchmarking test MUST be documented.

1322	7.3.3.2.  Test Equipment Configuration Parameters

1324	   Test equipment configuration parameters MUST conform to the
1325	   requirements defined in Section 4.3.  The following parameters MUST
1326	   be documented for this benchmarking test:

1328	   Client IP address range defined in Section 4.3.1.2

1330	   Server IP address range defined in Section 4.3.2.2

1332	   Traffic distribution ratio between IPv4 and IPv6 defined in
1333	   Section 4.3.1.2

1335	   Target inspected throughput: Aggregated line rate of interface(s)
1336	   used in the DUT/SUT or the value defined based on requirement for a
1337	   specific deployment scenario
1338	   Initial throughput: 10% of "Target inspected throughput" Note:
1339	   Initial throughput is not a KPI to report.  This value is configured
1340	   on the traffic generator and used to perform Step 1: "Test
1341	   Initialization and Qualification" described under Section 7.3.4.

1343	   Number of HTTP response object requests (transactions) per
1344	   connection: 10

1346	   RECOMMENDED HTTP response object size: 1, 16, 64, 256 KByte, and
1347	   mixed objects defined in Table 4.

1349	           +=====================+============================+
1350	           | Object size (KByte) | Number of requests/ Weight |
1351	           +=====================+============================+
1352	           | 0.2                 | 1                          |
1353	           +---------------------+----------------------------+
1354	           | 6                   | 1                          |
1355	           +---------------------+----------------------------+
1356	           | 8                   | 1                          |
1357	           +---------------------+----------------------------+
1358	           | 9                   | 1                          |
1359	           +---------------------+----------------------------+
1360	           | 10                  | 1                          |
1361	           +---------------------+----------------------------+
1362	           | 25                  | 1                          |
1363	           +---------------------+----------------------------+
1364	           | 26                  | 1                          |
1365	           +---------------------+----------------------------+
1366	           | 35                  | 1                          |
1367	           +---------------------+----------------------------+
1368	           | 59                  | 1                          |
1369	           +---------------------+----------------------------+
1370	           | 347                 | 1                          |
1371	           +---------------------+----------------------------+

1373	                          Table 4: Mixed Objects

[minor] Interesting/useful data. If there was any reference/explanation how these
numbere where derived, that would be great to add.

1375	7.3.3.3.  Test Results Validation Criteria

1377	   The following criteria are the test results validation criteria.  The
1378	   test results validation criteria MUST be monitored during the whole
1379	   sustain phase of the traffic load profile.

1381	   a.  Number of failed application transactions (receiving any HTTP
1382	       response code other than 200 OK) MUST be less than 0.001% (1 out
1383	       of 100,000 transactions) of attempt transactions.

1385	   b.  Traffic SHOULD be forwarded at a constant rate (considered as a
1386	       constant rate if any deviation of traffic forwarding rate is less
1387	       than 5%).

1389	   c.  Concurrent TCP connections MUST be constant during steady state
1390	       and any deviation of concurrent TCP connections SHOULD be less
1391	       than 10%. This confirms the DUT opens and closes TCP connections
1392	       at approximately the same rate.

1394	7.3.3.4.  Measurement

1396	   Inspected Throughput and HTTP Transactions per Second MUST be
1397	   reported for each object size.

1399	7.3.4.  Test Procedures and Expected Results

1401	   The test procedure is designed to measure HTTP throughput of the DUT/
1402	   SUT.  The test procedure consists of three major steps: Step 1
1403	   ensures the DUT/SUT is able to reach the performance value (Initial
1404	   throughput) and meets the test results validation criteria when it
1405	   was very minimal utilized.  Step 2 determines the DUT/SUT is able to
1406	   reach the target performance value within the test results validation
1407	   criteria.  Step 3 determines the maximum achievable performance value
1408	   within the test results validation criteria.

1410	   This test procedure MAY be repeated multiple times with different
1411	   IPv4 and IPv6 traffic distribution and HTTP response object sizes.

1413	7.3.4.1.  Step 1: Test Initialization and Qualification

1415	   Verify the link status of all connected physical interfaces.  All
1416	   interfaces are expected to be in "UP" status.

1418	   Configure traffic load profile of the test equipment to establish
1419	   "Initial inspected throughput" as defined in Section 7.3.3.2.

1421	   The traffic load profile SHOULD be defined as described in
1422	   Section 4.3.4.  The DUT/SUT SHOULD reach the "Initial inspected
1423	   throughput" during the sustain phase.  Measure all KPI as defined in
1424	   Section 7.3.3.4.

1426	   The measured KPIs during the sustain phase MUST meet the test results
1427	   validation criteria "a" defined in Section 7.3.3.3.  The test results
1428	   validation criteria "b" and "c" are OPTIONAL for step 1.

1430	   If the KPI metrics do not meet the test results validation criteria,
1431	   the test procedure MUST NOT be continued to "Step 2".

1433	7.3.4.2.  Step 2: Test Run with Target Objective

1435	   Configure test equipment to establish the target objective ("Target
1436	   inspected throughput") defined in Section 7.3.3.2.  The test
1437	   equipment SHOULD start to measure and record all specified KPIs.
1438	   Continue the test until all traffic profile phases are completed.

1440	   Within the test results validation criteria, the DUT/SUT is expected
1441	   to reach the desired value of the target objective in the sustain
1442	   phase.  Follow step 3, if the measured value does not meet the target
1443	   value or does not fulfill the test results validation criteria.

1445	7.3.4.3.  Step 3: Test Iteration

1447	   Determine the achievable inspected throughput within the test results
1448	   validation criteria and measure the KPI metric Transactions per
1449	   Second.  Final test iteration MUST be performed for the test duration
1450	   defined in Section 4.3.4.

1452	7.4.  HTTP Transaction Latency

[nit] It would be nice to have explanatory text explaining why 7.4 requires different
test runs as opposed to just measuring the transaction latency as part of 7.2 and 7.3.
I have not tried to compare in detail the descriptions here to figure out the differences
in test runs, but even if there are differences, why would transaction latency not
also be measured in 7.2 and 7.3 as a metric ?

1454	7.4.1.  Objective

1456	   Using HTTP traffic, determine the HTTP transaction latency when DUT
1457	   is running with sustainable HTTP transactions per second supported by
1458	   the DUT/SUT under different HTTP response object sizes.

1460	   Test iterations MUST be performed with different HTTP response object
1461	   sizes in two different scenarios.  One with a single transaction and
1462	   the other with multiple transactions within a single TCP connection.
1463	   For consistency both the single and multiple transaction test MUST be
1464	   configured with the same HTTP version

1466	   Scenario 1: The client MUST negotiate HTTP and close the connection
1467	   with FIN immediately after completion of a single transaction (GET
1468	   and RESPONSE).

1470	   Scenario 2: The client MUST negotiate HTTP and close the connection
1471	   FIN immediately after completion of 10 transactions (GET and
1472	   RESPONSE) within a single TCP connection.

1474	7.4.2.  Test Setup

1476	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1477	   specific testbed configuration changes (number of interfaces and
1478	   interface type, etc.)  MUST be documented.

1480	7.4.3.  Test Parameters

1482	   In this section, benchmarking test specific parameters SHOULD be
1483	   defined.

1485	7.4.3.1.  DUT/SUT Configuration Parameters

1487	   DUT/SUT parameters MUST conform to the requirements defined in
1488	   Section 4.2.  Any configuration changes for this specific
1489	   benchmarking test MUST be documented.

1491	7.4.3.2.  Test Equipment Configuration Parameters

1493	   Test equipment configuration parameters MUST conform to the
1494	   requirements defined in Section 4.3.  The following parameters MUST
1495	   be documented for this benchmarking test:

1497	   Client IP address range defined in Section 4.3.1.2

1499	   Server IP address range defined in Section 4.3.2.2

1501	   Traffic distribution ratio between IPv4 and IPv6 defined in
1502	   Section 4.3.1.2

1504	   Target objective for scenario 1: 50% of the connections per second
1505	   measured in benchmarking test TCP/HTTP Connections Per Second
1506	   (Section 7.2)

1508	   Target objective for scenario 2: 50% of the inspected throughput
1509	   measured in benchmarking test HTTP Throughput (Section 7.3)

1511	   Initial objective for scenario 1: 10% of "Target objective for
1512	   scenario 1"

1514	   Initial objective for scenario 2: 10% of "Target objective for
1515	   scenario 2"

1517	   Note: The Initial objectives are not a KPI to report.  These values
1518	   are configured on the traffic generator and used to perform the
1519	   Step1: "Test Initialization and Qualification" described under the
1520	   Section 7.4.4.

1522	   HTTP transaction per TCP connection: Test scenario 1 with single
1523	   transaction and test scenario 2 with 10 transactions.

1525	   HTTP with GET request requesting a single object.  The RECOMMENDED
1526	   object sizes are 1, 16, and 64 KByte.  For each test iteration,
1527	   client MUST request a single HTTP response object size.

1529	7.4.3.3.  Test Results Validation Criteria

1531	   The following criteria are the test results validation criteria.  The
1532	   Test results validation criteria MUST be monitored during the whole
1533	   sustain phase of the traffic load profile.

1535	   a.  Number of failed application transactions (receiving any HTTP
1536	       response code other than 200 OK) MUST be less than 0.001% (1 out
1537	       of 100,000 transactions) of attempt transactions.

1539	   b.  Number of terminated TCP connections due to unexpected TCP RST
1540	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
1541	       connections) of total initiated TCP connections.

1543	   c.  During the sustain phase, traffic SHOULD be forwarded at a
1544	       constant rate (considered as a constant rate if any deviation of
1545	       traffic forwarding rate is less than 5%).

1547	   d.  Concurrent TCP connections MUST be constant during steady state
1548	       and any deviation of concurrent TCP connections SHOULD be less
1549	       than 10%. This confirms the DUT opens and closes TCP connections
1550	       at approximately the same rate.

1552	   e.  After ramp up the DUT MUST achieve the "Target objective" defined
1553	       in Section 7.4.3.2 and remain in that state for the entire test
1554	       duration (sustain phase).

1556	7.4.3.4.  Measurement

1558	   TTFB (minimum, average, and maximum) and TTLB (minimum, average and
1559	   maximum) MUST be reported for each object size.

1561	7.4.4.  Test Procedures and Expected Results

1563	   The test procedure is designed to measure TTFB or TTLB when the DUT/
1564	   SUT is operating close to 50% of its maximum achievable connections
1565	   per second or inspected throughput.  The test procedure consists of
1566	   two major steps: Step 1 ensures the DUT/SUT is able to reach the
1567	   initial performance values and meets the test results validation
1568	   criteria when it was very minimally utilized.  Step 2 measures the
1569	   latency values within the test results validation criteria.

1571	   This test procedure MAY be repeated multiple times with different IP
1572	   types (IPv4 only, IPv6 only and IPv4 and IPv6 mixed traffic
1573	   distribution), HTTP response object sizes and single and multiple
1574	   transactions per connection scenarios.

1576	7.4.4.1.  Step 1: Test Initialization and Qualification

1578	   Verify the link status of all connected physical interfaces.  All
1579	   interfaces are expected to be in "UP" status.

1581	   Configure traffic load profile of the test equipment to establish
1582	   "Initial objective" as defined in Section 7.4.3.2.  The traffic load
1583	   profile SHOULD be defined as described in Section 4.3.4.

1585	   The DUT/SUT SHOULD reach the "Initial objective" before the sustain
1586	   phase.  The measured KPIs during the sustain phase MUST meet all the
1587	   test results validation criteria defined in Section 7.4.3.3.

1589	   If the KPI metrics do not meet the test results validation criteria,
1590	   the test procedure MUST NOT be continued to "Step 2".

1592	7.4.4.2.  Step 2: Test Run with Target Objective

1594	   Configure test equipment to establish "Target objective" defined in
1595	   Section 7.4.3.2.  The test equipment SHOULD follow the traffic load
1596	   profile definition as described in Section 4.3.4.

1598	   The test equipment SHOULD start to measure and record all specified
1599	   KPIs.  Continue the test until all traffic profile phases are
1600	   completed.

1602	   Within the test results validation criteria, the DUT/SUT MUST reach
1603	   the desired value of the target objective in the sustain phase.

1605	   Measure the minimum, average, and maximum values of TTFB and TTLB.

1607	7.5.  Concurrent TCP/HTTP Connection Capacity

[nit] again a summary comparison of the traffic in 7.5 vs. the prior traffic profiles
would be helpful to understand the benefit of these test runs. Is this about any
real-world reqirement or more a synthetic performance number for unrealistic HTTP
connections (which would still be a useful number IMHO, just want to know) ? 

The traffic profile below is somewhat strange because
it defines the rate of GET within a TCP connection based not on real-world application
behavior, but just to create some rate of GET per TCP connection over the steady state.
I guess the goal is something like "measure the maximum sustainable number of TCP/HTTP
connctions, wehreas each connection carries as little as possible traffic and a sufficiently
low number of HTTP (GET) transactions that the DUT is not too much performance loaded
with the HTTP level inspection, but mostly with HTTP/TCP flow maintainance ??

In general, describing for each of the 7.x section upfront the goal and design criteria
of the test runs in those high-level terms is IMHO very beneficial for reviewers to
vet if/how well the detailled description does meet the goals. Otherwise one is somewhat
left puzzling about that question. Aka: enhance the 7.x.1 objective sessions with that
amount of details.

1609	7.5.1.  Objective

1611	   Determine the number of concurrent TCP connections that the DUT/ SUT
1612	   sustains when using HTTP traffic.

1614	7.5.2.  Test Setup

1616	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1617	   specific testbed configuration changes (number of interfaces and
1618	   interface type, etc.)  MUST be documented.

1620	7.5.3.  Test Parameters

1622	   In this section, benchmarking test specific parameters SHOULD be
1623	   defined.

1625	7.5.3.1.  DUT/SUT Configuration Parameters

1627	   DUT/SUT parameters MUST conform to the requirements defined in
1628	   Section 4.2.  Any configuration changes for this specific
1629	   benchmarking test MUST be documented.

1631	7.5.3.2.  Test Equipment Configuration Parameters

1633	   Test equipment configuration parameters MUST conform to the
1634	   requirements defined in Section 4.3.  The following parameters MUST
1635	   be noted for this benchmarking test:

1637	      Client IP address range defined in Section 4.3.1.2

1639	      Server IP address range defined in Section 4.3.2.2

1641	      Traffic distribution ratio between IPv4 and IPv6 defined in
1642	      Section 4.3.1.2

1644	      Target concurrent connection: Initial value from product datasheet
1645	      or the value defined based on requirement for a specific
1646	      deployment scenario.

1648	      Initial concurrent connection: 10% of "Target concurrent
1649	      connection" Note: Initial concurrent connection is not a KPI to
1650	      report.  This value is configured on the traffic generator and
1651	      used to perform the Step1: "Test Initialization and Qualification"
1652	      described under the Section 7.5.4.

1654	      Maximum connections per second during ramp up phase: 50% of
1655	      maximum connections per second measured in benchmarking test TCP/
1656	      HTTP Connections per second (Section 7.2)

1658	      Ramp up time (in traffic load profile for "Target concurrent
1659	      connection"): "Target concurrent connection" / "Maximum
1660	      connections per second during ramp up phase"

1662	      Ramp up time (in traffic load profile for "Initial concurrent
1663	      connection"): "Initial concurrent connection" / "Maximum
1664	      connections per second during ramp up phase"

1666	   The client MUST negotiate HTTP and each client MAY open multiple
1667	   concurrent TCP connections per server endpoint IP.

1669	   Each client sends 10 GET requests requesting 1 KByte HTTP response
1670	   object in the same TCP connection (10 transactions/TCP connection)
1671	   and the delay (think time) between each transaction MUST be X
1672	   seconds.

1674	   X = ("Ramp up time" + "steady state time") /10

1676	   The established connections SHOULD remain open until the ramp down
1677	   phase of the test.  During the ramp down phase, all connections
1678	   SHOULD be successfully closed with FIN.

1680	7.5.3.3.  Test Results Validation Criteria

1682	   The following criteria are the test results validation criteria.  The
1683	   Test results validation criteria MUST be monitored during the whole
1684	   sustain phase of the traffic load profile.

1686	   a.  Number of failed application transactions (receiving any HTTP
1687	       response code other than 200 OK) MUST be less than 0.001% (1 out
1688	       of 100,000 transaction) of total attempted transactions.

1690	   b.  Number of terminated TCP connections due to unexpected TCP RST
1691	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
1692	       connections) of total initiated TCP connections.

1694	   c.  During the sustain phase, traffic SHOULD be forwarded at a
1695	       constant rate (considered as a constant rate if any deviation of
1696	       traffic forwarding rate is less than 5%).

1698	7.5.3.4.  Measurement

1700	   Average Concurrent TCP Connections MUST be reported for this
1701	   benchmarking test.

1703	7.5.4.  Test Procedures and Expected Results

1705	   The test procedure is designed to measure the concurrent TCP
1706	   connection capacity of the DUT/SUT at the sustaining period of
1707	   traffic load profile.  The test procedure consists of three major
1708	   steps: Step 1 ensures the DUT/SUT is able to reach the performance
1709	   value (Initial concurrent connection) and meets the test results
1710	   validation criteria when it was very minimally utilized.  Step 2
1711	   determines the DUT/SUT is able to reach the target performance value
1712	   within the test results validation criteria.  Step 3 determines the
1713	   maximum achievable performance value within the test results
1714	   validation criteria.

1716	   This test procedure MAY be repeated multiple times with different
1717	   IPv4 and IPv6 traffic distribution.

1719	7.5.4.1.  Step 1: Test Initialization and Qualification

1721	   Verify the link status of all connected physical interfaces.  All
1722	   interfaces are expected to be in "UP" status.

1724	   Configure test equipment to establish "Initial concurrent TCP
1725	   connections" defined in Section 7.5.3.2.  Except ramp up time, the
1726	   traffic load profile SHOULD be defined as described in Section 4.3.4.

1728	   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
1729	   concurrent TCP connections".  The measured KPIs during the sustain
1730	   phase MUST meet all the test results validation criteria defined in
1731	   Section 7.5.3.3.

1733	   If the KPI metrics do not meet the test results validation criteria,
1734	   the test procedure MUST NOT be continued to "Step 2".

1736	7.5.4.2.  Step 2: Test Run with Target Objective

1738	   Configure test equipment to establish the target objective ("Target
1739	   concurrent TCP connections").  The test equipment SHOULD follow the
1740	   traffic load profile definition (except ramp up time) as described in
1741	   Section 4.3.4.

1743	   During the ramp up and sustain phase, the other KPIs such as
1744	   inspected throughput, TCP connections per second, and application
1745	   transactions per second MUST NOT reach the maximum value the DUT/SUT
1746	   can support.

1748	   The test equipment SHOULD start to measure and record KPIs defined in
1749	   Section 7.5.3.4.  Continue the test until all traffic profile phases
1750	   are completed.

1752	   Within the test results validation criteria, the DUT/SUT is expected
1753	   to reach the desired value of the target objective in the sustain
1754	   phase.  Follow step 3, if the measured value does not meet the target
1755	   value or does not fulfill the test results validation criteria.

1757	7.5.4.3.  Step 3: Test Iteration

1759	   Determine the achievable concurrent TCP connections capacity within
1760	   the test results validation criteria.

1762	7.6.  TCP/HTTPS Connections per Second

[minor] The one big performance factor that i think is not documented or suggested
to be compared is the cost of certificate (chain) validation for different key-length
certificates used for the TCP/HTTPs connections. The parameters for TLS 1.2 and TLS 1.3
mentioned in before in the document do not cover that.  I think it would be prudent
to figure out an Internet common minimum (fastest to process) certificate and a 
common maximum complexity certificate. The latter one may simply be when revocation
is enabled, e.g.: checking the server certificate against a revocation list.

Just saying because server certificate verification may monopolise connection setup
performance - unless you want to make the argument that it is irrelevant because
due to the limited number of servers in the test, the DUT is assumed/known to be able
to cache server certificate validation results during ramput phase so it does become
irrelevant during steady state phase. But it would be at least good to describe this in text.

1763	7.6.1.  Objective

1765	   Using HTTPS traffic, determine the sustainable SSL/TLS session
1766	   establishment rate supported by the DUT/SUT under different
1767	   throughput load conditions.

1769	   Test iterations MUST include common cipher suites and key strengths
1770	   as well as forward looking stronger keys.  Specific test iterations
1771	   MUST include ciphers and keys defined in Section 7.6.3.2.

1773	   For each cipher suite and key strengths, test iterations MUST use a
1774	   single HTTPS response object size defined in Section 7.6.3.2 to
1775	   measure connections per second performance under a variety of DUT/SUT
1776	   security inspection load conditions.

1778	7.6.2.  Test Setup

1780	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1781	   specific testbed configuration changes (number of interfaces and
1782	   interface type, etc.)  MUST be documented.

1784	7.6.3.  Test Parameters

1786	   In this section, benchmarking test specific parameters SHOULD be
1787	   defined.

1789	7.6.3.1.  DUT/SUT Configuration Parameters

1791	   DUT/SUT parameters MUST conform to the requirements defined in
1792	   Section 4.2.  Any configuration changes for this specific
1793	   benchmarking test MUST be documented.

1795	7.6.3.2.  Test Equipment Configuration Parameters

1797	   Test equipment configuration parameters MUST conform to the
1798	   requirements defined in Section 4.3.  The following parameters MUST
1799	   be documented for this benchmarking test:

1801	   Client IP address range defined in Section 4.3.1.2

1803	   Server IP address range defined in Section 4.3.2.2

1805	   Traffic distribution ratio between IPv4 and IPv6 defined in
1806	   Section 4.3.1.2

1808	   Target connections per second: Initial value from product datasheet
1809	   or the value defined based on requirement for a specific deployment
1810	   scenario.

1812	   Initial connections per second: 10% of "Target connections per
1813	   second" Note: Initial connections per second is not a KPI to report.
1814	   This value is configured on the traffic generator and used to perform
1815	   the Step1: "Test Initialization and Qualification" described under
1816	   the Section 7.6.4.

1818	   RECOMMENDED ciphers and keys defined in Section 4.3.1.3

1820	   The client MUST negotiate HTTPS and close the connection with FIN
1821	   immediately after completion of one transaction.  In each test
1822	   iteration, client MUST send GET request requesting a fixed HTTPS
1823	   response object size.  The RECOMMENDED object sizes are 1, 2, 4, 16,
1824	   and 64 KByte.

1826	7.6.3.3.  Test Results Validation Criteria

1828	   The following criteria are the test results validation criteria.  The
1829	   test results validation criteria MUST be monitored during the whole
1830	   test duration.

1832	   a.  Number of failed application transactions (receiving any HTTP
1833	       response code other than 200 OK) MUST be less than 0.001% (1 out
1834	       of 100,000 transactions) of attempt transactions.

1836	   b.  Number of terminated TCP connections due to unexpected TCP RST
1837	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
1838	       connections) of total initiated TCP connections.

1840	   c.  During the sustain phase, traffic SHOULD be forwarded at a
1841	       constant rate (considered as a constant rate if any deviation of
1842	       traffic forwarding rate is less than 5%).

1844	   d.  Concurrent TCP connections MUST be constant during steady state
1845	       and any deviation of concurrent TCP connections SHOULD be less
1846	       than 10%. This confirms the DUT opens and closes TCP connections
1847	       at approximately the same rate.

1849	7.6.3.4.  Measurement

1851	   TCP connections per second MUST be reported for each test iteration
1852	   (for each object size).

1854	   The KPI metric TLS Handshake Rate can be measured in the test using 1
1855	   KByte object size.

1857	7.6.4.  Test Procedures and Expected Results

1859	   The test procedure is designed to measure the TCP connections per
1860	   second rate of the DUT/SUT at the sustaining period of traffic load
1861	   profile.  The test procedure consists of three major steps: Step 1
1862	   ensures the DUT/SUT is able to reach the performance value (Initial
1863	   connections per second) and meets the test results validation
1864	   criteria when it was very minimally utilized.  Step 2 determines the
1865	   DUT/SUT is able to reach the target performance value within the test
1866	   results validation criteria.  Step 3 determines the maximum
1867	   achievable performance value within the test results validation
1868	   criteria.

1870	   This test procedure MAY be repeated multiple times with different
1871	   IPv4 and IPv6 traffic distribution.

1873	7.6.4.1.  Step 1: Test Initialization and Qualification

1875	   Verify the link status of all connected physical interfaces.  All
1876	   interfaces are expected to be in "UP" status.

1878	   Configure traffic load profile of the test equipment to establish
1879	   "Initial connections per second" as defined in Section 7.6.3.2.  The
1880	   traffic load profile SHOULD be defined as described in Section 4.3.4.

1882	   The DUT/SUT SHOULD reach the "Initial connections per second" before
1883	   the sustain phase.  The measured KPIs during the sustain phase MUST
1884	   meet all the test results validation criteria defined in
1885	   Section 7.6.3.3.

1887	   If the KPI metrics do not meet the test results validation criteria,
1888	   the test procedure MUST NOT be continued to "Step 2".

1890	7.6.4.2.  Step 2: Test Run with Target Objective

1892	   Configure test equipment to establish "Target connections per second"
1893	   defined in Section 7.6.3.2.  The test equipment SHOULD follow the
1894	   traffic load profile definition as described in Section 4.3.4.

1896	   During the ramp up and sustain phase, other KPIs such as inspected
1897	   throughput, concurrent TCP connections, and application transactions
1898	   per second MUST NOT reach the maximum value the DUT/SUT can support.
1899	   The test results for specific test iteration SHOULD NOT be reported,
1900	   if the above mentioned KPI (especially inspected throughput) reaches
1901	   the maximum value.  (Example: If the test iteration with 64 KByte of
1902	   HTTPS response object size reached the maximum inspected throughput
1903	   limitation of the DUT, the test iteration MAY be interrupted and the
1904	   result for 64 KByte SHOULD NOT be reported).

1906	   The test equipment SHOULD start to measure and record all specified
1907	   KPIs.  Continue the test until all traffic profile phases are
1908	   completed.

1910	   Within the test results validation criteria, the DUT/SUT is expected
1911	   to reach the desired value of the target objective ("Target
1912	   connections per second") in the sustain phase.  Follow step 3, if the
1913	   measured value does not meet the target value or does not fulfill the
1914	   test results validation criteria.

1916	7.6.4.3.  Step 3: Test Iteration

1918	   Determine the achievable connections per second within the test
1919	   results validation criteria.

1921	7.7.  HTTPS Throughput

1923	7.7.1.  Objective

1925	   Determine the sustainable inspected throughput of the DUT/SUT for
1926	   HTTPS transactions varying the HTTPS response object size.

1928	   Test iterations MUST include common cipher suites and key strengths
1929	   as well as forward looking stronger keys.  Specific test iterations
1930	   MUST include the ciphers and keys defined in Section 7.7.3.2.

1932	7.7.2.  Test Setup

1934	   Testbed setup SHOULD be configured as defined in Section 4.  Any
1935	   specific testbed configuration changes (number of interfaces and
1936	   interface type, etc.)  MUST be documented.

1938	7.7.3.  Test Parameters

1940	   In this section, benchmarking test specific parameters SHOULD be
1941	   defined.

1943	7.7.3.1.  DUT/SUT Configuration Parameters

1945	   DUT/SUT parameters MUST conform to the requirements defined in
1946	   Section 4.2.  Any configuration changes for this specific
1947	   benchmarking test MUST be documented.

1949	7.7.3.2.  Test Equipment Configuration Parameters

1951	   Test equipment configuration parameters MUST conform to the
1952	   requirements defined in Section 4.3.  The following parameters MUST
1953	   be documented for this benchmarking test:

1955	   Client IP address range defined in Section 4.3.1.2

1957	   Server IP address range defined in Section 4.3.2.2

1959	   Traffic distribution ratio between IPv4 and IPv6 defined in
1960	   Section 4.3.1.2

1962	   Target inspected throughput: Aggregated line rate of interface(s)
1963	   used in the DUT/SUT or the value defined based on requirement for a
1964	   specific deployment scenario.

1966	   Initial throughput: 10% of "Target inspected throughput" Note:
1967	   Initial throughput is not a KPI to report.  This value is configured
1968	   on the traffic generator and used to perform the Step1: "Test
1969	   Initialization and Qualification" described under the Section 7.7.4.

1971	   Number of HTTPS response object requests (transactions) per
1972	   connection: 10

1974	   RECOMMENDED ciphers and keys defined in Section 4.3.1.3

1976	   RECOMMENDED HTTPS response object size: 1, 16, 64, 256 KByte, and
1977	   mixed objects defined in Table 4 under Section 7.3.3.2.

1979	7.7.3.3.  Test Results Validation Criteria

1981	   The following criteria are the test results validation criteria.  The
1982	   test results validation criteria MUST be monitored during the whole
1983	   sustain phase of the traffic load profile.

1985	   a.  Number of failed Application transactions (receiving any HTTP
1986	       response code other than 200 OK) MUST be less than 0.001% (1 out
1987	       of 100,000 transactions) of attempt transactions.

1989	   b.  Traffic SHOULD be forwarded at a constant rate (considered as a
1990	       constant rate if any deviation of traffic forwarding rate is less
1991	       than 5%).

1993	   c.  Concurrent TCP connections MUST be constant during steady state
1994	       and any deviation of concurrent TCP connections SHOULD be less
1995	       than 10%. This confirms the DUT opens and closes TCP connections
1996	       at approximately the same rate.

1998	7.7.3.4.  Measurement

2000	   Inspected Throughput and HTTP Transactions per Second MUST be
2001	   reported for each object size.

2003	7.7.4.  Test Procedures and Expected Results

2005	   The test procedure consists of three major steps: Step 1 ensures the
2006	   DUT/SUT is able to reach the performance value (Initial throughput)
2007	   and meets the test results validation criteria when it was very
2008	   minimally utilized.  Step 2 determines the DUT/SUT is able to reach
2009	   the target performance value within the test results validation
2010	   criteria.  Step 3 determines the maximum achievable performance value
2011	   within the test results validation criteria.

2013	   This test procedure MAY be repeated multiple times with different
2014	   IPv4 and IPv6 traffic distribution and HTTPS response object sizes.

2016	7.7.4.1.  Step 1: Test Initialization and Qualification

2018	   Verify the link status of all connected physical interfaces.  All
2019	   interfaces are expected to be in "UP" status.

2021	   Configure traffic load profile of the test equipment to establish
2022	   "Initial throughput" as defined in Section 7.7.3.2.

2024	   The traffic load profile SHOULD be defined as described in
2025	   Section 4.3.4.  The DUT/SUT SHOULD reach the "Initial throughput"
2026	   during the sustain phase.  Measure all KPI as defined in
2027	   Section 7.7.3.4.

2029	   The measured KPIs during the sustain phase MUST meet the test results
2030	   validation criteria "a" defined in Section 7.7.3.3.  The test results
2031	   validation criteria "b" and "c" are OPTIONAL for step 1.

2033	   If the KPI metrics do not meet the test results validation criteria,
2034	   the test procedure MUST NOT be continued to "Step 2".

2036	7.7.4.2.  Step 2: Test Run with Target Objective

2038	   Configure test equipment to establish the target objective ("Target
2039	   inspected throughput") defined in Section 7.7.3.2.  The test
2040	   equipment SHOULD start to measure and record all specified KPIs.
2041	   Continue the test until all traffic profile phases are completed.

2043	   Within the test results validation criteria, the DUT/SUT is expected
2044	   to reach the desired value of the target objective in the sustain
2045	   phase.  Follow step 3, if the measured value does not meet the target
2046	   value or does not fulfill the test results validation criteria.

2048	7.7.4.3.  Step 3: Test Iteration

2050	   Determine the achievable average inspected throughput within the test
2051	   results validation criteria.  Final test iteration MUST be performed
2052	   for the test duration defined in Section 4.3.4.

2054	7.8.  HTTPS Transaction Latency

2056	7.8.1.  Objective

2058	   Using HTTPS traffic, determine the HTTPS transaction latency when
2059	   DUT/SUT is running with sustainable HTTPS transactions per second
2060	   supported by the DUT/SUT under different HTTPS response object size.

2062	   Scenario 1: The client MUST negotiate HTTPS and close the connection
2063	   with FIN immediately after completion of a single transaction (GET
2064	   and RESPONSE).

2066	   Scenario 2: The client MUST negotiate HTTPS and close the connection
2067	   with FIN immediately after completion of 10 transactions (GET and
2068	   RESPONSE) within a single TCP connection.

2070	7.8.2.  Test Setup

2072	   Testbed setup SHOULD be configured as defined in Section 4.  Any
2073	   specific testbed configuration changes (number of interfaces and
2074	   interface type, etc.)  MUST be documented.

2076	7.8.3.  Test Parameters

2078	   In this section, benchmarking test specific parameters SHOULD be
2079	   defined.

2081	7.8.3.1.  DUT/SUT Configuration Parameters

2083	   DUT/SUT parameters MUST conform to the requirements defined in
2084	   Section 4.2.  Any configuration changes for this specific
2085	   benchmarking test MUST be documented.

2087	7.8.3.2.  Test Equipment Configuration Parameters

2089	   Test equipment configuration parameters MUST conform to the
2090	   requirements defined in Section 4.3.  The following parameters MUST
2091	   be documented for this benchmarking test:

2093	   Client IP address range defined in Section 4.3.1.2

2095	   Server IP address range defined in Section 4.3.2.2
2096	   Traffic distribution ratio between IPv4 and IPv6 defined in
2097	   Section 4.3.1.2

2099	   RECOMMENDED cipher suites and key sizes defined in Section 4.3.1.3

2101	   Target objective for scenario 1: 50% of the connections per second
2102	   measured in benchmarking test TCP/HTTPS Connections per second
2103	   (Section 7.6)

2105	   Target objective for scenario 2: 50% of the inspected throughput
2106	   measured in benchmarking test HTTPS Throughput (Section 7.7)

2108	   Initial objective for scenario 1: 10% of "Target objective for
2109	   scenario 1"

2111	   Initial objective for scenario 2: 10% of "Target objective for
2112	   scenario 2"

2114	   Note: The Initial objectives are not a KPI to report.  These values
2115	   are configured on the traffic generator and used to perform the
2116	   Step1: "Test Initialization and Qualification" described under the
2117	   Section 7.8.4.

2119	   HTTPS transaction per TCP connection: Test scenario 1 with single
2120	   transaction and scenario 2 with 10 transactions

2122	   HTTPS with GET request requesting a single object.  The RECOMMENDED
2123	   object sizes are 1, 16, and 64 KByte.  For each test iteration,
2124	   client MUST request a single HTTPS response object size.

2126	7.8.3.3.  Test Results Validation Criteria

2128	   The following criteria are the test results validation criteria.  The
2129	   Test results validation criteria MUST be monitored during the whole
2130	   sustain phase of the traffic load profile.

2132	   a.  Number of failed application transactions (receiving any HTTP
2133	       response code other than 200 OK) MUST be less than 0.001% (1 out
2134	       of 100,000 transactions) of attempt transactions.

2136	   b.  Number of terminated TCP connections due to unexpected TCP RST
2137	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
2138	       connections) of total initiated TCP connections.

2140	   c.  During the sustain phase, traffic SHOULD be forwarded at a
2141	       constant rate (considered as a constant rate if any deviation of
2142	       traffic forwarding rate is less than 5%).

2144	   d.  Concurrent TCP connections MUST be constant during steady state
2145	       and any deviation of concurrent TCP connections SHOULD be less
2146	       than 10%. This confirms the DUT opens and closes TCP connections
2147	       at approximately the same rate.

2149	   e.  After ramp up the DUT/SUT MUST achieve the "Target objective"
2150	       defined in the parameter Section 7.8.3.2 and remain in that state
2151	       for the entire test duration (sustain phase).

2153	7.8.3.4.  Measurement

2155	   TTFB (minimum, average, and maximum) and TTLB (minimum, average and
2156	   maximum) MUST be reported for each object size.

2158	7.8.4.  Test Procedures and Expected Results

2160	   The test procedure is designed to measure TTFB or TTLB when the DUT/
2161	   SUT is operating close to 50% of its maximum achievable connections
2162	   per second or inspected throughput.  The test procedure consists of
2163	   two major steps: Step 1 ensures the DUT/SUT is able to reach the
2164	   initial performance values and meets the test results validation
2165	   criteria when it was very minimally utilized.  Step 2 measures the
2166	   latency values within the test results validation criteria.

2168	   This test procedure MAY be repeated multiple times with different IP
2169	   types (IPv4 only, IPv6 only and IPv4 and IPv6 mixed traffic
2170	   distribution), HTTPS response object sizes and single, and multiple
2171	   transactions per connection scenarios.

2173	7.8.4.1.  Step 1: Test Initialization and Qualification

2175	   Verify the link status of all connected physical interfaces.  All
2176	   interfaces are expected to be in "UP" status.

2178	   Configure traffic load profile of the test equipment to establish
2179	   "Initial objective" as defined in the Section 7.8.3.2.  The traffic
2180	   load profile SHOULD be defined as described in Section 4.3.4.

2182	   The DUT/SUT SHOULD reach the "Initial objective" before the sustain
2183	   phase.  The measured KPIs during the sustain phase MUST meet all the
2184	   test results validation criteria defined in Section 7.8.3.3.

2186	   If the KPI metrics do not meet the test results validation criteria,
2187	   the test procedure MUST NOT be continued to "Step 2".

2189	7.8.4.2.  Step 2: Test Run with Target Objective

2191	   Configure test equipment to establish "Target objective" defined in
2192	   Section 7.8.3.2.  The test equipment SHOULD follow the traffic load
2193	   profile definition as described in Section 4.3.4.

2195	   The test equipment SHOULD start to measure and record all specified
2196	   KPIs.  Continue the test until all traffic profile phases are
2197	   completed.

2199	   Within the test results validation criteria, the DUT/SUT MUST reach
2200	   the desired value of the target objective in the sustain phase.

2202	   Measure the minimum, average, and maximum values of TTFB and TTLB.

2204	7.9.  Concurrent TCP/HTTPS Connection Capacity

2206	7.9.1.  Objective

2208	   Determine the number of concurrent TCP connections the DUT/SUT
2209	   sustains when using HTTPS traffic.

2211	7.9.2.  Test Setup

2213	   Testbed setup SHOULD be configured as defined in Section 4.  Any
2214	   specific testbed configuration changes (number of interfaces and
2215	   interface type, etc.)  MUST be documented.

2217	7.9.3.  Test Parameters

2219	   In this section, benchmarking test specific parameters SHOULD be
2220	   defined.

2222	7.9.3.1.  DUT/SUT Configuration Parameters

2224	   DUT/SUT parameters MUST conform to the requirements defined in
2225	   Section 4.2.  Any configuration changes for this specific
2226	   benchmarking test MUST be documented.

2228	7.9.3.2.  Test Equipment Configuration Parameters

2230	   Test equipment configuration parameters MUST conform to the
2231	   requirements defined in Section 4.3.  The following parameters MUST
2232	   be documented for this benchmarking test:

2234	      Client IP address range defined in Section 4.3.1.2

2236	      Server IP address range defined in Section 4.3.2.2
2237	      Traffic distribution ratio between IPv4 and IPv6 defined in
2238	      Section 4.3.1.2

2240	      RECOMMENDED cipher suites and key sizes defined in Section 4.3.1.3

2242	      Target concurrent connections: Initial value from product
2243	      datasheet or the value defined based on requirement for a specific
2244	      deployment scenario.

2246	      Initial concurrent connections: 10% of "Target concurrent
2247	      connections" Note: Initial concurrent connection is not a KPI to
2248	      report.  This value is configured on the traffic generator and
2249	      used to perform the Step1: "Test Initialization and Qualification"
2250	      described under the Section 7.9.4.

2252	      Connections per second during ramp up phase: 50% of maximum
2253	      connections per second measured in benchmarking test TCP/HTTPS
2254	      Connections per second (Section 7.6)

2256	      Ramp up time (in traffic load profile for "Target concurrent
2257	      connections"): "Target concurrent connections" / "Maximum
2258	      connections per second during ramp up phase"

2260	      Ramp up time (in traffic load profile for "Initial concurrent
2261	      connections"): "Initial concurrent connections" / "Maximum
2262	      connections per second during ramp up phase"

2264	   The client MUST perform HTTPS transaction with persistence and each
2265	   client can open multiple concurrent TCP connections per server
2266	   endpoint IP.

2268	   Each client sends 10 GET requests requesting 1 KByte HTTPS response
2269	   objects in the same TCP connections (10 transactions/TCP connection)
2270	   and the delay (think time) between each transaction MUST be X
2271	   seconds.

2273	   X = ("Ramp up time" + "steady state time") /10

2275	   The established connections SHOULD remain open until the ramp down
2276	   phase of the test.  During the ramp down phase, all connections
2277	   SHOULD be successfully closed with FIN.

2279	7.9.3.3.  Test Results Validation Criteria

2281	   The following criteria are the test results validation criteria.  The
2282	   Test results validation criteria MUST be monitored during the whole
2283	   sustain phase of the traffic load profile.

2285	   a.  Number of failed application transactions (receiving any HTTP
2286	       response code other than 200 OK) MUST be less than 0.001% (1 out
2287	       of 100,000 transactions) of total attempted transactions.

2289	   b.  Number of terminated TCP connections due to unexpected TCP RST
2290	       sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
2291	       connections) of total initiated TCP connections.

2293	   c.  During the sustain phase, traffic SHOULD be forwarded at a
2294	       constant rate (considered as a constant rate if any deviation of
2295	       traffic forwarding rate is less than 5%).

2297	7.9.3.4.  Measurement

2299	   Average Concurrent TCP Connections MUST be reported for this
2300	   benchmarking test.

2302	7.9.4.  Test Procedures and Expected Results

2304	   The test procedure is designed to measure the concurrent TCP
2305	   connection capacity of the DUT/SUT at the sustaining period of
2306	   traffic load profile.  The test procedure consists of three major
2307	   steps: Step 1 ensures the DUT/SUT is able to reach the performance
2308	   value (Initial concurrent connection) and meets the test results
2309	   validation criteria when it was very minimally utilized.  Step 2
2310	   determines the DUT/SUT is able to reach the target performance value
2311	   within the test results validation criteria.  Step 3 determines the
2312	   maximum achievable performance value within the test results
2313	   validation criteria.

2315	   This test procedure MAY be repeated multiple times with different
2316	   IPv4 and IPv6 traffic distribution.

2318	7.9.4.1.  Step 1: Test Initialization and Qualification

2320	   Verify the link status of all connected physical interfaces.  All
2321	   interfaces are expected to be in "UP" status.

2323	   Configure test equipment to establish "Initial concurrent TCP
2324	   connections" defined in Section 7.9.3.2.  Except ramp up time, the
2325	   traffic load profile SHOULD be defined as described in Section 4.3.4.

2327	   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
2328	   concurrent TCP connections".  The measured KPIs during the sustain
2329	   phase MUST meet the test results validation criteria "a" and "b"
2330	   defined in Section 7.9.3.3.

2332	   If the KPI metrics do not meet the test results validation criteria,
2333	   the test procedure MUST NOT be continued to "Step 2".

2335	7.9.4.2.  Step 2: Test Run with Target Objective

2337	   Configure test equipment to establish the target objective ("Target
2338	   concurrent TCP connections").  The test equipment SHOULD follow the
2339	   traffic load profile definition (except ramp up time) as described in
2340	   Section 4.3.4.

2342	   During the ramp up and sustain phase, the other KPIs such as
2343	   inspected throughput, TCP connections per second, and application
2344	   transactions per second MUST NOT reach to the maximum value that the
2345	   DUT/SUT can support.

2347	   The test equipment SHOULD start to measure and record KPIs defined in
2348	   Section 7.9.3.4.  Continue the test until all traffic profile phases
2349	   are completed.

2351	   Within the test results validation criteria, the DUT/SUT is expected
2352	   to reach the desired value of the target objective in the sustain
2353	   phase.  Follow step 3, if the measured value does not meet the target
2354	   value or does not fulfill the test results validation criteria.

2356	7.9.4.3.  Step 3: Test Iteration

2358	   Determine the achievable concurrent TCP connections within the test
2359	   results validation criteria.

[mayor] I would really love to see DUT power consumption numbers captured and reported
for the 10% and the maximum achieved rates for the 7.x tests (during steady state).

Energy consumption is becoming a more and more important factor in networking, and all the
high-touch operations of security devices are amongst the most power/compute hungry operations
of any network device, but with a wide variety depending on how its implemented.
Its also extremely simple to just plug a power-meter into the supply line of the DUT.

This would encourage DUT vendors to reduce power consumption, something that often
can be achieved by just selecting appropriate components (lowest power CPU options, going
FPGA etc.. routes).

Personally, i am of course also interested in easily derived performance factors such as
comparing 100% power consumption for the HTTP vs. HTTPS case - cost of end-to-end security
that is. If a DUT just shows linerate for both HTTP and HTTPS, but with double the
power consumption when using HTTPs, that may even impact deployment - even in small cases
with a small 19" rack, some ventilation and some amount of power - 100..500W makes a difference
whethre its 100 or 500W.

2361	8.  IANA Considerations

2363	   This document makes no specific request of IANA.

2365	   The IANA has assigned IPv4 and IPv6 address blocks in [RFC6890] that
2366	   have been registered for special purposes.  The IPv6 address block
2367	   2001:2::/48 has been allocated for the purpose of IPv6 Benchmarking
2368	   [RFC5180] and the IPv4 address block 198.18.0.0/15 has been allocated
2369	   for the purpose of IPv4 Benchmarking [RFC2544].  This assignment was
2370	   made to minimize the chance of conflict in case a testing device were
2371	   to be accidentally connected to part of the Internet.

[minor] I don't think the secnd paragraph belongs into an IANA considerations
section. This section is usually resesrved only for actions IANA is supposed to
take for this document. I would suggest to move this paragraph to an earlier
section, maybe even simply make one up "Addressing for tests".

2373	9.  Security Considerations

2375	   The primary goal of this document is to provide benchmarking
2376	   terminology and methodology for next-generation network security
2377	   devices for use in a laboratory isolated test environment.  However,
2378	   readers should be aware that there is some overlap between
2379	   performance and security issues.  Specifically, the optimal
2380	   configuration for network security device performance may not be the
2381	   most secure, and vice-versa.  The cipher suites recommended in this
2382	   document are for test purpose only.  The cipher suite recommendation
2383	   for a real deployment is outside the scope of this document.

2385	10.  Contributors

2387	   The following individuals contributed significantly to the creation
2388	   of this document:

2390	   Alex Samonte, Amritam Putatunda, Aria Eslambolchizadeh, Chao Guo,
2391	   Chris Brown, Cory Ford, David DeSanto, Jurrie Van Den Breekel,
2392	   Michelle Rhines, Mike Jack, Ryan Liles, Samaresh Nair, Stephen
2393	   Goudreault, Tim Carlin, and Tim Otto.

2395	11.  Acknowledgements

2397	   The authors wish to acknowledge the members of NetSecOPEN for their
2398	   participation in the creation of this document.  Additionally, the
2399	   following members need to be acknowledged:

2401	   Anand Vijayan, Chris Marshall, Jay Lindenauer, Michael Shannon, Mike
2402	   Deichman, Ryan Riese, and Toulnay Orkun.

2404	12.  References

2406	12.1.  Normative References

2408	   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
2409	              Requirement Levels", BCP 14, RFC 2119,
2410	              DOI 10.17487/RFC2119, March 1997,
2411	              <https://www.rfc-editor.org/info/rfc2119>.

2413	   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2414	              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
2415	              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

2417	12.2.  Informative References

2419	   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
2420	              Network Interconnect Devices", RFC 2544,
2421	              DOI 10.17487/RFC2544, March 1999,
2422	              <https://www.rfc-editor.org/info/rfc2544>.

2424	   [RFC2647]  Newman, D., "Benchmarking Terminology for Firewall
2425	              Performance", RFC 2647, DOI 10.17487/RFC2647, August 1999,
2426	              <https://www.rfc-editor.org/info/rfc2647>.

2428	   [RFC3511]  Hickman, B., Newman, D., Tadjudin, S., and T. Martin,
2429	              "Benchmarking Methodology for Firewall Performance",
2430	              RFC 3511, DOI 10.17487/RFC3511, April 2003,
2431	              <https://www.rfc-editor.org/info/rfc3511>.

2433	   [RFC5180]  Popoviciu, C., Hamza, A., Van de Velde, G., and D.
2434	              Dugatkin, "IPv6 Benchmarking Methodology for Network
2435	              Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May
2436	              2008, <https://www.rfc-editor.org/info/rfc5180>.

2438	   [RFC6815]  Bradner, S., Dubray, K., McQuaid, J., and A. Morton,
2439	              "Applicability Statement for RFC 2544: Use on Production
2440	              Networks Considered Harmful", RFC 6815,
2441	              DOI 10.17487/RFC6815, November 2012,
2442	              <https://www.rfc-editor.org/info/rfc6815>.

2444	   [RFC6890]  Cotton, M., Vegoda, L., Bonica, R., Ed., and B. Haberman,
2445	              "Special-Purpose IP Address Registries", BCP 153,
2446	              RFC 6890, DOI 10.17487/RFC6890, April 2013,
2447	              <https://www.rfc-editor.org/info/rfc6890>.

2449	   [RFC7230]  Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
2450	              Protocol (HTTP/1.1): Message Syntax and Routing",
2451	              RFC 7230, DOI 10.17487/RFC7230, June 2014,
2452	              <https://www.rfc-editor.org/info/rfc7230>.

2454	   [RFC8446]  Rescorla, E., "The Transport Layer Security (TLS) Protocol
2455	              Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
2456	              <https://www.rfc-editor.org/info/rfc8446>.

2458	   [RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
2459	              Multiplexed and Secure Transport", RFC 9000,
2460	              DOI 10.17487/RFC9000, May 2021,
2461	              <https://www.rfc-editor.org/info/rfc9000>.

2463	Appendix A.  Test Methodology - Security Effectiveness Evaluation

[nit] /Evaluation/Test/ - called test in the rest of this doc.

2464	A.1.  Test Objective

2466	   This test methodology verifies the DUT/SUT is able to detect,

[nit] /verifies the/ verifies that the/

2467	   prevent, and report the vulnerabilities.

2469	   In this test, background test traffic will be generated to utilize
2470	   the DUT/SUT.  In parallel, the CVEs will be sent to the DUT/SUT as
2471	   encrypted and as well as clear text payload formats using a traffic
2472	   generator.  The selection of the CVEs is described in Section 4.2.1.

2474	   The following KPIs are measured in this test:

2476	   *  Number of blocked CVEs

2478	   *  Number of bypassed (nonblocked) CVEs

2480	   *  Background traffic performance (verify if the background traffic
2481	      is impacted while sending CVE toward DUT/SUT)

2483	   *  Accuracy of DUT/SUT statistics in term of vulnerabilities
2484	      reporting

2486	A.2.  Testbed Setup

2488	   The same testbed MUST be used for security effectiveness test and as
2489	   well as for benchmarking test cases defined in Section 7.

2491	A.3.  Test Parameters

2493	   In this section, the benchmarking test specific parameters SHOULD be
2494	   defined.

[nit] /SHOULD/are/ - a requirement against the authors of the document to write
desirable text in the document is not normative.

2496	A.3.1.  DUT/SUT Configuration Parameters

2498	   DUT/SUT configuration parameters MUST conform to the requirements
2499	   defined in Section 4.2.  The same DUT configuration MUST be used for
2500	   Security effectiveness test and as well as for benchmarking test
2501	   cases defined in Section 7.  The DUT/SUT MUST be configured in inline
2502	   mode and all detected attack traffic MUST be dropped and the session

[nit] /detected traffic/detected CVE traffic/ - there is also background traffic, which i guess shouldnot be dropped, right ?

[nit] /the session/its session/ ?

2503	   SHOULD be reset

2505	A.3.2.  Test Equipment Configuration Parameters

2507	   Test equipment configuration parameters MUST conform to the
2508	   requirements defined in Section 4.3.  The same client and server IP
2509	   ranges MUST be configured as used in the benchmarking test cases.  In
2510	   addition, the following parameters MUST be documented for this
2511	   benchmarking test:

2513	   *  Background Traffic: 45% of maximum HTTP throughput and 45% of
2514	      Maximum HTTPS throughput supported by the DUT/SUT (measured with
2515	      object size 64 KByte in the benchmarking tests "HTTP(S)
2516	      Throughput" defined in Section 7.3 and Section 7.7).

[nit] RECOMMENDED Background Traffic ?

2518	   *  RECOMMENDED CVE traffic transmission Rate: 10 CVEs per second

2520	   *  It is RECOMMENDED to generate each CVE multiple times
2521	      (sequentially) at 10 CVEs per second

2523	   *  Ciphers and keys for the encrypted CVE traffic MUST use the same
2524	      cipher configured for HTTPS traffic related benchmarking tests
2525	      (Section 7.6 - Section 7.9)

2527	A.4.  Test Results Validation Criteria

2529	   The following criteria are the test results validation criteria.  The
2530	   test results validation criteria MUST be monitored during the whole
2531	   test duration.

[nit] /criteria are/lists/ - duplication of criteria in sentence.

2533	   a.  Number of failed application transaction in the background
2534	       traffic MUST be less than 0.01% of attempted transactions.

2536	   b.  Number of terminated TCP connections of the background traffic
2537	       (due to unexpected TCP RST sent by DUT/SUT) MUST be less than
2538	       0.01% of total initiated TCP connections in the background
2539	       traffic.

[comment] That is quite high. Shouldn't this at least be 5 nines of
success ? 99.999% -> 0.001% maximum rate of errors ? I thought thats the common
lore service provider product quality requirement minimum.

2541	   c.  During the sustain phase, traffic SHOULD be forwarded at a
2542	       constant rate (considered as a constant rate if any deviation of
2543	       traffic forwarding rate is less than 5%).

[minor] This seems underspecified. I guess in the ideally behaving DUT case
all background traffic is passed unmodified and all CVE connection traffic is dropped.
So the total amount of traffic with CVE events must be configured to be less then
5% ?! What additional information would this 5% tell me that i do not already
get from a. and b. ? E.g.: if i fail some background connection, then the impact
depends on how big that connection would have been, but it doesn't seem as if
i get new information if a big NetFlix background flow got killed and therefore
5 Gigabyte less background traffic where observed, or if the same happened to
a 200KByte amazon shopping connection. It would just cause DUT to maybe do
less inspection on big flows in fear of triggering false resets on them ?? Is
that what we want from DUTs ?

2545	   d.  False positive MUST NOT occur in the background traffic.

[comment]  I do not understand d. When a background transaction from a. fails,
how is that different from false-positively being classified as a CVE - it would
be droppen then, right ? Or are you saying that a./b. is the case that the
background traffic receives errors from the DUT even though the DUT does NOT
recognize it as a CVE ?  Any example reason why that would happen ?

2547	A.5.  Measurement

2549	   Following KPI metrics MUST be reported for this test scenario:

2551	   Mandatory KPIs:

2553	   *  Blocked CVEs: It SHOULD be represented in the following ways:

2555	      -  Number of blocked CVEs out of total CVEs

2557	      -  Percentage of blocked CVEs

2559	   *  Unblocked CVEs: It SHOULD be represented in the following ways:

2561	      -  Number of unblocked CVEs out of total CVEs

2563	      -  Percentage of unblocked CVEs

2565	   *  Background traffic behavior: It SHOULD be represented one of the
2566	      followings ways:

2568	      -  No impact: Considered as "no impact'" if any deviation of
2569	         traffic forwarding rate is less than or equal to 5 % (constant
2570	         rate)

2572	      -  Minor impact: Considered as "minor impact" if any deviation of
2573	         traffic forwarding rate is greater than 5% and less than or
2574	         equal to10% (i.e. small spikes)

2576	      -  Heavily impacted: Considered as "Heavily impacted" if any
2577	         deviation of traffic forwarding rate is greater than 10% (i.e.
2578	         large spikes) or reduced the background HTTP(S) throughput
2579	         greater than 10%

[minor] I would prefer reporting the a./b. numbers, e.g.: percentage of
failed background connections. As mentioned before, i find the total background
traffic rate impact a rather problematic/less valuable metric.

2581	   *  DUT/SUT reporting accuracy: DUT/SUT MUST report all detected
2582	      vulnerabilities.

2584	   Optional KPIs:

2586	   *  List of unblocked CVEs

[minor] I think this KPI is a SHOULD or even MUST. Otherwise one can not trace
security impacts (when one does not know which CVE it is). This is still the
security effectiveness appendix, and reporting is not effective without this.

2588	A.6.  Test Procedures and Expected Results

2590	   The test procedure is designed to measure the security effectiveness
2591	   of the DUT/SUT at the sustaining period of the traffic load profile.
2592	   The test procedure consists of two major steps.  This test procedure
2593	   MAY be repeated multiple times with different IPv4 and IPv6 traffic
2594	   distribution.

2596	A.6.1.  Step 1: Background Traffic

2598	   Generate background traffic at the transmission rate defined in
2599	   Appendix A.3.2.

2601	   The DUT/SUT MUST reach the target objective (HTTP(S) throughput) in
2602	   sustain phase.  The measured KPIs during the sustain phase MUST meet
2603	   all the test results validation criteria defined in Appendix A.4.

2605	   If the KPI metrics do not meet the acceptance criteria, the test
2606	   procedure MUST NOT be continued to "Step 2".

2608	A.6.2.  Step 2: CVE Emulation

2610	   While generating background traffic (in sustain phase), send the CVE
2611	   traffic as defined in the parameter section.

2613	   The test equipment SHOULD start to measure and record all specified
2614	   KPIs.  Continue the test until all CVEs are sent.

2616	   The measured KPIs MUST meet all the test results validation criteria
2617	   defined in Appendix A.4.

2619	   In addition, the DUT/SUT SHOULD report the vulnerabilities correctly.

2621	Appendix B.  DUT/SUT Classification

2623	   This document aims to classify the DUT/SUT in four different
2624	   categories based on its maximum supported firewall throughput
2625	   performance number defined in the vendor datasheet.  This
2626	   classification MAY help user to determine specific configuration
2627	   scale (e.g., number of ACL entries), traffic profiles, and attack
2628	   traffic profiles, scaling those proportionally to DUT/SUT sizing
2629	   category.

2631	   The four different categories are Extra Small (XS), Small (S), Medium
2632	   (M), and Large (L).  The RECOMMENDED throughput values for the
2633	   following categories are:

2635	   Extra Small (XS) - Supported throughput less than or equal to1Gbit/s

2637	   Small (S) - Supported throughput greater than 1Gbit/s and less than
2638	   or equal to 5Gbit/s

2640	   Medium (M) - Supported throughput greater than 5Gbit/s and less than
2641	   or equal to10Gbit/s

2643	   Large (L) - Supported throughput greater than 10Gbit/s

2645	Authors' Addresses

2647	   Balamuhunthan Balarajah
2648	   Berlin
2649	   Germany

2651	   Email: [email-masked]
2652	   Carsten Rossenhoevel
2653	   EANTC AG
2654	   Salzufer 14
2655	   10587 Berlin
2656	   Germany

2658	   Email: [email-masked]

2660	   Brian Monkman
2661	   NetSecOPEN
2662	   417 Independence Court
2663	   Mechanicsburg, PA 17050
2664	   United States of America

2666	   Email: [email-masked]

EOF

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 31 Jan 2022 10:40:47 +0100

On Mon, Jan 31, 2022 at 08:20:00AM +0100, Carsten Rossenhoevel wrote:
>  * NGFW and UTM are fixed terms in the industry. 

The terminology is fine. I just felt it would be fairly easy to somewhat
rewrite the intro such that it correctly introduces those terms and how
they relate to each other instead of expecting them to be known. Wrt to
your argument that everybodt knows these terms: Just mentor a few young folks
coming to IETF primarily to learn, and then revisit whether your statement
holds true to the target readers you want to address as opposed to the
expert you work with ;-)

>  * We have had extensive discussions of goodput, throughput, and other
>    terms describing how many packets of which type make it from source
>    to destination.

A summary of what was changed in scope and terminology from RFC3511.
Such a change summary section is also quite common (and i'd say required)
for RFCs that are meant to obsolete another draft. Just write there
that goodput has fallen out of fashion or the like.

Technically, my main concern is just badly working TCP stacks in the
NGFW that create unnecessary much retransmission in the face of bad
paths to either client or server, but let me check if/what standard
TCP measurements are recommended for that case and get back to you.

>  * Adding a terminology section would not have increased readability
>    for the target audience.
>  * This document cannot and does not aim to introduce the whole world
>    of network security terms.
>  * This is just a  benchmarking methodology document.  We have to assume
>    that readers are network security professionals understanding the basics
>    of today's network security functions.

Is there any other IETF RFC that introduces/explain the network security
device terminology/architecture ? If not RFC then any other reference
you could add ?

>    I am not sure how to respond to some of your questions (e.g.
>    about making SSL inspection mandatory

I did not ask for that. I just wondered about the text "MUST explain the implication".

>    , or recommending to implement Web Filtering across the
>    industry [row 294]).

Row 294 comments where not about web filtering.
I was asking for row294 whether my understanding of your term
"Certificate Validation" is correct, because the way i understand it,
it could never be optional for a security device: The NGFW must 
verify the server certificate if it is putting itslf in the middle of
a TLS session client/server.

>    DDoS is not a firewall feature (it is an attack type), so it is not
>    listed as a security feature in table 3.

DDoS is listed in Table 1 which is titled "NGFW Security Features".
Maybe you want to rename that table 1 line enty to something like "DDoS protection".
But in any case there seem to be some NGFW feature related to DDoS,
and i jut wonder if/how that feature is reflected in table 3.

>    DPI is an obsolete term.

Why then is it listed in Table 2 ? Or do you consider the written out term
"Deep Packet Inspection" (as in Table 2) to be something different than DPI ?
oI was just doing consistency checking. Table 1 and Table 2 claim to be
NGFW/NGIPS Security Features, and Table 3 claims to be "Security Feature Description",
so i was just trying to check completeness of Table 3.

>    Any NGFW does some type of deep-packet
>    inspection.  More detailed terms in table 3 describe the different
>    aspects of packet inspections.

It would be easier to read if the terms used across Tables 1,2,3 where consistent.
Maybe just put only those terms from Table 3 into Table 3 that apply to Table 2 ?

>  *   The time of ACL-based routers acting as firewalls is over.
>    Managing thousands of ACL entries is difficult and often bears a
>    security risk in itself.  It is not a typical application scenario
>    today.  Security is achieved by advanced features such as IDS and
>    malware detection.

That would be a good simple explanation for the rather low tested ACL value
that should be somewhere into the document, because it too is a part of the
evolution from RFC3511.

As an example counterpoint: There are new IETF protocols, such as MUD (RFC8520), that
ultimately result in automation of ACL building, for example for each IoT
device on the client side (lets say large enterprise) an ACL with a
few entries (defining the permitted traffic flows between this IoT device
and the Internet). So if the enterprise has maybe 4000 IoT devices, it could easily
be a (SrcIP,DstIP,Proto,*,DstPort) ACL list of maybe 15000 entries.

If such novel IETF work can not be put into a NGFW because that new firewall
will not support such long ACLs, that is your NGFW perogative, except that i would
feel a lot safer if that was written somewhere. Such as in the list of
excemptions not covered by this document.

I have seen similar automation of security through combinations of controllers
and firewalls in other areas as well, such as Telephony. 

>  * "security and measurement traffic" in row 411 can be changed if you
>    feel that the use of these descriptive words requires a definition.
>       If you like, we can remove "security and measurement", keeping
>    "traffic".

It seems to me that there are two type of traffic. One is the traffic passing
through the DUT to measure it, the other one is all the other collateral
traffic to make the test setup work. Those two type of traffic just need
to terms thart are defined. Unfortunately, i am not aware if/what any
two industry agreed upon terms are for these two type of traffic.

> *Obsoleting RFC3511*: RFC3511 shall indeed be obsoleted; we have had
> extensive discussions about it e.g. in IETF110
> <https://datatracker.ietf.org/meeting/110/materials/minutes-110-bmwg-01.pdf>
> and on the BMWG mailing list. Rationales:  1) Allowing the use of more than
> 18-year old benchmarking methodology for the same group of network security
> solutions in parallel to the new one would really confuse the market and not
> be good SDO workmanship.  2) All relevant benchmarks in RFC3511 are
> technically substituted and improved by the new draft; only a few L4
> (TCP/UDP) test cases have deliberately been obsoleted.  These test cases do
> not make any sense for today's NGFWs.  Even old firewalls (pre-NGFW) can be
> tested much more accurate with the new methodology.

I wasn't challenging the goal, i was just wondering if the document in
its current version has all the bells and whistles normally expected
for a document doing the obsoletion. Such as that "diff-from-obsoleted-rfc"
section and explanation such as the ones in your paragraph above.

> *Energy efficiency measurements* are of paramount importance - I agree. 
> Large-scale groups in ATIS and other SDOs are working on standardizing
> energy efficiency measurements.  I invite contributors to create a new draft
> for NGFW energy efficiency benchmarks. Unfortunately, attaching a power
> meter is not sufficient and is not a key performance indicator (KPI) by
> itself.

But it does become  very useful KPI for reders of reports as soon as 
they compare different products with for their purpose appropriate 
security/performance features set for their particular use-case. 

I actually ran through similar vetting with routers and customers. They
effectively where looking for Capex (equipment cost, space cost),
Opex (power consumption), vs. performance for a particular feature set.

> A firewall with fewer security functions is not better because it
> takes less power per Gigabit than a strong firewall. Balancing these KPIs
> with power usage is a difficult and sensitive task that requires a lot of
> compromises and industry consensus.

I am trying to parse what you say but have difficulties. Are you
afraid that people in the industry would fear that captiuring the power
consumption number during the steady state run of their devices would
aggravate people in the industry because they fear the power consumption
numbers would make their devices look bad and therefore you would prefer
for BMWG to not ask for taking such power measurement numbers ?

IMHO, A simple power consumption measurement taken the way i suggest to
(for the 10% and max-performance measurement points) is IMHO a very 
useful start, because it can help to motivate vendors within their
existing designs to optimize. For example CPU based NGFW would have more
of an incentive to go beyond simple PMD (Poll Mode Driver) for their
CPU forwarding plane (which is the worst CPU burner), and enabling
 of low-power mode dynamically anywere where the HW supports it when
only low performance is required. A lot of firewalls at the edge of the internet have
a lot of time where only low-performance is needed, but they need to
be bought for peak performance (such as most enterprise offices outside office
hours).

> I sincerely hope that the IETF
> recognizes the unparalleled importance of energy-efficient standardization
> and provides guidance to all areas and WGs; then the BMWG and NGFW
> benchmarks could well be expanded in future work.

I don't think we are setting a good example by simply pushing off the
task instead of taking care of low hanging fruits when they offer themselves,
like i think they do in this case.

> The *test setups* (Figure 2) are to be used as specified:
> 
>  * The goal of this benchmarking draft is to enable reproducible
>    results in controlled, minimum lab setups.  Of course we could
>    complicate it with mixing trusted and untrusted zones, adding more
>    security zones, etc.  Within the four years of development, the
>    group concluded that adding such complexity would not improve the
>    reproducibility and readibility of results.  "actual deployments"
>    and "typical deployments" relate to the "parameters and security
>    features" (line 258) - not to the network topology setup. 
>    Reproducibility is gained by adequate documentation of the
>    parameters and security features (see section 6 of the draft as
>    well) - not by nailing down specific configurations.  The NGFWs are
>    too diverse to aim for such a goal.

Agreed. But i just came to that realization after going through the whole
document and also not bing able to come up with a lot broader testing without
a lot more complexity. 

I think a sentence like the following would be good explanation for the
text: "The DUT test topology and test traffic flows do not aim
to exercise the variety of real-world traffic flow and security zones
often attached to an NGIPS/NGFW, but to represent the most simple topology
in which the performance of traffic flows through the DUT can be measured.".

>  * Of course, measurements with different sets of parameters will yield
>    different results (your comment on line 380).  Detailed reporting
>    (section 6) will allow readers to interpret results correctly.  One
>    potential use of this draft is to establish an external
>    certification program ("NetSecOPEN" initiative).  For such a
>    program, parameter sets need to be defined in more detail.  But the
>    consensus among authors and the WG was that the draft shall not be
>    limited to very specific certification setups.

Thats fine. Its again that the text is terse and lets one guess at what
you so much better explain above.

>  * "maximum security coverage" is a blanket clause, indicating that
>    should focus configurations on best security not only on achieving
>    maximum performance.  This is a typical conflict of goals in network
>    security benchmark testing, specifically if vendors carry out tests.

Seems to me as if you could simply delete "in order to achieve maximum network security coverage",
and the remaining text is still perfectly fine, and you avoid having readers wonder
about an undefined blanket clause.

>  * The technique of aggregating lower-speed interfaces from test
>    equipment for a higher-speed DUT interface is considered common lab
>    knowledge and thus not explained in this draft.
> 
>  * In line 255, the word "unique" could be misunderstood indeed. Maybe
>    the word "single" would explain it better?

single common ?
I am not a native english speaker, so please apply your best knowledge
to resolve those language nits if you agree with them ;-)

> *DUT classifications* into XS, S, M, and L were made in the main document to
> ensure this classification bears some weight.  It is important for
> apples-to-apples comparisons of benchmarks. While the requirements for
> number of rules per DUT classification is expected to be stable, the actual
> device scaling will change faster due to innovations.  This is why the DUT
> types are specified in Appendix B.  If you feel differently, please suggest
> text.

It just seems to me that the numbers in Figure 3 are completely dependent on the
XS/S/M/L classifications. If you'd make all XS/S/M/L devices in the future
10x faster, then you would need to equally change the numbers in Figure 3.
Tht makes the attempt of putting some numbers into an appendix, but not its
dependent numbers somewhat feeble.

Aka: I wouldn't t bother with appendix B. Just inline the text, makes the document
also easier to read. Or else you may also want to move out Figure 3 into
appendix B. Those seem to be the two consistent options to me.

And yes, sorry, this is also just structural text nitpicking, nothing
substantial, but hopefully adds to text quality.

> *Test Case descriptions*.
> 
>  * Your comment "Section 7 is a lot of work to get right" is
>    interesting. Procedural replication is intentional in test plans, to
>    make sure that each test case is complete.  Readers do not typically
>    appreciate complex referrals and footnotes when executing test cases
>    (speaking from a few years of experience). Being very descriptive in
>    test case descriptions improves the quality and reproducibility of
>    test execution.

Yes. Indeed. This was mostly from the concerns of a poor reviewer trying to
compare details across tests. Carry on.

(in real test plans i have often seen also big tables across test runs
 making comparisons easy when having large printers.... oh well ;-)

>  * Test case scale goals are already aligned with promised vs. measured
>    performance., as per the text in rows 1046/1047
>  * Row 1070 relates to foreground measurement traffic, appendix A.4 to
>    background traffic failure rates

Ah, ok. Thanks. But just because from the security perspective it's called
background traffic does not mean to me that 0.01% failure rate is appropriate.
That traffic would be potentially all business critical traffic and a good
amount of it might not be using resilient application code that recovers
from failures well.

E.g.: how did you folks come up with the 0.01% ?

Cheers
    Toerless

> Best regards, Carsten

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Mon, 31 Jan 2022 08:20:00 +0100

Dear Toerless,

Thank you very much for the tremendous detail of your review, and for 
your kind words regarding the readiness of the draft for publication.

Let me summarize my responses here instead of in-line, as it will be 
easier to read.

Regarding *terminology*:

  * NGFW and UTM are fixed terms in the industry.  NGFW is by far the
    overwhelmingly used term describing today's network security devices
    (firewalls with additional features).  There are informal
    definitions from Gartner, Wikipedia, and a number of vendors
    including Cisco.  We decided to use these terms because they are
    well-known to each network security professional, and there are no
    better ones. "Next-generation" sounds silly, but a lot of terms in
    our industry sound silly if read verbatim :-)

  * We have had extensive discussions of goodput, throughput, and other
    terms describing how many packets of which type make it from source
    to destination.  I would like to refer you to the mailing list
    archive <https://mailarchive.ietf.org/arch/browse/bmwg/?q=goodput>.
    In short, for NGFWs, the application layer is the important metric
    to measure.  This is why we do not refer to the generic "throughput"
    (L3) as defined in router benchmarking, but to "HTTP throughput" and
    "HTTPS throughput" (sections 7.3 and 7.7, respectively).  Your
    reference to QUIC is covered under HTTP/3 as mentioned in section
    4.3.1.3 of the draft.

  * Adding a terminology section would not have increased readability
    for the target audience.  We introduce major terms in table 3.  The
    draft is not an introductory tutorial to the network security
    architecture; it expects the reader to understand the purpose and
    configuration of perimeter firewalls (e.g. connected with a minimum
    of two interfaces between trusted and untrusted networks).  The term
    "active" is only used to explain the term "inline" in conjunction
    with Figures 1 and 2. There is no circular reference for human
    readers from my point of view (AI algorithms are not part of the
    target audience).

  * This document cannot and does not aim to introduce the whole world
    of network security terms.  It would become a compendium / tutorial
    for network security which is an explicit non-goal. This is just a
    benchmarking methodology document.  We have to assume that readers
    are network security professionals understanding the basics of
    today's network security functions. This is not aimed to criticize
    the area director review process, but I am not sure how to respond
    to some of your questions (e.g. about making SSL inspection
    mandatory, or recommending to implement Web Filtering across the
    industry [row 294]). I suspect this is because most IETF areas are
    related to Internet transport, not to security.

  * Tables 1 and 2 relate to different types of DUTs.  Due to the
    limitations of table text formatting in RFCs, we could not put both
    sets of values into a single table.

  * I agree that we can move table 3 to the top, making it the new table
    1, and renumber tables 1 -> 2 and 2 -> 3.
  *

  *

    DPI is an obsolete term.  Any NGFW does some type of deep-packet
    inspection.  More detailed terms in table 3 describe the different
    aspects of packet inspections.

*Obsoleting RFC3511*: RFC3511 shall indeed be obsoleted; we have had 
extensive discussions about it e.g. in IETF110 
<https://datatracker.ietf.org/meeting/110/materials/minutes-110-bmwg-01.pdf> 
and on the BMWG mailing list. Rationales:  1) Allowing the use of more 
than 18-year old benchmarking methodology for the same group of network 
security solutions in parallel to the new one would really confuse the 
market and not be good SDO workmanship.  2) All relevant benchmarks in 
RFC3511 are technically substituted and improved by the new draft; only 
a few L4 (TCP/UDP) test cases have deliberately been obsoleted.  These 
test cases do not make any sense for today's NGFWs.  Even old firewalls 
(pre-NGFW) can be tested much more accurate with the new methodology.

*Energy efficiency measurements* are of paramount importance - I agree.  
Large-scale groups in ATIS and other SDOs are working on standardizing 
energy efficiency measurements.  I invite contributors to create a new 
draft for NGFW energy efficiency benchmarks.  Unfortunately, attaching a 
power meter is not sufficient and is not a key performance indicator 
(KPI) by itself.  A firewall with fewer security functions is not better 
because it takes less power per Gigabit than a strong firewall. 
Balancing these KPIs with power usage is a difficult and sensitive task 
that requires a lot of compromises and industry consensus.  I sincerely 
hope that the IETF recognizes the unparalleled importance of 
energy-efficient standardization and provides guidance to all areas and 
WGs; then the BMWG and NGFW benchmarks could well be expanded in future 
work.

  * In line 255, the word "unique" could be misunderstood indeed. Maybe
    the word "single" would explain it better?
  * Line 389: As the document reads, logging of all traffic at the flow
    level is recommended.  Details of external log receivers are left to
    the individual implementation.

*DUT classifications* into XS, S, M, and L were made in the main 
document to ensure this classification bears some weight.  It is 
important for apples-to-apples comparisons of benchmarks. While the 
requirements for number of rules per DUT classification is expected to 
be stable, the actual device scaling will change faster due to 
innovations.  This is why the DUT types are specified in Appendix B.  If 
you feel differently, please suggest text.

  * Your comment "Section 7 is a lot of work to get right" is
    interesting. Procedural replication is intentional in test plans, to
    make sure that each test case is complete.  Readers do not typically
    appreciate complex referrals and footnotes when executing test cases
    (speaking from a few years of experience). Being very descriptive in
    test case descriptions improves the quality and reproducibility of
    test execution.
  * Test case scale goals are already aligned with promised vs. measured
    performance., as per the text in rows 1046/1047
  * Row 1070 relates to foreground measurement traffic, appendix A.4 to
    background traffic failure rates
  * The difference between HTTP throughput, HTTP transaction latency,
    and number of HTTP connections test cases is expected to be
    understood by the reader.  Do you really want us to provide a
    tutorial about HTTP benchmarking in general?  From my point of view
    such a tutorial does not belong into an RFC standardizing benchmark
    methodology.  Tutorials are not a typical development goal for RFCs
    it seems to me.

Best regards, Carsten

Am 1/30/2022 um 1:16 PM schrieb [email-masked]:
> Please fix the following nits - fromhttps://www.ietf.org/tools/idnits
>>    Checking nits according tohttps://www.ietf.org/id-info/checklist  :
> 35	   Drafts is athttps://datatracker.ietf.org/drafts/current/.
>
> 2415	              May 2017,<https://www.rfc-editor.org/info/rfc8174>.
>
> 2436	              2008,<https://www.rfc-editor.org/info/rfc5180>.
>
> 2651	   Email:[email-masked]
> 2658	   Email:[email-masked]
>
> 2660	   Brian Monkman
> 2661	   NetSecOPEN
> 2666	   Email:[email-masked]
>
> EOF

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked],https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Mon, 31 Jan 2022 18:00:20 +0100

Hi Toerless,

Thanks for your lighting-fast and detailed response:

- We agree to rewrite the intro slightly, introducing the terms NGFW and 
UTM and detail the assumptions of minimum working knowledge expected by 
readers.  Unfortunately IETF has not introduced network security 
architectures in other RFCs.  Earlier we got discouraged to reference 
external sources such as Wikipedia regarding an introduction to the 
general topic of network security, because it seems that RFCs typically 
do not provide such references and they are objected to by the IESG.  If 
you have some advice please let me know.

- We will add a paragraph explaining the rationale to obsolete RFC3511, 
and the differences of the new document.  Note that we do not plan to 
obsolete RFC2647; we will mention that as well and explain briefly that 
we will just amend the terminology with additional terms such as HTTP 
throughput.  We will also explain how firewalls have evolved from simple 
stateless ACL filters since the time when RFC3511 was published.

- We will rename DDoS in table 1 to "DDoS protection"

- Regarding the term DPI, we will enable consistency between tables 1, 
2, and 3 (and re-order them as agreed before, moving table 3 before table 1)

- We will add a note to the ACL numbers in figure 3, explaining that 
these are minimum numbers that can be increased at the discretion of the 
test lab.  Initially, we had much intended higher ACL numbers, but NGFW 
vendors explained that these numbers do not influence the performance of 
firewalls during operations as much anymore.  Our PoC tests with a 
number of DUT candidates confirmed these claims.

- We will rename "security and measurement traffic" to "test traffic" to 
avoid confusion

- Regarding your proposed caveat to the test setups, we agree to adopt 
it with some small modifications.  The test topology should be as simple 
as possible, but the test traffic flows certainly shall not be 
over-simplified.  In fact, we are defining multiple different traffic 
mixes to adapt to real-world traffic in a reproducible way.

- We can remove "in order to achieve maximum network security" from 
section 4.2

- We will rename "A unique DUT/SUT configuration" to "The same DUT/SUT 
configuration" in the first sentence of section 4.2.

More feedback to your comments that we believe does not require 
modifications of the draft:

- Badly working TCP stacks would result in bad performance measured 
according to the test methodology in section 7.  From our point of view 
there should not be a specific discussion of underlying protocol issues 
in this application-layer testing document.  We discussed the TCP issues 
within the BMWG for some time.   By the way, the moment we start talking 
about TCP details, immediately IETF people show up that ask "what about 
TCPv6" and "what about QUIC"?

- Certain use case scenarios allow validating SSL certificates (such as 
protecting web server farms with well-known certificates including the 
private keys). But in perimeter scenarios, SSL private keys are 
unknown.  And even a simple SSL certificate validation ("does the public 
key have a valid keychain") cannot be done exhaustively.  The NGFW is in 
/between /client and server, but not necessarily /in the middle///- as 
in decoding and re-encoding things- for all application scenarios.

- We think the word "unique" is not confusing and would like to keep the 
original wording.

- Appendix B was created following guidance from the BMWG co-chairs, as 
the numbers in figure 3 are affecting the benchmarking methodology but 
the numbers in Appendix B are not. We would like to keep it this way 
unless the co-chairs agree with the suggested change.

- The acceptable failure rates are a result of extensive discussions 
between the authors and contributors from the BMWG and is a compromise.  
In the end it is a finite number.  Some people wanted zero (zero 
packets) failure rate, some much higher failure rates to be permitted.   
The agreed-to number was also influenced by extensive benchmarking of 
the BMWG and related Linux Foundation work regarding failure rates by 
virtualized network functions. The foreground failure rates are lower by 
one order of magnitude because foreground traffic is considered more 
important and DUTs/SUTs are expected to act more precisely on foreground 
traffic.

Best regards,
      Bala, Brian, and Carsten

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 31 Jan 2022 18:44:26 +0100

Inline

On Mon, Jan 31, 2022 at 06:00:20PM +0100, Carsten Rossenhoevel wrote:
> - We agree to rewrite the intro slightly, introducing the terms NGFW and UTM
> and detail the assumptions of minimum working knowledge expected by
> architectures in other RFCs.

Shame. But it's typical. There is so much proprietary, yet "industry standard"
functionality in that space that doesn't seem to require interop and therefore
hasn't seen a lot of industry push to be done in IETF. And sometimes then we even
try to reinvent the wheel in the space in IETF. Requires some ambitious
individuals to write informational RFCs about such subject matter.

> Earlier we got discouraged to reference
> external sources such as Wikipedia regarding an introduction to the general
> topic of network security, because it seems that RFCs typically do not
> provide such references and they are objected to by the IESG.  If you have
> some advice please let me know.

I am not aware of general guidelines that object wikipedia. I can see how
one risk of wikipedia is that its not a stable reference unless you know how to
refer to a particular wikipedia article snapshot for the time you looked it up
and felt it was what you wanted. Other SDO documents might be easier to justify
when they're recognized as experts on the subject matter. Not sure if/what SDO
three might be for these security devices (if any). In my last rfc8994 i have
two wikipedia references for technical terms. Aka: if the references are for
terms and you feel they are stable in wikipedia i'd suggest to give it a try.

> - We will add a paragraph explaining the rationale to obsolete RFC3511, and
> the differences of the new document.  Note that we do not plan to obsolete
> RFC2647; we will mention that as well and explain briefly that we will just
> amend the terminology with additional terms such as HTTP throughput.  We
> will also explain how firewalls have evolved from simple stateless ACL
> filters since the time when RFC3511 was published.

If you introduce new terminology, you can think of updating rfc2647, e.g.:
you can have a section updates to rfc2647 that lists the new terms you use
in the document. If you where about to put those new term explanations into
a single section, its almost a no-add-on-effort to make that section be an
update to rfc2647. But that something like bonus work.
> - Regarding the term DPI, we will enable consistency between tables 1, 2,
> and 3 (and re-order them as agreed before, moving table 3 before table 1)
> 
> - We will add a note to the ACL numbers in figure 3, explaining that these
> are minimum numbers that can be increased at the discretion of the test
> lab.  Initially, we had much intended higher ACL numbers, but NGFW vendors
> explained that these numbers do not influence the performance of firewalls
> during operations as much anymore.  Our PoC tests with a number of DUT
> candidates confirmed these claims.

Thats what i expected. compressed, large, fast parse ACLs have been standard
for quite a while. If anything, i've seen unexpected delays when compiling
those acls (e.g.: config to operational delay which most people - including
this doc do not have to care about). 

> - Regarding your proposed caveat to the test setups, we agree to adopt it
> with some small modifications.  The test topology should be as simple as
> possible, but the test traffic flows certainly shall not be
> over-simplified.  In fact, we are defining multiple different traffic mixes
> to adapt to real-world traffic in a reproducible way.
> 
> - We can remove "in order to achieve maximum network security" from section
> 4.2
> 

Thanks a lot.

> there should not be a specific discussion of underlying protocol issues in
> this application-layer testing document.  We discussed the TCP issues within
> the BMWG for some time.   By the way, the moment we start talking about TCP
> details, immediately IETF people show up that ask "what about TCPv6" and
> "what about QUIC"?

I guess the way i would describe it: The HTTP(s) tests will indirectly uncover
any TCP stack issues of the DUT. What the tests will not provide is the
root cause analysis. E.g.: If performance of DUT sucks, it may need additional
work/measurement to isolate whether its from HTTP or TCP layer. Which probably is fine for
the purpose of this document. I was obviously coming from the view of somone
who likes to dig deeper as well ;-)

TCP6 should be equally be covered by the doc purely by running the tests
with IPv6 addresses, so i am not concerned about that.

I don't really understand what difference to a security device QUIC vs. TCP makes
except that any security setup has fewer roundtrips than TCP+TLS, so the
transaction performance should be higher...

> private keys). But in perimeter scenarios, SSL private keys are unknown. 
> And even a simple SSL certificate validation ("does the public key have a
> valid keychain") cannot be done exhaustively.  The NGFW is in /between
> /client and server, but not necessarily /in the middle///- as in decoding
> and re-encoding things- for all application scenarios.

Right. This is what confuses me. I understand that there are all those different
models, in the worst case the security device can not really see anything inside
a session and starting with TLS 1.3 it can not even see the server certificte
in the clear. In that case it's not clear to me how useful all those HTTP/HTTPs
tests are for those flows (from section 7) because at best the security device can
try to filter based on spoofing of DNS or similar contextual security.
On the other hand, if the security device is/can be set up as a trusted proxy, then
it can and will do a lot more with the flow, down to payload verification.

The document seems to not distinguish between these options.. 

> - Appendix B was created following guidance from the BMWG co-chairs, as the
> numbers in figure 3 are affecting the benchmarking methodology but the
> numbers in Appendix B are not. We would like to keep it this way unless the
> co-chairs agree with the suggested change.

Hmm... don't think that when numbers of table 3 would need to increase that
this would be different from when the DUT would have faster interfaces..
But i made my point, you folks decide ;-)

> - The acceptable failure rates are a result of extensive discussions between
> the authors and contributors from the BMWG and is a compromise.  In the end
> it is a finite number.  Some people wanted zero (zero packets) failure rate,
> some much higher failure rates to be permitted.   The agreed-to number was
> also influenced by extensive benchmarking of the BMWG and related Linux
> Foundation work regarding failure rates by virtualized network functions.
> The foreground failure rates are lower by one order of magnitude because
> foreground traffic is considered more important and DUTs/SUTs are expected
> to act more precisely on foreground traffic.

Thanks for the explanations. It was really curious to me, because from older
experiences with (old ;-) firewalls, i'd never expected that such high failure
rates would be acceptable (0.01% of "normal/good" flows). I wonder what specific
evolution of technology would be the cause of such high failure rates.

Cheers
    Toerless

-- 
---
[email-masked]