Subject: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

From: Roman Danyliw via Datatracker <[email-masked]>
Date: Wed, 02 Feb 2022 14:39:37 -0800

Roman Danyliw has entered the following ballot position for
draft-ietf-bmwg-ngfw-performance-13: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
for more information about how to handle DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

** A key element of successfully running the throughput tests described in
Section 7, appears to be ensuring how to configure the device under test. 
Section 4.2. helpfully specifies feature sets with recommendations
configurations.  However, it appears there are elements of under-specification
given the level of detail specified with normative language.  Specifically:

-- Section 4.2.1 seems unspecified regarding all the capabilities in Table 1
and 2.  The discussion around vulnerabilities (CVEs) does not appear to be
relevant to configuration of anti-spyware, anti-virus, anti-botnet, DLP, and
DDOS.

-- Recognizing that NGFW, NGIPS and UTM are not precise product categories,
offerings in this space commonly rely on statistical models or AI techniques
(e.g., machine learning) to improve detection rates and reduce false positives
to realize the capabilities in Table 1 and 2.  If even possible, how should
these settings be tuned?  How should the training period be handled when
describing the steps of the test regime (e.g., in Section 4.3.4? Section 7.2.4?)

** Appendix A.  The KPI measures don’t seem precise here – CVEs are unlikely to
be the measure seen on the wire.  Wouldn’t it be exploits associated with a
particular vulnerability (that’s numbered via CVE)?  There can be a one-to-many
relationship between the vulnerability and exploits (e.g., multiple products
affected by a single CVE); or the multiple implementations of an exploit.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

** Abstract.  NGFW, NGIPS and UTM are fuzzy product categories.  Do you want to
them somewhere?  How do they differ in functionality?  UTM is mentioned here,
but not again in the document.

** Section 1.
The requirements
   for network security element performance and effectiveness have
   increased tremendously since then.  In the eighteen years since
   [RFC3511] was published, recommending test methodology and
   terminology for firewalls, requirements and expectations for network
   security elements has increased tremendously.

I don’t follow how the intent of these two sentences is different.  Given the
other text in this paragraph, these sentences also appear redundant.

** Section 3. Per “This document focuses on advanced, …”, what makes a testing
method “advanced”?

** Section 4.2.  The abstract said that testing for NGFW, NGIPS and UTM would
be provided.  This section is silent on UTM.

** Section 4.2.  Should the following additional features be noted as a feature
of NGFWs and NGIPS (Tables 1 and 2)?

-- reconnaissance detection

-- geolocation or network topology-based classification/filtering

** Section 4.2. Thanks for the capability taxonomies describe here.  Should it
be noted that “Table 1 and 2 are approximate taxonomies of features commonly
found in currently deployed NGFW and NGIDS.  The features provided by specific
implementations may be named differently and not necessarily have configuration
settings that align to the taxonomy.”

** Table 1.  Is there a reason that DPI and Anti-Evasion (listed in Table 2 for
NGIPS) are not mentioned here (for NGFW).  I don’t see how many (all?) of the
features listed as RECOMMENDED could be done without it.

** Table 3.  For Anti-Botnet, should it read “detects and blocks”?

** Table 3.  For Web Filtering, is this scoped to be classification and threat
detection by URI?

** Table 3.  This table is missing a description for DoS from Table 1 and DPI
and Anti-Evasion from Table 2.

** Section 4.2.  Per “Logging SHOULD be enabled.”  How does this “SHOULD” align
with “logging and reporting” being a RECOMMENDED in Table 1 and 2?  Same
question on “Application Identification and Control SHOULD be configured”

** Section 4.3.1.1.  Why is such well-formed and well-behaved traffic assumed
for a security device?

** Section 4.3.1.  What cipher suites should be used for TLS 1.3 based tests?
The text is prescriptive for TLS 1.2 (using a RECOMMEND) but simply restates
all of those registered by RFC8446.

** Section 9.  Given that the configurations of these test will include working
exploits, it would be helpful to provide a reminder on the need control access
to them.

** Section A.1.
In parallel, the CVEs will be sent to the DUT/SUT as
   encrypted and as well as clear text payload formats using a traffic
   generator.

This guidance doesn’t seem appropriate for all cases.  Couldn’t the
vulnerability being exploited involve a payload in the unencrypted part or a
phase in the communication exchange before a secure channel is negotiated?

** Editorial nits
-- Section 1.  Editorial. s/for firewalls initially/for firewalls/

-- Section 5.  Typo. s/as test equipments/as test equipment/

---

From: "MORTON JR., AL" <[email-masked]>
Date: Wed, 2 Feb 2022 23:14:01 +0000

Authors,

Roman's DISCUSS ballot is particularly important to address. Specifically, the points above
----------------------------------------------------------------------
COMMENT:
line, but all the points should be addressed eventually.

If you can provide some proposed resolutions before the IESG meets Thursday, it will give our AD Warren some backup for any exchanges with Roman during the live meeting.

Status as of Wednesday evening:
There are surprisingly few IESG ballots cast (4)
https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/ballot/

I previously sent the link to the webex for the meeting, so you can listen-in and possibly gain some insights.

Al

-----Original Message-----
From: Roman Danyliw via Datatracker <[email-masked]> 
Sent: Wednesday, February 2, 2022 5:40 PM
To: The IESG <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Al Morton <[email-masked]>; [email-masked]
Subject: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

Please refer to https://urldefense.com/v3/__https://www.ietf.org/blog/handling-iesg-ballot-positions/__;!!BhdT!wz4NWwr2EhVf2SVpgeLrJ37BbODitrSIIGTSdWLeVIHQAuDPrMeDZojEcJCm$ 
for more information about how to handle DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/__;!!BhdT!wz4NWwr2EhVf2SVpgeLrJ37BbODitrSIIGTSdWLeVIHQAuDPrMeDZpdwKyaS$ 

** Section 1.
The requirements

** Section A.1.

** Editorial nits

---

From: <[email-masked]>
Date: Wed, 2 Feb 2022 19:18:34 -0500

Thanks for the feedback Al. These are all very useful comments. However, given the complexity of the questions/comments and given that the IESG meeting is early tomorrow I am concerned that there won't be enough time to provide an adequate answer.  We will give it our best shot. 

Brian

-----Original Message-----
From: MORTON JR., AL <[email-masked]> 
Sent: Wednesday, February 2, 2022 6:14 PM
To: Carsten Rossenhoevel <[email-masked]>; [email-masked]; [email-masked]
Cc: Sarah Banks <[email-masked]>; 'Warren Kumari' <[email-masked]>; [email-masked]
Subject: FW: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

Authors,

Status as of Wednesday evening:
There are surprisingly few IESG ballots cast (4) https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/ballot/

Al

-----Original Message-----
From: Roman Danyliw via Datatracker <[email-masked]>
Sent: Wednesday, February 2, 2022 5:40 PM
To: The IESG <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Al Morton <[email-masked]>; [email-masked]
Subject: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)

** Section 1.
The requirements

** Section A.1.

** Editorial nits

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Thu, 3 Feb 2022 01:48:11 +0100

Dear Roman,

thank you for your comments and questions. Primarily, the document is 
focused on performance benchmarking as mentioned in 4.2.1.  This is the 
area where the authors and contributors saw a major gap in test 
methodology.  Today's datasheets of commercial firewalls are sometimes 
difficult to interpret.  Performance benchmarking methodology applies 
across all types of security features (per Table 1).

Thus, we describe performance benchmarking methodology in great detail.  
Sections 4.3 and 7 focus on representative and reproducible load 
profiles.  (Note - Training phases would not include threat emulation.  
This is normal and not mentioned otherwise in the document - do you 
recommend we change any text / add an explanation?)

Advanced security features use a wide range of functions and methods, as 
you mentioned.  It would be very difficult (and a race against time) to 
cover white-box test methodology exhaustively for such security features 
in a standards document.  Therefore in this document, we:

a) describe the recommended and optional security features at high level 
(section 4.2)

b) suggest evaluating the correct functional security effectiveness of 
these features in advance of performance benchmarking, outside of the 
core performance benchmarking methodology (mentioned in section 4.2.1, 
second paragraph)

c) safeguard against the false testing of a firewall without proper 
security features enabled (mentioned in section 4.2.1, first paragraph 
and 2nd sentence in second paragraph)

d) describe security effectiveness test methodology verifying protection 
against common vulnerabilities and exposures (CVEs) in Appendix A.  This 
test methodology is a black-box approach not aiming to verify/compare 
advanced concepts such as sandboxing, zero-day threat protection, and 
other AI-based functions.   We are working on test methodology for these 
areas in the NetSecOPEN group; but it is difficult to come up with 
reasonable test solutions (specifically if they should be reproducible 
and test tool-agnostic), which is why this document does not include 
detailed methodology yet.

Regarding "CVE" nomenclature, I agree that the language in Appendix A is 
not absolutely precise.  The term CVE has become informally used for the 
(NIST/Mitre) database entries, the firewall capability to protect, and 
for the test tool attack type.   The authors are open to improving the 
terminology.  What do you suggest (without making each sentence a mouthful)?

I also agree with your observation that CVEs are often not atomic and 
that test tool implementations and firewall implementations may cover 
parts of a CVE.  This is an issue of the CVE database - there is no 
consistent numbering of sub-CVEs.  We have suffered from this issue in 
our security effectiveness methodology verification program we have 
conducted since early 2021.  Our goal is to create a standard document 
that is not tied to any specific test tool or lab implementation.  Two 
test labs (UNH-IOL and EANTC) have worked with two test tool vendors 
(Spirent and Keysight) and multiple firewall vendors.  We  have 
evaluated thousands of CVEs, comparing test coverage and results.

Each test tool vendor uses individual implementations to emulate threats 
aligned with CVEs, and their implementations sometimes differ in the 
sub-CVE coverage.  Firewall vendors also have their individual code and 
do not always cover all sub-CVEs - often for good reasons that need to 
be analyzed on the application layer individually (very high effort).  
We concluded that we have to accept the state of the art in CVE 
granularity and that we have to live with the uncertainties of sub-CVE 
coverage in general.  (In detail we will continue to work with test tool 
and firewall vendors to align implementations and aim for best-working 
CVE threat protection.)  If you have any ideas how this draft document 
could help improve the industry more quickly, we are seriously open for 
suggestions.

Re your other comments:

- The introduction section and tables 1/2/3 are being rephrased 
editorially to increase their readability and add more explanations, 
based on feedback from other AD reviewers.

- In section 4.2, I am not sure if I understand your comment about 
logging.  According to RFC 2119, "RECOMMEND" and "SHOULD" consistently 
describe the same level of requirement?

- Well-defined TCP stack attributes are required for a network security 
device because transport layer attributes influence its performance 
greatly.  Many issues found in EANTC tests in the past originated from 
incorrect TCP stack attributes, and non-reproducible behavior of a 
firewall is often related to undocumented/uncontrolled TCP stack attributes.

- From our understanding, TLS 1.3 cipher suites are all considered 
adequate in the industry today, while some of the TLS 1.2 cipher suites 
are considered deprecated. This is why we specify them in detail.  That 
said, the note below the list of TLS 1.2 recommended ciphers encourages 
the use of ciphers appropriate for evolving use cases.  This might lead 
to a selection of specific TLS 1.3 ciphers in the future as well.

- Good point regarding the use of emulated exploits in section 9.  We 
agree to add a note.

- Regarding 2nd paragraph of A.1, second sentence ("In parallel..."), we 
agree that the text is imprecise.  Threats that apply to the connection 
setup cannot be encrypted.  We'll edit the text to apply to these cases 
as well.

Best regards, Carsten

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked], https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025

---

From: <[email-masked]>
Date: Wed, 9 Feb 2022 10:34:49 -0500

Roman,

Do you have response to the questions Carsten asked in response to you?

Brian

-----Original Message-----
From: Carsten Rossenhoevel <[email-masked]> 
Sent: Wednesday, February 2, 2022 7:48 PM
To: Roman Danyliw <[email-masked]>; The IESG <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Al Morton <[email-masked]>
Subject: Re: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

Dear Roman,

thank you for your comments and questions. Primarily, the document is focused on performance benchmarking as mentioned in 4.2.1.  This is the area where the authors and contributors saw a major gap in test methodology.  Today's datasheets of commercial firewalls are sometimes difficult to interpret.  Performance benchmarking methodology applies across all types of security features (per Table 1).

Thus, we describe performance benchmarking methodology in great detail. Sections 4.3 and 7 focus on representative and reproducible load profiles.  (Note - Training phases would not include threat emulation. This is normal and not mentioned otherwise in the document - do you recommend we change any text / add an explanation?)

Advanced security features use a wide range of functions and methods, as you mentioned.  It would be very difficult (and a race against time) to cover white-box test methodology exhaustively for such security features in a standards document.  Therefore in this document, we:

a) describe the recommended and optional security features at high level (section 4.2)

b) suggest evaluating the correct functional security effectiveness of these features in advance of performance benchmarking, outside of the core performance benchmarking methodology (mentioned in section 4.2.1, second paragraph)

c) safeguard against the false testing of a firewall without proper security features enabled (mentioned in section 4.2.1, first paragraph and 2nd sentence in second paragraph)

d) describe security effectiveness test methodology verifying protection against common vulnerabilities and exposures (CVEs) in Appendix A.  This test methodology is a black-box approach not aiming to verify/compare advanced concepts such as sandboxing, zero-day threat protection, and other AI-based functions.   We are working on test methodology for these areas in the NetSecOPEN group; but it is difficult to come up with reasonable test solutions (specifically if they should be reproducible and test tool-agnostic), which is why this document does not include detailed methodology yet.

Regarding "CVE" nomenclature, I agree that the language in Appendix A is not absolutely precise.  The term CVE has become informally used for the
(NIST/Mitre) database entries, the firewall capability to protect, and for the test tool attack type.   The authors are open to improving the terminology.  What do you suggest (without making each sentence a mouthful)?

I also agree with your observation that CVEs are often not atomic and that test tool implementations and firewall implementations may cover parts of a CVE.  This is an issue of the CVE database - there is no consistent numbering of sub-CVEs.  We have suffered from this issue in our security effectiveness methodology verification program we have conducted since early 2021.  Our goal is to create a standard document that is not tied to any specific test tool or lab implementation.  Two test labs (UNH-IOL and EANTC) have worked with two test tool vendors (Spirent and Keysight) and multiple firewall vendors.  We  have evaluated thousands of CVEs, comparing test coverage and results.

Each test tool vendor uses individual implementations to emulate threats aligned with CVEs, and their implementations sometimes differ in the sub-CVE coverage.  Firewall vendors also have their individual code and do not always cover all sub-CVEs - often for good reasons that need to be analyzed on the application layer individually (very high effort). We concluded that we have to accept the state of the art in CVE granularity and that we have to live with the uncertainties of sub-CVE coverage in general.  (In detail we will continue to work with test tool and firewall vendors to align implementations and aim for best-working CVE threat protection.)  If you have any ideas how this draft document could help improve the industry more quickly, we are seriously open for suggestions.

Re your other comments:

- The introduction section and tables 1/2/3 are being rephrased editorially to increase their readability and add more explanations, based on feedback from other AD reviewers.

- In section 4.2, I am not sure if I understand your comment about logging.  According to RFC 2119, "RECOMMEND" and "SHOULD" consistently describe the same level of requirement?

- Well-defined TCP stack attributes are required for a network security device because transport layer attributes influence its performance greatly.  Many issues found in EANTC tests in the past originated from incorrect TCP stack attributes, and non-reproducible behavior of a firewall is often related to undocumented/uncontrolled TCP stack attributes.

- From our understanding, TLS 1.3 cipher suites are all considered adequate in the industry today, while some of the TLS 1.2 cipher suites are considered deprecated. This is why we specify them in detail.  That said, the note below the list of TLS 1.2 recommended ciphers encourages the use of ciphers appropriate for evolving use cases.  This might lead to a selection of specific TLS 1.3 ciphers in the future as well.

- Good point regarding the use of emulated exploits in section 9.  We agree to add a note.

- Regarding 2nd paragraph of A.1, second sentence ("In parallel..."), we agree that the text is imprecise.  Threats that apply to the connection setup cannot be encrypted.  We'll edit the text to apply to these cases as well.

Best regards, Carsten

Am 2/2/2022 um 11:39 PM schrieb Roman Danyliw via Datatracker:
> email addresses included in the To and CC lines. (Feel free to cut 
> this introductory paragraph, however.)
>
>
> Please refer to 
> https://www.ietf.org/blog/handling-iesg-ballot-positions/
> ** A key element of successfully running the throughput tests 
> described in Section 7, appears to be ensuring how to configure the device under test.
> configurations.  However, it appears there are elements of 
> under-specification given the level of detail specified with normative language.  Specifically:
>
> -- Section 4.2.1 seems unspecified regarding all the capabilities in 
> Table 1 and 2.  The discussion around vulnerabilities (CVEs) does not 
> appear to be relevant to configuration of anti-spyware, anti-virus, 
> anti-botnet, DLP, and DDOS.
>
> -- Recognizing that NGFW, NGIPS and UTM are not precise product 
> categories, offerings in this space commonly rely on statistical 
> models or AI techniques (e.g., machine learning) to improve detection 
> rates and reduce false positives to realize the capabilities in Table 
> 1 and 2.  If even possible, how should these settings be tuned?  How 
> should the training period be handled when describing the steps of the 
> test regime (e.g., in Section 4.3.4? Section 7.2.4?)
>
> ** Appendix A.  The KPI measures don’t seem precise here – CVEs are 
> unlikely to be the measure seen on the wire.  Wouldn’t it be exploits 
> associated with a particular vulnerability (that’s numbered via CVE)?  
> There can be a one-to-many relationship between the vulnerability and 
> exploits (e.g., multiple products affected by a single CVE); or the multiple implementations of an exploit.
>
>
> ** Abstract.  NGFW, NGIPS and UTM are fuzzy product categories.  Do 
> you want to them somewhere?  How do they differ in functionality?  UTM 
> is mentioned here, but not again in the document.
>
> ** Section 1.
> The requirements
> I don’t follow how the intent of these two sentences is different.  
> Given the other text in this paragraph, these sentences also appear redundant.
>
> ** Section 3. Per “This document focuses on advanced, …”, what makes a 
> testing method “advanced”?
>
> ** Section 4.2.  The abstract said that testing for NGFW, NGIPS and 
> UTM would be provided.  This section is silent on UTM.
>
> ** Section 4.2.  Should the following additional features be noted as 
> a feature of NGFWs and NGIPS (Tables 1 and 2)?
>
> ** Section 4.2. Thanks for the capability taxonomies describe here.  
> Should it be noted that “Table 1 and 2 are approximate taxonomies of 
> features commonly found in currently deployed NGFW and NGIDS.  The 
> features provided by specific implementations may be named differently 
> and not necessarily have configuration settings that align to the taxonomy.”
>
> NGIPS) are not mentioned here (for NGFW).  I don’t see how many (all?) 
> of the features listed as RECOMMENDED could be done without it.
>
> ** Table 3.  For Web Filtering, is this scoped to be classification 
> and threat detection by URI?
>
> ** Table 3.  This table is missing a description for DoS from Table 1 
> and DPI and Anti-Evasion from Table 2.
>
> ** Section 4.2.  Per “Logging SHOULD be enabled.”  How does this 
> “SHOULD” align with “logging and reporting” being a RECOMMENDED in 
> Table 1 and 2?  Same question on “Application Identification and Control SHOULD be configured”
>
> ** Section 4.3.1.1.  Why is such well-formed and well-behaved traffic 
> assumed for a security device?
>
> The text is prescriptive for TLS 1.2 (using a RECOMMEND) but simply 
> restates all of those registered by RFC8446.
>
> ** Section 9.  Given that the configurations of these test will 
> include working exploits, it would be helpful to provide a reminder on 
> the need control access to them.
>
> ** Section A.1.
> vulnerability being exploited involve a payload in the unencrypted 
> part or a phase in the communication exchange before a secure channel is negotiated?
>
> ** Editorial nits
--
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center) Salzufer 14, 10587 Berlin, Germany office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721 [email-masked], https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany EU VAT No: DE812824025