Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

From: Sarah Banks <[email-masked]>
Date: Mon, 12 Jul 2021 14:19:39 -0700

Hi Brian et al,
    First, my apologies for the delay, and I very much appreciate your patience. I also appreciate the time and effort that went into the reply to my comments, which can be even more difficult to do as a large group :) Please see inline.

Thank you,
Sarah (as a participant)

>> 
>> Hi Sarah,
>> 
>> As I mentioned in the previous message, we will remove reference to 
>> IDS from the draft. Given that, none of the IDS related 
>> comments/questions are being addressed.

SB// Makes sense, thank you.

>> - The draft aims to replace RFC3511, but expands scope past Firewalls, 
>> to "next generation security devices". I'm not finding a definition of 
>> what a "next generation security device is", nor an exhaustive list of 
>> the devices covered in this draft. A list that includes is nice, but 
>> IMO not enough to cover what would be benchmarked here - I'd prefer to 
>> see a definition and an exhaustive list.
>> 
>> [bpm] "We avoid limiting the draft by explicitly adding a list of NG 
>> security devices currently available in the market only. In the 
>> future, there may be more and more new types of NG security devices 
>> that will appear on the market.

SB// I think there are 2 types of devices called out; I'm not seeing a definition of what a "NG security device" is, and I'm not comfortable with a draft that has a blanket to encompass what would come later. Who knows what new characteristics would arrive with that new device? I think the scope here is best suited for the devices we know about today and can point to and say we're applying knowledgeable benchmarking tests against.

>> 
>> [bpm] This draft includes a list of security features that the 
>> security device can have ( RFC 3511 doesn't have such a list). Also, 
>> we will describe in the draft that the security devices must be configured ""in-line"" mode.
>> We believe these two points qualifying the definition of next 
>> generation security.
>> 

SB// I strongly disagree. Well, I mean OK, for active inline devices maybe this is OK, but to say that the only way a device can be "NG" is to be active/inline, I disagree with. And if there is, have we gathered all of the items we'd want to actively test for in that case? For example, what about their abilities to handle traffic when a failure occurs? (fail open/closed). What about alerts and detections and the whole federation of tests around positives/false positives/false negatives, etc? I'm onboard with expanding the scope, but then we have to do the devices benchmarking justice, and I feel we're missing a lot here.

>> - What is a NGIPS or NGIDS? If there are standardized definitions 
>> pointing to them is fine, otherwise, there's a lot of wiggle room here.
>> 
>> [bpm] See above. We are removing NGIDS from the draft.

SB// Understood, thank you.

>> 
>> - I still have the concern I shared at the last IETF meeting, where 
>> here, we're putting active inline security devices in the same 
>> category as passive devices. On one hand, I'm not sure I'd lump these 
>> three together in the first place; on the other, active inline devices 
>> typically include additional functions to allow administrators to 
>> control what happens to packets in the case of failure, and I don't 
>> see those test cases included here.
>> 
>> [bpm] This draft focuses on ""in-line"" mode security devices only. We 
>> will describe this in section 4 in more detail.

SB// Understood, thank you.

>> 
>> [bpm] Additionally, the draft focuses mainly on performance tests. The 
>> DUT must be configured in ""fail close"" mode. We will describe this 
>> under section 4. Any failure scenarios like ""fail open"" mode is out of scope.

SB// OK, but I think an RFC that is going to encompass this device under the "NG security devices" classification is missing out on large portions of what customers will want to test. It'll also beg for another draft to cover them, and then I'm not sure we're serving the industry as well as we could. 

>> 
>> - Section 4.1 - it reads as if ANY device in the test setup cannot 
>> contribute to network latency or throughput issues, including the DUTs 
>> - is that what you intended?
>> 
>> [bpm] "Our intention is, if the external devices (routers and 
>> switches) are used in the test bed, they should not negatively impact DUT/SUT performance.
>> To address this, we added a section ( section 5 ""Test Bed 
>> Considerations"") which recommends a pre-test.  We can rename this as 
>> reference test or baseline test. "
>> 

SB// Thank you for the clarification. I think there's still a concern there. Who defines what "negative impact" is? You're traversing at least another L2 or L3 step in the network with each bump, which contributes some amount of latency. If they don't serve in control plane decisions and are passively passing data on, then we could consider removing them from the setup and removing the potential skew on results.

>> - Option 1: It'd be nice to see a specific, clean, recommended test bed.
>> There are options for multiple emulated routers. As a tester, I expect 
>> to see a specific, proscribed test bed that I should configure and 
>> test against.
>> 
>> [bpm] The draft describes that Option 1 is the recommended test setup.
>> However. We added emulated routers as optional in option 1. The reason 
>> for
>> that:
>> Some type of security devices for some deployment scenarios requires 
>> routers between test client/server and the DUT (e.g., NGFW) and some 
>> DUT/SUT doesn't need router (e.g. NGIPS )
>> 
>> - Follow on: I'm curious as to the choice of emulated routers here. 
>> The previous test suggests you avoid routers and switches in the topo, 
>> but then there are emulated ones here. I'm curious as to what 
>> advantages you think these bring over the real deal, and why they 
>> aren't subject to the same limitations previously described?
>> 
>> [bpm] Comparing real router, the emulated router gives more advantages 
>> for
>> L7 testing.
>> 
>> [bpm] - Emulated router doesn't add latency. Even if it adds delay due 
>> to the routing process, the test equipment can report the added 
>> latency, or it can consider this for the latency measurement.
>> 
>> [bpm] - Emulated routers simply do routing function only. But in a "real"
>> router, we are not sure what else the router is doing with the packets.

SB// Maybe I'm missing something here - a device can't perform a function for free, right? Even if it's impact is negligible, it's an impact of some sort. We're saying the emulated router is doing the routing - OK - but I think the same thing applies to the physical router - how do you know what else the emulated router is doing? if the test gear can call out the latency, I'd like to see clarification around how it's doing that and distinguishing the latency introduced by Device A, versus Device B, versus the DUT, etc.

>> 
>> [bpm] Your question regarding the need for routers:
>> 
>> [bpm] - We avoid impacting the DUT/SUT performance due to ARP or ND 
>> process
>> 
>> [bpm] - Represent realistic scenario (In the production environment 
>> the security devices will not be directly connected with the clients.)
>> 
>> [bpm] - Routing (L3 mode) is commonly used in the NG security devices.
>> 
>> [bpm] However, in both figures we mentioned that router including 
>> emulated router is optional. If there is no need have routing 
>> functionality on the test bed (e.g., if we used very small number of 
>> clients and server IPs or the DUT operates in Layer 2 mode), it can be ignored.
>> 
>> [bpm] Also, we described in Option 1, that the external devices are if 
>> there is need to aggregate the interfaces of the tester or DUT.  For 
>> an example, DUT has 2 Interfaces, but tester need to use it's 4 
>> interfaces to achieve the performance. So here we need switch/router 
>> to aggregate tester interface from 4 to 2.
>> 
>> - In section 4.1 the text calls out Option 1 as the preferred test 
>> bed, which includes L3 routing, but it's not clear why that's needed?
>> 
>> [bpm] See above.
>> 
>> - The difference between Option 1 and Option 2 is the inclusion of 
>> additional physical gear in Option 2 - it's not clear why that's 
>> needed, or why the tester can't simply directly connect the test 
>> equipment to the DUT and remove extraneous devices from potential influence on results?
>> 
>> [bpm] See above.
>> 
>> - Section 4.2, the table for NGFW features - I'm not sure what the 
>> difference is between RECOMMENDED and OPTIONAL? (I realize that you 
>> might be saying that RECOMMENDED is the "must have enabled" features, 
>> where as optional is at your discretion, but would suggest that you 
>> make that clear)
>> 
>> [bpm] The definition for OPTIONAL and RECOMMENDED is described in,  
>> and recommended, RFC2119. We already referenced this under the section 
>> 2 "Requirements".
>> 

SB// Thanks!

>> - Proscribing a list of features that have to be enabled for the test, 
>> or at least more than 1, feels like a strange choice here - I'd have 
>> expected tests cases that either test the specific features one at a 
>> time, or suggest several combinations, but that ultimately, we'd tell 
>> the tester to document WHICH features were enabled, to make the test 
>> cases repeatable? This allows the tester to apply a same set of apples 
>> to apples configurations to different vendor gear, and omit the 1 
>> feature that doesn't exist on a different NGFW (for example), but hold a baseline that could be tested.
>> 
>> - Table 2: With the assumption that NGIPS/IDS are required to have the 
>> features under "recommended", I disagree with this list. For example, 
>> some customers break and inspect at the tap/agg layer of the network - 
>> in this case, the feed into the NGIDS might be decrypted, and there's 
>> no need to enable SSL inspection, for example.
>> 
>> [bpm] IDS is being removed.

SB// OK...I'm not sure this addresses the feedback though :) A NGFW for sure will do break/inspect as well, right?

>> 
>> - Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This 
>> behaviour might be suitable for an NGIPS, but the NGIDS is not a bump 
>> on the wire, and often isn't decrypting and re-encrypting the traffic.
>> 
>> [bpm] IDS is being removed.

SB// See comment above.

>> 
>> - Table 3: An NGIDS IMO is still a passive device - it wouldn't be 
>> blocking anything, but agree that it might tell you that it happened after the fact.
>> 
>> [bpm] IDS is being removed.
SB// Thanks!

>> 
>> - Table 3: Anti-evasion definition - define "mitigates". 
>> 
>> [bpm] Not sure why you are asking this as mitigate is not an uncommon 
>> term/word.
>> 
>> - Table 3: Web-filtering - not a function of an NGIDS.
>> 
>> [bpm] IDS is being removed.
>> 
>> - Table 3: DLP: Not applicable for an NGIDS.
>> 
>> [bpm] IDS is being removed.
>> 
>> - Can you expand on "disposition of all flows of traffic are logged" - 
>> what's meant here specifically, and why do they have to be logged? 
>> (Logging, particularly under high loads, will impact it's own 
>> performance marks, and colours output)
>> 
>> [bpm] We intentionally recommended enabling logging which will impact 
>> the performance. The draft is not aiming to get high performance 
>> number with minimal DUT/SUT configuration. In contrast, it aims to get 
>> reasonable performance number with realistic DUT configuration. The 
>> realistic configuration can vary based on DUT/SUT deployment scenario.
>> 
>> [bpm] In most of the DUT/SUT deployment scenarios or customer 
>> environments, logging is enabled as default configuration.
>> 
>> [bpm] "Disposition of all flows of traffic are logged": means that the 
>> DUT/SUT need to log all the traffic at the flow level not each packet.
>> 
>> [bpm] We will add more clarification for the meaning of "disposition 
>> of all flows of traffic are logged".
>> 

SB// Thanks!

>> - ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic 
>> :)
>> 
>> [bpm] IDS is being removed.
>> 
>> - It might be helpful to testers to say something like "look, here's 
>> one suggested set of ACLs. If you're using them, great, reference 
>> that, but otherwise, make note of the ACLs you use, and use the same 
>> ones for repeatable testing".
>> 
>> [bpm] The draft gives guidance how to choose the ACL rules. We 
>> describe here a methodology to create ACL.
>> 
>> - 4.3.1.1 The doc proscribes specific MSS values for v4/v6 with no 
>> discussion around why they're chosen - that color could be useful to 
>> the reader.
>> 
>> [bpm] We will add some more clarification that these are the default 
>> number used in most of the client operating systems currently.
>> 

SB// Thanks!

>> - 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
>> that should be changed.
>> 
>> [bpm] Thank you.
>> 
>> - 4.3.1.1 - As a tester with long time experience with major test 
>> equipment manufacturers, I can't possibly begin to guess which ones of 
>> them would conform to this - or even if they'd answer these questions. 
>> How helpful is this section to the non test houses? I suggest 
>> expansion here, ideally with either covering the scope of what you 
>> expect to cover, or hopefully which (open source/generally available) 
>> test tools or emulators could be considered for use as examples.
>> 
>> [bpm] We extensively discussed with Ixia and Spirent about this section.
>> This section was developed with significant input from these test 
>> tools vendors in addition to others.

SB// OK, that's really good to know, but there are plenty of us working with and looking for more cost effective options to Ixia and Spirent. :) I think the expansion would be good here.

>> 
>> - 4.3.1.3 - Do the emulated web browser attributes really apply to 
>> testing the NGIPS?
>> 
>> [bpm] Yes, we performed many PoC tests with test tools. Ixia and 
>> Spirent confirmed this.
>> 
>> - 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration 
>> option here?
>> 
>> [bpm] Yes
>> 
>> - 4.3.4 - I'm surprised to see the requirement that all sessions 
>> establish a distinct phase before moving on to the next. You might 
>> clarify why this is a requirement, and why staggering them is specifically rejected?
>> 
>> [bpm] This draft doesn't describe that all sessions establish a 
>> distinct phase before moving on to the next. We will remove the word 
>> "distinct" from the 1st paragraph in section 4.3.4.

SB// Thanks!

>> 
>> [bpm] Unlike Layer 2/3 testing, Layer 7 testing requires several 
>> phases in the traffic load profile. The traffic load profile described 
>> in the draft is the common profile mostly used for Layer 7 testing.
>> 
>> - 5.1 - I like the sentence, but it leaves a world of possibilities 
>> open as to how one confirmed that the ancillary switching, or routing 
>> functions didn't limit the performance, particularly the virtualized components?
>> 
>> [bpm] The sentence says, "Ensure that any ancillary switching or 
>> routing functions between the system under test and the test equipment 
>> do not limit the performance of the traffic generator."
>> 
>> [bpm] Here we discuss the traffic generator performance, and this can 
>> be confirmed by doing reference test.
>> 
>> [bpm] The section 5 recommends reference test to ensure that the 
>> maximum desired traffic generator's performance. Based on the 
>> reference test results it can be identified, if the external device 
>> added any impact on traffic generator's performance.
>> 
>> [bpm] We will add more content in section 5 to provide more details 
>> about reference test.
>> 

SB// Thanks!

>> - 5.3 - this is a nice assertion but again, how do I reasonably make 
>> the assertion?
>> 
>> [bpm] We will change the word from "Assertion" to "Ensure". Also, we 
>> will add more clarity about reference testing.

SB// Thanks!

>> 
>> - 6.1 - I would suggest that the test report include the configuration 
>> of ancillary devices on both client/server side as well
>> 
>> [bpm] We believe that adding configuration of the ancillary devices 
>> doesn't add more value in the report. Instead of this, we will 
>> recommend documenting the configuration of the ancillary devices by 
>> doing reference test. We will add this under the section 5 "Test bed consideration".
>> 

SB// I think including them assists greatly in the repeatability of the testing, for what it's worth.

>> - 6.3 - Nothing on drops anywhere?
>> 
>> [bpm] Are you referring to packet drops? If you are, there is no 
>> packet loss in stateful traffic. Instead of packet loss, the stateful 
>> traffic has retransmissions.
>> 
>> - 7.1.3.2 - Where are these numbers coming from? How are you 
>> determining the "initial inspected throughput"? Maybe I missed that in 
>> the document overall, but it's not clear to me where these KPIs are 
>> collected? I suggest this be called out.
>> 
>> [bpm] We will add more clarification in the next version. Thank you.

SB// Thanks!

>> 
>> - 7.1.3.3 - what is a "relevant application traffic mix" profile?
>> 
>> [bpm] This is described in section7.1.1  (2nd paragraph). We will add 
>> the word "relevant" in the 1st sentence of the 2nd pragraph.so the 
>> sentence will be "Based on customer use case, users can choose the 
>> relevant application traffic mix for this test.  The details about the 
>> traffic mix MUST be documented in the report.  At least the following 
>> traffic mix details MUST be documented and reported together with the test results:

SB// A set of example(s) could be helpful. Not required, just helpful.

>> 
>> - 7.1.3.4 - where does this monitoring occur?
>> 
>> [bpm] The monitoring or measurement must occur in the test equipment.
>> Section 4.3.4 describes this.
>> 
>> - 7.1.3.4 - This looks a bit like conformance testing -  Why does item 
>> (b) require a specific number/threshold?
>> 
>> [bpm] These numbers are synonymous with the zero-packet loss criteria 
>> for [RFC2544] Throughput and recognize the additional complexity of 
>> application layer performance. This was agreed by the IETF BMWG.
>> 
>> - 9: Why is the cipher squite recommendation for a real deployment 
>> outside the scope of this document?
>> 
>> [bpm] Because new cipher suites are frequently developed. Given that 
>> the draft will not be easily updated once it is accepted as an RFC we 
>> wanted to ensure there was flexibility to use future developed cipher suites.
>> 
>> Brian Monkman on behalf of....
>> 
>> Alex Samonte (Fortinet), Amritam Putatunda (Ixia/Keysight), Bala 
>> Balarajah (NetSecOPEN), Carsten Rossenhoevel (EANTC), Chris Brown 
>> (UNH-IOL), Mike Jack (Spirent), Ryan Liles (Cisco), Tim Carlin 
>> (UNH-IOL), Tim Otto (Juniper)
>>

---

From: <[email-masked]>
Date: Tue, 13 Jul 2021 10:44:26 -0400

Looping in the others on my original post.

Thanks Sarah,

Good to see the issues are being whittled down. Of the remaining five or six
issues I doubt we will have a response prior to IETF 111. 

Al,

Could we schedule an interim meetup/call for some time early August?

Brian

-----Original Message-----
From: Sarah Banks <[email-masked]> 
Sent: Monday, July 12, 2021 5:20 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Brian et al,
    First, my apologies for the delay, and I very much appreciate your
patience. I also appreciate the time and effort that went into the reply to
my comments, which can be even more difficult to do as a large group :)
Please see inline.

Thank you,
Sarah (as a participant)

>> - The draft aims to replace RFC3511, but expands scope past 
>> Firewalls, to "next generation security devices". I'm not finding a 
>> definition of what a "next generation security device is", nor an 
>> exhaustive list of the devices covered in this draft. A list that 
>> includes is nice, but IMO not enough to cover what would be 
>> benchmarked here - I'd prefer to see a definition and an exhaustive list.
>> 

SB// I think there are 2 types of devices called out; I'm not seeing a
definition of what a "NG security device" is, and I'm not comfortable with a
draft that has a blanket to encompass what would come later. Who knows what
new characteristics would arrive with that new device? I think the scope
here is best suited for the devices we know about today and can point to and
say we're applying knowledgeable benchmarking tests against.

>> We believe these two points qualifying the definition of next 

SB// I strongly disagree. Well, I mean OK, for active inline devices maybe
this is OK, but to say that the only way a device can be "NG" is to be
active/inline, I disagree with. And if there is, have we gathered all of the
items we'd want to actively test for in that case? For example, what about
their abilities to handle traffic when a failure occurs? (fail open/closed).
What about alerts and detections and the whole federation of tests around
positives/false positives/false negatives, etc? I'm onboard with expanding
the scope, but then we have to do the devices benchmarking justice, and I
feel we're missing a lot here.

>> three together in the first place; on the other, active inline 
>> devices typically include additional functions to allow 
>> administrators to control what happens to packets in the case of 
>> failure, and I don't see those test cases included here.
>> 
>> [bpm] This draft focuses on ""in-line"" mode security devices only. 
>> We will describe this in section 4 in more detail.

>> 
>> [bpm] Additionally, the draft focuses mainly on performance tests. 
>> The DUT must be configured in ""fail close"" mode. We will describe 
>> this under section 4. Any failure scenarios like ""fail open"" mode is
out of scope.

SB// OK, but I think an RFC that is going to encompass this device under the
"NG security devices" classification is missing out on large portions of
what customers will want to test. It'll also beg for another draft to cover
them, and then I'm not sure we're serving the industry as well as we could. 

>> reference test or baseline test. "

SB// Thank you for the clarification. I think there's still a concern there.
Who defines what "negative impact" is? You're traversing at least another L2
or L3 step in the network with each bump, which contributes some amount of
latency. If they don't serve in control plane decisions and are passively
passing data on, then we could consider removing them from the setup and
removing the potential skew on results.

>> There are options for multiple emulated routers. As a tester, I 
>> expect to see a specific, proscribed test bed that I should configure 
>> and test against.
>> 
>> However. We added emulated routers as optional in option 1. The 
>> reason for
>> that:
>> The previous test suggests you avoid routers and switches in the 
>> topo, but then there are emulated ones here. I'm curious as to what 
>> [bpm] Comparing real router, the emulated router gives more 
>> advantages for
>> L7 testing.
>> 
>> [bpm] - Emulated router doesn't add latency. Even if it adds delay 
>> due to the routing process, the test equipment can report the added 

SB// Maybe I'm missing something here - a device can't perform a function
for free, right? Even if it's impact is negligible, it's an impact of some
sort. We're saying the emulated router is doing the routing - OK - but I
think the same thing applies to the physical router - how do you know what
else the emulated router is doing? if the test gear can call out the
latency, I'd like to see clarification around how it's doing that and
distinguishing the latency introduced by Device A, versus Device B, versus
the DUT, etc.

>> [bpm] Also, we described in Option 1, that the external devices are 
>> if there is need to aggregate the interfaces of the tester or DUT.  
>> For an example, DUT has 2 Interfaces, but tester need to use it's 4 

SB// Thanks!

>> - Proscribing a list of features that have to be enabled for the 
>> test, or at least more than 1, feels like a strange choice here - I'd 
>> have expected tests cases that either test the specific features one 
>> at a time, or suggest several combinations, but that ultimately, we'd 
>> tell the tester to document WHICH features were enabled, to make the 
>> test cases repeatable? This allows the tester to apply a same set of 
>> apples to apples configurations to different vendor gear, and omit 
>> the 1 feature that doesn't exist on a different NGFW (for example), but
hold a baseline that could be tested.
>> 
>> - Table 2: With the assumption that NGIPS/IDS are required to have 
>> the features under "recommended", I disagree with this list. For 
>> example, some customers break and inspect at the tap/agg layer of the 
>> network - in this case, the feed into the NGIDS might be decrypted, 
>> and there's no need to enable SSL inspection, for example.
>> 

SB// OK...I'm not sure this addresses the feedback though :) A NGFW for sure
will do break/inspect as well, right?

SB// See comment above.

SB// Thanks!

>> - Can you expand on "disposition of all flows of traffic are logged" 
>> - what's meant here specifically, and why do they have to be logged?
>> number with minimal DUT/SUT configuration. In contrast, it aims to 
>> get reasonable performance number with realistic DUT configuration. 
>> The realistic configuration can vary based on DUT/SUT deployment
scenario.
>> [bpm] "Disposition of all flows of traffic are logged": means that 
>> the DUT/SUT need to log all the traffic at the flow level not each
packet.

SB// Thanks!

SB// Thanks!

>> equipment manufacturers, I can't possibly begin to guess which ones 
>> of them would conform to this - or even if they'd answer these questions.

SB// OK, that's really good to know, but there are plenty of us working with
and looking for more cost effective options to Ixia and Spirent. :) I think
the expansion would be good here.

SB// Thanks!

>> phases in the traffic load profile. The traffic load profile 
>> described in the draft is the common profile mostly used for Layer 7
testing.
>> routing functions between the system under test and the test 
>> equipment do not limit the performance of the traffic generator."
>> 

SB// Thanks!

SB// Thanks!

>> 
>> - 6.1 - I would suggest that the test report include the 
>> configuration of ancillary devices on both client/server side as well
>> 

SB// I think including them assists greatly in the repeatability of the
testing, for what it's worth.

>> determining the "initial inspected throughput"? Maybe I missed that 
>> in the document overall, but it's not clear to me where these KPIs 
>> are collected? I suggest this be called out.
>> 

SB// Thanks!

>> relevant application traffic mix for this test.  The details about 
>> the traffic mix MUST be documented in the report.  At least the 
>> following traffic mix details MUST be documented and reported together
with the test results:

---

From: "MORTON JR., AL" <[email-masked]>
Date: Tue, 13 Jul 2021 15:00:37 +0000

Hi Brian,

Our next opportunity to discuss this draft and comment resolution is during the BMWG session at IETF-111.
Let's try to make that work for everyone.

thanks!
Al

> From: [email-masked] <[email-masked]>
> Sent: Tuesday, July 13, 2021 10:44 AM
> To: 'Sarah Banks' <[email-masked]>; 'MORTON, ALFRED C (AL)'
> <[email-masked]>
> Cc: [email-masked]; [email-masked]; [email-masked];
> 'Bala Balarajah' <[email-masked]>; 'Carsten Rossenhoevel'
> <[email-masked]>; 'Christopher Brown' <[email-masked]>; 'Jack, Mike'
> <[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]>; 'Timothy
> Carlin' <[email-masked]>; 'Timothy Otto' <[email-masked]>
> Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance
>

---

From: <[email-masked]>
Date: Tue, 13 Jul 2021 11:17:00 -0400

Thanks for your response Al.

Given that the BMWG session is only 9 working days away I doubt very much we
will be able to get together as a group on our side and work through the
remaining issues. It might be possible to get our response out by the end of
next week, but I don't think that will provide enough time for it to be
reviewed and a response formulated by the meeting on July 26th.

I think it would be preferable to schedule an interim. 

Brian

-----Original Message-----
From: MORTON JR., AL <[email-masked]> 
Sent: Tuesday, July 13, 2021 11:01 AM
To: [email-masked]; 'Sarah Banks' <[email-masked]>; 'MORTON,
ALFRED C (AL)' <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked];
'Bala Balarajah' <[email-masked]>; 'Carsten Rossenhoevel'
<[email-masked]>; 'Christopher Brown' <[email-masked]>; 'Jack, Mike'
<[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]>;
'Timothy Carlin' <[email-masked]>; 'Timothy Otto' <[email-masked]>
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Brian,

Our next opportunity to discuss this draft and comment resolution is during
the BMWG session at IETF-111.
Let's try to make that work for everyone.

thanks!
Al

> Cc: [email-masked]; [email-masked]; 
> [email-masked]; 'Bala Balarajah' <[email-masked]>;
'Carsten Rossenhoevel'
> <[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]>; 
> 'Timothy Carlin' <[email-masked]>; 'Timothy Otto' 
> <[email-masked]>
> Good to see the issues are being whittled down. Of the remaining five 
> or six issues I doubt we will have a response prior to IETF 111.
> 
> Al,
> 
> patience. I also appreciate the time and effort that went into the 
> reply to my comments, which can be even more difficult to do as a 
> large group :) Please see inline.
> 
> Thank you,
> Sarah (as a participant)
> 
> >>
> >> Hi Sarah,
> >>
> >> [bpm] "We avoid limiting the draft by explicitly adding a list of 
> >> NG security devices currently available in the market only. In the 
> definition of what a "NG security device" is, and I'm not comfortable 
> with a draft that has a blanket to encompass what would come later. 
> Who knows what new characteristics would arrive with that new device? 
> I think the scope here is best suited for the devices we know about 
> today and can point to and say we're applying knowledgeable benchmarking
tests against.
> >> security device can have ( RFC 3511 doesn't have such a list). 
> >> Also, we will describe in the draft that the security devices must 
> >> be
> configured ""in-line"" mode.
> SB// I strongly disagree. Well, I mean OK, for active inline devices 
> maybe this is OK, but to say that the only way a device can be "NG" is 
> to be active/inline, I disagree with. And if there is, have we 
> gathered all of the items we'd want to actively test for in that case? 
> For example, what about their abilities to handle traffic when a failure
occurs? (fail open/closed).
> What about alerts and detections and the whole federation of tests 
> around positives/false positives/false negatives, etc? I'm onboard 
> with expanding the scope, but then we have to do the devices 
> benchmarking justice, and I feel we're missing a lot here.
> 
> >> category as passive devices. On one hand, I'm not sure I'd lump 
> >> these three together in the first place; on the other, active 
> >> inline devices typically include additional functions to allow 
> SB// OK, but I think an RFC that is going to encompass this device 
> under the "NG security devices" classification is missing out on large 
> portions of what customers will want to test. It'll also beg for 
> another draft to cover them, and then I'm not sure we're serving the
industry as well as we could.
> >> switches) are used in the test bed, they should not negatively 
> >> impact
> DUT/SUT performance.
> >> Considerations"") which recommends a pre-test.  We can rename this 
> >> as reference test or baseline test. "
> >>
> 
> Who defines what "negative impact" is? You're traversing at least 
> another L2 or L3 step in the network with each bump, which contributes 
> some amount of latency. If they don't serve in control plane decisions 
> and are passively passing data on, then we could consider removing 
> them from the setup and removing the potential skew on results.
> 
> 
> >> expect to see a specific, proscribed test bed that I should 
> >> configure and test against.
> >>
> >> Some type of security devices for some deployment scenarios 
> >> requires routers between test client/server and the DUT (e.g., 
> >> NGFW) and some DUT/SUT doesn't need router (e.g. NGIPS )
> >>
> SB// Maybe I'm missing something here - a device can't perform a 
> function for free, right? Even if it's impact is negligible, it's an 
> impact of some sort. We're saying the emulated router is doing the 
> routing - OK - but I think the same thing applies to the physical 
> router - how do you know what else the emulated router is doing? if 
> the test gear can call out the latency, I'd like to see clarification 
> around how it's doing that and distinguishing the latency introduced 
> by Device A, versus Device B, versus the DUT, etc.
> 
> 
> >>
> >> functionality on the test bed (e.g., if we used very small number 
> >> of clients and server IPs or the DUT operates in Layer 2 mode), it 
> >> can be
> ignored.
> >>
> >> interfaces to achieve the performance. So here we need 
> >> switch/router to aggregate tester interface from 4 to 2.
> >>
> >> might be saying that RECOMMENDED is the "must have enabled" 
> >> features, where as optional is at your discretion, but would 
> >> suggest that you make that clear)
> >>
> >> test, or at least more than 1, feels like a strange choice here - 
> >> I'd have expected tests cases that either test the specific 
> >> features one at a time, or suggest several combinations, but that 
> >> ultimately, we'd tell the tester to document WHICH features were 
> >> enabled, to make the test cases repeatable? This allows the tester 
> >> to apply a same set of apples to apples configurations to different 
> >> vendor gear, and omit the 1 feature that doesn't exist on a 
> >> different NGFW (for example), but
> >> example, some customers break and inspect at the tap/agg layer of 
> >> the network - in this case, the feed into the NGIDS might be 
> >> decrypted, and there's no need to enable SSL inspection, for example.
> >>
> SB// OK...I'm not sure this addresses the feedback though :) A NGFW 
> for sure will do break/inspect as well, right?
> 
> >>
> >> - Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. 
> >> This behaviour might be suitable for an NGIPS, but the NGIDS is not 
> >> a bump on the wire, and often isn't decrypting and re-encrypting the
traffic.
> >> blocking anything, but agree that it might tell you that it 
> >> happened
> after the fact.
> >>
> >> [bpm] Not sure why you are asking this as mitigate is not an 
> >> uncommon term/word.
> >>
> >> [bpm] We intentionally recommended enabling logging which will 
> >> impact the performance. The draft is not aiming to get high 
> >> performance number with minimal DUT/SUT configuration. In contrast, 
> >> it aims to get reasonable performance number with realistic DUT
configuration.
> >> [bpm] We will add more clarification for the meaning of 
> >> "disposition of all flows of traffic are logged".
> >>
> 
> SB// Thanks!
> 
> >> - It might be helpful to testers to say something like "look, 
> >> here's one suggested set of ACLs. If you're using them, great, 
> >> reference that, but otherwise, make note of the ACLs you use, and 
> >> use the same ones for repeatable testing".
> >>
> >> discussion around why they're chosen - that color could be useful 
> >> to the reader.
> >>
> >> [bpm] We will add some more clarification that these are the 
> >> default number used in most of the client operating systems currently.
> >>
> 
> SB// Thanks!
> 
> >> expect to cover, or hopefully which (open source/generally 
> >> available) test tools or emulators could be considered for use as
examples.
> SB// OK, that's really good to know, but there are plenty of us 
> working with and looking for more cost effective options to Ixia and 
> Spirent. :) I think the expansion would be good here.
> 
> >>
> >> - 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a 
> >> configuration option here?
> >>
> >> [bpm] Yes
> >>
> >> distinct phase before moving on to the next. We will remove the 
> >> word "distinct" from the 1st paragraph in section 4.3.4.
> 
> SB// Thanks!
> 
> >>
> >> open as to how one confirmed that the ancillary switching, or 
> >> routing functions didn't limit the performance, particularly the 
> >> virtualized
> components?
> >>
> >> [bpm] Here we discuss the traffic generator performance, and this 
> >> can be confirmed by doing reference test.
> >>
> >> - 5.3 - this is a nice assertion but again, how do I reasonably 
> >> make the assertion?
> >>
> >> [bpm] We will change the word from "Assertion" to "Ensure". Also, 
> >> we will add more clarity about reference testing.
> 
> 
> SB// Thanks!
> 
> >>
> >> doing reference test. We will add this under the section 5 "Test 
> >> bed
> consideration".
> >>
> 
> >> packet loss in stateful traffic. Instead of packet loss, the 
> >> stateful traffic has retransmissions.
> >>
> >> [bpm] This is described in section7.1.1  (2nd paragraph). We will 
> >> add the word "relevant" in the 1st sentence of the 2nd pragraph.so 
> >> the sentence will be "Based on customer use case, users can choose 
> >> the relevant application traffic mix for this test.  The details 
> >> about the traffic mix MUST be documented in the report.  At least 
> >> the following traffic mix details MUST be documented and reported 
> >> together
> with the test results:
> 
> >> [bpm] These numbers are synonymous with the zero-packet loss 
> >> criteria for [RFC2544] Throughput and recognize the additional 
> >> complexity of application layer performance. This was agreed by the
IETF BMWG.
> >> [bpm] Because new cipher suites are frequently developed. Given 
> >> that the draft will not be easily updated once it is accepted as an 
> >> RFC we wanted to ensure there was flexibility to use future 
> >> developed cipher
> suites.
> >>

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Tue, 13 Jul 2021 17:19:22 +0100

The question is, how much more time will be required to discuss this 
topic in an IETF meeting (whether regular or interim)?
Al (as chair) and Sarah (as contributor), what do you think? Would it 
fit into the regular BMWG meeting at IETF111 at all anyway?

My personal view is that it will take a considerable time to explain the 
positions and reach consensus, specifically related to the rather 
fundamental questions about the scope of the next-gen security device 
industry.

Sarah, is it acceptable to kindly ask you to prepare constructive 
editorial suggestions how to resolve her concerns?

The WGLC process has already been delayed by yet another IETF meeting 
cycle, and I would like to ask for your support to avoid any further 
unnecessary delays.  I would like to remind everyone of RFC 7154 
<https://tools.ietf.org/search/rfc7154> (IETF Guidelines for Conduct) 
section 2.4.

Best regards, Carsten

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked], https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025

---

From: "MORTON JR., AL" <[email-masked]>
Date: Tue, 13 Jul 2021 17:13:14 +0000

Hi Carsten and all,
please see replies below, [acm]

From: Carsten Rossenhoevel <[email-masked]>
Sent: Tuesday, July 13, 2021 12:19 PM
To: [email-masked]; MORTON JR., AL <[email-masked]>; 'Sarah Banks' <[email-masked]>; 'MORTON, ALFRED C (AL)' <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; 'Bala Balarajah' <[email-masked]>; 'Christopher Brown' <[email-masked]>; 'Jack, Mike' <[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]>; 'Timothy Carlin' <[email-masked]>; 'Timothy Otto' <[email-masked]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

The question is, how much more time will be required to discuss this topic in an IETF meeting (whether regular or interim)?
Al (as chair) and Sarah (as contributor), what do you think?  Would it fit into the regular BMWG meeting at IETF111 at all anyway?
[acm] Given the current agenda requests and our session scheduled for 120 minutes, we could easily allocate 60 minutes (maybe more) to resolve issues raised on a mature working group draft like this. Perhaps not all of the issues, but the most important ones! Others may need to be deferred to follow-up activity.
In my view, resolving comments at a faster pace than possible with e-mail exchange in exactly what our IETF sessions are intended to accomplish. While the WG drafts have time priority, other discussions took place on the list since we last met and it seems fair to allocate some time to those as well. The good news is that we have a sufficiently active list to more-than-justify a live session during this IETF. That’s the next step.
Al
bmwg co-chair

My personal view is that it will take a considerable time to explain the positions and reach consensus, specifically related to the rather fundamental questions about the scope of the next-gen security device industry.

Sarah, is it acceptable to kindly ask you to prepare constructive editorial suggestions how to resolve her concerns?

The WGLC process has already been delayed by yet another IETF meeting cycle, and I would like to ask for your support to avoid any further unnecessary delays.  I would like to remind everyone of RFC 7154<https://urldefense.com/v3/__https:/tools.ietf.org/search/rfc7154__;!!BhdT!x_nyyuYbLuR_bxaZQzRebn-62Pq05wFObCEJ0-gdObGoaHM0WyFnsfztUjSU$> (IETF Guidelines for Conduct) section 2.4.

Best regards, Carsten

On 13.07.2021 16:17, [email-masked]<mailto:[email-masked]> wrote:

Brian

-----Original Message-----

From: MORTON JR., AL <[email-masked]><mailto:[email-masked]>

To: [email-masked]<mailto:[email-masked]>; 'Sarah Banks' <[email-masked]><mailto:[email-masked]>; 'MORTON,

ALFRED C (AL)' <[email-masked]><mailto:[email-masked]>

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>;

'Bala Balarajah' <[email-masked]><mailto:[email-masked]>; 'Carsten Rossenhoevel'

<[email-masked]><mailto:[email-masked]>; 'Christopher Brown' <[email-masked]><mailto:[email-masked]>; 'Jack, Mike'

<[email-masked]><mailto:[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]><mailto:[email-masked]>;

'Timothy Carlin' <[email-masked]><mailto:[email-masked]>; 'Timothy Otto' <[email-masked]><mailto:[email-masked]>

Hi Brian,

thanks!

Al

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>

To: 'Sarah Banks' <[email-masked]><mailto:[email-masked]>; 'MORTON, ALFRED C (AL)'

<[email-masked]><mailto:[email-masked]>

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>;

[email-masked]<mailto:[email-masked]>; 'Bala Balarajah' <[email-masked]><mailto:[email-masked]>;

'Carsten Rossenhoevel'

<[email-masked]><mailto:[email-masked]>; 'Christopher Brown' <[email-masked]><mailto:[email-masked]>; 'Jack, Mike'

<[email-masked]><mailto:[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]><mailto:[email-masked]>;

'Timothy Carlin' <[email-masked]><mailto:[email-masked]>; 'Timothy Otto'

<[email-masked]><mailto:[email-masked]>

Subject: RE: [bmwg] WGLC on New version of

draft-ietf-bmwg-ngfw-performance

Thanks Sarah,

Al,

Brian

-----Original Message-----

From: Sarah Banks <[email-masked]><mailto:[email-masked]>

To: [email-masked]<mailto:[email-masked]>

Cc: [email-masked]<mailto:[email-masked]>

Subject: Re: [bmwg] WGLC on New version of

draft-ietf-bmwg-ngfw-performance

Hi Brian et al,

Thank you,

Sarah (as a participant)

Hi Sarah,

benchmarked here - I'd prefer to see a definition and an exhaustive

list.

tests against.

be

generation security.

this under section 4. Any failure scenarios like ""fail open"" mode

is

out of scope.

contribute to network latency or throughput issues, including the

DUTs

impact

DUT/SUT performance.

SB// Thank you for the clarification. I think there's still a concern

there.

- Option 1: It'd be nice to see a specific, clean, recommended test

bed.

reason for

that:

advantages for

L7 testing.

[bpm] - Emulated routers simply do routing function only. But in a

"real"

process

the security devices will not be directly connected with the

clients.)

can be

ignored.

[bpm] See above.

equipment to the DUT and remove extraneous devices from potential

influence on results?

[bpm] See above.

and recommended, RFC2119. We already referenced this under the

section

2 "Requirements".

SB// Thanks!

traffic.

SB// See comment above.

happened

after the fact.

SB// Thanks!

uncommon term/word.

configuration.

scenario.

packet.

SB// Thanks!

- ACLs wouldn't apply to an IDS because IDS's aren't blocking

traffic

:)

to the reader.

SB// Thanks!

- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK).

and"

that should be changed.

[bpm] Thank you.

of them would conform to this - or even if they'd answer these

questions.

examples.

[bpm] We extensively discussed with Ixia and Spirent about this

section.

testing the NGIPS?

Spirent confirmed this.

[bpm] Yes

clarify why this is a requirement, and why staggering them is

specifically rejected?

SB// Thanks!

described in the draft is the common profile mostly used for Layer

7

testing.

virtualized

components?

about reference test.

SB// Thanks!

make the assertion?

SB// Thanks!

configuration of ancillary devices on both client/server side as

well

bed

consideration".

SB// I think including them assists greatly in the repeatability of

the testing, for what it's worth.

SB// Thanks!

together

with the test results:

- 7.1.3.4 - This looks a bit like conformance testing -  Why does

item

IETF BMWG.

developed cipher

suites.

--

Carsten Rossenhövel

[email-masked]<mailto:[email-masked]>, https://www.eantc.de<https://urldefense.com/v3/__https:/www.eantc.de__;!!BhdT!x_nyyuYbLuR_bxaZQzRebn-62Pq05wFObCEJ0-gdObGoaHM0WyFnsSV1q7Jm$>

EU VAT No: DE812824025

---

From: Sarah Banks <[email-masked]>
Date: Tue, 13 Jul 2021 10:23:37 -0700

Carsten,
     While I find some of the comments below a tad offensive, I want to focus in a bit on the draft text itself. For me, the disconnect starts in the Abstract:

This document provides benchmarking terminology and methodology for
   next-generation network security devices including next-generation
   firewalls (NGFW), next-generation intrusion detection and prevention
   systems (NGIDS/NGIPS) and unified threat management (UTM)
   implementations.  This document aims to strongly improve the
   applicability, reproducibility, and transparency of benchmarks and to
   align the test methodology with today's increasingly complex layer 7
   security centric network application use cases.  The main areas
   covered in this document are test terminology, test configuration
   parameters, and benchmarking methodology for NGFW and NGIDS/NGIPS to
   start with.

I read that and my takeaway is that the draft wants to be the RFC for all next generation security devices. I am willing to accept your assertion that perhaps I don't understand the fundamental concept of what a NG security device is, and in my feedback I asked you to point me at such a definition. What I was gently pointing out is that I'm not sure one exists, and that the text might be best suited to include one. The draft points out required features to be enabled, and I wonder, if I don't have that feature on my device, are you saying I'm not an NG security device? Am I disqualified from such a test?

Which leads me to my general concern - I am not supportive of a draft that covers "NG security devices", describes tests for 2 of them, and that's it. We don't know what's next to be invented, so having a blanket device that covers them seems strange to me at best. The abstract itself describes a methodology for testing "unified threat management" implementations, and while the tests to be performed are well described, they're lighter than I'd expect. Further, if I were an NGIPS in a bakeoff, wouldn't I want to test the performance of some of the features from 4.2 to be benchmarked? Do you see what I mean? I realize that this might not be the point of your testing here, but I'm trying to share that were I to come to this draft as written, I might reasonably expect that the draft cover some or all of those features with test cases. 

Ways forward: I believe a definition of, or pointing to such, of NG <x> is required. While I'm not entirely convinced that the draft actually covers the entire set of features I'd want to test on a NGFW or on an IPS, I'd be supportive of a draft that covered those devices, since test cases for them are called out. I am not supportive of a draft that blanket-covers what's next, and I think we should strongly consider adding more test cases that allow someone to more thoroughly benchmark the NGFW or IPS, as the abstract suggests.

Kind regards,
Sarah

> On Jul 13, 2021, at 9:19 AM, Carsten Rossenhoevel <[email-masked]> wrote:
> 
> The WGLC process has already been delayed by yet another IETF meeting cycle, and I would like to ask for your support to avoid any further unnecessary delays.  I would like to remind everyone of RFC 7154 <https://tools.ietf.org/search/rfc7154> (IETF Guidelines for Conduct) section 2.4. 
> 
> Best regards, Carsten
> 
> 
> 
> 
> 
> On 13.07.2021 16:17, [email-masked] <mailto:[email-masked]> wrote:
>> From: MORTON JR., AL <[email-masked]> <mailto:[email-masked]> 
>> To: [email-masked] <mailto:[email-masked]>; 'Sarah Banks' <[email-masked]> <mailto:[email-masked]>; 'MORTON,
>> ALFRED C (AL)' <[email-masked]> <mailto:[email-masked]>
>> Cc: [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>;
>> 'Bala Balarajah' <[email-masked]> <mailto:[email-masked]>; 'Carsten Rossenhoevel'
>> <[email-masked]> <mailto:[email-masked]>; 'Christopher Brown' <[email-masked]> <mailto:[email-masked]>; 'Jack, Mike'
>> <[email-masked]> <mailto:[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]> <mailto:[email-masked]>;
>> 'Timothy Carlin' <[email-masked]> <mailto:[email-masked]>; 'Timothy Otto' <[email-masked]> <mailto:[email-masked]>
>>> From: [email-masked] <mailto:[email-masked]> <[email-masked]> <mailto:[email-masked]>
>>> To: 'Sarah Banks' <[email-masked]> <mailto:[email-masked]>; 'MORTON, ALFRED C (AL)'
>>> <[email-masked]> <mailto:[email-masked]>
>>> Cc: [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>; 
>>> [email-masked] <mailto:[email-masked]>; 'Bala Balarajah' <[email-masked]> <mailto:[email-masked]>;
>> 'Carsten Rossenhoevel'
>>> <[email-masked]> <mailto:[email-masked]>; 'Christopher Brown' <[email-masked]> <mailto:[email-masked]>; 'Jack, Mike'
>>> <[email-masked]> <mailto:[email-masked]>; 'Ryan Liles (ryliles)' <[email-masked]> <mailto:[email-masked]>; 
>>> 'Timothy Carlin' <[email-masked]> <mailto:[email-masked]>; 'Timothy Otto' 
>>> <[email-masked]> <mailto:[email-masked]>
>>> From: Sarah Banks <[email-masked]> <mailto:[email-masked]>
>>> To: [email-masked] <mailto:[email-masked]>
>>> Cc: [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>, https://www.eantc.de <https://www.eantc.de/>
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: <[email-masked]>
Date: Tue, 13 Jul 2021 13:45:07 -0400

Sarah,

I understand your comments.

Could you say whether changing the draft to explicitly including only NGFW and NGIPS as the target would impact your follow-on comments.

And with respect to your comment regarding the number of test cases, could you tell us what it is you would like to see. I ask because we have had the same test cases enumerated since the initial draft. This is the first time, I believe, that a suggestion has been raised that there are not enough. We think there are. However, I understand you don’t. But it would be helpful to us if you could detail what you think is needed in addition to the test cases currently included.

Brian

From: Sarah Banks <[email-masked]> 
Sent: Tuesday, July 13, 2021 1:24 PM
To: Carsten Rossenhoevel <[email-masked]>
Cc: [email-masked]; ALFRED MORTON <[email-masked]>; MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]; Jack, Mike <[email-masked]>; Timothy Otto <[email-masked]>; Bala Balarajah <[email-masked]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Carsten,

Kind regards,

Sarah

On Jul 13, 2021, at 9:19 AM, Carsten Rossenhoevel <[email-masked] <mailto:[email-masked]> > wrote:

Best regards, Carsten

 

Brian
 
-----Original Message-----
From: MORTON JR., AL  <mailto:[email-masked]> <[email-masked]> 
Sent: Tuesday, July 13, 2021 11:01 AM
To: [email-masked] <mailto:[email-masked]> ; 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON,
ALFRED C (AL)'  <mailto:[email-masked]> <[email-masked]>
Cc: [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> ;
'Bala Balarajah'  <mailto:[email-masked]> <[email-masked]>; 'Carsten Rossenhoevel'
 <mailto:[email-masked]> <[email-masked]>; 'Christopher Brown'  <mailto:[email-masked]> <[email-masked]>; 'Jack, Mike'
 <mailto:[email-masked]> <[email-masked]>; 'Ryan Liles (ryliles)'  <mailto:[email-masked]> <[email-masked]>;
'Timothy Carlin'  <mailto:[email-masked]> <[email-masked]>; 'Timothy Otto'  <mailto:[email-masked]> <[email-masked]>
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance
 
Hi Brian,

thanks!
Al

-----Original Message-----
From: [email-masked] <mailto:[email-masked]>   <mailto:[email-masked]> <[email-masked]>
Sent: Tuesday, July 13, 2021 10:44 AM
To: 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON, ALFRED C (AL)'
 <mailto:[email-masked]> <[email-masked]>
Cc: [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> ; 
[email-masked] <mailto:[email-masked]> ; 'Bala Balarajah'  <mailto:[email-masked]> <[email-masked]>;

'Carsten Rossenhoevel'

 <mailto:[email-masked]> <[email-masked]>; 'Christopher Brown'  <mailto:[email-masked]> <[email-masked]>; 'Jack, Mike'
 <mailto:[email-masked]> <[email-masked]>; 'Ryan Liles (ryliles)'  <mailto:[email-masked]> <[email-masked]>; 
'Timothy Carlin'  <mailto:[email-masked]> <[email-masked]>; 'Timothy Otto' 
 <mailto:[email-masked]> <[email-masked]>
Subject: RE: [bmwg] WGLC on New version of 
draft-ietf-bmwg-ngfw-performance

Thanks Sarah,

Al,

Brian

-----Original Message-----
From: Sarah Banks  <mailto:[email-masked]> <[email-masked]>
Sent: Monday, July 12, 2021 5:20 PM
To: [email-masked] <mailto:[email-masked]> 
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [bmwg] WGLC on New version of 
 
Hi Brian et al,
 
Thank you,
Sarah (as a participant)

Hi Sarah,

list.

tests against.

 

 

out of scope.

 

DUT/SUT performance.

there.

bed.

 

 

"real"

 

 

ignored.

 
[bpm] See above.

influence on results?

[bpm] See above.

SB// Thanks!

 

traffic.

SB// See comment above.

after the fact.

SB// Thanks!

 

 

configuration.

scenario.

packet.

SB// Thanks!

 

 

SB// Thanks!

and"

that should be changed.
 
[bpm] Thank you.

questions.

examples.

section.

 

[bpm] Yes

specifically rejected?

SB// Thanks!

testing.

components?

 

SB// Thanks!

SB// Thanks!

consideration".

 

SB// Thanks!

with the test results:

 

IETF BMWG.

suites.

 

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked] <mailto:[email-masked]> , https://www.eantc.de <https://www.eantc.de/> 

_______________________________________________
bmwg mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Wed, 14 Jul 2021 07:15:18 +0100

Sarah,

At this point I think it has become quite clear what you /don't 
/support.  As an editor, referencing the Code of Conduct was meant to 
inspire you to provide /constructive suggestions what you actually would 
support/.  The editors have spent a considerable time to review your 
previous set of comments and have not taken the task lightly.  However, 
I regret to say, we have been unable to come up with reasonable 
suggestions for some of your comments because we did not know in which 
direction you wanted to inspire us with your rejection of parts of the 
document.  We tried, but seemingly failed.

Innovation is best fueled by reviewers who contribute to improve 
drafts.  Indeed, continually being asked to provide suggestions is 
arduous: It requires considering the consequences of a comment, and 
creates more work for the contributor the more fundamental the comments 
are.  I regret if you find this offensive; it is part of the IETF process.

For the upcoming IETF 111 BMWG session, I kindly ask all subscribers to 
this mailing list to consider the following:

- If you would like to have a definition of a "next-gen security 
device", please be prepared to suggest specific text that is in line 
with the contents of the document

- If you would like to add test cases, please prepare at least a list of 
test purposes ("verify that ...") and key performance indicators (KPIs) 
you suggest to be measured in addition to what the draft covers today.  
We can then discuss the desired scope extension during the session.

Thank you very much,
Carsten

On 13.07.2021 18:23, Sarah Banks wrote:
> Carsten,
>      While I find some of the comments below a tad offensive, I want 
> to focus in a bit on the draft text itself. For me, the disconnect 
> starts in the Abstract:
>
> I read that and my takeaway is that the draft wants to be the RFC for 
> all next generation security devices. I am willing to accept your 
> assertion that perhaps I don't understand the fundamental concept of 
> what a NG security device is, and in my feedback I asked you to point 
> me at such a definition. What I was gently pointing out is that I'm 
> not sure one exists, and that the text might be best suited to include 
> one. The draft points out required features to be enabled, and I 
> wonder, if I don't have that feature on my device, are you saying I'm 
> not an NG security device? Am I disqualified from such a test?
>
> Which leads me to my general concern - I am not supportive of a draft 
> that covers "NG security devices", describes tests for 2 of them, and 
> that's it. We don't know what's next to be invented, so having a 
> blanket device that covers them seems strange to me at best. The 
> abstract itself describes a methodology for testing "unified threat 
> management" implementations, and while the tests to be performed are 
> well described, they're lighter than I'd expect. Further, if I were an 
> NGIPS in a bakeoff, wouldn't I want to test the performance of some of 
> the features from 4.2 to be benchmarked? Do you see what I mean? I 
> realize that this might not be the point of your testing here, but I'm 
> trying to share that were I to come to this draft as written, I might 
> reasonably expect that the draft cover some or all of those features 
> with test cases.
>
> Ways forward: I believe a definition of, or pointing to such, of NG 
> <x> is required. While I'm not entirely convinced that the draft 
> actually covers the entire set of features I'd want to test on a NGFW 
> or on an IPS, I'd be supportive of a draft that covered those devices, 
> since test cases for them are called out. I am not supportive of a 
> draft that blanket-covers what's next, and I think we should strongly 
> consider adding more test cases that allow someone to more thoroughly 
> benchmark the NGFW or IPS, as the abstract suggests.
>
> Kind regards,
> Sarah
>
>
>> On Jul 13, 2021, at 9:19 AM, Carsten Rossenhoevel <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Al (as chair) and Sarah (as contributor), what do you think?  Would 
>> it fit into the regular BMWG meeting at IETF111 at all anyway?
>>
>> My personal view is that it will take a considerable time to explain 
>> the positions and reach consensus, specifically related to the rather 
>>> From: MORTON JR., AL<[email-masked]>  
>>> To:[email-masked]; 'Sarah Banks'<[email-masked]>; 'MORTON,
>>> ALFRED C (AL)'<[email-masked]>
>>> Cc:[email-masked];[email-masked];[email-masked];
>>> 'Bala Balarajah'<[email-masked]>; 'Carsten Rossenhoevel'
>>> <[email-masked]>; 'Christopher Brown'<[email-masked]>; 'Jack, Mike'
>>> <[email-masked]>; 'Ryan Liles (ryliles)'<[email-masked]>;
>>> 'Timothy Carlin'<[email-masked]>; 'Timothy Otto'<[email-masked]>
>>>> From:[email-masked]  <[email-masked]>
>>>> To: 'Sarah Banks'<[email-masked]>; 'MORTON, ALFRED C (AL)'
>>>> <[email-masked]>
>>>> Cc:[email-masked];[email-masked];
>>>> [email-masked]; 'Bala Balarajah'<[email-masked]>;
>>> 'Carsten Rossenhoevel'
>>>> <[email-masked]>; 'Christopher Brown'<[email-masked]>; 'Jack, Mike'
>>>> <[email-masked]>; 'Ryan Liles (ryliles)'<[email-masked]>;
>>>> 'Timothy Carlin'<[email-masked]>; 'Timothy Otto'
>>>> <[email-masked]>
>>>> From: Sarah Banks<[email-masked]>
>>>> To:[email-masked]
>>>> Cc:[email-masked]
>> [email-masked],https://www.eantc.de
>>

---

From: Sarah Banks <[email-masked]>
Date: Fri, 16 Jul 2021 11:09:55 -0700

Hi Brian,
    Please see inline.

Thanks, 
Sarah (as a participant)

> On Jul 13, 2021, at 10:45 AM, <[email-masked]> <[email-masked]> wrote:
> 
> Sarah,
>  

SB// I support the draft as a draft that covers benchmarks that cover specifically NGFW and NGIPS, provided the scope is not open ended to cover the undefined "NG*" and future devices that we haven't invented yet. :)

> And with respect to your comment regarding the number of test cases, could you tell us what it is you would like to see. I ask because we have had the same test cases enumerated since the initial d…

SB// I can certainly understand the frustration with this comment at the late hour. At first, I was focused on "can the NGIDS piece fit into this draft as is?" and "does it cover the general scope of any upcoming NG* device?" - and once I sorted that out and settled on the idea that this draft makes a lot of sense in the NGFW/IPS sense, it occurred to me - if I wanted to benchmark one NGFW versus another, or one IPS versus another, there's a lot more here I'd like to test. The features called out in section 4.2 would certainly be items I'd want to understand when comparing different but like devices. This expectation is set from the very abstract itself - where we write, "This document provides benchmarking terminology and methodology for next-generation network security devices including next-generation firewalls (NGFW), next-generation intrusion detection and prevention systems (NGIDS/NGIPS) and unified threat management (UTM) implementations." It's to these features that I think test cases would meet the stated objective in the abstract. 

Last, we use a term that is not defined - NG* - and it begs the question, what characterizes a next generation device? Let me use an IDS as the example (I realize the IDS is out of scope now! But it highlights some of the questions) - is it performance (amount of traffic a device can ingest at ingress? the amount of records/meta data/flows/alerts/events it can send out per second?) Does it extend to an architecture? Does it cover a breadth of analyzers/protocols it can perceive and interpret? Would it include a stance on the effectiveness of the detections (positive/false positive hits)? I believe the due diligence on undefined terms here is on the authors, as I'm not sure I understand what's meant by an "NGIPS" - hence the question - and hence the reason I can't propose the text for the authors. 

> Brian
>  
> From: Sarah Banks <[email-masked] <mailto:[email-masked]>> 
> To: Carsten Rossenhoevel <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>; ALFRED MORTON <[email-masked] <mailto:[email-masked]>>; MORTON, ALFRED C (AL) <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; Jack, Mike <[email-masked] <mailto:[email-masked]>>; Timothy Otto <[email-masked] <mailto:[email-masked]>>; Bala Balarajah <[email-masked] <mailto:[email-masked]>>
>> On Jul 13, 2021, at 9:19 AM, Carsten Rossenhoevel <[email-masked] <mailto:[email-masked]>> wrote:
>>  
>> https://www.ietf.org/mailman/listinfo/bmwg <https://www.ietf.org/mailman/listinfo/bmwg>
>  
> https://www.ietf.org/mailman/listinfo/bmwg <https://www.ietf.org/mailman/listinfo/bmwg>

---

From: <[email-masked]>
Date: Fri, 16 Jul 2021 14:39:22 -0400

I’m confused. You have cited IDS as examples even though we have explicitly stated our that including IDS is out of scope. As we have not *yet* created an updated draft it does confuse things. We felt it would be a waste of time to create a new draft given several of our comments were still open.

I suppose since you “can’t propose” anything specific we will have to discuss it during the meeting. 

Thanks for taking the time to respond.

Brian

From: Sarah Banks <[email-masked]> 
Sent: Friday, July 16, 2021 2:10 PM
To: [email-masked]
Cc: Carsten Rossenhoevel <[email-masked]>; ALFRED MORTON <[email-masked]>; Jack, Mike <[email-masked]>; MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]; Timothy Otto <[email-masked]>; Bala Balarajah <[email-masked]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Brian,

    Please see inline.

Thanks, 

Sarah (as a participant)

On Jul 13, 2021, at 10:45 AM, <[email-masked] <mailto:[email-masked]> > <[email-masked] <mailto:[email-masked]> > wrote:

Sarah,

 

 

Brian

From: Sarah Banks <[email-masked] <mailto:[email-masked]> > 
Sent: Tuesday, July 13, 2021 1:24 PM
To: Carsten Rossenhoevel <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> ; ALFRED MORTON <[email-masked] <mailto:[email-masked]> >; MORTON, ALFRED C (AL) <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> ; Jack, Mike <[email-masked] <mailto:[email-masked]> >; Timothy Otto <[email-masked] <mailto:[email-masked]> >; Bala Balarajah <[email-masked] <mailto:[email-masked]> >
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Carsten,

Kind regards,

Sarah

Best regards, Carsten

 

Brian
 
-----Original Message-----
From: MORTON JR., AL  <mailto:[email-masked]> <[email-masked]> 
Sent: Tuesday, July 13, 2021 11:01 AM
To: [email-masked] <mailto:[email-masked]> ; 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON,
 
Hi Brian,

thanks!
Al

-----Original Message-----
From: [email-masked] <mailto:[email-masked]>   <mailto:[email-masked]> <[email-masked]>
Sent: Tuesday, July 13, 2021 10:44 AM
To: 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON, ALFRED C (AL)'
 <mailto:[email-masked]> <[email-masked]>
Cc: [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> ; 

'Carsten Rossenhoevel'

 
Thanks Sarah,

Al,

Brian

-----Original Message-----
From: Sarah Banks  <mailto:[email-masked]> <[email-masked]>
Sent: Monday, July 12, 2021 5:20 PM
To: [email-masked] <mailto:[email-masked]> 
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [bmwg] WGLC on New version of 
 
Hi Brian et al,
 
Thank you,
Sarah (as a participant)

Hi Sarah,

list.

tests against.

 

 

out of scope.

 

DUT/SUT performance.

there.

bed.

 

 

"real"

 

 

ignored.

 
[bpm] See above.

influence on results?

[bpm] See above.

SB// Thanks!

 

traffic.

SB// See comment above.

after the fact.

SB// Thanks!

 

 

configuration.

scenario.

packet.

SB// Thanks!

 

 

SB// Thanks!

and"

that should be changed.
 
[bpm] Thank you.

questions.

examples.

section.

 

[bpm] Yes

specifically rejected?

SB// Thanks!

testing.

components?

 

SB// Thanks!

SB// Thanks!

consideration".

 

SB// Thanks!

with the test results:

 

IETF BMWG.

suites.

 

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked], https://www.eantc.de <https://www.eantc.de/> 

_______________________________________________
bmwg mailing list
 <mailto:[email-masked]> [email-masked]
 <https://www.ietf.org/mailman/listinfo/bmwg> https://www.ietf.org/mailman/listinfo/bmwg

---

From: "MORTON JR., AL" <[email-masked]>
Date: Thu, 22 Jul 2021 18:39:09 +0000

Hi all,

I think there might be a way forward for the most recent discussions below, and this might be the most important area to agree since it pertains to the scope (even the title) of the draft/RFC we are working on.

Perhaps what we have in this draft is best described as a set of benchmarks that *test specific features* of next-gen firewalls (and some are applicable to IDS, too?).

Don’t claim to test all features of *any* device: the scope is the set of features for which benchmarks and methods have been developed. The scope defines what is in the document. If there is a need to add more tests/benchmarks to cover additional features later, another document can take-up that work easily.

Something like:

   *several features commonly found in*

   systems (NGIDS/NGIPS) and unified threat management (UTM) implementations.

   The features/benchmarks are < list >.

Also, I didn’t get hung-up on the term “Next-Generation” in the doc, because we were just finishing RFC3511 when I joined BMWG and so next-gen had an inherent meaning to me.  It might be somewhat awkward to refer to RFC 3511 in the explanation of the “next-generation”, but we clearly mean *modern* devices and using the term “modern” may help.

In any case, if we concentrate on the features we benchmark, it matters much less whether these features are part of existing, current, modern, or even next-generation devices.

I hope this helps,
Al
(as your friendly wg co-chair, trying to move the discussion forward productively)

From: [email-masked] <[email-masked]>
Sent: Friday, July 16, 2021 2:39 PM
To: 'Sarah Banks' <[email-masked]>
Cc: 'Carsten Rossenhoevel' <[email-masked]>; MORTON JR., AL <[email-masked]>; 'Jack, Mike' <[email-masked]>; 'MORTON, ALFRED C (AL)' <[email-masked]>; [email-masked]; 'Timothy Otto' <[email-masked]>; 'Bala Balarajah' <[email-masked]>
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian

From: Sarah Banks <[email-masked]<mailto:[email-masked]>>
Sent: Friday, July 16, 2021 2:10 PM
To: [email-masked]<mailto:[email-masked]>
Cc: Carsten Rossenhoevel <[email-masked]<mailto:[email-masked]>>; ALFRED MORTON <[email-masked]<mailto:[email-masked]>>; Jack, Mike <[email-masked]<mailto:[email-masked]>>; MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Timothy Otto <[email-masked]<mailto:[email-masked]>>; Bala Balarajah <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Brian,
    Please see inline.

Thanks,
Sarah (as a participant)

On Jul 13, 2021, at 10:45 AM, <[email-masked]<mailto:[email-masked]>> <[email-masked]<mailto:[email-masked]>> wrote:

Sarah,

Brian

From: Sarah Banks <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, July 13, 2021 1:24 PM
To: Carsten Rossenhoevel <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; ALFRED MORTON <[email-masked]<mailto:[email-masked]>>; MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Jack, Mike <[email-masked]<mailto:[email-masked]>>; Timothy Otto <[email-masked]<mailto:[email-masked]>>; Bala Balarajah <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Carsten,

   start with.

Kind regards,
Sarah

On Jul 13, 2021, at 9:19 AM, Carsten Rossenhoevel <[email-masked]<mailto:[email-masked]>> wrote:

The WGLC process has already been delayed by yet another IETF meeting cycle, and I would like to ask for your support to avoid any further unnecessary delays.  I would like to remind everyone of RFC 7154<https://urldefense.com/v3/__https:/tools.ietf.org/search/rfc7154__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPV0W6Qa9$> (IETF Guidelines for Conduct) section 2.4.

Best regards, Carsten

Brian

-----Original Message-----

Hi Brian,

thanks!

Al

-----Original Message-----

'Carsten Rossenhoevel'

Thanks Sarah,

Al,

Brian

-----Original Message-----

Hi Brian et al,

Thank you,

Sarah (as a participant)

Hi Sarah,

list.

tests against.

be

generation security.

is

out of scope.

DUTs

impact

DUT/SUT performance.

there.

bed.

reason for

that:

advantages for

L7 testing.

"real"

process

clients.)

can be

ignored.

[bpm] See above.

influence on results?

[bpm] See above.

section

2 "Requirements".

SB// Thanks!

traffic.

SB// See comment above.

happened

after the fact.

SB// Thanks!

uncommon term/word.

configuration.

scenario.

packet.

SB// Thanks!

traffic

:)

to the reader.

SB// Thanks!

and"

that should be changed.

[bpm] Thank you.

questions.

examples.

section.

testing the NGIPS?

Spirent confirmed this.

[bpm] Yes

specifically rejected?

SB// Thanks!

7

testing.

virtualized

components?

about reference test.

SB// Thanks!

make the assertion?

SB// Thanks!

well

bed

consideration".

SB// Thanks!

together

with the test results:

item

IETF BMWG.

developed cipher

suites.

--

Carsten Rossenhövel

[email-masked]<mailto:[email-masked]>, https://www.eantc.de<https://urldefense.com/v3/__https:/www.eantc.de/__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPZUV4CZF$>

EU VAT No: DE812824025
_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPT3gep8V$>

_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPT3gep8V$>

---

From: <[email-masked]>
Date: Thu, 22 Jul 2021 15:12:31 -0400

Thanks for this Al. Good suggestions.

I think it is also worth noting that the feature list contained in section 4.2, table 1 are either recommended or optional. We have enumerated the common features but do not mandate that any one feature is required. We do Recommend, a term I believe is supported in RFC 2119. I think the modification you suggest will strengthen that.

We are not resistant to the idea of making it clear that this document is only looking at NGFW and NGIPS.

We would prefer to not use the term “modern” as it isn’t really an industry recognized term.  Hopefully this can be something that is acceptable. Compromise being the word of the day. 

Thanks again Al.

Brian

From: MORTON JR., AL <[email-masked]> 
Sent: Thursday, July 22, 2021 2:39 PM
To: [email-masked]; 'Sarah Banks' <[email-masked]>
Cc: 'Carsten Rossenhoevel' <[email-masked]>; 'Jack, Mike' <[email-masked]>; 'MORTON, ALFRED C (AL)' <[email-masked]>; [email-masked]; 'Timothy Otto' <[email-masked]>; 'Bala Balarajah' <[email-masked]>
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi all,

Something like:

 

I hope this helps,

Al

 

From: [email-masked] <mailto:[email-masked]>  <[email-masked] <mailto:[email-masked]> > 
Sent: Friday, July 16, 2021 2:39 PM
To: 'Sarah Banks' <[email-masked] <mailto:[email-masked]> >
Cc: 'Carsten Rossenhoevel' <[email-masked] <mailto:[email-masked]> >; MORTON JR., AL <[email-masked] <mailto:[email-masked]> >; 'Jack, Mike' <[email-masked] <mailto:[email-masked]> >; 'MORTON, ALFRED C (AL)' <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> ; 'Timothy Otto' <[email-masked] <mailto:[email-masked]> >; 'Bala Balarajah' <[email-masked] <mailto:[email-masked]> >
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian

Cc: Carsten Rossenhoevel <[email-masked] <mailto:[email-masked]> >; ALFRED MORTON <[email-masked] <mailto:[email-masked]> >; Jack, Mike <[email-masked] <mailto:[email-masked]> >; MORTON, ALFRED C (AL) <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> ; Timothy Otto <[email-masked] <mailto:[email-masked]> >; Bala Balarajah <[email-masked] <mailto:[email-masked]> >

Hi Brian,

    Please see inline.

Thanks, 

Sarah (as a participant)

Sarah,

 

 

 

Brian

Carsten,

Kind regards,

Sarah

The WGLC process has already been delayed by yet another IETF meeting cycle, and I would like to ask for your support to avoid any further unnecessary delays.  I would like to remind everyone of RFC 7154 <https://urldefense.com/v3/__https:/tools.ietf.org/search/rfc7154__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPV0W6Qa9$>  (IETF Guidelines for Conduct) section 2.4. 

Best regards, Carsten

 

Brian
 
-----Original Message-----
From: MORTON JR., AL  <mailto:[email-masked]> <[email-masked]> 
Sent: Tuesday, July 13, 2021 11:01 AM
To: [email-masked] <mailto:[email-masked]> ; 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON,
 
Hi Brian,

thanks!
Al

-----Original Message-----
From: [email-masked] <mailto:[email-masked]>   <mailto:[email-masked]> <[email-masked]>
Sent: Tuesday, July 13, 2021 10:44 AM
To: 'Sarah Banks'  <mailto:[email-masked]> <[email-masked]>; 'MORTON, ALFRED C (AL)'
 <mailto:[email-masked]> <[email-masked]>
Cc: [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> ; 

'Carsten Rossenhoevel'

 
Thanks Sarah,

Al,

Brian

-----Original Message-----
From: Sarah Banks  <mailto:[email-masked]> <[email-masked]>
Sent: Monday, July 12, 2021 5:20 PM
To: [email-masked] <mailto:[email-masked]> 
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [bmwg] WGLC on New version of 
 
Hi Brian et al,
 
Thank you,
Sarah (as a participant)

Hi Sarah,

list.

tests against.

 

 

out of scope.

 

DUT/SUT performance.

there.

bed.

 

 

"real"

 

 

ignored.

 
[bpm] See above.

influence on results?

[bpm] See above.

SB// Thanks!

 

traffic.

SB// See comment above.

after the fact.

SB// Thanks!

 

 

configuration.

scenario.

packet.

SB// Thanks!

 

 

SB// Thanks!

and"

that should be changed.
 
[bpm] Thank you.

questions.

examples.

section.

 

[bpm] Yes

specifically rejected?

SB// Thanks!

testing.

components?

 

SB// Thanks!

SB// Thanks!

consideration".

 

SB// Thanks!

with the test results:

 

IETF BMWG.

suites.

 

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked] <mailto:[email-masked]> , https://www.eantc.de <https://urldefense.com/v3/__https:/www.eantc.de/__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPZUV4CZF$> 

_______________________________________________
bmwg mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/bmwg <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPT3gep8V$> 

_______________________________________________
bmwg mailing list
 <mailto:[email-masked]> [email-masked]
 <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bmwg__;!!BhdT!344XpUuQ7YIyFQ3UZmSxHG88B5cVP-J29Mgk7T1SDS0ZYCrD8cayPT3gep8V$> https://www.ietf.org/mailman/listinfo/bmwg

---

From: Gábor LENCSE <[email-masked]>
Date: Thu, 22 Jul 2021 22:19:15 +0200

2021.07.22. 21:12 keltezéssel, [email-masked] írta:
> We would prefer to not use the term “modern” as it isn’t really an 
> industry recognized term.

What about "state-of-the-art" or perhaps "contemporary"?

I am just trying to find a synonym, that is acceptable for both Sarah 
and the Authors. I myself am OK, with "next-generation", too. I feel all 
of them (including "modern") as "dated", and I think they all likely to 
mean "obsolete" in 10 or 20 years.

Best regards,

Gábor

---

From: Carsten Rossenhoevel <[email-masked]>
Date: Thu, 22 Jul 2021 23:13:59 +0200

Thanks Al and Gábor for your suggestions!

Let's find some compromises next week.

Regarding the name, a Google search revealed (search including the 
quotation marks to avoid matches on a subset of the terms):

*Name*
	*# Google Hits
*
"next-generation firewall"
	656,000
NGFW
	1,280,000
"modern firewall"
	13,800
"contemporary firewall"
	5,280

The term next-generation firewall or NGFW is most widely recognized in 
the industry, as one can see.  The other terms would rather confuse the 
audience from my point of view.  If a definition of the term needs to be 
added or refined in the draft's glossary it can certainly be discussed 
during the BMWG session next week.

Best regards, Carsten

On 7/22/21 10:19 PM, Gábor LENCSE wrote:
> and the Authors. I myself am OK, with "next-generation", too. I feel 
> all of them (including "modern") as "dated", and I think they all 
> likely to mean "obsolete" in 10 or 20 years.
>
> Best regards,
>
> Gábor
>
>

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
[email-masked], http://www.eantc.de

---

From: Gábor LENCSE <[email-masked]>
Date: Sun, 25 Jul 2021 14:17:10 +0200

Dear Sarah,

Please see my comments and suggestions in-line.

2021.07.16. 20:09 keltezéssel, Sarah Banks írta:
[...]
>
> SB// I support the draft as a draft that covers benchmarks that cover 
> specifically NGFW and NGIPS, provided the scope is not open ended to 
> cover the undefined "NG*" and future devices that we haven't invented 
> yet. :)
>

First of all, let me admit, that when I read the draft, I was not so 
careful as you. I have silently accepted the implicit definition of 
"next-generation firewall", and did not check what the term was used 
for. Now, I tried to find a formal definition, but I was not really 
successful.

What I found: https://en.wikipedia.org/wiki/Next-generation_firewall

As I teach my students not to use Wikipedia as a reference (e.g. because 
it is not a stable document), but rather read its sources, I tried to do 
the same.

For example, its first source is: 
https://www.esecurityplanet.com/products/intro-to-next-generation-firewalls/

It seems to me that the definition of NGFW is rather informal, 
especially from the following paragraph:

"These application-aware firewalls are commonly cited as a 
next-generation firewall (NGFW) but they are, basically, a form of a 
unified threat management (UTM) solution. However, the term UTM is 
usually applied to products that lack the true application-awareness and 
are targeted towards the SMB market. UTM products usually offer 
additional functions over traditional firewalls, such as antivirus, 
antispam, or even intrusion prevention systems (IPS)."

As an academic, I really appreciate your earlier suggestion that the 
draft should contain a definition of the term NGFW, however, as this 
term seems to be rather loosely defined, perhaps the Authors could 
include a sentence describing the situation. E.g. something like:

Next-generation firewall (NGFW): This term is widely used for the 
modern, state-of-the-art technology firewalls (as of 2021) that can do 
application-level traffic inspection including several, sometimes 
optional features. (Perhaps a few of them could be listed here.)

I just intended it a sample, what I want to suggest is that the daft 
should say something about what the term is used for, but admitting that 
it is not a formal definition.

What do you think of it?

As for your concerns about the technologies yet to be invented, my past 
experience suggests that new technologies are likely to be invented and 
they will likely make the draft obsolete. Or at least it should be 
updated. (In the same way as with RFC 2544, RFC 5180, RFC 8219.) The 
above definition (or something similar) could make it clear that we do 
not believe to define a benchmarking methodology for the eternity. :-)

Best regards,

Gábor