Subject: Re: [bmwg] Work item proposal: extending firewall benchmarking methodology and terminology

From: "Eyal Manor" <[email-masked]>
Date: Sun, 6 Apr 2008 18:24:46 +0300

Hi Scott,

"All in one firewalls" having IPS/IDS and UTM capabilities are very popular,
being delivered by most Firewall companies. 

Evidently, these firewalls do not have the entire range of capabilities that
exist in different "point products" in each area, however, the firewall is
the first "line of defense" in the entity using it, and as such, should
demonstrate basic abilities in those areas.

We aim to define the methodologies in a way that will benchmark the
performance of a firewall when such features like IPS/IDS, Anti Virus, Anti
Spam or URL Filtering are activated, and with relevant test scenario. 

The scope is not very large, and summarizes with 2-3 tests for each of those
features.

Regards,

/Eyal. 

  _____  

From: Scott Poretsky [mailto:[email-masked]] 
Sent: Sunday, April 06, 2008 12:37 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [bmwg] Work item proposal:
extendingfirewallbenchmarkingmethodology and terminology

"IDS/IPS capabilities, deep inspection, enhanced DoS mitigation, and UTM
features such as Anti Virus, Anti Spam, and Web Filtering."

This is a very large scope.  I suggest we pick one rather than trying to
wrestle the universe with one work item.

Scott

----- Original Message -----
From: Eyal Manor <[email-masked]>
To: Scott Poretsky; 'Kevin Dubray' <[email-masked]>; [email-masked]
<[email-masked]>
Sent: Sun Apr 06 03:47:21 2008
Subject: RE: [bmwg] Work item proposal:
extendingfirewallbenchmarkingmethodology and terminology

Hello Scott,

This suggested proposal and the Accelerated work item have different goals:

As I understand the "Accelerated Stress" work item, it comes to benchmarks
the stability of routers in different scenarios under load of multiple
routing protocols, interface flapping, routes flapping, management
operations, etc'

Our suggested proposal comes to extend existing Firewall performance
benchmarking RFCs (2647 and 3511). We aim to define the criterions of
benchmarking the performance of "next generation" firewalls that provide
IDS/IPS capabilities, deep inspection, enhanced DoS mitigation, and UTM
features such as Anti Virus, Anti Spam, and Web Filtering.

The main motivation is to benchmark the performance of these features, and
less to "stress" the Firewall.

If you still see an overlap between these two, I'll be happy to continue
discussing about that with you together with the group of people that work
on this new suggestion.

/Eyal.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Scott Poretsky
Sent: Thursday, April 03, 2008 11:56 PM
To: Kevin Dubray; [email-masked]
Subject: Re: [bmwg] Work item proposal:
extendingfirewallbenchmarkingmethodology and terminology

Excerpts provided here.  Scott
-------------

                    Methodology for Benchmarking
              Accelerated Stress with Operational Security
             <draft-ietf-bmwg-acc-bench-meth-opsec-00.txt>

   ABSTRACT
   Routers in an operational network are simultaneously configured with
   multiple protocols and security policies while forwarding traffic and

   being managed.  To accurately benchmark a router for deployment it is

   necessary that the router be tested in these simultaneous operational

   conditions, which is known as Stress Testing.  This document provides

   the Methodology for performing Stress Benchmarking of networking
   devices when subjected to instability as described in [7]. 
   Descriptions of test topology, benchmarks and reporting format are
   provided in addition to procedures for conducting various test cases.

   This methodology is based upon the accelerated stress methodology
   guidelines [6] and is to be used with the companion terminology
   document [4].

    4.1 Restart Under Load
        Objective
        The purpose of this test is to benchmark the performance of the
DUT
        during restart when stress conditions are applied.

     4.2 Destination Control Processor
        Objective
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the Control Processor of the DUT.

     4.3 Destination Control Processor with Rate-Limiting
        Objective
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the Control processor of the DUT.

     4.4 Destination Interfaces
        Objective
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the interfaces of the DUT.

     4.5 DoS Attack
        Objective
        The purpose of this test is to benchmark the performance of the
        DUT during stress conditions while experiencing a DoS attack.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, April 03, 2008 11:50 AM
To: [email-masked]
Subject: Re: [bmwg] Work item proposal: extending
firewallbenchmarkingmethodology and terminology

In which dimensions are the overlap, Scott?  AccStress seems to want to
describe coexisting applications (breadth?) in a generic manner; the
extension proposed seems to increase depth in specific context.

I think the effort has merit; I would like to see a I-D (or at least a
possible table of contents) to better reconcile possible overlap..
(whether AccStress, IPsec, etc.)

-Kevin

Scott Poretsky wrote:
>
> This proposal as written seems to have quite a bit of overlap with the

> existing Accelerated Stress work item.
>
> Scott
>

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Eyal Manor" <[email-masked]>
Date: Sun, 6 Apr 2008 18:22:11 +0300

Hi Scott,

Regards,

/Eyal. 

  _____  

Scott

----- Original Message -----
From: Eyal Manor <[email-masked]>
To: Scott Poretsky; 'Kevin Dubray' <[email-masked]>; [email-masked]
<[email-masked]>

Hello Scott,

/Eyal.

-----Original Message-----

   ABSTRACT

    4.1 Restart Under Load
        Objective

     4.5 DoS Attack
        Objective

-----Original Message-----

-Kevin

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Eyal Manor
> Sent: Thursday, April 03, 2008 5:26 AM
> To: [email-masked]
> Subject: [bmwg] Work item proposal: extending firewall
> benchmarkingmethodology and terminology
>
> Dear BMWG members,
>
> I would like to have the group's feedback on the following proposal,
with
> the goal of adding it to the BMWG charter. We would like to extend RFC

> 2647
> and RFC 3511 (firewall benchmarking terminology and methodology) to
bring
> them up to date, where the majority of today's "firewall" devices have

> much
> more extensive functionality than what these documents cover.
>
>

> As currently implemented and widely deployed, security devices go well
> beyond firewall capabilities to include a large number of additional
> functionalities such as IPsec and/or SSL virtual private networking;
> intrusion detection and prevention; and unified threat management
(UTM)
> capabilities such as antivirus scanning, URL filtering, and content
> filtering.
>
> Implementers and end-users have serious difficulties validating the
> performance aspects of these all-in-one security devices, as the
existing
> terminology and methodology documents, RFCs 2647 and 3511, cover only
> firewall performance measurement.
>
> As a result, implementers and end-users may deploy security devices
with
> limited understanding of these devices' performance characteristics,
> especially when activating additional capabilities beyond the access
> control
> of a firewall.
>
> We would like to propose updated terminology and methodology documents
to
> provide guidelines for key tests validating the performance of each of
the
> security subsystems for both allowed and rejected (malicious) traffic.
>
> As with the earlier RFCs, the primary focus of these documents is
> performance measurement. While there is some overlap between
> performance and
> security, assessment of device security is beyond the scope of this
> document.
>
>
> Goal:
> The objective of this work item is to extend RFC 2647 and RFC 3511
beyond
> firewall benchmarking.
> This new effort will add new definitions and tests to measure the
> performance of all-in-one security devices.
> New procedures will allow testers to benchmark the performance of each
> security technology under conditions that are repeatable, stressful,
and
> meaningful.
>
> In addition to introducing tests of more security technologies, this
> proposal expands and updates the existing RFCs by adding tests with
> multiple
> interfaces on each side of a device as well as redundancy and failover
> capabilities.
>
>
> Scope:
> Performance of the following functionalities will be measured:
firewall,
> IPsec and SSL VPN gateway, intrusion detection/prevention, and UTM
> capabilities such as antivirus, URL filtering, and content filtering.
>
> Each element will be measured in three steps: performance with allowed
> traffic, performance with rejected (malicious) traffic and performance

> with
> mixed allowed and rejected traffic.
>
> For each element, our methodology will describe procedures for
measuring
> throughput, latency, session rate, and maximum concurrent active
> connections
> for each protocol used.
>
> The extension to the Terminology RFC will update and add the
> definitions of
> the new tested functionalities.
>
> This proposal does not intend to introduce new IPsec tests, but rather
to
> refer to specific terms and procedures from the IPsec documents
currently
> under review by the BMWG.
> Similarly, this proposal will look to use existing terminology and
> methodologies for application-layer performance measurement, for
example
> from PMOL, wherever possible.
>
>
> Motivation:
> In the past few years, stateful inspection firewalls have evolved far
> beyond
> a device that enforces an access control policy between networks (RFC
> 2647).  Today security devices include not only firewall capabilities
but
> also IPsec/SSL VPN, IDS/IPS, Anti Virus, Anti Spam, URL filtering, and

> other
> security technologies.
>
> Implementers and end-users alike require standardized tests to
understand
> the performance of different elements in security devices to make
informed
> decisions on device performance and capacity.
>
>
> Proposed Schedule:
> 04/15/2008 - Submit this proposal to BMWG
> 06/15/2008 - Submit terminology and methodology drafts to BMWG, as
> individual contributions before Dublin.
> Next steps TBD.
>
>
========================================================================
===
>
> Best Regards,
> Eyal Manor
>
>
>
> 
> This message may contain confidential and/or proprietary information,
> and is
> intended only for the person/entity to whom it was originally
addressed.
> The content of this message may contain private views and opinions
> which do
> not constitute a formal disclosure or commitment unless specifically
> stated.
>
>
>
>
>
>
>
>

Scanned by Check Point Total Security Gateway.

---

From: Al Morton <[email-masked]>
Date: Sun, 06 Apr 2008 14:23:14 -0400

Folks,

The replies to this thread are now exceeding the single message

octet limit. Please clip your replies as much as possible, use

plain text, and think of all the hard drives you are saving space
on...

That said, please continue sorting through the scope of this

proposed item on the list.

technical comment to follow,

Al

bmwg chair

At 11:22 AM 4/6/2008, Eyal Manor wrote:

Hi Scott,

"All in one firewalls" having IPS/IDS and UTM capabilities are
very popular, being delivered by most Firewall companies.

---

From: Al Morton <[email-masked]>
Date: Sun, 06 Apr 2008 14:42:19 -0400

...

"All in one
firewalls" having IPS/IDS and UTM capabilities are very popular,
being delivered by most Firewall companies.

Evidently, these
firewalls do not have the entire range of capabilities that exist in
different "point products" in each area, however, the firewall
is the first “line of defense” in the entity using it, and as such,
should demonstrate basic abilities in those
areas.

...

Sometime earlier, Scott Poretsky wrote:

Excerpts provided here.
Scott

-------------

ABSTRACT

Routers in an operational network are simultaneously
configured with

multiple protocols and security policies while forwarding
traffic and

being managed.  To accurately benchmark a router for
deployment it is

necessary that the router be tested in these simultaneous
operational

conditions, which is known as Stress Testing.

<snip>

The scope of the ACC work started rather broad, we have tried to
achieve

focus by limiting the methodologies, and it's still in
progress...

I note that the ACC work addresses a specific device (a router),

and suggestions to "add SIP configurations" were discussed and
discouraged

at the last meeting.

Maybe we can get some additional scope clarity by defining

"firewall" as a dedicated device or a networking technology for

the purposes of this effort.

(is it a floor wax, or a dessert topping? I know, it's both.)

Al

(participant)

---

From: Al Morton <[email-masked]>
Date: Sun, 06 Apr 2008 14:46:18 -0400

I think this is a message from Scott that was also caught

by the size filter...

-----Original Message-----

From: [email-masked]
[mailto:[email-masked]]
On Behalf Of

Scott Poretsky

-------------

ABSTRACT

conditions, which is known as Stress Testing.  This
document provides

the Methodology for performing Stress Benchmarking of
networking

Descriptions of test topology, benchmarks and reporting
format are

provided in addition to procedures for conducting various
test cases.

This methodology is based upon the accelerated stress
methodology

guidelines [6] and is to be used with the companion
terminology

document [4].

4.1 Restart Under Load

Objective

The purpose of this test is to
benchmark the performance of the

DUT

during restart when stress
conditions are applied.

Objective

The purpose of this test is to
benchmark the performance

of the DUT during stress
conditions when traffic is destined for

the Control Processor of the
DUT.

4.3 Destination Control Processor with
Rate-Limiting

Objective

The purpose of this test is to
benchmark the performance

of the DUT during stress
conditions when traffic is destined for

the Control processor of the
DUT.

Objective

The purpose of this test is to
benchmark the performance

of the DUT during stress
conditions when traffic is destined for

the interfaces of the
DUT.

4.5 DoS Attack

Objective

The purpose of this test is to
benchmark the performance of the

DUT during stress conditions
while experiencing a DoS attack.

---

From: David Newman <[email-masked]>
Date: Sun, 06 Apr 2008 11:58:02 -0700

> (is it a floor wax, or a dessert topping? I know, it's both.)

Some history: When I proposed what became RFC 2647 (firewall performance 
terminology) at IETF 39 in 1997, I intended to describe performance 
testing of a generic "security device" in the same way RFCs 1242 and 
2544 describe performance testing of a "network interconnect device".

At the time -- considering that firewalls 10-15 years ago focused mainly 
or exclusively on access control -- there was consensus to focus on that 
function only. (Even then, the definition of firewall was kind of hazy.)

Since then, for better or worse, the definition of "firewall" has 
expanded considerably. Virtually all firewalls sold today incorporate -- 
and, as implemented, commonly use -- many other functions above and 
beyond lower-layer access control. We don't have a standardized way of 
describing performance of these devices, leading to the kind of 
"specsmanship" confusion mentioned in 1242.

I'll be the first to agree that we can't describe the performance of the 
whole without first defining the performance of all parts. Then again, 
we can't ignore that firewalls today have multiple parts.

dn

---

From: "Scott Poretsky" <[email-masked]>
Date: Mon, 7 Apr 2008 11:32:10 -0400

This proposed work item needs to decide if the scope is to test
"Firewalls" or test "Security Devices".  "Security Devices" is too broad
of a scope.  Things like IDS/IPS are available as standalone products.
An IPS/IDS could be benchmarked whether it is standalone or integrated
into a firewall.  

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
David Newman
Sent: Sunday, April 06, 2008 2:58 PM
Cc: [email-masked]
Subject: Re: [bmwg] Work item proposal:
extendingfirewallbenchmarkingmethodology and terminology

dn