Subject: [bmwg] Work item proposal: extending firewall benchmarking methodology and terminology

From: "Eyal Manor" <[email-masked]>
Date: Thu, 3 Apr 2008 12:26:07 +0300

Dear BMWG members,

I would like to have the group's feedback on the following proposal, with
the goal of adding it to the BMWG charter. We would like to extend RFC 2647
and RFC 3511 (firewall benchmarking terminology and methodology) to bring
them up to date, where the majority of today's "firewall" devices have much
more extensive functionality than what these documents cover.

I would like to thank Gregory Fresnais, David Newman, Timmons Player, Yaron
Sheffer and Hari K. Tadepalli who participated in preparing this proposal.

===========================================================================

Proposal for security device benchmarking terminology and methodology:
Extending RFC 2647 and RFC 3511

Summary:
This is a proposal for a BMWG work item to define and extend test
terminology and methodologies for firewalls and related security
technologies.

As currently implemented and widely deployed, security devices go well
beyond firewall capabilities to include a large number of additional
functionalities such as IPsec and/or SSL virtual private networking;
intrusion detection and prevention; and unified threat management (UTM)
capabilities such as antivirus scanning, URL filtering, and content
filtering. 

Implementers and end-users have serious difficulties validating the
performance aspects of these all-in-one security devices, as the existing
terminology and methodology documents, RFCs 2647 and 3511, cover only
firewall performance measurement.

As a result, implementers and end-users may deploy security devices with
limited understanding of these devices' performance characteristics,
especially when activating additional capabilities beyond the access control
of a firewall.

We would like to propose updated terminology and methodology documents to
provide guidelines for key tests validating the performance of each of the
security subsystems for both allowed and rejected (malicious) traffic.

As with the earlier RFCs, the primary focus of these documents is
performance measurement. While there is some overlap between performance and
security, assessment of device security is beyond the scope of this
document.

Goal:
The objective of this work item is to extend RFC 2647 and RFC 3511 beyond
firewall benchmarking.
This new effort will add new definitions and tests to measure the
performance of all-in-one security devices.
New procedures will allow testers to benchmark the performance of each
security technology under conditions that are repeatable, stressful, and
meaningful.

In addition to introducing tests of more security technologies, this
proposal expands and updates the existing RFCs by adding tests with multiple
interfaces on each side of a device as well as redundancy and failover
capabilities.

Scope:
Performance of the following functionalities will be measured: firewall,
IPsec and SSL VPN gateway, intrusion detection/prevention, and UTM
capabilities such as antivirus, URL filtering, and content filtering.

Each element will be measured in three steps: performance with allowed
traffic, performance with rejected (malicious) traffic and performance with
mixed allowed and rejected traffic.

For each element, our methodology will describe procedures for measuring
throughput, latency, session rate, and maximum concurrent active connections
for each protocol used.

The extension to the Terminology RFC will update and add the definitions of
the new tested functionalities.

This proposal does not intend to introduce new IPsec tests, but rather to
refer to specific terms and procedures from the IPsec documents currently
under review by the BMWG.
Similarly, this proposal will look to use existing terminology and
methodologies for application-layer performance measurement, for example
from PMOL, wherever possible.

Motivation:
In the past few years, stateful inspection firewalls have evolved far beyond
a device that enforces an access control policy between networks (RFC
2647).  Today security devices include not only firewall capabilities but
also IPsec/SSL VPN, IDS/IPS, Anti Virus, Anti Spam, URL filtering, and other
security technologies.

Implementers and end-users alike require standardized tests to understand
the performance of different elements in security devices to make informed
decisions on device performance and capacity.

Proposed Schedule:
04/15/2008 - Submit this proposal to BMWG
06/15/2008 - Submit terminology and methodology drafts to BMWG, as
individual contributions before Dublin.
Next steps TBD.

===========================================================================

Best Regards,
Eyal Manor

This message may contain confidential and/or proprietary information, and is
intended only for the person/entity to whom it was originally addressed.
The content of this message may contain private views and opinions which do
not constitute a formal disclosure or commitment unless specifically stated.

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Scott Poretsky" <[email-masked]>
Date: Thu, 3 Apr 2008 05:37:49 -0400

This proposal as written seems to have quite a bit of overlap with the existing Accelerated Stress work item.

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Eyal Manor
Sent: Thursday, April 03, 2008 5:26 AM
To: [email-masked]
Subject: [bmwg] Work item proposal: extending firewall benchmarkingmethodology and terminology

Dear BMWG members,

Summary:

Goal:

Scope:

Motivation:

Proposed Schedule:

Best Regards,
Eyal Manor

---

From: Kevin Dubray <[email-masked]>
Date: Thu, 03 Apr 2008 11:49:49 -0400

In which dimensions are the overlap, Scott?  AccStress seems to want to 
describe coexisting applications (breadth?) in a generic manner; the 
extension proposed seems to increase depth in specific context.

I think the effort has merit; I would like to see a I-D (or at least a 
possible table of contents) to better reconcile possible overlap.. 
(whether AccStress, IPsec, etc.)

-Kevin

---

From: "Scott Poretsky" <[email-masked]>
Date: Thu, 3 Apr 2008 16:55:42 -0400

Excerpts provided here.  Scott
-------------

  	            Methodology for Benchmarking 
              Accelerated Stress with Operational Security 
             <draft-ietf-bmwg-acc-bench-meth-opsec-00.txt>

   ABSTRACT
   Routers in an operational network are simultaneously configured with 
   multiple protocols and security policies while forwarding traffic and

   being managed.  To accurately benchmark a router for deployment it is

   necessary that the router be tested in these simultaneous operational

   conditions, which is known as Stress Testing.  This document provides

   the Methodology for performing Stress Benchmarking of networking 
   devices when subjected to instability as described in [7].  
   Descriptions of test topology, benchmarks and reporting format are 
   provided in addition to procedures for conducting various test cases.

   This methodology is based upon the accelerated stress methodology 
   guidelines [6] and is to be used with the companion terminology 
   document [4].

    4.1 Restart Under Load
        Objective
        The purpose of this test is to benchmark the performance of the
DUT 
        during restart when stress conditions are applied.

     4.2 Destination Control Processor
        Objective 
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the Control Processor of the DUT.

     4.3 Destination Control Processor with Rate-Limiting
        Objective 
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the Control processor of the DUT.

     4.4 Destination Interfaces
        Objective 
        The purpose of this test is to benchmark the performance
        of the DUT during stress conditions when traffic is destined for

        the interfaces of the DUT.

     4.5 DoS Attack 
        Objective 
        The purpose of this test is to benchmark the performance of the 
        DUT during stress conditions while experiencing a DoS attack.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, April 03, 2008 11:50 AM
To: [email-masked]
Subject: Re: [bmwg] Work item proposal: extending
firewallbenchmarkingmethodology and terminology

-Kevin

Scott Poretsky wrote:
>
> This proposal as written seems to have quite a bit of overlap with the

> existing Accelerated Stress work item.
>
> Scott
>
> the goal of adding it to the BMWG charter. We would like to extend RFC

> 2647
> them up to date, where the majority of today's "firewall" devices have

> much

> traffic, performance with rejected (malicious) traffic and performance

> with
> also IPsec/SSL VPN, IDS/IPS, Anti Virus, Anti Spam, URL filtering, and

> other
========================================================================
===

---

From: "Eyal Manor" <[email-masked]>
Date: Sun, 6 Apr 2008 10:47:21 +0300

Hello Scott,

This suggested proposal and the Accelerated work item have different goals:

As I understand the "Accelerated Stress" work item, it comes to benchmarks
the stability of routers in different scenarios under load of multiple
routing protocols, interface flapping, routes flapping, management
operations, etc'

Our suggested proposal comes to extend existing Firewall performance
benchmarking RFCs (2647 and 3511). We aim to define the criterions of
benchmarking the performance of "next generation" firewalls that provide
IDS/IPS capabilities, deep inspection, enhanced DoS mitigation, and UTM
features such as Anti Virus, Anti Spam, and Web Filtering.

The main motivation is to benchmark the performance of these features, and
less to "stress" the Firewall. 

If you still see an overlap between these two, I'll be happy to continue
discussing about that with you together with the group of people that work
on this new suggestion.

/Eyal. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Scott Poretsky
Sent: Thursday, April 03, 2008 11:56 PM
To: Kevin Dubray; [email-masked]
Subject: Re: [bmwg] Work item proposal:
extendingfirewallbenchmarkingmethodology and terminology

   ABSTRACT

    4.1 Restart Under Load
        Objective

     4.5 DoS Attack 
        Objective 

-----Original Message-----

-Kevin

Scanned by Check Point Total Security Gateway.