Subject: [bmwg] Providing context - re: draft-ietf-bmwg-ngfw-performance-13

From: <[email-masked]>
Date: Mon, 7 Feb 2022 20:26:39 -0500

Folks,

After reviewing the Discuss submissions again we, the authors, feel that
providing everyone some context on the how's and why's of the draft would be
useful. It is our hope that this will put us all on the same page and might
lead to easier resolution of your Discuss ballots.

This draft is intended to provide guidance on the testing of network
security products, in a tightly controlled environment, namely a test lab,
using test tools designed for this task. The benchmarking specifications are
not meant to address the variables that exist in a live environment or in
the billion of network stacks on internet hosts.

Modern day test tools, such as the ones created by contributors from
Spirent, Keysight (Ixia) and Viavi, are very complex and highly
configurable. Hence the detailed guidance provided for test tool
configuration and testbed setup. The TCP options are many and are not
something you would run into, or would be needed by most users, in a typical
live networking environment.

Configuration guidance was provided to give enough information to testers to
meet a key goal of this draft: allow the use of disparate test tools, yet
provide comparable results. Achieving this goal will provide metrics and
results for datasheets that allow apples-to-apples comparisons for
enterprises to evaluate security products. This goal is consistent with
BMWG's charter since 1989.

This goal is supported by the three test tool vendors cited, by multiple
security product vendor experts and test labs. And it has been put to the
test multiple times over the last two years. Seven vendors have gone public
with their test results. Additionally, a number of other security product
vendors have indicated they are waiting until this draft has been formally
approved to submit product for testing. If you wish to look at the published
test reports, they are all available at www.netsecopen.org.

This draft does not try to provide direction on how to configure all the
variables present on live production networks. In fact, much of the traffic
present on production networks will not be present on a well-run test/lab
network. As stated above, it is meant to provide guidance so that test tools
can be used to run tests in a consistent manner, such that there is
confidence in the results (e.g. could be repeatable by others).

Hopefully, this will provide the reviewers helpful context.

We will be responding to the individual Discuss submissions throughout the
week.

Brian Monkman, Carsten Rossenhoevel and Bala Balarajah

---------
Brian Monkman
Executive Director, NetSecOPEN
Office: +1-717-610-0808 
Fax: +1-717-506-0460
Mobile: +1-717-462-5422

https://www.netsecopen.org