Subject: [bmwg] Last call comments on the firewall benchmarking draft

From: [email-masked]
Date: Mon, 8 Jul 2002 16:27:31 +0300

Dear Authors of the firewall benchmarking draft and Dear BMWG members,

I work for Stonesoft, which is a software company making firewall software
and other security solutions. We would like to suggest some modifications
to
the firewall benchmarking draft before moving it forward. I apologize for
commenting only at such a late phase of the draft, but I hope you take our
points into consideration. Naturally, we are willing to contribute to the
drafting
work, if considered appropriate by the authors.

The problem I see with the current draft is that the test traffic is
perhaps
not defined accurately enough. Rather than standardizing firewall
configurations
(such as security policies), I would like to standardize the test loads
(traffic patterns)
as accuarately as possible, as well as the quantities that can be measured
from
the received traffic.

For IP layer traffic there is not much problem, it is very much like what
is described
in RFC 2544. Also UDP is quite easy because it doesn't retransmit. However,
TCP
and the applications running over it (e.g. HTTP) seem quite problematic
because
there are some TCP parameters at the end-points (client/server) that
influence the traffic flow, most notably the TCP window size. The firewall
between
the TCP end-points may very well be a "long, fat pipe" as described in RFC
1323,
which means that the TCP or application throughput is not limited by the
firewall but
by the TCP window size at the client/server end-points.

Therefore I would suggest characterizing very accurately the traffic flows
and the test client/server behavior in the benchmarks. For TCP benchmarking
it would
be very useful to use a bit simplified TCP traffic so that it would have
e.g. "infinite window
size", which would mean that the end-points never paused transmitting even
if they didn't
receive certain acknowledgements. Another simplification might be to use
constant
RTT (round-trip time) instead of dynamic RTT measurement, so that a
possible
retransmission would always take place a constant time (e.g. 1s) after the
original
transmission (or use an infinite RTT meaning that the TCP never
retransmits). Even
if this traffic is not "real life TCP traffic", it would be very useful for
benchmarking
because the offered load would be easily controlled. The firewalls (in most
cases)
wouldn't see any difference between the "simplified" and "real" TCP
traffic.

Also the TCP connection closing sequences should be defined accurately.
There
are a few ways to close a connection, and in some firewalls it makes a
difference
whether the closed connection is considered to enter the TIME_WAIT state or
not.

In general, I would like to move the focus of the draft from application
(HTTP) level
benchmarking to TCP level benchmarking. For example, I'm missing a TCP
transfer rate test, as well as a TCP transaction test that both establishes
and closes
connections in the same test. In some sense a simple application level test
(e.g.
HTTP transfer rate with simple GET methods) can be used for these purposes
by measuring transport layer goodput as well as application  goodput, but I
feel
it would be clearer to define separate TCP benchmarks for cases where
application
data is not important, and separate application level  tests e.g for HTTP
if the
firewall happens to apply some sort of proxying, URL filtering or content
scanning.

I would also like to define which quantities are measured and reported for
each
traffic pattern,  including error statistics. The error information is
valuable for example
when bencharking fault tolerant firewall systems, where the firewall is
able to
recover from a HW or link failure by switching over to a redundant link or
HW unit. In
these cases some packets or connections are most probably lost, and it
would be
valuable to know how many. In any case the end condition of a test should
not
necessarily be a single error, but the tests should in general continue and
just
measure and report the errors. The current draft defines this kind of error
logging
for some tests, but I'd like to have it done for all tests where possible
and maybe
a bit more extensively.

I'd appreaciate your comments on the points I've tried to make above. They
are
mostly based on practical experience on testing our firewall software's
performance.
We've had quite a lot of trouble finding suitable tests equipment for
measuring TCP
performance, typically we've run into limitations of the test equipment
before
hitting the limits of the firewall, and also the available test cases don't
sometimes
make much sense . So we would really like to see some good TCP benchmarks
standardised. Of course we would also like to benchmark with application
protocols,
but currently we are mostly missing TCP benchmarks.

Sincerely,

Juha Pirkola
Software Specialist
Stonesoft Oyj, Finland

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg