Subject: Thoughts on firewall performance draft

From: [email-masked] (Kevin Dubray)
Date: Mon, 22 Jun 1998 15:02:24 -0400

From: "David Newman" <[email-masked]>

BMWGers,

Thanks for your input on the firewall draft at the LA meeting. I have
finally gotten around to addressing these issues. Generally, the to-do
items fall into two groups: better definitions of forwarding rate and
connections.

1. Forwarding rate
At the LA meeting, Kevin and others noted that the forwarding rate
definition, in its present form, is not sufficiently different from the RFC
2285 definition to merit inclusion here. My intent in the draft was to note
that many firewalls work at the application layer and thus frame-based
metrics aren't terribly meaningful. Bit- or byte-specific metrics are
useful.

There were various proposals for bit-specific terms. The one that makes the
most sense to me is bit forwarding rate. I would define it as follows:

Bit forwarding rate
The number of bits per second of allowed traffic a firewall can be observed
to transmit successfully to the correct destination interface in response
to a specified offered load.

This definition is firewall-specifc in two ways: First, it specifies
measurement in bits rather than frames. Second, it specifies measurement of
allowed traffic--traffic the DUT/SUT is *supposed* to forward as defined by
a given rule set. Not also that "specified offered load"=!"allowed traffic"
since it may be desirable to offer a mix of allowed and illegal traffic.

As an aside, John Dawson remarked after the meeting that buggy
implementations may forward illegal traffic, and the current definition for
"rejected traffic" doesn't cover that case (it simply assumes a firewall
will forward all allowed traffic and drop everything else). To address this
I plan on adding a definition for "illegal traffic" (traffic not allowed by
the rule set, regardless of whether it's forwarded or rejected).

2. Connections and sessions
The draft attempts to distinguish between "connection" and "session," with
the former defined as the existence of some link and the latter as the flow
of data over that link. This dichomoty is troublesome, as it doesn't allow
for cases where multiple sessions may flow over a single physical link and
doesn't adequately define what state information the DUT/SUT may hold about
either a connection or state.

It would be much simpler to collapse "connection" and "state" into a single
definition. At the LA meeting, Kim Martin suggested using the term
"circuit" for this unified definition, a term I liked because it conveys
both the existence of a link (a state in which transactions can occur, as
Kevin Dubray put it) and the possibility of data flowing over the link.

However, on further reflection I'm more inclined to stick with the term
connection. Couple of reasons here: First, connections is widely used in
describing firewall performance, as in, "How many concurrent connections
can the firewall handle?" I haven't heard "circuit" used in the same
context. Second, circuit also carries the meaning of physical link, as in a
T1 circuit, and that's *not* what I had in mind. The issue here is that
multiple virtual connections (TCP, ATM, whatever) can ride over a single
circuit.

Here's what I would propose:

Connection
A state in which nodes agree to exchange data using a known protocol.

This is purposefully vague. As we have discussed in LA and in Washington it
is desirable to allow for connections at multiple layers (e.g., not just
TCP but also UDP, ATM, etc.)

One thing this definition does *not* cover is the flow of data over that
connection. It seems to me that the real heavy lifting as far as the
firewall is concerned comes with opening a socket for a connection request;
the existence of an open socket has a memory cost (and by extension, a
performance impact) regardless of whether data flows through that socket.

Those are my two main proposals for revision. In addition, it's been
pointed out to me that my definitions of proxies do not adequately
distinguish between application- and circuit-type proxies, and I will
clarify that point in the next draft.

I would welcome other comments on these definitions.

Regards
David Newman

---

From: [email-masked] (Kevin Dubray)
Date: Thu, 25 Jun 1998 09:55:04 -0400

From: Edward Gonen <[email-masked]>
To: "'Firewall Performance Group'" <[email-masked]>
Subject: RE:Thoughts on firewall performance draft
Date: Thu, 25 Jun 1998 10:45:47 +0200
Return-Receipt-To: Edward Gonen <[email-masked]>

	Dear > BMWGers,

	Could you please add me to this mailing list?  Below I repost my
comments and David's answers regarding performance draft. There was a kind
of misunderstanding - I thought we are talking about methodology but as
David Newman explained it is rather terminology definition stage so I'll
keep my comments until we get to the methodology.

	Edward Gonen
	NetGuard. Ltd.
	[email-masked] <mailto:[email-masked]> 

Thanks very much for your thoughts, Edward. By the way I would strongly
encourage you to repost this to the bmwg mailing list, if you're not already
on the list. I've commented in ((double parentheses)) on a few points below.
dn

Edward Gonen <[email-masked] <mailto:[email-masked]> > on
06/24/98 05:28:31 AM
To:	David Newman/NYC/CMPNotes, "[email-masked]'"
	<[email-masked] <mailto:[email-masked]> >
Subject:	RE:Thoughts on firewall performance draft

David,

I've got your mail "Thoughts on firewall performance draft" somehow
forwarded to me and here are some points I'd like to mention:
		     > > Bit forwarding rate. > The number of bits per
second of allowed
traffic a firewall can be > observed > to >transmit >successfully to the
correct destination interface in > response > to a specified offered load.
		Yes this definition is better than "frames" or "packets".
But how do
you calculate this number? If we are talking about TCP/IP then there are
MAC
header, IP header, TCP, UDP ... and eventually - the data. Some of the
performance benchmarks measure only the application data size (HTML page
size or FTP files), while others calculate total traffic including MAC
header. In my opinion only the data part (what comes after TCP / UDP
headers) of the TCP or UDP should be included.  Of course with packets
	~1Kbyte you may not count the header but when the number of packets
is huge
and they are smaller the addition of headers is quite important.
((I don't agree that only the payload should be counted. This may reflect my
bias toward router and switch-type benchmarks, but remember that a firewall
has to process the entire datagram, including header. As you note this may
be a statistically significant amount of work, especially with smaller
packets. Since the header comprises part of the workload handled by the
firewall, I feel it should be counted in measuring forwarding rate.))

Another thing - traffic direction. In most of the benchmark tests there is
device under test (firewall) installed between a large number of clients and
a smaller number of servers thus the traffic is mostly one-way road - most
of the data is coming from the servers toward the clients. In my opinion in
the test environment the clients and servers (receivers and senders) should
be also swept places. The result may be calculated as an average value of
both tests.

There is also question regarding which traffic is measured - UDP or TCP. In
case of TCP there is an issue of acknowledges sent in the opposite
direction. But we can ignore that if the clients and servers are swept.
((Can you please define the term "swept places"? I'm not familiar with it.
Past versions of the draft had included definitions of unidirectional and
bidirectional traffic, but the consensus within the bmwg was that the
definitions were not sufficiently different from existing definitions to
merit inclusion here.

This purpose of this memo is to describe *terminology* used in benchmarking
firewall performance. The bmwg generally follows a two-step process, with a
terminology document first and a methodology second. Traffic direction is
very germane to the methodology document.))

Another issue is which services are used in the test. I think that
performance of the application-proxy firewall depends on which service is
used - HTTP, FTP, SMTP etc. The differences may be very noticeable.
((Indeed. I've included definitions of proxies in this terminology document.
Describing which proxies are invoked is a methodology issue, however.))

Now is another point - how the firewall should be prepared to the tests. I
think it is a good idea that (like in the past tests) vendor's
representatives should prepare the firewall for performance tests, but it is
crucial that after that preparation and before the performance tests, the
firewall is tested for security. It must be proven that there is no security
degradation when the firewall is configured for the performance tests. It
may be also a good idea to perform the security tests even after the
performance tests.
((Again, this is largely a methodology issue. This is going to be a very
difficult issue to sort out.

The task of "proving there's no security degradation" is very open-ended, to
say the least. The terminology document only contemplates that a given rule
set is enforced. What about security issues outside the rule set, like the
use or nonuse of NAT and hardening of the device against various
denial-of-service attacks? I don't have an answer to this as yet and would
welcome your thoughts.))

	Next time I'll write about connections and sessions..

((I'd very much appreciate your input on this as well.))

	I'll be glad to hear your opinions.

Edward Gonen
Vice President of R&D
NetGuard Ltd.

================================================================
eMail: [email-masked] <mailto:[email-masked]>
<mailto:[email-masked] <mailto:[email-masked]> >
Phone:+972-6-6449944 Mobile:+972-52661113 Fax:+972-6-6540124
URL:http://www.ntguard.com <URL:http://www.ntguard.com>
<URL:http://www.ntguard.com <URL:http://www.ntguard.com> >
Address: LanOptics Building, P.O.Box 184,
Ramat Gabriel Industrial Park, Migdal Ha-Emek 10551, Israel

---

From: David Newman <[email-masked]>
Date: Fri, 26 Jun 1998 17:00:52 -0400

>     Kevin> Kevin, I'd like a clarification of what's meant by "connection
>     Kevin> overhead." The memory cost of opening a socket is
>
>   Given n current connections, how much time does it take for the n+1'th
> connection to be opened.
>
Great! That's a very good way of expressing connection setup time.

But we have to define what's meant by connection in the first place. .
.I'd be interested in your comments on the original proposal:

>     Kevin> implementation-specific, and a DUT/SUT that uses 100 kbytes
> of
>     Kevin> memory for each connection may or may not move bits on the
> wire
>     Kevin> more slowly than one requiring only 10 kbytes. That's a
> firewalls
>
>   The internals issue is more to do with the kind of data structure
> that
> is used to store the connection state. Also, whether or not a new
> process
> needs to be created to handle that connection.

Let's not get into internals; this is an effort to benchmark
characteristics observed on the *outside* of the box. ;-)

dn