Subject: RE:Thoughts on firewall performance draft

From: [email-masked] (Kevin Dubray)
Date: Thu, 25 Jun 1998 09:55:04 -0400

From: Edward Gonen <[email-masked]>
To: "'Firewall Performance Group'" <[email-masked]>
Subject: RE:Thoughts on firewall performance draft
Date: Thu, 25 Jun 1998 10:45:47 +0200
Return-Receipt-To: Edward Gonen <[email-masked]>

	Dear > BMWGers,

	Could you please add me to this mailing list?  Below I repost my
comments and David's answers regarding performance draft. There was a kind
of misunderstanding - I thought we are talking about methodology but as
David Newman explained it is rather terminology definition stage so I'll
keep my comments until we get to the methodology.

	Edward Gonen
	NetGuard. Ltd.
	[email-masked] <mailto:[email-masked]> 

Thanks very much for your thoughts, Edward. By the way I would strongly
encourage you to repost this to the bmwg mailing list, if you're not already
on the list. I've commented in ((double parentheses)) on a few points below.
dn

Edward Gonen <[email-masked] <mailto:[email-masked]> > on
06/24/98 05:28:31 AM
To:	David Newman/NYC/CMPNotes, "[email-masked]'"
	<[email-masked] <mailto:[email-masked]> >
Subject:	RE:Thoughts on firewall performance draft

David,

I've got your mail "Thoughts on firewall performance draft" somehow
forwarded to me and here are some points I'd like to mention:
		     > > Bit forwarding rate. > The number of bits per
second of allowed
traffic a firewall can be > observed > to >transmit >successfully to the
correct destination interface in > response > to a specified offered load.
		Yes this definition is better than "frames" or "packets".
But how do
you calculate this number? If we are talking about TCP/IP then there are
MAC
header, IP header, TCP, UDP ... and eventually - the data. Some of the
performance benchmarks measure only the application data size (HTML page
size or FTP files), while others calculate total traffic including MAC
header. In my opinion only the data part (what comes after TCP / UDP
headers) of the TCP or UDP should be included.  Of course with packets
	~1Kbyte you may not count the header but when the number of packets
is huge
and they are smaller the addition of headers is quite important.
((I don't agree that only the payload should be counted. This may reflect my
bias toward router and switch-type benchmarks, but remember that a firewall
has to process the entire datagram, including header. As you note this may
be a statistically significant amount of work, especially with smaller
packets. Since the header comprises part of the workload handled by the
firewall, I feel it should be counted in measuring forwarding rate.))

Another thing - traffic direction. In most of the benchmark tests there is
device under test (firewall) installed between a large number of clients and
a smaller number of servers thus the traffic is mostly one-way road - most
of the data is coming from the servers toward the clients. In my opinion in
the test environment the clients and servers (receivers and senders) should
be also swept places. The result may be calculated as an average value of
both tests.

There is also question regarding which traffic is measured - UDP or TCP. In
case of TCP there is an issue of acknowledges sent in the opposite
direction. But we can ignore that if the clients and servers are swept.
((Can you please define the term "swept places"? I'm not familiar with it.
Past versions of the draft had included definitions of unidirectional and
bidirectional traffic, but the consensus within the bmwg was that the
definitions were not sufficiently different from existing definitions to
merit inclusion here.

This purpose of this memo is to describe *terminology* used in benchmarking
firewall performance. The bmwg generally follows a two-step process, with a
terminology document first and a methodology second. Traffic direction is
very germane to the methodology document.))

Another issue is which services are used in the test. I think that
performance of the application-proxy firewall depends on which service is
used - HTTP, FTP, SMTP etc. The differences may be very noticeable.
((Indeed. I've included definitions of proxies in this terminology document.
Describing which proxies are invoked is a methodology issue, however.))

Now is another point - how the firewall should be prepared to the tests. I
think it is a good idea that (like in the past tests) vendor's
representatives should prepare the firewall for performance tests, but it is
crucial that after that preparation and before the performance tests, the
firewall is tested for security. It must be proven that there is no security
degradation when the firewall is configured for the performance tests. It
may be also a good idea to perform the security tests even after the
performance tests.
((Again, this is largely a methodology issue. This is going to be a very
difficult issue to sort out.

The task of "proving there's no security degradation" is very open-ended, to
say the least. The terminology document only contemplates that a given rule
set is enforced. What about security issues outside the rule set, like the
use or nonuse of NAT and hardening of the device against various
denial-of-service attacks? I don't have an answer to this as yet and would
welcome your thoughts.))

	Next time I'll write about connections and sessions..

((I'd very much appreciate your input on this as well.))

	I'll be glad to hear your opinions.

Edward Gonen
Vice President of R&D
NetGuard Ltd.

================================================================
eMail: [email-masked] <mailto:[email-masked]>
<mailto:[email-masked] <mailto:[email-masked]> >
Phone:+972-6-6449944 Mobile:+972-52661113 Fax:+972-6-6540124
URL:http://www.ntguard.com <URL:http://www.ntguard.com>
<URL:http://www.ntguard.com <URL:http://www.ntguard.com> >
Address: LanOptics Building, P.O.Box 184,
Ramat Gabriel Industrial Park, Migdal Ha-Emek 10551, Israel

---

From: "Michael C. Richardson" <[email-masked]>
Date: Fri, 26 Jun 1998 16:22:26 -0400

>>>>> "Kevin" == Kevin Dubray <[email-masked]> writes:
    >> This is wonderful as clarifying concept or term on which to build.  As
    >> a stand-alone metric, I'm not so sure .  You referred to measuring
    >> connection concurrency.  This is something that I can and probably
    >> should measure wrt firewalls.  A connection overhead metric may yield
    >> better insight how an IUT "lifts" or handles connection maintainance.

    Kevin> Kevin, I'd like a clarification of what's meant by "connection
    Kevin> overhead." The memory cost of opening a socket is

  Given n current connections, how much time does it take for the n+1'th
connection to be opened.

    Kevin> implementation-specific, and a DUT/SUT that uses 100 kbytes of
    Kevin> memory for each connection may or may not move bits on the wire
    Kevin> more slowly than one requiring only 10 kbytes. That's a firewalls

  The internals issue is more to do with the kind of data structure that
is used to store the connection state. Also, whether or not a new process
needs to be created to handle that connection.

    Kevin> Another option is to make the connection definition(s) very
    Kevin> specific to the workings of TCP, UDP, and ICMP, and count only
    Kevin> those conversations which have successfully navigated the setup
    Kevin> procedures of each protocol.  I'm reluctant to do that; better to
    Kevin> have a more general definition.

  ICMP isn't a protocol in the same way TCP and UDP are. TCP and UDP make use
of IP services. Some of these services are carried by ICMP datagrams.
  A firewall that doesn't consider an ICMP MUST FRAGMENT to be associated
with a TCP connection (i.e. Path MTU discovery) is violating the IP protocol.
[The firewall needn't let it in: it could process it locally, adjusting
its own view of fragment size and perhaps emitting an ICMP out the other
end.]

  The only ICMP that may consistute a protocol *may* be ECHO/ECHO RESPONSE.

   :!mcr!:            |  Sandelman Software Works Corporation, Ottawa, ON  
   Michael Richardson |	SSH IPsec: http://www.ssh.fi/. Secure, strong, international
 Personal: <A HREF="http://www.sandelman.ottawa.on.ca/People/Michael_Richardson/Bio.html">[email-masked]</A>. PGP key available.
 Corporate: <A HREF="http://www.sandelman.ottawa.on.ca/SSW/">[email-masked]</A>.

---

From: David Newman <[email-masked]>
Date: Fri, 26 Jun 1998 17:00:52 -0400

>   Given n current connections, how much time does it take for the n+1'th
Great! That's a very good way of expressing connection setup time.

But we have to define what's meant by connection in the first place. .
.I'd be interested in your comments on the original proposal:

A state in which nodes agree to exchange data using a known protocol.

> needs to be created to handle that connection.

Let's not get into internals; this is an effort to benchmark
characteristics observed on the *outside* of the box. ;-)

dn