Subject: Re: Naming those hunks of data

From: David Newman <[email-masked]>
Date: Fri, 02 Oct 1998 19:05:48 -0400

Thanks to all for the constructive comments. 

Let's get back to the original problem: BMWGers raised concerns that a
bit- or byte-based measurement of forwarding rate was insufficient _by
itself_ in describing forwarding rate. After some characteristically
bullheaded resistance on my part, I agreed that having some sort of
datagram/packet/frame/hunk o' data definition would be useful. 

I thought it was these discrete pieces of data that needed naming, and
that was the task I had in mind in proposing "transaction." But I 
overshot the mark when I cited some examples that were packet-like and
others that were session-like (an FTP session, a DNS query and reply). 

Complete-session measurements are _very_ useful, to be sure, but they're
not atomic. I don't think we can use a session metric without first
defining what goes into a session.

I'd propose we add two sets of definitions here: One to describe
individual pieces of data, and one to describe a complete session and
its subcomponents.

In the "pieces of data" category, I think steady-state forwarding rate
is the thing we're after, not setup and teardown costs or the cost of
security association (we can and should define those as subcategories of
a session). What I really want to know is how many frames/packets/cells
I can push through a firewall interface each second. If I'm wrong on
this, please correct me, but I believe this is the essential goal of a
datagram-type metric.

With all the good suggestions made, I'm hard-pressed to choose one
label--they all work pretty well. In my mind "units of transfer"
narrowly edges out "benchmarking units" or "benchmarking data units"
only because in our acronym-laden industry, something beginning with b
will be shortened to bps and that will lead to confusion (especially
since it's not bits per second!). At least with "unit of transfer," the
acronym is ups and presumably there won't be any confusion with
uninterruptible power supplies (well, one never knows...).

Here's a proposed definition:

-----------------------------

3.XX Unit of transfer

Definition:
A discrete collection of bytes comprising at least one header and
optional user data.

Discussion:
This metric is intended chiefly to describe steady-state forwarding rate
of the DUT/SUT. 

The unit of transfer (UOT) definition is deliberately left open to
interpretation, allowing the broadest possible application. Examples of
UOTs include TCP segments, IP packets, Ethernet frames, and ATM cells.

While the definition is deliberately broad, its interpretation should
not be. The tester MUST describe what type of UOT will be offered to the
DUT/SUT, and MUST offer these UOTs at a consistent rate. Traffic
measurement MUST begin after all session establishment routines complete
and before any session completion routine begins. Further, measurements
MUST begin after any security associations (SAs) are established and
before any SA is revoked.

Testers also MUST compare only like UOTs. It is not appropriate, for
example, to compare forwarding rates by offering 1,500-byte Ethernet
UOTs to one DUT/SUT and 53-byte ATM cells to another. 

Measurement units:
Units of transfer
Units of transfer per second

Issues:

See also:
Bit forwarding rate
Session

I'll still need to add definitions for sessions and their pieces
(establishment, teardown, and security association). Before I do, I
wanted to check if other BMWGers think this is the way to go.

dn

> At 22:55 01.10.98 -0400, [email-masked] wrote:
> >     Harald:
> >
> >     True enough; however, within the unit of work, you are masking a
> >     multitude of sins (or virtues, depending on your view point  ;->), in
> >     that different authentication mechanisms and/or encryption algorithms
> >     will add varying amounts of overhead to the transfer of "the first
> >     byte of the HTTP command arriving to the last byte of the HTTP
> >     response."  Because of this "apples to oranges" problem, I suggest
> >     that the elementary unit of data transfer be measured only after:
> >
> >     1. All security associations have been checked and verified
> >     2. All connection oriented hands have been shaken
> >
> >     Again, this is not to say that the time required to complete the above
> >     two operations should not be part of the benchmarking terminology and
> >     methodology.   Rather, I suggest that these measurements be separate
> >     from the actual data transfer, which allows a more unbiased assessment
> >     of a particular firewall's capability.
> 
> Maybe.....I think this is a case-by-case thing; the ways in which setup
> cost is associated with units of work can be many and confusing
> (in HTTP/1.0 "traditional" it would make much more sense to measure from the
> SYN bit to the FIN bit, since every page lookup required a separate TCP
> connection).
> (OTOH, in HTTP, authentication typically requires two transactions; one
> to know that it's needed, and one to attempt to authenticate. A third
> lookup in the same auth-domain would not require a third round)
> 
>                 Harald A