Subject: RE: bmwg Internet tunnelling protocol benchmarking

From: [email-masked]
Date: Tue, 9 Feb 1999 15:41:25 -0500

A couple years ago at a BMWG meeting, I proposed a throughput / goodput
definition which was partly aimed at the question of performance in the
face of imperfection (real world traffic conditions).

"Throughput" in the canonical definition is the performance with no loss.
I proposed a set of sliding assumptions about a traffic stream on which to
base a "loss / credit" scheme.  My focus at the time was router testing but
it might fit your needs.  "Effective throughput" was the name I gave it.

I proposed that a testing scenario state an assumption about how many
individual streams or conversations might be borne within a given flow (say
25).  Then assume that every lost packet eventually causes some higher
layer retransmission event, e.g. TCP.  In a standard ACK window environment
(not SACK) we can state assumptions about how long this takes and how much
of the flow (1/25 times n seconds) is effectively wasted.  The "goodput" or
effective throughput is the net of this calculation.

Obviously there are many possible assumptions and the assumptions can be
slid to one end or the other of the scale, e.g. assume ONE conversation and
the whole bandwidth for, say one second, is lost by a single packet loss.
Assume 5000 conversations and only 1/5000th of one second's bandwidth is
"lost."

The suggestion, by the way, was soundly rejected by the group at the time!
One reason was that the range of possible assumptions was too wide and a
closely related point was that there was absolutely NO consensus about what
normal traffic and normal loss might be.  Both of these are open to
question still, but I think it might be a starting point for you.

Jim McQuaid
NetScout Systems

[email-masked] on 02/09/99 02:08:54 PM

To:   [email-masked]
cc:   [email-masked] (bcc: Jim McQuaid/NetScout)
Subject:  RE: bmwg Internet tunnelling protocol benchmarking

I think the problem that I am trying to address is somewhat simple in
concept but complicated in practice.  I think it is best summarized by the
question:  "How does one performance test a VPN/tunneling protocol
implementation in the lab so as to provide a (somewhat) accurate
representation of what one can expect on the true Internet or other IP
network?"  This one question has a couple of issues that need to be
addressed.

Some of the information I saw on the NLANR was very useful in classifying
Internet behavior at large.  This is issue #1.  If you pick a certain
scenario to simulate, you can say that Internet performance is this good or
better 99% of the time so issue #1 is possibly contained.  But as we use
this data to simulate different Internet conditions, we have to consider
what we do with the data with regard to different protocol implementations.
This is issue #2.  This is the subjective, difficult to gain consensus
area.

If I am able to construct a "realistic" Internet simulation in my lab over
which I run a tunneling protocol, how do I know which perf numbers to use?
Virtually every tunneling protocol is going to add packet loss to the
existing loss of the system. This can be due to encryption methods, large
packet fragmentation, etc.. How much additional packet loss is acceptable
so
I can determine which perf numbers to report?  If I say introduced packet
loss +0% additional loss, this may be an impossible mark to hit due to
aformentioned complications of tunneling.  So I have to say at introduced
loss +X% is where I am going to say my acceptable amount of loss is and
therefore the point where I take my perf numbers.  So who is to say what
"X"
is and why don't I use "2X" since I could probably show some performance
improvement if I allow for a bit more packet loss?

I hope this makes more sense.  In a nutshell, I can't use the existing line
in the sand of no dropped packets.  I also can't test this in a scenario of
no loss.  Analysis of Internet traffic patterns will help me draw some
boundaries for my simulations but then the analysis of your results becomes
somewhat subjective depending upon how much additional loss the tester
deems
"acceptable."  If you ask a developer how much loss to accept, somehow I
believe the answer is going to be "wherever we get the best numbers as
compared to the competition."  ;-) Better yet, I often hear that a
particular protocol was designed for certain Internet conditions so we
should test under those conditions.  With either situation, if you have to
compare two different tunneling implementations you end up with an apples
to
watermelons comparison.  And even if I test them in the same scenarios
which
I deem "realistic" or "acceptable," this will probably not coincide with
another lab.  As an engineer, I am looking for some standardization of
methods for testing VPNs so the data is somewhat universally applicable.
>From there, you can design tunneling infrastructures, predict their
deployed
throughput (within reason), and make the case why the protocol you picked
was the appropriate choice.

Thanks-
Ron

> -----Original Message-----
> From:   Kevin Dubray [SMTP:[email-masked]]
> Sent:   Tuesday, February 09, 1999 7:06 AM
> To:     Ron Johnson
> Cc:     [email-masked]
> Subject:     Re: bmwg Internet tunnelling protocol benchmarking
>
> Ron Johnson wrote:
> >
> >
> > Any experience or thoughts you may have on this are invited.  If such
> > discussion leads to a possible methodology and the consensus is that IP
> > tunnelling is a viable technology, I would be very interested in
helping
> to
> > draft an informational RFC.
> >
> > Sincerely-
> > Ron Johnson
>
> Hi Ron,
>
> I would presume that the folks working on MPLS are betting that
> tunnelling is a viable Internet technology! ;-)
>
> In any event, your original mail brings up several good points.  I
> glean two central themes:
>
>      1. Ways to benchmark tunnelling technologies.
>      2. Need to inject "realism" into clinical evaluations.
>
> But in order to make progress, I believe you need to answer the
> first question Mr. McQuaid put forth, namely, what do you want
> to measure and where?
>
> If attempting to characterize application-level performance using a
> given end-to-end network technology, then simulating the aggregated
> loss/delay/sequencing errors/etc. propagated by an abstracted network
> may give the understanding that you seek.(Much work has been done in
> this area for TCP, so you already have founding work.)
>
> It may be beneficial to try to characterize the forwarding
> performance of networking element using protocols that are dependent
> on a historical event as a condition of forwarding (packet sequencing,
> packet reassembly, etc.).  Then, again, not all tunnelling protocols
> have such requirements, so loss may have no impact on the
> discrete action of forwarding by NEs.
>
> Bottomline, there are many ways to skin the cat. The question is
> _which_ cat?  ;-)  If the BMWG were to formally undertake such work, I
> for one, would like to see a crisper problem set so that we could
> make forward progress.
>
> -Kevin

---

From: "Cynthia.E.Martin" <[email-masked]>
Date: Thu, 18 Feb 1999 14:42:35 -0500

All,

Jeff Dunn and I resubmitted the Call/Cell ID and submitted the new Frame
Relay ID.  At the last IETF, we determined it would be best to separate
ATM and Frame Relay into two separate terminology documents.  Please
refer to the following web site for the presentation given at the last
IETF, these slides give a good overview of the objectives for these
ID's:

http://www.ietf.org/proceedings/98dec/slides/bmwg-cell-98dec/index.html

A brief summary of our strategy is as follows:
1) Provide terminology related to the technology and measurements as it
relates to TCP/IP over a particular link layer.
2) Utilizing the terminology ID, generate the methodology ID as it
relates to TCP/IP over a particular link layer.

A brief summary of many changes in the Cell/Call ID follows:
1) Our focus is the effect of the physical layer, link layer, and
adaptation layer impairments on TCP round trip time calculation, packet
and segment loss and/or corruption, TCP performance and TCP throughput.
2) We believe that several of the ATM traffic types, such as UBR and
ABR, negatively impact TCP services; however, in some instances and with
particular rate values, the impact can be minimized.

We are hoping that everyone will have an opportunity to read the ID's
and comment on them before the IETF in March. 

Thanks
Cynthia Martin

-- 
Cynthia E. Martin
Phone: 410 730 6300
FAX: 410 730 1412
Email: [email-masked]