Subject: Re: bmwg Internet tunnelling protocol benchmarking

From: [email-masked]
Date: Wed, 3 Feb 1999 13:57:51 -0500

Some questions:

Where do you want to test? desktop to desktop or network node to network
node?

I assume that you want to benchmark, meaning, lab conditions not Internet
conditions.

Or are you primarily interested in how performance varies in the face of
[Internet-like] ambient traffic conditions leading to loss, high latency,
etc.?

Jim McQuaid
NetScout Systems

[email-masked] on 02/02/99 06:48:35 PM

To:   [email-masked]
cc:    (bcc: Jim McQuaid/NetScout)
Subject:  bmwg Internet tunnelling protocol benchmarking

Hello BMWG-

At Kevin Dubray's suggestion, I would like to solicit your opinions on an
appropriate way to benchmark tunnelling protocol implementation such as
GRE.
In my experience, I find that performance measurements of networking gear
as
per Scott Bradner's informational RFC fairly closely represents what we can
expect in "real" world use.  However, due to the variability in Internet
conditions, it seems nearly impossible to come up with a testing
methodology
for tunnelling protocols that will provide the same applicability of data.

Outside of the actually variability of Internet conditions, the protocol
and/or encryption implementation could further aggravate the end node's
actual throughput.  It is possible such a protocol implementation could be
excellent in a no/low loss situation and very poor once packet rates and
latency increases.  This is what I am trying to avoid.

Any experience or thoughts you may have on this are invited.  If such
discussion leads to a possible methodology and the consensus is that IP
tunnelling is a viable technology, I would be very interested in helping to
draft an informational RFC.

Sincerely-
Ron Johnson
Sr. Network Engineer

---

From: Ron Johnson <[email-masked]>
Date: Wed, 3 Feb 1999 11:56:04 -0800

Currently, my test set up is very basic with two routers connected via a
rolled cable and Smartbits ports on the outer edges.  This is my no
loss/negligible latency scenario.  In the next week or so, I plan on
contstructing a "simulated" Internet with two more routers connected via an
Adtech WAN simulator.  I have access to both a T1/multiplexed T1 and a DS3
simulator.  This allows me to introduce latency and I need to do some
research on whether their ability to introduce bit errors would be an
effective simulation of dropped packets.

So I guess the answer to your questions are that this is a lab set up with
both pristine conditions and I will soon have a simulation of less than
perfect Internet transport.

Ron

> -----Original Message-----
> From:	[email-masked] [SMTP:[email-masked]]
> Sent:	Wednesday, February 03, 1999 10:58 AM
> To:	Ron Johnson
> Cc:	[email-masked]
> Subject:	Re: bmwg Internet tunnelling protocol benchmarking
> 
> Some questions:
>

---

From: Ron Johnson <[email-masked]>
Date: Wed, 3 Feb 1999 13:55:09 -0800

I will look into all of these.  Thanks much for the suggestions.

> From:	A Mukherjee [SMTP:[email-masked]]
> Sent:	Wednesday, February 03, 1999 2:02 PM
> To:	Ron Johnson; [email-masked]
> Cc:	[email-masked]; Hans-Werner Braun (E-mail)
> Subject:	RE: bmwg Internet tunnelling protocol benchmarking
> 
> Ron:
> 
> My name is Amarnath Mukherjee, and I work in benchmarking/network
> performance analysis. What you are doing is very interesting.
> 
> My $0.02:
> 
> 1.
> If you need traffic workload, check out Hans Werner Braun's NLANR website:
> 	http://moat.nlanr.net
> NLANR (National Laboratory for Advanced Networking Research) is collecting
> multiple CDs 
> worth of data from multiple backbone sites each day.
> 
> 2.
> Another possibility: for the AdTech simulator, use a time-series model to
> generate realistic traffic workloads. 
> We have  a paper on this on www.knoltex.com. See "Time Series Models of
> Internet Traffic" among "Traffic Papers"
> 
> 2a.
> If you want to model errors due to buffer overflows (the usual case), you
> can explicitly do it in the simulator. 
> 
> 2b.
> For bit errors, one could use an iid model or a 2-state markov chain (one
> state is when line is relatively clean, another
> when it is noisy). The latter makes sense for wireless, e.g., because of
> weather conditions, but do not know
> if it makes sense for wired networks.
> 
> Regards,
> 
> 
> -----------------------------------------------
> Amarnath Mukherjee
> Knoltex Corp.
> 
> Email: [email-masked]
> Tel:    408-615-1611
> Web:  http://www.knoltex.com
> Surface Mail: Box 9707, San Jose, CA 95157
> 
> 
> From:	Ron Johnson [SMTP:[email-masked]]
> Sent:	Wednesday, February 03, 1999 11:56 AM
> To:	[email-masked]
> Subject:	RE: bmwg Internet tunnelling protocol benchmarking
>

---

From: A Mukherjee <[email-masked]>
Date: Wed, 3 Feb 1999 14:02:14 -0800

Ron:

My name is Amarnath Mukherjee, and I work in benchmarking/network performance analysis. What you are doing is very interesting.

My $0.02:

1.
If you need traffic workload, check out Hans Werner Braun's NLANR website:
	http://moat.nlanr.net
NLANR (National Laboratory for Advanced Networking Research) is collecting multiple CDs 
worth of data from multiple backbone sites each day.

2.
Another possibility: for the AdTech simulator, use a time-series model to generate realistic traffic workloads. 
We have  a paper on this on www.knoltex.com. See "Time Series Models of Internet Traffic" among "Traffic Papers"

2a.
If you want to model errors due to buffer overflows (the usual case), you can explicitly do it in the simulator. 

2b.
For bit errors, one could use an iid model or a 2-state markov chain (one state is when line is relatively clean, another
when it is noisy). The latter makes sense for wireless, e.g., because of weather conditions, but do not know
if it makes sense for wired networks.

Regards,

Email: [email-masked]
Tel:    408-615-1611

-----Original Message-----
From:	Ron Johnson [SMTP:[email-masked]]
Sent:	Wednesday, February 03, 1999 11:56 AM
To:	[email-masked]
Cc:	[email-masked]
Subject:	RE: bmwg Internet tunnelling protocol benchmarking

Ron

---

From: Ron Johnson <[email-masked]>
Date: Tue, 9 Feb 1999 11:08:54 -0800

I think the problem that I am trying to address is somewhat simple in
concept but complicated in practice.  I think it is best summarized by the
question:  "How does one performance test a VPN/tunneling protocol
implementation in the lab so as to provide a (somewhat) accurate
representation of what one can expect on the true Internet or other IP
network?"  This one question has a couple of issues that need to be
addressed.

Some of the information I saw on the NLANR was very useful in classifying
Internet behavior at large.  This is issue #1.  If you pick a certain
scenario to simulate, you can say that Internet performance is this good or
better 99% of the time so issue #1 is possibly contained.  But as we use
this data to simulate different Internet conditions, we have to consider
what we do with the data with regard to different protocol implementations.
This is issue #2.  This is the subjective, difficult to gain consensus area.

If I am able to construct a "realistic" Internet simulation in my lab over
which I run a tunneling protocol, how do I know which perf numbers to use?
Virtually every tunneling protocol is going to add packet loss to the
existing loss of the system. This can be due to encryption methods, large
packet fragmentation, etc.. How much additional packet loss is acceptable so
I can determine which perf numbers to report?  If I say introduced packet
loss +0% additional loss, this may be an impossible mark to hit due to
aformentioned complications of tunneling.  So I have to say at introduced
loss +X% is where I am going to say my acceptable amount of loss is and
therefore the point where I take my perf numbers.  So who is to say what "X"
is and why don't I use "2X" since I could probably show some performance
improvement if I allow for a bit more packet loss?

I hope this makes more sense.  In a nutshell, I can't use the existing line
in the sand of no dropped packets.  I also can't test this in a scenario of
no loss.  Analysis of Internet traffic patterns will help me draw some
boundaries for my simulations but then the analysis of your results becomes
somewhat subjective depending upon how much additional loss the tester deems
"acceptable."  If you ask a developer how much loss to accept, somehow I
believe the answer is going to be "wherever we get the best numbers as
compared to the competition."  ;-) Better yet, I often hear that a
particular protocol was designed for certain Internet conditions so we
should test under those conditions.  With either situation, if you have to
compare two different tunneling implementations you end up with an apples to
watermelons comparison.  And even if I test them in the same scenarios which
I deem "realistic" or "acceptable," this will probably not coincide with
another lab.  As an engineer, I am looking for some standardization of
methods for testing VPNs so the data is somewhat universally applicable.
>From there, you can design tunneling infrastructures, predict their deployed
throughput (within reason), and make the case why the protocol you picked
was the appropriate choice.

Thanks-
Ron

> From:	Kevin Dubray [SMTP:[email-masked]]
> Sent:	Tuesday, February 09, 1999 7:06 AM
> To:	Ron Johnson
> Ron Johnson wrote:
> Hi Ron,
> I would presume that the folks working on MPLS are betting that
> tunnelling is a viable Internet technology! ;-)
> 
> In any event, your original mail brings up several good points.  I
> glean two central themes:
> 
>      1. Ways to benchmark tunnelling technologies.
>      2. Need to inject "realism" into clinical evaluations.
> 
> But in order to make progress, I believe you need to answer the 
> first question Mr. McQuaid put forth, namely, what do you want
> to measure and where?
> 
> If attempting to characterize application-level performance using a
> given end-to-end network technology, then simulating the aggregated
> loss/delay/sequencing errors/etc. propagated by an abstracted network
> may give the understanding that you seek.(Much work has been done in
> this area for TCP, so you already have founding work.)
> 
> It may be beneficial to try to characterize the forwarding 
> performance of networking element using protocols that are dependent 
> on a historical event as a condition of forwarding (packet sequencing,
> packet reassembly, etc.).  Then, again, not all tunnelling protocols
> have such requirements, so loss may have no impact on the
> discrete action of forwarding by NEs.
> 
> Bottomline, there are many ways to skin the cat. The question is
> _which_ cat?  ;-)  If the BMWG were to formally undertake such work, I
> for one, would like to see a crisper problem set so that we could
> make forward progress. 
> 
> -Kevin

---

From: Ron Johnson <[email-masked]>
Date: Tue, 9 Feb 1999 13:57:55 -0800

This is an interesting definition which I had not considered up to this
point.  I had been more focused on bulk throughput without regard to
individual streams but I'll work through some numbers and see if I can make
this scale from a few convesations up to thousands.  I appreciate the idea.

What I find interesting is that there are people using tunneling right now
except without a standard benchmarking method. I guess they must be modeling
their own view of the Internet or their Internet peering policy and testing
each protocol as they expect to deploy it.  Does anyone know how those who
deploy VPNs set expectation for performance?

Thanks-
Ron

> Sent:	Tuesday, February 09, 1999 12:41 PM
> To:	Ron Johnson
> Cc:	[email-masked]; [email-masked]
> A couple years ago at a BMWG meeting, I proposed a throughput / goodput
> definition which was partly aimed at the question of performance in the
> face of imperfection (real world traffic conditions).
> 
> "Throughput" in the canonical definition is the performance with no loss.
> I proposed a set of sliding assumptions about a traffic stream on which to
> base a "loss / credit" scheme.  My focus at the time was router testing
> but
> it might fit your needs.  "Effective throughput" was the name I gave it.
> 
> I proposed that a testing scenario state an assumption about how many
> individual streams or conversations might be borne within a given flow
> (say
> 25).  Then assume that every lost packet eventually causes some higher
> layer retransmission event, e.g. TCP.  In a standard ACK window
> environment
> (not SACK) we can state assumptions about how long this takes and how much
> of the flow (1/25 times n seconds) is effectively wasted.  The "goodput"
> or
> effective throughput is the net of this calculation.
> 
> Obviously there are many possible assumptions and the assumptions can be
> slid to one end or the other of the scale, e.g. assume ONE conversation
> and
> the whole bandwidth for, say one second, is lost by a single packet loss.
> Assume 5000 conversations and only 1/5000th of one second's bandwidth is
> "lost."
> 
> The suggestion, by the way, was soundly rejected by the group at the time!
> One reason was that the range of possible assumptions was too wide and a
> closely related point was that there was absolutely NO consensus about
> what
> normal traffic and normal loss might be.  Both of these are open to
> question still, but I think it might be a starting point for you.
> 
> Jim McQuaid
> NetScout Systems
> 
> 
> 
> 
> 
> 
> 
> [email-masked] on 02/09/99 02:08:54 PM
> 
> To:   [email-masked]
> cc:   [email-masked] (bcc: Jim McQuaid/NetScout)
> > > tunnelling is a viable technology, I would be very interested in
> helping
> > to

---

From: Vern Paxson <[email-masked]>
Date: Tue, 09 Feb 1999 16:21:05 PST

> individual streams or conversations might be borne within a given flow (say
> layer retransmission event, e.g. TCP.  In a standard ACK window environment
> of the flow (1/25 times n seconds) is effectively wasted.  The "goodput" or

Note, there is recent theoretical work on analytic formula correlating loss
rate and RTT with TCP throughput.  The most recent appeared in SIGCOMM '98,
"Modeling TCP Throughput: A Simple Model and its Empirical Validation", J.
Padhye, V. Firoiu, D. Towsley and J. Kurose, no doubt available off of
umass.edu somewhere.

		Vern