Subject: RE: Latency and load

From: David Newman <[email-masked]>
Date: Fri, 20 Aug 1999 10:33:03 -0400

Hi Jim,

Comments inline:

> 1.	Given that there is a maximum throughput with (probably) maximum
> latency, we establish the OUTSIDE limit of performance for highest sustained
> forwarding rate.  "Latency@throughput_rate"
> 
> 2.	Does the device have some OTHER performance "edge" where it exhibits
> the lowest possible latency at some [less than throughput] rate?  And what
> is this rate?  This is "rate@min_latency"
> 
> If believe that devices will exhibit lowest latency at very low loads and
> that latency will increase in various jumps and increments until
> "latency@throughput_rate" is reached.  

Two comments here:

1. I like the two metrics! But rate@min_latency isn't necessarily a low
number. I've seen numerous cases where the latency "knee" is well above
90 percent load. Thus, there might not be a significant difference in
min_latency at 1 percent and 99 percent load.

2. Some sort of step test that finds latency@tput and rate@min_latency
would be interesting. Maybe this is something for the editors of the
switch methodology to consider.

> After that, packets are dropped.  For
> each device architecture, there is probably a graph showing these jumps or
> "edges" within the maximum performance envelope.
> 
> The related question is: why bother?  Clearly the max load performance is
> major constraint.  

This isn't necessarily true. As I mentioned, I have benchmarked devices
where latency is orders of magnitude lower when the offered load is just
below max (say, 98 percent or 96 percent). Further, I know of cases
where the "knee" for forwarding rate is *different* than the knee for
latency (i.e., I can still get terrible latency numbers even when there
is no frame loss). 

For some applications it would be very relevant to know rate@min_latency
as well as latency@tput.

Is it possible to control offered load in practice to the
> point of achieving lower latency most of the time?  What about latency per
> QOS stream?  I think these are all fascinating topics but I'm not sure about
> the benchmarking value of most of them.

There are plenty of ISPs and their customers that would be *very*
interested in benchmarking the latency of a particular QOS stream. The
question is, How should that measurement be taken? 

As a long-time student of the stress-testing-is-good school, I share
your reluctance to try to define "non-perfect performance." I am VERY
allergic to suggestions that the offered load for latency tests must be
98 percent or 50 percent or whatever, because whatever will differ by
device. 

But I don't like ambiguity either. RFC 2544's approach might produce a
far higher latency measurement than the ISP would see in practice. IMO
some sort of step testing would be more useful than a single data point
that says, essentially, this box will always have terribly high latency.

dn

---

From: Steve Bell <[email-masked]>
Date: Fri, 20 Aug 1999 09:08:51 -0700

This is a test methodology problem we've encountered in virtual every switch
we've tested.  Early on, we went back to 1944/2544 getting the idea to
measure latency while offering a load at the throughput rate.  However that
isn't very satisfactory, as we've found many devices exhibit interesting
behaviours with higher latency at the throughput rate, falling off
dramatically below that, etc.  There seem to be several different "U" shaped
latency curves, ramping up at high and low extremes of offered loads (with a
range of Polynomials to describe them :).

As an interim solution, in a full benchmarking where we're interested in
latency we typically run a "latency profile" with a 10% step, offering a 10%
to 100% load, using three frame sizes.  Between the throughput rate and 100%
we sometimes need to decrease the step size to obtain a good profile.
However, a better methodology is needed.  Measuring latency at a light load
(15%), moderate (50%), throughput rate, and 100% offered load (at two or
three frame sizes) would in most cases be equally useful, and much faster to
test.  There is probably a superior solution that could be devised, though
by those on this reflector!  A uniform solution is needed, I guess 1944
neglected this a bit since it was conceived more in the era of router
testing.

Steve Bell
The Silicon Valley Networking Lab, Inc.
San Jose, CA
www.svnl.com