Subject: RE: Latency and load

From: Jim McQuaid <[email-masked]>
Date: Fri, 20 Aug 1999 08:39:59 -0400

David, thinking out loud about the issue you raise.

The theory about latency & throughput (in the canonical 1242 sense) is that
since throughput is, by definition, the steady state forwarding rate of the
DUT,  it should be able to pump frames all day at this rate without dropping
any.  The latency under this load clearly will include full buffers, queues,
hash tables whatever and should exhibit (my belief) the *highest* latency
the device will ever have in practice.  In fact any higher latency will
amount to dropping a frame.  

Obviously, pipelined designs increase total throughput but add latency which
is what you are getting at.  So the real question you are raising (I think)
is this: 

1.	Given that there is a maximum throughput with (probably) maximum
latency, we establish the OUTSIDE limit of performance for highest sustained
forwarding rate.  "Latency@throughput_rate"

2.	Does the device have some OTHER performance "edge" where it exhibits
the lowest possible latency at some [less than throughput] rate?  And what
is this rate?  This is "rate@min_latency"

If believe that devices will exhibit lowest latency at very low loads and
that latency will increase in various jumps and increments until
"latency@throughput_rate" is reached.  After that, packets are dropped.  For
each device architecture, there is probably a graph showing these jumps or
"edges" within the maximum performance envelope.

The related question is: why bother?  Clearly the max load performance is
major constraint.  Is it possible to control offered load in practice to the
point of achieving lower latency most of the time?  What about latency per
QOS stream?  I think these are all fascinating topics but I'm not sure about
the benchmarking value of most of them.

my thoughts,

Jim McQuaid
Ganymede Software

-----Original Message-----
From: David Newman [mailto:[email-masked]]
Sent: Thursday, August 19, 1999 4:46 PM
To: [email-masked]
Subject: Latency and load

BMWGer's,

I'd like to start a discussion regarding the offered load to be used
when benchmarking latency of routers and switches.

I'm asking for two reasons:

1. Numerous devices from numerous vendors exhibit relatively high
latency when offered loads are at line rate, and substantially lower
latency with lower offered loads.

I believe this occurs because of congestion internal to the DUT/SUT. In
other words, congestion can still occur even when the offered load
itself should not produce congestion. In my experience, reducing the
offered load (to somewhere between 1 percent and 99 percent--it varies
by device) will clear the internal congestion and produce far lower
latency measurements.

2. I haven't been able to locate clear documentation on what offered
load should be used when making latency measurements. Section 26.2 of
RFC 2544 instructs testers to "first determine throughput" and then
"send a stream of frames ... at the determined throughput rate." 

This is problematic, since there isn't necessarily a strong correlation
between throughput and latency. I have encountered devices with
line-rate throughput that nonetheless exhibit less-than-optimal latency
when the offered load is 100 percent. Although RFC 1242 makes clear that
throughput doesn't necessarily mean line rate, there's still a problem:
I know of at least one device where throughput is 99 percent of the
theoretical maximum, yet latency doesn't settle down until the offered
rate is 96 percent or less. 

Maybe this issue should be clarified in the switching methodology. 

I would be very curious to hear what offered loads other BMWGer's use in
measuring latency.

dn

---

From: David Newman <[email-masked]>
Date: Fri, 20 Aug 1999 10:04:59 -0400

I have observed the phenomenon described with multiple vendors' test
equipment, including Radcom's. 

In this regard no vendor's test equipment is "truer" than any other, as
loopback tests will demonstrate. This is about methodology, not choice
of test equipment.

dn

> It's quite possible that your test equipment  may be applying a timestamp to
> the message prior to transmission.
> 
> If this is the case, then the measured latency will reflect the time in
> queue awaiting transmission at the test equipment.
> 
> You can either calibrate your tests by looping the test equipment back on
> itself (if possible) in order to evaluate the latency due just to the
> tester, or better yet, apply a dual port monitor at the ingress and egress
> points to make a truer measurement.
> 
> 
> Mike Winslow
> Director of Products, North America
> Radcom Equipment Inc
> 1-800-RADCOM4 ext. 304
> (pager) 1-800-609-9283
> (car)  201-220-3218
> (text page) [email-masked]
> www.radcom-inc.com
> 
> 
>

---

From: David Newman <[email-masked]>
Date: Sun, 22 Aug 1999 21:30:05 -0400

Hi John,

> At 99% load, the device may not be able to completely keep up with the
> input, but it can be close enough such that it can float the difference in
> it's buffers.  This affects the latency measurement.  Each packet received
> drives a little deeper into the buffers and hence has a slightly worse
> latency.  You can test this theory by making several latency measurements:
> one at five seconds, one at ten seconds, and one at twenty seconds.  

As usual, you've come up with an accurate description of the problem and
an innovative way of solving it. So far all the discussion has assumed
that we should hold duration constant and use load as the variable;
you've switched the two. 

In the end I'm not sure there will be a difference, since varying
duration *or* load will produce the buffering effect you describe with
different measurements resulting at various points. 

There's been a lot of discussion recently about what duration to use for
forwarding rate measurements, with a consensus settling around a
30-second standard (and a requirement that any deviation from this MUST
be declared).  

Do you think latency measurements *shouldn't* use a single duration and
varying offered loads? I think they're the same, but I may be missing
something.

Thanks

dn

---

From: David Newman <[email-masked]>
Date: Sun, 22 Aug 1999 21:55:36 -0400

Hi Steve,

> Our test engineers tell me that a 30 second test has been adequate for most
> of the 100Mbps switches
> being tested today, to avoid skewing the results due to buffering dynamics.
> For example, we just tested a
> workgroup switch from a far-Eastern manufacturer -- running the latency test
> at 30, 60, and 90 seconds.
> The results showed the same latency regardless of the test period, to a
> fraction of a microsecond.

This sort of steady-state behavior is not the problem I'm describing.
Rather, I've seen cases where a switch will exhibit a small amount of
frame loss during a 30-second run--and it turns out any loss, even if
trivial, is a *very* significant event in terms of its impact on
latency. Moreover, if I use the method John Dawson suggested (hold load
constant and vary trial duration) I will see different measurement for
different length runs.

The problem I originally set out to describe is that the "knee" for
latency occurs at a different place that it does for forwarding rate.
For example I know of some devices that can forward without loss with a
99 percent offered load, yet latency goes off the chart whenever I drive
a load of 96 percent of more. 

Jim McQuaid's approach of "regions" should demonstrate this. 

dn

> We checked it for all RFC2544 frame sizes, in store and forward mode.  This
> is for a normal latency test,
> in it's "linear packet processing" region; that is not saturating the switch
> so that it is dropping packets.
> 
> There may be some 10Mbps devices, or some WAN devices with huge buffers
> where the 30 second test period becomes
> an issue so it's hard to generalize, so some calculation on a range of
> devices (buffer depth in terms of
> time) to verify that 30 seconds is an adequate test period to allow the
> buffers to stabilize.
> 
> -Steve Bell
>  SVNL
>  www.svnl.com
> 
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Jim McQuaid
> Sent: Friday, August 20, 1999 1:17 PM
> To: 'John Dawson'; [email-masked]
> Subject: RE: Latency and load
> 
> 
> So what you are saying, I think, is that most devices will exhibit THREE
> regions in this -hypothetical- latency characteristic curve we are
> proposing.  And that NO region can be characterized without a time of
> "adequate" duration.
> 
> Lowest latency - region one
> 	The device can handle every input without buffering and output is
> therefore as "instantaneous" as it can ever be.
> 
> Rising latency - region two
> 	The device can't quite handle every input instantly and begins to
> pile up inputs, therefore latency rises. . . .
> 
> Maximum latency - region three +
> 	The device can't quite handle inputs instantly and buffers are now
> full.  The additional processing time provided by buffering = high latency.
> This should corespond to behavior at max_throughput rate.
> 
> Hit the wall - region four?
> 	The device can't quite keep up; many packets make it through with
> high latency but some number have infinite latency, i.e. get dropped.
> 
> Jim McQuaid
> Ganymede Software
> 
> 
> 
> From: John Dawson [mailto:[email-masked]]
> Sent: Friday, August 20, 1999 2:43 PM
> Subject: Re: Latency and load
> 
> 
> David,
> 
> My two cents.
> 
> Throughput tests are affected by the buffering depth of the DUT.  The
> Throughput results for a 30 second test can be different then the results
> for a 60 second test versus a 5 minute test.
> 
> The goal of the Throughput test is to find the steady-state operating
> point.  If the throughput tests is too short, then the steady-state
> operating point does not get found and buffering depth can make up the
> difference.  If the tests are too short, then a DUT may not actually keep
> up with the input but buffering helps prevent dropping any packets.  If the
> test is allowed to run long enough then no amount of buffering will
> cover-up for a DUT that can not keep up with the input.
> 
> So how does this apply to Latency?
> 
> one at five seconds, one at ten seconds, and one at twenty seconds.  I
> would suspect that if your DUT is not in steady state but relying on
> buffers, then plotted the latency on a time line you would see a straight
> line curve start developing (latency gets worse with test duration).
> 
> When the load is dropped below a certain threshold, then the DUT can keep
> up with the input without using it's buffers as a crutch.  In this
> situation, the traffic flows freely and hence the latency achieves a steady
> state characteristic.  If you repeat the measurement process at several
> different time intervals (5 sec, 10 sec, and 20 sec) you would see less
> variation and it would be independent of the test duration.
> 
> This actually suggests that multiple latency measurements at staggered time
> intervals can tell you whether the DUT is relying on buffering depth in
> order to pass the throughput test or whether the DUT has actually acheived
> steady state performance.
> 
> John
> 
> At 04:46 PM 8/19/99 -0400, you wrote:
> >
> >BMWGer's,
> >

---

From: Kevin Dubray <[email-masked]>
Date: Tue, 24 Aug 1999 08:00:25 -0400

-------- Original Message --------
Subject: BOUNCE [email-masked]:    Non-member submission
from ["Robert Mandeville" <[email-masked]>]
Date: Tue, 24 Aug 1999 02:39:27 -0400 (EDT)
From: [email-masked]
To: [email-masked]

Dunn is wrong and Newman is right on this one. Check out any results I
have
published over the years. The only issue here is whether the DUT is
blocking
or not. If it is there will be a point where numbers go south.
Bob

[email-masked]
usa +1-949-400-44-44
europe +33 (0)6 07 47 67 10

----- Original Message -----
From: Jeffrey Dunn <[email-masked]>
To: David Newman <[email-masked]>
Cc: bmwg <[email-masked]>
Sent: Tuesday, August 24, 1999 4:06 AM
Subject: Re: Latency and load

> David,
>
> I absolutely disagree.  In the case of devices such as ATM and FR
> switches or routers, the number of cells/frames/packets per second per
> port/interface and, more importantly, the number of forwarding
> discussions per port/interface per second will significantly effect both
> the latency and loss statistics.  In any device which bases forwarding
> decisions on information in multi-level caches, the number of ports and
> destinations per port involved in a test will drastically impact
> performance both in latency and loss.
>
> Jeff Dunn
> Advanced Network Consultants
>
> >David Newman wrote:
> >
> > What Jim said.
> >
> > The particular instances I mentioned always involved more than two
> > ports, but I don't think that should be a requirement in determining a
> > "correct" method of measurement. The methodology should work independent
> > of the number of ports. Of course, the number of ports, offered load,
> > and traffic orientation should be declared by the tester.
> >
> > dn
> >
> > > I think the current discussion is about what might be the ingredients
of
> > > "correct way" since there is no specific definition in 1944/2544 about
> > > multiport devices.  Multiport devices probably have different latency
> > > characteristics for different paths but that is a real rathole!!
> > >
> > > Jim McQuaid
> > > Ganymede Software
> > >
> > > From: [email-masked]
> > > [mailto:[email-masked]]
> > > Sent: Monday, August 23, 1999 1:31 PM
> > > I have another question regarding the Latency and Load issue.
> > >
> > > How many port pairs are you utilizing when running the latency test
??

---

From: David Newman <[email-masked]>
Date: Tue, 24 Aug 1999 10:53:48 -0400

Jeff,

I urge you, in the nicest possible way, to reread my message. Nowhere do
I suggest that numbers of ports or packets are not important
determinants of latency and forwarding rate. Rather, I am arguing the
against *static* requirements of testing with 2 ports or 12 ports or N
ports. Further, I emphasize that all conditions should be declared by
the tester. 

Not sure how you arrived at the conclusions you did, but you're arguing
against something I didn't say.

dn

---

From: Jerry Perser <[email-masked]>
Date: Tue, 24 Aug 1999 09:15:33 -0700

I think Bob has come up with a good solution.

The problem I have been running into is: what offered load should latency be
measured at?  I have been asked this several times by different vendors.  My
answer was at MFR, your latency will be high and at half MFR, your latency
will be low.  Every vendor wants to show the best performance.  So most of
the time they test at half MFR.

Bob's solution has this in the minimum latency measurement as well as the
worse case in the maximum latency measurement.  In one test run, I can get
both set of numbers.  You can absolutely state that "real world" latency
will fall somewhere in between these two numbers.

This would also be simpler than running the latency for different durations.
We run the standard 30 second test.  The minimum latency should happen
within the first few frames.  I am assuming that the DUT/SUT has already
learned the flow.

> Date: Tue, 24 Aug 1999 02:25:38 -0400 (EDT)
> Per the ongoing discussion on latency, a possible approach is as
> follows:
> - run a single latency test at 100% load for selected packet sizes
> - use send and receive timestamps and sequence numbers on all packets
> - report the minimum, average and maximum latencies on a thirty second
> test
> run.
> 
> The advantages of this approach are:
> - a single test run to provide a complete set of latency results
> - a minimum value for latency of the first few packets going 
> through the
> DUT
> before buffers fill up. This time would correspond to
> the low load latency that Jim rightly wants to see.
> - a maximum latency value which would give an appreciation of 
> the total
> buffer depth
> available on the DUT.
> The average latency would give some indication of how quickly 
> the buffer
> fills
> up. Jitter or latency variation could also be calculated as standard
> deviation of the
> delta values of per packet in and out timestamps.
> Bob

---

From: Jim McQuaid <[email-masked]>
Date: Fri, 27 Aug 1999 08:37:12 -0400

The real question is your assumption that a DUT can never "catch up."  Since
some devices begin emitting the packet before the entire packet has been
received, they might have a hope of catching up, especially with the larger
packet size offered load.

BTW the HTML version was just fine at my end.

Jim McQuaid
Ganymede Software

-----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Thursday, August 26, 1999 8:29 PM
To: [email-masked]; 'Jim McQuaid'; dnewman; Jerry Perser
Cc: bmwg
Subject: Latency and Load

The question of what load is appropriate for latency measurements is still
an open issue.  I have tried to put together some thoughts on this matter
(see HTML attached document).  

I hope these documents transfer.  If not, then I would appreciate some
suggestions on what format to use for embedding pictures in documents (I
have been fighting microsoft word for the last hour and will not use this
program).  

Give the document a read (it is very short, about a page and a half), and
if this matter is of any importance to you then let me know what you think.

John