Subject: bmwg Latency Under Load

From: Kevin Dubray <[email-masked]>
Date: Sat, 13 Feb 1999 17:17:06 -0500

I was reviewing the presentation that you gave on Latency
Under Load in Orlando.

(http://www.ietf.org/proceedings/98dec/slides/bmwg-latency-98dec/index.html)

You make some good points for enhanced latency measurements. I 
have a few follow-on questions/comments.

The notion of Latency under Load is not new in the BMWG.  RFC 1944
suggests that latency be taken in the presence of other forwarded
traffic, presumably from the same sets of ports on which
the latency measurement occurs.  Additionally, RFC 2432 introduces
the notion of a "Burdened Response," specifically, a Forwarding
Burdened Multicast Latency metric - presumably a multi-ported
measurement not unlike what you are advocating. (The related FBML
methodology is currently being defined.) All this said, it is generally 
acknowledged that "loaded" metrics have a less "clinical" smell 
about them ;-) and are, therefore, more insightful - more "real 
world." 

Is it your opinion that your proposed heuristics replace or enhance
the methodologies of these historical terms? Or do you advocate that
your proposals obviate these terms entirely?  

As you mention, the control of offered load can be quite difficult
(though you made this comment in the context of bi-directional,
half-duplex measurements).  Is the synchronization of multi-stream
events too difficult to yield consistently repeatable results?

Do you see Latency Under Load (LUL) metrics better suited for
single-device characterization or are they suitable to form the
basis of comparison between different devices?

Lastly, if load is a primary factor in a latency metric, any
ideas on how much load is desireable?  What's the preferred 
load distribution across available ports?  Undoubtably, folks
will ultimately want to use LUL as a basis of comparison.  How 
do we "level the playing field?"

-Kevin

---

From: [email-masked] (John Dawson)
Date: Tue, 16 Feb 1999 13:51:30 -0800

>your proposals obviate these terms entirely?  

My hope is to enhance the existing methodologies -- not replace.  However I
believe my results thus far have shown that some existing methodologies do
not apply in a generic fashion to latency measurements.

It is my opinion that the methodology perscribed in RFC 1944 will yield
inconsistent latency results in a number of critical cases.  For instance,
one point that must be considered is the definition of how multi-port
streams should be defined.  Section 16 of RFC 1944 specifies that streams
should be structured in a many-to-many fashion (all inputs to all outputs).
 My presentation discusses the inherient difficulty with this type of
stream structure.  Some implementations have bypassed this issue by using
multiple discrete one-to-one port pairings rather than a many-to-many
oriented approach.  Another significant issue relates to the use of
bi-directional traffic patterns.  Bi-directional measurements are not a
good approach when it comes to half-duplex latency measurements.

With respect to RFC 2432, latency measurements in section 3.3 are based
upon a single port transmitting to all other ports in the same multicast
group.  There is a very valid and relevant measurement.  However, extending
the measurement such that all ports transmit to all other ports at the same
time is going to be a problem.  In the extended case transmit
synchronization is not controlled and the results become random.

>events too difficult to yield consistently repeatable results?

Offered load is independent of whether or not the load is synchronized.  At
any given load, the load can be either synchronized or un-synchronized.  My
comments with respect to bi-directional half-duplex streams are that
offered load can not be programmed -- no matter what is programmed, the
resulting load must still be measured in some fashion.  So now which should
be reported, as programmed or as measured?

I believe that the synchronization of multi-streamed events is possible,
but the test equipment currently on the market is very limited in how it
maintains synchronization over time.  I have not seen any commonly
available equipment that can remaim synchronized beyond the first couple of
packets transmitted.  Hence my presentation focussed on impluse-response
types of measurements that could remain synchronized.

Without synchronization, multi-stream measurements lead to random results.
Which is why a lot of latency measurements these days are showing
bell-shaped curves.  A bell shaped curve is completely expected when
sampling uncontrolled processes.

>basis of comparison between different devices?

I believe that the proposed latency under load metrics are suited for both
single-device characterization and for comparision purposes.  I also
believe that this metric will give consistently repeatable results that can
be duplicated at different test sites.  

One of my primary concerns with the current measurement process is how
results will vary anytime a test source is changed with a similar test
source that uses a slightly different clock (test source clocks are not a
constant).

However, I do not think that my LUL measurements are the only measurements
that need to be made.  There are other process that I feel are more
relevant to larger complex systems that involve multiple interconnected
devices.  I believe that there are exteremely important merits to a number
of other test processes -- no one metric is going to tell you everything.
In simplier terms, if you are testing a large system consisting of multiple
boxes, I would chose another test process for measuring latency. 

>will ultimately want to use LUL as a basis of comparison.

There are definitely undesirable loads.  A load becomes undesireable when
it changes the fundamental measurement you are trying to make.  For
instance if bi-directional half-duplex traffic is being used as a load
during a latency measurement at what point does the measurement actually
become a measurement of  collision back-off characteristics and not latency?

Anything that is not 'undesirable' could probably be claimed as desireable
and meaningful by someone -- everybody has some idea as to which ranges or
levels of loading are significant.

Unfortunately, I also think that this question leads us down a path of
testing a never-ending variety of load patterns.  The basic LUL
measurements I had proposed treated load as an impulse on every port.  I
believe that this impulse-response result is very good for directly
comparing products.

Variations of the basic impulse-response test could define loads as a
burst, or multiple bursts, etc.  However, I don't think that we want a
process that needs a large quantity of different tests and/or different
load patterns to make sense out of what is happening.  Is it possible for
us to augment LUL with a process that can discover the natural latency
response of a device under test - without having to adjust all the streams
in an iterative fashion?  (I believe this is entirely possible.)

>How do we "level the playing field?"

I believe that the best way to level the playing field is to require that
the test process include a detailed theory for explaining results.  To not
require any interpretation or forethought into the results allows a
free-for-all to ensue (everyone becomes a spin-doctor).  If no bounds are
placed on a test result, how does the tester know when the results are
within expectations or when the device under test is cheating the measurement?

A second way to level the playing field is to start with some processes
that are so simple (obvious) that the results can only be spun one way.  I
do have a proposal in this area.

John