Subject: FW: Feedback on L3 QoS Benchmarking Workplan

From: "David Newman" <[email-masked]>
Date: Thu, 2 Mar 2000 16:51:39 -0500

Private mail between John Dawson and myself, forwarded with John's
permission.

dn

> > > I am allergic to labeling any measurement a "good" or "bad" one. The
> > > goal of this effort is to define performance benchmarks, not to place
> > > judgments on the acceptability (or otherwise) of those benchmarks.
> >
> > Maybe I created some confusion here.  When I mentioned graded channel
> > capacities, I meant tiers of service (i.e. is a channel capable of
> > sustaining certain parametric requirements).  I did not mean to attach a
> > good or bad connotation to the word graded.
> >
> > I feel that being able to parameterize channels in terms of bandwidth,
> > packet loss rate, and delay/jitter is important.  If an application
> > requires a certain bandwidth with a maximum packet loss rate and delay
> > interval, then the application will either work on a channel or
> > it will not
> > work.  It is not a matter of good or bad, it is a matter of
> > meeting certain
> > absolute channel characteristics.
> >
> > To say that a channel can sustain a certain bandwidth with 100
> > milliseconds
> > of delay at a 10% loss rate does not label the channel as good or bad.
> > However, it does help a user determine whether this level of
> service will
> > be sufficient for his or her application.
>
Good point, and my apologies if I misinterpreted your comments.
Fully agree.

The use of multiple metrics concurrently is what I had in mind
when I proposed forwarding rate plus packet loss in place of
throughput. As Conrad points out, throughput has the advantage of
giving the user one result without having to do lots of
intermediate step tests. Concurrency has the result of examining
how different metrics may affect one another (e.g., does at what
forwarding rate does loss begin, does latency increase linearly
with loss, and so on).
>
> >
> > > even deterministic schemes like ATM have some non-guaranteed
> > > service levels
> >
> > Yes, I agree.  However, should I be able to expect one, none, or some
> > guaranteed service levels for my critical applications.  If QOS can not
> > meet any guarantees for mission critical traffic, then isn't QOS just
> > another way of saying best effort?  To what degree can a user
> expect to be
> > able to shape traffic flows (i.e. brick-wall filters or gradual
> > attenuation)?  One analogy from the world of analog filters
> > relates quality
> > of a channel to the slope of the roll-off above certain
> frequency levels.
> >
> > > we all do step testing today to determine device performance
> > > at various load levels
> >
> > I understand this, but I also believe that some QOS service
> levels should
> > be sustainable independent of system load (i.e. Topology Hello
> > Packets must
> > remain stable even when the system is massively over-subscribed).
>
Yes, agreed. To do this it seems we'll need not only a static
metric (like latency, loss, or whatever) but also the need to
measure over time -- some sort of histogram. And on top of this
we'll need to define concurrency (e.g., forwarding rate and loss
together). This is getting really interesting.
>
> >
> > > determining whether a device performance corresponds to predefined
> > > IP-based service parameters is something for engineers -- not
> > > marketeers -- to determine. That's why we're doing this
> >
> > I also agree, but if done wrong, the marketeers will start
> > over-riding good
> > engineering practice in order to cheat the benchmarks.

Not that this has ever been a problem in the past. . .

dn

---

From: "Conrad C. Nobili" <[email-masked]>
Date: Thu, 2 Mar 2000 22:20:54 -0500 (EST)

Sorry, this turned out longer than I intended.

I just want to clarify a few things related to how the methodologies
generate the metrics in an actual test.  I realize this is clear to
many of you, but I figure not everyone has the same amount of lab
experience.  While David may well know what he means, someone with
less lab experience might get a little confused.

Don't expect all the terminology and details to be exactly correct
here -- you can look those up.  This is intended to give a general
feel for the big picture of how this stuff goes together when doing
actual benchmarking tests.

> forwarding rate does loss begin, does latency increase linearly

Well the tests are made up of lots of intermediate step trials,
even for a throughput test.  The difference is in how it's all
put together as a particular kind of test.

A forwarding rate test and a packet loss rate test are pretty
much the same thing, so it's really one test, rather than two
tests concurrently.  What you're doing is offering a known load
to a device and measuring what comes out.  This is the Forwarding
Rate at the given Offered Load.  But of course the Packet Loss
at that OL is just OL - FR.  So you're just doing one test, from
which you get two results.  Obviously you need to do lots of these
trials at different offered loads.

This *trial* is the fundamental building block for forwarding rate,
packet loss rate, and throughput tests.  The only difference is in
the sequence of offered loads that defines the sequence of trials.
There is no difference of course in forwarding rate and packet loss
rate tests, as these really are equivalent.  What you do is divide
the range of offered load up into equally spaced amounts, either by
percentage or packets per second or whatever.  This determines the
test granularity.  Then you measure the FR and PL for those different
OLs.  Now you've done both a FR and PLR test.  You may find that the
maximum forwarding rate is not at the highest offered load.  You've
also got a pretty good idea as to the shape of the performance curve
of the device if your granularity was good enough.

A throughput test is built with these same trials, but here one uses
a binary search to find the Zero Loss Rate.  Again, there is a choice
of granularity here.  It's going to take a few more trials to find
the ZLR (throughput) to 1 Packet Per Second on a Gigabit device than
on a 10Mbps device.  It may be easier to wait out the extra trials
to get 1 PPS granularity than it is to explain a different granularity
choice to vendor marketing droids.  ;-)  A throughput test produces
a single value for the ZLR.

There's also a single value for the Maximum Forwarding Rate.  This
may be higher than the ZLR, in which case it comes from the FR/PLR
test.

While you may have a good idea about the throughput of a device by
looking at the data produced by a FR/PLR test, you really shouldn't
say anything about ZLR unless you've run the proper methodology to
measure it.  The ZLR may well be somewhere between offered loads of
the FR/PLR test -- in a blind spot.  You can't really be sure that
a device will be "perfect" on a ZLR test even if it doesn't show any
packet loss in the FR/PLR test.  There tend to be more trials in a
ZLR test, so it is better at chasing out robustness problems.  Also
I have seen devices that exhibit unexpected behavior at specific
offered loads.  A ZLR test is more likely to tickle these sorts of
problems than a FR/PLR test.

I hope that clarifies David's "concurrency" and "intermediate step"
test comments above.

> together). This is getting really interesting.

Generally people pick a trial duration that represents an effective
infinity (like more than the device could buffer).  You want to try
and find the steady state performance of the device.  If you want to
analyze performance with bursts of overload then it gets different
and more complicated and there are other sorts of test techniques
which come into play.

As I explained above forwarding rate and packet loss rate tests are
pretty much identical and therefore "concurrent".  When reporting
either you must report the OL, and thus both the FR and PL can be
deduced, so might as well just report all three.

> Not that this has ever been a problem in the past. . .

Keep it simple and strict and you can usually keep the marketeers
from mucking things up too badly.  I think we did a pretty good job
of keeping them at bay in Scott's lab.

--cn

Conrad C. Nobili  [email-masked]  Harvard University NDTL/NOC

---

From: "David Newman" <[email-masked]>
Date: Fri, 3 Mar 2000 11:25:24 -0500

<snip>
> of granularity here.

<snip>

Agreed on all of this. This is a very good explanation of the differences
between tput and forwaring rate/loss tests.

I was actually making a slightly different point, pertaining solely to
presentation of test results: Throughput has the advantage of presenting
*one* measurement, while a forwarding rate number, by itself, isn't terribly
meaningful without also presenting a packet loss measurement. Even then, it
may be more useful to present multiple steps of forwarding rate/packet loss
to get a sense of device performance.

There are many instances where presentation of fewer numbers is preferable.
Conversely, sometimes more really is more. Scott Poretsky has suggested we
include throughput *and* forwarding rate/packet loss in our draft, and I
concur.

dn

---

From: "Conrad C. Nobili" <[email-masked]>
Date: Fri, 3 Mar 2000 15:17:58 -0500 (EST)

On Fri, 3 Mar 2000, David Newman wrote:

> *one* measurement,

It's one *number*, which is the result of *many* *measurements*.  ;-)

This number has valid semantic content, so it's useful on its own.

> while a forwarding rate number, by itself, isn't terribly
> meaningful without also presenting a packet loss measurement.

It's entirely meaningless to present a Forwarding Rate number without
also presenting either the Offered Load or the Packet Loss (either can
be deduced from the other and the FR) with which it was measured.  I
would not present it without its associated OL.  I thought we had a
recommendation or requirement to that effect somewhere.

> Even then, it

The more slices through the performance curve you can show, the more
illuminating it will be.  But the granularity is entirely a matter of
choice (much more so than with a throughput test).

> There are many instances where presentation of fewer numbers is preferable.

Absolutely!  But they've still got to make sense.  So it's perfectly
reasonable to show a single FR result, but I would argue that that
result MUST include information about the OL at which it was measured.
This "test" could be a single trial.  It has meaning and it's potentially
useful.  The results of more trials would make a better test and be more
useful, but that depends on the situation.  You may just need to know the
behavior at a certain OL.

--cn