Subject: RE: Last Call : L3 QoS Benchmarking Workplan ???

From: Kevin Dubray <[email-masked]>
Date: Mon, 28 Feb 2000 13:32:05 -0500

From: "Conrad C. Nobili" <[email-masked]>
To: David Newman <[email-masked]>
cc: John Dawson <[email-masked]>, [email-masked],
        Scott Bradner <[email-masked]>
Subject: RE: Last Call : L3 QoS Benchmarking Workplan ???

On Sat, 26 Feb 2000, David Newman wrote:

> I may be to blame for some confusion on this point. As noted in section 3 of
> the proposal, the four main metrics we plan to define are forwarding rate,
> packet loss, latency, and jitter. Scott originally suggested throughput (per
> RFC 1242) rather than forwarding rate (per RFC 2285) but I objected on
> grounds opposite to yours -- that a throughput metric wouldn't tell us
> anything about forwarding rates once packet loss begins.

Scott's right that throughput is very important.  But of course David is
also correct in that forwarding rate can be much more illuminating.  Heck,
if you do enough forwarding rate tests with fine enough granularity, you
could even infer the results of a throughput test without actually doing
it.  Methodologically this is incorrect, but conceptually it's possible.

> The gist of my argument for using forwarding rate was that 1) we were
> already measuring packet loss and 2) we'd *gain* thhe ability to measure
> under lossy scenarios and we wouldn't *lose* any information we might have
> gleaned had we used a throughput metric instead.

I don't think forwarding rate tests can substitute for a throughput test
unless done with finer granularity than anyone ever does them.  Valuable
information would be lost without a throughput test.

> That said, I would be glad to consider arguments for using a throughput
> metric instead of or in addition to forwarding rate.

I think both are necessary.  They measure two different concepts.  And I
think that if we only define one there will be a lot of people who will
try to use it to make conclusions about the other -- and most of these
people won't be clever, careful, or thorough enough to do it correctly.
So I think it's much safer to define both and not leave room for such
improvisation.

Forwarding rate and throughput cover load-related phenomena reasonably
well.  But I've seen lots of non-load-related phenomena in devices too.
These are fundamentally different from the load-related phenomena, and
are sometimes as or even more important than them.  I wonder if it may
be useful to discuss non-load-related phenomena a bit?  Some are purely
statistical or environmental, some are even clock-related.  There isn't
any -- I hope I'm right ;-) -- mention of non-load-related phenomena in
1242, 1944, or 2285, yet it is impossible to do testing for load-related
phenomena using these methodologies without being supremely aware of any
non-load-related phenomena which may be occurring.  I wonder if it would
be useful to have some clarifying discussion in a document somewhere.

--cn

Conrad C. Nobili  [email-masked]  Harvard University NDTL/NOC

---

From: Scott Poretsky <[email-masked]>
Date: Mon, 28 Feb 2000 16:37:55 -0500

We will update the Proposal to include measurement of Throughput in 
addition to Forwarding Rate.

Scott