Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 24 Oct 2013 13:27:11 -0400

I can make a little time to pick-up this discussion again,
so maybe we can close on some advice for the draft authors.

If as David agreed below, the e2e jitter estimation is a valuable feature,
then it helps to narrow the field of applicable definitions for 
delay variation.  We know how to do this (albeit with some mathematical
complexity) for the form of delay variation distribution that is essentially
the histogram of delays with the minimum delay subtracted from all values.
http://tools.ietf.org/html/rfc6049#section-6

We don't know how to do comparable e2e estimation math for an alternate form 
of delay variation, usually called jitter, which emphasizes the change in delay
between successive packets.

Both these forms are defined and compared in 
http://tools.ietf.org/html/rfc5481#section-4
as PDV and IPDV, respectively.

Over the last two months, I've had some discussions with folks who want
to support Data Center trouble-shooting with IPv6 optional header time-stamp
assistance. These folks have worked to broadly inform various IETF working 
groups of their goals and proposed solutions (v6ops, ippm, and even tictoc
for some synchronization related aspects). One thing I've learned from the
exchange is that the histogram view of delay variation can assist their 
efforts. This may be more of an operational aspect than something we
need to measure in device benchmarking, but the synergy with e2e estimation
is worth noting.

regards,
Al
(as a participant)

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> David Newman
> Sent: Wednesday, July 31, 2013 2:38 PM
> Cc: [email-masked]
> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
> 
> On 7/31/13 11:07 AM, MORTON JR., ALFRED C (AL) wrote:
> 
> > I assumed we have benchmarked a single device, and wanted to
> > estimate how some number of these devices would affect delay
> > variation when deployed on the path. Clearly, if we can benchmark
> > a particular black-box SUT with a specific combination of these devices,
> > there's no need to do the estimation for that case.
> > But having the ability to estimate e2e variation for an unknown number
> on the path
> > seems valuable to me, and I wonder if Data Center Engineers agree.
> >
> > it's been a long day here, hope this makes sense,
> 
> Mostly. Agreed 100 percent on the requirement for e2e estimation.
> 
> Unclear how that estimate would differ from a single-box estimate times
> some number of identical boxes, modulo cable lengths.
> 
> And regardless of box count, there's still the question of which  method
> of jitter measurement to use.
> 
> That was my original point: Given so many definitions to choose from,
> it'd be nice to spell out one definition in this document, either
> referencing something else, or defining something new (as in the use of
> FILO for latency measurement).
> 
> dn
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Thu, 24 Oct 2013 10:56:16 -0700

> need to measure in device benchmarking, but the synergy with e2e estimation

I've found histograms to be very useful in lab as well as production
settings. There's still the question of "histograms of what?" though.

My main concern is having enough specifity in the term "jitter" (and
indeed all terms in all bmwg docs) so that everyone implementing the
test will make the same measurement.

The current reference, to the second definition in RFC 3393 section 1.1,
avoids giving a specific measurement (is it average or minimum?) and
even goes on to say the term jitter should be avoided.

Though I have a bias toward the definition produced by this WG, in RFC
4689, I'm less concerned with what definition we choose (3393, 4689,
6049, other) than with having that definition clearly spelled out.

Underspecification can lead to different RFC-compliant test sets
producing different results, and no one wants that.

dn

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 24 Oct 2013 14:28:40 -0400

Thanks for adding our "local" references and more, David.

With many possibilities on the table, I hope we'll hear from some 
additional folks who've joined us for data center-related work...

Al
(I can say that as co-chair, so have)

> Sent: Thursday, October 24, 2013 1:56 PM
> Cc: [email-masked]

---

From: "Rapp, Jacob (HP Networking)" <[email-masked]>
Date: Thu, 24 Oct 2013 18:50:19 +0000

David,
I agree, the main concern is to have the single definition, but let me address why we are proposing a histogram and what that histogram means too.

The biggest reason we wanted to clarify jitter is it is a term that is widely used when talking about benchmarking switches and it is used in a wide variety of ways. We did see RFC3393 saying avoid the term, but it hasn't been followed and the common uses have expanded past what is called out in RFC3393. Rather than continuing to avoid using the term, which hasn't worked, we need to take the time to define it. 

The confusion I usually see when someone uses the term jitter, they mean one of two things. 
1. Average delay variation - the average of the delta of packet arrival  times of the same size packets(in some cases I've heard people talk about jitter between different size packets, but that would have inherent different latency). Say the test consists of 100 packets sent all at 64 bytes. Each packet latency is measured to be a fixed value. The jitter is usually expressed as say, average jitter of 5ns, meaning the delay variation average of each packet is 5ns on average.
2. Some people mean jitter to be in the form of max, min and average latency of the same packet arrival. With our same example of 100 packets sent all at 64 bytes, the min latency is 10ns average is 12ns max was 15ns.

Looking back at the draft again, I see how it needs some refinement.
In the end, to clear up the confusion, we settled on proposing:
1. Jitter is valid only when sending packets of the same size.
2. There are two important consideration.
	2a. Packet to Packet delay variation
	2b. The min and max
3. Ultimately displaying a histogram of packet-to-packet latency change throughout the entire test gives the best view. Maybe back in 2000 this wasn't entirely possible, but now it is. Displaying jitter in the histogram form alleviates the need to abstract out information by using averages. 

--
Jacob Rapp

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Newman
Sent: Thursday, October 24, 2013 10:56 AM
Cc: [email-masked]
Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)

On 10/24/13, 10:27 AM, MORTON, ALFRED C (AL) wrote:
> I can make a little time to pick-up this discussion again, so maybe we 
> can close on some advice for the draft authors.
> 
> If as David agreed below, the e2e jitter estimation is a valuable 
> feature, then it helps to narrow the field of applicable definitions 
> for delay variation.  We know how to do this (albeit with some 
> mathematical
> complexity) for the form of delay variation distribution that is 
> essentially the histogram of delays with the minimum delay subtracted from all values.
> We don't know how to do comparable e2e estimation math for an 
> alternate form of delay variation, usually called jitter, which 
> emphasizes the change in delay between successive packets.
> 
> Over the last two months, I've had some discussions with folks who 
> want to support Data Center trouble-shooting with IPv6 optional header 
> time-stamp assistance. These folks have worked to broadly inform 
> various IETF working groups of their goals and proposed solutions 
> (v6ops, ippm, and even tictoc for some synchronization related 
> aspects). One thing I've learned from the exchange is that the 
> histogram view of delay variation can assist their efforts. This may 
> be more of an operational aspect than something we need to measure in 
> device benchmarking, but the synergy with e2e estimation is worth noting.

I've found histograms to be very useful in lab as well as production settings. There's still the question of "histograms of what?" though.

My main concern is having enough specifity in the term "jitter" (and indeed all terms in all bmwg docs) so that everyone implementing the test will make the same measurement.

The current reference, to the second definition in RFC 3393 section 1.1, avoids giving a specific measurement (is it average or minimum?) and even goes on to say the term jitter should be avoided.

Though I have a bias toward the definition produced by this WG, in RFC 4689, I'm less concerned with what definition we choose (3393, 4689, 6049, other) than with having that definition clearly spelled out.

Underspecification can lead to different RFC-compliant test sets producing different results, and no one wants that.

dn

>> From: [email-masked] [mailto:[email-masked]] On Behalf 
>> Of David Newman
>>> variation when deployed on the path. Clearly, if we can benchmark a 
>>> particular black-box SUT with a specific combination of these 
>>> devices, there's no need to do the estimation for that case.
>> Unclear how that estimate would differ from a single-box estimate 
>> times some number of identical boxes, modulo cable lengths.
>>
>> And regardless of box count, there's still the question of which  
>> method of jitter measurement to use.
>>
>> referencing something else, or defining something new (as in the use 
>> of FILO for latency measurement).
>>
>> dn
>>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 24 Oct 2013 15:35:29 -0400

Hi Jacob, thanks for joining in.

> 	2a. Packet to Packet delay variation

Just to be clear, you take the one-way delay of packet "x",
and subtract the one-way delay of packet "x-1", and that 
yields a single value of delay variation (one of many definitions).

The sequence of packets and their delays is important to this metric,
because we never compare the one-way delay of packet "x" to 
any other packet except "x-1".

So, if I have your definition right, what performance related question
does the 2a metric help to answer?  What physical quantity does it define?

I'd prefer that we had some compelling reasons to choose one definition
over the others.  If e2e estimation of n-DUT delay variation is one of our
goals, this definition won't help us, AFAIK.

> 	2b. The min and max

I think we are on the same page with 2b, min and max,
but not quite the same paragraph yet. 
These are the extremes of the one-way delay
histogram, which exhibits some spread between min and max one-way delay...

In your example, the min and max were the one-way delay values (12 and 15ns),
so the delay *variation* was 3ns maximum (min is zero, that's the basis for
variation in the definition I'm using here, every one-way delay value has the 
minimum delay subtracted to yield the variation).

This definition of delay variation (subtract the minimum) can support the
e2e estimation of n-DUT delay variation.

So, would you agree that's a useful thing to be able to do,
and something that data center architects might want to know?

Al
(as participant)

> Rapp, Jacob (HP Networking)
> Sent: Thursday, October 24, 2013 2:50 PM
> To: David Newman
> Cc: [email-masked]
> I agree, the main concern is to have the single definition, but let me
> address why we are proposing a histogram and what that histogram means
> too.
> 
> The biggest reason we wanted to clarify jitter is it is a term that is
> widely used when talking about benchmarking switches and it is used in a
> wide variety of ways. We did see RFC3393 saying avoid the term, but it
> hasn't been followed and the common uses have expanded past what is called
> out in RFC3393. Rather than continuing to avoid using the term, which
> hasn't worked, we need to take the time to define it.
> 
> 1. Average delay variation - the average of the delta of packet arrival
> times of the same size packets(in some cases I've heard people talk about
> jitter between different size packets, but that would have inherent
> different latency). Say the test consists of 100 packets sent all at 64
> bytes. Each packet latency is measured to be a fixed value. The jitter is
> usually expressed as say, average jitter of 5ns, meaning the delay
> variation average of each packet is 5ns on average.
> 2. Some people mean jitter to be in the form of max, min and average
> latency of the same packet arrival. With our same example of 100 packets
> sent all at 64 bytes, the min latency is 10ns average is 12ns max was
> 15ns.
> 
> 3. Ultimately displaying a histogram of packet-to-packet latency change
> throughout the entire test gives the best view. Maybe back in 2000 this
> wasn't entirely possible, but now it is. Displaying jitter in the
> histogram form alleviates the need to abstract out information by using
> averages.
> 
> --
> Jacob Rapp
> 
> 
> indeed all terms in all bmwg docs) so that everyone implementing the test
> will make the same measurement.
> 
> avoids giving a specific measurement (is it average or minimum?) and even
> goes on to say the term jitter should be avoided.
> 
> 4689, I'm less concerned with what definition we choose (3393, 4689, 6049,
> other) than with having that definition clearly spelled out.
> 
> Underspecification can lead to different RFC-compliant test sets producing
> different results, and no one wants that.
> 
> dn
> 
> >
> > regards,
> > Al
> > (as a participant)
> >
> >

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Fri, 25 Oct 2013 09:43:40 -0700

Hi Al,

I'd like to explain further, please see inline:

> any other packet except "x-1".

absolutely, it's we should compare two packets of the 'same' size and x 
compared to x-1, we will clarify this. the metric here measured is the 
'absolute delta=[x-(x-1)]'=jitter.

> does the 2a metric help to answer?  What physical quantity does it define?

It tries to define in my opinion a more precise jitter variation were we 
state that jitter is only significant between x and x-1 packets, when x 
and x-1 have the *same* packet characteristics.

> e2e estimation of n-DUT delay variation.

just to be sure I am clear, what we try to say is that the DUT can yield 
actually difference results for the variation 'absolute delta=[x-(x-1)]' 
and there will be a lowest [min] value at the end of the test and a max.

The idea between the histogram is to represent the statistical 
population of the results so we can see what the spread is around min 
and max.