Subject: [bmwg] comments on draft-dcbench-def-00 (part 1)

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Wed, 24 Jul 2013 13:03:01 -0400

Hi Jacob and Lucien,

Thanks for your efforts to adopt BMWG's traditional template
for your draft on terminology. This draft appears to provide a good
foundation for your methodology, and also for some fundamental 
discussions on benchmarking, so I share a few specific comments below.

Al (as a participant)

-=-=-=-=-=-=-=-=-=-

1.2. Definition format

   Term to be defined. (e.g., Latency)

   Definition: The specific definition for the term.

   Discussion: A brief discussion about the term, it's application and
   any restrictions on measurement procedures.

   Measurement: Methodology for the measure and units used to report
   measurements of this term, if applicable.

Comment: The last item "Measurement" is usually just "Measurement Units" in our
traditional format, because we usually add many details in the course of developing
BMWG methodology drafts. In the past, we have included the headings of "Issues" and 
"See Also" for references to other terms. 

>>>>>Perhaps we should have a short discussion
     of the usefulness of our current Definition format and see if there is need 
     for change. We can also consider the RFC 6390 template which presents guidelines 
     for new metrics.

-=-=-=-=-=-=-=-=-=-

2.  Latency

2.1. Definition

   Latency is a the amount of time it takes a frame to transit the DUT.

   Latency can be measured with the following methods, irrespectively of
   the type of switching device (bit forwarding aka cut-through or store
   forward type of device)

   FILO (First In Last Out) The time interval starting when the end of
   the first bit of the input frame reaches the input port and ending
   when the last bit of the output frame is seen on the output port

Several comments and suggestions:

* The paragraphs for FILO, FIFO, LILO, LIFO are actually different definitions
of Latency (not methods per se), but this is easily fixed:

OLD
   Latency can be measured with the following methods,
NEW
   The Latency interval can be assessed between different combinations of events,

* Another way to summarize the 4 different definitions is to simply refer to
the bit position as they normally occur, input to output.  This would change the
names as follows:

FILO -> First bit, Last bit
FIFO -> First bit, First bit
LILO -> Last bit, Last bit
LIFO -> Last bit, First bit

and these can be summarized FbLb or just FL.

* FILO is the definition recommended, but this has been avoided in the past
(in BMWG) to exclude the frame serialization time from Latency. It makes for easier
comparison when you are testing with multiple frame sizes, especially between
large and small packets (and that was preferred in RFC 1242, even with the possibility
of negative delays).

* If BMWG agrees to make this change, it would mean that this draft updates RFC 1242,
which we should do very carefully, IMO.

>>>>>> Let's plan to discuss Latency further during the BMWG session.

-=-=-=-=-=-=-=-=-=-

3 Jitter

3.1 Definition

   The definition of Jitter is covered extensively in RFC 3393. This
   definition is not meant to replace that definition, but it is meant
   to provide guidance of use for data center network devices.

   The use of Jitter is in according with the variation delay definition
   from RFC 3393:

   The second meaning has to do with the variation of a metric (e.g.,
   delay) with respect to some reference metric (e.g., average delay or
   minimum delay).  This meaning is frequently used by computer
   scientists and frequently (but not always) refers to variation in
   delay.

Comment:  Even with the reference to RFC 3393, there are many definitions
of "jitter" possible. Fortunately this is not the first draft to examine 
the possibilities :-)

I find that a comment I made on the Traffic Management draft is applicable here:
>...
>   - Packet Delay Variation (PDV): the Packet Delay Variation metric is
>   the variation between the timestamp of the received egress port
>   packets and specified in RFC 1889.
>
> Comment: since you have One-way Delay (PD defined above PDV), why resort to the RFC 1889 definition?  
> What does this give you?
> Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many 
> reasons given in http://tools.ietf.org/html/rfc5481#page-27
> (or call this IPDV to avoid confusion).

In the Isolated test lab environment, we don't have time clock drift issues that
we can't solve, in many cases the same clock is used for ingress and egress timestamps.
So, we are free to choose the delay variation metric(s) that serve our audience best.

>>>> I think "Jitter" is another topic that we can advance through face2face discussion.
     What do we really want to learn from the "jitter" metric?
     How will these results be used? (in network design, application planning, etc.)

(I'll have to do this review in multiple parts, it seems)

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Mon, 29 Jul 2013 09:14:48 -0700

Hi Al,

Thank you very much for taking the time to write this detailed feedback.
Comments inline

> discussions on benchmarking, so I share a few specific comments below.

We can rename the structure we use to Measurement Units, but we dont 
have always just a unit per say, we go beyond it. For example, for 
latency, the unit is time for the values, but in the measurement section 
we define what should be measured for this benchmarking: filo as must, 
fifo as may. If you feel appropriate that we call it measurement unit, 
then we can change each one to match.

Our intent was to have this paragraph to be the recommendation on how to 
measure a specific topic.

>       for change. We can also consider the RFC 6390 template which presents guidelines

That is good idea, it will add more clarity, we will add the above in 
the section

> large and small packets (and that was preferred in RFC 1242, even with the possibility

FILO would be the way to fairly compare a store forward switch to a 
cut-through switch, hence this recommendation, to get the users closer 
to an apple-apple comparison.

Also this will show the end consumers of this benchmarking, the real 
latency as seen per the applications. Usually applications can process 
data, once all the packet is been received. Only certain type of 
application, using for example an FPGA card, can start processing data 
with the first bits, without having to wait for the whole packet.

So we still strongly believe FIFO is very important, but we want to have 
the users to be given the FILO measure as well so they can later compare 
different method of switches and also see the performance from an 
'application' perspective versus just a 'network' latency point of view.

>>>>>>> Let's plan to discuss Latency further during the BMWG session.

For the Data Center benchmarking, we found that the rfc 3393 definition 
was the easiest to consume for its applicability to the data center. we 
wanted to add to this that what is key is to have packets of the same 
size when jitter is measured. It's mentioned in the RFC 3393 but in a 
more complex method as follows:

'‘+ L, a packet length in bits. The packets of a Type P packet stream 
from which the singleton ipdv metric is taken MUST all be of the same 
length.’’

We want to make this simpler to the end data center user by calling it 
packets of the same size.

Also we add to this definition, by explaining the necessity for a 
histogram with latency buckets, this will help understand where there is 
higher jitter for instance.

> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
>

---

From: David Newman <[email-masked]>
Date: Mon, 29 Jul 2013 14:35:28 -0700

On 7/29/13 9:14 AM, Lucien Avramov (lavramov) wrote:

>>> Suggest: Specify PDV as defined in
>>> http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many

RFC 4689, approved by this wg a few years ago, provides a simpler
definition of jitter:

http://tools.ietf.org/html/rfc4689#section-3.2.5

|D(i) - D(i-1)|, where D equals the Forwarding Delay
      and i is the order the packets were received.

This paragraph compares the RFC 3393 and 4689 definitions:

"Jitter is related to the IPDV (IP Delay Variation) taking the absolute
value of the ipdv. The two metrics will produce different mean values.
Mean Jitter will produce a positive value, where the mean ipdv is
typically zero. Also, IPDV is undefined when one packet from a pair is
lost."

The RFC 4689 definition may be more applicable here.

dn

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Tue, 30 Jul 2013 07:39:16 -0400

Hi David and Lucien,

Thanks for the additional reference, David.

The definition in 4689 is like the Interpacket delay variation definition
that can be formulated using 3393, but these definitions both depend on
the sequence of delays. Also, the absolute value function used in 4689
colapses the positive and negative inter-packet delays together,
and sometimes these have different meanings or causes. 

I think the question I'm asking is 
"How with the Jitter results be used by Datacenter Engineers?"

If, for example, they wish to know the overall jitter from a path
with ten DUTs in series, we can estimate that value but we have to
measure the right metric at the start to do it.

hope this helps, or gets people thinking at least. 
regards,
Al

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of David Newman [[email-masked]]
Sent: Monday, July 29, 2013 5:35 PM
To: [email-masked]
Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)

dn

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Tue, 30 Jul 2013 04:57:26 -0700

Thanks Al and David,

I also find the 4689 to add more details on jitter, but yet we don't 
have a simple explanation for the datacenter consumers of the 
benchmarking results.

Having been doing datacenter benchmarking and meeting customers to whom 
we show results, explain results or guide for product benchmarking for 
years, the problem we have faced the most is how to make jitter a simple 
and significant metric when evaluating a data center switch.

In the datacenter, where performance is often times quite critical [high 
performance compute clusters, high frequency trading datacenter 
environments; to name a few], the jitter is part of the performance for 
a DUT.

The problem we try to solve with jitter in terms of these two drafts is:
-how to make jitter a simple term for datacenter engineers to understand 
and to measure.

We don't have the ambition to redefine jitter or re-question previous 
drafts, rather place them in datacenter context with these two key parts 
of this draft-dcbench-def-00:

3.2 Discussion

    Jitter can be measured in different scenarios:
   -packet to packet delay variation
   -delta between min and max packet delay variation for all
    packets sent.

3.3 Measurement

    The jitter MUST be measured when sending packets of the same size.
    Jitter MUST be measured as packet to packet delay variation and delta
    between min and max packet delay variation of all packets sent. A
    histogram MAY be provided as a population of packets measured per
    latency or latency buckets.

We have observed certain benchmarking reports to abuse and not be RFC 
compliant when measuring jitter, not allowing anymore an apple-to-apple 
comparison between different DUTs.

The hope here is to have an informational document providing guidance 
for the datacenter benchmarking [including the jitter].

Al you raise a key point. When the jitter results are provided to a 
datacenter engineer, it will give him an idea of the variance of 
performance in a steady environment to understand what is the actual 
real latency addition that he may observe.

For example a 200 nanosecond latency for a DUT under FIFO, with a 10 
nanosecond of jitter at a given packet size will provide a 200+-10 
variance of latency the end user can expect. This gives an order of 
magnitude for the end performance one can actually have in production 
environment. If the jitter on a DUT is high [for the previous example 
say 100 ns] then a user will understand that the latency can vary up to 
50% instead of 5% [with the 10ns example].

Hope that clarifies it,

Cheers,
Lucien

---

From: David Newman <[email-masked]>
Date: Tue, 30 Jul 2013 15:45:31 -0700

On 7/30/13 4:39 AM, MORTON JR., ALFRED C (AL) wrote:

> the absolute value function used in 4689

Both the RFC 3393 and 4689 definitions use absolute values.

3393 gives two methods of measuring jitter. One references RFC 2598, the
other references RFC 1889, and both of these absolute values. RFC 3393
explicitly mentions the use of absolute values for both methods.

Thus, this is not really a meaningful differentiator between the 3393
and 4689 jitter definitions.

Indeed, it would be very problematic *not* to use absolute numbers.
Without them, the mean of a uniform distribution of positive and
negative values around some midpoint would give us -- zero.

Given Lucien's stated goal of "how to make jitter a simple term for
datacenter engineers to understand and to measure" (presumably by
representing it as a single number, the mean), and given his examples of
5% and 50% variance, it's unclear how a metric that is typically zero
would describe that variance in any simple way.

> measure the right metric at the start to do it

Also unclear, given the bmwg's black-box approach to measurement, why
the number of elements within a single SUT would matter.

Agreed that the key question here is what constitutes the "right metric"
regardless of whether we measure jitter for one box or lots of boxes as
a system.

dn

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Wed, 31 Jul 2013 01:26:37 -0700

Hi David,

Please see inline

> would describe that variance in any simple way.

I wanted to just give a representation of the jitter variance, the 
example values [5%,50%] were just meant to show how it is significant 
for a user and illustrate my point, but it's *not* the jitter 
definition. The goal is to first measure it correctly and what it means 
and then have a comprehensive optional chart histogram representing the 
jitter variance, this is usually done by packet size as on figure 10 for 
example here:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/ps12581/white_paper_c11-716751.pdf

The *key* to me is to first *explain* simply what jitter means for the 
datacenter engineer which I re-quote here from draft-dcbench-def-00

3.2 Discussion

3.3 Measurement

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Wed, 31 Jul 2013 14:07:53 -0400

Hi David,

Although RFC 3393 includes the absolute value, it's
only by way of reference to 1889 and 2598, and the
IPPM consensus was to deprecate the term jitter.
Thus the many other 3393 metrics for delay variation 
use differences without the absolute value, and 
they also supply some summary statistics in section 4.
I don't think they include the mean.

Also, in my example 
> <Data Center Engineers> wish to know the overall jitter from a path

I assumed we have benchmarked a single device, and wanted to
estimate how some number of these devices would affect delay 
variation when deployed on the path. Clearly, if we can benchmark
a particular black-box SUT with a specific combination of these devices,
there's no need to do the estimation for that case.
But having the ability to estimate e2e variation for an unknown number on the path
seems valuable to me, and I wonder if Data Center Engineers agree.

it's been a long day here, hope this makes sense,
Al

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Lucien Avramov (lavramov)
> Sent: Wednesday, July 31, 2013 4:27 AM
> To: David Newman
> Cc: [email-masked]

---

From: David Newman <[email-masked]>
Date: Wed, 31 Jul 2013 11:32:45 -0700

On 7/31/13 1:26 AM, Lucien Avramov (lavramov) wrote:

> The goal is to first measure it correctly

Hi Lucien,

This implies there is One True Method for jitter measurement. I wish
that were true, but it's not.

As it is, we have at least three definitions: Two in RFC 3393 (and that
document also deprecates the term), and another in RFC 4689, which came
out of this working group. There are others too.

This reinforces Al's point about the merits of the bmwg approach of
terminology and then methodology documents.

If nothing else, I think you'd be doing testers and implementers of test
instruments a big service by defining exactly which method is in use.

The two bullet points in section 3.2 are a good start, but note that
there are different ways to measure "variation," each with its own pros
and cons. (For example, for some applications, in-sequence is critical,
for others it doesn't matter, and so on.)

Thanks!

dn

---

From: David Newman <[email-masked]>
Date: Wed, 31 Jul 2013 11:38:29 -0700

> it's been a long day here, hope this makes sense,

Mostly. Agreed 100 percent on the requirement for e2e estimation.

Unclear how that estimate would differ from a single-box estimate times
some number of identical boxes, modulo cable lengths.

And regardless of box count, there's still the question of which  method
of jitter measurement to use.

That was my original point: Given so many definitions to choose from,
it'd be nice to spell out one definition in this document, either
referencing something else, or defining something new (as in the use of
FILO for latency measurement).

dn