Subject: Re: [bmwg] Benchmarking MPLS Forwarding

From: Curtis Villamizar <[email-masked]>
Date: Mon, 10 Dec 2012 15:58:22 -0500

Jay, Al,

Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
the comments.

See inline.

Curtis

In message <[email-masked]>
"Jay Karthik (jakarthi)" writes:
> 
> Al,
>  
> I took a look at this draft and in my opinion, most (if not all) of
> the clarifications in section 1.1 would need to be benchmarked
> appropriately.  Many of these scenarios described are beyond the scope
> of previous work (MPLS Forwarding Compliance and Performance
> Requirements/RFC5695) which focused on label stack depth of 1.

Label stack depth is one variable that has not been considered so far
in BMWG.

Another is a look at packet forwarding rate vs packet size with a fine
granularity of packet size.  When doing so, an sawtooth is observed.
In my experience, the sawtooth is almost always due to memory width
and an optimization toward packets of with 64 byte payloads (payload
padded in ethernet, plus header yielding on the order of 84 byte
times) and then multiples of some power of two, often 64.  It is
common to store an internal representation of destination in a
relative small number of bytes, such as 16, and then store the payload
separately in other banks of RAM.  The result is that the worst
performance is that experienced by packts with payloads that are a
little larger than 64 bytes.  This is bad news if carrying something
like pseudowires that may have small payloads, but a little larger
than 64 bytes.

> The entire section 4 (Forwarding Compliance and Performance Testing)
> could potentially be a new work item.

I agree but also think it might be worth breaking this up functionally
into more than one work item.  There has also been discussion off list
aimed at expanding on the pseudowire requirements and adding mpls-tp
oam forwarding requirements (currently the impact on POP and counters
is called out, but not the OAM itself).

It is nice to hear that there is interest in BMWG.  It might be best
though to wait for this work to settle a bit in MPLS so as not to be
working toward too much of a moving target.

> Here are some comments on draft-villamizar-mpls-forwarding-00.

I'm reading -01-preview1, but I think it is the same here.

> If T#1 in section 4 fails, the max supported depth should be
> determined for a given implementation.

The current text says "While packets with more than a dozen label
entriess are unlikely to be used in any practical scenario today, it
is useful to know if limitations exists."

So we appear to be in full agreement.  If this is not clear enough,
please suggest new wording.

> Subcase T#3b needs to be modified. While benchmarking, a device is
> either a tester/tool or a DUT. Specifically, the procedure described
> should apply to any DUT and not a router that has certain capability.

I think the issue is the sentence "One way to accomplish this is to
use a router with higher priority set on the interfaces on which small
packets are sent to it."  That router is not the DUT.  It is another
router front ending the DUT and providing the merged flow of traffic.
I can make that more clear by moving the entire suggested test setup
to an appendix with an ascii line drawing and referring to it from the
main body of the document.

The point here is that router A (front end) gets packets on more than
one interface and sends them out one interface.  Router B, the DUT
receives the packets complete with bursts of small packets on one
interface.  Router A has two or more forwarding engines working on the
problem and therefore is more likely to not bog down the its
forwarding engines.  The output side of router A could also be a
bottleneck but only if the issue is purely output side buffering and
is not related to the decision engine.  Clearly more than a paragraph
in the main part of the document is needed.

> T#5 is a generic functional validation that could apply to most cases
> (eg T#1). Same comment applies to other cases, which validates
> functionality or tests conformance (eg T#10). These tests could be
> part of the validation procedure but may not represent a benchmark.

The point in T#5 is that we've seen equipment in which the on-chip
counters are small enough that if all traffic went to the same LSP,
the counters would have to be polled in under 2 seconds, under 1
second if you prefer to avoid a rollover by half the counter capacity.
Since you don't know apriori which LSP is going to have a long burst
of activity, a brute force solution would require polling all of the
counters every second to update the 64 bit external DRAM version.
Some architectures are close to unworkable due to very small on chip
counters.  This really is a performance issue.  There is no reason
that the on-chip counters must be 64 bit, as long as there is an
efficient way to offload the counters and get them into 64 bit
counters in DRAM for the purpose of returning MIB values.

I can see your argument that T#10 is really a basic conformance test,
however it is now under "Multipath Capabilities and Performance" and
while it is conformance, it is conformance specific to multipath.  I
think T#10 is fine where it is.

> A few more test scenarios could be created to measure the max label
> stack size with PW control word, flow label and Entropy label. This is
> different from T#1 where the focus is to measure in general, the
> maximum stack size supported by DUT, instead of characterizing the max
> supported in different scenarios.

Andy Malis and I have had a long email thread off list.  Expect to see
a lot more in the 01 draft on pseudowire.

> -Jay
>  
> On 11/8/12 9:51 AM, "Al Morton" <[email-masked]> wrote:
>  
> >Folks,
> >
> >Curtis suggested I take a look at this (I did),
> >http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-11
> >and I think others should take a look, too.
> >
> >Food for thought (at least) as we consider new work.
> >
> >regards,
> >Al
> >bmwg chair 

Thanks again for the comments.  It would be best to keep this on the
MPLS mailing list for now and return to BMWG later.  That said, the
MPLS WG list has been silent on this but I have been getting quite a
few comments in private email.  (Mostly from service providers.)

Curtis

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sat, 15 Dec 2012 22:30:14 +0000

Curtis,

This is a very nicely written document that calls out important topics. Please find some bmwg-focused comments inline.

> Thanks for looking at the mpls-forwarding draft.  Jay - thank you for

Currently the document filename is draft-villamizar-mpls-forwarding, from which it could be extrapolated (1id-guidelines) that it targets the mpls wg, with a document name of "forwarding". However, there seem to be two multiple to this document: clarification on misconceptions, and conformance and performance requirements (and benchmarking?). More on this below.

> working toward too much of a moving target.

I also tend to thing that S4 is a bit its own document (or plural if extended in scope). I wonder also how much of a continuation of RFC 5695 some parts should be. 

> in the main part of the document is needed.

One thought is that perhaps defining up front a topology and methodology can make these easier to follow and more mechanic.

> second if you prefer to avoid a rollover by half the counter capacity.

While I like testing for this, and like this test, it might appear that it is a bit specific as compared to others. For example, these is no mention of spreading or dispersing the forwarding among multiple LSPs versus sending all the traffic in one LSP for forwarding performance; however there is this mention in the context of counters.

> think T#10 is fine where it is.

The location of T#10 makes sense in terms of multipath testing. However, it is not necessarily obvious or clear the methodology to make such determination (especially as it is proving a negative).

> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
> 

Thanks,

-- Carlos.

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 19 Dec 2012 14:56:00 -0500

Carlos,

Thanks for the comments.  Please keep in mind that this discussion is
a "heads up" from Al to BMWG about work in the MPLS WG.  The MPLS work
is more about requirements and is not about precise specification of
benchmarks which would be better dealt with in BMWG.

See inline.

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
> 
> Curtis,
>  
> This is a very nicely written document that calls out important
> topics. Please find some bmwg-focused comments inline.
>  
> Currently the document filename is draft-villamizar-mpls-forwarding,
> from which it could be extrapolated (1id-guidelines) that it targets
> the mpls wg, with a document name of "forwarding". However, there seem
> to be two multiple to this document: clarification on misconceptions,
> and conformance and performance requirements (and benchmarking?). More
> on this below.

You are correct.  draft-villamizar-mpls-forwarding is work intended
for MPLS WG about MPLS forwarding.

There is a bit of text in the draft on motivation for this work (a few
chip vendors got it very wrong, a few with a common theme - hence the
clarification on misconceptions).

> I also tend to thing that S4 is a bit its own document (or plural if
> extended in scope). I wonder also how much of a continuation of RFC
> 5695 some parts should be.

Section 4 is intended to be an outline of things to test for, without
specifying a methodology for testing in the way that BMWG does.

I should reference RFC 5695 as existing work.

However, RFC 5695, like other BMWG work misses the sawtooth that can
only be discovered using smaller increments in packet size, typically
large powers of two starting with 64 bytes.  This is highly
*counterproductive* since chip vendors may then design for the
benchmark and put forward designs which are poorly suited for real
world traffic.

There is a tendency going all the way back to the Bradner IP tests in
the early 1990s (hi Scott, sorry to bring this up again, but..) in
BMWG to precisely specify benchmarks whish prove inadequate and are
therefore counterproductive.

IMHO this tendency can be corrected by first considering the
requirements in greater detail, and *then* specifying a set of
benchmarks.  The focus must be less on making the benchmarks quick and
easy to perform and report and more on making the benchmarks
adequately reflect real world network requirements.

Doing work on requirements in MPLS WG first, then work on benchmarks
in BMWG *might* better accomplish this.

> One thought is that perhaps defining up front a topology and
> methodology can make these easier to follow and more mechanic.

The goal of the MPLS work is *not* to precisely specify a set of
benchmarks, although BMWG is welcome to do so.

There was some text in Subcase T#3b regarding the feasibility of
testing (for very high speed interfaces such as 100Bb/s interfaces)
which was too terse to be clear (as Jay pointed out).

That text should be either removed entirely or moved to an appendix
and expanded on.  If expanded on it should be made clear that the text
only exists to indicate a means of inserting bursts of small packets
of known burst durection into streams of larger packets.  That text
was entirely for the benefit of anyone that might wonder how it would
be feasible to do this and maybe should just be removed.

> While I like testing for this, and like this test, it might appear
> that it is a bit specific as compared to others. For example, these is
> no mention of spreading or dispersing the forwarding among multiple
> LSPs versus sending all the traffic in one LSP for forwarding
> performance; however there is this mention in the context of counters.

In the update that I've been working on after IETF-85, there is more
coverage of large flow and large number of flow issues.  That section
was prompted by a conversation just after the MPLS WG presentation and
some follow up private email.

The counter problem has been overlooked by some chip vendors
accounting for it getting into the list of tests before other large
flow and large number of flow issues.

> The location of T#10 makes sense in terms of multipath
> testing. However, it is not necessarily obvious or clear the
> methodology to make such determination (especially as it is proving a
> negative).

Currently T#10 reads:

       T#10  Determine whether reserved labels are included in the
             label stack hash.  They MUST NOT be included.

The MUST NOT is new and the change from SHOULD NOT is due to many
comments asking for change.  It is needed for OAM to work right.

I could change the wording to

       T#10  Determine whether reserved labels are excluded from the
             label stack hash.  They MUST be excluded.

The way to make this determination is send traffic and then send the
same traffic with a router-alert or GAL label down the stack a ways.
If the traffic splits differently with the reserved label, then you
have identified a problem.

You could also just carry multiple MPLS-TP LSP inside MPLS LSP without
EL (TP label must be at the very bottom) and use the TP LM OAM and see
if it comes out right.  If the counts are off a little , the reserved
labels were used and there was a little bit of reordering.

[ Or you could wait for it to happen in the field and then deal with
the complaints.  just kidding. ]

> > https://www.ietf.org/mailman/listinfo/bmwg

Thanks again for the comments.  

I will do the following: 1) point out the applicability of RFC 5695,
2) reword T#10 to eliminate the double negative, 3) do something
(exactly what is TBD) about the extraneous methodology hints in T#3b
that Jay pointed out.

Curtis

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Dec 2012 15:30:34 +0000

Thanks for the detailed response, Curtis.

A couple of follow-ups inline.

> have identified a problem.

Yes. But if the traffic does not split differently, it does not mean you do not have a problem. How much precision do you feel is needed in the specification of testing cases like this? Should all testings use the same methodology to perform the determination in T#10?

> Thanks again for the comments.  

Thank you!

> (exactly what is TBD) about the extraneous methodology hints in T#3b

Sounds good. Thanks,

-- Carlos.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 20 Dec 2012 13:10:36 -0500

Carlos,

Inline.

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
> Yes. But if the traffic does not split differently, it does not mean
> you do not have a problem. How much precision do you feel is needed in
> the specification of testing cases like this? Should all testings use
> the same methodology to perform the determination in T#10?

Use far more than one label stack in the test so that you can't
possibly get a false indication of success.  But this MPLS document is
not about testing methodology.  My response is just part of a
discussion with you on feasibility of doing this sort of testing.
This could end up in a BMWG document *later* but doesn't belong in
the MPLS document.

> > (exactly what is TBD) about the extraneous methodology hints in T#3b

Thanks again for the comments and followup.  I will send you diffs off
list when the next authors 01-preview comes out and if you like you
can check to see if your comments are adequately addressed before we
send out an 01 version.

Curtis