Subject: RE: [bmwg] comments on draft-sdry-bmwg-mvpnscale-00

From: "Silvija Andrijic Dry \(sdry\)" <[email-masked]>
Date: Tue, 13 Feb 2007 08:51:09 -0500

Thank you for the feedback Al ! 
We will rewrite scope section to be more clear in 01 rev that we are
working on now.
We will also reword use of "failure" throughout the document.

Regarding following question you had:
> I'd also be interested to know if the configuration parameters are
sufficiently generic in the 
> opinion of other MVPN implementors.

Authors belive that config parameters are sufficiently generic. I am
personaly familiar with implementations of two major router vendors and
all configuration parameters from this draft are supported by both
vendors. 
Does anyone else have opinion on this one ?

Thx,
Silvija

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Wednesday, January 31, 2007 1:01 PM
To: [email-masked]
Subject: [bmwg] comments on draft-sdry-bmwg-mvpnscale-00

Silvija reminded me to type-up a few comments/suggestions I had when I
read this draft last year.

Al (as a participant)

Scope:

As a candidate work item, it's important to have a clear Scope so that
everyone understands what is proposed.
The Introduction section has some material in the 2nd paragraph that
should be part of the existing Scope section:

>    There are multiple proposals on architectures and protocols
currently
>    in IETF for implementing MVPN service. This draft will describe
>    methodology for benchmarking MVPN scalability only for the
>    implementations following [ROSEN-8] or portion of [L3VPN-MCAST]
which
>    uses PIM protocol to create 'tunnels' that instantiate MI-PMSIs (or
>    MDT tunnel) and S-PMSIs (or data MDT). We are limiting scope only
to
>    above mentioned implementation as that is the only one deployed
>    today.

Also, spell-out all the acronyms at first usage:
>...MI-PMSIs (or MDT tunnel) and S-PMSIs (or data MDT).

These are some other points, spread around in the document, that
contribute to the Scope (you can find them with a search):

>       ...In this document it will be assumed
>      that MVPN service is deployed on the top of a "minimized" unicast
>      control plane.

>    We choose to limit the scope of this document so it is suitable for
>    standard comparison of disparate implementations. We included set 
> of
                                                                  ^a
>    test cases (8.12-8.14) that will be helpful to operators with
network
>    engineering for their deployments. Choices of values of variables
in
>    test cases 8.12-8.14 were made using information from the MVPN
>    requirements survey conducted as part of [MVPN-REQ].
(I thought the paragraph above was very good.)

>    All P facing interfaces MUST use OSPF as IGP.

>    As the focus of this document is on the control plane scalability
and
>    not on forwarding performance the data rate and packet size of
>    traffic flows can be chosen by user but it MUST be reported in the
>    test results.

>6  Test Categories, Stimulus and Execution Methodology
>
>    Following are two major test categories that this document
addresses:
>
>   . Steady State Testing: DUT and network as a whole are not subject
to
>      any failure stimulus/control plane events.
>   . Failure Recovery Testing: DUT and or network components are
subject
>      to different failure stimulus that introduce one or more control
>      plane events.
(these categories are mentioned briefly in the current scope)

Other comments (non-scope-related):

3rd paragraph Intro:
>    Before describing the detailed test methodology, it is important to
>    review the key factors that impact scalability of MVPN deployments:
>
>    o The MVPN Metric: a set of variables that when combined indicates
>      the scalability capabilities of a PE device. MVPN scalability is
>      multi-dimensional and can not be quantified with single parameter
>      thus defining such a metric set is necessary. MVPN Metric will be
>      defined in section 4. The rest of this document focuses on a
>      methodology that characterizes different aspects of MVPN Metric.

It seems that you're talking about a set of related metrics or
variables,
13 in all if you look ahead to section 4.  It might be more clear to
say:

     o The MVPN Metric is a 13-tuple comprised of a set of variables
that
       indicate the scalability of a PE device in various dimensions.
...

The current Scope section has a paragraph:

>    DUT (Device Under Test) term will be used interchangeably with MVPN
>    PE device. VPN related terms used in this document are defined in
>    RFC4364 and RFC2547bis. MVPN related terms used in this document
are
>    defined in [ROSEN-8] and [L3VPN-MCAST].

We usually put this sort of info in an "Existing Terms" section.

In Section 6.1, item 6):

>      6) All multicast traffic MUST be received at all expected
>         destination test ports without any packet drops. This MUST be
>         verified using external test apparatus. If this state can not
be
>         reached within 10 minutes of execution of step 5 test case
>         instance is considered failed and next test case instance with
>         reduced value of scaled variable/s needs to be performed.

Pass/Fail testing is not in the BMWG charter so this must be re-worded.
Something like:
                  ...  If this state can not be
         reached within 10 minutes of execution of step 5 test case,
>>> then the "recovery time" is Undefined, and next test case instance
with
         reduced value of scaled variable/s needs to be performed.
(not sure that you want to use recovery time, above, maybe just
"recovery")

There are lots of other instances of "failure" in the draft, these need
to be re-worded, too.

I'd also be interested to know if the configuration parameters are
sufficiently generic in the opinion of other MVPN implementors.

That's it for now...

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Rajiv Asati \(rajiva\)" <[email-masked]>
Date: Mon, 19 Feb 2007 17:19:48 -0500

> > opinion of other MVPN implementors.

I am of opinion that the config parameters listed in the draft are quite generic.
Thanks.

Cheers,
Rajiv

> From: Silvija Andrijic Dry (sdry) 
> Sent: Tuesday, February 13, 2007 8:51 AM
> To: Al Morton; [email-masked]
> Subject: RE: [bmwg] comments on draft-sdry-bmwg-mvpnscale-00
>

---

From: Thomas Morin <[email-masked]>
Date: Tue, 20 Feb 2007 17:38:28 +0100

Hi Silvija, Al,

I had a bunch of comments, which I made in San Diego, but which I didn't
find the time to write down before now. 

Here is a summary:
- having benchmarking guidelines on multicast vpn is definitely
something useful and a welcome initiative
- there is an issue regarding the terminology: the terms used comes most
of the time from draft-rosen-vpn-mcast-08 (has expired 1.5 year ago)
though the document is said to apply to both draft-rosen-vpn-mcast-08
and a subset of draft-l3vpn-2547bis-mcast (said differently: normative
references only should be required for understanding the document)
- I generally feel that it is a unwise use of resources to only target a
subset of a standard, especially when many of the benchmarks (if not
most) are relevant independently of the implementation choice (like it
is the case here).
- there is an issue regarding the mvpn implementation option to which
this draft applies : to exactly which sections of
draft-l3vpn-2547bis-mcast does it apply ? the document says it targets
implementations which "[use] PIM protocol to create tunnels that
instantiate MI-PMSIs and S-PMSIs", but reading the doc it seem to
actually implicitly make the additional assumption that option "Full
Per-MVPN PIM Peering Across a MI-PMSI" (described in section 5.2.1 of
draft-l3vpn-2547bis-mcast) also has to be used...  It seems to me that
the draft actually targets a _very_ restricted subset of the procedures
described in draft-l3vpn-2547bis-mcast. 
- some of the tests proposed are relevant to multicast in general,and
not specifically to multicast when used to carry VPN traffic. I believe
such tests would deserve being placed in a more generic document (when
benchmarks defined in RFC3918 are not enough).  Numerous tests in this
document are related to control plane stress due to how the mvpn
application uses multicast as provided by the operator infrastructure :
this is relevant not only for PE-routers, but also for P-routers. To
evaluate a platform or engineering options, multicast scalability should
separately be benchmarked, whether or not mvpn will be used. I think
such tests belong to a separate doc.

My suggestion would be to restructure the document so that everything
that is generic enough is described without making too much assumption
on how mVPN is implemented. When and if required, specific tests
applicable to a specific way of implementing mvpn could be also
described. Also, when applicable pointers to benchmarking documents
related to multicast should be referred (e.g. : RFC3918, RFC2432).
This would have the benefit of making the document more readable (eg.
the number of parameters would be greatly reduced).

Then, I would expect a review of such a benchmarking document by the
l3vpn working group, at some point.

[ Inlined below are a few more detailed comments on different sections,
including suggestions about which tests/section I feel should be
"factored", which could be kept separate, and which could be maybe
replaced by pointers to more generic documents.  ]

Cheers,

-Thomas Morin

>   1  Introduction...................................................3 

This implementation is rather dry (no pun intended):
- maybe the introduction could first expand the MVPN acronym ?
- the use of the "overlay" term is surprising.
- also "BGP VPNs" should probably be more precisely explained :
"BGP/MPLS VPNs [RFC4364]" ?

(But this is cosmetic considering the amount of revision that I think
would be required elsewhere.)

>    2  Document Scope.................................................4 

It seems to me that benchmarking "steady state" and "failure state" is
fine for unicast, but that multicast-related benchmarking would require
testing also the "membership changes stress" case (i.e. how a device
performs when a lot of multicast membership changes happen).

>    3  Key Words to Reflect Requirements..............................5 
>    4  MVPN Metric Definition.........................................5 

>    The MVPN Metric is defined as a tuple of the following 13 variables: 
> 
>    Global Domain Related Variables 
> 
>   1. Num_mVPN: Number of multicast VPN routing instances configured on 
>      DUT that have MI-PMSI active and forwarding. 
>   2. Num_*G_P: Total number of (*,G) multicast routes on DUT capable of 
>      forwarding and created by PIM P-instance on DUT. 
>   3. Num_SG_P: Total number of (S,G) multicast routes on DUT capable of 
>      forwarding and created by PIM P-instance. 
>   4. Num_OIF_P: Total number of OIFs (outgoing interfaces) on DUT across 
>      all multicast routes created by PIM P-instance. 

Here is an example of how things may not be abstracted enough in this
draft: "Variables" 2 and 3 above are not in fact variables independent
from any other variables, they are dependent on the number of VPNs,
likely the average number of PEs in each multicast VPN, the option
chosen to carry multicast in the provider backbone. But they are not
intrinsic metrics of the mvpn deployment/simulation.

>   VPN Domains Related Variables: 
> 
>   1. Num_MC_C_ints: Number of PIM C-interfaces on DUT 
>   2. Num_PIM_C_neigh: Total number of PIM neighbors in PIM C-instances 
>      across all mVPNs on DUT. 
>   3. Num_*G_C: Total number of (*,G) multicast routes across all MVPNs 
>      on DUT capable of forwarding and created by PIM C-instances. 
>   4. Num_SG_C: Total number of (S,G) multicast routes across all MVPNs 
>      on DUT capable of forwarding and created by PIM C-instances. 
>   5. Num_OIF_C: Total number of OIFs on DUT across all multicast routes 
>      created by PIM C-instances. 

Here we have I think another example: "Total number of PIM neighbors in
PIM C-instances across all mVPNs on DUT" should I think be reformulated
into "Total number of remote PEs across all mVPNs defined on DUT". It is
equivalent, more generic, and I think a little easier to read.

>    5  Test Environment...............................................6 
>       5.1   Test Topologies..........................................6 
>       5.2   Unicast Control Plane Setup..............................7 
>       5.3   Multicast Control Plane Setup............................7 

> If there are multiple sources per group in a C-instance then they 
>    MUST be located behind the same PE router. 

(Why so ?)

[...]

> Test apparatus ports that are physically directly connected to DUT 
>    (port A1 in Figures 1 and 2) MUST not use IGMP protocol to emulate 
>    multicast receivers. Instead PIM protocol must be used, i.e. the DUT 
>    MUST not be the last hop router.  

Why such a limitation ?
It should surely be precised in a report which of IGMP and PIM was used,
and the use of PIM should be recommended but I fail to see why
benchmarking the PE device as a last hop device would be precluded.

>       5.4   Data Traffic Characteristics.............................8 
>       5.5   Test Apparatus Considerations............................8 

[...]
> 5.5 Test Apparatus Considerations 
> 
>    Different test tools must generate PIM protocol control messages in a 
>    consistent way since they are directly connected to the DUT. 
> 
>    The following MUST be implemented on all PIM sessions on the test 
>    apparatus: 
> 
>      1) PIM Join/Prune aggregation MUST be supported and set such that 
>         80 PIM J/P messages are aggregated in each PDU      
> 
>      2) PIM Join/Prune aggregated PDUs MUST be sent at 10 PDUs/sec rate 
>         per PIM session, i.e. this translates to maximum of 
>         80*10*60=48,000 state per minute. 
> 
>      3) In order to closer mimic realistic deployments test apparatus 
>         MUST send all control plane messages in 10 equal size batches 
>         with at least 5 seconds between each batch. 

(Why these values ?)

>       5.6   Considerations for distributed architecture platforms....9 
>    6  Test Categories, Stimulus and Execution Methodology............9 
>       6.1   Steady State Testing....................................10 
>       6.2   Failure Recovery Testing................................11 

> 6.2 Failure Recovery Testing 
> 
> [...]
>      7) If any one of following conditions are reached the test case 
>        reduced value of scaled variable/s needs to be performed: 
> 
>           a. Value of MVPN metric in steady state reached after failure 
>              stimuli (step 6 above) is not the same as in original 
>              steady state. 
> 
>           b. Multicast latency [RFC2432] averaged over all C-instance 
>              multicast flows in steady state after failure recovery 
>              stimuli is more than 10% larger than in original steady 
>              state 

I'm not sure of the reason for this "10%" : e.g. having latency go from
50ms to 60ms is no sign of failure, and no issue either.  Maybe
introducing a lower value under which the test is considered a success,
would be a solution (eg. 80ms) ?

>    7  Results Content and Reporting Format..........................13 
>       7.1   Steady State Testing....................................13 
>       7.2   Failure Recovery Testing................................14 
>    8  Test Cases - Steady State Testing.............................14 
>       8.1   "Empty" MVPNs Scale.....................................14 

>       8.2   PIM Enabled VPN C-Interfaces Scale......................16 
>       8.3   PIM Neighborships Scale.................................18 
> 
>       8.4   Default MDT's Multicast State Scale.....................20 
>       8.5   VRF Multicast State Scale...............................22 
>       8.6   VRF Multicast OIF Scale.................................24 
>       8.7   Joined Data MDT Scale...................................26

[ This one is an example of one of my introductory comment on including
tests related to multicast in general in this mvpn specific draft. 8.4
and 8.7 actually test the same thing : how PE devices scale in terms of
multicast states. This is duplicated work. Notwithstanding the fact I
would expect such a test to be described in a more generic document
(note that RFC3918 section 7.1 describes something similar).  ]

>       8.8   Sourced Data MDT Scale..................................28 
>       8.9   Data MDT Reuse..........................................30 

This specific test is I think complete nonsense : it implicitly suggests
that a PE device allowing more data-MDT groups to be reused is better.
This is nonsense because, depending on the deployment context, the
operator may very well have reason to actually prefer that no reuse
would happen, or to have controlled reuse (reuse data mdt groups up
to...) of groups allocated for data-MDTs.

[ This is IMHO a very illustrative example of a benchmarking suite
written by one vendor for his own platform. ]

>       8.10  PIM J/P Suppression Effectiveness.......................31 
>       8.11  Additional Tests for Devices Lacking "Efficient" Join/Prune 
>       Suppression...................................................34 

These two tests are the only ones that are very specific to the use of
PIM LAN procedures (a la draft-rosen). All the other tests in this
section are I think generally relevant to any choice of mvpn
implementation and could be rewritten to be more generic.

>       8.12  Scale of mVPNs spanning large number of PEs.............34 
>       8.13  Scale of mVPNs with larger amount of state..............36 
>       8.14  Scale of "average" size mVPNs...........................38 
> 
> 
>    9  Security Considerations.......................................40 
>    10    IANA Considerations........................................40 
>    11    Acknowledgments............................................41 
>    12    References.................................................41 
>       12.1  Normative References....................................41 
>       12.2  Informative References..................................41 
>