Subject: [bmwg] BMWG Status (Feb 2009)
From: Al Morton <[email-masked]>
To: [email-masked]
Date: Fri, 06 Feb 2009 15:09:42 -0500
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>

BMWG folks,

We ended 2008 with several items in last call
and/or out for comment. Here's where we stand
right now (AFAICT).

Please note that March 9 is the final cutoff
for submission of revised I-Ds:
http://www.ietf.org/meetings/74/cutoff-dates.html

Al
bmwg chair

***********************************************
IGP Dataplane Convergence drafts

Preliminary versions with WGLC comments addressed
have been sent to all those who commented, and
awaiting agreement before posting revised versions.

***********************************************
WG Adoption of draft-poretsky-sip-bench-term-05
...objections on the list
before the end of day, December 16, 2008.

No objections or alternate proposals, and with
the interest expressed along the pre-chartering path,
I think we can adopt this as a WG draft.

************************************************
WG Last Call period for the Internet-Draft on
"MPLS Forwarding Benchmarking Methodology"
draft-ietf-bmwg-mpls-forwarding-meth-01.txt
open from 2 December through 23 December, 2008.
(mpls WG was added to the WGLC)

Extensive comments (both substantive and editorial)
from Carlos Pignataro.
https://www.ietf.org/mailman/private/bmwg/2008-December/000095.html

*************************************************
WG Last Call period for the Internet-Drafts on
Sub-IP Layer Protection Mechanism Performance Benchmarking
draft-ietf-bmwg-protection-term-05.txt
draft-ietf-bmwg-protection-meth-04.txt
open through 30 January, 2009.

Active Review Templates completed by:
Mohan Nanduri,
supported both drafts, no other comments
Andrey Kiselev,
strongly supported both drafts (based on his current
experience with benchmarking solutions), one editorial comment
on the method draft.
Raveesh Janardan,
strong support and complements on the Methodology draft.
Karu Ratnam,
support for both term and meth, and some uncertainties expressed.
The general concern about repeatability should be pursued with
this reviewer to see if a few considerations can be added.

Also, your chairman has not been able to make time to review the
term and meth drafts in their current form. Will do so ASAP.

**********************************************
Comments on Jan Novak's ipfix proposal from Benoit Claise,
https://www.ietf.org/mailman/private/bmwg/2008-November/000082.html
and Jan's response
https://www.ietf.org/mailman/private/bmwg/2008-November/000083.html

Date: Fri, 06 Feb 2009 15:13:09 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] BMWG Status (Feb 2009)

One omission:
Arun Gandhi also expressed support for the methodology draft
and filled out the template.

sorry Arun!

At 03:09 PM 2/6/2009, Al Morton wrote:
>*************************************************
>WG Last Call period for the Internet-Drafts on
>Sub-IP Layer Protection Mechanism Performance Benchmarking
>draft-ietf-bmwg-protection-term-05.txt
>draft-ietf-bmwg-protection-meth-04.txt
>open through 30 January, 2009.
>
>Active Review Templates completed by:
>Mohan Nanduri,
>supported both drafts, no other comments
>Andrey Kiselev,
>strongly supported both drafts (based on his current
>experience with benchmarking solutions), one editorial comment
>on the method draft.
>Raveesh Janardan,
>strong support and complements on the Methodology draft.
>Karu Ratnam,
>support for both term and meth, and some uncertainties expressed.
>The general concern about repeatability should be pursued with
>this reviewer to see if a few considerations can be added.

Date: Mon, 16 Feb 2009 16:19:29 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

...-bmwg-protection-meth-04 Authors,

Since there have been some comments during WGLC,
I hope you can take these late comments into account.
I only just made time to do a review during a flight.

I've raised some issues below that the test equipment community
may be interested to comment on.

Al (as a participant)

General:
--------

This memo is very nearly ready for publication request,
and closing the loop on a several details will make it so.

There are lots of section references that need +2 (3.1 -> 5.1)
or other fix-up.

Employing the "standard" Intro/Security paragraphs would help.

This work effort uses the IGP-Dataplane convergence work as a
foundation, and might also benefit by adding a new section on:
>    3.2.9 Tester Capabilities
>    It is RECOMMENDED that the Tester used to execute each test case
>    have the following capabilities:
>       1. Ability to insert a timestamp in each data packet's IP
>          payload.
>       2. An internal time clock to control timestamping, time
>          measurements, and time calculations.
>       3. Ability to distinguish traffic load received on the Preferred
>          and Next-Best interfaces.
>       4. Ability to disable or tune specific Layer-2 and Layer-3
>          protocol functions on any interface(s).
and some of this is applicable to MPLS-FRR.  A section like this would
be a good place to expand on the capabilities needed to distinguish
between impaired and unimpaired packets (per-flow sequence numbers).

I can make a lot of word-smithing suggestions,
but they would be too tedious to type-up explicitly,
and then for someone else to locate and change.

One key set of related issues to reconcile in section 7:
--------------------------------------------------------
Definition of "maximum forwarding rate (pps)"
(currently assessed in section 7.5)
In my opinion,
  - this is a setup condition needed for all the other
procedures in section 7, so this section should come first.
  - there should be a reference to the existing definitions
of Forwarding Rate and Maximum Forwarding Rate
http://tools.ietf.org/html/rfc2285#section-3.6.3
and their methods of measurement
(or possibly use Throughput instead, and refer to RFC 2544 and
draft-ietf-bmwg-mpls-forwarding-meth-01.txt for the method).

Also, there is no test to verify that *all* the offered load
is switched to the backup path and that stability is achieved
before turning off the load and assessing switching time.

Another key issue, assumption of Reversion performance:
-------------------------------------------------------
The draft says (in many places) that Reversion should not produce
packet loss, but this really depends on the implementation.
Also, our charter states:
>Because the demands of a particular technology may vary from deployment
>to deployment, a specific non-goal of the Working Group is to define
>acceptance criteria or performance requirements.
I think we've managed to convey a "desirable outcome" in the past
without wording that might be misinterpreted as a requirement

Other specific comments:
------------------------

S1 Intro:
---------
The 3rd and 4th paragraphs describe correlated failures and
planned failures, but they aren't particularly useful in the
remainder of the memo:
  - The scenarios that test correlated failures aren't identified
as such.  Sec 5.1 has a few examples (Parent Interface shut,
Reload/Crash Protected Node), but most are single failures.
  - The notion of a "Planned Failure" that takes a node or interface
out-of-service involves other activities to minimize the customer
impact, such as pre-notification, time of day, and diverting traffic
before taking action (anything out of service). Since my day-job
is with a service provider, I think that unplanned failures should
be the main emphasis here.

S2 Scope:
---------
I think that the last paragraph of Sec 1 should be part of the Scope
section.

The current 1st paragraph mentions "scenarios" twice, but section
5.1 lists and defines all the failure "events", and events are not
mentioned.  Maybe this would be better:
    This document provides detailed test cases along with different
    topologies and scenarios that should be considered to effectively
    benchmark MPLS protection mechanisms and failover times. Different
    failure *events* and scaling considerations are also provided in
    this document, in addition to reporting formats for the observed
    results.

The last sentence of the Scope:
    Benchmarking of unexpected correlated failures is currently out of
    scope of this document.
Wouldn't a Node Crash event be an example of an unexpected correlated failure?

S3 General ref and sample topology
----------------------------------

2nd sentence:
    ... TG & TA represents
    Traffic Generator & Analyzer respectively.
suggest change to:
TG & TA stand for Traffic Generator & Traffic Analyzer respectively,
and these components comprise the "tester".

Figure 1 is too wide!
I count 85 characters, including 5 leading spaces. The max is 72.
See 
http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-meth-04.txt
for more stuff like this...

Section 5.3 last paragraph reads:
---------------------------------
    In addition, this methodology considers the packets in error and
    duplicate packets that could have been generated during the failover
    process. In scenarios, where separate measurement of packets in
    error and duplicate packets is difficult to obtain, these packets
    should be attributed to lost packets.
A duplicated packet should *never* be considered the same as a lost packet.
I think you want to say that some of the methodologies consider lost,
out-of-order, and duplicate packets to be impaired packets that
all determine the failure recovery time.

Section 5.5, Selection of IGP
-----------------------------
Add a sentence/reference:
     See [IGP-DATAPLANE] for IGP options to consider and record.

Section 5.6, Reversion
----------------------
    ...In all test cases listed here
    Reversion should not produce any packet loss, out of order or
    duplicate packets. Each of the test cases in this methodology
    document provides a check to confirm that there is no packet loss.
This is not strictly true, especially since the presence of
out-of-order packets depends on the difference in delay between
backup and primary paths.

I suggest:
    ...In all test cases listed here,
    Reversion is actuated to observe any packet loss, out of order or
    duplicate packets. <delete last sentence>

Section 5.7, Traffic Generation
-------------------------------
The recommendation against round-robin traffic generation to prefixes
seems to be inconsistent with section 3.9 of [IGP-DATAPLANE]:
    ...The destination addresses
    for the offered load MUST be distributed such that all routes are
    matched and each route is offered an equal share of the total
    Offered Load.  This requirement for the Offered Load to be
    distributed to match all destinations in the route table creates
    separate flows that are offered to the DUT.  The capability of the
    Tester to measure packet loss for each individual flow (identified
    by the destination address matching a route entry) and the scale
    for the number of individual flows for which it can measure packet
    loss should be considered when benchmarking Route-Specific
    Convergence [Po07t].
This difference needs to be sorted-out.

Section 6
---------
Some nouns may be missing:
    f) PRI is Primary *Node or Path*   ??

    g) BKP denotes Backup Node *or Path*  ??

Section 6.1 (and beyond)
------------------------
The Label stack only makes sense if PE and P are
indicated in (all) the Figure(s).

Section 8, Reporting Format
---------------------------

Suggest adding:
     Note the presence of out-of-order and duplicate packets, when measured
as an item in the Benchmarks list.

Also:
       3. Timestamp Based Method (TBM): This method of failover
         calculation is based on the timestamp that gets transmitted as
         payload in the packets originated by the generator. The Traffic
         Analyzer records the timestamp of the last packet received
         before the failover event and the first packet after the
         failover and derives the time based on the difference between
         these 2 timestamps. Note: The payload could also contain
         sequence numbers for out-of-order packet calculation and
         duplicate packets.
The capabilities required to do this should be identified in the
new section on Tester capabilities.

(it seems I had more than a just few comments...)

Date: Tue, 17 Feb 2009 11:14:10 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Authors of protection meth and term drafts,

It seems that the comment below,
from the Dublin meeting notes:
http://www.ietf.org/proceedings/08jul/minutes/bmwg.html

is not yet incorporated in the drafts:

>Nick: (regarding the Timestamp-Based Method, TBM) out of order impairment
>measurement. Many tools support that, but order can change do to 
>normal parallel
>processing.
>AL: So what we need is the out of order WITHIN a flow. If out of order across
>different flows, then it doesn't affect the customer. So who cares 
>;-) - need to
>consider this in the metrics.
>Jay: We request folks to kindly review the Sub-IP ID
>excluding the formatting issues that the authors are already aware of and are
>working on fixing them.

At 04:19 PM 2/16/2009, Al Morton wrote:
>...-bmwg-protection-meth-04 Authors,
>
>Since there have been some comments during WGLC,
>I hope you can take these late comments into account.
>I only just made time to do a review during a flight.
>
>I've raised some issues below that the test equipment community
>may be interested to comment on.
>
>Al (as a participant)
>
>General:
>--------
>
>This memo is very nearly ready for publication request,
>and closing the loop on a several details will make it so.
>
>There are lots of section references that need +2 (3.1 -> 5.1)
>or other fix-up.
>
>Employing the "standard" Intro/Security paragraphs would help.
>
>This work effort uses the IGP-Dataplane convergence work as a
>foundation, and might also benefit by adding a new section on:
>>    3.2.9 Tester Capabilities
>>    It is RECOMMENDED that the Tester used to execute each test case
>>    have the following capabilities:
>>       1. Ability to insert a timestamp in each data packet's IP
>>          payload.
>>       2. An internal time clock to control timestamping, time
>>          measurements, and time calculations.
>>       3. Ability to distinguish traffic load received on the Preferred
>>          and Next-Best interfaces.
>>       4. Ability to disable or tune specific Layer-2 and Layer-3
>>          protocol functions on any interface(s).
>and some of this is applicable to MPLS-FRR.  A section like this would
>be a good place to expand on the capabilities needed to distinguish
>between impaired and unimpaired packets (per-flow sequence numbers).
>
>I can make a lot of word-smithing suggestions,
>but they would be too tedious to type-up explicitly,
>and then for someone else to locate and change.
>
>One key set of related issues to reconcile in section 7:
>--------------------------------------------------------
>Definition of "maximum forwarding rate (pps)"
>(currently assessed in section 7.5)
>In my opinion,
>  - this is a setup condition needed for all the other
>procedures in section 7, so this section should come first.
>  - there should be a reference to the existing definitions
>of Forwarding Rate and Maximum Forwarding Rate
>http://tools.ietf.org/html/rfc2285#section-3.6.3
>and their methods of measurement
>(or possibly use Throughput instead, and refer to RFC 2544 and
>draft-ietf-bmwg-mpls-forwarding-meth-01.txt for the method).
>
>Also, there is no test to verify that *all* the offered load
>is switched to the backup path and that stability is achieved
>before turning off the load and assessing switching time.
>
>Another key issue, assumption of Reversion performance:
>-------------------------------------------------------
>The draft says (in many places) that Reversion should not produce
>packet loss, but this really depends on the implementation.
>Also, our charter states:
>>Because the demands of a particular technology may vary from deployment
>>to deployment, a specific non-goal of the Working Group is to define
>>acceptance criteria or performance requirements.
>I think we've managed to convey a "desirable outcome" in the past
>without wording that might be misinterpreted as a requirement
>
>
>Other specific comments:
>------------------------
>
>S1 Intro:
>---------
>The 3rd and 4th paragraphs describe correlated failures and
>planned failures, but they aren't particularly useful in the
>remainder of the memo:
>  - The scenarios that test correlated failures aren't identified
>as such.  Sec 5.1 has a few examples (Parent Interface shut,
>Reload/Crash Protected Node), but most are single failures.
>  - The notion of a "Planned Failure" that takes a node or interface
>out-of-service involves other activities to minimize the customer
>impact, such as pre-notification, time of day, and diverting traffic
>before taking action (anything out of service). Since my day-job
>is with a service provider, I think that unplanned failures should
>be the main emphasis here.
>
>S2 Scope:
>---------
>I think that the last paragraph of Sec 1 should be part of the Scope
>section.
>
>The current 1st paragraph mentions "scenarios" twice, but section
>5.1 lists and defines all the failure "events", and events are not
>mentioned.  Maybe this would be better:
>    This document provides detailed test cases along with different
>    topologies and scenarios that should be considered to effectively
>    benchmark MPLS protection mechanisms and failover times. Different
>    failure *events* and scaling considerations are also provided in
>    this document, in addition to reporting formats for the observed
>    results.
>
>The last sentence of the Scope:
>    Benchmarking of unexpected correlated failures is currently out of
>    scope of this document.
>Wouldn't a Node Crash event be an example of an unexpected correlated failure?
>
>
>S3 General ref and sample topology
>----------------------------------
>
>2nd sentence:
>    ... TG & TA represents
>    Traffic Generator & Analyzer respectively.
>suggest change to:
>TG & TA stand for Traffic Generator & Traffic Analyzer respectively,
>and these components comprise the "tester".
>
>Figure 1 is too wide!
>I count 85 characters, including 5 leading spaces. The max is 72.
>See 
>http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-meth-04.txt
>for more stuff like this...
>
>
>Section 5.3 last paragraph reads:
>---------------------------------
>    In addition, this methodology considers the packets in error and
>    duplicate packets that could have been generated during the failover
>    process. In scenarios, where separate measurement of packets in
>    error and duplicate packets is difficult to obtain, these packets
>    should be attributed to lost packets.
>A duplicated packet should *never* be considered the same as a lost packet.
>I think you want to say that some of the methodologies consider lost,
>out-of-order, and duplicate packets to be impaired packets that
>all determine the failure recovery time.
>
>
>Section 5.5, Selection of IGP
>-----------------------------
>Add a sentence/reference:
>     See [IGP-DATAPLANE] for IGP options to consider and record.
>
>
>Section 5.6, Reversion
>----------------------
>    ...In all test cases listed here
>    Reversion should not produce any packet loss, out of order or
>    duplicate packets. Each of the test cases in this methodology
>    document provides a check to confirm that there is no packet loss.
>This is not strictly true, especially since the presence of
>out-of-order packets depends on the difference in delay between
>backup and primary paths.
>
>I suggest:
>    ...In all test cases listed here,
>    Reversion is actuated to observe any packet loss, out of order or
>    duplicate packets. <delete last sentence>
>
>Section 5.7, Traffic Generation
>-------------------------------
>The recommendation against round-robin traffic generation to prefixes
>seems to be inconsistent with section 3.9 of [IGP-DATAPLANE]:
>    ...The destination addresses
>    for the offered load MUST be distributed such that all routes are
>    matched and each route is offered an equal share of the total
>    Offered Load.  This requirement for the Offered Load to be
>    distributed to match all destinations in the route table creates
>    separate flows that are offered to the DUT.  The capability of the
>    Tester to measure packet loss for each individual flow (identified
>    by the destination address matching a route entry) and the scale
>    for the number of individual flows for which it can measure packet
>    loss should be considered when benchmarking Route-Specific
>    Convergence [Po07t].
>This difference needs to be sorted-out.
>
>Section 6
>---------
>Some nouns may be missing:
>    f) PRI is Primary *Node or Path*   ??
>
>    g) BKP denotes Backup Node *or Path*  ??
>
>Section 6.1 (and beyond)
>------------------------
>The Label stack only makes sense if PE and P are
>indicated in (all) the Figure(s).
>
>Section 8, Reporting Format
>---------------------------
>
>Suggest adding:
>     Note the presence of out-of-order and duplicate packets, when measured
>as an item in the Benchmarks list.
>
>
>Also:
>       3. Timestamp Based Method (TBM): This method of failover
>         calculation is based on the timestamp that gets transmitted as
>         payload in the packets originated by the generator. The Traffic
>         Analyzer records the timestamp of the last packet received
>         before the failover event and the first packet after the
>         failover and derives the time based on the difference between
>         these 2 timestamps. Note: The payload could also contain
>         sequence numbers for out-of-order packet calculation and
>         duplicate packets.
>The capabilities required to do this should be identified in the
>new section on Tester capabilities.
>
>(it seems I had more than a just few comments...)

Content-class: urn:content-classes:message
Date: Wed, 25 Feb 2009 19:21:25 +0200
From: "Scott Poretsky" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

By definition in RFC 4689, Out-of-Order Packets is measured for a single
stream.

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Al Morton
Sent: Tuesday, February 17, 2009 11:14 AM
To: [email-masked]
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Authors of protection meth and term drafts,

It seems that the comment below,
from the Dublin meeting notes:
http://www.ietf.org/proceedings/08jul/minutes/bmwg.html

is not yet incorporated in the drafts:

>Nick: (regarding the Timestamp-Based Method, TBM) out of order
impairment
>measurement. Many tools support that, but order can change do to=20
>normal parallel
>processing.
>AL: So what we need is the out of order WITHIN a flow. If out of order
across
>different flows, then it doesn't affect the customer. So who cares=20
>;-) - need to
>consider this in the metrics.
>Jay: We request folks to kindly review the Sub-IP ID
>excluding the formatting issues that the authors are already aware of
and are
>working on fixing them.

Content-class: urn:content-classes:message
Date: Wed, 25 Feb 2009 19:21:31 +0200
From: "Scott Poretsky" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

No.  That is an unplanned uncorrelated failure.  Correlated failures
require more than one Failover Event.

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Al Morton
Sent: Tuesday, February 17, 2009 11:14 AM
To: [email-masked]
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

>The last sentence of the Scope:
>    Benchmarking of unexpected correlated failures is currently out of
>    scope of this document.
>Wouldn't a Node Crash event be an example of an unexpected correlated
failure?

Date: Wed, 25 Feb 2009 14:13:34 -0500
To: "Scott Poretsky" <[email-masked]>, <[email-masked]>
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Scott,

At 12:21 PM 2/25/2009, Scott Poretsky wrote:
>By definition in RFC 4689, Out-of-Order Packets is measured for a single
>stream.

That's a good point, but the definition of a Stream is
very flexible, and it seems that streams need only have
one aspect in common, such as size (because of the "or"):
http://tools.ietf.org/html/rfc4689#section-3.3.2

I can easily construct two streams that are compliant
with the definition where one might experience reordering
in transfer across the DUT, and another stream (with more qualifications)
would not be reordered (we want the second type of stream).

The draft needs to include the point that the streams
evaluated for reordering must be sufficiently qualified
(in terms of destination address/port, source address/port,
DSCP or IP precedence) that reordering does not occur (across the
DUT) outside of failure events or reversion.

This point probably needs to be included in several places
for clarity.  Otherwise the reordering evaluation will be
useless.

thanks,
Al
(as a participant)

>Scott
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Al Morton
>Sent: Tuesday, February 17, 2009 11:14 AM
>To: [email-masked]
>Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
>Authors of protection meth and term drafts,
>
>It seems that the comment below,
>from the Dublin meeting notes:
>http://www.ietf.org/proceedings/08jul/minutes/bmwg.html
>
>is not yet incorporated in the drafts:
>
> >Nick: (regarding the Timestamp-Based Method, TBM) out of order
>impairment
> >measurement. Many tools support that, but order can change do to
> >normal parallel
> >processing.
> >AL: So what we need is the out of order WITHIN a flow. If out of order
>across
> >different flows, then it doesn't affect the customer. So who cares
> >;-) - need to
> >consider this in the metrics.
> >Jay: We request folks to kindly review the Sub-IP ID
> >excluding the formatting issues that the authors are already aware of
>and are
> >working on fixing them.
>
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

Date: Wed, 25 Feb 2009 14:27:07 -0500
To: "Scott Poretsky" <[email-masked]>, <[email-masked]>
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

It seems that a Node Crash would be viewed by adjacent nodes
as more than one of the Link Failure events listed in the definition
of Failure Event, below.  It all depends on how much of the
network you are looking at, and a crash seems to have the same kind of
"dependency on a common physical resource" mentioned in the definition
of correlated events: >1 links depend on the nodes.

>       ...Some of these failure events are listed below.
>
>    Link failure events
>        - Interface Shutdown on PLR side with POS Alarm
>        - Interface Shutdown on remote side with POS Alarm
>        - Interface Shutdown on PLR side with RSVP hello enabled
>        - Interface Shutdown on remote side with RSVP hello enabled
>        - Interface Shutdown on PLR side with BFD
>        - Interface Shutdown on remote side with BFD
>        - Fiber Pull on the PLR side (Both TX & RX or just the TX)
>        - Fiber Pull on the remote side (Both TX & RX or just the RX)
>        - Online insertion and removal (OIR) on PLR side
>        - OIR on remote side
>        - Sub-interface failure (e.g. shutting down of a VLAN)
>        - Parent interface shutdown (an interface bearing multiple sub-
>          interfaces

At 12:21 PM 2/25/2009, Scott Poretsky wrote:
>No.  That is an unplanned uncorrelated failure.  Correlated failures
>require more than one Failover Event.
>
>Scott
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Al Morton
>Sent: Tuesday, February 17, 2009 11:14 AM
>To: [email-masked]
>Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
> >The last sentence of the Scope:
> >    Benchmarking of unexpected correlated failures is currently out of
> >    scope of this document.
> >Wouldn't a Node Crash event be an example of an unexpected correlated
>failure?
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

Content-class: urn:content-classes:message
Date: Wed, 25 Feb 2009 21:23:14 +0200
From: "Scott Poretsky" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Thanks Al,

We will add that note to section 5.7 offered Load to narrow the
definition of "stream" for context of this work.

Scott=20

-----Original Message-----
From: Al Morton [mailto:[email-masked]]=20
Sent: Wednesday, February 25, 2009 2:14 PM
To: Scott Poretsky; [email-masked]
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Scott,

At 12:21 PM 2/25/2009, Scott Poretsky wrote:
>By definition in RFC 4689, Out-of-Order Packets is measured for a
single
>stream.

That's a good point, but the definition of a Stream is
very flexible, and it seems that streams need only have
one aspect in common, such as size (because of the "or"):
http://tools.ietf.org/html/rfc4689#section-3.3.2

I can easily construct two streams that are compliant
with the definition where one might experience reordering
in transfer across the DUT, and another stream (with more
qualifications)
would not be reordered (we want the second type of stream).

The draft needs to include the point that the streams
evaluated for reordering must be sufficiently qualified
(in terms of destination address/port, source address/port,
DSCP or IP precedence) that reordering does not occur (across the
DUT) outside of failure events or reversion.

This point probably needs to be included in several places
for clarity.  Otherwise the reordering evaluation will be
useless.

thanks,
Al
(as a participant)

>Scott
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Al Morton
>Sent: Tuesday, February 17, 2009 11:14 AM
>To: [email-masked]
>Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
>Authors of protection meth and term drafts,
>
>It seems that the comment below,
>from the Dublin meeting notes:
>http://www.ietf.org/proceedings/08jul/minutes/bmwg.html
>
>is not yet incorporated in the drafts:
>
> >Nick: (regarding the Timestamp-Based Method, TBM) out of order
>impairment
> >measurement. Many tools support that, but order can change do to
> >normal parallel
> >processing.
> >AL: So what we need is the out of order WITHIN a flow. If out of
order
>across
> >different flows, then it doesn't affect the customer. So who cares
> >;-) - need to
> >consider this in the metrics.
> >Jay: We request folks to kindly review the Sub-IP ID
> >excluding the formatting issues that the authors are already aware of
>and are
> >working on fixing them.
>
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

Sensitivity: Normal
Importance: Normal
To: "Al Morton" <[email-masked]>,[email-masked]
From: "Scott Poretsky" <[email-masked]>
Date: Wed, 25 Feb 2009 20:38:55 +0000
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
Reply-To: [email-masked]

Date: Wed, 25 Feb 2009 12:41:38 -0800
From: David Newman <[email-masked]>
Organization: Network Test Inc.
To: [email-masked]
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

On 2/25/09 12:38 PM, Scott Poretsky wrote:
> Yes, but in the BMWG we focus on a single DUT

Not necessarily. Many bmwg documents use the term DUT/SUT, where SUT may
be comprised of multiple systems (just not on a production network).

dn

Content-class: urn:content-classes:message
Date: Wed, 25 Feb 2009 22:46:30 +0200
From: "Scott Poretsky" <[email-masked]>
To: "David Newman" <[email-masked]>, <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

David,

Are you making a general statement or a point relevant to this thread?

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
David Newman
Sent: Wednesday, February 25, 2009 3:42 PM
To: [email-masked]
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

On 2/25/09 12:38 PM, Scott Poretsky wrote:
> Yes, but in the BMWG we focus on a single DUT

Not necessarily. Many bmwg documents use the term DUT/SUT, where SUT may
be comprised of multiple systems (just not on a production network).

dn

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

Date: Wed, 25 Feb 2009 16:06:02 -0500
To: [email-masked], [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Scott,

The meth draft has lots of SUT topologies --
there may be one device you are focusing on (the
point of local repair, PLR), but the system's
performance is measured from TG to TA:

            --------      --------      --------      --------
           |  R1    |    |  R2    |    |   R3   |    |   R4   |
        TG-|  HE    |    |  MID   |PRI |  MID   |PRI |  TE    |-TA
           |        |----|  PLR   |----|        |----|        |
            --------      --------      --------      --------
                           |                            |
                        BKP|          --------          |
                           |         |   R6   |         |
                            ---------|  BKP   |---------
                                     |  MID   |
                                      --------

                              Figure 7.

Also, the node that crashes is never the DUT,
so the system sees correlated failures.

Al

At 03:38 PM 2/25/2009, Scott Poretsky wrote:
>Yes, but in the BMWG we focus on a single DUT, so correlated events 
>are from the point of view of the DUT.
>
>Scott
>Sent via BlackBerry.
>
>-----Original Message-----
>From: "Al Morton" <[email-masked]>
>
>Date: Wed, 25 Feb 2009 14:27:07
>To: Scott Poretsky<[email-masked]>; <[email-masked]>
>Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
>
>It seems that a Node Crash would be viewed by adjacent nodes
>as more than one of the Link Failure events listed in the definition
>of Failure Event, below.  It all depends on how much of the
>network you are looking at, and a crash seems to have the same kind of
>"dependency on a common physical resource" mentioned in the definition
>of correlated events: >1 links depend on the nodes.
>
> >       ...Some of these failure events are listed below.
> >
> >    Link failure events
> >        - Interface Shutdown on PLR side with POS Alarm
> >        - Interface Shutdown on remote side with POS Alarm
> >        - Interface Shutdown on PLR side with RSVP hello enabled
> >        - Interface Shutdown on remote side with RSVP hello enabled
> >        - Interface Shutdown on PLR side with BFD
> >        - Interface Shutdown on remote side with BFD
> >        - Fiber Pull on the PLR side (Both TX & RX or just the TX)
> >        - Fiber Pull on the remote side (Both TX & RX or just the RX)
> >        - Online insertion and removal (OIR) on PLR side
> >        - OIR on remote side
> >        - Sub-interface failure (e.g. shutting down of a VLAN)
> >        - Parent interface shutdown (an interface bearing multiple sub-
> >          interfaces
>
>
>At 12:21 PM 2/25/2009, Scott Poretsky wrote:
> >No.  That is an unplanned uncorrelated failure.  Correlated failures
> >require more than one Failover Event.
> >
> >Scott
> >
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]] On Behalf Of
> >Al Morton
> >Sent: Tuesday, February 17, 2009 11:14 AM
> >To: [email-masked]
> >Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
> >
> > >The last sentence of the Scope:
> > >    Benchmarking of unexpected correlated failures is currently out of
> > >    scope of this document.
> > >Wouldn't a Node Crash event be an example of an unexpected correlated
> >failure?
> >
> >_______________________________________________
> >bmwg mailing list
> >[email-masked]
> >https://www.ietf.org/mailman/listinfo/bmwg
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

Sensitivity: Normal
Importance: Normal
To: "Al Morton" <[email-masked]>,[email-masked]
From: "Scott Poretsky" <[email-masked]>
Date: Wed, 25 Feb 2009 21:40:28 +0000
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
Reply-To: [email-masked]

Date: Wed, 25 Feb 2009 16:59:04 -0500
To: [email-masked], [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

OK, that's a welcome clarification.

There was a line of thought I was pursuing in this thread
(whether Node Crashes are correlated failures or not),
and it sounds like we're closing in on this as the conclusion:

Although Node Crashes may cause the other devices in the network
to experience distinctly separate failure events, the DUT only
observes the crash as a single failure event on its connecting link.

re-booting,
Al

At 04:40 PM 2/25/2009, Scott Poretsky wrote:
>In the test topologies for this methodology document there is only 
>one DUT.  The DUT is always a PLR.  The benchmarks are specific to 
>the DUT.   The Tester may emulate all of the other nodes.  The test 
>topology may have additional devices as the other nodes if the 
>Tester does not emulate those.  This is the reason the baseline 
>throughput test cases are so important to perform first.
>
>This note will be added to the Introduction and Scope.
>
>Scott
>Sent via BlackBerry.
>
>-----Original Message-----
>From: "Al Morton" <[email-masked]>
>
>Date: Wed, 25 Feb 2009 16:06:02
>To: <[email-masked]>; <[email-masked]>
>Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
>
>Scott,
>
>The meth draft has lots of SUT topologies --
>there may be one device you are focusing on (the
>point of local repair, PLR), but the system's
>performance is measured from TG to TA:
>
>             --------      --------      --------      --------
>            |  R1    |    |  R2    |    |   R3   |    |   R4   |
>         TG-|  HE    |    |  MID   |PRI |  MID   |PRI |  TE    |-TA
>            |        |----|  PLR   |----|        |----|        |
>             --------      --------      --------      --------
>                            |                            |
>                         BKP|          --------          |
>                            |         |   R6   |         |
>                             ---------|  BKP   |---------
>                                      |  MID   |
>                                       --------
>
>                               Figure 7.
>
>Also, the node that crashes is never the DUT,
>so the system sees correlated failures.
>
>Al
>
>At 03:38 PM 2/25/2009, Scott Poretsky wrote:
> >Yes, but in the BMWG we focus on a single DUT, so correlated events
> >are from the point of view of the DUT.
> >
> >Scott
> >Sent via BlackBerry.
> >
> >-----Original Message-----
> >From: "Al Morton" <[email-masked]>
> >
> >Date: Wed, 25 Feb 2009 14:27:07
> >To: Scott Poretsky<[email-masked]>; <[email-masked]>
> >Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
> >
> >
> >It seems that a Node Crash would be viewed by adjacent nodes
> >as more than one of the Link Failure events listed in the definition
> >of Failure Event, below.  It all depends on how much of the
> >network you are looking at, and a crash seems to have the same kind of
> >"dependency on a common physical resource" mentioned in the definition
> >of correlated events: >1 links depend on the nodes.
> >
> > >       ...Some of these failure events are listed below.
> > >
> > >    Link failure events
> > >        - Interface Shutdown on PLR side with POS Alarm
> > >        - Interface Shutdown on remote side with POS Alarm
> > >        - Interface Shutdown on PLR side with RSVP hello enabled
> > >        - Interface Shutdown on remote side with RSVP hello enabled
> > >        - Interface Shutdown on PLR side with BFD
> > >        - Interface Shutdown on remote side with BFD
> > >        - Fiber Pull on the PLR side (Both TX & RX or just the TX)
> > >        - Fiber Pull on the remote side (Both TX & RX or just the RX)
> > >        - Online insertion and removal (OIR) on PLR side
> > >        - OIR on remote side
> > >        - Sub-interface failure (e.g. shutting down of a VLAN)
> > >        - Parent interface shutdown (an interface bearing multiple sub-
> > >          interfaces
> >
> >
> >At 12:21 PM 2/25/2009, Scott Poretsky wrote:
> > >No.  That is an unplanned uncorrelated failure.  Correlated failures
> > >require more than one Failover Event.
> > >
> > >Scott
> > >
> > >-----Original Message-----
> > >From: [email-masked] [mailto:[email-masked]] On Behalf Of
> > >Al Morton
> > >Sent: Tuesday, February 17, 2009 11:14 AM
> > >To: [email-masked]
> > >Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
> > >
> > > >The last sentence of the Scope:
> > > >    Benchmarking of unexpected correlated failures is currently out of
> > > >    scope of this document.
> > > >Wouldn't a Node Crash event be an example of an unexpected correlated
> > >failure?
> > >
> > >_______________________________________________
> > >bmwg mailing list
> > >[email-masked]
> > >https://www.ietf.org/mailman/listinfo/bmwg
> >
> >_______________________________________________
> >bmwg mailing list
> >[email-masked]
> >https://www.ietf.org/mailman/listinfo/bmwg

Date: Thu, 26 Feb 2009 14:41:18 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [bmwg] Fwd: Initial Version I-D Submission Deadline Extended to March 4, 2009

FYI:

>The IESG has extended the deadline for initial version (00)
>submissions of Internet Drafts by 2 days (48 hours). The new deadline
>is March 4, 2009 at 1700 Pacific (March 5, 2009 at  0100 UTC / GMT)
>and the extension is for IETF 74 only.  The deadline has been extended
>due to the copyright legend text alternative being recently finalized,
>approved and implemented.
>
>Please note that the date for updated I-D versions has NOT been
>extended, and is still March 9, 2009.

Date: Sat, 28 Feb 2009 10:58:12 -0800
From: Jerry Perser <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt

Scott,

My view is both.

One, the BWMG focus is on a SUT/DUT.  If you can draw a box around what 
you are testing and some of the network is outside the box, then it 
falls under the BMWG.  If you can't draw a box around what you are 
testing or none of the network is outside the box, then is falls under 
the IPPM.  My notion of what a DUT is was destroyed by a company called 
Avici Systems.  Those guys can take 14 routers, each in their own rack, 
and connect them to look like a single DUT.  From the networks point of 
view, the routers look like a single DUT.  If your draft can not support 
SUT, you should be clear about it.  I assumed that the draft supported 
both SUT and DUT.  Please clarify if we need another draft for SUT.

Two, does Al's comment about a Node Crash also apply to SUT, but not a DUT?

Jerry Perser

Scott Poretsky wrote:
> David,
>
> Are you making a general statement or a point relevant to this thread?
>
> Scott
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> David Newman
> Sent: Wednesday, February 25, 2009 3:42 PM
> To: [email-masked]
> Subject: Re: [bmwg] Comments on draft-ietf-bmwg-protection-meth-04.txt
>
> On 2/25/09 12:38 PM, Scott Poretsky wrote:
>   
>> Yes, but in the BMWG we focus on a single DUT
>>     
>
> Not necessarily. Many bmwg documents use the term DUT/SUT, where SUT may
> be comprised of multiple systems (just not on a production network).
>
> dn
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
>
>
>   

-- 
Jerry Perser