Subject: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 19 Jul 2010 07:53:01 -0500

Timmons et all, 

Thank you very much for  your feedback, the  original version of the
draft completely ignored the traffic generator behavior, since at least
the ones I am familiar with already consider the PHY status @ providing
stats, so we thought that there was no any  clarification on this
subject ....

We received previous comments from J Dawson (pasted @ the end of the my
reply) requesting us to address the PHY up / down /  packets blaster
concern  he had and that's where we came out with this short paragraph,
since three of the most popular tools in the market offer this option
while configuring the supported media 

Here is our original exchange @ this subject , if you think that is
applicable or if further updates to the doc are needed please let us
know 

Kind  Regards 

The Authors ....

_____ From: DAWSON, JOHN E (JOHN) [mailto:[email-masked]] 
Sent: Friday, April 16, 2010 1:49 PM
To: Carlos Pignataro (cpignata)
Cc: Fernando Calabria (fcalabri); Rajiv Asati (rajiva);
[email-masked]
Subject: RE: draft-asati-bmwg-reset-03

My biggest concern can be alleviated so long as you create an explicit
test requirement that states that offered load can not stop when link
status goes down.  Your packet blasters must continue sending into dead
links.  Without this stated explicitly, your test methodology will be
severly compromised.

I think we thoughts about this already (Transmit Ignore Link Status)
support from modern traffic generators, we can for sure add one
paragraph making this a requirement,

A limitation of this process is still that you will not be able to
characterize how long link is up, yet no data is being forwarded.  This
can be important for redundant pathway devices, but since you never
claim to investigate this characteristic it can just be a known
limitation.

Correct, ew just care about overall convergence not link / up / down
characteristics, 

Fernando 

John D.

-----Original Message-----
From: Carlos Pignataro [mailto:[email-masked]]
Sent: Thursday, April 15, 2010 7:50 AM
To: DAWSON, JOHN E (JOHN)
Cc: Fernando Calabria (fcalabri); Rajiv Asati (rajiva);
[email-masked]
Subject: Re: draft-asati-bmwg-reset-03

Hi John,

Have you been able to parse Fernando's responses to your comments? We
are queuing up a new revision and would like to ensure we integrate your
comments.

Thanks,

-- Carlos.

On 3/19/2010 11:08 PM, DAWSON, JOHN E (JOHN) wrote:
> Yes, but I haven't had time to go through them yet.
>
> John D.
>
> -----Original Message-----
> From: Carlos Pignataro [mailto:[email-masked]]
> Sent: Friday, March 19, 2010 8:06 PM
> To: Fernando Calabria (fcalabri)
> Cc: DAWSON, JOHN E (JOHN); Rajiv Asati (rajiva);
[email-masked]
> Subject: Re: draft-asati-bmwg-reset-03
>
> Hello John,
>
> Have you been able to see our responses from Fernando?
> Thanks again for the feedback, it is much appreciated.
>
> -- Carlos.
>
> On 3/13/2010 9:56 AM, Fernando Calabria (fcalabri) wrote:
>> Hello John ,
>>
>> First of all, many thanks for taking time to review the document and
get
>> back to us with such intelligent and appropriate comments ...
>>
>> I wanted to follow up with you to see if you are ok with some of my
>> replies and what we would need to modify from the current doc to
>> incorporate your comments that may have not being addressed already ,
>>
>> Again from all of the authors thank you  very much for helping us
with
>> the document
>>
>> Kind Regards
>>
>> Fernando Calabria
>>
>>
>> -----Original Message-----
>> From: Fernando Calabria (fcalabri)
>> Sent: Friday, March 12, 2010 11:04 AM
>> To: Carlos Pignataro (cpignata); John Dawson
>> Cc: Rajiv Asati (rajiva); [email-masked]
>> Subject: RE: draft-asati-bmwg-reset-03
>>
>> These are my replies , (inline)
>>
>>
>>
>>
>> -----Original Message-----
>> From: Carlos Pignataro (cpignata)
>> Sent: Thursday, March 11, 2010 10:04 PM
>> To: John Dawson
>> Cc: Rajiv Asati (rajiva); Fernando Calabria (fcalabri);
>> [email-masked]
>> Subject: Re: draft-asati-bmwg-reset-03
>>
>> Hi John,
>>
>> Many thanks for your very comprehensive and detailed feedback. It is
>> much appreciated.
>>
>> You caught us in a bit of a combined interrupt-driven time, and we
>> really want to analyze and respond in detail, so we'll keep this
email
>> in the input hold queue for a little while. I hope this is OK.
>>
>> It really means that we need some time to respond, but I wanted to
>> Syn+Ack.
>>
>> We will add a bullet to the bmwg timeslot preso saying that we
received
>> this feedback and is pending.
>>
>> Thank you for understanding,
>>
>> -- Carlos.
>>
>> On 3/9/2010 7:44 PM, John Dawson wrote:
>>> Hi Folks,
>>>
>>> I was reviewing your Device Reset Characterization draft
>>> (_http://tools.ietf.org/html/draft-asati-bmwg-reset-03_) and had
many
>>> questions as to how you infer reset characteristics from RFC 2544
>> frame
>>> loss results.
>>>
>>> My first concern is how you intend to measure recovery time.  Based
>> upon
>>> RFC 2544 you have to actually capture packets and look at the
>>> time-stamps for the packets on both sides of the reset-event to get
>> the
>>> recovery time.  How do you accomplish this capture process when you
>> are
>>> pushing 14 million packets per second across a 10G link?
>>>
>>>>>>> FER
>> we are measuring the recovery time / outage by simply dividing the
total
>> number of packets lost / by the offered Rate
>>
>> So if we are sending 10K PPS and lost 1564 packets (as accounted for
the
>> test tool) (and any test tool can measure this for any or multiple
given
>> traffic streams) the duration of the outage will be : 1564/10,000 =
156
>> msec
>>
>> Doing the match while sending LR for an 768 interface will not be a
>> problem either but , this is not a benchmarking / throughput test so
no
>> need to send line Rate / non-dropping rate  to characterize a reset
...
>>
>>
>>
>>
>>
>>
>>> If you don't have a solid packet capture technique, then how can you
>>> infer any real meaning from a frame loss characteristic?  A primary
>>> concern is that there are two different characteristics that can
come
>>> into play relative to most traffic generators:
>>>
>>>         1)  Packet blasting continues while PHY is down.
>>>         2)  Packet blasting stops while PHY is down.
>>>
>>> Since RFC 2544 was not designed to actively being playing with PHY
>>> characteristics, this was not a primary concern for that
methodology.
>>> However, if your traffic generator stops sending frames when the PHY
>> is
>>> in link-state down -- because you are forcing resets -- then frame
>> loss
>>> will be suspended while the device under test is going through the
>> bulk
>>> of its reset processes.  Although you likely will have frame loss
>> after
>>> the PHY goes back into an active link-state -- due to spanning-tree
>>> learning, or while address and/or routing tables are in a state of
>> flux
>>> --  that frame loss may or may not be significant relative to the
>>> overall reset period.
>>>>> for benchmark / toughput test we just deal with static routing and
in
>> the worst case of Ethernet , an ARP can be a problem but most of the
>> time we set it in the traffic generator in a static fashion)
>> For absolutely all of the modern packet generators you can configure
the
>> option to stop sending traffic while PHY is down or continue to do so
>> (for the sake of accounting for lost frames) (look at for example the
>> option in N2x / IXIA ...
>>
>> Transmit Mode Select :
>>
>>
>>  Advanced Scheduler If selected, the traffic will be transmitted as
>> interleaved streams.
>>
>>
>>  Transmit Ignore Link Status If checked, the transmit operation
ignores
>> whether the link status of the port is up or down.   ************
>> ------------------------------
>>
>>
>>
>>
>>> This suggests that you are going to have much more stringent
>>> requirements relative to reporting intended load while actually
>>> measuring offered load.  You may or may not need to adjust your loss
>>> metrics to include any difference between intended and offered load.
>>> RFC 2544 has been really abused in this regard over the years and I
>>> don't believe there is any hard and fast process in place relative
to
>>> how one measures offered load -- which becomes absolutely critical
in
>>> link-state down scenarios.
>>>
>>> I believe that the above considerations make your overall
measurement
>>> process not so precise and subject to variations in both
>> implementation
>>> and interpretation.
>>>>>> Fer
>> As always , this is a best effort approach that with the case of a
>> benchmarking RFCs may not be perfect but good enough to do a precise
>> characterization
>> I don't see  1 or 2 or even 10 msecs making a  difference while
>> characterizing a device reset , remember we may be talking of an
event
>> that may take minutes , or so , so even if it's a minute long outage
to
>> reset an RP ,  100 msecs will be practically a .16% of sampling
error
>> ,,
>>
>>> Additionally, if you are using wire-speed data flows to characterize
>>> your reset interval, how do you account for the fact that you could
>> have
>>> flooding between streams as your device re-stabilizes.  Any flooding
>>> would lead to an over-subscription of a wire-speed flow and hence
lead
>>> to loss -- which again would affect the accuracy of your
measurement.
>>> Do you consider periods where data is flooding -- due to address
table
>>> instabilities or certain modules not coming on-line -- to be within
>> the
>>> measured reset period or outside the measured reset period?
Although
>>> this might not be a big deal for Layer-3 DUTs, it is going to come
>> into
>>> play for a Layer-2 switch.
>>
>> We may not necessary have to send Line Rate / wire speed for this
test,
>> moments of instabilities are accounted cause if you are losing
frames,
>> those are accounted by the test tool as part of the total number
>> But again if you are doing a pure troughput / benchmarking test you
want
>> to keep your control plane as minimum as possible (not to run any IGP
or
>> even SPT if it's a l2 device) , so you will deal with a static mac
entry
>> and or a static route cause you don't; want device specific
mechanisms
>> like pak_priority and so on to affect your troughput measurement, in
>> other words OSPF hellos do not have to consume any BW and have to
>> compete with your test streams
>>
>>
>>
>>
>>> I personally think that you have made a number of over-simplifying
>>> assumptions, and incorrectly equated the use of wire-speed flows
with
>> a
>>> more accurate measurement interval.  If you are forcing a power
>>> interruption on the order of 15 seconds, and you might expect a
device
>>> power-up interval in the 30 to 300 second range, then why mandate a
>>> packet rate that infers a measurement resolution in the
>> sub-microsecond
>>> range.
>> Again I don't see where we have to deploy wire-speed flows, whatever
>> rate that will give you a desired / good enough granularity will be
>> perfect,
>>
>> So if you care to go up to a 1 seconds granularity why not send 1
packet
>> per second :)
>>
>>
>>
>>
>>
>>> If your measurement is within 1% isn't that good enough?  Why would
a
>>> packet rate of 10 packets per second not be close enough accuracy
for
>>> this type of measurement?  A packet rate of 10 pps with a reset
>> interval
>>> of 20 seconds, would yield a 1% accuracy.
>>>>> I personally think 1 % is more than good enough , see my orevous
>> example, we may even get better without having to be close to sany
>> modern media wire speed ,,,
>>> Since most traffic generators have the ability to send packets with
an
>>> embedded time-stamp and capture packets along with a receive
>>> time-stamp.  You are much better off to start-up a number of data
flow
>>> at 10 pps, and do a time-stamp based capture over a 300 to 400
second
>>> time period.  Then wade through the 3,000+ packets that you sent
>> between
>>> each of the port pairs and create the following characterization.
>>>
>>>             a)  Time periods where no frames were seen by the
traffic
>>>             generator;  This would reflect both link-down intervals
>> and
>>>             no forwarding intervals.
>>>
>>>             b)  Time periods where frames were flooding between
ports
>> --
>>>             as reflected by the fact that you captured a stream on a
>>>             port that was not desired.
>>>
>>>             c)  Time periods where frames were received with no loss
>> --
>>>             as reflected by the fact that you received the packet on
>> the
>>>             intended port.
>>>
>>>             d)  Was there any bouncing in the forwarding
>>>             characterization; did data start to flow and then stops
>> for
>>>             a period, and then starts up again.
>>>
>>>             e)  The time interval between link going active and the
>>>             onset of the data flows; This one is a little trickier
as
>> it
>>>             does require some assumptions to be made, but it is
>>>             important for systems that employ fail-over
technologies.
>>>             You don't want your link going active to early, and
having
>> a
>>>             redundant path algorithm start using that path, if the
>>>             forwarding processes are not active along that path.
>>>>> yes I don't see anything wrong with this approach
>>
>>> If you have 16 ports (8 bi-directional or meshed streams for that
>>> matter), then how long would it take a typical test system to look
at
>>> 60,000 or so packet time-stamps?
>>>>> not a lot, really , this can be processed by modern test tools in
a
>> few seconds ...
>>
>>
>>> Although this alternative process does not scale so well to 100
ports,
>>> as the time required to collect all the time-stamps does start to
>> become
>>> excessive, I also think you would be hard pressed taking a 100 port
>>> frame-loss test result and making a blank statement about whether
the
>>> reset period was uniformly distributed and easily averaged across
>> every
>>> port.
>>>>> I don't  see a requirement to have to deploy 100+ ports to
>> characterize an outage ....
>>
>>
>>
>>> So what is the overall thought process here?  Is this an exercise in
>>> attempting to find another use of 2544 frame-loss data (with
>>> questionable correlative value)?  Or is your intention to accurately
>>> characterize reset characteristics?
>>>>>  the intention is to provide a set of methodologies to be
>> implemented in order to characterize a device reset, since different
>> platforms have different capabilities we want to level the play filed
>> with common methodologies that can be applied regardless platform
>> specific implementations
>> In other words, not all resets are created equally  when it comes to
>> different platforms / vendors
>>> I don't mean to throw cold water over your efforts, but I do think
>> that
>>> you are heading down a questionable path.
>>>
>>> Thank you in advance for your consideration in this matter.
>>>>>>> Please all your comment are very valuable and "help" us to make
>> this drat as good as it can / should be
>>
>>
>>> John Dawson
>>> Avaya / Automation Architect for Stackable Data Products
>>> Email: [email-masked]
>>>
>>>
>>> The author works for Avaya, Inc. ("Avaya") who is solely responsible
>> for
>>> this email and its contents. All enquiries regarding this email
should
>>> be addressed to Avaya, Inc. Nortel has provided the use of the
Nortel
>>> email messaging system to Avaya in connection with this email solely
>> for
>>> the purpose of connectivity and Nortel Networks Inc. has no
liability
>>> for the email or its contents. Avaya's web site is
>> _http://www.avaya.com_
>>
>

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Timmons C. Player
Sent: Friday, July 16, 2010 4:01 PM
To: [email-masked]
Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-reset-00

On 7/9/2010 8:39 AM, Al Morton wrote:
> Another reminder, there's still time to comment...
>

Howdy, all!  This comment applies to both versions, but I'm using the 
section numbers from 01.

This draft repeats the following comment several times (In section 
4.1.1.1, 4.1.1.2, 4.1.2, 4.2.1, 4.2.2, and 4.3.1):

      Modern traffic generators support the feature of transmitting
      packets while ignoring the link status. The operator / tester MUST
      ensure that this feature is enabled

I'm going to be snarky and say that really modern traffic generators can

measure this interval directly without counting phantom test frames.

The real point here is that I don't think any methodology needs to 
specify explicitly how the test tool should make a measurement.  For 
time based measurements especially, it really only needs to specify the 
goal posts for starting and stopping.  As long as the test equipment 
complies with the goal posts, it can make the measurement in all sorts 
of creative ways, including methods not forseen by the authors of the
draft.

If we take this document as written, then only test tools that claim 
they transmitted a frame when they really didn't can perform this 
methodology, and that behavior strikes me as broken (though I do 
understand the intention behind it).

Timmons

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Timmons C. Player" <[email-masked]>
Date: Mon, 19 Jul 2010 10:04:06 -0400

> applicable or if further updates to the doc are needed please let us
Wow, that's a lot to read.

...

Ok, I do think an update is necessary, because I think there are better 
ways to measure service outages than by counting lost/un-transmitted 
frames.  The discussion you added to this thread seems to point that out 
as well.

At a high level, I don't think methodology authors really need to 
concern themselves with HOW a device makes a measurement.  We really 
should just state WHAT we want to measure.  I mean, I don't see any 
drafts which explicitly tell me HOW my test hardware should measure 
latency, but I do see explicitly defined goalposts that tell me when the 
latency clock should start and stop.

But back to the draft...  Suppose I have a device that 
generates/analyzes test frames and records the rx timestamp for every 
received test frame.  I then say the service outage duration is:

(first non-flooded rx timestamp after service outage) - (last rx 
timestamp before outage)

This measurement is probably more accurate than the (missing/rate) 
measurement in some cases, because it is explicitly ignoring flooded 
frames, but it wouldn't be considered draft-compliant.

However, this technical discussion is all beside the point.  I would 
like there to see a change to this draft.  I would like there to be 
explicit start/stop goal posts for the reset measurement, so that a 
person performing this test knows when an outage is defined to start and 
when it is defined to stop.

The (missing/rate) methodology is fine for an approximation, but better 
methods of measurement shouldn't be excluded from the draft entirely.

Thanks.

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 19 Jul 2010 09:36:11 -0500

Hi there , 

We believe that the described methodology / approximation is quite
enough since  it will provide well under sub-second figures  and also is
inline with the "loss-derived convergence time" used in the
IGP-Dataplane drafts, but a reference to a start / stop action like you
suggest makes sense as well, 

Would you agree with the following  addition ?? 

Currently we have:
---
     Finally, the characterization is completed by measuring the
     complete frame loss and calculating the recovery time from the
     moment the active RP is hardware-reset until traffic is re-
     established, by using the following equation:
     Recovery_time = Packet_loss (packets)/Offered_rate (packets per
     second).
---

New: 
---
     Finally, the characterization is completed by measuring the
     complete frame loss and measuring or calculating the recovery time
from the
     moment the active RP is hardware-reset (start) until traffic is re-
     established (stop), by using the following equation:
     Recovery_time = Packet_loss (packets)/Offered_rate (packets per
     second) or other method measuring the interval between start and
stop times.
---

Rgds 

Fernando 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 10:04 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

...

as well.

Thanks.

Timmons

---

From: David Newman <[email-masked]>
Date: Mon, 19 Jul 2010 07:50:37 -0700

Two nits with this response:

1. Wrt this passage:

"We believe that the described methodology / approximation is quite
enough since it will provide well under sub-second figures"

For reasons similar to Timmons' concerns about bmwg methodologies
getting into test instrument implementation details, I am wary of using
a specific time resolution as an acceptance criterion.

I hope this methodology is wildly successful and that we use it at least
for as long as we have used RFC 1242, which was published in 1991.

Network speeds have increased >1000x since then; in that light, we
probably should not be making guesses about how short an interval will
be "good enough" on future networks.

2. Just tacking on the words "or other method" does not address Timmons'
concern. Now, instead of the problem of a bmwg methodology specifying
one test instrument implementation method, it suggests two or more...

dn

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 19 Jul 2010 10:04:33 -0500

Well, if we talk to the specific events @ this draft , well under
sub-second is more than enough now  if you mix everything and then
compare this with  a 50 msec fast reroute implementation for example
then you may have a point, but yet again it is beyond the scope of the
current work .... if you have a generic point to make with the BMWG it
not only applies to this work but to many other drafts that uses this
methodology (we already mentioned this)

We are no adding just the words "or other method" if you read the
changes we are also making reference to the requested start / stop
comment / concern

I would like to hear directly from  Timmons if this addresses his
concerns or not , if you have another one  or want to piggy back on his
please do so but do not reply saying that his concern is not address ...
I think he can do it for himself ... 

Rgds 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
David Newman
Sent: Monday, July 19, 2010 10:51 AM
To: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

1. Wrt this passage:

dn

---

From: "Timmons C. Player" <[email-masked]>
Date: Mon, 19 Jul 2010 11:53:35 -0400

> suggest makes sense as well,
Great!

>       second) or other method measuring the interval between start and

Not quite...

I was trying to point out that you don't need frame loss to measure 
recovery time, so you only need say...

---
Finally, the characterization is completed by measuring the recovery 
time from the moment the active RP is hardware-reset (start) until 
traffic is re-established (stop).  This MAY be accomplished by measuring 
frame loss and using the following equation:

Recovery_time = Packet_loss (packets)/Offered_rate (packets per second)
---

I would assume that explicit definitions for start/stop are forthcoming, 
as I currently don't know how those are defined.

Also, I'd like to recommend that the following text:

---
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
---

be stricken all 6 times that it occurs, or at least changed to state:

---
If frame loss is used to approximate recovery time, then the traffic 
generator MUST ignore link status when transmitting test frames.
---

or something to that effect...

Thanks.

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 19 Jul 2010 11:00:50 -0500

Great , thanks a lot, let us work out the detailed text  and we will
distributed it shortly 

Rgds 

Fernando 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 11:54 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Great!

Not quite...

---

---

---

Thanks.

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 19 Jul 2010 15:43:26 -0500

Timmons, David, Al et all, 

what do you think in regards to these modifications ?? 

We wanted to also  detail the start / stop definitions as well ... 

>       Recovery_time = Packet_loss (packets)/Offered_rate (packets per

To: .....

Third, perform the Route Processor (RP) hardware reset at this point
(start time) . This entails for example physically removing the RP to
later re-insert it, or triggering a hardware reset by other means (e.g.,
command line interface, physical switch, etc.)....

Finally, the characterization is completed by providing  the  frame loss
counters (reported by the Test Apparatus)  and recovery time from the
moment the RP is re-initialized or reinserted and all traffic recovered
(stop time) . The tester / operator MAY use the following formula for
reporting and characterization proposes: 

Recovery_time = Packet_loss (packets)/Offered_rate (packets per second).

...........

We also change the reporting format the Time definition from:

Seconds, with sufficient resolution to convey meaningful info

to: 

Milliseconds, with sufficient resolution to convey meaningful info

Please let us know your thoughts ,,,, since we believe that with this we
leave the door open for the tester to reference the start / stop timers
if he can provide a 1/1000s of a sec precision .... which is what I
believe the root of this discussion .... 

Rgds 

Fernando 

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: Monday, July 19, 2010 12:01 PM
To: 'Timmons C. Player'
Cc: [email-masked]
Subject: RE: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Rgds 

Fernando 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 11:54 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Great!

Not quite...

---

---

---

Thanks.

Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Tue, 20 Jul 2010 09:44:21 -0400

> Recovery_time = Packet_loss (packets)/Offered_rate (packets per second).

Well, I think this is still assuming the only method for doing this test 
is by tracking frame loss counters...  Should I care about frame loss 
counters if I'm not using them?  This text says I should, but I don't 
know why...

> believe the root of this discussion ....
No, accuracy isn't the root of my point at all.  Of course, I'd like my 
test equipment to be as accurate as possible, but the root cause was 
having a methodology that tells test equipment manufacturers how to make 
their hardware instead of telling them what they ought to measure (which 
is why I was asking for goal posts).

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Tue, 20 Jul 2010 09:01:07 -0500

Timmons, 

It's a MAY, not a MUST anymore so we are not saying that this is the
only available one ,.. 

Fer, 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Tuesday, July 20, 2010 9:44 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

second).

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Tue, 20 Jul 2010 09:10:46 -0500

Inline: 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Tuesday, July 20, 2010 9:44 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

second).

>>>> no, since it's a MAY not a MUST like before 

>>> I am not referring about accuracy but the possibility of use of the
start / stop timers if the methodology implemented can provide a
granularity of msecs
>>> see that we are sending the smallest possible frame at NDR ,,,

Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Tue, 20 Jul 2010 10:17:10 -0400

> It's a MAY, not a MUST anymore so we are not saying that this is the

I was referring to this particular part of the text:
---
Finally, the characterization is completed by providing  the  frame loss
counters (reported by the Test Apparatus)  and recovery time from the
moment the RP is re-initialized or reinserted and all traffic recovered
(stop time).
---

This states I need to provide the frame loss counters, which is fine, 
But if my test tool

a) only counts transmitted test frames as trasmitted
-and-
b) uses a non-frame loss based method to measure the recovery time

then providing the frame loss counters seems not entirely useful.

Additionally, in this case I MAY NOT use the formula to approximate 
recovery time.

So, it seems we're back to having the test tool behave a certain way 
instead of measuring a specific event.

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Tue, 20 Jul 2010 09:34:10 -0500

Are you really aware of such a test tool ??

----
But if my test tool

a) only counts transmitted test frames as transmitted
-and-
b) uses a non-frame loss based method to measure the recovery time
----

I am personally not aware of any tool that will not account for frames
lost ... unless you are sending / blasting  frames and there is no RX
port 

Again you can use the start / stop timers if you don't want to use the
formula / methodology provided 
As it's a MAY ... there is also a list of parameters / values to be
provided with each test case listed @ the draft ... 
 
But I am still missing you point since I don't see how / where we are
influencing the use of certain packet blaster against other

There may be a million ways to measure convergence, (like icmp / pings
instead ?? ) but we feel that describe or even list some of the possible
methodologies 
Are beyond the current work, we just want to be inline with the current
loss derived convergence methodology  implemented in current drafts (for
consistency among the WG)

How would you rephrase this ?? 

Rgds 

Fer

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Tuesday, July 20, 2010 10:17 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Timmons

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 20 Jul 2010 11:38:58 -0400

I suggest the following wording:

  Frame loss MUST be used to measure recovery time.  During recovery,
  protocols come up and routes get installed.  Recovery is complete
  when all routes are installed and are forwarding traffic.  The
  elapsed time from the first lost frame to the last lost frame MUST
  be reported as the recovery time.

In my experience working at or with providers and router vendors, this
is how they want recovery time measured.

You might also add:

  The conditions in terms of the type of routing protocols used in
  testing, the number of peers, number of routes, and NMS activity
  (SNMP polling, etc) may be varied by the tester to obtain results
  more representative of real world performance.

  The traffic source MUST provide a diverse set of destinations (and
  for some testing source and destination) in order to fully exercise
  the entries in the packet forwarding plane.

You can leave the exact conditions as out of scope.  I suggest that
you not try to provide default conditions since that will be latched
onto by marketroids who may influence product in a non-real-world
direction to acheive marketing bragging rights (it happens).

For fallback to a redundant RP, a failure of working RP followed by a
set of route changes is also a useful measure.  Route changes SHOULD
be at a fixed time after the RP failure if possible.  BFD may be
sufficient to satisfy this timing requirement.  The time it takes for
the standby RP to come up fully and then process the set of forwarding
changes is measured.  Multiple changes should be made to insure that
the worst case delay in changing forwarding entries is measured.

Curtis

In message <[email-masked]>
"Timmons C. Player" writes:

Great!

Not quite...

---

---

---

Thanks.

Timmons

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 20 Jul 2010 16:17:40 -0500

Timmons,

Thanks for the feedback, please see inline,

> then providing the frame loss counters seems not entirely useful.

Then, such a test tool is not compliant with RFC2544 (please see section
26.3 for frame loss rate calculation).

Also, the test tool will be out of sync with other upcoming BMWG
specifications such as IGP convergence benchmarking. Please see 2nd para
of section 1.
http://datatracker.ietf.org/doc/draft-ietf-bmwg-igp-dataplane-conv-meth/

<...
   IGP convergence time is measured on the data plane at the Tester by
   observing packet loss through the DUT.  All factors contributing to
   convergence time are accounted for by measuring on the data plane, as
...>

Please note that draft-ietf-bmwg-reset document does recommend the test
tool to report the frame-loss for each testcase.

Cheers,
Rajiv

---

From: "Timmons C. Player" <[email-masked]>
Date: Tue, 20 Jul 2010 17:34:12 -0400

> 26.3 for frame loss rate calculation).

No, no, no.  The context for this statement was counting test frames as 
transmitted when there is no physical link.  Neither RFC 2544 nor the 
convergence draft REQUIRE the test tool to increment the transmitted 
frame counter when there is no link on which to transmit any frames.

If there is no link, then a device cannot physically transmit frames, 
because there is nothing to transmit them on.  I know some test tools 
continue to increment their transmit counter at a fixed rate, even if 
there is no link on their test port.  What I'm trying to point out is 
that the current version of this draft REQUIRES this non-standard 
behavior and that it shouldn't.  There are alternative methods for 
measuring convergence, even when the transmitting or receiving ports 
loose link state.

Timmons

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 20 Jul 2010 16:43:16 -0500

Hi Timmons,

Thanks for the clarification. So, you are objecting to the following
para, nothing more, nothing less. Correct?

If so, then I am in agreement with you to removing it. Please confirm.

Cheers,
Rajiv

> Sent: Tuesday, July 20, 2010 5:34 PM
> To: Rajiv Asati (rajiva)
> Cc: Fernando Calabria (fcalabri); [email-masked]

---

From: Al Morton <[email-masked]>
Date: Tue, 20 Jul 2010 18:02:38 -0400

BMWG,

This is a very useful discussion. Let's keep it constructive.
I'm watching carefully.

In re-reading our references, I see that the RFC 1242 definition
of Frame Loss Rate is based on frames "not forwarded" as opposed
to "not offered" when the tester device's link goes down, and
the reporting section clearly mentions offered frames.

Al
bmwg chair

>3.6 Frame Loss Rate
>
>
>         Definition:
>                 Percentage of frames that should have been forwarded
>                 by a network device under steady state (constant)
>                 load that were not forwarded due to lack of
>                 resources.
>
>         Discussion:
>                 This measurement can be used in reporting the
>                 performance of a network device in an overloaded
>                 state.  This can be a useful indication of how a
>                 device would perform under pathological network
>                 conditions such as broadcast storms.
>
>         Measurement units:
>                 Percentage of N-octet offered frames that are dropped.
>                 To be reported as a graph of offered load vs frame loss.

At 05:34 PM 7/20/2010, Timmons C. Player wrote:
>No, no, no.  The context for this statement was counting test frames 
>as transmitted when there is no physical link.  Neither RFC 2544 nor 
>the convergence draft REQUIRE the test tool to increment the 
>transmitted frame counter when there is no link on which to transmit 
>any frames.
>
>If there is no link, then a device cannot physically transmit 
>frames, because there is nothing to transmit them on.  I know some 
>test tools continue to increment their transmit counter at a fixed 
>rate, even if there is no link on their test port.  What I'm trying 
>to point out is that the current version of this draft REQUIRES this 
>non-standard behavior and that it shouldn't.  There are alternative 
>methods for measuring convergence, even when the transmitting or 
>receiving ports loose link state.
>
>Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Tue, 20 Jul 2010 18:29:20 -0400

> If so, then I am in agreement with you to removing it. Please confirm.
Yes, assuming the previously discussed changes are also included, e.g. 
the definition for start/stop, so that other methods besides frame loss 
can be used to accurately measure the recovery time.

Thanks!

Timmons

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 20 Jul 2010 17:34:09 -0500

> If so, then I am in agreement with you to removing it. Please confirm.

I should have rather said - If so, then I am in agreement with you to
removing it for non-frame-loss method for recovery time measurement. 

Talking about non-frame-loss method, timestamp method is the only method
that is reasonable to include in this document.

Cheers,
Rajiv

> Rajiv Asati (rajiva)
> Sent: Tuesday, July 20, 2010 5:43 PM
> To: Timmons C. Player
> Cc: [email-masked]
frames,

---

From: "Timmons C. Player" <[email-masked]>
Date: Tue, 20 Jul 2010 18:42:30 -0400

> that is reasonable to include in this document.

Agreed.  I also should have said, "non-offered frames are considered 
lost" instead of frame-loss.

Timmons

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Wed, 21 Jul 2010 10:19:04 +0200

Hi,

I have gone through the 01 version and would have couple of 
comments/suggestions:

1) there is lots of repetitive text - as far as I could see the
reporting format is exactly same for all the tests and most of
the procedure text under each test is also same - it would benefit
may be from sticking the repetitive text into one generic-for-all
sections

2) In section 4.1.2 - the first sentence of the procedure -
"all destinations MUST be directly connected" - is confusing to me
at least - why is this specific to LC reset and I guess it wants
to say physically directly connected but surely the test traffic
destinations MUST NOT be directly connected else we would be
testing something else than forwarding performance (I appreciate
here the objective is to measure recovery time but it also
states it uses NDR style traffic to do so and if the traffic is directly
connected the resulting high CPU or processing related to such traffic
can influence the time measurement ...) ??

Jan

The climate of Edinburgh is such that the weak succumb young .... 
and the strong envy them.

                                 Dr. Johnson

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Fernando Calabria (fcalabri)
Sent: 19 July 2010 17:01
To: Timmons C. Player
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Rgds 

Fernando 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 11:54 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Great!

Not quite...

---

---

---

Thanks.

Timmons

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Wed, 21 Jul 2010 08:05:07 -0500

Jan, thanks a lot for your comments , 

inline 

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Wednesday, July 21, 2010 4:19 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: [bmwg] draft-ietf-bmwg-reset-01

Hi,

>> makes a lot of sense, we will work on this 

>> not sure I follow this one, let me know if I got it right ,,,
we are not measuring forwarding perf here, we just want to be sure that
the 
Destination  is only reachable via the LC that will we be reset to  we
avoid  any loop-holes per say, 
what about if the UUT has two LCs and the destination is behind a second
router connected in sort of a "Y" fashion to the UUT?
 ... (traffic may recover before the LC that was reset comes up  or
there may even be  a small bump in the traffic)
So this will be just be like routing convergence more than a LC
characterization exercise
Basically here we want  to be in line with the setup proposed by RFC
2544 
As far as the cpu goes I see you point but really this is getting into
the specific implementation of each UUT 
which is outside the scope of this work.
We are sending as much traffic as possible to be able to provide as
accurate and precise counters / values as possible
And  we believe this is fare when it comes to benchmarking different
platforms ,,, 

Let me know your thought 

Fer

Jan

                                 Dr. Johnson

-----Original Message-----

Rgds 

Fernando 

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 11:54 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]
Subject: Re: [bmwg] FW: WGLC: draft-ietf-bmwg-reset-00

Great!

Not quite...

---

---

---

Thanks.

Timmons

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Wed, 21 Jul 2010 15:41:34 +0200

> Let me know your thought 

2544 has the concept of senders/receivers/tester - so this
could read:

all receivers/testers MUST be directly connected ??

Jan

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Wed, 21 Jul 2010 09:09:44 -0500

Yes, if you look at the topologies @ 2544  this is the case 

Rgds

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Wednesday, July 21, 2010 9:42 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Jan

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Wed, 21 Jul 2010 19:16:42 -0500

Jan are we ok with this exchange ? something I missed ? 

So far we will :

1# place the reporting part into a new section and reference to it as
applicable 

2# I tries to explain why the sender and receiver (test tool) needs to
be directly connected to the UUT 
   Mainly to follow 2544 and avoid any possible routing back door ...

3# LC CPU as we understand your point while sending high rates of
traffic, if the LC punts @ the cpu / interrupt or does HW forwarding
   This is beyond the scope of this work .... 

Let me know if I addressed all these  points to your satisfaction  or
something else comes up 

Kind Regards 

Fer 

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: Wednesday, July 21, 2010 10:10 AM
To: Jan Novak (janovak)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Rgds

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Wednesday, July 21, 2010 9:42 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Jan

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Thu, 22 Jul 2010 10:25:07 +0200

Hi,

yes, sure, thanks for such a detailed follow up.

Jan

                                 Dr. Johnson

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: 22 July 2010 01:17
To: Fernando Calabria (fcalabri); Jan Novak (janovak)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

So far we will :

Kind Regards 

Fer 

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: Wednesday, July 21, 2010 10:10 AM
To: Jan Novak (janovak)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Rgds

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Wednesday, July 21, 2010 9:42 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Jan

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Thu, 22 Jul 2010 08:00:57 -0500

Thank you for taking the time to read the doc, and come back to us with
your feedback 

Rgds 

Fernandi 

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Thursday, July 22, 2010 4:25 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Hi,

Jan

                                 Dr. Johnson

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: 22 July 2010 01:17
To: Fernando Calabria (fcalabri); Jan Novak (janovak)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

So far we will :

Kind Regards 

Fer 

-----Original Message-----
From: Fernando Calabria (fcalabri) 
Sent: Wednesday, July 21, 2010 10:10 AM
To: Jan Novak (janovak)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Rgds

-----Original Message-----
From: Jan Novak (janovak) 
Sent: Wednesday, July 21, 2010 9:42 AM
To: Fernando Calabria (fcalabri)
Cc: [email-masked]'
Subject: RE: [bmwg] draft-ietf-bmwg-reset-01

Jan