Subject: [bmwg] Comments on Accelerated Stress Benchmarking Terminology & Methodology

From: "Silvija Andrijic Dry \(sdry\)" <[email-masked]>
Date: Tue, 31 Oct 2006 14:49:35 -0500

Hi Scott,Shankar
 
Hopefully is not too late for the comments on these two drafts...
 
First let me say that I like a lot idea of  trying to address
benchmarking in more realistic operational scenarios by implementing
multiple technologies at the same time and applying operational and
failure stimuli to the DUT. That is very complex topic to be addressed
in a standard way so I commend the work that you have done on this
draft.
 
Now here are some comments and suggestions:
 
A) draft-ietf-bmwg-acc-bench-term-10.txt
 
1) 3.1.7 Uncontrolled Session Loss/ Discussion:
    For some protocols such as PIM it is not sufficient to check only
test equipment side to ensure that "session" is established but both
sides should be checked (DUT and test equipment).  This is because in
some faulty instances you might end up with situation where DUT would be
sending PIM hellos but let's say is having a bug where it's not
processing hellos coming from the test tool. In such a case test tool
would show PIM neighbor as operational while DUT wouldn't, i.e. you
would miss some bugs if looking only at the test tool. I had experience
of seeing such bugs in practice especially after instability conditions.
 
2) 3.2.3 Management Plane
    I like the fact that you are having telnet/SSH and serial console
sessions active as management plane is often overlooked. I would suggest
to take it one step further where you would define operational scenarios
to be excuted over such connections. In particular it would be useful to
do things such as provisioning new sessions/users/customers, i.e.
mimicing type of things let's say SP would do in their day to day
operations. This is very important for all of SPs as they want to ensure
that adding new customers/users is not going to affect existing ones. 
 
3) 3.3.3.1 Control Plane Configuration Set
  Why is this limited to Routing protocols, multicast and MPLS
protocols? Maybe note stating that others can be used in similar manner
would be nice here?
 
4) 3.3.4.1 Data Plane Configuration Set
 I think QoS features need to be included here or somewhere else in the
document as today pretty much all SPs and entreprises alike utilize QoS
thus any impact of having QoS features and larger number of policies,
etc should be assessed.
 
5) 3.4.2.2 Aggregate Forwarding Rate Degradation
 I think it would be useful to have this given as a % rather than
absolute value
 
6) 3.5.2.1 Recovered Aggregate Forwarding Rate/ Discussion:
   statement that starts with 'Ideally, ..." doesn't seem correct as
during transient state after instability is removed each measurement of
recovered forwarding rate won't be equal to stable aggregate fw rate. I
know you are aware of that as you are asking in 3.5.2.3 for measuring
"recovery time" so I think just this sentance seems misleading and could
use some rewriting.
 
7) 3.5.2.3 Recovery time
In addition to recovery time as defined in this section I would suggest
adding two  more recovery time related benchmarks that are often useful
and that is time from the instance instability is removed to the time
when the first failed session recovers and time when first packet is
forwarded. basically this gives you feel as to when did actuall recovery
start, i.e. gives you sense of how long total loss lasted versus time
during which some flows/sessions were recovered but not all. Graphing
things that you already have such as forwarding rate degradation as a
function of time will give you indication of this but it would be nice
to have explicit benchmark that actually gives you value of that time
interval.
 
B) draft-ietf-bmwg-acc-bench-meth-06.txt
 
1) 3.2 Test Considerations
 
1st sentance : ".......can be applied in service provider test
environments..." : as this methodology is not limited to SP environment
I would mention here enterprise environments as well.
 
2) 3.3. Configuration Sets
In Example multicast config set I would add following:
- Number of PIM neighborships
- Bi-dir PIM (enabled/disabled)
instead of "number of multicast groups" I would add following :
- number of (S,G) multicast routes
- number of (*,G) multicast routes
- total number of OIFs accross all multicast routes
 
3) 3.3 Configuration Sets:
If you would like I can also provide example of MVPN config set that can
be added to your draft, just let me know.
 
4) 3.3 Configuration Sets:
I think QoS should be added and actually be part of mandatory
configuration set as it's pretty much used by everyone today
 
5) 3.3.4 Benchmarks 
In addition to "Session count" in all thress phases I would suggest
adding also "route count" (unicast mandatory and optionaly multicast
routes if mcast used): Stable Route Count,Unstable Uncontrolled Routes
Lost and Recovered Route Count
 
Thanks,
 
Silvija

 

 

 
 
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg