Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 12 Oct 2015 10:56:20 -0400

BMWG Folks, (this is the thread I was referring to in reply to Jacob)

I support the adoption of these drafts as WG items.

I'd also like to see some comments below addressed (e.g., Brian's),
if they haven't already. We would have to look at controller-clusters
to do add of these metrics, I believe.

regards,
Al
(as a participant)

-----Original Message-----
From: Castelli, Brian [mailto:[email-masked]] 
Sent: Monday, June 23, 2014 11:39 AM
To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL); 'Banks, Sarah'; [email-masked]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

I really like Al¹s suggestion to generalize SDN controller benchmarking because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not define SDN. OpenFlow is one of several southbound protocols that can be used between a controller function and network node functions. Regardless of the maturity of current OpenFlow implementations, I believe that we can define a family of benchmarks that have general applicability to SDN environments because there are common functions/services that will be provided independent of protocol. At the end of the day, protocols are commodities. It¹s the service they provide that adds value.

The current draft contains many of the elements required to sufficiently define a benchmarking methodology for a controller function in an SDN environment. We should follow Al¹s suggestion, define the taxonomy used in the draft, and consolidate wording to make the document easier to read and apply.

What follows is an attempt to define a higher-level set of SDN controller test cases. It builds on the work done in the draft. I will provide some detail on the first test case in order to establish a cadence. Subsequent test cases will not have the detail, but the ability to iterate over variables and so on is implied.

The SDN Controller test taxonomy would include:

- Asynchronous Message Response Rate. A test case to measure AMRR would cover a variety of conditions that are analogous to the following OpenFlow-specific events:

  o Packet_in received
  o Flow expiration received
  o Link down received

In other words, AMRR would be used to measure the ability of an SDN controller to respond to a variety of asynchronous (i.e. unexpected) conditions. One methodology for measurement would be specified by the benchmark. The tester would then apply that methodology to packet_ins, flow expiration, link down, and whatever other AMR testing is appropriate.

The AMRR methodology should include iteration variables and goal seeking.
The tester ought to be able to use the methodology to determine, for example, the maximum packet_in AMRR for a variety of conditions, such as variations in the number of connected nodes, the complexity of the packet_ins, and so on. Variables also include negative testing with invalid packets, for example.

- Synchronous Message Response Rate. A test case for measuring SMRR would cover a variety of conditions analogous to the following OpenFlow-specific
events:

  o Hello packet exchange
  o Echo request/response exchange

- Node Acquisition Rate. A test case for measuring NAR would be a special case of SMRR, but it specifically measures the rate at which an SDN controller can connect to nodes in a network.

- Node Acquisition Maximum. A test case for NAM would measure the maximum number of network nodes that an SDN controller can connect to without error.

- FailOver Response Time. There are at least two test cases for measuring FORT. One would measure a control cluster¹s ability to survive a controller failure. How long does it take the controller function to be assumed by another controller element? Another would measure the controller¹s ability to respond to node failures to do things like re-route traffic. How long does it take for the controller to send out commands that handle the error? This is a bit like measuring convergence time, but care must be taken to keep the node variables out of the measurement.

- FailBack Response Time. Test cases for FBRT would be used to measure the response time when the controller comes back up or the node comes back online. How long does it take the controller to send the commands that restore service?

On 6/23/14, 10:53 AM, "Bhuvan (Veryx Technologies)"
<[email-masked]> wrote:

>Hi Al,
>
>Thanks for your comment. I do agree with you that there needs to be a 
>common benchmarking mechanism for controller designs performing the 
>same tasks.
>But the technology is still relatively immature and the approach has 
>various dimensions including centralized control vs distributed control 
>(controller less).
>Again the centralized control approach uses different programming 
>methods (OpenFlow vs Non OpenFlow).
>
>Having said that, for better usability/understanding of the draft and 
>to avoid misinterpretation, I personally feel it would be good to have 
>separate benchmarking methodology for each approach (though the metrics 
>remain same).
>Also we would be happy to continue the effort to extend the same 
>metrics for other approaches too.
>
>Please let me know if I've missed something.
>
>Sarah/Brain, please also share your thoughts on this.
>
>Thanks,
>Bhuvan
>
>-----Original Message-----
>From: MORTON, ALFRED C (AL) [mailto:[email-masked]]
>Sent: Saturday, June 21, 2014 6:00 PM
>To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies); 
>[email-masked]
>Cc: 'Anton Basil'; 'Vishwas Manral'
>Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow 
>SDN Controller Performance
>
>Hi Brian, Sarah, Bhuvan, and all,
>
>a couple of quick answers and a comment:
>
>> -----Original Message-----
>> From: bmwg [mailto:[email-masked]] On Behalf Of Castelli, 
>> Brian
>...
>> My two main concerns are getting the terminology right and making 
>> sure that the proposed set of tests will achieve our goals. I believe 
>> those goals ought to include a common taxonomy, enabling 
>> apples-to-apples comparisons, and minimizing the time/work required 
>> to execute the test cases.
>
>Great, that's exactly what we are chartered to do in BMWG.
>http://datatracker.ietf.org/wg/bmwg/charter/
>
>
>>
>> I am willing to help improve the benchmark. I understand that it is 
>> an early draft. The time is right to make sure that we are developing 
>> the best specification that we can.
>>
>> How do we continue this process? Via email to this list? I am willing 
>> to help.
>
>It's principally e-mail (just as you've been doing) and face-to-face 
>meetings three times a year.  Sarah mentions that our next meeting is 
>in her home town:
>http://www.ietf.org/meeting/90/index.html
>
>Make some text proposals for the draft, and we can discuss them on the 
>list.
>
>Having said that, and recognizing that there appears to be a comparable 
>activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to 
>add value to the industry is to approach the SDN controller problem 
>more generically, such that different controller designs performing the 
>same tasks could be benchmarked as black-boxes.  I realize this 
>approach has substantial implications for the draft, but the benefit of 
>wider applicability.
>
>food for thought,
>Al
>(as a participant)
>
>

Spirent Communications E-mail confidentiality.
-----------------------------------------------------------------------
This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
26750 Agoura Road, Calabasas, CA, 91302, USA.
Tel No. 1-818-676- 2300

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 12 Oct 2015 11:17:42 -0400

...to add *some* of these metrics, I believe.  
(just to be clear, not sure what happened there,
something out of sync...)

Al

> From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Monday, October 12, 2015 10:56 AM
> To: [email-masked]
> Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-
> benchmarking (terms and meth)
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> 
> I'd also like to see some comments below addressed (e.g., Brian's), if
> they haven't already. We would have to look at controller-clusters to do
> add of these metrics, I believe.
> 
> regards,
> Al
> (as a participant)
> 
> I really like Al¹s suggestion to generalize SDN controller benchmarking
> because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not
> define SDN. OpenFlow is one of several southbound protocols that can be
> used between a controller function and network node functions.
> Regardless of the maturity of current OpenFlow implementations, I
> believe that we can define a family of benchmarks that have general
> applicability to SDN environments because there are common
> functions/services that will be provided independent of protocol. At the
> end of the day, protocols are commodities. It¹s the service they provide
> that adds value.
> 
> The current draft contains many of the elements required to sufficiently
> define a benchmarking methodology for a controller function in an SDN
> environment. We should follow Al¹s suggestion, define the taxonomy used
> in the draft, and consolidate wording to make the document easier to
> read and apply.
> 
> What follows is an attempt to define a higher-level set of SDN
> controller test cases. It builds on the work done in the draft. I will
> provide some detail on the first test case in order to establish a
> cadence. Subsequent test cases will not have the detail, but the ability
> to iterate over variables and so on is implied.
> 
> - Asynchronous Message Response Rate. A test case to measure AMRR would
> cover a variety of conditions that are analogous to the following
> OpenFlow-specific events:
> 
>   o Packet_in received
> In other words, AMRR would be used to measure the ability of an SDN
> controller to respond to a variety of asynchronous (i.e. unexpected)
> conditions. One methodology for measurement would be specified by the
> benchmark. The tester would then apply that methodology to packet_ins,
> flow expiration, link down, and whatever other AMR testing is
> appropriate.
> 
> The tester ought to be able to use the methodology to determine, for
> example, the maximum packet_in AMRR for a variety of conditions, such as
> variations in the number of connected nodes, the complexity of the
> packet_ins, and so on. Variables also include negative testing with
> invalid packets, for example.
> 
> - Synchronous Message Response Rate. A test case for measuring SMRR
> would cover a variety of conditions analogous to the following OpenFlow-
> specific
> events:
> 
>   o Hello packet exchange
> - Node Acquisition Rate. A test case for measuring NAR would be a
> special case of SMRR, but it specifically measures the rate at which an
> SDN controller can connect to nodes in a network.
> 
> - Node Acquisition Maximum. A test case for NAM would measure the
> maximum number of network nodes that an SDN controller can connect to
> without error.
> 
> - FailOver Response Time. There are at least two test cases for
> measuring FORT. One would measure a control cluster¹s ability to survive
> a controller failure. How long does it take the controller function to
> be assumed by another controller element? Another would measure the
> controller¹s ability to respond to node failures to do things like re-
> route traffic. How long does it take for the controller to send out
> commands that handle the error? This is a bit like measuring convergence
> time, but care must be taken to keep the node variables out of the
> measurement.
> 
> - FailBack Response Time. Test cases for FBRT would be used to measure
> the response time when the controller comes back up or the node comes
> back online. How long does it take the controller to send the commands
> that restore service?
> 
> 
> 
> This e-mail contains confidential and / or privileged information
> belonging to Spirent Communications plc, its affiliates and / or
> subsidiaries. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution and / or the taking
> of any action based upon reliance on the contents of this transmission
> is strictly forbidden. If you have received this message in error please
> notify the sender by return e-mail and delete it from your system.
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg