Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance

From: "Jay Karthik (jakarthi)" <[email-masked]>
Date: Sat, 8 Nov 2014 06:45:30 +0000

Hi Bhuvan, Anton, Vishwas, and Mark,

I zoomed in, on one of your ‘Performance Benchmarking Tests’, (7.1.4 Path Provisioning Time).

It appears to me, that the procedure for 'proactive path provisioning’ may have to be clarified a bit further. To begin with, when the controller is benchmarked, the dependency on the SDN Nodes/Routing Elements ('Network Devices' as described in the draft Al points us to -http://tools.ietf.org/html/draft-irtf-sdnrg-layer-terminology-04#page-11) could distort the actual measurements due to processing, propagation or buffering delays experienced by the routing elements. Agree ?

“Install the flow with the learnt source and destination address through controller's northbound or management interface.” Not following this. The northbound I/f of the controller is the application. Is this a case where the application layer is simulating/creating a ‘demand’ from a source to a destination ?

“Record the time when a successful response for the flow installation is received (Tp) from the controller.” Does ‘ low installation' denote, programming the forwarding plane or the arrival of the first packet/frame corresponding to this test stream ? How will the controller determine this and is the controller notifying the SDN nodes ?

How is ‘the expiry of test interval (To)' used in the equation ?

Regards,
Jay

From: Bhuvaneswaran Vengainathan <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, September 30, 2014 at 3:04 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: vishwas manral <[email-masked]<mailto:[email-masked]>>, 'Anton Basil' <[email-masked]<mailto:[email-masked]>>, "Tassinari, Mark A" <[email-masked]<mailto:[email-masked]>>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance

Hi folks,

We have submitted next version (https://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-01) of the draft addressing the feedbacks and comments received on the previous version. The draft is based on the points that were discussed during the IETF 90 meeting (http://www.ietf.org/proceedings/90/slides/slides-90-bmwg-2.pdf).

Highlights of changes:

1.       Redefined metrics and methodologies to benchmark wide range of controller implementations independent of southbound and northbound protocols.

2.       Defined additional metrics including Topology Discovery Time, Path Provisioning Time and Rate.

3.       Mapped the defined benchmarks in 3x3 matrix against Performance, Scalability and Reliability.

We would love to hear any comments and queries on the same.

Thanks,
Authors

---

From: "Bhuvaneswaran Vengainathan" <[email-masked]>
Date: Sun, 9 Nov 2014 12:21:09 +0530

Hi Jay,

Thanks for providing your valuable comments.

Please find my responses inline for your queries with tag [Bhuvan]

Thank you,

Bhuvan | Cloud and SDN Solutions

Phone: +91 44 4567 2222 x242

Mobile: +91 988 428 6693 

Description: Description: Description: Description: Description:
Description: Description: Description: Description: Description: Veryx-Logo
for Webinar

 <http://www.veryxtech.com/> www.veryxtech.com

From: Jay Karthik (jakarthi) [mailto:[email-masked]] 
Sent: Saturday, November 08, 2014 12:16 PM
To: Bhuvaneswaran Vengainathan; Tassinari, Mark A; 'Anton Basil'; vishwas
manral
Cc: [email-masked]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance

I zoomed in, on one of your 'Performance Benchmarking Tests', (7.1.4 Path
Provisioning Time). 

It appears to me, that the procedure for 'proactive path provisioning' may
have to be clarified a bit further. To begin with, when the controller is
benchmarked, the dependency on the SDN Nodes/Routing Elements ('Network
Devices' as described in the draft Al points us to
-http://tools.ietf.org/html/draft-irtf-sdnrg-layer-terminology-04#page-11)
could distort the actual measurements due to processing, propagation or
buffering delays experienced by the routing elements. Agree ?

[Bhuvan] I agree that the routing elements could distort the actual
measurements but we assumed that this could be of negligible value. One
other approach to handle this situation is to consider the time of last flow
setup message form the controller for the corresponding flow instead of the
time of 1st received data traffic. We will appropriately address this in the
next version. 

"Install the flow with the learnt source and destination address through
controller's northbound or management interface." Not following this. The
northbound I/f of the controller is the application. Is this a case where
the application layer is simulating/creating a 'demand' from a source to a
destination ?

[Bhuvan] Typically for the proactive flow provisioning, the demand for flow
provisioning would be placed through northbound I/F, mostly using REST API
calls. The demand would be created/simulated by the application (e.g.,
BSS/OSS, OpenStack).

"Record the time when a successful response for the flow installation is
received (Tp) from the controller." Does ' low installation' denote,
programming the forwarding plane or the arrival of the first packet/frame
corresponding to this test stream ? How will the controller determine this
and is the controller notifying the SDN nodes ?

[Bhuvan] This denotes, programming the forwarding plane. The controller
determines this based on the network topology mechanism (mentioned as part
of pre-requisite). 

How is 'the expiry of test interval (To)' used in the equation ?

[Bhuvan] The expiry of the test interval (a.k.a test duration) is used as a
criterion to avoid indefinite wait for the completion of the test. This is
not used in the equation.

Regards,

Jay

From: Bhuvaneswaran Vengainathan <[email-masked]>
Date: Tuesday, September 30, 2014 at 3:04 AM
To: "[email-masked]" <[email-masked]>
Cc: vishwas manral <[email-masked]>, 'Anton Basil'
<[email-masked]>, "Tassinari, Mark A" <[email-masked]>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance

Hi folks,

We have submitted next version (
<https://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-01
https://tools.ietf.org/html/draft-bhuvan-bmwg-of-controller-benchmarking-01)
of the draft addressing the feedbacks and comments received on the previous
version. The draft is based on the points that were discussed during the
IETF 90 meeting (
<http://www.ietf.org/proceedings/90/slides/slides-90-bmwg-2.pdf>
http://www.ietf.org/proceedings/90/slides/slides-90-bmwg-2.pdf).  
 
Highlights of changes:
1.  Redefined metrics and methodologies to benchmark wide range of
controller implementations independent of southbound and northbound
protocols.
2.  Defined additional metrics including Topology Discovery Time, Path
Provisioning Time and Rate.
3.  Mapped the defined benchmarks in 3x3 matrix against Performance,
Scalability and Reliability.

Thanks,

Authors