Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

From: Jacob Rapp <[email-masked]>
Date: Mon, 12 Oct 2015 15:24:06 +0000

I think the problem that I have is that I’m not sure if the southbound protocol can really be decoupled here. There are very different functionality if you use one protocol or another. It may not be a totally fair comparison, but it is like saying I can write a draft that tests a “router” that tests routing protocols, but I don’t define if it is for OSFP of BGP.
It would be one thing, if you were testing just controller HA, but the drafts test how the flows or paths or etc are programmed, which may be different and have special terminology depending on the southbound protocol.

—
Jacob

On 10/12/15, 4:56 PM, "bmwg on behalf of MORTON, ALFRED C (AL)" <[email-masked] on behalf of [email-masked]> wrote:

>BMWG Folks, (this is the thread I was referring to in reply to Jacob)
>
>I support the adoption of these drafts as WG items.
>
>I'd also like to see some comments below addressed (e.g., Brian's),
>if they haven't already. We would have to look at controller-clusters
>to do add of these metrics, I believe.
>
>regards,
>Al
>(as a participant)
>
>-----Original Message-----
>From: Castelli, Brian [mailto:[email-masked]] 
>Sent: Monday, June 23, 2014 11:39 AM
>To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL); 'Banks, Sarah'; [email-masked]
>Cc: 'Anton Basil'; 'Vishwas Manral'
>Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance
>
>I really like Al¹s suggestion to generalize SDN controller benchmarking because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not define SDN. OpenFlow is one of several southbound protocols that can be used between a controller function and network node functions. Regardless of the maturity of current OpenFlow implementations, I believe that we can define a family of benchmarks that have general applicability to SDN environments because there are common functions/services that will be provided independent of protocol. At the end of the day, protocols are commodities. It¹s the service they provide that adds value.
>
>The current draft contains many of the elements required to sufficiently define a benchmarking methodology for a controller function in an SDN environment. We should follow Al¹s suggestion, define the taxonomy used in the draft, and consolidate wording to make the document easier to read and apply.
>
>What follows is an attempt to define a higher-level set of SDN controller test cases. It builds on the work done in the draft. I will provide some detail on the first test case in order to establish a cadence. Subsequent test cases will not have the detail, but the ability to iterate over variables and so on is implied.
>
>The SDN Controller test taxonomy would include:
>
>- Asynchronous Message Response Rate. A test case to measure AMRR would cover a variety of conditions that are analogous to the following OpenFlow-specific events:
>
>  o Packet_in received
>  o Flow expiration received
>  o Link down received
>
>In other words, AMRR would be used to measure the ability of an SDN controller to respond to a variety of asynchronous (i.e. unexpected) conditions. One methodology for measurement would be specified by the benchmark. The tester would then apply that methodology to packet_ins, flow expiration, link down, and whatever other AMR testing is appropriate.
>
>The AMRR methodology should include iteration variables and goal seeking.
>The tester ought to be able to use the methodology to determine, for example, the maximum packet_in AMRR for a variety of conditions, such as variations in the number of connected nodes, the complexity of the packet_ins, and so on. Variables also include negative testing with invalid packets, for example.
>
>- Synchronous Message Response Rate. A test case for measuring SMRR would cover a variety of conditions analogous to the following OpenFlow-specific
>events:
>
>  o Hello packet exchange
>  o Echo request/response exchange
>
>- Node Acquisition Rate. A test case for measuring NAR would be a special case of SMRR, but it specifically measures the rate at which an SDN controller can connect to nodes in a network.
>
>- Node Acquisition Maximum. A test case for NAM would measure the maximum number of network nodes that an SDN controller can connect to without error.
>
>- FailOver Response Time. There are at least two test cases for measuring FORT. One would measure a control cluster¹s ability to survive a controller failure. How long does it take the controller function to be assumed by another controller element? Another would measure the controller¹s ability to respond to node failures to do things like re-route traffic. How long does it take for the controller to send out commands that handle the error? This is a bit like measuring convergence time, but care must be taken to keep the node variables out of the measurement.
>
>- FailBack Response Time. Test cases for FBRT would be used to measure the response time when the controller comes back up or the node comes back online. How long does it take the controller to send the commands that restore service?
>
>
>
>On 6/23/14, 10:53 AM, "Bhuvan (Veryx Technologies)"
><[email-masked]> wrote:
>
>>Hi Al,
>>
>>Thanks for your comment. I do agree with you that there needs to be a 
>>common benchmarking mechanism for controller designs performing the 
>>same tasks.
>>But the technology is still relatively immature and the approach has 
>>various dimensions including centralized control vs distributed control 
>>(controller less).
>>Again the centralized control approach uses different programming 
>>methods (OpenFlow vs Non OpenFlow).
>>
>>Having said that, for better usability/understanding of the draft and 
>>to avoid misinterpretation, I personally feel it would be good to have 
>>separate benchmarking methodology for each approach (though the metrics 
>>remain same).
>>Also we would be happy to continue the effort to extend the same 
>>metrics for other approaches too.
>>
>>Please let me know if I've missed something.
>>
>>Sarah/Brain, please also share your thoughts on this.
>>
>>Thanks,
>>Bhuvan
>>
>>-----Original Message-----
>>From: MORTON, ALFRED C (AL) [mailto:[email-masked]]
>>Sent: Saturday, June 21, 2014 6:00 PM
>>To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies); 
>>[email-masked]
>>Cc: 'Anton Basil'; 'Vishwas Manral'
>>Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow 
>>SDN Controller Performance
>>
>>Hi Brian, Sarah, Bhuvan, and all,
>>
>>a couple of quick answers and a comment:
>>
>>> -----Original Message-----
>>> From: bmwg [mailto:[email-masked]] On Behalf Of Castelli, 
>>> Brian
>>...
>>> My two main concerns are getting the terminology right and making 
>>> sure that the proposed set of tests will achieve our goals. I believe 
>>> those goals ought to include a common taxonomy, enabling 
>>> apples-to-apples comparisons, and minimizing the time/work required 
>>> to execute the test cases.
>>
>>Great, that's exactly what we are chartered to do in BMWG.
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__datatracker.ietf.org_wg_bmwg_charter_&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_GjrhU&s=gJv3CuKB6C3osQoVuA3JGltD3bMZFTfuubj__2xVgMU&e= 
>>
>>
>>>
>>> I am willing to help improve the benchmark. I understand that it is 
>>> an early draft. The time is right to make sure that we are developing 
>>> the best specification that we can.
>>>
>>> How do we continue this process? Via email to this list? I am willing 
>>> to help.
>>
>>It's principally e-mail (just as you've been doing) and face-to-face 
>>meetings three times a year.  Sarah mentions that our next meeting is 
>>in her home town:
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ietf.org_meeting_90_index.html&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_GjrhU&s=PyQn5inWQ2S-mFAiKMB3UaRcIfzBr2syenp80SlaYxE&e= 
>>
>>Make some text proposals for the draft, and we can discuss them on the 
>>list.
>>
>>Having said that, and recognizing that there appears to be a comparable 
>>activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to 
>>add value to the industry is to approach the SDN controller problem 
>>more generically, such that different controller designs performing the 
>>same tasks could be benchmarked as black-boxes.  I realize this 
>>approach has substantial implications for the draft, but the benefit of 
>>wider applicability.
>>
>>food for thought,
>>Al
>>(as a participant)
>>
>>
>
>Spirent Communications E-mail confidentiality.
>-----------------------------------------------------------------------
>This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.
>
>Spirent Communications plc
>Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
>Tel No. +44 (0) 1293 767676
>Fax No. +44 (0) 1293 767677
>
>Registered in England Number 470893
>Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
>
>Or if within the US,
>
>Spirent Communications,
>26750 Agoura Road, Calabasas, CA, 91302, USA.
>Tel No. 1-818-676- 2300
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_GjrhU&s=K2L3UKeL6pXwVtMOMvWjpLaa6gT-IIk7WZ9-2384xZI&e=

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Tue, 13 Oct 2015 19:32:24 +0530

Hi Jacop,

Thanks for bringing this point for discussion in the mailing list. As you mentioned, the functionality of one protocol will be different from the another one. 

But the intent of the draft is to define a generic methodology to benchmark the controller performance as a black-box basis independent of protocols it support. For instance, to benchmark the controller message processing rate (throughput), we need to measure the input/output  rate i.e., the input (a.k.a request such as OpenFlow Packet-ins or BGP route updates or Route install) to the controller could be from southbound/northbound interface and the output (response such as OpenFlow Packet-Out/Flow-Mod or XMPP Publish) could be sent to southbound interface. So this way we want to derive a methodology abstracting the protocols to benchmark different controller implementations.

If you feel some definitions or terminologies need to be clarified further/added, we would be very happy to take such inputs and work on it.

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob

On 10/12/15, 4:56 PM, "bmwg on behalf of MORTON, ALFRED C (AL)" <[email-masked] on behalf of [email-masked]> wrote:

>I'd also like to see some comments below addressed (e.g., Brian's), if 
>they haven't already. We would have to look at controller-clusters to 
>do add of these metrics, I believe.
>
>regards,
>Al
>(as a participant)
>
>- Synchronous Message Response Rate. A test case for measuring SMRR 
>would cover a variety of conditions analogous to the following 
>OpenFlow-specific
>events:
>
>  o Hello packet exchange
>>various dimensions including centralized control vs distributed 
>>control (controller less).
>>separate benchmarking methodology for each approach (though the 
>>metrics remain same).
>>> sure that the proposed set of tests will achieve our goals. I 
>>> believe those goals ought to include a common taxonomy, enabling 
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__datatracker.ietf.o
>>rg_wg_bmwg_charter_&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt
>>-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_
>>GjrhU&s=gJv3CuKB6C3osQoVuA3JGltD3bMZFTfuubj__2xVgMU&e=
>>
>>
>>>
>>> an early draft. The time is right to make sure that we are 
>>> developing the best specification that we can.
>>>
>>> How do we continue this process? Via email to this list? I am 
>>> willing to help.
>>
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ietf.org_meeti
>>ng_90_index.html&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uE
>>s&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_Gjr
>>hU&s=PyQn5inWQ2S-mFAiKMB3UaRcIfzBr2syenp80SlaYxE&e=
>>
>>Having said that, and recognizing that there appears to be a 
>>comparable activity in ONF that (I assume) is OpenFlow-specific, 
>>perhaps a way to add value to the industry is to approach the SDN 
>>controller problem more generically, such that different controller 
>>designs performing the same tasks could be benchmarked as black-boxes.  
>>I realize this approach has substantial implications for the draft, 
>>but the benefit of wider applicability.
>>
>>food for thought,
>>Al
>>(as a participant)
>>
>>
>
>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailm
>an_listinfo_bmwg&d=BQIFAw&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs
>&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_GjrhU
>&s=K2L3UKeL6pXwVtMOMvWjpLaa6gT-IIk7WZ9-2384xZI&e=
_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Boris Hassanov <[email-masked]>
Date: Wed, 14 Oct 2015 19:32:21 +0000 (UTC)

Hi all!
My 0,05$:I would agree with Bhuvan here, IMHO  common methodology for testing is definitely needed . I see here SDN-controller as general term, SBIs can be different today and probably there will be even more of them tomorrow. It will be very hard to support all of their message details in one draft. May be hierarchical approach( when details for particular SBI/protocol testing in another document will be correlated/referenced to the general methodology ), could be better in this case.
Jacob, may be this is not perfect example, but we have RFC2544 which defines methodology that can be used for testing of devices with IPv4, labeled traffic etc.
Thank you.
SY,Boris

     On Tuesday, October 13, 2015 5:02 PM, Bhuvan (Veryx Technologies) <[email-masked]> wrote:

 Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob

---

From: Jacob Rapp <[email-masked]>
Date: Wed, 14 Oct 2015 19:56:59 +0000

I think it's worth a conversation live. I would argue that controllers are functionally so different right now, we are not at the point where we can define this. I would simply take the example of a Cisco ACI controller, VMWare Controller cluster & Open Daylight controllers and apply the standard to them and see the result. I'm happy to go along a help improve the draft to include them if we can, but functionally they are so different.

At some point we start defining what an SDN controller is and that is not our charter.

--
Jacob

On Oct 14, 2015, at 9:34 PM, Boris Hassanov <[email-masked]<mailto:[email-masked]>> wrote:

Hi all!

My 0,05$:
I would agree with Bhuvan here, IMHO  common methodology for testing is definitely needed . I see here SDN-controller as general term, SBIs can be different today and probably there will be even more of them tomorrow. It will be very hard to support all of their message details in one draft. May be hierarchical approach( when details for particular SBI/protocol testing in another document will be correlated/referenced to the general methodology ), could be better in this case.

Thank you.

SY,
Boris

On Tuesday, October 13, 2015 5:02 PM, Bhuvan (Veryx Technologies) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob

On 10/12/15, 4:56 PM, "bmwg on behalf of MORTON, ALFRED C (AL)" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

>From: Castelli, Brian [mailto:[email-masked]<mailto:[email-masked]>]
>To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL); 'Banks, Sarah';
>[email-masked]<mailto:[email-masked]>
><[email-masked]<mailto:[email-masked]>> wrote:
>
>>Hi Al,
>>
>>From: MORTON, ALFRED C (AL) [mailto:[email-masked]<mailto:[email-masked]>]
>>[email-masked]<mailto:[email-masked]>
>>> From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Castelli,
>>> Brian
>>...
>[email-masked]<mailto:[email-masked]>
_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>

_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>

---

From: Jacob Rapp <[email-masked]>
Date: Wed, 14 Oct 2015 20:09:16 +0000

Also for the RFC 2544 is focused on the data plane not the control plane.

--
Jacob

On Oct 14, 2015, at 9:57 PM, Jacob Rapp <[email-masked]<mailto:[email-masked]>> wrote:

--
Jacob

Hi all!

My 0,05$:

Thank you.

SY,
Boris

Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob

_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=mEwjLokjWWse4IGGeCVWyjlSoL-paCEYOSoERPwppfs&s=6XoMjPRvZDmcQHZUqjsHF-QFBiRwxG9Q_hAJbgUrluc&e=

---

From: "Castelli, Brian" <[email-masked]>
Date: Wed, 14 Oct 2015 21:20:57 +0000

I think I’m beginning to understand Jacob’s reservations about the claim that this draft covers all SDN controllers. SDN is so broad that it may include controllers of devices that are not switches, do not have flows, etc. What if we choose a 3rd alternative: Call this a benchmark for controllers of SDN switches. That is specific enough to rule out inappropriate usage while avoiding coupling with one specific protocol. Controlling switches implies many if not all of the tests included in the draft. Would that satisfy the concerns?

From: bmwg <[email-masked]<mailto:[email-masked]>> on behalf of Jacob Rapp <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, October 14, 2015 at 4:09 PM
To: Boris Hassanov <[email-masked]<mailto:[email-masked]>>
Cc: "MORTON, ALFRED C (AL)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

--
Jacob

--
Jacob

Hi all!

My 0,05$:

Thank you.

SY,
Boris

Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Wed, 14 Oct 2015 14:26:42 -0700

Some thinking points here:

1)How would end users apply this draft to benchmark 'sdn controllers'? 
2)How would this help to do an apple to apple comparison?

Some are have hardware, some are mainy software as Jacob pointed out. 
The performance will be different but the criterias of testing will 
change drastically between each one, for example changing of the cpu on 
a software controller could help, or tuning the kernel or using bypass 
NIC cards. But on the hardware based solutions it will be different.

3) Why do we need to benchmark the control plane? The end users in the 
data center care about data plane performance as this is what will 
impact the traffic going through it.

On 10/14/15 2:20 PM, Castelli, Brian wrote:
> I think I’m beginning to understand Jacob’s reservations about the claim
> that this draft covers all SDN controllers. SDN is so broad that it may
> include controllers of devices that are not switches, do not have flows,
> etc. What if we choose a 3rd alternative: Call this a benchmark for
> controllers of SDN switches. That is specific enough to rule out
> inappropriate usage while avoiding coupling with one specific protocol.
> Controlling switches implies many if not all of the tests included in
> the draft. Would that satisfy the concerns?
>
> From: bmwg <[email-masked] <mailto:[email-masked]>> on
> behalf of Jacob Rapp <[email-masked] <mailto:[email-masked]>>
> To: Boris Hassanov <[email-masked] <mailto:[email-masked]>>
> Cc: "MORTON, ALFRED C (AL)" <[email-masked]
> <mailto:[email-masked]>>, "[email-masked] <mailto:[email-masked]>"
> <[email-masked] <mailto:[email-masked]>>
> Subject: Re: [bmwg] WG ADOPTION:
> draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)
>
>> I think it's worth a conversation live. I would argue that controllers
>> are functionally so different right now, we are not at the point where
>> we can define this. I would simply take the example of a Cisco ACI
>> controller, VMWare Controller cluster & Open Daylight controllers and
>> apply the standard to them and see the result. I'm happy to go along a
>> help improve the draft to include them if we can, but functionally
>> they are so different.
>>
>>> I would agree with Bhuvan here, IMHO  common methodology for testing
>>> is definitely needed . I see here SDN-controller as general term,
>>> SBIs can be different today and probably there will be even more of
>>> them tomorrow. It will be very hard to support all of their message
>>> details in one draft. May be hierarchical approach( when details for
>>> particular SBI/protocol testing in another document will be
>>> correlated/referenced to the general methodology ), could be better
>>> in this case.
>>>
>>> Jacob, may be this is not perfect example, but we have RFC2544 which
>>> defines methodology that can be used for testing of devices with
>>> IPv4, labeled traffic etc.
>>>
>>> Thank you.
>>>
>>> SY,
>>> Boris
>>>
>>>
>>>
>>>
>>> On Tuesday, October 13, 2015 5:02 PM, Bhuvan (Veryx Technologies)
>>> <[email-masked]
>>> <mailto:[email-masked]>> wrote:
>>>
>>>
>>> Hi Jacop,
>>>
>>> Thanks for bringing this point for discussion in the mailing list. As
>>> you mentioned, the functionality of one protocol will be different
>>> from the another one.
>>>
>>> But the intent of the draft is to define a generic methodology to
>>> benchmark the controller performance as a black-box basis independent
>>> of protocols it support. For instance, to benchmark the controller
>>> message processing rate (throughput), we need to measure the
>>> input/output  rate i.e., the input (a.k.a request such as OpenFlow
>>> Packet-ins or BGP route updates or Route install) to the controller
>>> could be from southbound/northbound interface and the output
>>> (response such as OpenFlow Packet-Out/Flow-Mod or XMPP Publish) could
>>> be sent to southbound interface. So this way we want to derive a
>>> methodology abstracting the protocols to benchmark different
>>> controller implementations.
>>>
>>> If you feel some definitions or terminologies need to be clarified
>>> further/added, we would be very happy to take such inputs and work on it.
>>>
>>> Thanks,
>>> Bhuvan
>>>
>>> From: bmwg [mailto:[email-masked]
>>> <mailto:[email-masked]>] On Behalf Of Jacob Rapp
>>> To: MORTON, ALFRED C (AL); [email-masked] <mailto:[email-masked]>
>>> Subject: Re: [bmwg] WG ADOPTION:
>>> draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)
>>>
>>> I think the problem that I have is that I’m not sure if the
>>> southbound protocol can really be decoupled here. There are very
>>> different functionality if you use one protocol or another. It may
>>> not be a totally fair comparison, but it is like saying I can write a
>>> draft that tests a “router” that tests routing protocols, but I don’t
>>> define if it is for OSFP of BGP.
>>> It would be one thing, if you were testing just controller HA, but
>>> the drafts test how the flows or paths or etc are programmed, which
>>> may be different and have special terminology depending on the
>>> southbound protocol.
>>>
>>> —
>>> Jacob
>>>
>>>
>>>
>>> On 10/12/15, 4:56 PM, "bmwg on behalf of MORTON, ALFRED C (AL)"
>>> <[email-masked] <mailto:[email-masked]> on behalf of
>>> [email-masked] <mailto:[email-masked]>> wrote:
>>>
>>> >From: Castelli, Brian [mailto:[email-masked] <mailto:[email-masked]>]
>>> >[email-masked] <mailto:[email-masked]>
>>> >I really like Al¹s suggestion to generalize SDN controller benchmarking because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not define SDN. OpenFlow is one of several southbound protocols that can be used between a controller function and network node functions. Regardless of the maturity of current OpenFlow
>>> implementations, I believe that we can define a family of benchmarks
>>> that have general applicability to SDN environments because there are
>>> common functions/services that will be provided independent of
>>> protocol. At the end of the day, protocols are commodities. It¹s the
>>> service they provide that adds value.
>>> >
>>> >- FailOver Response Time. There are at least two test cases for measuring FORT. One would measure a control cluster¹s ability to survive a controller failure. How long does it take the controller function to be assumed by another controller element? Another would measure the controller¹s ability to respond to node failures
>>> to do things like re-route traffic. How long does it take for the
>>> controller to send out commands that handle the error? This is a bit
>>> like measuring convergence time, but care must be taken to keep the
>>> node variables out of the measurement.
>>> >
>>> ><[email-masked]
>>> <mailto:[email-masked]>> wrote:
>>> >
>>> >>Hi Al,
>>> >>
>>> >>From: MORTON, ALFRED C (AL) [mailto:[email-masked] <mailto:[email-masked]>]
>>> >>[email-masked] <mailto:[email-masked]>
>>> >>> From: bmwg [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of
>>> Castelli,
>>> >>> Brian
>>> >>...
>>> >This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of
>>> this transmission is strictly forbidden. If you have received this
>>> message in error please notify the sender by return e-mail and delete
>>> it from your system.
>>> >
>>> >[email-masked] <mailto:[email-masked]>
>>> [email-masked] <mailto:[email-masked]>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
>>>
>>> [email-masked] <mailto:[email-masked]>
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
>>>
>>>
>> [email-masked] <mailto:[email-masked]>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 14 Oct 2015 17:27:14 -0400

...and, as I  suggested in a separated thread (approximately), we say
"For the purposes of this memo, the SDN Controller is a
function that manages and controls SDN switches, <adding more qualifications>".

This is about scope of work, and the title needs to represent the scope.
Al

From: Castelli, Brian [mailto:[email-masked]]
Sent: Wednesday, October 14, 2015 5:21 PM
To: Jacob Rapp; Boris Hassanov
Cc: MORTON, ALFRED C (AL); [email-masked]
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

I think I'm beginning to understand Jacob's reservations about the claim that this draft covers all SDN controllers. SDN is so broad that it may include controllers of devices that are not switches, do not have flows, etc. What if we choose a 3rd alternative: Call this a benchmark for controllers of SDN switches. That is specific enough to rule out inappropriate usage while avoiding coupling with one specific protocol. Controlling switches implies many if not all of the tests included in the draft. Would that satisfy the concerns?

--
Jacob

--
Jacob

My 0,05$:

Thank you.

SY,
Boris

Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

I think the problem that I have is that I'm not sure if the southbound protocol can really be decoupled here. There are very different functionality if you use one protocol or another. It may not be a totally fair comparison, but it is like saying I can write a draft that tests a "router" that tests routing protocols, but I don't define if it is for OSFP of BGP.
It would be one thing, if you were testing just controller HA, but the drafts test how the flows or paths or etc are programmed, which may be different and have special terminology depending on the southbound protocol.

-
Jacob

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Wed, 14 Oct 2015 14:30:30 -0700

some sdn controllers only 'manage' switches and don't control. Other are 
in the control plane. It's a very broad terminology prone to a lot of 
confusion, to be useful, the benchmark needs to clearly test one type of 
device or scope of performance (data center benchmarking?) and not go 
into a broad SDN controller term which i agree is not our charter to 
define.

On 10/14/15 2:27 PM, MORTON, ALFRED C (AL) wrote:
> “For the purposes of this memo, the SDN Controller is a
>
> function that manages and controls SDN switches, <adding more
> qualifications>”.
>
> *From:*Castelli, Brian [mailto:[email-masked]]
> *Sent:* Wednesday, October 14, 2015 5:21 PM
> *To:* Jacob Rapp; Boris Hassanov
> *Cc:* MORTON, ALFRED C (AL); [email-masked]
> *Subject:* Re: [bmwg] WG ADOPTION:
> *From: *bmwg <[email-masked] <mailto:[email-masked]>> on
> *Date: *Wednesday, October 14, 2015 at 4:09 PM
> *To: *Boris Hassanov <[email-masked] <mailto:[email-masked]>>
> *Cc: *"MORTON, ALFRED C (AL)" <[email-masked]
> *Subject: *Re: [bmwg] WG ADOPTION:
>     I think it's worth a conversation live. I would argue that
>     controllers are functionally so different right now, we are not at
>     the point where we can define this. I would simply take the example
>     of a Cisco ACI controller, VMWare Controller cluster & Open Daylight
>     controllers and apply the standard to them and see the result. I'm
>     happy to go along a help improve the draft to include them if we
>     can, but functionally they are so different.
>
>         I would agree with Bhuvan here, IMHO  common methodology for
>         testing is definitely needed . I see here SDN-controller as
>         general term, SBIs can be different today and probably there
>         will be even more of them tomorrow. It will be very hard to
>         support all of their message details in one draft. May be
>         hierarchical approach( when details for particular SBI/protocol
>         testing in another document will be correlated/referenced to the
>         general methodology ), could be better in this case.
>
>         Jacob, may be this is not perfect example, but we have RFC2544
>         which defines methodology that can be used for testing of
>         devices with IPv4, labeled traffic etc.
>
>         Thank you.
>
>         SY,
>
>         Boris
>
>         On Tuesday, October 13, 2015 5:02 PM, Bhuvan (Veryx
>         Technologies) <[email-masked]
>         Thanks for bringing this point for discussion in the mailing
>         list. As you mentioned, the functionality of one protocol will
>         be different from the another one.
>
>         But the intent of the draft is to define a generic methodology
>         to benchmark the controller performance as a black-box basis
>         independent of protocols it support. For instance, to benchmark
>         the controller message processing rate (throughput), we need to
>         measure the input/output  rate i.e., the input (a.k.a request
>         such as OpenFlow Packet-ins or BGP route updates or Route
>         install) to the controller could be from southbound/northbound
>         interface and the output (response such as OpenFlow
>         Packet-Out/Flow-Mod or XMPP Publish) could be sent to southbound
>         interface. So this way we want to derive a methodology
>         abstracting the protocols to benchmark different controller
>         implementations.
>
>         If you feel some definitions or terminologies need to be
>         clarified further/added, we would be very happy to take such
>         inputs and work on it.
>
>         Thanks,
>         Bhuvan
>
>         different functionality if you use one protocol or another. It
>         may not be a totally fair comparison, but it is like saying I
>         can write a draft that tests a “router” that tests routing
>         protocols, but I don’t define if it is for OSFP of BGP.
>         It would be one thing, if you were testing just controller HA,
>         but the drafts test how the flows or paths or etc are
>         programmed, which may be different and have special terminology
>         depending on the southbound protocol.
>
>         —
>         Jacob
>
>
>
>         <[email-masked] <mailto:[email-masked]> on behalf
>         of [email-masked] <mailto:[email-masked]>> wrote:
>
>         >>> From: bmwg [mailto:[email-masked] <mailto:[email-masked]>] On Behalf
>         Of Castelli,
>         >>> Brian
>         >>...

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 14 Oct 2015 17:40:19 -0400

Hi Lucien,

> 2)How would this help to do an apple to apple comparison?

I think there are enough alternatives in the "Controller of SDN Switches"
category.  When you generalize, you can't test every unique bell and whistle
of a specific implementation + version. But that's ok. Even when we've
looked at specific protocols in control-plane and data-plane, we try to
define the most important characteristics as benchmarks, rather than
exhaustive coverage.

> impact the traffic going through it.

Yes, dataplane is important and what users care about. I think many users
will want different components in the "controller" and "switch" roles.
We've got drafts that intend to characterize the (v)Switch separately.
Bhuvan's draft intends to characterize the controller separately.
That's a good start, IMO. At some future point we need the whole enchilada.

it must be dinner time, somewhere,
Al

> From: bmwg [mailto:[email-masked]] On Behalf Of Lucien Avramov
> (lavramov)
> Sent: Wednesday, October 14, 2015 5:27 PM
> To: [email-masked]
> > I think I'm beginning to understand Jacob's reservations about the
> > claim that this draft covers all SDN controllers. SDN is so broad that
> > it may include controllers of devices that are not switches, do not
> > have flows, etc. What if we choose a 3rd alternative: Call this a
> > benchmark for controllers of SDN switches. That is specific enough to
> > rule out inappropriate usage while avoiding coupling with one specific
> protocol.
> >> happy to go along a help improve the draft to include them if we can,
> >> but functionally they are so different.
> >>
> >>> Thanks for bringing this point for discussion in the mailing list.
> >>> As you mentioned, the functionality of one protocol will be
> >>> different from the another one.
> >>>
> >>> benchmark the controller performance as a black-box basis
> >>> independent of protocols it support. For instance, to benchmark the
> >>> controller message processing rate (throughput), we need to measure
> >>> the input/output  rate i.e., the input (a.k.a request such as
> >>> OpenFlow Packet-ins or BGP route updates or Route install) to the
> >>> controller could be from southbound/northbound interface and the
> >>> output (response such as OpenFlow Packet-Out/Flow-Mod or XMPP
> >>> Publish) could be sent to southbound interface. So this way we want
> >>> to derive a methodology abstracting the protocols to benchmark
> >>> different controller implementations.
> >>>
> >>> I think the problem that I have is that I'm not sure if the
> >>> not be a totally fair comparison, but it is like saying I can write
> >>> a draft that tests a "router" that tests routing protocols, but I
> >>> don't define if it is for OSFP of BGP.
> >>> >if they haven't already. We would have to look at
> >>> >controller-clusters to do add of these metrics, I believe.
> >>> >
> >>> >regards,
> >>> >Al
> >>> >(as a participant)
> >>> >
> >>> >To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL); 'Banks,
> >>> >Sarah'; [email-masked] <mailto:[email-masked]>
> >>> >I really like Al¹s suggestion to generalize SDN controller
> >>> >benchmarking because 1) the ONF effort is OpenFlow specific and 2)
> >>> >OpenFlow does not define SDN. OpenFlow is one of several southbound
> >>> >protocols that can be used between a controller function and
> >>> >network node functions. Regardless of the maturity of current
> >>> >OpenFlow
> >>> that have general applicability to SDN environments because there
> >>> are common functions/services that will be provided independent of
> >>> >The current draft contains many of the elements required to
> sufficiently define a benchmarking methodology for a controller function
> in an SDN environment. We should follow Al¹s suggestion, define the
> taxonomy used in the draft, and consolidate wording to make the document
> easier to read and apply.
> >>> >
> >>> >What follows is an attempt to define a higher-level set of SDN
> controller test cases. It builds on the work done in the draft. I will
> provide some detail on the first test case in order to establish a
> cadence. Subsequent test cases will not have the detail, but the ability
> to iterate over variables and so on is implied.
> >>> >
> >>> >- Asynchronous Message Response Rate. A test case to measure AMRR
> would cover a variety of conditions that are analogous to the following
> OpenFlow-specific events:
> >>> >
> >>> >  o Packet_in received
> >>> >In other words, AMRR would be used to measure the ability of an SDN
> >>> >controller to respond to a variety of asynchronous (i.e.
> >>> >unexpected) conditions. One methodology for measurement would be
> >>> >specified by the benchmark. The tester would then apply that
> >>> >methodology to packet_ins, flow expiration, link down, and whatever
> >>> >other AMR
> >>> testing is appropriate.
> >>> >
> >>> >The tester ought to be able to use the methodology to determine,
> for example, the maximum packet_in AMRR for a variety of conditions,
> such as variations in the number of connected nodes, the complexity of
> the packet_ins, and so on. Variables also include negative testing with
> invalid packets, for example.
> >>> >
> >>> >- Node Acquisition Rate. A test case for measuring NAR would be a
> special case of SMRR, but it specifically measures the rate at which an
> SDN controller can connect to nodes in a network.
> >>> >
> >>> >- Node Acquisition Maximum. A test case for NAM would measure the
> maximum number of network nodes that an SDN controller can connect to
> without error.
> >>> >
> >>> >- FailOver Response Time. There are at least two test cases for
> >>> >measuring FORT. One would measure a control cluster¹s ability to
> >>> >survive a controller failure. How long does it take the controller
> >>> >function to be assumed by another controller element? Another would
> >>> >measure the controller¹s ability to respond to node failures
> >>> >- FailBack Response Time. Test cases for FBRT would be used to
> measure the response time when the controller comes back up or the node
> comes back online. How long does it take the controller to send the
> commands that restore service?
> >>> >
> >>> >
> >>> >
> >>> >>Thanks for your comment. I do agree with you that there needs to
> >>> >>be a common benchmarking mechanism for controller designs
> >>> >>performing the same tasks.
> >>> >>But the technology is still relatively immature and the approach
> >>> >>has various dimensions including centralized control vs
> >>> >>distributed control (controller less).
> >>> >>Having said that, for better usability/understanding of the draft
> >>> >>and to avoid misinterpretation, I personally feel it would be good
> >>> >>to have separate benchmarking methodology for each approach
> >>> >>(though the metrics remain same).
> >>> >>Subject: RE: [bmwg] Feedback for Benchmarking Methodology for
> >>> >>OpenFlow SDN Controller Performance
> >>> >>
> >>> >>> <mailto:[email-masked]>] On Behalf Of
> >>> Castelli,
> >>> >>> Brian
> >>> >>...
> >>> >>> My two main concerns are getting the terminology right and
> >>> >>> making sure that the proposed set of tests will achieve our
> >>> >>> goals. I believe those goals ought to include a common taxonomy,
> >>> >>> enabling apples-to-apples comparisons, and minimizing the
> >>> >>> time/work required to execute the test cases.
> >>> >>
> >>> >>-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3
> >>> >>Dku_ GjrhU&s=gJv3CuKB6C3osQoVuA3JGltD3bMZFTfuubj__2xVgMU&e=
> >>> >>
> >>> >>
> >>> >>>
> >>> >>> I am willing to help improve the benchmark. I understand that it
> >>> >>> is an early draft. The time is right to make sure that we are
> >>> >>It's principally e-mail (just as you've been doing) and
> >>> >>face-to-face meetings three times a year.  Sarah mentions that our
> >>> >>next meeting is in her home town:
> >>> >>s&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku
> >>> >>_Gjr hU&s=PyQn5inWQ2S-mFAiKMB3UaRcIfzBr2syenp80SlaYxE&e=
> >>> >>
> >>> >>Make some text proposals for the draft, and we can discuss them on
> >>> >>the list.
> >>> >>
> >>> >>controller problem more generically, such that different
> >>> >>controller designs performing the same tasks could be benchmarked
> as black-boxes.
> >>> >>I realize this approach has substantial implications for the
> >>> >>draft, but the benefit of wider applicability.
> >>> >>
> >>> >>food for thought,
> >>> >>Al
> >>> >>(as a participant)
> >>> >>
> >>> >>
> >>> >
> >>> >-------------------------------------------------------------------
> >>> >---- This e-mail contains confidential and / or privileged
> >>> >information belonging to Spirent Communications plc, its affiliates
> >>> >and / or subsidiaries. If you are not the intended recipient, you
> >>> >are hereby notified that any disclosure, copying, distribution and
> >>> >/ or the taking of any action based upon reliance on the contents
> >>> >of
> >>> message in error please notify the sender by return e-mail and
> >>> delete it from your system.
> >>> >
> >>> >Registered in England Number 470893 Registered at Northwood Park,
> >>> >Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
> >>> >
> >>> >Or if within the US,
> >>> >
> >>> >Spirent Communications,
> >>> >&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_G
> >>> >jrhU &s=K2L3UKeL6pXwVtMOMvWjpLaa6gT-IIk7WZ9-2384xZI&e=
> >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_m
> >>> ailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMN
> >>> tXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU6
> >>> 8-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >>>
> >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_m
> >>> ailman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMN
> >>> tXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU6
> >>> 8-I_C4AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >>>
> >>>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> >> lman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt
> >> -uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=mEwjLokjWWse4IGGeCVWyjlSoL-paCEYOSoER
> >> Pwppfs&s=6XoMjPRvZDmcQHZUqjsHF-QFBiRwxG9Q_hAJbgUrluc&e=
> >
> >

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 14 Oct 2015 17:47:00 -0400

Hi Lucien,

after agreeing on the beginnings of a definition
(the one I dashed-off below):

  ... function that manages *and* controls SDN switches, <adding more qualifications>".

the scope police would say, "if you haven't got control capability,
the controller is out of scope for this memo".

ok?
Al

> From: Lucien Avramov (lavramov) [mailto:[email-masked]]
> Sent: Wednesday, October 14, 2015 5:31 PM
> To: MORTON, ALFRED C (AL); Castelli, Brian; Jacob Rapp; Boris Hassanov
> Cc: [email-masked]
> > qualifications>".
> <mailto:[email-masked]>
> >         can write a draft that tests a "router" that tests routing
> >         protocols, but I don't define if it is for OSFP of BGP.
> >         >Subject: Re: [bmwg] Feedback for Benchmarking Methodology for
> OpenFlow
> protocols that can be used between a controller function and network
> node functions. Regardless of the maturity of current OpenFlow
> common functions/services that will be provided independent of protocol.
> At the end of the day, protocols are commodities. It¹s the service they
> provide that adds value.
> >         >
> >         >What follows is an attempt to define a higher-level set of
> SDN controller test cases. It builds on the work done in the draft. I
> will provide some detail on the first test case in order to establish a
> >         >- Asynchronous Message Response Rate. A test case to measure
> AMRR would cover a variety of conditions that are analogous to the
> following OpenFlow-specific events:
> >         >
> >         >  o Packet_in received
> >         >In other words, AMRR would be used to measure the ability of
> an SDN controller to respond to a variety of asynchronous (i.e.
> specified by the benchmark. The tester would then apply that methodology
> to packet_ins, flow expiration, link down, and whatever other AMR
> testing is appropriate.
> >         >
> >         >The tester ought to be able to use the methodology to
> determine, for example, the maximum packet_in AMRR for a variety of
> conditions, such as variations in the number of connected nodes, the
> complexity of the packet_ins, and so on. Variables also include negative
> testing with invalid packets, for example.
> >         >
> >         >- Node Acquisition Rate. A test case for measuring NAR would
> be a special case of SMRR, but it specifically measures the rate at
> which an SDN controller can connect to nodes in a network.
> >         >
> >         >- Node Acquisition Maximum. A test case for NAM would measure
> the maximum number of network nodes that an SDN controller can connect
> to without error.
> >         >
> >         >- FailOver Response Time. There are at least two test cases
> for measuring FORT. One would measure a control cluster¹s ability to
> measure the controller¹s ability to respond to node failures to do
> things like re-route traffic. How long does it take for the controller
> to send out commands that handle the error? This is a bit like measuring
> convergence time, but care must be taken to keep the node variables out
> of the measurement.
> >         >
> >         >- FailBack Response Time. Test cases for FBRT would be used
> to measure the response time when the controller comes back up or the
> node comes back online. How long does it take the controller to send the
> making
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=4LKgWZFGzgq2ZNUk2Q5XudH3NI16VJ0lK3Dku_
> face-to-face
> controller
> >         >This e-mail contains confidential and / or privileged
> information belonging to Spirent Communications plc, its affiliates and
> / or subsidiaries. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution and / or the taking
> of any action based upon reliance on the contents of this transmission
> is strictly forbidden. If you have received this message in error please
> notify the sender by return e-mail and delete it from your system.
> >         >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4
> > AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_bmwg&d=BQMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_C4
> > AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> > man_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-u
> > Es&r=4eVZvh2C5Un5N5XXv1rh7g&m=mEwjLokjWWse4IGGeCVWyjlSoL-paCEYOSoERPwp
> > pfs&s=6XoMjPRvZDmcQHZUqjsHF-QFBiRwxG9Q_hAJbgUrluc&e=
> >
> >
> >

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Thu, 15 Oct 2015 19:16:22 +0530

Hello everyone,

Based on the below discussion, I think the term SDN controller definition
has to be refined further in this memo as suggested by Al/Brain.
Also as suggested by Al, we will include the scope statement "If an SDN
controller does not have control capability, the controller is out of scope
for this memo".

Hope this fine with everyone.

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: 15 October 2015 03:17
To: Lucien Avramov (lavramov); Castelli, Brian; Jacob Rapp; Boris Hassanov
Cc: [email-masked]
Subject: Re: [bmwg] WG ADOPTION:

Hi Lucien,

after agreeing on the beginnings of a definition (the one I dashed-off
below):

  ... function that manages *and* controls SDN switches, <adding more
qualifications>".

the scope police would say, "if you haven't got control capability, the
controller is out of scope for this memo".

ok?
Al

> some sdn controllers only 'manage' switches and don't control. Other 
> are in the control plane. It's a very broad terminology prone to a lot 
> of confusion, to be useful, the benchmark needs to clearly test one 
> type of device or scope of performance (data center benchmarking?) and 
> not go into a broad SDN controller term which i agree is not our 
> charter to define.
>
>
>
> > claim that this draft covers all SDN controllers. SDN is so broad 
> > that it may include controllers of devices that are not switches, do 
> > not have flows, etc. What if we choose a 3rd alternative: Call this 
> > a benchmark for controllers of SDN switches. That is specific enough 
> > to rule out inappropriate usage while avoiding coupling with one 
> > specific
> protocol.
> > Controlling switches implies many if not all of the tests included 
> > in the draft. Would that satisfy the concerns?
> >
> > *From: *bmwg <[email-masked] <mailto:[email-masked]>> 
> > on behalf of Jacob Rapp <[email-masked] <mailto:[email-masked]>>
> At the end of the day, protocols are commodities. It¹s the service 
> they provide that adds value.
> >         >
> sufficiently define a benchmarking methodology for a controller 
> function in an SDN environment. We should follow Al¹s suggestion, 
> define the taxonomy used in the draft, and consolidate wording to make 
> the document easier to read and apply.
> >         >
> will provide some detail on the first test case in order to establish 
> a cadence. Subsequent test cases will not have the detail, but the 
> ability to iterate over variables and so on is implied.
> >         >
> other AMR testing is appropriate.
> >         >
> complexity of the packet_ins, and so on. Variables also include 
> negative testing with invalid packets, for example.
> >         >
> to send out commands that handle the error? This is a bit like 
> measuring convergence time, but care must be taken to keep the node 
> variables out of the measurement.
> >         >
> node comes back online. How long does it take the controller to send 
> the commands that restore service?
> >         >
> >         >
> >         >
> >         >>Having said that, for better usability/understanding of 
> > the
> draft and
> >         >>separate benchmarking methodology for each approach 
> > (though
> the
> >         >>metrics remain same).
> >         >>Also we would be happy to continue the effort to extend 
> > the
> same
> >         >>> apples-to-apples comparisons, and minimizing the 
> > time/work
> required
> >         >>> How do we continue this process? Via email to this list? 
> > I
> am
> >         >>> willing to help.
> >         >>
> and / or subsidiaries. If you are not the intended recipient, you are 
> hereby notified that any disclosure, copying, distribution and / or 
> the taking of any action based upon reliance on the contents of this 
> transmission is strictly forbidden. If you have received this message 
> in error please notify the sender by return e-mail and delete it from your
system.
> >         >Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 
> > 9XN,
> United Kingdom.
> > ai
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_
> > C4 AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> > ai
> > uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=B5kq3hQUCK15sk9bAe4fPf-KMQVMJQsU68-I_
> > C4 AJzU&s=WW_J6bWjO5D-Bc3laLMFroyH_w03jRhE2Z-UZ3ciH8U&e=>
> >
> > Es&r=4eVZvh2C5Un5N5XXv1rh7g&m=mEwjLokjWWse4IGGeCVWyjlSoL-paCEYOSoERP
> > wp pfs&s=6XoMjPRvZDmcQHZUqjsHF-QFBiRwxG9Q_hAJbgUrluc&e=
> >
> >
> >

---

From: Boris Hassanov <[email-masked]>
Date: Fri, 16 Oct 2015 01:01:53 -0700

Hi,

Yes, I know the difference with 2544.

My point was that we can use 2544 benchmarking methodology for testing any type of router architecture:  NP-based, CPU-based or even SW-based.

Of course common sense should be used for results interpretation, because obviously NP-based router with distributed architecture will be much faster than SW-based. But it does not mean that they cannot be compared at all.

Thus i belive that similar approach is nèeded here.

SY,

Boris

От:"Jacob Rapp" <[email-masked]>
Дата:ср, Окт 14, 2015 в 23:09
Тема:Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

--

Jacob

On Oct 14, 2015, at 9:57 PM, Jacob Rapp <[email-masked]> wrote:

--

Jacob

On Oct 14, 2015, at 9:34 PM, Boris Hassanov <[email-masked]> wrote:

Hi all!

My 0,05$:

Thank you.

SY,

Boris

Hi Jacop,

Thanks,
Bhuvan

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of Jacob Rapp
Sent: 12 October 2015 20:54
To: MORTON, ALFRED C (AL); [email-masked]
Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

—
Jacob