Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

From: "Marius Georgescu" <[email-masked]>
Date: Sat, 1 Aug 2015 17:17:20 +0900

Dear Bhuvan et alia,

Please find attached my inline suggestions for the new draft. They were
marked with ###MG .

Thank you for taking care of so many comments.  

Best regards,

Marius 

From: bmwg [mailto:[email-masked]] On Behalf Of Bhuvan (Veryx
Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To: [email-masked]
Cc: 'Anton Basil' <[email-masked]>; Tassinari, Mark A
<[email-masked]>; vishwas.manral <[email-masked]>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance -
Updated Draft Version

Dear BMWG Members,

We have updated the draft
(draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
00> ) about SDN Controller benchmarking addressing comments received in
IETF-92 meeting. Thank you very much for providing your valuable comments.
The latest draft can be found in
draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01> 

Summary of Changes:

a.      Updated test setup diagram following the comment from Scott Bradner.

b.      Added recommendations for test topology, test iterations etc., to
use for benchmarking.

c.      Provided reference test topologies.

d.      Split Path Provisioning tests into two different tests - Proactive
and Reactive Path Provisioning tests.

e.      Provided more clarity on test procedure for some of the tests.

f.       Fixed IETF normative language usage.

We would love to hear any comments and queries on the same.

Thanks,

Authors

---

From: Sarah Banks <[email-masked]>
Date: Sat, 1 Aug 2015 21:08:21 -0700

Thank you Marius, for your time and inline suggestions. We'll review and touch base soon.

Thanks
Sarah

> On Aug 1, 2015, at 1:17 AM, Marius Georgescu <[email-masked]> wrote:
> 
> Dear Bhuvan et alia,
>  
> Please find attached my inline suggestions for the new draft. They were marked with ###MG .
>   <>
> From: bmwg [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Bhuvan (Veryx Technologies)
> To: [email-masked] <mailto:[email-masked]>
> Cc: 'Anton Basil' <[email-masked] <mailto:[email-masked]>>; Tassinari, Mark A <[email-masked] <mailto:[email-masked]>>; vishwas.manral <[email-masked] <mailto:[email-masked]>>
> Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version
>  
> Dear BMWG Members,
>  
> We have updated the draft (draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00 <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00>) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01>
>  
> Summary of Changes:
> b.      Added recommendations for test topology, test iterations etc., to use for benchmarking.
> d.      Split Path Provisioning tests into two different tests - Proactive and Reactive Path Provisioning tests.
> <draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT>_______________________________________________
> bmwg mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/bmwg <https://www.ietf.org/mailman/listinfo/bmwg>

---

From: Khasanov Boris <[email-masked]>
Date: Fri, 7 Aug 2015 07:07:13 +0000

Hi Sarah and all,
I am sorry  for jumping late into discussion.
Just few thoughts or my 0,02$:

1)      4.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-4.2> Test Traffic. IMHO, it would be better to say there explicitly  that in case of performance testing of forwarding plane we are looking for NDRs for different frame sizes according to RFC2544, if I got it correct.

2)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time May be would be good to add a couple of words for clarification - why it is important to SDN controller to process such messages as Error or Flow-Removed as quick as possible. It may sound obvious, but I believe that for many customers such clarification will be important.

3)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time Would it make sense to specify explicitly which messages need to be generated? I am not SW engineer so please correct me if my understanding is wrong, but I think different async messages may have different processing time on SDN controller, so results may be different and hard to compare (like comparison of best results but these results will be collected for different messages on different controllers).  We also have 6.3.1 test which measures correlation between error packets and performance which provides more details about messages vs. 6.1.2

4)      Same comment as previous one for 6.1.3

5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test procedure says about sending single traffic stream between end nodes. But it does not say about traffic stream parameters  (frame size, rate, IPv4 or something else etc.). Would be good to specify it  IMHO (as I mentioned above in 4.2 are not many details in current draft) . Another option could be just comment that these parameters should be defined before testing by customer itself.

6)      6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5. Also here we have to send traffic with some rate and calculate rate at the receiver end.  Sounds that we are looking for NDR here, may be we need to mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually say that traffic should be sent with NDR, so we need to find it during baseline tests.

7)      6.1.8 Network Topology Change Detection Time. Procedure Step 1, should we bring down ANY active node?  Does not matter which one?

8)      6.2 Scalability tests. Shouldn't we put there multidimensional test to find out the maximum numbers of  parameters from 6.2.1-6.2.3 but altogether?

9)      6.4.2  Network Re-Provisioning Time. Again it says about sending bi-directional traffic stream but without any stream details.

Thank you.

SY,
Boris

From: bmwg [mailto:[email-masked]] On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: [email-masked]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Thanks
Sarah

On Aug 1, 2015, at 1:17 AM, Marius Georgescu <[email-masked]<mailto:[email-masked]>> wrote:

Dear Bhuvan et alia,

Best regards,
Marius

From: bmwg [mailto:[email-masked]] On Behalf Of Bhuvan (Veryx Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To: [email-masked]<mailto:[email-masked]>
Cc: 'Anton Basil' <[email-masked]<mailto:[email-masked]>>; Tassinari, Mark A <[email-masked]<mailto:[email-masked]>>; vishwas.manral <[email-masked]<mailto:[email-masked]>>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Dear BMWG Members,

We have updated the draft (draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00>) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01>

Summary of Changes:

Thanks,
Authors
<draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT>_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Fri, 7 Aug 2015 17:01:52 +0530

Hi Boris,

Thank you very much for taking time to review this draft and providing your
valuable comments.  

I've provided my responses in line to your comments.

Best regards,

Bhuvan

From: bmwg [mailto:[email-masked]] On Behalf Of Khasanov Boris
Sent: 07 August 2015 12:37
To: Sarah Banks
Cc: [email-masked]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

Hi Sarah and all,

1)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-4.2> 4.2 Test Traffic. IMHO, it would be better to say there
explicitly  that in case of performance testing of forwarding plane we are
looking for NDRs for different frame sizes according to RFC2544, if I got it
correct. 

[Bhuvan] I would like to clarify that the draft focusses on defining test
methodology for control plane. So we believe that measuring NDRs for
different frame sizes would not be applicable. But it is important to
perform the defined tests for various frame sizes and traffic type.  So
which we are referring to RFC 2544.

2)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time May be would be
good to add a couple of words for clarification - why it is important to SDN
controller to process such messages as Error or Flow-Removed as quick as
possible. It may sound obvious, but I believe that for many customers such
clarification will be important.  

               [Bhuvan] Sure. We will add some clarification in the
discussion section (2.3.1.2) of
draft-bhuvan-bmwg-sdn-controller-benchmark-term-00 draft.

3)
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01#section-6.1.2> 6.1.2 Asynchronous Message Processing Time Would it make
sense to specify explicitly which messages need to be generated? I am not SW
engineer so please correct me if my understanding is wrong, but I think
different async messages may have different processing time on SDN
controller, so results may be different and hard to compare (like comparison
of best results but these results will be collected for different messages
on different controllers).  We also have 6.3.1 test which measures
correlation between error packets and performance which provides more
details about messages vs. 6.1.2

[Bhuvan] I completely agree with you. But the draft defines methodology
generic to all controllers implementation, I'm not sure if we able to
explicitly specify/recommend any particular message. We will provide some
examples for reference.

5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test
procedure says about sending single traffic stream between end nodes. But it
does not say about traffic stream parameters  (frame size, rate, IPv4 or
something else etc.). Would be good to specify it  IMHO (as I mentioned
above in 4.2 are not many details in current draft) . Another option could
be just comment that these parameters should be defined before testing by
customer itself.

[Bhuvan] Typically controller provisions the network path for a flow in the
SDN nodes (switches) when it sees the first packet of a flow (reactive).
Once the path is provisioned, all the subsequent packets are forwarded by
the SDN node (e.g., switch) based on the provisioned flow. Here we are
measuring the time that the controller takes to pave the entire path for a
flow in the SDN nodes. So I believe that the packet size or the rate may not
influence the test. 

6)      6.1.6 and 6.1.7 Reactive and Proactive rates. Same question as in 5.
Also here we have to send traffic with some rate and calculate rate at the
receiver end.  Sounds that we are looking for NDR here, may be we need to
mention it? Because in other tests like 6.4.1 (Procedure Step1) we actually
say that traffic should be sent with NDR, so we need to find it during
baseline tests.

[Bhuvan] Sure. We will mention the rate.

7)      6.1.8 Network Topology Change Detection Time. Procedure Step 1,
should we bring down ANY active node?  Does not matter which one?

[Bhuvan]  This will not matter for this test.

8)      6.2 Scalability tests. Shouldn't we put there multidimensional test
to find out the maximum numbers of  parameters from 6.2.1-6.2.3 but
altogether?

[Bhuvan] Could you please provide some more clarification about this
comment?

9)      6.4.2  Network Re-Provisioning Time. Again it says about sending
bi-directional traffic stream but without any stream details.

[Bhuvan] We will provide the details.

Thank you.

SY,

Boris

From: bmwg [mailto:[email-masked]] On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: [email-masked]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

Thank you Marius, for your time and inline suggestions. We'll review and
touch base soon.

Thanks

Sarah

On Aug 1, 2015, at 1:17 AM, Marius Georgescu <[email-masked]>
wrote:

Dear Bhuvan et alia,

Best regards,

Marius 

From: bmwg [ <mailto:[email-masked]> mailto:[email-masked]] On
Behalf Of Bhuvan (Veryx Technologies)
Sent: Tuesday, July 21, 2015 8:23 PM
To:  <mailto:[email-masked]> [email-masked]
Cc: 'Anton Basil' < <mailto:[email-masked]>
[email-masked]>; Tassinari, Mark A <
<mailto:[email-masked]> [email-masked]>; vishwas.manral <
<mailto:[email-masked]> [email-masked]>
Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance -
Updated Draft Version

Dear BMWG Members,

We have updated the draft (
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
00> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00) about SDN Controller
benchmarking addressing comments received in IETF-92 meeting. Thank you very
much for providing your valuable comments. The latest draft can be found in
<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-
01> draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01

Summary of Changes:

Thanks,

Authors

<draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01
MG.TXT>_______________________________________________
bmwg mailing list
 <mailto:[email-masked]> [email-masked]
 <https://www.ietf.org/mailman/listinfo/bmwg>
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Khasanov Boris <[email-masked]>
Date: Fri, 7 Aug 2015 12:39:28 +0000

Hi Bhuvan,

Thank you.

I put my answer below under [BKH].

SY,
Boris

From: Bhuvan (Veryx Technologies) [mailto:[email-masked]]
Sent: Friday, August 07, 2015 2:32 PM
To: Khasanov Boris
Cc: [email-masked]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Boris,

Thank you very much for taking time to review this draft and providing your valuable comments.
I've provided my responses in line to your comments.

Best regards,
Bhuvan

From: bmwg [mailto:[email-masked]] On Behalf Of Khasanov Boris
Sent: 07 August 2015 12:37
To: Sarah Banks
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Sarah and all,

1)      4.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-4.2> Test Traffic. IMHO, it would be better to say there explicitly  that in case of performance testing of forwarding plane we are looking for NDRs for different frame sizes according to RFC2544, if I got it correct.
[Bhuvan] I would like to clarify that the draft focusses on defining test methodology for control plane. So we believe that measuring NDRs for different frame sizes would not be applicable. But it is important to perform the defined tests for various frame sizes and traffic type.  So which we are referring to RFC 2544.
2)      6.1.2<http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01#section-6.1.2> Asynchronous Message Processing Time May be would be good to add a couple of words for clarification - why it is important to SDN controller to process such messages as Error or Flow-Removed as quick as possible. It may sound obvious, but I believe that for many customers such clarification will be important.
               [Bhuvan] Sure. We will add some clarification in the discussion section (2.3.1.2) of draft-bhuvan-bmwg-sdn-controller-benchmark-term-00 draft.

4)      Same comment as previous one for 6.1.3
[Bhuvan] I completely agree with you. But the draft defines methodology generic to all controllers implementation, I'm not sure if we able to explicitly specify/recommend any particular message. We will provide some examples for reference.

5)      6.1.4 and 6.1.5 Reactive and Proactive Path Provisioning time. Test procedure says about sending single traffic stream between end nodes. But it does not say about traffic stream parameters  (frame size, rate, IPv4 or something else etc.). Would be good to specify it  IMHO (as I mentioned above in 4.2 are not many details in current draft) . Another option could be just comment that these parameters should be defined before testing by customer itself.
[Bhuvan] Typically controller provisions the network path for a flow in the SDN nodes (switches) when it sees the first packet of a flow (reactive). Once the path is provisioned, all the subsequent packets are forwarded by the SDN node (e.g., switch) based on the provisioned flow. Here we are measuring the time that the controller takes to pave the entire path for a flow in the SDN nodes. So I believe that the packet size or the rate may not influence the test.

[Bhuvan] Could you please provide some more clarification about this comment?

[BKH]  I meant the following, in 6.2.1 maximum number of control sessions will be found, in 6.2.2 network size is measured, in 6.2.3 maximum number of flow entries. But everything is measured separately (single dimension). In real life customer may have  big network (many nodes)  so he will have maximum number of control sessions, maximum number of flows and big network size simultaneously (multidimensional (MD) case). That is why I personally would propose to create such MD test case.

Thank you.

SY,
Boris

From: bmwg [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Sarah Banks
Sent: Sunday, August 02, 2015 7:08 AM
To: Marius Georgescu
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Thanks
Sarah

Dear Bhuvan et alia,

Best regards,
Marius

Dear BMWG Members,

Summary of Changes:

Thanks,
Authors

---

From: "Bhuvan \(Veryx Technologies\)" <[email-masked]>
Date: Mon, 10 Aug 2015 22:19:45 +0530

Hi Boris,

Thanks for the clarification.

I really like the proposal of having an MD test case for scalability.  But
at the same time, the measurement/result comparison would be difficult when
multiple dimensions vary simultaneously . Also there may be chances that the
result of one dimension could influence the result of other.

Best regards,

Bhuvan

From: Khasanov Boris [mailto:[email-masked]] 
Sent: 07 August 2015 18:09
To: Bhuvan (Veryx Technologies)
Cc: [email-masked]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

Hi Bhuvan,

Thank you.

SY,

Boris

From: Bhuvan (Veryx Technologies)
[mailto:[email-masked]] 
Sent: Friday, August 07, 2015 2:32 PM
To: Khasanov Boris
Cc: [email-masked]; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance
- Updated Draft Version

Hi Boris,

Best regards,

Bhuvan

Hi Sarah and all,

1)

2)

 

3)

[BKH]  I meant the following, in 6.2.1 maximum number of control sessions
will be found, in 6.2.2 network size is measured, in 6.2.3 maximum number of
flow entries. But everything is measured separately (single dimension). In
real life customer may have  big network (many nodes)  so he will have
maximum number of control sessions, maximum number of flows and big network
size simultaneously (multidimensional (MD) case). That is why I personally
would propose to create such MD test case.

Thank you.

SY,

Boris

Thanks

Sarah

Dear Bhuvan et alia,

Best regards,

Marius 

Dear BMWG Members,

Summary of Changes:

Thanks,

Authors

---

From: "Tassinari, Mark" <[email-masked]>
Date: Mon, 10 Aug 2015 18:34:59 +0000

Hi Boris,

I also agree this this is an excellent test.  However, as Bhuvan pointed out testing multiple vectors at their limit would be difficult to benchmark because of the multiple vectors.  We have found that any one vector can be maximized at the cost of the other two but running all three at the limit is a challenge.  This also invites the real possibility of exhausting CPU and/or memory which can impact those results.

Regards,
Mark

From: bmwg [mailto:[email-masked]] On Behalf Of Bhuvan (Veryx Technologies)
Sent: Monday, August 10, 2015 9:50 AM
To: 'Khasanov Boris'
Cc: [email-masked]
Subject: Re: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Boris,

Thanks for the clarification.
I really like the proposal of having an MD test case for scalability.  But at the same time, the measurement/result comparison would be difficult when multiple dimensions vary simultaneously . Also there may be chances that the result of one dimension could influence the result of other.

Best regards,
Bhuvan

Cc: [email-masked]<mailto:[email-masked]>; 'Sarah Banks'

Hi Bhuvan,

Thank you.

SY,
Boris

From: Bhuvan (Veryx Technologies) [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Friday, August 07, 2015 2:32 PM
To: Khasanov Boris
Cc: [email-masked]<mailto:[email-masked]>; 'Sarah Banks'
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Boris,

Best regards,
Bhuvan

Hi Sarah and all,

Thank you.

SY,
Boris

Thanks
Sarah

Dear Bhuvan et alia,

Best regards,
Marius

Dear BMWG Members,

Summary of Changes:

Thanks,
Authors

---

From: Khasanov Boris <[email-masked]>
Date: Tue, 25 Aug 2015 11:30:55 +0000

Hi Mark, Bhuvan
This is clear that such test will be challenging. This is just my personal opinion taken from testing HW based high-end routers.

Thank you.

SY,
Boris

From: Tassinari, Mark [mailto:[email-masked]]
Sent: Monday, August 10, 2015 9:35 PM
To: Bhuvan (Veryx Technologies); Khasanov Boris
Cc: [email-masked]
Subject: RE: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version

Hi Boris,

Regards,
Mark

Hi Boris,

Best regards,
Bhuvan

Hi Bhuvan,

Thank you.

SY,
Boris

Hi Boris,

Best regards,
Bhuvan

Hi Sarah and all,

Thank you.

SY,
Boris

Thanks
Sarah

Dear Bhuvan et alia,

Best regards,
Marius

Dear BMWG Members,

Summary of Changes:

Thanks,
Authors