Subject: [bmwg] Benjamin Kaduk's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 19 Apr 2018 05:22:11 -0700

Benjamin Kaduk has entered the following ballot position for
draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

In the Abstract:

   This document defines the methodologies for benchmarking control
   plane performance of SDN controllers.

Why "the" methodologies?   That seems more authoritative than is
appropriate in an Informational document.

Why do we need the test setup diagrams in both the terminology draft
and this one?  It seems like there is some excess redundancy, here.

In Section 4.1, how can we even have a topology with just one
network device?  This "at least 1" seems too low.  Similarly, how
would TP1 and TP2 *not* be connected to the same node if there is
only one device?

Thank you for adding consideration to key distribution in Section
4.4, as noted by the secdir review.  But insisting on having key
distribution done prior to testing gives the impression that keys
are distributed once and updated never, which has questionable
security properties.  Perhaps there is value in doing some testing
while rekeyeing is in progress?

I agree with others that the statistical methodology is not clearly
justified, such as the sample size of 10 in Section 4.7 (with no
consideration for sample relative variance), use of sample vs.
population veriance, etc.

It seems like the measurements being described sometimes start the
timer at an event at a network element and other times start the
timer when a message enters the SDN controller itself (similarly for
outgoing messages), which seems to include a different treatment of
propagation delays in the network, for different tests.  Assuming
these differences were made by conscious choice, it might be nice to
describe why the network propagation is/is not included for any
given measurement.

It looks like the term "Nxrn" is introduced implicitly and the
reader is supposed to infer that the 'n' represents a counter, with
Nrx1 corresponding to the first measurement, Nrx2 the second, etc.
It's probably worth mentioning this explicitly, for all fields that
are measured on a per-trial/counter basis.

I'm not sure that the end condition for the test in Section 5.2.2
makes sense.

It seems like the test in Section 5.2.3 should not allow flexibility
in "unique source and/or destination address" and rather should
specify exactly what happens.

In Section 5.3.1, only considering 2% of asynchronous messages as
invalid implies a preconception about what might be the reason for
such invalid messages, but that assumption might not hold in the
case of an active attack, which may be somewhat different from the
pure DoS scenario considered in the following section.

Section 5.4.1 says "with incremental sequence number and source
address" -- are both the sequence number and source address
incrementing for each packet sent?  This could be more clear.
It also is a little jarring to refer to "test traffic generator TP2"
when TP2 is just receiving traffic and not generating it.

Appendix B.3 indicates that plain TCP or TLS can be used for
communications between switch and controller.  It seems like this
would be a highly relevant test parameter to report with the results
for the tests described in this document, since TLS would introduce
additional overhead to be quantified!

The figure in Section B.4.5 leaves me a little confused as to what
is being measured, if the SDN Application is depicted as just
spontaneously installing a flow at some time vaguely related to
traffic generation but not dependent on or triggered by the traffic
generation.

---

From: Sarah B <[email-masked]>
Date: Thu, 19 Apr 2018 09:32:38 -0700

Hi Benjamin,
	Thanks for your review. A few comments inline below with SB//

Thanks
Sarah

Sure, we can remove this.

> while rekeyeing is in progress?

We'll discuss internally.. the goal here wasn't to test rekeying, which is why the test doesn't call for anything other than doing the distribution prior to the test, and remember, this is lab testing; this isn't what happens on a live network, but it's an interesting point, and we'll discuss.

> population veriance, etc.

As a draft, we aren't saying you only have to do 10; it's been a point of discussion in BMWG for some time, what number do you put? We haven't always found a happy medium to this. We're suggesting you perform the test 10 times, but there's nothing that stops a tester from making that <n> times; for example, 100 times. It's important to note how many times the test was performed and move on. We can clarify this text to reflect that, if it helps. However, note, BMWG is about repeatability, and that was the goal here.

> are measured on a per-trial/counter basis.
SB// OK

> additional overhead to be quantified!

SB// Agreed. The intent is to have the scenario specifically documented by the tester to reflect the choice made; we'll clarify the text.

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 19 Apr 2018 11:41:53 -0500

> Sure, we can remove this.

(I probably should have explicitly said that "a methodology" is
fine.)

> We'll discuss internally.. the goal here wasn't to test rekeying, which is why the test doesn't call for anything other than doing the distribution prior to the test, and remember, this is lab test…

Okay.  I'm just approaching this from a "social norms" perspective
(regarding expectations that re-keying will be done), but I
definitely understand that there is less need for benchmarking
rekeying performance than the cases already covered.

> As a draft, we aren't saying you only have to do 10; it's been a point of discussion in BMWG for some time, what number do you put? We haven't always found a happy medium to this. We're suggesting…

Well, there's a huge amount of literature (and, e.g., textbooks) on
statistical anlaysis and how to decide that a series of measurements
has converged on a stable result.  So it might be better to just say
that benchmarkers must ensure that the data has converged and note
that this will require running many repeated trials, but not
necessarily list a specific value.

-Benjamin

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 19 Apr 2018 12:02:45 -0500

This is Benjamin's ballot thread, but ...

On Thu, Apr 19, 2018 at 11:41 AM, Benjamin Kaduk <[email-masked]> wrote:

> > We'll discuss internally.. the goal here wasn't to test rekeying, which
> is why the test doesn't call for anything other than doing the distribution
> prior to the test, and remember, this is lab testing; this isn't what
> happens on a live network, but it's an interesting point, and we'll discuss.
>
> > As a draft, we aren't saying you only have to do 10; it's been a point
> of discussion in BMWG for some time, what number do you put? We haven't
> always found a happy medium to this. We're suggesting you perform the test
> 10 times, but there's nothing that stops a tester from making that <n>
> times; for example, 100 times. It's important to note how many times the
> test was performed and move on. We can clarify this text to reflect that,
> if it helps. However, note, BMWG is about repeatability, and that was the
> goal here.
>

FWIW, that's the kind of thing I was asking about in my own ballot. If 10
repetitions has been a good heuristic in testing so far, that could be
worth saying, but if people are going to have to look at results and figure
out whether they've repeated a test often enough, you might as well put it
in those terms.

IMO, of course.

Spencer