Subject: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

From: Eric Rescorla <[email-masked]>
Date: Tue, 17 Apr 2018 14:14:12 -0700

Eric Rescorla has entered the following ballot position for
draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Rich version of this review at:
https://mozphab-ietf.devsvcdev.mozaws.net/D3948

COMMENTS
>      reported.
>   
>   4.7. Test Repeatability
>   
>      To increase the confidence in measured result, it is recommended
>      that each test SHOULD be repeated a minimum of 10 times.

Nit: you might be happier with "RECOMMENDED that each test be repeated
..."

Also, where does 10 come from? Generally, the number of trials you
need depends on the variance of each trial.

>      Test Reporting
>   
>      Each test has a reporting format that contains some global and
>      identical reporting components, and some individual components that
>      are specific to individual tests. The following test configuration
>      parameters and controller settings parameters MUST be reflected in

This is an odd MUST, as it's not required for interop.

>      5. Stop the trial when the discovered topology information matches
>        the deployed network topology, or when the discovered topology
>        information return the same details for 3 consecutive queries.
>      6. Record the time last discovery message (Tmn) sent to controller
>        from the forwarding plane test emulator interface (I1) when the
>        trial completed successfully. (e.g., the topology matches).

How large is the TD usually? How much does 3 seconds compare to that?

>                                                   Total Trials
>   
>                                              SUM[SQUAREOF(Tri-TDm)]
>      Topology Discovery Time Variance (TDv)  ----------------------
>                                                  Total Trials -1
>   

You probably don't need to specify individual formulas for mean and
variance. However, you probably do want to explain why you are using
the n-1 sample variance formula.

>   
>   Measurement:
>   
>                                              (R1-T1) + (R2-T2)..(Rn-Tn)
>      Asynchronous Message Processing Time Tr1 = -----------------------
>                                                          Nrx

Incidentally, this formula is the same as \sum_i{R_i} - \sum_i{T_i}

>      messages transmitted to the controller.
>   
>      If this test is repeated with varying number of nodes with same
>      topology, the results SHOULD be reported in the form of a graph. The
>      X coordinate SHOULD be the Number of nodes (N), the Y coordinate
>      SHOULD be the average Asynchronous Message Processing Time.

This is an odd metric because an implementation which handled overload
by dropping every other message would look better than one which
handled overload by queuing.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 17 Apr 2018 22:20:18 +0000

Eric, a quick reply on your last comment, below.
Al
doc shepherd

...
> handled overload by queuing.
[acm] 
If processing time were the only number reported, you're right.

Although the early generation of controller benchmarking tools
overlooked the important combinations of metrics,
the Reporting Format adds the success/loss message performance:

   The report should capture the following information in addition to
   the configuration parameters captured in section 5.

   - Successful messages exchanged (Nrx)

   - Percentage of unsuccessful messages exchanged, computed using the
   formula (1 - Nrx/Ntx) * 100), Where Ntx is the total number of
   messages transmitted to the controller.

BUT, it would be better if SHOULD or RECOMMENDED terms were used,
to cover the case you identified.

> 
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=8vh-
> jwFdXG0pU1mgRW3_HbkfU1msplWbXuJZesLmozs&s=mfmCArWKOyMJgNPTRNNf467DrHNCKxkp
> QVKXcOCQKQA&e=

---

From: [email-masked]
Date: Wed, 18 Apr 2018 08:26:55 +0000

Hi Eric,
Thank you for your feedback on this draft. Please find below my response inline with tag [Bhuvan] 

@Al, Thanks for taking care of Eric's last feedback.
Best regards,
Bhuvan

On Wed, Apr 18, 2018 at 02:44 AM, Eric Rescorla  wrote:
Eric Rescorla has entered the following ballot position for
draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html (https://www.ietf.org/iesg/statement/discuss-criteria.html)
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/ (https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/)

Rich version of this review at:
https://mozphab-ietf.devsvcdev.mozaws.net/D3948 (https://mozphab-ietf.devsvcdev.mozaws.net/D3948)

COMMENTS
reported.

4.7. Test Repeatability

...."

Also, where does 10 come from? Generally, the number of trials you
need depends on the variance of each trial.
[Bhuvan] The RECOMMENDED number 10 was arrived based on our experience during the benchmarking. I will discuss with other authors for changing SHOULD to RECOMMENDED. 
Test Reporting

This is an odd MUST, as it's not required for interop.[Bhuvan] The intent of specifying MUST is to capture relevant test parameters to enable Apple to Apple comparison of test test results across two testers/test runs. 

How large is the TD usually? How much does 3 seconds compare to that?
[Bhuvan] The test duration varies depends on the size of the test topology. For a smaller topology (3 - 10) the TD was within a minute. So we kept the query interval of 3 seconds to accomodate smaller and larger topologies.
Total Trials

SUM[SQUAREOF(Tri-TDm)]

[Bhuvan] We have added both formulas based on the feedback received in the mailing list.  We are using n-1, as it is commonly used variance measure. Do we need an explanation here or providing any reference to this is sufficient?
Measurement:

Incidentally, this formula is the same as sum_i{R_i} - sum_i{T_i} 

[Bhuvan]  Good suggestion, we will incorporate in the next revision. 

[Bhuvan] Believe Al has clarified this feedback. As suggested by Al, we will update the BCP language in the reporting section.
_______________________________________________
bmwg mailing list
[email-masked] (mailto:[email-masked])
https://www.ietf.org/mailman/listinfo/bmwg (https://www.ietf.org/mailman/listinfo/bmwg)

DISCLAIMER: Privileged and/or Confidential information may be
contained in this message. If you are not the addressee of this message,
you may not copy, use or deliver this message to anyone. In such
event,you should destroy the message and kindly notify the sender by
reply e-mail.
It is understood that opinions or conclusions that do not relate to the
official business of the company are neither given nor endorsed by the
company.

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 18 Apr 2018 05:56:48 -0700

On Wed, Apr 18, 2018 at 1:26 AM, <[email-masked]>
wrote:

> On Wed, Apr 18, 2018 at 02:44 AM, Eric Rescorla <[email-masked]> wrote:
>
> [Bhuvan] The RECOMMENDED number 10 was arrived based on our experience
> during the benchmarking. I will discuss with other authors for changing
> SHOULD to RECOMMENDED.
>
>
The SHOULD versus RECOMMENDED has the same normative force in 2119. It's
just editorial and I thought it
would read better.

Test Reporting
> [Bhuvan] The intent of specifying MUST is to capture relevant test
> parameters to enable Apple to Apple comparison of test test results across
> two testers/test runs.
>

OK

> [Bhuvan] The test duration varies depends on the size of the test
> topology. For a smaller topology (3 - 10) the TD was within a minute. So we
> kept the query interval of 3 seconds to accomodate smaller and larger
> topologies.
>
>
So, 3 seconds is a pretty big fraction of that. It introduces non-trivial
random (I think) error.

As for n-1, I *think* it's the right one here, but I'm not sure. It's what
you use for "sample variance" typically. Have you talked to a statistician?

-Ekr

Total Trials
> [Bhuvan] We have added both formulas based on the feedback received in the
> mailing list.  We are using n-1, as it is commonly used variance measure.
> Do we need an explanation here or providing any reference to this is
> sufficient?
>
>
Well, my point was that you could specify mean and variance in one place
and not repeat them over and over

Well, n-1 is typically used for sample variance. This is something a little
different.

> [Bhuvan] Believe Al has clarified this feedback. As suggested by Al, we
> will update the BCP language in the reporting section.
> https://www.ietf.org/mailman/listinfo/bmwg
>
> DISCLAIMER: Privileged and/or Confidential information may be contained in
> this message. If you are not the addressee of this message, you may not
> copy, use or deliver this message to anyone. In such event,you should
> destroy the message and kindly notify the sender by reply e-mail. It is
> understood that opinions or conclusions that do not relate to the official
> business of the company are neither given nor endorsed by the company.
>