Subject: Re: [Gen-art] Second last call review of draft-ietf-bmwg-sip-bench-meth

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 03 Nov 2014 17:30:27 -0600

On Oct 2, 2014 Tom Taylor wrote:

 > Summary: Ready with minor issues and nits. I did not check out the
 > correctness of the code in Appendix A.

Tom: Thank you for your detailed review and your time on our draft.

Please see inline for resolutions.

> Minor issues:
> ------------
>
> (1) Sec. 4.10 paragraphs 3 and 4: I believe these are about setting
> the initial value for the attempt rate. The first paragraph offers the
> alternative of using a vendor-provided value R as a starting point or
> using r=R as a starting point. Doesn't seem like two alternatives.
>
> The second paragraph ignores all this and sets r=100 as the starting
> point. But then it repeats text about using a vendor-supplied value.
>
> The two paragraphs need cleaning up.

This is a very good comment.  Here is what I propose:

OLD:

    If the DUT vendor provides a value for R, the tester can use this
    value.  Alternatively, in cases where the DUT vendor does not provide
    a value for R, or in cases where the tester wants to ascertain a
    vendor provided value using local media characteristics, the
    algorithm could be run by setting "r = R" and observing the value at
    convergence.

    The algorithm proceeds by initializing "r = 100"; "r" is the session
    attempt rate.  The algorithm dynamically increases and decreases "r"
    as it converges to the a maximum sps value for R. The dynamic
    increase and decrease rate is controlled by the weights "w" and "d",
    respectively.  If the DUT vendor provides a value for R, the tester
    can use that value; however, because the requirements and media
    characteristics are a function of the test environment, it is best
    that the tester reflect these requirements during testing and allow
    the algorithm to converge to R.

NEW:

    If the DUT vendor provides a value for R, the tester can use this
    value to initialize "r = R", and proceed with the algorithm.  In
    cases where the DUT vendor does not provide a value for R, or in
    cases where the tester wants to ascertain a vendor provided value
    using local media characteristics, it is best that the tester
    reflect these requirements during testing and allow the algorithm to
    converge to R.  In such cases, the algorithm proceeds by initializing
    "r = 100";  "r" is the session attempt rate.  The algorithm
    dynamically increases and decreases "r" as it converges to the a
    maximum sps value for R. The dynamic increase and decrease rate is
    controlled by the weights "w" and "d", respectively.

> (2) The algorithm in 4.10 seems fairly unsophisticated, and amounts
> to ten iterations of constrained binary search. Was application of
> Newton's Method
>
> http://www.ima.umn.edu/%7Emiller/Newton-MVT.pdf
>
> to give you faster convergence and shorter test runs considered?

Very pertinent feedback (thanks!).  Pursuant to our offline
conversations, I believe that the algorithm as specified in bmwg-sip-
bench-meth-11 does not need to be modified.  In the end, the algorithm
in the draft is a version of gradient descent, or in our case, ascent.
Such algorithms may not be fast, but they do find the maxima (minima)
on a bounded number of iterations.

An important reason why the algorithm in bmwg-sip-bench-meth-11 appears
in the form that is does is for the simple reason that the tester may
not know R a-priori.  Recall that R is a SIP session attempt rate,
which the DUT can process with zero errors over a defined, extended
period.

Our draft and the algorithm takes the conservative approach that the
tester may not be given the value of R by the vendor a-priori, or
the tester may independently want to approximate R to double-check
vendor claim using locally pertinent media characteristics.

The binary search method depends on knowing the value of R, since that
will become a ceiling to the process.  The Newton (N-R) method also
depends on knowing the value of R, since to get N-R going, we would
start the first run at the claimed steady-state (R) capacity.  I have
no doubt that we can make binary search or N-R work such that R is not
known a-priori. But by doing so I suspect that we'd be close to the
gradient descent that we have now.

Based on the simulation results for a binary search I shared with
you, I suggest adding a sentence in the revision to bench-meth as
follows:

OLD:
    The pseudo-code corresponding to the description above follows, and a
    simulation written in the R statistical language is provided in
    Appendix A.

NEW:
    The pseudo-code corresponding to the description above follows, and a
    simulation written in the R statistical language is provided in
    Appendix A.  We note that it is possible to have other algorithms
    that converge faster to R than the one shown below (e.g., an
    algorithm based on a binary search), however, such algorithms may not
    produce an optimal value for R, that is, they may sacrifice some
    accuracy for speed.  Such algorithms are for further study.

> Nits/editorial comments:
> -----------------------
>
> IDNits requests that the code in Appendix A be preceded by a <CODE
> BEGINS> line and terminated by a <CODE ENDS> line.
>
> Sec. 3 para 2 last line: s/arrangements/arrangement/

Got it.  Thanks!

> Next para: spell out EA as presumably "Emulated Agents" on first
> use?Term is from the terminology document.

Done.

> Add <CODE BEGINS> and <CODE ENDS> around the pseudo-code in 4.10.
>
> Reference [I-D.sip-bench-term] is now dated July, 2014 (version -11).

Good catch.  Updated.

I will skip the <CODE BEGINS> and <CODE ENDS> nits, if it is okay with
you.

The results will show up in -12 that will be released shortly.  Thanks
a lot, Tom, for your insights and feedback on this work.

Cheers,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA)
Email: vkg@{bell-labs.com,acm.org} / [email-masked]
Web: http://ect.bell-labs.com/who/vkg/  | Calendar: http://goo.gl/x3Ogq

---

From: joel jaeggli <[email-masked]>
Date: Mon, 03 Nov 2014 17:29:53 -0800

Thanks guys.

joel

On 11/3/14 3:30 PM, Vijay K. Gurbani wrote:
>> Summary: Ready with minor issues and nits. I did not check out
>> the correctness of the code in Appendix A.
> 
>> Minor issues: ------------
>> 
>> (1) Sec. 4.10 paragraphs 3 and 4: I believe these are about
>> setting the initial value for the attempt rate. The first
>> paragraph offers the alternative of using a vendor-provided value
>> R as a starting point or using r=R as a starting point. Doesn't
>> seem like two alternatives.
>> 
>> The second paragraph ignores all this and sets r=100 as the
>> starting point. But then it repeats text about using a
>> vendor-supplied value.
>> 
> value.  Alternatively, in cases where the DUT vendor does not
> provide a value for R, or in cases where the tester wants to
> ascertain a vendor provided value using local media
> characteristics, the algorithm could be run by setting "r = R" and
> observing the value at convergence.
> 
> The algorithm proceeds by initializing "r = 100"; "r" is the
> session attempt rate.  The algorithm dynamically increases and
> decreases "r" as it converges to the a maximum sps value for R. The
> dynamic increase and decrease rate is controlled by the weights "w"
> and "d", respectively.  If the DUT vendor provides a value for R,
> the tester can use that value; however, because the requirements
> and media characteristics are a function of the test environment,
> it is best that the tester reflect these requirements during
> testing and allow the algorithm to converge to R.
> 
> NEW:
> 
> reflect these requirements during testing and allow the algorithm
> to converge to R.  In such cases, the algorithm proceeds by
> initializing "r = 100";  "r" is the session attempt rate.  The
> algorithm dynamically increases and decreases "r" as it converges
> to the a maximum sps value for R. The dynamic increase and decrease
> rate is controlled by the weights "w" and "d", respectively.
> 
>> (2) The algorithm in 4.10 seems fairly unsophisticated, and
>> amounts to ten iterations of constrained binary search. Was
>> application of Newton's Method
>> 
> conversations, I believe that the algorithm as specified in
> bmwg-sip- bench-meth-11 does not need to be modified.  In the end,
> the algorithm in the draft is a version of gradient descent, or in
> our case, ascent. Such algorithms may not be fast, but they do find
> the maxima (minima) on a bounded number of iterations.
> 
> An important reason why the algorithm in bmwg-sip-bench-meth-11
> appears in the form that is does is for the simple reason that the
> tester may not know R a-priori.  Recall that R is a SIP session
> attempt rate, which the DUT can process with zero errors over a
> defined, extended period.
> 
> Our draft and the algorithm takes the conservative approach that
> the tester may not be given the value of R by the vendor a-priori,
> or the tester may independently want to approximate R to
> double-check vendor claim using locally pertinent media
> characteristics.
> 
> The binary search method depends on knowing the value of R, since
> that will become a ceiling to the process.  The Newton (N-R) method
> also depends on knowing the value of R, since to get N-R going, we
> would start the first run at the claimed steady-state (R) capacity.
> I have no doubt that we can make binary search or N-R work such
> that R is not known a-priori. But by doing so I suspect that we'd
> be close to the gradient descent that we have now.
> 
> OLD: The pseudo-code corresponding to the description above
> follows, and a simulation written in the R statistical language is
> provided in Appendix A.
> 
> NEW: The pseudo-code corresponding to the description above
> follows, and a simulation written in the R statistical language is
> provided in Appendix A.  We note that it is possible to have other
> algorithms that converge faster to R than the one shown below
> (e.g., an algorithm based on a binary search), however, such
> algorithms may not produce an optimal value for R, that is, they
> may sacrifice some accuracy for speed.  Such algorithms are for
> further study.
> 
>> Nits/editorial comments: -----------------------
>> 
>> IDNits requests that the code in Appendix A be preceded by a
>> <CODE BEGINS> line and terminated by a <CODE ENDS> line.
>> 
> I will skip the <CODE BEGINS> and <CODE ENDS> nits, if it is okay
> with you.
> 
> The results will show up in -12 that will be released shortly.
> Thanks a lot, Tom, for your insights and feedback on this work.
> 
> Cheers,
> 
> - vijay