Subject: [bmwg] A few more comments on SIP meth-00 draft

From: Al Morton <[email-masked]>
Date: Tue, 31 Oct 2006 15:33:00 -0500

Intro:
>      ...   Session Setup
>   Rate could possibly be impacted by the selected value for Maximum
>   Sessions Attempted.  The benchmark for Session Setup Rate is
>   measured with a fixed value for Maximum Sessions Attempted.

It seems to me that this detail is worthy of more discussion
in the methodology section.  Can the Max Sessions limit cause the
offered setup rate to be incorrect?  It's not completely clear what
the "impact" of setting Max Sessions Attempted will be
(especially at this point, in the Intro).

>3. Test Setup
>
>    3.1 Test Topologies
>
>    Figures 1 through 5 below provide various topologies to perform
>    the SIP Performance Benchmarking.  These figures show the Device
>    Under Test (DUT) to be a single server or a System Under Test (SUT).

Perhaps these Figures are very simplified, with respect to how SIP
messages flow between the tester and the DUT/SUT.  IOW, they seem
sufficient for Registration, but not when the DUT is a Proxy Server.
It might be more efficient to discuss this when we're face-to-face.

-----------

I suggest adding the units of measurement in the reporting format:

>    3.3 Reporting Format
>
>       Test Setup
>         SIP Transport Protocol      = ___________
>         Session Duration            = ___________seconds
>         Attempted Session Rate      = ___________sessions/sec
>         Maximum Sessions Attempted  = ___________sessions
>         Media Streams per Session   = ___________streams/session
>         Media Protocol              = ___________
>
>       Device Benchmarks
>         Failed Session Attempts            = ___________attempts
>         Session Capacity                   = ___________sessions
>         Zero-Failure Session Setup Rate    = ___________sessions/sec
>         Maximum Retransmits                = ___________
>         Mean Session Setup Delay           = ___________msec
>         Mean Session TearDown Delay        = ___________msec

>    4.6 Session Setup Rate with Media
>
>...
>       2. Configure Tester for SIP UDP with an Attempted Session Rate =
>          100 SPS, Session Duration = 30 sec, Maximum Sessions Attempted
>          = 100,000 and media streams per session = 1.  The rate of
>          offered load for each media stream SHOULD be
>
>          (eq 1) Offered Load per Media Stream =
>                 Throughput / Maximum Sessions Attempted,
>                     where Throughput is defined in [Ba99].

Not sure if we can use RFC 1242/2544 Throughput definition here,
since we're not checking for packet loss on the DUT output.

from 2544:
    "The throughput is the fastest rate at which the count of test frames
    transmitted by the DUT is equal to the number of test frames sent to
    it by the test equipment."

Same comment on
>    4.7 Session Capacity

and also:
>       5. If a Failed Session Attempt is recorded then reduce the
>          Maximum Sessions Attempted configured on the Tester by 5,000.
>
>       6. If no Failed Session Attempt is recorded then increase the
>          Maximum Sessions Attempted configured on the Tester by 10,000.

This seems to be a rather high granularity for iterations, perhaps the
test operator should have the option to modify these values.

Same comment on
>    4.8 Session Capacity with Media

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg