Subject: [bmwg] comments on draft-ietf-bmwg-sip-bench-term-00

From: Al Morton <[email-masked]>
Date: Wed, 06 May 2009 16:58:07 -0400

BMWG,

I have the following comments/suggestions on term-00.

Al
(as a bmwg participant)

----------------------------------------------------------------------

OLD
3.1.2.  Signaling Plane

    Definition:
       The control plane in which SIP messages [RFC3261] are exchanged
       between SIP Agents [RFC3261] to establish a connection for media
       exchange.

    Discussion:
       SIP messages are used to establish sessions in several ways:
       directly between two User Agents [RFC3261], through a Proxy Server
       ...

SUGGEST:
3.1.2.  Signaling Plane

    Definition:
       The plane in which SIP messages [RFC3261] are exchanged
          ^
       between SIP Agents [RFC3261] to establish a connection for media
       exchange.

    Discussion:
|     The Signaling Plane is sometimes considered part of the
|     Network Control Plane.
       SIP messages are used to establish sessions in several ways:
       directly between two User Agents [RFC3261], through a Proxy Server
       ...

-----------------------------------------------

OLD
3.1.3.  Media Plane

    Definition:
       The data plane in which one or more media streams and their
       associated media control protocols are exchanged after a media
       connection has been created by the exchange of signaling messages
       in the Signaling Plane.

    Discussion:
       Media may also be known as the the "payload" or "bearer channel".
       ...
SUGGEST:
3.1.3.  Media Plane

    Definition:
       The plane in which one or more media streams and their
          ^
       associated media control protocols are exchanged after a media
       connection has been created by the exchange of signaling messages
       in the Signaling Plane.

    Discussion:
|     The Media Plane is sometimes called the Data Plane.
       Media may also be known as the the "payload" or "bearer channel".
       ...

------------------------------------------------------

3.1.4.  Associated Media

    Definition:
       Media that corresponds to an 'm' line in the SDP payload of the
       Signaling Plane.

Comment:  This doesn't seem to define the term, it's the answer to the
question, "where can I find the specification for the media stream
controlled by this signalling message?"

Suggest:

3.1.4.  Associated Media
    Definition:
       Media (stream) that is the subject of a specific signaling plane
       exchange.

    Discussion:
       The specification of Associated Media may be found in the 'm' line
       in the SDP payload on the Signaling Plane.
       Any Media Protocol MAY be used.
       ...
----------------------------------------------------
OLD
3.1.6.  Session Attempt

    Definition:
       A SIP Session for which the Emulated Agent has sent the SIP INVITE
       or SUBSCRIBE NOTIFY and has not yet received a message response
       from the DUT/SUT.
SUGGEST:
3.1.6.  Session Attempt

    Definition:
       A SIP Session for which the Emulated Agent has sent the SIP INVITE
       or SUBSCRIBE NOTIFY and expects to receive a message response
                               ^^^^^^^^^^^^^^^^^^^
       from the DUT/SUT.

----------------------------------------------------------

OLD
3.1.7.  Established Session

    Definition:
       A SIP session for which the Emulated Agent acting as the UE/UA has
       received a 200OK message from the DUT/SUT.

SUGGEST:
move this section after 3.1.8 and 3.1.9, so that IS and NS are defined
before you read the "Established Session" Definition.
THEN, move the IS and NS Established Session Discussion text
to this Definition, as below:

3.1.7.  Established Session

    Definition:
       A SIP session for which the Emulated Agent acting as the UE/UA has
|     sent an INVITE or a SUBSCRIBE NOTIFY, and
       received a 200OK message from the DUT/SUT.

|     An IS is successfully established if the
|     following two conditions are met:
|     1.  Sess.sig is established by the end of Establishment Threshold
|         Time (c.f.  Section 3.3.3), and
|     2.  The media session described in the SDP body of the signaling
|         message is present.

|     An NS is successfully established if the Session Attempt via a
|     non-INVITE request results in the EA receiving a 2xx or 3xx reply
|     from the DUT/SUT before the expiration of the Establishment
|     Threshold timer (c.f., Section 3.3.3).

(the 2 paragraphs above are deleted from Discussion in IS and NS sections)

-------------------------------------------------------------
OLD
3.1.10.  Session Attempt Failure

    Definition:
       A session attempt that does not result in an Established Session.

    Discussion:
       The session attempt failure may be indicated by the following
       observations at the Emulated Agent:
       1.  Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session
           Attempt.
       2.  The lack of any received SIP response to a Session Attempt.
                       ***

QUESTION:  How is an attempt that results in a 180 classified?

----------------------------------------------------

OLD
3.1.11.  Standing Sessions

    Definition:
       The number of Sessions currently established on the DUT/SUT at any
       instant.

    Discussion:
       The number of Standing Sessions is influenced by the Session
       Duration and the Session Attempt Rate.  Benchmarks MUST be
       reported with the maximum and average Standing Sessions for the
       DUT/SUT.  In order to determine the maximum and average Standing
       Sessions on the DUT/SUT for the duration of the test it is
       necessary to make periodic measurements of the number of Standing
       Sessions on the DUT/SUT.  The recommended value for the
       measurement period is 1 second.

QUESTIONS:
A counter of the number of sessions can be read in an instant,
so why is there a recommended measurement period?
Is this how frequently the sessions counter should be read?

On the other hand, the Discussion may be implying that there are
actually three terms being defined: an instantaneous count of
Standing Sessions, the Average Standing Sessions (over a time interval
with many readings of Standing Sessions), and the Maximum Standing Sessions
(over a time interval with many readings of Standing Sessions).

To clarify Standing Sessions, SUGGEST:
    Definition:
       The count of Sessions established on the DUT/SUT at any instant.

New terms and defs needed for Average and Max.

-------------------------------------------------------------

3.3.1.  Session Attempt Rate

    Definition:
       Configuration of the Emulated Agent for the number of sessions
       that the Emulated Agent attempts to establish with the DUT/SUT
       over a specified time interval.

    Discussion:
       The Session Attempt Rate can cause variation in performance
       benchmark measurements.  Since this is the number of sessions
       configured on the Tester, some sessions may not be successfully
       established on the DUT.  A session may be either an IS or an NS.

    Measurement Units:
       Session attempts per second

Comment:

I see some wiggle-room here for interpretation that might have
a significant effect on the results.  Suppose one tester generates
all Session Attempts with equal spacing and linearly maintains the
attempt rate even when viewed at sub-second intervals.
Another tester bursts all Session Attempts at the line rate,
and then waits for the beginning of the next second to produce
another Attempt burst.

I imagine that the DUT performance may differ under these circumstances,
and suggest to tighten this wording if all agree this is an issue.

------------------------------------------------

OLD
3.3.3.  Establishment Threshold Time
SUGGEST
3.3.3.  Establishment Failure Time Threshold

-----------------------------------------------------

OLD
3.3.6.  Media Session Hold Time

    Definition:
       Configuration of the Emulated Agent for the amount of time
       measured at the Emulated Agent for the Associated Media for an
       Established Session of type IS media to flow between the Emulated
       Agent and the DUT/SUT.

Comment:  That's a tough definition to read. Let me try...
SUGGEST
    Definition:
       The time duration of Associated Media (for a specific IS),
       flowing from an EA to the DUT/SUT, as measured at the EA.

-------------------------------------------------------

I also have some comments to type up on section 3.4, Benchmarks.
but they will have to wait...

---

From: Al Morton <[email-masked]>
Date: Thu, 07 May 2009 13:51:09 -0400

At 04:58 PM 5/6/2009, Al Morton wrote:
><snip>
>I also have some comments to type up on section 3.4, ...

I'm interested in some feedback from the WG, especially the
author team, before proceeding to suggest more changes like
the ones below.

In section 3.4:

- There are several benchmarks which attempt to discern
the largest value from among a set of values.
- The values themselves may be time averages (attempts per second)
or simple counts, and metric definitions are needed to
produce the values.

IMO, the terms and definitions should clearly distinguish between
these two aspects, because it helps them to be clear and
unambiguous.

<rant>
When I step back for a moment and take a fresh look at term names,
it's always been a rough point that we have metrics with simple
names, like "Throughput", which is actually the maxima of
offered load under lossless conditions, obtained over multiple iterations.
It's really "THE Throughput Benchmark" [RFC1242, RFC2544].
Just saying "throughput" and ignoring all the other existing
definitions of this term can certainly lead to confusion
beyond bmwg-list, or beyond the benchmarking community in general.
We might avoid some confusion with SIP device benchmarking now,
if we try. We might also produce some SIP metrics that are useful
*beyond benchmarking*.
</rant>

So, I propose to organize all of section 3.4 to have
pairs of sections: a metric definition, and then the benchmark
based on that metric.

OLD
3.4  Benchmarks
3.4.1.  Registration Rate

    Definition:
       The maximum number of registrations that can be successfully
       completed by the DUT/SUT in a given time period.

    Discussion:
       This benchmark is obtained with zero failure in which 100% of the
       registrations attempted by the Emulated Agent are successfully
       completed by the DUT/SUT.  The maximum value is obtained by
       testing to failure.  This means that the registration rate
       provisioned on the EA is raised progressively until a registration
       attempt failure is observed.

    Measurement Units:
       registrations per second (rps)

    Issues:
       None.

    See Also:

SUGGEST:
3.4  Metrics and Benchmarks

3.4.1.  Registration Rate

    Definition:
       The number of registrations successfully
       established by the DUT/SUT in a given time period.
       ^^^^^^^^^^^
    Discussion:
|     The time period is defined at the EA as the time basis
|     for outgoing registrations that are eventually successful.
|     However, time is also needed for the EA to evaluate the success
|     or failure of each registration attempt, thus there are really
|     two time intervals involved.

    Measurement Units:
       registrations per second (rps)

    Issues:
       None.

    See Also:

NEW:
3.4.2.  Maximum Registration Rate Benchmark

    Definition:
       The maximum Registration Rate accompanied by zero Ineffective
       Registration Attempts [Malas] in a given time period.

    Discussion:
       This benchmark is obtained with zero failure in which 100% of the
       registrations attempted by the Emulated Agent are successfully
       completed by the DUT/SUT.  The maximum value is obtained by
       testing to failure over multiple trials at fixed Registration Rates.
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       This means that the registration rate
       provisioned on the EA is raised incrementally until a registration
                                       ^^^^^^^^^^^^^
       attempt failure is observed, and then decremented to find the
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       highest successful Registration Rate in a repeatable trial.
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    Measurement Units:
       registrations per second (rps)

    Issues:
       None.

    See Also:

where [Malas] is draft-ietf-pmol-sip-perf-metrics-