Subject: [bmwg] Comments on SIP term-02 and meth-00 drafts

From: Al Morton <[email-masked]>
Date: Mon, 30 Oct 2006 17:44:51 -0500

Scott, Vijay, and Carol,

Just a few comments as I read through...

(first, with my chairman's hat on)
The WG discussed the scope and purpose of this work at IETF-66, and
a few key points needing attention were captured in the minutes:
  - be more precise about the difference between device performance
    and network performance (division of the work with SIPPING).
  - be clear *who* will use the results of the benchmarking and how,
    otherwise there is a risk that no one will benefit.
So, I expected to see these points directly addressed in the drafts,
perhaps in a section titled "Scope and Purpose".
To be fair, I think the text of the Intro adequately addresses the
technological aspects of the Scope in terms of what DUT types
can be benchmarked, and what "planes" will be benchmarked, but our
discussion indicated that more specifics were needed.

Comments that follow are as a participant, term draft first
(original text quotes are indented below).
Al

>Intro:
P-CSCF is Proxy Call Session Control Function, right?
Need to spell this out.

>    3.1 Test Components
>        3.1.1 SIP Control Plane
>
>            Definition:
>            The logical plane in which SIP Signaling messages are
>            exchanged between SIP Agents.
>
>            Discussion:
>            SIP signaling messages are used to establish SIP signaling
>            sessions in several ways: ...

I think that SIP signaling messages are used to establish Media Sessions,
and may also spawn associated signaling exchanges.

>         3.1.2 SIP Media Plane

Isn't this just the Media Plane?

>            Definition:
>            The logical plane in which media streams, also known as
>            the "payload" or "bearer channel", established by the
>            SIP Signaling messages are exchanged.
>
>            Discussion:
>            The Media Plane is analagous to the Data Plane.
>            Packets for the SIP Control Plane and the SIP Media Plane
>            traverse different paths, which can produce variation
>            in performance.  For this reason it is necessary to
>            benchmark performance of the SIP Control Plane and the
>            SIP Media Plane.

Let me suggest this:

Discussion:
The Media Plane is analagous to the Data Plane, as described in
[reference to IGP dataplane convergence drafts].
SIP Control Plane packets and Media Plane packets may traverse
different paths and experience differences in packet transfer
performance.  Therefore, it is appropriate to benchmark both planes.

>         3.1.3 Emulated Agent
>
>            Definition:
>            Device in test topology that initates/responds to SIP
>            signaling as a session endpoint and sources/receives
>            associated media for established connections.

... for established sessions. (? why mix-in connection-oriented terms ?)

>         3.1.4 Session Server
>
>            Definition:
>            Device in test topology that acts as proxy between Emulated
>            Agents.  This device is either a DUT or component of a SUT.
>
>            Discussion:
>            The Session Server MUST be a RFC 3261 [Ro02] compliant
>            device.  It MAY be a Proxy Server, Session Border Controller
>            (SBC), or other type of device that is RFC 3261 compliant.

Question:  Is there more detail we can add here? Many organizations have
prepared compliance test suites, although IETF is not in the business
of granting certifications...

>         3.1.6 Invite-Initiated Control Session (IICS)
>
>            Definition:
>            A SIP signaling exchange that includes an initial SIP INVITE
>            message exchanged between Emulated Agent and DUT/SUT.
>
>            Discussion:
>
>            An Invite-Initiated Control Session (IICS) MAY have
>            associated media.  The inclusion of media is test case
>            dependent. An IICS may be in one of several different
>            states:  Attempting , Established, Disconnecting. These
>            states are distinguished as follows:
>
>            Attempting - the state after the invite is sent by or
>            received at the Tester and before the ack to that Invite
>            is received at or sent by the Tester. This definition
>            includes possible reinvites as well as redirects. It also
>            includes all Invites that are rejected for lack of
>            authentication information.
>
>            Established - the state after the 200 OK for the initiating
>            Invite is sent by or received at the Tester. This definition
>            includes possible reinvites as well as redirects.  It also
>            includes all invites that are rejected for lack of
>            authentication information.
>
>            Disconnecting - the state after a BYE is sent by or received
>            at the Tester.

The Discussion section is a part of the IICS Definition, IMO.

>        3.1.8 Registration
>
>            Definition:
>            A NIICS whose initial SIP message is a REGISTER request.
>
>            Discussion:
>            Registrations represent a percent of SIP network traffic.
Suggest:
s/a percent/some fraction/

>        3.2.1 SIP Transport Protocol
>
>...
>
>            Measurement Units:
>            TCP or UDP

These aren't really measurement units, more like allowed values
(others are listed, too).

>         3.2.2 Intended Session Duration
>
>            Definition:
>            Configuration on the Emulated Agent for time from an IICS
>            establishment

(the time that the EA receives 200 OK) ?

>            to BYE.

Replace with:  to the time that the EA sends the BYE. ?

>This is the duration of the
>            Established State.

>         3.2.3 Measured Session Duration
>
>            Definition:
>            Average measurement on the DUT/SUT for time from session
>            establishment to BYE.

Need to define one term at a time here, first Measured Session Duration,
then the Average Session Duration taken over many measurements.
Also, need to nail down the start and end time instants for the
Duration.

>         3.2.4 Session Attempt Rate
>
>            Definition:
>            Configuration on the Emulated Agent for number of
>            sessions to be established at the DUT per continuous
>            one-second intervals.

... for the rate of session *attempts* submitted to the DUT on a
per-second basis.

>         3.2.4.1 Media Session Attempt Rate
>
>            Definition:
>            Configuration on the Emulated Agent for number of
>            media sessions to be established at the DUT per
>            continuous one-second intervals.

similar revision to the Rate definition above.

>         3.2.4.2 NIICS Attempt Rate

similar revision to the Rate definition above.

>         3.3.1 Registration Rate
>
>            Definition:
>            Maximum number of registrations successfully
>            completed by the DUT/SUT.
>
>            Discussion
>            This benchmark is obtained with zero failure in which
>            100% of the registrations attempteded by the Emulated Agent
>            are successfully completed by the DUT/SUT.
>            The maximum value is obtained by testing to failure.
Suggest:  The maximum value is obtained by repeated trials and
           iterating the NIICS/Registration Rate.

Similar revision to
>         3.3.2 Session Rate
           ...obtained by repeated trials and
           iterating the session attempt Rate.

>         3.3.3 Session Capacity
>            Definition:
>            The maximum number of SIP sessions that the DUT/SUT
>            can simultaneously have established.
Suggest:  ...can be observed to simultaneously establish and maintain.

>         3.3.5 Session Setup Delay
>
>            Definition:
>            The average time for a session to establish.
>
>            Discussion:
>            Time is from the Emulated Agent to signal the first
>            INVITE.  Session Setup Delay MUST be measured for
>            every established session to calculate the average.
>            Session Setup Delay MUST be measured at the
>            Successful Setup Attempt Rate.

This is an Average over many sessions, which is requires
a definition for each measured Setup Delay.

It overlaps with Session Request Delay and its counterparts
in draft-malas-performance-metrics-05.txt.

If the Average is to be computed here, it should reference the
SRD definition in draft-malas-...

>         3.3.6 Session Teardown Delay
>
>            Definition:
>            The average time for a session to teardown.

Again, an average, which is requires
a definition for each measured Teardown Delay.
This time, the reference should be to Session Disconnect Delay
in draft-malas-performance-metrics-05.txt.

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Daryl Malas <[email-masked]>
Date: Tue, 31 Oct 2006 15:11:46 -0700

I just read through these two drafts, and I agree with Al's assessment
on the last two metrics: Session Setup Delay and Session Teardown Delay.
These overlap with the metrics in the draft-malas-perfor...-05.txt.
IMO, they seem very "end-to-end" in nature, as opposed to benchmarking a
single DUT/SUT.

Or...are you benchmarking them in the following manner:
Session Setup Delay: From the time a UAC (DUT) receives an INVITE to the
time it responds with a 18x message

Session Teardown Delay: From the time the DUT/SUT receives a BYE to the
time it is able to respond with a 200

--Daryl

---

From: Daryl Malas <[email-masked]>
Date: Tue, 31 Oct 2006 15:18:44 -0700

I see when I was editing my comments I messed up something...correction
in-line...sorry for any confusion.

On Tue, 2006-10-31 at 15:11 -0700, Daryl Malas wrote:
> Session Setup Delay: From the time a "UAS" (DUT) receives an INVITE to the
By the way, this is really reaching in my opinion...this metric implies
a dependency on a UAC...as described in the SIPPING draft.