Subject: [Sipping] SIP Performance Metrics

From: Al Morton <[email-masked]>
Date: Thu, 14 Sep 2006 16:20:29 -0400

Hi Daryl,

I finally started through your latest draft:
http://www.ietf.org/internet-drafts/draft-malas-performance-metrics-04.txt
here's my comments so far.

Al

1.  Introduction
--- Your current text comes close to defining the Scope,
but doesn't quite nail it down, although I think you've done
a good job of saying what is out of scope. Let me suggest
revised wording and re-ordering of points that might be more
straight-forward.  I think there may be some more tweaking of
the Scope, but that process might go more quickly with this
sort of organization.
==============================================================================

1. Introduction and Scope

    SIP has become a widely-used standard among many service providers,
    vendors, and end users.  Although there are many different standards
    for measuring the performance of signaling protocols, none of them
    specifically address SIP.

    The scope of this document is limited to the definitions of a standard
    set of metrics for measuring and reporting SIP performance from an
    end-to-end perspective.  The metrics introduce a common foundation for
    understanding and quantifying performance expectations between
    service providers, vendors, and the users of services based on SIP.

    Measurements of the metrics described in this document are affected
    by variables external to SIP.  The following is a non-exhaustive list
    of examples:

      . Network connectivity

      . Switch and router performance

      . Server processes and hardware performance

    Note that some metrics in this document may not apply to all
    applications of SIP.  This document provides an overview of pertinent
    metrics, which may be used individually or as a set based on the
    usage of SIP within the context of a given service.

    The metrics defined in this document DO NOT take into consideration
    the impairment or failure of actual application processing of a
    request or response.  The metrics do not distinguish application
    processing time from other sources of delay, such as packet transfer
    delay.

    Metrics designed to quantify single device application processing
    performance are beyond the scope of this document.

    This document does not provide any numerical objectives or acceptance
    threshold values for the SIP performance metrics defined below, as
    these items are beyond the scope of IETF activities, in general.

=============================================================================
Section 2. Terminology

The definitions Time Begin (TB) and Time Stop (TS) could be cast as
externally observable events, so that a passive monitoring system
could make the measurements while a typical UA1 participates.

I suggest:

    Time Begin (TB) - This is the time instant that starts a continuous
    time interval running until the related response is received.  TB occurs
    when the designated request has been processed by the application and
    last bit of the request packet has been sent from the proxy or UA
    (and is externally observable at some physical interface).

    Time Stop (TS) - This is the time instant that ends a continuous
    time interval running from when the related request is sent.  TS occurs
    when the last bit of the designated response is received at the
    requesting device (and is externally observable at some physical
    interface).

============================================================================

Section 3, first paragraph:
---  I'm struggling for the meaning of the following sentence:

      ... In regards to all of the following metrics, message
    re-transmissions must be excluded in order to provide accurate metric
    results.

---  Let's say the first INVITE msg packet is lost, so it's not observed
      crossing the UA2 interface.  The re-transmitted INVITE is successful:

                UA1                    UA2
                 |                      |
                 |INVITE                |
          TB---->|-------------X        |
             /\  |                      |
             ||  |                      |
             ||  |INVITE                |
             ||  |--------------------->|
            SRD  |                      |
             ||  |                      |
             ||  |                      |
             \/  |                   180|
          TS---->|<---------------------|
                 |                      |

---- Is this the correct TB and TS for SRD, depicted above?
IOW, the second INVITE is a re-transmission (determined somehow)
and ignored.
(If so, then it may be sufficient to say that the first copy of a SIP
message observed at UA1 is used to determine the TB or TS)

Or, does this get into the impairment/failure cases that you said are
out of scope?

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Daryl Malas <[email-masked]>
Date: Thu, 14 Sep 2006 15:40:27 -0600

Al, thanks for the feedback...comments in-line:

> the Scope, but that process might go more quickly with this
I'll take a look at this, as more clarity is always good.
I suppose this is an applicable solution, although I think the results
will be somewhat inaccurate if you are measuring from points downstream
from the UAC or upstream from the UAS, because it will not depict the
actual UA experience of the duration.  Using this method, the signaling
would still have a "length" of network to traverse as well as layer 1 -
3 processing before layer 5.
The reason the first INVITE is used; it describes UA1's perspective of
when the call began.  Think of PDD, since this is what SRD is adapted
from.  As soon as someone finishes typing in their TN or URI, the wait
begins for ring-back or a connect.  They are oblivious to whether or not
multiple retransmissions take place or not.  I believe the TB is correct
in this case.  I will review all of the TB and TS's throughout the
document to confirm their validity. --Daryl

---

From: Daryl Malas <[email-masked]>
Date: Fri, 15 Sep 2006 11:34:12 -0600

Al,

Please see my comments in-line...

> the Scope, but that process might go more quickly with this
I accepted your changes with this...Thanks!!
I accepted your changes with this...Thanks!!
I have changed the paragraph to read like this:

"In regards to all of the following metrics, TB begins with the first
SIP message sent by the UAC, and is not reset if the UAC must retransmit
the same request multiple times.  The first SIP message indicates the TB
associated with the user or application TB expectation associated with
the request."

Let me know if this clarifies this.  Thanks!!

---

From: Al Morton <[email-masked]>
Date: Fri, 15 Sep 2006 17:57:59 -0400

Daryl, I'm glad the comments were helpful.

>Let me know if this clarifies this.  Thanks!!

This is good, although we have to deal with the TS cases, too.
What if two 200OK arrive at UA1? (because an ACK was lost?)

Also, in the "Sum of REGISTER Requests" and "Sum of INVITE Requests",
it needs to be clear that the re-transmissions are not counted.

Al