Subject: [Sipping] Comments on draft-malas-performance-metrics-05.txt

From: "Hedayat, Kaynam" <[email-masked]>
Date: Sat, 14 Oct 2006 18:05:52 -0400

This is a very useful draft and is much needed in the industry. Some comments and suggestions:

- It is useful and more general if metrics are defined as seen at any point in the network and not just by the UAC and UAS. The general terminology will apply to UAC, UAS, and monitoring devices in the middle.

- The Session Establishment Rate seems to be defined as having a Total # Invite Requests as the divisor.  It looks more accurate if it is defined as total number of sessions with INVITE. For example if an endpoint consistently issues the invite twice (e.g. due to dropped packets or authentication required), but with all call successfully connecting, the value should be 100% vs. 50% (as seems to be suggested by the current text). 

- Session Completion Ratio state that it is only used when at least one proxy is involved, despite the fact that it can occur due to a failure of a UA to respond. Is this correct?

- Session Duration time for caller is time from receiving 200 response to call termination (sending BYE or receiving a BYE). The latter is not distinguished in the diagram. 

- Some metrics in the draft apply to both UAC and UAS. From those by nature some are measured differently by the UAC and the UAS. These metrics should be identified and defined. For example Session Duration time for the UAS is the time from sending 200 response to session termination time.

- Section 3 last paragraph: it would be helpful to outline the types of applications (e.g. SLA monitoring, Operational monitoring, troubleshooting and diagnostics, etc.).

- Generally we are not keen about averaging of statistics. The problem with average is that they hide problems. Instead we tend to favor what call Acceptable Quality Indexes which simply translate to percent of events that passed a threshold for quality. For example % of time where Session Request Delay was lower than 2 seconds. In many cases, not all, this is a better metric for setting thresholds and easier one for operations to understand. I suggest we review each metric and decide which aggregation applies to which one.

- I suggest the definition of the following metrics in the draft:
-- Initial Response Time - Metric for the caller. It is the time from INVITE to first SIP response (provisional or final OK, whatever it is).
-- Authentication Time - Isolating authentication related time (in call setup, registration etc.). It is the time from the first invite to the first provisional response for the last invite.
-- Call Setup Time - Metric for the caller and the called party. For the caller it is the time from INVITE to reception of 200 OK. For the called party it is the time from reception of INVITE to reception of ACK
-- Media Start Time - Metric for the caller and the called party. For the caller it is the time from INVITE to reception of first media packet. For called party it is the time from receiving INVITE to receiving first media packet
-- Post Pickup Delay - Metric for the called party. It is the time from sending a 200 response to receiving first media packet.
-- Network Effectiveness Ratio defined as (answered-calls + no-answer-calls, + busy-calls) / all_calls

Regards,
Kaynam 

-----------------------------------------
Kaynam Hedayat
Chief Technology Officer
Vice President, Engineering

Brix Networks
285 Mill Road
Chelmsford, MA 01824
tel. 978.367.5611
fax. 978.367.5700
[email-masked]
www.brixnet.com

Service Quality Matters. Test the performance and quality of your VoIP or IP video service at http://www.TestYourVoIP.com  <http://www.testyourvoip.com/>  and http://www.TestYourIPVideo.com <http://www.testyouripvideo.com/>  

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Sanjay Sinha \(sanjsinh\)" <[email-masked]>
Date: Tue, 31 Oct 2006 10:10:16 -0500

But according to RFC 3264, shouldn't the offerer be ready to receive
media for streams it is offering. So why is the cut-thru timing
important or any different than what is described in the rfc.

Sanjay 

>-----Original Message-----
>From: Dolly, Martin C, ALABS [mailto:[email-masked]] 
>Sent: Tuesday, October 31, 2006 8:59 AM
>To: Michael Hammer (mhammer); Gonzalo Camarillo; sipping
>Cc: [email-masked]
>Subject: RE: [Sipping] Comments on 
>draft-malas-performance-metrics-05.txt
>
>The timing of cut-thru is extremely important, as it relates 
>to tones and announcements that occur before answer. 
>
>-----Original Message-----
>From: Michael Hammer (mhammer) [mailto:[email-masked]]
>Sent: Tuesday, October 31, 2006 8:51 AM
>To: Gonzalo Camarillo; sipping
>Cc: [email-masked]
>Subject: RE: [Sipping] Comments on
>draft-malas-performance-metrics-05.txt
>
>Is there an issue associated with when "cut through" really occurs?
>
>In other words, in some cases each media path may be available 
>at different times.  So, perhaps the setup times should be 
>relative to each unidirectional path.
>
>Also, is the focus of "session" the media or signaling?  Just 
>trying to be sure it is clear what we intend to measure.
>
>Mike
> 
>
>> -----Original Message-----
>> From: Gonzalo Camarillo [mailto:[email-masked]]
>> Sent: Tuesday, October 31, 2006 3:00 AM
>> To: sipping
>> Cc: [email-masked]
>> Subject: [Sipping] Comments on draft-malas-performance-metrics-05.txt
>> 
>> Hi,
>> 
>> a few comments on draft-malas-performance-metrics-05.txt.
>> 
>> The Section "Conventions used in this document", which currently 
>> appears between the Abstract and the TOC, is usually part of the 
>> Terminology Section instead.
>> 
>> Expand acronyms (e.g., UAC, UAS) on their first use.
>> 
>> Section 3.1.2: RRD is defined as:
>> 
>> "the interval is defined from the initial REGISTER request and the 
>> final response indicating a failure received from the destination 
>> registrar or interim proxies."
>> 
>> However, in the example, the UAC receives an error response, 
>it sends 
>> a new REGISTER, and then gives up. RRD is calculated until the UAC 
>> gives up. Therefore, the description needs to clarify which final 
>> responses are taking into account to calculate RRD (i.e., 
>when the UAC 
>> gives up) and which ones are not (e.g., authentication challenges).
>> 
>> Section 3.2 talks about the Session Request Delay (SSD and says that 
>> is telephony applications of SIP is known as Post Dial Delay.
>> 
>> The ITU defines the post-selection delay as:
>> 
>> "Post-selection delay (en bloc sending) is defined as the time 
>> interval from the instant the first bit of the initial SETUP message 
>> containing all the selection digits is passed by the calling 
>terminal 
>> to the access signalling system until the last bit of the first 
>> message indicating call disposition is received by the calling 
>> terminal (ALERTING message in case of successful call)."
>> 
>> However, the draft defines Time Begin as:
>> 
>>    "TB occurs when the designated request has been processed by the
>>     application and last bit of the request packet has been 
>sent from 
>> the
>>     proxy or UA (and is externally observable at some physical
>>     interface)."
>> 
>> Therefore, while for the ITU, TB relates to the first bit of the 
>> message, for the draft TB relates to the last bit of the message. We 
>> should either use the first bit of the message as well of 
>clarify that 
>> the Post Dial Delay is not the same thing as SRD.
>> 
>> Section 3.4 defines the Session Duration Time as an interval 
>starting 
>> with a 200 OK response. However, session establishments involving an 
>> answer in the ACK cannot be considered to be established 
>until the ACK 
>> arrives to the UAS. These sessions can be considered to be 
>"accepted", 
>> though. Section 3.4 should clarify this point.
>> 
>> The example in Section 4 does not seem to relate to any of 
>the ways to 
>> provide finer granularity, which are described right before the 
>> example.
>> 
>> Cheers,
>> 
>> Gonzalo
>> 
>> This list is for NEW development of the application of SIP Use 
>> [email-masked] for questions on current sip Use 
>> [email-masked] for new developments of core SIP
>> 
>
>This list is for NEW development of the application of SIP Use 
>[email-masked] for questions on current sip 
>This list is for NEW development of the application of SIP Use 
>[email-masked] for questions on current sip