Subject: [Sipping] RE: draft-malas-performance-metrics-06

From: "Malas, Daryl" <[email-masked]>
Date: Tue, 13 Feb 2007 16:02:03 -0700

Marian...Comments inline...Also, I copied the Sipping mailing list,
because it is helpful to allow others within the Sipping community to
provide insight, thoughts into your comments as well.  I hope you do not
mind.

Thanks...--Daryl

> -----Original Message-----
> From: Marian Delkinov (LN/EAB) [mailto:[email-masked]]
> Sent: Tuesday, February 06, 2007 12:45 AM
> To: Malas, Daryl
> Subject: draft-malas-performance-metrics-06
> 
> Hi,
> 
> I'm working on implementing some IMS performance metrics into our SLA
> solutions. After performing a lot of searching through the different
> standardization bodies, the most useful document I found was your
draft.
> 
> 
> Would like to comment on some of the metrics and perhaps on the
generic
> approach in the document.
> 
> Generic approach:
> 
> It would be good to give some background information about what are
the
> major reasons for SIP calls being lost. A picture similar to the one
> used in ITU E411 recommendation for the interpretation of the network
> performance data would really help. Please, see example in attachment.
> 
[DM] I could do this if the general mailing list agrees with the
request.  I would like to get their opinion as well.  In either case, I
will consider it for future revisions of the document.  I agree that
diagrams, in general, usually help provide clarity around concepts.
 
> Metrics:
> 
> Since dealing with SLAs, I would like to put the things more into a
> Customer perspective. Therefore, I'm more focused on how the service
> looks for the customer. First of all what is the service accessibility
> and secondly what is the service performance. Perhaps you can help me
> find out the most suitable metrics that you discuss in your document.
> 
> 1) I try to define the SIP based telephony service effectiveness:
> 
> The probability that an end user can access the IMS (SIP based)
> Telephony Service.
> 
> In the PSTN/ISDN terms this is defined by ITU in Recommendation 411 as
> Network Effectiveness Rate:
> 
>             Seizures Delivered to the far end terminal
> NER% = ---------------------------------------------------- x 100
> 		       Total Number of Seizures
> 
> Note that in this case the end user behaviour is excluded, so seizures
> that reach answer signal, or user busy, or ring no answer are counted
as
> successful.
> 
> When I look in your draft document, I don't clearly see similar
metric.
[DM]  If you look in the "Additional Metrics for Consideration" section
of the draft, I think you will find a similar metric to what you are
describing. Again, if you would like to see the metric included in the
document, please state that on the mailing list.  Thanks...

> However if used some of your definitions, it could perhaps be defined
as
> Session Effectiveness Rate (or any other suitable definition, e.g.
> Telephony Service Availability) as follows:
> 
>          Total # of Session Requests - # of Failed Session Completions
> SEFR% = --------------------------------------------------------------
x
> 100
> 			Total # of Session Requests
> 
> Or, if in the above the "# of Failed Session Completion" includes the
> end user behaviour, then perhaps a better definition would be:
> 
>          Total # of Session Requests - # of ISA
> SEFR% = --------------------------------------------------------------
x
> 100
> 			Total # of Session Requests
> 
> So, I would be really grateful if you share your opinion about that.
> 
> 2) I also try to define the SIP based telephony call completion ratio.
> According to your document the Session Completion Rate (SCR) is the
> parameter that matches that. However I think of a more useful
> definition, that would really present the ability of the network to
> maintain successfully initiated session in the time. So, I would like
to
> hear what you think about the following definition proposal:
> 
> Session Completion Rate is the probability that a successfully
> established session is maintained for a predetermined time until it is
> released intentionally by any of the UAs.
> 
> According to your document it is defined as:
>          # of Successfully Completed Sessions
> SCR% = ---------------------------------------
>           Total # of Session Requests
> 
> But if we follow my proposal it would be:
> 
>          # of Successfully Completed Sessions
> SCR% = ------------------------------------------ x 100
>          # of Successfully Established Sessions
> 
> 
> 
[DM] In concept I understand what you are trying to get at with this
change; however, this will require defining what an "Established
Session" means.  This will become very tricky.  Instead, I think you can
use the SCR as it is defined and then provide greater granularity by
calculating it for calls above and below a time certain duration.  What
do you think?

This is actually something we can perhaps easily measure, however it
> would be good if we can show this parameter in dependence with the
> Session Duration and that I haven't taken into account. In many cases,
> when jitter and packet loss increase, the sessions are terminated
> unintentionally. And the probability this to happen raises in time.
For
> example, for short sessions the SCR may look good, but for long
sessions
> it deteriorates.
> 
> So, I wonder how to link this with the session duration time in order
to
> have a single parameter that really represents the network
performance.
> Will be happy to exchange ideas.
> 
> Best regards!
> Mario.
> 
> Marian Delkinov
> Senior Consultant
> 
> Network Management and Service Assurance
> 
> ERICSSON AB
> Lindholmspiren 11
> SE-417 56 Gothenburg
> Sweden
> 

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Marian Delkinov \(LN/EAB\)" <[email-masked]>
Date: Fri, 23 Feb 2007 10:48:40 +0100

Hi,

(Of course - no problem to copy to the mailing list. I don't mind and
would be happy to see more opinions)

Regarding including diagrams, did you receive more support about it?

Regarding the metrics:

1) Chapter 5.6. Additional Metrics for Consideration
Session Establishment Efficiency Rate: Please, see my comments in a
separate mail with subject "[Sipping] Session Establishment Efficiency
Rate".

2) Session Completion Rate (SCR)

My proposal was:

Your comment:

[MD] I thought that as "Established Session" we can consider one that
got a 200 OK response as result of an INVITE. Perhaps the definition in
chapter 2 of the draft can be used:

Session Establishment - Session establishment occurs when a 200 OK 
   response from the UAS has been received, in response to a 
   corresponding UAC's INVITE setup request, indicating the session 
   setup request was successful. 

It's another question how we can get both counters from above in order
to calculate the metric. Some vendors provide # of BYEs and # of OKs,
but in those cases the counters include the repeated BYEs and OKs, which
makes the metric wrong. Some other of course have counters for the
sessions in fact. But I believe that if this gets into an RFC, most of
the vendors will include the session counters as needed. 

Best regards!
Mario.

-----Original Message-----
From: Malas, Daryl [mailto:[email-masked]] 
Sent: Wednesday, 14 February, 2007 00:02
To: Marian Delkinov (LN/EAB)
Cc: sipping
Subject: [Sipping] RE: draft-malas-performance-metrics-06

Thanks...--Daryl

draft.
generic
the

>             Seizures Delivered to the far end terminal NER% = 
> ---------------------------------------------------- x 100
> 		       Total Number of Seizures
> 

as
metric.

as

x
>          Total # of Session Requests - # of ISA SEFR% = 
> --------------------------------------------------------------
x
to

>          # of Successfully Completed Sessions SCR% = 
> ---------------------------------------
>          # of Successfully Completed Sessions SCR% = 
> ------------------------------------------ x 100

For
sessions
to
performance.

---

From: Daryl Malas <[email-masked]>
Date: Fri, 23 Feb 2007 08:56:42 -0700

Okay, I just wanted to make sure our definitions of "Established
Session" and "Session Establishment" were the same.  I assumed so, but
wanted to make sure you were not inferring a different definition.

Thanks for clarifying...---Daryl