Subject: [Sipping] reviewer comments draft-malas-performance-metrics-04

From: Carol Davids <[email-masked]>
Date: Tue, 05 Sep 2006 15:58:20 -0500

Review notes for draft-malas-performance-metrics-04 are attached (also copied below.) 

Reviewer:  Carol Davids, [email-masked]

Date: 9/4/6

Daryl,

Draft 4 looks good.  Please see my comments below.

Overall:

I think we should include a discussion of the logical and physical location of observation points (measurement points) in this document. 

1. Physical location of points on the network at which measurements will be taken: 

The metrics defined in this document are classified as end-to-end.  Sections 3.7 through 3.10 contain the phrase, "The input variables for this metric are captured from the originating UAC or proxy perspective as relative to the end-to-end network under measurement.  "   This phrase could be applied to all the metrics except the AHI, Section 3.4. The inputs required for the calculation of the AHI are identified on two networks that are possibly separate and therefore the collection of data is required at two different points on the inter-network.  How will the data used to calculate this metric be collected and correlated?  The current document notes this as an issue in Section 5.2, and suggests several methods. Should we be able to calculate all metrics using inputs obtained at a single network observation point?   Doing so might require the definition of new SIP headers, SIP Parameters, or SNMP objects. Has this possibility been considered?   

2. Logical location of points on the network at which measurements will be taken: 

The collection of data used to compute these metrics must be logically separated from the processes that make up the network under measurement. For example, a piece of software that includes the SIP UA under measurement should not be used as the source of values used to calculate these metrics.  The current document needs to specify that logs associated with such software should not be used for the calculation of the metrics.  Else the process becomes something similar to the situation in which you try to proof read a document that you have written.

Section 2: Terminology:

Consider using a more limited definition of the Time Begin (TB) and Time Stop (TS) in this section.  Something like the following:

"TB and TS refer to time stamps in data frames captured by software at observation points on the network under measurement. The definition of the SIP request or response whose TB or TS is to be recorded will be described in the definition of the individual metrics. "

Section 3.3 Session Completion Delay (SCD)

Consider changing the name to "Session Disconnect Delay" (SDD).  The term "completion" is already being used for other purposes in this document (see 3.10 for example.) The term "completed call" is used to refer to the fact that there has been some sort of disconnection activity associated with the call.  The definition in section 3.3 refers to the actual time it takes to disconnect the call.  Since this metric deals specifically with that disconnection, the name of the metric should reflect that fact.  

Section 3.4 Average hops per Invite (AHI)

The definition implies that there may be two observation (measurement) points.  One at the "interface" between UA1 and Proxy 1 and another at the "interface" between Proxy 2 and UA2.  How will we synchronize the observations made at these two points?  

a. Originating and terminating UA are both on the same network:

The observation point on the originating network can capture messages between PROXY 2 and UA2 as well as between PROXY 1 and UA1.  

b. Originating and terminating UA are on different networks:

If information is captured at the terminating side of the session, how will it be communicated to the originating side for incorporation into the metric?  Is there a defined MIB object for this purpose?  Should one be proposed?  Alternatively, is there a header or header parameter that can be used at the terminating side to send the data back to the originator in a response?  Should one be proposed?

Section 3.5 Session Duration Time (SDT)

The definitions of "successful completion SDT" and "failed completion SDT" need to be clarified.  

1. In 3.5.2 we do not know that a BYE was sent so we cannot know when timer F expires and so do not know how to populate the variable TS.

2. Perhaps we should say the Time Stop (TS) value in 3.5.1 is the time stamp on the receipt of the 200 OK associated with the BYE, rather than on the BYE itself.   Then we could make the parallel definition for 3.5.2 saying the Time Stop value in 3.5.2 is the time stamp on the first BYE  + the value of the F timer.  

Section 3.10 Session Completion Rate (SCR)

Just a small point, I think.  You refer to successful session completion and unsuccessful session completion.  For consistency, perhaps we should divide this section into 3.10.1 and 3.10.2 as was done in Sections 3.2 SRD and 3.3 SCD.  The success refers to completions, and this just makes that clearer.

Yours,

Carol

Carol Davids
Professor ITM & Director of VoIP Activities
Illinois Institute of Technology - Center for Professional Development
Office: (630) 682-6024
Mobile: (630) 292-9417
Email: [email-masked]
Web: http://www.itm.iit.edu/voip/
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Daryl Malas <[email-masked]>
Date: Wed, 06 Sep 2006 11:24:39 -0600

Thanks Carol.  I'll review the draft relative to your comments.  I
provided some comments in-line...I would appreciate the mailing list's
opinion on these as well. Thanks...--Daryl

On Tue, 2006-09-05 at 15:58 -0500, Carol Davids wrote:
> I think we should include a discussion of the logical and physical
> location of observation points (measurement points) in this document. 
> 
>  
> 
> The metrics defined in this document are classified as end-to-end.
> Sections 3.7 through 3.10 contain the phrase, “The input variables for
> this metric are captured from the originating UAC or proxy perspective
> as relative to the end-to-end network under measurement.  “   This
> phrase could be applied to all the metrics except the AHI, Section
> 3.4. The inputs required for the calculation of the AHI are identified
> on two networks that are possibly separate and therefore the
> collection of data is required at two different points on the
> inter-network.  How will the data used to calculate this metric be
> collected and correlated?  The current document notes this as an issue
> in Section 5.2, and suggests several methods. Should we be able to
> calculate all metrics using inputs obtained at a single network
> observation point?
[DM] This would be a nice thing to have; however, the ability to
calculate AHI requires information from other sources.  Perhaps this is
not a necessary metric, and could be removed from the draft.  This would
simplify the ability to capture the data from the ingress point as
described in the other metrics.
>    Doing so might require the definition of new SIP headers, SIP
> Parameters, or SNMP objects. Has this possibility been considered?
[DM] I would really like to avoid defining new things, as you describe.
I think the easiest approach is also the easiest to implement and/or
calculate for SNMP trap triggers.  I did mention the possibility of
adding the input variables as SNMP trap variables.  This is still a
possibility; however, some vendors may wish to perform these
calculations external to their elements in a "performance server" or
similar.
> The collection of data used to compute these metrics must be logically
> separated from the processes that make up the network under
> measurement. For example, a piece of software that includes the SIP UA
> under measurement should not be used as the source of values used to
> calculate these metrics.  The current document needs to specify that
> logs associated with such software should not be used for the
> calculation of the metrics.  Else the process becomes something
> similar to the situation in which you try to proof read a document
> that you have written.
[DM] I'm assuming you are simply stating no 3rd party software will be
used to capture the input variables, rather the protocol itself must be
used?
> 
>  
> 
> Section 2: Terminology:
> 
>  
> 
> Consider using a more limited definition of the Time Begin (TB) and
> Time Stop (TS) in this section.  Something like the following:
> 
> “TB and TS refer to time stamps in data frames captured by software at
> observation points on the network under measurement. The definition of
> the SIP request or response whose TB or TS is to be recorded will be
> described in the definition of the individual metrics. “
[DM] I can do that.  I thought it provided more clarity by separating
them.  Any other thoughts on this?
> Consider changing the name to “Session Disconnect Delay” (SDD).  The
> term “completion” is already being used for other purposes in this
> document (see 3.10 for example.) The term “completed call” is used to
> refer to the fact that there has been some sort of disconnection
> activity associated with the call.  The definition in section 3.3
> refers to the actual time it takes to disconnect the call.  Since this
> metric deals specifically with that disconnection, the name of the
> metric should reflect that fact.
[DM] Someone else made a comment about this as well.  I like your
suggested change.  Thanks...
> The definition implies that there may be two observation (measurement)
> points.  One at the “interface” between UA1 and Proxy 1 and another at
> the “interface” between Proxy 2 and UA2.  How will we synchronize the
> observations made at these two points?
[DM] I would really like to know if the working group would even like to
keep this metric.  If you would, then I can work to provide greater
clarity around this point.
> The observation point on the originating network can capture messages
> between PROXY 2 and UA2 as well as between PROXY 1 and UA1.  
> 
>  
> 
> If information is captured at the terminating side of the session, how
> will it be communicated to the originating side for incorporation into
> the metric?  Is there a defined MIB object for this purpose?  Should
> one be proposed?  Alternatively, is there a header or header parameter
> that can be used at the terminating side to send the data back to the
> originator in a response?  Should one be proposed?
> 
>  
> 
> The definitions of “successful completion SDT” and “failed completion
> SDT” need to be clarified.  
> 
>  
> 
> 1. In 3.5.2 we do not know that a BYE was sent so we cannot know when
> timer F expires and so do not know how to populate the variable TS.
> 
>  
> 
> 2. Perhaps we should say the Time Stop (TS) value in 3.5.1 is the time
> stamp on the receipt of the 200 OK associated with the BYE, rather
> than on the BYE itself.   Then we could make the parallel definition
> for 3.5.2 saying the Time Stop value in 3.5.2 is the time stamp on the
> first BYE  + the value of the F timer.  
> 
>  
> 
> Just a small point, I think.  You refer to successful session
> completion and unsuccessful session completion.  For consistency,
> perhaps we should divide this section into 3.10.1 and 3.10.2 as was
> done in Sections 3.2 SRD and 3.3 SCD.  The success refers to
> completions, and this just makes that clearer.
> 
>  
> 
>  
> 
> Yours,
> 
>  
> 
> Carol
> 
>  
> 
> 
> Carol Davids

---

From: Daryl Malas <[email-masked]>
Date: Fri, 15 Sep 2006 11:56:02 -0600

Carol,

Please see my comments in-line...

> calculate all metrics using inputs obtained at a single network
Yes, I have removed AHI and applied the variable capture point to all
metrics.  I believe this simplifies it.
> Parameters, or SNMP objects. Has this possibility been considered? 
I don't think this would require the establishment of a new SIP header
or SIP parameter, and I think I would like to stray from this point.  I
have contact Jean-Francois (SIP MIB package) to talk about how these
metrics may be able to fit or if it makes sense.
> similar to the situation in which you try to proof read a document
Are you talking about a "log" writing application could skew the
timestamps, and should not be used to calculate the metrics?  I didn't
want to specify WHAT external application should be used, but I could
specify the signaling protocol information should be used directly and
not an external application.
> described in the definition of the individual metrics. “
I changed these per some new language from Al Morton.  Please review
Al's comments and let me know if you have further comments.
> metric should reflect that fact. 
I changed this per your recommendation.  I agree it provides better
clarity as to the purpose of the metric.  Thanks!!
> originator in a response?  Should one be proposed?
This metric is no longer in the document.
> completions, and this just makes that clearer.
I updated this per your suggestion.  Thanks!!