Subject: [Sipping] Draft: SIP Performance Metrics

From: "Malas, Daryl" <[email-masked]>
Date: Wed, 24 May 2006 11:40:36 -0600

(re:
http://www.ietf.org/internet-drafts/draft-malas-performance-metrics-00.t
xt)

Based on a couple of emails, I think the following few things should be
considered for this draft and I'm looking for feedback on the direction:

1.  Include relevant metrics from the work done in documents such as
SIPStone.

2.  Detail the ability to include the metric outputs for reporting in
the SIP MIB draft
(http://www.ietf.org/internet-drafts/draft-ietf-sip-mib-10.txt)
--This may include new metrics, which are calculated on the server
itself in addition to post-processing methods.

3.  Standardize the acronyms for the metrics. (For example: Post Dial
Delay, Post Selection Delay, Post INVITE Delay...which term should be
used.

I am sure other work has been done regarding metrics besides SIPStone.
Are there any other thoughts on this?

Thanks for your feedback thus far.

--Daryl

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 24 May 2006 13:39:31 -0500

Daryl: There is work starting in the BMWG along SIP performance
metrics, being worked by Scott Poretsky and me.  At the very
least, we should co-ordinate your draft with the thoughts there,
and then can figure out with the help of chairs and ADs as to where
the work should proceed.

I have also heard about some performance metrics work in TISPAN
and SPEC, but I do not have any more details about these.

Thanks,

- vijay

-- 
Vijay K. Gurbani  vkg@{lucent.com,research.bell-labs.com,acm.org}
Bell Laboratories, Lucent Technologies, Inc.
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 24 May 2006 14:44:01 -0400

A working group has just been formed (this week) in the SPEC consortium.

---

From: [email-masked]
Date: Fri, 26 May 2006 14:51:17 -0400

> and SPEC, but I do not have any more details about these.

I have been organizing the SPEC SIP benchmarking effort, and would
love to get participation from IETF folks.  It seems like the two
activities could be very complementary.  

Based on what I've seen, it seems like the IETF BMWG and Daryl Malas' 
draft posted on SIPPING a few days ago are more oriented towards defining 
the proper methods and metrics to measure (e.g., call success rate, defects 
per million, etc.).  

In contrast, SPEC wants to develop and distribute common code for 
benchmarking, as is done with SPECWeb and SPECJAppServer.  That code can 
and should use the standardized peformance definitions agreed to by 
SIPPING and/or BMWG.  

For example, if you look at the SPECWeb design docs and run rules,
you'll see:

"The benchmark environment shall be governed by the following standards:

    * RFC 2616 Hypertext Transfer Protocol 1.1 -- HTTP 1.1
    * RFC 791 Internet Protocol (IPv4) (Standard)
      o updated by RFC1349 Type of Service in the Internet Protocol Suite 
    * RFC 793 Transmission Control Protocol (TCP) (Standard)
    * RFC 1122 Requirements for Internet Hosts - Communication Layers "

A SIP benchmark would be similar.

Right now there are two mechanisms in place to participate in defining 
and developing a SIP benchmark: a mailing list and a bi-weekly conference 
call.  However, to participate, your organization must be a SPEC member.
which costs money, around $500 a year after a $1000 initiation fee.  
However, that entitles you to all the source code.  You can find out 
more about SPEC membership at

	http://www.spec.org/spec/membership.html

Current members tend to be server-side hardware and software vendors 
such as Dell, IBM, HP, Microsoft, Oracle, BEA, etc.  It would be good
to see some Lucent/Nortel/Cisco/Ubiquity type names as well.  

If you or someone in your organization would like to participate, please
contact me.

Thanks,

-Erich

-- 
Erich M. Nahum                  IBM T.J. Watson Research Center
Research Staff Member           P.O. Box 704
[email-masked]            Yorktown Heights NY 10598