Subject: RE: [bmwg] FW: IETF BMWG Work Item Proposal for SIP PerformanceBenchmarking

From: "Romascanu, Dan \(Dan\)" <[email-masked]>
Date: Wed, 24 May 2006 17:39:10 +0300

I hear what you are saying but I am not sure that I can agree. The
question is why should RTP plane benchmarking be within SIP Performance
Benchmarking scope. What does SIP have to do with media plane
performance anyway?  SIP can be used to establish sessions with a
different data plane than RTP, and we may want to benchmark RTP
performance even if signaling is not SIP. 

I guess if you want to say that you are using SIP to route media through
an overlay network with media B2BUA's, then sure, there is a delay
penalty to pay at each B2BUA for looking up routing information,
re-encapsulating packets, and possible other packet transformations, but
that doesn't appear to be the intent of the proposal. 

I am not saying that RTP media plane benchmarking is not useful, but I
believe that it should be within the scope of a different document. 

Dan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> Sent: Wednesday, May 24, 2006 5:19 PM
> To: Romascanu, Dan (Dan); [email-masked]
> Cc: [email-masked]; [email-masked]; 
> [email-masked]; [email-masked]
> Subject: RE: [bmwg] FW: IETF BMWG Work Item Proposal for SIP 
> PerformanceBenchmarking
> 
> Hi Dan,
> 
> SIP benchmarking requires measurement of the control (SIP 
> signaling) and data (RTP) planes.  It is more reasonable with 
> SIP to refer to the data plane as the media plane.  The 
> relationship between the two planes is that the media will 
> not flow until the control sessions are established.  It is 
> important to benchmark performance of the media plane with 
> control load, and vice-versa.
> 
> Scott
> 
> -----Original Message-----
> From: Romascanu, Dan (Dan) [mailto:[email-masked]]
> Sent: Wednesday, May 24, 2006 4:20 AM
> To: [email-masked]; [email-masked]
> Cc: Cullen Jennings; Magnus Westerlund
> Subject: RE: [bmwg] FW: IETF BMWG Work Item Proposal for SIP 
> PerformanceBenchmarking
> 
> 
> I have a clarification question. How do you define 'SIP Media 
> Plane'? It looks to me that all the benchmark metrics and 
> test parameters included under this label are generic for any 
> RTP stream benchmark and not necessarily related to SIP. Am I 
> missing something? 
> 
> Dan
> 
> 
>  
>  
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]
> > Sent: Wednesday, May 24, 2006 12:50 AM
> > To: [email-masked]
> > Subject: [bmwg] FW: IETF BMWG Work Item Proposal for SIP 
> > PerformanceBenchmarking
> > Importance: High
> > 
> > 
> > Fellow BMWG-ers,
> > 
> > Please find this IETF Work Item Proposal for your review 
> and comment.  
> > The authors will be submitting a first draft Terminology and 
> > Methodology for the Montreal meeting.
> > 
> > Scott
> > 
> > >  -----Original Message-----
> > > From: 	Poretsky, Scott  
> > > Sent:	Thursday, May 11, 2006 12:44 PM
> > > To:	[email-masked]'
> > > Subject:	IETF BMWG Work Item Proposal for SIP Performance
> > > Benchmarking
> > > Importance:	High
> > > 
> > > Hi Al,
> > > 
> > [snip]
> > 
> > > Thanks,
> > > Scott
> > > 
> > > ########################
> > > BMWG Work Item Proposal
> > > 
> > > TITLE: Session Initiation Protocol (SIP) Performance Benchmarking
> > > 
> > > AUTHOR TEAM:
> > > Dr. Carol Davids, Illinois Institute of Technology Dr. 
> > Vijay Gurbani,
> > > Lucent Technologies
> > Scott Poretsky, Reef Point Systems
> > >  
> > > GOALS
> > > The purpose of this work item is to provide a single
> > terminology and
> > > methodology from which SIP equipment vendors and VoIP service 
> > > providers can measure performance benchmarking metrics for
> > comparison and selection.
> > > It is intended to develop terms, benchmarks, and 
> methodologies that 
> > > can be applied to any type of IP device including SIP
> > Servers, Session
> > > Border Controllers (SBCs), and Security Gateways (SEGs).
> > > 
> > > MOTIVATION
> > > Service Providers are now planning VoIP and Multimedia network 
> > > deployments using the IETF developed Session Initiation Protocol 
> > > (SIP).  Through SIP, service providers will be able to have rich 
> > > service offerings and many even plan to turn off their
> > Public Switch Telephone Network within five
> > > years.   VoIP has led to development of new networking 
> > devices including
> > > SIP Servers, Session Border Controllers, and Security
> > Gateways.  The
> > > mix of voice and IP functions in this variety of devices
> > has produced
> > > inconsistencies in vendor reported performance metrics and
> > has caused
> > > confusion in the service provider community.
> > > 
> > > SCOPE:
> > > This work item will provide terms, benchmarks, and
> > methodologies for
> > > performance benchmarking the SIP control and media planes.  The 
> > > methodologies can be used for benchmarking SIP performance of SIP 
> > > servers, SBCs, and SEGs.  The media used for the
> > benchmarking will be
> > > the IETF defined Real-Time Protocol (RTP).  Test cases will allow 
> > > testing of VoIP or Multimedia over IP by varying the number
> > of media
> > > streams per SIP call from 1 to higher.  There will be at 
> least one 
> > > test case to measure the impact of a SIP DOS Attack on the
> > performance
> > > metrics.  All SIP traffic will be in the clear; no
> > encryption using TLS or IPsec will be used.
> > > Similar benchmarks and methodologies using encryption is a
> > potential
> > > work item for the BMWG that could be considered in the future.
> > > 
> > > There will be separate test cases in the methodology document for 
> > > obtaining measurements for each of the Benchmarking Metrics
> > defined in
> > > the Terminology.  The benchmarking metrics to be measured
> > for the SIP
> > > Control Plane and SIP Media Plane are as follow:
> > > 
> > > 	Benchmark Metrics - SIP Control Plane 
> > > 	Standing Calls, maximum (calls)
> > > 	Calls Per Second, maximum (CPS)
> > > 	Call Attempts Per Second, maximum (CAPS)
> > > 	Busy Hour Call Attempts, maximum (BHCA)
> > > 	Busy Hour Call Connects, maximum (BHCC)
> > > 	Call Completion Rate (%)
> > > 	Call Setup Delay, average (msec)
> > > 	Call Teardown Delay, average (msec)
> > > 
> > > 	Benchmark Metrics- SIP Media Plane
> > > 	RTP Media Throughput, per Media Stream (pps)
> > > 	RTP Media Throughput, Aggregate (pps)
> > > 	RTP Packet Loss, average (pps)
> > > 	RTP Packet Delay, average (msec)
> > > 	RTP Packet Jitter, average (msec)
> > > 
> > > In order to obtain measurements for the benchmarking
> > metrics it will
> > > be necessary to configure the test setup with certain test
> > parameters.  
> > > Some of these test parameters may also be the benchmarking metric 
> > > being measured for another test case.  The test parameters to be 
> > > configured for the SIP Control Plane and SIP Media Plane
> > are as follow:
> > > 
> > > 	Test Parameters- SIP Control Plane 
> > > 	Call Duration (msec)
> > > 	Call Per Second (CPS)
> > > 	Call Attempts Per Second (CAPS)
> > > 
> > > 	Test Parameters - SIP Media Plane
> > > 	RTP Media Streams per Call (streams per call)
> > > 	RTP Packet Size (bytes)
> > > 	RTP Media Offered Load, per Media Stream (pps)
> > > 	RTP Media Offered Load, Aggregate (pps)
> > > 
> > > The basic test topology to be used for this benchmarking is
> > as follows:
> > > 
> > > Emulated Agents<--> SEG (optional) <--> SIP Proxy Server
> > (or SBC) <-->
> > > SIP Server (optional) <--> Emulated Agents
> > > 
> > > PROPOSED MILESTONES
> > > 06/07 First Draft Terminology and Methodology
> > > 02/07 First WG Last Call
> > > 05/07 SIP WG and SIPPING WG Review
> > > 06/07 Final BMWG Last Call for Terminology and Methodology
> > > 08/07 Submittal for IESG review
> > > 
> > > 
> > > 
> > 
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/bmwg
> > 
> 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 24 May 2006 09:54:32 -0500

> performance even if signaling is not SIP. 

Dan: You are absolutely right.

> believe that it should be within the scope of a different document. 

I am curious: is there such an existing document?  Has there been
any work in avt or mmusic for RTP benchmarking?  Would the bench-
mark be specific to a codec?

It would be nice to gauge some consensus in the WG around whether
the "Media Plane" is inline with the notion of SIP benchmarking
or not.

Thanks,

- vijay
-- 
Vijay K. Gurbani  vkg@{lucent.com,research.bell-labs.com,acm.org}
Bell Laboratories, Lucent Technologies, Inc.
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 25 May 2006 14:04:08 +0200

Hi,

As AVT WG chair I do have some opinions on this topic.

> mark be specific to a codec?

There are at least two performance monitoring frameworks for RTP. The 
first one is RTCP with additional metrics, see RFC 3550 and RFC 3611. 
There is also the RMONMIB work, which provides another framework for 
real-time monitoring of performance.
http://www.ietf.org/internet-drafts/draft-ietf-rmonmib-raqmon-framework-16.txt

There is a very difficult problem when trying to benchmark end to end 
performance of media. The reason is that the performance depends on the 
users subjective quality impression. Errors affects the subjective 
quality different, and are also dependent on the consuming individual. 
Thus the connection between objective measurements and subjective 
quality is very difficult. A common method to get somewhere is to 
calibrate a objective measurement method towards subjective assessments. 
That way one can get decent quality of the objective measurement. 
However the important factor here is that theses measurements are only 
valid within the confines of the method used in the test. Thus certain 
system changes or simply values outside of the correlated values lack 
meaning.

The problem with the media plane is that beyond the network 
characteristics there is so much implementation dependent behavior that 
affect quality. It is also very difficult to setup probes in meaningful 
ways within these implementations that are comparable between 
implementations.

> the "Media Plane" is inline with the notion of SIP benchmarking

In my knowledge there there is clearly separate parts which are linked 
in that the full performance of a VoIP "system" clearly is the sum of 
the signalling and the media plane behavior. The debate about how to 
correlate the signalling with the media plane measurements that we have 
had regarding the RFC 3611 update in AVT WG clearly shows there is 
important factors to consider here. The bottom line is that it is 
important to be able to correlate a signalling measurement with the 
media plane measurements to determine the system performance.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 25 May 2006 14:04:08 +0200

Hi,

Cheers

Magnus Westerlund