Subject: [bmwg] SIP Benchmarking methodology and terminology - version 4 updates

From: Carol Davids <[email-masked]>
Date: Mon, 12 Mar 2012 21:53:45 -0500

Hello BMWG,

There are new drafts on SIP performance benchmarking available at
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-04 and
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-04.

The following changes were made:

Changes to Terminology:
1. Revised the definitions of Standing Sessions Count,  Registration Rate,
Session Establishment
Rate, and Session Capacity to make them consistent with the description of
the
test algorithm and pseudocode introduced in the associated Methodology
Document.

2. Added a benchmarking model, entitled ŒBaseline
performance model of the Emulated Agent without a DUT present¹, to be used
to
baseline the test apparatus itself.  Made
an editorial change to reflect the fact that there are now 10 benchmarking
models illustrated in the document.

Changes to Methodology:

1. Added a description of the test algorithm to be used to
discover the Session Establishment Rate.
There are both a prose description and a pseudocode description in the
new document.

2. Revised all test descriptions to refer to the test algorithm.

The authors invite your comments and discussion. We have asked for agenda
time at IETF 83. 

Best Regards,

Carol Davids

---

From: "Aamer Akhter (aakhter)" <[email-masked]>
Date: Tue, 27 Mar 2012 04:28:19 -0500

Please find my comments for draft-ietf-bmwg-sip-bench-*-04 below:

General comment:

1. Can we please add a statement with regards to the relationship
between this work and rfc6076? (something more deeper than BMWG is in
the lab). 

2. Have anybody checked to see if the metrics out of rfc6076 and this
work are comparable?

3. "However, benchmarking SIP
      presence is not a part of this work item.". Is there a reason for
this. Presence would be one of the more interesting cases to understand.

4. "Only session establishment is considered for the performance
      benchmarks.  Session disconnect is not considered in the scope of
      this work item." I would think that given connectivity churn
(links going up and down, power outages, etc), both establishment as
well as disconnect are interest items. Consider the cleanup work needed
after a disconnect... some of this might be a high workload than simply
maintaining a connection.

5. term-04, figure 12 seems misformatted, or it needs a better
explanation of how to read it...

6. For the case of video conferencing the SDP can get quite large. I am
wondering if this (SDP size) is an interesting axis to explore.

-------- Original Message --------
Subject: 	SIP Benchmarking methodology and terminology - version 4
updates
Date: 	Mon, 12 Mar 2012 21:53:45 -0500
From: 	Carol Davids <[email-masked]>
To: 	bmwg.ietf.org <[email-masked]>
Newsgroups: 	gmane.ietf.bmwg

Hello BMWG,

Changes to Terminology:
1. Revised the definitions of Standing Sessions Count, Registration
Rate, Session Establishment Rate, and Session Capacity to make them
consistent with the description of the test algorithm and pseudocode
introduced in the associated Methodology Document.

2. Added a benchmarking model, entitled 'Baseline performance model of
the Emulated Agent without a DUT present', to be used to baseline the
test apparatus itself. Made an editorial change to reflect the fact that
there are now 10 benchmarking models illustrated in the document.

Changes to Methodology:

1. Added a description of the test algorithm to be used to discover the
Session Establishment Rate.
There are both a prose description and a pseudocode description in the
new document.

The authors invite your comments and discussion. We have asked for
agenda time at IETF 83.

Best Regards,

Carol Davids