Subject: [bmwg] Methodology/Terminology for Benchmarking SIP Networking Devices -drafts -02 - Description of changes

From: Carol Davids <[email-masked]>
Date: Tue, 20 Jul 2010 00:11:09 -0500

All,
	
Below is a summary of the changes we made to 'Methodology for Benchmarking
SIP Networking Devices', draft-ietf-bmwg-sip-bench-meth-02.txt and
'Terminology for Benchmarking Session Initiation Protocol (SIP) Networking
Devices', draft-ietf-bmwg-sip-bench-term-02.txt. 

Your comments and questions are greatly appreciated.  Thanks.

Changes to the Terminology Draft:

1. Introduction: Added a note regarding the need to record and report
failures.

2. Scope: Added clarification that the cause of failure is outside the scope
of this document.

3. Section 2.2: Benchmarking Models: Fixed format of Figure 6.

4. Section 3.1.5: Overload: Revised this section to remove the dependency on
the Overload Working Group activities.

5. Section 3.1.8 Invite-initiated Session (IS): Revised to make clear the
relationship between associated media and IS's.

6. Section 3.1.10: Session Attempt Failure: Added the phrase, "within the
Establishment Threshold Time x.x.x" to Discussion Item 2, to ensure that
popping a defined timeout threshold constitutes a failure.

7. Section 3.1.11: Standing Sessions: Changed the term "Standing Sessions"
to a new term "Standing Sessions Count." Since the metric is a number, we
believed that this fact should be reflected in the term's name.  Else, the
term was ambiguous and appeared to refer the session itself rather than to a
number of sessions. Revised the Discussion section accordingly. 

8. Global: Changed the related-term "Session Duration" throughout the
document to "Media Session Duration." This is because "Media Session
Duration" is a parameter of test, but we never defined "Session Duration".  

9. Section 3.3.3: Establishment Threshold Time: Rewrote the discussion
section for clarity.  Did not change intent.

10. Added a new Section 3.3.4: Session Duration, to align with the
Methodology Document.

11. Section 3.3.7: Media Session Hold Time: Added material to the discussion
section distinguishing between a call, a dialog and a media session, per Al
Morton's request.

12. Section 3.4.2: Session Establishment Rate: Changed the wording in the
definition to, "The average maximum rate at which..." instead of "The
maximum average rate at which...".

13. Section 3.4.3: Session Capacity: 
a. Added a note to the effect that we can control the session duration of an
IS with or without media by identifying the time interval before we send the
BYE. In this situation, we wanted the session duration to be set to
infinity.  This can be accomplished by not sending a BYE during the duration
of the test.

b. Replace undefined term "Established Sessions" by "Standing Sessions
Count."  

14. Section 3.4.4: Session Overload Capacity: We think that this section
should be removed, since we have defined overload to be outside the scope of
the document.  We are interested in hearing any discussion or feedback.

15. Security Considerations: Added the BMWG standard paragraph. 

Changes to the Methodology Draft:

1. Global: 
a. Changed the labels on the figures to be consistent with the terminology
document.  Thus, Tester became EA (emulated agent) and Server became
Signaling Server.

b. Edited references to Figures to reflect the reduction in figures
identified below.

c. Changed term Server to Signaling Server throughout.

d. Changed several "must"s to "MUST"s.

2. Section 3: Test Topologies:  
a. Collapsed the separate Figures that showed a firewall, NAT and relay into
a single figure, with the central box  labeled "media relay OR firewall OR
NAT.  

b. Added a baseline performance measurement test topology, the new Figure 4.
corresponding to the Figure 1 Benchmarking Model in the Terminology document

3. Section 4: Test Considerations:
a. Section 4.4: Changed Title from "Selection of Associated Media Protocol"
to "Selection of Associated Media Transport Protocol"

4. Section 5: Reporting Format:
a. Section 5.1: Test Setup Report: Added the units of measure to the Test
Setup Report.. 
b. Section 5.2: Device Benchmarks for IS: Deleted Session Disconnect Delay,
since it is out of scope.  Added the units of measure to the report. 
c. Section 5.3: Device Benchmarks for NS: Added the units of measure to the
report.

5. Section 6: Test Cases:
a. Section 6.1: Baseline Session Establishment Rate: Added a baseline
performance test, corresponding the Figure 4 in this document and to Figure
1 in the Terminology document.

b. Removed the reference to setting the tester to SIP UDP and SIP TCP
throughout all tests, since one of the parameters of test is the transport
protocol.

c. Section 6.8: Session Establishment Rate with SIP Flooding: Did not change
this, but think we need to add a definition of SIP Flooding to the
Terminology document.

Yours,

Carol

Carol Davids
Todd Professor / VoIP Research Lab Director
IIT - CPD
Office: 1-630-682-6024
Mobile: 1-630-292-9417
Email: [email-masked]
Web: voip.itm.iit.edu

---

From: Al Morton <[email-masked]>
Date: Tue, 20 Jul 2010 17:11:04 -0400

>Devices', draft-ietf-bmwg-sip-bench-term-02.txt.

Thanks Carol and Vijay, and Scott.  I like the inherent efficiency
of circulating these changes by e-mail before we meet...

>Your comments and questions are greatly appreciated.  Thanks.

And the latest drafts are here:
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-02
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-02

We'll discuss next steps on this topic during our IETF-78 session
and on the list.

regards,
Al
bmwg chair

---

From: "Carol Davids" <[email-masked]>
Date: Tue, 20 Jul 2010 17:46:23 -0500

Hi Al,

Thanks.  I will send you a slide deck as well.  

Yours,

Carol

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Al
Morton
Sent: Tuesday, July 20, 2010 4:11 PM
To: Carol Davids; [email-masked]
Subject: Re: [bmwg] Methodology/Terminology for Benchmarking SIP Networking
Devices -drafts -02 - Description of changes

regards,
Al
bmwg chair

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg