Subject: [OPS-AREA] APM BOF Proposal

From: "Alan Clark" <[email-masked]>
Date: Mon, 21 May 2007 07:15:55 -0400

We are proposing a BOF for the Chicago meeting to address the need for a WG
or Directorate related to application performance metrics.  We would welcome
any comments or feedback on this, and if the BOF is accepted then would very
much welcome your participation.
 
Regards
 
Alan Clark, Al Morton

APM BOF 

Application Performance Metrics 

Mailing Lists: TBD 

BOF Chairs: 
Alan Clark [[email-masked]] 
Al Morton [[email-masked]] 

Possible Area Directorates: Operations and Management Area, TSV, RAI
Ultimate Area Directorate: TBD 

BOF Problem Statement:

There have always been some uncertainties about the performance and
suitability of new technologies and applications for their intended
audience, and the Internet is no exception. Most uncertainties are 
effectively addressed through quantified assessment of key performance
indicators.  Standardized performance metrics add the desirable features
of consistent implementation, interpretation, and comparison.

Although the IETF has two Working Groups dedicated to the development
of performance metrics, they each have strict limitations in their 
charters:
 
 - The Benchmarking Methodology WG has addressed a range of networking 
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but their charter strictly
limits their performance characterizations to the laboratory environment.

 - The IP Performance Metrics WG has the mandate to develop metrics 
applicable to live IP networks, but it is specifically prohibited from 
developing metrics that characterize traffic (such as a VoIP stream).

The IETF does have current and completed activities related to the 
reporting of application performance metrics (e.g. RAQMON) and is 
also actively involved in the development of reliable transport 
protocols which would affect the relationship between IP performance 
and application performance. 

APM proposals that are beyond the scope of bmwg and ippm have begun to
appear.
Some recent drafts include:
http://www.ietf.org/internet-drafts/draft-malas-performance-metrics-06.txt 
http://tools.ietf.org/html/draft-venna-ippm-app-loss-metrics-00

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for application-layer protocols that
can be used to characterize traffic on live networks.
 
There is a need for an application performance framework and 
application performance metrics that enable IP based applications 
to be characterized (and hence managed) when operating over 
UDP, TCP, FEC and the other robust transport protocols under development 
within IETF. However, the work undertaken must be strictly evaluated 
on an item by item basis, to avoid overlap with standards organizations 
already developing application performance metrics in this general area, 
ITU-T SG 12 for example. This is the problem to be solved. 

Potential Solutions:

Two possible approaches to solving this problem have been suggested.

 + APM Directorate 
This Directorate would advise working groups initiating APM work (e.g. AVT
for 
Voice and IP video quality metrics, sipping for SIP performance metrics, idr

for BGP metrics) and provide expert reviews. Members of the directorate
could
review protocols for manageability from a performance perspective. The
Directorate members would be drawn from experts currently involved in ippm, 
bmwg, ipfix, and other working groups. The APM directorate 
would be available to consult in all IETF areas, like other directorates.
The members of the Directorate would create a BCP document that would 
provide guidance to the reviewers and reference for editors of IETF APM 
metrics documents. 

 + APM WG 
This WG would be responsible for the development of RFCs that characterize
the performance of applications, especially those utilizing IETF protocols.
The WG would be responsible for advancing the RFCs along the standards
track, according to the process defined in draft-bradner-metricstest-01.txt
.
The WG would partner with a specific protocol development WG whenever 
possible, to take advantage of willing experts and ensure that the metrics
developed are both appropriate and relevant. Last, the WG would create a 
BCP document that would provide guidance for development of IETF APM 
metrics documents, including the requirement to include relevant protocol 
experts in the development.  The BCP may also define the process to 
adopt new work in the APM WG, where a unique feature could be that proposals
would be vetted in the relevant protocol WG before reaching the APM WG.
The APM WG would avoid overlap with existing SDOs, and seek IESG
approval for each new Metric RFC before the WG spends any resources
on it.
(more is needed for a complete charter)

BOF Agenda:

Examination of a perceived gap in IETF coverage and Problem Statement

Comparison of Possible Solutions:
 + APM Directorate (with working process)
 + APM WG (with charter and working process)

Discussion

Assess consensus on the following questions:

1. Should IETF commit its resources to the formal development of 
application performance metrics in one or more key areas?

2. If yes, is there a preferred approach to manage metric development:
 + Protocol Development WGs with advice from an APM Directorate 
 + APM WG with participation by experts from experts from the Protocol WG.
 + Both an APM Directorate and an APM WG are needed
 + Neither an APM Directorate or an APM WG are needed

3. If the APM Directorate approach is desirable, is the proposed method
of working acceptable, or are modifications needed?

4. If the APM WG approach is desirable, is the proposed charter
acceptable, or are modifications needed?

_______________________________________________
OPS-AREA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ops-area