Subject: BMWG New Work Proposal: L3 QoS Benchmarking

From: Kevin Dubray <[email-masked]>
Date: Thu, 03 Feb 2000 08:26:05 -0500

BMWG,

Below is a proposal for the BMWG to undertake new work in the area of
"network-layer quality of service testing."  This new work appears to be
consistent with the BMWG charter; the proposal is offered for your 
consideration as a new work item for the WG.

A 2 week open discussion period, ending 17 Feb 00, will allow the WG to
question, comment on, or suggest modifications to the proposal.

A 1 week WG Last Call, ending 24 Feb 00, will be issued on the proposal, or 
its successor, to determine if the BMWG wishes to move forward with this 
new work.

The WG chairperson reserves the right to extend either the open discussion 
or the Last Call period based on the volume and quality of commentary.

-Kevin

----------------------------------------------------------------------------

Workplan Proposal: Network-Layer Quality of Service Testing 

S. Poretsky, Avici Systems 
D. Newman, Network Test

January 2000

This memo proposes the development of standard practices for evaluating 
the performance of network-layer quality of service (QoS) mechanisms. We 
propose to develop terminology and methodology documents, prepared under 
the auspices of the Benchmarking Working Group (BMWG) of the Internet 
Engineering Task Force (IETF).

1. Effort's Motivation

Driven by exponential traffic growth, demand for bandwidth-hungry 
applications, and desire for guaranteed service levels, vendors of Internet 
routers and switches had added mechanisms intended to ensure quality of 
service (QoS) levels for IP traffic. There is currently no standard practice 
for evaluating whether these QoS mechanisms actually work as advertised. As a 
result, there are widely varying definitions of QoS terms and methodologies 
for evaluating QoS performance. The lack of a standard testing methodology 
makes it difficult to perform meaningful comparisons of vendor implementations 
for the same QoS mechanism. It is equally difficult to compare performance 
of different QoS mechanisms.

2. Effort's Goals

The general goal of this effort is to develop terminology and methodology 
documents for benchmarking the performance characteristics of Layer-3 QoS 
mechanisms. QoS can be applied at many layers. This work will focus on QoS 
at Layer 3 (Network Layer) using two mechanisms, the IP precedence and 
differentiated services code point (DSCP) fields defined in the IP header.

The authors' intent with these documents is to enable even and fair comparison 
of routing/switching devices that use dissimilar QoS hardware or software. We 
plan to apply the same conceptual framework established for benchmarking QoS 
performance of routers and switches so that each device under test will be 
identically configured and offered identical traffic.

3. Plan and Milestones

This effort will follow the standard BMWG procedure of producing terminology 
and methodology documents. The QoS terminology document will serve as the 
foundation for the methodology document by defining QoS metrics and other 
concepts related to IP precedence and DSCP settings. The main metrics to be 
addressed in the terminology document include:

 --forwarding rate
 --packet loss
 --delay
 --jitter

The terminology document also will define QoS concepts and parameters that 
will be used in the methodology document.

Since IP precedence and DSCP fields are different mechanisms with different 
intended effects on traffic, there will be two methodology documents -- one 
apiece to describe measurement procedures for devices using IP precedence and 
the DSCP field. These methodology documents will describe procedures for 
measuring QoS levels, building on the concepts and metrics described in the 
terminology document. The methodology document also will include source 
traffic profiles used to drive the measurements.

3.1 Terminology Document  

This document will define and describe the various terms used in benchmarking 
router/switch QoS performance. It will follow the framework established in 
RFC 1242 [2], including for each term a definition, discussion, measurement 
units, issues, and list of related terms.

As noted, the terminology document will introduce metrics and concepts used 
in measurement of QOS mechanisms based on IP precedence and DSCP fields.

3.1.1 Terminology Milestones

We propose the following timetable for completion of the QoS terminology 
document:

  February 2000    First draft presented to BMWG 
  December 2000    WG Last Call on QoS Benchmarking Terminology I-D.

3.2 Methodology Documents

After the BMWG reaches rough consensus on the terminology document, the 
authors propose to write companion methodology documents. As noted, the 
authors propose separate methodology documents for measuring QoS performance 
of devices using IP precedence and DSCP field mechanisms. These documents will 
follow the format used in RFC 2544 in describing specific requirements of the 
test bed and test traffic, along with specific procedures to follow. The 
procedures will follow the objective/procedure/reporting format framework used 
in RFC 2544.

3.2.1 Methodology Milestones

We propose the following timetable for completion of the QoS test methodology 
documents.

3.2.1.1 IP Precedence Methodology

  December 2000   First draft of Benchmarking Methodology for QoS Using IP 
                    Precedence I-D presented to BMWG. 
  June 2001       WG Last Call on Benchmarking Methodology for QoS Using IP
                    Precedence I-D.

3.2.2.2 DSCP Methodology

  March 2001     First draft of Benchmarking Methodology for QoS Using DSCP 
                    I-D presented to BMWG.
  November 2001  WG Last Call on Benchmarking Methodology for QoS Using DSCP.