Subject: Router Benchmarking Framework and Metric

From: Kevin Dubray <[email-masked]>
Date: Fri, 17 Mar 2000 16:59:39 -0500

There are two documents that are to be developed this year.  We believe
that the first "Framework for Router Metrics and Benchmarking" is a
document that MUST be written before any other router documents are
submitted. This framework document will be format of RFC 2330 "Framework
for IP Performance Metrics", May 1998.  We also believe that router
documents can be submitted when most of the "Framework for Router
Benchmarking" is at completed, there is no need to wait to have this
published as an RFC before other work can begin.

Cynthia Martin
Jeff Dunn
ANC, Inc.
-----------------------------------------------------------------------

Work Plan for "Framework for Router Metrics and Benchmarking"

I. Introduction

The purpose of this memo is to define a general framework for particular
metrics to be developed by the Benchmarking Methodology Working Group
(BMWG) of the Operational Requirements Area. This memo extends existing
work, specifically RFC 2330 "Framework for IP Performance Metrics", May
1998.

This document will lay out several criteria for the routing metrics to be
developed.  These criteria are designed to promote a BMWG effort that will
maximize an accurate common understanding by Internet users and Internet
providers of the performance and reliability both of end-to-end paths
through the Internet and of specific 'IP clouds' that comprise portions of
those paths.

This document will reference Internet vocabulary, clearly describing
Internet components such as routers, routing protocols, and router MIB
element definitions.  Any additional router related vocabulary necessary to
develop router metrics will be defined in this document. Measurement
uncertainties and errors will be described, including how they relate to
the analytical framework shared by many aspects of the Internet engineering
discipline.

The remainder of the document will provide the definitions of configuration
and state parameter sets which effect IP packet forwarding performance in
routers.  Specifically, the following four classes of configuration and
state parameter sets will be defined:
1)   static configuration parameters
2)   dynamic configuration parameters
3)   static state parameters
4)   dynamic state parameters

II. Framework Document Milestones

April 2000 - First draft of framework document submitted to working group

July 2000 - Rough consensus as to framework

December 2000 - Final thrashing out of document and last call

Work Plan for "Metrics for IPv4 Router Benchmarking"

I. Introduction

This memo will discuss and define the basic metrics associated with
performance benchmarking tests and the results of these tests in the
context of router devices supporting IPv4. Terminology for router and
routing protocols is already defined in various IETF RFC's and ID's and
will be not discussed in the memo.

This document assumes that necessary services are available and active to
provide routing services. This document covers only exterior routing
protocols, interior routing protocols, and their interaction.  The metrics
are generic and can be applied to all IETF defined routing protocols for
IPv4.  It may be necessary to define routing protocol specific metrics in
separate documents.  This document presents only the metrics associated
with benchmarking IP performance in routers; therefore, it does not
represent a total compilation of router test metrics.

II. Objective

The goal of this effort is to produce a set of metrics, from which a
methodology can be derived, to characterize the effects of a router's
configuration and state on IP packet forwarding performance as defined in
RFC 2544 "Benchmarking Methodology for Network Interconnect Devices".
"Framework for Router Metrics and Benchmarking" ID should be consulted
before attempting to make use of this document.  Metrics will be defined in
accordance with the framework specified in RFC 1242 "Benchmarking
Terminology for Network Interconnect Devices".  Specifically, this effort
will focus on the effects of the following configuration and states
parameter sets on IP packet forwarding performance:

1. Static configuration parameters, e.g., route cache vs. total available
memory size

2. Dynamic configuration parameters, e.g., BGP4
MinRouteAdvertisementInterval

3. Static states, e.g., response to BGP4 NLRI updates

4. Dynamics states, e.g., response to conflicting BGP4 NLRI updates (route
flapping)

Metrics will be defined which characterize both the impact on IP packet
forwarding performance and router response to route updates based on these
states.  Packet forwarding performance assessment will be based on metrics
described in RFCs 1242, 2285 and 2761.  The assessment of router response
will be based on the values of MIB objects described in RFC 1850 and the
upcoming BGP MIB document.

Specific metrics will include:

1. IP packet: Delay Variation, Error Ratio, Loss Rate, Latency,
Throughput and many more

2. Router: Interface and Interface Metric Table state, MIB Alignment
Time, Neighbor and Virtual Neighbor Table state, Route Update Response
Time and many more

III. Metric Document Milestones

April 2000 - First draft of metric document submitted to working group

July 2000 - Rough consensus as to metric definitions

December 2000 - Final thrashing out of document and last call

IV. Future developments

December 2000 - First draft of methodology document based on rough
consensus on metrics

March 2001 - Final draft of methodology document and last call