Subject: [bmwg] Multicast Methodology draft suggestions.

From: Kevin Dubray <[email-masked]>
Date: Tue, 05 Jun 2001 15:21:47 -0400

Hi,

I volunteered to provide the multicast methodology editors some
alternative wording to the min/max multicast latency metric so that
it may be closer in line to the intent of RFC 2432. (The current
wording seems to provide for a wide spectrum of descriptive
statistics, where RFC 2432 is very specific.)

You'll find that "alternative wording" for section 5.2 below. You'll
also find "alternatives" for section 5.1, Multicast Latency, and 
section 2, Keywords (RFC 2119).  The modifications of 5.1 were
to better align section 5.1 with section 5.2 and put more requirements 
on the reporting aspects.  Section 2 was modified to better
articulate the role of RFC 2119 and this non-standards track
document.

Anywho, for your consideration.

-Kevin

*******************************************************************

5.2 Min/Max Multicast Latency

Objective: To produce a specific, derived metric that demonstrates the
difference between the maximum latency measurement and the minimum
latency measurement from a set of multicast latency measurements.

Test Notes:  RFC 2432, the corresponding multicast benchmarking 
terminology document to this methodology document, suggests that 
this derived metric might be useful in characterizing a device's 
implementation, suggesting a more DUT-specific focus than SUT 
focus.

While the difference in time between the DUT/SUT presenting the
first tagged latency packet and the last tagged latency packet to
the receiving test apparatus might provide some insight into a 
device's implementation, this metric might have yield beneficial 
insight in a multiple device (SUT) scenario.  Regardless, most of 
the methodological details related to DUT/SUT scenarios lie with 
the Multicast Latency methodology specified in section 5.1.

However, particular attention to the test topology and packet 
egress paths is advised when attempting to interpret SUT-based 
results for this metric.  (E.g., How should the metric's 
result be interpreted if the test traffic has a different number 
of hops to each of its final SUT egress interfaces?)

Procedure: 

A.  The number of tested egress interfaces of the DUT/SUT, n, specified
    in Step A of Section 5.1, Multicast Latency, MUST be greater
    than 1.

B.  Execute the procedure for collecting multicast latency as prescribed
in section 5.1.  This will produce a set of multicast latencies, M,
where M is composed of individual forwarding latencies between DUT
packet ingress and DUT packet egress interface pairs. E.g.:

    M = { L(I,E0), L(I,E1), ..., L(I,En) }

where L is the multicast latency between a tested ingress
interface, I, and a specific, tested multicast egress interface 
of the DUT/SUT.  (E0 ... En) identify the unique, tested 
egress interfaces on the DUT/SUT.  

C.  From the collected multicast latency measurements in set M, 
identify MAX(M), where MAX is a function that yields the largest 
latency value from set M.  

Identify MIN(M), when MIN is a function that yields the smallest 
latency value from set M.  

D.  Determine the Max/Min value: 

        Result = MAX(M) - MIN(M) 

Reporting format: 

The result MUST be represented as a single numerical value in time 
units consistent with the corresponding latency measurements.  In 
addition, the number of tested egress interfaces on the DUT/SUT MUST 
be reported. 

All required reporting parameters of section 5.1, Multicast Latency,
MUST be reflected in the results report for this metric.

If the metric is to be used as a basis of comparison between assorted
devices, any differences in device configuration, test methodology, or
test parameters of the test trials MUST be reported.

******************************************************************

5.1 Multicast Latency

Objective: To produce a set of multicast latency measurements from a
a single, multicast ingress port of a DUT or SUT through multiple, 
egress multicast ports of that same DUT or SUT as provided for by the
metric "Multicast Latency" in RFC 2432.

The procedures highlighted below attempt to draw from the collection
methodology for latency in RFC 2544 to the degree possible.  The 
methodology addresses two topological scenarios: one for a single 
device (DUT) characterization; a second scenario is presented 
for multiple device (SUT) characterization.

Test Notes:

The procedure outlined below is written without regard 
for specific physical layer or link layer protocols. The
methodology further assumes a uniform medium topology. Issues
regarding mixed transmission media, such as speed mismatch, headers 
differences, etc., are not specifically addressed. Moreover,
no provisions are made for traffic-affecting factors, such as
congestion control or service differentiation mechanisms.     
Modifications to the specified collection procedures might need 
to be made to accommodate the transmission media actually tested.  
These accommodations MUST be presented with the test results.

The Offered Load accompanying the latency-measured packet can
affect the DUT/SUT packet buffering, which may subsequently impact 
measured packet latency.  This might be a consideration when 
selecting an Intended Load in Step B of this methodology.
  
Lastly, RFC 1242 and RFC 2544 draws distinction between two classes
of devices: "store and forward" and "bit-forwarding." Each class 
impacts how latency is collected and subsequently presented. See
the related RFCs for more information.  In practice, much of the
test equipment will collect the latency measurement for one class
or the other, and, if needed, mathematically derive the reported value
by the addition or subtraction of values accounting for medium
propagation delay of the packet, bit times to the timestamp trigger
within the packet, etc. Test equipment vendors SHOULD provide
documentation regarding the composition and calculation latency values
being reported.  The user of this data SHOULD understand the nature of
the latency values being reported, especially when comparing results
collected from multiple test vendors. (E.g., If test vendor A presents 
a "store and forward" latency result and test vendor B presents 
a "bit-forwarding" latency result, the user may erroneously conclude 
the DUT has two differing sets of latency values.)

Procedure: 

A.  If the test trial is to characterize latency across a single
Device Under Test (DUT), an example test topology might take the 
form of Figure 1 in section 3.  That is, a single DUT with one 
ingress interface receiving the multicast test traffic from
packet-transmitting component of the test apparatus and n egress
interfaces on the same DUT forwarding the multicast test traffic back 
to the packet-receiving component of the test apparatus.  Note that n 
reflects the number of TESTED egress interfaces on the DUT 
actually expected to forward the test traffic (as opposed to 
configured but untested, non-forwarding interfaces, for example).

If the multicast latencies are to be taken across multiple devices 
forming a System Under Test (SUT), then test packets are offered to
a single ingress interface on a device of the SUT, subsequently 
routed across the SUT topology, and finally forwarded to the 
test apparatus' packet-receiving components by the test egress 
interface(s) of devices in the SUT. Figure 3 offers an 
example SUT test topology.  If a SUT is tested, the details of 
the test topology MUST be disclosed  with the corresponding test 
results.

   +--------+                       +----------------+    +--------+
   |        |     +------------+    |DUT B Egress E0(-)-->|        |
   |        |     |DUT A       |--->|                |    |        |
   | Test   |     |            |    |      Egress E1(-)-->| Test   |
   | App.   |--->(-)Ingress, I |    +----------------+    | App.   |
   | Traffic|     |            |    +----------------+    | Traffic|
   | Src.   |     |            |--->|DUT C Egress E2(-)-->| Dest.  |
   |        |     +------------+    |                |    |        |
   |        |                       |      Egress En(-)-->|        |
   +--------+                       +----------------+    +--------+
   
                                 Figure 3 

Further, the DUT/SUT is to be configured with any required group
membership and multicast routing functionality to facilitate the
successful forwarding of the test traffic from the tested ingress 
interfaces to the tested egress interfaces. Again, the details of the 
forwarding enabling configuration SHOULD be disclosed in the test
report.  

B. The test equipment SHOULD be configured to generate a Traffic Class 
of a single source address, destined to a single multicast group, having
a uniform packet size, equal inter-packet gap for all packets 
constituting the test traffic, and a specific Intended Load [Ma 98] 
for a specific trial duration.  The Intended Load presented the DUT's 
ingress interface SHOULD NOT oversubscribe any egress interface. The 
trial duration SHOULD be 120 seconds.  Departures to the suggested 
traffic class guidelines MUST be disclosed with the respective trial 
results.  The nature of the latency measurement, "store and 
forward" or "bit forwarding," MUST be associated with the related 
test trial(s) and disclosed in the results report.  

C. End-to-end reachability of the test traffic path SHOULD be verified
prior to the engagement of a test trial.  This implies that subsequent
measurements are intended to characterize the latency across the tested
device's or devices' normal traffic forwarding path (e.g., faster 
hardware-based engines) of the device(s) as opposed a non-standard 
traffic processing path (e.g. slower, software-based exception 
handlers).  If the test trial is to be executed with the intent of 
characterizing a non-optimal, forwarding condition, then a description 
of the exception processing conditions being characterized MUST be 
included with the trial's results.

D. A test traffic stream is presented to the DUT in accordance the
to the configuration parameters specified in Step B.  At the
mid-point of the trial's duration, the test apparatus MUST inject a
uniquely identifiable ("tagged") packet into the test traffic packets 
being presented.  This tagged packet will be the basis for the 
latency measurements. By "uniquely identifiable," it is meant that 
the test apparatus MUST be able to discern the "tagged" packet from 
the other packets comprising the test traffic set.  A packet 
generation timestamp, Timestamp A, reflecting the completion of 
the transmission of the tagged packet by the test apparatus, MUST 
be determined. 

The test apparatus then monitors packets from the DUT's tested egress
port(s) for the expected tagged packet(s) until the cessation of 
traffic generation at the end of the configured trial duration.
A value of the Offered Load presented the DUT/SUT MUST be noted.

The test apparatus MUST record the time of the successful detection 
of a tagged packet from a tested egress interface with a timestamp, 
Timestamp B.  A set of Timestamp B values MUST be collected for all 
tested egress interfaces of the DUT/SUT.

Note that the nature of the latency measurement (i.e., "store and
forward" or "bit-forwarding"), defined in section 26.1 of RFC 1242,
impacts the conditions of Timestamp A and Timestamp B. 

E. A trial MUST be considered INVALID should any of the following
conditions occur in the collection of the trial data:

    * Forwarded test packets directed to improper destinations.
    * Unexpected differences between Intended Load and Offered Load 
      or unexpected differences between Offered Load and the
      resulting Forwarding Rate(s) on the DUT/SUT egress ports.
    * Forwarded test packets improperly formed or packet header
      fields improperly manipulated.
    * Failure to forward required tagged packet(s) on all expected
      egress interfaces.
    * Reception of a tagged packet by the test apparatus outside the
      configured test duration interval or 5 seconds, whichever is
      greater.

Data from invalid trials SHOULD be considered inconclusive.  Data from
invalid trials MUST not form the basis of comparison.

F. The set of latency measurements, M, composed from each latency
measurement taken from every ingress/tested egress interface
pairing MUST be determined from a valid test trial:

    foreach (e of Timestamps_B)

        M[e] = Timestamps_B[e] - Timestamp_A;

where Timestamps_B is the set of all tagged packet detection event 
timestamps individually associated with each tested egress interface 
of the DUT/SUT; "e" uniquely identifies a tested egress interface
of the DUT/SUT; Timestamp A is the single tagged packet transmission 
complete event timestamp noted for the DUT/SUT ingress interface. 
M[e] reflects an element of M representing the latency between the 
DUT/SUT ingress interface and a unique, tested DUT/SUT egress 
interface.

Said another way, a set of multicast latencies, M, is produced from
the collection of every latency measurement across the DUT/SUT ingress
interface and each tested DUT/SUT egress interface(s).

      M = { (Timestamp B(E0) - Timestamp A), 
            (Timestamp B(E1) - Timestamp A), ...
            (Timestamp B(En) - Timestamp A) }

where (E0 ... En) represents the range of all tested egress interfaces
and Timestamp B represents a tagged packet detection event for a given
DUT/SUT tested egress interface. 

Reporting format: 

Two types of information MUST be reported: 1) the set of latency
measurements and 2) the significant environmental, methodological, or
device particulars giving insight into the test or its results.

Specifically, when reporting the results of a VALID test trial, the set 
of ALL latencies related to the tested ingress interface and each tested
egress DUT/SUT interface of MUST be presented.  The time units of the
presented latency MUST be uniform and with sufficient precision for the
medium or media being tested.  Results MAY be offered in tabular format
and SHOULD preserve the relationship of latency to ingress/egress
interface to assist in trending across multiple trials. 

The Offered Load of the test traffic presented the DUT/SUT, size
of the "tagged" packet, trial duration, and nature (i.e., store-and-
forward or bit-forwarding) of the trial's measurement MUST be 
associated with any reported test trial's result.

If the attributes of the test traffic offered the DUT/SUT do not 
reflect the suggested guidelines of Step B, the following MUST be 
reported for the associated test trials:
  
  * Alternate traffic addressing  
  * Alternate interpacket-gap
  * Alternate packet size distribution 
  * Alternate trial duration

Tested device configuration(s) SHOULD be associated with any reported
test trial's results.

Test attributes MAY be report as a group for trials of like kind. 

Other statistical derivations might be applied to the Multicast Latency
results.  The scope and specification of these statistical exercises are 
beyond the scope of this metric.  However, if statistical techniques are 
utilized to summarize or otherwise provide insight regarding Multicast 
Latency results, the associated heuristics applied to Multicast Latency 
result set(s) MUST be disclosed in the test report.

Finally, if the results are to be used as a basis of comparison 
between assorted devices, any differences in device configuration, 
test methodology, or test parameters of the test trials MUST be 
reported.

*******************************
Section 2. Key Words to Reflect Requirements

In addition to the single paragraph on interpretation of words form 
RFC 2119, I'd add something to the effect:

RFC 2119 implies the usage of these keywords in IETF Standards-Track 
documents.  While this document will utilize the RFC 2119 interpretation
of these keywords, the reader should note that this benchmarking 
methodology document is not a Standards-track specification.

_______________________________________________
bmwg mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/bmwg