Subject: [bmwg] bmwg-multicast document review

From: "John Dawson" <[email-masked]>
Date: Mon, 29 Oct 2001 12:20:50 -0800

Hi All,

In a one-on-one discussion I told Kevin about some concerns that I had
relative to the multicast benchmarking.

He had asked that I please review the document again and put my concerns
down in writing for the entire group.

I apologize in advance for the length of the document, but there are many
issues that I believe MUST be
addressed relative to the proposed methodology.

If you think I am off-base, please say so.

John
------------------------------------------------------------------------
>>  Section 3.1. Test Considerations 
   
>> Moreover, no provisions are 
>> made for traffic-affecting factors, such as congestion control or 
>> service differentiation mechanisms.  Modifications to the specified 
>> collection procedures might need to be made to accommodate the 
>> transmission media actually tested.  These accommodations MUST be 
>> presented with the test results. 

I don't agree with this language.  We have suffered for too many years with
throughput and performance
tests that allow congestion control and forward pressure techniques to skew
results.  It is time to close
these holes.  This paragraph essentially grants license to change any
collection procedures to accommodate
some designer agenda.  If the procedures laid out in this RFC can not be
used to characterized a specific
media, then this RFC should not be used as a basis for analyzing that
technology.  Although the authors
specify that any accommodations must be presented with the test results,
this action allows anyone to
re-engineer the test process and then leaves it to the ingenuity of the
reader to catch the tricks that may
be employed to mis-direct the less educated.

Are we agreeing to a process here? or are we granting everyone free license
to re-write their own test?

>> Section 4.1 Mixed Class Throughput
>> Result 
>> Parameters to be measured SHOULD include the frame loss and percent 
>> loss for each class of traffic per destination port.  The ratio of 
>> unicast traffic to multicast traffic MUST be reported. 

Let's specify what must be measured, not what should be measured.

Proposed Change:

	Parameters to be measured MUST include the test duration, number of
frames offered,
	and number of frames properly received for the aggregated unicast
and multicast traffic
	stream per destination port.  The test duration MUST be a measure of
the actual time 
	required for the source port to transmit the number of frames
offered.

The proposed change reflects the fact that OFFERED load requires both a TIME
and FRAME COUNT
measurement.  If you do not measure both components, then you are reporting
intended load numbers.
The terminology definitions for the classical RFC "Throughput Test" clearly
states that OFFERED load 
is what matters, not intended load.  The weakness of the requirements
provided here is that it does 
not require any tracking of traffic improperly delivered.  Does our
loss-less forwarding requirement assume
that unicast frames should not appear at the wrong ports?  Traditionally, I
don't believe this has been required.

>> The nature of the traffic stream contributing to the result MUST be 
>> reported.  All required reporting parameters of mixed class 
>> throughput MUST be reflected in the results report, such as the 
>> transmitted packet size(s) and offered load of the packet stream. 

Let's specify what must be reported -- What are the required parameters?
Don't make me guess,
STATE IT HERE...

Proposed Change:

	Results for mixed class throughput MUST include the following
parameters:  a).  The maximum
	aggregate offered frames per second, from the source port, at which
no loss forwarding is observed.  
	b).  The ratio of unicast traffic to multicast traffic at each
destination port.  c).  The transmitted 
	packet size(s).  d). The number of source and destination ports
within the multicast group.   
	e).  The test duration.  f).  The granularity of the measurement
process.

The proposed change goes back to what is the objective of the measurement
and what is needed to put
this number into proper context.  By specifically binding the offered Load
measurement to the source port
it prevents people from adding lots of destination ports and counting each
multicast packet more than once.
By requiring the ratio (unicast to multicast) to be measured at each
destination port, it shows what is 
happening to the traffic ratio after switching occurs.  Also by requiring
reporting at each destination, either 
all ports must have the same ratio mix or a separate number must be reported
for each port (if the ports are 
not uniformly loaded). The number of source ports is expected to be one,
reporting the number of destination
ports allows the ratio of unicast to multicast to be recalculated relative
to the source port.  The granularity of 
the measurement process prevents someone from using a coarse measurement to
unfairly exaggerate results 
-- consider if one product is known to have a throughput of 21% while
another is at 19%, by using a coarse 
measurement one product gets a 20% result while another gets a 15% or 10%
result.

>> Section 4.2 Scaled Group Forwarding Matrix
>> Results
>>     Parameters to be measured SHOULD include the frame loss and percent 
>>     loss per destination port for each multicast group address. 
>>     The nature of the traffic stream contributing to the result MUST be 
>>     reported.  All required reporting parameters MUST be reflected in 
>>     the results report, such as the transmitted packet size(s) and 
>>     offered load of the packet stream. 

I have the same type of comments for this section.  The section indicates
what should or could be measured
but makes no real requirements for what MUST be reported.  What is required
to describe the nature of the
traffic streams?  What are the required parameters that MUST be reported?
Don't make me guess, don't
let me cheat!!

>> Section 4.3 Aggregate Multicast Throughput
>> Procedure 
>>     Multicast traffic is sent at a fixed percent of maximum offered 
>>     load with a fixed number of groups at a fixed frame length for a 
>>     fixed duration of time. 
>>     The initial number of receive ports of the tester will join the 
>>     group(s) and the sender will transmit to the same groups after a 
>>     certain delay (a few seconds). 
>>     Then the an incremental number of receive ports will join the same 
>>     groups and then the Multicast traffic is sent as stated. 
>>     The receive ports will continue to be added and multicast traffic 
>>     sent until a user defined maximum number of ports is reached.

Reading the procedure I could not tell that throughput measurements were
actually the point of the exercise.
Initially I thought this was trying to measure how many ports could be added
before loss started to occur.  

However, I do see a big difference between the original terminology and this
methodology.  The terminology
document specifies the results as an ordered set  (N,t) implying that the
number of IGMP address being used in
the group is one.  However, when I read this methodology I can not determine
if the fixed number of groups relates
to ports or IGMP addresses.  Is there only one IGMP address being tested?
Is there only one group of ports, or are
different IGMP addresses being used by different groups of ports?  I can
read the methodology and structure several
different test scenarios.

I believe that what this attempts to test is one multicast group that has
its size as the variable.  If this is the case,
then lets get rid of any reference to groups and group(s).  Otherwise our
terminology is not (N,t) it is (Ngroups, Mports, t).

My comments relative to reporting are the same for this section as the prior
sections.  I wouldn't have a clue what I
should be reporting based upon the current form of the document.

>> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput 

The figure shows several measurement points (a, b, c, d, e, and f).  The
points B and F are duplicates of A and E.
If these measurement points are not identical then it is not appropriate to
show them being directly connected to
one another.

>> 4.4.1. Encapsulation Throughput
>>Procedure 
>>     Traffic is sent through a DUT/SUT that has been configured to 
>>     encapsulate the frames. Traffic is received on a test port prior to 
>>     decapsulation and throughput is calculated based on RFC2544. 
      
I would like to see the generic "traffic" term replace with "multicast
traffic" and some reference to the fact that this
involves a single IGMP group address.

>>     Results 
>>     Parameters to be measured SHOULD include the measured throughput 
>>     per tunnel. 
>>     The nature of the traffic stream contributing to the result MUST be 
>>     reported.  All required reporting parameters of encapsulation 
>>     throughput MUST be reflected in the results report, such as the 
>>     transmitted packet size(s) and offered load of the packet stream. 

Proposed Change :
	Results

	Parameters to be measured MUST include the test duration, number of
frames offered,
	and number of frames properly received for the multicast traffic
group address at tunnel point C.
	The test duration MUST be a measure of the actual time required for
the source port to transmit 
	the number of frames offered.

	Results for Encapsulation Throughput MUST include the following
parameters:  a).  The maximum
	offered multicast frames per second, from the source port, at which
no loss forwarding is observed.  
	b).  The transmitted packet size(s).  c). The test duration. d).
The granularity of the measurement 
	process.

Let's specify where the measurement is made -- is it tunnel point C or
tunnel point D.  If it is tunnel point D,
then the characteristics of the Network between C and D is a big factor and
we are no longer reporting
encapsulation throughput for the DUT but rather for the SUT (which is now a
DUT and Network device).  I am
pretty much against trying to perform this measurement at tunnel point D.  I
would prefer to see the Network
connection replaced by a passive tap or an active layer 1 repeater (don't
allow the network device to be a switch).

>> Section   4.4.2. Decapsulation Throughput 
>>     Procedure 
>>     Encapsulated traffic is sent through a DUT/SUT that has been 
>>     configured to decapsulate the frames. Traffic is received on a test 
>>     port after decapsulation and throughput is calculated based on 
>>     RFC2544. 

Can we add some words saying that this traffic is multicast traffic.

>>     Results 
>>     Parameters to be measured SHOULD include the measured throughput 
>>     per tunnel. 
>>     The nature of the traffic stream contributing to the result MUST be 
>>     reported.  All required reporting parameters of decapsulation 
>>     throughput MUST be reflected in the results report, such as the 
>>     transmitted packet size(s) and offered load of the packet stream. 
       
Proposed Change :
	Results

	Parameters to be measured MUST include the test duration, number of
frames offered at tunnel
	point C, and number of frames properly received from the multicast
traffic group address at tunnel 
	point E.  The test duration MUST be a measure of the actual time
required for the source port 
	(client A) to transmit the number of frames offered.

	Results for Decapsulation Throughput MUST include the following
parameters:  a).  The maximum
	offered multicast frames per second, from tunnel point C, at which
no loss forwarding is observed.  
	b).  The transmitted packet size(s).  c). The test duration. d).
The granularity of the measurement 
	process.

These changes attempt to remove any ambiguity as to where the throughput
measurement is made (i.e. 
between tunnel points C and E and not A to F).

>> Section  4.4.3. Re-encapsulation Throughput

This section should explicitly cite that we are considering multicast
traffic for a single IGMP group address.
The results section should be modified to include what MUST be measured and
reported.

>> Section  5. Forwarding Latency 
>>     The tests should also take into account the 
>>     DUT's/SUT's need to cache the traffic in its IP cache, fastpath 
>>     cache or shortcut tables since the initial part of the traffic will 
>>     be utilized to build these tables. 

I would strike this entire portion.  It appears to give permission to the
test author to change the test for each
device architecture.  Do we have one test process or a multitude?  Maybe the
only words that need to be 
removed are "should also" -- this RFC is either engineered to be meaningful
or it is not.

>> Section 5.1 Multicast Latency
>>  If the test trial is to be 
>>     executed with the intent of characterizing a non-optimal, 
>>     forwarding condition, then a description of the exception 
>>     processing conditions being characterized MUST be included with the 
>>     trial's results. 

I can't understand what this is trying to say -- What are my requirements
here?

>> A trial MUST be considered INVALID should any of the following 
>> conditions occur in the collection of the trial data: 
>>   . Forwarded test packets directed to improper destinations.

If this is a multicast packet destined for all ports, how can test packets
go to the wrong destinations?
Is this test mixing unicast and multicast?  I don't see this described in
the procedure.

>>  . Failure to forward required tagged packet(s) on all expected egress
interfaces. 

If the test is below the throughput rate, then this exception should not
happen.  I think this case should
be included in the results and specifically flagged as a failing case.

>>     Data from invalid trials SHOULD be considered inconclusive.  Data 
>>     from invalid trials MUST not form the basis of comparison. 

Recommended change :

	Data from invalid trials SHOULD be considered inconclusive and
represent specific failure conditions.
	The number of invalid trials and the reason for invalidating data
MUST be reported in the final report.

>> Results 
>>     Two types of information MUST be reported: 1) the set of latency 
>>     measurements and 2) the significant environmental, methodological, 
>>     or device particulars giving insight into the test or its results. 

Information in category 2 is meant to be specified by this RFC and is not to
be left to variation.  Why should the
methodology be changing?  What purpose is served by requiring that device
particulars MUST be reported?

In general, a single latency measurement in isolation is not extremely
meaningful.  Many RFCs require more 
than one set of measurements.  This RFC suggests that multiple sets of
measurements are made but does 
not require any minimum set of measurements.  Is a single measurement set
sufficient?

>> Section   6.1. Group Join Delay 
   
>>     Procedure 
>>     Traffic is sent on the source port at the same time as the IGMP 
>>     JOIN Group message is transmitted from the destination ports.  The 
>>     join delay is the difference in time from when the IGMP Join is 
>>     sent (timestamp A) and the first frame is forwarded to a receiving 
>>     member port (timestamp B). 

How is timestamp A observed?  You have a couple cases to consider.  The DUT
either supports a VLAN 
collection of ports where one port acts as the join requestor for the
collection of ports (in which case the 
IGMP join request is forwarded and observable), or snooping functions are
being used -- where the DUT 
does not forward requests between ports and each port is allowed a separate
set of multicast groups.  

If we are talking about the Snooping case, the transmitted time stamp A is
observable only by the DUT.  
Unless the test equipment source has the ability to track timestamps in both
the transmitted and received
direction, this measurement is going to be difficult.  I guess I will have
it take it on faith that test equipment
suppliers will be able to meet these requirements.

One way to make the test more observable is to slightly alter the procedure
such that the joining station
actually sends more than one packet.  If the joining station where to send a
unicast frame, a join frame, and 
a second unicast frame in rapid succession, then timestamps for the unicast
frames could be observed on
a different port.  This would require only received data to be captured.
The timestamp for the join message
could then be estimated to be half way between the two unicast time stamps.

>> Section 6.2  Group Leave Delay

I have the same concerns as expressed in section 6.1.

>> Section 8.1 and 8.2

There is no real methodology here.  There are no real reporting requirements
specified here.

Latency under load numbers are quite tricky and this section glosses over
that complexity. 

I believe that this section of the document needs a methodology that can be
defined, implemented, and 
defended.  However, since the BMWG has taken on no prior latency under load
efforts, this document is
left to define everything -- which I believe is impractical.  I believe that
these sections should be identified
as not being possible to complete until such time as agreement as to
appropriate latency under load
measurements can be standardized.

If you don't want to wait for methodology to be developed, then at least
give me some methodology to actually
follow.  Don't just repeat the terminology and give me a for example as to
how the test might be useful.

-----------------------------------------------------------------END of
Comments -------------------------------------------------------