Subject: I-D ACTION:draft-ietf-bmwg-mcastm-04.txt

From: Kevin Dubray <[email-masked]>
Date: Tue, 11 Jul 2000 18:17:49 -0400

Subject: I-D ACTION:draft-ietf-bmwg-mcastm-04.txt
Date: Tue, 11 Jul 2000 06:29:18 -0400
From: [email-masked]
To: IETF-Announce: ;
CC: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology Working Group of the
IETF.

	Title		: Methodology for IP Multicast Benchmarking
	Author(s)	: H. Soor, D. Stopp, R. Daniels
	Filename	: draft-ietf-bmwg-mcastm-04.txt
	Pages		: 16
	Date		: 10-Jul-00
	
The purpose of this draft is to describe methodology specific to  the
benchmarking  of  multicast IP forwarding devices. It builds upon the
tenets set forth in RFC 2544, RFC 2432 and  other  IETF  Benchmarking
Methodology  Working  Group  (BMWG)  efforts.  This document seeks to
extend these efforts to the multicast paradigm.
The BMWG  produces  two  major  classes  of  documents:  Benchmarking
Terminology  documents  and  Benchmarking  Methodology documents. The
Terminology documents present the benchmarks and other related terms.
The  Methodology  documents define the procedures required to collect
the benchmarks cited in the corresponding Terminology documents.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-04.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-bmwg-mcastm-04.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-bmwg-mcastm-04.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-bmwg-mcastm-04.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-bmwg-mcastm-04.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

---

From: "John Dawson" <[email-masked]>
Date: Tue, 18 Jul 2000 11:23:43 -0700

Sorry for the length of the reply.

John

>4.1    Mixed Class Throughput
>The mixture of multicast and unicast traffic MUST be 
>set up in one of two ways: 
>
> a) As a percent of the total traffic flow resulting 
> in a ratio.  For example, 20 percent multicast traffic
> to 80 percent unicast traffic. 
>
> b) In evenly distributed bursts of multicast and 
> unicast traffic, resulting in a 50-50 ratio of multicast
> to unicast traffic. 
>
>The transmission of the frames MUST be set up so that 
>they form a deterministic distribution while still 
>maintaining the specified forwarding rates.

Is this traffic ratio relative to the transmitter or the receiver?

>Result 
>   
>Parameters to be measured SHOULD include the frame loss 
>and percent loss for each class of traffic per destination
>port.  The ratio of unicast traffic to multicast traffic 
>MUST be reported. 
>   
>In addition, the transmit and receive rates in frames per 
>second for each source and destination port for both unicast
>and multicast traffic, together with the number of frames
>transmitted and received per port per class type traffic 
>SHOULD be reported. 

Previous benchmarking RFC's have failed to resolve the issue of Offered vs.
Intended load.  This RFC has ignored the issue entirely.  Is this RFC
intended to be used in systems that support Pause capabilities and/or
half-duplex traffic.  If so, then the meaning of transmit and receive rates
in frames per second should be defined.  Are these average rates, maximum
rates, or instanteous rates that are sampled at some arbitrary point during
the middle of the test?  Are the numbers reported meant to be intended or
offered loads?

>4.4.2      Decapsulation Throughput 
>In Figure 4, the encapsulation process takes place in 
>DUT/SUT A. This may effect the throughput of the DUT/SUT B.
>Therefore, two test ports should be used to separate 
>the encapsulation and decapsulation processes. Client A 
>is replaced with the test port A which will generate a 
>multicast frame that will be encapsulated by DUT/SUT A. 
>Another test port B is inserted between DUT/SUT A and 
>DUT/SUT B that will receive the encapsulated frames and 
>forward it to DUT/SUT B. Test port C will receive the 
>decapsulated frames and measure the throughput. 

This is the first time I have ever seen the throughput tests using a
two-port device to both measure traffic rates and act as a forwarding point
within the test.  What kind of device is required for this test, a repeater,
or a switch, or a router?  

This test would appear to be opening pandora's box in terms of
reproducibility and accuracy of measurements.  My assumption is that this
two-port test device should be a passive tap or a repeater that can provide
an unbuffered replication of the original signal -- the tap signal is then
monitored by normal test equipment.  What are other people thinking this
two-port device would be?

>4.4.2 Decapsulation Throughput RESULTS
>Parameters to be measured SHOULD include the frame loss and
>percent loss per destination port for each multicast group 
>address. 
>
>In addition, the transmit and receive rates in frames per second
>for each source and destination port for all multicast groups,
>together with the number of frames transmitted and received per
>port per multicast groups SHOULD be reported. 

Is test port B considered a destination port?

Are loss rates at test port C made relative to transmitted rates from test
port B or relative to the transmitted rates from test port A?

>5.1    Multicast Latency 
>
>According to RFC 2544, a tagged frame is sent half way through 
>the transmission that contains a timestamp used for calculation 
>of latency. In the multicast situation, a tagged frame is sent 
>to all destinations for each multicast group and latency calculated
>on a per multicast group basis. Note that this test MUST be run 
>using the transmission rate that is less than the multicast 
>throughput of the DUT/SUT. Also, the test should take into account
>the DUT's/SUT's need to cache the traffic in its IP cache, fastpath
>cache or shortcut tables since the initial part of the traffic will
>be utilized to build these tables. 

Discussions of the previous RFC's have highlighted, time and time again,
that the throughput rate for a DUT is the rate at which the buffers are
completely full at the end of the test interval.  Latency tests that run at
or slightly below a DUT throughput rate are therefore likely to include
buffer depth as a significant component within the latency measurement.  Is
there any way to specify a more exact traffic rate for latency measurements
so that we are not measuring buffer depth -- use a pre-defined percentage of
the throughput rate that should not fill the buffers (i.e. 25 or 50%)?

>5.1    Multicast Latency Result 
>The parameter to be measured is the latency value for each multicast
>group address per destination port. An aggregate latency MAY also be 
>reported. 

The transmission rate (in frames per second) would seem to be important
here, would it not?

>5.2    Min/Max/Average Multicast Latency 
>Definition 
>First determine the throughput for DUT/SUT at each of the listed
>frame sizes determined by the forwarding and throughput tests of 
>section 4. Send a stream of frames to a fixed number of multicast 
>groups through the DUT at the determined throughput rate.

Latency measurements performed at the DUT throughput rate will be measuring
buffer depth, not the latency through the DUT. Is there any way to specify a
more exact traffic rate for latency measurements so that we are not
measuring buffer depth -- use a pre-defined percentage of the throughput
rate that should not fill the buffers (i.e. 25 or 50%)?

> -----Original Message-----
> From:	Kevin Dubray [SMTP:[email-masked]]
> Sent:	Tuesday, July 11, 2000 3:18 PM
> To:	[email-masked]

---

From: "Debby Stopp" <[email-masked]>
Date: Tue, 25 Jul 2000 09:50:07 -0700

> Is this traffic ratio relative to the transmitter or the receiver?

The traffic ratio is relative to the transmitter - we have reworded this
section as follows as a further clarification:

*****
The mixture of multicast and unicast traffic MUST be set up in one of two
ways:

        a) As a percentage of the total traffic flow employing maximum
bandwidth utilization. Thus, each type of traffic is transmitted at the
maximum available bandwidth. This also implies that the offered load,
regardless of the type of traffic, remains constant.

        b) As a percentage of the total traffic flow employing a
proportionate bandwidth utilization. Thus, each type of traffic is
transmitted at a fraction of the available bandwidth proportional to the
specified ratio. This also implies that the offered load for each traffic
type varies in proportion to its specified ratio.
*****

> the middle of the test?  Are the numbers reported meant to be intended or

We will reword to clarify offered load vs. intended load.

> Is test port B considered a destination port?

Yes - it's monitoring the progress of the encapsulated frames.

> port B or relative to the transmitted rates from test port A?

They are relative to the transmitted rates from test port A - test port B
does not transmit anything, it only listens.  We will clarify this section a
bit to reflect your comments.

> the throughput rate that should not fill the buffers (i.e. 25 or 50%)?

I don't think we can use a pre-defined percentage; what, for example, would
happen if your DUT/SUT throughput rate was 20%... then you would be back to
measuring the buffer depth again.  Perhaps an additional comment w.r.t
specifically the buffer depth should be added, ie:

'Note that this test MUST be run using the transmission rate that is less
than the multicast throughput of the DUT/SUT.  In addition,  the
transmission rate SHOULD be less than the rate at which the buffers are
depleted in order to avoid measuring buffer depth.'

> The transmission rate (in frames per second) would seem to be important

Good comment; we will update the document & add this parameter as a
'SHOULD' in the result report.

> rate that should not fill the buffers (i.e. 25 or 50%)?

Same comment from previous section; we can add the following comment from
the previous section:

'Note that this test MUST be run using the transmission rate that is less
than the multicast throughput of the DUT/SUT.  In addition,  the
transmission rate SHOULD be less than the rate at which the buffers are
depleted in order to avoid measuring buffer depth.'

Debby Stopp
Principal Software Engineer
(818) 444-2365

Ixia
26601 W. Agoura Rd.
Calabasas
CA 91302
Tel: 818 871 1800 (US) +1 818 871 1800 (international)
http://www.ixiacom.com

Ixia...
      When the test really counts.