Subject: Throughput Questions

From: "Talbert, Brian" <[email-masked]>
Date: Thu, 6 Jul 2000 14:56:41 -0400

I have a question regarding the measurement of throughput.   I am using an
Ixia chassis to perform RFC 2544 throughput testing.    The test setup
involves two routers with a WAN connection, similar to the following:

Ixia sending port ---------> RouterA ----- T1 circuit ----RotuerB --------->
Ixia receive port

It appears that the sending port begins offering frames at the given rate
and size and also begins a duration counter.   It sends frames until the
duration counter reaches zero.   The receive port collects all marked frames
and counts them.    The problem I appear to be having with this is that
because of buffering at just about every interface, the receive port may
count frames longer than the duration timer.   The results look like rates
that exceed T1 line speed (I've gotten bps rates over 1.7 mbps).   I would
think the timer should be on the receive port, should it not?    

Does anyone have any specifc input that my help shed some more light on
this? 

Thanks,

______________________________________
Brian Talbert, Network Consulting Engineer
UUNET an MCI WorldCom Company
* email:  [email-masked]
* phone:  703-886-5812

---

From: Scott Bradner <[email-masked]>
Date: Sat, 8 Jul 2000 07:06:23 -0400 (EDT)

> am i missing something or would would one not also want to specify the
> inter-packet delay distribution?

most of the performance testing assumes constant inter packet interval 
(anything else gets too hard to interpret)

Scott

---

From: "Perser, Jerry" <[email-masked]>
Date: Mon, 10 Jul 2000 11:57:10 -0700

The results are technically incorrect.  RFC 1242 states that the DUT/SUT can
not drop any frames offered.  It does not state how long the DUT/SUT has to
forward these frames.

RFC 2544 (sections 23 and 24) states a trail duration of 60 second and to
"wait for two seconds for any residual frames to be received."  This will
give an accuracy of 3 1/3 percent.  On a 1.544 Mbps T1 line, the maximum
throughput reading should be 1.595 Mbps due to buffering.

I know of two techniques to compensate for the buffering.  The first one was
mentioned by John Dawson.  Increasing the test duration minimizes the
buffering effect.  The problem that arises is how long of a duration to use?
Longer duration mean longer test time and fewer results per day.

I use another technique with the SmartBits.  I run the throughput test per
the RFC.  This tells me at what offered load the DUT/SUT does not drop any
frames.  Then I run a single SmartMetrics Latency test (standard feature on
all of the SmartMetrics modules) at that offered load. 

SmartMetrics Latency test gives me minimum and maximum latency of all the
frames forwarded.  The minimum latency represents the quickest path through
the DUT/SUT (no buffering).  The maximum latency represents longest path
through the DUT/SUT (full buffering) before dropping.  The difference is the
buffering time.  Add the buffer time to the test duration for calculating
the throughput.  The modified formula is:

                            input bits
   Throughput = ----------------------------------------------
                (trial duration) + (max latency - min latency)

The formula does not take into account back pressure.  The "Benchmarking
Methodology for LAN Switching Devices" draft, Appendix B discusses ways to
compensate for back pressure.  Also note that most of the buffering will be
done by router A.

___________________________________________________________ 
Jerry Perser               Phone: 818.676.2320 
SmartLab Manager           Lab:   818.676.2337 
Netcom Systems Inc.        Fax:   818.880.9165     
26750 Agoura Road          Corp:  818.676.2300 
Calabasas, CA, 91302       Pager: [email-masked]

---

From: "John Dawson" <[email-masked]>
Date: Mon, 10 Jul 2000 13:10:48 -0700

Jerry,

Here is the grind.  The RFC also states to wait for the transmitter to
transmit all frames.  In some cases (i.e. half-duplex and pause based
scenario's) the test can not transmit all required frames in 60 seconds.
How is the test program supposed to know when to stop waiting for residual
frames if the trial duration is in question in the first place?

Your equations are essentially trying to calculate offered load in a more
precise fashion.  However, it is also changing some of the base requirments
from a Transmit based process to a Receive based process.

Since none of the RFC's have addressed exactly how offered load is
determined (its' still in draft), and since the RFC makes no explicit
requirement about measuring latency during a throughput test sequence,  I
don't think it would be fair to call the results incorrect.

I would agree that Throughput results are subject to a certain amount of
measurement error, but I believe that this error is inherent to the process
(no process is perfect).  However, I would be curious as to the time
duration that was used in the original test sequence that started this
entire discussion.  If the original test was 10 second or 30 second
duration, then your error assumptions are going to change substantially.
Are we worrying about something that is a non-issue/known-issue? 

John

> -----Original Message-----
> From:	Perser, Jerry [SMTP:[email-masked]]
> Sent:	Monday, July 10, 2000 11:57 AM
> To:	BMWG (E-mail)
> Subject:	RE: Throughput Questions
> 
> The results are technically incorrect.  RFC 1242 states that the DUT/SUT
> can not drop any frames offered.  It does not state how long the DUT/SUT
> has to forward these frames.
> 
> I know of two techniques to compensate for the buffering.  The first one
> was mentioned by John Dawson.  Increasing the test duration minimizes the
> buffering effect.  The problem that arises is how long of a duration to
> use?  Longer duration mean longer test time and fewer results per day.
> 
> frames.  Then I run a single SmartMetrics Latency test (standard feature
> on all of the SmartMetrics modules) at that offered load. 
> 
> frames forwarded.  The minimum latency represents the quickest path
> through the DUT/SUT (no buffering).  The maximum latency represents
> longest path through the DUT/SUT (full buffering) before dropping.  The
> difference is the buffering time.  Add the buffer time to the test
> duration for calculating the throughput.  The modified formula is:
> 
>                             input bits 
> compensate for back pressure.  Also note that most of the buffering will
> be done by router A.
>

---

From: "Talbert, Brian" <[email-masked]>
Date: Mon, 10 Jul 2000 18:07:18 -0400

I brought up the discussion to begin with ... the engineer that performed
the original test sequence used Ixia Scriptmate, using the RFC2544
throughput script.   The duration was set to 60 seconds.   The test setup
was as follows:
 
Ixia 10/100 port --------- Cisco 2620 -----T1(Frame Relay encap) ----- Cisco
7513 ------- Ixia 10/100 port
 
Using frame sizes of 1518 she got a throughput rate of 149 packets per
second.   This works out to well over 1.595bps that is mentioned as the
maximum with buffering considerations so I am fairly certain there is
something going on other than a simple calculation discrepancy.  
 
Brian Talbert

-----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Monday, July 10, 2000 4:11 PM
To: 'Perser, Jerry'
Cc: BMWG (E-mail)
Subject: RE: Throughput Questions

Jerry, 

John 

	-----Original Message----- 
From:   Perser, Jerry [SMTP:[email-masked]] 
Sent:   Monday, July 10, 2000 11:57 AM 
To:     BMWG (E-mail) 
Subject:        RE: Throughput Questions 

	The results are technically incorrect.  RFC 1242 states that the
DUT/SUT can not drop any frames offered.  It does not state how long the
DUT/SUT has to forward these frames.

	RFC 2544 (sections 23 and 24) states a trail duration of 60 second
and to "wait for two seconds for any residual frames to be received."  This
will give an accuracy of 3 1/3 percent.  On a 1.544 Mbps T1 line, the
maximum throughput reading should be 1.595 Mbps due to buffering.

	I know of two techniques to compensate for the buffering.  The first
one was mentioned by John Dawson.  Increasing the test duration minimizes
the buffering effect.  The problem that arises is how long of a duration to
use?  Longer duration mean longer test time and fewer results per day.

	I use another technique with the SmartBits.  I run the throughput
test per the RFC.  This tells me at what offered load the DUT/SUT does not
drop any frames.  Then I run a single SmartMetrics Latency test (standard
feature on all of the SmartMetrics modules) at that offered load. 

	SmartMetrics Latency test gives me minimum and maximum latency of
all the frames forwarded.  The minimum latency represents the quickest path
through the DUT/SUT (no buffering).  The maximum latency represents longest
path through the DUT/SUT (full buffering) before dropping.  The difference
is the buffering time.  Add the buffer time to the test duration for
calculating the throughput.  The modified formula is:

	                            input bits

	The formula does not take into account back pressure.  The
"Benchmarking Methodology for LAN Switching Devices" draft, Appendix B
discusses ways to compensate for back pressure.  Also note that most of the
buffering will be done by router A.

---

From: "Perser, Jerry" <[email-masked]>
Date: Mon, 10 Jul 2000 15:57:13 -0700

> transmit all frames.

I can't find this in RFC 1242 or 2544.  My opinion agrees with this
statement.  My point is: I don't think it is stated.  Please correct me on
this.

> In some cases (i.e. half-duplex and pause based scenario's) the test can 
> based scenario transmit all required frames in 60 seconds.  How is the
test > program supposed to know when to stop waiting for residual frames if
the 
> trial duration is in question in the first place?

Brian issue was not back pressure.  All the frames were transmitted within
the test duration.  Brian's problem was that "The maximum rate at which none
of the offered frames are dropped" (RFC 1242 section 3.17) exceeded both the
Maximum Forwarding Rate and the BPS of the T1 line.

> precise fashion.  However, it is also changing some of the base
requirements 
> don't think it would be fair to call the results incorrect.

Per RFC 2544 section 23, the receive counter is active 60 seconds (step c)
plus 2 seconds (step d).  All the transmitted frames MUST be received in
those 62 seconds with no loss.  A 60 second offered load could be stretched
to a 62 seconds.  This would be an error of 3 1/3 percent.

Suppose we transmit 1.7 Mbps for 60 seconds; we receive 1.5 Mbps for 66
seconds; the scripts do not comply with RFC 2544 section 23 step d.  This
would explain Brian's symptoms.  The results are invalid to either 1)
operator error or 2) Scripts do not comply with RFC 2544. 

> Are we worrying about something that is a non-issue/known-issue? 

The issue is the reliability of the test equipment and test methodology.
Brain's concern (and mine in the past) is why does a foot measure 13 inches.
My next troubleshooting step would to use a stop watch.  Measure the
transmit time and the receive time.  This would be an independent way to
confirm the trail duration.

John

---

From: "John Dawson" <[email-masked]>
Date: Mon, 10 Jul 2000 15:55:19 -0700

Brian,

The 1.595 Mbps number that Jerry mentioned is the maximum possible without
buffering and assuming a maximum of 2 seconds as a cut-off between the end
of transmission and the end of reception.

A buffer size of 2 megabytes on the Cisco would be enough to account for the
discrepency you mention.

Here is how I calculated that:
  1518 bytes packets at 149 packets per second is roughly 1.809 Mbits per
second (less preamble and gap).
  Since the T1 channel is running at 1.544 Mbps (assuming no compression)
there is a delta of 265k bits per second.
  Over a 60 second test, the buffer would have to hold 60 * 265kbits =
15.9Mbits or about 2 MBytes

This would suggest that even though the transmitter ran for 60 seconds, the
receiver continued running an additional 10 seconds past the end of
transmission (70 seconds total).  However, if the Cisco was employing any
compression on the data stream then all the numbers would change.

Since the RFC bases Throughput on the transmitted rate and not the receive
rate, the channel capacity will be over-stated.  There will always be
measurement error between the offered load and the channel rate for a
Throughput tests that is dependent upon the size of the buffers in the DUT.
Throughput was not designed to measure channel rates in the fashion you are
expecting, it is more of a buffering rate test.

The maximum forwarding rate test would be a more appropriate test to measure
the channel rate.  The forwarding rate test is based upon maximum bits per
second rates at the receiver. 

John
==================================

> From:	Talbert, Brian [SMTP:[email-masked]]
> Sent:	Monday, July 10, 2000 3:07 PM
> To:	Dawson, John [SC6:382:EXCH]; 'Perser, Jerry'
> Cc:	BMWG (E-mail); [email-masked]'
> Ixia 10/100 port --------- Cisco 2620 -----T1(Frame Relay encap) -----
> Cisco 7513 ------- Ixia 10/100 port
>  
> 	Here is the grind.  The RFC also states to wait for the transmitter
> to transmit all frames.  In some cases (i.e. half-duplex and pause based
> 	Your equations are essentially trying to calculate offered load in a
> more precise fashion.  However, it is also changing some of the base
> requirments from a Transmit based process to a Receive based process.
> 
> 	I would agree that Throughput results are subject to a certain
> amount of measurement error, but I believe that this error is inherent to
> the process (no process is perfect).  However, I would be curious as to
> the time duration that was used in the original test sequence that started
> this entire discussion.  If the original test was 10 second or 30 second
> 		The results are technically incorrect.  RFC 1242 states that
> the DUT/SUT can not drop any frames offered.  It does not state how long
> the DUT/SUT has to forward these frames.
> 
> 		RFC 2544 (sections 23 and 24) states a trail duration of 60
> second and to "wait for two seconds for any residual frames to be
> received."  This will give an accuracy of 3 1/3 percent.  On a 1.544 Mbps
> T1 line, the maximum throughput reading should be 1.595 Mbps due to
> buffering.
> 
> 		I know of two techniques to compensate for the buffering.
> The first one was mentioned by John Dawson.  Increasing the test duration
> minimizes the buffering effect.  The problem that arises is how long of a
> duration to use?  Longer duration mean longer test time and fewer results
> per day.
> 
> 		I use another technique with the SmartBits.  I run the
> throughput test per the RFC.  This tells me at what offered load the
> DUT/SUT does not drop any frames.  Then I run a single SmartMetrics
> Latency test (standard feature on all of the SmartMetrics modules) at that
> offered load. 
> 
> 		SmartMetrics Latency test gives me minimum and maximum
> latency of all the frames forwarded.  The minimum latency represents the
> quickest path through the DUT/SUT (no buffering).  The maximum latency
> represents longest path through the DUT/SUT (full buffering) before
> dropping.  The difference is the buffering time.  Add the buffer time to
> the test duration for calculating the throughput.  The modified formula
> is:
> 
> 		                            input bits
> discusses ways to compensate for back pressure.  Also note that most of
> the buffering will be done by router A.
>

---

From: "John Dawson" <[email-masked]>
Date: Mon, 10 Jul 2000 16:45:02 -0700

Jerry,

If you want to debate the RFC literally.

Section 5. Requirements 
   An implementation is not compliant if it fails to satisfy one or more
   of the MUST requirements for the protocols it implements.  

Section 23. Trial description
   c) Run the test trial.
   d) Wait for two seconds for any residual frames to be received.

Section 23 contains no mandatory requirements. It has no must, could, or
shoulds.

I believe that you are streching the meaning of the spirit of the document
to say that the test trial must time the receive path to within 2 seconds of
the transmitted duration.  If you want to interpret this literally, then the
test duration is strictly the only thing specified.  For a sixty (60) second
test, no test would be allowed to transmit longer than 60 seconds or recieve
longer than 62 seconds.  If a test transmits a fixed number of frames
(independent of time) then it would not meet your definition of compliant.
I would then argue that just about every implementation that supports either
half-duplex or pause control is non-compliant.

However, lets look at some mandatory requirements in section 24.

24. Trial duration
   The aim of these tests is to determine the rate continuously
   supportable by the DUT.  The actual duration of the test trials must
   be a compromise between this aim and the duration of the benchmarking
   test suite.  

The fact that we are dealing with a slow channel and reasonably large
buffers, would suggest that a two second period for residual frames may not
be sufficient.  Section 24 would therefore state that the actual trial
duration must be reconsidered.  This section could also suggest that a 60
second test is insufficient for such a slow channel.  A longer test duration
may therefore be needed to get the results you desire.

I do not believe that the Throughput test was ever designed to report
channel capacities with the accuracy you are attempting to achieve.  The
maximum forwarding rate test would be the more appropriate test.

John

---

From: "Debby Stopp" <[email-masked]>
Date: Tue, 18 Jul 2000 19:52:53 -0700

RE: Throughput Questions<I've tried to forward this several times; hopefully
this time it will make it through, sorry for the lateness of the response>
 Subject: RE: Throughput Questions

Here's something more from my soapbox...

I would like to suggest that the apparent discrepancy between the throughput
results and theoretical frame rate of the T1 link could be due to the
difference in inter-device encapsulation on the frame relay link compared to
the Ethernet links. Has this been investigated?

It is not the concern of the test equipment how the packets are delivered to
the receiving end. The fact is, the packets *are* showing up. Are there any
sources of potential error here?

1. The transmitter may be saying that it is transmitting more frames than it
really is.

No. The transmitter is quite accurate.

2. The receive counter is saying that it is receiving more packets than it
really is.

Again, no. The received frames are counted based on a specific match on a
pattern inserted in the payload at the transmitter. It is unlikely that the
intervening equipment is sourcing packets that contain the same proprietary
pattern.

If buffering is responsible for the eventual delivery of the packets to the
receiving end, then extending the duration of the test traffic should cause
the reported throughput to asymptotically approach the true bandwidth of the
intermediate bandwidth-limited pipe (as stated before). If there is still a
discrepancy between empiricism and theory, then it is up to the equipment
vendor to determine exactly how the packets are being delivered across the
link. The fact remains, the packets *are* showing up at the receiving end.

As to whether or not using the last successful *transmit* rate is
appropriate under these circumstances, that is for greater minds than mine
to contemplate.

Best Regards,

- Gene

Gene Stopp
Ixia
Ph. 818-871-1800 ext. 3214

Ixia…
            When the test really counts.

  -----Original Message-----
  From: Debby Stopp [mailto:[email-masked]]
  Sent: Tuesday, July 11, 2000 9:16 AM
  To: [email-masked]
  Cc: [email-masked]; [email-masked]
  Subject: FW: Throughput Questions

  Here's some of Gene's comments (not yet public, to be cleaned up before
expose)

  d.

  -----Original Message-----
  From: Gene Stopp [mailto:[email-masked]]
  Sent: Monday, July 10, 2000 4:58 PM
  To: Debby Stopp; Hardev Soor
  Cc: Cliff Hannel; [email-masked]; [email-masked]
  Subject: RE: Throughput Questions

  I am prepared to respond to this if needed. I just ran testTP from
100M --> 10M --> 100M (100M test with a 10M bottleneck in the middle). I am
looking at the numbers now. Our test correctly uses the transmit *count* of
the last passing (i.e. no drop) iteration, and calculates the througput in
frames per second based on this count and the transmit duration. There is no
math mistake.

  If a customer gets a number, it is the correct number. We are PUTting the
packets THROUGH, and we are printing the THROUGH-PUT. Do the math, read the
RFC. The frames are getting through at a certain TRANSMIT rate. Any
discrepancies are therefore due to buffering effects in the intermediate
devices, which should be reduced by extending the duration of the test.

  Mark Robinson has also pointed out a potentially important fact - the T1
link is not using Ethernet encapsulation, but rather Frame Relay, which has
a substantially lower overhead on a per-packet basis (3 bytes FR header, 2
byte CRC, 1 Flag as opposed to 14 byte Ethernet header and 4 byte CRC). Mark
also says that Cisco uses a proprietary FR header that uses even fewer
bytes.
  Best Regards,

  - Gene

  Gene Stopp
  Ixia
  Ph. 818-871-1800 ext. 3214

  Ixia…
              When the test really counts.

    -----Original Message-----
    From: Debby Stopp [mailto:[email-masked]]
    Sent: Monday, July 10, 2000 4:45 PM
    To: Hardev Soor
    Cc: Gene@Ixiacom. Com; Cliff Hannel
    Subject: RE: Throughput Questions

    I have been following this discussion very closely for the past week,
just waiting for an opportune time to jump in if required.  However, I
believe that John Dawson is very nicely stating our point of view wrt how
the RFC was originally intended.

    Gene; I think it may be time for that well-intended doc we've been
talking about to be drafted, if for no other reason than to support John
Dawson in his suppositions.

    d.
      -----Original Message-----
      From: Hardev Soor [mailto:[email-masked]]
      Sent: Monday, July 10, 2000 4:16 PM
      To: Debby Stopp
      Cc: Gene@Ixiacom. Com; Cliff Hannel
      Subject: FW: Throughput Questions

      Debby, it may come to a point when they might start asking us
questions. Have you seen these emails? Do you know why they would see this
discrepancy in our throughput test?
      I am attaching Brian's original email.
      Thanks.
      (o>-------------------------------------
      Hardev Soor
      Product Manager
      Ixia
      26601 West Agoura Road
      Calabasas, CA 91302
      Tel: (818) 444-3213
      Mobile: (818) 422-1236
      Email: [email-masked]
      www.ixiacom.com

      Ixia ...
           When the test really counts.

      -----Original Message-----
      From: [email-masked]
[mailto:[email-masked]]On Behalf Of Talbert, Brian
      Sent: Monday, July 10, 2000 3:07 PM
      To: 'John Dawson'; 'Perser, Jerry'
      Cc: BMWG (E-mail); [email-masked]'
      Subject: RE: Throughput Questions

      I brought up the discussion to begin with ... the engineer that
performed the original test sequence used Ixia Scriptmate, using the RFC2544
throughput script.   The duration was set to 60 seconds.   The test setup
was as follows:

      Brian Talbert
        -----Original Message-----
        From: John Dawson [mailto:[email-masked]]
        Sent: Monday, July 10, 2000 4:11 PM
        To: 'Perser, Jerry'
        Cc: BMWG (E-mail)
        Subject: RE: Throughput Questions

        Jerry,

        I would agree that Throughput results are subject to a certain
amount of measurement error, but I believe that this error is inherent to
the process (no process is perfect).  However, I would be curious as to the
time duration that was used in the original test sequence that started this
entire discussion.  If the original test was 10 second or 30 second
duration, then your error assumptions are going to change substantially.
Are we worrying about something that is a non-issue/known-issue?

        John

          -----Original Message-----

          I know of two techniques to compensate for the buffering.  The
first one was mentioned by John Dawson.  Increasing the test duration
minimizes the buffering effect.  The problem that arises is how long of a
duration to use?  Longer duration mean longer test time and fewer results
per day.

                                      input bits