Subject: [bmwg] WGLC on IPv6 Benchmarking

From: Al Morton <[email-masked]>
Date: Tue, 20 Feb 2007 22:49:27 -0500

BMWG:

A WG Last Call period for the Internet-Draft on

"IPv6 Benchmarking Methodology"
<draft-ietf-bmwg-ipv6-meth-01.txt>

will be open from 20 Feb 2007 through 20 March 2007.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ipv6-meth-01.txt

Version 01 incorporates changes suggested during development
prior to the draft becoming a chartered item, and more recent
comments that have been shared with the authors.
The authors believe that all comments from the working group
have been addressed, and they would also like to
point out that their mixed traffic methodology has been proven with an
actual test tool implementation of their draft recommendations.

This draft is entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the
BMWG Last Call Review Template in the message of 4/27/04
titled, "Refinements to the BMWG Last Call Process".
Volunteers should contact [email-masked]
A copy of the template is available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review,
please express your opinion on whether or not this Internet-Draft
should be submitted to the Area Directors for consideration to
publish the draft as an Informational RFC.  Send your comments
to this list or [email-masked]

Al
BMWG Co-Chair 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Tue, 20 Feb 2007 22:49:37 -0800

> actual test tool implementation of their draft recommendations.

Some nits:

- - Appendices A.1 and A.2 refer to "maximum throughput." There is no such
metric. RFC 1242 defines throughput as a single rate, not a range with
minimum and maximum values.

- - Appendix A.1 erroneously rounds up some maximum rates.

For example, the maximum rate for 64-byte frames on gigabit Ethernet is
given as 1,488,096 fps. In fact, the formula given in this same section
yields the correct rate of 1,488,095.24 fps.

Even if a rate were x.74 fps, the number should either be rounded down
if expressed in integer form or (better yet, in my opinion) presented as
a floating-point number.

- - Ethernet's maximum rates are subject to some tolerance due to clock
slop. It would be useful to note that the rates given are theoretical
maximums, and that actual rates may vary by +/- 100 ppm (with
conversions for the appropriate pps for each flavor of Ethernet).

dn

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Wed, 21 Feb 2007 07:33:16 -0500

Thank you David for your review and for your comments! We will integrate
them in the next version of the document.

Best Regards,
Chip 

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Wednesday, February 21, 2007 1:50 AM
To: Al Morton
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

On 2/20/07 7:49 PM, Al Morton wrote:
> Version 01 incorporates changes suggested during development prior to 
> the draft becoming a chartered item, and more recent comments that 
> have been shared with the authors.
> The authors believe that all comments from the working group have been

> addressed, and they would also like to point out that their mixed 
> traffic methodology has been proven with an actual test tool 
> implementation of their draft recommendations.

Some nits:

- - Appendices A.1 and A.2 refer to "maximum throughput." There is no
such metric. RFC 1242 defines throughput as a single rate, not a range
with minimum and maximum values.

dn

> At this time, we are seeking volunteers to complete the BMWG Last Call

> Review Template in the message of 4/27/04 titled, "Refinements to the 
> BMWG Last Call Process".
> Volunteers should contact [email-masked] A copy of the template is 
> available here:
> Whether or not you decide to volunteer for the directed review, please

> express your opinion on whether or not this Internet-Draft should be 
> submitted to the Area Directors for consideration to publish the draft

> as an Informational RFC.  Send your comments to this list or 
> [email-masked]
> 
> Al
> BMWG Co-Chair
>

---

From: "Timmons C. Player" <[email-masked]>
Date: Thu, 22 Feb 2007 13:04:34 -0500

I have a couple questions/comments.

1.  I'm glad to see the back-to-back test being deprecated.  I've always
thought that it was mostly useless for modern DUT's.

2.  Why is 64 bytes used as the minimum frame size for SONET?  I
understand that 64 is the minimum for Ethernet, but the minimum for
SONET is either 46 or 48 (40 byte packets), so it seems like that should
be included in the recommended frame size list...

Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Thu, 22 Feb 2007 13:23:55 -0500

Cool, but I actually was referring to section 5.1.2. Frame Sizes to be
used on SONET.

Timmons

Chip Popoviciu (cpopovic) wrote:
> Thank you Timmons for your review and comments. It would be no problem
> to include the smaller frame size since this is done in the Annex as a
> reference.
> 
> Best Regards,
> Chip  
> 
> From: Timmons C. Player [mailto:[email-masked]] 
> Sent: Thursday, February 22, 2007 1:05 PM
> To: [email-masked]

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Thu, 22 Feb 2007 13:20:22 -0500

Best Regards,
Chip  

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:05 PM
To: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Thu, 22 Feb 2007 13:23:55 -0500

Timmons

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Thu, 22 Feb 2007 13:20:22 -0500

Best Regards,
Chip  

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:05 PM
To: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

Timmons

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Thu, 22 Feb 2007 13:33:10 -0500

Putting this into the annex is straight forward, but including in 5.1.2
presents the following interesting scenario:
- on one hand we wanted to take this opportunity to add to RFC2544 by
including SONET (something needed for both IPv6 and IPv4)
- on the other, a 48 byte frame size would be useless in the case of
IPv6 (with its 40 bytes main header)

This is why my thought went straight to the Annex. What are your (the
list) thoughts on this? 

Best Regards,
Chip

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:24 PM
To: Chip Popoviciu (cpopovic)
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

Timmons

> 1.  I'm glad to see the back-to-back test being deprecated.  I've 
> always thought that it was mostly useless for modern DUT's.
> 
> SONET is either 46 or 48 (40 byte packets), so it seems like that 
> should be included in the recommended frame size list...
> 
> Timmons

---

From: "McQuaid, Jim" <[email-masked]>
Date: Thu, 22 Feb 2007 12:37:18 -0600

I think the 64-byte test is important for Sonet, because sonet is not a
"last mile" connection.  It will have one performance characteristic for
smaller data, but likely a much worse one for 64-byte frames.  However,
in today's world, the ethernet-concentric value is the test figure of
merit, I believe.

Jim McQuaid

Jim McQuaid | Senior Product Manager  
NetQoS, Inc. | 4601 Creekstone Drive, Suite 180, Durham, NC 27703
512.674.9541 desk |  919.619.3220 mobile | [email-masked]
NetQoS: Performance Experts
www.netqos.com

-----Original Message-----
From: Chip Popoviciu (cpopovic) [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:33 PM
To: Timmons C. Player
Cc: [email-masked]
Subject: RE: [bmwg] WGLC on IPv6 Benchmarking

Best Regards,
Chip

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:24 PM
To: Chip Popoviciu (cpopovic)
Cc: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

Timmons

---

From: "Timmons C. Player" <[email-masked]>
Date: Thu, 22 Feb 2007 13:40:21 -0500

But won't a 64 byte Ethernet frame end up as a 46 byte IP packet on Sonet?

Timmons

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Fri, 23 Feb 2007 10:59:39 -0500

Timmons,

I think Jim's point regarding SONET not being the last mile technology
is important in the overall discussion. Of course, the same argument
might make us think that in order to tie the access layer traffic to the
core layer traffic we would keep the IP datagram sizes that correspond
to the recommended Ethernet frame sizes (access) and add to them the
SONET layer two overhead. This will lead to a new set of recommended
frame sizes for SONET. One thing to consider is the fact that with such
an approach, the recommendations might become more complex since we have
various L2 encaps for a given technology. For example, the 64 bytes
frame for Ethernet leads to a 44 bytes IP packet which in the case of
SONET would be a 51 bytes (with a 2 bytes FCS) or a 53 bytes (with a 4
bytes FCS) frame.

Has this approach been considered within the WG in the past? Looking for
opinions on this topic. I think that in the case of Ethernet and SONET
this correlation is logical. The changes you suggest would be also
justified (independent of the above mentioned correlation) by the
RFC2544 approach which defines the frame sizes by identifying the
smallest possible and largest possible:

"Specifically, the sizes SHOULD include the maximum and minimum
   legitimate sizes for the protocol under test on the media under test
   and enough sizes in between to be able to get a full characterization
   of the DUT performance."

Looking forward to thoughts and suggestions.

Best Regards,
Chip  

-----Original Message-----
From: Timmons C. Player [mailto:[email-masked]] 
Sent: Thursday, February 22, 2007 1:40 PM
To: McQuaid, Jim
Cc: Chip Popoviciu (cpopovic); [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

But won't a 64 byte Ethernet frame end up as a 46 byte IP packet on
Sonet?

Timmons

McQuaid, Jim wrote:
> I think the 64-byte test is important for Sonet, because sonet is not 
> a "last mile" connection.  It will have one performance characteristic

> for smaller data, but likely a much worse one for 64-byte frames.  
> However, in today's world, the ethernet-concentric value is the test 
> figure of merit, I believe.
> 
> Jim McQuaid
> 
> 
> Jim McQuaid | Senior Product Manager NetQoS, Inc. | 4601 Creekstone 
> Drive, Suite 180, Durham, NC 27703
> Putting this into the annex is straight forward, but including in 
> 5.1.2 presents the following interesting scenario:

>>> to include the smaller frame size since this is done in the Annex as

>>> a

---

From: David Newman <[email-masked]>
Date: Fri, 23 Feb 2007 15:48:18 -0800

On 2/23/07 7:59 AM, Chip Popoviciu (cpopovic) wrote:

> this correlation is logical.

bmwg scope is limited both explicitly and historically to testing boxes
in the lab, not on live production networks.

For that reason I do not think it makes sense to rule in or out given
packet lengths because of where boxes *might* be used in production
today or at some future point. We can't predict the future. We can
design tests so they stress the technology in front of us right now.

For example: When 1242 and 1944/2544 were written 10-15 years ago,
Ethernet boxes weren't flinging packets over distances of 40 km or more.
Instead, the notion was that Ethernet supported a 64-byte minimum frame,
so that's what should be tested.

I would hope the same approach would apply here. Sonet supports 40-byte
packets, the shortest length possible for IPv6. Ergo, that length should
be included in any methodology, regardless of where a Sonet device
ultimately gets deployed.

dn

---

From: "Chip Popoviciu \(cpopovic\)" <[email-masked]>
Date: Sat, 24 Feb 2007 01:31:30 -0500

David,

I think this is reasonable. 

So for IPv4 we are using the minimum packet size of 40 bytes plus the 9
bytes overhead. The minimum IPv4 header is 20 bytes so such a packet
leaves us 20 bytes of payload to work with. With IPv6, 40 bytes provides
room just for the main header so is that a reasonable test? If we give
the IPv6 packet a 20 bytes payload as well then the minimum frame size
becomes 69 bytes. Under the circumstances, it almost makes sense to
stick to the original 64 bytes minimum frame size for SONET.

Best Regards,
Chip  

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Friday, February 23, 2007 6:48 PM
To: [email-masked]
Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

On 2/23/07 7:59 AM, Chip Popoviciu (cpopovic) wrote:

> Has this approach been considered within the WG in the past? Looking 
> for opinions on this topic. I think that in the case of Ethernet and 
> SONET this correlation is logical.

dn

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 24 Feb 2007 11:52:34 -0500

David,

Some nits on your nits.

See inline.

In message <[email-masked]>
David Newman writes:
>  
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>  

That is sort of a bug and could be reported as errata against that
RFC.  The term "throughput" is commonly referred to as an average rate
that is acheived given a particular load.  In an operating network,
the throughput is the number of bits actually flowing, not the maximum
acheivable rate.  For example, if a 10g interface has 1g of traffic on
it, the throughput is 1g, not 10g.  The maximum acheivable throughput
might be 10g (we can pretend it is, though for practical reasons it
rarely is).  It makes no sense for the benchmarking community to
redefine a commonly used term that has a somewhat different meaning to
the rest of the community.  I don't think that was ever the intention
of RFC1242 but I can't speak for the author.  Scott can clarify.

Perhaps if we wanted a concise definition which was consistent with
the wider Internet community usage, RFC1242 could have defined
"maximum throughput" and then noted that for the purpose of
benchmarking "throughput" is commonly understood to be mean "maximum
throughput" (sometimes called "device throughput" or with sterling
clarity "maximum throughput acheivable with a given device").  If we
wanted to be very clear RFC1242 could also have stated that in network
operations (more common) "throughput" is commonly understood to be
mean the throughput acheived in practice and observed at any given
instant on an operational network and in the transport protocol
community "throughput" is commonly understood to be mean the acheived
end-to-end throughput observed by a given host pair.

I think we can live with RFC1242 as is, as long as the above is
understood and need not be explicitly stated.  Most readers will find
your point and my response to be incredibly pedantic.

> yields the correct rate of 1,488,095.24 fps.

Personally I think 7 significant figures is more than enough.
Considering that in a real network 95% utilization would be considered
spectacular, it seems silly to put 9 significant figures in the RFC.
(Of course real networks has nothing to do with marketing figures so
it may depend on what you think the purpose of benchmarking is.)  A
note that the figure has been rounded could be made.

> if expressed in integer form or (better yet, in my opinion) presented as

Or reported as "just over 1.488 mpps".

> conversions for the appropriate pps for each flavor of Ethernet).

Which makes reporting "just over 1.488 mpps" even more reasonable.  If
+/- 100ppm is accurate (~150pps), then it could be under 1.488 mpps so
maybe "approximately 1.488 mpps" might be better.  Reporting more
significant figures would be contrary to normal scientific and
engineering practice of reporting only the number of significant
figures known to be accurate.  A note about rounding and clock rate
inaccuracy wouldn't hurt.

Curtis

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (Darwin)
>  
> iD8DBQFF2+uByPxGVjntI4IRAiZOAJ4uF3c85HBTZafNyMj/c7mTUBBiwACfWAkQ
> B5pEtNcwVge7xJQZheeOYMw=
> =JIeu
> -----END PGP SIGNATURE-----
>

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 24 Feb 2007 12:47:16 -0500

In message <[email-masked]>
> thought that it was mostly useless for modern DUT's.

RFC2544 indicates that 1024 frames back to back would be called for
for testing any protocol.  Its a SHOULD, not a MUST.  IMHO the figure
of 1024 is too small and testing using bursts of small packets is also
called for.  The figure you'll get is not as important as passing the
test without packet drops.  The recommendation in the draft is based
on people who do vendor benchmarking not liking non-repeatable figures
as opposed to people who do ISP benchmarking not liking equipment
which can't be relied upon to deliver busty traffic.  I think your
point is that the latter is or at least should be extinct.

ISPs (smart ones) will test for this whether its in the spec as a
SHOULD or a MUST or a SHOULD NOT as it is here.  You might be
surprised by some "modern DUTs" and smart ISPs prefer to be surprised
in their test lab rather than on their production network.  If they
don't get surprised in the lab its still time well spent.

> be included in the recommended frame size list...

Some discussion a while back had voice over micrograms, smaller than
TCP ACKs.  I can dig that up to get the packet size but I do agree
that bursts of 40 byte payloads should be considered.  Which goes back
to the importance of bursty traffic testing.  Average packet sizes
in real networks have never been as low as 64 bytes so 64 bytes as a
means to determine pps limitations is reasonable.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 24 Feb 2007 13:10:05 -0500

In message <[email-masked]>
> in today's world, the ethernet-concentric value is the test figure of

SONET is often a "last mile" connection.  It might not be a last meter
connection.  Big science used to be known for the oddest last mile and
last meter arrangements but were never known for using tinygrams over
these.  Your point about ethernet being the host system interface of
choice seems valid though it might be common in the real world for
multiple 10gE (and even more common for a lot more that 10 1gE) to be
feeding a router with a single OC192 (or less).  Ethernet carries
small packets with padding, so they can end up being 40 bytes.  With
TCP ACK being the only common source of IPv4 40 byte packets, a lot of
40 byte IPv6 packets does seem very unlikely.  VOIP micrograms (to the
extent that they exist) are not going to arrive at full line rate
except perhaps at very small values of SONET line rate.

Curtis