Subject: Re: [bmwg] WGLC on IPv6 Benchmarking

From: [email-masked] (Scott O. Bradner)
Date: Sun, 25 Feb 2007 17:33:49 -0500 (EST)

> > 1.  I'm glad to see the back-to-back test being deprecated.  I've always
> > thought that it was mostly useless for modern DUT's.
> 
> RFC2544 indicates that 1024 frames back to back would be called for
> for testing any protocol.  Its a SHOULD, not a MUST.  IMHO the figure
> of 1024 is too small and testing using bursts of small packets is also
> called for.  The figure you'll get is not as important as passing the
> test without packet drops.  The recommendation in the draft is based
> on people who do vendor benchmarking not liking non-repeatable figures
> as opposed to people who do ISP benchmarking not liking equipment
> which can't be relied upon to deliver busty traffic.  I think your
> point is that the latter is or at least should be extinct.

my experience (a long time ago with hardware from a long time ago) was
that there was no useful repeatability in the back-to-back test (17% 
variance in one product between runs, in the same range in a number
of other products) - do I think that the b2b test should be deprecated

but I do agree with Curtis that some useful information can be found -
if the max burst is very short that could be an indication of problems
that could hurt but its more of a warning sign than anything quanitative

and I agree that 1024 is too short a burst these days - not sure what it
should be though

> ISPs (smart ones) will test for this whether its in the spec as a
> SHOULD or a MUST or a SHOULD NOT as it is here.  You might be
> surprised by some "modern DUTs" and smart ISPs prefer to be surprised
> in their test lab rather than on their production network.  If they
> don't get surprised in the lab its still time well spent.

agree

Scott

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: [email-masked] (Scott O. Bradner)
Date: Sun, 25 Feb 2007 17:37:32 -0500 (EST)

> I would observe at the same time that RFC 2544 is out of synch by
> two generations in what Ethernet standard (non-jumbo) maximum frame
> length is concerned. The figure of 1518 mentioned all over RFC 2544 was
> replaced long time back by 1522 in order to accommodate the IEEE 802.1Q
> VLAN header, and 1522 was more recently replaced by 2000, as per IEEE
> 802.3as

true - but I'm not sure how much of a performance diffeence it would
make (other than time-on-the-wire ) since the max payload size has not
changed.

Scott