Subject: Re: [homegate] Jumbo frames

From: Joe St Sauver <[email-masked]>
Date: Tue, 16 Jun 2009 10:36:05 -0700

Kirk commented:

#Point taken on your chicken and egg problem. But if you are going to do 
#jumbo frames, why stop at 9K? For a vendor building a gateway with off 
#the shelf components, 12K capability is reasonably common, though it 
#may be locked out in the software build that operates the MAC.

Indeed, my colleague Matt Mathis at PSC makes a persuasive case for what 
some call "super jumbo frames" for very high speed links, (e.g., see
http://staff.psc.edu/mathis/MTU/#pushing ) however my MTU ambitions are
a bit more incremental. No arguments with Matt's fine work, I'd just
get where he wants to go more gradually. (And 9K is a value that's 
currently widely supported on at least much of the wide area R&E 
infrastructure, so that's a target that I'd be most comfortable with 
for the time being)

Regards,

Joe

---

From: Joe St Sauver <[email-masked]>
Date: Mon, 22 Jun 2009 12:09:50 -0700

Hi Mike,

You mentioned:

#100baseT is the baseline for all use models in this space because it is 
#"free" and the bandwidth is more than adequate.

By "free" I assume you mean "is bundled with shipping products by default."

If I've interpreted that correctly, note that even systems as modest as 
Apple's Mac Mini (a ~$600-class box) now support gigabit ethernet, and 
if a PC ships without a gig NIC, adding one is a <$30 correction (if not
less, checking www.pricewatch.com/browse/networking/gigabit_adapter/ )

I thus don't think it is safe to assume that customers will only have 
10/100-capable hosts any more, and I think it would be a mistake to fail
to acknowledge the market reality of 1000/100/10baseT NICs in a growing 
number of consumer-class systems.

And when it comes to the adequacy of 100Mbps worth of bandwidth, I think
that applications like the network backup of increasingly disk-heavy 
systems to cheap TB-class external disks will drive demand for 
better-than-fast-ethernet speeds (assuming users don't abandon ethernet 
entirely for those sort of external disk systems).

#Finally, I think there is no reason to be concerned about the 1500 byte
#limit for MTUs w/r/t TCP performance over Ethernet as long as a reasonably
#modern NIC is used. The really cheap 1G and 100M NICs that we currently 
#ship have automatic IP packetizing (load up a DMA buffer with 32k bytes 
#and the NIC will break it into the appropriately formatted and sized TCP 
#frames).

The problem is that we just don't see people getting the throughput they
"should," even when running on high bandwidth and zero loss networks. See,
for example http://netflow.internet2.edu/weekly/20090615/#select

-- the 50th %-ile for bulk file transfers is only 3.6Mbps
-- the 99th %-ile for bulk file transfers is only 68.2Mbps

We can discuss why that may be true, and believe me, many folks have, but 
the reality is that if your performance focus is on single stream high 
throughput TCP flows (rather than low jitter), we really need all the help 
we can get, and that includes jumbo frame support for gig links.

Failing that, I think we'll increasingly see a lot of fairly ugly things,
like large numbers of parallel TCP flows in an effort to increase their
throughput to what it "should be."

Regards,

Joe St Sauver ([email-masked])
http://www.uoregon.edu/~joe/
Disclaimer: all opinions strictly my own