Subject: Re: [homegate] Jumbo frames

From: Joe St Sauver <[email-masked]>
Date: Tue, 16 Jun 2009 11:13:27 -0700

Marshall commented:

#Is there a definition yet for Jumbo frames ? I thought that IEEE  had  
#said no to this.

I believe you are correct, and yet 9K (or better) is broadly supported
by the vendor community and widely deployed (at least in the R&E community).

For example, the Large Scale Networking (LSN) Joint Engineering Team (JET),
(see http://www.nitrd.gov/subcommittee/lsn/jet/ ) has endorsed 9000 byte
MTUs, see http://www.nitrd.gov/subcommittee/lsn/jet/9000_mtu_statement.pdf

#Your own web site, http://darkwing.uoregon.edu/~joe/jumbo-clean-gear.html
#shows what a mess this is in the absence of a standard.

You know me, I'm just eager to be able to add home gateway devices 
somewhere on that already messy page. :-;

(And FWIW, I do think that 9000 is the defacto standard for jumbo frames 
in practice, even if, unfortunately, not in the IEEE paperwork)

But I truly don't mean to side track this broader discussion.

I'm just hoping that we DO NOT end up with a specification which says 
that home gateway devices 

   "MUST NOT [or SHOULD NOT] support MTUs greater than 1500 bytes"

Rather, I'd hope that we DO end up with a statement that reads

   "SHOULD [or at least MAY] support MTUs of at least 9000 bytes"

instead.

Regards,

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 16 Jun 2009 21:41:57 +0200

On 16 jun 2009, at 20:13, Joe St Sauver wrote:

>> Is there a definition yet for Jumbo frames ? I thought that IEEE   
>> had said no to this.

> by the vendor community and widely deployed (at least in the R&E  

Yes, all the vendors that get together at the IEEE and refuse to do  
1500+ also happily build products that support larger packet sizes on  
802.3.

The IEEE's problem is that they need to retain compatibility with  
older versions of ethernet. So if I hook up my 10GE router to a 10GE  
switch with a gigabit ethernet port and that port to a cheap  
10/100/1000 switch and that switch to a prehistoric half duplex  
10baseT NIC, the packets need to be understood by the 10 Mbps system.  
So they can't be longer than 1500 bytes.

I've been working on a draft for a few years that allows IP hosts to  
negotiate non-standard frame sizes between them:

http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-02

Don't spend too much time on this text though, I plan to completely  
overhaul it for Stockholm.

Note that although 9000 bytes is a common jumboframe size, it's far  
from universal. Some NICs and many hosts and routers support other  
nonstandard sizes. A lot of cheap switches don't even break the 2k  
barrier but even a few dozen extra bytes can be useful to allow  
tunneling without path MTU discovery issues.

The point is that we break away from the "1500-only" assumption and  
get to use MTU sizes that fit the job at hand, not that we replace one  
limitation with another.

The problem with a home gateway is that it potentially has three  
interfaces: WAN, LAN and WLAN. On the LAN size, a 9000-byte MTU would  
be nice, but on the WAN side this may not be appropriate. For  
instance, ATM uses a ~4.5k default MTU and using 9000-byte packets  
with 128 kbps upstream capacity means that a packet occupies the wire  
more than half a second. I remember some time reading that 802.11 uses  
a 4k MTU under the hood but that the wifi alliance requires a 1500- 
byte MTU for wifi certification. In any event, I've never seen a wifi  
MTU that was user settable. Also, on high error links such as wifi  
using very large packets is bad for performance.

So a home gateway that supports jumboframes would almost certainly  
have to manage different MTUs on different interfaces. It would even  
be a good idea to only send 1500 byte packets on ports that are  
running at 10 or 100 Mbps because jumboframes are very uncommon with  
sub-gigabit gear.

Now this shouldn't be a problem if the home gateway can generate  
"packet too big" ICMP/ICMPv6 messages even when it's switching. So two  
hosts connected to the HGW @ 1000 Mbps could talk 9000-byte packets  
between them but be limited to 4.5k for packets going out to the  
service provider and 1500 for packets going to hosts connected to the  
HGW over wifi. It would still be necessary to set MTU sizes on the  
different interfaces, though.

---

From: "Michael Johas Teener" <[email-masked]>
Date: Mon, 22 Jun 2009 11:25:14 -0700

Um ... As one of those IEEE 802 people, and as a contributor to Broadcom's
significant Ethernet and switch expertise, I *do* have an opinion on
larger-than-standard MTUs. They are, in general, a bad idea in any network
that includes less than gigabit links (which will likely be the case for a
very, very long time).

The problem is that a 1500 byte packet occupies the wire for a good 120us or
so at 100Mbit/s. This is just barely tolerable when there are various
classes of real-time traffic that have to be multiplexed together. We are
just now finish a set of standards within 802.1 that will provide assured
delays on the order of 2ms through 7 bridges (switches) using 100baseT. 2ms
is a magic number in the audio biz, since that is the maximum network delay
for live music (the actual acceptable "hit the drum, hear the sound" delay
is 10ms, but that includes processing delays and the speed of sound from
monitor speakers to the drummer's ears). 100baseT is the baseline for all
use models in this space because it is "free" and the bandwidth is more than
adequate.

Anyway, that 120us wiretime introduces an equivalent head-of-queue delay in
all egress ports, so we are already up to 8*120us = 960us just because the
existing packets can be 1500 bytes. There is only another 130us per egress
port that we can allow before we violate our starting requirement ... Which
goes towards shared delays between other realtime streams as well as wire
and pipeline delays.

There are some other traffic management concerns about longer MTUs,
frequently related to link round-trip delays. Waiting around for jumbo
frames introduces extra delays in any control loop, and should be avoided.

Finally, I think there is no reason to be concerned about the 1500 byte
limit for MTUs w/r/t TCP performance over Ethernet as long as a reasonably
modern NIC is used. The really cheap 1G and 100M NICs that we currently ship
have automatic IP packetizing (load up a DMA buffer with 32k bytes and the
NIC will break it into the appropriately formatted and sized TCP frames).
All of our competitors can do the same (with the possible exception of the
Intel chipsets). The resulting packet stream is within a few percent as
efficient as a jumbo frame, without interfering with the QoS of other
streams

--  
Michael D. Johas Teener ‹ [email-masked] / [email-masked]
  office +1-408-922-7542 cell +1-831-247-9666 fax +1-206-339-6331
      http://xri.net/=Michael.Johas.Teener - PGP ID 0x3179D202

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 22 Jun 2009 21:26:11 +0200

> that includes less than gigabit links (which will likely be the case  

Ok, I can go along with that for the most part.

Not sure though if it's reasonable to expect the networking stack to  
know the speed at which the interface is operating at any given time...

> 100baseT is the baseline for all

Right. At 100 Mbps we don't really need jumboframes yet so there  
shouldn't be a conflict between our respective goals.

> goes towards shared delays between other realtime streams as well as  

With ethernet we have a 32-bit CRC that has a hamming distance of 3  
upto 11454 byte packets. So going to 11455 bytes increases the  
potential for undetected errors quite significantly. So I don't see  
gigabit ethernet vendors going beyond that, which means your  
serialization delay would be some 91 us for 11k packets on gigabit  
ethernet vs 120 for 1.5k packets on fast ethernet.

> modern NIC is used.

Unfortunately, the NIC is only part of the story. Between to MacBook  
Pros (which I don't think offload anything to the NIC except the  
checksum) I can do 86 MB/s with regular packets and 116 MB/s with  
jumboframes.

Routers and switches may need more power in order to switch larger  
numbers of packets for the same amount of data.

Another issue is that there is a packet size dependency in the TCP  
algorithms which means that for a given loss probability you're going  
to run much faster with larger packets.

> The really cheap 1G and 100M NICs that we currently ship

But they don't put them back together again on the receiving side.  :-)

---

From: Marshall Eubanks <[email-masked]>
Date: Mon, 22 Jun 2009 15:41:17 -0400

> shouldn't be a conflict between our respective goals.

I disagree (about the Fast Ethernet, not the jumboframes). Anything we  
do will be for the 2011 and after time frame.
By that point, I would expect Gig-E to be routine in SOHO use, at  
least, if not in home use as well.

Anyone who is doing home video work, for example, is likely to be on  
Gig-E already. I would also dare say that
many people running Gig-E in the home don't know it.

My personal feeling is to stay away from jumbo, but I would suggest  
not doing anything to forestall fast LANs.

Regards
Marshall

> algorithms which means that for a given loss probability you're  
> going to run much faster with larger packets.
>
> _______________________________________________
> homegate mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homegate
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 22 Jun 2009 21:44:41 +0200

> I disagree (about the Fast Ethernet, not the jumboframes).

I don't understand what you're disagreeing with.

Do you want jumboframes on fast ethernet too?

Do you NOT want jumboframes on gigabit ethernet either?

Why?

---

From: Marshall Eubanks <[email-masked]>
Date: Mon, 22 Jun 2009 16:14:59 -0400

> Do you NOT want jumboframes on gigabit ethernet either?

I want this effort to support Gig-E and even 10Gig-E. I do not want  
decisions to be taken strictly assuming Fast-Ethernet. (There might  
not be any other
issues besides jumboframes, but if there are ...).

I personally would like jumboframe support IF that can be done in a  
sane fashion. I don't see how you can define
jumboframe support at the present without defining jumboframes, and I  
am not sure that this is within scope for the present effort.

There is a SDO failure here -
there are many jumboframes in use because IEEE won't define one. It  
seems to me that some group will step up and create a definition that  
will stick,
probably as a consequence of some other standard that needs  
jumboframes. I am just not sure that this should be that group.

Regards
Marshall

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 22 Jun 2009 22:48:37 +0200

On 22 jun 2009, at 22:14, Marshall Eubanks wrote:

> I want this effort to support Gig-E

Is anyone saying it shouldn't? Michael mentioned fast ethernet because  
it's basically the minimum installed base currently, 10 Mbps ethernet  
stuff is getting pretty scarce. Although gigabit ethernet is becoming  
very common, you can't really assume that it's universally available  
in the home at this point. 10/100 switches are still a lot cheaper  
than 10/100/1000 ones and cheap boxes like home gateways etc usually  
don't do gigabit, probably because their CPUs aren't up to the task  
anyway.

> and even 10Gig-E.

:-)

I got to play with a Mac with a 10GE card in it a few years ago.  
Pretty sweet. Used 16k jumboframes, BTW.

> I do not want decisions to be taken strictly assuming Fast-Ethernet.

Ethernet is ethernet, it's all compatible. I'm sure more and more  
vendors will build in gigabit the coming years and I don't think  
anyone on this list is opposed to that or has anything that would get  
in the way of that.

What I'm assuming is that if we can, we'll probably want to use  
jumboframes on gigabit ports and disable jumboframes on ports running  
at 10 or 100 Mbps.

> jumboframe support at the present without defining jumboframes

You mean a size?

That's exactly the wrong thing to do. We have 30 years of 1500 byte  
installed base. So we need to be able to determine whether we're  
talking to someone who can do 1500 or someone who can do larger than  
1500. We can of course make this a one bit decision between 1500 or  
9000 or some such, but if we're going to do all this work we really  
don't want to have to do it again when someone builds a NIC that can  
do 12000. So rather than 1 bit for jumbo yes/no we should use 32 bits  
that can hold the exact packet size that's supported even when we grow  
beyond 64k.

(Note that I've personally worked with ethernet gear that supported  
9000, 9216, 16384 and 64000. Looking at vendor documentation I came up  
with at least a dozen other sizes. Trying to standardize a jumboframe  
size is going to be very hard.)

> and I am not sure that this is within scope for the present effort.

Right. I suggest that those interested in this go to the intarea list,  
I'm going to post an update of my draft there later this week.

---

From: "Richard Bennett" <[email-masked]>
Date: Mon, 22 Jun 2009 14:09:12 -0700

I'd like to remind people that frame size is a MAC protocol issue, and not
something that the IETF needs to have a specific opinion about, beyond "be
conservative in what you send and liberal in what you accept." If 802
permits or mandates support for jumbo frames, it will happen, and if they
don't it probably won't. 

Richard Bennett
-----Original Message-----
From: Iljitsch van Beijnum [mailto:[email-masked]] 
Sent: Monday, June 22, 2009 1:49 PM
To: Marshall Eubanks
Cc: Michael Johas Teener; [email-masked]
Subject: Re: [homegate] Jumbo frames

:-)

You mean a size?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 22 Jun 2009 23:20:23 +0200

On 22 jun 2009, at 23:09, Richard Bennett wrote:

> conservative in what you send and liberal in what you accept."

The problem is that IP assumes that every subnet has a fixed MTU.  
Since obviously the 1500-byte stuff isn't going anywhere, the addition  
of jumboframe-capable hardware creates a problem, because now we have  
systems with two (or more) different MTUs on a subnet, which doesn't  
work in the general case although the TCP MSS option hides a multitude  
of sins. (Not the sin of having a 1500-byte switch sit between all the  
jumbo-capable systems, though.)

((I keep telling people that RFC 894 (IP over ethernet) is an april  
fools day RFC, published on april 1 1984, and we shouldn't take it  
seriously but nobody listens.))

> If 802

Also not exactly true. Like I wrote before, the same vendors that make  
up IEEE 802.3 also build products that ignore the 1500-byte MTU limit  
in 802.3.

---

From: "Richard Bennett" <[email-masked]>
Date: Mon, 22 Jun 2009 14:32:02 -0700

Aren't jumbo frames generally used within the LAN? When they are, there's no
need for fragmentation. When they pass through the NAT toward the Internet,
it's a different story, of course, but we don't have so many Gigagbit WAN
connections today, outside of high-end consumers in Japan. 

Richard Bennett

---

From: Lars Eggert <[email-masked]>
Date: Tue, 23 Jun 2009 12:25:35 +0300

> efficient as a jumbo frame, without interfering with the QoS of other

As an aside, existing hardware mechanisms for TCP segmentation  
offloading may stop working if TCP changes in a major way. (The  
Multipath TCP BOF in Stockholm - if successful - could eventually lead  
to such TCP changes.)

Lars

---

From: Iñigo Artundo <[email-masked]>
Date: Tue, 23 Jun 2009 11:26:01 +0200

Related to Gb WANs and the use of 10/100/1000 Ethernet ports in 
computers and network equipment, please remember as someone already 
said, that all the thinking being done here is for a midterm future, not 
for the current market, which means that technologies that now are not 
mainstream, it may very well be in a 5-8 years timeframe. Think ahead.

And I agree that the discussion about jumboframes should be taken out 
from this homegate list to other more relevant wg.

Iñigo

-- 

Iñigo Artundo, PhD
iTEAM - Grupo de Comunicaciones Opticas
Universidad Politecnica de Valencia (UPV)
http://www.iteam.upv.es

---

From: David R Oran <[email-masked]>
Date: Fri, 26 Jun 2009 15:08:27 -0400

On Jun 22, 2009, at 5:20 PM, Iljitsch van Beijnum wrote:

> Since obviously the 1500-byte stuff isn't going anywhere, the  
> addition of jumboframe-capable hardware creates a problem, because  
> now we have systems with two (or more) different MTUs on a subnet,  
> which doesn't work in the general case although the TCP MSS option  
> hides a multitude of sins. (Not the sin of having a 1500-byte switch  
> sit between all the jumbo-capable systems, though.)
>
Which is one among many reasons we should be designing for multiple  
subnets on a single home network, instead of assuming the network is a  
single sea of L2 connectivity.

> Also not exactly true. Like I wrote before, the same vendors that  
> make up IEEE 802.3 also build products that ignore the 1500-byte MTU  
> limit in 802.3.

---

From: "Richard Bennett" <[email-masked]>
Date: Fri, 26 Jun 2009 12:13:30 -0700

Good point.

Richard Bennett