Subject: Re: Reopening jumbo frames in IS-IS

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 09:59:31 -0400

In message <[email-masked]>
Tony Li writes:
>  
>  
> > Jumbo frames exists only to avoid the performance drop that would
> > otherwise potentially cripple core networks using GbE or 10GbE and
> > aggregating higher MTU links.
>  
>  
> Another, and perhaps more common, reason for jumbo frames is to support
> high speed data center and storage applications.  Many modern systems
> have an 8KB primary memory page size, and can be most efficient at bulk
> transfers when they can make use of the page as a transfer unit.
>  
> Tony

Excluding NFS and a few other less used protocols there is little
reason for end system jumbo frames.  It is widely understood that NFS
8K UDP packets are a bad design.

IMHO endorsing jumbo frames in end systems goes to much against what
IEEE would like to see and does so for the benefit of protocols that
should have a new revision that eliminates the need by being more MTU
aware.  This also can improve their performance radically when used
over a WAN or VPN.

Curtis

_______________________________________________
routing-discussion mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/routing-discussion

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 15 Jul 2005 16:55:06 +0200

Curtis Villamizar wrote:
> reason for end system jumbo frames.

There is lots of practical evidence that this isn't true; it's one of
the reasons that Infiniband is very successful in environments such
as high performance redundant database clusters, for example.

Not that this is relevant to the IS-IS issue.

> It is widely understood that NFS

That is irrelevant to the IS-IS issue.

> aware.  This also can improve their performance radically when used

But the typical use of jumbo frames is in highly tuned LAN-based clusters.

Not that this is relevant to the IS-IS issue.

Also, we're not trying to rewrite the IEEE standard; we're trying to deal
with practical reality in the data center.

    Brian

---

From: Radia Perlman <[email-masked]>
Date: Fri, 15 Jul 2005 10:05:55 -0700

>aware.  This also can improve their performance radically when used
What are the technical reasons that IEEE does not like large packets?

Radia

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 13:20:00 -0400

In message <[email-masked]>
> What are the technical reasons that IEEE does not like large packets?

Is existing hardware a technical reason?  It seems to be a good excuse
to not change things in incompatible ways in the IETF so I'd think
that IEEE could use the same reason.  At the very least, disabling by
default should be a requirement (again imho).

I was only aware of the long standing ISP issue with jumbo frames but
not the data center issues that Tony and Brian brought up so I'll step
aside on further conversations about whether jumbo frames are needed
in end systems.

Curtis

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jul 2005 10:27:16 -0700

> What are the technical reasons that IEEE does not like large packets?

I can't speak for IEEE, but the reasons usually brought up include  
implementation costs in terms of buffer depths, and mutual jitter  
between competing traffic streams. If you have a session that sends a  
packet every 20 ms and depends on that being mostly maintained,  
having another session send packets that are 30 ms long and can get  
several into the queue ahead of you can be a real pain.

---

From: [email-masked] (Scott Bradner)
Date: Fri, 15 Jul 2005 13:30:19 -0400 (EDT)

also - there is no layer-2 way to say "packet too big"
so that big packets sent through paths that do not support big packets
go into a black hole and the software never knows what happened

Scott

----
>From [email-masked]  Fri Jul 15 13:27:20 2005
Cc: [email-masked], [email-masked], [email-masked],
	[email-masked], Brian E Carpenter <[email-masked]>,
	Scott Bradner <[email-masked]>
From: Fred Baker <[email-masked]>
Subject: Re: Reopening jumbo frames in IS-IS
Date: Fri, 15 Jul 2005 10:27:16 -0700
To: Radia Perlman <[email-masked]>
IIM-SIG: v:"1.1"; h:"imail.cisco.com"; d:"cisco.com"; z:"home"; m:"krs";
	t:"1121448351.73505"; x:"432200"; a:"rsa-sha1"; b:"nofws:421";
	e:"Iw=="; n:"sQYarK2E51MdcTiUqeif3F7cWdxIfoCiXhdfb9vD5ee/j0jXL15gbFxF2p"
	"XIweAblu0N6XAgK7k+wrbr7bQDJaCDqOmzqpRUBjIRQAXQ7NzadpmR3pUL6wxaRUtW+c43sl9jC"
	"50Qg1sXHpPjt8Y+Y16ioyQAQAdSunM4YhevURc=";
	s:"PRq9DVgsHUgdLZP6sRxe94EDYyx7rXYtgeFGzLriIX75ui9ef1uLnh7XQ1nYkEgZgUbgUrce"
	"j9ixfAHCKzd2oKC2SPQxG2Vnw1cm9o0mM/fNAmN9DIstyE5Px0hi+06+oVrzgFpZxxJnerl1okq"
	"hNk5G8/GFnmG/344JKdaVCXE=";
	c:"From: Fred Baker <[email-masked]>";
	c:"Subject: Re: Reopening jumbo frames in IS-IS";
	c:"Date: Fri, 15 Jul 2005 10:27:16 -0700"
IIM-VERIFY: s:"y"; v:"y"; r:"60"; h:"imail.cisco.com";
	c:"message from imail.cisco.com verified; "

---

From: Bob Hinden <[email-masked]>
Date: Fri, 15 Jul 2005 11:01:01 -0700

Radia,

>What are the technical reasons that IEEE does not like large packets?

I can't speak for IEEE, but I have always thought that one of there reasons 
that Ethernet has been so successful is that that IEEE tried very hard to 
insure backward compatibility between the different versions.  It made it 
easier to bridge between 10M/100M/1G/10G/etc. versions, new versions didn't 
break any protocols that ran over Ethernet, and it is easier to build NICs 
that supported a range of variants.

It also avoided having to build things like a FDDI/Ethernet bridge I once 
heard about that supported IP fragmentation.  I bet you remember that :-)

Bob

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:28 -0500

Thus spake "Curtis Villamizar" <[email-masked]>
> default should be a requirement (again imho).

The IETF also has a history of allowing incompatible changes in protocols 
_provided that there is a mechanism to first determine if those changes are 
supported on both ends_.

It seems fair to me to allow jumbos to be on by default iff there were a 
standard means to determine that the other host also supports them.

> aside on further conversations about whether jumbo frames are needed

I am mainly familiar with jumbos used in HPC, though I've also seen them 
used on "back side" networks used for backups or file serving. 
Communication between application and DB servers is similar and doesn't 
surprise me.  However, all cases I'm aware of are closed networks with 
processes for _humans_ to ensure that all hosts support a nonstandard MTU. 
Jumbos can't move to general use until humans are removed from the loop.

Also, there are large numbers of routers (and "L3 switches") that are 
limited in the number of pps they can handle.  Jumbos allow a higher 
bandwidth for the same pps.  OTOH, as more applications with small packets 
(e.g. VoIP) come around, we may need jumbos just to maintain current pps 
levels on a given pipe.

S

Stephen Sprunk      "Those people who think they know everything
CCIE #3723         are a great annoyance to those of us who do."
K5SSS                                             --Isaac Asimov

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:26 -0700

On Jul 15, 2005, at 10:59 AM, Stephen Sprunk wrote:

> It seems fair to me to allow jumbos to be on by default iff there  
> were a standard means to determine that the other host also  
> supports them.

The issue isn't the other host; there usually is a way to determine  
that, like a TCP MSS negotiation. The issue is the network - is there  
some buried switch that won't support them? Is there a tunnel in the  
path that reduces the MTU below what the end hosts expect?

I tend to like draft-ietf-pmtud-method-04.txt's proposal, although I  
tend to think it is a bit over-optimized. Basically, send a large  
packet every so often and see if it gets through. If it demonstrably  
does, you can decide to use them in regular conversation. Otherwise,  
do what works. Thinking in terms of TCP, we now suggest that TCP  
sessions send 2-4 segments in their first burst. well, gee. What if  
the session sent a 9000 byte packet plus three 1460 byte packets? The  
Ack would either report something 9K into the session or something 4K  
into the session. Seems like an excellent start to me... Or, have the  
delivery API include with it the size of the largest fragment folded  
into a delivered datagram. If the number falls, the receiving TCP  
could renegotiate the MSS. Maybe that's too simple.

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 15:01:21 -0500

Thus spake "Bob Hinden" <[email-masked]>
> I can't speak for IEEE, but I have always thought that one of there
> reasons that Ethernet has been so successful is that that IEEE tried
> very hard to insure backward compatibility between the different
> versions.  It made it easier to bridge between 10M/100M/1G/10G/etc.
> versions, new versions didn't break any protocols that ran over Ethernet,
> and it is easier to build NICs that supported a range of variants.
>

Wait a minute...  If Ethernet supported jumbo frames, the FDDI-Ethernet 
bridge wouldn't have needed to support fragmentation -- just set the 
Ethernet side's MTU to 4470 and all would be well.

I dealt with this many times when bridging Token Ring and Ethernet, and the 
only solution that worked in every case was to drop the Token Ring's MTU 
(network-wide, since TR implies SRB) down to 1500 -- a horrible kludge. 
Sometimes other tactics worked, including dropping oversized frames with no 
fragmentation, but some SNA apps were really touchy about that.  IP handled 
things a bit better, but still not as well as jumbos would have.

S

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 16:25:36 -0400

In message <[email-masked]>
Fred Baker writes:

> several into the queue ahead of you can be a real pain. 

btw- 30 msec at 1 Gb/s is 30 mbit or about 4mB so the delay jitter
argument falls apart for GbE.  Its 400 kB for 100baseT.  It is a
problem for 10baseT and 40 kB in the queue.  For chips that put on the
order of 10 packets into an on chip hardware queue and don't consider
the length of the packets in the queue, 4-8KB MTU might get you 30
msec with 10baseT and 5-10 packets in the hardware queue.  The IEEE
ethernet types will remind us that a GbE and 10baseT might be bridged
together.  Still the jitter buffer argument is a very weak argument.

The memory requirements is not really an issue either.  You need the
same buffer depth (a lot more than a few packets) for any MTU.

Therefore the cost of changing the implementation is the only cost.
The cost of interoperability problems is an operational cost that
needs to be considered and is the reason to procede with caution.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 17:15:59 -0400

In message <026601c58978$13564970$6501a8c0@dax>
> CCIE #3723         are a great annoyance to those of us who do."

Maybe the problem is that bridging dissimilar interfaces is itself a
horrible hack.  That's what routing is for.  Yes, some protocols don't
support routing but they are now dinosaurs, SNA included.  I thought
the days of building networks around the needs of netbios, netware 3,
and sna were over.

Curtis

---

From: Bob Hinden <[email-masked]>
Date: Fri, 15 Jul 2005 14:54:11 -0700

Stephen,

>Ethernet side's MTU to 4470 and all would be well.

That was back in the days of 10M Ethernet and it didn't support jumbo 
frames.  Seems to me the issue would still occur unless both FDDI and 
Ethernet (w/ jumbo frames) both had the same max MTU.  Someone would want 
to use the max of the bigger of the two.

Bob

p.s. I also find it ironic that in the other side of the house there are 
folks who think 53 byte packets are ideal :-)

---

From: Tony Li <[email-masked]>
Date: Fri, 15 Jul 2005 15:06:10 -0700

>> Wait a minute...  If Ethernet supported jumbo frames, the
>> FDDI-Ethernet bridge wouldn't have needed to support fragmentation --
>> just set the Ethernet side's MTU to 4470 and all would be well.
> 
> Ethernet (w/ jumbo frames) both had the same max MTU.  Someone would
> want to use the max of the bigger of the two.

While I appreciate the free trip down memory lane, I'm kind of missing
the point of the discussion here.  I think that we all now acknowledge
that Jumbo Frames exist, that like it or not, they are in use.  The
question now is what are we going to do about it...

Tony

---

From: Tony Li <[email-masked]>
Date: Sun, 24 Jul 2005 07:55:48 -0700

Curtis,

> aware.  This also can improve their performance radically when used

I would agree that using UDP for NFS in the first place was a
questionable call, exacerbated when UDP checksums were not enabled by
default on the implementations in question.

However, the real issue here is the end system efficiency, not MTU
awareness.  With a larger LAN MTU, the end system can be more efficient
by transfering an entire page at a time, rather than fragmenting it at
the UDP level or at the TCP level.  For some systems, this is a
non-trivial consideration.  When you couple this with ever increasing
link speeds (e.g., 10GigE), there seems little reason to preclude using
larger MTU sizes.  You'll note that the supercomputer folks have been
pushing us in this direction for years and we also had this discussion
relative to IPv6.

As to what the IEEE would like to see, that would seem to be a lesser
consideration than delivering the technology that our customers (i.e.,
end users) would like to see.  Those votes have already been cast and
are clearly in favor of jumbo frames.

The IETF's larger job, besides pushing this work forward, should be to
consider the role and scope of the MTU in the Internet in general.  It
is a fine point of the Internet architecture, and we have been remiss in
  not attending to this issue.  Right now, much of the Internet supports
a 4096 byte MTU, thanks primarily to some predominant POS
implementations.  However, most access links are still constrained to
1500B.  It would be very helpful, operationally, if the IETF could come
up with a recommended set of MTUs and set some expectations for end
users on the predominant MTU sizes.

While many folks may feel that this is a trivial and pointless detail,
it has a decided economic impact.  A fine example is classical IP over
ATM.  There the IETF specified a large default MTU (9120B, if memory
serves).  This caused equipment prices to rise as packet buffering
memory needed to be sized accordingly.  However, due to the lack of
transit capability for this MTU size, this capability has been largely
wasted.  Admittedly, this is indeed trivial with respect to the overall
waste on ATM technology, but it would be best if we did not repeat this
particular experiment.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 24 Jul 2005 22:08:01 -0400

> pushing us in this direction for years and we also had this discussion

That was then and now is now.  Network interfaces are now faster than
disk drives by an order of magnitude or more.  Yes, there is probably
some Cray out there front ended with an Amdol putting 4K frames ont a
HIPPI interface because at the time breaking up the disk block by
anything but a fixed small factor of two was unthinkable.

> are clearly in favor of jumbo frames.

At least we agree on the conclusion.

> users on the predominant MTU sizes.

Unfortunately in the past each new interface seems to want a bigger
MTU than the last.  I'm not sure where the 9K+ for IP/ATM came from.
I don't think its at all likely that the IETF will be going back and
changing MTUs just to make them all the same.  Why would they?  So
that anyone building a new token ring or FDDI interface can use the
new MTU?  No one builds token ring interfaces anymore.

> waste on ATM technology, but it would be best if we did not repeat this

That doesn't make sense.  In switches if you tossed out the ATM
traffic management and actually tried to make use of most of the
bandwidth you'd need delay*bandwidth sized buffers and the early
switches had 128-256 cells.  That had nothing to do with MTU.  In
routers the were two big problems.  One was packet buffers which is
unrelated to MTU.  The other was SAR buffering.  The problem existed
when carrying Internet traffic over ATM and then reassembling at the
other end and so also had nothing to do with IP/ATM MTU because there
was no IP/ATM traffic involved.  In NICs if cells didn't get reordered
(which shouldn't happen) then you'd need at most a packet per
concurrent open connection.  If you had thousands of concurrent open
connections (such as in a web server) you'd need megabytes of SAR
memory.  With an ether NIC and a jumbo MTU you don't need that sort of
memory oncard because you get an entire frame at a time, not small
pieces of many frames.  With jumbo frame ether you still need the TCP
buffers (which should always larger by at least a factor of 4) but
that's in processor main memory.  So it was really hopelessly
inadequate buffering in switches and problems with SAR on routers and
NICs and not the large MTU that was ATM's problem.

> Tony

We're straying off topic.  We agree on the conclusion that IETF should
support jumbo frames for ethernet.  If you bringing up further
examples to support extending the jumbo MTU to end systems and not
just routers and switches, that's fine with me.

Curtis

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 24 Jul 2005 20:32:52 -0700

On Sun, Jul 24, 2005 at 10:08:01PM -0400, Curtis Villamizar wrote:
> disk drives by an order of magnitude or more.

But that's not necessary the right metric to judge the cost of
packetizing. Distributed applications can easily shuffle data just between
memory without having to resolve to disk transfer. I am not sure if
there are any research community agreed upon metrics (sounds like a
question to Scott). I personally would look for packetization to be
acceptable as long as it takes up less % of CPU than the DMA of the data
takes up in % of memory bandwidth. 

Just observing from past experience - not really wanting to make a
statement one way or the other about research computers nowadays.

Just for completeness (not that i'd expect anyone to care):
For multicast to be able to leverage higher MTU than 1280 effectively and
safely would require something very close to the architectural minimum limit
to be raised (aka: ubiquitous availability of larger MTU across the Internet,
end-to-end).

So, i would certainly be happy if the agreed upon minimum MTU even in
ethernet was raised to something like let's say at minimum the 4k so
common in core networks anyhow (which also happens to be larger than many
video frame sizes, so that could avoid fragmentation on many typical video
encoders).

Cheers
	Toerless

---

From: Tony Li <[email-masked]>
Date: Mon, 25 Jul 2005 06:25:59 +0100

> just routers and switches, that's fine with me.

Shutting the hell up...

Tony