Subject: [Int-area] Larger MTUs

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 23 Jul 2007 18:46:56 -0500

[see conclusion at the end if you skip some stuff]

After the feedback this afternoon I read RFC 4821, which is about  
discovering the path MTU by probing rather than depending on ICMP  
messages.

This approach has the advantage that it can work just as well over a  
layer 2 path with an unknown MTU as over a layer 3 path where  
traditional path MTU discovery doesn't work because ICMP "too big"  
messages aren't sent or received.

This means that it would be possible for hosts implementing this  
mechanism to simply set the maximum MTU search size to their local  
non-standard MTU and everything will work.

In my approach, I wanted to avoid sending oversized packets that  
don't make it through the layer 2 network as much as possible to  
avoid the problems this may cause. In the degenerative case, a  
10/100/1000 Mbit host sends a bunch of oversized packets in a short  
time because a number of TCP sessions are searching for the MTU and  
this happens on an old 10 Mbps network where this leads to some kind  
of exception state with further negative impact. (Hubs/switches that  
disconnect ports with too many errors, that kind of thing.)

Another difference is that in my draft, routers can announce a TCP  
MSS value but the MTU discovery overrides this information on the  
local subnet, which makes it easy to stick to 1500 byte (or smaller)  
packets across the net but use larger packets locally. Routers can  
also announce a maximum allowed MTU so it's easy to make sure that  
hosts don't send packets larger than a certain size administratively  
if desired. Obviously it's also possible to simply announce the  
largest possible MSS so large packets can be used across the internet  
(I think this may have been unclear this afternoon).

Last but not least, the RFC 4821 mechanism must be implemented per  
transport protocol, while the mechanism in my draft works at the IP  
layer so it doesn't introduce new logic in transports.

The idea of having switches send "too big" messages isn't very  
attractive for three reasons:

1. This isn't very robust at the IP layer with traditional PMTUD
2. A node could send packets that are so large that the switch can't  
receive them so it's not possible to send an ICMP message back either
3. Nodes would need to know whether the switches support this before  
they can send larger packets, which is more or less the same reason  
why most subnets aren't configured for jumboframes today

First reaction to issues with neighbor discovery over tunnels:  
tunnels have problems with MTUs in general and PMTUD in particular.  
There are many opportunities for problems, but I think in practice  
the mechanism I proposed wouldn't lead to much additional trouble  
because the MTU for the tunnel interface is generally low enough that  
the mechanism isn't used anyway, and probing + neighbor  
unreachability detection (for IPv6) will make sure it's possible to  
avoid problems and/or recover from them.

Multiple paths with multiple MTUs: you can't have loops in your  
ethernet topology, so the only way to do this is with 802.3ad link  
aggregation. As far as I can tell, at least some Cisco equipment  
makes sure bundled links all use the same MTU. Not sure if 802.3ad  
says anything about this. Also, switches don't send packets belonging  
to the same session over different links in a bundle to avoid packet  
reordering. So if an MTU failure occurs, it will be consistent.  
Because the actual traffic and the ICMP probe message aren't  
necessarily the same "session" it's possible that MTU probes and data  
traffic see different MTUs. Neighbor unreachability detection will  
have to detect the problem so the neighbor MTU is reset.

About 9000 bytes is not enough: all MTU fields are 32 bits in the  
draft.  :-)

Someone made me aware of this:

http://grouper.ieee.org/groups/802/3/frame_study/index.html

This effort doesn't increase the payload size of ethernet packets,  
though.

My conclusion:

Wide scale implementation of RFC 4821 makes the MTU probing packets  
unnecessary, but this and the other options and messages can still be  
useful for severa reasons:

- skip probing steps because neighbor MTU is known immediately
- allow administrators to limit MTU sizes
- use different MTUs for different link speeds for jitter/delay  
control and interaction with nodes/switches with limited capabilities
- allow unmodified transports to use larger packets

I'm thinking that it's probably possible and desireable to make all  
messages and options optional, with the exception of something that  
allows administrators to limit the MTU subnet-wide with one setting.  
But maybe cases can be made for completely removing some messages or  
options because it's unlikely they'll be implemented or provide many  
benefits if implemented. However, please note that although the  
number of new options and messages may seem a bit high, the way in  
which they work is actually very straightforward with very simple  
decision making logic and only a single new timer introduced.

The goal is to allow the use of larger packets between supporting  
nodes on a subnet. Whatever gets that done without breaking any old  
stuff that's reasonably still in use is fine by me.

_______________________________________________
Int-area mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 26 Jul 2007 15:44:08 -0500

> discovering the path MTU by probing rather than depending on ICMP  

[...]

> non-standard MTU and everything will work.

However, for this to work reliably, it's necessary that a host only  
sends large packets if it actually supports RFC 4821. What could  
happen if that host A1 on subnet A where RFC 4821 in combination with  
a variable MTU is used by systems supporting jumboframes communicates  
with host B2 on subnet B where RFC 4821 isn't used, but the  
administrator has set a larger than 1500 byte MTU on all nodes on the  
subnet (the way jumboframes are done traditionally). If host B2 then  
sends a packet that is too large for the switch that host A1 connects  
to, the router on subnet A keeps sending packets that are too large  
into the layer 2 network and host B2 never recovers from the lack of  
reachability.

---

From: John Heffner <[email-masked]>
Date: Fri, 27 Jul 2007 14:05:19 -0400

Iljitsch van Beijnum wrote:
> sends large packets if it actually supports RFC 4821. What could happen 
> if that host A1 on subnet A where RFC 4821 in combination with a 
> variable MTU is used by systems supporting jumboframes communicates with 
> host B2 on subnet B where RFC 4821 isn't used, but the administrator has 
> set a larger than 1500 byte MTU on all nodes on the subnet (the way 
> jumboframes are done traditionally). If host B2 then sends a packet that 
> is too large for the switch that host A1 connects to, the router on 
> subnet A keeps sending packets that are too large into the layer 2 
> network and host B2 never recovers from the lack of reachability.

For hosts communicating on a mixed-MTU switched network, it should be
safe for hosts implementing 4821 to increase their interface MTU beyond
that network's configured MTU.  (Assuming nothing bad happens to the
network gear by sending oversized frames -- as you mentioned, this has
potential to cause some problems on very old ethernet gear.)  Hosts
implementing 4821 that have a jumbo-clean path between them will
automatically take advantage of this, while still communicating with
non-4821 hosts using the standard MTU.

The difficult problem is the router's behavior.  If a subnet is running
a mixed MTU, it's not clear what that router's interface MTU to that
subnet should be.  It would be nice for it to forward the largest
packets that it can support; however, to be compliant with the spec it
must generate ICMP PTB messages for any packets that are too large to be
delivered.  (Note, I assume here that MTU==MRU in all cases.)  Since the
router does not know the MTU of each host on the subnet, the only
compliant thing for it to do is to use the smallest possible (e.g.,
standard) MTU for that network.  I believe what you are describing above
is the failure mode if the router interface is configured with an MTU
higher than the standard MTU.  Hosts sending from outside the router
using 4821 will work fine, but any host outside the router with a larger
MTU (that may be valid on its own network), and still relying on
classical 1122/1982 pmtud, will break.

In my mind, the question of what the router should do is the most
important in any discussion of mixed-MTU subnets.  In order to generate
the correct ICMP PTB messages, a route with a large-MTU interface on a
mixed-MTU subnet would need to know the individual MTUs of each host on
that subnet.  Neighbor discovery seems like the place you'd want to do
this.  However, I think this is a tricky problem, and I'm not sure your
proposal solves it in full.  (Think this over and tell me if I'm wrong
-- I'm not sure I understand your "The MTU detection message"s fully.)

There was some talk a while back about doing jumbo ARP or jumbo DHCP to
allow routers to discover the MTU of their neighbors.  Does anyone have
pointers on this?  At least one Cisco router has error counters for
jumbo ARP/DHCP (google jumbo arp).  I've been kicking around some ideas
in this space but none of them are fully baked, possibly not even
half-baked. ;)

One thing to note is that if you imagine a world where the vast majority
of hosts with large MTUs use 4821, it may be operationally acceptable to
deliberately break classical pmtud by not generating ICMP PTB messages.
  This may not be totally far-fetched, since classical pmtud is so
fragile in practice that large MTUs are currently quite rare.  This is
an imperfect solution, but with some luck it may be just good enough.

   -John

---

From: Matt Mathis <[email-masked]>
Date: Fri, 27 Jul 2007 15:10:52 -0400 (EDT)

> into the layer 2 network and host B2 never recovers from the lack of

This is the same failure that John mentions.  In short: a jumbo enabled
non-RFC4821 host trying to connect via a jumbo path to a standard MTU host on
a mixed MTU subnet.  Fixing this case requires the router at the mixed MTU
subnet to know the MTU of the host and send the proper PTB message back to the
non-RFC 4821 host.

Years ago there was a lot of work in this space, including the use of jumbo
arp to deduce the MTU of the router's peer devices.  I think this failed
because it was deemed to be too hard to deploy sufficiently to enable the
global deployment of jumbograms.  However, in conjunction with 4821, it is
only needed to solve the above corner case.  RFC 4821, 1191 and 1981 should be
sufficient for all other reasonable deployment scenarios.

And as John Heffner noted, the community can decide to evade this problem.
All we have to do is declare it to be a bug to be trying to do jumbo from a
non-4821 host.  Then RFC 4821, 1191 and 1981 are definitely sufficient.

Jumbo ARP would actually be pretty efficient, since the first packet of a
connection is (nearly) always small.  The router does a regular ARP for the
SYN (or whatever), and then has lots of time: a full end-to-end RTT before the
first data packet arrives after the SYN-ACK.  During this time it can test the
largest MTU and possibly even do RFC 4821 style MTU searching before the first
data packet arrives.

I could not find any of the old documents..... pointers anyone?

(BTW: 4821 unilaterally solves a lot of other problems with tunnels and the
like, so I expect it to be deployed even without a solution to the router part
of the global MTU problem).

Thanks,
--MM--

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 27 Jul 2007 12:26:45 -0700

Matt,

> non-4821 host.  Then RFC 4821, 1191 and 1981 are definitely
sufficient.

IF the community can declare it a bug for non-4821 hosts to send
jumbos (presumably anything larger than 1500 bytes) then this makes
matters easy for tunnels; the tunnel could simply look at the LinkMTU
of the underlying physical interface and admit the packet if it is
no larger than LinkMTU. If the community *cannot* declare it to be
a bug, then tunnels will have to perform some special handling to
support jumbos. Right?

Thanks - Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 27 Jul 2007 14:31:55 -0500

On 27-jul-2007, at 13:05, John Heffner wrote:

> (Assuming nothing bad happens to the
> potential to cause some problems on very old ethernet gear.)

Right, that's why I feel it's very important to have a mechanism that  
makes it possible to tell all hosts to reduce their MTU with one  
setting in one place, rather than have to do this on all hosts  
individually.

> (Note, I assume here that MTU==MRU in all cases.)  Since the
> standard) MTU for that network.

Why is this true for the router but not for other hosts? The issues  
are the same, assuming that all hosts (including the one reachable  
through the router) implement RFC 4821.

Obviously, that's an "interesting" assumption to make, especially as  
there's no obvious way to validate its veracity.

> Hosts sending from outside the router

Right.

> -- I'm not sure I understand your "The MTU detection message"s fully.)

This is extremely simple: put the MTU in neighbor discovery packets  
so that nodes learn their neighbor's MTU along with the MAC address.

> an imperfect solution, but with some luck it may be just good enough.

Why would you want to do that? Even with full RFC 4821 support  
regular PMTUD is still extremely useful because it gives you the  
exact value of the path MTU immediately after sending an oversized  
packet.

Iljitsch

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 27 Jul 2007 14:58:13 -0500

> non-4821 host.  Then RFC 4821, 1191 and 1981 are definitely  

I don't think that's a workable approach. For ethernet, using packets  
larger than 1500 bytes is illegal, strictly speaking, but there are  
no limits on other link technologies. So if I get some FDDI equipment  
in a garage sale or connect to the internet over ATM, SONET or even  
PPP, I will / can have an MTU larger than 1500 bytes without breaking  
any specification. Retroactively making this illegal would be a  
curious move to say the least.

Also, this is something that will only lead to deploment problems,  
because there is no way to know if someone who is sending you large  
packets is implementing RFC 4821, using "illegal" jumboframes without  
RFC 4821 or is simply running old 1500+ gear. Right now, you can be  
pretty sure that it's not the first case, because RFC 4821 isn't in  
wide use, unless I'm mistaken. So at what point do we get to assume  
that it is?

A better approach is to exchange MTU information at the neighbor  
discovery / ARP stage. That way, you're not breaking anything that  
your correspondent may be depending on.

I'm not sure how jumbo ARPs work. My initial thoughts for backporting  
the stuff in my draft to IPv4 was to use neighbor discovery for IPv4  
addresses, but later it occurred to me that it's easier to do regular  
ARP and let that trigger the sending of a new message type to  
announce the supported MTU.

---

From: John Heffner <[email-masked]>
Date: Fri, 27 Jul 2007 16:08:07 -0400

Iljitsch van Beijnum wrote:
> makes it possible to tell all hosts to reduce their MTU with one setting 
> in one place, rather than have to do this on all hosts individually.

It's a good goal.  I'd like to address this in a separate email once 
I've given your draft some more careful thought.  I think Matt Mathis 
has some input on this as well.

> Why is this true for the router but not for other hosts? The issues are 
> the same, assuming that all hosts (including the one reachable through 
> the router) implement RFC 4821.

The router is different because it must generate ICMP PTB messages with 
the correct next hop MTU.  There's no way for the router to do 4821 like 
hosts can.

> there's no obvious way to validate its veracity.

Sorry for the confusion.  This assumption is only for simplicity of 
language.  It doesn't affect the validity of what I was saying.

> This is extremely simple: put the MTU in neighbor discovery packets so 
> that nodes learn their neighbor's MTU along with the MAC address.

I wouldn't go so far as to say it's extremely simple.  Making the 
neighbor discovery work correctly in all cases, in the presence of 
switches that don't participate, and without too much overhead, seems 
tricky to me.  Again, I'll think a bit harder on this and address it in 
another email.

> Why would you want to do that? Even with full RFC 4821 support regular 
> PMTUD is still extremely useful because it gives you the exact value of 
> the path MTU immediately after sending an oversized packet.

It's not what I would want in an ideal world.  However, if other 
solutions don't materialize, this is one possible avenue of evolution. 
Attacking the problem in multiple complimentary ways is always a good 
thing. :)

   -John

---

From: John Heffner <[email-masked]>
Date: Fri, 27 Jul 2007 16:24:59 -0400

Iljitsch van Beijnum wrote:
> larger than 1500 bytes is illegal, strictly speaking, but there are no 
> limits on other link technologies. So if I get some FDDI equipment in a 
> garage sale or connect to the internet over ATM, SONET or even PPP, I 
> will / can have an MTU larger than 1500 bytes without breaking any 
> specification. Retroactively making this illegal would be a curious move 
> to say the least.

I think this need to be made a bit more clear.  In general right now, 
only a single MTU is supported on a single broadcast domain.  It is 
usually enforced by standard (Ethernet) or DHCP / ipv6 mtu 
announcements.  The point is that with a host capable of sending larger 
packets, using a protocol that implements RFC 4821, can exceed this MTU 
without breaking MTU discovery.  (Other protocols on that same host not 
implementing RFC 4821 should not exceed the "safe" MTU.)

This does not impose any sort of global 1500-byte limit on hosts doing 
classical PMTUD.  It is only saying that it might be safe for 
4821-capable protocols to ignore certain existing MTU limits.

   -John

---

From: Matt Mathis <[email-masked]>
Date: Fri, 27 Jul 2007 17:07:49 -0400 (EDT)

> curious move to say the least.

Most highend Ethernet gear supports jumbo (or "superjumbo" as the IEEE calls
it).  See http://darkwing.uoregon.edu/~joe/jumbo-clean-gear.html for a list.
The IEEE refuses to standardize it in part because it breaks the zero
configuration "plug and play" usage model for ethernet.

> your correspondent may be depending on.

The problem with an explicit protocol solution is that you have to get both
parties to deploy it before anybody gets any gain.  Why should the router
vendors implement it if no hosts support it and vice versa?  This is why it is
important that RFC 4821 unilaterally solves some other problems.

Although jumbo ARP is a hack^H^H^H^H workaround, it has the advantage that any
vendor can unilaterally implement and market a product that solves the router
part of mixed MTU subnet problem (or so I think).  Furthermore they can do so
even in advance of a standard!  (In the style of 4821 you send ARP padded out
to the MTU that you want to test.)

Which is why I think it is so awesome that the first hit I see when googling
"jumbo arp" is online documentation for one of the more popular switch/routers
out in the field today.

You could consider adding a padding option to neighbor discovery, in the style
of RFC4820.

Thanks,
--MM--

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 28 Jul 2007 20:30:27 -0500

On 27-jul-2007, at 15:08, John Heffner wrote:

> The router is different because it must generate ICMP PTB messages  
> with the correct next hop MTU.  There's no way for the router to do  
> 4821 like hosts can.

Yes, it seems you are correct. I was about to say that you can even  
easily have hosts with different MTUs on the same subnet without  
trouble if there are no routers on the subnet, but that's not  
entirely true: I think that A/V streaming happens with < 1500 packets  
anyway, but EDNS0 has the potential for packets > 1500 bytes without  
the ability to recover from MTU mismatches, and traditional NFS with  
8k UDP packets certainly has that problem. But we tend to have  
routers on our subnets anyway.

However, if you consider the case of subnets used by multiple routers  
with different MTU sizes, and the fact that RFC 4821 isn't  
universally deployed, then a way to learn the maximum MTU between any  
two routers would be very useful. This way, the routers can generate  
the appropriate packet too big messages.

> solutions don't materialize, this is one possible avenue of  
> evolution. Attacking the problem in multiple complimentary ways is  
> always a good thing. :)

I still don't see how deliberately breaking classical PMTUD is  
helpful...

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 28 Jul 2007 20:40:12 -0500

> parties to deploy it before anybody gets any gain.

That's not completely accurate. If an OS implement such a mechanism,  
then it becomes possible for hosts or routers running this OS to  
exchange larger packets even on subnets where jumboframes aren't  
configured explicitly.

So it could even help vendors sell more gear if they're the first to  
implement this.  :-)

> You could consider adding a padding option to neighbor discovery,  

[Can't look up the RFC right now, on my way back home from Chicago.]

Would you send such an ND packet as the first one? That would be  
suboptimal, because if the other side doesn't support your packet  
size the packet is lost and you need to retry after a timeout. If  
you're doing such an ND after a regular one, then there is no real  
need for it to be an ND. An ICMP echo request would accomplish the  
same. (Except in the case of Windows, of course, which filters them.)

Iljitsch

---

From: Fred Baker <[email-masked]>
Date: Tue, 31 Jul 2007 16:11:27 +0800

On Jul 28, 2007, at 2:05 AM, John Heffner wrote:

> The difficult problem is the router's behavior.  If a subnet is  
> running a mixed MTU, it's not clear what that router's interface  
> MTU to that subnet should be.  It would be nice for it to forward  
> the largest packets that it can support; however, to be compliant  
> with the spec it must generate ICMP PTB messages for any packets  
> that are too large to be delivered.

here's an interesting gotcha. Macs run a 1000/100/10 Ethernet  
interface and run a 1500 byte MTU/MRU regardless. The router can  
correctly observe that it is 1 GBPS and send the jumbogram, the  
switch can support the jumbogram, and have the Mac not accept the  
packet.

Hence, there are variations of this that are beyond the router's  
knowledge.

I would suggest that the router respond with the ICMP when the packet  
is too big for the configured interface MTU and not try to predict  
the end system's or the switch's behavior. 4821 encourages the sender  
to use the largest segment size that it can verify delivery of. Leave  
that to the end system.

---

From: John Heffner <[email-masked]>
Date: Tue, 31 Jul 2007 08:47:56 -0400

Fred Baker wrote:
>> running a mixed MTU, it's not clear what that router's interface MTU 
>> to that subnet should be.  It would be nice for it to forward the 
>> largest packets that it can support; however, to be compliant with the 
>> spec it must generate ICMP PTB messages for any packets that are too 
>> large to be delivered.
> 
> here's an interesting gotcha. Macs run a 1000/100/10 Ethernet interface 
> and run a 1500 byte MTU/MRU regardless. The router can correctly observe 
> that it is 1 GBPS and send the jumbogram, the switch can support the 
> jumbogram, and have the Mac not accept the packet.
> 
> Hence, there are variations of this that are beyond the router's knowledge.

Exactly.

> I would suggest that the router respond with the ICMP when the packet is 
> too big for the configured interface MTU and not try to predict the end 
> system's or the switch's behavior. 4821 encourages the sender to use the 
> largest segment size that it can verify delivery of. Leave that to the 
> end system.

The question then is how to configure the router's MTU.  The only safe 
way that won't break 1122/1981 for an Ethernet interface would be to set 
it to 1500 bytes.  But then it won't forward larger packets so 
jumbo-capable devices don't get the benefit of jumbo frames beyond their 
local subnet.  Setting the router MTU larger will allow 4821 probing to 
work for jumbo-capable hosts, but then you risk breaking connectivity to 
hosts on your subnet with smaller MTUs, from outside hosts/protocols 
with larger MTUs that rely on classical PMTUD (and aren't protected by 
the TCP MSS option).

   -John

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 31 Jul 2007 16:08:45 +0200

On 31-jul-2007, at 10:11, Fred Baker wrote:

> interface and run a 1500 byte MTU/MRU regardless.

(On many Macs, you can manually set an MTU size of upto 9000 bytes  
regardless of the link speed.)

> The router can correctly observe that it is 1 GBPS and send the  
> jumbogram, the switch can support the jumbogram, and have the Mac  
> not accept the packet.

> Hence, there are variations of this that are beyond the router's  

Right. Hence the new mechanisms that I propose in my draft:

http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-00.txt

However, it would be good if benefits could be had without  
introducing such a relatively large number of new options and messages.

My current thinking is that if hosts implement RFC 4821, they can use  
a larger-than-standard MTU for the transport protocols that use RFC  
4821, but they'd still need to maintain the standard MTU for packets  
that aren't generated by those transport protocols OR they have to  
implement the neighbor MTU negotiation laid out in my draft (or  
something similar).

So for full end-to-end jumbo use across the internet the additional  
mechanisms need to be implemented on all intermediate ethernets (or  
jumboframes must be enabled administratively) but with only RFC 4821  
you can still have out-of-the-box jumboframe capability for TCP and  
maybe some other transports on the local subnet.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 31 Jul 2007 23:01:31 +0200

On 31-jul-2007, at 16:08, Iljitsch van Beijnum wrote:

> jumboframes must be enabled administratively) but with only RFC  
> 4821 you can still have out-of-the-box jumboframe capability for  
> TCP and maybe some other transports on the local subnet.

Rethinking the type of MTU detection packets: a new mechanism that  
must be supported by both sides, or something that can be implemented  
by just one side?

For the latter, the question would be: what type of packet is most  
appropriate? I see two options:

1. ICMP echo request/reply
2. Oversized ARP and neighbor discovery packets

The advantage of ICMP is that using echo request/reply doesn't  
introduce new uses for an existing message, but it has several  
downsides:

- replying to large packets with large packets could be abused for  
denial of service
- replying with large packets is unnecessary but inherent to ICMP echo
- many software firewalls, some enabled by default, block these messages

I haven't been able to find a description of "jumbo ARP" but I assume  
this is simply an ARP request padded to the desired length and sent  
as unicast. (Multicasting large packets if it can be avoided is a big  
no-no on wifi because multicasts are typically sent at the lowest  
supported rate.)

If an answer comes back, the target host is able to receive packets  
of the length used when sending. ARP packets are already often padded  
to 46 bytes to fill a minimum ethernet frame, so I assume this will  
work in many cases. IPv6 ND would work much the same way except that  
a padding option would have to be added to increase the packet size.  
Some systems may not be prepared to handle ND packets of arbitrary  
sizes.

In each case, the test packet would have to be sent shortly after  
discovering a neighbor's MAC address using regularly-sized ARP or ND.  
Since it is unknown whether the neighbor supports jumboframes at all,  
let alone what size, it's probably prudent to test in two stages:  
first send a test packet of the minimum usable jumbo size (I'm  
thinking 1520 bytes, that would be 1500 + 20 byte IP-in-IP header).  
This wastes less time on the wire and is likely to upset old gear to  
a lesser degree than a maximum size jumboframe.

If an answer comes back, note the increased MTU and the next try is  
with the largest packet size supported locally. If an answer comes  
back for that one, again note the larger MTU. If no answer comes  
back, do nothing. If the other side supports a smaller maximum size,  
then it will probe with that size at some point, and that will be our  
cue to try one last time with that size. This way, it's possible to  
find out the largest supported MTU between two hosts that both  
implement this mechanism without trying a large number of sizes.

However, if only one end supports the mechanism, additional probes  
are necessary to discover the maximum supported MTU, or a  
conservative maximum must be chosen and the real maximum between two  
neighbors is never discovered.

With IPv6 neighbor discovery, we can add an MTU option so that if  
both neighbors support the new mechanism, they'll tell the other  
their MTU without the need to send multiple large packets. (Also very  
useful to signal lack of jumboframe support.) If nodes run both IPv4  
and IPv6, this information can be reused for IPv4, and it only has to  
be validated by sending a jumbo ARP. In this case, we can probably  
get away with trying just 9000 bytes or the local maximum, whichever  
is smaller, if no neighbor specific information is known, as few  
hosts (but a larger number of switches) implement jumboframes < 9000  
bytes.

Unless someone has other insights to offer, I think I'll update my  
draft according to the above.

---

From: Fred Baker <[email-masked]>
Date: Wed, 1 Aug 2007 05:40:32 +0800

On Aug 1, 2007, at 5:01 AM, Iljitsch van Beijnum wrote:

> must be supported by both sides, or something that can be  
> implemented by just one side?

see previous note. I don't think the mechanism has to even be known  
to both sides necessarily, just observable by the side that needs the  
answer.

One way to implement RFC 4821 would be to literally send each of the  
enumerated segment sizes in the first burst (send n segments, of  
sizes 9000, 8192, 4096, 2048, 1500, 1460, 1400, and 1280, and see how  
much data is acknowledged in the first acknowledgment). One could  
imagine the same kind of thing happening here. If I send you an ARP  
request in a 9K byte packet, will you care? You will no doubt respond  
with the same packet you would have normally. But in doing so, you  
will have demonstrated that you received the 9K-gram.

---

From: Fred Baker <[email-masked]>
Date: Wed, 1 Aug 2007 05:33:39 +0800

personally, I would detect the speed (Macs do that) and set to 1500  
for 10/100 and 9K for 1000 (and 10000). It might be of value to try  
sending a jumbogram to a neighbor, perhaps in the context of an ARP/ 
ND request (send a 9K packet containing an ARP request or ND  
solicitation and see if you get a response) before doing so.

On Jul 31, 2007, at 8:47 PM, John Heffner wrote:

>> I would suggest that the router respond with the ICMP when the  
>> packet is too big for the configured interface MTU and not try to  
>> predict the end system's or the switch's behavior. 4821 encourages  
>> the sender to use the largest segment size that it can verify  
>> delivery of. Leave that to the end system.
>
> The question then is how to configure the router's MTU.  The only  
> safe way that won't break 1122/1981 for an Ethernet interface would  
> be to set it to 1500 bytes.  But then it won't forward larger  
> packets so jumbo-capable devices don't get the benefit of jumbo  
> frames beyond their local subnet.  Setting the router MTU larger  
> will allow 4821 probing to work for jumbo-capable hosts, but then  
> you risk breaking connectivity to hosts on your subnet with smaller  
> MTUs, from outside hosts/protocols with larger MTUs that rely on  
> classical PMTUD (and aren't protected by the TCP MSS option).
>
>   -John

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 31 Jul 2007 14:46:18 -0700

I may have missed this, but how are we defining "jumbogram"?
Is it anything larger than 1500bytes? Will it still be the
same value going forward into the future?

Thanks - Fred

> -----Original Message-----
> From: Fred Baker [mailto:[email-masked]] 
> Sent: Tuesday, July 31, 2007 2:34 PM
> To: John Heffner
> Cc: Internet Area
> Subject: Re: [Int-area] Larger MTUs
>

---

From: Fred Baker <[email-masked]>
Date: Wed, 1 Aug 2007 05:54:34 +0800

for the moment, trying to distinguish between 10/100's 1500 byte  
datagram and GigEthernet's 9K datagram, 9K is a jumbogram. The  
numbers may not transfer, but the concept should transfer to other  
distinctions.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 1 Aug 2007 00:02:40 +0200

> to both sides necessarily, just observable by the side that needs  
> the answer.

Right. My conclusion was also that having a one-sided mechanism is a  
good start.

But adding some extra logic that allows two sides to determine the  
maximum size they both support, including the situation where one end  
doesn't support jumboframes, without unnecessary test packets, is  
also helpful in addition to that.

> One way to implement RFC 4821 would be to literally send each of  
> the enumerated segment sizes in the first burst (send n segments,  
> of sizes 9000, 8192, 4096, 2048, 1500, 1460, 1400, and 1280, and  
> see how much data is acknowledged in the first acknowledgment).

When you have RFC 4821, life is simple.  :-)

However, off the top of my head I can think of two UDP applications,  
one that absolutely uses large packets (NFS over UDP), and another  
that could in the future (DNS with EDNS0 in the presence of IPv6  
roots and/or DNSSEC) that would fail badly by simply setting the MTU  
to "jumbo" and letting RFC 4821 take care of the difference.

And routers don't know about RFC 4821 so they necessarily have to be  
more conservative than RFC 4821 compliant hosts, and with a router  
with a 1500 byte MTU in the middle life isn't exactly jumbo-sized.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 31 Jul 2007 15:07:19 -0700

> to "jumbo" and letting RFC 4821 take care of the difference.

Add also routing protocols that send large Hellos, LSAs,
etc. over tunnels where neighbors may be separated by
many sub-IP layer hops...

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 31 Jul 2007 15:31:41 -0700

AFAICT, the only significance of 1500 bytes is that it is
the traditional MTU for 10/100 Ethernet gear - that, plus
per [RFC2460] it is the minMRU for IPv6.

Given the tradition to want to "make everything look like
an Ethernet", some have suggested that 1500 bytes can be
considered as the practical "Internet cell size" in today's
Internet. By that measure, perhaps 1501 bytes could be
considered as a "jumbogram"?

It might be convenient to have an 'N' such that we could
say: "for packets no larger than N, we support doing things
the old way and for packets larger than N we recommend doing
things the new way". Is 1500 a reasonable value for N? (Or
*is* there even a reasonable value for N?)

Thanks - Fred

> Sent: Tuesday, July 31, 2007 2:55 PM
> To: Templin, Fred L
> Cc: John Heffner; Internet Area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 1 Aug 2007 00:53:46 +0200

On 1-aug-2007, at 0:31, Templin, Fred L wrote:

> the traditional MTU for 10/100 Ethernet gear

It's not just tradition: all members of the ethernet family have a  
maximum packet size of 1500 bytes. That includes 1 and 10 Gbit  
ethernet. This is necessary to allow ethernets of different speeds to  
be connected together without trouble.

> Internet.

I'm guestimating that for 99% of all packets flowing over the  
internet, there is at least one ethernet hop in the path, so  
practically, yes, this is true as an upper limit. You don't always  
get that, though, because of tunneling. Traditionally, the internet  
has supported many different link technologies with many different  
MTU sizes. Hopefully, we can revive that tradition.

> By that measure, perhaps 1501 bytes could be

First line of RFC 2675:

    A "jumbogram" is an IPv6 packet containing a payload longer than
    65,535 octets.

 From http://en.wikipedia.org/wiki/Jumbo_Frames

In computer networking, jumbo frames are Ethernet frames with more  
than 1500 bytes of payload (MTU).