Subject: [Softwires] MTU issue for 6RD (and 6to4)

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 Nov 2009 13:28:02 +1300

Strangely, I only realised yesterday that Teredo has a
recommended MTU of 1280, whereas 6to4 doesn't, and simply
refers to the general discussion of MTU in the basic
transition tunnel specs. 6RD follows the 6to4 approach.

The reason I realised this difference was a comment on
an IPv6 list we run in NZ; here is an out of context extract:

> On 2009-11-08 15:37, TreeNet Admin wrote:
>> > Donald Neal wrote:
> 
> ...
>>> >> The step-wise progression of the graphs for these larger packets seems
>>> >> to reflect the way in which packets are fitted into the available MTU. A
>>> >> reduction in MTU from 1500 to 1280 with a resulting throughput
>>> >> loss of 13% wouldn't be undertaken lightly. The 4.9% throughput
>>> >> improvement gained by taking MTU up to a level where 4096 bytes of
>>> >> payload will fit inside the MTU might be more attractive.
>> > 
>> > Interesting things to note here:
>> > 
>> >  Default MTU of teredo:  1280
>> >  Default MTU of 6to4:    1500
>> >    (requires manual reduction to ~1400 for trouble evasion)

Note the last line.

Should we be specifying a default MTU of 1280 for 6RD too?

(The same question applies retroactively to 6to4, of course.)

    Brian

---

From: james woodyatt <[email-masked]>
Date: Sun, 8 Nov 2009 16:57:23 -0800

> (The same question applies retroactively to 6to4, of course.)

They're not the same, of course.  In the 6to4 case, I've found that  
setting the MTU to 1280, despite what the RFC says, is the only  
sensible way to keep the various accidental and deliberate path MTU  
discovery obstacles on public IPv4 networks from causing mysterious  
connectivity problems.  In the 6RD case, because the relay is  
bidirectional, clamping the MTU at 1280 might be overkill.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 8 Nov 2009 17:01:21 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of james woodyatt
> Sent: Sunday, November 08, 2009 4:57 PM
> To: [email-masked]
> Subject: Re: [Softwires] MTU issue for 6RD (and 6to4)
> 

I don't get the "because the relay is bidirectional" part;
path MTU is by nature a unidirectional consideration.

Fred
[email-masked]

> _______________________________________________
> Softwires mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/softwires

---

From: "Lee, Yiu" <[email-masked]>
Date: Sun, 8 Nov 2009 20:09:02 -0500

James has a point. My question is if the the 6rd CE receives ICMP Packet Too
BIG, should it translate to ICMPv6 Packet too BIG to the host behind the 6rd
CE?

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Nov 2009 10:12:22 +0900

> (The same question applies retroactively to 6to4, of course.)

this is what the draft says:

"6rd's scope is limited to a service provider network. If the MTU is  
well-managed
such that the IPv4 MTU on the 6rd CE WAN interface is set so that no
fragmentation occurs within the boundary of the SP, then the IPv6 MTU
should be set to the IPv4 MTU minus the size of the encapsulating IPv4
header (20 bytes)."

the assumption being that the service provider maintains a consistent  
MTU across its network.

there is more reason for 6to4 to have a default MTU of 1280 though.

cheers,
Ole

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Nov 2009 10:14:50 +0900

> BIG, should it translate to ICMPv6 Packet too BIG to the host behind  

yes, it should follow RFC4213, section 3.4

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 8 Nov 2009 17:26:47 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ole Troan
> Sent: Sunday, November 08, 2009 5:15 PM
> To: Lee, Yiu
> Cc: [email-masked]
> Subject: Re: [Softwires] [Softwire] MTU issue for 6RD (and 6to4)
> 

Two points about this. First, it requires that ICMPs
be delivered without filtering and that ICMPs cannot
be spoofed. This may be possible only within certain
carefully controlled environments. Second, the translated
ICMPs will always go through to the original source hosts
that would repeatedly be inconvenienced by excessive
interactions with path MTU discovery. (Note also that
translation is not always guaranteed to be possible
based on the amount of information conveyed in ICMPs.)

Most importantly, dynamic path MTU discovery within the
tunnel does not hide the effects of adding encapsulation.
The overhead for encapsulation will be seen by the
original sources in any case.

Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 Nov 2009 17:21:17 +1300

> the assumption being that the service provider maintains a consistent

Sure. If the v4 MTU is 1500, set the 6rd MTU to 1480.

But shouldn't we specify a default for shipped products,
and shouldn't it be 1280 (as the least bad choice)?

> there is more reason for 6to4 to have a default MTU of 1280 though.

I agree, and I wish we had specified a default.

All this is orthogonal to PMTUD and ICMP translation; we're talking
about a default for when PMTUD fails to work at all.

   Brian

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Sun, 8 Nov 2009 22:26:04 -0600

It's not as simple as that.  What if the 6rd functionality is running on
the access concentrator like a CMTS in cable?  In a cable deployment,
due to the added len from docsis headers, the MTU in cable between the
home and the cable access concentrator is actually less than the typical
MTU for Ethernet LANs.  So do we really have to specify an MTU for the
6rd specification?

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Brian E Carpenter
Sent: Monday, November 09, 2009 1:21 PM
To: Ole Troan
Cc: [email-masked]
Subject: Re: [Softwires] MTU issue for 6RD (and 6to4)

   Brian

---

From: Dave Thaler <[email-masked]>
Date: Mon, 9 Nov 2009 04:30:35 +0000

[...]
> I agree, and I wish we had specified a default.

FWIW, Windows defaults to 1280 for 6to4.

-Dave

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Sun, 8 Nov 2009 22:40:57 -0600

Specifying a low default like 1280 sounds fine by me because after
accounting for cable docsis headers this value will work if 6rd
functionality is running in the cable access concentrator.

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Dave Thaler
Sent: Monday, November 09, 2009 1:31 PM
To: Brian E Carpenter; Ole Troan
Cc: [email-masked]
Subject: Re: [Softwires] MTU issue for 6RD (and 6to4)

[...]

-Dave

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 8 Nov 2009 21:03:10 -0800

If the tunnel endpoints want to avoid fragmentation
and reassembly, then setting the lower MTU seems
prudent. If the tunnel endpoints are willing to
incur a little fragmentation and reassembly, then
the MTU could possibly be pushed to 1500 or more.

As far as I can see, there is no way to truly sweep
the extra overhead for encapsulation under the carpet
without invoking fragmentation and reassembly within
the tunnel.

Fred

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Hemant Singh
> (shemant)
> Sent: Sunday, November 08, 2009 8:41 PM
> To: Dave Thaler; Brian E Carpenter; Ole Troan
> Cc: [email-masked]

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Nov 2009 14:32:29 +0900

Hemant,

> MTU for Ethernet LANs.  So do we really have to specify an MTU for the

in that case the IPv4 MTU is wrong too right?
the 6rd assumption is that the IPv4 WAN interface MTU is correct and  
the MTU in the SP network is well managed. I'd prefer that to be the  
default as opposed to a fixed value of 1280. with a default of 1280 it  
appears you would need 6rd specific provisioning if one wants a  
different value.

cheers,
Ole

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Sun, 8 Nov 2009 23:40:03 -0600

>From: Ole Troan [mailto:[email-masked]] On Behalf Of Ole Troan
>Sent: Monday, November 09, 2009 2:32 PM
>To: Hemant Singh (shemant)
>Cc: Brian E Carpenter; [email-masked]

>in that case the IPv4 MTU is wrong too right?

Yes.

>appears you would need 6rd specific provisioning if one wants a  

Totally agree.  That is why I said why even bother specifying any value
for MTU default in the 6rd document when the default has to be equal to
the IP4 MTU.  Other documents have got to exist that define MTU values
for tunnels. 

Hemant

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 Nov 2009 18:49:37 +1300

Sure, (v4MTU - 20) is a reasonable formula once you
know the value of v4MTU, as long as the result is at
least 1280. But if you don't know v4MTU, you have to
have *something* as the default, and I can't see anything
better than 1280.

Regards
   Brian Carpenter

On 2009-11-09 18:32, Ole Troan wrote:
> the 6rd assumption is that the IPv4 WAN interface MTU is correct and the
> MTU in the SP network is well managed. I'd prefer that to be the default
> as opposed to a fixed value of 1280. with a default of 1280 it appears
> you would need 6rd specific provisioning if one wants a different value.
> 
> cheers,
> Ole
> 
>

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Nov 2009 15:00:14 +0900

Brian,

> have *something* as the default, and I can't see anything

agree, but in which cases would you not know the v4MTU?

cheers,
Ole

---

From: james woodyatt <[email-masked]>
Date: Sun, 8 Nov 2009 22:09:24 -0800

On Nov 8, 2009, at 21:03 , Templin, Fred L wrote:

> [...] As far as I can see, there is no way to truly sweep the extra  
> overhead for encapsulation under the carpet without invoking  
> fragmentation and reassembly within the tunnel.

I'm confused.  It was my impression that IPv4 showed us how expecting  
the network to handle fragmentation and reassembly on a hop-by-hop  
basis wasn't such a good idea after all, and that we lifted  
fragmentation and reassembly up to the level of destination option  
processing precisely to simplify the forwarding path in routers.  Am I  
missing something?

I mean, when I do the back of the envelope calculations about  
overhead, I don't see numbers that make a compelling argument for  
unburdening hosts from the need to do path MTU discovery, especially  
when the first-mile IPv6 link is tunneled over IPv4.  You're always  
going to have an IPv4 header in every ~1500 octet packet. After that,  
you have two alternatives:

   A) the IPv6 host has discovered the path MTU including the IPv4  
encapsulation, and there is an IPv6 header in every IPv4 packet;

   B) the IPv6 host has been saved from learning the smaller path MTU  
for the IPv4 tunnel and learns instead the larger path MTU, which is  
probably not going to be more than 1500 kB for at least as long as 6RD  
is expected to live, so here there are two subcases:

     1) the IPv6 packet is small enough that it doesn't need  
fragmentation, e.g. empty TCP ACK packets, in which case there is an  
IPv6 header in the IPv4 packet, just like the first alternative;

     2) the IPv6 packet is larger than the tunnel MTU, e.g. a full- 
size TCP segment for the IPv6 path MTU, and so there are two or more  
IPv4 packets and the IPv6 header is amortized between them.

If the IPv4 path MTU between the 6RD CPE router and the 6RD relay is  
the typical 1500 octets, then you don't win anything by doing fragment  
reassembly at the tunnel endpoints unless the IPv6 path MTU is larger  
than 2960 octets.  That's not happening any time soon for basically  
anyone in the intended user community for 6RD.

It's going to be a long time before residential subscriber networks  
can be reasonably expected to support a link MTU large enough for this  
path MTU business to be important.  People keep deploying heterogenous  
bridges between various link layers with different native MTU, e.g. Wi- 
Fi with 2304 and/or 4000 depending, Gig-E with 9000.  Sadly, IPv6  
neighbor discovery doesn't cope well with jumbo frames on such  
networks, so everyone uses 1500 octets for the same historical reasons  
we've always had.  They're not going away.

Until IPv6 neighbor discovery can cope with the issue of path MTU  
discovery on local links [*], I'm not getting excited about the  
encapsulation overhead for hosts behind 6RD gateways.

[*] c.f. <http://tools.ietf.org/html/draft-van-beijnum-multi-mtu>

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Nov 2009 15:24:11 +0900

> have *something* as the default, and I can't see anything

re-reading RFC4213 section 3.2.
it does say that the tunnel MTU MUST be between 1280 and 1480 and it  
SHOULD be 1280 by default.
this is a general problem and really shouldn't be solved in 6rd. if we  
think the MTU discussion in 4213 is missing something we should update  
that RFC instead.

is the current text in the draft acceptable or should we say even less  
and just reference 4213:

"MTU and fragmentation issues for IPv6 in IPv4 tunnelling is discussed  
in detail in section 3.2 of RFC4213 [RFC4213]. 6rd's scope is limited  
to a service provider network. If the MTU is well-managed such that  
the IPv4 MTU on the 6rd CE WAN interface is set so that no  
fragmentation occurs within the boundary of the SP, then the IPv6 MTU  
should be set to the IPv4 MTU minus the size of the encapsulating IPv4  
header (20 bytes). IPv4 Path MTU discovery MAY be used to adjust the  
MTU of the tunnel as described in section 3.2.2 of RFC4213 [RFC4213]  
or the IPv6 tunnel MTU may be explicitly configured."

cheers,
Ole

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 Nov 2009 20:02:23 +1300

I certainly agree this is nothing specific to 6RD.
I also think that RFC4213 is correct (section 3.2.1.,
Static Tunnel MTU), but it seems as if at least some 6to4
implementations don't follow it (probably because they are
old enough to have consulted RFC2893, or because they
don't believe that the requirement applies).

You also asked:

> in which cases would you not know the v4MTU?

I have no idea. Maybe some very strange implementation?

   Brian

On 2009-11-09 19:24, Ole Troan wrote:
> in detail in section 3.2 of RFC4213 [RFC4213]. 6rd's scope is limited to
> a service provider network. If the MTU is well-managed such that the
> IPv4 MTU on the 6rd CE WAN interface is set so that no fragmentation
> occurs within the boundary of the SP, then the IPv6 MTU should be set to
> the IPv4 MTU minus the size of the encapsulating IPv4 header (20 bytes).
> IPv4 Path MTU discovery MAY be used to adjust the MTU of the tunnel as
> described in section 3.2.2 of RFC4213 [RFC4213] or the IPv6 tunnel MTU
> may be explicitly configured."
> 
> cheers,
> Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 8 Nov 2009 23:02:46 -0800

James,

> Sent: Sunday, November 08, 2009 10:09 PM
> To: [email-masked]

Yes, you are missing something that I didn't explain well; I
do not want outer IP fragmentation nor inner IP fragmentation.
I am referring to a mid-layer segmentation and reassembly
capability that is neither/nor. That is something that can
be coordinated btw tunnel endpoints without incurring the
harmful aspects of IP fragmentation and reassembly.

> you have two alternatives:

Its not necessarily the first-mile IPv6 link and
oftentimes will not be. There may be an arbitrarily
complex IPv6 network stacked up behind the CPE router,
for example. When hosts have to constantly participate 
in path MTU discovery, there will always be packet loss
and there will always be round-trip delays before the
host can reduce its packet size and try again. It would
be good to avoid this if possible.  

> is expected to live, so here there are two subcases:

Why do you say no more than 1500? As long as the tunnel
egress is willing to reassemble larger (e.g., 9K) then
the ingress can accommodate the larger packet size using
segmentation and reassembly even if the physical links
on the path only configure smaller MTUs (e.g., 1500).

> IPv4 packets and the IPv6 header is amortized between them.

If you mean the IPv6 header is amortized due to segmentation
and reassembly between the tunnel endpoints, then that is
good and I agree.

> anyone in the intended user community for 6RD.

As explained above, with segmentation and reassembly the tunnel
MTU is really independent of the MTUs of the underlying links
on the path. All that matters is the size the egress is willing
to reassemble, so we can happily do 9KB across the tunnel even
if the links in the path only support 1500.

> we've always had.  They're not going away.

Residential users don't need to move to larger MTUs before
this becomes important; the benefits are there even at 1500.
What matters is that the tunnel not unnecessarily reduce
the size the hosts see on their native links regardless
of what that size is. 

> encapsulation overhead for hosts behind 6RD gateways.

Huh? This is all about supporting multiple MTUs on the
same link. Using SEAL, multi-MTU handling occurs at a
layer below IPv6 neighbor discovery, and ND never even
needs to get involved.

Multi-MTUs on the same link are to be expected and are
naturally accommodated by SEAL.

Fred

---

From: Rémi Després <[email-masked]>
Date: Mon, 9 Nov 2009 18:52:57 +0900

Le 9 nov. 09 à 15:24, Ole Troan a écrit :

> it does say that the tunnel MTU MUST be between 1280 and 1480 and  
> it SHOULD be 1280 by default.
> this is a general problem and really shouldn't be solved in 6rd. if  
> we think the MTU discussion in 4213 is missing something we should  
> update that RFC instead.
>
> is the current text in the draft acceptable or should we say even  
> less and just reference 4213:
>
> "MTU and fragmentation issues for IPv6 in IPv4 tunnelling is  
> discussed in detail in section 3.2 of RFC4213 [RFC4213]. 6rd's  
> scope is limited to a service provider network. If the MTU is well- 
> managed such that the IPv4 MTU on the 6rd CE WAN interface is set  
> so that no fragmentation occurs within the boundary of the SP, then  
> the IPv6 MTU should be set to the IPv4 MTU minus the size of the  
> encapsulating IPv4 header (20 bytes). IPv4 Path MTU discovery MAY  
> be used to adjust the MTU of the tunnel as described in section  
> 3.2.2 of RFC4213 [RFC4213] or the IPv6 tunnel MTU may be explicitly  
> configured."

After the second sentence, what about something in the following flavor:

"SPs that use 6rd MUST ensure that IPv4 packets that encapsulate IPv6  
packets are not fragmented within each part of their network to which  
they have assigned a 6rd SP prefix.

For this, each such 6rd network part MUST have for all its possible  
paths an IPv4 MTU of at least 1300 octets, i.e. 1280, the minimum  
IPv6 MTU, plus 20 octets, the length of the encapsulating header.

If a SP can guarantee, for a given a 6rd CPE or relay that the MTU of  
all its paths betwen 6rd CPEs and relays is some value M larger than  
1300 octets, then this node MAY set its IPv6 MTU to M - 20 octets.  
For example, if a 6rd network part has MTUs of at least 1500 octets  
everywhere, each of its 6rd CPE or relay may individually set its  
accepted IPv6 MTU to 1480 octets."

In my understanding, it covers what has been discussed, and expresses  
more rigorously. The first paragraph is essential. The two others are  
just clarifications.
`
>
> cheers,
> Ole_______________________________________________
> Softwires mailing list
> [email-masked]