Subject: Re: [BEHAVE] 64 MTU and fragmentation handling

From: Zhen Cao <[email-masked]>
Date: Thu, 10 Sep 2009 21:30:05 +0800

Hi Iljitsch,

The current discussion only focused on the black hole for 1280
fragments where an IPv4 router reports MTU <1280. IMO, the black hole
may happen in many scenarios.  For instance, if MTU is 1600 for ipv6
side and 1500 for ipv4 side, an IPv6 sender starts sending 1600-byte
packets without fragmentation header to discover the PMTU. But the
ICMP PktTooBig message is blocked or lost due to whatever reasons, the
sender continues to send 1600-byte packets without frag header,
resulting DF=1 translated ipv4 packets longer than 1500 and
communication is lost then.  Has this case been handled by some
existing specfications?

Moreover, current IPv4 network always has MTU greater than 1400, so
the possibility of reporting a <1280 MTU is slim.

Thanks,
Zhen

On Thu, Aug 20, 2009 at 11:57 PM, Iljitsch van Beijnum
<[email-masked]> wrote:
> Hi,
>
> We had a fruitful discussion about MTU and fragmentation handling in the
> online interim meeting yesterday. My original options for handing < 1280
> MTUs on the IPv4 side for stateful translation:
>
> 1. Depend on RFC 2460/2765 behavior where 1280-byte packets with a fragment
> header are sent after a < 1280 "too big" message, translator sets DF=0 when
> translating packets with a fragment header, DF=1 otherwise.
>
> The problem with this is that there is the possibility of a black hole when
> the operator disables path MTU discovery on the IPv6 side by setting the MTU
> to 1280 and filtering "too big" messages. In that case, the fragment header
> is never included, the translator generates a 1260-byte IPv6 packet with
> DF=1 and communication is impossible. This requires unusual, but not unheard
> of cases on both the v4 and v6 sides (< 1260 MTU, PMTUD disabled), but the
> problem is that in the pure IPv4 and pure IPv6 cases these wouldn't cause
> trouble, it's only when communicating through a translator that the black
> hole presents itself.
>
> Solution in the draft:
>
> 3. Rewrite "too big" messages to provide a 1280-byte MTU and have the
> translator set DF=0 for packets <= 1280 bytes and DF=1 for packets > 1280
> bytes.
>
> Alternative solution:
>
> 2. Adopt the RFC 2460/2765 behavior, except that to avoid the black hole,
> packets <= 1280 are sent with DF=0 even if there is no fragment header.
>
> Additional solution mentioned yesterday:
>
> 4. Let the translator track path MTU sizes for IPv4 destinations and
> fragment packets at the source where needed (just like normal IPv6 hosts do
> for IPv6, and in our specification, the translator also does this for
> packets going towards the IPv6 side).
>
> One argument that was made yesterday was the applicability to stateless
> translation. Obviously a mechanism that works for both stateful and
> stateless would be better than separate mechanisms for both. (However, note
> that the identification value handling needs to be different between
> stateful and stateless translation regardless of path MTU discovery
> handling.)
>
> However, solutions 2 and 4 equally apply to the stateful and stateless
> cases. Solution 3 is somewhat less suited for the stateless case, because it
> requires the translator to come up with an identification value. If traffic
> is flowing through multiple translators the identification values could
> clash and result in reassembly errors.
>
> All solutions produce fragments (except perhaps option 4 in the case of TCP
> when TCP is resegmented rather than fragmented), with the associated risk of
> incorrect reassemly and subsequent dependence on the weak TCP/UDP checksum
> to avoid data corruption.
>
> Option 4 has the disadvantage of creating more work for the translator and
> generating more packets and bits, taking over this work from the router in
> front of the < 1280 byte MTU link on the IPv4 side.
>
> As such, I favor option 2 for both stateful and stateless translation, and
> if implementers feel 4 is a viable option I would be willing to include that
> as a not specifically recommended alternative.
>
> Thoughts?
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 10 Sep 2009 16:15:54 +0200

> communication is lost then.  Has this case been handled by some

This is true. However, someone sending 1600-byte packets and filtering  
too big messages would very quickly find out that that combination of  
settings is problematic and fix this one way or another. (Although  
there will be some PMTUD black holes when the sender of 1600-byte  
packets accepts too bigs but the other side fails to generate them.)  
So this situation is orthogonal to the IPv6/IPv4 translation.

The only good solution for this is RFC 4821, but I understand that  
there are other PMTUD blackhole detection mechanisms, and that  
Microsoft implements one of those. Not sure how they work.

> the possibility of reporting a <1280 MTU is slim.

I agree that this wouldn't be a common situation, but "always" is too  
optimistic, I'm afraid...

Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 11 Sep 2009 10:56:05 -0700

Iljitsch,

> -----Original Message-----
> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Thursday, September 10, 2009 7:16 AM
> To: Zhen Cao
> Cc: Behave WG
> Subject: Re: [BEHAVE] 64 MTU and fragmentation handling
> 

I would go one step farther to say that any host sending
DF=1 packets larger than 1500 SHOULD use RFC 4821 or another
suitable PMTUD blackhole detection mechanism. And, failure
to observe the SHOULD may result in unpredictable behavior.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 11 Sep 2009 21:26:42 +0200

We're getting a bit off topic for BEHAVE, but ok...

> to observe the SHOULD may result in unpredictable behavior.

I don't see the relationship with 1500 bytes. You never know when you  
may be communicating through a path that supports a smaller MTU than  
your own MTU, so discovering the path MTU by modulating transport  
segment sizes as per RFC 4821 is ALWAYS a good idea. However, this is  
a complex solution and it only addresses the case where the sender can  
easily manipulate packet sizes. For many non-TCP protocols that isn't  
the case, especially not if the packets are created by the application  
rather than the OS.

Also note that using an MTU larger than 1500 is far less dangerous  
than using an MTU smaller than 1500. In the latter case, if there is a  
problem, the party with the non-1500-byte MTU is in no position to fix  
the problem: the too bigs are traveling towards the correspondent  
host, and are presumably filtered before they get there. And  
presumably increasing the MTU is infeasible, otherwise it wouldn't be  
so small in the first place. In the case of a larger than 1500-byte  
MTU, the operator is in a better position to remove stuff that blocks  
the too bigs, and it's always possible to reduce the MTU to 1500 as a  
last resort.

Last but not least, the TCP MSS option hides a multitude of sins. Only  
when both ends have an MTU larger than the path MTU there may be  
problems.

So I'm not worried about people using MTUs larger than 1500.

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 11 Sep 2009 14:22:22 -0700

Hi Iljitsch,

> Sent: Friday, September 11, 2009 12:27 PM
> To: Templin, Fred L
> Cc: Behave WG
> segment sizes as per RFC 4821 is ALWAYS a good idea.

Right.

> However, this is

So, RFC4821 does have something to say about implementation
in packetization layers other than TCP.
 
> the too bigs, and it's always possible to reduce the MTU to 1500 as a

Not sure I get the distinction you are making, but I will
agree that a 1500- MTU is more likely to be closer toward
the final destination while a 1500+ MTU is more likely to
be closer toward the original source. So that would tend
to reduce the exposure to ICMP filtering simply due to
fewer hops traversed along the reverse path.

> when both ends have an MTU larger than the path MTU there may be

I put MSS clamping in the "band aid solution" category. When
it works, it works great, but IMHO we will begin to see more
and more non-TCP communications that can't use it.

Fred

---

From: Zhen Cao <[email-masked]>
Date: Sat, 12 Sep 2009 14:04:34 +0800

> We're getting a bit off topic for BEHAVE, but ok...

I guess you are saying that the black hole problem not only targets
translation. But translation, especially v4-to-v6 which lengthens the
packet, makes this problem worse. So translation algorithm should take
special care of this problem?

> I don't see the relationship with 1500 bytes. You never know when you may be
> communicating through a path that supports a smaller MTU than your own MTU,
> so discovering the path MTU by modulating transport segment sizes as per RFC
> 4821 is ALWAYS a good idea. However, this is a complex solution and it only
> addresses the case where the sender can easily manipulate packet sizes. For
> many non-TCP protocols that isn't the case, especially not if the packets
> are created by the application rather than the OS.
>
> Also note that using an MTU larger than 1500 is far less dangerous than
> using an MTU smaller than 1500. In the latter case, if there is a problem,
> the party with the non-1500-byte MTU is in no position to fix the problem:
> the too bigs are traveling towards the correspondent host, and are
> presumably filtered before they get there. And presumably increasing the MTU
> is infeasible, otherwise it wouldn't be so small in the first place. In the
> case of a larger than 1500-byte MTU, the operator is in a better position to
> remove stuff that blocks the too bigs, and it's always possible to reduce
> the MTU to 1500 as a last resort.

Not sure i have understood your point correctly. Do you mean that it
is more comfortable for operator to handle 1500+ bytes packets if they
exceed the limit?

I thought they were as dangerous if the ICMP error message could get through.

>
> Last but not least, the TCP MSS option hides a multitude of sins. Only when
> both ends have an MTU larger than the path MTU there may be problems.
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Sep 2009 00:05:48 +0200

> in packetization layers other than TCP.

Right. It's not necessarily impossible, but considering we don't have  
it for TCP yet...

> Not sure I get the distinction you are making,

If you use a larger-than-mainstream MTU then the too bigs are sent to  
you for the packets you send so you can debug them. If you use a  
smaller-than-mainstream MTU the too bigs go from some place in the  
middle to the other end, there's nothing you can see or do in that case.

> and more non-TCP communications that can't use it.

No, I wasn't talking about clamping. Although this is a useful hack in  
a PPPoE box in today's IPv4 internet, implementing MSS clamping in  
pretty much all the routers across the internet isn't feasible and  
also extremely ugly. The point is that when you talk to someone with a  
1500-byte MTU you will be sending 1500-byte (or smaller) TCP packets  
even without clamping.

> special care of this problem?

In an earlier version of my text I took this into account, but since  
then I have changed my mind: I don't think it's worth the trouble to  
do this. The common situation will be for all the systems involved to  
have 1500-byte MTUs. In that case, the IPv6 host will advertise a 1440- 
byte MSS (1500 - 40 bytes IPv6 - 20 bytes TCP) so the IPv4 host will  
send TCP packets of 1480 bytes (1440 + 20 bytes IPv4 + 20 bytes IPv4)  
which the translator then translates into 1500-byte IPv6 packets.

---

From: Zhen Cao <[email-masked]>
Date: Mon, 14 Sep 2009 12:06:13 +0800

On Mon, Sep 14, 2009 at 6:05 AM, Iljitsch van Beijnum
<[email-masked]> wrote:
> Right. It's not necessarily impossible, but considering we don't have it for
> TCP yet...
>
> If you use a larger-than-mainstream MTU then the too bigs are sent to you
> for the packets you send so you can debug them. If you use a
> smaller-than-mainstream MTU the too bigs go from some place in the middle to
> the other end, there's nothing you can see or do in that case.
>
> No, I wasn't talking about clamping. Although this is a useful hack in a
> PPPoE box in today's IPv4 internet, implementing MSS clamping in pretty much
> all the routers across the internet isn't feasible and also extremely ugly.
> The point is that when you talk to someone with a 1500-byte MTU you will be
> sending 1500-byte (or smaller) TCP packets even without clamping.
>
> In an earlier version of my text I took this into account, but since then I
> have changed my mind: I don't think it's worth the trouble to do this. The
> common situation will be for all the systems involved to have 1500-byte
> MTUs. In that case, the IPv6 host will advertise a 1440-byte MSS (1500 - 40
> bytes IPv6 - 20 bytes TCP) so the IPv4 host will send TCP packets of 1480
> bytes (1440 + 20 bytes IPv4 + 20 bytes IPv4) which the translator then
> translates into 1500-byte IPv6 packets.
>
For TCP, yes, For UDP, it is really a problem.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Sep 2009 09:45:23 +0200

On 14 Sep 2009, at 6:06 , Zhen Cao wrote:

>> MTUs. In that case, the IPv6 host will advertise a 1440-byte MSS

> For TCP, yes, For UDP, it is really a problem.

But since UDP doesn't have an MSS option, how do you propose to solve  
this?

As long as UDP packets are sent with DF=0 there shouldn't be too much  
of a problem.

However, I've seen a root DNS server send UDP DNS packets with DF=1 a  
few years ago... Since in IPv4 the OS doesn't do path MTU discovery  
and unless I'm very much mistaken, neither does DNS server software,  
this is a big problem if the DNS server sends a packet larger than the  
path MTU.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 15 Sep 2009 09:10:16 +1000

In message <[email-masked]>, Iljitsch van Beijn
> this is a big problem if the DNS server sends a packet larger than the  

Which apart from on Solaris the nameserver can turn off.  When I
noticed this brokeness on the part of the kernel I modified named
to turn DF off on UDP sockets on all platforms where there was a
per socket api to do this.  Other nameserver vendors should also
be aware of this by now.

Now there are two question.

1. Do we need to provided a Advanced Socket API for IPv4 that includes
   this?

2. Do we need to do a "DF and UDP, by default, Considered Harmful" RFC?

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]