Subject: [BEHAVE] Fwd: IPv6 hosts sending <1280 byte packets

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 7 Feb 2010 20:40:06 +0100

Ben Stasiewics has done some testing to see if hosts actually respond with a fragment header after a < 1280 too big message, and the results are surprising.

Looks like we can't depend on hosts implementing this correctly.

Begin forwarded message:

> From: Ben Stasiewicz <[email-masked]>
> Date: 7 februari 2010 19:39:08 GMT+01:00
> To: [email-masked]
> Cc: Matthew Luckie <[email-masked]>
> Subject: Re: IPv6 hosts sending <1280 byte packets [was RE: RRG discussion of SEAL, IPTM - and my critique of RFC4821]

> On 29/01/10 08:40, Dan Wing wrote:
>> Ben, would it be possible to conduct a test to see how hosts react to PTB
>> smaller than 1280?

> I conducted such a test and found that 299 (43.46%) of the 688
> IPv6-capable web servers that I tested did include an IPv6 fragmentation
> header in their response packets after they were sent an ICMPv6 PTB
> message specifying an MTU < 1280 bytes. The other 389 (56.54%) did not.

> I am happy to answer any questions about the test.

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 08:21:06 -0800

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
> Sent: Sunday, February 07, 2010 11:40 AM
> To: [email-masked] WG
> Subject: [BEHAVE] Fwd: IPv6 hosts sending <1280 byte packets
> 
> Ben Stasiewics has done some testing to see if hosts actually 
> respond with a fragment header after a < 1280 too big 
> message, and the results are surprising.
> 

To be clear, Ben Stasiewicz's test does not determine if the problem is hosts
failing to implement the last paragraph of Section 5 of RFC2460 correctly, or
if host-/network-based firewalls are blocking ICMPv6 packet-too-big messages
or blocking IPv6 packets with the fragment header.

I recently received email from TAHI (http://www.tahi.org), which does IPv6
conformance testing.  Their test is listed at
http://www.ipv6ready.org/docs/Core_Conformance_Latest.pdf and its test
v6LC.4.1.6 ("Receiving MTU Below IPv6 Minimum Link MTU") which appears to have
been tested since 2005.  So any recipient of the IPv6 Ready certification will
implement things correctly.  And there may well be sysctl settings to tweak
this specific behavior.

But no matter the reason, I agree that a greater than 50% failure on the
Internet means the function is effectively broken.  In my view, Ben's testing
invalidates the consensus reached in the meeting at IETF76 (Hiroshima), and I
have asked authors of xlate and xlate-stateful to revise the documents
accordingly.

If anyone thinks we should keep the IETF76 consensus, please speak up now!

-d

> > Subject: Re: IPv6 hosts sending <1280 byte packets [was RE: 
> RRG discussion of SEAL, IPTM - and my critique of RFC4821]
> 
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 08 Feb 2010 17:52:35 +0100

El 08/02/10 17:21, Dan Wing escribió:
> invalidates the consensus reached in the meeting at IETF76 (Hiroshima),and I
>    

I am not sure what the translator should do now... what do you have in mind?

Regards, marcelo

>>> From: Ben Stasiewicz<[email-masked]>
>>> Cc: Matthew Luckie<[email-masked]>
>>> Subject: Re: IPv6 hosts sending<1280 byte packets [was RE:
>>>        
>>>> Ben, would it be possible to conduct a test to see how
>>>>          
>> hosts react to PTB
>>      
>>>> smaller than 1280?
>>>>          
>>      
>>> IPv6-capable web servers that I tested did include an IPv6
>>>        
>> fragmentation
>>      
>>> message specifying an MTU<  1280 bytes. The other 389
>>>        
>> (56.54%) did not.
>>
>>

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 09:46:13 -0800

> [mailto:[email-masked]] On Behalf Of marcelo bagnulo braun
> Sent: Monday, February 08, 2010 8:53 AM
> To: [email-masked]
> Subject: Re: [BEHAVE] Fwd: IPv6 hosts sending <1280 byte packets
> 
> > have asked authors of xlate and xlate-stateful to revise 
> > the documents accordingly.
> >
> >    
> 

We can't rely on sending ICMPv6 packet-too-big to IPv6 hosts
if the packet-too-big indicates an MTU less than 1280.  Thus,
the text from RFC2765 (SIIT),

   However, [IPv6]
   requires that IPv6 nodes handle such an ICMP "packet too big" message
   by reducing the path MTU to 1280 and including an IPv6 fragment
   header with each packet.  This allows end-to-end path MTU discovery
   across the translator as long as the path MTU is 1280 bytes or
   greater.  When the path MTU drops below the 1280 limit the IPv6
   sender will originate 1280 byte packets that will be fragmented by
   IPv4 routers along the path after being translated to IPv4.

is true, but testing results on the Internet show it fails more than
50% of the time.  Thus, we can't rely on it.  draft-ietf-behave-v6v4-xlate-08 
has incorporated that same text from RFC2765 and the current design 
relies on that function.  

For stateless translation, we need to simply translate small packets
(which was smaller than 1280 on the IPv6 side) with IPv4 DF=0, and allow
it to be fragmented on the IPv4 side.  This is because if the translator
does receive an IPv4 ICMP packet-too-big describing an MTU smaller than 
1280, and translated it to IPv6, more than 50% of the IPv6 hosts would 
be unable to successfully process it.  This means we lose end-to-end 
PMTUD across the translator for links smaller than 1280.  However, we 
still have end-to-end PMTUD for links greater than 1280, so translators
will allow moving to larger packets in the future.

For stateful translation, it is _possible_ for the translator
to include MTU in its IPv4 and IPv6 tables, and do provide end-to-end
PMTUD even for links below 1280.  To do this, a stateful translator
would maintain the MTU for all IPv4 hosts that currently have a mapping,
and go ahead and translate with DF=1.  Then, if an IPv4 ICMP PTB is
received and describes an MTU smaller than 1280, go ahead and translate
that to ICMPv6 (in the hopes of hitting the ~45% of hosts that will
successfully process it), but also modify the MTU in the mapping table
so that subsequent packets are always fragmented by the stateful
translator.  This allows the IPv6 host to realize it isn't getting
end-to-end PMTUD (because it was sent the ICMPv6 PTB message, 
indicating a smaller-than-1280 MTU), and also allows the stateful
translator to set DF=0 only for those IPv4 destinations that need
it.  But, this all seems complicated for stateful translation; I 
don't know if we need to get into such an optimization, especially at
this late date with our specifications.

Thoughts?

-d

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 8 Feb 2010 21:27:00 +0100

On 8 feb 2010, at 18:46, Dan Wing wrote:

> and go ahead and translate with DF=1.

So in this case the translator receives IPv6 packets with or without a fragment header that are 1280 bytes but the IPv4 destination has a path MTU of (say) 576 bytes. This means the translator has to fragment.

So essentially this means doing more work (keeping PMTUD state for the IPv4 path) in order to do even more work later (fragment). Not sure how that makes sense... If you simply set DF=0 for packets < 1280 bytes then the people who thought it was a good idea to set an unreasonably small MTU are the ones who get to fragment, which to me seems how it should be.

(Although in practice few fragments will happen because in IPv4 PMTUD is only supported for TCP and TCP has the MSS option.)

> Then, if an IPv4 ICMP PTB is
> successfully process it),

You can always do that, yes.

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 14:34:25 -0800

> From: Iljitsch van Beijnum [mailto:[email-masked]] 
> Sent: Monday, February 08, 2010 12:27 PM
> To: Dan Wing
> Cc: 'marcelo bagnulo braun'; [email-masked]
> So in this case the translator receives IPv6 packets with or 
> without a fragment header that are 1280 bytes but the IPv4 
> destination has a path MTU of (say) 576 bytes. This means the 
> translator has to fragment.
> 
> So essentially this means doing more work (keeping PMTUD 
> state for the IPv4 path) in order to do even more work later 
> (fragment). Not sure how that makes sense... If you simply 
> set DF=0 for packets < 1280 bytes then the people who thought 
> it was a good idea to set an unreasonably small MTU are the 
> ones who get to fragment, which to me seems how it should be.

Yes, it means the translator is doing the effort of fragmenting
rather than the router immediately in front of the small-MTU
link.  However, by doing it this way, we can maintain end-to-end
PMTUD for those IPv6 hosts that receive ICMPv6 PTB and send
packets with the fragmentation header.  Those hosts now know
their packets are being fragmented.  Certainly there was some
perceived value in the IPv6 knowing its packets are being
fragmented.

> (Although in practice few fragments will happen because in 
> IPv4 PMTUD is only supported for TCP and TCP has the MSS option.)

I agree that nearly 100% of PMTUD on IPv4 is with TCP.

But there isn't anything prohibiting PMTUD for UDP.  
For UDP, draft-petithuguenin-behave-stun-pmtud describes how to do
PMTUD and uses the technique described in RFC4821 (which doesn't 
rely on receiving ICMP messages).

> You can always do that, yes.

I tend to think, though, we don't want to figure this out
for stateful translators at this point in time.  If it is
found useful/necessary, it can be specified later.  It feels
like an optimization to retain PMTUD awareness in the IPv6
network for less-than 1280 payloads for stateful translation.
It does not seem necessary, because we can maintain PMTUD
end-to-end for packets bigger than 1280 bytes.  Thus, 
stateful (or stateful) translation does not harm the hoped-
for transition to larger MTUs in the future.

-d

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 00:02:25 +0100

On 8 feb 2010, at 23:34, Dan Wing wrote:

>> (fragment). Not sure how that makes sense...

> packets with the fragmentation header.

No, the two issues are orthogonal.

At some point, an IPv6 host sends a packet that is larger than the IPv4 PMTU. A router sends back a too big message. Issue A is how to handle this too big message. Issue B is what happens to packets that the IPv6 host subsequently sends.

I think there is agreement that A should be handled by simply translating the IPv4 too big too IPv6 and adjusting it as per the header size differences, but otherwise let it through transparently. (I once argued for rewriting it to 1280, though.)

Nothing has changed here.

But now for part B. A stateless translator can't monitor the IPv4 PMTU. Not because it's stateful (we're talking per destination state, not per session state, it would be doable to track this) but because there may be more than one translator so any given translator may not see all packets. So for stateless translation when the translator has a < 1280 packet with no fragment header we don't know whether this is because PMTUD was succesful and the packet size fits in the PMTU, or the IPv6 host ignored the < 1280 too big message and is omitting the fragment header in violation of the relevant RFCs. In order to be compatible with the latter the only thing we can do is set DF to 0.

(I just realize that we don't know what happens when for instance the PMTU is 1000 and the IPv6 host wants to send a 1100-byte packet: does it include a fragment header or not?)

Note that for part B, if there _is_ a fragment header, that will also trigger the translator to set DF to 0. So in the case where the IPv6 hosts are well behaved there is no change to either parts A or B.

In the stateful case we know all packets flow through the same translator so we get to track the IPv4 PMTU so when there is an IPv6 packet that needs to be translated we know whether it's bigger or smaller than the PMTU so we can send packets that are bigger with DF=1 and packets that are smaller with DF=0 and if we are to, also immediately fragment them. However, there is no advantage to doing this extra work because even if we get to set DF=1 on some packets that are smaller than the PMTU that only lets us discover a reduction in the PMTU, but that discovery doesn't buy us anything because the IPv6 host isn't going to reduce its packet size. Fragmenting at the translator also doesn't buy the translator anything and it may even get the packet blocked or create more work if there's a firewall with a > 1280 MTU before the path hits the < 1280 MTU which would need to reassemble the packet to observe the port numbers. And it could be an attack vector: attackers could send too bigs with tiny MTUs to make the translator work harder.

> Certainly there was some

I don't think so. Even if the application somehow learns this information, what is it supposed to do then?

> But there isn't anything prohibiting PMTUD for UDP.

With sufficient thrust pigs fly just fine...

Generally, it's not easy for UDP applications to limit their packet sizes. So the protocol would have to support changing the packet size on the fly and then the application would have to react to too big messages. Not impossible, but I've never seen it happen. But if BitTorrent goes to UDP it's likely they'll put this in in some way.

> found useful/necessary, it can be specified later.

Right.

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 15:28:38 -0800

> Sent: Monday, February 08, 2010 3:02 PM
> To: Dan Wing
> At some point, an IPv6 host sends a packet that is larger 
> than the IPv4 PMTU. A router sends back a too big message. 
> Issue A is how to handle this too big message. Issue B is 
> what happens to packets that the IPv6 host subsequently sends.

A stateless translator cannot identify 'subsequent' packets,
because it doesn't remember previous state.  

> I think there is agreement that A should be handled by simply 
> translating the IPv4 too big too IPv6 and adjusting it as per 
> the header size differences, but otherwise let it through 
> transparently. (I once argued for rewriting it to 1280, though.)
> 
> But now for part B. A stateless translator can't monitor the 
> IPv4 PMTU. Not because it's stateful (we're talking per 
> destination state, not per session state, it would be doable 
> to track this) but because there may be more than one 
> translator so any given translator may not see all packets. 
> So for stateless translation when the translator has a < 1280 
> packet with no fragment header we don't know whether this is 
> because PMTUD was succesful and the packet size fits in the 
> PMTU, or the IPv6 host ignored the < 1280 too big message and 
> is omitting the fragment header in violation of the relevant 
> RFCs. In order to be compatible with the latter the only 
> thing we can do is set DF to 0.
> 
> (I just realize that we don't know what happens when for 
> instance the PMTU is 1000 and the IPv6 host wants to send a 
> 1100-byte packet: does it include a fragment header or not?)
> 
> Note that for part B, if there _is_ a fragment header, that 
> will also trigger the translator to set DF to 0. So in the 
> case where the IPv6 hosts are well behaved there is no change 
> to either parts A or B.

So, I believe we are in agreement:  IPv6 packets less than
1280 need to be translated to IPv4 and sent with DF=0.

> In the stateful case we know all packets flow through the 
> same translator so we get to track the IPv4 PMTU so when 
> there is an IPv6 packet that needs to be translated we know 
> whether it's bigger or smaller than the PMTU so we can send 
> packets that are bigger with DF=1 and packets that are 
> smaller with DF=0 and if we are to, also immediately fragment 
> them. However, there is no advantage to doing this extra work 
> because even if we get to set DF=1 on some packets that are 
> smaller than the PMTU that only lets us discover a reduction 
> in the PMTU, but that discovery doesn't buy us anything 
> because the IPv6 host isn't going to reduce its packet size. 

I agree the IPv6 host won't reduce its packet size below
1280.  But the IPv6 host becomes aware that the path MTU
is smaller than 1280.  I don't know if the IPv6 host finds
that useful/valuable to know the path MTU is less than the
IPv6 minimum MTU.

> Fragmenting at the translator also doesn't buy the translator 
> anything and it may even get the packet blocked or create 
> more work if there's a firewall with a > 1280 MTU before the 
> path hits the < 1280 MTU which would need to reassemble the 
> packet to observe the port numbers. And it could be an attack 
> vector: attackers could send too bigs with tiny MTUs to make 
> the translator work harder.
> 
> > Certainly there was some
> I don't think so. Even if the application somehow learns this 
> information, what is it supposed to do then?

Don't know.  It is obviously easier for the 6/4 translator to
simply fragment the packet (or translate and send with DF=0) without 
informing the IPv6 host, but there is long-standing text in both
RFC2460 and its predecessor (RFC1883) that says to send back
ICMPv6 PTB and the IPv6 host is then supposed to send packets
with the fragmentation header.

> Generally, it's not easy for UDP applications to limit their 
> packet sizes. So the protocol would have to support changing 
> the packet size on the fly and then the application would 
> have to react to too big messages.

draft-petithuguenin-behave-stun-pmtud does not rely on
ICMP packet too big messages.

> Not impossible, but I've never seen it happen.

Me, neither.  I'm just trying to temper the "only TCP does
PMTUD", because UDP can do PMTUD.  And of course there is value
in sending the largest packets possible.

> But if BitTorrent goes to UDP it's 
> likely they'll put this in in some way.

As of version 2.x, the most popular BitTorrent client on
Windows (uTorrent) has implemented uTP, for whatever that's
worth.  I don't know if they're doing PMTUD, though.  Beta
versions of uTorrent are available for OSX, too, and I expect
they will soon support uTP (if they aren't already).

> Right.=

-d

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Feb 2010 16:01:56 -0800

The concern with setting DF=0 on the IPv4 side for packets
that are 1280 or smaller on the IPv6 side is the possibility
for RFC4963 data corruption if we let it run at line rate.

There are certainly plenty of websites on the IPv4 Internet
that set DF=0 on every packet they send, and many even use
packet sizes larger than 1280. But, how many of them send
large packets at line rate?

Fred
[email-masked]  

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dan Wing
> Sent: Monday, February 08, 2010 3:29 PM
> To: 'Iljitsch van Beijnum'
> Cc: [email-masked]

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 16:24:12 -0800

> From: Templin, Fred L [mailto:[email-masked]] 
> Sent: Monday, February 08, 2010 4:02 PM
> To: Dan Wing; 'Iljitsch van Beijnum'
> Cc: [email-masked]
> Subject: RE: [BEHAVE] Fwd: IPv6 hosts sending <1280 byte packets
> 

At high speed with the same source/destination addresses, and when
fragmentation on the IPv4 network occurs.  That is, sending lots of packets
with DF=0 does not, by itself, cause a problem even if they are fragmented on
the same link -- so long as they have different source or different
destination addresses.  I agree we should make a note of that.  

Is there another approach that should be considered to avoid that concern?

> large packets at line rate?

Content delivery networks?

-d

> [mailto:[email-masked]] On Behalf Of Dan Wing

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Feb 2010 17:11:06 -0800

Dan,

> Sent: Monday, February 08, 2010 4:24 PM
> To: Templin, Fred L; 'Iljitsch van Beijnum'
> Cc: [email-masked]

Yes, the concern is for same src,dst operating at line 
rate. IPv6 apps assume that there is no possibility for
fragmentation in the network, hence they may not be as
shy as IPv4 apps when it comes to sending large packets
at high data rates. Having the translator set DF=0 can
break that assumption.
 
> Is there another approach that should be considered to avoid that concern?

I still believe placement of the translator matters.
Placing the translator closer to the IPv4 host
reduces the portion of the path that is exposed to
IPv4 hops. This is true whether the DF=0 or DF=1
approach is used.

Fred

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Feb 2010 17:16:26 -0800

> [mailto:[email-masked]] On Behalf Of Templin, Fred L
> Sent: Monday, February 08, 2010 5:11 PM

There are three approaches discussed in new text in
http://tools.ietf.org/html/draft-ietf-behave-v6v4-xlate-08#page-18,
which only set DF=0 for "small packets" (packets that were
less than 1280 on the IPv6 side).

I don't think that eliminates your concern, though.

> IPv4 hops. This is true whether the DF=0 or DF=1

Agreed.

However, that isn't always possible.  For example,
someone operating an IPv6-only network will have to
place their IPv6/IPv4 translator at their network
border.

But perhaps some text could be added to 
draft-ietf-behave-v6v4-xlate which highlighted
this recommendation to place the translator as
close to the IPv4 host as possible.  Or, similarly,
to ensure the IPv4 network has an MTU greater than
1280 (minus whatever the IPv6/IPv4 header differences
are).

-d

---

From: "Ramji Vaithianathan (rvaithia)" <[email-masked]>
Date: Mon, 8 Feb 2010 19:04:26 -0800

>From my understanding, this will be a problem only if the 
  1. MTU on some link on IPv4 side is less than 1280
  2. We have high speed traffic less than 1280 bytes but greater than
the above MTU value.

Generally most links on intermediate routers are 1500 or greater.  

So this may not be a common case, unless there is a link attached to an
end host is of smaller MTU (< 1280) and can take in high traffic as well
in the range MTU_value .. 1280. 

Regards,

Ramji
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Dan Wing (dwing)
Sent: Tuesday, February 09, 2010 5:54 AM
To: 'Templin, Fred L'; 'Iljitsch van Beijnum'
Cc: [email-masked]
Subject: Re: [BEHAVE] Fwd: IPv6 hosts sending <1280 byte packets

> The concern with setting DF=0 on the IPv4 side for packets that are 
> 1280 or smaller on the IPv6 side is the possibility for RFC4963 data 
> corruption if we let it run at line rate.

At high speed with the same source/destination addresses, and when
fragmentation on the IPv4 network occurs.  That is, sending lots of
packets with DF=0 does not, by itself, cause a problem even if they are
fragmented on the same link -- so long as they have different source or
different destination addresses.  I agree we should make a note of that.

Is there another approach that should be considered to avoid that
concern?

> There are certainly plenty of websites on the IPv4 Internet that set 
> DF=0 on every packet they send, and many even use packet sizes larger 
> than 1280. But, how many of them send large packets at line rate?

-d

> > > >> So essentially this means doing more work (keeping PMTUD state 
> > > >> for the IPv4 path) in order to do even more work later 

> > > At some point, an IPv6 host sends a packet that is larger than the

> > > IPv4 PMTU. A router sends back a too big message.
> > > Issue A is how to handle this too big message. Issue B is what 
> > > happens to packets that the IPv6 host subsequently sends.
> > 
> > A stateless translator cannot identify 'subsequent' packets, because

> > it doesn't remember previous state.
> > 
> > > translating the IPv4 too big too IPv6 and adjusting it as per the 
> > > header size differences, but otherwise let it through 
> > > destination state, not per session state, it would be doable to 
> > > track this) but because there may be more than one translator so 
> > > any given translator may not see all packets.
> > > because PMTUD was succesful and the packet size fits in the PMTU, 
> > > or the IPv6 host ignored the < 1280 too big message and is 
> > > omitting the fragment header in violation of the relevant RFCs. In

> > > order to be compatible with the latter the only thing we can do is

> > > set DF to 0.
> > >
> > > (I just realize that we don't know what happens when for instance 
> > > the PMTU is 1000 and the IPv6 host wants to send a 1100-byte 
> > > packet: does it include a fragment header or not?)
> > >
> > > Note that for part B, if there _is_ a fragment header, that will 
> > > also trigger the translator to set DF to 0. So in the case where 
> > > the IPv6 hosts are well behaved there is no change to either parts

> > > A or B.
> > 
> > So, I believe we are in agreement:  IPv6 packets less than 1280 need

> > to be translated to IPv4 and sent with DF=0.
> > 
> > > In the stateful case we know all packets flow through the same 
> > > translator so we get to track the IPv4 PMTU so when there is an 
> > > IPv6 packet that needs to be translated we know whether it's 
> > > bigger or smaller than the PMTU so we can send packets that are 
> > > bigger with DF=1 and packets that are smaller with DF=0 and if we 
> > > are to, also immediately fragment them. However, there is no 
> > > advantage to doing this extra work because even if we get to set 
> > > DF=1 on some packets that are smaller than the PMTU that only lets

> > > us discover a reduction in the PMTU, but that discovery doesn't 
> > > buy us anything because the IPv6 host isn't going to reduce its 
> > > packet size.
> > 
> > I agree the IPv6 host won't reduce its packet size below 1280.  But 
> > the IPv6 host becomes aware that the path MTU is smaller than 1280.

> > I don't know if the IPv6 host finds that useful/valuable to know the

> > path MTU is less than the
> > IPv6 minimum MTU.
> > 
> > > anything and it may even get the packet blocked or create more 
> > > work if there's a firewall with a > 1280 MTU before the path hits 
> > > the < 1280 MTU which would need to reassemble the packet to 
> > > observe the port numbers. And it could be an attack
> > > vector: attackers could send too bigs with tiny MTUs to make the 
> > > translator work harder.
> > >
> > > > Certainly there was some
> > Don't know.  It is obviously easier for the 6/4 translator to simply

> > fragment the packet (or translate and send with DF=0) without 
> > ICMPv6 PTB and the IPv6 host is then supposed to send packets with 
> > the fragmentation header.
> > 
> > > packet sizes. So the protocol would have to support changing the 
> > > packet size on the fly and then the application would have to 
> > > react to too big messages.
> > 
> > draft-petithuguenin-behave-stun-pmtud does not rely on ICMP packet 
> > too big messages.
> > 
> > Me, neither.  I'm just trying to temper the "only TCP does PMTUD", 
> > because UDP can do PMTUD.  And of course there is value in sending 
> > the largest packets possible.
> > 
> > > But if BitTorrent goes to UDP it's likely they'll put this in in 
> > > some way.
> > 
> > As of version 2.x, the most popular BitTorrent client on Windows 
> > (uTorrent) has implemented uTP, for whatever that's worth.  I don't 
> > know if they're doing PMTUD, though.  Beta versions of uTorrent are 
> > available for OSX, too, and I expect they will soon support uTP (if 
> > they aren't already).
> > 
> > > > I tend to think, though, we don't want to figure this out for 
> > > > stateful translators at this point in time.  If it is found 
> > > > useful/necessary, it can be specified later.
> > >
> > > Right.=
> > 
> > -d
> >

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 08:28:55 +0100

> for RFC4963 data corruption if we let it run at line rate.

But how can this be avoided by the translator?

In theory if the IPv6 host sees a < 1280 too big it could deduce that its packets are going to be translated to IPv4 with DF=0 and then slow down to avoid wrapping the ID during a packet lifetime, but I don't see that happening...

If an IPv4 network is stupid enough to use such a small MTU then incorrect reassemly of fragments is a risk they are apparently willing to run.

But again, note that few fragments will happen because the TCP MSS will usually make the IPv6 host send packets that don't require fragmentation. The only two applications that I know of that send high rates of non-TCP packets are RTSP video streaming, which I assume won't be done much from an IPv6 client towards and IPv4 server, and tunneling, which I assume will also be rare across a NAT64. Maybe in the future there will also be UDP BitTorrent, but BitTorrent usually doesn't send too much traffic to any given destination and it uses application layer checksums.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 07:51:27 -0800

Ramji,

> From: Ramji Vaithianathan (rvaithia) [mailto:[email-masked]]
> Sent: Monday, February 08, 2010 7:04 PM
> To: Dan Wing (dwing); Templin, Fred L; Iljitsch van Beijnum
> Cc: [email-masked]

I think it is reasonable to assume that the vast
majority of intermediate routers configure an MTU
of 1500 or greater. Where there might be a concern
for links with smaller MTUs is within end sites.

Right.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 09:51:38 -0800

Dan,

About the placement of the translator, in the IPv6->IPv4
direction the packet incurs a size reduction due to the
substitution of an IPv4 header in place of the IPv6 header.
In the IPv4->IPv6 direction, however, the size of the packet
is inflated. So, e.g., if the translator receives a 1500byte
IPv4 packet and has to translate it into a 1520byte IPv6
packet, there is a chance that it may be dropped due to a
link MTU restriction somewhere in the IPv6 network and an
ICMPv6 PTB sent back toward the translator. Presumably, the
translator would translate that into an ICMPv4 PTB message
that it forwards on to the IPv4 host.

If I have that right, there may be a concern for whether
the ICMPv4 PTB would be dropped in the IPv4 network before
it arrives at the IPv4 host. I ran some tests and found
that path MTU discovery in IPv4 seems to be problematic
for a substantial number of the top 1000 websites:

http://www.ietf.org/mail-archive/web/rrg/current/msg05907.html

This suggests to me that path MTU discovery in the IPv4
network "works" simply because it is rarely invoked. In
other words, the IPv4 network seems to be getting by
based on the good fortune that most links in the Internet
core configure an MTU of 1500 or larger. But, this state
of affairs may be disrupted if more and more translators
are used that cause IPv4 packets to be inflated in size
after they are translated into IPv6 packets.

So, if the translator were located at the site border of
the IPv4 host (i.e., rather than at the site border of
the IPv6 host) the bulk of the path would be over IPv6
instead of IPv4, hence the incidence of ICMPv4 filtering
would be isolated to the end site of the IPv4 host.

Has the translator deployment analysis taken IPv4 path
MTU-related black holing into consideration?

Fred

> From: Dan Wing [mailto:[email-masked]]
> Sent: Monday, February 08, 2010 5:16 PM

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 19:01:34 +0100

On 9 feb 2010, at 18:51, Templin, Fred L wrote:

> So, e.g., if the translator receives a 1500byte

If the IPv6 host is connected through a link with a 1500-byte MTU, it will advertise a maximum segment size (MSS option) in its TCP SYN packet of 1500 - 40 (IPv6 header) - 20 (TCP header) = 1440. So the IPv4 host will send 1480-byte IPv4 packets which are then translated into 1500-byte IPv6 packets and there are no issues.

Only when the IPv6 host has an MTU larger than 1500 but the translator doesnt the IPv4 PTMUD black holes will be triggered.

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 9 Feb 2010 10:09:24 -0800

> Sent: Tuesday, February 09, 2010 9:52 AM
> network "works" simply because it is rarely invoked. 

Agreed.

> In other words, the IPv4 network seems to be getting by

Agreed.

Which is why tweaking the TCP MSS is a helpful workaround
to blocked ICMP PTB.  But tweaking TCP MSS restricts us to 
the lowest common denominator (that is, limiting TCP MSS 
means we can't expect to use larger-than-1500 frames).

I know Lars Eggert had concerns with a translator tweaking
TCP MSS in a translator, and there is only a very brief 
mention in draft-ietf-behave-v6v4-xlate.

> would be isolated to the end site of the IPv4 host.

That places the onus on the content provider to make their
content available to IPv6.  And there are millions of 
content providers from the local school district's website
to magazine sites, online stores, and so on.  

They can't add IPv6 support in time:  there simply isn't
enough time left.

> MTU-related black holing into consideration?

No.

-d

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 19:18:08 +0100

On 9 feb 2010, at 19:09, Dan Wing wrote:

> to blocked ICMP PTB.

There very little need to do anything to the MSS option, see my previous message.

Only in the case where the IPv6 host supports an MTU larger than 1500 but the translator doesn't, there is a possible optimization by having the translator rewrite the MSS to the maximum that the translator supports so the translator wouldn't have to send too bigs and there is less risk of PMTUD black holes. However, it's not clear to me that this case is worth the trouble.

> But tweaking TCP MSS restricts us to 

Obviously you would be advertising the minimum of what the IPv6 host and the translator can handle and not simply hardcode 1500. If both do more than 1500 bytes then you advertise more than 1500 bytes and you get to use more than 1500 bytes. Note that this won't trigger any black holes because the IPv4 hosts will send 1500-byte packets. (The intersection between users with broken PMTUD setups and users with an MTU larger than 1500 is rather small.)

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 10:34:45 -0800

Iljitsch,

You may be missing the "dumbbell configutation" where
both the IPv6 host and translator are on 9KB links but
there are 1500 links in between?

Fred

> Sent: Tuesday, February 09, 2010 10:18 AM
> To: Dan Wing
> Cc: Templin, Fred L; [email-masked]
> Only in the case where the IPv6 host supports an MTU larger than 1500 but the translator doesn't,
> there is a possible optimization by having the translator rewrite the MSS to the maximum that the
> translator supports so the translator wouldn't have to send too bigs and there is less risk of PMTUD
> black holes. However, it's not clear to me that this case is worth the trouble.
> 
> Obviously you would be advertising the minimum of what the IPv6 host and the translator can handle
> and not simply hardcode 1500. If both do more than 1500 bytes then you advertise more than 1500 bytes
> and you get to use more than 1500 bytes. Note that this won't trigger any black holes because the
> IPv4 hosts will send 1500-byte packets. (The intersection between users with broken PMTUD setups and
> users with an MTU larger than 1500 is rather small.)

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 9 Feb 2010 10:50:32 -0800

> Cc: 'Templin, Fred L'; [email-masked]
> Only in the case where the IPv6 host supports an MTU larger 
> than 1500 but the translator doesn't, there is a possible 
> optimization by having the translator rewrite the MSS to the 
> maximum that the translator supports so the translator 
> wouldn't have to send too bigs and there is less risk of 
> PMTUD black holes. However, it's not clear to me that this 
> case is worth the trouble.

There are long-standing desires to MTUs larger than 1500
with IPv6, under the theory that IPv6 Did Things Correctly.

-d

> Obviously you would be advertising the minimum of what the 
> IPv6 host and the translator can handle and not simply 
> hardcode 1500. If both do more than 1500 bytes then you 
> advertise more than 1500 bytes and you get to use more than 
> 1500 bytes. Note that this won't trigger any black holes 
> because the IPv4 hosts will send 1500-byte packets. (The 
> intersection between users with broken PMTUD setups and users 
> with an MTU larger than 1500 is rather small.)=

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 19:50:03 +0100

> there are 1500 links in between?

Yes, that could happen. And it would suck.  :-)

For the IPv6-to-IPv4 direction there is no problem, because the IPv6 host simply runs PMTUD. For the IPv4-to-IPv6 direction with DF=1 any black holes on the IPv4 side will be triggered and with DF=0 I believe the current text specifies always fragmenting to 1280, which I very much disagree with and which we haven't had any discussion about.

But you really don't want to run a translator with jumboframes if you can't reach at least _some_ translator clients with large packets.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 20:05:39 +0100

> with IPv6, under the theory that IPv6 Did Things Correctly.

You can send large packets with IPv4, too. Having a bigger than 1500 byte MTU causes much fewer problems than having a smaller than 1500 byte MTU for a number of reasons which are explained in http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-02

However, if you have a 1500+ MTU on the IPv6 side (but the translator has 1500), the MSS doesn't fix the PMTUD black holes that are caused by the fact that that the IPv6 host has a 1480 path MTU seen from the IPv4 side while it advertising a larger one in the MSS.

If we want to avoid this particular black hole the solution is to rewrite the MSS in the translator to the minimum of the IPv6 host's MSS and the translator MTU on the IPv6 side + sizeof(IPv4) - sizeof(IPv6).

I don't remember the discussion about MSS rewriting in the softwires context, just that my conclusion was that that issue doesn't apply here. In any event, MSS clamping has been a staple of cheap NAT/PPPoE boxes and I don't believe it causes any problems.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 11:10:19 -0800

Iljitsch,

Another concern I should have called out earlier is what
if the IPv6 host is on a 1500 link and opens a TCP
connection to the IPv4 server with MSS=1440. The IPv4
server sends a 1480b IPv4 packet that gets translated
into a 1500b IPv6 packet. The packet makes its way
through the IPv6 network toward the IPv6 host, but
first it hits a 1480 MTU link (e.g., and IPv6 over IPv4
tunnel). The ICMPv6 PTB gets triggered, then translated
into ICMPv4 fragmentation needed, then dropped by
a filtering gateway in the IPv4 Internet.

Black hole, right?

Fred

> Sent: Tuesday, February 09, 2010 10:50 AM
> To: Templin, Fred L
> Cc: Dan Wing; [email-masked]
> For the IPv6-to-IPv4 direction there is no problem, because the IPv6 host simply runs PMTUD. For the
> IPv4-to-IPv6 direction with DF=1 any black holes on the IPv4 side will be triggered and with DF=0 I
> believe the current text specifies always fragmenting to 1280, which I very much disagree with and
> which we haven't had any discussion about.
>

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 9 Feb 2010 12:14:58 -0800

> Sent: Tuesday, February 09, 2010 11:06 AM
> To: Dan Wing
> You can send large packets with IPv4, too.

Yes, but experience on the IPv4 Internet shows that ICMP packet-too-big are
unreliable.  So many have given up on IPv4.  Closed environments, including
where tunnels are operating over equipment you own/control, are different.
But that is why I said 'desires' and put 
'Did Things Correctly' in quotes.  My goal, as chair, is to make sure
translators do not make deployment of large-MTU networks worse.  If they do,
it will complicate acceptance by IESG.

-d

> Having a bigger 
> than 1500 byte MTU causes much fewer problems than having a 
> smaller than 1500 byte MTU for a number of reasons which are 
> explained in http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-02
> 
> However, if you have a 1500+ MTU on the IPv6 side (but the 
> translator has 1500), the MSS doesn't fix the PMTUD black 
> holes that are caused by the fact that that the IPv6 host has 
> a 1480 path MTU seen from the IPv4 side while it advertising 
> a larger one in the MSS.
> 
> If we want to avoid this particular black hole the solution 
> is to rewrite the MSS in the translator to the minimum of the 
> IPv6 host's MSS and the translator MTU on the IPv6 side + 
> sizeof(IPv4) - sizeof(IPv6).
> 
> I don't remember the discussion about MSS rewriting in the 
> softwires context, just that my conclusion was that that 
> issue doesn't apply here. In any event, MSS clamping has been 
> a staple of cheap NAT/PPPoE boxes and I don't believe it 
> causes any problems.=

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 21:25:44 +0100

> Black hole, right?

Indeed. Note however that the same black hole would exist if both ends are IPv6 or both ends are IPv4 and there is no translator. I don't think we can reasonably expect NAT64s to fix all possible black holes.

In practice, I think NAT64 users would have to avoid reduced MTUs between the translator and the IPv6 hosts (and I would assume that a deployment that needs NAT64 is far enough along that it doesn't need IPv6-in-IP tunneling) or artificially lower the IPv6 host MTU or the translator MTU / rewrite the MSS if a reduced MTU is present between the translator and the IPv6 hosts.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 21:40:54 +0100

> unreliable.  So many have given up on IPv4.

I don't know about the latter, but please don't think that issues you see with too bigs in one direction translate to the same issues with too bigs in the other direction.

The reason PMTUD black holes are common is because people filter them out in their firewalls or their hosts because they have some irrational fear of ICMP messages and/or they don't think they need to see those messages. If you have a reduced MTU, you depend on the side with the regular 1500-byte MTU handling PMTUD properly. If they don't, you have a problem that _you_ can't fix. If you use jumboframes and communicate with someone with a smaller MTU, you have to run PMTUD properly _yourself_ to avoid problems. So you would probably not want to filter too big messages. Or if somehow you can't get the PMTUD to work, you can always reduce your MTU to avoid the problem. People with small MTUs generally can't increase those.

So the only issue that I see is the case where too bigs aren't properly generated in the first place. I'm sure there will be a few networks where this is the case, but these are outgoing ICMP messages and as a rule network operators don't filter ICMP, at least not as often as server operators. You also only have a jumbo-clean path towards a limited number of networks, so you don't need 100% of the server operators to handle PMTUD correctly but only a few percent of the network operators. So you will very likely not see any problems. If problems are rare they're also more likely to be fixed when they occur.

Maybe we should ask people on the MTU list about their experiences.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Feb 2010 12:48:03 -0800

Hi Iljitsch,

> Sent: Tuesday, February 09, 2010 12:26 PM
> To: Templin, Fred L
> Indeed. Note however that the same black hole would exist if both ends are IPv6 or both ends are IPv4
> and there is no translator.

Eh? For IPv6<->IPv6, we have been assured that PMTUD
is fixed. Is that not the case?

Also, for IPv4-to-IPv4 we have the convenient fact that
links with MTU<1500 are rare in the Internet core so
tripping over a low MTU link is a rare event. Granted
that does not get us on the path to 9KB MTUs, but at
least it works in some fashion through good fortune.

> I don't think we can reasonably expect NAT64s to fix all possible black
> holes.

It's not a matter of fixing all black holes; it is
rather an analysis of what new sorts of black holes
are made possible when translators are used.

> In practice, I think NAT64 users would have to avoid reduced MTUs between the translator and the IPv6
> hosts (and I would assume that a deployment that needs NAT64 is far enough along that it doesn't need
> IPv6-in-IP tunneling)

AFAICT, it may still be some time before ISP networks can
enable ubiquitous native IPv6. For those that can't there
is IPv6-in-IPv4 tunneling for the near term. Does that
mean that those ISPs can't deploy translators?

> or artificially lower the IPv6 host MTU or the translator MTU / rewrite the MSS
> if a reduced MTU is present between the translator and the IPv6 hosts.

Artificially lowering the IPv6 host MTU may be unsatisfactory
if the host is connected to a network where it could otherwise
be using larger packet sizes. And, MSS clamping only works
for TCP.

Fred

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 09 Feb 2010 14:21:54 -0800

> after they are translated into IPv6 packets.

well note that as (rfc 2923) observes and some platforms implement pmtu
black -hole avoidance which involves dropping back to 576 (rfc 791).
Appart from being kind of suboptimal that will work in places where
pmutd won't such as when you have a 9k mtu on your nic, but need to talk
to a printer on the same subnet.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 9 Feb 2010 23:55:35 +0100

On 9 feb 2010, at 21:48, Templin, Fred L wrote:

>> Indeed. Note however that the same black hole would exist if both ends are IPv6 or both ends are IPv4 and there is no translator.

> is fixed. Is that not the case?

Fixed how? I know Microsoft has PMTUD blackhole detection and maybe others as well, but that has nothing to do with IPv6. I've had my share of black holes with IPv6, although not recently.

> tripping over a low MTU link is a rare event.

PPP over ethernet is pretty common with ADSL and reduces a user's MTU below 1500, usually without the hosts knowing it. This is solved with adjusting (clamping) the value in the MSS option.

> Granted that does not get us on the path to 9KB MTUs,

Like I said before, you can do this without trouble. I didn't even mention the MSS, which will save you if there's a network in the middle that doesn't generate too bigs correctly.

> are made possible when translators are used.

Right. But your example wasn't very particular to the translation case.

> is IPv6-in-IPv4 tunneling for the near term.

What kind of tunneling do you foresee? Obviously not 6to4. Other types of tunneling are more complex, native IPv6 makes life much simpler.

> Does that mean that those ISPs can't deploy translators?

> be using larger packet sizes. And, MSS clamping only works

If you have a better solution I'm all ears.

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 09 Feb 2010 15:31:12 -0800

Dan Wing wrote:
>> Only in the case where the IPv6 host supports an MTU larger than
>> 1500 but the translator doesn't, there is a possible optimization
>> by having the translator rewrite the MSS to the maximum that the
>> translator supports so the translator wouldn't have to send too
>> bigs and there is less risk of PMTUD black holes. However, it's not
>> clear to me that this case is worth the trouble.
> 
> There are long-standing desires to MTUs larger than 1500 with IPv6,
> under the theory that IPv6 Did Things Correctly.

There are longing standing desires for the same under ipv4... ;)

it even works some of the time, and sometimes pmtud works, sometimes
it doesn't.

my first experience with end-to-end 9k mtu's across the internet was
over a decade ago. Things have gotten a little better since then I
guess, or not.