Subject: [BEHAVE] NAT64 fragmentation

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 13 Apr 2009 17:12:16 +0200

Hi all,

here is text about fragmentation that I've been working on for the  
NAT64 draft.

Comments are appreciated, especially about the choice for DF=0 <= 1280  
and DF=1 > 1280 byte packet sizes for IPv6->IPv4.

Iljitsch

Packet sizes

It's the job of the network layer to adapt to different maximum packet  
sizes as packets move through the network. There are three mechanisms  
that handle this: transport layer negotiations such as the TCP MSS  
option, path MTU discovery and fragmentation. The difference between  
the IPv4 and IPv6 header sizes requires some handling in a NAT64  
translator, and there are complications because of the differences  
between how IPv4 and IPv6 handle fragmentation, as well as the issue  
of how to demultiplex fragmented IPv4 packets.

There are two approaches to path MTU discovery and fragmentation when  
translating from IPv6 to IPv4:

1. Set DF to 0 in the translated packets. This avoids path MTU  
discovery issues but leads to significant numbers of fragments.

2. Set DF to 1 in the translated packets. This supports path MTU  
discovery on the IPv4 side so unnecessary fragments are avoided, but  
it doesn't address the issue that IPv6 hosts are not required to  
perform PMTUD when sending packets of 1280 bytes or smaller.

The choice made in this document is to support option 1 for packets  
upto 1280 bytes, and option 2 for packets larger than 1280 bytes.

A NAT64 translator MUST have an MTU of at least 1280 on all of its  
interfaces, both IPv4 and IPv6 interfaces.

TCP MSS option

The TCP MSS option [RFC 793] is used during the three-way handshake by  
the two hosts involved to inform each other about the maximum TCP  
segment size (assuming IP and TCP headers without options) that the  
host can receive.

In practice, the MSS option is often used to make TCP work in the  
presence of broken path MTU discovery.

To avoid unnecessary path MTU discovery cycles, a NAT64 SHOULD rewrite  
the MSS option in SYN packets to the minimum of the original MSS  
option, the NAT64's MTU on the IPv6 side - 60 and the NAT64's MTU on  
the IPv4 side - 40. This applies to SYNs in both the IPv4-to-IPv6  
direction and the IPv6-to-IPv4 direction.

Path MTU discovery

The vast majority of both IPv4 and IPv6 hosts use path MTU discovery  
[RFC 1191] [RFC 1981]. With IPv4, PMTUD can be enabled on a per-packet  
basis by setting the DF bit to 1. With IPv6, there is no need for  
PMTUD for packets up to 1280 bytes because all IPv6 hosts are required  
to be able to receive 1280-byte packets without fragmentation. When  
sending larger packets, IPv6 hosts implicitly use PMTUD.

IPv6-to-IPv4

If the NAT64 has the same MTUs on its IPv6 and IPv4 interfaces, it  
will never have to generate "packet too big" messages for incoming  
IPv6 packets because the translation from IPv6 to IPv4 reduces the  
packet size by 20 bytes, more if the IPv6 packet has extension headers  
that are removed during the translation, such as the fragment header.  
If the MTU on the IPv6 side is larger than 1280 bytes and more than 20  
bytes smaller than the MTU on the IPv4 side, the NAT64 MUST generate  
the appropriate "packet too big" messages on the IPv6 side.

To support PMTUD, for translated packets that are larger than 1260  
bytes on the IPv4 side (1280 bytes IPv6 packets with 20 byte size  
reduction through the translation), the DF bit is set to 1 in the  
resulting IPv4 packet.

IPv4 routers may generate "packet too big" messages indicating a  
supported MTU size smaller than 1280 bytes. In those cases, the IPv6  
hosts will continue to send packets larger than what the IPv4 path MTU  
can support. To allow packets to be delivered successfully in this  
case, the DF bit is set to 0 in all translated packets smaller than or  
equal to 1260 bytes, to allow these packets to be fragmented in the  
IPv4 network.

Note: it is highly recommended for IPv4 hosts running services that  
may be used by IPv6 clients through a NAT64 translator to use an MTU  
size of at least 1260 bytes and to properly generate "packet too big"  
messages.

When a NAT64 translates "packet too big" messages from IPv6 to IPv4,  
it adjusts the advertised MTU to the minimum of the original  
advertised MTU + 20, the NAT64's MTU on the IPv6 side + 20 and the  
NAT64's MTU on the IPv4 side.

IPv4-to-IPv6

Because it may be necessary to include a fragmentation header or other  
extension header, the NAT64 MUST be prepared to generate "packet too  
big" messages for packets with the DF bit set to 1 received from the  
IPv4 side, regardless of the MTU sizes on the IPv4 and IPv6  
interfaces. If the packet is larger than can be transmitted on the  
IPv6 side after translation, the NAT64 returns a "packet too big"  
message indicating the maximum IPv4 packet size that would be  
supported using the same translation as the current packet. This can  
be calculated as IPv4-packet-size - (IPv6-packet-size - IPv6-total- 
length) + 20.

When a NAT64 translates "packet too big" messages from IPv4 to IPv6,  
it adjusts the advertised MTU to the minimum of the original  
advertised MTU - 20, the NAT64's MTU on the IPv6 side and the NAT64's  
MTU on the IPv4 side - 20. However, if the advertised MTU in "packet  
too big" messages is smaller than 1260 bytes, the value put into the  
translated "packet too big" message is 1280. This makes sure that the  
IPv6 host will limit its packet sizes to 1280 bytes, so its packets  
are subsequently translated into IPv4 packets with DF set to 0. (This  
deviates from [RFC 2765].)

Fragmentation

Because NAT deviates from normal router behavior, the limitation that  
IPv6 packets or IPv4 packets with DF set to 1 are not fragmented by  
routers doesn't apply to a NAT64 translator. Where appropriate, these  
packets are fragmented after translation as described below.

Demultiplexing

Because NAT64 provides a stateful many-to-one (perhaps even many-to- 
many) translation, it is necessary to recognize which session a given  
packet belongs to. For this, the TCP or UDP port numbers must be  
known, but these only occur in the first fragment of a fragmented  
packet. There are two possible ways to deal with this:

1. Reassemble the packet before translating it.

2. Create translation state for the fragments belonging to the same  
packet so each packet can be translated.

Strategy 2 is attractive in large installations because it requires  
less storage and processing. However, it may still be necessary to  
buffer fragments for some time, as the fragment containing the first  
part of the packet (and with that, the port numbers) may not be the  
first one to arrive.

Note: based on the assumptions that hosts generate fragments in-order  
and that reordering must happen through parallel network links and  
that the path between these parallel links and a NAT64 supports speeds  
of at least 10 Mbps, there is a very high probability that two out-of- 
order fragments making up a packet will arrive at the NAT64 within 50  
to 100 milliseconds. Further assuming that fragmented traffic makes up  
less than 10% of all traffic, this only requires a buffer of 6 to  
12,500 fragments (50 ms at 10 Mbps to 100 ms at 10 Gbps).

In some cases, especially in the IPv6-to-IPv4 direction, there may  
only be a single session matching the fragment's source and  
destination addresses and protocol number. In these cases, it would be  
possible to translate the fragments out-of-order. A NAT64 translator  
MAY do this for TCP, however, it MUST NOT translate UDP packets before  
the first fragment is available. The reason for this is that the  
fragment could be part of a packet setting up a new session. However,  
with TCP session establishment packets don't carry data, so it's  
extremely unlikely that they are fragmented. This is not the case with  
UDP, and in the IPv4-to-IPv6 direction, a UDP packet may have a zero  
checksum, which must be recalculated when translating to IPv6, for  
which the entire packet must be available.

IPv6-to-IPv4

For all IPv4 packets that the NAT64 creates through translation, the  
translator generates an ID value. This applies to all packets,  
regardless of their size or the value of the DF field. A NAT64  
translator MAY employ strategies to avoid reusing an ID value for a  
certain source, destination, protocol tuple as long as possible. If  
the IPv4 packets are fragments of an IPv6 packet, then state is  
created that makes it possible for all the fragments to have the same  
ID value on the IPv4 side.

[RFC 2765] specifies copying the lower bits from the IPv6 ID field in  
a fragment header (if present) to the IPv4 ID field, but this runs the  
risk of two IPv6 hosts talking to the same IPv4 destination through  
the NAT64 using the same ID value.

Otherwise, when translating IPv6 packets with a fragmentation header,  
the fragments are translated as per [RFC 2765].

IPv4-to-IPv6

Because packets coming in on the IPv4 side may be larger than 1280  
bytes after translation, a NAT64 MUST implement PMTUD on the IPv6  
side. In other words, it must react to "packet too big" messages for  
any IPv6 destination that it communicates with by limiting the size of  
the packets that it sends to the advertised maximum.

In the case where, after translation from IPv4 to IPv6, a packet is  
larger than a destination's PMTU, the NAT64 returns a "packet too big"  
as outlined earlier in the case that the DF bit was set to 1 in the  
IPv4 packet. If the DF bit was set to 0, the translator first  
translates the IPv4 packet, and then fragments the resulting IPv6  
packets using normal IPv6 fragmentation rules. The value in the ID  
field is generated locally by the NAT64. If the IPv4 packet was a  
fragment, state is created that allows the same ID value to be used  
for all IPv6 packets or fragments that are part of the same original  
IPv4 packet.

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 13 Apr 2009 16:33:12 -0700

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
> Sent: Monday, April 13, 2009 8:12 AM
> To: Behave WG
> Subject: [BEHAVE] NAT64 fragmentation
> 
> Hi all,
> 

(Not as chair.)

Lots of your text seems redundant with draft-baker-behave-v4v6-translation-02,
for example Section 2 which includes:

   When the IPv4 node performs path MTU discovery (by setting the DF bit
   in the header) the path MTU discovery can operate end-to-end i.e.
   across the translator.  In this case either IPv4 or IPv6 routers
   might send back ICMP "packet too big" messages to the sender.  When
   these ICMP errors are sent by the IPv6 routers they will pass through
   a translator which will translate the ICMP error to a form that the
   IPv4 sender can understand.  In this case an IPv6 fragment header is
   only included if the IPv4 packet is already fragmented.

   However, when the IPv4 sender does not perform path MTU discovery the
   translator has to ensure that the packet does not exceed the path MTU
   on the IPv6 side.  This is done by fragmenting the IPv4 packet so
   that it fits in 1280 byte IPv6 packet since IPv6 guarantees that 1280
   byte packets never need to be fragmented.  Also, when the IPv4 sender
   does not perform path MTU discovery the translator MUST always
   include an IPv6 fragment header to indicate that the sender allows
   fragmentation.  That is needed should the packet pass through an IP/
   ICMP translator.

but there are other places in draft-baker-behave-v4v6-translation-02 where
fragmentation is discussed (e.g., section 3).  I expect the stateful
translation document only needs to discuss unique aspects about fragmenting
that relate to stateful translation.

A few comments below.

> direction and the IPv6-to-IPv4 direction.

I noticed that draft-baker-behave-v4v6-translation-02 doesn't mention anything
about TCP MSS.  I wonder if it should or if we want expect only a stateful
translator to benefit from adjusting TCP MSS?  It does not require state to
rewrite TCP MSS.

> equal to 1260 bytes, to allow these packets to be fragmented in the  

That paragraph needs to be turned into a requirement for the 
implementation.  I believe something like this would be what you
need,

  To allow packets to be delivered successfull in this case,
  the NAT64 notices the 'packet too big' ICMP error has a 
  'Next-Hop MTU' field less than 1280 (including a Next-Hop MTU 
  of zero) and then, for the duration of that flow, it MUST 
  forward packets on the IPv4 network with DF=0.

This seems a feature exclusive to a stateful translator.

> size of at least 1260 bytes and to properly generate "packet 
> too big" messages.
> 
> packet belongs to.

The many-to-one translation appears also true of the stateless 
mode, per 
http://tools.ietf.org/html/draft-xli-behave-ivi-01#section-5.1
which states, in part:

   To further increase the utilization ratio of the public IPv4
   addresses, the port multiplexing inside the ISP(i)'s /32 can be
   deployed [RFC2766] [RFC4966].  This is to say that a single IPv4
   address (IVI4(i)) can be used for multiple IVI6(i) addresses.  The
   mapping scheme is to use the least significant bits in the IVI6(i) to
   define the multiple mapping and combine the transport-layer port
   number to perform uniquely the mapping from IVI4(i) to IVI6(i).

Which, if it is still being pursued for standardization would be
text that should be in the stateless document.

However, I don't see a similar description in Fred's IVI document
(draft-baker-behave-ivi-01) nor in the translation 
(draft-baker-behave-v4v6-translation-02) or framework
(draft-baker-behave-v4v6-framework-02) documents -- so maybe the
idea has been abandoned for standardization.  I'm not sure.  

> For this, the TCP or UDP port numbers must be  

The text above should be rolled into the stateless translation
draft.

> for all IPv6 packets or fragments that are part of the same original  

-d

---

From: Lars Eggert <[email-masked]>
Date: Tue, 14 Apr 2009 17:16:14 +0300

Hi,

please run your proposal on TCP MSS clamping by the TCPM WG. I'm not  
aware of any existing RFC that describes/recommends this technique  
(although it's obviously implemented in some products), and we need to  
carefully discuss what we want to say here.

Lars

On 2009-4-13, at 18:12, Iljitsch van Beijnum wrote:

> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

---

From: "Durand, Alain" <[email-masked]>
Date: Tue, 14 Apr 2009 10:29:20 -0400

The question of fragmentation/MTU adaptation & TCP MSS came also in Softwire
for Dual-stack lite.

See:
http://www.ietf.org/mail-archive/web/softwires/current/msg00793.html
As a response to:
http://www.ietf.org/mail-archive/web/softwires/current/msg00790.html

  - Alain.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 14 Apr 2009 20:50:50 +0200

[Added even more CCs, please prune where appropriate]

> http://www.ietf.org/mail-archive/web/softwires/current/msg00790.html

I don't think TCP authentication issues apply because the translation  
breaks those anyway.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 15 Apr 2009 09:55:16 -0700

Iljitsch,

> From: Iljitsch van Beijnum [mailto:[email-masked]]

Yikes; that's a lot of text - any way to compress?
It would also be easier to insert comments if the
text were in (pseudo) Internet draft form.
 
> Comments are appreciated, especially about the choice for DF=0 <= 1280

This seems to deviate from RFC2460 and RFC2765 behavior
wrt IPv6 path MTU underflow, and if so you must justify
exactly why. In RFC2460, hosts may receive a PTB with an
MTU smaller than 1280 from a v6/v4 translator, and are
thereafter required to insert a fragment header so that
translator can have a well-behaved ID for generating
IPv4 fragments. Why does this approach deviate?

> and DF=1 > 1280 byte packet sizes for IPv6->IPv4.

Enabling IPv4 path MTU discovery opens the system to
MTU-related black holing for the portion of the path
that is conveyed as IPv4. It is therefore desirable to
locate the NAT64 as close as possible to the IPv4 end
system such that the surface for ICMP spoofing and
black holing is reduced. (AFAICT, that includes
performing the translation on the end system itself.)

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 15 Apr 2009 19:53:59 +0200

Hi Fred,

Thanks for the comments.

> Yikes; that's a lot of text - any way to compress?

Sure, gzip or bzip2?  :-)

> text were in (pseudo) Internet draft form.

It was supposed to be, just need to drop in the XML, but I want the  
text to be close to final before I start with that.

> wrt IPv6 path MTU underflow,

Not sure what you're getting at with regard to RFC 2460 (the IPv6  
spec, for IPv4-only BEHAVErs), but you're right about RFC 2765. This  
specifies:

(PTB = packet too big)

> IPv4 fragments. Why does this approach deviate?

This seemed unnecessarily complex to me, just setting DF=0 on <= 1280  
byte packets and DF=1 on larger packets is simpler. So I asked Erik  
Nordmark, author of RFC 2765, about this and he explained that SIIT  
does this because it's stateless, so different fragments may flow  
through different translators, so it's important that the source host  
puts the ID value in packets. I think there is a slight issue here,  
but that's not important here, as in the _stateful_ translation we  
require that all packets flow through the same translator in any event.

Additionally, there is actually a risk if we use the ID generated by  
the IPv6 hosts: suppose that two hosts both talk to google.com and  
they both choose ID 0xdead. Suppose the normal large first fragment /  
small second and last fragment case. Google could then easily  
reassemble a packet from fragment 1 from host A and fragment 2 from  
host B. This case becomes less of a corner case when there is a loss:  
if fragment 2 from host A is lost, and host B sends (for instance) 130  
packets during the time between when host A sends fragment 1 and it  
retransmits fragment 2, there is a one in 500 chance that the ID  
fields will clash.

That's why I want the translator to generate new ID values for all  
IPv4 packets.

> that is conveyed as IPv4.

Is it a goal to make PMTUD for NAT64 better than for untranslated  
IPv4? If there's a black hole, you're going to have trouble regardless  
of the translation.

We can of course set DF=0 for all packets but then, pretty much nobody  
does that for IPv4 today, even people who filter incoming ICMP, like  
microsoft.com.

> It is therefore desirable to

Unfortunately, the translation that we're talking about here happens  
fairly close to the IPv6 hosts = far from the IPv4 host.

---

From: Philip Matthews <[email-masked]>
Date: Wed, 15 Apr 2009 09:10:05 -0400

On Mon, 13-Apr-09, at 11:12 , Iljitsch van Beijnum wrote:

> Comments are appreciated, especially about the choice for DF=0 <=  
> 1280 and DF=1 > 1280 byte packet sizes for IPv6->IPv4.

Iljitsch:

Under your proposal, what would happen if a NAT64 receives an IPv6  
packet with the Fragmentation Header?

- Philip

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 15 Apr 2009 13:57:03 -0700

> > Lots of your text seems redundant with draft-baker-behave-v4v6- 
> > translation-02
> 
> 
> Let me propose that we begin by figuring out how we want to handle  
> fragmentation. We can then work out how to divide the text amongst  
> documents.

Sure.  But they don't align.  I don't know if that is intentional
or accidental.  If intentional it would be helpful to call it 
out rather than relying on everyone to analyze the differences --
that is wasted effort.

-d

---

From: Philip Matthews <[email-masked]>
Date: Wed, 15 Apr 2009 09:13:49 -0400

- Philip

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 15 Apr 2009 23:24:46 +0200

This is what I wrote:

> [RFC 2765] specifies copying the lower bits from the IPv6 ID field  
> in a fragment header (if present) to the IPv4 ID field, but this  
> runs the risk of two IPv6 hosts talking to the same IPv4 destination  
> through the NAT64 using the same ID value.

> Otherwise, when translating IPv6 packets with a fragmentation  
> header, the fragments are translated as per [RFC 2765].

And before that:

> destination addresses and protocol number. In these cases, it would  
> be possible to translate the fragments out-of-order. A NAT64  
> translator MAY do this for TCP, however, it MUST NOT translate UDP  
> packets before the first fragment is available.

I made a mistake here: in the IPv6-to-IPv4 direction, there are no  
demultiplexing or UDP checksum issues so fragments can be translated  
individually, as long as we make sure that there are no issues with  
the ID field.

A reasonably robust but annoyingly complex way to handle the IDs:

For each IPv4 destination address, all unfragmented packets and each  
IPv6 source of fragmented packets needs ID space. Obtain this space in  
blocks, for instance of 256. For unfragmented packets, simply cycle  
through the block. For fragmented packets, copy the low order bits of  
the IPv6 ID in the low order bits of the IPv4 ID and the block number  
in the high order bits of the IPv4 ID. Count first fragments so we  
know when 256 ID numbers are used. Whenever a block is used up, get a  
new one in least recently used fashion. I guess the block size could  
be made dynamic for more efficiency with few fragment sources but  
support for a large number of fragment sources when required.

Or perhaps it makes sense to just ignore the issue, and write the  
lower 15 bits of the IPv6 ID in the IPv4 ID with the high bit 0 for  
IPv6 fragments and generate a 15-bit ID with the high bit 1 for  
unfragmented IPv6 packets. This WILL lead to incorrect reassembly when  
the number of fragment sources and number of fragments per second  
increases and in 1/65535th of those cases that will go undetected and  
lead to data corruption, but maybe we'll have to live with that if we  
want to have fast translators.

Thoughts?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 16 Apr 2009 11:12:26 +0200

On 15 apr 2009, at 22:57, Dan Wing wrote:

If intentional it would be helpful to call it
out rather than relying on everyone to analyze the differences --
that is wasted effort.

Ok, here are the differences between draft-baker-behave-v4v6- 
translation-02 and my text that I sent to the list a few days aog  
relevant to packet size and fragmentation issues that I found:

trans02 says:

    Fragmented IPv4 UDP packets that do not contain a UDP checksum (i.e.
    the UDP checksum field is zero) are not of significant use over  
wide-
    areas in the Internet and will not be translated by the IP/ICMP
    translator [Miller].

our text says to compute a checksum. This is also what the SIIT RFC  
specifies. Unfortunately, the reference is very old and not specific  
enough to retrieve.

trans02:

    One of the differences between IPv4 and IPv6 is that in IPv6 path  
MTU
    discovery is mandatory but it is optional in IPv4.

This isn't entirely correct. RFC 1981 says:

    IPv6 nodes SHOULD implement Path MTU Discovery in order to discover
    and take advantage of paths with PMTU greater than the IPv6 minimum
    link MTU [IPv6-SPEC].  A minimal IPv6 implementation (e.g., in a  
boot
    ROM) may choose to omit implementation of Path MTU Discovery.

    Nodes not implementing Path MTU Discovery use the IPv6 minimum link
    MTU defined in [IPv6-SPEC] as the maximum packet size.

trans02:

    When the IPv4 node performs path MTU discovery (by setting the DF  
bit
    in the header) the path MTU discovery can operate end-to-end i.e.
    across the translator.  In this case either IPv4 or IPv6 routers
    might send back ICMP "packet too big" messages to the sender.  When
    these ICMP errors are sent by the IPv6 routers they will pass  
through
    a translator which will translate the ICMP error to a form that the
    IPv4 sender can understand. In this case an IPv6 fragment header is
    only included if the IPv4 packet is already fragmented.

This doesn't address the situation where the IPv4 packet is translated  
into an IPv6 packet that is too large to forward without fragmentation  
like we do.

trans02:

    However, when the IPv4 sender does not perform path MTU discovery  
the
    translator has to ensure that the packet does not exceed the path  
MTU
    on the IPv6 side.  This is done by fragmenting the IPv4 packet so
    that it fits in 1280 byte IPv6 packet

In our text this is done differently: the translator performs PMTUD on  
the IPv6 side and only fragments the IPv6 packet if the IPv6  
destination's PMTU requires it. Note that in the trans02 case this  
would create a relatively large number of IPv6 fragments, depending on  
how much traffic is >1280 and DF=0. For TCP this should be rare but  
for video streaming over UDP this would be the common case.

trans02:

    Also, when the IPv4 sender
    does not perform path MTU discovery the translator MUST always
    include an IPv6 fragment header to indicate that the sender allows
    fragmentation.  That is needed should the packet pass through an IP/
    ICMP translator.

This isn't done in our text.

    In addition, the rules use the presence of an
    IPv6 fragment header to indicate that the sender might not be using
    path MTU discovery i.e. the packet should not have the DF flag set
    should it later be translated back to IPv4.

Our requirement that the translator perform PMTUD on the IPv6 side  
means that this isn't an issue for us.

trans02:

    If the DF flag is not set and the IPv4 packet will result in an IPv6
    packet larger than 1280 bytes the IPv4 packet MUST be fragmented
    prior to translating it.

Doesn't it make more sense to translate first and then fragment?  
(That's what we specify.) But I guess implementers can decide this  
issue for themselves as long as the end result is the same.

trans02:

    When a translator receives the first fragment of a fragmented UDP
    IPv4 packet and the checksum field is zero the translator SHOULD  
drop
    the packet and generate a system management event specifying at  
least
    the IP addresses and port numbers in the packet.  When it receives
    fragments other than the first it SHOULD silently drop the packet,
    since there is no port information to log.

It's not clear whether the last sentence is meant to only apply to the  
checksum=0 case. In that case, state would have to be maintained to  
recognize additional fragments. If not, then UDP fragments won't work  
because non-first fragments are consistently dropped so the  
destination can never reassemble.

There is also no specification of what happens when fragments arrive  
out-of-order.

trans02:

                Code 4 (fragmentation needed and DF set):  Translate  
to an
                ICMPv6 Packet Too Big message (Type 2) with code 0.  The
                MTU field needs to be adjusted for the difference  
between
                the IPv4 and IPv6 header sizes.  Note that if the IPv4
                router did not set the MTU field i.e. the router does  
not
                implement [RFC1191], then the translator must use the
                plateau values specified in [RFC1191] to determine a
                likely path MTU and include that path MTU in the ICMPv6
                packet.  (Use the greatest plateau value that is less
                than the returned Total Length field.)

Our text also specifies taking the translator's MTUs into account when  
adjusting the MTU difference.

We don't handle the case where no MTU value is returned. Since RFC  
1191 was published 18 years ago and PMTUD is as good as ubiquitous, do  
we need to support this case?

The text in trans02 about PMTUD and fragmentation starting at the top  
of page 15 suggests that PMTUD for UDP can't find the optimal packet  
size. However, this doesn't account for the differences between an  
IPv4 MTU of 1260 or smaller vs one that's bigger. In the latter case,  
PMTUD can work without any issues. (Note that PMTUD for UDP generally  
doesn't work with IPv4 because it must be handled by the transport  
protocol = effectively the application in IPv4, rather than the IP  
layer, like in IPv6.)

What about TCP? Can we assume that TCP fragments never happen with IPv6?

trans02:

    Identification:  All zero.

I have personally set up a system to fragment packets with DF=1 anyway  
way back when in my sordid past. It would be better if packets have a  
valid ID field even with DF=1.

trans02 (for unfragmented IPv6 packets):

       Flags:  The More Fragments flag is set to zero.  The Don't  
Fragments
       flag is set to one.

Note the difference with our text that specifies DF=0 for <= 1280 and  
DF=1 for > 1280 byte packets.

trans02, now talking about IPv6 packets with a fragment header:

       Identification:  Copied from the low-order 16-bits in the
       Identification field in the Fragment header.

See my message from yesterday about the issues with clashing ID values  
from different IPv6 hosts in the case of copying the ID bits from IPv6  
to IPv4. Also note that because of the way PMTUD is handled, there  
will be much more IPv6 packets with a fragment header so this issue  
will be more significant. In our solution, in many cases the  
translator will generate the ID because the IPv6 packet doesn't have a  
fragment header.

Additional issues that our text discusses:

"A NAT64 translator MUST have an MTU of at least 1280 on all of its  
interfaces, both IPv4 and IPv6 interfaces."

"To avoid unnecessary path MTU discovery cycles, a NAT64 SHOULD  
rewrite the MSS option in SYN packets to the minimum of the original  
MSS option, the NAT64's MTU on the IPv6 side - 60 and the NAT64's MTU  
on the IPv4 side - 40. This applies to SYNs in both the IPv4-to-IPv6  
direction and the IPv6-to-IPv4 direction."

Discussion of out-of-order fragment handling.

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 16 Apr 2009 09:48:11 -0700

Iljitsch,

> Sent: Wednesday, April 15, 2009 10:54 AM
> To: Templin, Fred L
> Cc: Behave WG; [email-masked] Nordmark
> Subject: Re: [BEHAVE] NAT64 fragmentation
> 
> Hi Fred,
> 
> Thanks for the comments.
> 
> >> 1280
> through different translators,

If the IPv6 host fragments a packet into multiple IPv6
fragments before sending, then for the reasons you
mentioned the translators must honor the ID which the
IPv6 host supplies (else, fragments of the same packet
flowing through different translators would not be
reassembled correctly). But, if the IPv6 host sends a
singleton fragment (fragment header included, offset=0
and MF=0) then shouldn't the translator likewise also
honor the ID?

> so it's important that the source host

Now you have me worried. We have a 32-bit ID field in IPv6,
hence probabilities of collision are rather small. But, we
are only using the low-order 16bits of the ID when translating
into IPv4, hence a much greater probability of collision. But
with a service such as google, we may see many orders of
magnitude more IPv6 hosts trying to talk to google through
the translator such that the incidence of collision goes way
up *even if the translator (and not the source hosts) is the
one who chooses the IDs*.

We also have the transport port numbers to consider in terms
of translation stream separation, so inter-stream interference
may be ameliorated somewhat. But fragment misassociations
within a single stream at high data rates are still a problem
in any case.

> fairly close to the IPv6 hosts = far from the IPv4 host.

Then, I expect there may be pathological interactions with
IP fragmentation in any case; especially when you set DF=0
and there is a link on the path with MTU<1280.

An alternative is to set DF=1 always, and if you get an
ICMP fragmentation needed with MTU<1280 declare the path
broken and go get the marginal link fixed. But, the ICMPs
might be filtered such that communications would silently
black hole - not good. 

All of this would be made much more robust if the translator
could be located close to the IPv4 hosts; even if that means
tunneling IPv6 over IPv4 over an extended span.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 17 Apr 2009 10:32:46 +0200

> and MF=0) then shouldn't the translator likewise also

Well, I don't see a pressing need to do that in this case, but then  
again, I also don't see why it would make sense to specifically check  
for this condition.

Thinking a bit more about the ID issues, I've come to the conclusion  
that we should just ignore the issue, for the following reasons:

1. There doesn't seem to be a way to solve this well
2. Solving it "better" incurs significant complexity
3. Fragments in the IPv6-to-IPv4 direction SHOULD be rare

About that last point: basically, you get fragments when the  
application sends large packets, or the path MTU is small. The former  
case occurs with the DNS where DNS servers typically advertise a 2 or  
4 k buffer so if there is a big answer, this could be larger than what  
fits in 1500 bytes so the answer needs to be fragmented at the source.  
However, even if we assume many large answers few NAT64 users would  
have to answer thousands of DNS queries per second through a  
translator, so in practice this isn't an issue.

It's pretty much impossible for TCP fragments to occur with IPv6. With  
video streaming over UDP there could be a relatively large number of  
UDP packets, but applications limit their packet sizes to avoid  
fragmentation. Also, since the IPv6 hosts are _clients_, whenever  
there is video streaming it's probably not from many IPv6 hosts to a  
single IPv4 hosts (when the ID issues come up) more more like from  
each IPv6 host to a different IPv4 host.

I guess it's possible to have fragments generated by tunnels of some  
sort, but again, that would probably not be many IPv6 hosts to a  
single IPv4 host.

So my earlier suggestion to just copy the lower 15 bits of the IPv6 ID  
into the IPv4 ID when there is a fragment header or generate a new ID  
if there isn't with a different high order bit is probably sufficient.

[...]

> one who chooses the IDs*.

Right, but not for TCP, which I believe Google and the like use for  
the vast majority of their services.

> and there is a link on the path with MTU<1280.

Right. So don't use links with MTUs that small.

Can you think of a reasonable way to make this work better?

> broken and go get the marginal link fixed.

I don't see that happen.

> tunneling IPv6 over IPv4 over an extended span.

You'd still need to cross that < 1280 byte link. If you do that with  
IPv4-in-IP tunneling then you'd need to fragment the outer header and  
now ALL packets will need a good ID value, while before we didn't have  
a problem with TCP.

I'm also quite uncomfortable by doing this much work for people who  
are too lazy to either:

1. upgrade to IPv6, or
2. use a reasonable MTU

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 17 Apr 2009 10:38:40 +0200

On 14 apr 2009, at 16:16, Lars Eggert wrote:

> (although it's obviously implemented in some products), and we need  
> to carefully discuss what we want to say here.

Ok, so far no reactions from tcpm.

However, after some more consideration I think it's better to drop  
this. There isn't much value in doing this, because in the common case  
where everyone involved has a 1500-byte MTU, it's not needed: in the  
IPv6-to-IPv4 direction the IPv6 host will think it can send 1520 byte  
packets after seeing a 1460 MSS but it obviously won't because it's  
limited to 1500 bytes itself. In the IPv4-to-IPv6 direction the IPv4  
host will see a 1440 MSS so it will send 1480 byte packets which will  
be 1500 bytes after translation so that works just fine as well.

Only when the IPv6 host supports larger packets it will send those  
1520-byte packets which will probably hit a 1500-byte MTU link so  
PMTUD will kick in but jumboframe-capable IPv6 hosts would be expected  
to see a lot of PMTUD activity so this shouldn't be a problem.

---

From: Lars Eggert <[email-masked]>
Date: Fri, 17 Apr 2009 12:18:33 +0300

If you drop this, I'd suggest to include the argument below in the  
draft, so that implementors understand why it isn't needed. (They  
might otherwise decide to do the clamping anyway, "because the IETF  
forgot it.")

Lars

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 17 Apr 2009 08:23:18 -0700

Iljitsch,

> Sent: Friday, April 17, 2009 1:33 AM
> To: Templin, Fred L

You get fragments when the IPv6 host performs host
fragmentation, e.g., to a maximum fragment size of 1280.
I don't think it is possible to say (sic) "its only DNS",
or "it will never be TCP", etc. Fragments are fragments,
and the translator has no way to control an IPv6 host that
wants to produce them.

> if there isn't with a different high order bit is probably sufficient.

So, you are suggesting to code the IP ID?

> Right. So don't use links with MTUs that small.

We can't control what links are out there in the wild; we
have to find some way to live with them.

> Can you think of a reasonable way to make this work better?

Yes - attach the translator directly to the IPv4 host.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 17 Apr 2009 17:32:06 +0200

Hi Fred,

> and the translator has no way to control an IPv6 host that

Of course.

However, we know we can make basic fragmentation work fairly well for  
limited numbers of fragmented packets per second. As soon as ID space  
wrapping issues start to crop up (4k fragmented packets per second?)  
we're in trouble, and there is no real solution. The question is  
whether it's worth the extra trouble to go from a 60% solution to a  
80% solution.

Based on the logic explained in my previous message, my position is  
that the number of fragments will almost certainly be so low that ID  
wrapping issues should be extremely rare to spending extra resources  
to improve this are not worth it.

> So, you are suggesting to code the IP ID?

Not sure what you mean by that.

> Yes - attach the translator directly to the IPv4 host.

That's not going to happen in the scenario that we're trying to  
address with the NAT64 draft.