Subject: [tsvwg] Is it feasible to perform fragmentation on UDP encapsulated packets.

From: Xuxiaohu <[email-masked]>
Date: Fri, 27 May 2016 10:08:28 +0000

<Note that I have changed the subject of the email hence it has nothing to do with the WG adoption call now. It's just a discussion on a particular issue which is related to those WGs which are working on UDP tunnels. The reason for containing the old email is to use it as a background which may be useful for better understanding of this particular issue>

The possible side-effect of performing fragmentation on UDP encapsulated packets is to worsen the reassembly burden on tunnel egress since fragments of UDP encapsulated packets are more likely to be forwarded across different paths towards the tunnel egress than those of IP or GRE encapsulated packets.

It seems that most X-over-UDP proposals choose to prohibit the tunnel ingress from performing fragmentation on UDP encapsulated packets. See the following quoted text regarding fragmentation from those X-over-UDP drafts:

LISP:

When an ITR receives a packet from a site-facing interface and adds H
   octets worth of encapsulation to yield a packet size greater than L
   octets, it resolves the MTU issue by first splitting the original
   packet into 2 equal-sized fragments.  A LISP header is then prepended
   to each fragment.

VXLAN:

VTEPs MUST NOT fragment VXLAN packets.  Intermediate routers may
   fragment encapsulated VXLAN packets due to the larger frame size.
   The destination VTEP MAY silently discard such VXLAN fragments.

VXLAN-GPE:

VTEPs MUST never fragment an encapsulated VXLAN GPE packet, and when
   the outer IP header is IPv4, VTEPs MUST set the DF bit in the outer
   IPv4 header.

GEVENE:

   To prevent fragmentation and maximize performance, the best practice
   when using Geneve is to ensure that the MTU of the physical network
   is greater than or equal to the MTU of the encapsulated network plus
   tunnel headers.

GUE:

    If a packet is fragmented before encapsulation in GUE, all the
    related fragments must be encapsulated using the same source port
    (inner flow identifier). An operator may set MTU to account for
    encapsulation overhead and reduce the likelihood of fragmentation.

GRE/UDP

Regarding packet fragmentation, an encapsulator/decapsulator SHOULD
   be compliant with [RFC7588] and perform fragmentation before the
   encapsulation.

However, the above choice seems conflict with the requirements as described in https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02

I wonder whether the IETF should reach a consensus on whether or not the fragmentation on UDP encapsulated packets should be allowed.
  
Best regards,
Xiaohu

> -----Original Message-----
> From: Xuxiaohu
> Sent: Thursday, May 26, 2016 4:35 PM
> To: 'Joe Touch'; Fred Baker (fred); Wassim Haddad
> Cc: [email-masked]
> Subject: RE: [Int-area] Call for adoption of draft-xu-intarea-ip-in-udp-03
> 
> 
> 
> > -----Original Message-----
> > From: Joe Touch [mailto:[email-masked]]
> > Sent: Thursday, May 26, 2016 2:11 AM
> > To: Xuxiaohu; Fred Baker (fred); Wassim Haddad
> > Cc: [email-masked]
> > Subject: Re: [Int-area] Call for adoption of
> > draft-xu-intarea-ip-in-udp-03
> >
> >
> >
> > On 5/24/2016 7:24 PM, Xuxiaohu wrote:
> > > Hi Joe,
> > >
> > > I wonder whether you want to tell me the following truth by the
> > > example that you gave: no matter whatever improvements we had done
> > > with this draft, those persons who dislike it by the light of nature
> > > would dislike it in the end.
> >
> > The only improvements that would make this doc useful would be to add
> > capabilities already in GRE/UDP or GUE/UDP, which we already have.
> 
> Let's go over the four things you mentioned earlier in GRE/UDP and GUE/UDP:
> 
> 	- stronger checksums
> 
> In GRE/UDP, in order to use UDP-zero-checksum, it gave the following
> restrictions:
> " 6. UDP Checksum Handling
> 
>    6.1. UDP Checksum with IPv4
> 
>    For UDP in IPv4, the UDP checksum MUST be processed as specified in
>    [RFC768] and [RFC1122] for both transmit and receive. The IPv4
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 12]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation             March 2016
> 
>    header includes a checksum which protects against mis-delivery of
>    the packet due to corruption of IP addresses. The UDP checksum
>    potentially provides protection against corruption of the UDP header,
>    GRE header, and GRE payload. Disabling the use of checksums is a
>    deployment consideration that should take into account the risk and
>    effects of packet corruption.
> 
>    When a decapsulator receives a packet, the UDP checksum field MUST
>    be processed. If the UDP checksum is non-zero, the decapsulator MUST
>    verify the checksum before accepting the packet. By default a
>    decapsulator SHOULD accept UDP packets with a zero checksum. A node
>    MAY be configured to disallow zero checksums per [RFC1122]; this may
>    be done selectively, for instance disallowing zero checksums from
>    certain hosts that are known to be sending over paths subject to
>    packet corruption. If verification of a non-zero checksum fails, a
>    decapsulator lacks the capability to verify a non-zero checksum, or
>    a packet with a zero-checksum was received and the decapsulator is
>    configured to disallow, the packet MUST be dropped and an event MAY
>    be logged.
> 
>    6.2. UDP Checksum with IPv6
> 
>    For UDP in IPv6, the UDP checksum MUST be processed as specified in
>    [RFC768] and [RFC2460] for both transmit and receive.
> 
>    When UDP is used over IPv6, the UDP checksum is relied upon to
>    protect both the IPv6 and UDP headers from corruption. As such, A
>    default GRE-in-UDP Tunnel MUST perform UDP checksum; A TMCE GRE-in-
>    UDP Tunnel MAY be configured with the UDP zero-checksum mode if the
>    traffic-managed controlled environment or a set of closely
>    cooperating traffic-managed controlled environments (such as by
>    network operators who have agreed to work together in order to
>    jointly provide specific services) meet at least one of following
>    conditions:
> 
>    a. It is known (perhaps through knowledge of equipment types and
>       lower layer checks) that packet corruption is exceptionally
>       unlikely and where the operator is willing to take the risk of
>       undetected packet corruption.
> 
>    b. It is judged through observational measurements (perhaps of
>       historic or current traffic flows that use a non-zero checksum)
>       that the level of packet corruption is tolerably low and where
>       the operator is willing to take the risk of undetected packet
>       corruption.
> 
> 
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 13]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation             March 2016
> 
>    c. Carrying applications that are tolerant of mis-delivered or
>       corrupted packets (perhaps through higher layer checksum,
>       validation, and retransmission or transmission redundancy) where
>       the operator is willing to rely on the applications using the
>       tunnel to survive any corrupt packets.
> 
>    The following requirements apply to a TMCE GRE-in-UDP tunnel that
>    use UDP zero-checksum mode:
> 
>      a. Use of the UDP checksum with IPv6 MUST be the default
>         configuration of all GRE-in-UDP tunnels.
> 
>      b. The GRE-in-UDP tunnel implementation MUST comply with all
>         requirements specified in Section 4 of [RFC6936] and with
>         requirement 1 specified in Section 5 of [RFC6936].
> 
>      c. The tunnel decapsulator SHOULD only allow the use of UDP zero-
>         checksum mode for IPv6 on a single received UDP Destination
>         Port regardless of the encapsulator. The motivation for this
>         requirement is possible corruption of the UDP Destination Port,
>         which may cause packet delivery to the wrong UDP port. If that
>         other UDP port requires the UDP checksum, the mis-delivered
>         packet will be discarded.
> 
>      d. It is RECOMMENDED that the UDP zero-checksum mode for IPv6 is
>         only enabled for certain selected source addresses. The tunnel
>         decapsulator MUST check that the source and destination IPv6
>         addresses are valid for the GRE-in-UDP tunnel on which the
>         packet was received if that tunnel uses UDP zero-checksum mode
>         and discard any packet for which this check fails.
> 
>      e. The tunnel encapsulator SHOULD use different IPv6 addresses for
>         each GRE-in-UDP tunnel that uses UDP zero-checksum mode
>         regardless of the decapsulator in order to strengthen the
>         decapsulator's check of the IPv6 source address (i.e., the same
>         IPv6 source address SHOULD NOT be used with more than one IPv6
>         destination address, independent of whether that destination
>         address is a unicast or multicast address). When this is not
>         possible, it is RECOMMENDED to use each source IPv6 address for
>         as few UDP zero-checksum mode GRE-in-UDP tunnels as is feasible.
> 
>      f. When any middlebox exists on the path of a GRE-in-UDP tunnel,
>         it is RECOMMENDED to use the default mode, i.e. use UDP
>         checksum, to reduce the chance that the encapsulated packets to
>         be dropped.
> 
> 
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 14]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation             March 2016
> 
>      g. Any middlebox that allows the UDP zero-checksum mode for IPv6
>         MUST comply with requirement 1 and 8-10 in Section 5 of
>         [RFC6936].
> 
>      h. Measures SHOULD be taken to prevent IPv6 traffic with zero UDP
>         checksums from "escaping" to the general Internet; see Section
>         8 for examples of such measures.
> 
>      i. IPv6 traffic with zero UDP checksums MUST be actively monitored
>         for errors by the network operator. For example, the operator
>         may monitor Ethernet layer packet error rates.
> 
>      j. If a packet with a non-zero checksum is received, the checksum
>         MUST be verified before accepting the packet. This is
>         regardless of whether the tunnel encapsulator and decapsulator
>         have been configured with UDP zero-checksum mode.
> 
>    The above requirements do not change either the requirements
>    specified in [RFC2460] as modified by [RFC6935] or the requirements
>    specified in [RFC6936].
> 
>    The requirement to check the source IPv6 address in addition to the
>    destination IPv6 address, plus the strong recommendation against
>    reuse of source IPv6 addresses among GRE-in-UDP tunnels collectively
>    provide some mitigation for the absence of UDP checksum coverage of
>    the IPv6 header. A traffic-managed controlled environment that
>    satisfies at least one of three conditions listed above in this
>    section provides additional assurance.
> 
>    A GRE-in-UDP tunnel is suitable for transmission over lower layers
>    in the traffic-managed controlled environments that are allowed by
>    the exceptions stated above and the rate of corruption of the inner
>    IP packet on such networks is not expected to increase by comparison
>    to GRE traffic that is not encapsulated in UDP.  For these reasons,
>    GRE-in-UDP does not provide an additional integrity check except
>    when GRE checksum is used when UDP zero-checksum mode is used with
>    IPv6, and this design is in accordance with requirements 2, 3 and 5
>    specified in Section 5 of [RFC6936].
> 
>    Generic Router Encapsulation (GRE) does not accumulate incorrect
>    state as a consequence of GRE header corruption. A corrupt GRE
>    packet may result in either packet discard or forwarding of the
>    packet without accumulation of GRE state. Active monitoring of GRE-
>    in-UDP traffic for errors is REQUIRED as occurrence of errors will
>    result in some accumulation of error information outside the
>    protocol for operational and management purposes. This design is in
>    accordance with requirement 4 specified in Section 5 of [RFC6936].
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 15]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation             March 2016
> 
>    The remaining requirements specified in Section 5 of [RFC6936] are
>    not applicable to GRE-in-UDP.  Requirements 6 and 7 do not apply
>    because GRE does not include a control feedback mechanism.
>    Requirements 8-10 are middlebox requirements that do not apply to
>    GRE-in-UDP tunnel endpoints (see Section 7.1 for further middlebox
>    discussion).
> 
>    It is worth mentioning that the use of a zero UDP checksum should
>    present the equivalent risk of undetected packet corruption when
>    sending similar packet using GRE-in-IPv6 without UDP [RFC7676] and
>    without GRE checksums.
> 
>    In summary, a TMCE GRE-in-UDP Tunnel is allowed to use UDP-zero-
>    checksum mode for IPv6 when the conditions and requirements stated
>    above are met. Otherwise the UDP checksum need to be used for IPv6
>    as specified in [RFC768] and [RFC2460]. Use of GRE checksum is
>    RECOMMENED when the UDP checksum is not used.
> "
> 
> In GUE, to support UDP-checksum-zero, it said
> 
> " Therefore, when GUE is used over
>     IPv6, either the UDP checksum must be enabled or the GUE header
>     checksum must be used.  An encapsulator MAY set a zero UDP checksum
>     for performance or implementation reasons, in which case the GUE
>     header checksum MUST be used or applicable requirements for using
>     zero UDP checksums in [GREUDP] MUST be met. If the UDP checksum is
>     enabled, then the GUE header checksum should not be used since it is
>     mostly redundant."
> 
> It's easy for me to add the similar words to the IP-in-UDP draft like "the
> applicable requirements for using zero UDP checksum in [GREUDP] MUST be
> met when zero UDP checksum is used by the tunnel ingress". However, the
> major goal for disabling the UDP checksum is to improve the performance.
> When GUE header checksum is used and/or the bunch of applicable
> requirements as described in GRE/UDP are verified, is the goal of improving
> performance still achievable? If not, why not directly enable the UDP-checksum
> instead?
> 
> 	- fragmentation support
> 
> In GRE/UDP, it said
> 
> " 4.1. MTU and Fragmentation
> 
>    Regarding packet fragmentation, an encapsulator/decapsulator SHOULD
>    be compliant with [RFC7588] and perform fragmentation before the
>    encapsulation. The size of fragments SHOULD be less or equal to the
>    PMTU associated with the path between the GRE ingress and the GRE
>    egress tunnel endpoints minus the GRE and UDP overhead ..."
> 
> in GUE, it said
> 
> " 4.9. MTU and fragmentation
> 
>     Standard conventions for handling of MTU (Maximum Transmission Unit)
>     and fragmentation in conjunction with networking tunnels
>     (encapsulation of layer 2 or layer 3 packets) should be followed.
>     Details are described in MTU and Fragmentation Issues with In-the-
>     Network Tunneling [RFC4459]... "
> 
> It seems that the only missing thing in the IP-in-UDP draft is to allow the outer
> fragmentation. However, as it said in
> (https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02#page-13), " ...IPsec
> performs only Outer Fragmentation; this distinguishes it from IP-in-IP, which
> performs only Inner Fragmentation. " Note that IP-in-IP is the dominant
> encapsulation choice within Softwires networks. In other words, performing
> only inner fragmentation works very well in practice. Furthermore, the outer
> fragmentation issue (e.g., reassembly cost for the egress) would become even
> worse since the fragments of X-in-UDP packets are more likely to be forwarded
> across different paths than those of X-in-IP and X-in-GRE packets. Hence, I'm
> wondering whether it's worthwhile to support the outer fragmentation on UDP
> encapsulated packets which seems useless in practice.
> 
> 	- signalling support (e.g., to test whether a tunnel is up or
> 	to measure MTUs)
> 
> I haven't found any description of this in both GRE/UDP and GUE. Did you?
> 
> 	- support for robust ID fields (related to fragmentation,
> 	e.g., to overcome the limits of IPv4 ID as per RFC 6864)
> 
> I haven't found any description of this in both GRE/UDP and GUE. Did you?
> 
> Xiaohu
> 
> > It is not our obligation to find a way for your document to proceed -
> > that onus is on you.
> >
> > Joe

---

From: [email-masked]
Date: Fri, 27 May 2016 12:50:02 +0200

> I wonder whether the IETF should reach a consensus on whether or not the fragmentation on UDP encapsulated packets should be allowed.

Having just implemented fragmentation and reassembly support in a MAP BR...

MAP does IPv4 over IPv6 tunnels with shared IPv4 address (aka routing on ports).

- Inner IPv4 fragmentation is relatively easy, cause you don't need to reassemble you can do virtual reassembly
- Outer IPv6 reassembly is much harder, cause you have to do it on the tunnel egress.

It is not possible to implement reassembly complying with IETF RFCs.
The IPv4 reassembly buffer is specified in IPv4 as 15 seconds, in IPv6 as 60 seconds.
My implementation does upwards of 100Mpps, you do the numbers.

If you can:
 - Guarantee always in sequence
 - Maximum fragment chain of 2
 - Maximum time and packet gap between first and last fragment of _very_small_

Then sure it is implementable, but if not then you're just setting yourself up for a DOS attack.
And if you have that much control over the environment, just increase the MTU in the tunnel domain.

Code is here:
https://git.fd.io/cgit/vpp/tree/vnet/vnet/map/ip6_map.c#n530

So in short, IETF can say whatever they like, that's not going to change reality.

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 May 2016 08:06:18 -0700

> The possible side-effect of performing fragmentation on UDP encapsulated packets is to worsen the reassembly burden on tunnel egress since fragments of UDP encapsulated packets are more likely to b…

Xiaohu,

I don't understand why UDP encapsulation would make things worse than
other encapsulations. Fragmentation would be needed at tunnel ingress
when packet size exceeds MTU of the tunnel. RFC4459 describes the
issues and general solutions that apply to the different techniques of
IP tunneling.

> It seems that most X-over-UDP proposals choose to prohibit the tunnel ingress from performing fragmentation on UDP encapsulated packets. See the following quoted text regarding fragmentation from t…
Please look a draft-herbert-gue-fragmentation-02. This is a method
where the fragmentation/reassembly is performed in the encapsulation
layer instead of (outer or inner) IP layer.

Tom

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 28 May 2016 01:26:55 +0000

Hi Ole,

> From: lisp [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Friday, May 27, 2016 6:50 PM
> To: Xuxiaohu
> Cc: Softwires WG; [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [lisp] [Softwires] Is it feasible to perform fragmentation on UDP
> encapsulated packets.
> 
> > <Note that I have changed the subject of the email hence it has nothing to do
> with the WG adoption call now. It's just a discussion on a particular issue which is
> related to those WGs which are working on UDP tunnels. The reason for
> containing the old email is to use it as a background which may be useful for
> better understanding of this particular issue>
> >
> > The possible side-effect of performing fragmentation on UDP encapsulated
> packets is to worsen the reassembly burden on tunnel egress since fragments of
> UDP encapsulated packets are more likely to be forwarded across different
> paths towards the tunnel egress than those of IP or GRE encapsulated packets.
> >
> > It seems that most X-over-UDP proposals choose to prohibit the tunnel ingress
> from performing fragmentation on UDP encapsulated packets. See the following
> quoted text regarding fragmentation from those X-over-UDP drafts:
> >
> > LISP:
> >
> > However, the above choice seems conflict with the requirements as described
> in https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02
> >
> >
> > I wonder whether the IETF should reach a consensus on whether or not the
> fragmentation on UDP encapsulated packets should be allowed.
> 

Thanks a lot for sharing your implementation experience!

> And if you have that much control over the environment, just increase the MTU

I couldn't agree more.

> So in short, IETF can say whatever they like, that's not going to change reality.

Wouldn't it be better if the IETF guidelines on tunnel fragmentation could be more in conformity with the reality?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 28 May 2016 02:59:00 +0000

Hi Tom,

> From: Tom Herbert [mailto:[email-masked]]
> Sent: Friday, May 27, 2016 11:06 PM
> To: Xuxiaohu
> Cc: [email-masked]; Softwires WG; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [Int-area] Is it feasible to perform fragmentation on UDP
> encapsulated packets.
> 
> I don't understand why UDP encapsulation would make things worse than other
> encapsulations. Fragmentation would be needed at tunnel ingress when packet

When performing fragmentation after UDP encapsulation (a.k.a., outer fragmentation), the first fragment (still a UDP packet) and the subsequent fragments (not a UDP packet anymore) of a given UDP encapsulated packet are more likely to be routed over different paths in contrast to IP or GRE encapsulated packets. As a result, the reordering issue on the tunnel egress which is required to do reassembly would become even worse.

> size exceeds MTU of the tunnel. RFC4459 describes the issues and general
> solutions that apply to the different techniques of IP tunneling.

Partially agree. The use of UDP as a tunneling protocol was not popular at the time when RFC4459 was published. Otherwise, the worse recording issue associated with the outer fragmentation of UDP encapsulated packets may need more attention in that RFC.

By the way, RFC4459 has illustrated the risk of performing [outer] fragmentation on tunneled packets clearly and said clearly that "...   So, if reassembly could be made to work sufficiently reliably, this would be one acceptable fallback solution but only for IPv6." However, I just occasionally noticed that in section 3.3.2.1 (Tunneling GRE over IPv4) of RFC7588, it said "By default, the GRE ingress node does not fragment delivery packets. However, the GRE ingress node includes a configuration option that allows delivery packet fragmentation." Does it mean there is a conflict between those two RFCs regarding whether outer fragmentation is needed for GRE over IPv4?

> Please look a draft-herbert-gue-fragmentation-02. This is a method where the
> fragmentation/reassembly is performed in the encapsulation layer instead of
> (outer or inner) IP layer.

It's an interesting idea. However, since it needs a dedicated fragmentation field in the GUE header, it seems that this approach is not suitable for the compressed GUE packet (a.k.a., in the same format of IP-in-UDP) where the fragmentation field and even the whole GUE header is gone.

Best regards,
Xiaohu

---

From: Tom Herbert <[email-masked]>
Date: Mon, 30 May 2016 09:30:38 -0700

> When performing fragmentation after UDP encapsulation (a.k.a., outer fragmentation), the first fragment (still a UDP packet) and the subsequent fragments (not a UDP packet anymore) of a given UDP e…
Presumably you are referring to ECMP and devices that parse the
transport header to use port information in the hash. There are two
ways this can be "fixed" in devices that do ECMP. 1) If the packet is
a fragment don't parse the transport header. This way all fragments
will be ECMP routed based on just L3 information so they all take the
same path. 2) Use non-zero IPv6 flow label and don't parse L4 header
to do ECMP. This means all fragments and all non-fragments of a flow
will follow the same path. The flow label approach also works with
protocols other than TCP and UDP as well as any arbitrary list of
extension headers. Both of these techniques have been implement in
Linux stack.

> Partially agree. The use of UDP as a tunneling protocol was not popular at the time when RFC4459 was published. Otherwise, the worse recording issue associated with the outer fragmentation of UDP e…
Yes, but UDP was popular long before it was used for tunneling and the
problems with it of fragmentation/reassembly were well known.
Considering tunneling with UDP doesn't really change anything, the
same solutions (like those listed in RFC4459) still apply.

> It's an interesting idea. However, since it needs a dedicated fragmentation field in the GUE header, it seems that this approach is not suitable for the compressed GUE packet (a.k.a., in the same f…
Different header formats can be used for different packets in the same
flow. The outer UDP header and flow label are sufficient for ECMP, as
long devices aren't parsing into the GUE payload to do ECMP then all
packets for a flow (fragmented or not) should follow the same path.

Tom