Subject: [multipathtcp] how to deal with GRO when we use MPTCP?

From: 王帅 <[email-masked]>
Date: Wed, 22 Mar 2017 15:32:21 +0800 (GMT+08:00)

Hi,

     Some NICs has GRO(generic-receive-offload), which works by aggregating multiple incoming packets from a single stream into a larger buffer before they are passed higher up the networking stack, thus reducing the number of packets that have to be processed. But when we use MPTCP, GRO may aggregate packets whose subflow sequence number is continuous but data sequence number is discontinuous. This is not expected. How can we deal with this problem? Is it necessary to disable GRO?

Best wishes,
Wang Shuai

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 22 Mar 2017 09:54:08 +0100

Hello,
>
>      Some NICs has GRO(generic-receive-offload), which works by
> aggregating multiple incoming packets
> <https://en.wikipedia.org/wiki/Packet_(information_technology)> from a
> single stream <https://en.wikipedia.org/wiki/Stream_(computing)> into a
> larger buffer before they are passed higher up the networking stack,
> thus reducing the number of packets that have to be processed. But when
> we use MPTCP, GRO may aggregate packets whose subflow sequence number is
> continuous but data sequence number is discontinuous. This is not
> expected. How can we deal with this problem? Is it necessary to disable GRO?

GRO should not aggregate packets that contain different TCP options that 
are unknown to the network interface. Our experience with GRO and 
Multipath TCP has been positive until now with the Linux MPTCP stack.

GRO is typically combined with TSO. What we usually observe is the 
following behaviour :

Sender creates a large packet with DSN, Timestamp, data
TSO splits the packet with each packet containing a copy of the DSN and 
the Timestamp. All DSNs are the same in the packets sent by the network 
interface.

On the receiver side, GRO sees a sequence of packets with the same DSN 
and Timestamp options. These packets are combined together to present a 
large packet containing DSN, Timestamp, data to the stack

GRO should not coalesce packets from an MPTCP sender that does not use 
TSO and thus send packets with different DSN values. If you see a 
network interface that behaves badly, please report this on the 
mptcp-dev mailing list

Olivier

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Mar 2017 08:42:05 -0700

FYI - we had experience with GRO on Linux where it acted against spec,
coalescing packets with EDO headers even when these options were
different between adjacent packets. We also saw performance plummet when
GRO was enabled even when it was not supposed to coalesce packets.

See Sec 5.2 of the following:

H. Trieu, J. Touch, T. Faber, “Implementation of the TCP Extended Data
Offset Option <http://www.isi.edu/touch/pubs/isi-tr-2015-696.pdf>,”
USC/ISI Tech. Report ISI-TR-696, Mar. 2015.

Note that we were unable to pinpoint the issue, which might have been in
our implementation.

Joe

On 3/22/2017 1:54 AM, Olivier Bonaventure wrote:
> ...
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp