Subject: [multipathtcp] Sequence numbers in payload, Data ACKs in options

From: "SCHARF, Michael" <[email-masked]>
Date: Wed, 28 Jul 2010 15:31:16 +0200

Dear all,

I thought a little bit about whether it is possible to combine the two
protocols proposals on the table: Use TLV encoding for the sequence
numbers (somehow similar to MCTCP), but use data ACKs+flow control in
options, where they seem to be more reasonable. According to Phil's
table, this would actually be a somehow reasonable solution, combining
the benefits of both.

This actually seems to avoid the protocol problem that option encoding
has with unreliable transport of data sequence mappings, and the
performance issues of payload transport of data ACKs.

I thought a little bit about how I would implement this in Linux.
Actually, I think that a user-space implementation of such a protocol is
possible, in combination with a kernel patch that is not much more
complex than the one that I already have. The simplest case would be if
the data ACKs are only send in pure ACKs without other payload.
Again, I do not claim that this is the most efficient variant, in
particular concernin memory, but, as I pointed out before, I am trying
to think of protocol variants that can easily be deployed. And
full-featured kernel-implementation would probably very similar to the
current one of MPTCP.

I know that this solution is technically not perfect neither. But I'd
like to better understand the technical downsides of it. Any comments on
that?

Michael

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 28 Jul 2010 15:39:54 +0200

Michael,
> table, this would actually be a somehow reasonable solution, combining

Using TLV encoding for sequence numbers looks difficult from a
performance viewpoint to me. Since a long time, TCP stacks have been
tuned to avoid copying the user data as much as possible.

In the Linux implementation, it is possible to directly send a set of
pages (e.g. a file) on a TCP connection without passing through user
space memory. This is used for example by video streaming servers.

By using a TLV encoding of the data, you would need to add some
information in front of the data on the sender side. This might require
moving data in memory depending on the size of the chunks that you
consider and the size of the buffer passed through the send system call.
On the receiver side, this is possibly worse as the receiver has to
parse the received bytestream to extract the TLV information and remove
the TLV encoding before passing the information to the userspace. This
again potentialle requires touching all the received data.

Olivier

-- 
http://inl.info.ucl.ac.be , UCLouvain, Belgium

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 28 Jul 2010 14:49:17 +0100

I belive the only way TLV would be viable to work together with Offload
HW, would be to place a "MPTCP TLV" Option (2 bytes) in those segments
containing TLV data.

However, certain hardware might avoid the problem of touching the data
part after the TLV. You can prepare a sub-flow TCP Header (with the
MPTCP option "MPTCP TLV") and add the TLV to the descriptor of the
header; send one segment like that, then adjust the TCP header (remove
option & TLV) and send out the remainder - however, it's likely that the
data will not be aligned by that time (or the first Option+TLV segment
might be of some smaller size, ie, TLV+1k payload), for efficiency
reasons....

Regards,
   Richard

> -----Original Message-----
> From: Olivier Bonaventure [mailto:[email-masked]] 
> Sent: Mittwoch, 28. Juli 2010 15:40
> To: SCHARF, Michael
> Cc: multipathtcp
> Subject: Re: [multipathtcp] Sequence numbers in payload, Data 
> ACKs in options
> 
> Michael,
> > 
> Using TLV encoding for sequence numbers looks difficult from 
> a performance viewpoint to me. Since a long time, TCP stacks 
> have been tuned to avoid copying the user data as much as possible.
> 
> In the Linux implementation, it is possible to directly send 
> a set of pages (e.g. a file) on a TCP connection without 
> passing through user space memory. This is used for example 
> by video streaming servers.
> 
> By using a TLV encoding of the data, you would need to add 
> some information in front of the data on the sender side. 
> This might require moving data in memory depending on the 
> size of the chunks that you consider and the size of the 
> buffer passed through the send system call.
> On the receiver side, this is possibly worse as the receiver 
> has to parse the received bytestream to extract the TLV 
> information and remove the TLV encoding before passing the 
> information to the userspace. This again potentialle requires 
> touching all the received data.
> 
> 
> Olivier
> 
> 
> 
> 
> --
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: "SCHARF, Michael" <[email-masked]>
Date: Wed, 28 Jul 2010 17:15:27 +0200

> touching all the received data.

If the chunk sizes are large, we could e. g. send the TLV header in a
single packet (with PUSH flag), and the rather large chunk size
afterwards without memory copy, using today's TCP memory copy features.
Unless the bytestream gets resegmented, the receiver just needs to pass
the received bytes with a certain offset to the user space, in chunks of
a rather large size (e. g., 64K, or more).

Of course, this is not perfect, but if a DSN option is needlessly set on
every packet, there seems to be a significant processing overhead at the
receiver, too. I am not familiar enough with OS design to say what is
worse.

If resegmentation occurs in a NIC or in a middlebox, we will IMHO have a
problem both with payload and option encoding, as the data sequence
number mappings will not be aligned with packet boundaries and the
receiver has to touch the received data in both cases.

Michael

---

From: Robert Watson <[email-masked]>
Date: Wed, 28 Jul 2010 17:29:22 +0100 (BST)

On Wed, 28 Jul 2010, Olivier Bonaventure wrote:

>> protocols proposals on the table: Use TLV encoding for the sequence numbers 
>> (somehow similar to MCTCP), but use data ACKs+flow control in options, 
>> where they seem to be more reasonable. According to Phil's table, this 
>> would actually be a somehow reasonable solution, combining the benefits of 
>> both.
>
> Using TLV encoding for sequence numbers looks difficult from a performance 
> viewpoint to me. Since a long time, TCP stacks have been tuned to avoid 
> copying the user data as much as possible.
>
> In the Linux implementation, it is possible to directly send a set of pages 
> (e.g. a file) on a TCP connection without passing through user space memory. 
> This is used for example by video streaming servers.

The benefit here isn't just through the avoidance of copying itself, but also 
the reduction in kernel memory use accomplished by multiple references to the 
same underlying memory -- i.e., a single copy of the file in the page cache. 
Instead of 30 copies of the file being bounced through userspace to 
independent kernel memory, you just have the meta-data referencing the file 
deirectly in the cache.  This means you can keep many more files in your 
cache, improving scalability.

However, interfaces such as sendfile(2) already provide the ability to prepend 
headers and append trailers, and I'm not convinced that conservative injection 
of TLV data would be a big problem.  At least in BSD-style stacks, the 
meta-data is stored in mbufs (Linux - skbufs), which then reference underlying 
objects, such as VM pages.  Those references can be at arbitrary offsets 
inside the underlying object, so while you would pick up additional overhead 
in managing new mbufs (primarily, allocating and freeing them, but also some 
queueing costs), no additional copying is needed, and the costs of allocation 
will scale with the amount of meta-data, as opposed to with the size of the 
file being sent.

Similarly, if you're allowed to place the TLV headers based on existing 
breaking points when copying from userspace, it shouldn't cause additional 
data copies: we copy data in various size chunks, but have to allocate 
additional storage as we go, so there are natural breaking points where data 
can be cheaply inserted even at lower levels of the stack.  In fact, that 
process closely resembles TCP segmentation and header insertion that occurs 
anyway.

Which is to say: the cost of inserting TLV headers will correspond to the 
number of headers inserted, not to the number of bytes in the actual data 
stream, from an implementation perspective.  The frequency of TLV headers, of 
course, might well scale with the number of bytes in the file...

> By using a TLV encoding of the data, you would need to add some information 
> in front of the data on the sender side. This might require moving data in 
> memory depending on the size of the chunks that you consider and the size of 
> the buffer passed through the send system call. On the receiver side, this 
> is possibly worse as the receiver has to parse the received bytestream to 
> extract the TLV information and remove the TLV encoding before passing the 
> information to the userspace. This again potentialle requires touching all 
> the received data.

The parsing problem on the recipient end, on the other hand, seems like a more 
critical problem.  However, is much "parsing" actually needed in a 
walk-the-bytes sense?  If intermediate headers contains relative offset 
information to point forward through the stream, it shouldn't be necessary to 
walk each individual byte (i.e., pulling in each cache line) searching for the 
next header.  You would need to walk the meta-data describing the segment 
chain, however.

The in-band TLV does worry me as well, especially because it sounds stateless 
offload-unfriendly, but it seems for slightly different reasons.

Robert