Subject: [multipathtcp] Why payload transport of Data ACKs is actually possible

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 29 Jul 2010 11:11:31 +0200

Dear all,

I thought about payload transport of data ACKs again, and in particular
about the examples showed by Costin on Tuesday. Unless I miss something,
I don't think that the severe performance problem shown for TCP-over-TCP
with reverse traffic exists for MPTCP with data ACKs in the payload.

The fact that a comparison of MPTCP with TCP-over-TCP is inappropriate
should actually have already been pointed out during the session on
Tuesday.

Here is some explanation of the difference between MPTCP and
TCP-over-TCP, as far as I understand it right now:

* TCP-over-TCP uses the inner ACKs for data release at the sender,
retransmissions, flow control _and_ congestion control. Most
importantly, on the inner connection, retransmissions due to RTOs do
trigger the inner congestion control. This is likely to occur with
reliable transport on the outer congestion, and small RTOs values in the
inner connection.

* In contrast, to my understanding, multipath transport with data ACKs
in the payload use data ACKs only for data release at the sender,
retransmissions over different subflows, and flow control. An RTOs on
data ACKs would probably cause a retransmission over another subflow,
but this does _not_ trigger congestion control.

This is a very fundamental difference.

Therefore we have to be very careful with a comparison of TCP-over-TCP
and data ACK payload transport in multipath transport. I expect that the
performance degradation in Costin's TCP-over-TCP measurements mostly
comes from the inner congestion control loop. Due to reliable delivery,
the inner loop is likely to suffer from RTOs, which triggers
Slow-Starts. This cannot occur in case of multipath transport. Unlike
TCP-over-TCP, payload encoding of data ACKs would not suffer from
problems with an inner congestion control loop. This is why the
comparison is misleading.

If the sender is neither limited by the receive buffer, nor by a
connection-level flow control, and if the paths are operational, a
potential delay of data ACKs due to reliable, in-order delivery does not
affect the overall throughput at all. It only affects the memory
constraints in the sender and the receiver. The only difference to
unreliable data delivery is that reliable, in-order delivery of data
ACKs requires larger buffers. I want to point out once again that IMHO
in many use cases of multipath transport, memory is not a major
constraint. A GB of memory is today very cheap.

Obviously, there is a head-of-line blocking issue if the sender is
constrainted by session-level flow control, and in this case unreliable
delivery would indeed be better, and also architectureally cleaner. But
in this case it is rather likely that a sender cannot make use of the
aggregate bandwidth on all paths anyway. The root cause is the
insufficient buffer dimensioning, not payload transport of data ACKs.

In other words: If sender and receiver have sufficiently large send and
receive buffers, and if paths are stable, payload transport of data ACKs
should not have any performance penalty. Or do I miss something?

I would offer to provide performance measurements for a MCTCP variant
with payload encoded data ACKs in a future meeting, to proof that
head-of-line blocking is not a major problem if buffers are reasonably
dimensioned.

Michael

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 29 Jul 2010 11:53:32 +0200

Costin,

> Now the data ACKs themselves, as you say, will be reliably 
> transmitted, are subfject to Head of Line blocking, 
> congestion control and flow control. They can be delayed 
> quite a bit (one timeout = 200ms or 1s according to the 
> spec); for a flow running at 100Mbps, the buffer needed is 
> 2.5MB/10MB. This will block the connection despite the cc 
> loop not being included.  

as I pointed out before, this is an issue of proper buffer dimensioning,
but not a protocol design problem. We will need buffers much larger than
10MB anyway. Then all what you explain is about a hypothetical problem
based on the assumption of buffer space scarcity.

> In general, people have made TCP over TCP work in certain 
> cases - as far as I understand, this is mostly through 
> overprovisioning the networks. I am sure that payload data 
> ACKs will work well in certain scenarios. I am also sure 
> there will always be cases where all these problems seriously 
> hurt performance, and there's not much we can do about it. 

IMHO the solution to the data ACK issues is pretty simple: Use buffer
overdimensioning. This a good idea for other reasons anyway.

Could you please describe a scenario with buffer overprovisioning where
payload transport of data ACKs hurts, as you claim?

Michael

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 29 Jul 2010 13:34:22 +0200

On 29 jul 2010, at 11:11, SCHARF, Michael wrote:

> affect the overall throughput at all.

Granted that under these highly unrealistic assumptions the data keeps flowing from the send buffer on the sender to the receive buffer on the receiver. But the application doesn't receive any data until the missing data has been retransmitted.

This can be a long wait if one of the subflows died in the middle of a transmission.

As I said before, this is a complete non-starter as it makes MPTCP less reliable than regular TCP.

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 29 Jul 2010 13:51:34 +0200

[Forgot to send this to the list when I replied to Michael. His reply refers to this email]

Hi Michael,

I did not imply that multipath TCP and tcp over tcp are the same thing. That's why I setup an experiment (following Jana's suggestion) where only the data ACKs are tunelled over tcp, and everything else is in the clear. The results were the same as presented.

I agree to the difference between Data ACKs and regular ACKs: data ACKs are not used for congestion control. 

However, as you say, they are used to release data at the sender. That means if they don't arrive, the send buffer will fill up and the sender cannot send anymore - it is waiting for data ACKs.
Now the data ACKs themselves, as you say, will be reliably transmitted, are subfject to Head of Line blocking, congestion control and flow control. They can be delayed quite a bit (one timeout = 200ms or 1s according to the spec); for a flow running at 100Mbps, the buffer needed is 2.5MB/10MB. This will block the connection despite the cc loop not being included.  

Hence, it's not clear to me that the interaction with the forward congestion control loop is all that matters. 

In general, people have made TCP over TCP work in certain cases - as far as I understand, this is mostly through overprovisioning the networks. I am sure that payload data ACKs will work well in certain scenarios. I am also sure there will always be cases where all these problems seriously hurt performance, and there's not much we can do about it. 

Cheers,
Costin

On 29 Jul 2010, at 11:11, SCHARF, Michael wrote:

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 29 Jul 2010 13:58:26 +0200

> based on the assumption of buffer space scarcity.

10MB for multipath vs ~100KB for single path TCP today is two orders of magnitude more. I don't think this will go down well with OS vendors. RAM is large, but finite. NetApp can probably do this in an app level protocol for one app, but if everybody uses this instead of TCP, things will get a lot worse.

> payload transport of data ACKs hurts, as you claim?

What you've done with overprovisioning is to solve only a part of the problem. There will be connection level timers based on data ack moving forward. How do you set these when your ack stream has frequent and possibly long lived "hickups"? I guess the only answer has to be, you set some huge values to cover this case. But of course, when things do fail you will take on the order of seconds to detect and use another path. If links are coming and going as when you move between wireless networks, this solution will perform much much worse. 

Cheers,
Costin

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 29 Jul 2010 14:26:26 +0200

> What you've done with overprovisioning is to solve only a 
> part of the problem. There will be connection level timers 
> based on data ack moving forward. How do you set these when 
> your ack stream has frequent and possibly long lived 
> "hickups"? I guess the only answer has to be, you set some 
> huge values to cover this case. But of course, when things do 

Yes, agreed, the connection-level timers must be larger than the RTO of
TCP to avoid interactions with TCP. But I don't think that this is an
issue, as a timeout on the data ACK is only the last remedy. I'd
personally set it to a value of the order of at least 10 sec, with
exponential backoff.

Note that I would use a Data NACK (or SACK) so that a receiver can
request for the retransmission of content on another subflow if it
observes that a subflow stalls (e. g., due to head-of-line blocking on
that subflow), and this increases the likelyhood of a data ACK on
another subflow (in particular, if we use unidirectional subflows). In
other words, a connection-level timeout should actually seldomly be
required for retransmissions.

> fail you will take on the order of seconds to detect and use 

True, connection-level recovery can be of the order of seconds in some
cases. 

> another path. If links are coming and going as when you move 
> between wireless networks, this solution will perform much 
> much worse.

Well, if the assumption is that paths come and go on time scales of the
order of tens of seconds, any transport layer "mobility" mechanism is
challenged anyway. For instance, congestion control is then really
tricky, the sender has to go through Slow-Start very frequently, etc. In
these cases, there are a lot of other issues that we probably have not
solved so far.

Michael