Subject: [multipathtcp] Receiver's gap problem in MPTCP

From: Bachir CHIHANI <[email-masked]>
Date: Thu, 8 Jul 2010 08:02:15 -0700 (PDT)

Dear,

The use of multipath TCP create the gap problem, which consist of receiving packet in an out of sequence manner.

Can we add a multipath option to the segment header that enable the receiver to state what data level sequence number he's expecting whenever he receive an out of sequence packet.
We can then use Fast Retransmit at the data level to send the corresponding packet whener the sender receive three duplicated ACK.

Best regards.
---------------------------------------

Bachir CHIHANI.

Tél: +33 4 92 94 52 77

portable:6 48 62 32 65

---

From: <[email-masked]>
Date: Thu, 8 Jul 2010 16:07:55 +0100

Isn't that the existing sequence number?

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bachir CHIHANI
Sent: 08 July 2010 16:02
To: [email-masked]
Subject: [multipathtcp] Receiver's gap problem in MPTCP

Dear,

Best regards.

---

From: Bachir CHIHANI <[email-masked]>
Date: Thu, 8 Jul 2010 08:15:17 -0700 (PDT)

Dear, 

There isn't an ACK sequence number at the data level, but only at the subflow level which is the existing one in TCP segments.

Best regards.

Bachir CHIHANI.

--- En date de : Jeu 8.7.10, [email-masked] <[email-masked]> a écrit :

De: [email-masked] <[email-masked]>
Objet: RE: [multipathtcp] Receiver's gap problem in MPTCP
À: [email-masked], [email-masked]
Date: Jeudi 8 juillet 2010, 8h07

Isn't that the existing sequence 
number?

From: [email-masked] 
[mailto:[email-masked]] On Behalf Of Bachir 
CHIHANI
Sent: 08 July 2010 16:02
To: 
[email-masked]
Subject: [multipathtcp] Receiver's gap problem 
in MPTCP

  
    Dear,

The use of multipath TCP create the gap 
      problem, which consist of receiving packet in an out of sequence 
      manner.

Can we add a multipath option to the segment header that 
      enable the receiver to state what data level sequence number he's 
      expecting whenever he receive an out of sequence packet.
We can then 
      use Fast Retransmit at the data level to send the corresponding packet 
      whener the sender receive three duplicated ACK.

Best 
      regards.
Bachir 
      CHIHANI.
Tél: +33 4 92 94 52 77
portable:6 48 62 32 
65

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 8 Jul 2010 12:32:54 -0700

Hi Bachir,

The DATA_ACK option is not enough for this purpose?
--
Yoshifumi Nishida
[email-masked]

2010/7/8 Bachir CHIHANI <[email-masked]>

> Can we add a multipath option to the segment header that enable the
> receiver to state what data level sequence number he's expecting whenever he
> receive an out of sequence packet.
> We can then use Fast Retransmit at the data level to send the corresponding
> packet whener the sender receive three duplicated ACK.
>
> Best regards.
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>
>

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 8 Jul 2010 21:35:37 +0100

I don't think it's a good idea to do fast retransmit at data ack - you could easily mistake data level reordering (which is unavoidable for multipath TCP) with segment loss. 
The main reason fast retransmit works for regular TCP is the fact that reorderings are rare. 

One of the motivations for going to multiple sequence spaces was exactly this: being able to disambiguate at data level segment loss from reordering. The individual 
flows detect segment loss using their regular mechanisms, and the data level is only used for reordering. Because of middleboxes, we now use an explicit  data level ACK
to allow retransmission in certain failure cases with pro-active acking middleboxes. However, this is supposed to be very rare and will be triggered by coarse grain timers
rather than duplicate acks.

A discussion  of our design decisions is at http://www.cs.ucl.ac.uk/staff/C.Raiciu/files/mtcp-design.pdf

Cheers,
Costin

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 8 Jul 2010 16:18:19 -0700

I agree. I also think that we don't have to send data_ack whenever data
level seq is advanced. I feel it can be sent more infrequently.
--
Yoshifumi Nishida
[email-masked]

2010/7/8 Costin Raiciu <[email-masked]>

> I don't think it's a good idea to do fast retransmit at data ack - you
> could easily mistake data level reordering (which is unavoidable for
> multipath TCP) with segment loss.
> One of the motivations for going to multiple sequence spaces was exactly
> this: being able to disambiguate at data level segment loss from reordering.
> The individual
> flows detect segment loss using their regular mechanisms, and the data
> level is only used for reordering. Because of middleboxes, we now use an
> explicit  data level ACK
> to allow retransmission in certain failure cases with pro-active acking
> middleboxes. However, this is supposed to be very rare and will be triggered
> by coarse grain timers
> A discussion  of our design decisions is at
> http://www.cs.ucl.ac.uk/staff/C.Raiciu/files/mtcp-design.pdf
>
> Cheers,
> Costin
>
> > The use of multipath TCP create the gap problem, which consist of
> receiving packet in an out of sequence manner.
> >
> > We can then use Fast Retransmit at the data level to send the
> corresponding packet whener the sender receive three duplicated ACK.
> >
> > Best regards.

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 9 Jul 2010 00:30:21 +0100

The reason for doing that is to save send buffer space. You have to buffer the data at the sender until you get a data ack covering the data.

Cheers,
Costin

On 9 Jul 2010, at 00:18, Yoshifumi Nishida wrote:

> I agree. I also think that we don't have to send data_ack whenever data level seq is advanced. I feel it can be sent more infrequently.
> --
> Yoshifumi Nishida
> [email-masked]
>

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 9 Jul 2010 08:05:10 +0200

Hi,

here a question comes into my mind: Apart from pro-active acking middleboxes, is there any other benefit of having separate data ACKs?

Pro-active acking middleboxes, if they should exist, break the end-to-end semantics of today's TCP over that path anyway. Solving that issue seems to be somehow orthogonal to multipath transport.

IMHO applications that do care about such failures use an app-level reliability mechanism already today, and a data ACK would not provide much additional benefit then. Or do I miss something?

(Of course, we might need a kind of ACK mechanism for working around middleboxes that modify the byte stream. But for this a simple failure notification instead of frequent ACKs may be sufficient.)

Michael

-----Ursprüngliche Nachricht-----
Von: [email-masked] im Auftrag von Costin Raiciu
Gesendet: Do 08.07.2010 22:35
An: Yoshifumi Nishida
Cc: [email-masked]; Bachir CHIHANI
Betreff: Re: [multipathtcp] Receiver's gap problem in MPTCP

Cheers,
Costin

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 9 Jul 2010 01:22:17 -0700

Yep, so It's trade-off between socket buffer overhead and network overhead.
It might be ok even if we don't send data ack until data level seq is
advanced by a certain amount? (say 4-8 MSS?)

Thanks,
--
Yoshifumi Nishida
[email-masked]

> The reason for doing that is to save send buffer space. You have to buffer
> the data at the sender until you get a data ack covering the data.
>
> Cheers,
> Costin
>

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 9 Jul 2010 09:32:40 +0100

Hi Michael,

Multipath TCP is supposed to help with client mobility - e.g. mobile phones. This is one of the better deployment scenarios. Most of these are on flakey wireless nets, and there are middleboxes on the flow. Some of these break the TCP flow to speed it up; they ack locally - before getting an ACK from the receiver. If the ACK never comes or is delayed (say a a few timeouts) the whole connection will break/suffer.  How often this is remains to be determined. 
This is one problem where the Data ACK can help, but in general explicit signalling is better than implicit - so I would bet that there will be other problems the data ACK solves.

Our latest examination of the payload encoding made us realize that it is NOT possible to use the Data ACK in payload; the reason is the possibility of deadlock: the receive window will control the control stream too, not only the data. This is fine for data mapping, but is not fine for data ACK - which is essential - if used - to free send buffer space.

To conclude, I'd favor using the data ACK if we do options. If we do payload we have to abandon it.

Cheers,
Costin

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 9 Jul 2010 11:18:31 +0200

Hi Costin,

> Multipath TCP is supposed to help with client mobility - e.g. 
> mobile phones. This is one of the better deployment 
> scenarios. Most of these are on flakey wireless nets, and 
> there are middleboxes on the flow. Some of these break the 
> TCP flow to speed it up; they ack locally - before getting an 
> ACK from the receiver. If the ACK never comes or is delayed 
> (say a a few timeouts) the whole connection will 
> break/suffer.  How often this is remains to be determined. 

as I said, this problem exists (or does not exist) for today's TCP over
such middleboxes as well, and as the problem is well-known, applications
should already have worked around it, if they care about reliability. A
TCP connection has never been 100% reliable. I still think that it is
actually orthogonal to multipath transport.

BTW, what would prevent a middlebox from proactively acking MPTCP data
ACKs?

If proactive acking in wireless networks is a relevant problem, let's
solve it for single-path TCP, too, as multipath transport is anyway a
corner case. Specifying an additional ack option for TCP would be pretty
straightforward, and it would be of added value to all single-path TCP
connections that suffer from proacking middleboxes - until those boxes
learn to process that option as well.

> This is one problem where the Data ACK can help, but in 
> general explicit signalling is better than implicit - so I 
> would bet that there will be other problems the data ACK solves.

I think that we have to be sure whether it would solve a relevant
problem.
 
> Our latest examination of the payload encoding made us 
> realize that it is NOT possible to use the Data ACK in 
> payload; the reason is the possibility of deadlock: the 
> receive window will control the control stream too, not only 
> the data. This is fine for data mapping, but is not fine for 
> data ACK - which is essential - if used - to free send buffer space.

I don't understand why there should be such a deadlock. Could you please
elaborate on it?

> If we do payload we have to abandon it.

I personally think that we don't need frequent data ACKs in neither of
the two cases. And IMHO we could realize them in the payload as well if
it was really a mandatory feature.

Michael

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 9 Jul 2010 10:38:35 +0100

Hey,

The difference between multipath and regular TCP is that multipath should be able to continue with one path failing. Our aim is to isolate performance problems on the path with problems, and not bring the whole connection down.
In TCP there is no route around the middlebox, so whether you know what it is doing or not is irrelevant.

As for the deadlock, think of this scenario, with hosts A and B sending to each other in the same connection.
Say the app on A does not read from its buffer, and its receive window closes. B is receiving data and sending back data ACKs; his flow of data to A stops at some point because of the zero receive window. At this point, B cannot send data ACKs - as these occupy window! So A will assume its segments are missing, retransmit them repeatedly and eventually time out the connection.

In regular TCP the flow A->B is unaffected by A's receive window. In Multipath TCP with payload encoding and data ACK, the flow just dies.

Cheers,
Costin

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 9 Jul 2010 12:23:03 +0200

Hi Costin, 

> The difference between multipath and regular TCP is that 
> multipath should be able to continue with one path failing. 
> Our aim is to isolate performance problems on the path with 
> problems, and not bring the whole connection down.
> In TCP there is no route around the middlebox, so whether you 
> know what it is doing or not is irrelevant.

is there a performance problem that you cannot detect with a NACK
mechanism at the receiver?

Let's assume for the moment that the NACKs are transported reliably, and
that the sender can implicitly decide how much data to buffer.

> As for the deadlock, think of this scenario, with hosts A and 
> B sending to each other in the same connection.
> Say the app on A does not read from its buffer, and its 
> receive window closes. B is receiving data and sending back 
> data ACKs; his flow of data to A stops at some point because 
> of the zero receive window. At this point, B cannot send data 
> ACKs - as these occupy window! So A will assume its segments 
> are missing, retransmit them repeatedly and eventually time 
> out the connection.
> 
> In regular TCP the flow A->B is unaffected by A's receive 
> window. In Multipath TCP with payload encoding and data ACK, 
> the flow just dies.

The root cause of this problem seems to be that the app on A does not
read from the buffer, right?

In case of MCTCP, it is not the app that reads the socket buffer, but
the MCTCP layer. The MCTCP layer will always be able to read at least
small amounts of data from the socket as long as the end system has free
memory, and if this is not the case, it will probably crash very soon
anyway. Thus, don't see an issue here. Or do I miss something?

We could also do lots of other tricks, such as mandating a MPTCP
receiver to announce non-zero windows from time to time, etc. We already
mandate that MPTCP shall not be enabled on a system that is short of
memory... If so, is this situation relevant?

Apart from that, this potential deadlock situation is not specific to
multipath, right? It seems to apply to all application layer protocols
that try to use an own acknowledgment mechanism on top of today's TCP.
So, you're basically saying that using a windowing mechanism on top of
TCP cannot work at all?

Michael

> scenarios. Most 
> >> of these are on flakey wireless nets, and there are middleboxes on 
> >> the flow. Some of these break the TCP flow to speed it up; 
> they ack 
> >> locally - before getting an ACK from the receiver. If the 
> ACK never 
> >> comes or is delayed (say a a few timeouts) the whole 
> connection will 
> > as I said, this problem exists (or does not exist) for today's TCP 
> > over such middleboxes as well, and as the problem is well-known, 
> > applications should already have worked around it, if they 
> care about 
> > reliability. A TCP connection has never been 100% reliable. I still 
> > think that it is actually orthogonal to multipath transport.
> > 
> > corner case. Specifying an additional ack option for TCP would be 
> > pretty straightforward, and it would be of added value to all 
> > single-path TCP connections that suffer from proacking 
> middleboxes - 
> > until those boxes learn to process that option as well.
> > 
> >> This is one problem where the Data ACK can help, but in general 
> >> explicit signalling is better than implicit - so I would bet that 
> >> there will be other problems the data ACK solves.
> > 
> realize that 
> >> it is NOT possible to use the Data ACK in payload; the 
> reason is the 
> >> possibility of deadlock: the receive window will control 
> the control 
> >> stream too, not only the data. This is fine for data 
> mapping, but is 
> >> not fine for data ACK - which is essential - if used - to 
> free send 
> >> buffer space.
> > 
> > I don't understand why there should be such a deadlock. Could you 
> > please elaborate on it?
> > 
> > the two cases. And IMHO we could realize them in the 
> payload as well 
> > if it was really a mandatory feature.
> > 
> > Michael
> 
>