Subject: Re: [multipathtcp] MPTCP connection termination

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 1 Nov 2011 02:54:26 -0700

Hi Olivier,

Sorry for picky questions, but could you elaborate the following points?

1: I'm a bit confused which side performs active close. Since you mentioned
    about avoiding CLOSE_WAIT state, it seems to be that an active opener
    performs active close. Am I right?
    This seems to mean a passive opener always needs to wait for active
    close from the peer to use this method. But, I'm guessing you might want
    to reset the connection without waiting for FIN in some cases..

2: How do mptcp stacks know I'm a passive opener?
    Does this mean implementations need to have a new parameter for this?

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/10/31 Olivier Bonaventure <[email-masked]>:
> Hello,
>
> During the last telco, we discussed briefly about the problem of
> terminating an MPTCP connection.
>
> Regular TCP supports two mechanisms to terminate a connection :
> - graceful release with the utilisation of the FIN flags
> - abrupt release with the utilisation of the RST flag
>
> The current MPTCP draft supports the graceful release with the
> utilisation of the DATA_FIN flags, but there is no equivalent of an
> abrupt release since the reception of a segment with the RST flag on a
> subflow only causes this particular subflow to be released and not the
> entire MPTCP connection.
>
> There are some applications, notably busy web servers, that abuse the
> utilisation of the RST flag in order to avoid entering in the CLOSE_WAIT
> state. A typical TCP connection with such as web server would end with a
> RST after all data sent by the server has been acknowledged by the client.
>
> MPTCP servers could probably also benefit from the ability to be able to
> release an MPTCP connection (and the underlying subflows) without having
> to maintain state in CLOSE_WAIT. The design of MPTCP is an opportunity
> to solve this problem in a cleaner way than by abusing with the RST flag.
>
> Our objective with this proposed modification is to allow the passive
> opener (i.e. the host that received the first SYN segment) to close the
> MPTCP connection without having to maintain any CLOSE_WAIT state. We
> only allow the passive opener is allowed to close the connection this
> way to avoid having to introduce a more complex state machine to deal
> with collisions when the two hosts try to close the MPTCP connection at
> the same time. We believe that avoiding CLOSE_WAIT is mainly of benefit
> for the passive opener.
>
> The proposed solution relies on an additional bit, let us call it the R
> bit in the DSS option :
>
>                          1                   2                   3
>      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>     +---------------+---------------+-------+----------------------+
>     |     Kind      |    Length     |Subtype| (reserved) |F|m|M|a|A|
>     +---------------+---------------+-------+----------------------+
>     |           Data ACK (4 or 8 octets, depending on flags)       |
>     +--------------------------------------------------------------+
>     |   Data Sequence Number (4 or 8 octets, depending on flags)   |
>     +--------------------------------------------------------------+
>     |              Subflow Sequence Number (4 octets)              |
>     +-------------------------------+------------------------------+
>     |  Data-level Length (2 octets) |      Checksum (2 octets)     |
>     +-------------------------------+------------------------------+
>
>                Figure 9: Data Sequence Signal (DSS) option
>
> The R bit can only be set when the F bit is set. If the R bit is clear,
> then the DSS option that contains the F bit is a regular DATA_FIN that
> is processed as explained in the current draft.
>
> When sending a DSS with the F bit set, the passive opener, and only the
> passive opener, can add the R bit to indicate that it requests the MPTCP
> connection to be released but it does not want to maintain any
> CLOSE_WAIT state. The DSS with both bits F and R set contains the
> following information :
>  - the last data sequence number that was sent by the passive opener.
> The DATA_FIN+R indicates that the seerver will not send data beyond this
> data sequence number
> - the data acknowledgement number that corresponds to the last data that
> was correctly received in sequence. The transmission of the DATA_FIN+R
> indicates that the server will not accept new data from the client
> beyond this data acknowledgement number
>
> Upon reception of a DATA_FIN+R, the client should stop sending any new
> data. When the DATA_FIN+R is acknowledged, the client should enter a
> CLOSE_WAIT for the MPTCP connection and the server can send a RST on all
> subflows to allow middleboxes to remove subflow state.
>
>
> Note that this proposal is slightly different from the current
> utilisation of RST on web servers. With regular TCP, web servers would
> send all data, wait for the last ack and then send a RST to close the
> connection. With DATA_FIN+R, a MPTCP server could send all data and then
> immediately send a DATA_FIN+R after the last data. The reception of the
> ack for the DATA_FIN+R will indicate that the data has been received
> correctly and the MPTCP connection can be closed (and the RST will be
> sent only to remove state in the middleboxes).
>
>
> Let me know whether you agree with this proposed approach
>
>
>
> Olivier
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 02 Nov 2011 16:09:49 +0100

Georg,

> I think the introduction of a MP RST flag makes perfect sense. 

Note that the proposal is not simply to mimic TCP RST for MPTCP.
Originally, TCP RST has been introduced to allow a TCP host to send a
RST segment upon reception of a segment belonging to a connection for
which it does not maintain any state. Then, TCP RST has been abused by
allowing hosts to send a TCP RST segment to teardown a connection
abruptly, but the host had state for the connection when it sent the TCP
RST.

With MPTCP, we do not need the first usage. When a TCP segment is
received on a subflow for which there is no state, a classical TCP RST
is sufficient to indicate that there is no state for the subflow. Note
that since the MPTCP segment does not contain the token except the SYN
segments, it is impossible to consider a MPTCP RST in this case.

The proposal that I sent tries to propose a cleaner solution for the
second (ab)use of the RST. The solution is cleaner because it ensures
that all data has been delivered reliably before the connection is closed.

> However, I wouldn't restrict this flag to the passive opener since such a restriction relies on certain assumptions about the application type (i.e. client-server content retrieval). In P2P applications, for instance, either side of the connection may decide to "hang up". Also for content upload, the client could the one who sends MP RST. Even for client-based content retrieval, MP RST could be sent by the client () when the user forcefully interrupts the application.

The motivation for restricting the utilisation to the passive opener is
to avoid a complex state machine when both the active and the passive
opener decide to release the connection without wanting to maintain state.

> I'm not certain I fully understand the complexity of R+F-flag signaling. Why can't the MP RST signaling follow along the same lines as TCP RST for conventional TCP connections? MP RST by itself would simply mean: "Tear down this MP connection" and the receiver would act upon it in any state. What am I missing here?

What you are proposing is to tear down the MPTCP connection without
trying to ensure that data has been delivered reliably. This is another
design point, but there is a subtle problem with MPTCP there.

Consider two hosts, A and B with one MPTCP connection composed of two
subflows. Both hosts have state for the MPTCP connection and A wants to
abruptly release the connection. Let's assume that we have the
equivalent of an MPTCP-level RST and that A sends this MPTCP RST on both
subflows and then immediately removes state. If the two MPTCP RSTs are
lost, they will never reach B and B will continue to have state for the
MPTCP connection. If B sends a segment to A, it will receive a TCP RST
for the corresponding subflow. This RST segment will cause the subflow
to be released but not the MPTCP connection and in fact B might be
trying to open another subflow to recover from the failed subflow. B's
state will only be removed after timeouts and several attempts to
reestablish subflows.

The only way for host A to avoid this problem is to indicate to B that
it wants to remove state and wait for an acknowledgement from B. The
DATA_FIN+R allows to do this without having to add complexity with both
A and B want to release the MPTCP connection at the same time.

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 3 Nov 2011 08:50:27 -0500

Olivier,

We agree that only "RST-abuse" cases have to be considered.

Before diving into implementation constraints, I'd like to understand what functionality we need to support. I think we agree on a scenario where one side of the MPTCP connection wants to preemptively tear down MPTCP-related state.

Your example shows why MP RST needs to be ACKed on connection level. In fact, the hosts need to follow through a proper 3-way handshake (RST -RST/ACK - ACK) which follows along the same lines as the FIN/ACK handshake. Also, the handshake initiator (host A) needs to remain in TIME_WAIT state for some time to be available if its last data ACK got lost and the peer (host B) retransmits the MP RST/ACK. 

The only difference between RST/ACK handshake and FIN/ACK handshake is that upon arrival of MP FIN, host B immediately terminates all data transmission/reception, responds with MP RST/ACK and goes into LAST ACK state, i.e. avoiding CLOSE WAIT state. In principle, host B could also respond with MP FIN/ACK instead of MP RST/ACK since it already knows that host A wants to shut down (there is no semantic difference between both messages).

I don't mind indicating MP RST with two flags, i.e. data RST + data FIN, instead of only data RST. However, I don't see any problem if MP RSTs are simultaneously sent by both hosts. In other words, either side of the connection should be able to initiate the RST handshake. Please explain. Thanks.

Georg

-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]] 
Sent: Wednesday, November 02, 2011 11:10 AM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Georg,

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 04 Nov 2011 18:10:43 +0100

Yoshifumi,
>     to reset the connection without waiting for FIN in some cases..

The proposal is to allow the passive opener (i.e. the host that received
the first SYN) to use DATA_FIN+RST. The active opener can still use
regular DATA_FIN to close the MPTCP connection.

>     Does this mean implementations need to have a new parameter for this?

You'd need one bit of state to remember that you received the first SYN
and did not send it.

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 04 Nov 2011 18:24:52 +0100

Georg,
> Before diving into implementation constraints, I'd like to understand what functionality we need to support. I think we agree on a scenario where one side of the MPTCP connection wants to preemptiv…

The question we need to ask is whether we want to preserve the reliable
delivery when performing this abrupt release or not. In the proposal
that I sent, I assume that the passive opener wants to ensure that its
data has been received correctly (i.e. all data up to and including
DATA_FIN). There are other designs that are possible.

> 
> Your example shows why MP RST needs to be ACKed on connection level. In fact, the hosts need to follow through a proper 3-way handshake (RST -RST/ACK - ACK) which follows along the same lines as the FIN/ACK handshake. 

This is another design point. Basically, when a host sends DATA_RST, it
indicates that it does not care about the in-flight data and wants to
release the entire MPTCP connection. This DATA_RST should contain some
information about the MPTCP connection being RST (e.g. the token that
identifies the MPTCP connection). Upon reception of a DATA_RST, the
receiving host should discard all data inside its buffers and return an
acknowledgement (DATA_RST_ACK) to confirm the reception of the DATA_RST.

The host that sent the DATA_RST_ACK must maintain state (say
DATA_RST_ACK state) for some time to be able to retransmit the
DATA_RST_ACK if needed.

The host that receives the DATA_RST_ACK could send a normal TCP RST on
all subflows to remove state in intermediate middleboxes.

The host that is in the DATA_RST_ACK state could leave this state once
all subflows have been reset.

> Also, the handshake initiator (host A) needs to remain in TIME_WAIT state for some time to be available if its last data ACK got lost and the peer (host B) retransmits the MP RST/ACK. 
> 

The difference is that DATA_FIN contains a dseq and allows to ensure
that all data have been reliably delivered. This is not the case with a
DATA_RST

> I don't mind indicating MP RST with two flags, i.e. data RST + data FIN, instead of only data RST. However, I don't see any problem if MP RSTs are simultaneously sent by both hosts. In other words,…

With your proposal, if both hosts sent DATA_RST, they should probably
both maintain state for some time in the DATA_RST_ACK state since they
will both send a DATA_RST_ACK

Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 7 Nov 2011 03:13:09 -0800

Hi Olivier,
Thanks for the response.

> regular DATA_FIN to close the MPTCP connection.

I might miss something, but I'm wondering how the state transition of this
proposal looks like. If active opener performs active close, I think the state
transition will be something like this.

                                 Active                            Passive
M_FINWAIT1        DATA_FIN    ->
  M_CLOSEWAIT
M_FINWAIT2                                           <- DATA_ACK
M_TIMEWAIT                                          <- DATA_FIN +R   M_LASTACK
                             DATA_ACK   ->
            M_CLOSE

If passive opener performs active close, I guess the state transition will
be something like this although I'm not very sure.

                                 Active                            Passive
M_CLOSEWAIT                                      <- DATA_FIN+R    M_FINWAIT1
M_TIMEWAIT        DATA_ACK->                                          M_CLOSE
(active opener doesn't send FIN in this case)

Does these state transitions match your proposal?
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 7 Nov 2011 13:43:35 -0600

Olivier, all,

I would prefer to have *one* MP RST solution to cover both use cases, i.e. (1) a webserver avoiding CLOSE_WAIT  and (2) a host forcefully aborting a connection.

I am proposing the following solution. Use case (2) is addressed first:

When host A wants to abort the connection it sends a DATA_RST message on all subflows, and it subsequently enters into the connection RST_WAIT state. When any of the following events occurs, host A transfers the connection to the CLOSED state and deletes all state right away:
-	Host A receives DATA_RST_ACK from host B
-	Host A receives TCP RST on all subflows
If neither of these events occurs, Host A retransmits DATA_RST.
 
When host B receives the DATA_RST message on any subflow, it responds with DATA_RST_ACK, enters CLOSED state and deletes all state associated with this connection right away. In case DATA_RST_ACK gets lost, host A will retransmit DATA_RST on all subflows and receive TCP RST on all subflows, which allows host A to go into CLOSED state right away.

If both hosts hang up at the same time, they both enter RST_WAIT state. When receiving DATA_RST from their peers they respond with DATA_RST_ACK and transition to CLOSED state right away.

Now to use case (1):
If host A (e.g. web server) sends DATA_RST together with DATA_FIN, host B won't respond with DATA_RST_ACK until it has received all data up to the DSN that is carried on the DATA_FIN. The rest goes as above.

Security: To avoid DoS it should be sufficient if SN and DSN of the DATA_RST-carrying packets are within the receive windows of subflow and data level, respectively.

This solution accomplishes quick connection tear down.

Georg

-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]] 
Sent: Friday, November 04, 2011 1:25 PM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Georg,

Olivier

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 9 Nov 2011 18:13:29 -0600

All,

The MPTCP-RST solution I proposed a few days can be simplified: We don't need a separate Data_RST_ACK but can use Data_RST instead.

Here how it works (may be a little clearer now):
a) Data_RST alone means: I want to tear down the connection right away assuming data loss.
b) Data_RST + Data_FIN + DSN means: I want to tear down as soon as I receive a Data_ACK to the enclosed DSN.

First to a):
When a host receives Data_RST in any connection state, it responds with Data_RST and tears down the connection right away (including all subflows). No retransmissions of Data_RST in this case.

When a host wants to tear down the connection while being in any connection state, it sends Data_RST on *all* subflows and goes into RST_WAIT state. 

The crucial idea is that a connection in RST_WAIT state can be torn down *either* through reception of a Data_RST *or* through TCP RSTs received on all subflows. Note that the latter *only* applies to RST_WAIT state and not to any other connection state. 

If neither of these two conditions is met within some timeout, the host retransmits Data_RST on all subflows, which will either provoke a Data_RST reply in case the peer's connection end point is still alive, or it will provoke TCP RSTs on all subflows. Either of them leads to immediate connection tear down. 

If both hosts simultaneously send Data_RST, they both have assumed RST_WAIT state and can tear down the connection as soon as they receive their respective peer's Data_RST.
 
Now to b):
When the host sends Data_RST together with Data_FIN, the peer will not respond with Data_RST until it has received all data up to the DSN contained in the Data_RST + Data_FIN packet.

Security: To mitigate vulnerability to DoS attacks, Data_RST should always include a DSN, which should lie within the data receive window. The subflow SN should also lie within the subflow receive window.

Comments are welcome. Thanks.

Regards,
Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Hampel, K Georg (K Georg)
Sent: Monday, November 07, 2011 2:44 PM
To: [email-masked]
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Olivier, all,

 

Now to use case (1):

Georg

-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]] 
Sent: Friday, November 04, 2011 1:25 PM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Georg,

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 10 Nov 2011 09:29:45 +0100

Georg,

> If neither of these events occurs, Host A retransmits DATA_RST.

If A sends DATA_RST on all subflows, should B reply on all subflows as 
well ? If yes, we need to specify how A should react when it receives a 
TCP segment that contains a DATA_RST_ACK but for which it has no state.
One possibility could be to place the DATA_RST_ACK inside a TCP RST segment.

> If host A (e.g. web server) sends DATA_RST together with DATA_FIN, host B won't respond with DATA_RST_ACK until it has received all data up to the DSN that is carried on the DATA_FIN. The rest goes…

This means that the semantics of DATA_RST differs depending on whether 
it is used together with DATA_FIN or not. I'd need to think a bit more 
about this.

> Security: To avoid DoS it should be sufficient if SN and DSN of the DATA_RST-carrying packets are within the receive windows of subflow and data level, respectively.

This should work.

Olivier

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 10 Nov 2011 08:36:53 -0600

Olivier,

Comments inside. Thanks.

Georg

[Comment GH]: I clarified the DATA_RST_ACK issue in my last email: We should replace DATA_RST_ACK with DATA_RST. That would eliminate the issue you're referring to. It also addresses situations where DATA_RSTs cross.
Host A must *retransmit* DATA_RST on *all* subflows to provoke TCP_RST on all subflows in response in case host B received host A's prior DATA_RST and already closed down. All other messages, i.e. host B's response and also host A's initial DATA_RST can be sent on any number of subflows. However, to make things simple and to also deal with border cases (e.g. DATA_RSTs cross) we should require that DATA_RSTs are always sent on all subflows.
One more thing: "All subflows" refers to "all subflows in synchronized state". Host A can send TCP RST on all other subflows right away. I will send out a separate email on this detail.

[GH comment] That's probably true.

This should work.

Olivier

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 10 Nov 2011 08:50:49 -0600

All,

We need to make a refinement for the host that is in or entering RST_WAIT state:

- The host sends Data_RST *only* on those subflows that are in synchronized state (but on all of those). It sends TCP RST on all other subflows and tears them down right away.

- The host tears down the connection if either Data_RST is received or no subflow is left in synchronized state.

There is only one ambiguous situation: If the host enters RST_WAIT during BBM when there's no subflow in synchronized state, it tears down the connection without informing its peer. 

Regards,

Georg

-----Original Message-----
From: Hampel, K Georg (K Georg) 
Sent: Wednesday, November 09, 2011 7:13 PM
To: Hampel, K Georg (K Georg); [email-masked]
Cc: multipathtcp
Subject: RE: [multipathtcp] MPTCP connection termination

All,

First to a):

 
Now to b):

Regards,
Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Hampel, K Georg (K Georg)
Sent: Monday, November 07, 2011 2:44 PM
To: [email-masked]
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Olivier, all,

 

Now to use case (1):

Georg

-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]] 
Sent: Friday, November 04, 2011 1:25 PM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Georg,

Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 13 Nov 2011 12:50:34 -0800

Hi Gorge,

I'm not sure if I understand Olivier's proposal well enough,
but, I think it has the following key points..

1: MPTCP RST should not make both endpoints wait too long.
   However, in mptcp, if an endpoint terminates connection
   without telling the peer, the peer has to wait for long time.
   Because once an endpoint terminates, all it can send
   is TCP RST since no mptcp state remains.
   From this reason, MPTCP RST has to be acked.

2: The endpoint that receives MPTCP_RST is needed to go
   M_TIMEWAIT state.
   When this endpoint send an ACK for MPTCP_RST, there's a
   possibility that it is lost. To confirm the ACK
   arrives at the peer successfully, the endpoint has to
   wait for 2MSL. If the ACK is lost, it will receive
   retransmitted MPTCP_RST from the peer.

I think I don't understand your proposal fully yet, but
I guess there'll be a case where an endpoint terminates
connection while the other endpoint doesn't know it.
(e.g. When both endpoints send DATA_RST and one of them is lost)
Is this OK in your proposal?

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 14 Nov 2011 12:36:14 -0600

Hi Yoshifumi,

Your points 1) and 2) summarize the issues very well. My proposal tries to cover the situations where data_RST messages get lost, or where they cross. 

The key question is: What is the fastest way to completely tear down a MPTCP connection so that NEITHER side holds any further connection-related state.

Host A could send TCP RST on all subflows. This, however, would NOT tear down the connection on host B (!).

The core idea of my proposal is to introduce a new state in the MPTCP statemachine (referred to as RST_WAIT), where "no more synchronized subflows" means "tear down connection". 

Further, reception of data_RST means: 
(1) If not in RST_WAIT, send TCP RST on all subflows and tear down the connection.
(2) If in RST_WAIT, tear down the connection.

Here's the flow:

When host A wants to tear down the connection, it sends Data_RST to host B and enters RST_WAIT. From now on, receiving either Data_RST or TCP RSTs on all synchronized subflows means that host A can terminate the connection. 

When host B receives Data_RST, it sends TCP RSTs on all subflows and immediately tears down the connection.

When host A receives the TCP RSTs on all subflows or the data_RST, it can also tear down the connection since it is in RST_WAIT state.

Delivery failure:

In case host B does not receive the initial data_RST message, host A won't see the desired response and it will retransmit data_RST on all synchronized subflows. 

In case host B does receive the initial message and it has already torn down the connection, the DATA_RST retransmissions will automatically provoke TCP RSTs on all subflows and host A will consequently tear down the connection as well.

Message crossing:
In case both hosts want to terminate at the same time, they both send Data_RSTs and they both are in RST_WAIT state. Consequently, each host can tear down its connection as soon as it receives the Data_RST from its peer. 

Is there any scenario I have missed? Thanks.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Sunday, November 13, 2011 3:51 PM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Hi Gorge,

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 22 Nov 2011 10:15:16 -0600

Olivier, all,

I found a nice link to the busy-webserver problem by T Faber et al:

http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/

After reading this doc, I'm a little puzzled on how web servers currently "fix" their TIME_WAIT problem by sending RST (?!). 

- Georg

-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]] 
Sent: Thursday, November 10, 2011 3:30 AM
To: Hampel, K Georg (K Georg)
Cc: multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Georg,

This should work.

Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 28 Nov 2011 02:23:10 -0800

Hi Georg, all,

Sorry for my slow response. I'm now back from vacation.
My little concern for this is it relies on a little implicit signaling
compared to using
acks for data_RST.
I think there'll be the following corner cases, although they might be
minor points.
and might be solved with some additional logics.

Corner case 1:

  Host A sent data_RST, but it was lost and didn't reach Host B.
  At the same time, Host B sent TCP RST to all connections in order to
switch networks.

  In this case, Host A considers data_RST arrived at Host B since it
received TCP RSTs,
  although this is not true.

Corner case 2:

  Host A sent data RST and Host B received it.
  Host B sent TCP RST on all subflows and tore down the connection, however
  one of paths somehow discarded all packets from both ends.

  In this case, Host A continues to send data RST to wait for TCP RST.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 28 Nov 2011 10:04:52 -0600

Yoshifumi, Olivier,

I have inserted comments on top of Olivier's comments. Thanks.

Georg

Yoshifumi,
>   although this is not true.

At least they both consider the MPTCP to be closed and have removed
their state.

[GH] As I understand this scenario, Host B's MPTCP connection is still up while host A's connection is already closed. This, indeed, is a corner case where data_RST does not do proper connection teardown. In such a scenario, host B would hang in until some timeout has been exceeded. 
Here is why we should not be too concerned about this corner case:
1. The data_RST solution properly handles the vast majority of abrupt teardown scenarios. 
2. This remaining corner case is not handled worse WITH data_RST than it is handled WITHOUT data_RST. Further, the only penalty is that host B's connection-related state is released with some delay.
3. At the moment, there doesn't seem to be a solution in sight that would address this corner case any better.

>   one of paths somehow discarded all packets from both ends.

At this point, Host B has lost its state for the MPTCP connection
>   In this case, Host A continues to send data RST to wait for TCP RST.

If A sends a Data_RST, it will receive a RST from B and this will allow
A to close the connection. If A never receives the RST, then it will
have to wait for the expiration of a timer to remove its state.

[GH] If host A does not receive Data_RST reply or TCP RSTs on all synchronized subflows within a short timeframe (e.g. RTO ~ 2 sec), it will retransmit Data_RST. Since this retransmission will occur on all synchronized subflows, it will produce the TCP RSTs needed to tear down the connection.

Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 29 Nov 2011 03:45:36 -0800

Hi Georg, Olivier,

Thanks for the response. I basically agree with your comments.
The corner case 1 might be a minor issue compared to the benefits.

As Olivier mentioned, the corner case 2 can be solvable by using a timer to
address the case where no response comes back from the peer due to routing
failures or middilebox failures or something.
But, I think this means if one of paths becomes no-response status, the peer
will need to wait for a certain amount of time. I'm wondering that this would be
acceptable or not..

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 29 Nov 2011 11:37:07 -0600

Hi Yoshifumi,

In case the host doesn't hear anything from its peer for an extended period of time, I think MPTCP should react in the same way as TCP.

How does TCP react under such circumstances? To the best of my knowledge, this depends on if keepalives are set or not. If not, TCP keeps hanging until the app is closed down.

Regards,
Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Tuesday, November 29, 2011 6:46 AM
To: Hampel, K Georg (K Georg)
Cc: Olivier Bonaventure; multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Hi Georg, Olivier,

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 30 Nov 2011 03:17:50 -0800

Hi Georg,

Sorry, my point was not clear enough.
Please correct me if I'm wrong. In my understanding, your proposal
requires to receive
TCP RST from ALL subflows to shutdown a MPTCP session.

Let's say we try to reset a MPTCP session which has 10 subflows. In this case,
even if it successfully receives 9 TCP RSTs, it might have to wait for
one TCP RST
from a non-response-path, which might take long time.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 30 Nov 2011 11:14:11 -0600

Hi Yoshifumi,

We have to file out the details. One solution to your problem may be:

- MPTCP_RST initiator sends DATA_RST on all subflows.

- Upon reception of at least one DATA_RST, peer responds with DATA_RST and TCP RST on all subflows and closes down right away. (Should we bundle DATA_RST and TCP RST in this case? I'm not so sure.)

- Upon reception of at least on DATA_RST, MPTCP_RST initiator sends TCP RST on all subflows (to close middleboxes) and closes down connection.

This solution would also address the situation where both hosts initiate DATA_RST at the same time.

Does this look better? Thanks.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Wednesday, November 30, 2011 6:18 AM
To: Hampel, K Georg (K Georg)
Cc: Olivier Bonaventure; multipathtcp
Subject: Re: [multipathtcp] MPTCP connection termination

Hi Georg,

Thanks,
--
Yoshifumi Nishida
[email-masked]