Subject: [multipathtcp] A proposal for MPTCP Robust session Establishment (MPTCP RobE)

From: <[email-masked]>
Date: Mon, 10 Jul 2017 16:15:57 +0000

Hi all,

My name is Markus Amend from Deutsche Telekom Research & Innovation and I want to propose an idea how to make MPTCP robust against outages during establishment process of the initial flow. I want to start the discussion right now in advance to the IETF 99 meeting, where I will present this topic for discussion. It would be appreciated to receive some feedback in advance.

--- Let's start with a short motivation ---

A proposal for MPTCP Robust session Establishment (MPTCP RobE), related to the real world problem: What happens if the default gateway (GW) does not provide end-to-end connectivity, but an additional route is available? 

I guess, all of you already faced the problem with your smartphone (or any other device with more than one interface), connected to a bad WiFi, where you manually switched off the WiFi interface to have at least connectivity or a more stable connection over cellular network. This happens, because your smartphone decides for you to use the cheaper way, the WiFi way, but it doesn't take care about reachability and stability. So, whenever a physical WiFi connectivity is given, the smartphone puts the default GW to it. In the worst case, the WiFi is not able to establish an end-to-end connectivity and it will fail, even if a proper cellular connectivity is available at the same time. The problem is not just related to smartphone implementations, it is related to any device with more than one interface available!

MPTCP implementations struggle with this as well and even more aggravated, because MPTCP promises apparently redundancy if more the one path to destination is available. But this isn't the case all the time, why?

MPTCP relies on a concept of an initial flow, which has to be established first, to negotiate MP capability and afterwards proceeds to establish subsequent flows to enable redundancy or even capacity aggregation. If the initial flow fails to establish a (MP-)TCP connection, MPTCP will never make use of other available paths and from an application view, there is no connection possible.

To summarize at this point again, first we need a proper initial flow establishment before subsequent flows can be established and exploit multipath capabilities. Also in a scenario where we have a lot of working paths available, but the path used for the initial flow (mostly the default GW, or more generic: default route) is unable to finish handshaking, we end in a zero connectivity scenario.

We demand:

"If there is at least one functional path, a connection must be possible"

With the idea of MPTCP Robust Establishment "MPTCP RobE" we try to solve or mitigate the challenge of blocked or unstable path during initial flow establishment and it would shift the full multipath capabilities to the very beginning of a MPTCP session.

--- stop motivation ---

--- start rough description ---

Over the last year we developed three proposals, to tackle the challenge. We tried to meet the following criteria:

1. Robustness: If there is at least one functional path, a connection must be possible

2. Overhead and latency: The solution should not introduce excessive amounts of overhead and latency compared to standard MPTCP 

3. Standard compliance: A solution should use and integrate with existing standards and work around limitations

In general, all proposals make use of redefining the initial flow towards a >potential< initial flow. That gives the freedom to use several potential initial flows at the very beginning of a session or shift the potential flows between different path. Technically it is based on duplicating the MP_CAPABLE option.

Following the proposals (which will be explained in detail first during the IETF 99 session), which all includes at least the first criteria:

1. "Downgrade" approach:
   Start on all available path potential initial flows, the first successful one remains as the initial flow the others are "downgraded" to subsequent ones.

2. "Break before Make" approach:
   Start on all available path potential initial flows, first successful one remains as the initial flow the others are reset. Standard MPTCP procedure follows to establish subsequent flows

3. "Timer" approach:
   One potential initial flow is started, if during a defined time slot no establishment becomes apparent, a separate potential initial flow is started on another path. 

Currently the "Downgrade" approach is preferred from our side, because it provides beside robustness the benefit of reducing handshaking time and sub flows are created simultaneously and not delayed by initial flow establishment -> The path providing the lowest latency determined the overall connection latency.
But there are still some minor drawbacks which needs to be solved. The solution of potential initial flows leads to the problem, that the SYN-Receiver cannot distinguish anymore between a valid Key A connection request or an invalid one, which is accidentally used by a foreign requester or deliberately by an attacker (e.g. brute force). MPTCP RobE tries to mitigate this challenge by reducing the time frame of accepting new SYN-Key A requests for a specific connection to the duration until the first initial flow is set up. But this is still not perfect and may be discussed in terms of security aspects. Also the question of how to handle Address IDs in the context of potential initial flows, needs to be discussed.

On the other hand the "Timer" approach can integrate with the existing standard best, but doesn't exploit the full potential of a possible MPTCP RobE standard/implementation and may be delay connection establishment for a long while. Whereas the "Break before Make" approach behaves like the first one but will not speed up the overall latency in the same way, but incorporates better with the MPTCP standard.

--- end rough description ---

--- start outlook ---

A first reference implementation was already done by us, based on MPTCP v0.90 and the "Downgrade" approach and is currently under lab investigation and optimization.

Btw., we know the presentation from NICT (https://www.ietf.org/proceedings/98/slides/slides-98-mptcp-sessa-synduplication-00.pdf), but we see fundamental differences, because robustness wasn't addressed there. 

--- end outlook ---

I'm looking forward to your comments.

Kind regards
Markus Amend

DEUTSCHE TELEKOM AG
T-Labs (Research & Innovation)
Markus Amend

LIFE IS FOR SHARING.  

You can find the obligatory information on www.telekom.com/compulsory-statement

BIG CHANGES START SMALL - CONSERVE RESOURCES BY NOT PRINTING EVERY E-MAIL.

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 12 Jul 2017 21:48:05 +0200

Markus,

The problem that you raise is very similar to the HappyEyeBalls 
solutions that is used by web browsers on dual-stack hosts. The final 
goal is to make sure that applications can connect quickly to a 
destination when there are two paths available and one of them is bad.

I think that there are two ways to solve this problem :

1. Enhance MPTCP so that MPTCP can support this use case without 
requiring any modification to the applications

2. Enhance the applications without adding complexity to MPTCP

Concerning the first approach, I guess that you solution is to let the 
server parse through all the received SYN with a given key to check 
whether this is a dual connection establishment or a regular connection 
establishment. The limitation of this approach is that server needs to 
lookup all the keys of the received SYNs (possibly during a given period 
of time as you mention in your email) everytime it receives a SYN. This 
increases the SYN processing time on servers and thus the risk of denial 
of service attacks. It would be interesting to see how your 
implementation copes with this issue. Note that it requires the presence 
of a key inside the SYN (as defined in RFC6824). This implies that this 
is not compatible with RFC6824bis.

The second approach has been used successfully with HappyEyeBalls by web 
browers. It could be integrated in any library or higher level 
programming language by simply trying to open separate MPTCP connections 
and reset that one that connects slowly. A kernel implementation could 
probably also include code to support these parallel establishments of 
MPTCP connections. This would allow all applications to benefit from 
this feature, even those that use the C socket API directly.

The main benefit of the second approach is that the client is in 
control. It can delay the transmission of the second SYN as in 
HappyEyeBalls to avoid doubling unnecessarily the number of SYNs that a 
server has to process. Since the client is less loaded than the server, 
it is less an issue to put some complexity (e.g. timers, remembering the 
best interface, ...) on the client side.

If you travel to Prague, I'd be happy to discuss with you. We'll also 
participate to the hackathon on Sunday if you'd like to discuss about 
your implementation.

Best regards,

Olivier

---

From: "Zuojing (2012 Laboratories)" <[email-masked]>
Date: Thu, 13 Jul 2017 03:32:42 +0000

Hi, Olivier and Markus,

I meet the same problem that the connection establishment fails due to weak WiFi signal although LTE is good.
The HappyEyeBalls is a good solution that we do not need modify the MPTCP. However, it requires the enhancement of the applications, as you mentioned. 
How about if the application does not support the enhancement, since we can't control the design of applications? 
I am considering that we may modify the MPTCP, e.g., let the MPTCP stack open two separate connections with indicated network IP addresses instead of the applications do. Moreover, this solution does not increase as much complexity as 'Duplicate SYN' does. Instead, the duplicate SYNs are sent to build two separate MPTCP connections without modifying the process of MPTCP connection establishment. How do you think?

Kind regards,
Jing

> -----Original Message-----
> From: multipathtcp [mailto:[email-masked]] On Behalf Of
> Olivier Bonaventure
> Sent: Thursday, July 13, 2017 3:48 AM
> To: [email-masked]; [email-masked]
> Subject: Re: [multipathtcp] A proposal for MPTCP Robust session Establishment
> (MPTCP RobE)
> 
> Markus,
> 
> The problem that you raise is very similar to the HappyEyeBalls solutions that is
> used by web browsers on dual-stack hosts. The final goal is to make sure that
> applications can connect quickly to a destination when there are two paths
> available and one of them is bad.
> 
> 1. Enhance MPTCP so that MPTCP can support this use case without requiring
> any modification to the applications
> 
> Concerning the first approach, I guess that you solution is to let the server parse
> through all the received SYN with a given key to check whether this is a dual
> connection establishment or a regular connection establishment. The limitation
> of this approach is that server needs to lookup all the keys of the received SYNs
> (possibly during a given period of time as you mention in your email) everytime
> it receives a SYN. This increases the SYN processing time on servers and thus the
> risk of denial of service attacks. It would be interesting to see how your
> implementation copes with this issue. Note that it requires the presence of a
> key inside the SYN (as defined in RFC6824). This implies that this is not
> compatible with RFC6824bis.
> 
> browers. It could be integrated in any library or higher level programming
> language by simply trying to open separate MPTCP connections and reset that
> one that connects slowly. A kernel implementation could probably also include
> code to support these parallel establishments of MPTCP connections. This
> would allow all applications to benefit from this feature, even those that use
> the C socket API directly.
> 
> The main benefit of the second approach is that the client is in control. It can
> delay the transmission of the second SYN as in HappyEyeBalls to avoid doubling
> unnecessarily the number of SYNs that a server has to process. Since the client
> is less loaded than the server, it is less an issue to put some complexity (e.g.
> timers, remembering the best interface, ...) on the client side.
> 
> If you travel to Prague, I'd be happy to discuss with you. We'll also participate to
> the hackathon on Sunday if you'd like to discuss about your implementation.
> 
> Best regards,
> 
> 
> Olivier
> 
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 13 Jul 2017 09:03:28 +0200

Jing,
> The HappyEyeBalls is a good solution that we do not need modify the MPTCP. However, it requires the enhancement of the applications, as you mentioned.

On smartphones, most applications use a library that manages the 
underlying socket. By changing this library, you can enable most 
applications to benefit from the change.

> I am considering that we may modify the MPTCP, e.g., let the MPTCP stack open two separate connections with indicated network IP addresses instead of the applications do. Moreover, this solution do…

It should be possible to change the implementation of the connect system 
call so that instead of selecting one source IP address and trying to 
create one connection it tries to create two MPTCP connections, 
typically one immediately with the default interface and the other one 
later after some timeout and then returns the first established 
connection and discards the other one (i.e. sends a RST upon reception 
of the SYN+ACK for the second).

Olivier

---

From: <[email-masked]>
Date: Thu, 13 Jul 2017 11:48:00 +0000

Hi Olivier,

thanks for your feedback, see inline...

BR

Markus 

> From: Olivier Bonaventure [mailto:[email-masked]]
> Sent: Mittwoch, 12. Juli 2017 21:48
> To: Amend, Markus <[email-masked]>; [email-masked]
> Subject: Re: [multipathtcp] A proposal for MPTCP Robust session 
> Establishment (MPTCP RobE)
> 
> Markus,
> 

That is exactly what we aim to, because of application transparency. MPTCP is a wonderful protocol which extends TCPs single path limitation to multipath support. But again, like the TCP it tries to improve in respect to multipath support, it depends during sessions establishment from a single path. We see the chance to make MPTCP both robust against default route blocking and at the same time reduce establishment latency to the fastest path in game. This idea of robustness and acceleration is implemented in our "Downgrade" proposal, without any network overhead, but obviously a proposal with higher complexity. Especially compared to an approach which only tries to solve robustness challenge.

> 2. Enhance the applications without adding complexity to MPTCP

You are absolutely right to raise this option, which was never in focus of us, because we think/thought it should be a MPTCP inherent capability, see above. But why not...
At the end it needs special treatment from application developers, where I ever thought, that MPTCP focus is to be application transparent.

> whether this is a dual connection establishment or a regular 
> connection establishment. The limitation of this approach is that 
> server needs to lookup all the keys of the received SYNs (possibly 
> during a given period of time as you mention in your
> email) everytime it receives a SYN. This increases the SYN processing 
> time on servers and thus the risk of denial of service attacks.

Yes, good observation. Indeed, it is a risk and I guess without additional processing, compared to MPTCP, it is hard to solve.

Our implementation uses the same functionality as exists in the MPTCP reference implementation for ensuring local uniqueness of keys respectively the corresponding token (hashtables). Thus the complexity is reduced to search the correct hash table entry and then only parse the linked list stored at this point.

> It would be interesting
> to see how your implementation copes with this issue. Note that it 
> requires the presence of a key inside the SYN (as defined in RFC6824). 
> This implies that this is not compatible with RFC6824bis.

It seems so, just read it the first time and yes, if KeyA is missing it is hard to figure out SYNs, related to the same request.

> The second approach has been used successfully with HappyEyeBalls by 
> web browers. It could be integrated in any library or higher level 
> programming language by simply trying to open separate MPTCP 
> connections and reset that one that connects slowly. A kernel 
> implementation could probably also include code to support these parallel establishments of MPTCP connections.
> This would allow all applications to benefit from this feature, even 
> those that use the C socket API directly.

It is like the "Break before make" or "Timer" proposal from my original email, which is very similar to HappyEyeBalls idea in kernel space.

> HappyEyeBalls to avoid doubling unnecessarily the number of SYNs that a server has to process.
> Since the client is less loaded than the server, it is less an issue 
> to put some complexity (e.g. timers, remembering the best interface, 
> ...) on the client side.

I guess, you mean the third "Timer" proposal, which fulfill the requirement of robustness as well. As you already wrote, we possibly delay!!! requests, that's why we don't prefer it currently, especially compared to the "Downgrade" proposal.
We know about the complexity of the first "Downgrade" proposal, but hey, to be honest, MPTCP is somehow also not easy :-)

I think in further discussion we have to clarify, if we raise with the question for MPTCP robust establishment a standard relevant topic or is it just implementation related?

> participate to the hackathon on Sunday if you'd like to discuss about 

Unfortunately, I arrive Sunday late around 8pm. But I like to discuss the ideas with you and anyone who is interested. Perhaps, we agree during the first MPTCP session on Tuesday some additional discussion? Otherwise you can contact me directly if you have some time on Sunday evening or Monday.
I'm looking forward to meet you in Prague.