Subject: [multipathtcp] MPTCP Schedulers

From: Vladimir Olteanu <[email-masked]>
Date: Fri, 15 Feb 2019 15:42:55 +0200

Hi guys,

I'm trying to tie up any loose ends in the SOCKS 6 draft. There is some 
functionality which lets you change the MPTCP scheduler used by the 
proxy. The available schedulers are basically the ones available in the 
Linux kernel implementation. 
(https://tools.ietf.org/html/draft-olteanu-intarea-socks-6-05#page-20)

Do you think it would be worthwhile to standardize the MPTCP schedulers? 
I think an informational or best practice RFC would be useful.

Cheers,

Vlad

---

From: <[email-masked]>
Date: Fri, 15 Feb 2019 16:18:05 +0000

Vlad,

personally I think there are lots of interesting issues associated with MPTCP scheduling which it would be interesting to discuss, if people can share results beyond those in https://tools.ietf.org/html/rfc8041#section-3.7

best wishes

phil

________________________________
From: multipathtcp <[email-masked]> on behalf of Vladimir Olteanu <[email-masked]>
Sent: 15 February 2019 13:42
To: multipathtcp
Subject: [multipathtcp] MPTCP Schedulers

Hi guys,

Cheers,

Vlad

_______________________________________________
multipathtcp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 15 Feb 2019 10:07:22 -0800

On 15/02/19 - 15:42:55, Vladimir Olteanu wrote:
> functionality which lets you change the MPTCP scheduler used by the proxy.
> The available schedulers are basically the ones available in the Linux
> kernel implementation.
> Do you think it would be worthwhile to standardize the MPTCP schedulers? I
> think an informational or best practice RFC would be useful.

In general, I think that would be good. The tricky part around MPTCP is that
the client has no direct control on the server's scheduling behavior.
The backup-bit provides some amount of control but it's not fine-grained enough.

Standardizing the schedulers or providing a more explicit "API/messaging"
from the client to the server about the client's policies would be very
useful.

That way, clients can connect to "any" MPTCP-server and make sure that the
server's behavior is respecting the client's policies about data-usage on
the different subflows.

Christoph

---

From: Alexander Frömmgen <[email-masked]>
Date: Fri, 15 Feb 2019 20:53:52 +0100

This reminds me that I once took some notes about a scheduling API https://github.com/AlexanderFroemmgen/draft-ietf-mptcp-api-for-preference-awareness

Best,

Alexander 

Am 15. Februar 2019 19:07:22 MEZ schrieb Christoph Paasch <cpaasch=[email-masked]>:
>proxy.

---

From: Zhen Cao <[email-masked]>
Date: Sat, 16 Feb 2019 21:53:58 +0800

> I think an informational or best practice RFC would be useful.

It's meaningful.  Documenting the existing schedulers (minRTT,
redundant, round robin) as informational documents will help the mptcp
practitioners to get better use of it.

In addition, path manager needs a better specification.  These can be
considered as the input to the co-chairs' ''call for future items of
the working group.  We may discuss it over at the upcoming meeting.

Cheers,
Zhen

---

From: Alexander Frömmgen <[email-masked]>
Date: Fri, 15 Feb 2019 20:45:39 +0100

Best,

Alexander

---

From: <[email-masked]>
Date: Sun, 17 Feb 2019 08:08:22 +0000

Hi Vladimir, hi Christoph,

> > I'm trying to tie up any loose ends in the SOCKS 6 draft. There is
> > some functionality which lets you change the MPTCP scheduler used by the
> proxy.
> > Do you think it would be worthwhile to standardize the MPTCP
> > schedulers? I think an informational or best practice RFC would be useful.
> 
> In general, I think that would be good.

ACK

> The tricky part around MPTCP is that

ACK, but that's also true for the other way round from server to client :-)

> from the client to the server about the client's policies would be very useful.
> 

ACK, even other scheduling algorithms which are not part of the reference
implementation are worthwhile to consider. In particular it would also
support the current 5G ATSSS standardization, which most probably relies on
MPTCP. Standardized or at least documented scheduling algorithms would
make the traffic steering behavior more predictable across different vendor
implementations.

> server's behavior is respecting the client's policies about data-usage on the
> different subflows.
> 

Half ACK :-), see comments above. From a 5G ATSSS view I would assume the UE
to be a client and the MPTCP Proxy in the 5G core to be a server, right? Than it
should be possible to exchange the preferred settings in both directions. But
independent from 5G I think we should not limit such an MPTCP API in one
direction. 

BR
Markus

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 17 Feb 2019 10:19:27 +0000

Hello,
> The backup-bit provides some amount of control but it's not fine-grained enough.

I agree, but this control could be provided by defining an MPTCP option 
that enables a client to request the utilisation of a specific scheduler 
by a remote server.

For this, we would need to :

1. agree on a small list of schedulers that should be supported by MPTCP 
implementations. The literature contains a variety of schedulers, we'd 
need to select some of them

2. agree on a way to exchange the information related to the requested 
scheduler

If we can define a small list of identifiers for 1, then we can use the 
same technique as the one described in recent work by Hoang Tran Viet

https://arxiv.org/abs/1901.01863

> from the client to the server about the client's policies would be very

There are two parts of the API:
A. an API that enables clients to interact with servers, likely based on 
MPTCP options
B. an API that enables an application to interact with the MPTCP stack

A. should be easy. For B., the socket API draft could be a starting 
point for such a discussion, but other APIs could be possible as well.

> server's behavior is respecting the client's policies about data-usage on

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 17 Feb 2019 10:20:51 +0000

Vlad,
> I think an informational or best practice RFC would be useful.

I think that this would make sense, but we could also encode the 
requested MPTCP scheduler inside an MPTCP option instead of putting it 
in an application level message as you propose in your draft. The 
advantage of using MPTCP options is that the same solution would work 
for any application.

Olivier

---

From: Alexander Frömmgen <[email-masked]>
Date: Sun, 17 Feb 2019 13:35:34 +0100

Hello,

some remarks and questions inline.

> think an informational or best practice RFC would be useful.

What level of scheduler standardization do you consider? While an informal description of the scheduler behavior should be feasible, I have the feeling that a detailed specification of the scheduler behavior is not possible, e.g., i) details of todays scheduler behaviors emerged from the dependencies to the Linux Kernel (round trip time is calculation, packet queues), and ii) intuitive descriptions, such as "redundant“, are very imprecise (prefer old or new packets for heterogenous subflow environments?). [0]

> need to select some of them

1.1 How to select the concrete schedulers? Given that there is a variety of proposed schedulers, as of today, the Linux implementation contains only three of them. Does this imply that there was no need for further schedulers outside of academia so far?
1.2 How should we deal with the concrete parameters of these schedulers and scheduler Evolution?

> same technique as the one described in recent work by Hoang Tran Viet

Sounds great. Can you elaborate a bit the concrete usecase? Why is the eBPF flexibility required? 

> B. an API that enables an application to interact with the MPTCP stack

It might worth to start with the high level  "goals" and the target audience of such an API. For application developers, for example, the choice of the concrete scheduler might be inappropriate. The Apple API [2], for example, is on a much higher abstraction level than explicitly selecting the scheduler.

A high level API to specify the preferences [3], e.g., the required throughput, might enable scheduler innovations to optimize for these preferences and speed up the MPTCP adoption by simplifying the application developers life.

Obviously, the high level API might be implemented on top of a lower level API. In this case, the goal for the lower level API should be to enable the implementation of the high level API on top.

Best,

Alexander

[0] http://progmp.net/froemmgen-middleware2017.pdf 
[1] http://tuprints.ulb.tu-darmstadt.de/7709/13/Dissertation_Froemmgen.pdf Section 2.2
[2] https://developer.apple.com/documentation/foundation/nsurlsessionmultipathservicetype 
[3] https://github.com/AlexanderFroemmgen/draft-ietf-mptcp-api-for-preference-awareness/blob/master/output.txt

---

From: Vladimir Olteanu <[email-masked]>
Date: Mon, 18 Feb 2019 02:46:24 +0200

> advantage of using MPTCP options is that the same solution would work

Hi,

I agree that it is worthwhile doing it at L4. As far as SOCKS is 
concerned, this would also open the door for the client controlling the 
scheduler used not only by the proxy, but also by the remote host.

I'd argue that being able to do it at the application layer is also 
useful, even if your proposed option gets standardized. The scheduler 
option will not fit inside a SYN + MP_CAPABLE + TFO (along with SACK, 
TS, MSS and Wscale) and you'll have to send it 1 RTT later. (In fact, 
not even the TFO option will fit if the cookie is large.) OTOH, there is 
on such pressure oh the SOCKS option space; you've got 16K to play with.

Cheers,

Vlad

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 18 Feb 2019 09:42:11 +0000

Vlad,
> scheduler used not only by the proxy, but also by the remote host.

Agreed

> on such pressure oh the SOCKS option space; you've got 16K to play with.

The scheduler is not that important in the first rtt since you only have 
one subflow at that time anyway. I agree that SOCKS has a larger space 
to encode options, but after the SOCKS exchange you cannot send SOCKS 
data anymore. This means that with a SOCKS solution you can set the 
scheduler for the entire connection, but never change it during the 
connection. An MPTCP option could be used at the beginning of a 
connection but also at any time during the connection lifetime

Olivier

---

From: <[email-masked]>
Date: Mon, 18 Feb 2019 09:51:54 +0000

A couple of things personally I’d be interested in hearing about

-          If there are differences for the schedulers deployed for end host deployments of MPTCP and the proxy deployments

-          Scheduling algorithms that prefer one interface for policy reasons (eg cost) ie the choice of interface is not primarily down to performance differences between the interfaces (but presumably kicks in at some point when it’s decided that the primary is down (how to decide that?) or that there isnt enough bandwidth on the primary interface (how to decide that?)
phil

From: multipathtcp [mailto:[email-masked]] On Behalf Of Alexander Frömmgen
Sent: 17 February 2019 12:36
To: Olivier Bonaventure <[email-masked]>; Christoph Paasch <cpaasch=[email-masked]>; Vladimir Olteanu <[email-masked]>
Cc: [email-masked]
Subject: Re: [multipathtcp] MPTCP Schedulers

Hello,

> B. an API that enables an application to interact with the MPTCP stack

It might worth to start with the high level  "goals" and the target audience of such an API.. For application developers, for example, the choice of the concrete scheduler might be inappropriate. The Apple API [2], for example, is on a much higher abstraction level than explicitly selecting the scheduler.

Best,

Alexander

[0] http://progmp.net/froemmgen-middleware2017.pdf
[1] http://tuprints.ulb.tu-darmstadt.de/7709/13/Dissertation_Froemmgen.pdf Section 2.2<http://tuprints.ulb.tu-darmstadt.de/7709/13/Dissertation_Froemmgen.pdf%20Section%202.2>
[2] https://developer.apple.com/documentation/foundation/nsurlsessionmultipathservicetype
[3] https://github.com/AlexanderFroemmgen/draft-ietf-mptcp-api-for-preference-awareness/blob/master/output.txt

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 18 Feb 2019 11:03:20 +0000

Phil,

> -If there are differences for the schedulers deployed for end host 
> deployments of MPTCP and the proxy deployments

Yes. In the hybrid access networks use case, the schedulers are 
optimised to first use the DSL network and then overlfow over the LTE 
network when the DSL is full.

> -Scheduling algorithms that prefer one interface for policy reasons (eg 
> cost) ie the choice of interface is not primarily down to performance 
> differences between the interfaces (but presumably kicks in at some 
> point when it’s decided that the primary is down (how to decide that?) 

Detecting the failure of a primary path can be done by using existing 
TCP retransmission techniques. If the same packet needs to be 
retransmitted n times over a path, the stack can infer that this path is 
bad and should not be used for new packets.

> or that there isnt enough bandwidth on the primary interface (how to 
> decide that?)

The simple techniques in hybrid access networks is to simply measure the 
load of the DSL interface and take it into account when performing a 
scheduling decision

Olivier

---

From: Vladimir Olteanu <[email-masked]>
Date: Mon, 18 Feb 2019 20:00:30 +0200

Hi,

On 2/17/2019 2:35 PM, Alexander Frömmgen wrote:
> What level of scheduler standardization do you consider? While an 
> informal description of the scheduler behavior should be feasible, I 
> have the feeling that a detailed specification of the scheduler 
> behavior is not possible, e.g., i) details of todays scheduler 
> behaviors emerged from the dependencies to the Linux Kernel (round 
> trip time is calculation, packet queues), and ii) intuitive 
> descriptions, such as "redundant“, are very imprecise (prefer old or 
> new packets for heterogenous subflow environments?). [0]

At the very least we should write a general description of the three 
algorithms that are available, what to expect from them and in which 
cases they are best suited.

In the absence of such a document, I'd have to include said descriptions 
in the SOCKS draft, which isn't the best place to go looking for such 
information.

Vlad

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 18 Feb 2019 19:51:18 +0000

Vlad,
> cases they are best suited.

Round-Robin was a proof of concept that Christoph wrote to describe the 
flexibility of the modular implementation. I don't think that anyone 
really considers to use it in production.

The redundant scheduler was proposed in a paper, but again I'm not aware 
of anyone using it in production.

The default one is the lowest rtt first. It has been tuned over the 
years, but we could describe it in a draft.

There are other schedulers that are used and would be worth documenting 
in an IETF draft.

> in the SOCKS draft, which isn't the best place to go looking for such 

We can prepare something before Prague based on the existing papers. 
However, the description will remain abstract and may not include all 
the details that have been added to the Linux kernel implementation over 
the years.

It would be more interesting to document the use cases that operators 
would want to have as schedulers.

Olivier