Subject: Re: Multi-path QUIC Extension Experiments

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 16 Jul 2021 13:14:25 +0000

Hi Yunfei,

thanks as well for you sharing your results! Can you explain even a bit more what you mean by MP-HoL Blocking? Is this because of the flow control limits? If so wouldn’t it make sense to reserve a certain “space” for each path?

Mirja

From: QUIC <[email-masked]> on behalf of "Ma, Yunfei" <yunfei.ma=[email-masked]>
Date: Thursday, 15. July 2021 at 04:18
To: Robin MARX <[email-masked]>, Yanmei Liu <[email-masked]>
Cc: "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>, Christian Huitema <[email-masked]>, "lucaspardue.24.7" <[email-masked]>, quic <[email-masked]>, Qing An <[email-masked]>
Subject: Re: Re: Multi-path QUIC Extension Experiments

Hi Robin,

Thanks so much for your questions!

First, the head of line blocking discussed here is called multi-path head-of-line blocking or MP-HoL blocking, and its root cause is quite different from the stream HoL blocking usually discussed in QUICv1. The MP-HoL blocking happens when one path blocks the other path, not when one stream blocks the other stream. Please note that we indeed use multiple streams, for example, different video requests are carried in different QUIC streams. QUIC’s stream multiplexing ability and its benefits still hold in this scenario.

Second, regarding packet scheduling mode, right now, in our Taobao A/B test, we transmit packets on multiple paths simultaneously. However, you can definitely use traffic switching only and choose to switch when one path could not meet your bandwidth requirement. Basically, if you use multiple paths simultaneously, you get the most elasticity from a resource pooling perspective. It really comes down on what your application needs. We will also update the packet scheduling section soon in a newer version of the draft, in which we plan to include more discussions on the packet scheduling policy.

Third, regarding the benefits of more bandwith versus the "downsides". Whether you want more bandwidth depends on your application. For videos, yes, more bandwidth is extremely helpful in improving the long tail QoE, which is an important target for Taobao. We find multi-path QUIC helps us improve two important metrics, rebuffer rate and video start-up delays. In the past, if you work on multi-path scheduling that does not collaborate close enough with applications such as MPTCP, the MP-HoL blocking becomes the downside that cripples the performance. However, the user space nature of QUIC provides us the opportunity to solve this problem, so now our conclusion is that you can enjoy the benefits of more bandwidth and more reliable connectivity from multi-path without much of the “downsides”.

I hope my answer is helpful, but feel free to let me know if you have any additional comments.

Cheers,
Yunfei

from Alimail macOS<https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https%3A%2F%2Fmail.alibaba-inc.com%2F>
------------------Original Mail ------------------
Sender:Robin MARX <[email-masked]>
Send Date:Wed Jul 14 07:39:37 2021
Recipients:Yanmei Liu <[email-masked]>
CC:quic <[email-masked]>, Ma, Yunfei <[email-masked]>, Christian Huitema <[email-masked]>, Qing An <[email-masked]>, 李振宇 <[email-masked]>, matt.joras <[email-masked]>, lucaspardue.24.7 <[email-masked]>
Subject:Re: Multi-path QUIC Extension Experiments
Hello Yanmei,

Thanks for the additional results on an interesting topic. I'm looking forward to reading the SIGCOMM paper.

I was a bit surprised to (apparently) see HOL blocking mentioned as a major issue, as that's one of the things QUIC aims to be better at than TCP.
It's a bit difficult to understand from the slides, but it seems like you're sending packets for a single stream (Stream ID 1 in the diagrams) on both the slow and fast path, which would indeed induce HOL blocking.
Consequently, I was wondering what the practical reasons are for you to multiplex packets for a single stream over multiple paths, as opposed to for example attaching a single stream to a single path (say: high priority streams use the fast path for all their packets).

I see this mentioned a bit in the draft under "packet scheduling", where it talks about switching paths once the cwnd is full for one. That indeed leads to the behaviour seen in the slides, but that's my question: why would you take those approaches then?
Are there so many cases where the additional "bandwidth" from using multiple path's cwnd for a single stream outweigh the downsides of HOL blocking? Relatedly: what are the packet loss rates you've observed on real networks?
Have you experimented with e.g., tying streams to paths more closely? Does that work better or worse? Why?

I'm mainly wondering how these tradeoffs evolve depending on the type of paths available and if it's possible to make a model to drive this logic.
I assume there is much existing work on this for MPTCP, but I also assume some of that changes due to QUIC's independent streams / stream prioritization flexibility.

Thank you in advance and with best regards,
Robin

On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=[email-masked]<mailto:[email-masked]>> wrote:
Hi everyone,

We have finished some experiments about deploying multi-path quic extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/) in Alibaba Taobao short-form video streaming, and the experiment results are concluded in the slides (attached file).
If anyone is interested in the experimental details about multi-path quic, please let us know.
All the feedbacks and suggestions are appreciated!

Best regards,
Yanmei

--

dr. Robin Marx
Postdoc researcher - Web protocols
Expertise centre for Digital Media

Cellphone +32(0)497 72 86 94

www.uhasselt.be<https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http%3A%2F%2Fwww.uhasselt.be%2F>
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

[Image removed by sender.]

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Jul 2021 19:12:18 +0000

I too am curious!
There are only two ways to handle flow control—overcommit, or don’t overcommit.

The “don’t overcommit” choice leads to blocking, since any of that resource allocated to one path can’t be used by the other.
The “overcommit” choice either leads to OOM, or throwing out some successfully transmitted and received data.

Underlying this is a fun question: Which inefficiency is worse? Not using resources that should be used (i.e. from choosing to not overcommit), or sometimes redundantly using a resource (from choosing to overcommit)?
I’m curious too about what implementation strategies we end up doing in general around this, and.. if enough implementations are choosing overcommit, if we need some different protocol mechanisms to bound the redundancy?
-=R

From: QUIC <[email-masked]> on behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
Date: Friday, July 16, 2021 at 6:15 AM
To: "Ma, Yunfei" <yunfei.ma=[email-masked]>, Robin MARX <[email-masked]>, Yanmei Liu <[email-masked]>
Cc: "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>, Christian Huitema <[email-masked]>, "lucaspardue.24.7" <[email-masked]>, quic <[email-masked]>, Qing An <[email-masked]>
Subject: Re: Multi-path QUIC Extension Experiments

Hi Yunfei,

Mirja

Hi Robin,

Cheers,
Yunfei

from Alimail macOS<https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/>
------------------Original Mail ------------------

We have finished some experiments about deploying multi-path quic extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)<https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)> in Alibaba Taobao short-form video streaming, and the experiment results are concluded in the slides (attached file).
If anyone is interested in the experimental details about multi-path quic, please let us know.
All the feedbacks and suggestions are appreciated!

Best regards,
Yanmei

--

dr. Robin Marx

www.uhasselt.be<https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/>
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

---

From: "Charles 'Buck' Krasic" <[email-masked]>
Date: Fri, 16 Jul 2021 12:50:18 -0700

"don't overcommit" includes the common practice of setting very large
limits on the client side, where in aggregate the case of server being flow
control limited is effectively non-existent.

I am curious to hear clarification of the precise definition of MP-HoL
blocking here.  is it not flow control, but rather path aliasing where
distinct paths are actually sharing some physical link(s)?

On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=[email-masked]>
wrote:

> The “don’t overcommit” choice leads to blocking, since any of that
> resource allocated to one path can’t be used by the other.
>
> The “overcommit” choice either leads to OOM, or throwing out some
> successfully transmitted and received data.
>
>
> Underlying this is a fun question: Which inefficiency is worse? Not using
> resources that should be used (i.e. from choosing to not overcommit), or
> sometimes redundantly using a resource (from choosing to overcommit)?
> I’m curious too about what implementation strategies we end up doing in
> general around this, and.. if enough implementations are choosing
> overcommit, if we need some different protocol mechanisms to bound the
> redundancy?
> -=R
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Mirja Kuehlewind
> <mirja.kuehlewind=[email-masked]>
> *Date: *Friday, July 16, 2021 at 6:15 AM
> *To: *"Ma, Yunfei" <yunfei.ma=[email-masked]>, Robin
> MARX <[email-masked]>, Yanmei Liu <[email-masked]>
> *Cc: *"matt.joras" <[email-masked]>, 李振宇 <[email-masked]>,
> Christian Huitema <[email-masked]>, "lucaspardue.24.7" <
> [email-masked]>, quic <[email-masked]>, Qing An <
> [email-masked]>
> *Subject: *Re: Multi-path QUIC Extension Experiments
>
>
>
> Hi Yunfei,
>
>
>
> thanks as well for you sharing your results! Can you explain even a bit
> more what you mean by MP-HoL Blocking? Is this because of the flow control
> limits? If so wouldn’t it make sense to reserve a certain “space” for each
> path?
>
>
>
> Mirja
>
>
>
>
>
> *From: *QUIC <[email-masked]> on behalf of "Ma, Yunfei" <yunfei.ma=
> [email-masked]>
> *Date: *Thursday, 15. July 2021 at 04:18
> *To: *Robin MARX <[email-masked]>, Yanmei Liu <
> [email-masked]>
> *Cc: *"matt.joras" <[email-masked]>, 李振宇 <[email-masked]>,
> Christian Huitema <[email-masked]>, "lucaspardue.24.7" <
> [email-masked]>, quic <[email-masked]>, Qing An <
> [email-masked]>
> *Subject: *Re: Re: Multi-path QUIC Extension Experiments
>
>
>
> Hi Robin,
>
>
>
> First, the head of line
> blocking discussed here is called multi-path head-of-line blocking or MP-HoL blocking, and its root cause is quite different from the stream HoL blocking usually discussed in
> QUICv1. The MP-HoL blocking happens when one path blocks the other path, not when one stream blocks the other stream. Please note that we indeed
> use multiple streams, for example, different video requests are carried in different QUIC streams. QUIC’s stream multiplexing ability and its benefits still hold in this scenario.
>
>
>
> Second, regarding packet scheduling mode,
> right now, in our Taobao A/B test, we transmit packets on multiple paths simultaneously. However, you can definitely use
> traffic switching only and choose to switch when one path could not meet
> your bandwidth requirement. Basically, if you use multiple paths
> simultaneously, you get the most elasticity from a resource pooling
> perspective.
> It really comes down on what your application needs. We will also update the packet scheduling section
> soon in a newer version of the
> draft, in which we plan to include more discussions on the packet scheduling
> policy.
>
>
>
> Third, regarding the benefits of more bandwith versus the "downsides".
> Whether you want more bandwidth depends on your application. For videos, yes, more bandwidth is
> extremely helpful in improving the long tail QoE, which is an important target for Taobao. We find multi-path QUIC helps us improve two important metrics, rebuffer rate and video start-up delays.
> In the past, if you work on multi-path scheduling that does not collaborate
> close enough with applications such as MPTCP, the MP-HoL blocking becomes
> the downside that cripples the
> performance. However, the user space nature of QUIC provides us the opportunity to solve this problem,
> so now our conclusion is that
> you can enjoy the benefits of more bandwidth and more reliable connectivity
> from multi-path without much of the “downsides”.
>
>
>
> from Alimail macOS
> <https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/>
>
> *Sender:*Robin MARX <[email-masked]>
>
> *Send Date:*Wed Jul 14 07:39:37 2021
>
> *Recipients:*Yanmei Liu <[email-masked]>
>
> *CC:*quic <[email-masked]>, Ma, Yunfei <[email-masked]>,
> Christian Huitema <[email-masked]>, Qing An <
> [email-masked]>, 李振宇 <[email-masked]>, matt.joras <
> [email-masked]>, lucaspardue.24.7 <[email-masked]>
>
> *Subject:*Re: Multi-path QUIC Extension Experiments
>
> Hello Yanmei,
>
>
>
> I was a bit surprised to (apparently) see HOL blocking mentioned as a
> major issue, as that's one of the things QUIC aims to be better at than TCP.
>
> It's a bit difficult to understand from the slides, but it seems like
> you're sending packets for a single stream (Stream ID 1 in the diagrams) on
> both the slow and fast path, which would indeed induce HOL blocking.
>
> Consequently, I was wondering what the practical reasons are for you to
> multiplex packets for a single stream over multiple paths, as opposed to
> for example attaching a single stream to a single path (say: high priority
> streams use the fast path for all their packets).
>
>
>
> I see this mentioned a bit in the draft under "packet scheduling", where
> it talks about switching paths once the cwnd is full for one. That indeed
> leads to the behaviour seen in the slides, but that's my question: why
> would you take those approaches then?
>
> Are there so many cases where the additional "bandwidth" from using
> multiple path's cwnd for a single stream outweigh the downsides of HOL
> blocking? Relatedly: what are the packet loss rates you've observed on
> real networks?
>
> I'm mainly wondering how these tradeoffs evolve depending on the type of
> paths available and if it's possible to make a model to drive this logic.
>
> I assume there is much existing work on this for MPTCP, but I also assume
> some of that changes due to QUIC's independent streams / stream
> prioritization flexibility.
>
>
>
> On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=
> [email-masked]> wrote:
>
> Hi everyone,
>
> We have finished some experiments about deploying multi-path quic
> extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/) in
> Alibaba Taobao short-form video streaming, and the experiment results are
> concluded in the slides (attached file).
> *dr. Robin Marx*
> *Cellphone *+32(0)497 72 86 94
>
>
>
> www.uhasselt.be
> <https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/>
>

---

From: Yunfei Ma <[email-masked]>
Date: Sun, 18 Jul 2021 01:16:32 -0700

Hi Charles, Roberto, and Mirja:

Thanks a lot for your questions. As all three of you are curious about the
definition of MP-HoL, I am putting my answer into one reply.

Short answer: the MP-HoL is not because of flow control, but rather, it is
related to the nature of path heterogeneity. In other words, MP-HoL can
happen when flow control limit is not reached (as pointed out by Charles,
you can set a large limit on the client side).

More specifically, when you want to send out packets on different paths at
the same time, there is a scheduler to decide how to split your packets and
put them on different paths. However, in mobile networks, the network paths
could have very different path delays. MP-HoL blocking arises when the
packets sent earlier at the slow path arrive later than the packets sent
later at the fast path, causing out-of-order arrival. As a consequence, the
out-of-order packets are not eligible to be submitted to applications, so
the fast path has to wait.

For example, say we want to send out two packets that belong to the same
video frame with a min-RTT scheduler, which is default in MPTCP. For
each packet, the scheduler selects a path for that packet to transmit. The
selection has two criterias: (1) the path's congestion window is not full
and (2) the path selected has a smaller RTT than the other. If somehow, at
the moment of transmitting, the fast path's cwnd is full (some traffic has
been sent before), the first packet is then put on the slow path by the
scheduler. Later, an ACK is received and the fast path becomes available,
so the scheduler puts the second packet on the fast path. As a result,
there is an out-of-order arrival.

What makes the problem even more difficult is that in mobile networks, the
RTTs can change quickly, which makes accurate prediction very difficult.
Worst case is that when the scheduler thinks it is using the fast path, it
is actually using the slow path instead. As you can see, in order to make
multi-path transport efficient, it is important to solve this problem and
that's what we are doing in this project .

I hope I have answered your questions. If not, please let me know.

Cheers,
Yunfei

On Fri, Jul 16, 2021 at 12:51 PM Charles 'Buck' Krasic <
[email-masked]> wrote:

> On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=
> [email-masked]> wrote:
>
>> I too am curious!
>> *From: *QUIC <[email-masked]> on behalf of "Ma, Yunfei" <yunfei.ma
>> =[email-masked]>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 18 Jul 2021 11:12:25 +0200

>> MP-HoL blocking arises when the packets sent earlier at the slow path arrive later than the packets sent later at the fast path, causing out-of-order 
>> arrival. As a consequence, the out-of-order packets are not eligible to be submitted to applications, so the fast path has to wait.

Sorry, but this doesn’t make sense to me.
In earlier mails it was stated that HoL happens even if all packets of one stream is always sent on the same path. If you have time dependent data like a video frame, and the frame is not, say, interlaced, then all the data should be placed on the same stream or at least the same path. Of course you can using interlacing on multiple streams and choose to display a lower quality frame if HoL is detected. So if you want to use bandwidth sharing across paths, then yes, there might be HoL blocking.

But I am interested in understanding what MP-HoL means when streams are not distributed on multiple paths because then at least the transport layer should not exhibit blocking beyond flow control. It can be that the application protocol has stream dependencies that introduce HoL blocking, but that is kind of a separe discussion, relevant as it may be.

Mikkel

> On 18 Jul 2021, at 10.16, Yunfei Ma <[email-masked]> wrote:
> 
> Thanks a lot for your questions. As all three of you are curious about the definition of MP-HoL, I am putting my answer into one reply.
> 
> Short answer: the MP-HoL is not because of flow control, but rather, it is related to the nature of path heterogeneity. In other words, MP-HoL can happen when flow control limit is not reached (as pointed out by Charles, you can set a large limit on the client side).
> 
> More specifically, when you want to send out packets on different paths at the same time, there is a scheduler to decide how to split your packets and put them on different paths. However, in mobile networks, the network paths could have very different path delays. MP-HoL blocking arises when the packets sent earlier at the slow path arrive later than the packets sent later at the fast path, causing out-of-order arrival. As a consequence, the out-of-order packets are not eligible to be submitted to applications, so the fast path has to wait.
> 
> For example, say we want to send out two packets that belong to the same video frame with a min-RTT scheduler, which is default in MPTCP. For each packet, the scheduler selects a path for that packet to transmit. The selection has two criterias: (1) the path's congestion window is not full and (2) the path selected has a smaller RTT than the other. If somehow, at the moment of transmitting, the fast path's cwnd is full (some traffic has been sent before), the first packet is then put on the slow path by the scheduler. Later, an ACK is received and the fast path becomes available, so the scheduler puts the second packet on the fast path. As a result, there is an out-of-order arrival.
> 
> What makes the problem even more difficult is that in mobile networks, the RTTs can change quickly, which makes accurate prediction very difficult. Worst case is that when the scheduler thinks it is using the fast path, it is actually using the slow path instead. As you can see, in order to make multi-path transport efficient, it is important to solve this problem and that's what we are doing in this project .
> 
> On Fri, Jul 16, 2021 at 12:51 PM Charles 'Buck' Krasic <[email-masked] <mailto:[email-masked]>> wrote:
> "don't overcommit" includes the common practice of setting very large limits on the client side, where in aggregate the case of server being flow control limited is effectively non-existent.
> 
> I am curious to hear clarification of the precise definition of MP-HoL blocking here.  is it not flow control, but rather path aliasing where distinct paths are actually sharing some physical link(s)?
> 
> On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=[email-masked] <mailto:[email-masked]>> wrote:
> I too am curious!
> From: QUIC <[email-masked] <mailto:[email-masked]>> on behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked] <mailto:[email-masked]>>
> To: "Ma, Yunfei" <yunfei.ma <http://yunfei.ma/>=[email-masked] <mailto:[email-masked]>>, Robin MARX <[email-masked] <mailto:[email-masked]>>, Yanmei Liu <[email-masked] <mailto:[email-masked]>>
> Cc: "matt.joras" <[email-masked] <mailto:[email-masked]>>, 李振宇 <[email-masked] <mailto:[email-masked]>>, Christian Huitema <[email-masked] <mailto:[email-masked]>>, "lucaspardue.24.7" <[email-masked] <mailto:[email-masked]>>, quic <[email-masked] <mailto:[email-masked]>>, Qing An <[email-masked] <mailto:[email-masked]>>
> From: QUIC <[email-masked] <mailto:[email-masked]>> on behalf of "Ma, Yunfei" <yunfei.ma <http://yunfei.ma/>=[email-masked] <mailto:[email-masked]>>
> To: Robin MARX <[email-masked] <mailto:[email-masked]>>, Yanmei Liu <[email-masked] <mailto:[email-masked]>>
> Cc: "matt.joras" <[email-masked] <mailto:[email-masked]>>, 李振宇 <[email-masked] <mailto:[email-masked]>>, Christian Huitema <[email-masked] <mailto:[email-masked]>>, "lucaspardue.24.7" <[email-masked] <mailto:[email-masked]>>, quic <[email-masked] <mailto:[email-masked]>>, Qing An <[email-masked] <mailto:[email-masked]>>
> from Alimail macOS <https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/>
> Sender:Robin MARX <[email-masked] <mailto:[email-masked]>>
> 
> Recipients:Yanmei Liu <[email-masked] <mailto:[email-masked]>>
> 
> CC:quic <[email-masked] <mailto:[email-masked]>>, Ma, Yunfei <[email-masked] <mailto:[email-masked]>>, Christian Huitema <[email-masked] <mailto:[email-masked]>>, Qing An <[email-masked] <mailto:[email-masked]>>, 李振宇 <[email-masked] <mailto:[email-masked]>>, matt.joras <[email-masked] <mailto:[email-masked]>>, lucaspardue.24.7 <[email-masked] <mailto:[email-masked]>>
> 
> On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi everyone,
> 
> We have finished some experiments about deploying multi-path quic extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/) <https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)> in Alibaba Taobao short-form video streaming, and the experiment results are concluded in the slides (attached file).
> www.uhasselt.be <https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/>

---

From: Roberto Peon <[email-masked]>
Date: Sun, 18 Jul 2021 21:51:55 +0000

It sounds like this problem is not inherent to single-connection multi-path, but will be present in any multi-path implementation, including multiple-tcp-connections used with application-layer muxing.
If this is correct, then it isn’t really a ‘QUIC problem, but rather an implementation/scheduling/CC problem.
That isn’t saying that it isn’t interesting or important to solve, but rather that the protocol itself need not change to solve the problem for generic-QUIC transport.

H3, OTOH, may suffer from this at L7+ proxies without some changes to QUIC or H3, but that is a much longer conversation that doesn’t require multi-path to happen.

-=R

From: Yunfei Ma <[email-masked]>
Date: Sunday, July 18, 2021 at 1:17 AM
To: Charles 'Buck' Krasic <[email-masked]>, Mirja Kuehlewind <[email-masked]>, Roberto Peon <[email-masked]>
Cc: "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>, Christian Huitema <[email-masked]>, Yanmei Liu <[email-masked]>, "lucaspardue.24.7" <[email-masked]>, quic <[email-masked]>, Qing An <[email-masked]>, Yunfei Ma <[email-masked]>
Subject: Re: Multi-path QUIC Extension Experiments

Cheers,
Yunfei

On Fri, Jul 16, 2021 at 12:51 PM Charles 'Buck' Krasic <[email-masked]<mailto:[email-masked]>> wrote:
"don't overcommit" includes the common practice of setting very large limits on the client side, where in aggregate the case of server being flow control limited is effectively non-existent.

On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=[email-masked]<mailto:[email-masked]>> wrote:
I too am curious!
There are only two ways to handle flow control—overcommit, or don’t overcommit.

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]<mailto:[email-masked]>>
Date: Friday, July 16, 2021 at 6:15 AM
To: "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=[email-masked]<mailto:[email-masked]>>, Robin MARX <[email-masked]<mailto:[email-masked]>>, Yanmei Liu <[email-masked]<mailto:[email-masked]>>
Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>, 李振宇 <[email-masked]<mailto:[email-masked]>>, Christian Huitema <[email-masked]<mailto:[email-masked]>>, "lucaspardue.24.7" <[email-masked]<mailto:[email-masked]>>, quic <[email-masked]<mailto:[email-masked]>>, Qing An <[email-masked]<mailto:[email-masked]>>
Subject: Re: Multi-path QUIC Extension Experiments

Hi Yunfei,

Mirja

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=[email-masked]<mailto:[email-masked]>>
Date: Thursday, 15. July 2021 at 04:18
To: Robin MARX <[email-masked]<mailto:[email-masked]>>, Yanmei Liu <[email-masked]<mailto:[email-masked]>>
Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>, 李振宇 <[email-masked]<mailto:[email-masked]>>, Christian Huitema <[email-masked]<mailto:[email-masked]>>, "lucaspardue.24.7" <[email-masked]<mailto:[email-masked]>>, quic <[email-masked]<mailto:[email-masked]>>, Qing An <[email-masked]<mailto:[email-masked]>>
Subject: Re: Re: Multi-path QUIC Extension Experiments

Hi Robin,

Cheers,
Yunfei

------------------Original Mail ------------------
Sender:Robin MARX <[email-masked]<mailto:[email-masked]>>
Send Date:Wed Jul 14 07:39:37 2021
Recipients:Yanmei Liu <[email-masked]<mailto:[email-masked]>>
CC:quic <[email-masked]<mailto:[email-masked]>>, Ma, Yunfei <[email-masked]<mailto:[email-masked]>>, Christian Huitema <[email-masked]<mailto:[email-masked]>>, Qing An <[email-masked]<mailto:[email-masked]>>, 李振宇 <[email-masked]<mailto:[email-masked]>>, matt.joras <[email-masked]<mailto:[email-masked]>>, lucaspardue.24.7 <[email-masked]<mailto:[email-masked]>>
Subject:Re: Multi-path QUIC Extension Experiments
Hello Yanmei,

Best regards,
Yanmei

--

dr. Robin Marx

Error! Filename not specified.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Jul 2021 10:29:52 -0700

Yunfei, Yanmei, Qing and their colleagues are pointing a very important 
issue in multipath protocol design: in some cases, using multiple paths 
results in worse quality than using a single path. It is rather 
important to delineate exactly how that happens, and to understand why 
cooperation between transport and application is required to solve the 
issue.

For simplification, let's assume a basic "equal cost multipath" 
scenario, in which the transport splits the traffic between two paths of 
equal delay and equal capacity. When both paths are working well, data 
arrives twice faster than if a single path was used, and everyone is 
happy. But of course, things do not keep working well all the time, 
which is when we will see bad results. For example:

* if one path experiences packet losses and the other does not, some 
packets sent on the lossy path will have to be transmitted again. This 
will cause delays, and may well cause "head of line blocking".

* if one path suddenly slows down, maybe because of radio issues, it 
will take some time for congestion control algorithms to detect the drop 
in capacity. Queues will build up during that time, which again may well 
cause HoB.

One extreme solution is to always send copies of the data on both paths, 
and let the application use whichever copy arrives first. That works, 
but at the cost of massive overhead. We don't want to incur that 
overhead all the time. I like the suggestion of using application state 
to drive when the transport should use this kind of redundancy, and when 
it should not.

-- Christian Huitema

---

From: Alexis Norech <[email-masked]>
Date: Mon, 19 Jul 2021 23:15:25 +0400

In the best case, where the data is sent successfully with minimal packet
loss, using multipath is quite good, and having an overhead should probably
be avoided. In these cases, send copies on both paths is truly not a
solution.

On the other hand, in the worst case, you need to detect and handle the
packet loss, and waiting for the data in the other path will surely cause
more delay than just sending the same data with only a single "good" path.

Resending the "bad path" packets to the "good path" looks like a solution,
but that's considering that the traffic is stable, which may not be the case
if we are in this "bad" case. That would just append packets to a queue,
and would ultimately create another bottleneck if the "good path" become
a "bad path" itself.

The network will ultimately be unstable in some cases, so it would be best
to not cause too much overhead in the paths where the traffic is already
unstable.

- A. Cheron

On 7/19/2021 21:30 +0400, Christian Huitema <[email-masked]> wrote:

-- Christian Huitema

It sounds like this problem is not inherent to single-connection
multi-path, but will be present in any multi-path implementation, including
multiple-tcp-connections used with application-layer muxing.
If this is correct, then it isn’t really a ‘QUIC problem, but rather an
implementation/scheduling/CC problem.
That isn’t saying that it isn’t interesting or important to solve, but
rather that the protocol itself need not change to solve the problem for
generic-QUIC transport.

H3, OTOH, may suffer from this at L7+ proxies without some changes to QUIC
or H3, but that is a much longer conversation that doesn’t require
multi-path to happen.

-=R

From: Yunfei Ma <[email-masked]>
Date: Sunday, July 18, 2021 at 1:17 AM
To: Charles 'Buck' Krasic <[email-masked]>, Mirja Kuehlewind <
[email-masked]>, Roberto Peon <[email-masked]>
Cc: "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>, Christian
Huitema <[email-masked]>, Yanmei Liu <[email-masked]>,
"lucaspardue.24.7" <[email-masked]>, quic <[email-masked]>, Qing
An <[email-masked]>, Yunfei Ma <[email-masked]>
Subject: Re: Multi-path QUIC Extension Experiments

video frame with a min-RTT scheduler, which is default in MPTCP. For each
packet, the scheduler selects a path for that packet to transmit. The

Cheers,
Yunfei

[email-masked]<mailto:[email-masked]>> wrote:

On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=[email-masked]
<mailto:[email-masked]>> wrote:
I too am curious!
There are only two ways to handle flow control—overcommit, or don’t
overcommit.

The “don’t overcommit” choice leads to blocking, since any of that resource
allocated to one path can’t be used by the other.
The “overcommit” choice either leads to OOM, or throwing out some
successfully transmitted and received data.

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf
of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]<mailto:
[email-masked]>>
Date: Friday, July 16, 2021 at 6:15 AM
To: "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=
[email-masked]<mailto:[email-masked]>>,
Robin MARX <[email-masked]<mailto:[email-masked]>>, Yanmei
Liu <[email-masked]<mailto:[email-masked]>>
Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>, 李振宇 <
[email-masked]<mailto:[email-masked]>>, Christian Huitema <
[email-masked]<mailto:[email-masked]>>, "lucaspardue.24.7" <
[email-masked]<mailto:[email-masked]>>, quic <
[email-masked]<mailto:[email-masked]>>, Qing An <[email-masked]
<mailto:[email-masked]>>
Subject: Re: Multi-path QUIC Extension Experiments

Hi Yunfei,

Mirja

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf
of "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=
[email-masked]<mailto:[email-masked]>>
Date: Thursday, 15. July 2021 at 04:18
To: Robin MARX <[email-masked]<mailto:[email-masked]>>,
Yanmei Liu <[email-masked]<mailto:[email-masked]>>
Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>, 李振宇 <
[email-masked]<mailto:[email-masked]>>, Christian Huitema <
[email-masked]<mailto:[email-masked]>>, "lucaspardue.24.7" <
[email-masked]<mailto:[email-masked]>>, quic <
[email-masked]<mailto:[email-masked]>>, Qing An <[email-masked]
<mailto:[email-masked]>>
Subject: Re: Re: Multi-path QUIC Extension Experiments

Hi Robin,

First, the head of line blocking discussed here is called multi-path
head-of-line blocking or MP-HoL blocking, and its root cause is quite
different from the stream HoL blocking usually discussed in QUICv1. The
MP-HoL blocking happens when one path blocks the other path, not when one
stream blocks the other stream. Please note that we indeed use multiple
streams, for example, different video requests are carried in different
QUIC streams. QUIC’s stream multiplexing ability and its benefits still
hold in this scenario.

Second, regarding packet scheduling mode, right now, in our Taobao A/B
test, we transmit packets on multiple paths simultaneously. However, you
can definitely use traffic switching only and choose to switch when one
path could not meet your bandwidth requirement. Basically, if you use
multiple paths simultaneously, you get the most elasticity from a resource
pooling perspective. It really comes down on what your application needs.
We will also update the packet scheduling section soon in a newer version
of the draft, in which we plan to include more discussions on the packet
scheduling policy.

Third, regarding the benefits of more bandwith versus the "downsides".
Whether you want more bandwidth depends on your application. For videos,
yes, more bandwidth is extremely helpful in improving the long tail QoE,
which is an important target for Taobao. We find multi-path QUIC helps us
improve two important metrics, rebuffer rate and video start-up delays. In
the past, if you work on multi-path scheduling that does not collaborate
close enough with applications such as MPTCP, the MP-HoL blocking becomes
the downside that cripples the performance. However, the user space nature
of QUIC provides us the opportunity to solve this problem, so now our
conclusion is that you can enjoy the benefits of more bandwidth and more
reliable connectivity from multi-path without much of the “downsides”.

I hope my answer is helpful, but feel free to let me know if you have any
additional comments.

Cheers,
Yunfei

from Alimail macOS<
https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/
------------------Original Mail ------------------
Sender:Robin MARX <[email-masked]<mailto:[email-masked]>>
Send Date:Wed Jul 14 07:39:37 2021
Recipients:Yanmei Liu <[email-masked]<mailto:
[email-masked]>>
CC:quic <[email-masked]<mailto:[email-masked]>>, Ma, Yunfei <
[email-masked]<mailto:[email-masked]>>, Christian
Huitema <[email-masked]<mailto:[email-masked]>>, Qing An <
[email-masked]<mailto:[email-masked]>>, 李振宇 <
[email-masked]<mailto:[email-masked]>>, matt.joras <[email-masked]
<mailto:[email-masked]>>, lucaspardue.24.7 <[email-masked]
<mailto:[email-masked]>>
Subject:Re: Multi-path QUIC Extension Experiments
Hello Yanmei,

Thanks for the additional results on an interesting topic. I'm looking
forward to reading the SIGCOMM paper.

I was a bit surprised to (apparently) see HOL blocking mentioned as a major
issue, as that's one of the things QUIC aims to be better at than TCP.
It's a bit difficult to understand from the slides, but it seems like
you're sending packets for a single stream (Stream ID 1 in the diagrams) on
both the slow and fast path, which would indeed induce HOL blocking.
Consequently, I was wondering what the practical reasons are for you to
multiplex packets for a single stream over multiple paths, as opposed to
for example attaching a single stream to a single path (say: high priority
streams use the fast path for all their packets).

I see this mentioned a bit in the draft under "packet scheduling", where it
talks about switching paths once the cwnd is full for one. That indeed
leads to the behaviour seen in the slides, but that's my question: why
would you take those approaches then?
Are there so many cases where the additional "bandwidth" from using
multiple path's cwnd for a single stream outweigh the downsides of HOL
blocking? Relatedly: what are the packet loss rates you've observed on real
networks?
Have you experimented with e.g., tying streams to paths more closely? Does
that work better or worse? Why?

On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=
[email-masked]<mailto:[email-masked]>>
wrote:
Hi everyone,

We have finished some experiments about deploying multi-path quic extension(
https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)<
https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)> in Alibaba
Taobao short-form video streaming, and the experiment results are concluded
in the slides (attached file).
If anyone is interested in the experimental details about multi-path quic,
please let us know.
All the feedbacks and suggestions are appreciated!

Best regards,
Yanmei

--

dr. Robin Marx

www.uhasselt.be<
https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/

---

From: "Charles 'Buck' Krasic" <[email-masked]>
Date: Mon, 19 Jul 2021 14:41:28 -0700

Yunfei, thank you for the clarification, it makes a great deal of sense.

It feels very reminiscent of QPACK to me, in the sense that it would be
helpful for the multi-path scheduling to have insight into application data
dependencies, so that it can use that information to avoid scheduling
decisions that would be vulnerable to HoL blocking, conversely to take
advantage when such dependencies are absent.

On Sun, Jul 18, 2021 at 1:17 AM Yunfei Ma <[email-masked]> wrote:

>>> Underlying this is a fun question: Which inefficiency is worse? Not
>>> using resources that should be used (i.e. from choosing to not overcommit),
>>> or sometimes redundantly using a resource (from choosing to overcommit)?
>>> *From: *QUIC <[email-masked]> on behalf of "Ma, Yunfei" <
>>> yunfei.ma=[email-masked]>
>>> I assume there is much existing work on this for MPTCP, but I also
>>> assume some of that changes due to QUIC's independent streams / stream

---

From: Roberto Peon <[email-masked]>
Date: Mon, 19 Jul 2021 22:19:40 +0000

If we have to send data along a path in order to discover properties about that path, then sending less data on the path means discovering less about that path.

The ideal would be to send *enough* data on any one path to maintain an understanding of its characteristics (including variance), and no more than that, and then to schedule the rest of the data to whichever path(s) are best at the moment.

Because the tradeoffs here are so deep, I don’t believe that there is an “optimal” that will work well for everything… however, it is interesting to think about a few canonical (mostly video) examples as motivations.
- Videoconferencing
- VoD with deep buffering
- small, reliable RPCs (e.g. financial transactions)

Each of these would likely wish to have different redundancy, efficiency, and latency tradeoffs.

I do believe that innovation in APIs to allow applications to express this kind of thing well is necessary if we wish to do a good job with multipath (or multiconnection).
-=R

From: Charles 'Buck' Krasic <[email-masked]>
Date: Monday, July 19, 2021 at 2:41 PM
To: Yunfei Ma <[email-masked]>
Cc: Mirja Kuehlewind <[email-masked]>, Roberto Peon <[email-masked]>, "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>, Christian Huitema <[email-masked]>, Yanmei Liu <[email-masked]>, "lucaspardue.24.7" <[email-masked]>, quic <[email-masked]>, Qing An <[email-masked]>, Yunfei Ma <[email-masked]>
Subject: Re: Multi-path QUIC Extension Experiments

It feels very reminiscent of QPACK to me, in the sense that it would be helpful for the multi-path scheduling to have insight into application data dependencies, so that it can use that information to avoid scheduling decisions that would be vulnerable to HoL blocking, conversely to take advantage when such dependencies are absent.

On Sun, Jul 18, 2021 at 1:17 AM Yunfei Ma <[email-masked]<mailto:[email-masked]>> wrote:
Hi Charles, Roberto, and Mirja:

Cheers,
Yunfei

Hi Yunfei,

Mirja

Hi Robin,

Cheers,
Yunfei

------------------Original Mail ------------------

Best regards,
Yanmei

--

dr. Robin Marx

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 19 Jul 2021 23:36:45 +0100

Interesting thread. There's a lot of stuff here.

Speaking as an individual, I'm reminded that congestion control and path
scheduling is an important factor for implementation of (the various)
multipath QUIC proposals that have been floated. But IIRC the view was that
these factors are not a core part of multipath QUIC protocol
specifications. Therefore, I'm keen to understand if there is something
that the QUIC transport protocol should or could provide to accommodate
these concerns. E.g. new or different forms of signalling to avoid or
improve on extant constraints.

Cheers
Lucas

On Mon, 19 Jul 2021, 23:19 Roberto Peon, <[email-masked]> wrote:

> If we have to send data along a path in order to discover properties about
> that path, then sending less data on the path means discovering less about
> that path.
>
>
> The ideal would be to send **enough** data on any one path to maintain an
> understanding of its characteristics (including variance), and no more than
> that, and then to schedule the rest of the data to whichever path(s) are
> best at the moment.
>
> Because the tradeoffs here are so deep, I don’t believe that there is an
> “optimal” that will work well for everything… however, it is interesting to
> think about a few canonical (mostly video) examples as motivations.
> - Videoconferencing
>
> I do believe that innovation in APIs to allow applications to express this
> kind of thing well is necessary if we wish to do a good job with multipath
> (or multiconnection).
>
> -=R
>
>
> *From: *Charles 'Buck' Krasic <[email-masked]>
> *Date: *Monday, July 19, 2021 at 2:41 PM
> *To: *Yunfei Ma <[email-masked]>
> *Cc: *Mirja Kuehlewind <[email-masked]>, Roberto Peon <
> [email-masked]>, "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>,
> Christian Huitema <[email-masked]>, Yanmei Liu <
> [email-masked]>, "lucaspardue.24.7" <
> [email-masked]>, Yunfei Ma <[email-masked]>
> *Error! Filename not specified.*
>
>
>
>

---

From: Yunfei Ma <[email-masked]>
Date: Mon, 19 Jul 2021 17:17:10 -0700

Hi Mikkel,

Thanks for the question. Let me clarify below:

> In earlier mails it was stated that HoL happens even if all packets of one
> stream is always sent on the same path. If you have time dependent data
> like a video frame, and the frame is not, say, interlaced, then all the
> data should be placed on the same stream or at least the same path. Of
> course you can using interlacing on multiple streams and choose to display
> a lower quality frame if HoL is detected. So if you want to use bandwidth
> sharing across paths, then yes, there might be HoL blocking.
>

In our case, each stream is allowed to transmit on multiple paths (Please
note that this is an implementation choice. The draft does allow you to
choose other strategies. I remember that Spencer Dawkins also had a draft
about the difference between switching, steering and aggregation.). We use
aggregation for a reason, if all packets of one stream are always sent on
the same path, the stream that is unfortunate to get a bad path(small
bandwidth, high loss or high latency) will suffer a lot. In such a
scenario, the overall robustness is not improved. I think an important goal
of using multi-path for wireless transport is to make your network more
stable. The elastic resource pooling ability when allowing each stream to
transmit on multiple paths is quite helpful; You are right. If you want to
enable bandwidth sharing, then, you need to deal with HoL blocking.

But I am interested in understanding what MP-HoL means when streams are not
> distributed on multiple paths because then at least the transport layer
> should not exhibit blocking beyond flow control. It can be that the
> application protocol has stream dependencies that introduce HoL blocking,
> but that is kind of a separe discussion, relevant as it may be.
>

I think the stream dependencies you mentioned here is a great point. In our
implementation, we introduced a stream-priority based reinjection which
tries to address such dependency (There is a figure in the material that
Yanmei sent). But we haven't tried when each stream is limited to a single
path. In our case, streams are distributed on multiple paths. I would
definitely want to hear more about the application you are dealing with,
and maybe for wired transport, such a design is needed.

Cheers,
Yunfei

---

From: Yunfei Ma <[email-masked]>
Date: Mon, 19 Jul 2021 17:30:02 -0700

Hi Roberto,

You are right. Basically it is the trade-off between redundancy(traffic
cost), latency, and bandwidth efficiency. That's what we are trying to
balance. An interesting thing we find is that you need QoE control feedback
to adjust your scheduling strategy in real time. For example, in VoD, when
the video buffer is high (you are far from rebuffering), your scheduling
can be conservative in order to reduce cost. But when your network becomes
unstable and you are about to encounter rebuffering, you should use more
aggressive scheduling, at the cost of more redundancy.

We also think different applications (videoconferencing, VoD, and RPC)
should use different policies and that is what QUIC is good at as a user
space protocol. I believe there is a lot more to be done as QUIC truly
enables accelerated innovations.

Cheers,
Yunfei

---

From: Yunfei Ma <[email-masked]>
Date: Mon, 19 Jul 2021 17:44:03 -0700

Hi Charles,

Your analogy to QPACK is intriguing. It feels exciting to see how the
underlying principles of different parts are so deeply connected.

Cheers,
Yunfei

On Mon, Jul 19, 2021 at 2:41 PM Charles 'Buck' Krasic <
[email-masked]> wrote:

>> Thanks a lot for your questions. As all three of you are curious about
>> the definition of MP-HoL, I am putting my answer into one reply.
>>
>> Short answer: the MP-HoL is not because of flow control, but rather, it
>> is related to the nature of path heterogeneity. In other words, MP-HoL can
>> More specifically, when you want to send out packets on different paths
>> at the same time, there is a scheduler to decide how to split your packets
>> and put them on different paths. However, in mobile networks, the network
>> paths could have very different path delays. MP-HoL blocking arises when
>> the packets sent earlier at the slow path arrive later than the packets
>> sent later at the fast path, causing out-of-order arrival. As a
>> consequence, the out-of-order packets are not eligible to be submitted to
>> applications, so the fast path has to wait.
>>
>> What makes the problem even more difficult is that in mobile networks,
>> the RTTs can change quickly, which makes accurate prediction very
>> difficult. Worst case is that when the scheduler thinks it is using the
>> fast path, it is actually using the slow path instead. As you can see, in
>> order to make multi-path transport efficient, it is important to solve this
>> problem and that's what we are doing in this project .
>>
>>>> I see this mentioned a bit in the draft under "packet scheduling",
>>>> where it talks about switching paths once the cwnd is full for one. That
>>>> indeed leads to the behaviour seen in the slides, but that's my question:
>>>> why would you take those approaches then?
>>>>
>>>> I'm mainly wondering how these tradeoffs evolve depending on the type
>>>> of paths available and if it's possible to make a model to drive this logic.
>>>>

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Jul 2021 18:10:17 -0700

> improve on extant constraints.

In first order, you can split the multipath issue in two. On one hand, 
the mechanics of setting up paths, acknowledging path specific data, 
etc. On the other hand, the fine art of setting path at the right time, 
scheduling the right amount of packets on these paths, sensing the 
properties of the paths, etc.

I think that the mechanics of multipath are well covered in 
https://datatracker.ietf.org/doc/draft-liu-multipath-quic/, and I think 
we know enough about the problem to adopt that in the WG and drive it to 
publication.

I also think that we have ample opportunities to learn more about the 
fine arts of multipath scheduling. I love the idea of sharing as much of 
that experience as possible between implementers, and I appreciate that 
the Ali Baba team has been doing just that. But we probably are still 
going to learn more about all that for several years.

If we really need to publish something, we might be able to publish some 
kind of basic scheduling algorithm, with known limitations and known 
applicability. A bit  like we published a Reno specification for QUIC 
and pretended that was good enough.

-- Christian Huitema

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 20 Jul 2021 07:22:33 +0100

Hi Christian,

> we know enough about the problem to adopt that in the WG and drive it to

As an individual: could you qualify a bit more about what you mean by "know
enough"? For instance, the draft contains things like path priority and QoE
control signals. I'm unsure if those are the solution to put on a standards
track document, it seems like we are still figuring things out. Placing
those aside, the proposal in the I-D does seem like it would allow
interoperable experimentation of multipath QUIC, which is a good way to
flush out the things we don't know. In the past, we've seen other
proposals, like draft-deconinck,  that seem like they allow similar
experimentation too. Has there been any progress on aligning/combining
proposals?

> going to learn more about all that for several years.

Agree. Thanks for sharing the results. As above, do the findings help
towards defining what interoperable QoE signals might be needed?

> and pretended that was good enough.

One important thing there was to provide a default congestion controller
that was known to be safe on the Internet. That way, QUIC implementations
could bootstrap everything else they needed. Are there such safety concerns
with multipath scheduling?

Cheers
Lucas

---

From: Lars Eggert <[email-masked]>
Date: Tue, 20 Jul 2021 10:14:28 +0300

> The ideal would be to send *enough* data on any one path to maintain an understanding of its characteristics (including variance), and no more than that, and then to schedule the rest of the data t…

^^^ This.

Because the Internet has no explicit network-to-endpoint signaling, an endpoint must build its understanding of the properties of a path by exercising it, and specifically exercising it to a degree that causes queues to form (to obtain "under load" RTTs, see bufferbloat) and congestion loss to happen (to obtain an understanding of available path capacity.) Some people have called this "putting pressure on a path".

There has been a long-standing assumption that if you exercised a path in the (recent) past you can probably assume that the properties haven't changed much if you want to start exercising it again. This is why heuristics like caching path properties (RTTs, etc.) are often of benefit - often, but not always, and maybe never in some scenarios (e.g., overcommitted CGNs.)

There has been some work on this in the past for MPTCP. For example, on mobile devices - which most often have multiple possible paths to a destination via WiFi and cellular - exercising multiple paths comes at a distinct increase in energy usage. So you need a heuristic to determine if the potential benefit of going multipath is worth the energy cost of probing multiple paths before you do so.

Thanks,
Lars

---

From: Robin MARX <[email-masked]>
Date: Tue, 20 Jul 2021 11:28:26 +0200

Hello Yunfei,

Thanks for your extended explanations on Multipath HoL-blocking and
especially:

> I think the stream dependencies you mentioned here is a great point. In
our implementation, we introduced a stream-priority based reinjection which

This is exactly what I was trying to explore in my previous mail. You're
basically intentionally causing (or perhaps risking?) HOL blocking because
you split a single stream over multiple paths.
As noted by Christian with the 'equal cost multipath', this can have
bandwidth usage benefits, but only if paths are usable/similar. If not, HOL
blocking might undo all the benefits you get from this setup (and using a
single path per stream would be better).
So my question was: where is the inflection point where you might decide to
switch modes? At which parameters is one better than the other?
I'd hoped you would have experimented with the fixed-path-per-stream setup
to get some insight into this.

In my mind, the idea of doing a purely transport-level multipath scheduler
(i.e., without taking into account application layer streams / data
dependencies / etc.)
has historically made some sense for TCP / for completely separated stacks,
as the transport didn't have that type of information available.
It is however utterly strange to me that this approach would continue for
QUIC (at least in endpoint multipath, not things like in-network
aggregators that have been discussed),
where we have clear splits between streams and (hopefully) already some
type of prioritization information for each stream.
For QUIC, I'd expect one-path-per-stream to be the default, with
multiple-paths-per-stream to be an edge case if you have a single,
high-traffic stream (which I do assume is your situation with a video
stream).

With best regards,
Robin

On Tue, 20 Jul 2021 at 09:15, Lars Eggert <[email-masked]> wrote:

> > If we have to send data along a path in order to discover properties
> about that path, then sending less data on the path means discovering less
> about that path.
> >
> > The ideal would be to send *enough* data on any one path to maintain an
> Because the Internet has no explicit network-to-endpoint signaling, an
> endpoint must build its understanding of the properties of a path by
> exercising it, and specifically exercising it to a degree that causes
> queues to form (to obtain "under load" RTTs, see bufferbloat) and
> congestion loss to happen (to obtain an understanding of available path
> capacity.) Some people have called this "putting pressure on a path".
>
> There has been a long-standing assumption that if you exercised a path in
> the (recent) past you can probably assume that the properties haven't
> changed much if you want to start exercising it again. This is why
> heuristics like caching path properties (RTTs, etc.) are often of benefit -
> often, but not always, and maybe never in some scenarios (e.g.,
> overcommitted CGNs.)
>
> There has been some work on this in the past for MPTCP. For example, on
> mobile devices - which most often have multiple possible paths to a
> destination via WiFi and cellular - exercising multiple paths comes at a
> distinct increase in energy usage. So you need a heuristic to determine if
> the potential benefit of going multipath is worth the energy cost of
> probing multiple paths before you do so.
>
> Thanks,
> Lars
>
>

-- 

dr. Robin Marx

www.uhasselt.be

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 20 Jul 2021 12:17:21 +0200

> There has been a long-standing assumption that if you exercised a path in the (recent) past you can probably assume that the properties haven't changed much if you want to start exercising it again…

This is rough assumption for WiFi. I just measured one laptop on a WiFi n network going from no service to 1Mb/s to 10Mb/s moving only a few meters, and then 100Mb/s moving some more. You can rapidly transition between a network being available and not available, and if you do not handle some heuristics, you can get some bad artefacts. Mobile can of course also vary a lot when going from 4G to 3G to no signal in a tunnel on a train.

Mikkel

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 20 Jul 2021 12:22:39 +0200

> On 20 Jul 2021, at 11.28, Robin MARX <robin.marx=[email-masked]> wrote:
> 
> It is however utterly strange to me that this approach would continue for QUIC (at least in endpoint multipath, not things like in-network aggregators that have been discussed),
> where we have clear splits between streams and (hopefully) already some type of prioritization information for each stream. 
> For QUIC, I'd expect one-path-per-stream to be the default, with multiple-paths-per-stream to be an edge case if you have a single, high-traffic stream (which I do assume is your situation with a video stream). 

Already with QUIC v1 there is a lot unsaid about the scheduling and stream prioritisation. As I understand, it is expected that certain application level protocols can place demands on the QUIC transports they support, but specifying a specific API for the transport would be very hard, and limiting. H3 naturally implies certain features in the transport API.

We did try being more elaborate with H3 stream priorities, even though this was only at application level, and even that proved too complex.

Thus, I doubt that tying the application layer closer to the transport layer will be verysuccessfull, but there will be some API features expected for the most successful MP app protocols, and it is certainly worth discussing what these should be, just as for H3, but it does not imply a tight coupling at the transport specification level.

Mikkel

---

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 20 Jul 2021 10:09:41 -0500

Hi Christian,

When "when we will see bad results" I think that returning to single path
QUIC should be OK.
Maintaining multipath may not be possible all the time, as we are talking
about transmission over very different technologies.

Behcet
On Mon, Jul 19, 2021 at 12:30 PM Christian Huitema <[email-masked]>
wrote:

> > H3, OTOH, may suffer from this at L7+ proxies without some changes to
> QUIC or H3, but that is a much longer conversation that doesn’t require
> multi-path to happen.
> >
> > -=R
> >
> > Cc: "matt.joras" <[email-masked]>, 李振宇 <[email-masked]>,
> [email-masked]<mailto:[email-masked]>> wrote:
> > I too am curious!
> > From: QUIC <[email-masked]<mailto:[email-masked]>> on
> behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]
> <mailto:[email-masked]>>
> Robin MARX <[email-masked]<mailto:[email-masked]>>,
> > Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>,
> 李振宇 <[email-masked]<mailto:[email-masked]>>, Christian Huitema <
> > From: QUIC <[email-masked]<mailto:[email-masked]>> on
> behalf of "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=
> > Cc: "matt.joras" <[email-masked]<mailto:[email-masked]>>,
> 李振宇 <[email-masked]<mailto:[email-masked]>>, Christian Huitema <
> > Sender:Robin MARX <[email-masked]<mailto:[email-masked]
> >>
> <mailto:[email-masked]>>, lucaspardue.24.7 <
> [email-masked]<mailto:[email-masked]>>
> extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)<

---

From: Christian Huitema <[email-masked]>
Date: Tue, 20 Jul 2021 10:51:56 -0700

> experimentation too. Has there been any progress on aligning/combining

Good question. I much prefer draft-liu, because it is defines fewer new 
frames and concepts and is easier to implement -- I did implement it in 
picoquic. There is possibly an argument that some of the extensions in 
draft-deconinck should be integrated in draft-liu, or maybe that some of 
the content of draft-liu should be moved out. I think adopting draft-liu 
would be force these discussions.

> towards defining what interoperable QoE signals might be needed?

The QoE signals carry preferences from one application end to the other, 
such as "my video buffer is nearly empty, please do something to prevent 
a stall." Such signals are largely application dependent, and there will 
always be a discussion between making that a common transport frame that 
may carry application specific data versus just make it part of the 
application. I like having the QoE frame, because it is a good place to 
hook a QoE API, as in "dear application, transport just received this 
QoE frame; should it change some of what it is doing?"

Draft-liu also includes a PATH-STATUS frame, which provides generic 
controls with two parameters, a "status" and a "priority". In my 
implementation, the status field is only used to distinguish between 
path that should be dropped (status=abandon) and other paths. The 
priority field is used in the scheduling algorithm. Packets are only 
sent over the paths with high priority, unless those paths are marked as 
"suspect", for example because high packet loss has been observed.

The priority field can be used in the common scenario in which a client 
has established both a Wi-Fi and a cellular path, but would rather not 
use the cellular path if that can be avoided.

> with multipath scheduling?

Define "safety"? I think there is common agreement that each path should 
be subject to its own congestion control, as specified in the base 
transport spec. Using congestion control addresses the main network 
safety concern, i.e., not causing undue congestion to other network 
users. There have been discussions in MP-TCP of "overall fairness", in 
which multipath transports would be subject to stricter congestion 
control than single path transports. For example, transports that have 
access to a good path should try not compete for capacity with 
single-path users of the other paths. Something like using Cubic or BBR 
on the main path, LEDBAT on the other paths. That could be done, with 
some extra work when the status of paths changes. But then, in QUIC, 
such restrictions would be almost impossible to enforce.

The basic algorithm for multipath scheduling is simple: consider all the 
available paths; send the next packet on a given path if congestion 
control allows it on that path. After that, transport need to take into 
account path priority and also some "health" indicator, for example 
whether high losses or high delays were observed recently. Sending data 
on a path with dubious health is likely to cause losses and delays, so 
it should either not be done at all, or only for redundant 
transmissions. Then of course there is a need for probing traffic to 
assess when paths are healthy, which can be either redundant traffic, or 
just made up traffic (picoquic just uses ping+pad). I think the baseline 
specification should stop there. After that, there are per-application 
issues, for example dealing with applications in which some traffic is 
higher priority than other -- which is what the QoE discussion is about.

-- Christian Huitema