Subject: Re: Multi-path QUIC Extension Experiments

From: Yunfei Ma <[email-masked]>
Date: Thu, 16 Sep 2021 23:38:11 -0700

Hi Robin,

Thanks for the question and sorry for the late reply. With regard to the
experiment, the goal was to achieve bandwidth, latency and reliability for
a video stream at the same time. Intuitively, adding more paths should give
you immediate improvement. But, as you pointed out, HoL might undo the
benefit. So what we found was a way to overcome this, and the technique was
to use QoE feedback in conjunction with scheduling techniques. What the
results tell us is if you have a stream that is time critical and bandwidth
intensive, with the user-space nature of QUIC, you now have a way to go
forward using multipath.

But please keep in mind that you can choose to use whatever scheduling
algorithm you like with the draft. In the experiment, we are trying to push
the limit, but it is indeed one of the many possibilities. If you want to
send streams that do not require every high reliability or high bandwidth,
you can definitely use the fixed-path-per-stream strategy. Recently, we
also have also done testing with fixed-path-per-stream with
draft-liu-multipath-quic for certain scenarios. Here are some observations,
if you have a large number of streams, having an assigned path for each can
actually give you more combined bandwidth utilization, but not every stream
benefits and some could suffer because of a bad path assignment. If you
really care about the long tail performance, then, I would recommend
sending a stream on multiple paths.

How to use multi-path and which scheduling mode depends on what you try to
optimize. I hope the draft can provide a starting point, and please feel
free to go beyond.

Cheers,
Yunfei

> Thanks for your extended explanations on Multipath HoL-blocking and
> especially:
>
> > I think the stream dependencies you mentioned here is a great point. In
> our implementation, we introduced a stream-priority based reinjection which
> tries to address such dependency (There is a figure in the material that
> Yanmei sent). But we haven't tried when each stream is limited to a single
> path. In our case, streams are distributed on multiple paths. I would
> definitely want to hear more about the application you are dealing with,
> and maybe for wired transport, such a design is needed.
>
> This is exactly what I was trying to explore in my previous mail. You're
> basically intentionally causing (or perhaps risking?) HOL blocking because
> you split a single stream over multiple paths.
> As noted by Christian with the 'equal cost multipath', this can have
> bandwidth usage benefits, but only if paths are usable/similar. If not, HOL
> blocking might undo all the benefits you get from this setup (and using a
> single path per stream would be better).
> So my question was: where is the inflection point where you might decide
> to switch modes? At which parameters is one better than the other?
> I'd hoped you would have experimented with the fixed-path-per-stream setup
> to get some insight into this.
>
> In my mind, the idea of doing a purely transport-level multipath scheduler
> (i.e., without taking into account application layer streams / data
> dependencies / etc.)
> has historically made some sense for TCP / for completely separated
> stacks, as the transport didn't have that type of information available.
> It is however utterly strange to me that this approach would continue for
> QUIC (at least in endpoint multipath, not things like in-network
> aggregators that have been discussed),
> where we have clear splits between streams and (hopefully) already some
> type of prioritization information for each stream.
> For QUIC, I'd expect one-path-per-stream to be the default, with
> multiple-paths-per-stream to be an edge case if you have a single,
> high-traffic stream (which I do assume is your situation with a video
> stream).
>
> With best regards,
> Robin
>
>
>
>
> On Tue, 20 Jul 2021 at 09:15, Lars Eggert <[email-masked]> wrote:
>
>> On 2021-7-20, at 1:19, Roberto Peon <fenix=[email-masked]>
>> wrote:
>> >
>> > If we have to send data along a path in order to discover properties
>> about that path, then sending less data on the path means discovering less
>> about that path.
>> >
>> > The ideal would be to send *enough* data on any one path to maintain an
>> understanding of its characteristics (including variance), and no more than
>> that, and then to schedule the rest of the data to whichever path(s) are
>> best at the moment.
>>
>> ^^^ This.
>>
>> Because the Internet has no explicit network-to-endpoint signaling, an
>> endpoint must build its understanding of the properties of a path by
>> exercising it, and specifically exercising it to a degree that causes
>> queues to form (to obtain "under load" RTTs, see bufferbloat) and
>> congestion loss to happen (to obtain an understanding of available path
>> capacity.) Some people have called this "putting pressure on a path".
>>
>> There has been a long-standing assumption that if you exercised a path in
>> the (recent) past you can probably assume that the properties haven't
>> changed much if you want to start exercising it again. This is why
>> heuristics like caching path properties (RTTs, etc.) are often of benefit -
>> often, but not always, and maybe never in some scenarios (e.g.,
>> overcommitted CGNs.)
>>
>> There has been some work on this in the past for MPTCP. For example, on
>> mobile devices - which most often have multiple possible paths to a
>> destination via WiFi and cellular - exercising multiple paths comes at a
>> distinct increase in energy usage. So you need a heuristic to determine if
>> the potential benefit of going multipath is worth the energy cost of
>> probing multiple paths before you do so.
>>
>> Thanks,
>> Lars
>>
>>
>
> --
>
> dr. Robin Marx
> Postdoc researcher - Web protocols
> Expertise centre for Digital Media
>
> *Cellphone *+32(0)497 72 86 94
>
> www.uhasselt.be
> Universiteit Hasselt - Campus Diepenbeek
> Agoralaan Gebouw D - B-3590 Diepenbeek
> Kantoor EDM-2.05
>
>
>

---

From: Ian Swett <[email-masked]>
Date: Fri, 17 Sep 2021 16:05:54 -0400

I think the PATH-STATUS frame is useful, and I think the 'Path Status'
field is clearly useful.  I'm unsure if the 'Path Priority' is capturing
quite what we might like.  Admittedly, priority is always hard, as we've
found before.

In particular, if I have a free Wifi connection that may be slow or only
somewhat connected, and a cell connection that's very solid, I'd like to
send a new HTTP/3 request on the cell link or maybe even both to minimize
tail latency, but maybe only smaller responses can come back via cell?  I'm
not sure how to encode this mix of metered and unmetered connections in
PATH_STATUS today?  The best I can think of is something along the lines of
Christian's approach of 'Always use Wifi and switch to cell if it fails',
but that seems like a suboptimal user experience.

As an editorial nit, I'd suggest you remove the word 'Path' from the
beginning of the field names in the PATH_STATUS frame.  Given they're all
in the PATH_STATUS frame, I don't think it's necessary.

Thanks, Ian

On Fri, Sep 17, 2021 at 2:39 AM Yunfei Ma <[email-masked]> wrote:

>> I'd hoped you would have experimented with the fixed-path-per-stream
>> setup to get some insight into this.
>>
>> In my mind, the idea of doing a purely transport-level multipath
>> scheduler (i.e., without taking into account application layer streams /
>> data dependencies / etc.)
>>> > The ideal would be to send *enough* data on any one path to maintain
>>> an understanding of its characteristics (including variance), and no more
>>> than that, and then to schedule the rest of the data to whichever path(s)
>>> are best at the moment.
>>>
>>> ^^^ This.
>>>
>>> There has been a long-standing assumption that if you exercised a path
>>> in the (recent) past you can probably assume that the properties haven't
>> *Cellphone *+32(0)497 72 86 94 <+32%20497%2072%2086%2094>
>>
>> www.uhasselt.be

---

From: Yunfei Ma <[email-masked]>
Date: Fri, 17 Sep 2021 18:39:42 -0700

Hi Ian,

Thanks for the comments. Please see my reply below.

> but that seems like a suboptimal user experience.

The problem you point out is a very important one, especially for
end-to-end multi-path use cases. Usually on the end-host side, wifi is
free, but cellular costs money, albeit In certain cases, cellular could be
free depending on the user's data plan. With the priority field in
PATH_STATUS, here is what one can do: sending data on paths that are
available and have the same priority level. For example, when the user has
an unlimited cellular data plan or he/she actually thinks the cellular cost
is acceptable, we can set the wifi and cellular path with the same
priority, and let the scheduler only choose from the path set consisting of
the highest priority level paths. In another example, if the user prefers
wifi, he/she can lower the priority of the cellular, thus only sending on
wifi when wifi is available and switch to cellular when wifi becomes
unavailable, as you just mentioned above.

In connection migration, you have the mechanism of migrating when a path
has faded and then switching back to the "default" network, so the cost
issue is well addressed. In multi-path, I think it is desirable to have a
switch that the end user can turn on/off which controls the path priority.
In our previous experiment, we used zero rating. We did so because we
wanted to ensure the results were not biased. We also want to learn better
approaches as it is directly related to the product design.

> in the PATH_STATUS frame, I don't think it's necessary.

Thanks for pointing this out. I think you are right and we should modify
the wording here a bit.

Cheers,
Yunfei

On Fri, Sep 17, 2021 at 2:39 AM Yunfei Ma <[email-masked]> wrote:
>> How to use multi-path and which scheduling mode depends on what you try
>> to optimize. I hope the draft can provide a starting point, and please feel
>> free to go beyond.
>>
>> Cheers,
>> Yunfei
>>
>>
>>
>>
>>> > I think the stream dependencies you mentioned here is a great point.
>>> In our implementation, we introduced a stream-priority based reinjection
>>> which tries to address such dependency (There is a figure in the material
>>> that Yanmei sent). But we haven't tried when each stream is limited to a
>>> single path. In our case, streams are distributed on multiple paths. I
>>> would definitely want to hear more about the application you are dealing
>>> with, and maybe for wired transport, such a design is needed.
>>>
>>> It is however utterly strange to me that this approach would continue
>>> for QUIC (at least in endpoint multipath, not things like in-network

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 21 Sep 2021 11:37:56 -0500

Hi, Yunfei,

> optimize. I hope the draft can provide a starting point, and please feel

This is an important point that we shouldn't lose sight of. I think we're
tripping over an assumption about how smart "QUIC with multiple paths"
should be about scheduling packets on each path, versus how much
applications should be involved in scheduling packets on each path.

Applications using MP-TCP, especially for applications like bandwidth
bonding, could reasonably let MP-TCP "do the right thing" - if you've got
paths, use them, and MP-TCP can manage multiple paths with different path
characteristics fairly well if the goal is to use all the available
bandwidth.

Applications using QUIC with multiple paths have a lot more possibilities.
For instance, using reliable streams and unreliable datagrams
simultaneously across multiple paths.

ISTM that we should be kind of clear about what QUIC implementations will
be responsible for, and what decisions applications will be responsible
for.

I have been maintaining a collection of various goals for path selection
(most recently, in
https://datatracker.ietf.org/doc/html/draft-dawkins-quic-multipath-selection-01#section-3).
It's not a short list.

If we're also including user preferences that would include knowing the
difference between cellular connections that are metered, cellular
connections that are unmetered but being throttled, and cellular
connections that are unmetered and unthrottled, and cross-matching them
with wifi connections that are likely to work better than cellular
connections vs. wifi connections that are only intermittently performing
well, that's not a small amount of complexity.

Maybe that complexity should be managed by applications until we have a
better understanding of what applications are actually doing?

Best,

Spencer

---

From: Christian Huitema <[email-masked]>
Date: Tue, 21 Sep 2021 11:31:05 -0700

> well, that's not a small amount of complexity.

This leads to the problem of asymmetric knowledge between client and 
server. On the client, applications can use system APIs and user 
preferences to identify which connections are "metered, cellular 
connections that are unmetered but being throttled, and cellular 
connections that are unmetered and unthrottled." But the server side 
does not know that. Yet in many scenarios most of the traffic volume 
originates from the server, and the scheduling choices of the server may 
cause increased bills or exhaustion of quotas for the client. Which 
implies that there should be some way for the client to signal how the 
server shall consider connection.

I think this "knowledge sharing" is independent from "application 
preferences". The work at Ali Baba shows that there is a lot of benefits 
in incorporating application states in scheduling decisions. For 
example, if the video streaming application notices that the video 
buffers are emptying too fast, it might tell the server to start using 
more bandwidth, even if that means using metered connections. But that 
application state, "video buffers are emptying", is not the same as the 
network state, "path 3 is on a metered connection".

I don't think that we will be able to fully standardize the way 
applications pass their preferences. That's why in draft-liu the 
"QOE_CONTROL_SIGNALS" is mostly used to pass application-defined 
signals. But we might be able to have a reasonable set of codes (or 
priorities) explaining the "state of a path", and use that in the 
"PATH_STATUS" frame.

-- Christian Huitema

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 21 Sep 2021 14:14:13 -0500

Hi, Christian,

> server shall consider connection.

Yes, very much so. My point was that this is complex, and your point about
asymmetric knowledge is well-taken. The idea that "clients" (UEs) know
things that "servers" (in this case, UPFs) don't know is also popping up in
3GPP ATSSS discussions (for example, in clause 5.32.8 of 3GPP TS 23.501
V17.1.1):

*UE-assistance indicator: *This indicator may be provided only when the
Steering Mode is Load-Balancing. When provided by the network, it indicates
that (a) the UE may decide how to distribute the UL traffic of the matching
SDF based on the UE's internal state (e.g. when the UE is in the special
internal state, e.g. lower battery level), and that (b) the UE may inform
the UPF how it decided to distribute the UL traffic of the matching SDF. In
the normal cases, although with this indicator provided, the UE shall
distribute the UL traffic as indicated by the network.

> network state, "path 3 is on a metered connection".

I like this (especially because what you're talking about is meaningful to
applications and independent of changes in underlying network technologies
along a path).

> priorities) explaining the "state of a path", and use that in the

You and I have shared the hope that we would be able to describe building
blocks that applications can use, rather than define new codes every time
someone comes up with a new way of distributing packets across multiple
paths, for a while now -
https://datatracker.ietf.org/doc/html/draft-dawkins-quic-what-to-do-with-multipath-03#section-2.4
was based on a discussion we had, and that was submitted last January.

I hope the QUIC working group's return to multipath discussions gives us a
chance to make progress on that.

Best,

Spencer

---

From: Yunfei Ma <[email-masked]>
Date: Tue, 21 Sep 2021 19:43:03 -0700

Dear Spencer,

Thanks for your comments. To achieve real performance gain, it might take
you to design some "smart" algorithms depending on what you are working on,
but what we were hoping to understand in the earlier experiment was what
basic features we should include to enable a larger design space. In other
words, the algorithm may be more complex and flexible, but the protocol
feature that enables it is simple and independent of the algorithm, like
the QoE_Control_Signal frame defined in draft-liu. Just like what has been
pointed out by Christian, it solves the problem of how to close the
asymmetric knowledge gap between a client and a server. The "knowledge
sharing" mechanism can definitely go beyond our early experiments. I think
the power-saving use case you just mentioned serves as another example
where such a knowledge sharing mechanism can help.

Cheers,
Yunfei