Subject: IETF Last Call for QUIC

From: Lars Eggert <[email-masked]>
Date: Fri, 25 Sep 2020 13:00:30 +0300

Hi,

you just saw a flurry of emails indicating that we requested publication for the latest revision of the "base drafts". This is a major milestone for the QUIC WG, and it would not have been possible without the huge amount of work over a number of years by all the WG participants, and especially our group of editors!

Our AD Magnus will now prepare the document set for their IETF Last Call, during which new issues are likely to be opened, by the wider IETF community but also possibly by WG participants. We'll tag those as "ietf-lc" in GitHub, and discuss and handle them there per our usual way of working.

In parallel to progressing the "base drafts" towards RFC publications, the WG should now also begin to pick up the pace on our other adopted work items (ops drafts, extensions, etc.)

One important other discussion item is what to do about the multipath extension milestone, which some have suggested should be dropped, while others still show interest to pursue it.

Thanks,
Lars, Lucas and Mark

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Sun, 27 Sep 2020 11:07:27 -0500

Dear QUIC working group,

On Fri, Sep 25, 2020 at 5:00 AM Lars Eggert <[email-masked]> wrote:

> In parallel to progressing the "base drafts" towards RFC publications, the
> WG should now also begin to pick up the pace on our other adopted work
> items (ops drafts, extensions, etc.)
>
> One important other discussion item is what to do about the multipath
> extension milestone, which some have suggested should be dropped, while
> others still show interest to pursue it.
>

So, I'd like to understand the suggestion to drop this milestone, before I
start trying to discuss that suggestion :-).

In conversations with individual folk, I've heard these concerns about QUIC
multipath:

- Whether it will be possible to evaluate multipath performance at scale,
both for evaluating proposals and testing implementations.

- The complexity involved in making decisions dynamically about which path
to send a given packet on (which could be a research topic, given certain
constraints and goals).

If I've misunderstood or misquoted, my apologies, of course. Please
correct me.

What other concerns do people have? I'd like to get all the objections out
at the beginning of the discussion.

Thanks!

Spencer

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 27 Sep 2020 09:37:42 -0700

At least I’d like to see multi-path with non-migration, for example to take
advantage of multiple network cards, and to prioritize a VPN link but with
WAN fallback at full capacity, or during service degradation. This
especially so for long running connections server to server in order to
avoid dealing with this at the application layer. Of course, it is
non-trivial to state preferences, and QoS / latency / loss must be
evaluated per link.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 27 September 2020 at 18.08.21, Spencer Dawkins at IETF (
[email-masked]) wrote:

Dear QUIC working group,

Thanks!

Spencer

---

From: "Flinck, Hannu (Nokia - FI/Espoo)" <[email-masked]>
Date: Tue, 29 Sep 2020 07:36:20 +0000

We see high relevance for multipath-quic for the use cases we are developing. The timing is of importance so that quic can be used in the traffic steering use cases from the beginning (see ATSSS draft for details). Unnecessarily delaying multipath quic work will lead to use of MPTCP which we know cannot meet fully the requirements set for 5G. There is a high cost migration cost if starting with MPTCP and then swapping later to multipath-quic impacting the industry at large.

Similarly to Spencer I would like to understand the rational why to consider to drop multipath-quic. If such a decision is made it should be clear why.  All those arguments that Spencer mentioned below have been faced and overcome with MPTCP. What makes quic so different that they cannot be solved (in a same way as with MPTCP)?

Best regards
Hannu

From: QUIC <[email-masked]> On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Sunday, September 27, 2020 7:38 PM
To: Spencer Dawkins at IETF <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Preparing for discussion on what to do about the multipath extension milestone (was: Re: IETF Last Call for QUIC)

At least I’d like to see multi-path with non-migration, for example to take advantage of multiple network cards, and to prioritize a VPN link but with WAN fallback at full capacity, or during service degradation. This especially so for long running connections server to server in order to avoid dealing with this at the application layer. Of course, it is non-trivial to state preferences, and QoS / latency / loss must be evaluated per link.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 27 September 2020 at 18.08.21, Spencer Dawkins at IETF ([email-masked]<mailto:[email-masked]>) wrote:
Dear QUIC working group,

On Fri, Sep 25, 2020 at 5:00 AM Lars Eggert <[email-masked]<mailto:[email-masked]>> wrote:
In parallel to progressing the "base drafts" towards RFC publications, the WG should now also begin to pick up the pace on our other adopted work items (ops drafts, extensions, etc.)

So, I'd like to understand the suggestion to drop this milestone, before I start trying to discuss that suggestion :-).

In conversations with individual folk, I've heard these concerns about QUIC multipath:

- Whether it will be possible to evaluate multipath performance at scale, both for evaluating proposals and testing implementations.

- The complexity involved in making decisions dynamically about which path to send a given packet on (which could be a research topic, given certain constraints and goals).

If I've misunderstood or misquoted, my apologies, of course. Please correct me.

What other concerns do people have? I'd like to get all the objections out at the beginning of the discussion.

Thanks!

Spencer

---

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 29 Sep 2020 10:38:44 -0500

On Sun, Sep 27, 2020 at 11:08 AM Spencer Dawkins at IETF <
[email-masked]> wrote:

>> In parallel to progressing the "base drafts" towards RFC publications,
>> the WG should now also begin to pick up the pace on our other adopted work

I saw evaluations  of individual implementations, I don't know if they have
an mpquic test suite but I think, there is some for quic interoperability
testing. These could be scaled easily (!), I think.

> This should be answered by implementers. 3GPP is eager to standardize
mpquic for 5G ATSSS which is basically traffic steering, i.e. telling
where/which path to steer the packet based on operators business policies,
etc.

Regards,
Behcet

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 29 Sep 2020 22:11:37 +0200

Spencer,
>     In parallel to progressing the "base drafts" towards RFC
>     publications, the WG should now also begin to pick up the pace on
>     our other adopted work items (ops drafts, extensions, etc.)
> 
>     One important other discussion item is what to do about the
>     multipath extension milestone, which some have suggested should be
>     dropped, while others still show interest to pursue it.
> 
> 
> So, I'd like to understand the suggestion to drop this milestone, before 
> I start trying to discuss that suggestion :-).

I'd like to understand this as well.
> - Whether it will be possible to evaluate multipath performance at 
> scale, both for evaluating proposals and testing implementations.

We already have plenty of experience with MPTCP with several large 
deployments, including :

- MPTCP on all iPhones since 2013 with a growing number of applications
- MPTCP on Android smartphones in South Korea for WiFi/4G offload
- MPTCP in hybrid access networks that are used by different network 
operators to combine xDSL and LTE

Multipath extensions to QUIC would be applicable in these different use 
cases

> - The complexity involved in making decisions dynamically about which 
> path to send a given packet on (which could be a research topic, given 
> certain constraints and goals).

The packet scheduling problem is a much simpler problem in multipath 
transport protocol than congestion control. I would not consider this as 
a research topic given all the experience we have with MPTCP

> out at the beginning of the discussion.

Same for me

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 29 Sep 2020 21:30:09 +0100

Hi Olivier,

> a research topic given all the experience we have with MPTCP

As an individual and co-author on the Extensible Prioirties scheme for
HTTP, I am very interested to know how MP-QUIC scheduling would interact
with HTTP/3 servers doing HTTP prioiritization scheduling when handling
concurrent requests. Are you aware of work that has looked at this in
either MP-QUIC or MPTCP+HTTP/2? Is there clear improvement in web browser
KPIs, for instance?

Cheers
Lucas

---

From: Matt Joras <[email-masked]>
Date: Tue, 29 Sep 2020 13:31:00 -0700

Two replies inline.

> I'd like to understand this as well.
I want to echo Jana's reply from the original discussion here
<https://mailarchive.ietf.org/arch/msg/quic/R-uJhzzXBz-93OTmYupXu8ooJdA/>.
Everyone agrees multi-path transports are an interesting problem, and IETF
participants love to solve interesting problems. It's not clear, however,
that it should be a primary milestone of the working group at this time.

> Multipath extensions to QUIC would be applicable in these different use
There is a difference between "Someone is doing these things with MP-TCP",
and "Someone has shown interest and intent in doing these things with
MP-QUIC". Additionally, the first example can, as I understand it, largely
be replicated with QUIC connection migration. The other two I would like to
hear more about, because it would surprise me if they amounted to a
significant deployment. Support for MP-TCP in Android is virtually
nonexistent, for example.

We have deployment experience with pre-IETF QUIC, and growing deployment
experience with IETF QUIC itself. However, even things like connection
migration in the base drafts have been fully exercised in the real world
yet. As far as I know there have not been any major implementers or
"championing" applications showing a lot of interest in deploying MP-QUIC
in the near future, with the exception of the 3GPP's desire to integrate it
into the ATSSS specification. As I understand it, this interest does not in
itself even imply a deployment in the near future. I would rather we not
endeavor on tackling what is certainly going to be a difficult design
problem simply under the hope that "If you build it, applications will turn
up".

> Matt Joras

---

From: Madhan Raj Kanagarathinam <[email-masked]>
Date: Wed, 30 Sep 2020 12:58:49 +0530

I second Pr.Olivier.

Multipath Protocol gives more benefits and flexibility for Smartphones.

Apart from the Aggregation (Cellular + Wi-Fi) and Reliability (Cellular <-> Wi-Fi), it can also provide much advantages (like ndiffmode).

The multipath scheduling, path management(pm), and congestion control(cc) is a bit of a challenge, but fairly explored and addressed area.

With the deployment of MPTCP in Smartphones for more than 4 years, I feel that MPTCP scheduler/pm/cc has worked pretty good.

Multipath QUIC, with inherent advantages of QUIC protocol, can provide much more benefits than MPTCP.

--------- Original Message ---------

Sender : Olivier Bonaventure <[email-masked]>

Date : 2020-09-30 01:42 (GMT+5:30)

Title : Re: Preparing for discussion on what to do about the multipath extension milestone

Spencer,

Same for me

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 30 Sep 2020 10:06:44 +0200

Matt,
>      >
>      > On Fri, Sep 25, 2020 at 5:00 AM Lars Eggert <[email-masked]
>     <mailto:[email-masked]>
>      > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>      >
> Everyone agrees multi-path transports are an interesting problem, and 
> IETF participants love to solve interesting problems. It's not clear, 
> however, that it should be a primary milestone of the working group at 
> this time.

I think that the main question is whether the working group wants to 
have a generic transport protocol which can support a variety of 
applications or wants a transport that is only tuned to the requirements 
of HTTP. Given QUIC's architecture and clean design, it is possible to 
do a generic and future proof transport that includes clean multipath 
support that is much better than MPTCP.
> 
>      >
>      > In conversations with individual folk, I've heard these concerns
>     about
>      > QUIC multipath:
>      >
> There is a difference between "Someone is doing these things with 
> MP-TCP", and "Someone has shown interest and intent in doing these 
> things with MP-QUIC". Additionally, the first example can, as I 
> understand it, largely be replicated with QUIC connection migration. 

With connection migration, QUIC has roughly the same features as SCTP's 
failover mechanism. Connection migration is a nice feature when you have 
a clear indication that one path has failed and thus move to another 
one. However, the experience reported by Apple with MPTCP on iPhones 
shows that there are many situations where the selection of one 
interface over the other is not a binary decision. When users walk, 
there are many cases where both cellular and WiFi need to be used in 
parallel for some period of time (seconds or tens of second) to achieve 
good quality of experience while none of the networks is perfect. During 
these periods, being able to send data over two paths is key.

> The 
> other two I would like to hear more about, because it would surprise me 
> if they amounted to a significant deployment. Support for MP-TCP in 
> Android is virtually nonexistent, for example.

For the second use case, the GigaLTE service was presented by KT in the 
MPTCP working group. AFAIK, a similar service has been deployed by other 
operators in South Korea. Several smartphone vendors (Samsung, LG 
notably) have ported the off-tree Multipath TCP kernel patches on their 
high-end smartphones to support this service. Huawei has also announced 
some MPTCP-enabled smartphones and one CDN service using MPTCP in China.
There is an ongoing effort that involves Redhat, intel, Apple, Tessares 
and others to rewrite MPTCP in the mainline Linux kernel. There have 
been several presentations about that at netdev.

The third use case is deployed in several countries to combine xDSL and 
LTE to provide fast broadband services in rural areas. There are several 
references on wikipedia
  https://en.wikipedia.org/wiki/Hybrid_Access_Networks

> "championing" applications showing a lot of interest in deploying 
> MP-QUIC in the near future, with the exception of the 3GPP's desire to 
> integrate it into the ATSSS specification. 
When MPTCP started within the IETF, there was no major deployment and no 
experience with a pre-IETF version. This did not prevent the MPTCP 
working group to design a protocol that has been widely deployed.

> As I understand it, this 
> interest does not in itself even imply a deployment in the near future. 
> I would rather we not endeavor on tackling what is certainly going to be 
> a difficult design problem simply under the hope that "If you build it, 
> applications will turn up".

Given the experience that we already have with MPTCP, I don't consider 
the multipath extensions to QUIC as a difficult design problem. In 
MPTCP, the main problems were middlebox interference, this does not 
exist with QUIC. The MPQUIC design that was initially proposed by 
Quentin De Coninck in 2017 has been updated to take into account the 
evolution of QUIC within the IETF. The changes made to QUIC have made it 
easier to support multipath capabilities, see
https://datatracker.ietf.org/doc/draft-deconinck-quic-multipath/

There is also running code inside pquic which is a fork of picoquic
https://github.com/p-quic/pquic

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 30 Sep 2020 10:14:12 +0100

> support that is much better than MPTCP.

As an HTTP/3 implementer, I believe it to be the best example we have of a
protocol that exercises the generic transport capabilities. Particularly
stream multiplexing, HoL avoidance, synchronisation across independent
streams, sensitivity to iwnd and cwnd, and large transfers in both
directions.

Yes, there are other application protocols. But I am not aware of ones that
can so clearly demonstrate impacts of stream interactions. This can be in
the form of user-perceived performance or industry metrics such as Web
Vitals [1].

HTTP is also a good example of a widely deployed application semantic that
has been implemented across different libraries that have different design
choices. This is an important consideration for the prioiritzation work
because we need to consider what information an application is able to
access from the transport, and how the application may/may not govern over
transport behaviour. QUIC transport defines a set functions on streams that
an application can rely upon [2], how does MP-QUIC augment that?

I'm concerned about generic transport capabilities that cannot be
reasonably used by applications without deep integration or layering
violations. So it would be very useful for me to understand how a piece of
HTTP server software (as an example of a real, complex application) would
leverage MP-QUIC.

[1] - https://web.dev/vitals/
[2] - https://tools.ietf.org/html/draft-ietf-quic-transport-31#section-2.4

---

From: Marten Seemann <[email-masked]>
Date: Wed, 30 Sep 2020 16:30:09 +0700

One of the pillars of standardization work at the IETF is running code.
Aside from some proof-of-concept implementations of multipath, I'm not
aware of any implementation supporting any kind of MPQUIC, let alone any
internet-wide experiments / deployments. From what I've seen on the interop
runner, a large number implementations are still struggling with connection
migration at the moment.

As others have said before, I do agree that multi-path is an intellectually
challenging problem. It's not a problem we're in a hurry to solve at the
current moment though: Given that QUIC is end-to-end encrypted, rolling out
MPQUIC should be a lot easier than rolling out MPTCP, as we won't have to
deal with ossified middleboxes.

Before devoting a significant amount of working group resources on this
particular problem, I'd like to see:

   1. one (or multiple) big players driving experimentation and
   internet-wide deployment and
   2. measurements that show that MPQUIC provides substantial benefits for
   real-world application protocols compared to vanilla QUIC with connection
   migration.

On Wed, Sep 30, 2020 at 4:14 PM Lucas Pardue <[email-masked]>
wrote:

>> > <
>> https://mailarchive.ietf.org/arch/msg/quic/R-uJhzzXBz-93OTmYupXu8ooJdA/>.
>>
> Yes, there are other application protocols. But I am not aware of ones
> that can so clearly demonstrate impacts of stream interactions. This can be
> in the form of user-perceived performance or industry metrics such as Web
> Vitals [1].
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 30 Sep 2020 12:09:26 +0200

Lucas,
> either MP-QUIC or MPTCP+HTTP/2? Is there clear improvement in web 
> browser KPIs, for instance?

The work that I know is Robin Marx's work, but AFAIK he has not studied 
MPTCP or MPQUIC.

Concerning prioritization, MPQUIC exposes streams like QUIC to the 
HTTP/3 application. It could be possible to expose some of the 
characteristics of MPQUIC to HTTP/3, but this is not required. MPTCP 
provides the same socket API to applications and enables them to work as 
if they were using TCP, except that they get better resilience and 
higher throughput for free.

The same is true for MPQUIC. MPQUIC allows the application above to be 
resilient to handovers, lower performance on one path, and benefit from 
bandwidth aggregation when operating over low-speed networks, without 
having to deal with all these problems themselves.

It is possible to expose some information from MPQUIC and the underlying 
paths to the application above, but I'm not convinced that this is 
really required. What worked well for Multipath TCP is the definition of 
policies that are used by the MPTCP implementation and relate to the 
deployment use case. Here are a few deployed examples :

  - on iPhones, some users (e.g. those with an unlimited data plan) want 
to use the best network interface
  - on iPhones, some users (e.g. those with a limited data plan) agree 
to use cellular when Wi-Fi becomes too bad. MPTCP measures the rtt and 
the timeouts and switches to cellular when performance decreases
  - in hybrid access networks, operators want to first use DSL before 
LTE. MPTCP fills the DSL pipe first and only uses LTE when DSL is fully used

I think that MPQUIC deployments would also use similar policies.

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 30 Sep 2020 12:28:13 +0200

Lucas,
> 
> As an HTTP/3 implementer, I believe it to be the best example we have of 
> a protocol that exercises the generic transport capabilities. 
> Particularly stream multiplexing, HoL avoidance, synchronisation across 
> independent streams, sensitivity to iwnd and cwnd, and large transfers 
> in both directions.
> 
> that can so clearly demonstrate impacts of stream interactions. This can 
> be in the form of user-perceived performance or industry metrics such as 
> Web Vitals [1].
> 
> HTTP is also a good example of a widely deployed application semantic 
> that has been implemented across different libraries that have different 
> design choices. This is an important consideration for the 
> prioiritzation work because we need to consider what information an 
> application is able to access from the transport, and how the 
> application may/may not govern over transport behaviour. QUIC transport 
> defines a set functions on streams that an application can rely upon 
> [2], how does MP-QUIC augment that?
> 
> violations. So it would be very useful for me to understand how a piece 
> of HTTP server software (as an example of a real, complex application) 
> would leverage MP-QUIC.
> 

MPQUIC would provide the same streams abstraction as QUIC, no changes 
are required to [2]. To support multiple streams a QUIC sender needs to 
include an algorithm to decide which data frame (from which stream) 
needs to be sent when there is a transmission opportunity (congestion 
window is open). MPQUIC is similar, except that it needs to decide on 
which path the data will be sent. We could expose some information about 
the different paths to the applications, but I don't think that this is 
required. A MPQUIC implementation will select the utilisation of the 
different paths based on policies that typically depend on the use case 
and the deployment model.

Different policies can be defined as different QUIC implementations 
might use different congestion control schemes or different stream 
schedulers.

Considering your HTTP server use case, here are a few possible policies 
and their impact on MPQUIC.

1. Consider a dual-stack HTTP server that wants to use the best 
performing path between IPv6 and IPv4. When a client connects over IPv4, 
the server advertises its IPv6 address and MPQUIC automatically starts 
to use the two path. If one is significantly better, traffic will move 
to that path without any involvment of the HTTP server

2. Consider again a HTTP server that serves customers in rural areas 
where DSL and 4G are slow and in cities where fiber is deployed. When 
smartphones connect, they advertise to MPQUIC their WiFi (attached to 
DSL/fiber) and 4G addresses. MPQUIC can be configured to only accept a 
second subflow over the 4G if the primary one does not perform 
correctly. With MPQUIC, the HTTP server would aggregate the 4G and DSL 
bandwidth in rural areas while only using fiber in cities.

The main benefit of putting multipath inside the transport is that the 
application does not need to handle the problems that occur on any of 
these paths. The transport has access to round-trip-times and loss 
measurements and has all the information required to manage the 
different paths efficiently.

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 30 Sep 2020 11:49:58 +0100

It's this special sauce that concerns me.
I don't know how I'd objectively measure that the MP-QUIC design and all
the implementation effort would actual result in improvement. For instance,
more bandwidth via aggregation can still be misused; some types of streams
will be more latency sensitive than others. Putting the decision making
into the transport library could also be seen as black box.

I also want to draw some parallels between uniflows and the HTTP/2 priority
tree. The tree was a fully expressive model that allowed a client to hint
to the server about the relationship between streams. The problem of
actioning this signal was left up to the server. Deployment experience
reveals that getting this well-tuned, just for a single TCP connection, is
hard. Some implementers just never got around to fixing some serious and
easily detectable performance problems.

Presenting a bunch of uniflows to a sever and leaving it responsible for
using them properly seems to me quite a similar problem.

Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 30 Sep 2020 08:18:52 -0500

If we're starting the discussion on Multipath, it's likely that starting
new email threads would be a good idea.

Beyond that ...

>From the beginning of time, people who needed some, but not all, of TCP's
transport capabilities were left using UDP, and figuring out the transport
capabilities in their own applications, with greater or less great success.

The quote I've heard so many times is that many UDP-based applications
ended up adding so many TCP-like transport capabilities that they might
just as well have used TCP.

We've had to write BCPs about that - guidelines for UDP usage, with
https://tools.ietf.org/html/rfc8085 replacing
https://tools.ietf.org/html/rfc5405 ten years later being the most obvious
example, but also circuit breakers, like
https://datatracker.ietf.org/doc/rfc8083/ (a Proposed Standard) and
https://datatracker.ietf.org/doc/rfc8084/.

Meanwhile, we've had to try to solve the "some but not all of TCP" problem
in transport ANYWAY - first with SCTP and DCCP, and now with QUIC.

So, I was hoping that we could learn from that example, and do work on
multipath within transport, so that the world would not be full of
applications managing multiple paths with greater or less success.

Do the right thing, of course.

Best,

Spencer

---

From: Robin MARX <[email-masked]>
Date: Wed, 30 Sep 2020 15:39:08 +0200

Hello all,

I don't want to get too deep into this, as I don't have enough experience
with multipath to have a well-founded opinion.

With regards to Lucas' questions on interfacing HTTP2/3 priorities with
multipath TCP/QUIC, I am aware of several recent works that have looked at
this problem (e.g., [1], [2], [3]).
I'm not entirely convinced of their results, methodology or conclusions
myself, but they all claim to see benefits for the Web page loading use
case.
I share the concern that this specifically is a generally difficult and
largely unsolved (research) problem and that APIs/interfaces for exposing
the full power of multipath to the application layer will be difficult to
create.
Then again, the same can be said for exposing QUIC-layer scheduling to the
application layer, and we seem to be happy to stick our heads in the sand
for that...

I also feel it's important to make the clear distinction between general
multipath scheduling (what I feel is mostly what Olivier is describing) and
"stream-aware" scheduling (what I'd call "prioritiy-driven", and my remarks
above allude to),
where I get the impression the first is a generally solved problem for
MPTCP with a large body of research (and deployment experience) that can be
relatively directly re-used for MP-QUIC.

As a personal critical remark on the use of multipath for the Web browsing
use case, I feel that the wider Web performance community in general is of
the opinion that the network itself matters less these days,
and that the (ab)use of e.g., JavaScript frameworks, loading of (too) large
images, etc. has a much more profound impact on the observed user
performance than (limited) network bandwidth,
especially on slow mobile devices with limited processing capabilities.

As such, I too posit that we should not consider Multipath just or
primarily for the HTTP use case (or hinge it on its application in that
area), but as a property of the general purpose transport that QUIC aims to
be.
In this context, like Olivier, I feel most of the lessons learned from
MPTCP can be transposed to QUIC, including its deployment experience.
If true, the question of the usefulness of MPQUIC devolves into the
usefulness of MPTCP, which I think the past decade of research and
implementation has shown to be considerable.
As such, I wonder if the amount of work necessary for MPQUIC isn't being
overestimated, as well as if (some of) the "outstanding questions" really
are unanswered.

Robin

[1] : A stream-aware multipath quic scheduler for heterogeneous paths -
Rabitsch et al. - https://dl.acm.org/doi/pdf/10.1145/3284850.3284855
[2] : SRPT-ECF: challenging Round-Robin for stream-aware multipath
scheduling - Jonglez et al. - https://hal.inria.fr/hal-02570686/document
[3] : PriorityBucket: A Multipath-QUIC Scheduler on Accelerating First
Rendering Time in Page Loading - Shi et al. -
https://dl.acm.org/doi/pdf/10.1145/3396851.3402923

On Wed, 30 Sep 2020 at 12:50, Lucas Pardue <[email-masked]>
wrote:

> I also want to draw some parallels between uniflows and the HTTP/2
> priority tree. The tree was a fully expressive model that allowed a client
> to hint to the server about the relationship between streams. The problem
> of actioning this signal was left up to the server. Deployment experience

-- 

Robin Marx
PhD researcher - web performance
Expertise centre for Digital Media

T +32(0)11 26 84 79 - GSM +32(0)497 72 86 94

www.uhasselt.be
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 30 Sep 2020 16:17:15 +0200

Lucas,

> It's this special sauce that concerns me. > I don't know how I'd objectively measure that the MP-QUIC design and all
> the implementation effort would actual result in improvement. For 
> instance, more bandwidth via aggregation can still be misused; some 
> types of streams will be more latency sensitive than others. Putting the 
> decision making into the transport library could also be seen as black box.

I would suggest that we start by considering MPQUIC as a blackbox as we 
did with MPTCP. This worked well and getting the multipath mechanisms 
will be easy.

> priority tree. The tree was a fully expressive model that allowed a 
> client to hint to the server about the relationship between streams. The 
> problem of actioning this signal was left up to the server. Deployment 
> experience reveals that getting this well-tuned, just for a single TCP 
> connection, is hard. Some implementers just never got around to fixing 
> some serious and easily detectable performance problems.
> 

That's a policy decision and policy can be very complex. In MPTCP, we 
have very limited support for policies:
- clients use a path manager to decide when they create subflows
- servers basically never create subflows (due to NATs and firewalls)
- clients and servers can use the backup bit to indicate that a subflow 
should only be used if all the non-backup subflows failed or have problems

We discussed multiple times how to exchange policies over MPTCP. It 
turned out that this was very difficult given the middleboxes. In the 
end, MPTCP does not support the exchange of policies and the current 
deployments embed the policies on the hosts that use MPTCP with a 
specific path manager. This works well enough.

For MPQUIC, the situation could be different as there will be no direct 
inteference from middleboxes that modify options. We could exchange 
information such as the priority of a flow, mapping streams to flows, 
rtt preference, capping flows, ... My fear is that if we open this, the 
discussion could never stop and result in something similar that is too 
complex to implement. I would suggest to focus initially on very simple 
policies that are implemented locally (i.e. the application that uses 
QUIC would control when and how to advertise addresses and when and how 
to create flows, QUIC implementations could include different packet 
schedulers) and discuss the possibility of exchanging this policy 
information for a revision of MPQUIC.

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 30 Sep 2020 15:29:21 +0100

On Wed, Sep 30, 2020 at 3:17 PM Olivier Bonaventure <
[email-masked]> wrote:

> Lucas,
>
> > It's this special sauce that concerns me. > I don't know how I'd
> objectively measure that the MP-QUIC design and all
Defining the capability, presenting little evidence to support its
applicability and punting on the problems is exactly what got HTTP/2
priorities into the pickle that it's in.

Personally, I think starting on a basis of ignoring QUIC transport's core
method of exchanging application data is a bad idea.

Cheers,
Lucas

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 30 Sep 2020 16:52:30 +0200

Lucas,

> 
> Personally, I think starting on a basis of ignoring QUIC transport's 
> core method of exchanging application data is a bad idea.

I'm not suggesting that. I suggest that we first agree on the basic 
multipath mechanisms without taking too much time to discuss different 
and diverging application requirements. These mechanisms obviously need 
to take into account the core characteristics of QUIC. One example is 
that in contrast with MPTCP, MPQUIC acknowledgements do not necessarily 
need to be transmitted over the same "path" as the data that they 
acknowledge. This gives more freedom to an MPQUIC implementation than an 
MPTCP one. Another example are the flow control frames. Once we agree on 
these application-independent mechanisms, then we can start to discuss 
about policies and how to handle them.

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 30 Sep 2020 16:29:14 +0100

> about policies and how to handle them.

Seems like tail wagging the dog? If an "integrated assembly of application
and MPQUIC" as a whole has no clue how to send stream data then deciding
the best way to send ACKs is an afterthought.

To put it a different way, the problem is about packetization over time
windows, before worrying about path selection. I get worried when I read
draft-deconinck-quic-multipath and it says

> TODO: Add a companion document discussing the packet scheduling and path
management considerations.

I appreciate that could be a placeholder, so I'm trying to find out if the
knowledge and experience of MPTCP does actually help in anyway bootstrap
understanding of independent streams in a transport.

---

From: Ian Swett <[email-masked]>
Date: Wed, 30 Sep 2020 12:12:36 -0400

I'm very receptive to Jana's arguments, but I think it's also worth
understanding how much work we believe this will be for the QUIC WG.

Many fairly complex multipath problems have already been dealt with by
supporting connection migration in QUIC v1, such as path validation and
linkability between flows.  Multiple packet number spaces during the
handshake means that implementations already have some ability to have
multiple loss recovery contexts, though they may not be easily adaptable to
multipath.

Scheduling seems like a big problem, but it's not clear to me that the QUIC
working group should be attempting to solve that.  I'd argue we should make
that explicitly out of scope if we took on this work.

The existing draft(draft-deconinck-quic-multipath-05
<https://tools.ietf.org/html/draft-deconinck-quic-multipath-05>) adds
ADD_ADDRESS/REMOVE_ADDRESS frames and modifies the ACK, NEW_CONNECTION_ID,
and RETIRE_CONNECTION_ID frames.  It also adds what appears to be a
debugging frame(UNIFLOWS) and a single transport parameter to indicate
support and indicate the max number of uniflows.  Having sets of CIDs per
path seems an unfortunate complication to me, but I haven't thought about
it enough.  The two new frames and modifications to the ACK frame are
simple enough.

Is there MORE work QUIC WG would have to do than what's described in the
draft?  If so, can people outline that work?

I certainly think MP-QUIC is a cleaner solution than MP-TCP.

I lean towards keeping multipath in scope, and I'd be willing to spend time
improving a proposal if it was adopted.  One reason is that I never want to
be forced to support MP-TCP because there is no MP-QUIC.  I also don't
think this has to be designed with HTTP in mind, even if that was the focus
of QUIC v1.  MASQUE/VPNs are a more compelling use case to me.

Ian

PS: It'd be good if someone from Apple could share their perspective on
whether MP-QUIC is necessary, since they have quite a bit of experience
with MP-TCP.

On Tue, Sep 29, 2020 at 4:31 PM Matt Joras <[email-masked]> wrote:

>> > <mailto:[email-masked]>> wrote:
>> >

---

From: Martin Duke <[email-masked]>
Date: Wed, 30 Sep 2020 10:16:24 -0700

(Speaking as an individual)

There is some back-and-forth as to whether these are useful cases are not.
I'll take it on faith, given the proponents, that there is a real hope of
deploying this. However, I share the desire to not have the WG fully
consumed by MP-QUIC for the foreseeable future.

I don't think the community has well-established solutions for many
problems in this space (e.g. scheduling). However, I think QUIC is a far
better platform for experimentation than the alternatives, and would
support a draft similar to draft-deconinck-multipath-quic
<https://tools.ietf.org/html/draft-deconinck-multipath-quic> that provided
the required protocol extensions to make that happen [1].

IIUC the hard, unsolved problems are common to all MP protocols, so I don't
think further research and future standards in this area are specific to
QUIC or appropriate for the QUIC Working Group. But experimental QUIC
extensions would accelerate this work, are appropriate for the WG, and may
get us to a place where we could confidently develop standards about it.

Martin Duke

[1] I would prefer that this draft be Experimental, and have numerous nits
about the design that are not relevant to this thread.

On Wed, Sep 30, 2020 at 9:13 AM Ian Swett <ianswett=
[email-masked]> wrote:

> Scheduling seems like a big problem, but it's not clear to me that the
> QUIC working group should be attempting to solve that.  I'd argue we should
> make that explicitly out of scope if we took on this work.
>
> I lean towards keeping multipath in scope, and I'd be willing to spend
> time improving a proposal if it was adopted.  One reason is that I never
> want to be forced to support MP-TCP because there is no MP-QUIC.  I also
> don't think this has to be designed with HTTP in mind, even if that was the
> focus of QUIC v1.  MASQUE/VPNs are a more compelling use case to me.
>
> Ian
>
>> MP-TCP", and "Someone has shown interest and intent in doing these things
>> with MP-QUIC". Additionally, the first example can, as I understand it,
>> largely be replicated with QUIC connection migration. The other two I would
>> like to hear more about, because it would surprise me if they amounted to a

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 30 Sep 2020 14:15:07 -0500

Hi, Martin,

Just a couple of thoughts here:

> consumed by MP-QUIC for the foreseeable future.

That sounds right. I'm assuming that getting the core QUIC specifications
published and doing any cleanup work necessary SHOULD/MUST take priority,
in the BCP 14 sense of those words.

As Lars' initial note said, I'd also like to see the manageability,
applicability, and datagram extension working group drafts, already adopted
by QUIC, moving forward.

> <https://tools.ietf.org/html/draft-deconinck-multipath-quic> that
> provided the required protocol extensions to make that happen [1].
>

I agree that scheduling is challenging - 3GPP is certainly spending time
defining different strategies for behaviors, even in addition to the ones
we described in
https://datatracker.ietf.org/doc/draft-bonaventure-quic-atsss-overview/.

And I agree that the QUIC protocol would be a better platform for
experimentation than anything I can think of (other suggestions are, of
course, welcome).

> IIUC the hard, unsolved problems are common to all MP protocols, so I
> don't think further research and future standards in this area are specific
> to QUIC or appropriate for the QUIC Working Group. But experimental QUIC

Targeting Experimental status for work in this area sounds like a fine plan
to me (much better than not thinking about multicast in the IETF for a
while longer).

I know you have a variety of tools at your disposal to direct this work
(MP-TCP was done in its own working group, for both Experimental and
Standards-Track versions of the protocol specifications). Do the right
thing, of course.

What do you and Magnus need from members of the community, to help move
forward on this?

Best,

Spencer

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 1 Oct 2020 09:48:35 +0900

I think that the discussion around prioritization might be showing the
problem behind how we have framed the dabate.

Multipath is a vague phrase, and to me it seems that people are referring
to different features of it. It could be symmetric path migration,
bandwidth aggregation, latency reduction by path selection, ...

In case of prioritization, maybe it does not matter a lot if what people
have in mind bandwidth aggregation. But if people want to run a
latency-sensitive application on aggregate paths that gave varying RTTs,
then path selection becomes a critical part of the application protocol.

I think it would be much more fruitful to discuss about each specific
feature of "multipath", what the use-case is, why it should be a WG item.
If we are to make changes to the milestones, I'd prefer the milestones to
clarify those specific features (and problems) that we will be solving,
rather than talking about "multipath" which IMO is a vague term that people
might disagree about the scope.

2020年9月30日(水) 22:39 Robin MARX <[email-masked]>:

> and that the (ab)use of e.g., JavaScript frameworks, loading of (too)
> large images, etc. has a much more profound impact on the observed user

-- 
Kazuho Oku

---

From: Ian Swett <[email-masked]>
Date: Wed, 30 Sep 2020 20:51:22 -0400

Given the responses, can we narrow down the way forward(ideally on a
different thread) to directions that are less open-ended?  I'll suggest
some options, but the chairs and/or ADs need to decide.
 1) No future work on multipath in the QUIC WG, in the belief the existing
connection migration functionality is sufficient.
 2) Adopt the existing draft as a starting point for QUIC multipath(
draft-deconinck-multipath-quic
<https://tools.ietf.org/html/draft-deconinck-multipath-quic>), with the
explicit goal of not expanding the scope of the document.
 3) Adopting multipath as a core QUIC WG deliverable.

I favor #2, but these may not be the right options.  Normally I'd say
people should work this out in person, but that doesn't seem viable at
the moment.  I'm happy to set up a long(3-4+hr) Google Meet to discuss this
via videoconference if that helps move the discussion forward.

Or we can form a design team, which typically takes O(3 months) to finish.

Ian

On Wed, Sep 30, 2020 at 3:15 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

>> There is some back-and-forth as to whether these are useful cases are
>> not. I'll take it on faith, given the proponents, that there is a real hope
>> of deploying this. However, I share the desire to not have the WG fully
> Targeting Experimental status for work in this area sounds like a fine
> plan to me (much better than not thinking about multicast in the IETF for a
> while longer).
>
>> [1] I would prefer that this draft be Experimental, and have numerous
>> nits about the design that are not relevant to this thread.
>>
>
>
>