Subject: Re: Preparing for discussion on what to do about the multipath extension milestone

From: Christian Huitema <[email-masked]>
Date: Thu, 1 Oct 2020 00:25:27 -0700

I am not sure that the current "mpquic" draft is the right approach.
Specifically, I do not agree that having one packet number space per
path is the right approach. This contradicts the design of QUIC V1, in
which data sent on multiple paths shares a common packet number space.
For example, in QUIC V1, we can start a connection on one path, migrate
to another path, and keep the same packet number space throughout. I
find that a very nice property -- and also an essential property if we
want to support NAT rebinding. Handling multipath with a single number
space requires some book-keeping on the sender side to match
acknowledgements and sending paths, but we have working code for that.

I am also not convinced that we properly understand the concept of
"path". There is very little in the QUIC V1 protocol that requires
transmission paths to be symmetric: any packet sent from a node to a
valid address of the peer will be accepted, provided the crypto works.
The linkage such requirement comes from the statement that a server
starts directing traffic to a validated path when it sees the client
using the same pair of addresses. This is an "implicit" linkage; I would
expect that the first role of a multipoint extension would be to replace
that by an "explicit" statement of preferences.

I am worried that we have a set of unresolved security issues around
paths, largely linked to the requirement to support NAT rebinding. If we
support NAT, the IP headers must be outside the authentication envelope
of the crypto. There are plausible attacks in which the attacker splices
a cryptographically valid packet and a forged IP header. We have some
defensive heuristics, but if we study multipath I hope we will end up
with something better.

-- Christian Huitema

On 9/30/2020 5:51 PM, Ian Swett wrote:
> Given the responses, can we narrow down the way forward(ideally on a
> different thread) to directions that are less open-ended?  I'll
> suggest some options, but the chairs and/or ADs need to decide.
>  1) No future work on multipath in the QUIC WG, in the belief the
> existing connection migration functionality is sufficient.
>  2) Adopt the existing draft as a starting point for QUIC
> multipath(draft-deconinck-multipath-quic
> <https://tools.ietf.org/html/draft-deconinck-multipath-quic>),
> with the explicit goal of not expanding the scope of the document.
>  3) Adopting multipath as a core QUIC WG deliverable.
>
> I favor #2, but these may not be the right options.  Normally I'd say
> people should work this out in person, but that doesn't seem viable at
> the moment.  I'm happy to set up a long(3-4+hr) Google Meet to discuss
> this via videoconference if that helps move the discussion forward.
>
> Or we can form a design team, which typically takes O(3 months) to finish.
>
> Ian
>
> On Wed, Sep 30, 2020 at 3:15 PM Spencer Dawkins at IETF
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     Hi, Martin, 
>
>     Just a couple of thoughts here: 
>
>     On Wed, Sep 30, 2020 at 12:16 PM Martin Duke
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         (Speaking as an individual)
>
>         There is some back-and-forth as to whether these are useful
>         cases are not. I'll take it on faith, given the proponents,
>         that there is a real hope of deploying this. However, I share
>         the desire to not have the WG fully consumed by MP-QUIC for
>         the foreseeable future.
>
>
>     That sounds right. I'm assuming that getting the core QUIC
>     specifications published and doing any cleanup work necessary
>     SHOULD/MUST take priority, in the BCP 14 sense of those words. 
>
>     As Lars' initial note said, I'd also like to see the
>     manageability, applicability, and datagram extension working group
>     drafts, already adopted by QUIC, moving forward. 
>      
>
>         I don't think the community has well-established solutions for
>         many problems in this space (e.g. scheduling). However, I
>         think QUIC is a far better platform for experimentation than
>         the alternatives, and would support a draft
>         similar to draft-deconinck-multipath-quic
>         <https://tools.ietf.org/html/draft-deconinck-multipath-quic> that
>         provided the required protocol extensions to make that happen [1].
>
>
>     I agree that scheduling is challenging - 3GPP is certainly
>     spending time defining different strategies for behaviors, even in
>     addition to the ones we described
>     in https://datatracker.ietf.org/doc/draft-bonaventure-quic-atsss-overview/.
>
>     And I agree that the QUIC protocol would be a better platform for
>     experimentation than anything I can think of (other suggestions
>     are, of course, welcome). 
>      
>
>         IIUC the hard, unsolved problems are common to all MP
>         protocols, so I don't think further research and future
>         standards in this area are specific to QUIC or appropriate for
>         the QUIC Working Group. But experimental QUIC extensions would
>         accelerate this work, are appropriate for the WG, and may get
>         us to a place where we could confidently develop standards
>         about it.
>
>
>     Targeting Experimental status for work in this area sounds like a
>     fine plan to me (much better than not thinking about multicast in
>     the IETF for a while longer). 
>
>     I know you have a variety of tools at your disposal to direct this
>     work (MP-TCP was done in its own working group, for both
>     Experimental and Standards-Track versions of the protocol
>     specifications). Do the right thing, of course.  
>
>     What do you and Magnus need from members of the community, to help
>     move forward on this?
>
>     Best,
>
>     Spencer
>      
>
>         Martin Duke
>
>         [1] I would prefer that this draft be Experimental, and have
>         numerous nits about the design that are not relevant to this
>         thread.
>
>
>      
>

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 01 Oct 2020 18:22:28 +1000

I share Christian's concerns about the draft, but it's not just ACKs, it's the entire Uniflow concept that I would call into question.

On Thu, Oct 1, 2020, at 17:25, Christian Huitema wrote:
> using the same pair of addresses. This is an "implicit" linkage; I 
> would expect that the first role of a multipoint extension would be to 
> replace that by an "explicit" statement of preferences.
> 
> paths, largely linked to the requirement to support NAT rebinding. If 
> we support NAT, the IP headers must be outside the authentication 
> envelope of the crypto. There are plausible attacks in which the 
> attacker splices a cryptographically valid packet and a forged IP 
> header. We have some defensive heuristics, but if we study multipath I 
> hope we will end up with something better.
> 
> -- Christian Huitema
> 
> > Given the responses, can we narrow down the way forward(ideally on a different thread) to directions that are less open-ended?  I'll suggest some options, but the chairs and/or ADs need to decide. 
> >  1) No future work on multipath in the QUIC WG, in the belief the existing connection migration functionality is sufficient.
> >  2) Adopt the existing draft as a starting point for QUIC multipath(draft-deconinck-multipath-quic <https://tools.ietf.org/html/draft-deconinck-multipath-quic>), with the explicit goal of not expanding the scope of the document.
> > I favor #2, but these may not be the right options.  Normally I'd say people should work this out in person, but that doesn't seem viable at the moment.  I'm happy to set up a long(3-4+hr) Google Meet to discuss this via videoconference if that helps move the discussion forward.
> > 
> > On Wed, Sep 30, 2020 at 3:15 PM Spencer Dawkins at IETF <[email-masked]> wrote:
> >> Hi, Martin, 
> >> 
> >> On Wed, Sep 30, 2020 at 12:16 PM Martin Duke <[email-masked]> wrote:
> >>> There is some back-and-forth as to whether these are useful cases are not. I'll take it on faith, given the proponents, that there is a real hope of deploying this. However, I share the desire to not have the WG fully consumed by MP-QUIC for the foreseeable future.
> >> 
> >> That sounds right. I'm assuming that getting the core QUIC specifications published and doing any cleanup work necessary SHOULD/MUST take priority, in the BCP 14 sense of those words. 
> >> 
> >> As Lars' initial note said, I'd also like to see the manageability, applicability, and datagram extension working group drafts, already adopted by QUIC, moving forward. 
> >>  
> >>> I don't think the community has well-established solutions for many problems in this space (e.g. scheduling). However, I think QUIC is a far better platform for experimentation than the alternatives, and would support a draft similar to draft-deconinck-multipath-quic <https://tools.ietf.org/html/draft-deconinck-multipath-quic> that provided the required protocol extensions to make that happen [1].
> >> 
> >> I agree that scheduling is challenging - 3GPP is certainly spending time defining different strategies for behaviors, even in addition to the ones we described in https://datatracker.ietf.org/doc/draft-bonaventure-quic-atsss-overview/.
> >> 
> >> And I agree that the QUIC protocol would be a better platform for experimentation than anything I can think of (other suggestions are, of course, welcome). 
> >>  
> >>> IIUC the hard, unsolved problems are common to all MP protocols, so I don't think further research and future standards in this area are specific to QUIC or appropriate for the QUIC Working Group. But experimental QUIC extensions would accelerate this work, are appropriate for the WG, and may get us to a place where we could confidently develop standards about it.
> >> 
> >> Targeting Experimental status for work in this area sounds like a fine plan to me (much better than not thinking about multicast in the IETF for a while longer). 
> >> 
> >> I know you have a variety of tools at your disposal to direct this work (MP-TCP was done in its own working group, for both Experimental and Standards-Track versions of the protocol specifications). Do the right thing, of course.  
> >> 
> >> What do you and Magnus need from members of the community, to help move forward on this?
> >> 
> >> Best,
> >> 
> >> Spencer
> >>  
> >>> Martin Duke
> >>> 
> >>> [1] I would prefer that this draft be Experimental, and have numerous nits about the design that are not relevant to this thread.
> >> 
> >>

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 1 Oct 2020 13:37:41 +0200

Lucas,
> 
>      >
>      > Personally, I think starting on a basis of ignoring QUIC transport's
>      > core method of exchanging application data is a bad idea.
> 
>     I'm not suggesting that. I suggest that we first agree on the basic
>     multipath mechanisms without taking too much time to discuss different
>     and diverging application requirements. These mechanisms obviously need
>     to take into account the core characteristics of QUIC. One example is
>     that in contrast with MPTCP, MPQUIC acknowledgements do not necessarily
>     need to be transmitted over the same "path" as the data that they
>     acknowledge. This gives more freedom to an MPQUIC implementation
>     than an
>     MPTCP one. Another example are the flow control frames. Once we
>     agree on
>     these application-independent mechanisms, then we can start to discuss
>     about policies and how to handle them.
> 
> Seems like tail wagging the dog? If an "integrated assembly of 
> application and MPQUIC" as a whole has no clue how to send stream data 
> then deciding the best way to send ACKs is an afterthought. >
> To put it a different way, the problem is about packetization over time 
> windows, before worrying about path selection. 

In a single-path QUIC implementation, the implementation sends the next 
frames according to the congestion control scheme. When the congestion 
window is open, new frames can be sent. The QUIC protocol does striclty 
not mandate how different frames from different streams will be sent. 
Robin's measurements show that different implementations use very 
different strategies. The same will be true for multipath, expect that 
there will be one congestion controller per path. Each of these 
congestion controllers will open opportunities to transmit frames and 
the QUIC implementation will send frames when any of the underlying 
congestion window is open. This can be further optimised depending on 
the policies that are function of the considered use case.

> I get worried when I read 
> draft-deconinck-quic-multipath and it says
 >
>  > TODO: Add a companion document discussing the packet scheduling and 
> path management considerations.

I'm glad to see that there are now technical discussions on mpquic and 
would be happy to incorporate in the next revision of the draft the 
result of the ongoing discussion.

> I appreciate that could be a placeholder, so I'm trying to find out if 
> the knowledge and experience of MPTCP does actually help in anyway 
> bootstrap understanding of independent streams in a transport.
> 

Packet schedulers and path managers do not need to be precisely defined 
inside IETF documents. Based on discusions about MPTCP, we have 
summarised some of the used packet schedulers in

https://datatracker.ietf.org/doc/html/draft-bonaventure-iccrg-schedulers-00

We can write a similar informational document for path management if 
there is interest

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 1 Oct 2020 14:03:46 +0100

> the policies that are function of the considered use case.

Right. But the opportunities to send STREAM frames are driven by flow
control not just congestion. There are some strict mandates there.
Approaches such as binding a stream to a uniflow could cause starvation of
the related-path due to flow control even when congestion window is open.
Flow control is driven by applications putting data into the transport, and
consuming from it. I suspect HTTP/2 over MPTCP does not have this problem,
mainly because TCP's flow control trumps. But maybe I'm wrong.

Some implementations like quiche decouple the transport library from UDP.
Applications hoping for MP-QUIC with such implementations will have to
accommodate some of the complexity, the transport cannot do it all.

I do agree that we don't have to solve all of these things upfront and in
one document. But I think there is a risk of defining mechanisms and
ignoring some pathological cases. As a parallel, QUIC provides congestion
control mechanisms and does not mandate any algorithms but it has taken
care to suggest one that is going to be safe to deploy on the Internet.

Lucas

[1] https://mailarchive.ietf.org/arch/msg/quic/OlwAHMyK2mAuRCRuCP-Lpyb7ntA/
[2] https://github.com/quicwg/base-drafts/issues/3332

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 1 Oct 2020 14:07:34 +0100

Hiya,

On Wed, Sep 30, 2020 at 5:13 PM Ian Swett <ianswett=
[email-masked]> wrote:

>
> The existing draft(draft-deconinck-quic-multipath-05
> <https://tools.ietf.org/html/draft-deconinck-quic-multipath-05>) adds
> ADD_ADDRESS/REMOVE_ADDRESS frames and modifies the ACK, NEW_CONNECTION_ID,
> and RETIRE_CONNECTION_ID frames.  It also adds what appears to be a
> debugging frame(UNIFLOWS) and a single transport parameter to indicate
> support and indicate the max number of uniflows.  Having sets of CIDs per
> path seems an unfortunate complication to me, but I haven't thought about
> it enough.  The two new frames and modifications to the ACK frame are
> simple enough.
>
> Is there MORE work QUIC WG would have to do than what's described in the
> draft?  If so, can people outline that work?
>
>
This reminded me of the comment I made on the mailing list in January [1]
about how QUIC extensions would play together. There was some further
discussion on GitHub [2] and in Zurich we decided to do nothing. But we did
note that maybe we need to figure things out when the problem comes up, is
that time now? Does adopting one extension force people into an
uncomfortable position of picking one optimization over another?

Cheers
Lucas

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 1 Oct 2020 09:33:08 -0400

On Thu, Oct 01, 2020 at 02:07:34PM +0100, Lucas Pardue wrote:
> discussion on GitHub [2] and in Zurich we decided to do nothing.
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That's not quite accurate.  We decided to do nothing with the
understanding that figuring out extension composability and interplay
is the responsibility of newer extensions; they'd have to "watch out
for the dragons" [1].

  - Dmitri.

1. https://blog.litespeedtech.com/2020/02/11/quic-working-group-interim-zurich-recap/#extension-dragons

---

From: Martin Duke <[email-masked]>
Date: Thu, 1 Oct 2020 07:20:41 -0700

The concerns that Christian and MT raise are the same ones I was alluding
to, but I do think a draft like this one that adds the necessary bits of
protocol to enable multipath experimentation is the right scope. I don't
feel strongly whether we should adopt this and fix it, or fix it and then
adopt it.

Martin

On Thu, Oct 1, 2020 at 1:23 AM Martin Thomson <[email-masked]> wrote:

> I share Christian's concerns about the draft, but it's not just ACKs, it's
> the entire Uniflow concept that I would call into question.
>
> different thread) to directions that are less open-ended?  I'll suggest
> some options, but the chairs and/or ADs need to decide.
> multipath(draft-deconinck-multipath-quic <
> https://tools.ietf.org/html/draft-deconinck-multipath-quic>), with the
> explicit goal of not expanding the scope of the document.
> people should work this out in person, but that doesn't seem viable at the
> moment.  I'm happy to set up a long(3-4+hr) Google Meet to discuss this via
> videoconference if that helps move the discussion forward.
> > >
> > >>> There is some back-and-forth as to whether these are useful cases
> are not. I'll take it on faith, given the proponents, that there is a real
> hope of deploying this. However, I share the desire to not have the WG
> fully consumed by MP-QUIC for the foreseeable future.
> > >>
> specifications published and doing any cleanup work necessary SHOULD/MUST
> take priority, in the BCP 14 sense of those words.
> > >>
> > >> As Lars' initial note said, I'd also like to see the manageability,
> applicability, and datagram extension working group drafts, already adopted
> by QUIC, moving forward.
> > >>
> > >>> I don't think the community has well-established solutions for many
> problems in this space (e.g. scheduling). However, I think QUIC is a far
> better platform for experimentation than the alternatives, and would
> support a draft similar to draft-deconinck-multipath-quic <
> https://tools.ietf.org/html/draft-deconinck-multipath-quic> that provided
> the required protocol extensions to make that happen [1].
> > >>
> > >> I agree that scheduling is challenging - 3GPP is certainly spending
> time defining different strategies for behaviors, even in addition to the
> ones we described in
> https://datatracker.ietf.org/doc/draft-bonaventure-quic-atsss-overview/.
> > >>
> experimentation than anything I can think of (other suggestions are, of
> course, welcome).
> > >>
> > >>> IIUC the hard, unsolved problems are common to all MP protocols, so
> I don't think further research and future standards in this area are
> specific to QUIC or appropriate for the QUIC Working Group. But
> experimental QUIC extensions would accelerate this work, are appropriate
> for the WG, and may get us to a place where we could confidently develop
> standards about it.
> > >>
> fine plan to me (much better than not thinking about multicast in the IETF
> for a while longer).
> > >>
> work (MP-TCP was done in its own working group, for both Experimental and
> Standards-Track versions of the protocol specifications). Do the right
> thing, of course.
> > >>
> numerous nits about the design that are not relevant to this thread.
> > >>
> > >>
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 1 Oct 2020 09:40:16 -0500

Hi, Martin,

> feel strongly whether we should adopt this and fix it, or fix it and then

I had asked you and Magnus yesterday what our next steps on multicast
should be.

Am I reading this correctly as "start working on this draft, on the QUIC
mailing list, and let the working group do the right thing"?

I'd be fine with that, but wanted to check.

Best,

Spencer

>> I share Christian's concerns about the draft, but it's not just ACKs,
>> it's the entire Uniflow concept that I would call into question.
>>
>> https://tools.ietf.org/html/draft-deconinck-multipath-quic> that

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 1 Oct 2020 10:02:30 -0500

Hi Christian,

What about MPTCP? It would be good to know what MPTCP does in this case?
Also does having a different packet number space create problems, or is it
just your personal preference?

Behcet

On Thu, Oct 1, 2020 at 2:58 AM Christian Huitema <[email-masked]>
wrote:

> Specifically, I do not agree that having one packet number space per path
> is the right approach. This contradicts the design of QUIC V1, in which
> data sent on multiple paths shares a common packet number space. For
> example, in QUIC V1, we can start a connection on one path, migrate to
> another path, and keep the same packet number space throughout. I find that
> a very nice property -- and also an essential property if we want to
> support NAT rebinding. Handling multipath with a single number space
> requires some book-keeping on the sender side to match acknowledgements and
> sending paths, but we have working code for that.
>
> I am also not convinced that we properly understand the concept of "path".
> There is very little in the QUIC V1 protocol that requires transmission
> paths to be symmetric: any packet sent from a node to a valid address of
> the peer will be accepted, provided the crypto works. The linkage such
> requirement comes from the statement that a server starts directing traffic
> to a validated path when it sees the client using the same pair of
> addresses. This is an "implicit" linkage; I would expect that the first
> role of a multipoint extension would be to replace that by an "explicit"
> statement of preferences.
>
> support NAT, the IP headers must be outside the authentication envelope of
> the crypto. There are plausible attacks in which the attacker splices a
> cryptographically valid packet and a forged IP header. We have some
> defensive heuristics, but if we study multipath I hope we will end up with
> something better.
>
> -- Christian Huitema
>  1) No future work on multipath in the QUIC WG, in the belief the existing
> connection migration functionality is sufficient.
>  2) Adopt the existing draft as a starting point for QUIC multipath(
> draft-deconinck-multipath-quic
> <https://tools.ietf.org/html/draft-deconinck-multipath-quic>), with the
> the moment.  I'm happy to set up a long(3-4+hr) Google Meet to discuss this
> via videoconference if that helps move the discussion forward.
>
>>> There is some back-and-forth as to whether these are useful cases are
>>> not. I'll take it on faith, given the proponents, that there is a real hope
>>> of deploying this. However, I share the desire to not have the WG fully
>>> consumed by MP-QUIC for the foreseeable future.
>>>
>>
>> That sounds right. I'm assuming that getting the core QUIC specifications
>> published and doing any cleanup work necessary SHOULD/MUST take priority,
>> in the BCP 14 sense of those words.
>>
>>> support a draft similar to draft-deconinck-multipath-quic
>> I agree that scheduling is challenging - 3GPP is certainly spending time
>> defining different strategies for behaviors, even in addition to the ones
>> we described in
>>> IIUC the hard, unsolved problems are common to all MP protocols, so I
>>> don't think further research and future standards in this area are specific
>>> to QUIC or appropriate for the QUIC Working Group. But experimental QUIC
>>> extensions would accelerate this work, are appropriate for the WG, and may
>>> get us to a place where we could confidently develop standards about it.
>>>
>>
>> Targeting Experimental status for work in this area sounds like a fine
>> plan to me (much better than not thinking about multicast in the IETF for a
>> while longer).
>>
>> I know you have a variety of tools at your disposal to direct this work
>> (MP-TCP was done in its own working group, for both Experimental and
>>> [1] I would prefer that this draft be Experimental, and have numerous
>>> nits about the design that are not relevant to this thread.
>>>
>>
>>
>>
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 1 Oct 2020 08:38:53 -0700

On 10/1/2020 8:27 AM, Roberto Peon wrote:
> I’m probably on the other side of the coin here.
> Were we doing MP, I’d prefer to have a separate packet-space per path,
> because you probably need a separate congestion-controller instance
> per path.

Picoquic today implements separate congestion controller instances per
paths, with a single number space. That's not hard.

-- Christian Huitema

---

From: Martin Duke <[email-masked]>
Date: Thu, 1 Oct 2020 09:36:30 -0700

Hi Spencer,

This is Magnus's charter so I don't want to be expansive in my comments as
AD.

I think the next step would be for the WG to come to consensus on
this work, whatever that happens to be. My gentle suggestion is that a
standards-track design for simultaneous flow on multiple paths is
premature, and probably not QUIC-specfiic. However, a short experimental
draft to enable this in QUIC would allow the research community to make
progress faster.

On Thu, Oct 1, 2020 at 7:40 AM Spencer Dawkins at IETF <
[email-masked]> wrote:

>>> > > I favor #2, but these may not be the right options.  Normally I'd
>>> say people should work this out in person, but that doesn't seem viable at
>>> many problems in this space (e.g. scheduling). However, I think QUIC is a
>>> far better platform for experimentation than the alternatives, and would
>>> > >>> IIUC the hard, unsolved problems are common to all MP protocols,
>>> so I don't think further research and future standards in this area are

---

From: Christian Huitema <[email-masked]>
Date: Thu, 1 Oct 2020 08:23:21 -0700

>> discussion on GitHub [2] and in Zurich we decided to do nothing.

We certainly did not "do nothing". The time stamp extension, for
example, was rewritten as an extension that can be composed with the ACK
frame, instead of as an extension changing the format of the ACK frame.
See https://datatracker.ietf.org/doc/draft-huitema-quic-ts/.

I think we should keep this design pattern for future extensions. For
example, instead of "rewrite the retire_connection_id" frame, we should
define an extension frame that composes with it.

-- Christian Huitema

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 1 Oct 2020 11:58:31 -0500

On Thu, Oct 1, 2020 at 10:28 AM Roberto Peon <[email-masked]> wrote:

> because you probably need a separate congestion-controller instance per
> path.
>

me too.

Behcet

> -=R
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Behcet Sarikaya <
> [email-masked]>
> *Reply-To: *"[email-masked]" <[email-masked]>
> *Date: *Thursday, October 1, 2020 at 8:04 AM
> *To: *Christian Huitema <[email-masked]>
> *Cc: *Matt Joras <[email-masked]>, Olivier Bonaventure <
> [email-masked]>, Spencer Dawkins at IETF <
> [email-masked]>, Ian Swett <ianswett=
> [email-masked]>, QUIC WG <[email-masked]>, Martin Duke <
> [email-masked]>
> *Subject: *Re: Preparing for discussion on what to do about the multipath
> extension milestone
>
>
>
> Hi Christian,
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Ddeconinck-2Dmultipath-2Dquic&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yV58k7v0W6KRG-4kJdhZ3Hs461h_EZKSw7VT6VHgsvU&s=qRLqBfssWffIVMcb3b7R5gxykJMN9tqDTp7pq9j5QCY&e=>),
> There is some back-and-forth as to whether these are useful cases are not.
> I'll take it on faith, given the proponents, that there is a real hope of
> deploying this. However, I share the desire to not have the WG fully
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Ddeconinck-2Dmultipath-2Dquic&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yV58k7v0W6KRG-4kJdhZ3Hs461h_EZKSw7VT6VHgsvU&s=qRLqBfssWffIVMcb3b7R5gxykJMN9tqDTp7pq9j5QCY&e=> that
> https://datatracker.ietf.org/doc/draft-bonaventure-quic-atsss-overview/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dbonaventure-2Dquic-2Datsss-2Doverview_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=yV58k7v0W6KRG-4kJdhZ3Hs461h_EZKSw7VT6VHgsvU&s=79s85o1Msi5birIYHLoQ2DCLdCA8M8KOYgh_gWY81EI&e=>
> .
>
>
>
> [1] I would prefer that this draft be Experimental, and have numerous nits
> about the design that are not relevant to this thread.
>
>
>
>
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 1 Oct 2020 12:09:05 -0500

Hi, Martin,

> This is Magnus's charter so I don't want to be expansive in my comments as

Sure. I wasn't sure whether you two had time to talk.

Part of my original question was whether you two thought this was QUIC work
or something else that might be chartered outside QUIC - I'm sorry if that
wasn't clear.

> draft to enable this in QUIC would allow the research community to make

Thank you for making that clearer (at least to me), and I agree completely
with that.

Best,

Spencer

>>> The concerns that Christian and MT raise are the same ones I was
>>> alluding to, but I do think a draft like this one that adds the necessary
>>> bits of protocol to enable multipath experimentation is the right scope. I
>>> don't feel strongly whether we should adopt this and fix it, or fix it and
>>> then adopt it.
>>>
>>
>>>> > > Given the responses, can we narrow down the way forward(ideally on
>>>> a different thread) to directions that are less open-ended?  I'll suggest
>>>> manageability, applicability, and datagram extension working group drafts,
>>>> already adopted by QUIC, moving forward.
>>>> > >>
>>>> spending time defining different strategies for behaviors, even in addition
>>>> to the ones we described in

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 2 Oct 2020 00:18:03 +0100

I was a bit unclear in my wording, sorry. A github issue was raised at the
time, suggesting we add text to the base specs about composability. The
outcome of discussion for that specific proposal was "do nothing".

I didn't mean to suggest that nothing else had been done. I cannot talk on
behalf of anyone else in that respect.

That the base specs say nothing, does not mean that there is nothing to be
said.

Cheers
Lucas

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 02 Oct 2020 09:48:54 -0700

Hello,

to answer Ian's question regarding Multipath:

We (Apple) are using both Wi-Fi and Cell simultaneously for a number of use
cases. The scheduling decision is different for each of the use cases based
on policy, quality of experience requirements and traffic patterns. Our
different scheduler modes (handover, interactive and aggregate) are
described in [1].

For Siri, we send traffic on Wi-Fi and Cell simultaneously depending on the
latencies of each path, using the MPTCP interactive mode [1]. This
per-packet scheduling allows it to have the best latency across Wi-Fi and
Cell. Wi-Fi/Cell latency can have huge short-term variances due to
signal-strength, at very short time scales. A pure migration solution would
not allow for a quick reaction on those time scales.

Apple Music uses Wi-Fi and Cell simultaneously whenever the need arises
where Wi-Fi signal is not strong enough. As music streaming can potentially
consume significant amounts of data, it is important to keep cellular data
usage to a strict minimum, and here it uses the MPTCP aggregation mode [1]
only when necessary to ensure a smooth music playback. A migration-based
solution would instead be a hard handover to cellular. This could consume
significantly greater amounts of cellular data (with all the costs that
entails), while simultaneously being very dependent on the good quality of
the cellular access.

Considering the benefits of using multipath for these use cases and the work
involved for the IETF to standardize MP-QUIC, I believe that the QUIC
Working Group should add MP-QUIC to the charter as a deliverable. Otherwise,
there will always be situations (like the above) where QUIC migration will
not be satisfactory in bad network environments. In those situations, one
must be able to leverage multiple paths simultaneously to get the best of
them.

Keeping the complexities of packet scheduling and path management
out-of-scope is a good idea, as both are generic research questions that
have already seen significant work as part of MPTCP and will still be worked
on - similar to QUIC's stance on more sophisticated congestion controllers.

[1] https://developer.apple.com/documentation/foundation/urlsessionconfiguration/multipathservicetype

Christoph

On 09/30/20 - 12:12, Ian Swett wrote:
> I'm very receptive to Jana's arguments, but I think it's also worth
> understanding how much work we believe this will be for the QUIC WG.
> 
> Many fairly complex multipath problems have already been dealt with by
> supporting connection migration in QUIC v1, such as path validation and
> linkability between flows.  Multiple packet number spaces during the
> handshake means that implementations already have some ability to have
> multiple loss recovery contexts, though they may not be easily adaptable to
> multipath.
> 
> Scheduling seems like a big problem, but it's not clear to me that the QUIC
> working group should be attempting to solve that.  I'd argue we should make
> that explicitly out of scope if we took on this work.
> 
> I certainly think MP-QUIC is a cleaner solution than MP-TCP.
> 
> I lean towards keeping multipath in scope, and I'd be willing to spend time
> improving a proposal if it was adopted.  One reason is that I never want to
> be forced to support MP-TCP because there is no MP-QUIC.  I also don't
> think this has to be designed with HTTP in mind, even if that was the focus
> of QUIC v1.  MASQUE/VPNs are a more compelling use case to me.
> 
> Ian
> 
> PS: It'd be good if someone from Apple could share their perspective on
> whether MP-QUIC is necessary, since they have quite a bit of experience
> with MP-TCP.
> 
> On Tue, Sep 29, 2020 at 4:31 PM Matt Joras <[email-masked]> wrote:
> 
> > Two replies inline.
> >
> > On Tue, Sep 29, 2020 at 1:12 PM Olivier Bonaventure <
> >> Spencer,
> >> > On Fri, Sep 25, 2020 at 5:00 AM Lars Eggert <[email-masked]
> >> > <mailto:[email-masked]>> wrote:
> >> >
> >> >     In parallel to progressing the "base drafts" towards RFC
> >> >     publications, the WG should now also begin to pick up the pace on
> >> >     our other adopted work items (ops drafts, extensions, etc.)
> >> >
> >> >     One important other discussion item is what to do about the
> >> >     multipath extension milestone, which some have suggested should be
> >> >     dropped, while others still show interest to pursue it.
> >> >
> >> >
> >> > So, I'd like to understand the suggestion to drop this milestone,
> >> before
> >> > I start trying to discuss that suggestion :-).
> >>
> >> I'd like to understand this as well.
> >>
> > I want to echo Jana's reply from the original discussion here
> > <https://mailarchive.ietf.org/arch/msg/quic/R-uJhzzXBz-93OTmYupXu8ooJdA/>.
> > Everyone agrees multi-path transports are an interesting problem, and IETF
> > participants love to solve interesting problems. It's not clear, however,
> > that it should be a primary milestone of the working group at this time.
> >
> >> >
> >> > In conversations with individual folk, I've heard these concerns about
> >> > QUIC multipath:
> >> >
> >> > - Whether it will be possible to evaluate multipath performance at
> >> > scale, both for evaluating proposals and testing implementations.
> >>
> >> We already have plenty of experience with MPTCP with several large
> >> deployments, including :
> >>
> >> - MPTCP on all iPhones since 2013 with a growing number of applications
> >> - MPTCP on Android smartphones in South Korea for WiFi/4G offload
> >> - MPTCP in hybrid access networks that are used by different network
> >> operators to combine xDSL and LTE
> >>
> >> Multipath extensions to QUIC would be applicable in these different use
> >> cases
> >>
> > There is a difference between "Someone is doing these things with MP-TCP",
> > and "Someone has shown interest and intent in doing these things with
> > MP-QUIC". Additionally, the first example can, as I understand it, largely
> > be replicated with QUIC connection migration. The other two I would like to
> > hear more about, because it would surprise me if they amounted to a
> > significant deployment. Support for MP-TCP in Android is virtually
> > nonexistent, for example.
> >
> > We have deployment experience with pre-IETF QUIC, and growing deployment
> > experience with IETF QUIC itself. However, even things like connection
> > migration in the base drafts have been fully exercised in the real world
> > yet. As far as I know there have not been any major implementers or
> > "championing" applications showing a lot of interest in deploying MP-QUIC
> > in the near future, with the exception of the 3GPP's desire to integrate it
> > into the ATSSS specification. As I understand it, this interest does not in
> > itself even imply a deployment in the near future. I would rather we not
> > endeavor on tackling what is certainly going to be a difficult design
> > problem simply under the hope that "If you build it, applications will turn
> > up".
> >
> >>
> >> > - The complexity involved in making decisions dynamically about which
> >> > path to send a given packet on (which could be a research topic, given
> >> > certain constraints and goals).
> >>
> >> The packet scheduling problem is a much simpler problem in multipath
> >> transport protocol than congestion control. I would not consider this as
> >> a research topic given all the experience we have with MPTCP
> >>
> >> > If I've misunderstood or misquoted, my apologies, of course. Please
> >> > correct me.
> >> >
> >> > What other concerns do people have? I'd like to get all the objections
> >> > out at the beginning of the discussion.
> >>
> >> Same for me
> >>
> >>
> >> Olivier
> >>
> >> Matt Joras
> >

---

From: Matt Joras <[email-masked]>
Date: Fri, 2 Oct 2020 10:32:58 -0700

Christoph, this is really great information regarding Apple's usage of
MPTCP, thank you for sharing.

In general I agree with what Ian and Martin Duke and others have said
above, which is that limiting the scope of the WG's work on multipath to be
the core logistics of getting the job done at the transport seems sound --
however in practice I do wonder how that would look. It is nice to say that
e.g. packet scheduling policy, path management, API, and HTTP/3
considerations are out of scope, but then how would implementers know we
have achieved parity with MPTCP and we aren't missing pieces in the design?
As an implementer having a deployment which is primarily HTTP/3, if we
wanted to start testing out MPQUIC as developed by the WG we would need to
map the existing work done for MPTCP for scheduling and policy onto QUIC.
This is arguably similar to, e.g., mapping non-Reno congestion control onto
QUIC, as we have done with BBR. I have concerns though that multipath is
more complicated than CCA integration, and this will delay feedback signals
on whether the MPQUIC has design defects.

Another question, and perhaps this is getting increasingly off-topic, but
now that we are getting into designing "significant" extensions, I think it
is worth asking whether this should be an extension at all. I.e. should
MPQUIC be a core feature of QUICv2? If not, I think the extension
composability problems stand, which the WG has mostly punted on addressing;
implementers interested in composing MPQUIC with any other extension would
be on their own.

Matt Joras

On Fri, Oct 2, 2020 at 9:48 AM Christoph Paasch <[email-masked]> wrote:

> [1]
> https://developer.apple.com/documentation/foundation/urlsessionconfiguration/multipathservicetype
>
>
>
> Christoph
>
>
> > > <
> https://mailarchive.ietf.org/arch/msg/quic/R-uJhzzXBz-93OTmYupXu8ooJdA/>.

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 2 Oct 2020 18:51:07 +0100

+1 to the suggestion to discuss use cases, it is helpful for clarity. And
thanks Christoph for sharing some specific ones.

For the use cases "Siri" and "Apple Music", since I'm not experienced with
multipath there's a gap in my understanding of how the proposal in
draft-deconinck-quic-multipath would be used to satisfy these use cases.
For instance, is the behaviour client-driven, server-driven or does it
require coordination? Would anyone be so kind as to share _an example_ of
how uniflows would be used within a QUIC connection that uses a
request/response paradigm for data exchange.

now that we are getting into designing "significant" extensions, I think it
is worth asking whether this should be an extension at all. I.e. should
MPQUIC be a core feature of QUICv2?

I tried to spin that out into another thread [1]. IIRC someone previously
suggested that MP-QUIC could start as an experimental version of it's own.
I can't find a citation for that though sorry.

[1] https://mailarchive.ietf.org/arch/msg/quic/p2II8y3y1FXy4kWMSuf1lgAXaFA/

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 2 Oct 2020 13:54:32 -0500

Hi, Lucas,

On Fri, Oct 2, 2020 at 12:51 PM Lucas Pardue <[email-masked]>
wrote:

> > Another question, and perhaps this is getting increasingly off-topic,
> but now that we are getting into designing "significant" extensions, I
> think it is worth asking whether this should be an extension at all. I.e.
> should MPQUIC be a core feature of QUICv2?
>

At a minimum, Martin and I have both been talking about experiments,
experimental drafts, and perhaps even experimental versions, but I think
the clearest citations is Martin's, at
https://mailarchive.ietf.org/arch/msg/quic/FH8JDTD5noAbnsSjr7s4sVuiI5o/.

He also noted that Magnus is the responsible AD for QUIC in his note.

Best,

Spencer

> [1]
> https://mailarchive.ietf.org/arch/msg/quic/p2II8y3y1FXy4kWMSuf1lgAXaFA/
>

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 02 Oct 2020 16:20:28 -0700

Hello,

On 10/02/20 - 18:51, Lucas Pardue wrote:
> require coordination?

MPTCP has some limited coordination capabilities. Namely, it allows to
signal to the peer if a "subflow" (in QUIC-language "uniflow") is using a
backup-interface (through the backup-bit in the MP_JOIN establishment or the
MP_PRIO-option - in case you want to look it up in the RFC).

That signaling is very limited in MPTCP though. And this is exactly the
place where MPQUIC can have a significant benefit over MPTCP as it can be more
descriptive on the scheduling because it does not have limitations that the
TCP-option space has.

> Would anyone be so kind as to share _an example_ of

I'm not entirely sure which parts of the MPQUIC-exchange are unclear, but
I believe the best description can be found at: https://multipath-quic.org/conext17-deconinck.pdf

In principle, the flow would be the following:

Assuming the client is on WiFi, whenever the client realizes that WiFi is
getting weaker (based on RTT-measurement or congestion-indication or
system-wide indicators), it establishes a uniflow on the cellular interface.
As soon as that uniflow is in ACTIVE state, it can be used for data
transmission.
The client can send the request (assuming it fits in a single packet) on
either WiFi or Cell, based on - for example - the measured RTT of
each of the paths.
The server (knowing that both uniflows are active) can then send the reply,
scheduling each packet on either WiFi or Cell (e.g., following one of the
schedulers I mentioned in the previous email).

Christoph

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 3 Oct 2020 00:56:22 +0100

Thanks Christoph, this is very useful!

> descriptive on the scheduling because it does not have limitations that the

So maybe I'm still missing it, but the MP-QUIC proposal seems to omit such
a MP_PRIO signal. So the gap I had was wondering if the client needed to
actively bring up or tear down uniflows in order to control the server
sending on a higher-cost cellular network (for however one measures cost).

Although I have drawn parallels with stream prioritization, there is a
difference. The server behaviour of stream prioritization cannot really do
much damage. The server behaviour for the multipath use case seems to be
higher stakes - getting it wrong could hurt end users.

I'm not saying that we have to state a scheduler now, or research one. But
I do wonder why the uniflows proposal does  not simply start with adopting
the same/similar signal that MPTCP already has.

Cheers
Lucas

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 02 Oct 2020 17:09:02 -0700

Hello Lucas,

> higher stakes - getting it wrong could hurt end users.

"could hurt end users", you mean consume significant cellular data? Or do
you mean hurt performance?

If it is the former, I agree. And that's why the client should not establish
a uniflow unless it actually is willing to receive data on that uniflow.
Sure, any scheduling-signaling we could add to QUIC could help alleviate
those problems, but one cannot "trust" the server to ever do the right
thing.

If it is the latter, I think that stream prioritization suffers from the
same potential to do severe damage in terms of performance.

> the same/similar signal that MPTCP already has.

I doubt there is a specific reason on why the MP-QUIC draft does not have
such an "MP_PRIO"-concept. In any case, that would be a minor point that can
easily be added.

Christoph

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 3 Oct 2020 01:53:02 +0100

Hey,

> a uniflow unless it actually is willing to receive data on that uniflow.

Yes I meant this former case. And by using the term end user I was very
much channeling RFC 8890.

> such an "MP_PRIO"-concept. In any case, that would be a minor point that

Thats a good suggestion, I think adding something on this would help.

Cheers
Lucas

---

From: Ian Swett <[email-masked]>
Date: Sun, 4 Oct 2020 10:56:43 -0400

Christoph, thanks for the great information.

Is the lack of MP-QUIC likely to cause QUIC to be disabled for certain key
use cases(ie: Siri and Apple Music), or can existing connection migration
be made to work?

For example, I can imagine working around true multipath by only using one
path at once, but sending PATH_CHALLENGE on the other path to probe
reliability and/or RTT.  Or even sending identical packets(ie: a request)
on both paths and the receiver will naturally respond on the path the
packet is received first and drop the other packet as a duplicate.

My key takeaway from the multipath scheduler papers I've read is that most
of the performance gains are obtained by always sending on the faster of
the two available paths, and in a large set of circumstances, nothing
should be sent on the slower path.

Thanks, Ian

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 4 Oct 2020 22:54:24 +0200

Lucas,
> Approaches such as binding a stream to a uniflow could cause starvation 
> of the related-path due to flow control even when congestion window is 
> open. 

I would suggest to start by not binding a stream to a specific uniflow. 
This might be added later if useful.

> Flow control is driven by applications putting data into the 
> transport, and consuming from it. I suspect HTTP/2 over MPTCP does not 
> have this problem, mainly because TCP's flow control trumps. But maybe 
> I'm wrong.

MPTCP provides the same bytestream as TCP and does not expose anything 
special AFAIK to HTTP/2

> Some implementations like quiche decouple the transport library from 
> UDP. Applications hoping for MP-QUIC with such implementations will have 
> to accommodate some of the complexity, the transport cannot do it all.

Path management (i.e. deciding when to create uniflows) is something 
that MPQUIC applications could do. The Linux MPTCP implementation 
initially included this feature as a set of kernel modules. It involved 
to libraries that interact with the kernel using netlink, see

https://inl.info.ucl.ac.be/publications/smapp-towards-smart-multipath-tcp-enabled-applications.html

> I do agree that we don't have to solve all of these things upfront and 
> in one document. But I think there is a risk of defining mechanisms and 
> ignoring some pathological cases. As a parallel, QUIC provides 
> congestion control mechanisms and does not mandate any algorithms but it 
> has taken care to suggest one that is going to be safe to deploy on the 
> Internet.

This is a reasonable approach

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 4 Oct 2020 23:17:14 +0200

Lucas,

> +1 to the suggestion to discuss use cases, it is helpful for clarity. 
> And thanks Christoph for sharing some specific ones.
> 
> For the use cases "Siri" and "Apple Music", since I'm not experienced 
> with multipath there's a gap in my understanding of how the proposal in 
> require coordination? Would anyone be so kind as to share _an example_ 
> of how uniflows would be used within a QUIC connection that uses a 

Let me try with a simple example on a moving smartphone. The application 
will send small amounts of data and receive variable amounts of data 
(depending on the type of requests).

We create a sending and a receiving uniflow on both the Wi-Fi and the 
cellular interface. The smartphone has two sending uniflows and the 
server as well.

To send a short request, the client duplicates it over its two sending 
uniflows since it does not know which of the two uniflows will be the best.

To return the response, the server could use the same scheduler if the 
response is short. However, if the response is long, this is not very 
efficient since data is sent over two paths. It could then use both 
paths to send the data and get the lowest delay to deliver the response.
This could be modulated by policies if the user pays on a per volume 
basis over one path and not over the other.

Another example is the hybrid access network scenario with a DSL and an 
LTE path. There, the objective is to send data over the LTE path only 
when the DSL is full.

In this case, the solution would differ. The client would first create 
sending and receiving uniflows over the DSL path. It then monitors the 
usage of this path. As long as the DSL is not fully used for some period 
of time (e.g. one or a few seconds), all data flows over the DSL path. 
Once the DSL path is saturated, the client creates a receiving uniflow 
(and possibly a sending one if the DSL upstream is saturated) over the 
LTE path. The second path can be used to offload traffic. In practice, 
the client and the server use a priority scheduler to always prefer the 
DSL over the LTE path, see

https://inl.info.ucl.ac.be/publications/increasing-broadband-reach-hybrid-access-networks.html

Another usecase is a device that has a high-speed satellite downlink 
(but very slow upling) and a low-speed cellular connection. In this 
case, it can use two receiving uniflows (one cellular and one satellite) 
and one sending one (over the cellular). Requests are sent over the 
cellular path and responses are received over the satellite uniflow and 
acks are sent over the cellular uniflow.

There is a generic discussion on schedulers in

https://datatracker.ietf.org/doc/html/draft-bonaventure-iccrg-schedulers-01

Olivier

> think it is worth asking whether this should be an extension at all. 
> I.e. should MPQUIC be a core feature of QUICv2?
> 
> I tried to spin that out into another thread [1]. IIRC someone 
> previously suggested that MP-QUIC could start as an experimental version 
> of it's own. I can't find a citation for that though sorry.
> 
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 4 Oct 2020 23:30:32 +0200

Hello,
> I believe the best description can be found at: https://multipath-quic.org/conext17-deconinck.pdf

The design changed to reflect the changes in QUIC since the publication 
of this paper. Quentin De Coninck describes the evolution of his design 
in his PhD thesis (chapters 5 and 6) :

https://inl.info.ucl.ac.be/publications/flexible-multipath-transport-protocols.html

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 4 Oct 2020 23:33:33 +0200

Lucas,
> So maybe I'm still missing it, but the MP-QUIC proposal seems to omit 
> such a MP_PRIO signal. So the gap I had was wondering if the client 
> needed to actively bring up or tear down uniflows in order to control 
> the server sending on a higher-cost cellular network (for however one 
> measures cost).

The description in the mpquic draft focuses on the basic principles. 
Once we agree on them, we can add different extensions such as the 
support for backup uniflows...

> difference. The server behaviour of stream prioritization cannot really 
> do much damage. The server behaviour for the multipath use case seems to 
> be higher stakes - getting it wrong could hurt end users.
> 
> I'm not saying that we have to state a scheduler now, or research one. 
> But I do wonder why the uniflows proposal does  not simply start with 
> adopting the same/similar signal that MPTCP already has.

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Sun, 4 Oct 2020 23:33:45 +0200

Hello,

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 5 Oct 2020 00:32:42 +0100

Olivier,

Many thanks for these resources. I'm slowly reading and digesting.
Quentin's Thesis I think has some good discussion about consideration
factors for an MP-QUIC abstract design, such as consideration of path
validation etc. I don't know if uniflow is the correct design but it
definitely helps to understand the motivation for it.

MP-QUIC appears to offer some things that conventional QUIC might not be
able to do*. But it, to me, also presents some problems that don't exist
today. As an individual commenter, not having an in-band signal for uniflow
prioritisation gives me cause for concern. If I were to deploy an MP-QUIC
general-purpose server, I don't have much confidence that I could meet the
use cases that clients want.

Cheers
Lucas

* Spencer occasionally mentions muticast in this thread, which I think is
the autocorrect kicking in. But it's mention is relevant because the
asymmetry of uniflows, and separation of data and control channels could
solve some of the problems that draft-pardue-quic-http-mcast goes to length
to design around.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Sun, 4 Oct 2020 18:52:49 -0500

Hi, Lucas,

> solve some of the problems that draft-pardue-quic-http-mcast goes to length

That would have been unintentional, but I'm glad if I make helpful errors
from time to time. ;-)

Best,

Spencer

---

From: Matt Joras <[email-masked]>
Date: Sun, 4 Oct 2020 22:26:16 -0700

Olivier, some questions/replies inline to your response to Lucas. In
general I have similar concerns about coordination, and I am wondering
whether MPQUIC is trying to solve too many problems at once.

> (depending on the type of requests).
I want to start by saying this is a real usecase, and a problem we see very
obviously through quality of experience outliers for people using our
products.

> basis over one path and not over the other.
This somewhat hand-waves away the scheduling problem. Most Internet traffic
flows in the direction of server -> client, where typical mobile clients
are obviously the ones with potentially multiple interfaces to the
Internet. The client also has the most information about the likely quality
of the first radio hop into the access network (e.g. signal quality). The
client also _may_ know about things like data pricing. However, the server
is the one which is responsible for scheduling most data across these
paths. To make good, proactive (rather than reactive) scheduling decisions
the server needs to be fed this information as input to its scheduler. This
seems a difficult thing to achieve in practice, and without it I wonder
whether the complexity of "full" multipath will be worth it until we solve
the signalling problem. What Ian is suggesting above, I think, is
essentially an Active/Passive extension to the existing connection
migration mechanisms we have today. What are your thoughts on this as an
initial direction? It would allow operators a way to solve this particular
problem with only mild modifications to the core protocol. Said another
way, I think in theory an omniscient packet scheduler can make very
intelligent decisions which would definitely benefit application quality.
In practice though I have concerns about how effective these schedulers
will be versus the naive approach which could be thought of as
"Active/Passive" or "Failover", eschewing bandwidth aggregation entirely.
I'd also like to echo what Kazuho said, which is that "Multipath" can mean
many things, and I'd prefer we narrow down the problem we want to solve in
the WG, which will drive our design direction.

> https://inl.info.ucl.ac.be/publications/increasing-broadband-reach-hybrid-access-networks.html

For these usecases, are you imagining that they would largely be used
internally for Internet operators? I always struggle with the hybrid access
examples, as they seem to assume a lot of knowledge about the underlying
networks that typical endpoints can't simply intuit from the ether. The
linked paper seems to suggest MPTCP as a proxy solution, which I gather is
largely the usecase things like ATSSS have for MPQUIC. However, as a mobile
application how do I know the DSL link is "full" and thus that I should
create a uniflow over the LTE path? The same question applies to the
server. It would be awesome if clients and servers had more active
information about the underlying network's state rather than being
reactive, but beyond things like ECN this seems difficult to achieve in
practice. If the Hybrid Access Network usecase of multipath presupposes a
deployment in an operator's network then I would argue it is somewhat
antithetical to the goals of QUIC, which deliberately puts more control at
the endpoints.

Matt Joras

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 5 Oct 2020 09:02:54 +0200

Ian,

> multiple loss recovery contexts, though they may not be easily adaptable 
> to multipath.
> 
> Scheduling seems like a big problem, but it's not clear to me that the 
> QUIC working group should be attempting to solve that.  I'd argue we 
> should make that explicitly out of scope if we took on this work.

I agree, this is what MPTCP did as well.
> ADD_ADDRESS/REMOVE_ADDRESS frames and modifies the ACK, 
> NEW_CONNECTION_ID, and RETIRE_CONNECTION_ID frames.  It also adds what 
> appears to be a debugging frame(UNIFLOWS) and a single transport 
> parameter to indicate support and indicate the max number of uniflows.  
> Having sets of CIDs per path seems an unfortunate complication to me, 
> but I haven't thought about it enough.  The two new frames and 
> modifications to the ACK frame are simple enough.
> 
> I lean towards keeping multipath in scope, and I'd be willing to spend 
> time improving a proposal if it was adopted.  One reason is that I never 
> want to be forced to support MP-TCP because there is no MP-QUIC.  I also 
> don't think this has to be designed with HTTP in mind, even if that was 
> the focus of QUIC v1.  MASQUE/VPNs are a more compelling use case to me.

I agree, the multipath VPN use case is probably more useful in the short 
term than HTTP/3 over MPQUIC

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 5 Oct 2020 09:07:15 +0200

Ian,

> this via videoconference if that helps move the discussion forward.

Thanks for proposing this excellent suggestions.

> Or we can form a design team, which typically takes O(3 months) to finish.

I think that it would be useful to continue the technical discussion 
that has started. I'll leave up to the WG to decide how best to proceed. 
I'd also suggest to initially keep the scope of the multipath discussion 
narrow. We can update the draft to include the feedback received so far 
or collect issues on github and then work to address them

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 5 Oct 2020 15:04:24 +0200

Ian,

> Is the lack of MP-QUIC likely to cause QUIC to be disabled for certain 
> key use cases(ie: Siri and Apple Music), or can existing connection 
> migration be made to work?
> 
> For example, I can imagine working around true multipath by only using 
> one path at once, but sending PATH_CHALLENGE on the other path to probe 
> reliability and/or RTT.  Or even sending identical packets(ie: a 
> request) on both paths and the receiver will naturally respond on the 
> path the packet is received first and drop the other packet as a duplicate.
> 
> My key takeaway from the multipath scheduler papers I've read is that 
> most of the performance gains are obtained by always sending on the 
> faster of the two available paths, and in a large set of circumstances, 
> nothing should be sent on the slower path.

This assumes that you have two paths that provide a good and stable 
performance. If path performance changes quickly because one of the 
hosts moves then it makes sense to use both paths simultaneously. This 
is true as well when the bandwidth of a single path is too slow. We've 
seen many reports form companies that use MPTCP to aggregate several 
low-speed Internet connection, DSL+DSL, DSL+4F, multiple 4G, ...

The most unexpected one was a television broadcaster that placed MPTCP 
in the car used by its journalists. With MPTCP, they could aggregate 
satellite, WiFi and several 4G interfaces to speed-up the upload of 
videos collected by their journalists in the field

https://www.youtube.com/watch?v=JMRWq7aqi9o

There are many situations where a single link is not sufficient to 
support the required applications. Multipath allows to address all these 
deployments.

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 5 Oct 2020 15:10:00 +0200

Lucas,
> today. As an individual commenter, not having an in-band signal for 
> uniflow prioritisation gives me cause for concern. If I were to deploy 
> an MP-QUIC general-purpose server, I don't have much confidence that I 
> could meet the use cases that clients want.

Prioritisation can be easily added to MPQUIC. It was not included in the 
current draft because we focused on the basic principles and did not 
want to add new features before the start of technical discussions on 
the mailing list. There are different ways to do prioritisation, 
starting from the one-bit solution used by MPTCP to other scenarios like 
putting an integer weigth on uniflows or limiting the bandwidth on a per 
uniflow basis, ...

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 5 Oct 2020 15:23:41 +0200

Matt,

> I want to start by saying this is a real usecase, and a problem we see 
> very obviously through quality of experience outliers for people using 
> our products.
> 
> 
> This somewhat hand-waves away the scheduling problem. Most Internet 
> traffic flows in the direction of server -> client, where typical mobile 
> clients are obviously the ones with potentially multiple interfaces to 
> the Internet. The client also has the most information about the likely 
> quality of the first radio hop into the access network (e.g. signal 
> quality). The client also _may_ know about things like data pricing. 

Agreed

> However, the server is the one which is responsible for scheduling most 
> data across these paths. To make good, proactive (rather than reactive) 
> scheduling decisions the server needs to be fed this information as 
> input to its scheduler. This seems a difficult thing to achieve in 
> practice, and without it I wonder whether the complexity of "full" 
> multipath will be worth it until we solve the signalling problem. 

There are several techniques that allow the server to learn about the 
different performance of the paths with MPTCP. Those can be applied to 
MPQUIC as well.

If the application uses short requests and short responses, then a 
simple heuristic for the server is to send the response on the path 
where the last packet (data or ack) has been received. The reception of 
a packet is a good indication that a link works.

If the application uses longer responses, then the congestion control 
used by the server over the two paths will enable it to easily find the 
best way to spread the load. If one path has a longer rtt or losses, 
then its congestion window will be slower than the other one and packets 
will naturally flow on the best path, but not only on this path. This 
does not require a specific scheduler and would work better than a 
strict weighted-round-robin scheduler where the client would indicate 
that it wants to receive 2/5 of the bytes over WiFi and 3/5 over cellular.

> What 
> Ian is suggesting above, I think, is essentially an Active/Passive 
> extension to the existing connection migration mechanisms we have today. 
> What are your thoughts on this as an initial direction? It would allow 
> operators a way to solve this particular problem with only mild 
> modifications to the core protocol. Said another way, I think in theory 
> an omniscient packet scheduler can make very intelligent decisions which 
> would definitely benefit application quality. In practice though I have 
> concerns about how effective these schedulers will be versus the naive 
> approach which could be thought of as "Active/Passive" or "Failover", 
> eschewing bandwidth aggregation entirely. I'd also like to echo what 
> Kazuho said, which is that "Multipath" can mean many things, and I'd 
> prefer we narrow down the problem we want to solve in the WG, which will 
> drive our design direction.

When coupled with congestion control, a simple packet scheduler such as 
lowest-rtt first (if paths have the same cost) or priority (for the 
lowest cost path) works well with MPTCP. The same would apply for MPQUIC

> internally for Internet operators? I always struggle with the hybrid 
> access examples, as they seem to assume a lot of knowledge about the 
> underlying networks that typical endpoints can't simply intuit from the 
> ether. The linked paper seems to suggest MPTCP as a proxy solution, 

Yes, that's deployed with MPTCP proxies running on home routers.

> which I gather is largely the usecase things like ATSSS have for MPQUIC. 
> However, as a mobile application how do I know the DSL link is "full" 
> and thus that I should create a uniflow over the LTE path? The same 
> question applies to the server. It would be awesome if clients and 
> servers had more active information about the underlying network's state 
> rather than being reactive, but beyond things like ECN this seems 
> difficult to achieve in practice. If the Hybrid Access Network usecase 
> of multipath presupposes a deployment in an operator's network then I 
> would argue it is somewhat antithetical to the goals of QUIC, which 
> deliberately puts more control at the endpoints.

On endpoints, you can get the same result by having a target bandwidth 
on the Wi-Fi interface. For example, a smartphone application could 
assume that if the current Wi-Fi bandwidth is below x Mbps then it 
should enable the LTE interface to boost a long download. Similarly, a 
videostreaming application could have a target that corresponds to the 
default quality chosen by the user and enable the LTE when it cannot 
receive video at the expected quality. The same applies for a radio 
streaming application.

Olivier

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 5 Oct 2020 19:36:49 -0700

After going through this thread, I'm still finding the framing of the
conversation to be: How do we get "multipath" into QUIC? Perhaps I'm not
being very charitable, but the title of this email thread isn't helping.

This framing is problematic and leads to the false scarcity of options we
see in this discussion, but I am still stuck at the premise. Let me explain.

Transport multipath is not a _problem_, it is a _class of solutions_ that
can be implemented in a transport. Multipath is simply the use of multiple
paths. The manner in which multiple paths are used depends on the problem
you are trying to solve.

For example, if you want to solve the handoff problem for reliability,
connection migration is a transport-level multipath solution for it. Note
that there are other ways to solve this problem. For example, if the
application is RESTful and simply fetching data (e.g. YouTube), then you
can implement this in the application quite easily. However, for other
applications, such as live transcription, where context is super important
and transport state is tied to application state, maintaining connection
state is necessary. We agreed that the problem was important -- I believe
that was in Melbourne -- and we then went about solving it with connection
migration with which, after many iterations, we now have a fragile peace.

So echoing what others have been saying on this thread: What is the problem
we are trying to solve? Do we agree on the importance of it? Is this
bandwidth aggregation across WiFi/Cellular (is that an important enough use
case?)? Is it latency reduction for HTTP? Is it increased reliability? In
what ways is it more than connection migration?

I don't want a laundry list of things that "multipath" can do. But an
analysis of the most important problem and why it needs to be solved. This
is where I'd like to engage, not around adoption of a solution, let alone
adoption of a draft.

I've still not heard folks rallying around one problem that we need to
solve right now, and I'm not hearing people saying that we need to do
this before we gain any experience with the existing multipath capabilities
of QUICv1. I definitely believe that we ought to wait for experience with
QUICv1.

Note that the problems MPTCP has solved for TCP can be quite different from
the problems it will need to solve with QUIC. For one, I agree with Lucas
that not talking about streams is like not talking about ordered delivery
with QUIC. Streams are QUIC's primary API model, and we need to understand
how an endpoint does streams across paths. This isn't an MPTCP problem, but
it is one we will need to solve when talking about QUIC. And I like to
remind myself that we spend time in working groups on engineering and
bit-coloring, and I suspect that there's plenty of that in just this one
single problem.

It's a fallacy to think that "this won't take too much time". While that is
never a good answer to why we should work on anything, it rings especially
hollow in the context of our experience building connection migration in
QUIC: it took us a couple of years and we kept stumbling and fixing things,
especially security vulnerabilities.

Because I like to hammer the point in: let's motivate the problem first and
agree on the shape of the problem and its importance. And let's see if we
agree to prioritize it/them over the other things that we are already
planning to work on (datagrams, version negotiation, deploying QUICv1 and
digesting experience.)

And I'll note this because the point has been raised. I agree with
Christian that _if we were to build "general-purpose multipath", I am
thinking of a far simpler (and potentially incremental) design -- maybe
using a single PN space...

... but trying to solve the protocol problem is putting the cart before the
horse. We are all good at protocol design, and we are all great at coloring
each bit just the right shade, even though we are terrible at agreeing on
what the right shade is. Before we dive into coloring the bits, let's
please adequately motivate spending engineering time and resources doing it.

- jana

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 6 Oct 2020 05:45:34 -0500

Hi, Jana,

> being very charitable, but the title of this email thread isn't helping.

Thanks for your helpful thoughts below.

Since the title of this thread was mine, it's fair to say that what I was
hoping for, was to identify the reasons people were thinking that dropping
the multipath charter deliverable was the right thing to do.

https://mailarchive.ietf.org/arch/msg/quic/cjjI9-uP6WXVI8p28pcTcq8vS-E/

Obviously, we moved far beyond what I was hoping for, without changing the
subject line ...

Best,

Spencer

> So echoing what others have been saying on this thread: What is the
> problem we are trying to solve? Do we agree on the importance of it? Is
> this bandwidth aggregation across WiFi/Cellular (is that an important
> enough use case?)? Is it latency reduction for HTTP? Is it increased
> reliability? In what ways is it more than connection migration?
>
> Note that the problems MPTCP has solved for TCP can be quite different
> from the problems it will need to solve with QUIC. For one, I agree with
> Lucas that not talking about streams is like not talking about ordered
> delivery with QUIC. Streams are QUIC's primary API model, and we need to
> understand how an endpoint does streams across paths. This isn't an MPTCP
> problem, but it is one we will need to solve when talking about QUIC. And I
> like to remind myself that we spend time in working groups on engineering
> and bit-coloring, and I suspect that there's plenty of that in just this
> one single problem.
>
> It's a fallacy to think that "this won't take too much time". While that
> is never a good answer to why we should work on anything, it rings
> especially hollow in the context of our experience building connection
> migration in QUIC: it took us a couple of years and we kept stumbling and
> fixing things, especially security vulnerabilities.
>
> Because I like to hammer the point in: let's motivate the problem first
> and agree on the shape of the problem and its importance. And let's see if
> we agree to prioritize it/them over the other things that we are already
> ... but trying to solve the protocol problem is putting the cart before
> the horse. We are all good at protocol design, and we are all great at
> coloring each bit just the right shade, even though we are terrible at
> agreeing on what the right shade is. Before we dive into coloring the bits,
> let's please adequately motivate spending engineering time and resources
> doing it.
>
> - jana
>

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 6 Oct 2020 14:35:53 +0100

> see in this discussion, but I am still stuck at the premise. Let me explain.

This is a clear and concise articulation of one of my concerns, thanks. To
give an example, bandwidth aggregation is literally throwing more bandwidth
at a problem. Other extensions might also help improve the goodput of
connections. My spin off thread about composability was in part trying to
shed light on single-path QUIC extensions that may mitigate some of the
problems clients have; where do those fit in?

Cheers
Lucas

---

From: Tommy Pauly <[email-masked]>
Date: Tue, 06 Oct 2020 12:17:35 -0700

> On Oct 4, 2020, at 7:56 AM, Ian Swett <ianswett=[email-masked]> wrote:
> 
> Is the lack of MP-QUIC likely to cause QUIC to be disabled for certain key use cases(ie: Siri and Apple Music), or can existing connection migration be made to work?

Since multipath support has proved to be a key part of optimizing performance for services like Siri and Apple Music, we wouldn’t plan to move Siri or Apple Music off of MPTCP onto QUIC until QUIC has support for “full” multipath. (This of course doesn’t affect adopting QUIC for applications that don’t use MPTCP to start with.)

Thanks,
Tommy

> 
> For example, I can imagine working around true multipath by only using one path at once, but sending PATH_CHALLENGE on the other path to probe reliability and/or RTT.  Or even sending identical packets(ie: a request) on both paths and the receiver will naturally respond on the path the packet is received first and drop the other packet as a duplicate.
> 
> My key takeaway from the multipath scheduler papers I've read is that most of the performance gains are obtained by always sending on the faster of the two available paths, and in a large set of circumstances, nothing should be sent on the slower path.
> 
> Thanks, Ian
>