Subject: My BoF report: multipath

From: "Martin Thomson" <[email-masked]>
Date: Fri, 23 Oct 2020 11:05:28 +1100

(I put a variation of this comment in the meeting and in slack, but I wanted to expand on it some.  Sorry, but this got long.  Four hours is not enough sleep.)

Multipath seems pretty clearly useful for certain cases.  I think that the meeting today answered at least the first two of the BoF questions I posed earlier on the list.  So if we are to regard this as a BoF, it meet its goals (thanks chairs).  There is some uncertainty about the first question about having a clear problem to solve, but I am of the view that we could muddle through with some combination of either ignoring our differences or working around them.  The third question regarding constituency is where I didn't find a satisfactory answer.  I want to be clear though, this is no fault of the proponents.  At the current time, I am convinced that formally starting work on multipath would be unwise.

Multipath aims to improve performance either through latency, robustness, or throughput.  Application awareness and involvement in scheduling seemed to be the key factor that enables finding the optimal usage pattern or scheduling algorithm that allow multipath to deliver on those goals.  Applications and users are in the best place to balance goals against other factors like cost or whatever else matters most.  (For reference, I recall the same point being made by Roberto and Christian most clearly, but several others made the same point.)  Christoph did a good job of showing how this applies to very specific use cases, and I thought I saw that in the Alibaba presentation also, but we didn't quite get enough time to get the necessary detail in either presentation.  One potential advantage in this regard is that QUIC implementations are often closer to applications, so they might be in a good position to integrate better.

However, many of the cases that were presented were exactly the sorts of opaque intermediation that is almost the antithesis of that ideal.  Similarly, David's assertion that multipath is orthogonal to MASQUE is reliant on the assumption that application involvement is not that important.  In these cases, it's not clear that using multipath is strictly good.  

I should unpack that a little.  For those people who are making scheduling decisions outside of the endpoint (possible examples being the satellite case and the 3GPP case), it's not clear that this is anything endpoints can prevent.  An endpoint probably can't stop a network provider from using ECMP either.  Similarly, it is not clear how an application endpoint could be aware of these decisions at a level that would allow them to understand and adapt to this treatment.  The result is that these cases have a far more ambiguous value proposition.  Improvements come with trade-offs: for instance, the application might get better throughput, but it comes at a cost to latency.  So I conclude that while these intermediary-based designs might provide an aggregate gain, they will probably not realize the full performance gains that come from end-to-end awareness and control.

For IETF insiders, see also the BANANA or LOOPS BoFs which were strictly network-based analogues of these.  Many of the same concerns that caused those BoFs to fail apply to those use cases.

Maybe we accept the application of the protocol to these questionable ends as acceptable collateral if we are able to deploy at the endpoints.  Maybe we allow intermediaries to seek marginal improvements, but try to ensure that we have a clear path to deploying something better in the long term.  But there is a risk that deployment in the network could interact poorly with more-ideal end-to-end solutions and even prevent those deployments.

These are systems-level questions that are large in scope and subtle in their effect.  I think that it will require considerable energy to resolve them.  Or, as seems more likely in my experience, it will take more time and effort to design a protocol where there are fundamental disagreements about the nature of the deployment models.

However, this isn't the only factor.  We are not deciding on the merits and value of multipath in a vacuum.  It was pretty clear that multipath has potential, at least in principle, or in certain cases.  I'm also mostly convinced now that we could produce a design.  There's some uncertainty, but it seems like we could tolerate that.  QUIC definitely wasn't a sure thing when we started out, I can't expect any large effort to be risk free.

So, with some uncertainty about uses cases, I might still conclude that we have satisfactory answers to the first two BoF questions.  My concern here is about the third: constituency.

What I think is most important at this point is understanding if this protocol will remain a single, coherent thing.  That we can keep building on the "synergies" that Spencer referred to.  No matter the technical merits of the protocol (it's great! probably!) that synergy is probably the most important feature that this working group has delivered with QUIC.  The details of the protocol matter less than the fact that we have a group of people committed to building and maintaining that protocol.  This working group needs to be the venue where work happens so that this community can continue to build on this success.

So for multipath, if we take it on, I'd only like to do so if I was convinced that a non-trivial proportion of the active deployments are committed to working on it and deploying the new extension or version.  That is, that this community wants to do the work.  I see no evidence of that yet, which is why I will claim that this fails to satisfactorily answer that third BoF question.

It is very easy for a splinter group to define a new version of QUIC that does anything.  draft-deconnick or draft-huitema could be the basis of that sort of effort and that could result in the definition of QUIC 84 or 0x0219c81 or whatever.  Call it QUICv2 if you really want.  But if that protocol is only used in certain narrow contexts, then it doesn't produce any of those synergies.  On the contrary, it works to undermine them, so I would prefer to avoid that.

So rather than ask whether multipath is doable, I think we need to instead decide what the QUIC working group - the group that built the core protocol - is doing next for that core protocol and the deployments that depend on it.  Personally, I don't think that we're ready for another large project.  We need deployment experience with the protocol.  We also need to go in and backfill those pieces of QUIC we need for the next thing, like version negotiation.  For me, that's more than enough.

I've now seen a lot of enthusiasm for the idea of multipath.  There were some great presentations with convincing use cases.  There might be too much diversity in use cases or a schism in approaches, but we probably could, with sufficient energy, overcome that.  However, I have to conclude that this is not a good time for starting that work.

I realize that this is likely unsatisfactory to those who want multipath.  I also recognize that deferring work when there is such clear demand could result in that demand manifesting in a bunch of non-interoperable protocols.  Those are risks that we each have to assess for ourselves.

This will change over time.  I don't know how long it will take.  But it's not now.

Thanks for reading this far,
Martin

---

From: Christian Huitema <[email-masked]>
Date: Thu, 22 Oct 2020 18:15:19 -0700

That's a great summary! Thanks, Martin.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 22 Oct 2020 22:17:57 -0500

Hi, Martin,

Thanks for this. I have a question and a couple of comments.

On Thu, Oct 22, 2020, 19:06 Martin Thomson <[email-masked]> wrote:

> (I put a variation of this comment in the meeting and in slack, but I
> wanted to expand on it some.  Sorry, but this got long.  Four hours is not
> enough sleep.)
>
> Multipath seems pretty clearly useful for certain cases.  I think that the
> meeting today answered at least the first two of the BoF questions I posed
> earlier on the list.  So if we are to regard this as a BoF, it meet its
> goals (thanks chairs).  There is some uncertainty about the first question
> about having a clear problem to solve, but I am of the view that we could
> muddle through with some combination of either ignoring our differences or
> working around them.  The third question regarding constituency is where I
> didn't find a satisfactory answer.  I want to be clear though, this is no
> fault of the proponents.  At the current time, I am convinced that formally
> starting work on multipath would be unwise.
>

I've been assuming that targeting Experimental would allow people who care
about multipath to work on it as part of the QUIC community without
derailing needed standards-track work, rather than working in isolation or
in a splinter group. At a minimum, the chairs could give priority to needed
standards-track work when that's required to make progress.

Does that make sense?

Multipath aims to improve performance either through latency, robustness,
> or throughput.  Application awareness and involvement in scheduling seemed
> to be the key factor that enables finding the optimal usage pattern or
> scheduling algorithm that allow multipath to deliver on those goals.
> Applications and users are in the best place to balance goals against other
> factors like cost or whatever else matters most.  (For reference, I recall
> the same point being made by Roberto and Christian most clearly, but
> several others made the same point.)  Christoph did a good job of showing
> how this applies to very specific use cases, and I thought I saw that in
> the Alibaba presentation also, but we didn't quite get enough time to get
> the necessary detail in either presentation.  One potential advantage in
> this regard is that QUIC implementations are often closer to applications,
> so they might be in a good position to integrate better.
>

One of the things I came away with today was an appreciation for at least
two kinds of applications - the kind that can do better if they handle
multiple paths themselves because the details of the application matters so
much, and the kind that don't care as much about - if you're using multiple
paths simultaneously just to make use of all your bandwidth, that's a lot
easier to delegate to a general purpose transport mechanism.

However, many of the cases that were presented were exactly the sorts of
> opaque intermediation that is almost the antithesis of that ideal.
> Similarly, David's assertion that multipath is orthogonal to MASQUE is
> reliant on the assumption that application involvement is not that
> important.  In these cases, it's not clear that using multipath is strictly
> good.
>
> I should unpack that a little.  For those people who are making scheduling
> decisions outside of the endpoint (possible examples being the satellite
> case and the 3GPP case), it's not clear that this is anything endpoints can
> prevent.  An endpoint probably can't stop a network provider from using
> ECMP either.  Similarly, it is not clear how an application endpoint could
> be aware of these decisions at a level that would allow them to understand
> and adapt to this treatment.  The result is that these cases have a far
> more ambiguous value proposition.  Improvements come with trade-offs: for
> instance, the application might get better throughput, but it comes at a
> cost to latency.  So I conclude that while these intermediary-based designs
> might provide an aggregate gain, they will probably not realize the full
> performance gains that come from end-to-end awareness and control.
>
> For IETF insiders, see also the BANANA or LOOPS BoFs which were strictly
> network-based analogues of these.  Many of the same concerns that caused
> those BoFs to fail apply to those use cases.
>
> Maybe we accept the application of the protocol to these questionable ends
> as acceptable collateral if we are able to deploy at the endpoints.  Maybe
> we allow intermediaries to seek marginal improvements, but try to ensure
> that we have a clear path to deploying something better in the long term.
> But there is a risk that deployment in the network could interact poorly
> with more-ideal end-to-end solutions and even prevent those deployments.
>

ISTM that this is extremely helpful observation. The presentation I gave
talked about short-term synergy (can't do multipath QUIC without a QUIC
stack, right?), but you're raising a really important question about opt-in
and bypass for intermediaries, that people who care about intermediaries
should be thinking about.

At a minimum, transparent interception for QUIC intermediaries will be a
lot harder than it was for TCP intermediaries ...

These are systems-level questions that are large in scope and subtle in
> their effect.  I think that it will require considerable energy to resolve
> them.  Or, as seems more likely in my experience, it will take more time
> and effort to design a protocol where there are fundamental disagreements
> about the nature of the deployment models.
>
> However, this isn't the only factor.  We are not deciding on the merits
> and value of multipath in a vacuum.  It was pretty clear that multipath has
> potential, at least in principle, or in certain cases.  I'm also mostly
> convinced now that we could produce a design.  There's some uncertainty,
> but it seems like we could tolerate that.  QUIC definitely wasn't a sure
> thing when we started out, I can't expect any large effort to be risk free.
>
> So, with some uncertainty about uses cases, I might still conclude that we
> have satisfactory answers to the first two BoF questions.  My concern here
> is about the third: constituency.
>
> What I think is most important at this point is understanding if this
> protocol will remain a single, coherent thing.  That we can keep building
> on the "synergies" that Spencer referred to.  No matter the technical
> merits of the protocol (it's great! probably!) that synergy is probably the
> most important feature that this working group has delivered with QUIC.
> The details of the protocol matter less than the fact that we have a group
> of people committed to building and maintaining that protocol.  This
> working group needs to be the venue where work happens so that this
> community can continue to build on this success.
>
> So for multipath, if we take it on, I'd only like to do so if I was
> convinced that a non-trivial proportion of the active deployments are
> committed to working on it and deploying the new extension or version.
> That is, that this community wants to do the work.  I see no evidence of
> that yet, which is why I will claim that this fails to satisfactorily
> answer that third BoF question.
>
> It is very easy for a splinter group to define a new version of QUIC that
> does anything.  draft-deconnick or draft-huitema could be the basis of that
> sort of effort and that could result in the definition of QUIC 84 or
> 0x0219c81 or whatever.  Call it QUICv2 if you really want.  But if that
> protocol is only used in certain narrow contexts, then it doesn't produce
> any of those synergies.  On the contrary, it works to undermine them, so I
> would prefer to avoid that.
>
> So rather than ask whether multipath is doable, I think we need to instead
> decide what the QUIC working group - the group that built the core protocol
> - is doing next for that core protocol and the deployments that depend on
> it.  Personally, I don't think that we're ready for another large project.
> We need deployment experience with the protocol.  We also need to go in and
> backfill those pieces of QUIC we need for the next thing, like version
> negotiation.  For me, that's more than enough.
>
> I've now seen a lot of enthusiasm for the idea of multipath.  There were
> some great presentations with convincing use cases.  There might be too
> much diversity in use cases or a schism in approaches, but we probably
> could, with sufficient energy, overcome that.  However, I have to conclude
> that this is not a good time for starting that work.
>
> I realize that this is likely unsatisfactory to those who want multipath.
> I also recognize that deferring work when there is such clear demand could
> result in that demand manifesting in a bunch of non-interoperable
> protocols.  Those are risks that we each have to assess for ourselves.
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 22 Oct 2020 22:08:02 -0700

On 10/22/2020 8:17 PM, Spencer Dawkins at IETF wrote:
> On Thu, Oct 22, 2020, 19:06 Martin Thomson <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     (I put a variation of this comment in the meeting and in slack,
>     but I wanted to expand on it some.  Sorry, but this got long. 
>     Four hours is not enough sleep.)
>
>     Multipath seems pretty clearly useful for certain cases.  I think
>     that the meeting today answered at least the first two of the BoF
>     questions I posed earlier on the list.  So if we are to regard
>     this as a BoF, it meet its goals (thanks chairs).  There is some
>     uncertainty about the first question about having a clear problem
>     to solve, but I am of the view that we could muddle through with
>     some combination of either ignoring our differences or working
>     around them.  The third question regarding constituency is where I
>     didn't find a satisfactory answer.  I want to be clear though,
>     this is no fault of the proponents.  At the current time, I am
>     convinced that formally starting work on multipath would be unwise.
>
>
> I've been assuming that targeting Experimental would allow people who
> care about multipath to work on it as part of the QUIC community
> without derailing needed standards-track work, rather than working in
> isolation or in a splinter group. At a minimum, the chairs could give
> priority to needed standards-track work when that's required to make
> progress.
>
> Does that make sense?

As Martin writes later in his report, anyone can use the extensibility
mechanisms of QUIC and design a multipath extension. I would expect to
see several of those. If applications really demand that, then some of
those extensions will be deployed, experience will be gained, etc. At
that point it might be time to pick draft a standard proposal based on
these prototype deployments.

>
>     Multipath aims to improve performance either through latency,
>     robustness, or throughput.  Application awareness and involvement
>     in scheduling seemed to be the key factor that enables finding the
>     optimal usage pattern or scheduling algorithm that allow multipath
>     to deliver on those goals.  Applications and users are in the best
>     place to balance goals against other factors like cost or whatever
>     else matters most.  (For reference, I recall the same point being
>     made by Roberto and Christian most clearly, but several others
>     made the same point.)  Christoph did a good job of showing how
>     this applies to very specific use cases, and I thought I saw that
>     in the Alibaba presentation also, but we didn't quite get enough
>     time to get the necessary detail in either presentation.  One
>     potential advantage in this regard is that QUIC implementations
>     are often closer to applications, so they might be in a good
>     position to integrate better.
>
>
> One of the things I came away with today was an appreciation for at
> least two kinds of applications - the kind that can do better if they
> handle multiple paths themselves because the details of the
> application matters so much, and the kind that don't care as much
> about - if you're using multiple paths simultaneously just to make use
> of all your bandwidth, that's a lot easier to delegate to a general
> purpose transport mechanism.

There are very few applications that "use multiple paths simultaneously
just to make use of all your bandwidth". Christopher told us for example
how Siri and Apple Music use very different strategies, one to minimize
latency in a low bandwidth application, the other to minimize use of the
expensive cellular link unless that's required to avoid stalls.
Multipath can easily go to "win some, lose some" trade-offs, an in
particular trade-offs between bandwidth and latency, or bandwidth an
cost of operation.

>
>     However, many of the cases that were presented were exactly the
>     sorts of opaque intermediation that is almost the antithesis of
>     that ideal.  Similarly, David's assertion that multipath is
>     orthogonal to MASQUE is reliant on the assumption that application
>     involvement is not that important.  In these cases, it's not clear
>     that using multipath is strictly good. 
>
>     I should unpack that a little.  For those people who are making
>     scheduling decisions outside of the endpoint (possible examples
>     being the satellite case and the 3GPP case), it's not clear that
>     this is anything endpoints can prevent.  An endpoint probably
>     can't stop a network provider from using ECMP either.  Similarly,
>     it is not clear how an application endpoint could be aware of
>     these decisions at a level that would allow them to understand and
>     adapt to this treatment.  The result is that these cases have a
>     far more ambiguous value proposition.  Improvements come with
>     trade-offs: for instance, the application might get better
>     throughput, but it comes at a cost to latency.  So I conclude that
>     while these intermediary-based designs might provide an aggregate
>     gain, they will probably not realize the full performance gains
>     that come from end-to-end awareness and control.
>
>     For IETF insiders, see also the BANANA or LOOPS BoFs which were
>     strictly network-based analogues of these.  Many of the same
>     concerns that caused those BoFs to fail apply to those use cases.
>
>     Maybe we accept the application of the protocol to these
>     questionable ends as acceptable collateral if we are able to
>     deploy at the endpoints.  Maybe we allow intermediaries to seek
>     marginal improvements, but try to ensure that we have a clear path
>     to deploying something better in the long term.  But there is a
>     risk that deployment in the network could interact poorly with
>     more-ideal end-to-end solutions and even prevent those deployments.
>
>
> ISTM that this is extremely helpful observation. The presentation I
> gave talked about short-term synergy (can't do multipath QUIC without
> a QUIC stack, right?), but you're raising a really important question
> about opt-in and bypass for intermediaries, that people who care about
> intermediaries should be thinking about. 
>
> At a minimum, transparent interception for QUIC intermediaries will be
> a lot harder than it was for TCP intermediaries ...

One of the priorities in QUIC development was to restore end-to-end
sanity by getting rid of middle-box interference. ATSSS appears to be
yet another design of middle boxes, so you can imagine that there is
very little enthusiasm for that scenario.

>
>     These are systems-level questions that are large in scope and
>     subtle in their effect.  I think that it will require considerable
>     energy to resolve them.  Or, as seems more likely in my
>     experience, it will take more time and effort to design a protocol
>     where there are fundamental disagreements about the nature of the
>     deployment models.
>
>     However, this isn't the only factor.  We are not deciding on the
>     merits and value of multipath in a vacuum.  It was pretty clear
>     that multipath has potential, at least in principle, or in certain
>     cases.  I'm also mostly convinced now that we could produce a
>     design.  There's some uncertainty, but it seems like we could
>     tolerate that.  QUIC definitely wasn't a sure thing when we
>     started out, I can't expect any large effort to be risk free.
>
>     So, with some uncertainty about uses cases, I might still conclude
>     that we have satisfactory answers to the first two BoF questions. 
>     My concern here is about the third: constituency.
>
>     What I think is most important at this point is understanding if
>     this protocol will remain a single, coherent thing.  That we can
>     keep building on the "synergies" that Spencer referred to.  No
>     matter the technical merits of the protocol (it's great!
>     probably!) that synergy is probably the most important feature
>     that this working group has delivered with QUIC.  The details of
>     the protocol matter less than the fact that we have a group of
>     people committed to building and maintaining that protocol.  This
>     working group needs to be the venue where work happens so that
>     this community can continue to build on this success.
>
>     So for multipath, if we take it on, I'd only like to do so if I
>     was convinced that a non-trivial proportion of the active
>     deployments are committed to working on it and deploying the new
>     extension or version.  That is, that this community wants to do
>     the work.  I see no evidence of that yet, which is why I will
>     claim that this fails to satisfactorily answer that third BoF
>     question.
>
>     It is very easy for a splinter group to define a new version of
>     QUIC that does anything.  draft-deconnick or draft-huitema could
>     be the basis of that sort of effort and that could result in the
>     definition of QUIC 84 or 0x0219c81 or whatever.  Call it QUICv2 if
>     you really want.  But if that protocol is only used in certain
>     narrow contexts, then it doesn't produce any of those synergies. 
>     On the contrary, it works to undermine them, so I would prefer to
>     avoid that.
>
>     So rather than ask whether multipath is doable, I think we need to
>     instead decide what the QUIC working group - the group that built
>     the core protocol - is doing next for that core protocol and the
>     deployments that depend on it.  Personally, I don't think that
>     we're ready for another large project.  We need deployment
>     experience with the protocol.  We also need to go in and backfill
>     those pieces of QUIC we need for the next thing, like version
We had a discussion recently on Slack, comparing the size of the early
prototype implementations, maybe 3,000 LOC, and the size of the current
implementations, often around 30,000 LOC. We discovered and solved a lot
of complex issues during the standardization process, and the increased
code size is an indicator of that additional complexity. Every
additional feature interacts with other features, potentially in a
multiplicative ways. Add multipath, and expect the code size to grow
significantly yet again. Hence the call for caution.

>
>     I've now seen a lot of enthusiasm for the idea of multipath. 
>     There were some great presentations with convincing use cases. 
>     There might be too much diversity in use cases or a schism in
>     approaches, but we probably could, with sufficient energy,
>     overcome that.  However, I have to conclude that this is not a
>     good time for starting that work.
>
>     I realize that this is likely unsatisfactory to those who want
>     multipath.  I also recognize that deferring work when there is
>     such clear demand could result in that demand manifesting in a
>     bunch of non-interoperable protocols.  Those are risks that we
>     each have to assess for ourselves.
>
Yes. I don't see how we can have a generic multipath solution now, not
without having some clear consensus from application developers about
the features that they want.

-- Christian Huitema

---

From: "Flinck, Hannu (Nokia - FI/Espoo)" <[email-masked]>
Date: Fri, 23 Oct 2020 08:04:53 +0000

Hello Martin r

You wrote: “.  There might be too much diversity in use cases or a schism in approaches, but we probably could, with sufficient energy, overcome that.  However, I have to conclude that this is not a good time for starting that work.”

Of course there are multiple use cases. If there was only one or two, one would argue that the application space is too narrow.
But ALL these use cases are about two things: 1) load balancing across two or more interfaces and 2) scheduling based on latency.

If you can do those you can implement the “diverse” use cases.

Best regards
Hannu

From: QUIC <[email-masked]> On Behalf Of Christian Huitema
Sent: Friday, October 23, 2020 8:08 AM
To: Spencer Dawkins at IETF <[email-masked]>; Martin Thomson <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: My BoF report: multipath

Thanks for this. I have a question and a couple of comments.
On Thu, Oct 22, 2020, 19:06 Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
(I put a variation of this comment in the meeting and in slack, but I wanted to expand on it some.  Sorry, but this got long.  Four hours is not enough sleep.)

I've been assuming that targeting Experimental would allow people who care about multipath to work on it as part of the QUIC community without derailing needed standards-track work, rather than working in isolation or in a splinter group. At a minimum, the chairs could give priority to needed standards-track work when that's required to make progress.

Does that make sense?

As Martin writes later in his report, anyone can use the extensibility mechanisms of QUIC and design a multipath extension. I would expect to see several of those. If applications really demand that, then some of those extensions will be deployed, experience will be gained, etc. At that point it might be time to pick draft a standard proposal based on these prototype deployments.

One of the things I came away with today was an appreciation for at least two kinds of applications - the kind that can do better if they handle multiple paths themselves because the details of the application matters so much, and the kind that don't care as much about - if you're using multiple paths simultaneously just to make use of all your bandwidth, that's a lot easier to delegate to a general purpose transport mechanism.

There are very few applications that "use multiple paths simultaneously just to make use of all your bandwidth". Christopher told us for example how Siri and Apple Music use very different strategies, one to minimize latency in a low bandwidth application, the other to minimize use of the expensive cellular link unless that's required to avoid stalls. Multipath can easily go to "win some, lose some" trade-offs, an in particular trade-offs between bandwidth and latency, or bandwidth an cost of operation.

ISTM that this is extremely helpful observation. The presentation I gave talked about short-term synergy (can't do multipath QUIC without a QUIC stack, right?), but you're raising a really important question about opt-in and bypass for intermediaries, that people who care about intermediaries should be thinking about.

At a minimum, transparent interception for QUIC intermediaries will be a lot harder than it was for TCP intermediaries ...

One of the priorities in QUIC development was to restore end-to-end sanity by getting rid of middle-box interference. ATSSS appears to be yet another design of middle boxes, so you can imagine that there is very little enthusiasm for that scenario.

We had a discussion recently on Slack, comparing the size of the early prototype implementations, maybe 3,000 LOC, and the size of the current implementations, often around 30,000 LOC. We discovered and solved a lot of complex issues during the standardization process, and the increased code size is an indicator of that additional complexity. Every additional feature interacts with other features, potentially in a multiplicative ways. Add multipath, and expect the code size to grow significantly yet again. Hence the call for caution.

Yes. I don't see how we can have a generic multipath solution now, not without having some clear consensus from application developers about the features that they want.

-- Christian Huitema

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 23 Oct 2020 10:42:41 +0200

Martin,

> (I put a variation of this comment in the meeting and in slack, but I wanted to expand on it some.  Sorry, but this got long.  Four hours is not enough sleep.)

Thanks expressing your viewpoint as a sumary. As suggested by the 
chairs, I continue the discussion on the mailing list and not on slack.

> Multipath aims to improve performance either through latency, robustness, or throughput.  Application awareness and involvement in scheduling seemed to be the key factor that enables finding the op…

There are two different things that must be considered in a multipath 
transport protocol:

  - the information that the two peers exchange
  - the algorithms that the two peers use

The MPTCP spec defines how the TCP options are use to carry information 
between peer and how the different subflows can be managed. It does not 
define how a path manager works and how a packet scheduler works, this 
level of specification is not required for interoperability.

Since the publication of RFC6824, different use cases have been deployed 
with MPTCP. As indicated by Christoph, those did not require any change 
to the protocol, only tuning the packet schedulers and path manager to 
better match the deployed use case.

I believe that the same distinction should apply to a multipath variant 
of QUIC. Compared to TCP, QUIC has a lot of protocol features that make 
it easy to add multipath capabilities and we are very close to having a 
multipath capable transport.

Concerning the algorithms (packet scheduler and path managers), these 
need to implement some kind of business logic related to the relative 
cost of the different links that are use. In MPTCP, we were constrained 
by the kernel/userspace boundary that made it more difficult from a 
systems viewpoint to allow the applications to easily tune the kernel 
mechanisms, but several solutions were found. Given that QUIC runs in 
userspace and is linked in the application, it becomes easier to allow 
the application to provide a set of functions that implement the 
application's business logic in a flexible manner.

> However, many of the cases that were presented were exactly the sorts of opaque intermediation that is almost the antithesis of that ideal.  

The fundamental reason why there is intermediation is that the current 
Internet architecture is not multipath capable. With widely deployed 
multipath transport, users of devices that need to aggregate the 
bandwidth of different paths to get the performance they need would be 
able obtain them without having to rely on various types of middleboxes 
that are deployed to overcome this architectural limitation.

If QUIC gets widely deployed beyond the current web-centric use case, 
there is an opportunity to have a transport protocol that can address 
all these problems in a clean and end-to-end manner.

Otherwise, all the Internet users that need to combine several Internet 
access links for performance (throughput, reliablity, ...) reasons will 
need to continue to rely on proprietary protocols or middleboxes that 
introduce their own problems to be able to fully use their network.

> Similarly, David's assertion that multipath is orthogonal to MASQUE is reliant on the assumption that application involvement is not that important.  In these cases, it's not clear that using multipath is strictly good.
> 

During the meeting, David complained about the fact that ATSSS what not 
beneficial for endusers. This is not totally true. Endusers want to be 
connected to the Internet using their mobile devices. This requires the 
deployment of mobile networks. One approach is to deploy a single mobile 
network (say 5G) and always attach the enduser's devices to this 
network. That's the best approach for single path protocols like TCP, 
UDP and QUIC. Another approach is to recognise that a mobile network 
like 5G can be combined with existing WiFi networks that are also widely 
deployed. Using techniques that allow mobile devices to roam over 
different WiFi networks makes it possible for them to use 5G and/or WiFi 
even while moving. This is what Apple does with Apple Music and Siri. 
Users do not understand that a solution that works with some application 
cannot work with all of them. This is the motivation for the deployment 
of ATSSS by network operators.

> For IETF insiders, see also the BANANA or LOOPS BoFs which were strictly network-based analogues of these.  Many of the same concerns that caused those BoFs to fail apply to those use cases.

If we had deployed multipath transport, BANANA and the hybrid access 
networks would never had to be discussed. The existence of these BoFs is 
a result of the architectural limitations of our current protocols.

> Maybe we accept the application of the protocol to these questionable ends as acceptable collateral if we are able to deploy at the endpoints.  Maybe we allow intermediaries to seek marginal improv…

In network deployments depend on business needs. There is a clear 
business need for combining different types of access networks. With 
Multipath QUIC it would become possible to use them on en end-to-end 
basis without any intermediary. If we stick to single path transport, 
then we will force the development and the deployment of new 
intermediaries...

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 23 Oct 2020 10:27:37 +0100

Posting as an individual.

Thanks to the time that people took to prepare, present and discuss, I
gained a better understanding of this area.

One main takeaway for me is that thinking of QUIC v1 as singlepath puts it
in an unfair box. Others, such as Jana, articulated this point better than
I.

QUIC is path independent, path aware and provides mechanisms, written in
the core document, for endpoints to interact  and interoperate about
path-related things. Through the course of specification development things
might have gone different, and we might be having a conversion now about
whether the group should adopt a document that describes for instance just
connection migration.

But migration is in the core and, based on the Slack discussion following
the interim, there appear to be several parties that have expressed an
interest in testing deployments of Connection migration. This can be seen
as some form of success.

The use cases indicate that things like active-active paths or bandwidth
aggregation are desirable. But I was unable to discern objectively how more
of an optimal experience they would provide over a well tuned QUIC v1
endpoint that use connection migration.

I encourage people to think about the charter goals for the QUIC protocol,
what success looks like, and whether they think connection migration
delivers a sufficiently good part of the multipath feature set that was
vaguely described when we started off.

Cheers
Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 23 Oct 2020 06:55:47 -0500

> interest in testing deployments of Connection migration. This can be seen

Could you please add me to the slack channel? I'm assuming that is still a
manual process.

Thanks,

Spencer

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 23 Oct 2020 10:15:02 -0500

Hi Lucas,

> endpoint that use connection migration.

Coming from IP layer, I have a message to QUIC enthusiasts:

Connection migration is a solution for mobility at transport layer.
Ideally, it should be IP layer. If a node moves, its connections normally
break. So connection migration of QUIC solves that issue at the transport
layer.

OTOH multipath QUIC is a solution for multiple interfaces. If a node can be
connected to  the Internet over more than one interface using them
simultaneously provides several advantages like increasing bandwidth, etc.

 Behcet

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 23 Oct 2020 17:34:11 +0100

Hi Behcet

On Fri, Oct 23, 2020 at 4:15 PM Behcet Sarikaya <[email-masked]>
wrote:

>> One main takeaway for me is that thinking of QUIC v1 as singlepath puts
>> it in an unfair box. Others, such as Jana, articulated this point better
>> than I.
>>
> OTOH multipath QUIC is a solution for multiple interfaces. If a node can
> be connected to  the Internet over more than one interface using them
As an individual. I'll repeat my comment, I've seen little evidence that
some design that enables bandwidth aggregation in the QUIC transport over
multiple paths offers advantages over a well-tuned QUIC stack that delivers
over a single path. If the benefits of bandwidth are so huge, I might
expect that people would be clambering to add this quickly with multiple
QUIC connections coordinated in the application layer. That would allow
deployment and testing without needing to wait for the transport to be
redesigned.

Let's not forget the disadvantages that bundling bandwidth can come with
too. These could be perfunctory or monetary.

Cheers
Lucas

---

From: Roland Zink <[email-masked]>
Date: Fri, 23 Oct 2020 21:34:16 +0200

A application works with a single connection most of the time. As 
application developer you probably can't afford to put any effort in 
improving this when it isn't absolute necessary. If you do it on the 
application layer then the application may also get location information 
out of it, like IP addresses or WiFi names, and this is a privacy issue.

Regards,

Roland

Am 23.10.2020 um 18:34 schrieb Lucas Pardue:
> Hi Behcet
>
> On Fri, Oct 23, 2020 at 4:15 PM Behcet Sarikaya 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi Lucas,
>
>     On Fri, Oct 23, 2020 at 4:28 AM Lucas Pardue
>     <[email-masked] <mailto:[email-masked]>>
>     wrote:
>
>         Thanks to the time that people took to prepare, present and
>         discuss, I gained a better understanding of this area.
>
>         One main takeaway for me is that thinking of QUIC v1 as
>         singlepath puts it in an unfair box. Others, such as Jana,
>         articulated this point better than I.
>
>         QUIC is path independent, path aware and provides mechanisms,
>         written in the core document, for endpoints to interact  and
>         interoperate about path-related things. Through the course of
>         specification development things might have gone different,
>         and we might be having a conversion now about whether the
>         group should adopt a document that describes for instance just
>         connection migration.
>
>         But migration is in the core and, based on the Slack
>         discussion following the interim, there appear to be several
>         parties that have expressed an interest in testing deployments
>         of Connection migration. This can be seen as some form of success.
>
>         The use cases indicate that things like active-active paths or
>         bandwidth aggregation are desirable. But I was unable to
>         discern objectively how more of an optimal experience they
>         would provide over a well tuned QUIC v1 endpoint that use
>         connection migration.
>
>
>
>     Connection migration is a solution for mobility at transport
>     layer. Ideally, it should be IP layer. If a node moves, its
>     connections normally break. So connection migration of QUIC solves
>     that issue at the transport layer.
>
>     OTOH multipath QUIC is a solution for multiple interfaces. If a
>     node can be connected to  the Internet over more than one
>     interface using them simultaneously provides several advantages
>     like increasing bandwidth, etc.
>
>
> As an individual. I'll repeat my comment, I've seen little evidence 
> that some design that enables bandwidth aggregation in the QUIC 
> transport over multiple paths offers advantages over a well-tuned QUIC 
> stack that delivers over a single path. If the benefits of bandwidth 
> are so huge, I might expect that people would be clambering to add 
> this quickly with multiple QUIC connections coordinated in the 
> application layer. That would allow deployment and testing without 
> needing to wait for the transport to be redesigned.
>
> Let's not forget the disadvantages that bundling bandwidth can come 
> with too. These could be perfunctory or monetary.
>
> Cheers
> Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 23 Oct 2020 21:31:56 +0100

Hi Roland,

> out of it, like IP addresses or WiFi names, and this is a privacy issue.

This is an interesting thought and I wonder if you might be able to
elaborate some. The userland QUIC libraries I'm familiar with require
applications to do a lot of their own lifting with regards to the network.

I'm particularly interested whether there is any difference say between:

1) An HTTP/3 client application that wants to open two QUIC connections
using a userland QUIC library.

2) An HTTP/3 client application that wants to open one multipath QUIC using
a userland QUIC library.

Is the expectation here that there is some TBD OS-level API would abstract
network layer information from the application in the name of privacy?

Cheers
Lucas

---

From: Roland Zink <[email-masked]>
Date: Fri, 23 Oct 2020 23:37:33 +0200

Hi Lucas,

1)  Just opening a second QUIC connection is not enough as it will most 
likely use the same path as the first. Some API to list interfaces and 
their properties is needed. Then the second second connection can be 
bound to the desired interface. If for example you want to check the 
available WiFi networks in Android then you need location permissions 
and the user needs to grant them and can revoke them anytime. In 
addition you need to write some legal text what you do with this data.

2) Is probably similar to 1) but may use some system library to do the 
job and hide it from the application itself.

I can't speak for others but my expectation would be that some OS-level 
API should abstract network layer information when possible. Still and 
e2e solution will reveal the clients changing external IP addresses on 
the server side. A longer lived multipath connection may be used to 
track the user.

Regards,

Roland

Am 23.10.2020 um 22:31 schrieb Lucas Pardue:
> Hi Roland,
>
> On Fri, Oct 23, 2020 at 8:34 PM Roland Zink <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     application developer you probably can't afford to put any effort
>     in improving this when it isn't absolute necessary. If you do it
>     on the application layer then the application may also get
>     location information out of it, like IP addresses or WiFi names,
>     and this is a privacy issue.
>
>
> 1) An HTTP/3 client application that wants to open two QUIC 
> connections using a userland QUIC library.
>
> 2) An HTTP/3 client application that wants to open one multipath QUIC 
> Is the expectation here that there is some TBD OS-level API would 
> abstract network layer information from the application in the name of 
> privacy?
>
> Cheers
> Lucas
>
>

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 23 Oct 2020 23:05:42 +0100

Hi Roland,

On Fri, Oct 23, 2020 at 10:37 PM Roland Zink <[email-masked]> wrote:

> their properties is needed. Then the second second connection can be bound
> to the desired interface. If for example you want to check the available
> WiFi networks in Android then you need location permissions and the user
> needs to grant them and can revoke them anytime. In addition you need to
> write some legal text what you do with this data.
>
>
> 2) Is probably similar to 1) but may use some system library to do the job
> and hide it from the application itself.
>
>
> API should abstract network layer information when possible. Still and e2e
> solution will reveal the clients changing external IP addresses on the
> server side. A longer lived multipath connection may be used to track the
> user.
>

Thanks for your thoughts.

One of the touted benefits of QUIC is the ability to iterate and experiment
with more freedom. Requiring OS or platform APIs would seem to provide
friction to that. There might be some interesting questions about the
deployment reality for QUIC extensions that need such capabilities,
compared to QUIC extensions that can operate in their own bubble.

Cheers
Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 23 Oct 2020 17:10:34 -0500

> server side. A longer lived multipath connection may be used to track the
I'm probably starting to sound like a broken record, but we chartered TAPS
in 2014 to think about hiding bizarre transport attributes from
applications (among other things) using an abstract application programming
interface, and I note that
https://datatracker.ietf.org/doc/draft-ietf-taps-interface/ is now
targeting Propose Standard, and
https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-5.2.13 is
about multipath.

Has anyone been looking at this? TAPS has been chugging away since before
QUIC was chartered, and certainly didn't want to do anything to slow QUICv1
work down, but now that QUICv1 is past Publication Requested, should we be?

Best,

Spencer

---

From: Christian Huitema <[email-masked]>
Date: Fri, 23 Oct 2020 15:18:59 -0700

On 10/23/2020 3:10 PM, Spencer Dawkins at IETF wrote:
> I'm probably starting to sound like a broken record, but we chartered
> TAPS in 2014 to think about hiding bizarre transport attributes from
> applications (among other things) using an abstract application
> programming interface, and I note
> that https://datatracker.ietf.org/doc/draft-ietf-taps-interface/ is
> now targeting Propose Standard,
> and https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-5.2.13
> is about multipath.

I think you are hearing practical reasons to not do what TAPS is
proposing, and instead to use the close coupling between the application
and the QUIC implementation to make better transport decisions. In
practice, application code is developed for a specific implementation of
QUIC, and that implementation gets ported to all platforms that provide
UDP sockets or equivalent.

> Has anyone been looking at this? TAPS has been chugging away since
> before QUIC was chartered, and certainly didn't want to do anything to
> slow QUICv1 work down, but now that QUICv1 is past Publication
> Requested, should we be?

It's a free world. Of course you can do that. Whether that's useful is
for you to decide.

-- Christian Huitema

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 24 Oct 2020 00:02:41 +0100

Maybe I just lack imagination but this is the part of the "put it in
transport and everyone gets it for free" line that I've not understood.

For example, Cloudflare's quiche implementation of QUIC is I/O agnostic,
you put application data in, it gives you QUIC packets back and you do with
them what you like. This flexibility means it's portable to lots of
platforms and applications. It's been able to slide into curl and nginx
implementations without a need to re-architect. More people using the
library means more diverse use cases and better coverage of protocol
features.

I don't know what a TAPS-quiche integration would look like or provide.
>From looking at
https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-7.1.7 a
lot of the decisions are "implementation specific". I'd hazard a guess that
each implementation would make its own decisions and want to expose
different things to the application. And even if *it* didn't expose network
information, it would need access. So either it sits in the application and
uses a platform API that could've been used all along, or it gets compiled
into the platform and now we lose agility.

Cheers
Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 23 Oct 2020 18:04:21 -0500

Hi, Christian,

On Fri, Oct 23, 2020 at 5:19 PM Christian Huitema <[email-masked]>
wrote:

> > and
> https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-5.2.13
> > is about multipath.
>
>

I'm sorry, I wasn't clear in my e-mail.

I think where we are, is that there are two flavors of applications - the
kind that would do better than a general purpose transport with multiple
paths, and the kind that doesn't need to handle the paths in detail.

I was thinking of the second category. I could usefully have included that
caveat in my note.

> It's a free world. Of course you can do that. Whether that's useful is

Sure. I was mostly looking at the author list for
https://datatracker.ietf.org/doc/draft-ietf-taps-interface/, and wondering
if anyone was already doing that.

Best,

Spencer

---

From: Eric Kinnear <[email-masked]>
Date: Fri, 23 Oct 2020 18:16:22 -0700

I generally agree with some of the thinking here, but I can definitely see a case where “everyone gets it for free” is a thing.
For example, one could imagine that Cloudflare might use quiche to allow its customers to enable HTTP/3 “for free” via a checkbox in their configuration, or even just enable it on their websites.

That’s not necessarily fully in sync with what the “QUIC is in userspace and therefore everyone should bring their own implementation that they can change at will” thinking is going towards, but I do expect there are quite a few folks creating end-user-facing software who don’t know that much about networking and are more interested in “hit this REST endpoint” than “tune my congestion control for the specific quirks of my traffic content that are unique to me and only me”.

I’m not saying anything against that type of tuning, that’s often how we can push the boundaries and really move things forward. 
I also think QUIC is in a really good place to enable a lot of awesome experimentation in that area.

But I do think we can acknowledge that there are folks (even if I personally might tend towards the “let’s experiment and push the boundaries” side of things) who have a very legitimate desire to abstract that away underneath something that generally does a “pretty good” job of handling it. If that's better than what exists today in terms of end-user visible benefits, even if not 100% optimally tuned for their specific traffic patterns, that seems like a good start.

There’s definitely a balance to strike between ultimate experimentation and moving meaningful numbers of people forward who either don’t have time or aren’t interested in that experimentation. Where are the cases where it’s most useful, leads to the most end-user benefit, or enables things not previously possible? Those seem like the things to focus on not abstracting away — is multipath in QUIC one of those? (I genuinely don’t have an answer to that.)

Thanks,
Eric

> On Oct 23, 2020, at 4:02 PM, Lucas Pardue <[email-masked]> wrote:
> 
> Maybe I just lack imagination but this is the part of the "put it in transport and everyone gets it for free" line that I've not understood. 

> 
> For example, Cloudflare's quiche implementation of QUIC is I/O agnostic, you put application data in, it gives you QUIC packets back and you do with them what you like. This flexibility means it's portable to lots of platforms and applications. It's been able to slide into curl and nginx implementations without a need to re-architect. More people using the library means more diverse use cases and better coverage of protocol features.
> 
> I don't know what a TAPS-quiche integration would look like or provide. From looking at https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-7.1.7 <https://tools.ietf.org/html/draft-ietf-taps-interface-09#section-7.1.7> a lot of the decisions are "implementation specific". I'd hazard a guess that each implementation would make its own decisions and want to expose different things to the application. And even if *it* didn't expose network information, it would need access. So either it sits in the application and uses a platform API that could've been used all along, or it gets compiled into the platform and now we lose agility. 
> 
> Cheers
> Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 24 Oct 2020 03:35:39 +0100

Hey Eric,

On Sat, Oct 24, 2020 at 2:16 AM Eric Kinnear <[email-masked]> wrote:

> I generally agree with some of the thinking here, but I can definitely see
> a case where “everyone gets it for free” is a thing.
> For example, one could imagine that Cloudflare might use quiche to allow
> its customers to enable HTTP/3 “for free” via a checkbox in their
> configuration, or even just enable it on their websites.
>
> That’s not necessarily fully in sync with what the “QUIC is in userspace
> and therefore everyone should bring their own implementation that they can
> change at will” thinking is going towards, but I do expect there are quite
> a few folks creating end-user-facing software who don’t know that much
> about networking and are more interested in “hit this REST endpoint” than
> “tune my congestion control for the specific quirks of my traffic content
> that are unique to me and only me”.
>
> I’m not saying anything against that type of tuning, that’s often how we
> can push the boundaries and really move things forward.
> But I do think we can acknowledge that there are folks (even if I
> personally might tend towards the “let’s experiment and push the
> boundaries” side of things) who have a very legitimate desire to abstract
> that away underneath something that generally does a “pretty good” job of
> handling it. If that's better than what exists today in terms of end-user
> visible benefits, even if not 100% optimally tuned for their specific
> traffic patterns, that seems like a good start.
>
> There’s definitely a balance to strike between ultimate experimentation
> and moving meaningful numbers of people forward who either don’t have time
> or aren’t interested in that experimentation. Where are the cases where
> it’s most useful, leads to the most end-user benefit, or enables things not
> previously possible? Those seem like the things to focus on *not* abstracting
> away — is multipath in QUIC one of those? (I genuinely don’t have an answer
> to that.)
>

To clarify a little. The point I was trying to make is that just adding
some support for $multipath frames$ parsing to something like quiche would
not be a complete solution. Something would still have to glue it to a
network. And I'm not sure if there's a consistent network API across
platforms that exposes path information required for something like that to
work. So the application has a lot of work to do to. If they have to build
to the nuances of every platform API, firmware or middleware, that's more
than just free. To build on Christian's earlier point, many projects have
picked their own way to do things, for good or bad.

The QUIC WG has benefited from its ability to openly and agily
interoperate. Many implementations are there for people to just pick up and
start running with. I'm not steadfast in belief that everyone has to use a
user-space library and bundle up everything. There's absolutely a value in
making technology accessible to folks that just want to get on with
business via simpler or stabler means. And both these worlds can exist (and
most definately should interoperate!). But I think it's prescient to
consider the accessibility of new transport features to different parts of
the community.

Cheers
Lucas

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 26 Oct 2020 20:43:04 -0700

Thanks for kicking off this thread, Martin.

Yet again, I want to focus on what is most important for us to work on
right now. Rather than answering the theoretical question of what a
well-rounded transport looks like, we need to focus on the engineering
decision of what problems we need to work on right now in QUIC.

We have multipath in QUIC. It may not be enough, but we agreed in this wg
that this would be a good first cut to have. I believe this satisfies the
charter text as we have it right now. We've just finished building the use
of multipath in QUIC, and I'd like us to get some experience with it before
considering how to expand its scope or how to improve it further.

For those who want to experiment with other ways of using multiple paths:
QUIC has many ways of extending the protocol. All you need to do is to
build, deploy, gain experience, and share it with the rest of this group. I
would love to hear more about such experiments going forward.

In the meantime, let's not forget that we still have important unfinished
business (version negotiation, tooling, load balancer work, potential
optimizations). We're about to see wider deployment of QUICv1 and HTTP/3,
which will be a massive shift in traffic. We will need bandwidth to discuss
issues related to this deployment that are inevitable, and deal with
immediate protocol issues that arise from our deployment experience of all
things QUIC, including the multipath that we will be deploying.

- jana

On Fri, Oct 23, 2020 at 12:34 PM Roland Zink <[email-masked]> wrote:

>>> But migration is in the core and, based on the Slack discussion
>>> following the interim, there appear to be several parties that have
>>> expressed an interest in testing deployments of Connection migration. This
>>> can be seen as some form of success.
>>>

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 28 Oct 2020 12:21:17 -0700

Thanks Lucas! Totally agree with what you’re saying here, was just noting that there we’re broadening the scope of implementation/experimentation possibility from “kernel vendors” to “platform/service vendors/someone who really wants to put a lot of work into a single application” — this is really really great, especially for many participants in the QUIC WG and the IETF, and as you say there’s still going to be a lot of folks depending on participants in the WG/platform/service vendors to supply APIs and configuration options to exercise much of this functionality.

I think one of the things that we’re coming to realize as we discuss multipath is that many of the considerations around the different options we could choose depend on external factors (like $$$ and policies of different industry players) more than the actual technology that’s under discussion. If we think those things will be solved (and probably not by the IETF), then great! 

From a technology standpoint, we’ve opened up some really awesome possibilities with QUIC in the realm of being able to move forwards more rapidly with deployment, as you noted, and I think on that front I’d be totally interested in defining a multipath extension for folks to go deploy, collect data, and then present the delta between those benefits for an end-user vs. connection migration. We can decide on how far down the standards process we’d want that to go, one could argue that we’re already at that point with the various I-Ds that are present.

Thanks,
Eric

> On Oct 23, 2020, at 7:35 PM, Lucas Pardue <[email-masked]> wrote:
> 
> Hey Eric,
> 
> On Sat, Oct 24, 2020 at 2:16 AM Eric Kinnear <[email-masked] <mailto:[email-masked]>> wrote:
> To clarify a little. The point I was trying to make is that just adding some support for $multipath frames$ parsing to something like quiche would not be a complete solution. Something would still have to glue it to a network. And I'm not sure if there's a consistent network API across platforms that exposes path information required for something like that to work. So the application has a lot of work to do to. If they have to build to the nuances of every platform API, firmware or middleware, that's more than just free. To build on Christian's earlier point, many projects have picked their own way to do things, for good or bad.
Absolutely! I do think it would be really nice to have a common set of ways to represent much of this functionality (even just connect-by-name is nice, long before you get to multipath). 
> 
> The QUIC WG has benefited from its ability to openly and agily interoperate. Many implementations are there for people to just pick up and start running with. I'm not steadfast in belief that everyone has to use a user-space library and bundle up everything. There's absolutely a value in making technology accessible to folks that just want to get on with business via simpler or stabler means. And both these worlds can exist (and most definately should interoperate!). But I think it's prescient to consider the accessibility of new transport features to different parts of the community.

---

From: Ian Swett <[email-masked]>
Date: Wed, 28 Oct 2020 17:21:30 -0400

Based on what I've heard, I think there might be enough people to work on
this, but it may not be the people most involved in QUIC v1.  If Eric and
the 3GPP could agree on a common(minimal) feature set and design they'd
like to deploy, that would make me more interested in accepting this into
the working group.  At this point, I think we're a ways from that.

In terms of the 3GPP liason request, if this is time sensitive, I'd suggest
publishing your own draft and once you have some deployment experience,
come back.  From personal experience, the IETF is not in the business of
preventing people from widely deploying experimental technologies.  I would
love to see people using QUIC multipath in production, and I do believe
there are use cases that would benefit.

Ian

On Wed, Oct 28, 2020 at 3:21 PM Eric Kinnear <ekinnear=
[email-masked]> wrote:

> Thanks Lucas! Totally agree with what you’re saying here, was just noting
> that there we’re broadening the scope of implementation/experimentation
> possibility from “kernel vendors” to “platform/service vendors/someone who
> really wants to put a lot of work into a single application” — this is
> really really great, especially for many participants in the QUIC WG and
> the IETF, and as you say there’s still going to be a lot of folks depending
> on participants in the WG/platform/service vendors to supply APIs and
> configuration options to exercise much of this functionality.
>
> I think one of the things that we’re coming to realize as we discuss
> multipath is that many of the considerations around the different options
> we could choose depend on external factors (like $$$ and policies of
> different industry players) more than the actual technology that’s under
> discussion. If we think those things will be solved (and probably not by
> the IETF), then great!
>
> From a technology standpoint, we’ve opened up some really awesome
> possibilities with QUIC in the realm of being able to move forwards more
> rapidly with deployment, as you noted, and I think on that front I’d be
> totally interested in defining a multipath extension for folks to go
> deploy, collect data, and then present the delta between those benefits for
> an end-user vs. connection migration. We can decide on how far down the
> standards process we’d want that to go, one could argue that we’re already
> at that point with the various I-Ds that are present.
>
> Thanks,
> Eric
>
>
>> I generally agree with some of the thinking here, but I can definitely
>> see a case where “everyone gets it for free” is a thing.
> Absolutely! I do think it would be really nice to have a common set of
> ways to represent much of this functionality (even just connect-by-name is
> nice, long before you get to multipath).
>
>