Subject: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 3 Apr 2020 13:53:59 +0000

Olivier, thanks for pointing out the different operation modes (handover, interactive, aggregate). I think it's also important to mention that these modes are already deployed for MPTCP and defining these modes was/is mostly an interface and scheduling question but did not require changes to the MPTCP protocol itself. With QUIC you could apply these modes as well and you can be even smarter than we often are with TCP today, as you can also schedule on a stream basis. If you have traffic in your transport protocol that you already know is independent of each other, being able utilizing multiple paths becomes even more useful.

Lars, I believe the most important use case here is where a mobile device have one cellular and one wifi connection. In such a setup RTTs are actually not as much different as it would in e.g. the fix network vs. cellular case, or satellite vs. cellular case but it is also mostly the handover or interactive mode that is most interesting (interactive especially in case your wifi is really crappy). Particularly for the handover case, being able to use both paths simultaneously for a short time can make your handover much smoother as you can initiate the handover earlier and also for the case where you are at the border of a coverage area, you can much easier switch forth an back when needed. Limited handover is possible with QUIC path migration already but given how common the handover case for mobile devices is, I think we should further optimizing for it.

On 03.04.20, 09:55, "QUIC on behalf of Olivier Bonaventure" <[email-masked] on behalf of [email-masked]> wrote:

    Lars,
    > 
    > what we have done with MPTCP is build a system that is reasonably good 
    > at pooling the capacity of multiple paths for one connection, *if* these 
    > paths have reasonably similar characteristics, esp. RTT.
     >
    > With paths with dissimilar RTTs, we can still pool the capacity, but 
    > only make it available for use to the application at pretty much the 
    > maximum of the individual RTTs. That’s still somewhat useful if you care 
    > about bulk throughput. It’s basically useless if you care about 
    > small-transfer latencies, which the web has been gravitating to for 
    > years. (Remember that the MPTCP design is ten years old - from a 
    > different world. Bandwidths have gone up a lot, making capacity pooling 
    > less urgent, but late cues have not decreased much.)
    
    This corresponds to the hybrid access network case that pools bandwidth.
    
    > To make multi path useful to today’s mostly latency-sensitive 
    > applications, we need a path scheduler that can pool (some of) the 
    > capacity of multiple dissimilar paths at some effective RTT that is much 
    > lower than the maximum of the individual ones. I am not aware of any 
    > proposed scheduler - for MPTCP or otherwise - that achieves that.

    AFAIK, this is exactly what Apple is doing with MPTCP. They call this 
    the "interactive mode" and the stack dynamically selects the path (wifi 
    or cellular) having the best performance (rtt, losses) for interactive 
    applications. See https://developer.apple.com/videos/play/wwdc2017/707/
    
    > Even if we change the argument from pooling latency to increasing 
    > resilience, I still doubt that increased resilience at the cost of 
    > much-increased late zu is a worthwhile trade off for many apps.
    
    AFAIK, Apple calls that the "handover mode", it is probably less useful 
    that the interactive one. Maybe Apple engineers who are on this list can 
    comment and provide more details.
    
    I don't think that Apple uses MPTCP to pool bandwidth. They use MPTCP to 
    improve user experience and mainly latency.

    Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 3 Apr 2020 16:17:03 +0200

Mirja,

> Olivier, thanks for pointing out the different operation modes (handover, interactive, aggregate). I think it's also important to mention that these modes are already deployed for MPTCP and defining these modes was/is mostly an interface and scheduling question but did not require changes to the MPTCP protocol itself.

Indeed. If one looks at the differences between RFC6824 (2013) and
RFC8684 (published last week), he/she can see that the protocol did not
have to change to accomodate any of these new deployments.

https://www.ietf.org/rfcdiff?url1=rfc8684&url2=rfc6824

The main differences concern the initial handshake and the advertisement 
of addresses that is more secure in RFC8684 than in RFC6824.

In MPTCP, the different use cases are mainly addressed by defining 
specific path managers and packet schedulers. The path manager is the 
algorithm that decides when a subflow should be used over a specific 
path. The packet scheduler is the algorithm that decides the path that 
will be used to transmit each packet (new or retransmitted).

> With QUIC you could apply these modes as well and you can be even smarter than we often are with TCP today, as you can also schedule on a stream basis. If you have traffic in your transport protocol that you already know is independent of each other, being able utilizing multiple paths becomes even more useful.

QUIC already includes several mechanisms that simplify a multipath 
transport protocol :
  - we do not need to take care of middlebox interference and for 
example a frame sent on paths 1 and 2 but acknowledged only on 1 does 
not need to be retransmitted on path 2 to cope with middleboxes that 
would expect data in sequence
  - the PING frame makes it easy to probe a path without sending real 
data over it
  - address advertisements are much easier in QUIC since everything is 
authenticated and we don't need to cope with the limite size of the TCP 
options
  - acknowledgements are simpler because they do not need to be sent on 
the reverse path of the data
  - ...

Then a multipath version of QUIC also needs a path manager and packet 
scheduler. In principle, these are similar to the algorithms used by 
MPTCP implementations. As you mention, the streams provide additional 
information for the scheduler that does not exist in MPTCP.

> Lars, I believe the most important use case here is where a mobile device have one cellular and one wifi connection. In such a setup RTTs are actually not as much different as it would in e.g. the…

Totally agree

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 3 Apr 2020 16:21:24 +0100

Hey Mirja, Olivier,

I'll acknowledge that HTTP/3 isn't the only use of QUIC, but it is a
comprehensive use of the transport features that has deployment experience.
So looking at MPQUIC thorough the lens of HTTP/3, streams may not be truly
independent because of their reliance on other streams such control or
QPACK. Furthermore, even if we model the streams as independent HTTP
requests, the reality is that there is often a higher layer that wants to
put them back together and there the ordering can matter. These are
elements of the puzzle for prioritizing multiplexing of streams.

Getting scheduling of HTTP streams right is really hard. With HTTP/2 and
TCP today we see several server implementations that just don't do it very
well and may never get around to fixing it. We've also seen issues in how
servers adapt their scheduling to client reprioritization signals. That
seems quite relevant to this use case which requires a server to select an
appropriate path based on client conditions.

The experience, to me, is a sign that people would struggle to build an
HTTP/3 implementation (particularly server-side response scheduling) that
achieves the theoretical goals of MPQUIC. Is it something that can be
tested and shown to demonstrably outperform HTTP/3 plus connection
migration in a statistically significant fashion? Maybe there are case
studies of  HTTP/2 implementations that have tried adapting to MPTCP, which
could inform an analysis for HTTP/3?

Cheers
Lucas

On Fri, Apr 3, 2020 at 3:19 PM Olivier Bonaventure <
[email-masked]> wrote:

> Mirja,
>
> > Olivier, thanks for pointing out the different operation modes
> (handover, interactive, aggregate). I think it's also important to mention
> that these modes are already deployed for MPTCP and defining these modes
> was/is mostly an interface and scheduling question but did not require
> changes to the MPTCP protocol itself.
>
> > With QUIC you could apply these modes as well and you can be even
> smarter than we often are with TCP today, as you can also schedule on a
> stream basis. If you have traffic in your transport protocol that you
> already know is independent of each other, being able utilizing multiple
> paths becomes even more useful.
>
> > Lars, I believe the most important use case here is where a mobile
> device have one cellular and one wifi connection. In such a setup RTTs are
> actually not as much different as it would in e.g. the fix network vs.
> cellular case, or satellite vs. cellular case but it is also mostly the
> handover or interactive mode that is most interesting (interactive
> especially in case your wifi is really crappy). Particularly for the
> handover case, being able to use both paths simultaneously for a short time
> can make your handover much smoother as you can initiate the handover
> earlier and also for the case where you are at the border of a coverage
> area, you can much easier switch forth an back when needed. Limited
> handover is possible with QUIC path migration already but given how common
> the handover case for mobile devices is, I think we should further
> optimizing for it.
>
> Totally agree
>
>
> Olivier
>
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 3 Apr 2020 19:00:51 +0200

Hello Lucas,

> comprehensive use of the transport features that has deployment 

I agree that HTTP/3 has been the main use case for QUIC and that it has 
influenced some parts of its design. However, I personally consider QUIC 
as the generic transport protocol that will support a broad set of 
applications that will probably have different requirements than HTTP/3.

QUIC has a very clean and extensible design and architecture. It opens 
new ways to think about the transport layer without all the problems 
that TCP has faced since roughly the last two decades.

> So looking at MPQUIC thorough the lens of HTTP/3, streams 
> may not be truly independent because of their reliance on other streams 
> such control or QPACK. 

Agreed, but there are applications that will use independant streams. A 
simple example is a standard file transfert and many existing 
applications that rely on a single bytestream. A TLS VPN that uses QUIC 
instead of TLS/TCP could map each transported TCP connection over a 
different QUIC stream.

> Furthermore, even if we model the streams as 
> independent HTTP requests, the reality is that there is often a higher 
> layer that wants to put them back together and there the ordering can 
> matter. These are elements of the puzzle for prioritizing multiplexing 
> of streams.
> 
> TCP today we see several server implementations that just don't do it 
> very well and may never get around to fixing it. We've also seen issues 
> in how servers adapt their scheduling to client reprioritization 
> signals. That seems quite relevant to this use case which requires a 
> server to select an appropriate path based on client conditions.
> 
> HTTP/3 implementation (particularly server-side response scheduling) 
> that achieves the theoretical goals of MPQUIC. Is it something that can 
> be tested and shown to demonstrably outperform HTTP/3 plus connection 
> studies of  HTTP/2 implementations that have tried adapting to MPTCP, 
> which could inform an analysis for HTTP/3?

In MPTCP, we were less ambitious than in HTTP/2 or HTTP/3 concerning the 
packet schedulers. We considered that each host is independant and that 
it uses its own packet scheduler. In contrast with HTTP/2, we did not 
define ways to prioritise one path over the other by exchanging 
information in the protocol (except the backup bit, but this disables a 
path). In practice, what we observe is that the application selects the 
packet scheduler that it needs locally. On Apple smartphones, this is 
the choice between the Handover and the Interactive modes for example. 
This is not as powerful as negotiating packet schedulers over the MPTCP 
connection, but seems a reasonable tradeoff from an engineering 
viewpoint based on deployment experience.

Olivier

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 3 Apr 2020 19:05:16 +0100

Hi Olivier,

See response inline,

> viewpoint based on deployment experience.

HTTP/3 is not the only show in town but it is an example of a
latency-sensitive application that fully utilises the capabilities of QUIC
and has a high probability to exercise concurrent multiplexing.

To clarify my question, I wasn't trying to compare HTTP/2 with MPTCP but
rather asking if anyone has experimented with scheduling of HTTP/2 streams
over a multipath TCP connection. That seems to be the closest to how the an
application would want to approach using MPQUIC for an application that
would care about stream multiplexing (at least from my understanding of the
proposal based on the I-D and Quentin's presentation to Zurich).

I agree with Roberto that the challenge is with the API. HTTP/2 stream
priorities with their limited dimensions have more-or-less failed. MPQUIC
would appear to have more dimensions and I just don't know how the client
and server would expect to measure and achieve their desired QoE
improvements. As an implementer I would like to have a framework that helps
me in some way to assess complexity vs. benefit in order to justify
spending the effort. If there is evidence or research in this respect I'd
appreciate some links I can read up on.

Cheers,
Lucas

On Fri, Apr 3, 2020 at 6:01 PM Roberto Peon <[email-masked]> wrote:

>
> Regardless of other changes that could solve some of the H3 issues at the
> protocol level, I believe that much of the problem is an API problem that
> we’re attempting to treat as a protocol problem.
>
> To be clear, I’m saying that multipath is primarily an API problem (and am
> happy to acknowledge that that “primarily” != “wholly”).
>
> The failure of H2 priorities was, in my opinion, in many ways because of
> this same API problem—until applications have good mechanisms by which to
> express the properties that they need (whether priorities or not), we’ll
> not be able to use protocol features in ways that matter to the application
> much of the time.
>
> If we want MPQUIC to work widely with application deployments (or PRHTTP
> or whatever), there must be an easy path for those applications to signal
> their needs, e.g. how much redundancy makes sense for the transmission of
> some item, or how latency sensitive is that thing?
>
> I actually love the idea of getting things out of order, since any
> application that is treating the order as something to be guaranteed is
> pretty much guaranteed to be provably incorrect, but I’m also sadly sure
> this wouldn’t be that popular an opinion.
> -=R
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Lucas Pardue <
> [email-masked]>
> *Date: *Friday, April 3, 2020 at 8:22 AM
> *To: *"[email-masked]" <[email-masked]
> >
> *Cc: *Ted Hardie <[email-masked]>, Mirja Kuehlewind <
> [email-masked]>, Lars Eggert <[email-masked]>, IETF QUIC
> WG <[email-masked]>, Martin Thomson <[email-masked]>, Spencer Dawkins
> at IETF <[email-masked]>
> *Subject: *Re: Use of multiple paths [was: Re: [Fwd: New Liaison
> Statement, "LS on need for Multi-Path QUIC for ATSSS"]]
>
>
>
> Hey Mirja, Olivier,
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_rfcdiff-3Furl1-3Drfc8684-26url2-3Drfc6824&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=rRICKNGIOM-uW98qnUJZABiZCBaHEXeqUvYzOGfuST8&s=VgA0Adyhx0GCjR-tQx_KyK_K96kqpEAhPMtRvzWPxRk&e=>
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 3 Apr 2020 18:43:26 +0000

Hi Lucas,

in both the handover and the interactive case, you would actually only use more or less one path at a time (with some overlap when switching). Then, if you have an application with mostly short streams, you could e.g. try to only switch when there is a new stream and thereby further optimize your transmission.

Even in the aggregation mode the observation was rather that if you have different streams, then it’s probably better to have a scheduler that works on a per stream basis than on a per frame basis. Of course if there would be a prioritization scheme that would be even better and could help to inform a scheduler. But again the assumption for the most common use cases is also that RTTs are actually not that different and in most cases the application would only select one of these modes and would not further need to worry about it. Yes, it would be possible to optimize even more but as you say that makes it also more complex.

Again as I said and Roberto also confirmed, changing that is just a question of how you design your interface and not about changing the protocol. So as long as we have one interface that provide benefits for the cases we are most interested in, which I think we have based on the experience with MPTCP deployment, there is room for further research if people are interested. But designing the protocol to be used with the interface we already have deployed is independent of that.

Mirja

From: Lucas Pardue <[email-masked]>
Date: Friday, 3. April 2020 at 20:05
To: Roberto Peon <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, Ted Hardie <[email-masked]>, Mirja Kuehlewind <[email-masked]>, Lars Eggert <[email-masked]>, IETF QUIC WG <[email-masked]>, Martin Thomson <[email-masked]>, Spencer Dawkins at IETF <[email-masked]>
Subject: Re: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Hi Olivier,

See response inline,

On Fri, Apr 3, 2020 at 6:00 PM Olivier Bonaventure <[email-masked]<mailto:[email-masked]>> wrote:

HTTP/3 is not the only show in town but it is an example of a latency-sensitive application that fully utilises the capabilities of QUIC and has a high probability to exercise concurrent multiplexing.

To clarify my question, I wasn't trying to compare HTTP/2 with MPTCP but rather asking if anyone has experimented with scheduling of HTTP/2 streams over a multipath TCP connection. That seems to be the closest to how the an application would want to approach using MPQUIC for an application that would care about stream multiplexing (at least from my understanding of the proposal based on the I-D and Quentin's presentation to Zurich).

I agree with Roberto that the challenge is with the API. HTTP/2 stream priorities with their limited dimensions have more-or-less failed. MPQUIC would appear to have more dimensions and I just don't know how the client and server would expect to measure and achieve their desired QoE improvements. As an implementer I would like to have a framework that helps me in some way to assess complexity vs. benefit in order to justify spending the effort. If there is evidence or research in this respect I'd appreciate some links I can read up on.

Cheers,
Lucas

On Fri, Apr 3, 2020 at 6:01 PM Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:

Regardless of other changes that could solve some of the H3 issues at the protocol level, I believe that much of the problem is an API problem that we’re attempting to treat as a protocol problem.

To be clear, I’m saying that multipath is primarily an API problem (and am happy to acknowledge that that “primarily” != “wholly”).

The failure of H2 priorities was, in my opinion, in many ways because of this same API problem—until applications have good mechanisms by which to express the properties that they need (whether priorities or not), we’ll not be able to use protocol features in ways that matter to the application much of the time.

If we want MPQUIC to work widely with application deployments (or PRHTTP or whatever), there must be an easy path for those applications to signal their needs, e.g. how much redundancy makes sense for the transmission of some item, or how latency sensitive is that thing?

I actually love the idea of getting things out of order, since any application that is treating the order as something to be guaranteed is pretty much guaranteed to be provably incorrect, but I’m also sadly sure this wouldn’t be that popular an opinion.
-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 3, 2020 at 8:22 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Ted Hardie <[email-masked]<mailto:[email-masked]>>, Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, Lars Eggert <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>, Spencer Dawkins at IETF <[email-masked]<mailto:[email-masked]>>
Subject: Re: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Hey Mirja, Olivier,

I'll acknowledge that HTTP/3 isn't the only use of QUIC, but it is a comprehensive use of the transport features that has deployment experience. So looking at MPQUIC thorough the lens of HTTP/3, streams may not be truly independent because of their reliance on other streams such control or QPACK. Furthermore, even if we model the streams as independent HTTP requests, the reality is that there is often a higher layer that wants to put them back together and there the ordering can matter. These are elements of the puzzle for prioritizing multiplexing of streams.

Getting scheduling of HTTP streams right is really hard. With HTTP/2 and TCP today we see several server implementations that just don't do it very well and may never get around to fixing it. We've also seen issues in how servers adapt their scheduling to client reprioritization signals. That seems quite relevant to this use case which requires a server to select an appropriate path based on client conditions.

The experience, to me, is a sign that people would struggle to build an HTTP/3 implementation (particularly server-side response scheduling) that achieves the theoretical goals of MPQUIC. Is it something that can be tested and shown to demonstrably outperform HTTP/3 plus connection migration in a statistically significant fashion? Maybe there are case studies of  HTTP/2 implementations that have tried adapting to MPTCP, which could inform an analysis for HTTP/3?

Cheers
Lucas

On Fri, Apr 3, 2020 at 3:19 PM Olivier Bonaventure <[email-masked]<mailto:[email-masked]>> wrote:
Mirja,

https://www.ietf.org/rfcdiff?url1=rfc8684&url2=rfc6824<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_rfcdiff-3Furl1-3Drfc8684-26url2-3Drfc6824&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=rRICKNGIOM-uW98qnUJZABiZCBaHEXeqUvYzOGfuST8&s=VgA0Adyhx0GCjR-tQx_KyK_K96kqpEAhPMtRvzWPxRk&e=>

Totally agree

Olivier

---

From: Ted Hardie <[email-masked]>
Date: Fri, 3 Apr 2020 12:02:08 -0700

Howdy,

On Fri, Apr 3, 2020 at 11:43 AM Mirja Kuehlewind <
[email-masked]> wrote:

> Hi Lucas,
>
>
>
> in both the handover and the interactive case, you would actually only use
> more or less one path at a time (with some overlap when switching). Then,
> if you have an application with mostly short streams, you could e.g. try to
> only switch when there is a new stream and thereby further optimize your
> transmission.
>
>
>
> Even in the aggregation mode the observation was rather that if you have
> different streams, then it’s probably better to have a scheduler that works
> on a per stream basis than on a per frame basis. Of course if there would
> be a prioritization scheme that would be even better and could help to
> inform a scheduler. But again the assumption for the most common use cases
> is also that RTTs are actually not that different
>

I think designing a system around this assumption is not, perhaps, the best
idea.  It was certainly not the case a few years back and, with the advent
of LEO paths, it may not be the case again in the relatively near future.
While you can certainly start with the assumption and measure deviation
using ping frames, I think even a per stream scheduler would need this if
it is using both paths.  Even for cases where there are relatively small
differences in path characteristics, you might find it best to select one
over the other for specific application types (e.g. those that mix
datagram-based media and regular streams).

> and in most cases the application would only select one of these modes and
> would not further need to worry about it. Yes, it would be possible to
> optimize even more but as you say that makes it also more complex.
>
>
>
> Again as I said and Roberto also confirmed, changing that is just a
> question of how you design your interface and not about changing the
> protocol.
>

The interface can only provide controls if they are in the protocol, and I
believe that getting MPQUIC right will require at least some new
facilities.  YMMV, of course.

best,

Ted

> So as long as we have one interface that provide benefits for the cases we
> are most interested in, which I think we have based on the experience with
> MPTCP deployment, there is room for further research if people are
> interested. But designing the protocol to be used with the interface we
> already have deployed is independent of that.
>
>
>
> Mirja
>
>
>
>
>
> *From: *Lucas Pardue <[email-masked]>
> *Date: *Friday, 3. April 2020 at 20:05
> *To: *Roberto Peon <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>,
> Ted Hardie <[email-masked]>, Mirja Kuehlewind <

---

From: Roberto Peon <[email-masked]>
Date: Fri, 3 Apr 2020 19:15:14 +0000

QUIC is currently missing a session layer other than H3.

I'm pretty unhappy about the idea of other protocols right now, since I believe moving forward without a generic session layer is a mistake we'll all pay for in the future ( again).

-=R

On 4/3/20, 10:01 AM, "QUIC on behalf of Olivier Bonaventure" <[email-masked] on behalf of [email-masked]> wrote:

    Hello Lucas,

    

    

    Olivier

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 6 Apr 2020 12:44:15 +0000

Hi Roberto,

Do you have a reference or can you expand on what functionalities you actually
expect from a session layer? 

Cheers

Magnus

On Fri, 2020-04-03 at 19:15 +0000, Roberto Peon wrote:
> I'm pretty unhappy about the idea of other protocols right now, since I
> believe moving forward without a generic session layer is a mistake we'll all
> pay for in the future ( again).
> 
> -=R
> 
> On 4/3/20, 10:01 AM, "QUIC on behalf of Olivier Bonaventure" <
> [email-masked] on behalf of [email-masked]> wrote:
> 
>     Hello Lucas,
>     
-- 
Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Networks, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 6 Apr 2020 17:20:41 -0500

For what it's worth,

>From https://www.ietf.org/jabber/logs/masque/2020-03-24.html (extracted,
please feel free to correct me if I'm misrepresenting anyone):

*[23:48:45]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:48:45.357561>
<[email-masked]
<[email-masked]>> Can someone explain why you couldn't you
have QUIC over QUIC? The payload would be the packet the proxy forwards.*
[23:48:46]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:48:46.658966>
<sftcd> could be, fine question though
[23:48:47]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:48:47.474340>
<hta> jonathan: I don't regard drop-from-queue as congestion control.
[23:48:55]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:48:55.904420>
<keithmoore> Magnus: but I also think that we're inevitably going to have
something like this, whether we want it or not, including the potential for
each app to negotiate its own relationship with an external proxy.   so we
might as well try to understand the hazards.
*[23:49:00]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:00.650063>
<[email-masked]
<[email-masked]>> carrick: you can have quic over quic. It's just not
specified yet*

*[23:49:02]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:02.352887>
<achernya> There's
no specification for anything-over-QUIC over than h3[23:49:10]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:10.785012>
<achernya> *other
than, rather*
[23:49:15]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:15.956316>
<Ted.h> @Martin I dropped Matt a note to ask for a citation; will forward
to you if he has one that can be referenced.
[23:49:18]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:18.593009> <
[email-masked]> I see, thanks
[23:49:25]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:25.634095>
<Martin
Thomson> Ted.h: appreciate it, thanks
*[23:49:30]
<https://www.ietf.org/jabber/logs/masque/2020-03-24.html#23:49:30.856183>
<spencerdawkins> That's
the problem - there's not a generalized upper interface for QUIC.  *

 and, in https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html
(extracted, and please feel free to correct me ...

[22:35:32]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:35:32.68890>
<Martin
Thomson> I also support adoption.  The requirements parts are mostly
solid.  I have concerns about the composition of transports in the overview
piece.
[22:35:45]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:35:45.447758>
<[email-masked]> I also support adoption of this draft.
[22:36:38]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:38.771346>
<[email-masked]/barnowl> schmansport sessions?

*[22:36:40]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:40.118114>
<hta> Session
transport![22:36:41]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:41.310567>
<spencerdawkins> Is
there an agreed upper interface for QUIC that is not HTTP/3?*
[22:36:48]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:48.169519>
<[email-masked]> +1 on what Mark said
[22:36:49]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:49.696124>
<[email-masked]> WebSockets/2
[22:36:49]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:49.855420>
<brong> SPLITTERS
[22:36:54]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:36:54.772286>
<[email-masked]> about "transport"
*[22:37:07]
<https://www.ietf.org/jabber/logs/webtrans/2020-03-27.html#22:37:07.443469>
<[email-masked]
<[email-masked]>> Spencer: draft-ietf-quic-transport ?  *

So, ISTM that this question is banging around from venue to venue. Some
people are assuming draft-ietf-quic-transport will work well enough for
protocols that aren't HTTP/3, and others are not so sure.

Definitely worth talking about, in what I hope is the near future ...

(It's also worth mentioning that
https://tools.ietf.org/html/draft-pardue-quic-siduck-00 appears in at least
one of those jabber logs)

Best,

Spencer

Cheers
> >     > Getting scheduling of HTTP streams right is really hard. With
> HTTP/2
> > and

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 7 Apr 2020 07:48:48 +0000

Thanks,

So I was groping after something more complex. So basically the definition of
reliable multi-stream usage of QUIC that has an ALPN to identify that there it
is up to the application to interpret how the streams are used. A sort of "TCP
socket" for QUIC? Which might also imply some basic constraints on the API
between the application and the QUIC stack? 

Magnus

On Mon, 2020-04-06 at 17:20 -0500, Spencer Dawkins at IETF wrote:
> From https://www.ietf.org/jabber/logs/masque/2020-03-24.htmll (extracted,
> [23:48:45] <[email-masked]> Can someone explain why you
> couldn't you have QUIC over QUIC? The payload would be the packet the proxy
> forwards.
> [23:48:46] <sftcd> could be, fine question though
> [23:48:47] <hta> jonathan: I don't regard drop-from-queue as congestion
> control.
> [23:48:55] <keithmoore> Magnus: but I also think that we're inevitably going
> to have something like this, whether we want it or not, including the
> potential for each app to negotiate its own relationship with an external
> proxy.   so we might as well try to understand the hazards.
> [23:49:00] <[email-masked]> carrick: you can have quic over quic. It's just
> not specified yet
> [23:49:02] <achernya> There's no specification for anything-over-QUIC over
> than h3
> [23:49:10] <achernya> *other than, rather
> [23:49:15] <Ted.h> @Martin I dropped Matt a note to ask for a citation; will
> forward to you if he has one that can be referenced.
> [23:49:18] <[email-masked]> I see, thanks
> [23:49:25] <Martin Thomson> Ted.h: appreciate it, thanks
> [23:49:30] <spencerdawkins> That's the problem - there's not a generalized
> upper interface for QUIC.  
> 
> [22:35:32] <Martin Thomson> I also support adoption.  The requirements parts
> are mostly solid.  I have concerns about the composition of transports in the
> overview piece.
> [22:35:45] <[email-masked]> I also support adoption of this draft.
> [22:36:38] <[email-masked]/barnowl> schmansport sessions?
> [22:36:40] <hta> Session transport!
> [22:36:41] <spencerdawkins> Is there an agreed upper interface for QUIC that
> is not HTTP/3?
> [22:36:48] <[email-masked]> +1 on what Mark said
> [22:36:49] <[email-masked]> WebSockets/2
> [22:36:49] <brong> SPLITTERS
> [22:36:54] <[email-masked]> about "transport"
> [22:37:07] <[email-masked]> Spencer: draft-ietf-quic-transport ?  
> 
> So, ISTM that this question is banging around from venue to venue. Some people
> are assuming draft-ietf-quic-transport will work well enough for protocols
> that aren't HTTP/3, and others are not so sure. 
> 
-- 
Cheers

Magnus Westerlund

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 7 Apr 2020 09:23:03 -0500

Hi, Magnus,

> between the application and the QUIC stack?

Thanks for this. I was thinking that what you're talking about, is what
other people are also groping for, assuming various levels of complexity :-)

Could you give an example or two of the kind of basic constraints you're
thinking of, so we can better understand?

Best,

Spencer

---

From: Mike Bishop <[email-masked]>
Date: Fri, 10 Apr 2020 15:44:39 +0000

Functionally, I think that what gets exposed to an application is a connection object with a set of streams.  An application needs to take some combination of two design choices:

  *   A new protocol model that is tailored to a multi-stream transport
  *   Running an instance of the protocol on each stream

HTTP/3 is a hybrid of the two, with the overall protocol handling multiple streams but requests on each stream might be extracted and routed separately.  An ALPN token inherently defines the protocol, and the protocol has to define how to do this.  There can’t be an ALPN protocol for “I dunno.”

When designing multi-path, you’ll have different focuses based on which model you assume the protocol is using.  For example, you can route each stream independently – but if you fail to deliver a stream that has cross-stream dependencies, then everything suffers.  Protocols over QUIC will inherently require a separate definition from the same protocol over TCP, unless you use a single stream.  (Protocols over SCTP wouldn’t sacrifice as much to be ported directly.)

If I’m extrapolating Roberto’s idea of a session layer correctly, it’s essentially an understanding in QUIC that would describe how the application intends to use the multiple streams and how they’re related to each other.  I don’t think you can do that generically, but if you could, I agree it would simplify a lot of concerns.

Ultimately, I think multi-path would have to manifest the same way multi-stream does – you expose it to the application layer, and let the app direct how to leverage it for the way it’s mapped onto the transport.

From: QUIC <[email-masked]> On Behalf Of Spencer Dawkins at IETF
Sent: Tuesday, April 7, 2020 10:23 AM
To: Magnus Westerlund <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Mirja Kuehlewind <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Hi, Magnus,

On Tue, Apr 7, 2020 at 2:48 AM Magnus Westerlund <[email-masked]<mailto:[email-masked]>> wrote:
Thanks,

Magnus

Thanks for this. I was thinking that what you're talking about, is what other people are also groping for, assuming various levels of complexity :-)

Could you give an example or two of the kind of basic constraints you're thinking of, so we can better understand?

Best,

Spencer

---

From: Roberto Peon <[email-masked]>
Date: Tue, 14 Apr 2020 18:48:25 +0000

My idea of ‘session’ is potentially very simple:
- Route some set of streams to the same ultimate endpoint, by having each of these streams use some same session-ID.
I’ve called this “stream grouping” in the past, but lets call it session-id for now.

There are other things one can throw-in, such as doing compression, or various settings (prioritization policy statements, for instance) on a per session-id basis.
Once you have that, you don’t need to deploy new proxies to support new protocols— each protocol on the connection can use a different session-id from that of the other protocols, and it is sufficient.

There is, separately, some fun API-related problems around multipath that are unrelated to sessions.
-=R

From: Mike Bishop <[email-masked]>
Date: Friday, April 10, 2020 at 8:44 AM
To: Spencer Dawkins at IETF <[email-masked]>, Magnus Westerlund <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, Roberto Peon <[email-masked]>, "[email-masked]" <[email-masked]>, Mirja Kuehlewind <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Hi, Magnus,

Magnus

Best,

Spencer

---

From: Ian Swett <[email-masked]>
Date: Tue, 14 Apr 2020 15:42:22 -0400

Given 0-RTT makes connection establishment fast, is the gain of sharing a
'connection' across multiple sessions a shared congestion controller or
something else?

Put another way: why not just open multiple connections, since that's
substantially easier?

On Tue, Apr 14, 2020 at 2:49 PM Roberto Peon <[email-masked]> wrote:

> - Route some set of streams to the same ultimate endpoint, by having each
> of these streams use some same session-ID.
> There are other things one can throw-in, such as doing compression, or
> various settings (prioritization policy statements, for instance) on a per
> session-id basis.
> Once you have that, you don’t need to deploy new proxies to support new
> protocols— each protocol on the connection can use a different session-id
> from that of the other protocols, and it is sufficient.
>
> *From: *Mike Bishop <[email-masked]>
> *Date: *Friday, April 10, 2020 at 8:44 AM
> *To: *Spencer Dawkins at IETF <[email-masked]>, Magnus
> Westerlund <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, Roberto Peon <
> [email-masked]>, "[email-masked]" <
> [email-masked]>, Mirja Kuehlewind <
> [email-masked]>, "[email-masked]" <
> [email-masked]>, "[email-masked]" <[email-masked]>, "
> [email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
> *Subject: *RE: Use of multiple paths [was: Re: [Fwd: New Liaison
> Functionally, I think that what gets exposed to an application is a
> connection object with a set of streams.  An application needs to take some
> combination of two design choices:
>
>    - A new protocol model that is tailored to a multi-stream transport
>    - Running an instance of the protocol on each stream
>
>
>
> HTTP/3 is a hybrid of the two, with the overall protocol handling multiple
> streams but requests on each stream might be extracted and routed
> separately.  An ALPN token inherently defines the protocol, and the
> protocol has to define how to do this.  There can’t be an ALPN protocol for
> “I dunno.”
>
>
>
> When designing multi-path, you’ll have different focuses based on which
> model you assume the protocol is using.  For example, you can route each
> stream independently – but if you fail to deliver a stream that has
> cross-stream dependencies, then everything suffers.  Protocols over QUIC
> will inherently require a separate definition from the same protocol over
> TCP, unless you use a single stream.  (Protocols over SCTP wouldn’t
> sacrifice as much to be ported directly.)
>
>
>
> If I’m extrapolating Roberto’s idea of a session layer correctly, it’s
> essentially an understanding in QUIC that would describe how the
> application intends to use the multiple streams and how they’re related to
> each other.  I don’t think you can do that generically, but if you could, I
> agree it would simplify a lot of concerns.
>
>
>
> Ultimately, I think multi-path would have to manifest the same way
> multi-stream does – you expose it to the application layer, and let the app
> direct how to leverage it for the way it’s mapped onto the transport.
>
>
>
> *From:* QUIC <[email-masked]> *On Behalf Of * Spencer Dawkins at
> IETF
> *Sent:* Tuesday, April 7, 2020 10:23 AM
> *To:* Magnus Westerlund <[email-masked]>
> *Cc:* [email-masked]; [email-masked]; [email-masked];
> Mirja Kuehlewind <[email-masked]>;
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> *Subject:* Re: Use of multiple paths [was: Re: [Fwd: New Liaison

---

From: Roberto Peon <[email-masked]>
Date: Tue, 14 Apr 2020 20:29:09 +0000

With multiple connections you almost always end up with multiple congestion controllers, which then fight amongst themselves, causing self-congestion, potentially additional latency (ramping the congestion window up) or other fun things.
Imho, it is desirable to minimize the total number of congestion-controllers involved wherever possible, supposing that these CCs have reasonable algorithms.

There was a lot of gaming of this in HTTP1 vs HTTP2 because of the “poor behavior” of the CCs deployed during that time—when there was loss you’d have multiplicatively less bandwidth reduction with multiple connections as compared to a single connection because of how the CCs worked. This is ultimately bad for the network and suboptimal for the application.

Separately, connections are expensive, and you can stretch things further/more cheaply if you don’t have to handle multiplicatively more. DoS attacks/malicious actors become more obvious when you do connection reuse, as compared to many single connections. Etc. etc.
-=R

From: Ian Swett <[email-masked]>
Date: Tuesday, April 14, 2020 at 12:42 PM
To: Roberto Peon <[email-masked]>
Cc: Mike Bishop <[email-masked]>, Spencer Dawkins at IETF <[email-masked]>, Magnus Westerlund <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, Mirja Kuehlewind <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Given 0-RTT makes connection establishment fast, is the gain of sharing a 'connection' across multiple sessions a shared congestion controller or something else?

Put another way: why not just open multiple connections, since that's substantially easier?

On Tue, Apr 14, 2020 at 2:49 PM Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
My idea of ‘session’ is potentially very simple:
- Route some set of streams to the same ultimate endpoint, by having each of these streams use some same session-ID.
I’ve called this “stream grouping” in the past, but lets call it session-id for now.

From: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 10, 2020 at 8:44 AM
To: Spencer Dawkins at IETF <[email-masked]<mailto:[email-masked]>>, Magnus Westerlund <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Roberto Peon <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Spencer Dawkins at IETF
Sent: Tuesday, April 7, 2020 10:23 AM
To: Magnus Westerlund <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: Use of multiple paths [was: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]]

Hi, Magnus,

Magnus

Best,

Spencer