Subject: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]

From: Magnus Westerlund <[email-masked]>
Date: Wed, 1 Apr 2020 16:15:17 +0000

QUIC WG,

The IETF has received the attached Liasion statement (
https://datatracker.ietf.org/liaison/1678/) that is intended for the QUIC WG.
Please review it and the WG should considerd sending information to 3GPP as
requested at key points in the process of the multi-path extension/version of
QUIC. 
 
Cheers

Magnus Westerlund 
TSV AD

Subject: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"
From: Liaison Statement Management Tool <[email-masked]>
To: "The IETF Chair" <[email-masked]>
Cc: The IETF Chair <[email-masked]>, The IESG <[email-masked]>, [email-masked], Gonzalo Camarillo <[email-masked]>
Date: Wed, 01 Apr 2020 08:09:28 -0700
List-Id: <iesg.ietf.org>

Title: LS on need for Multi-Path QUIC for ATSSS
Submission Date: 2020-04-01
URL of the IETF Web page: https://datatracker.ietf.org/liaison/1678/

From: Susanna Kooistra <[email-masked]>
To: The IETF Chair <[email-masked]>
Cc: The IESG <[email-masked]>,The IETF Chair <[email-masked]>,Gonzalo Camarillo <[email-masked]>
Response Contacts: [email-masked]
Technical Contacts: 
Purpose: For information

Body: 1. Overall Description:
3GPP has approved the study item on “Study on Access Traffic Steering, Switch and Splitting support in the 5G system architecture Phase 2” as part of Release 17.

This study aims to investigate the extension of Rel-16 ATSSS feature enabling the support of steering, switching and splitting the traffic of a PDU session supported over multiple accesses (namely 3GPP and Non-3GPP access), including the following objective:

-	Whether and how to support additional steering methods(s). Proposed solutions shall be based on IETF protocols or extension of such protocols (i.e. QUIC/MP-QUIC).

Please see attached document for details on study item objectives and timelines. 

The work on the study has not yet started in 3GPP, and there are thus no agreed conclusions. . The goal is to enable steering, switching and splitting of traffic (primarily UDP) across multiple accesses, including latency sensitive and real time traffic. Therefore 3GPP is interested to receive regular feedback on progress and prioritization on the multipath extensions to QUIC.

2. Actions:
To IETF group:
ACTION: 	3GPP SA kindly requests IETF to take the above information into account when discussing future work in prioritizing the multipath work for QUIC.

3. Date of Next TSG-SA Meetings:
3GPP TSG SA#88	17th – 19th June 2020	Malmö, SWEDEN
3GPP TSG SA#89	16th – 18th September 2020	Funchal (Madeira), Portugal
Attachments:

    SP-200095_S2-2002593_SP-190558_Revised_SID_FS_ATSSS_Ph2_rm
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2020-04-01-3gpp-ietf-ls-on-need-for-multi-path-quic-for-atsss-attachment-1.doc

    SP-200095_S2-2002593_SP-190558_Revised_SID_FS_ATSSS_Ph2_cl
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2020-04-01-3gpp-ietf-ls-on-need-for-multi-path-quic-for-atsss-attachment-2.doc

    SP-200289_SP-200264_cleaned
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2020-04-01-3gpp-ietf-ls-on-need-for-multi-path-quic-for-atsss-attachment-3.docx

---

From: Matt Joras <[email-masked]>
Date: Wed, 1 Apr 2020 09:49:40 -0700

On Wed, Apr 1, 2020 at 9:16 AM Magnus Westerlund <magnus.westerlund=
[email-masked]> wrote:

> ---------- Forwarded message ----------
> Cc: The IETF Chair <[email-masked]>, The IESG <[email-masked]>,
> [email-masked], Gonzalo Camarillo <[email-masked]>
> Bcc:
> Cc: The IESG <[email-masked]>,The IETF Chair <[email-masked]>,Gonzalo
> Camarillo <[email-masked]>
> 3GPP has approved the study item on “Study on Access Traffic Steering,
> Switch and Splitting support in the 5G system architecture Phase 2” as part
> of Release 17.
>
> This study aims to investigate the extension of Rel-16 ATSSS feature
> enabling the support of steering, switching and splitting the traffic of a
> PDU session supported over multiple accesses (namely 3GPP and Non-3GPP
> access), including the following objective:
>
> -       Whether and how to support additional steering methods(s).
> Proposed solutions shall be based on IETF protocols or extension of such
> protocols (i.e. QUIC/MP-QUIC).
>
> A small complaint. The attached documents are in a proprietary format
(Microsoft Word) that doesn't really render properly in freely available
software. Secondly, the information therein is pretty "raw", appearing to
be some sort minutes with annotations. I don't know how we are supposed to
get any usable information from these document without being actively
involved in 3GPP.

> The work on the study has not yet started in 3GPP, and there are thus no
> agreed conclusions. . The goal is to enable steering, switching and
> splitting of traffic (primarily UDP) across multiple accesses, including
> latency sensitive and real time traffic. Therefore 3GPP is interested to
> receive regular feedback on progress and prioritization on the multipath
> extensions to QUIC.
>
> When multipath was discussed in Zurich, the opinion of the room seemed to
be that we should not be endeavoring on this work without an eminently
clear use case and required feeatures. This statement from 3GPP does not
yet describe a clear use case. Without an explicit problem statement and an
enumeration of the features that are going to be required of a multipath
QUIC extension, I don't think the WG should begin work designing such an
extension, having other useful work that can be done.

>
> 2. Actions:
> To IETF group:
> ACTION:         3GPP SA kindly requests IETF to take the above information
> into account when discussing future work in prioritizing the multipath work
> for QUIC.
>
>
Matt Joras

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 1 Apr 2020 14:26:13 -0500

Dear QUICsters,

I have a couple of hats, but NONE of them apply in this e-mail. So, just
speaking for me.

On Wed, Apr 1, 2020 at 11:50 AM Matt Joras <[email-masked]> wrote:

> [email-masked] <[email-masked]..org>> wrote:
>
>> QUIC WG,
>>
> software.

That's fair. PDF versions are attached. If I can provide some other format,
please let me know.

> Secondly, the information therein is pretty "raw", appearing to be some
> sort minutes with annotations. I don't know how we are supposed to get any
> usable information from these document without being actively involved in
> 3GPP.
>

Not quite "minutes", but "output of the most recent SA plenary meeting,
reflecting all agreed contributions" (don't ask). These are usually
distributed in "rm" (revision marks) and "cl" (clean) formats. You can
safely ignore the "rm" version unless you are curious about what changed
recently.

Yes, you're right, the most effective way for any SDO to work with the IETF
is by sending humans to the IETF to participate as individuals. I'm now
participating in 3GPP ATSSS work as well as in the IETF, and I don't think
I'm the only one, so at least some crossover exists.

>> receive regular feedback on progress and prioritization on the multipath
IMO, this is actually a victory - for most of the time that 3GPP has been
trying to work with the IETF, it's been from the CT ("stage 3") protocol
groups), where there was a very short timeline between contact with the
IETF and the deadline for inclusion of IETF technology in a specification.
This is coming from SA, which does requirements and architecture, and even
then, they're just getting started. So that kind of running start ought to
be helpful.

(As an AD, I was begging liaisons from 3GPP to send us what they are
thinking about earlier in the process. This looks like that finally
happened, so I'd hate for the IETF to drop it on the ground, but I'm not an
AD, so that doesn't matter now)

> extension, having other useful work that can be done.

I understand most of what's in
https://datatracker.ietf.org/meeting/interim-2020-quic-01/materials/minutes-interim-2020-quic-01-202002050930.html#multipath-extensions-for-quic,
although I wasn't at the interim, and I've had conversations about
multipath with Lars (which included some concerns that weren't included in
the Zurich discussions).

Is it worth me trying to write something up, explaining where the QUIC
working group was on this, the last time it was discussed? That could go
back to 3GPP as a formal liaison response, or I could just tell the ATSSS
people on a 3GPP conference call - whatever works.

Best,

Spencer

> 2. Actions:
>> To IETF group:
>> ACTION:         3GPP SA kindly requests IETF to take the above
>> information into account when discussing future work in prioritizing the
>> multipath work for QUIC.
>>
>>

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 02 Apr 2020 08:56:11 +0900

If I read this liaison correctly, this is effectively a request to keep 3GPP in the loop.  The request to do multipath is more implied than anything else.  I think that the explicit request is fine.  The implicit one maybe less so.

I suspect that work here would go faster if we had more input on the hard multipath problems.  That's not the spelling of the protocol (that's almost trivial, as demonstrated in proposals), but the assignment of different information elements paths that might have different characteristics.  What I heard in Zürich was that this was less of an engineering problem than a research one.

I'd be interested in having active engagement from people who have use cases. Specifically, I would seek to learn whether this is truly a research question or whether there are simplifying assumptions that might make the problem tractable for some use cases.

The more I think on this subject, the more that I think it was a mistake to charter to do multipath.  I don't believe that we properly answered the standard BoF questions with respect to this topic.

On Thu, Apr 2, 2020, at 01:15, Magnus Westerlund wrote:
> * Email.eml
> * smime.p7s

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 1 Apr 2020 21:05:36 -0500

Hi, Martin,

On Wed, Apr 1, 2020 at 6:56 PM Martin Thomson <[email-masked]> wrote:

> If I read this liaison correctly, this is effectively a request to keep
> 3GPP in the loop.  The request to do multipath is more implied than
> anything else.  I think that the explicit request is fine.  The implicit
> one maybe less so.
>
> I suspect that work here would go faster if we had more input on the hard
> multipath problems.  That's not the spelling of the protocol (that's almost
> trivial, as demonstrated in proposals), but the assignment of different
> information elements paths that might have different characteristics.  What
> I heard in Zürich was that this was less of an engineering problem than a
> research one.
>
> I'd be interested in having active engagement from people who have use
> cases. Specifically, I would seek to learn whether this is truly a research
> question or whether there are simplifying assumptions that might make the
> problem tractable for some use cases.
>
> The more I think on this subject, the more that I think it was a mistake
> to charter to do multipath.  I don't believe that we properly answered the
> standard BoF questions with respect to this topic.
>

You rang? :-)

Multipath has been in the QUIC charter since the first proposed version
(see https://datatracker.ietf.org/doc/charter-ietf-quic/00-00/), at my
urging, because I couldn't imagine chartering a new transport protocol in
2016 without multipath, given that we'd had to retrofit that into both TCP
and SCTP. So this might have been a mistake, but it's not a recent one, and
I made it (and achieved IETF consensus, of course :-)

When we chartered QUIC, I don't think we had a clear picture of how much
MP-TCP was going to depend on use cases in order to justify deployment. I
think that now, if we know anything about multipath, we know that.

I agree that this could be a research topic (in at least one of two or
three existing research groups).

I note that 3GPP has been relying on MP-TCP for ATSSS, and I hope that we
might be able to better than that, for their use case.

But we definitely have some talking to do with them about their use case.

Best,

Spencer, who hopes Mirja told Magnus and Martin that they can ALWAYS blame
the previous AD ;-)

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 02 Apr 2020 14:44:07 +1100

On Thu, Apr 2, 2020, at 13:05, Spencer Dawkins at IETF wrote:
> urging, because I couldn't imagine chartering a new transport protocol 
> in 2016 without multipath, given that we'd had to retrofit that into 
> both TCP and SCTP. So this might have been a mistake, but it's not a 
> recent one, and I made it (and achieved IETF consensus, of course :-)

Hi Spencer,

I don't think that this is about blame.  It's about learning that sometimes you aspire to achieve things that are out of your reach.

Like you, I thought that at least someone understood this problem well enough for us to do the work.  I didn't have that expertise myself, but I could see SCTP and MP-TCP.  I've used SCTP multipath myself, many years ago.  What I didn't know then was how narrow that usage was and how difficult the problem is in the general case.  I suspect that many people reached a similar conclusion.

We could absolutely set out to do multipath for failover, and we basically have already.  Similarly, multipath for maximizing throughput (what most people seem to want to measure in a lot of testing), seems like it might be achievable.  This sort of general purpose usage is the most difficult without having a good set assumptions backing the work.

If this is truly desirable, I would suggest that some of those companies listed as supporting the liaison statement could send some people to feed in requirements, refine the scope, make proposals, and work on solutions.  The IETF is unlikely to magically create something that will meet 3GPP needs and the speed of interaction via liaison statement isn't conducive to working through this.

We do have other history here: the failed BANANA BoF at IETF 97 covered this exact topic: https://datatracker.ietf.org/doc/minutes-97-banana/  That's at a different layer, but the same challenges exist.

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 2 Apr 2020 09:53:07 +0100

A couple of words on SCTP and DCCP... as a TSVWG Chair...

First, I'm also really grateful for sharing this vision with the IETF. 
It will help to know, and discuss, what other SDOs would like to do.

The IETF has indeed worked on transport support for failover between 
different paths. SCTP has specified path failover, and as recently as 
2016, published RFC7829.   I agree that failover seems close to 
something that QUIC mechanisms can offer, and I wonder if there is value 
in explicitly writing something short that calls out path failover as 
one service offered by QUIC?

Over the years, SCTP has also considered concurrent multipath sharing. 
There are interesting and many useful papers on this, with data and 
considerations of interactions with congestion control, but no IETF work 
was adopted. That isn't to say that couldn't happen, far from it. I'm 
just noting that so-far the concurrent use of multiple paths remains 
something where TSVWG is looking to understand the general requirements 
for the service and protocol mechanisms.

There were recent requests to work on DCCP with multipath, these 
continue to be discussed in TSVWG as individual contributions. (MP-TCP 
work moved to the TCPM WG).

Finally, I'll observe (with no "hat") that the forwarding of latency 
sensitive and real time traffic along a different path to other traffic 
(or to a different endpoint) is work where the IETF could likely help, 
although that may not necessarilly imply a need for work on concurrent 
multipath sharing of capacity.

Gorry Fairhurst

(One of several TSVWG Co-Chairs)

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 2 Apr 2020 11:25:44 +0200

Martin,

> On 2 Apr 2020, at 01:56, Martin Thomson <[email-masked]> wrote:
> 

Since the publication of Multipath TCP in RFC6824, we have accumulated experience in deploying and operating solutions that leverage multipath transport. Here are a few pointers to documents that describe these deployments. Several of the people that were involved in multipath TCP and also active in QUIC and can bring their experience to finalise multipath extensions to QUIC.

https://www.ietfjournal.org/multipath-tcp-deployments/ <https://www.ietfjournal.org/multipath-tcp-deployments/>
https://tools.ietf.org/html/rfc8041 <https://tools.ietf.org/html/rfc8041>
https://arxiv.org/abs/1907.04570 <https://arxiv.org/abs/1907.04570>

I can provide additional information for some of these use cases

> The more I think on this subject, the more that I think it was a mistake to charter to do multipath.  I don't believe that we properly answered the standard BoF questions with respect to this topic.

I disagree. Given the flexibility of QUIC, it would be a mistake to not consider multipath. TCP should have been a multipath protocol from day one.

Olivier

-- 

Disclaimer: https://www.tessares.net/mail-disclaimer/ 
<https://www.tessares.net/mail-disclaimer/>

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 2 Apr 2020 11:40:43 +0200

Hello Martin,

> On 2 Apr 2020, at 05:44, Martin Thomson <[email-masked]> wrote:
> 
> 
> We could absolutely set out to do multipath for failover, and we basically have already.  

We should not stop at connection migration. Multipath allows to use two paths during the connection migration, which brings benefits on smartphones based on the lessons learned from using MPTCP on Apple phones.

> Similarly, multipath for maximizing throughput (what most people seem to want to measure in a lot of testing), seems like it might be achievable.  

We can leverage all the work done with MPTCP. QUIC simplifies the problem because we do not need to take care of any kind of middlebox interference. Many research papers have tried to maximise throughput because this fits well in lab environments, but deployments have different objectives :
- on smartphones, deployments try to minimise the utilisation of the cellular network when the Wi-Fi is good enough (with different definitions of good enough depending on the application)
- in hybrid access networks that combine xDSL and 4G, deployments try to only use the 4G network when the DSL is full  

These objectives are reached by using packet schedulers and path managers that are specific to each use case but have not been standardised within the IETF. Concerning the packet schedulers, we started to document them in a generic manner in 
https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00 <https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00>
and planned to present the draft in Vancouver

> This sort of general purpose usage is the most difficult without having a good set assumptions backing the work.

I’m ready to work on such a document. Feel free to contact me if you’d like to participate.

> We do have other history here: the failed BANANA BoF at IETF 97 covered this exact topic: https://datatracker.ietf.org/doc/minutes-97-banana/  That's at a different layer, but the same challenges e…

Two different approaches have been discussed around IETF97. A network layer based solution in BANANAS and an MPTCP solution. The network layer solution did not gain enough interest. 

The MPTCP solution has been included in the first version of ATSSS and is used in Hybrid Access Networks. It has been initially developed in the mptcp working group and then moved to the tcpw working group. See https://datatracker.ietf.org/doc/draft-ietf-tcpm-converters/ <https://datatracker.ietf.org/doc/draft-ietf-tcpm-converters/>

Olivier

--

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 2 Apr 2020 10:01:30 -0500

Hi, Martin,

On Wed, Apr 1, 2020 at 10:44 PM Martin Thomson <[email-masked]> wrote:

> I don't think that this is about blame.  It's about learning that
> sometimes you aspire to achieve things that are out of your reach.
>

I was mostly kidding, but see below.

> Like you, I thought that at least someone understood this problem well
> enough for us to do the work.  I didn't have that expertise myself, but I
> could see SCTP and MP-TCP.  I've used SCTP multipath myself, many years
> ago.  What I didn't know then was how narrow that usage was and how
> difficult the problem is in the general case.  I suspect that many people
> reached a similar conclusion.
>
> We could absolutely set out to do multipath for failover, and we basically
> have already.  Similarly, multipath for maximizing throughput (what most
> people seem to want to measure in a lot of testing), seems like it might be
> achievable.  This sort of general purpose usage is the most difficult
> without having a good set assumptions backing the work.
>

I think that's exactly right.

> If this is truly desirable, I would suggest that some of those companies
> listed as supporting the liaison statement could send some people to feed
> in requirements, refine the scope, make proposals, and work on solutions.
> The IETF is unlikely to magically create something that will meet 3GPP
> needs and the speed of interaction via liaison statement isn't conducive to
> working through this.
>

Yeah, and I'll do what I can to drag others along. Honestly, having both
3GPP and IETF going virtual for at least a few months should help that -
one of the big reasons given for limited interaction for the past several
years has been the number of meetings on the 3GPP side, and meeting
schedule conflicts for 3GPP meetings with IETF meetings. It's a lot easier
to time-slice via e-mail.

> We do have other history here: the failed BANANA BoF at IETF 97 covered
> this exact topic: https://datatracker.ietf.org/doc/minutes-97-banana/
> That's at a different layer, but the same challenges exist.
>

Quoting from https://tools.ietf.org/html/draft-irtf-panrg-what-not-to-do-07
,

1.  Introduction

   At the first meeting of the Path Aware Networking Research Group
   [PANRG], at IETF 99 [PANRG-99], Oliver Bonaventure led a discussion
   of "A Decade of Path Awareness" [PATH-Decade], on attempts, which
   were mostly unsuccessful for a variety of reasons, to exploit Path
   Aware techniques and achieve a variety of goals over the past decade.
   At the end of that discussion, two things were abundantly clear.

   o  The Internet community has accumulated considerable experience
      with many Path Aware techniques over a long period of time, and

   o  Although some path aware techniques have been deployed (for
      example, Differentiated Services, or DiffServ [RFC2475]), most of
      these techniques haven't seen widespread adoption and deplyment.
      Even "successful" techniques like DiffServ can face obstacles that
      prevents wider usage.  The reasons for non-adoption and limited
      adoption and deployment are many, and are worthy of study.

So, yeah, there's a LOT of history here (and I notice that BANANA isn't one
of the technologies mentioned in that draft, so there's more than I was
remembering).

Best,

Spencer

---

From: Ted Hardie <[email-masked]>
Date: Thu, 2 Apr 2020 10:32:37 -0700

On Thu, Apr 2, 2020 at 8:02 AM Spencer Dawkins at IETF <
[email-masked]> wrote:

>> We could absolutely set out to do multipath for failover, and we
>> basically have already.  Similarly, multipath for maximizing throughput
>> (what most people seem to want to measure in a lot of testing), seems like
>> it might be achievable.  This sort of general purpose usage is the most
>> difficult without having a good set assumptions backing the work.
>>
>
I am not a multipath expert, but I have built and tested a system that used
multiple paths to achieve its throughput goals.  That was 10 years ago, and
it used DTLS (
https://www.ietf.org/archive/id/draft-hardie-mdtls-session-00.txt has a
brief sketch of the protocol mechanics) to create a session layer on top of
multiple cellular links.

One thing that experience taught me was that even a discrete goal like
"maximize throughput" is not sufficiently granular to express what an
application needs.  That requires a good bit more detail on what the
tolerance for re-ordering is, what buffer capacities you can reasonably
expect, and what the application's tolerances are for jitter and delay.
We were building a system for transmitting 3-d video over multiple cellular
links, primarily to news rooms.  Given the cellular technology of the time
and the different network characteristics of the different paths, that
meant we had to have substantial playout buffers to handle reordering and
odd packet inter-arrival times.  That happened to fit in the tolerance of
our deployment because the buffer time and satellite delays were similar,
and the news readers could manage their reaction times in the same ways.
For lots of other cases, the playout buffer wouldn't have mattered at all
and in others it would have resulted in failure.

I don't want to generalize too much from that one data point, but it hints
to me that you don't really want to build a multipath transport around a
few use cases.  The chances that you are starting from an outlier are
pretty high.  What I think you want to do instead is build facilities into
the protocol that allow you to build multipath applications on top of
them.  If I were re-building the system from 10 years ago on top of QUIC, I
would be looking for a way to bundle connection IDs into session sets, so
that the data passed from QUIC up to the application could be marshaled
into the same application-level playout buffer.  Other applications might
use similar facilities in very different ways.

To put this slightly differently, I think if we're going to tackle how QUIC
handles multipath, I think we have to seriously consider how much of the
work is done in the application layer above it.  We may be able to handle a
wider variety of use cases with fewer primitives if we presume applications
are cooperating to achieve the goal.  For the ones which will remain
ignorant of it, we can obviously still provide failover.  It's not clear to
me, though, how much more we can do without restricting the options for the
ones that do have multipath aware applications.

Just my two cents, obviously,

Ted

---

From: "Qing An" <[email-masked]>
Date: Fri, 03 Apr 2020 11:34:27 +0800

>We can leverage all the work done with MPTCP. QUIC simplifies the problem because we do not need to take care of any kind of middlebox interference. Many research papers have tried to maximise >throughput because this fits well in lab environments, but deployments have different objectives :
>These objectives are reached by using packet schedulers and path managers that are specific to each use case but have not been standardised within the IETF. Concerning the packet schedulers, we >started to document them in a generic manner in 
>https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00

https://datatracker.ietf.org/doc/draft-an-multipath-quic-traffic-distribution/ describes several key components for Multipath QUIC traffic distribution. 

            >>If this is truly desirable, I would suggest that some of those companies listed as supporting the liaison statement could send some people to feed in requirements, refine the scope, make 
            >>proposals, and work on solutions.  The IETF is unlikely to magically create something that will meet 3GPP needs and the speed of interaction via liaison statement isn't conducive to working 
            >>through this.

>I’m ready to work on such a document. Feel free to contact me if you’d like to participate.
I'd like to participate and co-work on this document. 

Qing

---

From: Lars Eggert <[email-masked]>
Date: Fri, 3 Apr 2020 09:31:26 +0300

Hi,

what we have done with MPTCP is build a system that is reasonably good at pooling the capacity of multiple paths for one connection, *if* these paths have reasonably similar characteristics, esp. RTT.

With paths with dissimilar RTTs, we can still pool the capacity, but only make it available for use to the application at pretty much the maximum of the individual RTTs. That’s still somewhat useful if you care about bulk throughput. It’s basically useless if you care about small-transfer latencies, which the web has been gravitating to for years. (Remember that the MPTCP design is ten years old - from a different world. Bandwidths have gone up a lot, making capacity pooling less urgent, but late cues have not decreased much.)

To make multi path useful to today’s mostly latency-sensitive applications, we need a path scheduler that can pool (some of) the capacity of multiple dissimilar paths at some effective RTT that is much lower than the maximum of the individual ones. I am not aware of any proposed scheduler - for MPTCP or otherwise - that achieves that.

Even if we change the argument from pooling latency to increasing resilience, I still doubt that increased resilience at the cost of much-increased late zu is a worthwhile trade off for many apps.

So in short, I understand why the community is questioning if QUIC needs to be able to do multi path, and in what form.

Lars

-- 
Sent from a mobile device; please excuse typos.

> On Apr 2, 2020, at 20:33, Ted Hardie <[email-masked]> wrote:
> 
> 
>> On Thu, Apr 2, 2020 at 8:02 AM Spencer Dawkins at IETF <[email-masked]> wrote:
> 
>> 
> I am not a multipath expert, but I have built and tested a system that used multiple paths to achieve its throughput goals.  That was 10 years ago, and it used DTLS (MailScanner has detected a possible fraud attempt from "www.ietf.org" claiming to be https://www..ietf.org/archive/id/draft-hardie-mdtls-session-00.txt has a brief sketch of the protocol mechanics) to create a session layer on top of multiple cellular links.  
> 
> One thing that experience taught me was that even a discrete goal like "maximize throughput" is not sufficiently granular to express what an application needs.  That requires a good bit more detail on what the tolerance for re-ordering is, what buffer capacities you can reasonably expect, and what the application's tolerances are for jitter and delay.   We were building a system for transmitting 3-d video over multiple cellular links, primarily to news rooms.  Given the cellular technology of the time and the different network characteristics of the different paths, that meant we had to have substantial playout buffers to handle reordering and odd packet inter-arrival times.  That happened to fit in the tolerance of our deployment because the buffer time and satellite delays were similar, and the news readers could manage their reaction times in the same ways.  For lots of other cases, the playout buffer wouldn't have mattered at all and in others it would have resulted in failure.  
> 
> I don't want to generalize too much from that one data point, but it hints to me that you don't really want to build a multipath transport around a few use cases.  The chances that you are starting from an outlier are pretty high.  What I think you want to do instead is build facilities into the protocol that allow you to build multipath applications on top of them.  If I were re-building the system from 10 years ago on top of QUIC, I would be looking for a way to bundle connection IDs into session sets, so that the data passed from QUIC up to the application could be marshaled into the same application-level playout buffer.  Other applications might use similar facilities in very different ways.  
> 
> To put this slightly differently, I think if we're going to tackle how QUIC handles multipath, I think we have to seriously consider how much of the work is done in the application layer above it.  We may be able to handle a wider variety of use cases with fewer primitives if we presume applications are cooperating to achieve the goal.  For the ones which will remain ignorant of it, we can obviously still provide failover..  It's not clear to me, though, how much more we can do without restricting the options for the ones that do have multipath aware applications.
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 3 Apr 2020 09:55:30 +0200

Lars,
> 
> what we have done with MPTCP is build a system that is reasonably good 
> at pooling the capacity of multiple paths for one connection, *if* these 
> paths have reasonably similar characteristics, esp. RTT.
 >
> With paths with dissimilar RTTs, we can still pool the capacity, but 
> only make it available for use to the application at pretty much the 
> maximum of the individual RTTs. That’s still somewhat useful if you care 
> about bulk throughput. It’s basically useless if you care about 
> small-transfer latencies, which the web has been gravitating to for 
> years. (Remember that the MPTCP design is ten years old - from a 
> different world. Bandwidths have gone up a lot, making capacity pooling 
> less urgent, but late cues have not decreased much.)

This corresponds to the hybrid access network case that pools bandwidth.

> To make multi path useful to today’s mostly latency-sensitive 
> applications, we need a path scheduler that can pool (some of) the 
> capacity of multiple dissimilar paths at some effective RTT that is much 
> lower than the maximum of the individual ones. I am not aware of any 
> proposed scheduler - for MPTCP or otherwise - that achieves that.

AFAIK, this is exactly what Apple is doing with MPTCP. They call this 
the "interactive mode" and the stack dynamically selects the path (wifi 
or cellular) having the best performance (rtt, losses) for interactive 
applications. See https://developer.apple.com/videos/play/wwdc2017/707/

> Even if we change the argument from pooling latency to increasing 
> resilience, I still doubt that increased resilience at the cost of 
> much-increased late zu is a worthwhile trade off for many apps.

AFAIK, Apple calls that the "handover mode", it is probably less useful 
that the interactive one. Maybe Apple engineers who are on this list can 
comment and provide more details.

I don't think that Apple uses MPTCP to pool bandwidth. They use MPTCP to 
improve user experience and mainly latency.

Olivier

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 3 Apr 2020 09:35:57 +0100

On 03/04/2020 08:55, Olivier Bonaventure wrote:
> Lars,
>>
>> what we have done with MPTCP is build a system that is reasonably 
>> good at pooling the capacity of multiple paths for one connection, 
>> *if* these paths have reasonably similar characteristics, esp. RTT.
> >
>> maximum of the individual RTTs. That’s still somewhat useful if you 
>> care about bulk throughput. It’s basically useless if you care about 
>> different world. Bandwidths have gone up a lot, making capacity 
>> pooling less urgent, but late cues have not decreased much.)
>
>> capacity of multiple dissimilar paths at some effective RTT that is 
>> much lower than the maximum of the individual ones. I am not aware of 
>> any proposed scheduler - for MPTCP or otherwise - that achieves that.
>
> the "interactive mode" and the stack dynamically selects the path 
> (wifi or cellular) having the best performance (rtt, losses) for 
> interactive applications. See 
> https://developer.apple.com/videos/play/wwdc2017/707/
>
> AFAIK, Apple calls that the "handover mode", it is probably less 
> useful that the interactive one. Maybe Apple engineers who are on this 
> list can comment and provide more details.
>
> I don't think that Apple uses MPTCP to pool bandwidth. They use MPTCP 
> to improve user experience and mainly latency.
>
>
> Olivier

When you says "select" do you mean "change to send data using the new 
path"?

I f so, I think this is close in operation to a "path failover", even if 
it uses a trigger metric that isn't just a loss of connectivity, but the 
result of some other path metric - e.g. significantly lower RTT, 
availanility of PvD information, ... . In my mind, this is quite 
different to a proposal for somehow "sharing" traffic/capacity between 
several paths, which means understanding capacity and the impacts of 
variability in path characteristics, the impact of traffic on pooled 
bandwidth, etc.

It still does pose some issues in estimating the capacity on the "new" 
path, but ... The mechanisms to do this seem to exist within the design 
expected of QUIC.

Gorry

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 3 Apr 2020 11:09:06 +0200

Gorry,
> When you says "select" do you mean "change to send data using the new 

This is a packet scheduler decision. When new data needs to be sent and 
there are two subflows, the scheduler selects the most suitable one. I 
think that Apple uses its WiFi assist monitor to inform the packet 
scheduler of the most suitable path on the client. On paper that 
describes a similar approach is

https://inl.info.ucl.ac.be/publications/tuning-multipath-tcp-interactive-applications-smartphones.html

In MPTCP, it is difficult for the client and the server to easily 
exchange information about the quality of the paths due to middlebox 
interference and a set of heuristics are used. For example, the server 
replies on the subflow chosen by the smartphone to send the request.

With MPQUIC, we can use ping frames to measure the quality of paths and 
react accordingly. In MPTCP, this would require a complex machinery.

> availanility of PvD information, ... .

Apple's deployment shows that maintaining the two paths is useful 
because there are situations where latency sensite applications can use 
both paths, e.g. during the handover when the connectivity is weak but 
not yet broken.

> In my mind, this is quite 

The same techniques apply in these different cases.
> path, but ... The mechanisms to do this seem to exist within the design 

QUIC is clearly both cleaner and more powerful than MPTCP for this. You 
can find a detailed discussion on this in Quentin De Coninck's PhD 
thesis which is available from https://www.multipath-quic.org

Olivier

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 3 Apr 2020 10:17:38 +0100

On 03/04/2020 10:09, Olivier Bonaventure wrote:
>>>> With paths with dissimilar RTTs, we can still pool the capacity, 
>>>> but only make it available for use to the application at pretty 
>>>> much the maximum of the individual RTTs. That’s still somewhat 
>>>> useful if you care about bulk throughput. It’s basically useless if 
>>>> you care about small-transfer latencies, which the web has been 
>>>> gravitating to for years. (Remember that the MPTCP design is ten 
>>>> years old - from a different world. Bandwidths have gone up a lot, 
>>>> making capacity pooling less urgent, but late cues have not 
>>>> decreased much.)
>>>
>>>> much lower than the maximum of the individual ones. I am not aware 
>>>> of any proposed scheduler - for MPTCP or otherwise - that achieves 
>>>> that.
>>>
>>> AFAIK, this is exactly what Apple is doing with MPTCP. They call 
>>> this the "interactive mode" and the stack dynamically selects the 
>>> path (wifi or cellular) having the best performance (rtt, losses) 
>>> for interactive applications. See 
>>> useful that the interactive one. Maybe Apple engineers who are on 
>>> this list can comment and provide more details.
>>>
>>> I don't think that Apple uses MPTCP to pool bandwidth. They use 
>>> MPTCP to improve user experience and mainly latency.
>>>
>>>
>>> Olivier
>>
> This is a packet scheduler decision. When new data needs to be sent 
> and there are two subflows, the scheduler selects the most suitable 
> one. I think that Apple uses its WiFi assist monitor to inform the 
> packet scheduler of the most suitable path on the client. On paper 
> that describes a similar approach is
>
> With MPQUIC, we can use ping frames to measure the quality of paths 
> and react accordingly. In MPTCP, this would require a complex machinery.
>
>
>> I f so, I think this is close in operation to a "path failover", even 
>> if it uses a trigger metric that isn't just a loss of connectivity, 
>> but the result of some other path metric - e.g. significantly lower 
>> RTT, availanility of PvD information, ... .
>
> because there are situations where latency sensite applications can 
> use both paths, e.g. during the handover when the connectivity is weak 
> but not yet broken.
>
>> In my mind, this is quite different to a proposal for somehow 
>> "sharing" traffic/capacity between several paths, which means 
>> understanding capacity and the impacts of variability in path 
>> characteristics, the impact of traffic on pooled bandwidth, etc.
>
>> It still does pose some issues in estimating the capacity on the 
>> "new" path, but ... The mechanisms to do this seem to exist within 
>> the design expected of QUIC.
>
> QUIC is clearly both cleaner and more powerful than MPTCP for this. 
> You can find a detailed discussion on this in Quentin De Coninck's PhD 

Thanks for the reply, this helps confirm my thoughts on mechansims.

Gorry

---

From: <[email-masked]>
Date: Sun, 5 Apr 2020 17:05:56 +0000

Dear IETF community,

as one the co-authors of the 3GPP LS request and also being active at IETF mainly in the multipath transport area, I want to share some views from my side.

ATSSS is a fantastic tool to change customer connectivity experience by simultaneously use cellular and WiFi (multi-connectivity). Hence, it promises best reliability and performance. Especially the splitting mode (per-packet multipath decision) of ATSSS gives in this context highest flexibility and highest granularity for tuning the QoE. The almost finalized Rel. 16 defines the well known and mature MPTCP, but lacks in supporting UDP and in particular its raising share of QUIC traffic.

A logical step is now to close this gap in the 3GPP Rel. 17 phase, but also improving the documentation of multipath components.
Who cannot do this better than the multipath transport experts at IETF?

In general, I see IETF in the position to well document
- Protocols
- path manager
- traffic distribution scheduler (e.g. https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00)
- re-ordering mechanism
for multipath purposes which can be simply referenced by 3GPP (ATSSS), BBF (Hybrid Access) or any other SDO.

In case QUIC is seen as the protocol of choice to fulfill the demand on the LS request, the following extensions, already considered in the QUIC WG, might be mandatory or useful

- MP support
- Datagram support for flexible reliability and latency sensitivity demands
- Tunnel/Proxying mechanism (e.g. MASQUE)

Things that might be of further interest to be investigated (e.g. at ICCRG) independent from a particular protocol
- Congestion control over congestion control (e.g. E2E QUIC over MP-QUIC or E2E QUIC over MP-DCCP)
- Encryption over encryption (e.g. QUIC over MP-QUIC over 3GPP IPsec tunnel (untrusted path))

Independent from a potential (MP-)QUIC solution, SCTP and MP-DCCP were also mentioned as candidates in this ongoing email conversation. I would say 3GPP does not necessarily expect an answer from IETF which only includes a QUIC based solution. Rather it is the believe that IETF feedbacks the solution which fits best and also considers the tough timeline of 3GPP. This might also lead to an interim solution other than QUIC, until a final QUIC solution is mature. I strongly believe that (MP-)QUIC will in future be capable of supporting every requirements of the 3GPP ATSSS LS, it's just a matter of time, which might be the crux...

I want also to point out, that the MP-DCCP framework draft with its formulated requirements (https://tools.ietf.org/html/draft-amend-tsvwg-multipath-framework-mpdccp-01#section-2) most probably addresses most of the requirements of the 3GPP ATSSS LS, since it is made for ATSSS purposes (see Abstract/Introduction). Furthermore https://datatracker.ietf.org/meeting/104/materials/slides-104-tsvwg-sessb-43-markus-amend-multipath-dccp-00 might be worth reading, since it addresses the general challenges seen but also the challenges in particular using MP-QUIC (slide 3-5, 9)

BR

Markus

From: QUIC <[email-masked]> On Behalf Of Spencer Dawkins at IETF
Sent: Donnerstag, 2. April 2020 17:02
To: Martin Thomson <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: [Fwd: New Liaison Statement, "LS on need for Multi-Path QUIC for ATSSS"]

Hi, Martin, 

On Wed, Apr 1, 2020 at 10:44 PM Martin Thomson <mailto:[email-masked]> wrote:

[...] 

>> If this is truly desirable, I would suggest that some of those companies listed as supporting the liaison statement could send some people to feed in requirements, refine the scope, make proposals…

[...]

---

From: Lars Eggert <[email-masked]>
Date: Wed, 29 Apr 2020 17:27:29 +0300

QUIC WG participants,

the chairs have discussed a response to this statement and plan on sending the message below. If you have comments, please respond to this e-mail by May 13.

Thanks,
Lars

---

Thank you for your input to our specifications.

Multipath capabilities for QUIC are currently under active discussion in the
IETF's QUIC WG. Several individual proposals have been made, but the group is
also considering whether the already-specified connection migration capabilities
are sufficient to cover the majority of use cases.

We encourage 3GPP to detail their requirements for QUIC multipath capabilities
in an Internet-Draft, especially if the already-specified connection migration
capabilities are deemed insufficient. 3GPP's active involvement in any multipath
QUIC standardization would be the best way to remain informed of the progress of
any such work in the IETF.

Kind regards,
Mark Nottingham, Lucas Pardue and Lars Eggert, QUIC Working Group chairs

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 29 Apr 2020 12:57:35 -0500

Hi, Lars, et. al.

On Wed, Apr 29, 2020 at 9:29 AM Lars Eggert <[email-masked]> wrote:

> QUIC WG participants,
>
> the chairs have discussed a response to this statement and plan on sending
> the message below. If you have comments, please respond to this e-mail by
> May 13.
>
> Thanks,
> Lars
>
> ---
>

This seems very reasonable.

I have one stylistic comment, which is that I believe asking "3GPP to
detail their requirements" is begging for a long-RTT conversation in the
form of continued liaisons from quarterly SA plenaries (next SA plenary
ends July 3, then mid-September, then December). Perhaps asking "We
encourage 3GPP participants to provide an Internet-Draft that will help us
understand the considerations you'd like us to keep in mind, especially if
the already-specified connection migration capabilities are deemed
insufficient".

Gonzalo Camarillo and Lionel Morand can correct me, of course, but I
believe that SA participants can respond more more quickly if they don't
have to bounce between 3GPP SA1 ("requirements") and SA2 ("architecture")
in order to answer a question or express an opinion to the IETF.

Best,

Spencer