Subject: What to do about multipath in QUIC

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 4 Nov 2020 17:38:13 -0600

Lars said that if we wanted to move forward on multipath in QUIC, we should
be talking about multipath in QUIC on the mailing list, so this is my
suggestion as a starting point, based on my understanding of where we ended
up after the October virtual interim meeting and resulting threads on the
mailing list.

It's pretty clear that a lot of people have deploying and testing
connection migration as their priority in the near future, and we should
not distract them from that worthy task.

It's clear that at least some people think that connection migration onto a
new connection that has already been validated is a lightweight operation.
Deploying and testing connection migration will be a good basis for
verifying that theory.

It's clear that there are different ways of thinking about multipath in
QUIC - https://datatracker.ietf.org/doc/draft-deconinck-quic-multipath/ is
the proposal I have the most experience with, but Yanmei was presenting
https://datatracker.ietf.org/doc/draft-an-multipath-quic/ at the virtual
interim, and Christian submitted
https://datatracker.ietf.org/doc/draft-huitema-quic-mpath-option/ a day or
two before the virtual interim. That leads me to make two suggestions:

   - That we get experience with the proposals that we already have, and
   any proposals that pop up in the meantime.
   - That we discuss that experience and work on coming to a consensus
   about how multipath should work before moving forward.
   - That we publish (one or more) multipath proposals as Experimental, if
   and when that's the right thing to do.

Thoughts?

Best,

Spencer

---

From: Martin Duke <[email-masked]>
Date: Thu, 5 Nov 2020 12:59:43 -0800

Hi Spencer,

I shy away from deciding "how multipath should work." I think we should
figure how much protocol we need for the experimenters to figure out how
multipath should work. But maybe that's what you meant.

On Wed, Nov 4, 2020 at 3:39 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> Lars said that if we wanted to move forward on multipath in QUIC, we
> should be talking about multipath in QUIC on the mailing list, so this is
> my suggestion as a starting point, based on my understanding of where we
> ended up after the October virtual interim meeting and resulting threads on
> the mailing list.
>
> It's clear that at least some people think that connection migration onto
> a new connection that has already been validated is a lightweight
> operation. Deploying and testing connection migration will be a good basis
> for verifying that theory.
>
> QUIC - https://datatracker.ietf.org/doc/draft-deconinck-quic-multipath/
> is the proposal I have the most experience with, but Yanmei was presenting
> https://datatracker.ietf.org/doc/draft-huitema-quic-mpath-option/ a day
> or two before the virtual interim. That leads me to make two suggestions:
>
>    - That we publish (one or more) multipath proposals as Experimental,
>    if and when that's the right thing to do.
>
> Thoughts?
>
> Best,
>
> Spencer
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 5 Nov 2020 17:14:26 -0600

Hi, Martin,

> multipath should work. But maybe that's what you meant.

If I didn't mean that, I should have (especially after taking a closer look
at https://datatracker.ietf.org/doc/draft-huitema-quic-mpath-option/ today.
I think "how much protocol we need to figure out how multipath could work"
is a great revision to my third second bullet point.

Thanks for asking about that.

Best,

Spencer

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 5 Nov 2020 21:01:03 -0800

Hi Spencer,

I agree with your analysis that many of us think that our priority ought to
be to test our existing multipath capabilities (connection migration) first
before building more multipath capabilities. Which is why I agree with your
first suggestion, but not the other two.

> I don't understand this (and I've read your response to Martin's similar
comment). How is this a smaller work item than working on general-purpose
multipath?

> This will require the wg to adopt these proposals and engage on them.

I don't understand how the second and third suggestions follow from your
first suggestion of "get experience with the proposals that we already
have, and any proposals that pop up". This is what I want to see, but that
also means that we don't rush into adopting and working on proposals yet.

I want to remind folks that we need to have bandwidth in the wg for the
items that we _need_ to work on, and importantly, for experience that we
will have to share about our early QUIC and HTTP/3 deployments. We need to
have some room for it.

I've been asking this, and I'll do it again: Can someone please explain
what the urgency is in working on general-purpose multipath? Why are we not
waiting for deployment experience with existing multipath mechanisms in
QUICv1 before considering what to do next?

- jana

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Nov 2020 09:29:49 -0600

Hi, Jana,

Thanks for the thoughts. I have a couple of responses below, but would love
for other people to be chiming in, because I barely speak for Spencer,
these days.

On Thu, Nov 5, 2020 at 11:01 PM Jana Iyengar <[email-masked]> wrote:

> Hi Spencer,
>
> I agree with your analysis that many of us think that our priority ought
> to be to test our existing multipath capabilities (connection migration)
> first before building more multipath capabilities. Which is why I agree
> with your first suggestion, but not the other two.
>

I'm actually happy, if the path forward has "continue to work on multipath
proposals on the QUIC mailing list" as its first step.

I went further, but the second and third bullets were speculative, and
would depend on the QUIC working group, the QUIC working group chairs, and
the responsible AD deciding what to do at some point in the future. Further
direction on the path forward can be picked out along the way.

> comment). How is this a smaller work item than working on general-purpose

Thanks for asking the question this way.

One of the things I took away from the virtual interim meeting on
multipath, was that at least some of us have been talking about a
general-purpose multipath solution, but the use cases presented at the
virtual interim were different enough that we might need more than one
multipath solution.

Some of the use cases we talked about seemed to assume deep knowledge of
applications and of path characteristics, that would be difficult to meet
with a general-purpose multipath solution. Something like
https://datatracker.ietf.org/doc/draft-huitema-quic-mpath-option/ might be
all that they need.

Other use cases, especially the ones involving bandwidth aggregation, might
be a better fit for a general-purpose multipath solution.

But the point is, we don't know yet. So having a place that interested
parties can continue to discuss "multipath" seems most important.

So, I think the "how much protocol we need to figure out how multipath
could work" formulation I suggested to Martin is a reasonable proposal
here.

> will have to share about our early QUIC and HTTP/3 deployments. We need to

I trust the working group chairs to rate-limit adoptions in the QUIC
working group. Looking back, we didn't even adopt
draft-pauly-quic-datagram-00, submitted in 2018, as a working group item
until February 2020, so I think everyone understands that the QUIC
community has to have room for new work, for it to move forward.

> QUICv1 before considering what to do next?

As above - I think getting and understanding deployment experience with
"existing multipath mechanisms in QUICv1", which I'm reading as "connection
migration", to be a really important part of this discussion. I honestly
suspect that will meet the needs of many, but not all, use cases.

As an aside - I submitted a quick -00 draft from the Github repo at
https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath
on Monday (I-D cutoff), and posted a pointer on this mailing list, but I've
continued typing, and the current version in the repo now says this in
Section 2.2.4.

   One important output from the QUIC interim meeting was the
   recognition that some participants view "Traffic Switching" as
   something like "protection switching" from an active path to a
   standby path, that doesn't happen often, while other participants
   view QUIC connection migration as a way of responding frequently to
   changing path conditions.  This will be an important part of the
   conversation about multipath QUIC.

   If a QUIC endpoint opens and validates two or more connections, no
   additional setup or signaling is required for a sender to switch
   paths - that can begin with the next packet queued for transmission.
   This would allow "Traffic Splitting" for unordered QUIC datagram
   packets [I-D.ietf-quic-datagram], with an upper-layer mechanism
   providing flow ordering if that is needed.

That's an example of the kind of discussion I'm hoping that we continue to
have.

Again, thanks for your thoughts. I find them very helpful.

Best,

Spencer

p.s. Issues and PRs are, of course, welcomed in
https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath
...

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 6 Nov 2020 09:31:45 -0600

Hi Jana,

> QUICv1 before considering what to do next?

There is no existing multipath mechanisms in QUICv1. Period.
Please refer to my mail.

Behcet

Behcet

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 6 Nov 2020 16:16:32 +0000

Hi Behcet,

> Please refer to my mail.

That's a strong assertion. I'm not sure if members of the QUIC WG would
agree. I don't know what email you are referring to, for the sake of
discussion have you got a link or could you expand on why you think that
QUIC v1 doesn't support multiple paths?

Cheers,
Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Nov 2020 10:42:05 -0600

For what it's worth,

> QUIC v1 doesn't support multiple paths?

I don't remember anyone characterizing connection migration as "support for
multipath", before Jana's post on October 27, at
https://mailarchive.ietf.org/arch/msg/quic/AhDWbrNYsdevwq9COGeGc2YK168/,
but I could certainly have missed that - extracting from Jana's post:

"We have multipath in QUIC. It may not be enough, but we agreed in this wg
that this would be a good first cut to have. I believe this satisfies the
charter text as we have it right now. We've just finished building the use
of multipath in QUIC, and I'd like us to get some experience with it before
considering how to expand its scope or how to improve it further."

So, at least, I think that's what we're talking about - connection
migration from one path to another, as a type of "support for multipath" in
QUIC.

>From my to Jana in this thread:

Best,

Spencer

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 6 Nov 2020 17:17:01 +0000

Hey Spencer,

What I'm seeing (as an individual) is that the term multipath is too
nebulous.

There's a spectrum of use cases that can benefit from the use of
independent paths while using the same logical QUIC connection. On one end,
multiple paths are used during the lifetime of a single QUIC connection. On
the other end multipaths paths are kept alive (albeit maybe quiescent) for
redundancy or bandwidth aggregation.

If the connection migration feature can satisfy some part of the use case
range, then I think it's fair to say that QUIC supports "multipath use
cases", even if we don't  explicitly say there is a multipath protocol
feature. Maybe we didn't make this observation earlier but that doesn't
change the fact that path independence and migration is in the DNA of QUIC.

If folks want stuff on the other end of the spectrum, then it can help to
map how the additional possible technological solutions satisfy the use
cases. We should probably define some more precise terminology; getting
clarity now will help avoid "no true Scotsman" scenarios if people start
writing I-Ds and experimenting. This is especially important if any
proposals end up substituting or supplanting the existing connection
migration feature in the core QUIC.

I think this aligns with some of your thinking Spencer. Others could
disagree on some of my points. I just think there's a lot of shades of grey
in this topic.

Cheers
Lucas

On Fri, Nov 6, 2020 at 4:42 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> I don't remember anyone characterizing connection migration as "support
> for multipath", before Jana's post on October 27, at

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 6 Nov 2020 18:45:12 +0100

No doubt there are some overlaps between multipath and migration, but I think it would help the discussion to make a clear distinction. Migration means traffic moves majorily on one path at a time, and multipath means that traffic predominantly moves on multipath paths at a time.

Where this is an overlap is where multipath predominantly sends on one channel but can with short notice choose to send on another or on several path based on prevailing conditions. This is broader than migration where path availability seems to be the primary selection criteria, whereas multipath may choose on latency, bandwidth etc. - of course migration can also do that to an extend, which is why it is not clear cut.

Still migration does not allow for sustained concurrent traffic on multiple paths, while multipath does.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 6 November 2020 at 18.17.33, Lucas Pardue ([email-masked]) wrote:

Hey Spencer,

What I'm seeing (as an individual) is that the term multipath is too nebulous.

There's a spectrum of use cases that can benefit from the use of independent paths while using the same logical QUIC connection. On one end, multiple paths are used during the lifetime of a single QUIC connection. On the other end multipaths paths are kept alive (albeit maybe quiescent) for redundancy or bandwidth aggregation.

If the connection migration feature can satisfy some part of the use case range, then I think it's fair to say that QUIC supports "multipath use cases", even if we don't  explicitly say there is a multipath protocol feature. Maybe we didn't make this observation earlier but that doesn't change the fact that path independence and migration is in the DNA of QUIC.  

If folks want stuff on the other end of the spectrum, then it can help to map how the additional possible technological solutions satisfy the use cases. We should probably define some more precise terminology; getting clarity now will help avoid "no true Scotsman" scenarios if people start writing I-Ds and experimenting. This is especially important if any proposals end up substituting or supplanting the existing connection migration feature in the core QUIC.

I think this aligns with some of your thinking Spencer. Others could disagree on some of my points. I just think there's a lot of shades of grey in this topic.

Cheers
Lucas

On Fri, Nov 6, 2020 at 4:42 PM Spencer Dawkins at IETF <[email-masked]> wrote:
For what it's worth, 

On Fri, Nov 6, 2020 at 10:16 AM Lucas Pardue <[email-masked]> wrote:
Hi Behcet,

That's a strong assertion. I'm not sure if members of the QUIC WG would agree. I don't know what email you are referring to, for the sake of discussion have you got a link or could you expand on why you think that QUIC v1 doesn't support multiple paths?

I don't remember anyone characterizing connection migration as "support for multipath", before Jana's post on October 27, at https://mailarchive.ietf.org/arch/msg/quic/AhDWbrNYsdevwq9COGeGc2YK168/, but I could certainly have missed that - extracting from Jana's post:

So, at least, I think that's what we're talking about - connection migration from one path to another, as a type of "support for multipath" in QUIC.

As an aside - I submitted a quick -00 draft from the Github repo at https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath on Monday (I-D cutoff), and posted a pointer on this mailing list, but I've continued typing, and the current version in the repo now says this in Section 2.2.4.

That's an example of the kind of discussion I'm hoping that we continue to have. 

Best,

Spencer

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Nov 2020 12:52:25 -0600

Hi, Lucas,

> What I'm seeing (as an individual) is that the term multipath is too

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> disagree on some of my points. I just think there's a lot of shades of grey

I think that last part is exactly right.

Best,

Spencer

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Nov 2020 13:02:05 -0600

(I said I would try to post a bunch in this thread. Apparently I'm failing)

On Fri, Nov 6, 2020 at 11:45 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> No doubt there are some overlaps between multipath and migration, but I
> think it would help the discussion to make a clear distinction. Migration
> means traffic moves majorily on one path at a time, and multipath means
> that traffic predominantly moves on multipath paths at a time.
>
> Where this is an overlap is where multipath predominantly sends on one
> channel but can with short notice choose to send on another or on several
> path based on prevailing conditions. This is broader than migration where
> path availability seems to be the primary selection criteria, whereas
> multipath may choose on latency, bandwidth etc. - of course migration can
> also do that to an extend, which is why it is not clear cut.
>

I'm not sure what the protocol difference is between the two cases. I
understand the difference between sending on one path and sending on
another path, but once you've started sending on two (or more) paths, I'm
not sure what the protocol requirement is to stop sending on one of them.

I'm even more confused if we're talking about sending datagrams, rather
than reliable streams.

I understand why a scheduler might want to limit itself to sending on one
path in normal operation, but I'm trying to understand a different point.

Does my question make sense?

Best,

Spencer

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 6 Nov 2020 16:34:19 -0500

Below

On 6 November 2020 at 20.02.31, Spencer Dawkins at IETF (
[email-masked]) wrote:

Largely, the machinery that has to do with loss detection, bandwidth
estimation, congestion control, packet numbering is all set up to deal with
a single path in QUIC v1. When you migrate, you accept do a transactional
handover so both parties agree on the change, and you are very careful not
to get out of sync by one party changing path too frequently, and then
there are some issues with dealing with adversaries. You still allow for a
few packets on the old path, but but you can strictly place them as
originating from a point in the past (sort of like ballots being stamped no
later than election day).

With multiple paths, you have to decide have you number packets on
different paths and how you estimate loss, etc., and also more esoteric
details like spin bits and privacy in this context.

Ordering packets on multiple paths in the same stream is of course more of
a challenge in this case, as you point out, since packet numbering alone
might no longer be enough. But as stated, it there is a lot more to it than
just sequencing frames. Sequencing is a relatively small problem. The
bigger issues also apply to unordered datagrams.

Of course, these problems are far from unsolvable, it just takes som work.
The question is if it is worth the effort, given what you can already do
without true multipath, and if it is, exactly what are the important use
cases. And more remotely, if the additional API complexity with priotising
paths work out in praxis.

Mikkel

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 6 Nov 2020 17:18:20 -0600

Hi, Mikkel,

> bigger issues also apply to unordered datagrams.
Thanks - that was helpful, at least to me.

> paths work out in praxis.
I agree.

Best,

Spencer

---

From: Christian Huitema <[email-masked]>
Date: Fri, 6 Nov 2020 16:35:11 -0800

On 11/6/2020 3:18 PM, Spencer Dawkins at IETF wrote:
>
> Hi, Mikkel,
>
> On Fri, Nov 6, 2020 at 3:34 PM Mikkel Fahnøe Jørgensen
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Below
>
>     On 6 November 2020 at 20.02.31, Spencer Dawkins at IETF
>     ([email-masked]
>     <mailto:[email-masked]>) wrote:
>
>>     I'm not sure what the protocol difference is between the two
>>     cases. I understand the difference between sending on one path
>>     and sending on another path, but once you've started sending on
>>     two (or more) paths, I'm not sure what the protocol requirement
>>     is to stop sending on one of them. 
>>
>>     I'm even more confused if we're talking about sending datagrams,
>>     rather than reliable streams. 
>>
>>     I understand why a scheduler might want to limit itself to
>>     sending on one path in normal operation, but I'm trying to
>>     understand a different point. 
>
>     Largely, the machinery that has to do with loss detection,
>     bandwidth estimation, congestion control, packet numbering is all
>     set up to deal with a single path in QUIC v1. When you migrate,
>     you accept do a transactional handover so both parties agree on
>     the change, and you are very careful not to get out of sync by one
>     party changing path too frequently, and then there are some issues
>     with dealing with adversaries. You still allow for a few packets
>     on the old path, but but you can strictly place them as
>     originating from a point in the past (sort of like ballots being
>     stamped no later than election day).
>
>     different paths and how you estimate loss, etc., and also more
>     esoteric details like spin bits and privacy in this context.
>
>     Ordering packets on multiple paths in the same stream is of course
>     more of a challenge in this case, as you point out, since packet
>     numbering alone might no longer be enough. But as stated, it there
>     is a lot more to it than just sequencing frames. Sequencing is a
>     relatively small problem. The bigger issues also apply to
>     unordered datagrams.
>

Note there is no guarantee in QUIC that packets will be delivered in the
order in which they are sent. That would require head of line blocking,
and QUIC is specifically design to avoid that. So you could have three
packets:

PN 101: data on Stream 8, offset 0, length 1000

PN 102: data on Stream 12, offset 0, length 1000

PN 103: data on Stream 8, offset 1000, length 500, and data on Stream
12, offset 1000, length 500.

If packet 103 arrives after packet 101 but before packet 102 the data on
stream 8 will be delivered because it is in sequence, but most stack
will queue the data on stream 12 until the packet 102 arrives or its
content is repeated. I say most, because some applications may support
out of order delivery.

Having packets arrive "out of order" on multiple paths is merely a
generalization of that feature.

>     Of course, these problems are far from unsolvable, it just takes
>     som work. The question is if it is worth the effort, given what
>     you can already do without true multipath, and if it is, exactly
>     what are the important use cases. And more remotely, if the
>     additional API complexity with priotising paths work out in praxis.
>
> I agree.

That's the question indeed. Do we shove the multipath support in the
QUIC transport, or do we just tell applications to support multiple
connections. The more you ask the application to do something smart
about routing some traffic one way and some other another way, the more
the "multiple connection" solution seems appealing. Transport level
multipath makes the most sense when the paths are roughly equivalent, or
at least when the application policy can be described very simply.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 7 Nov 2020 02:57:25 -0500

So I can’t speak of all use cases, but take on this is two fold:

1. migration doesn’t currently support symmertric p2p server connections
when you want robust high performance datacenter connections. For example,
it is common for packet loss during equipment upgrades, and that could be
mitigated by multipath without much effort on the application.

2. Having two separate connections to deal with multi-path makes it
prohibitively difficult to coordinate a single stream For example a messing
service could increase performance on multi-path, but ensuring in-order
deliver on two connections require a new kind of application protocol on
the both ends.

Message ordering in actor bases systems can be very important in
simplifying algorithm complexity such as transactions: If you get message A
and but have not seen message B, then you know that it is safe to perform
task X. This is really hard to do over multiple connections. You have to
repeat a lot of the logic that is iinside QUIC or TCP.

Mikkel

On 7 November 2020 at 01.35.42, Christian Huitema ([email-masked])
wrote:

Hi, Mikkel,

Note there is no guarantee in QUIC that packets will be delivered in the
order in which they are sent. That would require head of line blocking, and
QUIC is specifically design to avoid that. So you could have three packets:

PN 103: data on Stream 8, offset 1000, length 500, and data on Stream 12,
offset 1000, length 500.

If packet 103 arrives after packet 101 but before packet 102 the data on
stream 8 will be delivered because it is in sequence, but most stack will
queue the data on stream 12 until the packet 102 arrives or its content is
repeated. I say most, because some applications may support out of order
delivery.

I agree.

That's the question indeed. Do we shove the multipath support in the QUIC
transport, or do we just tell applications to support multiple connections.
The more you ask the application to do something smart about routing some
traffic one way and some other another way, the more the "multiple
connection" solution seems appealing. Transport level multipath makes the
most sense when the paths are roughly equivalent, or at least when the
application policy can be described very simply.

-- Christian Huitema

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 9 Nov 2020 09:54:40 -0600

Hi Folks,
I agree with Mikkel's points.
To Lucas: I meant my short mail sometime ago I think it was before the
interim (?) where I explained that connection migration is mobility support
which should (from layering point of view) be in IP layer. In fact if IP
layer has this support then then no need for connection migration in QUIC,
so those procedures in the code do not get executed.

Multipath is multiple interface support. It seems more and more like
multipath probably better belongs in transport layer. Traffic in each
interface may go over different networks (in my case on over T Mobile and
the other AT&T). I believe a different PN is well justified in multipath as
we have it in the base draft because of these traffic conditions (no
offense to Christian).

Behcet.

On Sat, Nov 7, 2020 at 1:57 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

>> Ordering packets on multiple paths in the same stream is of course more
>> of a challenge in this case, as you point out, since packet numbering alone

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 9 Nov 2020 16:11:49 +0000

Hi Behcet,

> we have it in the base draft because of these traffic conditions (no

I still don't see why the current features of connection migration are not
in some way a form of multipath. When I look at RFC 6824 it says:

   A Multipath TCP connection provides a bidirectional bytestream
   between two hosts communicating like normal TCP and, thus, does not
   require any change to the applications.  However, Multipath TCP
   enables the hosts to use different paths with different IP addresses
   to exchange packets belonging to the MPTCP connection.  A Multipath
   TCP connection appears like a normal TCP connection to an
   application.

QUIC provides this, albeit restricting the paths from being used concurrently.

Cheers,

Lucas

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 9 Nov 2020 10:21:30 -0600

Hi Lucas,

> QUIC provides this, albeit restricting the paths from being used concurrently.
 This last sentence says it all. Also attention to the case I mentioned
above where the client has some kind of mobility support and its IP address
does not change (3GPP has that kind of support) but that does not
invalidate the "need" for multipath, i.e.

these two protocol mechanisms are orthogonal, you may need both, need one
or none.

Behcet

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 9 Nov 2020 17:31:26 +0100

Lucas,
> 
> On Mon, Nov 9, 2020 at 3:55 PM Behcet Sarikaya <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Folks,
>     To Lucas: I meant my short mail sometime ago I think it was before
>     the interim (?) where I explained that connection migration is
>     mobility support which should (from layering point of view) be in IP
>     layer. In fact if IP layer has this support then then no need for
>     connection migration in QUIC, so those procedures in the code do not
>     get executed.
> 
>     multipath probably better belongs in transport layer. Traffic in
>     each interface may go over different networks (in my case on over T
>     Mobile and the other AT&T). I believe a different PN is well
>     justified in multipath as we have it in the base draft because of
>     these traffic conditions (no offense to Christian).
> 
> 
> I still don't see why the current features of connection migration are 
> not in some way a form of multipath. 

You are right, connection migration is the weakest form of multipath.

> When I look at RFC 6824 it says:
> 

This paragraph of RFC6824 then continues as follows :

    However, to the network layer, each MPTCP subflow looks
    like a regular TCP flow whose segments carry a new TCP option type.
    Multipath TCP manages the creation, removal, and utilization of these
    subflows to send data.  The number of subflows that are managed
    within a Multipath TCP connection is not fixed and it can fluctuate
    during the lifetime of the Multipath TCP connection.

This is not really connection migration and MPTCP provides much more 
multipath capabilities than connection migration.

Olivier

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 9 Nov 2020 10:41:08 -0600

> You are right, connection migration is the weakest form of multipath.

Lucas admitted that connection migration does not support the paths being
used concurrently, possibly going over different networks.

Behcet

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 9 Nov 2020 16:51:04 +0000

Hey Olivier,

> You are right, connection migration is the weakest form of multipath.

Thanks. We heard use cases that would like stronger forms. I think it will
help continue to move the discussion forward if we can establish some
common ground on terms and capabilities.

> multipath capabilities than connection migration.

Yeah I follow. As someone coming from QUIC, the first sentence is kind of
easily negated (which is a benefit IIUC). I think the remainder of the
paragraph is partially satisfied by QUIC v1 if we consider
PATH_CHALLENGE/PATH_RESPONSE and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID.
But it starts to fall apart when you want to do more complicated things. I
think understanding the gaps in the transport signalling would be useful to
document in isolation to any specific solution.
draft-deconinck-quic-multipath has done some of that work already but it
gets a little too tied up with the solution IMO.

Cheers
Lucas

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 9 Nov 2020 10:58:07 -0600

Hi Lucas, Olivier,

> gets a little too tied up with the solution IMO.

I don't think Olivier would wish to undermine the most important feature of
multipath: multiple paths going over concurrently possible over different
networks.
Then he can not justify many features in draft-deconinck-quic-multipath.

Behcet

---

From: Roberto Peon <[email-masked]>
Date: Mon, 9 Nov 2020 17:32:03 +0000

I’m still concerned that we’re looking at solving this inside the connection, instead of providing a way for this to be solved irrespective of the connection.
There is a fundamental routing problem we have here that we could address (addressing a session), but we’re not addressing with what I’m seeing discussed (addressing a session within the same connection object).

If we consider this problem as making the session addressable, then applications can do it the way that makes sense for them, without having to put everything in every stack everywhere, plus new APIs to actually make them work.

I’m afraid if we add multipath, it’ll be like what happened with server push. The lack of appropriate APIs made using it with the browser fraught with tradeoffs with no reasonable way for an application to fix.

Solve the addressing-of-a-session problem, however, and we make it easier to solve the likely API problem that will accompany multipath.

Example:
I could have a virtual connection which is composed of a TCP connection on path A, and a QUIC connection on path B.
.. or maybe I want to try out a new version/extension on QUIC, so I have a virtual connection with QUIC and QUIC+extension.
I could declare that I’d like for data to flow down QUIC+extension path unless that is too slow, then duplicate the data onto the QUIC path.

In my mind, the application should establishes the virtual connection, and provide at least one path, and can optionally add (and remove) subsequent paths.
This is something we do already in “storage-land”, where diversity and separable failure domains are important, and where the use-cases are extremely diverse in latency, data-amounts, and cost.
-=R

From: QUIC <[email-masked]> on behalf of Behcet Sarikaya <[email-masked]>
Reply-To: "[email-masked]" <[email-masked]>
Date: Monday, November 9, 2020 at 8:58 AM
To: Lucas Pardue <[email-masked]>
Cc: Christian Huitema <[email-masked]>, Behcet Sarikaya <[email-masked]>, QUIC WG <[email-masked]>, Olivier Bonaventure <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>
Subject: Re: What to do about multipath in QUIC

Hi Lucas, Olivier,

On Mon, Nov 9, 2020 at 10:51 AM Lucas Pardue <[email-masked]<mailto:[email-masked]>> wrote:
Hey Olivier,

On Mon, Nov 9, 2020 at 4:31 PM Olivier Bonaventure <[email-masked]<mailto:[email-masked]>> wrote:
Lucas,
>
> On Mon, Nov 9, 2020 at 3:55 PM Behcet Sarikaya <[email-masked]<mailto:[email-masked]>
> <mailto:[email-masked]<mailto:[email-masked]>>> wrote:
>
>     Hi Folks,

Thanks. We heard use cases that would like stronger forms. I think it will help continue to move the discussion forward if we can establish some common ground on terms and capabilities.

Yeah I follow. As someone coming from QUIC, the first sentence is kind of easily negated (which is a benefit IIUC). I think the remainder of the paragraph is partially satisfied by QUIC v1 if we consider PATH_CHALLENGE/PATH_RESPONSE and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID. But it starts to fall apart when you want to do more complicated things. I think understanding the gaps in the transport signalling would be useful to document in isolation to any specific solution. draft-deconinck-quic-multipath has done some of that work already but it gets a little too tied up with the solution IMO.

I don't think Olivier would wish to undermine the most important feature of multipath: multiple paths going over concurrently possible over different networks.
Then he can not justify many features in draft-deconinck-quic-multipath.

Behcet
Cheers
Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 10 Nov 2020 03:37:06 -0800

But this doesn’t solve the serialization of a single stream over multiple
paths.

Also, it doesn’t really make sense to mix a video stream on QUIC on one
path and TCP on another. That would cause all kinds of problems, not to
mention privacy.

I agree that there is a risk of a complex unnecesary feature being poorly
implemented in QUIC. But true multipath cannot really be solved outiside of
QUIC. I’m fine with making it optional or giving it dedicated version.

Someone suggested building multipath on top of multiple QUIC connections. I
think that is viable if it is a newer QUIC version that delagates work to
older QUIC versions. The key point is that externally this happens
transparently, and there are optionas for the QUIC stacks to coordinate
locally or remotely via a signalling path. Not sure which solution is
ultimately the best, but you build upon what you already have.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 9 November 2020 at 18.32.06, Roberto Peon ([email-masked]) wrote:

I’m still concerned that we’re looking at solving this inside the
connection, instead of providing a way for this to be solved irrespective
of the connection.
There is a fundamental routing problem we have here that we could address
(addressing a session), but we’re not addressing with what I’m seeing
discussed (addressing a session within the same connection object).

If we consider this problem as making the session addressable, then
applications can do it the way that makes sense for them, without having to
put everything in every stack everywhere, plus new APIs to actually make
them work.

I’m afraid if we add multipath, it’ll be like what happened with server
push. The lack of appropriate APIs made using it with the browser fraught
with tradeoffs with no reasonable way for an application to fix.

Solve the addressing-of-a-session problem, however, and we make it easier
to solve the likely API problem that will accompany multipath.

Example:
I could have a virtual connection which is composed of a TCP connection on
path A, and a QUIC connection on path B.
.. or maybe I want to try out a new version/extension on QUIC, so I have a
virtual connection with QUIC and QUIC+extension.
I could declare that I’d like for data to flow down QUIC+extension path
unless that is too slow, then duplicate the data onto the QUIC path.

In my mind, the application should establishes the virtual connection, and
provide at least one path, and can optionally add (and remove) subsequent
paths.
This is something we do already in “storage-land”, where diversity and
separable failure domains are important, and where the use-cases are
extremely diverse in latency, data-amounts, and cost.
-=R

*From: *QUIC <[email-masked]> on behalf of Behcet Sarikaya <
[email-masked]>
*Reply-To: *"[email-masked]" <[email-masked]>
*Date: *Monday, November 9, 2020 at 8:58 AM
*To: *Lucas Pardue <[email-masked]>
*Cc: *Christian Huitema <[email-masked]>, Behcet Sarikaya <
[email-masked]>, QUIC WG <[email-masked]>, Olivier Bonaventure <
[email-masked]>, Mikkel Fahnøe Jørgensen <
[email-masked]>
*Subject: *Re: What to do about multipath in QUIC

Hi Lucas, Olivier,

Hey Olivier,

Lucas,

Behcet

Cheers

Lucas

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 10 Nov 2020 23:32:38 +0900

> But this doesn’t solve the serialization of a single stream over multiple

Right.

I would say that there are three categories.

1. Using different paths for different streams.

In this case, it is possible to achieve the goal by using different
connections.

2. Switching between multiple paths to serve one stream.

In this case, connection migration of QUIC v1 is sufficient.

3. Serving one stream using multiple paths concurrently.

This is the case that QUIC v1 does not support.

If there is an application that falls into the last category, then IMO it
would be worthwhile to discuss the requirements of the application (e.g.,
latency), because then we would would understand the requirements that we
need to accomplish.

> Someone suggested building multipath on top of multiple QUIC connections.
> I think that is viable if it is a newer QUIC version that delagates work to
> I don't think Olivier would wish to undermine the most important feature
> of multipath: multiple paths going over concurrently possible over
> different networks.
>

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 10 Nov 2020 06:40:36 -0800

On 10 November 2020 at 15.32.52, Kazuho Oku ([email-masked]) wrote:

This only works for the client. It doesn’t work server P2P. I’m alson
concerned with how fast it is possible to make the switch in latency
critical operations. Perhaps it can be done fast once, but then you have to
wait.

---

From: Lars Eggert <[email-masked]>
Date: Tue, 10 Nov 2020 16:44:39 +0200

On 2020-11-10, at 16:40, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> This only works for the client. It doesn’t work server P2P.

Given the presence of NATs and firewalls in most scenarios, the client will need to initiate a flow to the server first, before the server can respond (on a new path).

Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 10 Nov 2020 06:57:01 -0800

There are usually no NATs across datacenter hosts, and only firewalls that
you choose to set up on your own.
Such hosts may have multiple ethenet interfaces to maximize bandwidth. Or
there may be private and public IP addresses, billed differently.

I can see plenty of problems of bidirectional migration though. I suspect
P2P multipath will be easier.

On 10 November 2020 at 15.44.54, Lars Eggert ([email-masked]) wrote:

> This only works for the client. It doesn’t work server P2P.

Given the presence of NATs and firewalls in most scenarios, the client will
need to initiate a flow to the server first, before the server can respond
(on a new path).

Lars

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 10 Nov 2020 08:58:27 -0600

Hi, Mikkel,

> I’m alson concerned with how fast it is possible to make the switch in
> latency critical operations. Perhaps it can be done fast once, but then you
> have to wait.
>

One of the things I tool away from the multipath virtual interim was that I
had, and possibly other people had, the notion that "connection migration"
was a heavyweight operation (think, "failover to standby path"), but people
were saying that I should be thinking of it as lightweight ("once you've
validated two or more paths, you can just start sending on another path").

ISTM that "how fast it is possible to make the switch" would be the sum of

   1. How fast you can open and validate another connection - but this
   could happen before you need to use it
   2.  How fast you can send the first packet on another connection - my
   understanding is that you can just do that
   3. How you can ramp up the sending rate on another connection to roughly
   what you had achieved on the first connection - but is that critical for
   latency critical operations?

Are you thinking of other factors I'm missing here?

Best,

Spencer

---

From: Lars Eggert <[email-masked]>
Date: Tue, 10 Nov 2020 17:03:13 +0200

Hi,

On 2020-11-10, at 16:57, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> There are usually no NATs across datacenter hosts, and only firewalls that you choose to set up on your own.
> Such hosts may have multiple ethenet interfaces to maximize bandwidth. Or there may be private and public IP addresses, billed differently.

sure. But a datacenter-internal use of QUIC wasn't one of the scenarios that were brought up during the interim - those were IIRC all about use across the Internet. (Which is what the WG overall has focused on so far.)

I'd be curious to what benefits you see for using QUIC within a datacenter, given the performance gap that will still continue to exist for some time compared to TCP, and given that MPTCP hasn't really gotten traction there, either, even when that use case was investigated for it pretty early on?

Thanks,
Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 10 Nov 2020 07:20:06 -0800

I’m mostly concerned with cross+datacenter use, but it follows that you
would preferable use the same communication pattern for hosts on the same
datacenter also.

There are various concepts for datacenter use, such as Kubernetes, but it
isn't high performance, and it often isn’t cross datacenter. Sometimes you
can rely on private IP addresses being private, but that doesn’t always
extend to multiple datacenters of the same vendor, and certainly not
multi-vendors.

You can add a VPN to span multiple datacenters, but it can be a complex
setup without clear performance metrics, and they can potentially add
security holes by introducing a DMZ. Also, A VPN implementation could
possibly benefit from multipath QUIC.

A simple use case is connecting to a database server in a different
location. If for no other reason than geographical redundancy. A TCP
connection is far from ideal for database communication where you can have
many concurrent queries. For time sensitive data, or bandwidth constrained
data, or both, you want to use all the bandwidth that you can.

Another use case is cross platform messaging services.

Mikkel

On 10 November 2020 at 16.03.29, Lars Eggert ([email-masked]) wrote:

Hi,

sure. But a datacenter-internal use of QUIC wasn't one of the scenarios
that were brought up during the interim - those were IIRC all about use
across the Internet. (Which is what the WG overall has focused on so far.)

I'd be curious to what benefits you see for using QUIC within a datacenter,
given the performance gap that will still continue to exist for some time
compared to TCP, and given that MPTCP hasn't really gotten traction there,
either, even when that use case was investigated for it pretty early on?

Thanks,
Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 11 Nov 2020 07:46:54 -0800

You can’t do that at the app layer. You said yourself, wait for ACK on one
path before sending on the other.

I can see a protocol that uses both QUIC and TPC to overcome UDP blocks,
but then you could also argue that QUIC should support TCP as a transport
fallback. I think that would go down the wrong path so to speak. It is
better to let QUIC motivate passing UDP in more cases. So I’d rather have a
super QUIC doing multiple paths.

There is of course WebRTC that uses all sorts of UDP/TCP patterns, but it
is fairly complex to get right in all scenarios. Perhaps it is best to let
WebRTC handle TCP and move forward with new tech for QUIC. Then you can
hack WebRTC to also support QUIC if you can afford it.

On a related note:

I would be useful to be able to connect to an endpoint through multiple
different public access points, i.e. load balancers. The LBs have a
tendency to cut idle connections after a while, so being able to connect
through multiple would avoid downtime on that account. This is sort of
complicated because the connection establishment would see different IPs to
the same endpiont, but I guess that is what multipath is?

Kind Regards,
Mikkel Fahnøe Jørgensen

On 10 November 2020 at 18.33.44, Roberto Peon ([email-masked]) wrote:

I believe it does solve the serialization problem, but let’s talk it out! 😊

My example of TCP+QUIC is the kind of thing you’d do when you were
uncertain about one failing (perhaps UDP is being blocked).

What I’m talking about is a likely combination of two things:
1) Ability to get some new path or connection to a session (likely a single
“server”, but that is the decision of the “server”).
- can also be within the connection, so long as the path is addressable by
the application

- can be external to the app, where different connections are used.
2) Ability of application to schedule data to each path or session (i.e. to
mux/demux onto the paths).
- This could be via a standardized config, or
- It could be up to the application to figure it out

- An API for determining how packets are scheduled seems necessary
regardless of any other multipath implementation path.

An example standardized config could be something like:
  (race s=1 (path A) (delay 0.1 (path B)))
This would say that data should flow on path A by default (it is first),
and if you’ve not gotten an ack in 100ms (0.1 s), try path B.

If one wanted to send data in duplicate on all paths:
  (race s=2 (path A)  (path B))
If one wanted to send the first X bytes on path A, and the rest on path B:

(partition X (path A) (path B))

This kind of config is something we’ve been using for a couple of years for
video things, to good effect.
It is most certainly not perfect, but it does allow quite a bit of
flexibility with a fairly minimal non-Turing complete config.
I’m not claiming we should use it, just showing it absolutely can be done
in a way that supports about any application need without needing to write
code everywhere.

-=R

*From: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Date: *Tuesday, November 10, 2020 at 3:37 AM
*To: *Lucas Pardue <[email-masked]>, Roberto Peon <[email-masked]>,
"[email-masked]" <[email-masked]>
*Cc: *Olivier Bonaventure <[email-masked]>, Christian
Huitema <[email-masked]>, QUIC WG <[email-masked]>
*Subject: *Re: What to do about multipath in QUIC

Kind Regards,

Mikkel Fahnøe Jørgensen

Example:

Hi Lucas, Olivier,

Hey Olivier,

Lucas,

Behcet

Cheers

Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 11 Nov 2020 07:48:48 -0800

Sorry, regarding LBs, forgot say:

Some LBs cut connections even if they are not idle. Google L7 does that on
websockets on regular HTTPS 1.1
Thus you cannot just ping you way to liveness.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 11 November 2020 at 16.46.54, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

On a related note:

Kind Regards,
Mikkel Fahnøe Jørgensen

-=R

Kind Regards,

Mikkel Fahnøe Jørgensen

Example:

Hi Lucas, Olivier,

Hey Olivier,

Lucas,

Behcet

Cheers

Lucas

---

From: Roberto Peon <[email-masked]>
Date: Wed, 11 Nov 2020 20:39:04 +0000

What the application can do, or what it can cause to be done depends on the API.
Does the application get a callback or have opportunity to take action when data has been received/when you’ve received and ack or nack’d?
Does it have a way to express to the lower level stack what it should do?

In recent memory, I’ve said that H2 priorities in their original form are dead or should be.
Within the past few days we’re seeing Ian mention Chrome killing server push.
I believe that we’ve not had the appropriate APIs to make these features useful in the browsers and sometimes in servers, and that has resulted in their death on some platforms, because there has been no reasonable way to use them.

I believe we’re likely to suffer exactly the same thing here—a feature that has obvious theoretical advantages, but which has tradeoffs in use.
Because it has tradeoffs, it will fail in applications or browsers without the appropriate API which allows the application to deal with those tradeoffs and play/innovate in the space.

-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wednesday, November 11, 2020 at 7:47 AM
To: "[email-masked]" <[email-masked]>, Roberto Peon <[email-masked]>, Lucas Pardue <[email-masked]>
Cc: QUIC WG <[email-masked]>, Olivier Bonaventure <[email-masked]>, Christian Huitema <[email-masked]>
Subject: Re: What to do about multipath in QUIC

You can’t do that at the app layer. You said yourself, wait for ACK on one path before sending on the other.

I can see a protocol that uses both QUIC and TPC to overcome UDP blocks, but then you could also argue that QUIC should support TCP as a transport fallback. I think that would go down the wrong path so to speak. It is better to let QUIC motivate passing UDP in more cases. So I’d rather have a super QUIC doing multiple paths.

There is of course WebRTC that uses all sorts of UDP/TCP patterns, but it is fairly complex to get right in all scenarios. Perhaps it is best to let WebRTC handle TCP and move forward with new tech for QUIC. Then you can hack WebRTC to also support QUIC if you can afford it.

On a related note:

I would be useful to be able to connect to an endpoint through multiple different public access points, i.e. load balancers. The LBs have a tendency to cut idle connections after a while, so being able to connect through multiple would avoid downtime on that account. This is sort of complicated because the connection establishment would see different IPs to the same endpiont, but I guess that is what multipath is?

Kind Regards,
Mikkel Fahnøe Jørgensen

On 10 November 2020 at 18.33.44, Roberto Peon ([email-masked]<mailto:[email-masked]>) wrote:
I believe it does solve the serialization problem, but let’s talk it out! 😊

My example of TCP+QUIC is the kind of thing you’d do when you were uncertain about one failing (perhaps UDP is being blocked).

What I’m talking about is a likely combination of two things:
1) Ability to get some new path or connection to a session (likely a single “server”, but that is the decision of the “server”).
- can also be within the connection, so long as the path is addressable by the application
- can be external to the app, where different connections are used.
2) Ability of application to schedule data to each path or session (i.e. to mux/demux onto the paths).
- This could be via a standardized config, or
- It could be up to the application to figure it out
- An API for determining how packets are scheduled seems necessary regardless of any other multipath implementation path.

An example standardized config could be something like:
  (race s=1 (path A) (delay 0.1 (path B)))
This would say that data should flow on path A by default (it is first), and if you’ve not gotten an ack in 100ms (0.1 s), try path B.

This kind of config is something we’ve been using for a couple of years for video things, to good effect.
It is most certainly not perfect, but it does allow quite a bit of flexibility with a fairly minimal non-Turing complete config.
I’m not claiming we should use it, just showing it absolutely can be done in a way that supports about any application need without needing to write code everywhere.

-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, November 10, 2020 at 3:37 AM
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>, Roberto Peon <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Olivier Bonaventure <[email-masked]<mailto:[email-masked]>>, Christian Huitema <[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: What to do about multipath in QUIC

But this doesn’t solve the serialization of a single stream over multiple paths.

Also, it doesn’t really make sense to mix a video stream on QUIC on one path and TCP on another. That would cause all kinds of problems, not to mention privacy.

I agree that there is a risk of a complex unnecesary feature being poorly implemented in QUIC. But true multipath cannot really be solved outiside of QUIC. I’m fine with making it optional or giving it dedicated version.

Someone suggested building multipath on top of multiple QUIC connections. I think that is viable if it is a newer QUIC version that delagates work to older QUIC versions. The key point is that externally this happens transparently, and there are optionas for the QUIC stacks to coordinate locally or remotely via a signalling path. Not sure which solution is ultimately the best, but you build upon what you already have.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 9 November 2020 at 18.32.06, Roberto Peon ([email-masked]<mailto:[email-masked]>) wrote:
I’m still concerned that we’re looking at solving this inside the connection, instead of providing a way for this to be solved irrespective of the connection.
There is a fundamental routing problem we have here that we could address (addressing a session), but we’re not addressing with what I’m seeing discussed (addressing a session within the same connection object).

Example:

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Behcet Sarikaya <[email-masked]<mailto:[email-masked]>>
Reply-To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date: Monday, November 9, 2020 at 8:58 AM
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Cc: Christian Huitema <[email-masked]<mailto:[email-masked]>>, Behcet Sarikaya <[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>, Olivier Bonaventure <[email-masked]<mailto:[email-masked]>>, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Subject: Re: What to do about multipath in QUIC

Hi Lucas, Olivier,

Behcet
Cheers
Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 11 Nov 2020 13:51:23 -0800

I can’t say I disagree with server push og H2 priorities, and yes there is
a risk of going unused. But I doubt it. With 5G, 4G, WiFi and ethernet all
being available at once, I see this as a vast untapped market that just
hasnøt been viable so far. If your propasal was easy, it would be widely
deployed rather than limited to heavyweights end to end controlled cases
like FaceBook.

Building an application on top of dubious low level exposure is at best
lock in to one stack, and worst case not possible because the peer can’t
support it. So MP can have esoteric APIs for specifying available paths,
but all supporting stacks would have some way of configuring it.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 11 November 2020 at 21.39.42, Roberto Peon ([email-masked]) wrote:

What the application can do, or what it can cause to be done depends on the
API.
Does the application get a callback or have opportunity to take action when
data has been received/when you’ve received and ack or nack’d?
Does it have a way to express to the lower level stack what it should do?

In recent memory, I’ve said that H2 priorities in their original form are
dead or should be.
Within the past few days we’re seeing Ian mention Chrome killing server
push.
I believe that we’ve not had the appropriate APIs to make these features
useful in the browsers and sometimes in servers, and that has resulted in
their death on some platforms, because there has been no reasonable way to
use them.

I believe we’re likely to suffer exactly the same thing here—a feature that
has obvious theoretical advantages, but which has tradeoffs in use.
Because it has tradeoffs, it will fail in applications or browsers without
the appropriate API which allows the application to deal with those
tradeoffs and play/innovate in the space.

-=R

*From: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Date: *Wednesday, November 11, 2020 at 7:47 AM
*To: *"[email-masked]" <[email-masked]>, Roberto Peon <[email-masked]>,
Lucas Pardue <[email-masked]>
*Cc: *QUIC WG <[email-masked]>, Olivier Bonaventure <
[email-masked]>, Christian Huitema <[email-masked]>
*Subject: *Re: What to do about multipath in QUIC

On a related note:

Kind Regards,

Mikkel Fahnøe Jørgensen

-=R

Kind Regards,

Mikkel Fahnøe Jørgensen

Example:

Hi Lucas, Olivier,

Hey Olivier,

Lucas,

Behcet

Cheers

Lucas

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 12 Nov 2020 12:01:18 +1100

> Within the past few days we’re seeing Ian mention Chrome killing server 

https://groups.google.com/a/chromium.org/g/blink-dev/c/K3rYLvmQUBY/m/vOWBKZGoAQAJ for those who aren't watching.