Subject: Fwd: New Version Notification for draft-liu-multipath-quic-01.txt

From: "Yanmei Liu" <[email-masked]>
Date: Mon, 14 Dec 2020 14:13:42 +0800

Hi all, 

Here is an updated version of the draft which was posted in the mailing list last Friday. Please check this link: https://tools.ietf.org/html/draft-liu-multipath-quic-01
Regarding the questions in the mailing list for the previous version:
1. We have fixed the IANA registry problem in the new version. Thanks Lucas for pointing out this problem.
2. We currently use Github for editing, and we prefer discussions on the mailing list, so we removed the Github link in this draft. 

This version merges our draft and Christian’s draft, and here are some of the key features:

 * Re-use as much as possible mechanisms of QUIC-v1, which has
 supported connection migration and path validation.

 * To avoid the risk of packets being dropped by middleboxes (which
 may only support QUIC-v1), use the same packet header formats as
 QUIC V1.

 * Endpoints need a Path Identifier for each different path which is
 used to track states of packets. As we want to keep the packet
 header formats unchanged [QUIC-TRANSPORT], Connection IDs (and the
 sequence number of Connection IDs) would be a good choice of Path
 Identifier.

 * For the convenience of packet loss detection and recovery,
 endpoints use a different packet number space for each Path
 Identifier.

 * Congestion Control, RTT measurements and PMTU discovery should be
 per-path (following [QUIC-TRANSPORT])

Thanks Jana and Kazuho for reviewing the proposal.

We would like to know how people think about the draft, and all feedbacks and suggestions are welcome!

Thanks, 
Yanmei

------------------------------------------------------------------
From: internet-drafts <[email-masked]>
Date: 2020年12月14日(星期一) 12:36 
To：安勍(莳逸) <[email-masked]>; "Ma, Yunfei" <[email-masked]>; 刘彦梅(喵吉) <[email-masked]>; Christian Huitema <[email-masked]>; 李振宇 <[email-masked]>
Subject: New Version Notification for draft-liu-multipath-quic-01.txt

A new version of I-D, draft-liu-multipath-quic-01.txt
has been successfully submitted by Yanmei Liu and posted to the
IETF repository.

Name:  draft-liu-multipath-quic
Revision: 01
Title:  Multipath Extension for QUIC
Document date: 2020-12-14
Group:  Individual Submission
Pages:  18
URL:            https://www.ietf.org/archive/id/draft-liu-multipath-quic-01.txt
Status:         https://datatracker.ietf.org/doc/draft-liu-multipath-quic/
Htmlized:       https://datatracker.ietf.org/doc/html/draft-liu-multipath-quic
Htmlized:       https://tools.ietf.org/html/draft-liu-multipath-quic-01
Diff:           https://www.ietf.org/rfcdiff?url2=draft-liu-multipath-quic-01

Abstract:
   This document specifies multipath extension for the QUIC protocol to
   enable the simultaneous usage of multiple paths for a single
   connection.  The extension is compliant with the single-path QUIC
   design.  The design principle is to support multipath by adding
   limited extension to [QUIC-TRANSPORT].

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 01:38:58 -0800

Would it be practical to lift the 32-bit constraint on the CID sequence
number?

Even if it probably is enough for most use cases I find it unfortunate to
introduce a new constraint here because if affects design decisions
elsewhere in stack.

For example, it could be required that a key is updated at least once in
every 2^31 sequence numbers, or the IV could be hashed and use a
sufficiently frequent key update.

On 14 December 2020 at 07.14.07, Yanmei Liu ([email-masked])
wrote:

Hi all,

Here is an updated version of the draft which was posted in the mailing
list last Friday. Please check this link:
https://tools.ietf.org/html/draft-liu-multipath-quic-01
Regarding the questions in the mailing list for the previous version:
1. We have fixed the IANA registry problem in the new version. Thanks Lucas
for pointing out this problem.
2. We currently use Github for editing, and we prefer discussions on the
mailing list, so we removed the Github link in this draft.

This version merges our draft and Christian’s draft, and here are some of
the key features:

We would like to know how people think about the draft, and all feedbacks
and suggestions are welcome!

Thanks,
Yanmei

To：安勍(莳逸) <[email-masked]>; "Ma, Yunfei"
<[email-masked]>; 刘彦梅(喵吉) <[email-masked]>;
Christian Huitema <[email-masked]>; 李振宇 <[email-masked]>

Name:  draft-liu-multipath-quic
Revision: 01
Title:  Multipath Extension for QUIC
Document date: 2020-12-14
Group:  Individual Submission
Pages:  18
URL:
https://www.ietf.org/archive/id/draft-liu-multipath-quic-01.txt
Status:         https://datatracker.ietf.org/doc/draft-liu-multipath-quic/
Htmlized:
https://datatracker.ietf.org/doc/html/draft-liu-multipath-quic
Htmlized:       https://tools.ietf.org/html/draft-liu-multipath-quic-01
Diff:
https://www.ietf.org/rfcdiff?url2=draft-liu-multipath-quic-01

Abstract:

The IETF Secretariat

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 02:33:46 -0800

Some more comments on the proposed MP design

On protocol violations:

quote:

"If an endpoint receives MP frames from packets of other encryption
levels, it MAY return MP_PROTOCOL_VIOLATION as a connection error and
close the connection."

I think it is important to use strict MUST on easily detectable protocol
violations to ensure interop and security.

Also, to my previous point on path sequence numbers as IV source:

What is the motivation for using the same key on all paths? Is it
considered a resource constraint to have more keys? Considering a server
might have 10K+ connections, and perhaps 1.3 paths per connections (wild
guess), it doesn’t seem to much of an overhead.

In addition, having separate keys avoids the sync issues mentioned in the
document. As it stands, failure on one active path can block key update.
This sounds like an accidental or intentional attack vector, although I’m
not sure that it is.

On the document format:

I think the discussion on application influence is good, but maybe isolate
that discussion a bit. It seems that stream priorities are part of the MP
design, which isn’t really the case, even if it is relevant to MP
applications.

On QoE:
Maybe it is good to leave the interpretation open to applications, but it
makes it more difficult to design a protocol layer on top. Is it the
intention that, for example, a QUIC-MP video protocol is designed which
then documents how to interpret QoE? If so, does this need to be in the MP
protocol, or could it be in a control stream created by the app level
protocol? In other words, I’m not sure wha the value is at the transport
level if it doesn’t affect the transport layer directly (unless of course I
am missing something).

ACK_MP frame:
If this is an extension to QUIC-v1 it makes sense (and this seems to be the
case). If it is intended as a new protocol version, maybe replace the
regular ACK frame to avoid protocol bloat? I can see the counter argument
that a new frame ID makes it easier to reuse v1 implementations, but
protocol bloat is a concern as well.

The scheduling section has an example mentioning one of the two paths. I’m
sure this a left-over since there could be more than two active paths as I
understand.

Path state:
I do not understand this well enough, but I’m a bit concerned about one
endpoint dictating the behaviour of the other regarding priorities etc. It
seems like this could easily be abused, and perhaps this is also better
done in an application level protocol?

The text is not very clear about the client and the servers role in path
negotiation. It seems that that it follows he QUIC-v1 async design due to
relying on the v1 PATH packets. So presumably this also places constraints
on who can send path state updates.

If MP was to become more symmetric, I wonder if there is a natural
evolution from here, or the proposed design is just digging the async
design hole deeper, so to speak. Note that async MP is useful in its own
right.

Mikkel

On 14 December 2020 at 10.38.58, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Hi all,

Thanks,
Yanmei

Abstract:

The IETF Secretariat

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 02:38:10 -0800

Sorry, “async design “ should read asymmetric design, in that only one
endpoint can create multiple paths, as opposed to a symmetric design where
both can.

On 14 December 2020 at 11.33.46, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

async design

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 09:01:30 -0800

On 12/14/2020 1:38 AM, Mikkel Fahnøe Jørgensen wrote:
> Would it be practical to lift the 32-bit constraint on the CID 
> sequence number?
>
> Even if it probably is enough for most use cases I find it unfortunate 
> to introduce a new constraint here because if affects design decisions 
> elsewhere in stack.
>
> For example, it could be required that a key is updated at least once 
> in every 2^31 sequence numbers, or the IV could be hashed and use a 

It boils down to how much additional state you want to maintain in or to 
support multipath. The 32 bit constraint allows for a very simple 
implementation. I don't think applications are going to hit it in 
practice. Supposing one connection migration per second, the limit is 
not hit for one hundred years.

-- Christian Huitema

---

From: "Yanmei Liu" <[email-masked]>
Date: Tue, 15 Dec 2020 02:44:12 +0800

Hi Mikkel,

First of all, thanks a lot for reading the draft carefully and giving feedback about it.
My reply is behind the quote: 

> “If an endpoint receives MP frames from packets of other encryption
> I think it is important to use strict MUST on easily detectable protocol violations to ensure interop and security.

It’s a good advice and we will using MUST here in the next version.

> What is the motivation for using the same key on all paths? Is it considered a resource constraint to have more keys? Considering a server might have 10K+ connections, and perhaps 1.3 paths per connections (wild guess), it doesn’t seem to much of an overhead.
> In addition, having separate keys avoids the sync issues mentioned in the document. As it stands, failure on one active path can block key update. This sounds like an accidental or intentional attack vector, although I’m not sure that it is.
> 

We have discussed about using different keys in different paths. This requires adding a key derivation procedure for each path, and
also adding ways to identify the relevant key in the incoming packets, and we will have to modify the QUIC-v1 packet headers to support the feature. 
As we have explained in the basic design principles of the proposal, we don’t want to modify the packet header format of QUIC-v1, to avoid the risk of packets being dropped by middleboxes (which may only support QUIC-v1).

> On the document format:
> 
> I think the discussion on application influence is good, but maybe isolate that discussion a bit. It seems that stream priorities are part of the MP design, which isn’t really the case, even if it is relevant to MP applications.
> 

We do encounter scenarios where stream priority needs to be supported in multi-path in reality, and we will discuss this carefully.

> On QoE:
> Maybe it is good to leave the interpretation open to applications, but it makes it more difficult to design a protocol layer on top. Is it the intention that, for example, a QUIC-> MP video protocol is designed which then documents how to interpret QoE? If so, does this need to be in the MP protocol, or could it be in a control stream created by the app level protocol? In other words, I’m not sure wha the value is at the transport level if it doesn’t > affect the transport layer directly (unless of course I am missing something).
> The QOE_CONTROL_SIGNAL frame will definately affect the packet scheduler, as mentioned in the proposal. 
> And we didn’t restrict the inner value format, it’s open for implementors and researchers to play with it, and try their own scheduler algorithms with it. 
> We thought it would be convenient to play with an existing frame than adding a new frame. 
> ACK_MP frame:
> If this is an extension to QUICv1 it makes sense (and this seems to be the case). If it is intended as a new protocol version, maybe replace the regular ACK frame to avoid protocol bloat? I can see the counter argument that a new frame ID makes it easier to reuse v1 im> plementations, but protocol bloat is a concern as well.

I think that it’s important to keep compatible with QUIC-v1. If implementations send ACK frame in multiple paths, it still need to be processed without errors.

> The scheduling section has an example mentioning one of the two paths. I’m sure this a left-over since there could be more than two active paths as I understand.

We will add more examples about scenarios with more than two active paths.

> Path state:
> I do not understand this well enough, but I’m a bit concerned about one endpoint dictating the behaviour of the other regarding priorities etc. It seems like this could easily be abused, and perhaps this is also better done in an application level protocol?
> The text is not very clear about the client and the servers role in path negotiation. It seems that that it follows he QUIC-v1 async design due to relying on the v1 PATH packets. So presumably this also places constraints on who can send path state updates.
> If MP was to become more symmetric, I wonder if there is a natural evolution from here, or the proposed design is just digging the async design hole deeper, so to speak. Note that async MP is useful in its own right.

I’d like to express a little more, we have discussed a lot about this problem:
As the management of paths is largely specific to application, Why we will need a frame that one endpoint use it to dictate the behaviour of the peer? 
1. In uploading scenarios, as the sender side, client can decide the scheduler strategy itself, by providing APIs to let applications specify their path management preferences etc.
2. In downloading scenarios, as the receiver side, client need to express its preference of these paths. For example, client may need to tell server which path is preferred to use (maybe Wi-Fi path when there are both cellular and Wi-Fi environment). In the case client sends a management frame says "send on path #1 (the Wi-Fi path)", if server sends on path #2 (the cellular path), it will make users spend more traffic costs, and users will not be happy with that. Without the management frame in the transport, we will not be able to explain that preference of paths. Servers will never know which path client is prefer to use.
So we start from simple and clear design of path states (0- Abandon 1- Standby 2- Available), and we added priority for scenarios like endpoints want to send 20% packets on one path / 80% packets on the other.
Both client and server can send the PATH_STATUS frame. 

Thanks again for all the feedback!

Thanks,Yanmei

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 11:17:55 -0800

Hi Yanmei, thanks for explaining.

Comments inline:

We have discussed about using different keys in different paths. This
requires adding a key derivation procedure for each path, and
also adding ways to identify the relevant key in the incoming packets,
and we will have to modify the QUIC-v1 packet headers to support the
feature.

Why can’t you use the connection ID to find this information? It is needed
to find the connection anyway, and you need it in the IV, so it should be
trivial to store a local key ID along with CID lookup pointing to the
connection, or am I missing something?

It is true that you need a key derivation, but that machinery is already in
place in QUIC-TLS, altough some details need to be added.

As we have explained in the basic design principles of the proposal,
we don’t want to modify the packet header format of QUIC-v1, to avoid
the risk of packets being dropped by middleboxes (which may only
support QUIC-v1).

No, that would not be good. It also wouldn’t really be a QUIC-v1 extension.

I’d like to express a little more, we have discussed a lot about this problem:
As the management of paths is largely specific to application, Why we
will need a frame that one endpoint use it to dictate the behaviour of
the peer?
1. In uploading scenarios, as the sender side, client can decide the
scheduler strategy itself, by providing APIs to let applications
specify their path management preferences etc.
2. In downloading scenarios, as the receiver side, client need to
express its preference of these paths. For example, client may need to
tell server which path is preferred to use (maybe Wi-Fi path when
there are both cellular and Wi-Fi environment). In the case client
sends a management frame says "send on path #1 (the Wi-Fi path)", if
server sends on path #2 (the cellular path), it will make users spend
more traffic costs, and users will not be happy with that.

Yes, that all makes sense.

Without the management frame in the transport, we will not be able to
explain that preference of paths. Servers will never know which path
client is prefer to use.

This is what I do not understand. Why do you need it in transport instead
of the app level. Well, yes, as a general preference maybe it does make
sense. But sense transport cannot distinguish between different kinds of
payloads: different streams, different datagrams etc., all it can do is to
apply this to all content.

Furthermore, what if it is not upload or download, but something far more
complex, like a game server that sends realtime location data on game
objects and the game client sending player actions the other way, both at
low latency. At the same time the game server sends new scene data in high
bandwidth, high latency, in the background, and the client and server
uploads voice chat with different priorities.

Even if we forget about that complexity, what if both endpoints want to
change priorities of the path because of different content. Who gets to
decide?

So I agree about the need to manage paths, but I think it needs to be at
the application protocol level, just likely the now abandoned HTTP/3
prioritised streams were decided in the app protocol, not in the transport.

So we start from simple and clear design of path states
(0- Abandon 1- Standby 2- Available), and we added priority for
scenarios like endpoints want to send 20% packets on one path / 80%
packets on the other.
Both client and server can send the PATH_STATUS frame.

Yes, but I think this will lead to problems, even though I fully agree that
paths needs to be managed somehow.

If you compare to QUIC-v1, the transport spec does not say anything about
how to schedule and prioritize streams even though it does suggest using
pacing and does provide some guidance. This is not because scheduling is
not important, because it is, but transport is not the right place to do
it. Also here, you can imagine that it is the receiver, and not the sender,
that should decide. Yet there is no priority frame in the transport spec.

And thanks for the great work.

I think it is important work.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 11:34:29 -0800

On 12/14/2020 10:44 AM, Yanmei Liu wrote:

> > ACK_MP frame:
> > If this is an extension to QUICv1 it makes sense (and this seems to be the case). If it is intended as a new protocol version, maybe replace the regular ACK frame to avoid protocol bloat? I can see the counter argument that a new frame ID makes it easier to reuse v1 im> 
> plementations, but protocol bloat is a concern as well.
>

Some more explanations of this ACK frames issue. Sending packets and 
receiving acknowledgements forms a control loop, triggering actions like 
loss correction or changes in congestion control parameters. When 
implementing control loops, it is generally better to keep delays short. 
This has a clear effect in management of multiple paths. The shortest 
loops are obtained when the acknowledgements are sent on the shortest 
available path, which may not always be the same path over which data 
was sent. Since we use a separate number space for each path, supporting 
shortest-path-acknowledgements requires adding a path identifier in the 
ACK_MP.

Part of the problem there is that we end up with four frame type 
identifiers for acknowledgements, depending on whether the path 
identifier is present or not and whether the ECN counters are present or 
not. If I had to change something, I would put the ECN counters in a 
separate frame that could be composed with the ACK frame, just like time 
stamps can be carried in a separate time stamp frame. That way, we would 
end up with just three frame types for ACK, ACK_MP and ECN counters, 
instead of four. But the ECN counter decision was done a couple years 
ago, and it is too late to change it now.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 11:38:09 -0800

It boils down to how much additional state you want to maintain in or to
support multipath. The 32 bit constraint allows for a very simple
implementation. I don't think applications are going to hit it in practice.
Supposing one connection migration per second, the limit is not hit for one
hundred years.

Counting on fingers … hmm well yeah 136 to be exact. And for space travel
you can’t even migrate that fast, but then 100 years might mean something
else.

I still don’t like modifying the underlying semantics and I’m also a bit
concerned about the path sync issue, especially on forward stations /
offloads.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 12:01:17 -0800

On 12/14/2020 2:38 AM, Mikkel Fahnøe Jørgensen wrote:

> endpoint can create multiple paths, as opposed to a symmetric design 
> where both can.
>
>
> On 14 December 2020 at 11.33.46, Mikkel Fahnøe Jørgensen 
> ([email-masked] <mailto:[email-masked]>) wrote:
>
>> async design

The proposal indeed keeps the asymmetric design of QUIC v1. That design 
is grounded in practical realities, such as the prevalence of NAT and 
stateful firewalls that would drop "unexpected" packets sent to a 
client. Also, the client typically knows about the cost of paths better 
than the server, for example whether sending more data on a given path 
will generate larger invoices from the provider. The design goal is to 
let the client express these preference to the server using a simple 
priority scheme.

Anything more complex probably requires application level decisions. 
That why the design provides the QOE frame. That frame let applications 
exchange information about paths in an application specific way. The 
expectation is that the application posts them as it sees fit, and the 
transport delivers them as they are received. The application then uses 
locally provided API to affect the way data is sent over multiple paths.

We could debate whether the QOE frame is needed at all. Applications 
could also exchange the data as part of application protocol itself. I 
think that having a separate frame reduce the overall complexity, by 
separating application specific path control from the application 
protocol itself.

We could also debate whether the QOE frame should be specified in 
greater details. The current design simply passes a byte string. The DNS 
SVCB RR type addresses a related problem by an SvcParams field that 
contains a list of key-value pairs. It is possible that having a similar 
list of key-value pairs would be useful there. On the other hand, we do 
not yet have a lot of experience. And open design with a byte string is 
probably good enough for now.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 12:30:57 -0800

OK, thanks for explaining.

I wonder if the current asymmetric approach could be enhanced relatively
easily by allowing a server to announce a list of address/port/QoE tuples
that it can recommend the client to use as virtual endpoints, without
initiating anything. The client would then issue path challenges as usual,
but can do so to any one of the proposed endpoints. The server could
periodically propose new such tuples or update QoE on existing. If a server
wants to retire an endpoint, it could send that information too, but a
draining period would be needed.

This could also piggy back on the existing CID update system where a server
CID is associated with a virtual endpoint.

Mikkel

On 14 December 2020 at 21.01.27, Christian Huitema ([email-masked])
wrote:

async design

The proposal indeed keeps the asymmetric design of QUIC v1. That design is
grounded in practical realities, such as the prevalence of NAT and stateful
firewalls that would drop "unexpected" packets sent to a client. Also, the
client typically knows about the cost of paths better than the server, for
example whether sending more data on a given path will generate larger
invoices from the provider. The design goal is to let the client express
these preference to the server using a simple priority scheme.

Anything more complex probably requires application level decisions. That
why the design provides the QOE frame. That frame let applications exchange
information about paths in an application specific way. The expectation is
that the application posts them as it sees fit, and the transport delivers
them as they are received. The application then uses locally provided API
to affect the way data is sent over multiple paths.

We could debate whether the QOE frame is needed at all. Applications could
also exchange the data as part of application protocol itself. I think that
having a separate frame reduce the overall complexity, by separating
application specific path control from the application protocol itself.

We could also debate whether the QOE frame should be specified in greater
details. The current design simply passes a byte string. The DNS SVCB RR
type addresses a related problem by an SvcParams field that contains a list
of key-value pairs. It is possible that having a similar list of key-value
pairs would be useful there. On the other hand, we do not yet have a lot of
experience. And open design with a byte string is probably good enough for
now.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 14:37:12 -0800

On 12/14/2020 11:38 AM, Mikkel Fahnøe Jørgensen wrote:
>
>> It boils down to how much additional state you want to maintain in or 
>> to support multipath. The 32 bit constraint allows for a very simple 
> Counting on fingers … hmm well yeah 136 to be exact. And for space 
> travel you can’t even migrate that fast, but then 100 years might mean 
> something else.
>
> I still don’t like modifying the underlying semantics and I’m also a 
> bit concerned about the path sync issue, especially on forward 
> stations / offloads.
>
I am really not sure about the "underlying semantics" argument. We have 
hidden limits in the current design. For example, the MAX_DATA frame 
encodes the Maximum Data as a varint, thus no more than 2^62 bytes can 
be sent on a connection. Given a minimum MTU size of 1200 bytes, that 
means connections are unlikely to send more than 2^52 packets. That is a 
practical limit over all flows combined when using multipath. AEAD 
encryption imposes other limits, including limits on number of 
decryption failures that cannot be mitigated by key updates. The reality 
is that QUIC connections are not designed to last forever.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 14:49:32 -0800

On 12/14/2020 12:30 PM, Mikkel Fahnøe Jørgensen wrote:
> I wonder if the current asymmetric approach could be enhanced 
> relatively easily by allowing a server to announce a list of 
> address/port/QoE tuples that it can recommend the client to use as 
> virtual endpoints, without initiating anything. The client would then 
> issue path challenges as usual, but can do so to any one of the 
> proposed endpoints. The server could periodically propose new such 
> tuples or update QoE on existing. If a server wants to retire an 
> endpoint, it could send that information too, but a draining period 
> would be needed.
>
> This could also piggy back on the existing CID update system where a 
> server CID is associated with a virtual endpoint.

Yes, it is tempting to add a way for a server "to announce a list of 
address/port/QoE tuples that it can recommend the client to us" as part 
of the multipath design. I think that we should resist that temptation, 
and treat "annoucing addresses" as a separate problem. After all, even 
if the endpoint do not use the multipath extension, they might still use 
these server addresses as targets of V1-supported address migration.

Nick Banks was discussing this server address migration issue in the 
Slack chat room a couple of weeks ago. His use case was a server joined 
through a bandwidth limited VPN; there would be benefits in moving the 
connection to a direct path outside the VPN. It turns out that this 
migration requires more than merely communicating alternate server 
addresses to the peer: server and clients also need to cooperate to open 
a hole in the firewall that protects the server. This gets very close to 
problems encountered in peer to peer scenarios.

These are good problems to address, but I would rather address them in a 
targeted effort than as part of the multipath design.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 14:56:31 -0800

Good point and interesting description on VPN case.

I think it would be helpful for multipath to still consider this at an
abstract level such that it can be plugged in at a later point without
deciding on the exact details up front, if for no other reason than to
avoid hardcoding things that don’t have to be.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 14 Dec 2020 15:20:16 -0800

On 12/14/2020 2:56 PM, Mikkel Fahnøe Jørgensen wrote:

>> These are good problems to address, but I would rather address them 
>> in a targeted effort than as part of the multipath design.
>>

I don't think there is an hardcoded limit here. If we get a separate 
extension dealing with address announcements, it should automatically 
compose with the multipath extension.

-- Christian Huitema

---

From: Yunfei Ma <[email-masked]>
Date: Mon, 14 Dec 2020 15:26:22 -0800

Hi Mikkel,

I think this current protocol does not prevent you from doing that as it is
designed as a minimally-scoped extension of QUIC-v1. The goal is to make it
easier for people to try different things with multi-path. As pointed out
by Christian, announcing address/port/QoE does have some complexity. One
way is to let the client server cooperate, but you may also want to use a
STUN server to help you achieve that which is similar to the case of RTC.
We would appreciate it if you could try and let us know your experience.

Cheers,
Yunfei

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 14 Dec 2020 16:08:17 -0800

I am indeed working with a STUN WebRTC use case which could benefit from a
QUIC MP replacement of WebRTC. A problem here is that it is often necessary
to use WebRTC over TLS/TURN because of firewall policies. Hopefully a
future QUIC version could both provide multipath and avoid TURN overhead
provided QUIC gains enough traction to slip through firewalls by default.

But in this discussion I’m more thinking about possible server
configurations either server to server or client to server. A server might
implement itself on multiple hardware nodes, for resilience, load
balancing, or locality of reference, or a server might have access to
multiple network cards with different IP addresses.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 15 December 2020 at 00.27.13, Yunfei Ma ([email-masked]) wrote:

Hi Mikkel,

Cheers,
Yunfei

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 15 Dec 2020 01:22:57 -0800

Maybe the QoE per Path-ID is too granular when the ID is tied to the CID
because the CID may change for reasons unrelated to changes in the path
metrics.

On the client side what matters is largely the 2-tuple the decides between
WiFi, modem, or ethernet.

On a future multi-homed server, the endpoint likewise dictates QoE - but
here it could mean where the server relays chat traffic separate from game
state.

Mikkel

On 15 December 2020 at 01.08.17, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Kind Regards,
Mikkel Fahnøe Jørgensen

Hi Mikkel,

Cheers,
Yunfei

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 15 Dec 2020 01:41:26 -0800

I cant discuss the use case in detial but perhaps it is worth noting that
the WebRTC case I mention has a web client at one end, and a mobile server
at the other end, and here the multipath concern is primarily the
connectivity of the mobile server. Therefore the current multipath proposal
would not be sufficient (along with WebRTC on QUIC).

Mikkel

Kind Regards,
Mikkel Fahnøe Jørgensen

Hi Mikkel,

Cheers,
Yunfei

---

From: Quentin De Coninck <[email-masked]>
Date: Tue, 15 Dec 2020 12:47:07 +0100

Hi,

I'm glad to finally see some interesting discussion about some multipath 
proposals :-)

As one of the authors of the draft-deconinck-quic-multipath, I would 
like to discuss a little about the design differences between both 
drafts, which I currently understand to be small. Basically, all the 
basic design points you list are also valid for 
draft-deconinck-quic-multipath (reuse of QUICv1 mechanisms, unmodified 
header, PSN per path,...), except that we associate a separate "Uniflow 
ID" (which identifies the path) to a Connection ID . That is, we do not 
directly use the sequence number of the CID as the path identifier.

You propose to use the Connection IDs to identify the path in each 
direction. Its sequence number determines the Path ID. I see that in 
your Figure 1, you propose an example where you start a new "path" by 
using the sequence number 2 towards the server and the sequence number 1 
towards the client. So your proposal does not require to use the same 
path ID over a same network path, am I right?

In your same example, you state that the client must check that are 
unused available connection IDs at both sides before starting a new 
path. So if the client wants to reach the server over a new network path 
but the server does not start using a new path ID, what happens? Is the 
client unable to send data over it? It also seems odd to create a 
(bidirectional?) "path" with different identifiers at both sides.

For the scheduling section, I wonder what is really specific to 
Multipath QUIC. For me, many of the proposed schemes are applicable to 
any multipath transport protocol (like MPTCP). Why not discussing them 
in a document like draft-bonaventure-iccrg-schedulers-01? Also, why is 
the per-stream policy multipath specific?

I wonder why there is a MUST enforcing the ACK frame to return on the 
same path. Couldn't the ACK frame just refer to a specific, well-known 
path ID? For the remaining, the ACK_MP frame is identical to the MP_ACK 
frame.

Regarding the section discussing the differences with our draft:

1- Your draft states that your extension builds on the concept of 
bidirectional paths. But I see nothing in your description preventing a 
host from using only the initial path while receiving data on the other 
path. For instance, a server could receive data on two different paths 
but send all its data (including ACK_MP, PATH_RESPONSE,...) over the 
initial path, which is quite a unidirectional (or at least asymmetrical) 
usage of the second path.

2- I'm not sure to understand the difference here regarding 
"feedback-based dynamic scheduling strategy" using "ACK packet". Could 
you elaborate a bit more on this?

Again, thanks for bringing this to this list. I would be glad if we 
could at some point merge our both drafts into a single one and I am 
willing to contribute to this effort :-)

Best,
Quentin

> Hi all,
>
> Here is an updated version of the draft which was posted in the 
> mailing list last Friday. Please check this link: 
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-liu-multipath-quic-01&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091447064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=PNExv4lCxGy5uVoMRc1g8X52IZBMpjEk7NOx%2FWF55mk%3D&reserved=0>
> 2. We currently use Github for editing, and we prefer discussions on 
> the mailing list, so we removed the Github link in this draft.
>
>     <[email-masked]>; 刘彦梅(喵吉)
>     <[email-masked]>; Christian Huitema
>     <[email-masked]>; 李振宇 <[email-masked]>
>
>     <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-liu-multipath-quic-01.txt&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091457023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=qr8JPusFk9FKI26mNbYTtde1t4G1zyQSEibBqTbMIG0%3D&reserved=0>
>     <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-liu-multipath-quic%2F&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091457023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=2lNP%2BDxp3qlRncBcf6IPvoSUHx5cs7N7BqUkaMarQ5k%3D&reserved=0>
>     Htmlized:
>     <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-liu-multipath-quic&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091466978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=uBBdwodffWgFbeR7e6PESxrSkxPT9pu4tYwyPdx16A4%3D&reserved=0>
>     <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-liu-multipath-quic-01&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091466978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=0F1QrItnLruPTm4cRISETmA4pn0F0ygPqqmGWr9NLtk%3D&reserved=0>
>     Diff:
>     <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Frfcdiff%3Furl2%3Ddraft-liu-multipath-quic-01&data=04%7C01%7Cquentin.deconinck%40uclouvain.be%7C281d8515dee042881e9e08d89ff77c50%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637435233091466978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WBUZ%2Fm9vYpzuQ1a57XhKVlF8gdpYMXl02PiTDXmXpZ8%3D&reserved=0>
>
>     Abstract:

---

From: Yunfei Ma <[email-masked]>
Date: Tue, 15 Dec 2020 23:24:36 -0800

Dear Quentin,

Thanks so much for your comments. Indeed, we are looking forward to further
collaboration. For the time being, the most critical thing is to get as
much experimental experience as possible. We would love to talk in further
detail with you and Oliver.

Cheers,
Yunfei

On Tue, Dec 15, 2020 at 3:47 AM Quentin De Coninck <
[email-masked]> wrote:

> As one of the authors of the draft-deconinck-quic-multipath, I would like
> to discuss a little about the design differences between both drafts, which
> I currently understand to be small. Basically, all the basic design points
> you list are also valid for draft-deconinck-quic-multipath (reuse of QUICv1
> mechanisms, unmodified header, PSN per path,...), except that we associate
> a separate "Uniflow ID" (which identifies the path) to a Connection ID .
> That is, we do not directly use the sequence number of the CID as the path
> identifier.
>
> direction. Its sequence number determines the Path ID. I see that in your
> Figure 1, you propose an example where you start a new "path" by using the
> sequence number 2 towards the server and the sequence number 1 towards the
> client. So your proposal does not require to use the same path ID over a
> same network path, am I right?
>
> In your same example, you state that the client must check that are unused
> available connection IDs at both sides before starting a new path. So if
> the client wants to reach the server over a new network path but the server
> does not start using a new path ID, what happens? Is the client unable to
> send data over it? It also seems odd to create a (bidirectional?) "path"
> with different identifiers at both sides.
>
> For the scheduling section, I wonder what is really specific to Multipath
> QUIC. For me, many of the proposed schemes are applicable to any multipath
> transport protocol (like MPTCP). Why not discussing them in a document like
> draft-bonaventure-iccrg-schedulers-01? Also, why is the per-stream policy
> multipath specific?
>
> I wonder why there is a MUST enforcing the ACK frame to return on the same
> path. Couldn't the ACK frame just refer to a specific, well-known path ID?
> For the remaining, the ACK_MP frame is identical to the MP_ACK frame.
>
> path. For instance, a server could receive data on two different paths but
> send all its data (including ACK_MP, PATH_RESPONSE,...) over the initial
> path, which is quite a unidirectional (or at least asymmetrical) usage of
> the second path.
>
> "feedback-based dynamic scheduling strategy" using "ACK packet". Could you
> elaborate a bit more on this?
>
> Again, thanks for bringing this to this list. I would be glad if we could
> at some point merge our both drafts into a single one and I am willing to
> contribute to this effort :-)
>
> Best,
> Quentin
>
> Hi all,
>
> <[email-masked]>
> To：安勍(莳逸) <[email-masked]> <[email-masked]>; "Ma, Yunfei" <[email-masked]> <[email-masked]>; 刘彦梅(喵吉) <[email-masked]> <[email-masked]>; Christian Huitema <[email-masked]> <[email-masked]>; 李振宇 <[email-masked]> <[email-masked]>
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 16 Dec 2020 01:19:44 -0800

| It also seems odd to create a (bidirectional?) "path" with different
identifiers at both sides.

Hi Quentin, I have not read through your document, as I mentioned in an
earlier mail, I also have some concerns about the the Path-ID being too
granular because QoE does not relate to a path instance (sequence number)
but rather the tuples it runs over - at least most of the time.

But … the Path-ID in Liu uses the sequence number of the client CID.
Presumable both endpoints uses the same Path-ID for the same path, even if
server sends with a different CID. I had some concerns about PATH_STATE
frames conflicting on Path-ID when sent from different endpoints, but that
shouldn’t be a problem. I also especially had a concern about future
symmetric multipath with server announced endpoints, but if the client
always probes the path first, the client CID still remains a unique
identifier.

Now, I do think that a more stable Path-ID makes sense compare to the
(potentially) frequently changing CIDs. But, again thinking about a future
server announced endpoint, a server might want to use different paths for
traffic steering even if the endpoint tuple remains the same for multiple
virtual paths. The server CID could be computed differently for each
virtual endpoint and affect server side routing.

So I am not sure client side Path-ID should be client CID because it
changes often, but it should be sufficiently unique for both endpoints.
Also I am not sure that a tuple based Path-ID is a good decision because of
potential virtual server endpoints.

Mikkel

On 16 December 2020 at 08.25.34, Yunfei Ma ([email-masked]) wrote:

 It also seems odd to create a (bidirectional?) "path" with different
identifiers at both sides.

---

From: Quentin De Coninck <[email-masked]>
Date: Wed, 16 Dec 2020 10:48:36 +0100

Hi Mikkel,

Le 16/12/20 à 10:19, Mikkel Fahnøe Jørgensen a écrit :
> Hi Quentin, I have not read through your document, as I mentioned in 
> an earlier mail, I also have some concerns about the the Path-ID being 
> too granular because QoE does not relate to a path instance (sequence 
> number) but rather the tuples it runs over - at least most of the time.
+1, both the "forward path" and the "return path" should be considered.
> Presumable both endpoints uses the same Path-ID for the same path, 
> even if server sends with a different CID.
Probably, but how can we ensure that for a given Path ID, each endhost 
uses the same network road to reach its peer?
> I had some concerns about PATH_STATE frames conflicting on Path-ID 
> when sent from different endpoints, but that shouldn’t be a problem. I 
> also especially had a concern about future symmetric multipath with 
> server announced endpoints, but if the client always probes the path 
> first, the client CID still remains a unique identifier.
>
> (potentially) frequently changing CIDs. But, again thinking about a 
> future server announced endpoint, a server might want to use different 
> paths for traffic steering even if the endpoint tuple remains the same 
> for multiple virtual paths. The server CID could be computed 
> differently for each virtual endpoint and affect server side routing.
>
> changes often, but it should be sufficiently unique for both 
> endpoints. Also I am not sure that a tuple based Path-ID is a good 
> decision because of potential virtual server endpoints.

In my opinion, I don't think the "path" should be the element we should 
identify. Rather, I think we should identify "forward flows" and 
"backward flows" (what we call uniflows in our deconinck draft) and see 
paths as a combination of a forward flow and a backward one.

Best,

Quentin

>
> Mikkel
>
> On 16 December 2020 at 08.25.34, Yunfei Ma ([email-masked] 
> <mailto:[email-masked]>) wrote:
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 16 Dec 2020 02:27:38 -0800

In my opinion, I don't think the "path" should be the element we should
identify. Rather, I think we should identify "forward flows" and "backward
flows" (what we call uniflows in our deconinck draft) and see paths as a
combination of a forward flow and a backward one.

Maybe - it sounds appealing. But if the client is always responsible for
probing paths also for different server announced endpoints, then both
endpoints are naturally coupled. The alternative would be for the server to
also probe paths, and that can be difficult with NATs without STUN. There
are many things to consider ...

Mikkel

---

From: Quentin De Coninck <[email-masked]>
Date: Wed, 16 Dec 2020 14:08:35 +0100

>> In my opinion, I don't think the "path" should be the element we 
>> should identify. Rather, I think we should identify "forward flows" 
>> and "backward flows" (what we call uniflows in our deconinck draft) 
>> and see paths as a combination of a forward flow and a backward one.
>>
> Maybe - it sounds appealing. But if the client is always responsible 
> for probing paths also for different server announced endpoints, then 
> both endpoints are naturally coupled. The alternative would be for the 
> server to also probe paths, and that can be difficult with NATs 
> without STUN. There are many things to consider ...
>
Sure. Most of the time it is the client that will open the way and the 
server waits for the client's packets exposing reachable IP/port tuple. 
But I would find unfortunate to have a core design preventing such flow 
creation initiative by the server. And while the client can use, e.g., 
two flows to reach the server, the server may only want to keep one flow 
towards the client to limit its state and we have two possible circuits: 
1) flow 1 client to server/flow server to client and 2) flow 2 client to 
server/flow server to client.

Quentin

---

From: Christian Huitema <[email-masked]>
Date: Wed, 16 Dec 2020 08:36:12 -0800

On 12/16/2020 5:08 AM, Quentin De Coninck wrote:
>> both endpoints are naturally coupled. The alternative would be for 
>> the server to also probe paths, and that can be difficult with NATs 
> server waits for the client's packets exposing reachable IP/port 
> tuple. But I would find unfortunate to have a core design preventing 
> such flow creation initiative by the server. And while the client can 
> use, e.g., two flows to reach the server, the server may only want to 
> keep one flow towards the client to limit its state and we have two 
> possible circuits: 1) flow 1 client to server/flow server to client 
> and 2) flow 2 client to server/flow server to client.
>

The "symmetry of paths" is exactly the kind of issue for which we need 
experimentation with multiple implementations. We will know much more 
about that in a couple weeks.

The "symmetry of roles" is another matter. It is tied to discovery of 
addresses and traversal of NAT and firewalls. The address discovery 
issues are not specific to multipath -- they are prominent for example 
in P2P scenarios using QUIC V1. I think we should address them 
separately, as part maybe of an address discovery extension to QUIC V1. 
If that discovery extension was available, it could be composed nicely 
with multipath extensions.

I don't think there is anything in the current draft that would prevent 
that "composition with address discovery", but people are fallible. If 
there is, it will be fixed.

-- Christian Huitema