Subject: Re: What to do about multipath in QUIC

From: "Yanmei Liu" <[email-masked]>
Date: Thu, 12 Nov 2020 18:48:06 +0800

Hi Lucas,

> >
> > You are right, connection migration is the weakest form of multipath.
> >
> 
> Thanks. We heard use cases that would like stronger forms. I think it will
> help continue to move the discussion forward if we can establish some
> common ground on terms and capabilities.

I strongly agree that we need to establish some common ground on terms and capabilities, so I’d like to explain some design considerations in [draft-an-multipath-quic](https://tools.ietf.org/html/draft-an-multipath-quic-00), and we hope to get more feedbacks.

> >
> >     However, to the network layer, each MPTCP subflow looks
> >     like a regular TCP flow whose segments carry a new TCP option type.
> >     Multipath TCP manages the creation, removal, and utilization of these
> >     subflows to send data.  The number of subflows that are managed
> >     within a Multipath TCP connection is not fixed and it can fluctuate
> >     during the lifetime of the Multipath TCP connection.
> >
> > This is not really connection migration and MPTCP provides much more
> > multipath capabilities than connection migration.
> >
> 
> Yeah I follow. As someone coming from QUIC, the first sentence is kind of
> easily negated (which is a benefit IIUC). I think the remainder of the
> paragraph is partially satisfied by QUIC v1 if we consider
> PATH_CHALLENGE/PATH_RESPONSE and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID.
> But it starts to fall apart when you want to do more complicated things. I
> think understanding the gaps in the transport signalling would be useful to
> document in isolation to any specific solution.
> draft-deconinck-quic-multipath has done some of that work already but it
> gets a little too tied up with the solution IMO.

  1. We think QUICv1 has already laid down the foundation to build a general-purpose multi-path since migration can be viewed as a special type of multi-path. Therefore, we think one should reuse the design of migration in QUIC-v1 as much as possible, along with the features such as PATH_CHALLENGE/PATH_RESPONSE for path challenge and address validation, and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID for CID renegotiation of new path(which is called Sub-connection in our draft). Reusing these features of QUIC-v1 with small extensions has enabled us to get general-purpose multi-path features with very little efforts in Alibaba’s XQUIC(an implementation of QUIC-v1).
  2. We find that the simplest way to add a second path is to use a sub-connection. The concept of sub-connection is directly inherited from connection in QUIC-transport, defined as the logic session of each physical path. Similar to a connection, each sub-connection has its own packet number space for the purpose of loss detection and recovery. 
  3. To merge the gap between migration and the general-purpose multi-path, several new features need to be supported: 
  - (1) how to manage the lifecycles of individual sub-connections. 
  - (2) how to distinguish packets coming from different sub-connections.  
  - (3) how to co-operate with a multi-path scheduler.

We would appreciate hearing any thoughts, comments and suggestions.

Thanks,
Yanmei

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 12 Nov 2020 03:10:56 -0800

Hi Yanmei Liu

I think this sounds like a good approach. It takes all the work an
application would otherwise have to do and hides the complexity, without
completely introducing a lot of new protocol complexity. This allows an
application to open a stream and keep working on it regardless of which
connections are currently alive.

I wonder if there are cases where this is a bit too heavy. For example
handshakes could perhaps be simpler on secondary connections.

Also I wonder about privacy. For example if it is easy to detect that to
concurrent connections belong to the same endpoint, fingerprinting might be
easier. I guess that would always be a problem for MP.

There are also subtleties such as idle-timeout. What if each connection has
a different timeout, is it only the sub connection that dies, or the entire
connection?  There could even be a case where connections remain alive
without any currently open sub-connection in order to deal with aggressive
NAT gateways etc.

If secondary handshakes get simplified, I suspect it is more of an
implementation whether to see it as sub-connections, or just protocol
messages on different paths.

- Mikkel

On 12 November 2020 at 11.48.50, Yanmei Liu ([email-masked])
wrote:

Hi Lucas,

> common ground on terms and capabilities.

I
strongly agree that we need to establish some common ground on terms
and capabilities,
so I’d like to explain some design considerations in [draft-an-multipath-quic](
https://tools.ietf.org/html/draft-an-multipath-quic-00
), and we hope to get more feedbacks.

> gets a little too tied up with the solution IMO.

  1. We think QUICv1 has already laid down the foundation to build a
general-purpose multi-path since migration can be viewed as a special
type of multi-path. Therefore, we think one should reuse the design of
migration in QUIC-v1 as much as possible, along with the features such
as PATH_CHALLENGE/PATH_RESPONSE for path challenge and address
validation, and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID for CID
renegotiation of new path(which is called Sub-connection in our
draft). Reusing these features of QUIC-v1 with small extensions has
enabled us to get general-purpose multi-path features with very little
efforts in
Alibaba’s XQUIC(an implementation of QUIC-v1).
  2. We find that the simplest way to add a second path is to use a
sub-connection. The concept of sub-connection is directly inherited
from connection in QUIC-transport, defined as the logic session of
each physical path. Similar to a connection, each sub-connection has
its own packet number space for the purpose of loss detection and
recovery.
  3. To merge the gap between migration and the general-purpose
multi-path, several new features need to be supported:
  - (1) how to manage the lifecycles of individual sub-connections.
  - (2) how to distinguish packets coming from different sub-connections.
  - (3) how to co-operate with a multi-path scheduler.

Thanks,
Yanmei

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Nov 2020 07:21:28 -0600

Two things ...  one for Yanmei, one for Mikkel.

> easier. I guess that would always be a problem for MP.

For Mikkel,

ISTM that we are wobbling between "true multipath is easier to track" and
"connection migration is the weakest form of multipath, so if multipath is
trackable, so is connection migration", and I wonder if someone could point
me to an analysis of "how to avoid tracking for connection migration" - one
strategy I've learned a lot about at/since the virtual interim was
validating multiple paths early in the life of a connection, so that
failover can happen quickly, for instance.

One strategy that's happening in my world is "open one connection per QoS,
on each path". In my world, that's a total of four connections (the reasons
why don't matter). Is it better to open two connections on one path, and
wait SOME_RANDOM_TIME to open connections on the other path, or to open all
four paths roughly simultaneously?

I'm asking this in QUIC, because QUIC connection migration isn't limited to
two paths, and ISTM that the more paths you are going to use, the more we
should pay attention to tracking. I'm not a genius of privacy, but some of
them are here, and more are certainly available if we needed them, so I'd
like to understand the thinking here.

> There are also subtleties such as idle-timeout. What if each connection
> has a different timeout, is it only the sub connection that dies, or the
> entire connection?  There could even be a case where connections remain
> alive without any currently open sub-connection in order to deal with
> aggressive NAT gateways etc.
>
> For Yanmei,

>
> I
> strongly agree that we need to establish some common ground on terms and capabilities,
> I think "we need to establish some common ground on terms and
capabilities" is exactly right.
If I understood David Scinazi's comment to me about this at the virtual
interim, he found the steering/switching/splitting characterization in my
presentation helpful, so I started working on an individual draft in Github
here -
https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath
- to begin capturing understandings of ways in which multiple paths might
be used, and different strategies for using them.

The current version of the draft uses language that's ATSSS-specific,
although I don't think the ideas are closely tied to ATSSS. As I said in
the draft,

   This section uses terminology from 3GPP ATSSS {Access Traffic
   Steering, Switch and Splitting, [TS23501]) to describe three
   different patterns of packet forwarding that could be used when
   multiple disjoint paths are available.  Note that these terms
   describe concepts that are not ATSSS-specific, and could apply to any
   multipath environment.  If terminology more accessible to IETF QUIC
   working group participants presents itself, I'll change it.

If people agree that common ground on terms and capabilities would be
useful, they are invited to contribute.

And thanks for bringing your work to the QUIC working group.

Best,

Spencer

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 12 Nov 2020 14:38:38 +0100

In that light true MP could be less trackable than migration because you
can avoid the synchronization of traffic stopping on one connection when
another picks up. For ordinary migration you could also use MP to phase in
the new connection gradually if the old one keeps working somewhat.

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 12 Nov 2020 14:32:07 +0000

Hi Spencer,

On Thu, Nov 12, 2020 at 1:22 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> here - https://github.com/SpencerDawkins/draft-dawkins-q
> <https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath>
> *draft-dawkins-quic-what-to-do-with-multipath*
> uic-what-to-do-with-multipath
> <https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath>

Hi Spencer,

Reading the latest Markdown copy as an individual, and someone familiar
with the QUIC world, I find that the definition of flow is a little too
ambiguous. This is in part because it is not entirely clear to me what you
mean by packets. One interpretation of packets would be QUIC packets, a
different interpretation is application data. There are two examples of
where I find this particularly confusing. First, "a "flow" is "a set of
packets that share an ordering constraint"", is this ordering constraint
based on QUIC packet numbers or something else? If you're intending to mean
something like a stream, then I don't know where QUIC's control plane
traffic fits. Secondly, the statement "This would allow "Traffic Splitting"
for unordered QUIC datagram packets" is confusing because DATAGRAMS are
frames in packets. Are you intending to say a packet that contains only
DATAGRAM frames, or something else?

Perhaps it's just me but I also find the term forwarding confusing. The
only cases where this term is used in the QUIC Transport document are in
relation to describing attacks that would copy and forward QUIC packets.
For endpoint-managed multipath QUIC, I would assume that QUIC packets are
synthesized with particular paths in mind and then sent on those paths.

I wonder if a document such as this should talk more in terms of the
multiple bytestreams that QUIC provides, which TCP (and MPTCP) does not.
Leveraging QUIC streams seemed to be a use case enabler, either for
redundantly sending overlapping ranges, for efficiently striping
non-overlapping ranges for bandwidth, or for strict ordering.

I think there could be more clarity about the control plane and data plane.
IIUC an application might want to leverage splitting of stream data but
only steering of the corresponding ACKs or retransmissions to a particular
path (something like the highly-asynchronous Satellite use case)

Cheers
Lucas

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Nov 2020 09:04:00 -0600

Hi, Mikkel,

> the new connection gradually if the old one keeps working somewhat.

That's the way I thought it would work, but wanted to ask.

Thanks for your thoughts.

Best,

Spencer

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Nov 2020 09:32:22 -0600

Hi, Lucas,

>> useful, they are invited to contribute.

Thank you for this. All of it is helpful to me. I'll work on incorporating
your feedback in the draft.

> DATAGRAM frames, or something else?

I agree. The first comment I got about this draft was that I was using
"flows" and "connections" almost interchangeably, and that wasn't helpful
in my world, where so much will be relying on datagrams. So, yes, I need to
work on clarity here.

> synthesized with particular paths in mind and then sent on those paths.

That confusion sure wasn't intentional! Taking a quick tour through
https://tools.ietf.org/html/draft-ietf-quic-transport-32#section-12 makes
me think that I should be saying "sending". Does that sound about right?

> non-overlapping ranges for bandwidth, or for strict ordering.

I think this is the kind of thing that needs to go in Section 2.3, now to
be called Sending Strategies with Multiple Paths, if that's the right term.

> I think there could be more clarity about the control plane and data
> plane. IIUC an application might want to leverage splitting of stream data
> but only steering of the corresponding ACKs or retransmissions to a
> particular path (something like the highly-asynchronous Satellite use case)
>

Yes, I agree. The current sending strategies in Section 2.3 are roughly the
non-ATSSS-specific strategies from ATSSS, which got me started, but this
section will benefit from pushing down a level of detail or two. And your
"redundantly sending overlapping ranges, for efficiently striping
non-overlapping ranges for bandwidth, or for strict ordering" in the
previous comment would be a helpful clarification.

Thanks again so much for your thoughts. Much appreciated.

Best,

Spencer

---

From: Christian Huitema <[email-masked]>
Date: Thu, 12 Nov 2020 09:04:09 -0800

On 11/12/2020 3:10 AM, Mikkel Fahnøe Jørgensen wrote:
>   1. We think QUICv1 has already laid down the foundation to build a general-purpose multi-path since migration can be viewed as a special type of multi-path. Therefore, we think one should reuse the design of migration in QUIC-v1 as much as possible, along with the features such as PATH_CHALLENGE/PATH_RESPONSE for path challenge and address validation, and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID for CID renegotiation of new path(which is called Sub-connection in our draft). Reusing these features of QUIC-v1 with small extensions has enabled us to get general-purpose multi-path features with very little efforts in

Yes. That's a major investment in QUIC V1, and we should keep it.

>   2. We find that the simplest way to add a second path is to use a sub-connection. The concept of sub-connection is directly inherited from connection in QUIC-transport, defined as the logic sessi…

I did not want to do that in my own draft for a couple of reasons. The
main one is the interaction with encryption.

The use of AEAD is only safe if the same packet number is not reused
twice with the same key. If we use multiple packet number contexts, AEAD
is only safe if these contexts use different encryption keys. This
requires adding a key derivation procedure for the "sub connection", and
also adding ways to identify the relevant key in the incoming packets.
This gets complicated very quickly, especially if we want to keep the
possibility of using zero-length connection identifiers on the client side.

I use a concept very similar to the sub-connection, but only as a way to
manipulate paths, so the client can instruct the server when paths ought
to be abandoned. Otherwise, I just keep track of which PN maps to which
path.

> We would appreciate hearing any thoughts, comments and suggestions.

I think we need more work on the "multi-path scheduler". We have heard
of three application scenarios: maintaining the lowest RTT when sending
voice segments (Apple Siri), avoiding buffering delays when playing
music (Apple Music), and using two available links with equal preference
(Ali Baba "high speed train"). I wish that we could distillate that into
a couple of simple concepts.

-- Christian Huitema

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Nov 2020 12:27:35 -0600

FWIW ...

On Thu, Nov 12, 2020 at 11:05 AM Christian Huitema <[email-masked]>
wrote:

> I think we need more work on the "multi-path scheduler". We have heard of
> three application scenarios: maintaining the lowest RTT when sending voice
> segments (Apple Siri), avoiding buffering delays when playing music (Apple
> Music), and using two available links with equal preference (Ali Baba "high
> speed train"). I wish that we could distillate that into a couple of simple
> concepts.
>
I'm working to capture these in
https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath.
Thanks, Christian, for the framing.

There are others from the multipath virtual interim (for instance, sharing
bandwidth if multiple paths have roughly equivalent RTTs, and using the
path with the lowest RTT if they don't).

So, I agree with Christian about needing more work on this, and it's not
clear to me how many "multipath schedulers" we should be thinking about -
but that's a question for another day.

Best,

Spencer

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 12 Nov 2020 19:47:07 +0000

There is already work on this in ICCRG:

https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00

From: QUIC <[email-masked]> on behalf of Spencer Dawkins at IETF <[email-masked]>
Date: Thursday, 12. November 2020 at 19:28
To: Christian Huitema <[email-masked]>
Cc: "Ma, Yunfei" <[email-masked]>, "Liu, Hongqiang(洪强)" <[email-masked]>, healing4d <[email-masked]>, Yanmei Liu <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, quic <[email-masked]>, "安勍(莳逸)" <[email-masked]>
Subject: Re: What to do about multipath in QUIC

FWIW ...

On Thu, Nov 12, 2020 at 11:05 AM Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

I think we need more work on the "multi-path scheduler". We have heard of three application scenarios: maintaining the lowest RTT when sending voice segments (Apple Siri), avoiding buffering delays when playing music (Apple Music), and using two available links with equal preference (Ali Baba "high speed train"). I wish that we could distillate that into a couple of simple concepts.
I'm working to capture these in  https://github.com/SpencerDawkins/draft-dawkins-quic-what-to-do-with-multipath<https://protect2.fireeye.com/v1/url?k=732d78a9-2cb641af-732d3832-86ee86bd5107-819e3a2f60ca88ed&q=1&e=f5a6d870-fae3-4266-9a08-557fbd96f40c&u=https%3A%2F%2Fgithub.com%2FSpencerDawkins%2Fdraft-dawkins-quic-what-to-do-with-multipath>. Thanks, Christian, for the framing.

There are others from the multipath virtual interim (for instance, sharing bandwidth if multiple paths have roughly equivalent RTTs, and using the path with the lowest RTT if they don't).

So, I agree with Christian about needing more work on this, and it's not clear to me how many "multipath schedulers" we should be thinking about - but that's a question for another day.

Best,

Spencer

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Nov 2020 14:08:50 -0600

Hi, Mirja,

> https://tools.ietf.org/html/draft-bonaventure-iccrg-schedulers-00

Right, and the datatracker shows there's a -01. I know the authors are
still active on it as well.

My goal was to characterize what's being talked about in the QUIC context,
but the two efforts are definitely related, and will need to connect at an
appropriate time.

I'm assuming that Jana will tell me when that time is, since he's chairing
ICCRG :-)

Best,

Spencer

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 12 Nov 2020 20:17:12 +0000

See below.

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thursday, 12. November 2020 at 21:09
To: Mirja Kuehlewind <[email-masked]>
Cc: Christian Huitema <[email-masked]>, "Ma, Yunfei" <[email-masked]>, "Liu, Hongqiang(洪强)" <[email-masked]>, healing4d <[email-masked]>, Yanmei Liu <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, quic <[email-masked]>, "安勍(莳逸)" <[email-masked]>
Subject: Re: What to do about multipath in QUIC

Hi, Mirja,

On Thu, Nov 12, 2020 at 1:47 PM Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>> wrote:
There is already work on this in ICCRG:

Right, and the datatracker shows there's a -01. I know the authors are still active on it as well.

My goal was to characterize what's being talked about in the QUIC context, but the two efforts are definitely related, and will need to connect at an appropriate time.

They are definitely more than connected as the question about scheduling and the respective interface to the higher layer is really independent of any questions on how multipath support is signaled and realized on the wire. So any experience we have from MPTCP on scheduling is directly applicable to any other multipath capable protocol as well. That’s also why ICCRG is a good place to discuss these things.

Mirja

I'm assuming that Jana will tell me when that time is, since he's chairing ICCRG :-)

Best,

Spencer

---

From: Yanmei Liu <[email-masked]>
Date: Fri, 13 Nov 2020 19:01:20 +0800

Hi Mikkel,

Thanks a lot for the feedback.

Yeah, we also find that the handshake for the secondary sub-connections is
a little too heavy, so we are trying to simplify it.

We allow each sub-connection to have a different timeout, and when one of
them dies, the entire connection remain alive until all the sub-connections
die (in our implementation). but we will take the situation dealing with
NAT gateways into consideration.

And thank you again for all the advice :)

Thanks,
Yanmei

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 13 Nov 2020 09:29:32 -0600

I disagree with all these efforts to reinvent mpquic.
mpquic has already been specified we do have a draft which has already been
revised several times.
Why not revise that further to make it "prefect"?

I also suggest changing the name from mpquic which is too much TCP-oriented
to Multiple Interface QUIC or MIQUIC (a little bit of Spanish :)).

Behcet

On Thu, Nov 12, 2020 at 11:05 AM Christian Huitema <[email-masked]>
wrote:

> The use of AEAD is only safe if the same packet number is not reused twice
> with the same key. If we use multiple packet number contexts, AEAD is only
> safe if these contexts use different encryption keys. This requires adding
> a key derivation procedure for the "sub connection", and also adding ways
> to identify the relevant key in the incoming packets. This gets complicated
> very quickly, especially if we want to keep the possibility of using
> zero-length connection identifiers on the client side.
>
> manipulate paths, so the client can instruct the server when paths ought to
> be abandoned. Otherwise, I just keep track of which PN maps to which path.
>
>

---

From: Yanmei Liu <[email-masked]>
Date: Mon, 16 Nov 2020 07:24:52 +0800

Hi Christian and Lucas,

Thanks a lot for the advice :-)

> The use of AEAD is only safe if the same packet number is not reused
twice with the same key. If we use multiple packet number contexts, AEAD is
only safe if these contexts use different encryption keys. This requires
adding a key derivation procedure for the "sub connection", and also adding
ways to identify the relevant key in the incoming packets. This gets
complicated very quickly, especially if we want to keep the possibility of
using zero-length connection identifiers on the client side.

We have tried to use the same packet number space in all the paths (or
sub-connections) before, and have found that it brought much complexity in
implementation for loss recovery.
In the meanwhile, the AEAD security problem mentioned above should be
solved. Another way to solve this problem is using different keys in
different paths, but it also brings much complexity in key derivation as
you have mentioned.

We have found a third solution in
https://tools.ietf.org/id/draft-huitema-quic-mpath-req-01.txt : create
nonces by mixing the IV with both the path specific connection ID and the
packet sequence number.
If we can mix Destination Connection ID in for each packet's AEAD nonce,
then it will be safe to use the same key in all the paths with different
packet number spaces.
But since QUIC-TLS has been in the last-call period, would it be able to
add this modification into QUIC-TLS?

Thanks,
Yanmei

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Nov 2020 14:19:24 +0900

> then it will be safe to use the same key in all the paths with different

Or as an alternative, we can encode the sequence number of the connection
ID directly in the unused part of AEAD nonce (size of nonce is 96 bits in
AES-GCM, 128 bits in chacha20-poly1305, but we only use 62 bits). The
benefit of such an approach is that endpoints would not be required to have
additional state related to AEAD. Endpoints already have the mapping
between connection IDs and their sequence numbers, all they need to do is
pass that sequence number as part of the AEAD nonce.

> add this modification into QUIC-TLS?

I do not think we have to, especially if we embed the sequence number of
the connection ID into the AEAD nonce.

>> voice segments (Apple Siri), avoiding buffering delays when playing music
>> (Apple Music), and using two available links with equal preference (Ali
>> Baba "high speed train"). I wish that we could distillate that into a
>> couple of simple concepts.
>>
>> -- Christian Huitema
>>
>>

-- 
Kazuho Oku

---

From: "Ma, Yunfei" <[email-masked]>
Date: Mon, 16 Nov 2020 15:24:27 +0800

Dear Kazuho,

Thanks for providing this alternative solution. I think it is great, but please correct me if I am wrong. In the quic-tls-32 draft section 5.3., it reads:
    ​"The exclusive OR of the padded packet number and the IV forms the AEAD nonce."
So my question is: if we want to embed the sequence number of the connection ID into the AEAD nonce as you pointed out, don't we need to incoporate this method (or a sentence) into the section 5.3.?

Thanks,
Yunfei

from Alimail macOS
 ------------------Original Mail ------------------
Sender:Kazuho Oku <[email-masked]>
Send Date:Sun Nov 15 21:19:39 2020
Recipients:Yanmei Liu <[email-masked]>
CC:Christian Huitema <[email-masked]>, Ma, Yunfei <[email-masked]>, Hongqiang Liu <[email-masked]>, Yanmei Liu <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, quic <[email-masked]>, Qing An <[email-masked]>
Subject:Re: What to do about multipath in QUIC

Hi Christian and Lucas,

> The use of AEAD is only safe if the same packet number is not reused twice with the same key. If we use multiple packet number contexts, AEAD is only safe if these contexts use different encryption keys. This requires adding a key derivation procedure for the "sub connection", and also adding ways to identify the relevant key in the incoming packets. This gets complicated very quickly, especially if we want to keep the possibility of using zero-length connection identifiers on the client side. 
> I use a concept very similar to the sub-connection, but only as a way to manipulate paths, so the client can instruct the server when paths ought to be abandoned. Otherwise, I just keep track of which PN maps to which path.

We have tried to use the same packet number space in all the paths (or sub-connections) before, and have found that it brought much complexity in implementation for loss recovery. 
In the meanwhile, the AEAD security problem mentioned above should be solved. Another way to solve this problem is using different keys in different paths, but it also brings much complexity in key derivation as you have mentioned.

We have found a third solution in https://tools.ietf.org/id/draft-huitema-quic-mpath-req-01.txt : create nonces by mixing the IV with both the path specific connection ID and the packet sequence number. 
If we can mix Destination Connection ID in for each packet's AEAD nonce, then it will be safe to use the same key in all the paths with different packet number spaces. 

Or as an alternative, we can encode the sequence number of the connection ID directly in the unused part of AEAD nonce (size of nonce is 96 bits in AES-GCM, 128 bits in chacha20-poly1305, but we only use 62 bits). The benefit of such an approach is that endpoints would not be required to have additional state related to AEAD. Endpoints already have the mapping between connection IDs and their sequence numbers, all they need to do is pass that sequence number as part of the AEAD nonce.

But since QUIC-TLS has been in the last-call period, would it be able to add this modification into QUIC-TLS?

I do not think we have to, especially if we embed the sequence number of the connection ID into the AEAD nonce.

Thanks,
Yanmei

  2. We find that the simplest way to add a second path is to use a sub-connection. The concept of sub-connection is directly inherited from connection in QUIC-transport, defined as the logic session of each physical path. Similar to a connection, each sub-connection has its own packet number space for the purpose of loss detection and recovery. 
I did not want to do that in my own draft for a couple of reasons. The main one is the interaction with encryption.
The use of AEAD is only safe if the same packet number is not reused twice with the same key. If we use multiple packet number contexts, AEAD is only safe if these contexts use different encryption keys. This requires adding a key derivation procedure for the "sub connection", and also adding ways to identify the relevant key in the incoming packets. This gets complicated very quickly, especially if we want to keep the possibility of using zero-length connection identifiers on the client side.
I use a concept very similar to the sub-connection, but only as a way to manipulate paths, so the client can instruct the server when paths ought to be abandoned. Otherwise, I just keep track of which PN maps to which path.

-- 
Kazuho Oku

---

From: Yunfei Ma <[email-masked]>
Date: Sun, 15 Nov 2020 23:32:02 -0800

Dear Kazuho,

Thanks for providing this alternative solution. I think it's great, but
please correct me if I am wrong. In the quic-tls-32 section 5.3., it reads:

*"The exclusive OR of the padded packet number and the IV forms the
AEAD nonce."*

So my question is: if we want to embed the sequence number of the
connection ID into the AEAD nonce, don't we need to incorporate this
method into section 5.3.?

Thanks,

Yunfei

On Sun, Nov 15, 2020 at 9:19 PM Kazuho Oku <[email-masked]> wrote:

>> > I use a concept very similar to the sub-connection, but only as a way
>> to manipulate paths, so the client can instruct the server when paths ought

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Nov 2020 16:53:43 +0900

2020年11月16日(月) 16:32 Yunfei Ma <[email-masked]>:

> *"The exclusive OR of the padded packet number and the IV forms the AEAD nonce."*
>
>
> So my question is: if we want to embed the sequence number of the connection ID into the AEAD nonce, don't we need to incorporate this method into section 5.3.?
>
>
That's true. AEAD takes three inputs: secret, IV,  nonce.

In order to avoid reuse, we need to make either of the three a per-path
variable. My point is that changing the definition of nonce is the easiest,
because it is a value that is calculated for each encrypt / decrypt
operation, and because we have space to incorporate the path identifier
(being the sequence number included in the NEW_CONNECTION_ID frame).

That's going to be a small enough change that can be made in the multipath
extension.

-- 
Kazuho Oku

---

From: Florentin Rochet <[email-masked]>
Date: Mon, 16 Nov 2020 17:46:49 +0100

Hello,

I have suggested a similar idea than Kazuho in Quentin Deconinck's draft:

https://tools.ietf.org/html/draft-deconinck-quic-multipath-06#section-8.1

Playing with the IV allows the endpoints to do an implicit derivation of 
the new crypto material for the new paths (using the same key). Besides, 
a multi-key settings makes a security degradation [0], so it might be 
better to avoid it if it's easy enough :-)

Best,

Florentin

[0] https://www.isg.rhul.ac.uk/~kp/TLS-AEbounds.pdf  Section 1.3

Le 16/11/20 à 08:53, Kazuho Oku a écrit :
>
>
> 2020年11月16日(月) 16:32 Yunfei Ma <[email-masked] 
> <mailto:[email-masked]>>:
>
>     Dear Kazuho,
>
>     Thanks for providing this alternative solution. I think it's
>     great, but please correct me if I am wrong. In the quic-tls-32
>     section 5.3., it reads:
>
>     _"The exclusive OR of the padded packet number and the IV forms
>     the AEAD nonce."_
>
>     __
>
>     connection ID into the AEAD nonce, don't we need to incorporate
>     this method into section 5.3.?
>
>
> In order to avoid reuse, we need to make either of the three a 
> per-path variable. My point is that changing the definition of nonce 
> is the easiest, because it is a value that is calculated for each 
> encrypt / decrypt operation, and because we have space to incorporate 
> the path identifier (being the sequence number included in the 
> NEW_CONNECTION_ID frame).
>
> That's going to be a small enough change that can be made in the 
> multipath extension.
>
>     Thanks,
>
>     Yunfei
>
>
>     On Sun, Nov 15, 2020 at 9:19 PM Kazuho Oku <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>
>
>         2020年11月16日(月) 8:25 Yanmei Liu <[email-masked]
>         <mailto:[email-masked]>>:
>
>             Hi Christian and Lucas,
>
>             > The use of AEAD is only safe if the same packet number
>             is not reused twice with the same key. If we use multiple
>             packet number contexts, AEAD is only safe if these
>             contexts use different encryption keys. This requires
>             adding a key derivation procedure for the "sub
>             connection", and also adding ways to identify the relevant
>             key in the incoming packets. This gets complicated very
>             quickly, especially if we want to keep the possibility of
>             > I use a concept very similar to the sub-connection, but
>             only as a way to manipulate paths, so the client can
>             instruct the server when paths ought to be abandoned.
>             Otherwise, I just keep track of which PN maps to which path.
>
>             We have tried to use the same packet number space in all
>             the paths (or sub-connections) before, and have found that
>             it brought much complexity in implementation for loss
>             recovery.
>             In the meanwhile, the AEAD security problem mentioned
>             above should be solved. Another way to solve this problem
>             is using different keys in different paths, but it also
>             brings much complexity in key derivation as you have
>             mentioned.
>
>             https://tools.ietf.org/id/draft-huitema-quic-mpath-req-01.txt
>             : create nonces by mixing the IV with both the path
>             specific connection ID and the packet sequence number.
>             If we can mix Destination Connection ID in for each
>             packet's AEAD nonce, then it will be safe to use the same
>             key in all the paths with different packet number spaces.
>
>
>         Or as an alternative, we can encode the sequence number of the
>         connection ID directly in the unused part of AEAD nonce (size
>         of nonce is 96 bits in AES-GCM, 128 bits in chacha20-poly1305,
>         but we only use 62 bits). The benefit of such an approach is
>         that endpoints would not be required to have additional state
>         related to AEAD. Endpoints already have the mapping between
>         connection IDs and their sequence numbers, all they need to do
>         is pass that sequence number as part of the AEAD nonce.
>
>             But since QUIC-TLS has been in the last-call period, would
>             it be able to add this modification into QUIC-TLS?
>
>
>         I do not think we have to, especially if we embed the sequence
>         number of the connection ID into the AEAD nonce.
>
>
>
>             Thanks,
>             Yanmei
>
>
>
>
>             On Fri, 13 Nov 2020 at 01:04, Christian Huitema
>             <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>                 I did not want to do that in my own draft for a couple
>                 of reasons. The main one is the interaction with
>                 encryption.
>
>                 The use of AEAD is only safe if the same packet number
>                 is not reused twice with the same key. If we use
>                 multiple packet number contexts, AEAD is only safe if
>                 these contexts use different encryption keys. This
>                 requires adding a key derivation procedure for the
>                 "sub connection", and also adding ways to identify the
>                 relevant key in the incoming packets. This gets
>                 complicated very quickly, especially if we want to
>                 keep the possibility of using zero-length connection
>                 identifiers on the client side.
>
>                 I use a concept very similar to the sub-connection,
>                 but only as a way to manipulate paths, so the client
>                 can instruct the server when paths ought to be
>                 abandoned. Otherwise, I just keep track of which PN
>                 maps to which path.
>
>                 I think we need more work on the
>                 "multi-path scheduler". We have heard of three
>                 application scenarios: maintaining the lowest RTT when
>                 sending voice segments (Apple Siri), avoiding
>                 buffering delays when playing music (Apple Music), and
>                 using two available links with equal preference (Ali
>                 Baba "high speed train"). I wish that we could
>                 distillate that into a couple of simple concepts.
>
>                 -- Christian Huitema
>
>
>
>         -- 
>         Kazuho Oku
>
>
>
> -- 
> Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Mon, 16 Nov 2020 10:11:31 -0800

Kazuho,

You are proposing to use a transform, in which the AEAD "sequence" is
the conjunction of a sequence number space identifier and the number
within that space. I agree that doing that solves the uniqueness
requirement, but I have two concerns.

First, your original proposal allocates just 2 bits for the sequence
number space identifier. You observe that the sequence number is encoded
as a varint, which can only encode 62 bits of value, and thus we can
reuse the two high order bits to encode a number space identifier. But
what happens if the connection uses more than 4 number spaces?

Second, we have a synchronization issue. At what point do nodes start
using the "path-specific" number space? If we want to be robust, don't
we need to encode a number space identifier in the packet header?

-- Christian Huitema

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Nov 2020 07:05:42 +0900

> what happens if the connection uses more than 4 number spaces?

The "nonce" of AEAD, which is used for encoding the "packet number" of
QUIC, has more space than 64 bits. I know that some of the TLS stacks only
expose an AEAD API that lets applications specify 64-bit packet number, but
they are (I am) just lazy.

Quoting from RFC 5116,
> Each AEAD algorithm MUST accept any nonce with a length between N_MIN and
N_MAX octets, inclusive, where the values of N_MIN and N_MAX are specific
to that algorithm.  The values of N_MAX and N_MIN MAY be equal.  Each
algorithm SHOULD accept a nonce with a length of twelve (12) octets.

AES-GCM and chacha20-poly1305 have N_MIN and N_MAX of 12.

Therefore, we have 34-bit space (12 * 8 - 62 = 34) to encode the path
identifier.

> we need to encode a number space identifier in the packet header?

In QUIC, each CID has a sequence number, and the use of CIDs with a
non-zero sequence number happens after the handshake is confirmed, at which
point negotiation of the protocol features would have completed.

Therefore, multipath-aware endpoints can determine the path identifier
(i.e. sequence number) from the CID it sends / receives, and encode that as
part of the nonce that they use to encrypt / decrypt packets.

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Mon, 16 Nov 2020 15:12:10 -0800

On 11/16/2020 2:05 PM, Kazuho Oku wrote:
>
>
> 2020年11月17日(火) 3:11 Christian Huitema <[email-masked]
> <mailto:[email-masked]>>:
>
>     Kazuho,
>
> QUIC, has more space than 64 bits. I know that some of the TLS stacks
> only expose an AEAD API that lets applications specify 64-bit packet
> number, but they are (I am) just lazy.
>
> Quoting from RFC 5116,
> > Each AEAD algorithm MUST accept any nonce with a length between
> N_MIN and N_MAX octets, inclusive, where the values of N_MIN and N_MAX
> are specific to that algorithm.  The values of N_MAX and N_MIN MAY be
> equal.  Each algorithm SHOULD accept a nonce with a length of twelve
> (12) octets.
>

I suppose that 32 bits path-id would be enough, yes.

> non-zero sequence number happens after the handshake is confirmed, at
> which point negotiation of the protocol features would have completed.
>
> (i.e. sequence number) from the CID it sends / receives, and encode
> that as part of the nonce that they use to encrypt / decrypt packets.

Are you going to conflate CID sequence number and PN Number Space
identifier? That would have interesting effects on scenarios in which
the CID changes for other reasons than creating a new path. There would
also be an interesting relation between "retiring a CID" and "managing
acknowledgements". It would also make the creation of new CID
significantly more heavy weight than it is today -- the peer that
creates a new CID would have to allocate a selective ACK dashboard for
packets received through that CID.

Then there is the issue of peers who want to use zero length CID. I
suppose that if the peer has zero length CID, all packets sent to that
peer will be in the same number space. That may well be just fine if
that peer is not sending a lot of data, e.g., for clients that mostly
send short requests and acks.

Then there is the issue of key rotation. In this "path sequence" scheme,
we have only one key, so I assume that it will rotate simultaneously for
all CID.

-- Christian Huitema

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Nov 2020 08:33:43 +0900

2020年11月17日(火) 8:12 Christian Huitema <[email-masked]>:

> > Each AEAD algorithm MUST accept any nonce with a length between N_MIN
> and N_MAX octets, inclusive, where the values of N_MIN and N_MAX are
> specific to that algorithm.  The values of N_MAX and N_MIN MAY be equal.
> Each algorithm SHOULD accept a nonce with a length of twelve (12) octets.
>
> identifier? That would have interesting effects on scenarios in which the
> CID changes for other reasons than creating a new path. There would also be
> an interesting relation between "retiring a CID" and "managing
> acknowledgements". It would also make the creation of new CID significantly
> more heavy weight than it is today -- the peer that creates a new CID would
> have to allocate a selective ACK dashboard for packets received through
> that CID.
>
That's a good point. I think there isn't a strong reason to tie CID
sequence number to packet numbers, if we are going to have "path
identifiers" as a separate concept. Anyways, that identifier can be
embedded in the nonce.

> suppose that if the peer has zero length CID, all packets sent to that peer
> will be in the same number space. That may well be just fine if that peer
> is not sending a lot of data, e.g., for clients that mostly send short
> requests and acks.
>
IIUC, endpoints using zero-length CID cannot issue new connection IDs, and
therefore cannot sustain through intentional migration [1].

> we have only one key, so I assume that it will rotate simultaneously for
Agreed.

> -- Christian Huitema

[1]
https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-5.1.1-8

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Mon, 16 Nov 2020 15:59:46 -0800

On 11/16/2020 3:33 PM, Kazuho Oku wrote:
>
>
> 2020年11月17日(火) 8:12 Christian Huitema <[email-masked]
>>     The "nonce" of AEAD, which is used for encoding the "packet
>>     number" of QUIC, has more space than 64 bits. I know that some of
>>     the TLS stacks only expose an AEAD API that lets applications
>>     specify 64-bit packet number, but they are (I am) just lazy.
>>
>>     Quoting from RFC 5116,
>>     N_MIN and N_MAX octets, inclusive, where the values of N_MIN and
>>     N_MAX are specific to that algorithm.  The values of N_MAX and
>>     N_MIN MAY be equal.  Each algorithm SHOULD accept a nonce with a
>>     length of twelve (12) octets.
>>
>>     Therefore, we have 34-bit space (12 * 8 - 62 = 34) to encode the
>>     path identifier.
>
>>     In QUIC, each CID has a sequence number, and the use of CIDs with
>>     a non-zero sequence number happens after the handshake is
>>     confirmed, at which point negotiation of the protocol features
>>     would have completed.
>>
>>     Therefore, multipath-aware endpoints can determine the path
>>     identifier (i.e. sequence number) from the CID it sends /
>>     receives, and encode that as part of the nonce that they use to
>>     encrypt / decrypt packets.
>
>     identifier? That would have interesting effects on scenarios in
>     which the CID changes for other reasons than creating a new path.
>     There would also be an interesting relation between "retiring a
>     CID" and "managing acknowledgements". It would also make the
>     creation of new CID significantly more heavy weight than it is
>     today -- the peer that creates a new CID would have to allocate a
>     selective ACK dashboard for packets received through that CID.
>

But then we are back to square one. In order to decrypt the packet, you
need to know the number-space identifier. That means first identifying
the incoming path-ID based on four-tuple and CID, and then finding
whether the number space associated with the path. That leaves the issue
of the initial packets for a path, e.g., the first challenge packets.
For those, the path-id may not yet be established, and the path-specific
number space not yet initialized.

Maybe what we need is just one bit in the header to distinguish between
"this packet uses the default number space" and "this packet uses the
path specific number space". That, or trial decryption.

>     Then there is the issue of peers who want to use zero length CID.
>     I suppose that if the peer has zero length CID, all packets sent
>     to that peer will be in the same number space. That may well be
>     just fine if that peer is not sending a lot of data, e.g., for
>     clients that mostly send short requests and acks.
>
> IIUC, endpoints using zero-length CID cannot issue new connection IDs,
> and therefore cannot sustain through intentional migration [1].

That's incorrect. If the server supports non-zero CID, then the client
can do intentional migration, even if the client itself uses zero-length
CID. I have tested that behavior for Picoquic, and I believe it is also
used in Quic-Go.

>     Then there is the issue of key rotation. In this "path sequence"
>     scheme, we have only one key, so I assume that it will rotate
>     simultaneously for all CID.
>
> Agreed.
>
> [1]

-- Christian huitema

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Nov 2020 10:41:06 +0900

2020年11月17日(火) 9:26 Christian Huitema <[email-masked]>:

> need to know the number-space identifier. That means first identifying the
> incoming path-ID based on four-tuple and CID, and then finding whether the
> number space associated with the path. That leaves the issue of the initial
> packets for a path, e.g., the first challenge packets. For those, the
> path-id may not yet be established, and the path-specific number space not
> yet initialized.
>
Then, I might go back to encoding the CID sequence number in the AEAD nonce.

A sender does not have to reset PN for each CID. It can retain its internal
N:1 mapping between CIDs and the paths. A receiver can decode packets using
the CID and PN, retain ACK queue for each active CID, then send ACKs for
each CID identified by the sequence number.

> "this packet uses the default number space" and "this packet uses the path
> specific number space". That, or trial decryption.
>
> That's incorrect. If the server supports non-zero CID, then the client can
> do intentional migration, even if the client itself uses zero-length CID. I
> have tested that behavior for Picoquic, and I believe it is also used in
> Quic-Go.
>
Thank you for pointing that out. I think you are correct.

I tend to think that the multipath extension can be simpler if we avoid
introducing the concept of "path identifiers," and instead use CID sequence
numbers for identifying the paths, as stated above. But that does prohibit
clients using zero-length CIDs.

Personally I am fine with eliminating this asymmetry in the specification
that allows only clients to use zero-length CIDs, but maybe others have
different opinions.

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Tue, 17 Nov 2020 01:00:19 -0800

On 11/16/2020 5:41 PM, Kazuho Oku wrote:
>
>
> 2020年11月17日(火) 9:26 Christian Huitema <[email-masked]
>>>         number" of QUIC, has more space than 64 bits. I know that
>>>         some of the TLS stacks only expose an AEAD API that lets
>>>         applications specify 64-bit packet number, but they are (I
>>>         am) just lazy.
>>>
>>>         Quoting from RFC 5116,
>>>         > Each AEAD algorithm MUST accept any nonce with a length
>>>         between N_MIN and N_MAX octets, inclusive, where the values
>>>         of N_MIN and N_MAX are specific to that algorithm.  The
>>>         values of N_MAX and N_MIN MAY be equal.  Each algorithm
>>>         SHOULD accept a nonce with a length of twelve (12) octets.
>>>
>>>         Therefore, we have 34-bit space (12 * 8 - 62 = 34) to encode
>>>         the path identifier.
>>
>>>         In QUIC, each CID has a sequence number, and the use of CIDs
>>>         with a non-zero sequence number happens after the handshake
>>>         is confirmed, at which point negotiation of the protocol
>>>         features would have completed.
>>>
>>>         receives, and encode that as part of the nonce that they use
>>>         to encrypt / decrypt packets.
>>
>>         Are you going to conflate CID sequence number and PN Number
>>         Space identifier? That would have interesting effects on
>>         scenarios in which the CID changes for other reasons than
>>         creating a new path. There would also be an interesting
>>         relation between "retiring a CID" and "managing
>>         significantly more heavy weight than it is today -- the peer
>>         that creates a new CID would have to allocate a selective ACK
>>         dashboard for packets received through that CID.
>>
>>     That's a good point. I think there isn't a strong reason to tie
>>     CID sequence number to packet numbers, if we are going to have
>>     "path identifiers" as a separate concept. Anyways, that
>>     identifier can be embedded in the nonce.
>
>     But then we are back to square one. In order to decrypt the
>     packet, you need to know the number-space identifier. That means
>     first identifying the incoming path-ID based on four-tuple and
>     CID, and then finding whether the number space associated with the
>     path. That leaves the issue of the initial packets for a path,
>     e.g., the first challenge packets. For those, the path-id may not
>     yet be established, and the path-specific number space not yet
>     initialized.
>
> A sender does not have to reset PN for each CID. It can retain its
> internal N:1 mapping between CIDs and the paths. A receiver can decode
> packets using the CID and PN, retain ACK queue for each active CID,
> then send ACKs for each CID identified by the sequence number.

You could do that, but the sender has to guarantee that a given CID will
always map to the same PN Space. Otherwise, duplication can happen.

>  
>
>     Maybe what we need is just one bit in the header to distinguish
>     between "this packet uses the default number space" and "this
>     packet uses the path specific number space". That, or trial
>     decryption.
>
I think that if we go for "CID Sequence in PN Space", we probably need
this bit to distinguish between probe packets sent with a PN from the
space number 0 (the default) and further packets sent on a PN Space
negotiated per path.

>>         Then there is the issue of peers who want to use zero length
>>         CID. I suppose that if the peer has zero length CID, all
>>         packets sent to that peer will be in the same number space.
>>         That may well be just fine if that peer is not sending a lot
>>         of data, e.g., for clients that mostly send short requests
>>         and acks.
>>
>>     IIUC, endpoints using zero-length CID cannot issue new connection
>>     IDs, and therefore cannot sustain through intentional migration [1].
>
>     That's incorrect. If the server supports non-zero CID, then the
>     client can do intentional migration, even if the client itself
>     uses zero-length CID. I have tested that behavior for Picoquic,
>     and I believe it is also used in Quic-Go.
>
> I tend to think that the multipath extension can be simpler if we
> avoid introducing the concept of "path identifiers," and instead use
> CID sequence numbers for identifying the paths, as stated above. But
> that does prohibit clients using zero-length CIDs.

I am unconvinced. If we have different PN spaces, we need different ACK
spaces, a new ACK format that somehow includes an ID of the PN space.
Unless you have a struct 1-1 mapping between PN space and CID space, I
think that you end up requiring a new identifier.

>
> Personally I am fine with eliminating this asymmetry in the
> specification that allows only clients to use zero-length CIDs, but
> maybe others have different opinions.
>
>>         Then there is the issue of key rotation. In this "path
>>         sequence" scheme, we have only one key, so I assume that it
>>         will rotate simultaneously for all CID.
>>
>>     Agreed.
>>
>>     [1]
>>     <https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-5.1.1-8>
>
-- Christian Huitema

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Nov 2020 19:29:17 +0900

2020年11月17日(火) 18:00 Christian Huitema <[email-masked]>:

> packets using the CID and PN, retain ACK queue for each active CID, then
> send ACKs for each CID identified by the sequence number.
>
>
I believe we require an endpoint to use an unused CID whenever it attempts
to use a new path?

> this bit to distinguish between probe packets sent with a PN from the space
> number 0 (the default) and further packets sent on a PN Space negotiated
> per path.
>
I'm not sure if I agree.

I tend to think that there would be a M:1 mapping between CIDs and paths.
In most cases, M will be one, as the peers might stop issuing new CIDs at
any moment.

> think that you end up requiring a new identifier.
Yes, we need to change the ACK format to include the CID sequence number,
because the receiver does not have the concept of paths.

My point is that "path" can be an internal property of the sender, that
maps M:1 to the CID, where M can be 1.

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Tue, 17 Nov 2020 09:41:28 -0800

On 11/17/2020 2:29 AM, Kazuho Oku wrote:

>
>
> 2020年11月17日(火) 18:00 Christian Huitema <[email-masked]
>>>>             2020年11月17日(火) 3:11 Christian Huitema
>>>>             <[email-masked] <mailto:[email-masked]>>:
>>>>
>>>>                 Kazuho,
>>>>
>>>>             The "nonce" of AEAD, which is used for encoding the
>>>>             "packet number" of QUIC, has more space than 64 bits. I
>>>>             know that some of the TLS stacks only expose an AEAD
>>>>             API that lets applications specify 64-bit packet
>>>>             > Each AEAD algorithm MUST accept any nonce with a
>>>>             length between N_MIN and N_MAX octets, inclusive, where
>>>>             the values of N_MIN and N_MAX are specific to that
>>>>             algorithm.  The values of N_MAX and N_MIN MAY be
>>>>             equal.  Each algorithm SHOULD accept a nonce with a
>>>>             Therefore, we have 34-bit space (12 * 8 - 62 = 34) to
>>>>             encode the path identifier.
>>>
>>>>             In QUIC, each CID has a sequence number, and the use of
>>>>             CIDs with a non-zero sequence number happens after the
>>>>             handshake is confirmed, at which point negotiation of
>>>>             the protocol features would have completed.
>>>>
>>>>             Therefore, multipath-aware endpoints can determine the
>>>>             path identifier (i.e. sequence number) from the CID it
>>>>             sends / receives, and encode that as part of the nonce
>>>>             that they use to encrypt / decrypt packets.
>>>
>>>             Are you going to conflate CID sequence number and PN
>>>             Number Space identifier? That would have interesting
>>>             effects on scenarios in which the CID changes for other
>>>             reasons than creating a new path. There would also be an
>>>             interesting relation between "retiring a CID" and
>>>             "managing acknowledgements". It would also make the
>>>             creation of new CID significantly more heavy weight than
>>>             it is today -- the peer that creates a new CID would
>>>             have to allocate a selective ACK dashboard for packets
>>>             received through that CID.
>>>
>>>         That's a good point. I think there isn't a strong reason to
>>>         tie CID sequence number to packet numbers, if we are going
>>>         to have "path identifiers" as a separate concept. Anyways,
>>>         that identifier can be embedded in the nonce.
>>
>>         packet, you need to know the number-space identifier. That
>>         means first identifying the incoming path-ID based on
>>         four-tuple and CID, and then finding whether the number space
>>         associated with the path. That leaves the issue of the
>>         initial packets for a path, e.g., the first challenge
>>         packets. For those, the path-id may not yet be established,
>>         and the path-specific number space not yet initialized.
>>
>>     A sender does not have to reset PN for each CID. It can retain
>>     its internal N:1 mapping between CIDs and the paths. A receiver
>>     can decode packets using the CID and PN, retain ACK queue for
>>     each active CID, then send ACKs for each CID identified by the
>>     sequence number.
>
>
>     You could do that, but the sender has to guarantee that a given
>     CID will always map to the same PN Space. Otherwise, duplication
>     can happen.
>
> I believe we require an endpoint to use an unused CID whenever it
> attempts to use a new path?

Yes, unless it uses zero-length CID.

>>      
>>
>>         Maybe what we need is just one bit in the header to
>>         distinguish between "this packet uses the default number
>>         space" and "this packet uses the path specific number space".
>>         That, or trial decryption.
>>
>     I think that if we go for "CID Sequence in PN Space", we probably
>     need this bit to distinguish between probe packets sent with a PN
>     from the space number 0 (the default) and further packets sent on
>     a PN Space negotiated per path.
>
> I'm not sure if I agree.
>
> I tend to think that there would be a M:1 mapping between CIDs and
> paths. In most cases, M will be one, as the peers might stop issuing
> new CIDs at any moment.

Yes. That's the beauty of reusing as much as possible of the migration
work. We already have the infrastructure in place to issue and retire
CID, and to limit how many can be requested.

>
>>>             Then there is the issue of peers who want to use zero
>>>             length CID. I suppose that if the peer has zero length
>>>             CID, all packets sent to that peer will be in the same
>>>             number space. That may well be just fine if that peer is
>>>             not sending a lot of data, e.g., for clients that mostly
>>>         IIUC, endpoints using zero-length CID cannot issue new
>>>         connection IDs, and therefore cannot sustain through
>>>         intentional migration [1].
>>
>>         That's incorrect. If the server supports non-zero CID, then
>>         the client can do intentional migration, even if the client
>>         itself uses zero-length CID. I have tested that behavior for
>>         Picoquic, and I believe it is also used in Quic-Go.
>>
>>     avoid introducing the concept of "path identifiers," and instead
>>     use CID sequence numbers for identifying the paths, as stated
>>     above. But that does prohibit clients using zero-length CIDs.
>
>     I am unconvinced. If we have different PN spaces, we need
>     different ACK spaces, a new ACK format that somehow includes an ID
>     of the PN space. Unless you have a struct 1-1 mapping between PN
>     space and CID space, I think that you end up requiring a new
>     identifier.
>
> Yes, we need to change the ACK format to include the CID sequence
> number, because the receiver does not have the concept of paths.
>
> My point is that "path" can be an internal property of the sender,
> that maps M:1 to the CID, where M can be 1.

I have been thinking about variations of that. I think we are making
progress here.

If we follow your design, we get two constraints:

1) That the receive maintains an acknowledgement list based on the CID
through which the packets are received.

2) That the senders guarantee that the same sequence number will not be
used more than once with a specific CID.

The main implementation cost is for receivers. They have to allocate and
maintain a "SACK Dashboard" in the context of each CID that they issue.

Senders have lots of control. For example, the "only once" condition is
also met if a simple sender uses a single number space, as long as it
does not send more than 2^32 packets. That makes the design reasonably
easy to implement for constrained implementations.

Zero length CID are still possible, but that means the receiver supports
only one PN space per sender. Multipath is not impossible, but you end
up managing a single RTT and a single recovery structure. Not very good,
but similar to what happens if multipath is implemented at the IP level.

There is still an issue for coordinating the take down of a path.
Suppose that a client was using both Wi-Fi and LTE, and moves out of
Wi-Fi range. The server will find out eventually that the packets sent
to the Wi-Fi path are never acknowledged, but that may take some time.
It would be better if the client could send a message saying something
like "Abandon this path". That's not the same semantic as "retire this
CID". We need a new frame for that.

"Abandon this path" is an extreme case. There are half-way steps, like
manage the relative priority of a path.

>>     specification that allows only clients to use zero-length CIDs,
>>     but maybe others have different opinions.
>>
>>>             sequence" scheme, we have only one key, so I assume that
>>>             it will rotate simultaneously for all CID.
>>>
>>>         Agreed.
>>>
>>>         [1]
-- Christian Huitema

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 17 Nov 2020 15:08:44 -0600

Hi, Christian et. al.,

On Tue, Nov 17, 2020 at 11:42 AM Christian Huitema <[email-masked]>
wrote:

> work. We already have the infrastructure in place to issue and retire CID,
> and to limit how many can be requested.
>
>
I think so, too.

I didn't want to interrupt a valuable discussion, but it might be worth
changing the subject line, so it's easier for people who might have tuned
out earlier to notice the details you're working through here.

And thanks for the conversation.

Best,

Spencer

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 24 Nov 2020 19:34:56 -0800

(I'm taking Spencer's suggestion to spin off a new thread.)

Christian, Kazuho,

Slowly catching up on this, and apologies if I'm missing anything that was
previously discussed in the centi-thread earlier.

If I understand the design correctly, it makes sense to me, and is very
close to what we had implemented in Chromium a while ago.

Having thought about this problem several times in the past, I'd like to
share a few points that come to mind.

First though, a point on terminology: the receiver maintains a separate
"ReceivedPackets" for each CID, probably for each CID sequence number
(CSN). Let's please not call this a SACK Dashboard, to avoid confusion.

On the question of sending more than 2^32 packets, I think that resetting
the packet number (PN) is ok on new CIDs. I don't see why a sender would
need to maintain continuity across multiple paths anyways, since the CC and
loss recovery contexts are going to be different across paths. A sender
_could_ still maintain these packets in the same "SentPackets" structure if
it wants to, it would need an internal representation of CSN+PN to key off.

ACK Frames:
------------------
Kazuho pointed out that when acknowledging, the ACK frame format should
include CSN. I agree. I would argue for a design where a receiver uses an
ACK frame per CSN (and encodes the CSN explicitly). There are multiple
values for doing this, the primary being that you benefit from compression
when PNs are contiguous within a CSN.

Return Path:
-----------------
There are other ways to decide which return path to send an ACK on this,
but I would propose that a receiver respond on the most recently active
forward path. That is, the path on which a packet was most recently
received. This has the natural effect that a sender that wants to
distribute traffic in a particular way also causes ACKs to be distributed
similarly across the corresponding reverse paths.

RTT measurements:
---------------------------
The return path for ACK frames will impact RTT measurements. That is fine.
It is more important that information reach the sender as soon as possible
than that it should not affect RTT measurements; we can fix the sender
to measure and compensate as necessary. The estimated RTT statistics
reflect the distribution of samples, and if both paths are being used, then
the SmoothedRTT will reflect the expected value based on the traffic
distribution across paths.

That said, it might be useful to track some new stats, especially about how
much later is a "late ack" -- an ACK frame that contains no useful
information -- is received. I'd have to think a bit more about this, but I
think we can devise a stat here. This gives us useful information on the
longest return path, which we can then explicitly use as part of the PTO
computations, to compensate for the fact that the RTT is based on the
shortest return path. (I would _not_ use this stat in the time-based loss
detection timer,  but PTO ought to be fine.)

- jana

On Tue, Nov 17, 2020 at 9:42 AM Christian Huitema <[email-masked]>
wrote:

> also met if a simple sender uses a single number space, as long as it does
> not send more than 2^32 packets. That makes the design reasonably easy to
> implement for constrained implementations.
>
> only one PN space per sender. Multipath is not impossible, but you end up
> managing a single RTT and a single recovery structure. Not very good, but
> similar to what happens if multipath is implemented at the IP level.
>
> There is still an issue for coordinating the take down of a path. Suppose
> that a client was using both Wi-Fi and LTE, and moves out of Wi-Fi range.
> The server will find out eventually that the packets sent to the Wi-Fi path
> are never acknowledged, but that may take some time. It would be better if
> the client could send a message saying something like "Abandon this path".
> That's not the same semantic as "retire this CID". We need a new frame for
> that.
>

---

From: <[email-masked]>
Date: Thu, 26 Nov 2020 11:34:00 +0000

Dear all,

sry for hijacking this conversation. I’m not very familiar with the different multipath designs for QUIC, however I want to draw attention to multipath re-ordering which probably becomes important when multipath is combined with DATAGRAM.

As long as multipath QUIC is operated with strict reliability (similar to TCP), re-ordering on receiver side is a simple process known from MPTCP. Introducing unreliable DATAGRAM transmission makes it more challenging on receiver side to maintain the packet order, because it is not easy to differentiate between delayed and lost packets. To avoid HoL, a multipath re-ordering process may benefit from having connection and path sequencing. In https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01 we intend to describe this in section 5.6, how fast packet loss detection can be applied using these different packet sequence spaces. Still the description is meaningless und will be updated until next IETF, however we have successfully implemented this approach in a MP-DCCP prototype, which faces similar challenges in terms of re-ordering. That means, fast packet loss detection is very beneficial for the receiver re-ordering process to not lose time until an outstanding packet is assumed lost.

Br

Markus

From: QUIC <[email-masked]> On Behalf Of Jana Iyengar
Sent: Mittwoch, 25. November 2020 04:35
To: Christian Huitema <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>; Kazuho Oku <[email-masked]>
Subject: Packet number spaces in multipath (was Re: What to do about multipath in QUIC)

Christian, Kazuho,

Slowly catching up on this, and apologies if I'm missing anything that was previously discussed in the centi-thread earlier.

If I understand the design correctly, it makes sense to me, and is very close to what we had implemented in Chromium a while ago.

Having thought about this problem several times in the past, I'd like to share a few points that come to mind.

First though, a point on terminology: the receiver maintains a separate "ReceivedPackets" for each CID, probably for each CID sequence number (CSN). Let's please not call this a SACK Dashboard, to avoid confusion.

On the question of sending more than 2^32 packets, I think that resetting the packet number (PN) is ok on new CIDs. I don't see why a sender would need to maintain continuity across multiple paths anyways, since the CC and loss recovery contexts are going to be different across paths. A sender _could_ still maintain these packets in the same "SentPackets" structure if it wants to, it would need an internal representation of CSN+PN to key off.

ACK Frames:
------------------
Kazuho pointed out that when acknowledging, the ACK frame format should include CSN. I agree. I would argue for a design where a receiver uses an ACK frame per CSN (and encodes the CSN explicitly). There are multiple values for doing this, the primary being that you benefit from compression when PNs are contiguous within a CSN.

Return Path:
-----------------
There are other ways to decide which return path to send an ACK on this, but I would propose that a receiver respond on the most recently active forward path. That is, the path on which a packet was most recently received. This has the natural effect that a sender that wants to distribute traffic in a particular way also causes ACKs to be distributed similarly across the corresponding reverse paths.

RTT measurements:
---------------------------
The return path for ACK frames will impact RTT measurements. That is fine. It is more important that information reach the sender as soon as possible than that it should not affect RTT measurements; we can fix the sender to measure and compensate as necessary. The estimated RTT statistics reflect the distribution of samples, and if both paths are being used, then the SmoothedRTT will reflect the expected value based on the traffic distribution across paths.

That said, it might be useful to track some new stats, especially about how much later is a "late ack" -- an ACK frame that contains no useful information -- is received. I'd have to think a bit more about this, but I think we can devise a stat here. This gives us useful information on the longest return path, which we can then explicitly use as part of the PTO computations, to compensate for the fact that the RTT is based on the shortest return path. (I would _not_ use this stat in the time-based loss detection timer,  but PTO ought to be fine.)

- jana

On Tue, Nov 17, 2020 at 9:42 AM Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

I have been thinking about variations of that. I think we are making progress here.

1) That the receive maintains an acknowledgement list based on the CID through which the packets are received.

2) That the senders guarantee that the same sequence number will not be used more than once with a specific CID.

The main implementation cost is for receivers. They have to allocate and maintain a "SACK Dashboard" in the context of each CID that they issue.

Senders have lots of control. For example, the "only once" condition is also met if a simple sender uses a single number space, as long as it does not send more than 2^32 packets. That makes the design reasonably easy to implement for constrained implementations.

Zero length CID are still possible, but that means the receiver supports only one PN space per sender. Multipath is not impossible, but you end up managing a single RTT and a single recovery structure. Not very good, but similar to what happens if multipath is implemented at the IP level.

There is still an issue for coordinating the take down of a path. Suppose that a client was using both Wi-Fi and LTE, and moves out of Wi-Fi range. The server will find out eventually that the packets sent to the Wi-Fi path are never acknowledged, but that may take some time. It would be better if the client could send a message saying something like "Abandon this path". That's not the same semantic as "retire this CID". We need a new frame for that.

"Abandon this path" is an extreme case. There are half-way steps, like manage the relative priority of a path.

---

From: Christian Huitema <[email-masked]>
Date: Thu, 26 Nov 2020 09:53:31 -0800

If you want to send unreliable data with sequencing information, it 
might be simpler to use STREAM frame in "unreliable" mode than to use 
Datagram frames.

On 11/26/2020 3:34 AM, [email-masked] wrote:
>
> Dear all,
>
> sry for hijacking this conversation. I’m not very familiar with the 
> different multipath designs for QUIC, however I want to draw attention 
> to multipath re-ordering which probably becomes important when 
> multipath is combined with DATAGRAM.
>
> As long as multipath QUIC is operated with strict reliability (similar 
> to TCP), re-ordering on receiver side is a simple process known from 
> MPTCP. Introducing unreliable DATAGRAM transmission makes it more 
> challenging on receiver side to maintain the packet order, because it 
> is not easy to differentiate between delayed and lost packets. To 
> avoid HoL, a multipath re-ordering process may benefit from having 
> connection and path sequencing. In 
> https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01 
> <https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01> 
> we intend to describe this in section 5.6, how fast packet loss 
> detection can be applied using these different packet sequence spaces. 
> Still the description is meaningless und will be updated until next 
> IETF, however we have successfully implemented this approach in a 
> MP-DCCP prototype, which faces similar challenges in terms of 
> re-ordering. That means, fast packet loss detection is very beneficial 
> for the receiver re-ordering process to not lose time until an 
> outstanding packet is assumed lost.
>
> Br
>
> Markus
>
> **
>
> *From:*QUIC <[email-masked]> *On Behalf Of *Jana Iyengar
> *Sent:* Mittwoch, 25. November 2020 04:35
> *To:* Christian Huitema <[email-masked]>
> *Cc:* IETF QUIC WG <[email-masked]>; Kazuho Oku <[email-masked]>
> *Subject:* Packet number spaces in multipath (was Re: What to do about 
> multipath in QUIC)
>
> Slowly catching up on this, and apologies if I'm missing anything that 
> was previously discussed in the centi-thread earlier.
>
> If I understand the design correctly, it makes sense to me, and is 
> very close to what we had implemented in Chromium a while ago.
>
> First though, a point on terminology: the receiver maintains a 
> separate "ReceivedPackets" for each CID, probably for each CID 
> sequence number (CSN). Let's please not call this a SACK Dashboard, to 
> avoid confusion.
>
> On the question of sending more than 2^32 packets, I think that 
> resetting the packet number (PN) is ok on new CIDs. I don't see why a 
> sender would need to maintain continuity across multiple paths 
> anyways, since the CC and loss recovery contexts are going to be 
> different across paths. A sender _could_ still maintain these packets 
> in the same "SentPackets" structure if it wants to, it would need an 
> internal representation of CSN+PN to key off.
>
> ACK Frames:
>
> ------------------
>
> Kazuho pointed out that when acknowledging, the ACK frame format 
> should include CSN. I agree. I would argue for a design where a 
> receiver uses an ACK frame per CSN (and encodes the CSN explicitly). 
> There are multiple values for doing this, the primary being that you 
> benefit from compression when PNs are contiguous within a CSN.
>
> Return Path:
>
> -----------------
>
> There are other ways to decide which return path to send an ACK on 
> this, but I would propose that a receiver respond on the most recently 
> active forward path. That is, the path on which a packet was most 
> recently received. This has the natural effect that a sender that 
> wants to distribute traffic in a particular way also causes ACKs to be 
> distributed similarly across the corresponding reverse paths.
>
> RTT measurements:
>
> The return path for ACK frames will impact RTT measurements. That is 
> fine. It is more important that information reach the sender as soon 
> as possible than that it should not affect RTT measurements; we can 
> fix the sender to measure and compensate as necessary. The estimated 
> RTT statistics reflect the distribution of samples, and if both paths 
> are being used, then the SmoothedRTT will reflect the expected value 
> based on the traffic distribution across paths.
>
> That said, it might be useful to track some new stats, especially 
> about how much later is a "late ack" -- an ACK frame that contains no 
> useful information -- is received. I'd have to think a bit more about 
> this, but I think we can devise a stat here. This gives us useful 
> information on the longest return path, which we can then explicitly 
> use as part of the PTO computations, to compensate for the fact that 
> the RTT is based on the shortest return path. (I would _not_ use this 
> stat in the time-based loss detection timer,  but PTO ought to be fine.)
>
> - jana
>
>     1) That the receive maintains an acknowledgement list based on the
>     CID through which the packets are received.
>
>     2) That the senders guarantee that the same sequence number will
>     not be used more than once with a specific CID.
>
>     The main implementation cost is for receivers. They have to
>     allocate and maintain a "SACK Dashboard" in the context of each
>     CID that they issue.
>
>     Senders have lots of control. For example, the "only once"
>     condition is also met if a simple sender uses a single number
>     space, as long as it does not send more than 2^32 packets. That
>     makes the design reasonably easy to implement for constrained
>     implementations.
>
>     Zero length CID are still possible, but that means the receiver
>     supports only one PN space per sender. Multipath is not
>     impossible, but you end up managing a single RTT and a single
>     recovery structure. Not very good, but similar to what happens if
>     multipath is implemented at the IP level.
>
>     Suppose that a client was using both Wi-Fi and LTE, and moves out
>     of Wi-Fi range. The server will find out eventually that the
>     packets sent to the Wi-Fi path are never acknowledged, but that
>     may take some time. It would be better if the client could send a
>     message saying something like "Abandon this path". That's not the
>     same semantic as "retire this CID". We need a new frame for that.
>
>     "Abandon this path" is an extreme case. There are half-way steps,
>     like manage the relative priority of a path.
>

---

From: <[email-masked]>
Date: Fri, 27 Nov 2020 08:19:34 +0000

Hi Christian,

OK, good hint. From my understanding that will not change the general issue of out-of-order reception though.
For clarification, does this means, that a QUIC connection with DATAGRAM frames will not carry any sequence space or only one, the connection sequencing? And that using “unreliable” stream frame will provide two sequence spaces, a connection and a stream space?

Have not found anything about “unreliable” streams in the transport draft, is this exactly the same as with DATAGRAM, no HoL at all?

From: Christian Huitema <[email-masked]>
Sent: Donnerstag, 26. November 2020 18:54
To: Amend, Markus <[email-masked]>; [email-masked]
Cc: [email-masked]; von Hugo, Dirk <[email-masked]>; [email-masked]
Subject: Re: Packet number spaces in multipath (was Re: What to do about multipath in QUIC)

If you want to send unreliable data with sequencing information, it might be simpler to use STREAM frame in "unreliable" mode than to use Datagram frames.
On 11/26/2020 3:34 AM, [email-masked]<mailto:[email-masked]> wrote:
Dear all,

Br

Markus

From: QUIC <[email-masked]><mailto:[email-masked]> On Behalf Of Jana Iyengar
Sent: Mittwoch, 25. November 2020 04:35
To: Christian Huitema <[email-masked]><mailto:[email-masked]>
Cc: IETF QUIC WG <[email-masked]><mailto:[email-masked]>; Kazuho Oku <[email-masked]><mailto:[email-masked]>
Subject: Packet number spaces in multipath (was Re: What to do about multipath in QUIC)

Christian, Kazuho,

ACK Frames:
------------------

Return Path:
-----------------

RTT measurements:

- jana

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 27 Nov 2020 00:39:46 -0800

Hi Markus,

QUIC does not have connection sequencing, unlike TCP, that is one of the
key points of QUIC. Only streams carry sequencing.

QUIC Datagram frames do not carry sequencing or even any kind of flow
association. That was discussed, but it was found that the application
could do that at least as well - with the subtle exception of multiple
applications cooperating on the same QUIC connection.

Unreliable streams is something in between where there is stream
association and sequence information, but no delivery guarantees, and
probably no ordering either but the API could choose to offer that at the
end point. Unreliable streams are so far only a concept that has not been
fully explored but are expected to be important for video streaming and
similar use cases.

So you are right that datagram frames can be harder to reason about on
multiple paths, especially if an application makes assumptions based on the
received order. For the QUIC engine, the problem of loss detection is the
same for all QUIC packets and the problem of ordering within a stream gets
more costly due to more reordering, but otherwise remains the same.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 27 November 2020 at 09.20.08, [email-masked] (
[email-masked]) wrote:

Hi Christian,

OK, good hint. From my understanding that will not change the general issue
of out-of-order reception though.

For clarification, does this means, that a QUIC connection with DATAGRAM
frames will not carry any sequence space or only one, the connection
sequencing? And that using “unreliable” stream frame will provide two
sequence spaces, a connection and a stream space?

Have not found anything about “unreliable” streams in the transport draft,
is this exactly the same as with DATAGRAM, no HoL at all?

*From:* Christian Huitema <[email-masked]>
*Sent:* Donnerstag, 26. November 2020 18:54
*To:* Amend, Markus <[email-masked]>; [email-masked]
*Cc:* [email-masked]; von Hugo, Dirk <[email-masked]>;
[email-masked]
*Subject:* Re: Packet number spaces in multipath (was Re: What to do about
multipath in QUIC)

If you want to send unreliable data with sequencing information, it might
be simpler to use STREAM frame in "unreliable" mode than to use Datagram
frames.

Dear all,

sry for hijacking this conversation. I’m not very familiar with the
different multipath designs for QUIC, however I want to draw attention to
multipath re-ordering which probably becomes important when multipath is
combined with DATAGRAM.

As long as multipath QUIC is operated with strict reliability (similar to
TCP), re-ordering on receiver side is a simple process known from MPTCP.
Introducing unreliable DATAGRAM transmission makes it more challenging on
receiver side to maintain the packet order, because it is not easy to
differentiate between delayed and lost packets. To avoid HoL, a multipath
re-ordering process may benefit from having connection and path sequencing.
In https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01 we
intend to describe this in section 5.6, how fast packet loss detection can
be applied using these different packet sequence spaces. Still the
description is meaningless und will be updated until next IETF, however we
have successfully implemented this approach in a MP-DCCP prototype, which
faces similar challenges in terms of re-ordering. That means, fast packet
loss detection is very beneficial for the receiver re-ordering process to
not lose time until an outstanding packet is assumed lost.

Br

Markus

*From:* QUIC <[email-masked]> <[email-masked]> *On
Behalf Of *Jana
Iyengar
*Sent:* Mittwoch, 25. November 2020 04:35
*To:* Christian Huitema <[email-masked]> <[email-masked]>
*Cc:* IETF QUIC WG <[email-masked]> <[email-masked]>; Kazuho Oku
<[email-masked]> <[email-masked]>
*Subject:* Packet number spaces in multipath (was Re: What to do about
multipath in QUIC)

Christian, Kazuho,

ACK Frames:

------------------

Return Path:

-----------------

RTT measurements:

- jana

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 27 Nov 2020 00:58:59 -0800

I should perhaps clarify that QUIC does have packet numbers which may or
may not be path specific in a multipath scenario. However, QUIC does not
associate semantics with the packet numbering scheme with the exception
that unacked sufficiently old packet numbers can be considered lost because
it is not possible to track history indefinitely, and also because gaps can
be introduced to detect false acknowledgments (optimistic ack attack).
Specifically a multi-core QUIC engine and an endpoint with multiple
hardware offloads, can send packets out of sequence as an optimisation, and
later, the network can of course do the same.

Mikkel

On 27 November 2020 at 09.39.46, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Hi Markus,

Kind Regards,
Mikkel Fahnøe Jørgensen

Hi Christian,

Dear all,

Br

Markus

Christian, Kazuho,

ACK Frames:

------------------

Return Path:

-----------------

RTT measurements:

- jana