Subject: Multipath QUIC prototypes

From: Michael Eriksson <[email-masked]>
Date: Thu, 9 Sep 2021 06:59:35 +0000

Hi,

At Ericsson Research we are doing some prototyping of multipath QUIC (of the fully parallel kind). So far, we have very rudimentary support in our prototype but are about to take the next step.

Our initial implementation uses a single packet number space, which turned out to be a pretty clean addition to the path migration mechanism. Some care has to be taken with the ACK handling on both sides, but so far it has been straightforward. The key is that the sender has to keep track of over which path each packet was sent, but that is trivial to do. Some smartness with the ACK ranges is also needed, but you already need to prune them for the single path case.

This is what I think I know about existing "active" multipath implementations, please correct me if I'm wrong:

- picoquiq has experimental but undocumented support (open source)

- Alibaba are running A/B tests with real users and have published a paper about it. The implementation is planned to be open sourced (this seems to be delayed).

- Quentin De Coninck et al at UCLovain have a prototype and paper on multiflow QUIC, using unidirectional "uniflows" (open source)

Are there any other existing or planned multipath prototype activities in the QUIC community? I would be particularly interested in:

- Open source implementations (preferably in Rust :-))

- Public servers for interop testing

- Designs with a single packet number space and the experiences collected. Separate number spaces, as in draft-liu-multipath-quic, will of course work; however, it would be interesting to also understand the cleaner single PN space alternative.

Best regards,
Michael Eriksson

---

From: Christian Huitema <[email-masked]>
Date: Thu, 9 Sep 2021 05:46:39 -0700

> - picoquiq has experimental but undocumented support (open source)

Picoquic has in fact support for two variants of multipath, draft-liu 
and the single number option draft-huitema-quic-mpath-option-00.

Multipath has not yet been deployed in the picoquic demo program -- all 
tests are on simulated environments. I could easily add a listing of all 
IP addresses available at the client on start, given then a list of 
addresses to choose from, and more or less automatically trying them 
all. I should add a monitoring of this set of addresses using system 
APIs, but those APIs are of course system dependent so that's a bit of 
work. I would be motivated to do that if someone is using the feature...

> - Designs with a single packet number space and the experiences collected. Separate number spaces, as in draft-liu-multipath-quic, will of course work; however, it would be interesting to also unde…

For a discussion, see: 
https://huitema.wordpress.com/2021/02/14/how-many-packet-number-spaces-for-quic-multipath/. 
Both options work. The single number space option is simpler, the only 
problem being the number of ranges in the acknowledgements.

-- Christian Huitema

---

From: Michael Eriksson <[email-masked]>
Date: Fri, 10 Sep 2021 13:39:32 +0000

> work. I would be motivated to do that if someone is using the feature...

It seems simpler to let the application find and select the
interfaces. In a prototype phase, it should be enough to give a list
of local IP address as parameters to the test client. (Servers
normally have a single (public) interface, otherwise that applies to
them too.)

On Linux, for instance, you also need to set up source routing of the
IP packets, otherwise they go out on the default route independently
of which source address you have bound your UDP socket to. To set up
the source routing, you need to be root (or have similar privileges);
something you maybe don't want your experimental code that's
connecting over the Internet to do...

 > problem being the number of ranges in the acknowledgements.

Yes, I read your blog posts at the time, and they convinced me that a
single number space is a cleaner design. I think the number ranges in
the ACKs can be handled in a good way, maybe with a little help from
the sender. We'll see...

/Michael

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 10 Sep 2021 12:16:34 -0400

I am doing non-QUIC experiments in the same space.

The biggest and most important innovation in QUIC is that it eliminates the
dependency of transport area innovation on kernel updates. So it is far
from clear that QUIC is going to be THE replacement for TCP, it may just be
A replacement.

The other reason for my approach is that right now there is really little
to be gained for my project from layering over QUIC since I would have to
implement an already large spec myself. Publication of the QUIC RFCs is not
the same as consensus on the feature sets/APIs exposed by popular libraries
either.

And then there is my plan to encrypt every single bit of the UDP payloads
and to provide native onion routing support.

So I am very interested in this work and will be following but I am taking
a different path for now and it might well turn out to be the case that
there is room for a transport optimized for the needs of Web browsing and a
separate transport optimized for the needs of Web Services.

On Thu, Sep 9, 2021 at 2:59 AM Michael Eriksson <michael.eriksson=
[email-masked]> wrote:

> Hi,
>
> At Ericsson Research we are doing some prototyping of multipath QUIC (of
> the fully parallel kind). So far, we have very rudimentary support in our
> prototype but are about to take the next step.
>
> Our initial implementation uses a single packet number space, which turned
> out to be a pretty clean addition to the path migration mechanism. Some
> care has to be taken with the ACK handling on both sides, but so far it has
> been straightforward. The key is that the sender has to keep track of over
> which path each packet was sent, but that is trivial to do. Some smartness
> with the ACK ranges is also needed, but you already need to prune them for
> the single path case.
>
>
> This is what I think I know about existing "active" multipath
> implementations, please correct me if I'm wrong:
>
> - Alibaba are running A/B tests with real users and have published a paper
> about it. The implementation is planned to be open sourced (this seems to
> be delayed).
>
> - Quentin De Coninck et al at UCLovain have a prototype and paper on
> multiflow QUIC, using unidirectional "uniflows" (open source)
>
>
> Are there any other existing or planned multipath prototype activities in
> the QUIC community? I would be particularly interested in:
>
> - Designs with a single packet number space and the experiences collected.
> Separate number spaces, as in draft-liu-multipath-quic, will of course
> work; however, it would be interesting to also understand the cleaner
> single PN space alternative.
>
>
> Best regards,
> Michael Eriksson
>

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 13 Sep 2021 11:37:34 -0500

Good luck Philip.
Maybe you like to reinvent the wheel?

Behcet

On Fri, Sep 10, 2021 at 11:18 AM Phillip Hallam-Baker <[email-masked]>
wrote:

> The biggest and most important innovation in QUIC is that it eliminates
> the dependency of transport area innovation on kernel updates. So it is far
>> Our initial implementation uses a single packet number space, which
>> turned out to be a pretty clean addition to the path migration mechanism.
>> Some care has to be taken with the ACK handling on both sides, but so far
>> it has been straightforward. The key is that the sender has to keep track
>> of over which path each packet was sent, but that is trivial to do. Some
>> smartness with the ACK ranges is also needed, but you already need to prune
>> them for the single path case.
>>
>>
>> - Alibaba are running A/B tests with real users and have published a
>> paper about it. The implementation is planned to be open sourced (this
>> seems to be delayed).
>>
>> - Designs with a single packet number space and the experiences
>> collected. Separate number spaces, as in draft-liu-multipath-quic, will of
>> course work; however, it would be interesting to also understand the
>> cleaner single PN space alternative.
>>
>>
>> Best regards,
>> Michael Eriksson
>>
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 13 Sep 2021 13:08:48 -0400

> Maybe you like to reinvent the wheel?

That is exactly the objection people made to HTTP in the first place.

That is the objection to QUIC - why not stay with TCP?

QUIC is highly optimized to one use case. It is large, it is very complex.
At this point, I have not seen anyone show how to layer Web Services over
HTTP over QUIC. I do not expect *an* answer to that question, I expect
dozens of them only some of which are interoperable.

The assertion infrastructure adopted in SAML was originally designed as
part of TAXI which was a design commission I was given to re-invent PKIX in
XML.

I am not alone in looking at alternatives to QUIC going forward. If you
want to understand the design choices in QUIC going forward the best way is
to design your own.