Subject: [EToSat] Performance issue with QUIC on a satellite link with losses

From: PETROU Matthieu <[email-masked]>
Date: Wed, 18 Aug 2021 13:06:39 +0000

Hello everyone,

I’m currently running some tests to compare the download time of QUIC (picoquic) and TCP (iperf3) on a satellite connection, and I saw some wide differences in the results that I wanted to share with you.

I use OpenSAND to emulate the satellite link. The bandwidth between the server and the client is 12 Mb/s on the forward link, and 3 Mb/s on the return link. The buffer size is fixed on the gateway to a bit more than BDP (1.2 times the BDP). I emulate packet losses with time traces, generated with the modele Gilbert and Elliot, with p = 0.01 and q = 0.167. Overall, there is 2.56% of loss time with an average burst length of 16.70ms and a maximum burst length of 43ms.

For the tests, I download either 20 MB of data with iperf3 and picoquic. Either there is only one flow which downloads 20 MB, or five concurrent flows, with each one of them downloading 20 MB. To be able to compare the concurrent flows, I launch 5 different iperf3 and picoquic threads for each flow (on different ports). Both QUIC and TCP are using CUBIC and have hystart disabled. I’m using iperf 3.7+ (cJSON 1.5.2) and picoquic v0.34f.

Here is the download time medians and the standard deviation of 30 tests in those conditions :

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

91.42

47.48

1 QUIC flow

278.39

172.45

5 TCP flows

163.35

50.16

5 QUIC flows

400.74

127.81

If I remove all link losses, here is the download time medians and the standard deviation of 30 tests :

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

17.84

0.34

1 QUIC flow

18.35

0.20

5 TCP flows

65.29

8.73

5 QUIC flows

74.46

7.61

With those results, it seems that QUIC (picoquic) is way more impacted by losses on a satellite link. However, I don’t really understand why, as both are using CUBIC algorithm and hystart is disabled.

Is there something I’m missing here?

Thanks in advance for your help,
Matthieu

---

From: "Border, John" <[email-masked]>
Date: Wed, 18 Aug 2021 13:36:53 +0000

Normally I would say the TCP traffic is being PEPed.  Is it?  Since this is an emulated satellite link the answer seems like it should be no.

We ran some tests two years or so ago related to the rate of packet loss.  The results were presented at a couple of PANRG meetings (PANRG IETF 105<https://datatracker.ietf.org/meeting/105/materials/agenda-105-panrg-06.html> and PANRG Interim<https://datatracker.ietf.org/meeting/interim-2020-panrg-01/materials/agenda-interim-2020-panrg-01-panrg-01-04.htm>).  The long round trip definitely significant impacts recovery time and can stall progress.  But, we were testing with Google QUIC then and we were expecting IETF QUIC to be better.

John

From: EToSat <[email-masked]> On Behalf Of PETROU Matthieu
Sent: Wednesday, August 18, 2021 9:07 AM
To: [email-masked]
Subject: [EToSat] Performance issue with QUIC on a satellite link with losses

**EXTERNAL EMAIL**

Hello everyone,

I'm currently running some tests to compare the download time of QUIC (picoquic) and TCP (iperf3) on a satellite connection, and I saw some wide differences in the results that I wanted to share with you.

For the tests, I download either 20 MB of data with iperf3 and picoquic. Either there is only one flow which downloads 20 MB, or five concurrent flows, with each one of them downloading 20 MB. To be able to compare the concurrent flows, I launch 5 different iperf3 and picoquic threads for each flow (on different ports). Both QUIC and TCP are using CUBIC and have hystart disabled. I'm using iperf 3.7+ (cJSON 1.5.2) and picoquic v0.34f.

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

91.42

47.48

1 QUIC flow

278.39

172.45

5 TCP flows

163.35

50.16

5 QUIC flows

400.74

127.81

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

17.84

0.34

1 QUIC flow

18.35

0.20

5 TCP flows

65.29

8.73

5 QUIC flows

74.46

7.61

With those results, it seems that QUIC (picoquic) is way more impacted by losses on a satellite link. However, I don't really understand why, as both are using CUBIC algorithm and hystart is disabled.

Is there something I'm missing here?

---

From: PETROU Matthieu <[email-masked]>
Date: Wed, 18 Aug 2021 14:34:35 +0000

Thanks John for your response.

As you have already guessed, TCP traffic is not being PEPed here.

I agree with you, the long RTT impacts the recovery time. However, the impact should be the same on TCP and QUIC, as both have the same congestion control algorithm (CUBIC without hystart in my case).

Matthieu
________________________________
De : Border, John <[email-masked]>
Envoyé : mercredi 18 août 2021 15:36
À : PETROU Matthieu <[email-masked]>
Cc : [email-masked] <[email-masked]>
Objet : RE: Performance issue with QUIC on a satellite link with losses

Vous ne recevez pas souvent de courriers de la part de [email-masked]. Découvrez pourquoi cela est important<http://aka.ms/LearnAboutSenderIdentification>

We ran some tests two years or so ago related to the rate of packet loss.  The results were presented at a couple of PANRG meetings (PANRG IETF 105<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F105%2Fmaterials%2Fagenda-105-panrg-06.html&data=04%7C01%7CMatthieu.PETROU%40viveris.fr%7C0a6654f4e37d4b73a28808d9624d417a%7C34bab81c945c43f1ad13592b97e11b40%7C1%7C0%7C637648906227795360%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EuAeFSVMFDtSueMTotdFRhQHBEvU%2BFGM5zoOgsxDNy8%3D&reserved=0> and PANRG Interim<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2Finterim-2020-panrg-01%2Fmaterials%2Fagenda-interim-2020-panrg-01-panrg-01-04.htm&data=04%7C01%7CMatthieu.PETROU%40viveris.fr%7C0a6654f4e37d4b73a28808d9624d417a%7C34bab81c945c43f1ad13592b97e11b40%7C1%7C0%7C637648906227805354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1F%2Fdo1GyFr29rxAl0DKwkLol8wLGGVThKE8b6k223eU%3D&reserved=0>).  The long round trip definitely significant impacts recovery time and can stall progress.  But, we were testing with Google QUIC then and we were expecting IETF QUIC to be better.

John

**EXTERNAL EMAIL**

Hello everyone,

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

91.42

47.48

1 QUIC flow

278.39

172.45

5 TCP flows

163.35

50.16

5 QUIC flows

400.74

127.81

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

17.84

0.34

1 QUIC flow

18.35

0.20

5 TCP flows

65.29

8.73

5 QUIC flows

74.46

7.61

---

From: Christian Huitema <[email-masked]>
Date: Wed, 18 Aug 2021 09:40:45 -0700

For the Picoquic tests, did you capture a qlog?

You said Hystart was disabled. How did you do that on Picoquic? When using Cubic, I found that Hystart was critical for performance. Basic slowstart causes wild overshot and packet losses.

I have not been testing Cubic for a long time, most of the Picoquic tests use BBR. I wonder whether that would make a difference.

-- Christian Huitema 

> On Aug 18, 2021, at 6:36 AM, Border, John <[email-masked]> wrote:
> 
> 
> We ran some tests two years or so ago related to the rate of packet loss.  The results were presented at a couple of PANRG meetings (PANRG IETF 105 and PANRG Interim).  The long round trip definitely significant impacts recovery time and can stall progress.  But, we were testing with Google QUIC then and we were expecting IETF QUIC to be better.
>  
>  
> John
>  
>  
> _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat

---

From: Christian Huitema <[email-masked]>
Date: Wed, 18 Aug 2021 09:40:45 -0700

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Wed, 18 Aug 2021 09:40:45 -0700

-- Christian Huitema

---

From: PETROU Matthieu <[email-masked]>
Date: Thu, 19 Aug 2021 10:38:16 +0000

You will find two qlog files from the server attached to this mail. As there is a wide difference between download times, I put one with a good start (with the slow start reaching the 12 Mbps) and one with a bad start (the slow start quickly stops). I only put one flow case, not five concurrent flows.

To disable hystart, I changed the function "picoquic_hystart_test" (https://github.com/private-octopus/picoquic/blob/45f7412c7970ce8e92ca74fe85ac3d3abfa6372e/picoquic/cc_common.c#L134) so that it always returns zero.

I launched 30 tests in the same condition with CUBIC and hystart enabled, on both TCP and QUIC, here is the results:

Configuration

Download time median (s)

Standard Deviation (s)

1 TCP flow

135.73

43.45

1 QUIC flow

374.17

153.67

5 TCP flows

186.80

37.79

5 QUIC flows

375.26

116.36

The results are worse for TCP CUBIC with Hystart. For QUIC : worse with only one flow, and slightly better with five concurrent flows.

Of course, there are packet losses without Hystart, mainly because the slow start needs to overshoot to get out of the start phase. Hystart would avoid that with the measurement of the RTT which increases with buffer filling. However, the issue is that the flow goes out of the start phase when the RTT is greater by 16 ms than the minimum RTT. With a satellite, the jitter is important and a difference of 16 ms is easily reached. That’s why we decided to disable hystart for our tests with TCP and QUIC. We’re aiming more about the download time than packet losses.

I’ll launch some tests with BBR to see if that improves the results.

---

From: Christian Huitema <[email-masked]>
Date: Thu, 19 Aug 2021 18:21:58 -0700

From looking at the logs, I think I know what is happening. I see on 
the tests that the Window drops immediately after a packet loss. It will 
eventually converge to something like 1/loss-rate packets, which is way 
too slow. I think that the TCP implementation of Cubic in Linux has a 
some kind of filter, and it only reduces the window if it sees several 
packet losses. That's a way to disambiguate between  "congestion" losses 
that should trigger a slow down and "random" losses caused by 
transmission issues that should not. My implementation of Cubic does not 
do that, that's a bug that should be fixed. I opened an issue to track 
that (and eventually fix it): 
https://github.com/private-octopus/picoquic/issues/1221

BBR does not have that issue, it only reacts to losses if these are 
frequent enough. If you repeat the tests using BBR instead of Cubic it 
should work much better. Especially if you do not disable Hystart...

-- Christian Huitema