Subject: Re: [EToSat] Interop runner with satellite links

From: Sebastian Endres <[email-masked]>
Date: Fri, 18 Feb 2022 13:02:34 +0100

Dear all,

we've published a pre-print of our paper in which we present the QUIC-Interop-runner extended to include satellite scenarios and our measurement results using numerous publicly available QUIC implementations:

https://arxiv.org/abs/2202.08228

Best regards,

Sebastian

On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
> Dear all,
> 
> for my master's thesis we ran measurements of all publicly available QUIC implementations over an emulated satellite link. The results are available online: https://interop.sedrubal.de/
> 
> A click on the results also shows time-offset plots, but are not available for every combination.
> 
> In general, the performance of QUIC over high latency (e.g., geostationary satellites) is rather poor, especially if there is packet loss.
> 
> Would it make sense to add such tests with challenging link characteristics to the official QUIC interop runner?
> 
> Best regards,
> 
> Sebastian
> 
> 
> _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat
>

---

From: "Su, Chi-Jiun" <[email-masked]>
Date: Mon, 21 Feb 2022 18:16:36 +0000

Hi Sebastian,

Thanks for the great work.
Some comments/questions.

  *   Sec. IV,D, 5: Interesting to know "picoquic" does speculative retransmission. As you argue, this may not always help. Did you confirm with the author?
  *    The results show loss-based CC does not perform well compared to BBR
  *   Production software performs well more like picoquic or not ?
     *   how big difference is between production vs these implementations in the test?
  *   Any Time-Offset graphs for EUTELSAT case ?
  *   Research overview page: additional column on emulated or real Sat link will be helpful

Good useful work.
thanks.
cj
________________________________
From: QUIC <[email-masked]> on behalf of Sebastian Endres <[email-masked]>
Sent: Friday, February 18, 2022 7:02 AM
To: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [EToSat] Interop runner with satellite links

**EXTERNAL EMAIL**

Dear all,

https://urldefense.com/v3/__https://arxiv.org/abs/2202.08228__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg$

Best regards,

Sebastian

On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
> Dear all,
>
> for my master's thesis we ran measurements of all publicly available QUIC implementations over an emulated satellite link. The results are available online: https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ$
>

---

From: Joerg Deutschmann <[email-masked]>
Date: Mon, 21 Feb 2022 20:37:56 +0100

Dear CJ,

stepping in for Sebastian, who has earned a break after finishing his 
thesis.
https://interop.cs7.tf.fau.de/thesis.pdf

> Sec. IV,D, 5: Interesting to know "picoquic" does speculative retransmission. As you argue, this may not always help. Did you confirm with the author
No, we have not discussed results with any author, simply because the 
number of implementations is too high to understand each implementation 
in detail. I added Christian Huitema in Cc, maybe he wants to comment on 
this one.

> The results show loss-based CC does not perform well compared to BBR
Yes, but as mentioned in the paper, we are not sure about the CC of each 
and every implementation.

> Production software performs well more like picoquic or not? How big difference is between production vs these implementations in the test?
Good question, we don't know, should be noted as future work. Sebastian 
only has a very short section "Comparison with Results of Other 
Research" in his thesis. We assume that Docker does not have a big 
influence on the performance, but did not verify that. Also chrome not 
being part of the tests is unfortunate.

> Any Time-Offset graphs for EUTELSAT case?
Yes, on the https://interop.cs7.tf.fau.de website select "Run: 
real_2021-11-15" and then "Plots for eutelsat (EUT)". We plan further 
runs in the future.

> Research overview page: additional column on emulated or real Sat link will be helpful
Actually that column already exists, 5th column from the right.
Motivation for putting the overview on github was to allow pull 
requests, but visualization and markdown editing for such a large table 
is not optimal with github...

Thanks,
Joerg

---

From: Christian Huitema <[email-masked]>
Date: Fri, 25 Feb 2022 18:39:44 -0800

>    *   Sec. IV,D, 5: Interesting to know "picoquic" does speculative retransmission. As you argue, this may not always help. Did you confirm with the author?

This is indeed supported in picoquic. There are APIs that allows the 
implementation to turn the feature on or off: 
`picoquic_set_preemptive_repeat_policy(quic_context, do_repeat)` is used 
to set the policy per Quic context, i.e., for all new connections; 
`picoquic_set_preemptive_repeat_per_cnx(connection_context, do_repeat)` 
is used to set the policy for a specific connection.

The speculative retransmission happens at the lowest priority, i.e., 
only happens if there is nothing else to send. It is subject to 
congestion control and rate limiting. The "nothing to send" rule means 
that it will only kick after all data of a stream and the FIN mark have 
been sent. The selection of data for speculative retransmission is based 
on stream level acknowledgements. The code deduces the acknowledged 
parts of a data stream from the packet acknowledgements. It will look at 
data that has been sent, but not yet acknowledged.

As you say, this does not always help. If the packet loss rate is low, 
most of the preemptively repeated packets will be useless. On the other 
hand, when sending a file over a lossy link, the application may wait a 
long time for retransmission of packets lost in the last RTT. If loss 
rate and data rates are high enough, some of these packets will have to 
be repeated twice, or maybe three times. So we have a tradeoff: waste 
bandwidth, or waste time. The API allows the application to consider 
that tradeoff and decide whether it is useful or not.

I would very much like to replace the current implementation of 
preemptive repeat by some version of FEC. FEC in general is a poor fit 
for transmission of big files, because it is always less bandwidth 
efficient than just repeating the packets that are lost. But if the 
application knows that the file transmission is almost complete, because 
there is just one CWIN worth of data left, it could turn own FEC for the 
the duration of the last RTT. It will "waste" a modest number of FEC 
packets, while drastically reducing the impact of packet losses on the 
duration of the file transfer. We could even imagine only sending the 
FEC packets after the FIN mark has been sent, making sure that FEC does 
not increase the duration of transfer if there were no errors.

-- Christian Huitema

---

From: Nicolas Kuhn <[email-masked]>
Date: Sat, 26 Feb 2022 07:24:22 +0100

On Sat, Feb 26, 2022 at 3:40 AM Christian Huitema <[email-masked]>
wrote:

> >    *   Sec. IV,D, 5: Interesting to know "picoquic" does speculative
> retransmission. As you argue, this may not always help. Did you confirm
> with the author?
>
> On the interaction between congestion control and coding, you may be
interested in looking at :
https://datatracker.ietf.org/doc/html/draft-irtf-nwcrg-coding-and-congestion-12

There have been other interesting contributions on the integration of FEC
in QUIC :
- https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04
- https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/
-
https://inl.info.ucl.ac.be/publications/quic-fec-bringing-benefits-forward-erasure-correction-quic.html

I think the proposed approach (sending FEC packets when the file
transmission is over) had been experimented and will investigate whether
the results can be shared.

> > we've published a pre-print of our paper in which we present the
> QUIC-Interop-runner extended to include satellite scenarios and our
> measurement results using numerous publicly available QUIC implementations:
> >
> >
> >> for my master's thesis we ran measurements of all publicly available
> QUIC implementations over an emulated satellite link. The results are
> available online:
> https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
> >>
> >> In general, the performance of QUIC over high latency (e.g.,
> geostationary satellites) is rather poor, especially if there is packet
> loss.
> >>
> >> Would it make sense to add such tests with challenging link
> characteristics to the official QUIC interop runner?
> >>
> >> Best regards,
> >>
> >> Sebastian
> >>
> >>

---

From: Christian Huitema <[email-masked]>
Date: Sat, 26 Feb 2022 21:58:32 -0800

On 2/25/2022 10:24 PM, Nicolas Kuhn wrote:
> -https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04
> -https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/
> -

I will look, thanks.

In any case, I think that the adverse effects found by Sebastian are now 
fixed. There was an issue when the client did not increase the flow 
control window quickly enough. If the value of MAX_DATA or 
MAX_STREAM_DATA is too small, the sender becomes blocked. In the old 
builds, when the sender was blocked by flow control, the code decided 
that it could just as well repeat packets preemptively rather than doing 
nothing. This is somewhat debatable. It generally does not speed up 
transfer, because flow control does not affect loss recovery. The new 
code only does preemptive repeat at the end of file transfers: fewer 
transmission, probably lower energy consumption, and slightly faster 
completion of the file transfer compared to the old code.

I have uploaded a docket image with the new code in the interop runner. 
I don't know whether I need to do something special for the satellite 
interop runner.

-- Christian Huitema

---

From: "Su, Chi-Jiun" <[email-masked]>
Date: Mon, 28 Feb 2022 18:43:36 +0000

Hi Christian,

Thank you very much for coming up with various innovative approaches to improve QUIC over satellite links.
As the result shows, picoquic seems to perform very well over the GEO satellites.
We should continue this effort.

  *   It is a great challenge to improve user's application performance due to lack of corporation between end points and network
     *   picoquic does preemptive retransmission to speed up the data transfer
     *   Link layer may employ ARQ or packet level FEC in local links
     *   Neither end points nor local link is aware of optimization done by the other party.
     *   As a result, the performance may end up worse than without the optimization.
     *   Especially for upload cases, where return link in wireless/sat is more limited and resource allocation is more involved.
  *   Repetition code is simplest error correcting code with least complexity.
     *   Other FEC will offer better bandwidth efficiency.
     *   Your idea of using some form of FEC will be useful.

thanks.
cj

  *
     *

________________________________
From: EToSat <[email-masked]> on behalf of Christian Huitema <[email-masked]>
Sent: Friday, February 25, 2022 9:39 PM
To: Su, Chi-Jiun <Chi-Jiun.Su=[email-masked]>; Sebastian Endres <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [EToSat] Interop runner with satellite links

**EXTERNAL EMAIL**

-- Christian Huitema

> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$

_______________________________________________
EToSat mailing list
[email-masked]
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$

---

From: "Su, Chi-Jiun" <[email-masked]>
Date: Mon, 28 Feb 2022 18:49:06 +0000

This problem in the test may be gone with the new fix.
"Although only a file of 10 MiB is transferred, this leads to approximately 60 to 80 MiB of data being sent."

thanks.
cj

________________________________
From: EToSat <[email-masked]> on behalf of Christian Huitema <[email-masked]>
Sent: Sunday, February 27, 2022 12:58 AM
To: Nicolas Kuhn <[email-masked]>
Cc: Su, Chi-Jiun <Chi-Jiun.Su=[email-masked]>; [email-masked] <[email-masked]>; Sebastian Endres <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: [EToSat] Interop runner with satellite links

**EXTERNAL EMAIL**

interested in looking at :
https://datatracker.ietf.org/doc/html/draft-irtf-nwcrg-coding-and-congestion-12<https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-irtf-nwcrg-coding-and-congestion-12__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudZYRl-1xA$>

There have been other interesting contributions on the integration of FEC
in QUIC :
- https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04<https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudY_7bqPLg$>
- https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/<https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudYazujiig$>
-
https://inl.info.ucl.ac.be/publications/quic-fec-bringing-benefits-forward-erasure-correction-quic.html<https://urldefense.com/v3/__https://inl.info.ucl.ac.be/publications/quic-fec-bringing-benefits-forward-erasure-correction-quic.html__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudZKbpdRNw$>

I will look, thanks.

In any case, I think that the adverse effects found by Sebastian are now fixed. There was an issue when the client did not increase the flow control window quickly enough. If the value of MAX_DATA or MAX_STREAM_DATA is too small, the sender becomes blocked. In the old builds, when the sender was blocked by flow control, the code decided that it could just as well repeat packets preemptively rather than doing nothing. This is somewhat debatable. It generally does not speed up transfer, because flow control does not affect loss recovery. The new code only does preemptive repeat at the end of file transfers: fewer transmission, probably lower energy consumption, and slightly faster completion of the file transfer compared to the old code.

I have uploaded a docket image with the new code in the interop runner. I don't know whether I need to do something special for the satellite interop runner.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Mon, 28 Feb 2022 11:54:49 -0800

Thanks for the kind words, CJ.

A lot of the satellite innovation implemented in Picoquic results from 
the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan Emile 
and others in the ETOSAT community. In my mind, the most interesting 
contribution of that community was to define series of scenarios and 
expected results 
(https://datatracker.ietf.org/doc/draft-kuhn-quic-4-sat/). These 
scenarios act as a kind of "benchmark" for implementers. If implementers 
care about the scenarios, they will program the corresponding test cases 
in their test suites, and if needed ship the corresponding fixes. In 
order to work in these scenarios, Picoquic did the following:

1) Allow flow control windows to grow to values larger than the BDP. I 
believe that many bad results seen in Sebastian's tests are due to 
implementations capping the maximum flow control window to a low value.

2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK 
results in the absence of losses. BBR works well by default. Cubic works 
OK if using a filter to ignore isolated losses.

3) Perform ACK coalescing and avoid congesting a lower capacity return 
link. Picoquic implements the ACK frequency draft 
(https://datatracker.ietf.org/doc/draft-ietf-quic-ack-frequency/) and 
tries to not send too many acks.

4) Remember the bandwidth and delay observed in previous connections. 
Sebastian's benchmark did not test for that, but local tests show that 
implementing the 0RTT BDP draft 
(https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/) results in 
much better performance when resuming sessions over GEO satellite links.

These four points are all pretty much obvious, and I would expect to see 
them over time in most implementations. The only controversial one is 
the "flow control" issue. Some implementers may feel that opening flow 
control too much carries risks of memory exhaustion. On the other hand, 
some implementations are doing adaptive flow control already, i.e., 
"opening flow control enough but not too much". So there is hope.

On top of the four fairly standard points above, Picoquic implements two 
simple features:

5) Fast bandwidth estimate during slow start, in order to accelerate the 
slow start phase on high BDP paths 
(https://huitema.wordpress.com/2020/04/21/faster-slow-start-for-satellite-links/). 
Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows initial 
CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250 Mbps GEO 
path, or 8 for a 20 Mbps GEO path.

6) Preemptive repeat of unacknowledged data at the end of a file 
transfer, in order to avoid waiting 1 or 2 extra RTT for complete 
reception of a file in presence of packet losses.

I don't know if there is any interest in standardizing these two.

-- Christian Huitema