Subject: Re: [EToSat] Interop runner with satellite links

From: Emmanuel Lochin <[email-masked]>
Date: Tue, 1 Mar 2022 14:58:28 +0100

Hi all,

This is an interesting discussion. Nicolas and I get involved in a study 
on the performance of Picoquic/BBR and TCP/CUBIC within a Sliding Window 
FEC tunnel over SATCOM and clearly, BBR alone beats FEC coded CUBIC 
flows over SATCOM. We wrote a technical paper to present all our results 
here : https://hal.archives-ouvertes.fr/hal-03590651 and to confirm 
point 2) from Christian below and give some insights to previous CJ's email.

Emmanuel & Nicolas

Le 28/02/2022 à 20:54, Christian Huitema a écrit :
> Thanks for the kind words, CJ.
>
> A lot of the satellite innovation implemented in Picoquic results from 
> the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan 
> Emile and others in the ETOSAT community. In my mind, the most 
> interesting contribution of that community was to define series of 
> scenarios and expected results 
> (https://datatracker.ietf.org/doc/draft-kuhn-quic-4-sat/). These 
> scenarios act as a kind of "benchmark" for implementers. If 
> implementers care about the scenarios, they will program the 
> corresponding test cases in their test suites, and if needed ship the 
> corresponding fixes. In order to work in these scenarios, Picoquic did 
> the following:
>
> 1) Allow flow control windows to grow to values larger than the BDP. I 
> believe that many bad results seen in Sebastian's tests are due to 
> implementations capping the maximum flow control window to a low value.
>
> 2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK 
> results in the absence of losses. BBR works well by default. Cubic 
> works OK if using a filter to ignore isolated losses.
>
> 3) Perform ACK coalescing and avoid congesting a lower capacity return 
> link. Picoquic implements the ACK frequency draft 
> (https://datatracker.ietf.org/doc/draft-ietf-quic-ack-frequency/) and 
> tries to not send too many acks.
>
> 4) Remember the bandwidth and delay observed in previous connections. 
> Sebastian's benchmark did not test for that, but local tests show that 
> implementing the 0RTT BDP draft 
> (https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/) results 
> in much better performance when resuming sessions over GEO satellite 
> links.
>
> These four points are all pretty much obvious, and I would expect to 
> see them over time in most implementations. The only controversial one 
> is the "flow control" issue. Some implementers may feel that opening 
> flow control too much carries risks of memory exhaustion. On the other 
> hand, some implementations are doing adaptive flow control already, 
> i.e., "opening flow control enough but not too much". So there is hope.
>
> On top of the four fairly standard points above, Picoquic implements 
> two simple features:
>
> 5) Fast bandwidth estimate during slow start, in order to accelerate 
> the slow start phase on high BDP paths 
> (https://huitema.wordpress.com/2020/04/21/faster-slow-start-for-satellite-links/). 
> Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows 
> initial CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250 
> Mbps GEO path, or 8 for a 20 Mbps GEO path.
>
> 6) Preemptive repeat of unacknowledged data at the end of a file 
> transfer, in order to avoid waiting 1 or 2 extra RTT for complete 
> reception of a file in presence of packet losses.
>
> I don't know if there is any interest in standardizing these two.
>
> -- Christian Huitema
>
>
> On 2/28/2022 10:43 AM, Su, Chi-Jiun wrote:
>> Hi Christian,
>>
>> Thank you very much for coming up with various innovative approaches 
>> to improve QUIC over satellite links.
>> As the result shows, picoquic seems to perform very well over the GEO 
>> satellites.
>> We should continue this effort.
>>
>>    *   It is a great challenge to improve user's application 
>> performance due to lack of corporation between end points and network
>>       *   picoquic does preemptive retransmission to speed up the 
>> data transfer
>>       *   Link layer may employ ARQ or packet level FEC in local links
>>       *   Neither end points nor local link is aware of optimization 
>> done by the other party.
>>       *   As a result, the performance may end up worse than without 
>> the optimization.
>>       *   Especially for upload cases, where return link in 
>> wireless/sat is more limited and resource allocation is more involved.
>>    *   Repetition code is simplest error correcting code with least 
>> complexity.
>>       *   Other FEC will offer better bandwidth efficiency.
>>       *   Your idea of using some form of FEC will be useful.
>>
>> thanks.
>> cj
>>
>>    *
>>       *
>>
>> ________________________________
>> From: EToSat <[email-masked]> on behalf of Christian Huitema 
>> <[email-masked]>
>> Sent: Friday, February 25, 2022 9:39 PM
>> To: Su, Chi-Jiun <Chi-Jiun.Su=[email-masked]>; Sebastian 
>> Endres <[email-masked]>; [email-masked] <[email-masked]>; 
>> [email-masked] <[email-masked]>
>> Cc: [email-masked] <[email-masked]>
>> Subject: Re: [EToSat] Interop runner with satellite links
>>
>> **EXTERNAL EMAIL**
>>
>>
>> On 2/21/2022 10:16 AM, Su, Chi-Jiun wrote:
>>> Hi Sebastian,
>>>
>>> Thanks for the great work.
>>> Some comments/questions.
>>>
>>>     *   Sec. IV,D, 5: Interesting to know "picoquic" does 
>>> speculative retransmission. As you argue, this may not always help. 
>>> Did you confirm with the author?
>> This is indeed supported in picoquic. There are APIs that allows the
>> implementation to turn the feature on or off:
>> `picoquic_set_preemptive_repeat_policy(quic_context, do_repeat)` is used
>> to set the policy per Quic context, i.e., for all new connections;
>> `picoquic_set_preemptive_repeat_per_cnx(connection_context, do_repeat)`
>> is used to set the policy for a specific connection.
>>
>> The speculative retransmission happens at the lowest priority, i.e.,
>> only happens if there is nothing else to send. It is subject to
>> congestion control and rate limiting. The "nothing to send" rule means
>> that it will only kick after all data of a stream and the FIN mark have
>> been sent. The selection of data for speculative retransmission is based
>> on stream level acknowledgements. The code deduces the acknowledged
>> parts of a data stream from the packet acknowledgements. It will look at
>> data that has been sent, but not yet acknowledged.
>>
>> As you say, this does not always help. If the packet loss rate is low,
>> most of the preemptively repeated packets will be useless. On the other
>> hand, when sending a file over a lossy link, the application may wait a
>> long time for retransmission of packets lost in the last RTT. If loss
>> rate and data rates are high enough, some of these packets will have to
>> be repeated twice, or maybe three times. So we have a tradeoff: waste
>> bandwidth, or waste time. The API allows the application to consider
>> that tradeoff and decide whether it is useful or not.
>>
>> I would very much like to replace the current implementation of
>> preemptive repeat by some version of FEC. FEC in general is a poor fit
>> for transmission of big files, because it is always less bandwidth
>> efficient than just repeating the packets that are lost. But if the
>> application knows that the file transmission is almost complete, because
>> there is just one CWIN worth of data left, it could turn own FEC for the
>> the duration of the last RTT. It will "waste" a modest number of FEC
>> packets, while drastically reducing the impact of packet losses on the
>> duration of the file transfer. We could even imagine only sending the
>> FEC packets after the FIN mark has been sent, making sure that FEC does
>> not increase the duration of transfer if there were no errors.
>>
>> -- Christian Huitema
>>
>>
>>>     *    The results show loss-based CC does not perform well 
>>> compared to BBR
>>>     *   Production software performs well more like picoquic or not ?
>>>        *   how big difference is between production vs these 
>>> implementations in the test?
>>>     *   Any Time-Offset graphs for EUTELSAT case ?
>>>     *   Research overview page: additional column on emulated or 
>>> real Sat link will be helpful
>>>
>>> Good useful work.
>>> thanks.
>>> cj
>>> ________________________________
>>> From: QUIC <[email-masked]> on behalf of Sebastian Endres 
>>> <[email-masked]>
>>> Sent: Friday, February 18, 2022 7:02 AM
>>> To: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
>>> Cc: [email-masked] <[email-masked]>
>>> Subject: Re: [EToSat] Interop runner with satellite links
>>>
>>> **EXTERNAL EMAIL**
>>>
>>> Dear all,
>>>
>>> we've published a pre-print of our paper in which we present the 
>>> QUIC-Interop-runner extended to include satellite scenarios and our 
>>> measurement results using numerous publicly available QUIC 
>>> implementations:
>>>
>>> https://urldefense.com/v3/__https://arxiv.org/abs/2202.08228__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg$ 
>>>
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>> On Mittwoch, 29. September 2021 21:38:05 CET Sebastian Endres wrote:
>>>> Dear all,
>>>>
>>>> for my master's thesis we ran measurements of all publicly 
>>>> available QUIC implementations over an emulated satellite link. The 
>>>> results are available online: 
>>>> https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
>>>>
>>>> A click on the results also shows time-offset plots, but are not 
>>>> available for every combination.
>>>>
>>>> In general, the performance of QUIC over high latency (e.g., 
>>>> geostationary satellites) is rather poor, especially if there is 
>>>> packet loss.
>>>>
>>>> Would it make sense to add such tests with challenging link 
>>>> characteristics to the official QUIC interop runner?
>>>>
>>>> Best regards,
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> _______________________________________________
>>>> EToSat mailing list
>>>> [email-masked]
>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ$ 
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> EToSat mailing list
>>> [email-masked]
>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$ 
>>>
>> _______________________________________________
>> EToSat mailing list
>> [email-masked]
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw$ 
>>
>>
>>
>> _______________________________________________
>> EToSat mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/etosat
>
> _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat

-- 
Emmanuel LOCHIN
ENAC - Ecole Nationale de l'Aviation Civile
7, avenue Edouard Belin CS 54005, 31055 Toulouse Cedex 4 France
http://www.enac.fr
https://elochin.github.io/

---

From: "Su, Chi-Jiun" <[email-masked]>
Date: Tue, 1 Mar 2022 14:05:11 +0000

A good summary of progress made by the community. Thank you for all.
Geo Sat still plays a significant role to provide internet to underserved regions.
Hope more implementations adopt Satellite-friendly features as picoquic does.
thanks.
cj

________________________________
From: Christian Huitema <[email-masked]>
Sent: Monday, February 28, 2022 2:54 PM
To: Su, Chi-Jiun <[email-masked]>; Sebastian Endres <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [EToSat] Interop runner with satellite links

**EXTERNAL EMAIL**

A lot of the satellite innovation implemented in Picoquic results from
the efforts of Nicolas Kuhn, Gorry Fairhurst, John Border, Stephan Emile
and others in the ETOSAT community. In my mind, the most interesting
contribution of that community was to define series of scenarios and
expected results
(https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-kuhn-quic-4-sat/__;!!Emaut56SYw!j1a5MIVP-6cxYNb4bFqFoNUgcewYsapy3yFQgGmu6_E2avid3jHb2b3aMYxBbvSbBw$ ). These
scenarios act as a kind of "benchmark" for implementers. If implementers
care about the scenarios, they will program the corresponding test cases
in their test suites, and if needed ship the corresponding fixes. In
order to work in these scenarios, Picoquic did the following:

2) Use Cubic or BBR rather than Reno. Both Cubic and BBR would give OK
results in the absence of losses. BBR works well by default. Cubic works
OK if using a filter to ignore isolated losses.

3) Perform ACK coalescing and avoid congesting a lower capacity return
link. Picoquic implements the ACK frequency draft
(https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-quic-ack-frequency/__;!!Emaut56SYw!j1a5MIVP-6cxYNb4bFqFoNUgcewYsapy3yFQgGmu6_E2avid3jHb2b3aMYzWU8rv1w$ ) and
tries to not send too many acks.

4) Remember the bandwidth and delay observed in previous connections.
Sebastian's benchmark did not test for that, but local tests show that
implementing the 0RTT BDP draft
(https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/__;!!Emaut56SYw!j1a5MIVP-6cxYNb4bFqFoNUgcewYsapy3yFQgGmu6_E2avid3jHb2b3aMYyhVynyKg$ ) results in
much better performance when resuming sessions over GEO satellite links.

These four points are all pretty much obvious, and I would expect to see
them over time in most implementations. The only controversial one is
the "flow control" issue. Some implementers may feel that opening flow
control too much carries risks of memory exhaustion. On the other hand,
some implementations are doing adaptive flow control already, i.e.,
"opening flow control enough but not too much". So there is hope.

On top of the four fairly standard points above, Picoquic implements two
simple features:

5) Fast bandwidth estimate during slow start, in order to accelerate the
slow start phase on high BDP paths
(https://urldefense.com/v3/__https://huitema.wordpress.com/2020/04/21/faster-slow-start-for-satellite-links/__;!!Emaut56SYw!j1a5MIVP-6cxYNb4bFqFoNUgcewYsapy3yFQgGmu6_E2avid3jHb2b3aMYz6_DpORQ$ ).
Classic slow start needs log2(BDP/IW) RTTs. Fast estimate allows initial
CWIN estimation in 4 or 5 RTT, instead of 10 RTT for a 250 Mbps GEO
path, or 8 for a 20 Mbps GEO path.

-- Christian Huitema

On 2/28/2022 10:43 AM, Su, Chi-Jiun wrote:
> Hi Christian,
>
> Thank you very much for coming up with various innovative approaches to improve QUIC over satellite links.
> As the result shows, picoquic seems to perform very well over the GEO satellites.
>    *   It is a great challenge to improve user's application performance due to lack of corporation between end points and network
>       *   picoquic does preemptive retransmission to speed up the data transfer
>       *   Neither end points nor local link is aware of optimization done by the other party.
>       *   As a result, the performance may end up worse than without the optimization.
>       *   Especially for upload cases, where return link in wireless/sat is more limited and resource allocation is more involved.
>    *   Repetition code is simplest error correcting code with least complexity.
> From: EToSat <[email-masked]> on behalf of Christian Huitema <[email-masked]>
> To: Su, Chi-Jiun <Chi-Jiun.Su=[email-masked]>; Sebastian Endres <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
>>     *   Sec. IV,D, 5: Interesting to know "picoquic" does speculative retransmission. As you argue, this may not always help. Did you confirm with the author?
>>     *    The results show loss-based CC does not perform well compared to BBR
>>        *   how big difference is between production vs these implementations in the test?
>>     *   Research overview page: additional column on emulated or real Sat link will be helpful
>>
>> Good useful work.
>> thanks.
>> cj
>> From: QUIC <[email-masked]> on behalf of Sebastian Endres <[email-masked]>
>> we've published a pre-print of our paper in which we present the QUIC-Interop-runner extended to include satellite scenarios and our measurement results using numerous publicly available QUIC implementations:
>>
>>> for my master's thesis we ran measurements of all publicly available QUIC implementations over an emulated satellite link. The results are available online: https://urldefense.com/v3/__https://interop.sedrubal.de/__;!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ$
>>>
>>> A click on the results also shows time-offset plots, but are not available for every combination.
>>>
>>> In general, the performance of QUIC over high latency (e.g., geostationary satellites) is rather poor, especially if there is packet loss.
>>>
>>> Would it make sense to add such tests with challenging link characteristics to the official QUIC interop runner?
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/etosat__;!!Emaut56SYw!j1a5MIVP-6cxYNb4bFqFoNUgcewYsapy3yFQgGmu6_E2avid3jHb2b3aMYyklIe3zw$

---

From: François Michel <[email-masked]>
Date: Tue, 1 Mar 2022 14:39:52 +0000

Hello,

Thank you for the reference @Emmanuel, the study is interesting and we have similar results right here.
This illustrates quite well our point of view concerning the use of FEC within the transport
(in your article, you used it below the transport if I understand well): it seems better to use a
dedicated congestion control than trying to hide the losses with FEC
(it is never good to hide a signal IMO... :-) ). And your article demonstrates it well !

@everyone, and especially @Christian: I think sending FEC at the right moment, depending on the
use-case, can help a lot more than sending a fixed rate of redundancy, and what @Christian is doing
with the speculative retransmissions is IMO exactly sending redundancy at the right
moment for bulk transfer.

We worked quite a bit with FEC and QUIC during the
last months and years. We did it with Pluginized QUIC (PQUIC), but while PQUIC offers nice
features, the architecture of PQUIC made it hard to have the best results due to some bad
CPU with the PQUIC implem.
It think it becomes the moment where we start implementing FEC again in a clean QUIC implem,
and picoquic might be one of them. We did it several times now, so we are definitely willing to
look at it in the near future and it should not take much time. We now have cool use-cases and
diverse network accesses, including satellites, to experiment on. However, I think it will be
faster and cleaner if we look at it conjointly, so let me know if you're interested, we could
start a new FEC branch on picoquic and iterate on that.

It might also me a good moment to do a new pass on the design of quic-fec. After all this time using it,
I'm convinced we can improve it significantly. :-) So feel free to contact me for any of this.

Regards,

François

________________________________
From: EToSat <[email-masked]> on behalf of Emmanuel Lochin <[email-masked]>
Sent: 01 March 2022 14:58
To: [email-masked] <[email-masked]>
Cc: Nicolas Kuhn <[email-masked]>
Subject: Re: [EToSat] Interop runner with satellite links

Hi all,

here : https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhal.archives-ouvertes.fr%2Fhal-03590651&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=u7xI%2Bl4hQCxgMck6vyawzxbKnP30P4rRrLPeV0Nxf1M%3D&amp;reserved=0 and to confirm

Emmanuel & Nicolas

Le 28/02/2022 à 20:54, Christian Huitema a écrit :
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-4-sat%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=JlS7QpDgEM1i2ao6viQ2vTzFVPwqx7RrABGT2nVBDsA%3D&amp;reserved=0). These
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=DQ%2Bx4m4d8DYlPX8jkoqr%2B5KRaZvqS2h7mTekmg4gx4A%3D&amp;reserved=0) and
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-0rtt-bdp%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=lSfamWAkxRlrHO7bWhpeG5jys%2BNNEOK1nhBQyCuLxg8%3D&amp;reserved=0) results
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuitema.wordpress.com%2F2020%2F04%2F21%2Ffaster-slow-start-for-satellite-links%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=WpM735c%2FAmhNzm7KEX84FUeI8zoz%2BEKruNsaT3Svy%2Fg%3D&amp;reserved=0).
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Farxiv.org%2Fabs%2F2202.08228__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkW-JRj-Kg%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=Rq4pbHLGeeXqDSf1q3mfKQ7CZP33YivZ6VeZRknI3B0%3D&amp;reserved=0
>>>
>>>
>>> Best regards,
>>>
>>> Sebastian
>>>
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Finterop.sedrubal.de%2F__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkUlf0EgaQ%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=qUxmio7Yi6NDQ8cW4qOmAn4hPZA7y45Vzb%2BI8x2e53E%3D&amp;reserved=0
>>>>
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!mFvp0jheUG95wMrw7L6ZHr3AUjZZgp63MEXhAZnEYvwEkQITw9uIFauoKkXTZO8rhQ%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=z%2FL6ggN6uCRKUI6d0Na0n%2FAs4WtCQX1tKmxH%2F7gFLVc%3D&amp;reserved=0
>>>>
>>>>
>>>
>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=v5TH%2BJGED7M9jWJYCFuLYXT0Jywo21%2BHxj4t5f0gZPg%3D&amp;reserved=0
>>>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat__%3B!!Emaut56SYw!nXc1E9u7JuP0YnV6VSYRN1raD2YV53cI9SG-w15XNrNM6F4tNALqWp9LY3Xoq4YQvw%24&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=v5TH%2BJGED7M9jWJYCFuLYXT0Jywo21%2BHxj4t5f0gZPg%3D&amp;reserved=0
>>
>>
>>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0

--
Emmanuel LOCHIN
ENAC - Ecole Nationale de l'Aviation Civile
7, avenue Edouard Belin CS 54005, 31055 Toulouse Cedex 4 France
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.enac.fr%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=bY7XjNinIEUbGNxKWmyTQpBKAyNocNsqkSawuMeWkTI%3D&amp;reserved=0
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Felochin.github.io%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=UITTEYc2bMivTD3C%2B%2F64Gi7H6SgGqIqI1VttceO6oBk%3D&amp;reserved=0

_______________________________________________
EToSat mailing list
[email-masked]
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fetosat&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963647155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=%2FQAyUMsNxAM%2BnqS25RJL6vjBymeHQti0NaseC4LHm5c%3D&amp;reserved=0

---

From: Christian Huitema <[email-masked]>
Date: Tue, 1 Mar 2022 07:32:02 -0800

Hello François,

I am certainly willing to help prepare and review a pull request and 
eventually integrate the FEC code in Picoquic.

-- Christian Huitema

---

From: Nicolas Kuhn <[email-masked]>
Date: Wed, 2 Mar 2022 13:11:05 +0100

IMHO the QUIC-FEC drafts are very good starting points !
- https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04
- https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/

A QUIC FEC extension would be very interesting to discuss.
There should be cases other than SATCOM where this would help.

On Tue, Mar 1, 2022 at 4:32 PM Christian Huitema <[email-masked]>
wrote:

> > here :
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhal.archives-ouvertes.fr%2Fhal-03590651&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=u7xI%2Bl4hQCxgMck6vyawzxbKnP30P4rRrLPeV0Nxf1M%3D&amp;reserved=0
> and to confirm
> >> (
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-4-sat%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=JlS7QpDgEM1i2ao6viQ2vTzFVPwqx7RrABGT2nVBDsA%3D&amp;reserved=0).
> These
> >> (
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=DQ%2Bx4m4d8DYlPX8jkoqr%2B5KRaZvqS2h7mTekmg4gx4A%3D&amp;reserved=0)
> and
> >> (
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-kuhn-quic-0rtt-bdp%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=lSfamWAkxRlrHO7bWhpeG5jys%2BNNEOK1nhBQyCuLxg8%3D&amp;reserved=0)
> results
> >> (
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuitema.wordpress.com%2F2020%2F04%2F21%2Ffaster-slow-start-for-satellite-links%2F&amp;data=04%7C01%7Cfrancois.michel%40uclouvain.be%7C653be509ab674220f1cd08d9fb8b9dd7%7C7ab090d4fa2e4ecfbc7c4127b4d582ec%7C0%7C0%7C637817400963490958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0&amp;sdata=WpM735c%2FAmhNzm7KEX84FUeI8zoz%2BEKruNsaT3Svy%2Fg%3D&amp;reserved=0
> ).

---

From: Joerg Deutschmann <[email-masked]>
Date: Fri, 18 Mar 2022 17:09:30 +0100

Yes, that issue is now fixed.
https://interop.cs7.tf.fau.de/?run=logs_paper_2022-03-15
We'll update the paper accordingly, thanks for your feedback.

We'll also reference Robin Marx's "Same Standards, Different Decisions: 
A Study of QUIC and HTTP/3 Implementation Diversity" paper, which is 
obviously relevant in this context, especially regarding the influence 
of flow control.

On 28.02.22 19:49, Su, Chi-Jiun wrote:
> This problem in the test may be gone with the new fix.
> "Although only a file of 10 MiB is transferred, this leads to 
> approximately 60 to 80 MiB of data being sent."
> 
> thanks.
> cj
> 
> 
> ------------------------------------------------------------------------
> *From:* EToSat <[email-masked]> on behalf of Christian Huitema 
> <[email-masked]>
> *Sent:* Sunday, February 27, 2022 12:58 AM
> *To:* Nicolas Kuhn <[email-masked]>
> *Cc:* Su, Chi-Jiun <Chi-Jiun.Su=[email-masked]>; 
> [email-masked] <[email-masked]>; Sebastian Endres <[email-masked]>; 
> [email-masked] <[email-masked]>; [email-masked] 
> <[email-masked]>
> *Subject:* Re: [EToSat] Interop runner with satellite links
> 
> ***EXTERNAL EMAIL***
> 
> 
> 
> 
> On 2/25/2022 10:24 PM, Nicolas Kuhn wrote:
>>> On the interaction between congestion control and coding, you may be
>> interested in looking at :
>> https://datatracker.ietf.org/doc/html/draft-irtf-nwcrg-coding-and-congestion-12  <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-irtf-nwcrg-coding-and-congestion-12__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudZYRl-1xA$>
>>
>> There have been other interesting contributions on the integration of FEC
>> in QUIC :
>> -https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04  <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-swett-nwcrg-coding-for-quic-04__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudY_7bqPLg$>
>> -https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/  <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-roca-nwcrg-rlc-fec-scheme-for-quic/__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudYazujiig$>
>> -
>> https://inl.info.ucl.ac.be/publications/quic-fec-bringing-benefits-forward-erasure-correction-quic.html  <https://urldefense.com/v3/__https://inl.info.ucl.ac.be/publications/quic-fec-bringing-benefits-forward-erasure-correction-quic.html__;!!Emaut56SYw!jNkAqwXdSBCHAMUuYW8PPcglW-20gL3A6g374RkqkAJtdwP1lCyedBGvudZKbpdRNw$>
>>
>> I think the proposed approach (sending FEC packets when the file
>> transmission is over) had been experimented and will investigate whether
>> the results can be shared.
> 
> I will look, thanks.
> 
> In any case, I think that the adverse effects found by Sebastian are now 
> fixed. There was an issue when the client did not increase the flow 
> control window quickly enough. If the value of MAX_DATA or 
> MAX_STREAM_DATA is too small, the sender becomes blocked. In the old 
> builds, when the sender was blocked by flow control, the code decided 
> that it could just as well repeat packets preemptively rather than doing 
> nothing. This is somewhat debatable. It generally does not speed up 
> transfer, because flow control does not affect loss recovery. The new 
> code only does preemptive repeat at the end of file transfers: fewer 
> transmission, probably lower energy consumption, and slightly faster 
> completion of the file transfer compared to the old code.
> 
> I have uploaded a docket image with the new code in the interop runner. 
> I don't know whether I need to do something special for the satellite 
> interop runner.
> 
> -- Christian Huitema
>