Subject: Re: [EToSat] Re-chartering for extension work

From: Kuhn Nicolas <[email-masked]>
Date: Tue, 17 Dec 2019 17:27:49 +0000

Hi Mark, Lars,

Many meetings discussed the impacts of QUIC encryption on satellite Performance-enhancing proxy (PEP) and the benefit of exchanging additional transport parameters.
draft-kuhn-quic-0rtt-bdp was mentioned as a possible extension during « Quick QUIC Update » presentation.

We have updated draft-kuhn-quic-0rtt-bdp-05 as an extension of QUIC.
We propose to include it in the re-chartering discussion.

Regards,
Nicolas, Emile and Gorry

https://datatracker.ietf.org/meeting/106/materials/slides-106-tsvarea-quick-quic-update-mark-nottingham-00.pdf 
https://tools.ietf.org/html/draft-kuhn-quic-0rtt-bdp-05 

-----Message d'origine-----
De : QUIC <[email-masked]> De la part de Mark Nottingham
Envoyé : mercredi 11 décembre 2019 22:38
À : IETF QUIC WG <[email-masked]>
Cc : Lars Eggert <[email-masked]>
Objet : Re-chartering for extension work

We've just put out Calls for Adoption for extensions to QUICv1, as we believe that the group has some capacity to discuss them as it finishes work on the core protocol.

However, our charter [1] precludes work on at least some extensions. The specific text in question is:

"""
Extensions that will support partial reliability, and negotiation and use of Forward Error Correction schemes, are out of scope in this version of the working group charter.
"""

*If* we do decide we'd like to adopt, we'll need to update it to something like:

"""
Additionally, the Working Group will deliver [ adopted extensions ]. The Working Group may consider other extension work, but adopting further extensions requires updating this charter.
"""

Please take a look and discuss any concerns; we'll be asking our ADs for such a modification (with appropriate changes to the list of extensions adopted) once our Calls for Adoption complete.

Cheers,

1. https://datatracker.ietf.org/wg/quic/about/

--
Mark Nottingham   https://www.mnot.net/

---

From: Ian Swett <[email-masked]>
Date: Tue, 17 Dec 2019 12:37:07 -0500

I think the satellite use case is an important one, but I have a number of
concerns with the approach in that draft, so I don't believe it's ready for
adoption.

On Tue, Dec 17, 2019 at 12:29 PM Kuhn Nicolas <[email-masked]> wrote:

> Hi Mark, Lars,
>
> Many meetings discussed the impacts of QUIC encryption on satellite
> Performance-enhancing proxy (PEP) and the benefit of exchanging additional
> transport parameters.
> We've just put out Calls for Adoption for extensions to QUICv1, as we
> believe that the group has some capacity to discuss them as it finishes
> work on the core protocol.
>
> Extensions that will support partial reliability, and negotiation and use
> of Forward Error Correction schemes, are out of scope in this version of
> the working group charter.
> """
>
> Additionally, the Working Group will deliver [ adopted extensions ]. The
> Working Group may consider other extension work, but adopting further
> extensions requires updating this charter.
> """
>
> Please take a look and discuss any concerns; we'll be asking our ADs for
> such a modification (with appropriate changes to the list of extensions
> adopted) once our Calls for Adoption complete.
>
> Cheers,
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 17 Dec 2019 09:48:47 -1000

The 0RTT draft is a simple extension defining new transport parameters.
It is a fine draft, but it is only part of the solution, and I don't
know whether we need to engage the charter extension process just for
that. IMHO if we start adopting "satellite" work in QUIC, we should
start by the problem statement rather than the proposed solution.
Nicolas, you have a draft in progress defining "satellite scenarios". I
think that would be a very good starting point.

I am half way through the work to make the picoquic implementation work
well on satellite, and I found the scenario definitions very useful. I
did find the issue mentioned by you, Emile, John and Gorry in your
presentations, but I also found a bunch of other ones. The high BDP
tests stress parts of the code that are not exercised in the other
scenarios, such as reordering out of sequence arrival of frame data in a
very long list of buffers, managing ACK-of-ACK when the SACK list is
long and full of holes, or performing spurious loss detection when the
list of repeated packets is much longer than usual. These are of course
purely implementation issues. Of course, other implementations may or
may not have these specific issues. Maybe they will find different
issues. But the point remains that having a good set of well defined
tests will advance support for the satellite scenarios.

Another issue that I found was dealing with high variance of RTT
measurements induced by asymmetry between the upload and download
capacity. Gorry presented the issue in terms of "load on the ACK path"
and suggested reducing the ACK rate, which should definitely be part of
the recommended solution. But this is not just a load issue, it also
impacts RTT measurements which in turn impact a variety of algorithms --
especially if the implementation relies on delay based congestion
control to be robust in the presence of random packet losses. I dealt
with the RTT measurement issue by defining a one way delay measurement
extension, which has other usages than satellite and might also be a
candidate for WG adoption -- once I manage to actually write it in a
draft of course. There may be more work required to deal with the asymmetry.

-- Christian Huitema

On 12/17/2019 7:37 AM, Ian Swett wrote:
> I think the satellite use case is an important one, but I have a
> number of concerns with the approach in that draft, so I don't believe
> it's ready for adoption.
>
> On Tue, Dec 17, 2019 at 12:29 PM Kuhn Nicolas <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi Mark, Lars,
>
>     Many meetings discussed the impacts of QUIC encryption on
>     satellite Performance-enhancing proxy (PEP) and the benefit of
>     exchanging additional transport parameters.
>     draft-kuhn-quic-0rtt-bdp was mentioned as a possible extension
>     during « Quick QUIC Update » presentation.
>
>     De : QUIC <[email-masked] <mailto:[email-masked]>>
>     De la part de Mark Nottingham
>     À : IETF QUIC WG <[email-masked] <mailto:[email-masked]>>
>     Cc : Lars Eggert <[email-masked] <mailto:[email-masked]>>
>     We've just put out Calls for Adoption for extensions to QUICv1, as
>     we believe that the group has some capacity to discuss them as it
>     finishes work on the core protocol.
>
>     However, our charter [1] precludes work on at least some
>     extensions. The specific text in question is:
>
>     """
>     Extensions that will support partial reliability, and negotiation
>     and use of Forward Error Correction schemes, are out of scope in
>     this version of the working group charter.
>     """
>
>     Additionally, the Working Group will deliver [ adopted extensions
>     ]. The Working Group may consider other extension work, but
>     adopting further extensions requires updating this charter.
>     """
>
>     Please take a look and discuss any concerns; we'll be asking our
>     ADs for such a modification (with appropriate changes to the list
>     of extensions adopted) once our Calls for Adoption complete.
>
>     Cheers,
>
> _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat

---

From: Christian Huitema <[email-masked]>
Date: Tue, 17 Dec 2019 16:03:20 -1000

I am looking at this draft before trying to implement it. I think I
understand the spirit of the draft, but I have a couple of questions
regarding how this is supposed to work.

There is probably a typo in the formulation of the BDP as "last_bdp can
be set to min_rtt*congestion_window". This would give a value in
bytes*seconds, when we expect bytes. Using "min_rtt*received_rate" would
work, since clients can certainly measure the rate at which the server
was sending data. In fact, sending both the BDP and the RTT would make
sense.

By default, the number of RTT required to reach a full bandwidth BDP is
log2(BDP/IWbytes). In my tests of the "250Mbps/3Mbps" scenario, the
required BDP is reached in 3.6 seconds after starting with IW10.
Starting with IW160 would shave 2.4 seconds on that total, etc. We would
of course need RTT evaluation and pacing, but for high BDP scenarios we
need that anyhow. Learning the BDP in advance would help there.

All that sounds good, but I don't know how the client can safely
estimate IW. The draft says "use the previous sender value", but I don't
understand how the client learns that. I think that the client knows the
previous BDP, and it also knows how long ago the measurement was made.
If the time is short, there are good chances that the new conditions are
similar to the previous ones, but if the time is long there is a high
risk that they are different. Let's assume that the client sends a BDP
parameter with a large value. Should the server trust it? Should it
initialize CWIN to a fraction of that, and chose a smaller fraction if
the measurement was old? How does the server know how old the
measurement was?

There is also a security aspect there. The server that blindly trusts
the BDP option will blast a large number of packets. What if the client
is trying to sabotage the local network? Why should the server trust the
client? The security section does not cover that...

-- Christian Huitema

---

From: Lars Eggert <[email-masked]>
Date: Wed, 18 Dec 2019 08:32:40 +0200

Hi,

> We propose to include it in the re-chartering discussion.

all the extensions included in the recharter we are discussing here have been reviewed by the WG and are actually under adoption at the moment.

I agree that the satellite use case is important, but neither the problem nor the proposed solution is at the same stage, so I wouldn't want to include it in the current recharter.

(Same goes for pretty much *any* other extension, FWIW.)

Lars

---

From: Kuhn Nicolas <[email-masked]>
Date: Thu, 19 Dec 2019 09:33:13 +0000

Hi, 

> (Same goes for pretty much *any* other extension, FWIW.)

Thank you for your answer. 
We understand that the proposal is not at the same stage as others and understand the decision.
We mostly wanted to share with the group that we are working to improve the current version of the proposal and are happy to have feedback. 

Kind regards, 

Nicolas 

-----Message d'origine-----
De : Lars Eggert <[email-masked]> 
Envoyé : mercredi 18 décembre 2019 07:33
À : Kuhn Nicolas <[email-masked]>
Cc : Mark Nottingham <[email-masked]>; Gorry Fairhust <[email-masked]>; [email-masked]; IETF QUIC WG <[email-masked]>; [email-masked]
Objet : Re: Re-chartering for extension work

Hi,

Lars

---

From: Kuhn Nicolas <[email-masked]>
Date: Thu, 19 Dec 2019 10:39:40 +0000

Hi,

Thanks for your feedback – it helps us very much.
See inline for some comments to your email.

Cheers,

Nico

De : Christian Huitema <[email-masked]>
Envoyé : mardi 17 décembre 2019 20:49
À : Ian Swett <ianswett=[email-masked]>; Kuhn Nicolas <[email-masked]>
Cc : Gorry (erg) <[email-masked]>; [email-masked]; Mark Nottingham <[email-masked]>; [email-masked]; Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Objet : Re: [EToSat] Re-chartering for extension work

The 0RTT draft is a simple extension defining new transport parameters. It is a fine draft, but it is only part of the solution, and I don't know whether we need to engage the charter extension process just for that.

[NK] I think that 0RTT document is not focusing particularly on satellite (even if it was driven by a satellite use case at the beginning). If QUIC is used as a tunneling solution, there may be a large BDP and a connection would be bi-directional transmission of data. In this case, sharing information from the client and the server could be beneficial. Another thing is the optimization of asymmetric links that is not specific to satellite – see my comment below.

IMHO if we start adopting "satellite" work in QUIC, we should start by the problem statement rather than the proposed solution. Nicolas, you have a draft in progress defining "satellite scenarios". I think that would be a very good starting point.

I am half way through the work to make the picoquic implementation work well on satellite, and I found the scenario definitions very useful. I did find the issue mentioned by you, Emile, John and Gorry in your presentations, but I also found a bunch of other ones. The high BDP tests stress parts of the code that are not exercised in the other scenarios, such as reordering out of sequence arrival of frame data in a very long list of buffers, managing ACK-of-ACK when the SACK list is long and full of holes, or performing spurious loss detection when the list of repeated packets is much longer than usual. These are of course purely implementation issues. Of course, other implementations may or may not have these specific issues. Maybe they will find different issues. But the point remains that having a good set of well defined tests will advance support for the satellite scenarios.

[NK] Thank you for these feedback. We have indeed provided regression tests in the QUIC 4 SAT document. We plan on showing some results at next IETF. We also plan to provide referenced values for each test.

Another issue that I found was dealing with high variance of RTT measurements induced by asymmetry between the upload and download capacity. Gorry presented the issue in terms of "load on the ACK path" and suggested reducing the ACK rate, which should definitely be part of the recommended solution. But this is not just a load issue, it also impacts RTT measurements which in turn impact a variety of algorithms -- especially if the implementation relies on delay based congestion control to be robust in the presence of random packet losses. I dealt with the RTT measurement issue by defining a one way delay measurement extension, which has other usages than satellite and might also be a candidate for WG adoption -- once I manage to actually write it in a draft of course. There may be more work required to deal with the asymmetry.

[NK] One thing that is not particularly related to satellite but to asymmetry in general could be done through the 0-RTT proposed extension – such as we proposed in the section 5 of the current version https://tools.ietf.org/html/draft-kuhn-quic-0rtt-bdp-05#section-5 .

-- Christian Huitema
On 12/17/2019 7:37 AM, Ian Swett wrote:
I think the satellite use case is an important one, but I have a number of concerns with the approach in that draft, so I don't believe it's ready for adoption.

On Tue, Dec 17, 2019 at 12:29 PM Kuhn Nicolas <[email-masked]<mailto:[email-masked]>> wrote:
Hi Mark, Lars,

Regards,
Nicolas, Emile and Gorry

-----Message d'origine-----
De : QUIC <[email-masked]<mailto:[email-masked]>> De la part de Mark Nottingham
Envoyé : mercredi 11 décembre 2019 22:38
À : IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Cc : Lars Eggert <[email-masked]<mailto:[email-masked]>>
Objet : Re-chartering for extension work

"""

"""

Cheers,

EToSat mailing list

[email-masked]<mailto:[email-masked]>

---

From: Kuhn Nicolas <[email-masked]>
Date: Fri, 20 Dec 2019 10:29:45 +0000

Thank you for this feedback.
We are working on an updated version of the document. 
At least, we start a security discussion section. We also adapt the last_bdp parameter as you suggested.  
The parameters would likely to be (at least) ;
   o  last_bdp (0x000X): The BDP measured on the previous connection by
      the server.  It is an integer in unit of bytes.  Using the
      congestion_window defined in [I-D.ietf-quic-recovery], last_bdp
      can be set to congestion_window.
   o  last_rtt (0x000X): The minimum RTT measured on the previous
      connection by the server.  It is an integer in unit of
      milliseconds.  Using the min_rtt defined in
      [I-D.ietf-quic-recovery], last_rtt can be set to min_rtt.
   o  initial_max_pkt_number (0x000X): The maximum amount of packets
      that the server is allowed to send when reconnecting.  It is an
      integer in unit of packets.

Cheers, 

Nico

-----Message d'origine-----
De : Christian Huitema <[email-masked]> 
Envoyé : mercredi 18 décembre 2019 03:03
À : Kuhn Nicolas <[email-masked]>
Cc : 'Gorry (erg)' <[email-masked]>; [email-masked]' <[email-masked]>; [email-masked]
Objet : Re: BDP parameters draft (was: [EToSat] Re-chartering for extension work)

I am looking at this draft before trying to implement it. I think I understand the spirit of the draft, but I have a couple of questions regarding how this is supposed to work.

There is probably a typo in the formulation of the BDP as "last_bdp can be set to min_rtt*congestion_window". This would give a value in bytes*seconds, when we expect bytes. Using "min_rtt*received_rate" would work, since clients can certainly measure the rate at which the server was sending data. In fact, sending both the BDP and the RTT would make sense.

By default, the number of RTT required to reach a full bandwidth BDP is log2(BDP/IWbytes). In my tests of the "250Mbps/3Mbps" scenario, the required BDP is reached in 3.6 seconds after starting with IW10.
Starting with IW160 would shave 2.4 seconds on that total, etc. We would of course need RTT evaluation and pacing, but for high BDP scenarios we need that anyhow. Learning the BDP in advance would help there.

All that sounds good, but I don't know how the client can safely estimate IW. The draft says "use the previous sender value", but I don't understand how the client learns that. I think that the client knows the previous BDP, and it also knows how long ago the measurement was made.
If the time is short, there are good chances that the new conditions are similar to the previous ones, but if the time is long there is a high risk that they are different. Let's assume that the client sends a BDP parameter with a large value. Should the server trust it? Should it initialize CWIN to a fraction of that, and chose a smaller fraction if the measurement was old? How does the server know how old the measurement was?

There is also a security aspect there. The server that blindly trusts the BDP option will blast a large number of packets. What if the client is trying to sabotage the local network? Why should the server trust the client? The security section does not cover that...

-- Christian Huitema

---

From: Kuhn Nicolas <[email-masked]>
Date: Mon, 23 Dec 2019 16:07:26 +0000

Hi, 

Following your comments, we have updated the draft with (amongst various changes) a more accurate description of the last_bdp and a security discussion. 
https://datatracker.ietf.org/doc/draft-kuhn-quic-0rtt-bdp/ 
Do not hesitate to share your views with us. 

Merry Christmas, 

Nico

-----Message d'origine-----

-- Christian Huitema