Subject: Proposed extension for delayed acknowledgments

From: Jana Iyengar <[email-masked]>
Date: Thu, 23 Jan 2020 13:50:59 -0800

Hi all,

Ian and I have proposed an extension to enable sender-side control of
acknowledgement frequency: draft-iyengar-quic-delayed-ack-00
<https://datatracker.ietf.org/doc/draft-iyengar-quic-delayed-ack/>.

The problem this extension solves was originally discussed in #1978
<https://github.com/quicwg/base-drafts/issues/1978>, where we agreed that
this might be best done in an extension. The desire to reduce the frequency
of acknowledgements was more recently brought up and discussed in #3304
<https://github.com/quicwg/base-drafts/issues/3304>, where it seemed clear
that there was interest in working on this now. This proposed extension is
a result.

This is an individual draft. Please feel free to use the following github
repo, which is NOT the working group's repo, for filing issues and
discussion pertaining to the draft:
https://github.com/janaiyengar/ack-frequency

- jana

---

From: Ian Swett <[email-masked]>
Date: Thu, 23 Jan 2020 16:59:15 -0500

Thanks Jana!

This draft was motivated both by the fact Chrome(and Cronet) clients
already have a variation on this enabled(ack every 10 packets or 1/4
min_rtt), and it has shown huge(ie: 2x+) improvements in client CPU,
particularly on mobile clients.  Other clients have had similar experiences.

Additionally, testing on satellite networks has shown this is a critical
performance improvement, because these networks are commonly highly
asymmetric and they can't add proxies which terminate transport(ie: TCP)
separately from crypto for QUIC.

Additionally, by specifying the update_max_ack_delay field in microseconds,
this is a potential fix for Issue #3363
<https://github.com/quicwg/base-drafts/issues/3363>(max_ack_delay should be
in us, not ms).

I'm very excited about this extension and our team intends to implement it
relatively soon in Chrome and Google servers when an IETF QUIC draft is
enabled.

Ian

On Thu, Jan 23, 2020 at 4:51 PM Jana Iyengar <[email-masked]> wrote:

> <https://github.com/quicwg/base-drafts/issues/3304>, where it seemed
> clear that there was interest in working on this now. This proposed
> extension is a result.
>

---

From: Matt Joras <[email-masked]>
Date: Thu, 23 Jan 2020 14:25:05 -0800

Thank you Jana and Ian for doing this!

As I noted on #3304, we have also observed this sort of change is critical
to reducing client-originated ACK traffic for typical Internet content, and
for CPU performance of both client and server. Noting here that I have
opened an issue (#17
<https://github.com/janaiyengar/ack-frequency/issues/17>) for the
following: given that we have a basic heuristic that seems to work pretty
well for multiple deployments (default behavior for first 100 packets,
packet tolerance of 10 after), do we want to make that recommendation
explicit in the extension itself?

Matt Joras

On Thu, Jan 23, 2020 at 1:59 PM Ian Swett <ianswett=
[email-masked]> wrote:

> Additionally, by specifying the update_max_ack_delay field in
> microseconds, this is a potential fix for Issue #3363
> <https://github.com/quicwg/base-drafts/issues/3363>(max_ack_delay should
> be in us, not ms).
>
>> <https://github.com/quicwg/base-drafts/issues/1978>, where we agreed
>> that this might be best done in an extension. The desire to reduce the
>> frequency of acknowledgements was more recently brought up and discussed in
>> #3304 <https://github.com/quicwg/base-drafts/issues/3304>, where it
>> seemed clear that there was interest in working on this now. This proposed
>> extension is a result.
>>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 23 Jan 2020 13:53:34 -1000

On 1/23/2020 12:25 PM, Matt Joras wrote:

> As I noted on #3304, we have also observed this sort of change is
> critical to reducing client-originated ACK traffic for typical
> Internet content, and for CPU performance of both client and server.
> Noting here that I have opened an issue (#17
> following: given that we have a basic heuristic that seems to work
> pretty well for multiple deployments (default behavior for first 100
> packets, packet tolerance of 10 after), do we want to make that
> recommendation explicit in the extension itself?

+1

Also, I am a bit concerned of the slippery slope there. Suppose a peer
receives a frame that says "only every 22 packets please". Are we going
to flag a protocol violation if the receiver acks after 23 packets? Or
10? Or 1?

-- Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 23 Jan 2020 16:08:31 -0800

Christian,

No, that would be silly :-) There are exceptions allowed at a receiver --
such as when the receiver receives a batch of packets -- and we have to
allow for such things to happen. Also, ack loss, late timers, scheduler
delays, are all things that can cause a receiver to not follow the schedule
exactly. That's ok.

- jana

On Thu, Jan 23, 2020 at 3:53 PM Christian Huitema <[email-masked]>
wrote:

> receives a frame that says "only every 22 packets please". Are we going to
> flag a protocol violation if the receiver acks after 23 packets? Or 10? Or
> 1?
>
> -- Christian Huitema
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 27 Jan 2020 08:50:53 -0500

I'll commit to implementing this by the Interop next week if at least
one other implementation does so as well.

Takers?  (And let's pick the TP ID.)

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 28 Jan 2020 08:58:40 -0500

OK, thanks for the heads up: I'll wait for the updated draft.  I hope
it is ready in a day or two so that Matt Joras and I (and others?) can
have it by the Interop.

  - Dmitri.

On Mon, Jan 27, 2020 at 03:46:16PM -0800, Jana Iyengar wrote:
> Thanks for proposing the TP: I'll update the draft to use it, knowing that
> we're squatting on it.
> FWIW, I'm going to change the Reordering TP to a frame, per the discussion
> on #18 <https://github.com/janaiyengar/ack-frequency/issues/18>, so hold
> off on implementing that bit if you can.
> 
> On Mon, Jan 27, 2020 at 1:15 PM Mike Bishop <[email-masked]> wrote:
> 
> > The registry doesn’t exist until the document defining it ships, does it?
> > So technically, there’s nothing to make a provisional registration in at
> > this point; we’re just picking a value and squatting on it.
> >
> >
> >
> > (Which, to be clear, is a totally reasonable experimentation strategy
> > within the working group.)
> >
> >
> >
> > *From:* QUIC <[email-masked]> *On Behalf Of * Lucas Pardue
> > *Sent:* Monday, January 27, 2020 3:06 PM
> > *To:* Matt Joras <[email-masked]>; Jana Iyengar <[email-masked]>;
> > QUIC WG <[email-masked]>; Ian Swett <[email-masked]>
> > *Subject:* Re: Proposed extension for delayed acknowledgments
> >
> >
> >
> > Having been guilty of writing proposals that include "TBD" codepoints
> > myself, I think it would be a good idea if proposals started to exercise
> > the provisional registration policy defined in QUIC Transport draft 25
> > Section 22.1.1 [1]
> >
> >
> >
> >    Provisional registration of codepoints are intended to allow for
> >
> >    private use and experimentation with extensions to QUIC.  Provisional
> >
> >    registrations only require the inclusion of the codepoint value and
> >
> >    contact information.  However, provisional registrations could be
> >
> >    reclaimed and reassigned for another purpose.
> >
> > That saves interested folk from having to come up with their own in order
> > to do experimental interop. (thanks for picking something and soliciting it
> > Dmitri).
> >
> >
> >
> > Lucas
> >
> >
> >
> > [1] -
> > https://tools.ietf.org/html/draft-ietf-quic-transport-25#section-22.1
> >
> >
> >
> > On Mon, Jan 27, 2020 at 6:46 PM Dmitri Tikhonov <
> > [email-masked]> wrote:
> >
> > Cool.  As for TP, let's use 0xDE1A ("DEL"ayed "A"ck)?
> >
> >   - Dmitri.
> >
> > On Mon, Jan 27, 2020 at 10:01:35AM -0800, Matt Joras wrote:
> > > I'll bite. I can do my best to implement it for interop. We already have
> > a
> > > mechanism to do something with an out-of-band configuration.
> > >
> > > Matt
> > >
> > > On Mon, Jan 27, 2020 at 5:51 AM Dmitri Tikhonov <
> > [email-masked]>
> > > wrote:
> > >
> > > > that
> > > > clear
> > > > > that there was interest in working on this now. This proposed
> > extension
> > > > is
> > > > > a result.
> > > > >

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jan 2020 08:57:02 -0800

Yup, will try to get a PR out today.

On Tue, Jan 28, 2020, 5:58 AM Dmitri Tikhonov <[email-masked]>
wrote:

> > > > > > <
> https://datatracker.ietf.org/doc/draft-iyengar-quic-delayed-ack/>.
> > > > > >
> > > > > > This is an individual draft. Please feel free to use the
> following
> > > github

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 28 Jan 2020 19:19:09 +0000

On Mon, 27 Jan 2020, 21:15 Mike Bishop, <[email-masked]> wrote:

> Fair point, I agree.

As an individual, I might predict that we'll see a few proposals that need
code points before the registries get set up. I think there is benefit in
(provisional) provisional registration to help set a good example and adopt
the spirit of the guidance.

Lucas

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jan 2020 11:28:37 -0800

On Tue, Jan 28, 2020 at 11:19 AM Lucas Pardue <[email-masked]>
wrote:

>> The registry doesn’t exist until the document defining it ships, does
>> it?  So technically, there’s nothing to make a provisional registration in
>> at this point; we’re just picking a value and squatting on it.
>>
>> Fair point, I agree.
>

Given the space available, I think informal agreement among implementers
(as we're doing here and now) is probably adequate. Let's not set another
system up in preparation for the provisional registry that ought to happen
Real Soon Now.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 28 Jan 2020 19:44:24 +0000

On Tue, 28 Jan 2020, 19:28 Jana Iyengar, <[email-masked]> wrote:

> system up in preparation for the provisional registry that ought to happen

My observation was that of all the available space, none were selected. It
doesn't seem too onerous a habit for 00 drafts to pick a value and replace
IANA TBD sections with value and status fields. It might be squatting but
in practice aliases with the registration process pretty well.

Lucas

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jan 2020 13:08:05 -0800

> in practice aliases with the registration process pretty well.

Fair enough, and I agree.

---

From: Maxime Piraux <[email-masked]>
Date: Wed, 29 Jan 2020 13:13:53 +0000

Hi Jana,

Enabling sender-side control of the receiver behaviour is also part of 
our efforts with PQUIC <pquic.org>. We propose a new paradigm for 
deploying new extensions to QUIC. For some use-cases, instead of 
defining a standard wire format enabling the control of specific 
mechanisms, we propose to directly inject code that implements the 
required feature.

More concretely, adding a custom ack policy requires replacing a single 
function (aka protocol operation) in PQUIC. The protocol operation 
is_ack_needed determines whether sending an ack is required. An example 
of the code that could be injected to change the packet tolerance to 10 
and the max ack delay to 10ms can be found below.

/* The function returns whether an ack is needed for a given packet 
context */
protoop_arg_t is_ack_needed(picoquic_cnx_t *cnx)
{
     uint64_t current_time = (uint64_t) cnx->protoop_inputv[0]; /* 
Current time in microsec */
     picoquic_packet_context_enum pc = (picoquic_packet_context_enum) 
cnx->protoop_inputv[1];
     picoquic_path_t *path_x = (picoquic_path_t*) cnx->protoop_inputv[2];
     picoquic_packet_context_t *pkt_ctx = &path_x->pkt_ctx[pc];

     protoop_arg_t ack_needed = 0;
     uint64_t packet_tolerance = 10;  /* 10 packets */
     uint64_t max_ack_delay = 10000;  /* 10 ms */

     if (pkt_ctx->ack_needed) {
         if (pkt_ctx->highest_ack_sent + packet_tolerance <= 
pkt_ctx->first_sack_item.end_of_sack_range ||
             pkt_ctx->highest_ack_time + max_ack_delay <= current_time) {
             ack_needed = 1;
         }
     }

     return ack_needed;
}

Our approach can also be used to deploy an extension defining new frames 
as explained in [PQUIC]. Every time that there will be an interest in 
enabling the dynamic modification of a particular QUIC mechanism, having 
a common interface to inject code will shorten the time to deploy and 
experiment with those modifications.

[PQUIC]: Quentin De Coninck, François Michel, Maxime Piraux, et al. 
Pluginizing QUIC. SIGCOMM 2019. https://dl.acm.org/authorize?N680239

Maxime

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 29 Jan 2020 15:46:43 -0500

> Thanks for proposing the TP: I'll update the draft to use it, knowing that

And 0xAF (obvious reasons) for the ACK-FREQUENCY frame?

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Wed, 29 Jan 2020 16:34:28 -0500

Good point Dmitri, PR(https://github.com/janaiyengar/ack-frequency/pull/21)
created.

I didn't define a codepoint for the reordering field, because I believe it
needs more discussion and isn't critical for initial interop.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 29 Jan 2020 16:13:06 -0800

On 1/29/2020 1:34 PM, Ian Swett wrote:
> Good point Dmitri,
> PR(https://github.com/janaiyengar/ack-frequency/pull/21) created.
>
> I didn't define a codepoint for the reordering field, because I
> believe it needs more discussion and isn't critical for initial interop.
>
> On Wed, Jan 29, 2020 at 3:46 PM Dmitri Tikhonov
> <[email-masked] <mailto:[email-masked]>> wrote:
>

Cool. Good way to test encoding of frame type on two bytes!

-- Christian Huitema

---

From: David Schinazi <[email-masked]>
Date: Wed, 29 Jan 2020 17:33:16 -0800

To facilitate experimentation and to avoid collisions, I've created a
temporary registry on our wiki:

https://github.com/quicwg/base-drafts/wiki/Temporary-IANA-Registry

I'd encourage draft writers to avoid TBD and instead pick a random value
and add it to the wiki.

Hope this helps,
David

On Wed, Jan 29, 2020 at 4:13 PM Christian Huitema <[email-masked]>
wrote:

> On Wed, Jan 29, 2020 at 3:46 PM Dmitri Tikhonov <

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 30 Jan 2020 09:45:04 -0500

Thanks, David!

I added loss-bits and delayed-acked extensions to the registry.

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 30 Jan 2020 14:48:08 +0000

Agreed, thanks for taking the initiative and making that page David.

On Thu, 30 Jan 2020, 14:45 Dmitri Tikhonov, <[email-masked]>
wrote:

> > > Good point Dmitri, PR(
> https://github.com/janaiyengar/ack-frequency/pull/21)
> > > created.
> > >
> believe it
> > >> > Thanks for proposing the TP: I'll update the draft to use it,
> knowing
> > >> that
> > >> > we're squatting on it.
> > >>

---

From: Ian Swett <[email-masked]>
Date: Thu, 30 Jan 2020 21:42:07 -0500

Thanks for adding the page(and the ack-frequency codepoints).  Is there a
trick for getting there from the main Wiki page, or do I just need to save
the URL?
https://github.com/quicwg/base-drafts/wiki

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 31 Jan 2020 04:34:53 +0000

I've just added it as a link in the "Top Pages" section on the main wiki
page.

Githubs poor UI still doesn't help, we could reflect those link into the
main wiki page text too.

On Fri, 31 Jan 2020, 02:42 Ian Swett, <[email-masked]> wrote:

>>> > On Wed, Jan 29, 2020 at 4:13 PM Christian Huitema <[email-masked]
>>> >
>>> > wrote:
>>> >
>>> > >

---

From: Ian Swett <[email-masked]>
Date: Fri, 31 Jan 2020 09:13:20 -0500

Thanks Lucas!