Subject: Re: [quicwg/base-drafts] Changing the Default QUIC ACK Policy (#3529)

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 1 May 2020 16:01:28 +0100

These results/article on computational efficiency are really 
interesting. Please see below.

On 01/05/2020 05:40, Kazuho Oku wrote:
> Hi, Gorry
>
> Thank you for sharing your experiments. I'm delighted to see results 
> showing that use of ack10 gives us comparable throughput to ack2.
>
> Recently, I have been looking into how ack-frequency affects the 
> computational efficiency of the server [1]. Using the (almost) same 
> setup, I have today run a benchmark comparing three ACK strategies: 
> ack2, ack10, ack 1/8cwnd.
>
> ack2: 334Mbps
> ack10: 414Mbps
> ack 1/8cwnd: 487Mbps
>
> As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. 
> That's because there are still frequent ACKs. They take considerable 
> amount of CPU cycles.
>
This is useful, we see here there's a clear benefit here in using a 
larger ACK Ratio also in terms of server load.  We can see already the 
benefit at 1:10 and GSO helps as expected.
> If we want to see QUIC being as lightweight as TLS over TCP, it is 
> critically important to reduce ACK frequency as much as possible. This 
> is because the cost of processing ACK in QUIC is much more expensive 
> than TCP, due QUIC being a userspace transport.
>
We think that this is also really true. QUIC ACKs are also larger than 
TCP ACKs (see https://erg.abdn.ac.uk/users/gorry/ietf/QUIC/).
> Based on this, if we are to change what we have in the base drafts, 
> I'd much prefer adopting what we have in the proposed delayed ack 
> extension [2], rather than just changing the default ack policy from 
> ack2 to ack10. By adopting what we have in the delayed ack extension, 
> we can expect better performance, and the endpoints would have the 
> freedom to change the behavior when the default ack policy is deemed 
> insufficient (regardless of the default being ack2 or ack10).
>
> [1] 
> https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency
> [2] https://tools.ietf.org/html/draft-iyengar-quic-delayed-ack-00
>
I don't think the two approaches conflict.

The difference between the two approaches is that the small change to 
the base specification to use an ACK Ratio 1:10 ensures the default 
return path is at least as good as TCP for common traffic (even with 
common TCP ACK Thinning). Without this change, the end user could 
experience significant differences in performance between TCP and 
between different QUIC versions over asymmetric paths. 

Whereas to me, the transport parameter extension looks like an exciting 
way to enable many more things, but there are also many ways this could 
usefully evolve based on CC, offload, pacing etec and I think this will 
take much more to work-out what is needed, safe and best ... then to 
reach consensus. We still also need to specify a default ACK Ratio in 
this case as well.

Best wishes,

Gorry 
> 2020年4月28日(火) 20:37 Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>>:
> >
> > On 27/04/2020 17:19, Lars Eggert wrote:
> >
> > Folks who care about a change here need to prioritize progressing 
> this issue. We're close to being able to last-call the specs, and 
> would like this resolved in time..
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub, or unsubscribe.
> >
> > We’ve completed a round of updated analysis of the proposed change 
> to the QUIC spec, using quicly (changing to a 1:10 policy), and 
> chromium (which currently does 1:10 by default). Our hope is that this 
> change to the spec,  to actually make standard end-to-end QUIC work 
> better than TCP over asymmetric paths!
> >
> > This is a short slide deck about the performance of the proposed change:
> > https://erg.abdn.ac.uk/users/gorry/ietf/QUIC/QUIC-ack10-24-april-00.pdf
> >
> > There have been a number of other change made by the QUIC ID editors 
> relating to ACK processing, so we wanted to be sure these results were 
> up to date. (These results do not consider additional ACKs after 
> re-ordering - reflecting the latest text removing that.)
> >
> > We also really hope this will motivate a change the practice of 
> forcing of QUIC fallback to TCP (with PEP/Thining) over highly 
> asymmetric paths. People may also be interested in work for L4S to 
> evaluate the impact of ACK Thining for TCP-Prague.
> >
> > Please do provide feedback, or any questions on this - as Lars 
> notes, we think the time is running out for considering new changes, 
> even relatively simple ones like this.
> >
> > Gorry
> >
> > P.S. First email: 
> https://mailarchive.ietf.org/arch/msg/quic/NZBbuY88v0lrVspQE2A5br83asA/
>
>
>
> --
> Kazuho Oku

---

From: Matt Joras <[email-masked]>
Date: Fri, 1 May 2020 11:14:34 -0700

> Recently, I have been looking into how ack-frequency affects the
computational efficiency of the server [1]. Using the (almost) same setup,
I have today run a benchmark comparing three ACK strategies: ack2, ack10,
ack 1/8cwnd.
> As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd.
That's because there are still frequent ACKs. They take considerable amount
of CPU cycles.
> If we want to see QUIC being as lightweight as TLS over TCP, it is
critically important to reduce ACK frequency as much as possible. This is
because the cost of processing ACK in QUIC is much more expensive than TCP,
due QUIC being a userspace transport.
>
> Based on this, if we are to change what we have in the base drafts, I'd
much prefer adopting what we have in the proposed delayed ack extension
[2], rather than just changing the default ack policy from ack2 to ack10.
By adopting what we have in the delayed ack extension, we can expect better
performance, and the endpoints would have the freedom to change the
behavior when the default ack policy is deemed insufficient (regardless of
the default being ack2 or ack10).
I have thought about this a fair bit over the last few months, and I also
think we should consider bringing the delayed-ack mechanism into the core
transport. The main reasons to oppose at this stage are the risk of the
design process causing WG delays, and mandating further implementation
complexity in the core transport. For the latter, I will argue the
complexity is not significant. I am very curious to hear opinions from
other implementers as well.

The extension requires the following to be implemented to function as
intended:
1. The ability to update its maximum_ack_delay. I expect some
implementations are simply hard-coding the value of 25ms from the transport
draft. As long as this value can be updated mid-connection, no changes to
ACK sending logic are required.
2. Similarly, the implicit ACK threshold in the transport draft ("An ACK
frame SHOULD be generated for at least every second ack-eliciting packet.")
is likely implemented by most as some sort of threshold value which is
currently a constant of 2. If an implementation makes this a variable which
can be updated, again no changes to ACK sending are needed.
3. The last mechanism which is required is the ability to have a dynamic
threshold when declaring a packet lost. The theme is the same here, what is
likely a constant in the implementation has to be made a variable.
4. Handling of an additional transport parameter specifying the
minimum_ack_delay. This is straightforward, as it is simply a lower bound
for the variable in 1. which an implementation can tolerate.
5. Parsing of the ACK_FREQUENCY frame type. This should be trivial, as it
is 4 variable length integers.

This seems a reasonably small amount of work for an implementer, as it
mostly involves turning constants into variables. Anecdotally, it took me
an afternoon to implement a prototype in mvfst.

As for the need, I think there is a preponderance of evidence, both in the
form of research from Gorry, Kazuho, and Jana, as well as deployment
experience from us and Google to suggest that this is a problem which needs
to be addressed for QUIC to be a viable replacement for TCP. Though we have
two major deployments which have utilized ack-every-10 and we know it to be
a decent improvement, my opinion is that it is overly coarse and
inflexible. If we codify the ACK_FREQUENCY mechanism in the transport we
ensure flexibility on these policies in the future; through experimentation
we can arrive at "best practice" algorithms senders can implement to
balance the tradeoffs involved.

We should hear from others as well, especially those not engaged with the
delayed-ack work, as I know there is likely very little appetite for adding
new frames into the core documents at this point in the process.

Matt Joras

P.S. Removing the github email reply, as they are showing up on the linked
issue as replies from Gorry, which is overly confusing.

> 2020年4月28日(火) 20:37 Gorry Fairhurst <[email-masked]>:
> >
> > Folks who care about a change here need to prioritize progressing this
issue. We're close to being able to last-call the specs, and would like
this resolved in time..
> > We’ve completed a round of updated analysis of the proposed change to
the QUIC spec, using quicly (changing to a 1:10 policy), and chromium
(which currently does 1:10 by default). Our hope is that this change to the
spec,  to actually make standard end-to-end QUIC work better than TCP over
asymmetric paths!
> > There have been a number of other change made by the QUIC ID editors
relating to ACK processing, so we wanted to be sure these results were up
to date. (These results do not consider additional ACKs after re-ordering -
reflecting the latest text removing that.)
> >
> > We also really hope this will motivate a change the practice of forcing
of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths.
People may also be interested in work for L4S to evaluate the impact of ACK
Thining for TCP-Prague.
> >
> > Please do provide feedback, or any questions on this - as Lars notes,
we think the time is running out for considering new changes, even
relatively simple ones like this.