Subject: Questions about recovery draft 29

From: Jonas Reynders <[email-masked]>
Date: Thu, 9 Jul 2020 13:55:35 +0200

I have been looking at the recovery draft 29 of QUIC and comparing it to
TCP, specifically the linux implementation in kernel v5.4. I have a couple
of questions about the design choices that differ from TCP in linux.

The first question is in regards to ack-based loss detection where QUIC
uses both a packet-threshold and time-threshold method. Looking at the
linux implementation (kernel v5.4), TCP uses only a time-threshold method
(RACK) as default. From my understanding both methods were used in linux
some time ago, around the time it was clarified to use both methods for
QUIC as well. However since then TCP changed to RACK only as the default,
from what I found out it was due to tests at google detecting a 10%
reduction in recovery latency (reference:
https://patchwork.ozlabs.org/project/netdev/cover/[email-masked]/).
My question is why QUIC still recommends using both methods as default? I
can understand that in a datacenter context, where RTT values are really
low, RACK can struggle due to timers not being fine grained.

My second question is in regards to the time threshold calculation(6.1.2 in
recovery draft). QUIC uses a constant value (9/8) for the time threshold,
whilst in linux an adaptive threshold is used in the form of a reordering
window variable. In the recovery draft it is mentioned that adaptive
thresholds can be used, however it’s not mandatory. Is there a reason why
QUIC doesn’t use the linux method for an adaptive threshold? The draft also
mentions that reordering can be more common in QUIC and that adaptive
thresholds have the advantage of being more aggressive when there is no
reordering. The same question for the packet threshold, QUIC uses a
constant threshold of 3 packets whilst in linux there is an adaptive
threshold.

My last question is that of merging the TLP and RTO into PTO. If I
understand correctly, this was possible due to probe packet being
ack-eliciting. And, that the time-threshold method would detect loss in a
similar manner to RTO, with the RTO also causing a congestion collapse if
no acks were received for consecutive RTOs. I assume that keeping RTO would
unnecessarily complicate things, is that correct?

---

From: Matt Joras <[email-masked]>
Date: Thu, 9 Jul 2020 12:10:19 -0700

I won't offer specific answers to all your questions as Ian or Jana
likely have better ones, however I'd like to make a general point
about this line of questioning: Linux TCP != IETF TCP, or, said
another way, Implementations and Specifications are not the same
thing. Recall that QUIC is a series of specifications being developed
by the IETF, similar to the myriad RFCs that loosely define "IETF
TCP". Linux's TCP stack is one such interpretation/implementation of
TCP, and while it can inform the decisions of IETF workings, it should
not be an undue determiner of the development of said specifications.
Similarly we have an even wider and more diverse collection of QUIC
interpretations/implementations in various states of maturity and
deployment. Many diverge, sometimes heavily, from the specifications
especially insofar as recovery is concerned. Chrome and us (mvfst) use
BBR as the default congestion controller, others use Cubic or HSTCP or
who knows what else. Chrome, AFAIK has a notion of adaptive reordering
similar to Linux TCP and modifications to the draft-suggested time
thresholds and PTO strategy.

Whether some of these changes should be codified in the QUIC
specifications is a good question. My hope is that these best
practices will gain deployment experience and drive consensus at the
IETF in future iterations of QUIC recovery and encounter fewer
roadblocks than innovations based on TCP have historically
experienced. As it is now though, QUIC's charter, by design, largely
limits us to the adaptation of recovery and congestion control work
that has already gained consensus via other IETF RFCs. That being
said, what we have in the recovery specs may not be the "state of the
art" exactly, but it is still quite good and I would argue has other
significant advantages over what is commonly deployed in TCP, even
Linux TCP.

Matt Joras

On Thu, Jul 9, 2020 at 9:22 AM Jonas Reynders
<[email-masked]> wrote:
>
> I have been looking at the recovery draft 29 of QUIC and comparing it to TCP, specifically the linux implementation in kernel v5.4. I have a couple of questions about the design choices that differ from TCP in linux.
>
> The first question is in regards to ack-based loss detection where QUIC uses both a packet-threshold and time-threshold method. Looking at the linux implementation (kernel v5.4), TCP uses only a time-threshold method (RACK) as default. From my understanding both methods were used in linux some time ago, around the time it was clarified to use both methods for QUIC as well. However since then TCP changed to RACK only as the default, from what I found out it was due to tests at google detecting a 10% reduction in recovery latency (reference: https://patchwork.ozlabs.org/project/netdev/cover/[email-masked]/). My question is why QUIC still recommends using both methods as default? I can understand that in a datacenter context, where RTT values are really low, RACK can struggle due to timers not being fine grained.
>
> My second question is in regards to the time threshold calculation(6.1.2 in recovery draft). QUIC uses a constant value (9/8) for the time threshold, whilst in linux an adaptive threshold is used in the form of a reordering window variable. In the recovery draft it is mentioned that adaptive thresholds can be used, however it’s not mandatory. Is there a reason why QUIC doesn’t use the linux method for an adaptive threshold? The draft also mentions that reordering can be more common in QUIC and that adaptive thresholds have the advantage of being more aggressive when there is no reordering. The same question for the packet threshold, QUIC uses a constant threshold of 3 packets whilst in linux there is an adaptive threshold.
>
> My last question is that of merging the TLP and RTO into PTO. If I understand correctly, this was possible due to probe packet being ack-eliciting. And, that the time-threshold method would detect loss in a similar manner to RTO, with the RTO also causing a congestion collapse if no acks were received for consecutive RTOs. I assume that keeping RTO would unnecessarily complicate things, is that correct?

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 13 Jul 2020 09:38:43 -0500

Just to chime in here, so top-posting.

Matt is correct (that specifications != implementations), but I note that
the transport area has produced, and revised,
https://datatracker.ietf.org/doc/rfc7414/ (A Roadmap for Transmission
Control Protocol (TCP) Specification Documents", that go through what Matt
characterized as the 'myriad RFCs that loosely define "IETF TCP"'

The TCP community used Informational RFCs to document what's a good idea
and what's a bad idea at the per-specification level ("really important to
implement this, that is a really bad idea so don't implement it", because
RFCs were the tools we have.

It might be worth thinking about the form a "QUIC Roadmap" might take,
given that we now have wikis, gitbub, and other tools that could allow the
QUIC roadmap and implementations to be more aware of each other, especially
if this was more granular than "this document".

We talked about this briefly at IETF 106 in TSVAREA. From the minutes at
https://datatracker.ietf.org/meeting/106/materials/minutes-106-tsvarea-02
 ...

   - Ted: Lots of creativity right now. people need to understand what they
   are doing *to* the protocol, not just *with* it. We should write "the
   hitchhiker's guide to QUIC". Intended for all audiences.
   - Mirja: That's what applicability and mgmt docs are for.
   - Ted: That's not what I'm looking for.
   - Spencer: Imagine if TCP roadmap had started before there were 150 RFCs!

Did people have thoughts about that?

Best,

Spencer

On Thu, Jul 9, 2020 at 2:10 PM Matt Joras <[email-masked]> wrote:

> > My second question is in regards to the time threshold calculation(6.1.2
> in recovery draft). QUIC uses a constant value (9/8) for the time
> threshold, whilst in linux an adaptive threshold is used in the form of a
> reordering window variable. In the recovery draft it is mentioned that
> adaptive thresholds can be used, however it’s not mandatory. Is there a
> reason why QUIC doesn’t use the linux method for an adaptive threshold? The
> draft also mentions that reordering can be more common in QUIC and that
> adaptive thresholds have the advantage of being more aggressive when there
> is no reordering. The same question for the packet threshold, QUIC uses a

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Jul 2020 20:13:19 -0700

Jonas,

I agree with Matt's response overall. I'll add a couple of specific
responses to your questions.

On RACK: RACK's key insight is to use timestamps (and time) instead of TCP
sequence numbers for loss detection. RACK makes time-ordering the basis for
loss detection, instead of sequence number ordering.  This difference is
importnat when thinking about retransmissions, detection of spurious
retransmissions, and such. A big but subtle difference between TCP and QUIC
is that QUIC does not _reuse_ its packet numbers. This was a deliberate
design choice, at least in part to ensure that QUIC's packet numbers
represent time-ordering. As a result, using packet numbers for loss
detection in QUIC is already different than base (non-RACK) TCP.

The rest of it is around making constants adaptive, and as Matt notes, it's
not fully clear yet what the "right" answer here is, and we wanted to leave
room for experimentation here. As we learn more, we entirely expect the
mechanisms in the draft to evolve, first in implementations, and then in
specifications.

- jana

On Mon, Jul 13, 2020 at 7:40 AM Spencer Dawkins at IETF <
[email-masked]> wrote:

>    - Ted: Lots of creativity right now. people need to understand what
>    they are doing *to* the protocol, not just *with* it. We should write "the
>> > I have been looking at the recovery draft 29 of QUIC and comparing it
>> to TCP, specifically the linux implementation in kernel v5.4. I have a
>> couple of questions about the design choices that differ from TCP in linux.
>> >
>> > My second question is in regards to the time threshold
>> calculation(6.1..2 in recovery draft). QUIC uses a constant value (9/8) for
>> the time threshold, whilst in linux an adaptive threshold is used in the
>> form of a reordering window variable. In the recovery draft it is mentioned
>> that adaptive thresholds can be used, however it’s not mandatory. Is there
>> a reason why QUIC doesn’t use the linux method for an adaptive threshold?
>> The draft also mentions that reordering can be more common in QUIC and that

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Jul 2020 20:14:51 -0700

Spencer,

It's a good idea, but I think it's a bit early to think of a roadmap. The
core protocol components necessary for QUIC are exactly the three documents
we have right now, nothing more. Once we have more extensions, other
changes in more drafts, it might make sense to think of a roadmap.

- jana

---

From: Ian Swett <[email-masked]>
Date: Mon, 20 Jul 2020 10:14:59 -0400

+1 to Jana's point about QUIC not having a TCP-style sequence number, so
it's always time-based in some sense.

There was a recent discussion of whether adaptive thresholds should be
recommended(Issue #3571 <https://github.com/quicwg/base-drafts/issues/3571>)
which I opened after running experiments and enabling adaptive packet
thresholds by default in the Chromium implementation.  The current text
contains the consensus decision to highlight the value of adaptive
reordering, but not make any normative statements.

For TLP and PTO, QUIC recovery started with both, but it added excess
complexity without any recognizable value, so they were merged into PTO.
Even when both TLP and RTO existed in the recovery draft, they were subtly
different from TCP's, so a new mechanism with a new name made sense.

On Fri, Jul 17, 2020 at 11:15 PM Jana Iyengar <[email-masked]> wrote:

>> https://datatracker.ietf..org/doc/rfc7414/
>> <https://datatracker.ietf.org/doc/rfc7414/> (A Roadmap for Transmission
>> On Thu, Jul 9, 2020 at 2:10 PM Matt Joras <[email-masked]
>> <[email-masked]>> wrote:
>>
>>> > The first question is in regards to ack-based loss detection where
>>> QUIC uses both a packet-threshold and time-threshold method. Looking at the
>>> calculation(6.1...2 in recovery draft). QUIC uses a constant value (9/8)
>>> for the time threshold, whilst in linux an adaptive threshold is used in
>>> the form of a reordering window variable. In the recovery draft it is
>>> mentioned that adaptive thresholds can be used, however it’s not mandatory.
>>> Is there a reason why QUIC doesn’t use the linux method for an adaptive
>>> threshold? The draft also mentions that reordering can be more common in
>>> QUIC and that adaptive thresholds have the advantage of being more
>>> aggressive when there is no reordering. The same question for the packet
>>> threshold, QUIC uses a constant threshold of 3 packets whilst in linux
>>> there is an adaptive threshold.
>>> >

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 20 Jul 2020 10:22:46 -0500

Hi, Jana,

> changes in more drafts, it might make sense to think of a roadmap.

I think we're talking about a race condition between whether we know enough
about QUIC to start a roadmap now that would be useful when the tsunami of
extension proposals arrives (as envisioned in discussions about the NEXT
recharter), or whether that's still mysterious enough that we have no
choice except to wait for extensions (and other draft changes), before we
can produce something useful.

You may very well be right, that we don't have much to say yet, so waiting
is best.

But let's remember that we had this conversation in another year to 18
months, OK? :-)

Best,

Spencer