Subject: Handshake processing delays and impact on recovery

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Jul 2020 02:00:51 -0700

I'd like to bring this slightly tricky issue under the wg's eyes.
Discussions so far have yielded a strategy that I propose further below.
This summarizes conversations in 3821
<https://github.com/quicwg/base-drafts/issues/3821> and 3874
<https://github.com/quicwg/base-drafts/pull/3874> (and the outcomes might
touch 3889 <https://github.com/quicwg/base-drafts/issues/3889>). The
purpose of this email is to see if we can agree on the strategies outlined
here, so we can move forward on the PR there and/or other associated issues
and PRs.

Problems:

During the handshake, an endpoint can receive packets (carrying data or
acknowledgements) that it cannot decrypt because the requisite keys are not
yet available. This can happen because the crypto machinery at the endpoint
is waiting for earlier packets to be received to yield the keys. Similarly,
a client might not be able to read 1-RTT packets received from the server
because it is waiting for certificate validation to complete.

1.

The assumption in the TLS specification is that when an endpoint is busy
doing something (certificate validation or key generation for instance), it
won't be able to respond to received packets with acknowledgements (Section
4.1.4
<https://quicwg.org/base-drafts/draft-ietf-quic-tls.html#section-4.1.4-1>).

Contrary to the assumption in the spec, Chrome and Firefox both process the
certificate chain asynchronously, and therefore acks handshake packets
immediately, even the ones that arrive after they have started verifying
the certificate chain. But they buffer 1-RTT packets, and therefore the
issue exists for 1-RTT packets.

As a result, there's potential delay, sometimes significant, in responding
to some packets during the handshake. There are two side effects of this.

S1.

First, this means that an endpoint that runs a PTO timer and retransmits
data for 0-RTT and 1-RTT packets while the handshake is still in progress
might be doing this quite unnecessarily. The peer may well have received
this data and have queued it, waiting to be able to decrypt it.

S2.

Second, acks for these packets that are buffered and then later decrypted
and read can be sent much later than when the packets were received at the
peer, causing the ack_delay to potentially be much higher than the
advertised max_ack_delay. The endpoint ignores ack_delay larger than
max_ack_delay, and this inflates the smoothed_rtt and rtt_variance,
potentially inflating the PTO significantly (which can cause handshakes to
fail if the endpoint employs a "handshake timeout" period).

Proposed Solutions:

S1.

To solve the first problem, we already have a condition in the recovery
draft to _not_ arm the PTO on 1-RTT data until the handshake is
complete (Section
6.2.1
<https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.1-6>).
This is not quite adequate, since it still allows a client to arm a PTO
timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that
the server has received the CFIN. The proposed solution here is to change
this condition from "handshake complete" to "handshake confirmed". Meaning
that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows
that the peer has the keys necessary to read and write in 1-RTT, and can
therefore respond with an ACK.

We think this holds up, but we'd like to get this under everyone's eyes, to
ensure that the discussion isn't missing subtle deadlock issues.

S2.

To solve the second problem, the proposal is to use ack_delay as reported
by the endpoint that buffers the data appropriately. ack_delay is the
amount of the time the information (regardless of that being the packet
itself or the acknowledgement) being buffered intentionally. For this, we
need the following:

- endpoints MUST report an ack_delay that is the sum of the controlled
delay (i.e. the duration of the data and the ack being buffered)

- endpoints ignore max_ack_delay for acknowledgements of packets that were
sent prior to handshake confirmation. Those are the ones that might have
been buffered at the peer until keys were ready.

In addition, we explicitly recommend endpoints that asynchronously process
handshake messages to buffer 1-RTT packets, instead of dropping them, as
dropping packets has significant impact on congestion control. Note that
this matches what those implementations already do.

Thoughts?

- jana

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 20 Jul 2020 10:10:51 +1000

Hi Jana,

Thanks for writing this up.  I think that this is a good fix, even a necessary one.

On Fri, Jul 17, 2020, at 19:00, Jana Iyengar wrote:
> We think this holds up, but we'd like to get this under everyone's 
> eyes, to ensure that the discussion isn't missing subtle deadlock 
> issues.

I have satisfied myself that this is OK.  The only 1-RTT (or 0-RTT) packet content we depend on to get to "handshake confirmed" is the HANDSHAKE_DONE frame.  As this is sent by the server after confirming the handshake, it will be sent on PTO as normal.  Were this not the case, this would be disastrous as we couldn't rely on this getting through.

> - endpoints ignore max_ack_delay for acknowledgements of packets that 
> were sent prior to handshake confirmation. Those are the ones that 
> might have been buffered at the peer until keys were ready.

For this, endpoints need to retain an additional value: the highest packet number that was sent prior to confirmation (or the lowest packet number sent after confirmation).  It's a little annoying to have to do that, but it seems necessary.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 20 Jul 2020 19:07:01 -0700

Thanks, Martin.

> I have satisfied myself that this is OK.  The only 1-RTT (or 0-RTT) packet
> content we depend on to get to "handshake confirmed" is the HANDSHAKE_DONE
> frame.  As this is sent by the server after confirming the handshake, it
> will be sent on PTO as normal.  Were this not the case, this would be
> disastrous as we couldn't rely on this getting through.
>

Yes, this is perhaps the most delicate part of this change. As you say, at
that point, the server is at "handshake confirmed", and will therefore arm
the PTO timer for the HANDSHAKE_DONE frame.

One way to think about this would be that the server cannot read 1-RTT
packets (including ACK frames) until it has read the CFIN. The client
retransmits this on a PTO, since this is in Handshake space and not in
Application space, until it gets an ACK for it or receives HANDSHAKE_DONE.
The server's Application PTO does not do any good until it has received the
CFIN. So the client drives the machinery at this point.

One receiving the CFIN, the server moves to "handshake confirmed" and can
now arm a PTO for the HANDSHAKE_DONE that it sends next. So now, the server
drives the machinery. Once HANDSHAKE_DONE is received by the client, it is
free to set PTO on subsequent 1-RTT packets it sends.

> For this, endpoints need to retain an additional value: the highest packet
> number that was sent prior to confirmation (or the lowest packet number
> sent after confirmation).  It's a little annoying to have to do that, but
> it seems necessary.
>

Agreed.

---

From: Marten Seemann <[email-masked]>
Date: Tue, 21 Jul 2020 12:07:45 +0700

> - endpoints ignore max_ack_delay for acknowledgements of packets that
were sent prior to handshake confirmation. Those are the ones that might
have been buffered at the peer until keys were ready.

It took me a while to convince myself that this is ok. The danger I saw
here is that an adversarial endpoint might misreport the ack_delay to trick
the peer into coming up with an RTT estimate that's significantly shorter
than the actual RTT.
The reason why this attack doesn't work is that the only variables that
make use of the ack_delay are smoothed_rtt and rttvar. Specifically,
latest_rtt is just the difference between sending the packet and receipt of
the acknowledgment, and min_rtt is the minimum value of all latest_rtt
values measured. The ack_delay is only applied to the measurement of
smoothed_rtt if it doesn't lower the value below min_rtt.
Consequently, this means that a server reporting an ack_delay in the ACK
for the ClientHello won't have any effect.

On Tue, Jul 21, 2020 at 9:07 AM Jana Iyengar <[email-masked]> wrote:

>> I have satisfied myself that this is OK.  The only 1-RTT (or 0-RTT)
>> packet content we depend on to get to "handshake confirmed" is the
>> HANDSHAKE_DONE frame.  As this is sent by the server after confirming the
>> handshake, it will be sent on PTO as normal.  Were this not the case, this
>> would be disastrous as we couldn't rely on this getting through.
>>
>
>> For this, endpoints need to retain an additional value: the highest
>> packet number that was sent prior to confirmation (or the lowest packet
>> number sent after confirmation).  It's a little annoying to have to do
>> that, but it seems necessary.
>>
>
> Agreed.
>
>

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 21 Jul 2020 12:38:08 -0700

On Mon, Jul 20, 2020 at 10:07 PM Marten Seemann <[email-masked]>
wrote:

> > To solve the second problem, the proposal is to use ack_delay as
> reported by the endpoint that buffers the data appropriately. ack_delay is
> the amount of the time the information (regardless of that being the packet

Yes, thanks for noting this, Marten. That protection is exactly the point
of using only observed values for min_rtt -- the peer cannot reduce that
value below what is observed.

---

From: Ian Swett <[email-masked]>
Date: Tue, 21 Jul 2020 18:19:19 -0400

I'm not in love with us relying so strongly on HANDSHAKE_DONE, but I think
it works as intended.

One suggestion I'd make is to keep the SHOULD around coalescing Handshake
and 1-RTT packets when PTO is armed, even if it's prior to handshake
confirmed.  That can be worked out on the PR.

On Tue, Jul 21, 2020 at 3:38 PM Jana Iyengar <[email-masked]> wrote:

>> delay (i.e.. the duration of the data and the ack being buffered)