Subject: tcp rsts, data truncation, and the future of public reset

From: Patrick McManus <[email-masked]>
Date: Mon, 20 Feb 2017 13:58:48 -0500

Gentlefolk of QUIC-wg,

I have recently been revisiting the pain of TCP RST as applied to the HTTP
ecosystem. The layering between HTTP and TCP largely ignores RST and as a
result this is a recurring implementation pain point. I think it bears
thinking about the QUIC design - the language around close and public reset
is not yet well developed.

As you likely know, a received TCP RST essentially nulls out any existing
connection state with a giant hammer. Sometimes this obliterates already
ack'd data in socket buffers before it makes it to the application to say
nothing of making retransmissions unreliable.

Unfortunately, in several HTTP usage patterns TCP RST happens a lot. This
leads to data truncation and information loss.

I could go on at length, but a few scenarios where this has resulted in
data loss are: h1 pipelines[1], h1 servers early-rejecting a large incoming
message body[2], and h2 servers not knowing when it is ok to close a
connection[3]

TCP reset is in practice unhelpful in an http context except as a refusal
to connect at all or as a signal that a peer has totally lost the state of
the connection (i.e. crash).

While I'm keenly aware this is an http centric view of the problem, It
seems this would be a good footgun to keep out of quic. Public reset is
sort of loosely defined right now so I want to give some thought to whether
we are recreating the problematic use cases.

6.4 says

*Abrupt Shutdown: An endpoint may send a Public Reset packet at any time
during the connection to abruptly terminate an active connection. A Public
Reset packet SHOULD only be used as a final recourse. Commonly, a public
reset is expected to be sent when a packet on an established connection is
received by an endpoint that is unable decrypt the packet. For instance, if
a server reboots mid-connection and loses any cryptographic state
associated with open connections, and then receives a packet on an open
connection, it should send a Public Reset packet in return. (TODO:
articulate rules around when a public reset should be sent.)*
I think that the primary use case described by the text (lack of
cryptographic state) is the only time a public reset should be used - the
text should MUST that imo.

First, if you've got cryptographic state - use it to send an in-connection
error that can be authenticated!

Second, when else would you want to RESET? TCP resets have been classically
generated in conjunction with FIN state (though it seems rather
implementation dependent for some cases of when this happens). Wouldn't a
retransmission of a CONNECTION_CLOSE (or transmission of a new
CONNECTION_CLOSE if need be) be a better response in pretty much every
circumstance where you have the state of the connection (including
encryption state) available?

The transport document also has a todo re: TIME_WAIT (now opened as issue
328). When do you go into TIME_WAIT - as soon as CONNECTON_CLOSE is sent or
when it is ack'd.. do retransmissions happen in that state? If nothing is
unacked and you receive data do you just ignore it, or do you send a PUBLIC
RST (I hope not - based on above) or another CONNECTION_CLOSE with an error
code or...??

Obviously TIME_WAIT has scaling issues and dos implications, etc..

Given that we expect QUIC to be used in userspace applications, it would
probably be good to have some text describing

Thanks for any clarifying discussion.

-Patrick

[1] a client might send N requests in a http/1.1 pipeline, the server might
serve a valid response for #1 and then close the connection (legit http
behavior). As there are more requests sitting in the server's socket read
buffers when it closes the connection, a RST is also generated. The client
needs to be prepared to replay requests 2-N as part of the complexity of
implementing pipelining, but the RST likely obliterates or truncates
response #1. This unpredictability is a key element of why pipelines could
not be widely deployed.

[2] a client sends 1GB of data in a POST. The server knows right away it
cannot deal with that - perhaps it needs authentication on the POST,
perhaps it is going to issue a redirect, perhaps it has a 404 with a fancy
error page. who knows. The point is nobody wants to wait or pay for the GB
of data transfer. The server sends a response and closes the connection.
This almost certainly generates a RST as there is data streaming in when it
does so. Data does stop flowing, but when the client receives the reset it
may destroy its ability to also see the error. So it basically looks like
connection refused rather than a well formed http response - if the
response demanded authentication for the POST then we've got a
bootstrapping deadlock.

[3] an h2 server, reacting to resource constraints, would like to close an
active connection but it has 1 stream currently transmitting a response. So
it sends a GOAWAY with an appropriate last-stream-id and then a few more
data frames (completing that stream) and then a connection close. That
seems 7540 compliant - but it might very well generate a RST  based on
WINDOW_UPDATEs or even other HEADERS that are arriving at the server. This
RST in turn may very well obliterate those DATA frames on the client if
they are still sitting in socket buffers when the RST arrives, or at the
very least will prevent them from being retransmitted if they were lost -
another form of data loss.

---

From: Patrick McManus <[email-masked]>
Date: Mon, 20 Feb 2017 16:47:39 -0500

> text should MUST that imo.

I've received a couple requests for clarification on this.

What I mean to say is that a Public Reset mechanism for use when you don't
have state/keying material (crash/reboot/long timeout) is imo fine. (this
is the case the existing text imagines as its primary use). But we should
limit the use of it to that case only rather than allowing its use "anytime
to abruptly terminate a connection".. I think its that abruptness when
applied to otherwise orderly TCP connections that has caused all the pain
(and un-necessary data loss) around TCP RST.

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 20 Feb 2017 14:17:46 -0800

> (and un-necessary data loss) around TCP RST.

​Yes, I completely agree. PUBLIC_RESET should only be used when the server
is unable to send an in-band CONNECTION_CLOSE. Specifically, this happens
when a server receives a packet for a connection ID it does not have state
for, which is neither a handshake (null-encrypted) or 0-RTT packet. In all
other circumstances, an in-band CONNECTION_CLOSE is preferable.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 20 Feb 2017 14:18:24 -0800

I agree as well.

On Mon, Feb 20, 2017 at 2:17 PM, Ryan Hamilton <[email-masked]> wrote:

>> What I mean to say is that a Public Reset mechanism for use when you
>> don't have state/keying material (crash/reboot/long timeout) is imo fine.
>> (this is the case the existing text imagines as its primary use). But we
>> should limit the use of it to that case only rather than allowing its use
>> "anytime to abruptly terminate a connection".. I think its that abruptness
>> when applied to otherwise orderly TCP connections that has caused all the
>> pain (and un-necessary data loss) around TCP RST.
>>
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 20 Feb 2017 23:25:31 +0100

+1

(We'd talked about overloading reset for exposing end of connection. Some exposure of CONNECTION_CLOSE in the public header for state management purposes would be useful, but that's an entirely separate issue, and should be solved in a separate way.)

Cheers,

Brian

> On 20 Feb 2017, at 23:17, Ryan Hamilton <[email-masked]> wrote:
> 
> 
> On Mon, Feb 20, 2017 at 1:47 PM, Patrick McManus <[email-masked]> wrote:
> On Mon, Feb 20, 2017 at 1:58 PM, Patrick McManus <[email-masked]> wrote:
> I think that the primary use case described by the text (lack of cryptographic state) is the only time a public reset should be used - the text should MUST that imo.
> 
> 
> What I mean to say is that a Public Reset mechanism for use when you don't have state/keying material (crash/reboot/long timeout) is imo fine. (this is the case the existing text imagines as its primary use). But we should limit the use of it to that case only rather than allowing its use "anytime to abruptly terminate a connection".. I think its that abruptness when applied to otherwise orderly TCP connections that has caused all the pain (and un-necessary data loss) around TCP RST.
> 
> ​Yes, I completely agree. PUBLIC_RESET should only be used when the server is unable to send an in-band CONNECTION_CLOSE. Specifically, this happens when a server receives a packet for a connection ID it does not have state for, which is neither a handshake (null-encrypted) or 0-RTT packet. In all other circumstances, an in-band CONNECTION_CLOSE is preferable.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 20 Feb 2017 14:35:36 -0800

Completely agree.

In the case of TCP, the stack simply follows the application directive
(shutdown() vs close()), and at the receiver there's no way to know whether
the applciation at the other end caused it or if the stack lost state. QUIC
can make this difference explicit by never emitting a PUBLIC_RESET except
when connection state is absent. That is indeed the goal of this signal,
and we should make it more explicit and concrete in the draft where
possible.

On the point of what to do in TIME_WAIT, the server should hang on to some
state. At the least, hanging on to the connection ID protects the server by
preventing late packets from causing creation of new connection state, and
the server can simply discard incoming packets for connection IDs in
TIME_WAIT. (If a server wanted to hang on to more state, it could do so,
allowing it to retransmit CONNECTION_CLOSE frames later, but as you note,
this is more state and I wouldn't require it.)

On Mon, Feb 20, 2017 at 2:18 PM, Eric Rescorla <[email-masked]> wrote:

>> ​Yes, I completely agree. PUBLIC_RESET should only be used when the
>> server is unable to send an in-band CONNECTION_CLOSE. Specifically, this
>> happens when a server receives a packet for a connection ID it does not
>> have state for, which is neither a handshake (null-encrypted) or 0-RTT
>> packet. In all other circumstances, an in-band CONNECTION_CLOSE is
>> preferable.
>>
>
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 21 Feb 2017 03:48:46 +0000

I also agree that other alternatives are strongly preferred to PUBLIC RESET. However, I am worried about making it a MUST in this case.

First, a MUST leaves no wiggle room for unforeseen circumstances (see below). Second, it is easy to circumvent this MUST by deliberately losing all connection state (or just crypto state) and then immediately sending PUBLIC RESET (because I no longer have the state).

We can probably find a way to always avoid PUBLIC RESET when designing application protocols, since QUIC still allows the application to Reset individual streams (and those resets are not subject to connection-level flow control). But what do you do about a poorly designed application protocol that cannot solve some problem with a neat stream reset (assume you cannot update the application protocol on both peers quickly)? Would you want to force applications not to use PUBLIC RESET as a last resort workaround?

- Igor

-----Original Message-----
From: Jana Iyengar [[email-masked]]
Received: Monday, 20 Feb 2017, 5:35PM
To: Eric Rescorla [[email-masked]]
CC: IETF QUIC WG [[email-masked]]; Patrick McManus [[email-masked]]; Ryan Hamilton [[email-masked]]
Subject: Re: tcp rsts, data truncation, and the future of public reset

Completely agree.

In the case of TCP, the stack simply follows the application directive (shutdown() vs close()), and at the receiver there's no way to know whether the applciation at the other end caused it or if the stack lost state. QUIC can make this difference explicit by never emitting a PUBLIC_RESET except when connection state is absent. That is indeed the goal of this signal, and we should make it more explicit and concrete in the draft where possible.

On the point of what to do in TIME_WAIT, the server should hang on to some state. At the least, hanging on to the connection ID protects the server by preventing late packets from causing creation of new connection state, and the server can simply discard incoming packets for connection IDs in TIME_WAIT. (If a server wanted to hang on to more state, it could do so, allowing it to retransmit CONNECTION_CLOSE frames later, but as you note, this is more state and I wouldn't require it.)

On Mon, Feb 20, 2017 at 2:18 PM, Eric Rescorla <[email-masked]<mailto:[email-masked]>> wrote:
I agree as well.

On Mon, Feb 20, 2017 at 2:17 PM, Ryan Hamilton <[email-masked]<mailto:[email-masked]>> wrote:

On Mon, Feb 20, 2017 at 1:47 PM, Patrick McManus <[email-masked]<mailto:[email-masked]>> wrote:
On Mon, Feb 20, 2017 at 1:58 PM, Patrick McManus <[email-masked]<mailto:[email-masked]>> wrote:
I think that the primary use case described by the text (lack of cryptographic state) is the only time a public reset should be used - the text should MUST that imo.

?Yes, I completely agree. PUBLIC_RESET should only be used when the server is unable to send an in-band CONNECTION_CLOSE. Specifically, this happens when a server receives a packet for a connection ID it does not have state for, which is neither a handshake (null-encrypted) or 0-RTT packet. In all other circumstances, an in-band CONNECTION_CLOSE is preferable.

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 20 Feb 2017 21:31:37 -0800

On Mon, Feb 20, 2017 at 7:48 PM, Lubashev, Igor <[email-masked]> wrote:

> I also agree that other alternatives are strongly preferred to PUBLIC
> RESET. However, I am worried about making it a MUST in this case.
>
> First, a MUST leaves no wiggle room for unforeseen circumstances (see
> below). Second, it is easy to circumvent this MUST by deliberately losing
> all connection state (or just crypto state) and then immediately sending
> PUBLIC RESET (because I no longer have the state).
>
> We can probably find a way to always avoid PUBLIC RESET when designing
> application protocols, since QUIC still allows the application to Reset
> individual streams (and those resets are not subject to connection-level
> flow control). But what do you do about a poorly designed application
> protocol that cannot solve some problem with a neat stream reset (assume
> you cannot update the application protocol on both peers quickly)? Would
> you want to force applications not to use PUBLIC RESET as a last resort
> workaround?

​I don't think I understand how the use of CONNECTION_CLOSE is any
different to an application than PUBLIC_RESET? Under what circumstances
would an application protocol (admittedly a poorly designed one) work if a
PUBLIC_RESET were used, but would not work if a CONNECTION_CLOSED were?​

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 21 Feb 2017 11:24:08 +0000

If CONNECTION_CLOSE is like TCP FIN (an implicit FIN for all streams plus a GOAWAY), it implies that this side will continue to re-transit frames in lost packets and peer will deliver data to the application. Or is CONNECTION_CLOSE more like an implicit Reset for all streams plus a GOAWAY?

- Igor

-----Original Message-----
From: Ryan Hamilton [[email-masked]]
Received: Tuesday, 21 Feb 2017, 12:31AM
To: Lubashev, Igor [[email-masked]]
CC: [email-masked] [[email-masked]]; [email-masked] [[email-masked]]; [email-masked] [[email-masked]]; [email-masked] [[email-masked]]
Subject: Re: tcp rsts, data truncation, and the future of public reset

On Mon, Feb 20, 2017 at 7:48 PM, Lubashev, Igor <[email-masked]<mailto:[email-masked]>> wrote:
I also agree that other alternatives are strongly preferred to PUBLIC RESET. However, I am worried about making it a MUST in this case.

?I don't think I understand how the use of CONNECTION_CLOSE is any different to an application than PUBLIC_RESET? Under what circumstances would an application protocol (admittedly a poorly designed one) work if a PUBLIC_RESET were used, but would not work if a CONNECTION_CLOSED were??

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 21 Feb 2017 06:53:11 -0800

A CONNECTION_CLOSE immediately closes the connection. No data is
retransmitted. It is not equivalent to delivering a FIN on all open
streams. I would say that the equivalent of a TCP fin (which is the
graceful shutdown of one half of a bidirectional communication) is a stream
FIN. The CONNECTION_CLOSE sounds more like a TCP reset.

On Tue, Feb 21, 2017 at 3:24 AM, Lubashev, Igor <[email-masked]> wrote:

> If CONNECTION_CLOSE is like TCP FIN (an implicit FIN for all streams plus
> a GOAWAY), it implies that this side will continue to re-transit frames in
> lost packets and peer will deliver data to the application. Or is
> CONNECTION_CLOSE more like an implicit Reset for all streams plus a GOAWAY?
>
> - Igor
>
> *From:* Ryan Hamilton [[email-masked]]
> *Received:* Tuesday, 21 Feb 2017, 12:31AM
> *To:* Lubashev, Igor [[email-masked]]
> *CC:* [email-masked] [[email-masked]]; [email-masked] [[email-masked]];
> [email-masked] [[email-masked]]; [email-masked] [[email-masked]]
> *Subject:* Re: tcp rsts, data truncation, and the future of public reset
>
>

---

From: Patrick McManus <[email-masked]>
Date: Tue, 21 Feb 2017 10:13:57 -0500

> FIN. The CONNECTION_CLOSE sounds more like a TCP reset.
would you open an issue to add this type of language to the spec
(potentially in both the transport and loss drafts)? its really not defined
afaict.. as it would alternatively make sense to retransmit unacked stuff
after CLOSE - but then it wouldn't be behaving like reset (which honestly I
would rather it didn't behave like reset - reset is leads to pain.).

If the only difference between public_reset and connection_close is in the
auth (and I guess the retransmitting of close as necessary?) perhaps they
should share a name: public_connection_close or private_reset or somesuch?

---

From: Martin Duke <[email-masked]>
Date: Tue, 21 Feb 2017 08:23:58 -0800

> this is more state and I wouldn't require it.)

I don't understand this point. If we discarded the connection ID
immediately, wouldn't that just result in decryption failure for subsequent
packets and a public reset? There is no new state.

---

From: Martin Duke <[email-masked]>
Date: Tue, 21 Feb 2017 08:49:18 -0800

Never mind, no Public Reset on decryption failure. My mistake.

On Tue, Feb 21, 2017 at 8:23 AM, Martin Duke <[email-masked]>
wrote:

>> On the point of what to do in TIME_WAIT, the server should hang on to
>> some state. At the least, hanging on to the connection ID protects the
>> server by preventing late packets from causing creation of new connection
>> state, and the server can simply discard incoming packets for connection
>> IDs in TIME_WAIT. (If a server wanted to hang on to more state, it could do
>> so, allowing it to retransmit CONNECTION_CLOSE frames later, but as you
>> note, this is more state and I wouldn't require it.)
>>
>

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 21 Feb 2017 12:02:35 -0800

On Tue, Feb 21, 2017 at 7:13 AM, Patrick McManus <[email-masked]>
wrote:

> afaict..

​Sure. I filed https://github.com/quicwg/base-drafts/issues/330.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 21 Feb 2017 20:19:55 +0000

The streams mean that you have a two-layer close to the connection, which makes a graceful shutdown more complex.  As I understand it, the “clean” connection close flow is:

  *   GOAWAY – no new streams can open (implicit RST of streams that were racing the GOAWAY)
  *   Service and FIN (or RST as needed) all existing streams
  *   CONNECTION_CLOSE once all streams have closed and all data has been ACK’d

Graceless shutdown is simple as ever.  😊  If you skip straight to the CONNECTION_CLOSE, that’s effectively a RST of every stream that’s still open when you get there.

It sounds like what you’re concerned about is the window after sending all stream data.  If you transmit the CONNECTION_CLOSE too early, you risk losing data in the final flight, either through loss or reordering of the CONNECTION_CLOSE ahead of data packets.

It seems like the obvious API semantics would be similar to TCP’s – one option would GOAWAY, wait, send the CONNECTION_CLOSE, and finally return.  The other would send the CONNECTION_CLOSE immediately, and data falls where it may.  However, that invites the same application behavior you see with TCP today – the application doesn’t want to sit around, and will call the faster API if it has gotten everything it needs off the connection.

From: QUIC [mailto:[email-masked]] On Behalf Of Patrick McManus
Sent: Tuesday, February 21, 2017 7:14 AM
To: Ryan Hamilton <[email-masked]>
Cc: [email-masked]; [email-masked]; Lubashev, Igor <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: tcp rsts, data truncation, and the future of public reset

On Tue, Feb 21, 2017 at 9:53 AM, Ryan Hamilton <[email-masked]<mailto:[email-masked]>> wrote:
A CONNECTION_CLOSE immediately closes the connection. No data is retransmitted. It is not equivalent to delivering a FIN on all open streams. I would say that the equivalent of a TCP fin (which is the graceful shutdown of one half of a bidirectional communication) is a stream FIN. The CONNECTION_CLOSE sounds more like a TCP reset.

would you open an issue to add this type of language to the spec (potentially in both the transport and loss drafts)? its really not defined afaict.. as it would alternatively make sense to retransmit unacked stuff after CLOSE - but then it wouldn't be behaving like reset (which honestly I would rather it didn't behave like reset - reset is leads to pain.).
If the only difference between public_reset and connection_close is in the auth (and I guess the retransmitting of close as necessary?) perhaps they should share a name: public_connection_close or private_reset or somesuch?

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 21 Feb 2017 12:50:42 -0800

On Tue, Feb 21, 2017 at 12:19 PM, Mike Bishop <[email-masked]>
wrote:

> The streams mean that you have a two-layer close to the connection, which
> makes a graceful shutdown more complex.  As I understand it, the “clean”
> connection close flow is:
>
>    - GOAWAY – no new streams can open (implicit RST of streams that were
>    racing the GOAWAY)
>    - Service and FIN (or RST as needed) all existing streams
>    - CONNECTION_CLOSE once all streams have closed and all data has been
>    ACK’d
>
> ​*nod*
​

> Graceless shutdown is simple as ever.  😊  If you skip straight to the
> CONNECTION_CLOSE, that’s effectively a RST of every stream that’s still
> open when you get there.
>

Agreed. This makes sense to me and seems like the right way to think about
things.
​

> It sounds like what you’re concerned about is the window after sending all
> stream data.  If you transmit the CONNECTION_CLOSE too early, you risk
> losing data in the final flight, either through loss or reordering of the
> CONNECTION_CLOSE ahead of data packets.
>

​Since QUIC connections are multiplexed, it is common to keep them around
for some period of idle in expectation of new streams being created. As a
result, it's common to only send the CONNECTION_CLOSE after that period
expires. (Or if there is some sort of fatal error on the connection.)
​

>  It seems like the obvious API semantics would be similar to TCP’s – one
> option would GOAWAY, wait, send the CONNECTION_CLOSE, and finally return.
> The other would send the CONNECTION_CLOSE immediately, and data falls where
> it may.  However, that invites the same application behavior you see with
> TCP today – the application doesn’t want to sit around, and will call the
> faster API if it has gotten everything it needs off the connection.
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Patrick McManus
> *Sent:* Tuesday, February 21, 2017 7:14 AM
> *To:* Ryan Hamilton <[email-masked]>
> *Cc:* [email-masked]; [email-masked]; Lubashev, Igor <[email-masked]>;
> [email-masked]; [email-masked]

---

From: Martin Duke <[email-masked]>
Date: Tue, 21 Feb 2017 12:56:47 -0800

The draft could really use a clear description of a clean shutdown, similar
to what Mike described. However:

GOAWAY means that I will neither initiate streams nor accept new streams.
So how am I to know that the other side is done initiating streams? It
would be better if the GOAWAY were a sort of one-way connection FIN.

On Tue, Feb 21, 2017 at 12:50 PM, Ryan Hamilton <[email-masked]> wrote:

>
>
> On Tue, Feb 21, 2017 at 12:19 PM, Mike Bishop <
> [email-masked]> wrote:
>
>> It sounds like what you’re concerned about is the window after sending
>> all stream data.  If you transmit the CONNECTION_CLOSE too early, you risk
>> If the only difference between public_reset and connection_close is in
>> the auth (and I guess the retransmitting of close as necessary?) perhaps
>> they should share a name: public_connection_close or private_reset or
>> somesuch?
>>
>>
>>
>>
>>
>
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 21 Feb 2017 14:02:14 -0800

On 2/21/2017 12:56 PM, Martin Duke wrote:
> The draft could really use a clear description of a clean shutdown,
> similar to what Mike described. However:
>
> GOAWAY means that I will neither initiate streams nor accept new
> streams. So how am I to know that the other side is done initiating
> streams? It would be better if the GOAWAY were a sort of one-way
> connection FIN.
>

What should happen if a single packet contains the Connection Close
Frame and also a Stop Waiting frame with a delta = 0?

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 21 Feb 2017 22:59:55 +0000

> As I understand it, the “clean” connection close flow is:

I do not think this is sufficient. You need to talk about the the other side's semantics on receipt of CONNECTION_CLOSE. In particular, is the receiver of CONNECTION_CLOSE allowed to drop the connection state immediately? Or is it required to deliver ACKed data to the application first? If it is required to deliver ACKed data, is it all data or only data from properly closed streams (i.e. excluding streams reset due to STREAM_RST or CONNECTION_CLOSE).

My preference for requiring the delivery of all data belonging to properly closed streams and a SHOULD NOT for the delivery of any other data not yet delivered.

- Igor

-----Original Message-----
From: Mike Bishop [[email-masked]]
Received: Tuesday, 21 Feb 2017, 3:20PM
To: Patrick McManus [[email-masked]]; Ryan Hamilton [[email-masked]]
CC: [email-masked] [[email-masked]]; [email-masked] [[email-masked]]; Lubashev, Igor [[email-masked]]; [email-masked] [[email-masked]]
Subject: RE: tcp rsts, data truncation, and the future of public reset

Graceless shutdown is simple as ever.  ??  If you skip straight to the CONNECTION_CLOSE, that’s effectively a RST of every stream that’s still open when you get there.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 22 Feb 2017 15:10:59 +1100

> would be better if the GOAWAY were a sort of one-way connection FIN.

If you do as we allow in HTTP/2, GOAWAY can be sent many times.  GOAWAY
with a high number allows you to continue creating streams, but puts the
other side on notice of imminent shutdown.  Sending again with a lower
number promises that streams below that number are OK to complete, but
higher numbers are all reset.  And you can't sent with a higher number.  At
some point, you are either tired of waiting for streams to close, or they
are all closed.  Then send CONNECTION_CLOSE.

On 22 February 2017 at 09:59, Lubashev, Igor <[email-masked]> wrote:

> I do not think this is sufficient. You need to talk about the the other
> side's semantics on receipt of CONNECTION_CLOSE. In particular, is the
> receiver of CONNECTION_CLOSE allowed to drop the connection state
> immediately? Or is it required to deliver ACKed data to the application
> first? If it is required to deliver ACKed data, is it all data or only data
> from properly closed streams (i.e. excluding streams reset due to
> STREAM_RST or CONNECTION_CLOSE).

Well, if there is anything outstanding on a stream, that's a hard cut on
those things, throw them away.

Unlike TCP, which sometimes throws state away on a RST, I would say that an
implementation MUST allow completed streams that weren't delivered to
complete.  Any local state costs can continue to accrue to the application.

I don't know what the relevance of ACKed is when talking about received
data, you don't send an ACK for your own benefit.

My preference for requiring the delivery of all data belonging to properly
> closed streams and a SHOULD NOT for the delivery of any other data not yet
> delivered.

That's a MUST and a SHOULD NOT, which sounds fine.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 22 Feb 2017 16:46:09 +0000

Ø  My preference for requiring the delivery of all data belonging to properly closed streams and a SHOULD NOT for the delivery of any other data not yet delivered.

Great, I am glad you see it the same way about CONNECTION_CLOSE. The next question would be whether the same can be said for PUBLIC_RESET as well. If we really want to keep the two identical, it makes sense to require the same semantics on them. Any counter-examples?

Ø  I don't know what the relevance of ACKed is when talking about received data

You are correct. ACKs do not matter here.

-          Igor

From: Martin Thomson [mailto:[email-masked]]
Sent: Tuesday, February 21, 2017 11:11 PM
To: Martin Duke <[email-masked]>
Cc: Ryan Hamilton <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; Lubashev, Igor <[email-masked]>; Patrick McManus <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: tcp rsts, data truncation, and the future of public reset

On 22 February 2017 at 07:56, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
GOAWAY means that I will neither initiate streams nor accept new streams. So how am I to know that the other side is done initiating streams? It would be better if the GOAWAY were a sort of one-way connection FIN.

If you do as we allow in HTTP/2, GOAWAY can be sent many times.  GOAWAY with a high number allows you to continue creating streams, but puts the other side on notice of imminent shutdown.  Sending again with a lower number promises that streams below that number are OK to complete, but higher numbers are all reset.  And you can't sent with a higher number.  At some point, you are either tired of waiting for streams to close, or they are all closed.  Then send CONNECTION_CLOSE.
On 22 February 2017 at 09:59, Lubashev, Igor <[email-masked]<mailto:[email-masked]>> wrote:
I do not think this is sufficient. You need to talk about the the other side's semantics on receipt of CONNECTION_CLOSE. In particular, is the receiver of CONNECTION_CLOSE allowed to drop the connection state immediately? Or is it required to deliver ACKed data to the application first? If it is required to deliver ACKed data, is it all data or only data from properly closed streams (i.e. excluding streams reset due to STREAM_RST or CONNECTION_CLOSE).

Well, if there is anything outstanding on a stream, that's a hard cut on those things, throw them away.

Unlike TCP, which sometimes throws state away on a RST, I would say that an implementation MUST allow completed streams that weren't delivered to complete.  Any local state costs can continue to accrue to the application.

I don't know what the relevance of ACKed is when talking about received data, you don't send an ACK for your own benefit.
My preference for requiring the delivery of all data belonging to properly closed streams and a SHOULD NOT for the delivery of any other data not yet delivered.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Feb 2017 07:56:19 +1100

On 23 February 2017 at 03:46, Lubashev, Igor <[email-masked]> wrote:
> Great, I am glad you see it the same way about CONNECTION_CLOSE. The next
> question would be whether the same can be said for PUBLIC_RESET as well. If
> we really want to keep the two identical, it makes sense to require the same
> semantics on them. Any counter-examples?

Yes, there is a big difference, though we won't see it just yet.
Absent a connection-level cryptographic payload that is successfully
authenticated, Public Reset needs to kill a path, not an entire
connection.

---

From: Martin Duke <[email-masked]>
Date: Wed, 22 Feb 2017 13:56:37 -0800

I think the positions this are scattered about enough -- I have some
heretical ones not even ready for prime time -- that this probably should
be discussed in Chicago or Paris.Can someone open in issue?

---

From: Patrick McManus <[email-masked]>
Date: Wed, 22 Feb 2017 17:15:28 -0500

> Frame and also a Stop Waiting frame with a delta = 0?

my recollection from tokyo is that stop_waiting is being removed
https://github.com/quicwg/base-drafts/issues/66 is marked editor-ready

---

From: Patrick McManus <[email-masked]>
Date: Wed, 22 Feb 2017 17:54:01 -0500

Is this where we are?

Upon receipt of CLOSE/RESET an application MUST NOT abandon in-order queued
data for the application layer (whether or not it has a FIN for the stream
it is on) the way TCP currently does. But it SHOULD(?) drop future packets
received on the connection and MUST NOT send any on more on the connection.

CLOSE/RESET has the potential for data loss for any data whose ack has not
been received when CLOSE/RESET is generated. [I really think something like
that would be clarifying text in the document]

imo CLOSE ought to be named RESET.. generically close isn't normally a
destructive operation, while the semantics here certainly might be.

On Wed, Feb 22, 2017 at 11:46 AM, Lubashev, Igor <[email-masked]>
wrote:

> Ø  My preference for requiring the delivery of all data belonging to
> properly closed streams and a SHOULD NOT for the delivery of any other data
> not yet delivered.
>
>
>
> we really want to keep the two identical, it makes sense to require the
> same semantics on them. Any counter-examples?
>
>
>
>
>
> *From:* Martin Thomson [mailto:[email-masked]]
> *Sent:* Tuesday, February 21, 2017 11:11 PM
> *To:* Martin Duke <[email-masked]>
> *Cc:* Ryan Hamilton <[email-masked]>; Mike Bishop <
> [email-masked]>; [email-masked]; Lubashev, Igor <
> [email-masked]>; Patrick McManus <[email-masked]>;
> [email-masked]; [email-masked]
> Unlike TCP, which sometimes throws state away on a RST, I would say that
> an implementation MUST allow completed streams that weren't delivered to

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Feb 2017 10:05:01 +1100

> that would be clarifying text in the document]

I think that naturally follows.  Of course, the received might have
received data and therefore could have acted upon it.  That means that
unacknowledged data is in an ambiguous state.  Hopefully the graceful
close happens often enough that this isn't a common situation.

> destructive operation, while the semantics here certainly might be.

Reset implies an abort, which isn't really the case for a graceful
close.  Perhaps TERMINATE has the right connotations.

(I would also like to change RST_STREAM to CANCEL_STREAM and a few
other cosmetic changes.)

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 23 Feb 2017 00:01:49 +0000

On 23 February 2017 at 09:54, Patrick McManus <[email-masked]> wrote:
> Upon receipt of CLOSE/RESET an application MUST NOT abandon in-order 
> queued data for the application layer (whether or not it has a FIN for 
> the stream it is on)

I am not loving this.  I think it very much matters whether FIN has been received on the stream AND all data has been received on the stream prior to the CONNECTION_CLOSE/PUBLIC_RESET.  In my view, only such data MUST be delivered, and all other data (no FIN or not all data received) SHOULD NOT be delivered.

Doing anything else has two problems:
1) A "MUST" for any other condition does not provide guarantees to the sender what data the application will process, unless the sender is very careful (too error prone). If you've sent all your data and have seen all that data ACKed, just send FIN in the same packet as CONNECTION_CLOSED.

2a) If STREAM_RST requires a peer to abandon in-order queued data w/o FIN, the semantics of PUBLIC_RESET vs STREAM_RST are weird, as PUBLIC_RESET resets less state than STREAM_RST.

2b) If STREAM_RST also requires a peer to deliver in-order queued data w/o FIN, then there is no way to ask the transport to abandon its non-delivered data.  Having that data dropped by transport may be a desired feature for various applications, including transactional systems that may wish to abort an in-flight transaction (and therefore do not want the peer application to process all the buffered data just to throw it away later).

- Igor

P.S.

It seems that an orderly termination of a connection should look like:
1. GOAWAY 
2. Reset all incoming streams
3. Finish writing all streams, including the FIN.
4. Wait for all ACKs (and retransmit as needed)
5. CONNECTION_CLOSE

P.P.S.

Some cosmetic renaming is warranted, indeed.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Wednesday, February 22, 2017 6:05 PM
To: Patrick McManus <[email-masked]>
Cc: Lubashev, Igor <[email-masked]>; Martin Duke <[email-masked]>; Ryan Hamilton <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: tcp rsts, data truncation, and the future of public reset

On 23 February 2017 at 09:54, Patrick McManus <[email-masked]> wrote:
> Upon receipt of CLOSE/RESET an application MUST NOT abandon in-order 
> queued data for the application layer (whether or not it has a FIN for 
> the stream it is on) the way TCP currently does. But it SHOULD(?) drop 
> future packets received on the connection and MUST NOT send any on more on the connection.
>
> CLOSE/RESET has the potential for data loss for any data whose ack has 
> not been received when CLOSE/RESET is generated. [I really think 
> something like that would be clarifying text in the document]

I think that naturally follows.  Of course, the received might have received data and therefore could have acted upon it.  That means that unacknowledged data is in an ambiguous state.  Hopefully the graceful close happens often enough that this isn't a common situation.

> destructive operation, while the semantics here certainly might be.

Reset implies an abort, which isn't really the case for a graceful close.  Perhaps TERMINATE has the right connotations.

(I would also like to change RST_STREAM to CANCEL_STREAM and a few other cosmetic changes.)

---

From: Patrick McManus <[email-masked]>
Date: Wed, 22 Feb 2017 19:12:17 -0500

On Wed, Feb 22, 2017 at 6:05 PM, Martin Thomson <[email-masked]>
wrote:

> think that naturally follows.  Of course, the received might have
> unacknowledged data is in an ambiguous state.
>

indeed - but that's the state of things as soon as it is transmitted. no
takebacks as my 7 yr old says.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 22 Feb 2017 19:26:19 -0500

On Wed, Feb 22, 2017 at 7:01 PM, Lubashev, Igor <[email-masked]> wrote:

> I am not loving this.  I think it very much matters whether FIN has been
> received on the stream AND all data has been received on the stream prior
> to the CONNECTION_CLOSE/PUBLIC_RESET.  In my view, only such data MUST be
> delivered, and all other data (no FIN or not all data received) SHOULD NOT
> be delivered.
>
> If we do it the way I suggested then we get the nice property that no data
is lost for which an ack has been received before generating a
connection_close (modulo crash etc..) - which is a major improvement over
classic TCP which routinely loses data in surprising ways for application
developers (see endnotes in the first message in this thread).

Doing it the other way (requiring the FIN) just creates ambiguity and race
conditions about whether the data has already been streamed to the
application. It doesn't create a property you can rely on. I don't think we
should have a mechanism designed to "take back" already transmitted data -
an application can create that semantic through an application specific
signal on another stream if need be, it doesn't need to overload the
shutdown behavior.

 I would hope STREAM_RST would match. (I can't find any text right now on
the topic..)

> 4. Wait for all ACKs (and retransmit as needed)
basically agree, but you don't need #2 because of the last-good-stream-id
param of goaway. the rsts are implied.

I think TERMINATE sounds like a lovely shade of bikeshed.

---

From: Mike Bishop <[email-masked]>
Date: Thu, 23 Feb 2017 00:26:54 +0000

If we move to unidirectional resets, I'll note that you can completely ignore a RST_STREAM that arrives after the stream has already closed (i.e. all data and FIN).  The semantics of a RST_STREAM would be "I will no longer attempt to retransmit data on this stream; if you haven't received it, you're not going to."  An implementation could (and perhaps SHOULD) still deliver all data if the full sequence was received and assembled.  That would have the advantage of making that rule consistent.

However, the stream in the return direction might also have been closed, e.g. by a REQUEST_RST or a connection-level action.  The application layer, particularly a server, would need to check return-stream liveness before acting on a request, in this case.

-----Original Message-----
From: Lubashev, Igor [mailto:[email-masked]] 
Sent: Wednesday, February 22, 2017 4:02 PM
To: Martin Thomson <[email-masked]>; Patrick McManus <[email-masked]>
Cc: Martin Duke <[email-masked]>; Ryan Hamilton <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: tcp rsts, data truncation, and the future of public reset

- Igor

P.S.

It seems that an orderly termination of a connection should look like:
1. GOAWAY
2. Reset all incoming streams
3. Finish writing all streams, including the FIN.
4. Wait for all ACKs (and retransmit as needed) 5. CONNECTION_CLOSE

P.P.S.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Feb 2017 11:34:54 +1100

> developers (see endnotes in the first message in this thread).

I think that this is probably the right answer.  An application
protocol might decide that it doesn't want this feature and then
subsequently discard partially received streams, but we already have
the case in HTTP where data is progressively processed.  If a stream
carrying HTTP is reset in the middle, the data that has arrived might
still be delivered (even used).

If we model CONNECTION_CLOSE as a blanket reset on all open streams,
we have a nice basis upon which to model the situation.  As Mike says,
we can say that any received data gets delivered.  We also have a
single way to consider how this interacts.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 23 Feb 2017 02:35:08 +0000

---------- From: Mike Bishop [mailto:[email-masked]] ----------
> If we move to unidirectional resets, I'll note that you can completely ignore a RST_STREAM that arrives after the stream has already closed (i.e. all data and FIN).

A good point that should be made explicit.  (This is orthogonal to "require/do not require FIN".)

> The semantics of a RST_STREAM would be "I will no longer attempt to retransmit data on this stream; if you haven't received it, you're not going to."

"if you haven't received it, you're not going to" is not technically correct, since packets containing data and RST_STREAM could had been reordered.

---------- From: Martin Thomson [mailto:[email-masked]]  -----------
> we already have the case in HTTP where data is progressively processed.  If a stream carrying HTTP is reset in the middle, the data that has arrived might still be delivered (even used).

This is an interesting case that "require FIN and all data" cannot address fully.  It seems that a much more common case is that the application will not want to process an incomplete stream, though. But your case is still compelling.

---------- From: Patrick McManus [mailto:[email-masked]] ----------
> Doing it the other way (requiring the FIN) just creates ambiguity and race conditions about whether the data has already been streamed to the application. It doesn't create a property you can rely on.

Hm. The entire point was to avoid ambiguity and false assumptions for the application layer (not protocol layer).  The rule was to be "properly and timely close you streams if you want to be sure that every single byte is received".  No rule (whether requiring FIN or not requiring FIN) can possibly create a property you can rely on to tell you "whether the data has already been streamed to the application."  Receiving ACKs from transport definitely does not guarantee that transport already delivered anything to the application.  You would need some application logic for this.

The "not requiring FIN" rule, however, may indeed make additional guarantees that the transport "will" try to deliver certain bytes to the application at some point, if all ACKs have been received.

> I don't think we should have a mechanism designed to "take back" already transmitted data

It was supposed to be a hint to the transport; transport cannot undeliver anything, of course.

> an application can create that semantic through an application specific signal on another stream if need be, it doesn't need to overload the shutdown behavior.

This is possible, although cumbersome.  It would require "priority streams" (streams that are not subject to connection-wide flow control).  But we already require such streams for HTTP over QUIC, so that is not an undue burden.

If this is the way we decide to go "do not require FIN" route, I may suggest that QUIC libraries provides a way to choose "require FIN and all data" RST behavior per-stream (even if it is not the default).  Think sockopt.  This would likely be a subject of a different spec, though.

> basically agree, but you don't need #2 because of the last-good-stream-id param of goaway. the rsts are implied.

GOAWAY description implies that I cannot implicitly close streams that I've explicitly accepted and ACKed before.  If my endpoint does not want to hear anything new over the incoming streams, it should rst them.  Or do we want to allow such implicit RST of streams via GOAWAY with a low stream ID? Seems hacky but possibly useful.

- Igor