Subject: QUIC Connection Close Reliability

From: Nick Banks <[email-masked]>
Date: Mon, 23 Apr 2018 22:06:39 +0000

Hey Folks,

I have been trying to work through an issue with reliability, related to connection close and I am trying to understand if this is a protocol issue for if I need to change my implementation.

Suppose you have an application protocol on top of QUIC where each side of the connection may use one or more streams and then immediately, but gracefully, close the connection when they are done with all the streams. In that kind of application protocol, one side of the connection generally ends up closing immediately in response to receiving a FIN on a stream. Then you get into a state where the peer won't necessarily get that FIN acknowledged. Even if the endpoint sends an acknowledgment for the FIN (either as a separate packet or with the close) it's possible that ACK is lost. Ideally (I think), the ACK is sent in the same packet as the close, so you can control the atomic state change on the peer. If the ACK was separate, and lost, but the close wasn't lost, then the peer would treat the stream as abortively closed.

A -------- STREAM (w/FIN) ------> B
A <------- ACK, APP_CLOSE ------- B

But now the issue comes if the ACK/APP_CLOSE packet is lost. Per spec<https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#immediate-close>, the endpoint (B) that received the FIN and then closed the connection enters the closing period. While in this state, the spec states that the endpoint SHOULD respond to any packet it receives [without a corresponding close frame] with another packet containing a closing frame. If the endpoint does this, and includes the ACK again as well, I believe everything would work. The peer (A) would retransmit the STREAM packet, which would elicit a new ACK/APP_CLOSE packet from (B). My fear though (from current interop experience) is that everyone will try to short circuit connection closure and just immediately go away. If this behavior isn't completely standard across all implementations, then the application protocol would end up having different results on different implementations. Some implementations, might not send the ACK at all with the initial close, causing the peer (A) to most of the time interpret the stream as aborted. Other times, the endpoint (A) might not continue to send the ACK frame with the close, or just not send anything any more at all, resulting in a timeout on the peer side, also resulting in an abortive closure of the stream. This would probably cause the application protocol to end up creating it's own close semantics, on top of QUIC.

Now, folks might not think this is a very important scenario, but I feel we should aim for consistency here. Perhaps I have missed a different solution to the problem. I considered a design that always sends that last ACK with a PING frame, and waits for that to get acknowledged before closing, but I felt it was way too ugly.

Thanks,
- Nick

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Apr 2018 12:14:00 +0200

Hi Nick,

It seems that you are having a problem with getting agreement between
peers about what the close state is.  In your example, if A cares that
the stream was completely received by B, then this is not an
appropriate way to negotiate a close in that application protocol.

You might interpret the receipt of APPLICATION_CLOSE as implicit
acknowledgment of all streams to avoid this particular problem, or to
create additional signals.

HTTP over QUIC does this largely by driving this from the client end,
which simplifies things considerably.

On Tue, Apr 24, 2018 at 12:06 AM, Nick Banks
<nibanks=[email-masked]> wrote:
> Hey Folks,
>
>
>
> I have been trying to work through an issue with reliability, related to
> connection close and I am trying to understand if this is a protocol issue
> for if I need to change my implementation.
>
>
>
> Suppose you have an application protocol on top of QUIC where each side of
> the connection may use one or more streams and then immediately, but
> gracefully, close the connection when they are done with all the streams. In
> that kind of application protocol, one side of the connection generally ends
> up closing immediately in response to receiving a FIN on a stream. Then you
> get into a state where the peer won’t necessarily get that FIN acknowledged.
> Even if the endpoint sends an acknowledgment for the FIN (either as a
> separate packet or with the close) it’s possible that ACK is lost. Ideally
> (I think), the ACK is sent in the same packet as the close, so you can
> control the atomic state change on the peer. If the ACK was separate, and
> lost, but the close wasn’t lost, then the peer would treat the stream as
> abortively closed.
>
>
>
> But now the issue comes if the ACK/APP_CLOSE packet is lost. Per spec, the
> endpoint (B) that received the FIN and then closed the connection enters the
> closing period. While in this state, the spec states that the endpoint
> SHOULD respond to any packet it receives [without a corresponding close
> frame] with another packet containing a closing frame. If the endpoint does
> this, and includes the ACK again as well, I believe everything would work.
> The peer (A) would retransmit the STREAM packet, which would elicit a new
> ACK/APP_CLOSE packet from (B). My fear though (from current interop
> experience) is that everyone will try to short circuit connection closure
> and just immediately go away. If this behavior isn’t completely standard
> across all implementations, then the application protocol would end up
> having different results on different implementations. Some implementations,
> might not send the ACK at all with the initial close, causing the peer (A)
> to most of the time interpret the stream as aborted. Other times, the
> endpoint (A) might not continue to send the ACK frame with the close, or
> just not send anything any more at all, resulting in a timeout on the peer
> side, also resulting in an abortive closure of the stream. This would
> probably cause the application protocol to end up creating it’s own close
> semantics, on top of QUIC.
>
>
>
> Now, folks might not think this is a very important scenario, but I feel we
> should aim for consistency here. Perhaps I have missed a different solution
> to the problem. I considered a design that always sends that last ACK with a
> PING frame, and waits for that to get acknowledged before closing, but I
> felt it was way too ugly.
>
>
>
> Thanks,
>
> - Nick

---

From: Ian Swett <[email-masked]>
Date: Wed, 25 Apr 2018 13:05:15 +0000

Given the APP_CLOSE is closing the connection, I don't understand why the
ACK matters?  Once the connection is closed, the state is torn down and any
unacknowledged data is unnecessary.  I assume Martin's interpretation is
correct and this is a case where A wants to know the data was delivered,
and if you're immediately closing on fin receipt in your application,
Martin's suggestion that the APP_CLOSE implicitly acknowledges the data is
the obvious way to go.

However, I would expect most applications to have an idle period between
the last closed stream and when they close the connection, which ensures
the majority of connections are closed without unacknowledged data.

On Wed, Apr 25, 2018 at 6:14 AM Martin Thomson <[email-masked]>
wrote:

> > separate packet or with the close) it’s possible that ACK is lost..
> Ideally
> > this, and includes the ACK again as well, I believe everything would
> work..