Subject: Clean shutdown semantics

From: Martin Duke <[email-masked]>
Date: Thu, 23 Feb 2017 15:27:39 -0800

I am considering opening issue about clean shutdown semantics, but wanted
to bounce it off the list first in case there is something obvious I am
missing.

In TCP, after a clean shutdown both endpoints have a reasonable assurance
that (1) All data they sent was acknowledged, and (2) the peer was able to
deliver all the data it wanted. This is a property of the connection FIN
being a byte in a connection-wide, reliably delivered sequence space.

I see two aspects of QUIC that result in this assurance not quite being
achievable. This may be a result of editorial incompleteness, or something
baked into what Google implemented.

(1) GOAWAY is bidirectional. There is no way to say "I am done opening
streams, but you can open more streams if you need to." I don't see a way
to cleanly close the connection and know for sure that the peer didn't have
more streams to open.

(2) CONNECTION_CLOSE is ambiguous between abrupt and graceful shutdowns.
There are some related logic issues too.

(a) Say that I am endpoint A wrapping up a communication with endpoint B.
We've already sent GOAWAY(s), and I send a packet that sends the last data
and FIN for all remaining streams.

(b) I then receive a similar packet from B, delivering some data and FINing
all open streams, which I acknowledge.

(c) I receive a packet only containing an ACK frame for packet (a). I now
have assurance that all my data is delivered, and I have received
everything B has for me. There is no ack to send because there is only an
ACK frame.

(d) I therefore send CONNECTION_CLOSE.

But if the ack in (b) is lost, then endpoint B receives CONNECTION_CLOSE
while data is still outstanding. I believe the spec, as written, says to
treat this as an abrupt shutdown and throw the queued data on the floor.

Furthermore, B is required to acknowledge the CONNECTION_CLOSE frame. Even
if it were to keep the send queue after CONNECTION_CLOSE, the ack would
probably go out first. So A can tear down its connection before B has any
assurance that A got all its data.

There are several minor protocol edits to avoid all the problems in this
scenario. But I don't believe the spec as written is sufficient.

Martin Duke

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 24 Feb 2017 01:31:15 +0000

We’ve actually discussed related issues earlier today.  You can also see https://github.com/quicwg/base-drafts/issues/330

Ø  But if the ack in (b) is lost, then endpoint B receives CONNECTION_CLOSE while data is still outstanding. I believe the spec, as written, says to treat this as an abrupt shutdown and throw the queued data on the floor.

No data is actually outstanding in this scenario, since per (b), A received all data.  When B sees CONNECTION_CLOSE, it knows that either A received all data (including FIN), or A is no longer interested in any data from B. In any case, there is no problem for B to drop its queue and be done.

If B really desires to differentiate between whether A received all data or suddenly lost interest in data, there is an error code on CONNECTION_CLOSE. I suppose QUIC_STREAM_NO_ERROR would mean an orderly shutdown, which is the case of A receiving all data.  In any case, you cannot tell this one for sure with TCP either, since even if the transport can give you an assurance that all data has been received, there is no guarantee that the application would read that data.

-          Igor

From: Martin Duke [mailto:[email-masked]]
Sent: Thursday, February 23, 2017 6:28 PM
To: IETF QUIC WG <[email-masked]>
Subject: Clean shutdown semantics

I am considering opening issue about clean shutdown semantics, but wanted to bounce it off the list first in case there is something obvious I am missing.

In TCP, after a clean shutdown both endpoints have a reasonable assurance that (1) All data they sent was acknowledged, and (2) the peer was able to deliver all the data it wanted. This is a property of the connection FIN being a byte in a connection-wide, reliably delivered sequence space.

I see two aspects of QUIC that result in this assurance not quite being achievable. This may be a result of editorial incompleteness, or something baked into what Google implemented.

(1) GOAWAY is bidirectional. There is no way to say "I am done opening streams, but you can open more streams if you need to." I don't see a way to cleanly close the connection and know for sure that the peer didn't have more streams to open.

(2) CONNECTION_CLOSE is ambiguous between abrupt and graceful shutdowns. There are some related logic issues too.

(a) Say that I am endpoint A wrapping up a communication with endpoint B. We've already sent GOAWAY(s), and I send a packet that sends the last data and FIN for all remaining streams.

(b) I then receive a similar packet from B, delivering some data and FINing all open streams, which I acknowledge.

(c) I receive a packet only containing an ACK frame for packet (a). I now have assurance that all my data is delivered, and I have received everything B has for me. There is no ack to send because there is only an ACK frame.

But if the ack in (b) is lost, then endpoint B receives CONNECTION_CLOSE while data is still outstanding. I believe the spec, as written, says to treat this as an abrupt shutdown and throw the queued data on the floor.

Furthermore, B is required to acknowledge the CONNECTION_CLOSE frame. Even if it were to keep the send queue after CONNECTION_CLOSE, the ack would probably go out first. So A can tear down its connection before B has any assurance that A got all its data.

There are several minor protocol edits to avoid all the problems in this scenario. But I don't believe the spec as written is sufficient.

Martin Duke

---

From: Martin Thomson <[email-masked]>
Date: Fri, 24 Feb 2017 14:17:00 +1100

It might not be helpful to think of QUIC as whole being in one of two
states: cleanly closed, or broken.  Considering individual streams to
be either complete or cancelled helps a lot.

To that end, I model GOAWAY as a selective RST_STREAM that cuts the
stream space into two and cancels all the streams on one side.
CONNECTION_CLOSE then is a universal RST_STREAM.  That allows this to
be reduced to thinking in terms of the stream state machine.

On 24 February 2017 at 10:27, Martin Duke <[email-masked]> wrote:
> I am considering opening issue about clean shutdown semantics, but wanted to
> bounce it off the list first in case there is something obvious I am
> missing.
>
> streams, but you can open more streams if you need to." I don't see a way to
> cleanly close the connection and know for sure that the peer didn't have
> more streams to open.
>
> have assurance that all my data is delivered, and I have received everything
> B has for me. There is no ack to send because there is only an ACK frame.
>

---

From: Martin Duke <[email-masked]>
Date: Thu, 23 Feb 2017 20:51:33 -0800

Thanks, I don't quite agree with everything said here, but I'll dive in to
#330.

On Thu, Feb 23, 2017 at 7:17 PM, Martin Thomson <[email-masked]>
wrote:

> everything

---

From: Ian Swett <[email-masked]>
Date: Fri, 24 Feb 2017 10:11:42 -0500

When Ryan and I were discussing GOAWAY today, he noted it didn't indicate
what the largest stream the sender would create is.  It says "GOAWAY will
not initiate any additional streams and will not accept any new incoming
streams.", but there's no way to indicate the largest stream it has/will
create.

It says "active streams will continue to be processed", but that doesn't
actually indicate whether they'll be properly finished in some way, ie:
will all outstanding data be delivered by the sender of the GOAWAY and the
stream be properly fin'd or reset?

I filed this as a new issue for GOAWAY(#347
<https://github.com/quicwg/base-drafts/issues/347>), since I think it'd be
nice to clarify both connection close and goaway at the same time.