Subject: Move GOAWAY to HTTP draft

From: Martin Thomson <[email-masked]>
Date: Mon, 6 Mar 2017 09:16:44 +1100

Mirja asks an interesting question:

> I'm actually wondering if GOAWAY is really part of the transport semantics or should be an application layer signal...? -- https://github.com/quicwg/base-drafts/pull/354#issuecomment-283937992

I can't think of a reason not to move GOAWAY.  Should we?

---

From: Ian Swett <[email-masked]>
Date: Sun, 5 Mar 2017 17:36:17 -0500

I believe we want a way to do a clean shutdown and say what the GOAWAY
will/could say(once a second stream id is added):
"No more streams after these will be created and existing streams will be
finished, then we're done."

The question of whether we need GOAWAY is very much intertwined with the
conversations about CONNECTION_CLOSE and public reset.  Currently, I think
GOAWAY should be used for clean connection termination, analogous to TCP's
FIN WAIT.  In a multiplexed transport, a way to indicate that no new
streams will be opened seems critical for a clean close.  Conceptually,
it's a connection level FIN.

I can imagine doing this with CONNECTION_CLOSE instead, but then you'd have
to specify the max stream ids and use a reserved error code that has a very
different meaning from the error cases.

So my current thinking is:
 - clean shutdown = GOAWAY and then silent close once the fin wait style
timeout expires
 - error shutdown = CONNECTION_CLOSE or public reset(depending on how that
discussion resolves) which indicates no more data should be expected for
the connection.

That being said, we could rename it to CONNECTION_FIN if that made it feel
more transporty and less confusing than re-using the HTTP2 term.

On Sun, Mar 5, 2017 at 5:16 PM, Martin Thomson <[email-masked]>
wrote:

> > I'm actually wondering if GOAWAY is really part of the transport
> semantics or should be an application layer signal...? --
> https://github.com/quicwg/base-drafts/pull/354#issuecomment-283937992
>

---

From: Martin Thomson <[email-masked]>
Date: Mon, 6 Mar 2017 09:51:27 +1100

You see, I agree that GOAWAY - or something like it - is critical for
clean shutdown.  But it's just one potential design.  There are two
pieces of information that it carries:

1. the connection is going to go away
2. these streams never got used and so if you want to do something
with them elsewhere, go for it

For the first, you suggest that you need a signal so that you can
start a timer.  I don't think that the timer is necessary, or
something we should assume.  Some connections might take a long time
to settle down after GOAWAY and that is fine.  Some HTTP/2
implementations send GOAWAY with a value of 2^31-1; they want to
trigger a failover, but don't want to disrupt any current activity.

For that reason, I think that FIN WAIT is a nasty design choice.

The second is something that I don't think that the transport needs to
know.  I couldn't think of a reason.

And the application protocol knows even better which streams were
unused than QUIC.  QUIC only knows if bytes crossed the threshold
between it and the application protocol.  At that point, it has to
assume that something important happened.  On the other hand, the
application protocol can happily abandon streams that didn't result in
any real work getting done.  For instance, receiving a few QPACK
octets might not result in any real activity because the
implementation waits until the entire header block is present before
doing work.  Or an HTTP request was just on a queue awaiting
processing and was cleanly removed from that queue.

I said before that GOAWAY is just one potential design choice.
Another design doesn't even involve establishing whether streams can
be retried.  That's something that might be HTTP-specific.

On 6 March 2017 at 09:36, Ian Swett <[email-masked]> wrote:
> FIN WAIT.  In a multiplexed transport, a way to indicate that no new streams
> will be opened seems critical for a clean close.  Conceptually, it's a
> connection level FIN.
>
> discussion resolves) which indicates no more data should be expected for the
> connection.
>

---

From: Lucas Clemente <[email-masked]>
Date: Sun, 05 Mar 2017 23:27:06 +0000

One additional point is that there might be applications other than HTTP
where streams are more long lived, potentially even for the whole duration
of the connection. Even other applications might require more streams to be
opened for a clean shutdown. In both cases, a GOAWAY carrying information
about the highest stream number might not be very useful.

Since we're effectively defining part of the API that implementations will
expose to applications here, I feel that we should think more about its
general applicability.

On Sun, 5 Mar 2017 at 23:51 Martin Thomson <[email-masked]> wrote:

> streams
> the
> > On Sun, Mar 5, 2017 at 5:16 PM, Martin Thomson <[email-masked]
> >
> > wrote:
> >>

---

From: Ian Swett <[email-masked]>
Date: Sun, 5 Mar 2017 20:16:09 -0500

> about the highest stream number might not be very useful.

In both those cases, a GOAWAY could be useful, but may not be critical,
since it seems the application is defining the stream usage fairly clearly.

>> with them elsewhere, go for it
#2 would never have occurred to me if you hadn't pointed it out.  I think
the most critical thing GOAWAY indicates is when the connection is going to
be completed in a clean way, because it indicates what the final stream IDs
will be.

>> For that reason, I think that FIN WAIT is a nasty design choice.
If you care about both sides knowing all the data they sent was received
and all the data they received was acked, I believe you end up with a state
machine using timers that's not so different from TCP's TIME_WAIT.
Possibly we don't care about providing that, but TCP provides it, so I
assumed we'd want to?

>> know.  I couldn't think of a reason.
Totally agreed.

>> be retried.  That's something that might be HTTP-specific.
Can you explain what the other design is a bit more?

---

From: Martin Thomson <[email-masked]>
Date: Mon, 6 Mar 2017 14:04:56 +1100

> Possibly we don't care about providing that, but TCP provides it, so I

Assumptions are dangerous.  I definitely think that some
implementations will maintain a timer, but I can't say that ALL will.

The idea that you have an unbounded duration on a shutdown command
doesn't make a lot of sense.  On the other hand, there are cases where
graceful >> timely.  That was something we definitely learned in
HTTP/2 very late in development.  For example, a server might want to
shift load but don't care about the load that you have already taken
on.  I can imagine servers nudging new requests onto other nodes
something like Alt-Svc/ALTSVC and GOAWAY, but leaving the old session
open indefinitely.

This makes much less sense in HTTP, because *most* requests are fairly
tightly time bounded in one way or other.  But a general purpose
protocol doesn't need to have the same constraints.  If we were to
multiplex websockets with HTTP (leaving aside for the moment whether
that's a good idea or not) that might be an example where using simple
timers makes little sense.

> Can you explain what the other design is a bit more?

Let's say that you are doing the WebRTC thing where you send
individual video frames in separate streams (leaving aside for the
moment whether that's a good idea or not).  When you shut something
like that down, you might not care if frame 63412 made it through or
not.  You could just say "I'm going away" and that's it.

The stream identifiers in GOAWAY are only needed in protocols that a)
care about retries and b) aren't consistently idempotent.

---

From: Ian Swett <[email-masked]>
Date: Mon, 6 Mar 2017 08:37:49 -0500

> implementations will maintain a timer, but I can't say that ALL will.

Agreed, some applications may not need a graceful close.

So I guess that provides two options:
1) Make every application implement their own graceful close if they need
it and only provide a way to kill the connection immediately.
2) Provide a transport layer graceful close(GOAWAY, or CONNECTION_FIN, etc).

Not having a way to close the connection gracefully seems like a missing
feature, but I'm not hung up on having it if it's really not generally
useful.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 7 Mar 2017 09:44:57 +1100

> feature, but I'm not hung up on having it if it's really not generally

We made a similar decision with priority.  Multiplexing without
priority is - at best - lame.  But we now require that the application
protocol manage priority and any signaling that might be needed.  The
same decision could be made for graceful shutdown, and based on this
discussion, I'm inclined to think that it's the right decision.

---

From: Martin Duke <[email-masked]>
Date: Mon, 6 Mar 2017 15:06:43 -0800

So I think the GOAWAY-less QUIC clean-shutdown protocol behavior is as
follows, trying to imitate TCP semantics where possible:

Application sends close()
QUIC stops initiating streams
Finish delivering all outstanding streams, send a FIN bit, and close the
stream the FIN is acked.
Wait patiently for all the peer's stream FINs and ack them.
* *QUIC makes the assumption that the peer will open no more streams* *
If the acknowledgment of the peer's last FINs was not itself acked because
it had data with it, then start a TIME-WAIT timer.
If not, or when the TIME-WAIT timer expires, silently delete all state (or,
if we want to signal middleboxes, send a PUBLIC_RESET).

I think I'm alright with that, particularly if we send that Reset.

We wouldn't have to assume that the peer will open no more streams if we
were to repurpose GOAWAY to mean "I'm not starting any more streams", which
in my view would be a bit cleaner.

---

From: Ted Hardie <[email-masked]>
Date: Mon, 6 Mar 2017 15:29:42 -0800

> Finish delivering all outstanding streams, send a FIN bit, and close the

So, this seems subtly limiting.  If QUIC stops initiating streams, it could
notify the other endpoint that it had entered shutdown and then let the
other side complete the shutdown at any point from then on.  For some
applications, the emitter initiating shutdown may mean that the receiver is
also done; for others, the reliability of stream delivery required may be
variable depending on the application semantics and the availability of FEC.

How about

Application sends close
QUIC stops initiating streams, send FIN_START
QUIC delivers outstanding streams until receiving FIN_COMPLETE
If all streams delivered and no FIN_COMPLETE, start timer.

I believe that lets you get the same semantics in the set above, by simply
delivering the FIN_COMPLETE when all outstanding streams are delivered, but
it may open up other semantics as well.

regards,

Ted

> If not, or when the TIME-WAIT timer expires, silently delete all state
> (or, if we want to signal middleboxes, send a PUBLIC_RESET).
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 7 Mar 2017 10:40:14 +1100

I think that both are likely to run afoul of a simple problem in HTTP:
the control stream (stream 3) can't be closed.

The graceful shutdown I was imagining was, specifically for HTTP.

1. HTTP application calls shutdown() locally (assuming that close() is
immediate)
2. HTTP layer signals intent to close (including last stream identifiers)
3. HTTP layer stops opening new streams
4. HTTP layer waits for all open requests and pushes to complete or
error out (which means waiting for RST_STREAM or FIN in both
directions AND waiting for an ACK for any RST_STREAM or FIN that it
initiated itself)
X. The peer HTTP layer receives intent to close and does the same
5. HTTP layer (on either end) calls close()
6. QUIC layer sends CONNECTION_CLOSE and terminates the connection

That doesn't rely on any QUIC-layer signaling.

Also, stream 3 never closes.  It doesn't have to.  Its contents are
critical, but only as long as there are requests/responses/pushes in
flight.

On 7 March 2017 at 10:29, Ted Hardie <[email-masked]> wrote:
>> * QUIC makes the assumption that the peer will open no more streams *

---

From: Ted Hardie <[email-masked]>
Date: Tue, 7 Mar 2017 08:58:50 -0800

Howdy,

On Mon, Mar 6, 2017 at 3:40 PM, Martin Thomson <[email-masked]>
wrote:

> I don't think that's actually a problem with what I suggested.  It simply
means that for the HTTP mapping, the defined behavior is to send
FIN_COMPLETE after all open requests and pushes complete.  If it arrives
before then, you'd have to treat it as a protocol error for the mapping.
What I'm trying to avoid is baking in the requirement that FIN_COMPLETE (or
its moral equivalent) always occur then, since it will make using other
types of streams harder.

Possibly I have missed something, though; happy to hear more.

Ted

---

From: Mike Bishop <[email-masked]>
Date: Tue, 7 Mar 2017 17:15:07 +0000

We could also change the HTTP draft to say that as part of shutdown, you *do* close the control stream after all other streams have been closed.  If the control stream closes before any other stream, it's an error.

Sent from my Windows 10 phone

From: Ted Hardie<mailto:[email-masked]>
Sent: Tuesday, March 7, 2017 8:59 AM
To: Martin Thomson<mailto:[email-masked]>
Cc: Ian Swett<mailto:[email-masked]>; IETF QUIC WG<mailto:[email-masked]>; Lucas Clemente<mailto:[email-masked]>; Martin Duke<mailto:[email-masked]>
Subject: Re: Move GOAWAY to HTTP draft

Howdy,

On Mon, Mar 6, 2017 at 3:40 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
I think that both are likely to run afoul of a simple problem in HTTP:
the control stream (stream 3) can't be closed.

I don't think that's actually a problem with what I suggested.  It simply means that for the HTTP mapping, the defined behavior is to send FIN_COMPLETE after all open requests and pushes complete.  If it arrives before then, you'd have to treat it as a protocol error for the mapping.  What I'm trying to avoid is baking in the requirement that FIN_COMPLETE (or its moral equivalent) always occur then, since it will make using other types of streams harder.

Ted

On 7 March 2017 at 10:29, Ted Hardie <[email-masked]<mailto:[email-masked]>> wrote:
> On Mon, Mar 6, 2017 at 3:06 PM, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> On Mon, Mar 6, 2017 at 2:44 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>>
>> wrote:
>>>
>>> On 7 March 2017 at 00:37, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Martin Thomson <[email-masked]>
Date: Wed, 8 Mar 2017 10:20:32 +1100

On 8 March 2017 at 03:58, Ted Hardie <[email-masked]> wrote:
> types

Sure.  I had inferred that you were doing this entirely at the
transport layer, which doesn't really allow you to do that.

---

From: Ian Swett <[email-masked]>
Date: Tue, 7 Mar 2017 21:52:42 -0500

After thinking about it more, if no one else has objections to "Make every
application implement their own graceful close if they need it and only
provide a way to kill the connection immediately.", I'm fine with it as
well.

On Tue, Mar 7, 2017 at 6:20 PM, Martin Thomson <[email-masked]>
wrote:

> > On Mon, Mar 6, 2017 at 3:40 PM, Martin Thomson <[email-masked]
> >
> > wrote:
> >>

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 7 Mar 2017 19:16:44 -0800

After chatting with Martin offline, it's growing on me as well. The key
insights in my mind are:

1. It's preferable for each endpoint to indicate the streams it is closing,
to avoid requiring upper-layer retries of new streams in flight.

2. The semantics of stream creation are different for different apps. HTTP
generally has the client creating a stream and the server responds on the
same stream. A different application can have very different client-server
interactions. For instance, a client creates a stream which requires a a
server to create multiple streams in response, which in turn require
further client stream creations. In this case, QUIC has no idea what the
"last" stream ought to be, and this needs to be indicated by the
application.

3. Since in the general case the app protocol is expected to be involved in
figuring out what graceful close looks like, it makes sense to define it
per app. In other words, make it part of the app mapping to QUIC. For HTTP,
this would be a GOAWAY frame on Stream 3. Once all streams are closed, HTTP
closes Stream 3.

4. Graceful close by the app means that the QUIC connection can immediately
go into TIME_WAIT on a connection close from the app.

It is a clean separation of concerns for sure. I'd like to ruminate on this
a bit, since sometimes that helps.

This seems like good fodder for Chicago.

---

From: Martin Duke <[email-masked]>
Date: Wed, 15 Mar 2017 15:06:26 -0700

This doesn't match my perception of the contract between a TCP-like
transport layer and applications that use it in a "clean close", but then
neither does the existing bidirectional GOAWAY. A clean close should
indicate to the app that all written data has been received, and the peer
has successfully delivered all its intended data.

What QUIC needs is a signal that says "I am done creating new streams but
am still willing to receive new ones". Although there ought to be
stream-level shutdown() calls in a future socket API, but a
connection-level shutdown() would send this signal and also FIN all
remaining streams. After both QUIC endpoints clean all this up at the
transport layer, there is no need for a further signal from the app.

Forcing every app to create its own GOAWAY (or, in my vision, send-only
GOAWAY) mechanism seems rather heavyweight if it is a common feature of a
clean close.

On Tue, Mar 7, 2017 at 7:16 PM, Jana Iyengar <[email-masked]> wrote:

> 1. It's preferable for each endpoint to indicate the streams it is
> closing, to avoid requiring upper-layer retries of new streams in flight.
>
> 3. Since in the general case the app protocol is expected to be involved
> in figuring out what graceful close looks like, it makes sense to define it
> 4. Graceful close by the app means that the QUIC connection can
> immediately go into TIME_WAIT on a connection close from the app.
>
> It is a clean separation of concerns for sure. I'd like to ruminate on
> this a bit, since sometimes that helps.
>
>> After thinking about it more, if no one else has objections to "Make
>> every application implement their own graceful close if they need it and
>> only provide a way to kill the connection immediately.", I'm fine with it
>> as well.
>>

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Mar 2017 14:48:29 +1100

I don't think that "TCP-like" is the right benchmark here.  That would
apply to streams, but not the entire session.

Even in HTTP, your signal doesn't work in one direction.  A server
generates streams in response to client streams.  It can't say "I'm
done with creating streams, but you continue for now" without cutting
itself off from server push.  You might not be sad about that, but
you'd have to admit that would be a subjective decision.

Given that we're aiming for a general purpose protocol, we can't
guarantee that other applications will have clean transactional
semantics such that your signal will be actionable at the transport
layer.  Only the application protocol knows when it is "done", so any
signal of imminent shutdown is going to be up to them.  What would a
generic shutdown() do?  You can't just close all open streams where
they stand and hope that the application protocol will be happy.

The more I think about this problem, the more I am convinced that the
application needs to sort it out and then send a signal to the
transport.  That signal would be "start a timer (like FIN_WAIT2), if
you don't see anything new in that time, discard state".  HTTP would
do that after receiving its own "I'm done" signals on its connection
control stream and closing out any lingering streams and
retransmissions.  The real signaling (a GOAWAY or analogous) would
have to be exchanged a long time before that.

On 16 March 2017 at 09:06, Martin Duke <[email-masked]> wrote:
> What QUIC needs is a signal that says "I am done creating new streams but am
> still willing to receive new ones". Although there ought to be stream-level
> shutdown() calls in a future socket API, but a connection-level shutdown()
> would send this signal and also FIN all remaining streams. After both QUIC
> endpoints clean all this up at the transport layer, there is no need for a
> further signal from the app.
>
>> server to create multiple streams in response, which in turn require further
>> client stream creations. In this case, QUIC has no idea what the "last"
>> stream ought to be, and this needs to be indicated by the application.
>>
>>> only provide a way to kill the connection immediately.", I'm fine with it as
>>> well.
>>>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Mar 2017 23:05:26 -0700

I agree with what Martin (Thomson) said. The key thing to recognize is that
QUIC streams are similar to TCP connections, but the QUIC connection is
different --- it's a container. Applying TCP logic to QUIC streams makes
sense, but the QUIC connection itself is a different beast that doesn't
need to follow the logic of TCP connections.

I have some thoughts on how to make QUIC connections seem more like
containers, at least for connection close semantics. I'll try to write up
something tomorrow.

On Wed, Mar 15, 2017 at 8:48 PM, Martin Thomson <[email-masked]>
wrote:

> further

---

From: Martin Duke <[email-masked]>
Date: Thu, 16 Mar 2017 14:05:01 -0700

Alright, I think I'm convinced we can get rid of GOAWAY entirely.

I would still like to have two types of application close (in terms of the
application API), one that is not graceful (immediate PUBLIC_RESET) and one
that is graceful (as you describe, "wait for X sec to make sure there are
no non-PING, non-PADDING frames") that leads to a CONNECTION_CLOSE (or, as
I would argue in #353, a delayed Public Reset)

On Wed, Mar 15, 2017 at 11:05 PM, Jana Iyengar <[email-masked]> wrote:

> I agree with what Martin (Thomson) said. The key thing to recognize is
> that QUIC streams are similar to TCP connections, but the QUIC connection
> is different --- it's a container. Applying TCP logic to QUIC streams makes

---

From: Mike Bishop <[email-masked]>
Date: Thu, 16 Mar 2017 21:32:56 +0000

From an endpoint level, I’m content with the model of:

  *   A frame is “confirmed delivered” when an ACK is received which says the a packet containing the frame was received by the peer
  *   A stream is “confirmed closed” when all data, or a RST_STREAM, has been confirmed delivered to the far side and when all data or a RST_STREAM has been received and the ACK is confirmed delivered
  *   Once a shutdown has been triggered, a connection closes silently when all streams are either confirmed closed or idle.

There is a slight race, in that a peer-initiated stream could be opened during the shutdown wait.  As others have noted, that can be handled at the app layer by giving a pre-shutdown notification.

However, you want to make sure that frame has been delivered.  That implies having a way for the application to wait for the ACK of a particular piece of stream data; that’s not something we currently require the transport to expose, and I’d prefer not to make it a new requirement.  The application could also add an app-level ACK of some kind, or (for HTTP) the closing of stream 3 could be an implicit ACK.

And the other question, whether we want to expose to the path that the connection has definitively ended.

From: QUIC [mailto:[email-masked]] On Behalf Of Martin Duke
Sent: Thursday, March 16, 2017 2:05 PM
To: Jana Iyengar <[email-masked]>
Cc: Lucas Clemente <[email-masked]>; Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>; Ted Hardie <[email-masked]>
Subject: Re: Move GOAWAY to HTTP draft

I would still like to have two types of application close (in terms of the application API), one that is not graceful (immediate PUBLIC_RESET) and one that is graceful (as you describe, "wait for X sec to make sure there are no non-PING, non-PADDING frames") that leads to a CONNECTION_CLOSE (or, as I would argue in #353, a delayed Public Reset)

On Wed, Mar 15, 2017 at 11:05 PM, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:
I agree with what Martin (Thomson) said. The key thing to recognize is that QUIC streams are similar to TCP connections, but the QUIC connection is different --- it's a container. Applying TCP logic to QUIC streams makes sense, but the QUIC connection itself is a different beast that doesn't need to follow the logic of TCP connections.

I have some thoughts on how to make QUIC connections seem more like containers, at least for connection close semantics. I'll try to write up something tomorrow.

On Wed, Mar 15, 2017 at 8:48 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
I don't think that "TCP-like" is the right benchmark here.  That would
apply to streams, but not the entire session.

On 16 March 2017 at 09:06, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
> On Tue, Mar 7, 2017 at 7:16 PM, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> On Tue, Mar 7, 2017 at 6:52 PM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>> On Tue, Mar 7, 2017 at 6:20 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>>
>>> wrote:
>>>>
>>>> On 8 March 2017 at 03:58, Ted Hardie <[email-masked]<mailto:[email-masked]>> wrote:
>>>> >
>>>> > On Mon, Mar 6, 2017 at 3:40 PM, Martin Thomson
>>>> > <[email-masked]<mailto:[email-masked]>>
>>>> > wrote:
>>>> >>

---

From: Martin Duke <[email-masked]>
Date: Thu, 16 Mar 2017 14:38:12 -0700

Mike,

I agree with everything -- except that there is no way to "confirm
delivery" of an ack in the general case: you do not ack a packet consisting
only of acks. This is why we need some sort of timer at the end to make
sure that the peer isn't still retransmitting stuff due to a lost ACK.

On Thu, Mar 16, 2017 at 2:32 PM, Mike Bishop <[email-masked]>
wrote:

>    - A frame is “confirmed delivered” when an ACK is received which says
>    the a packet containing the frame was received by the peer
>    - A stream is “confirmed closed” when all data, or a RST_STREAM, has
>    been confirmed delivered to the far side and when all data or a RST_STREAM
>    has been received and the ACK is confirmed delivered
>    - Once a shutdown has been triggered, a connection closes silently
>    when all streams are either confirmed closed or idle.
>
>
>
> There is a slight race, in that a peer-initiated stream could be opened
> during the shutdown wait.  As others have noted, that can be handled at the
> app layer by giving a pre-shutdown notification.
>
>
>
> However, you want to make sure that frame has been delivered.  That
> implies having a way for the application to wait for the ACK of a
> particular piece of stream data; that’s not something we currently require
> the transport to expose, and I’d prefer not to make it a new requirement.
> The application could also add an app-level ACK of some kind, or (for HTTP)
> the closing of stream 3 could be an implicit ACK.
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Martin Duke
> *Sent:* Thursday, March 16, 2017 2:05 PM
> *To:* Jana Iyengar <[email-masked]>
> *Cc:* Lucas Clemente <[email-masked]>; Ian Swett <[email-masked]>;
> IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>;
> Ted Hardie <[email-masked]>
> *Subject:* Re: Move GOAWAY to HTTP draft
>
>
>
> >>>> > <[email-masked]>
> >>>> > wrote:
> >>>> >>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 17 Mar 2017 10:01:17 +1100

On 17 March 2017 at 08:38, Martin Duke <[email-masked]> wrote:
>
> I agree with everything -- except that there is no way to "confirm delivery"
> of an ack in the general case: you do not ack a packet consisting only of
> acks. This is why we need some sort of timer at the end to make sure that
> the peer isn't still retransmitting stuff due to a lost ACK.

Yes, I agree with this.  The model that I believe works best here is
that you have two timers, depending on whether the application agreed
to shutdown.  The idle timer runs when the state that Mike described
is reached, unless the application indicates that it's arranged to
shutdown, and then you need a timer to deal with extra retransmissions
and reordering and the like.

---

From: Mike Bishop <[email-masked]>
Date: Thu, 16 Mar 2017 23:04:00 +0000

But isn’t removal of STOP_WAITING predicated on getting ACKs of the ACK frames eventually?  But I guess that presumes there will be other traffic triggering ACK generation eventually, which will then mention the ACK-only packet.

We’re recommending the use of WINDOW_UPDATE or PING to trigger an ACK if you need one – perhaps we should note that during shutdown, you SHOULD trigger ACKs so you can verify final closure.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Thursday, March 16, 2017 4:01 PM
To: Martin Duke <[email-masked]>
Cc: Mike Bishop <[email-masked]>; Jana Iyengar <[email-masked]>; Lucas Clemente <[email-masked]>; Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>; Ted Hardie <[email-masked]>
Subject: Re: Move GOAWAY to HTTP draft

On 17 March 2017 at 08:38, Martin Duke <[email-masked]> wrote:
> of an ack in the general case: you do not ack a packet consisting only 
> of acks. This is why we need some sort of timer at the end to make 

Yes, I agree with this.  The model that I believe works best here is that you have two timers, depending on whether the application agreed to shutdown.  The idle timer runs when the state that Mike described is reached, unless the application indicates that it's arranged to shutdown, and then you need a timer to deal with extra retransmissions and reordering and the like.

---

From: Ryan Hamilton <[email-masked]>
Date: Thu, 16 Mar 2017 16:27:43 -0700

​​On Thu, Mar 16, 2017 at 4:04 PM, Mike Bishop <[email-masked]

> But isn’t removal of STOP_WAITING predicated on getting ACKs of the ACK
> frames eventually?

​Eventually yes. Immediately, no.​

> But I guess that presumes there will be other traffic triggering ACK
> generation eventually, which will then mention the ACK-only packet.
>

​Exactly! An ACK *will* eventually be acknowledged. But it won't be the
result of an ACK-only packet being received by the peer (since those never
solicit an ACK in response).

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 16 Mar 2017 19:08:25 -0700

I was trying to come up with a coherent way to capture these cases (thanks
Ryan, Ian and Martin for offline discussions). A few iterations later, I
have a simplified proposal.

When an app calls close() on a QUIC connection, the transport immediately
tears down all state.

We have to account for closing after idle and for the equivalent of a
FIN_WAIT to retransmit or ack retransmissions. Before describing that, I'll
define two time periods. IDLE is the amount of time that an endpoint closes
the connection after, and DRAIN is the amount of time that an endpoint
retransmits or sends acks of retransmissions at the tail of the connection.

After all streams are closed (via FIN or RST_STREAM), the app is
responsible for idle close. Assuming that the app negotiates this IDLE
timeout value (or we could negotiate this in QUIC and expose it to the app,
but that seems unnecessary), it waits for IDLE time, and then closed the
connection. Importantly, neither endpoint initiates new streams after IDLE
- DRAIN time has passed. Of course, if a new stream is created during this
period, the app is out of idle.

When the app wants to gracefully close the connection, the app peers can
agree in some manner to close the connection, by closing Stream 3 in HTTP
for example. The app waits for DRAIN time after all streams are closed and
then closes the connection.

Finally, we have the QUIC connection close, which I'll call error close
here, since that's what it really is. On error close, we could do one of
two things. QUIC could call back into the app, the app waits DRAIN period,
and the app then closes the connection. Alternatively, QUIC could have a
timer here that waits for DRAIN amount of time and then kills itself. I
much prefer the first idea since it keeps the waiting logic entirely inside
the app for all cases.

This proposal would basically remove any waiting inside QUIC and moves that
entire logic to the application.

I could be missing something here, so I'd like to think about this some
more. But y'all could poke holes in this sooner, so I'm sharing. I'm happy
to write this in a separate email thread and/or a PR and/or issue, but
airing this here for now.

What do you think? Plausible?

- jana

On Mar 16, 2017 4:27 PM, "Ryan Hamilton" <[email-masked]> wrote:

> ​​On Thu, Mar 16, 2017 at 4:04 PM, Mike Bishop <
> [email-masked]> wrote:
>

---

From: Martin Duke <[email-masked]>
Date: Thu, 16 Mar 2017 22:18:23 -0700

The DRAIN timer in your concept is common to most/all apps and should
therefore be in QUIC. To force every application developer to implement a
timer to perform a clean close seems pointlessly duplicative.

Moreover, in any case I think a packet needs to go over the wire at the
QUIC layer.

On Thu, Mar 16, 2017 at 7:08 PM, Jana Iyengar <[email-masked]> wrote:

> for example. The app waits for DRAIN time after all streams are closed
> and then closes the connection.
>
> This proposal would basically remove any waiting inside QUIC and moves
> that entire logic to the application.
>

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Mar 2017 00:00:47 -0700

> timer to perform a clean close seems pointlessly duplicative.

Yeah, I was in two minds about this. This is a reasonable expectation to
have from the transport (though not all apps need this), so we could move
the DRAIN time (after close() and error close) to inside QUIC. This would
make the error close logic more contained inside QUIC. The application
could still do the logic required for an idle close.

> QUIC layer.

I don't think this is required. What packet are you thinking about?

- jana

>> I was trying to come up with a coherent way to capture these cases
>> (thanks Ryan, Ian and Martin for offline discussions). A few iterations
>> later, I have a simplified proposal.
>>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 17 Mar 2017 20:51:01 +1100

On 17 March 2017 at 18:00, Jana Iyengar <[email-masked]> wrote:
> make the error close logic more contained inside QUIC. The application could
> still do the logic required for an idle close.

I had the same sense as Martin here, and this would be easier in some
ways.  Note that one consequence of your model is that the idle timer
moves under encryption because it would be an HTTP setting, not a
transport parameter.  I can see pros and cons to that.

> I don't think this is required. What packet are you thinking about?

I can't think of a reason either.  Any packet that QUIC sends is going
to cost radio/battery/bandwidth where it would otherwise be silent and
I would prefer silent unless there is a good reason.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 17 Mar 2017 10:06:33 -0400

> timer to perform a clean close seems pointlessly duplicative.
I think I agree with this - and it would also help to have a sensible
baseline here rather than having apps not think through the consequences
and just doing 0-idle after last write (which brings up the specter of
repeating TCP RST and unintentional data loss as part of supposedly clean
shutdown) - that seems like a transport responsibility.

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Mar 2017 09:58:39 -0700

On Fri, Mar 17, 2017 at 2:51 AM, Martin Thomson <[email-masked]>
wrote:

> could

Good point, that had not occurred to me. I'm wary of middleboxes reading
into the ClientHello messages to pull out a value, so it's more a pro from
my point of view than a con. But, it is a difference for sure.

---

From: Mike Bishop <[email-masked]>
Date: Fri, 17 Mar 2017 17:00:19 +0000

I guess I’m still a little stuck on why there’s a timer in the clean case anyway.  In the clean case, it should be a deterministic closure based on reaching a defined end state, shouldn’t it?  In an error close, you probably don’t have enough shared state to have that requirement, so a timer may be required.

I think we all agree that the connection being closed is equivalent to saying all streams are closed, though you can make a case for either being the cause of the other.  Going a step beyond that, the connection is “closed” (and state can be discarded) when all streams are either idle or closed, both sides agree on all stream states, and the app has indicated a desire to close the connection.

Based on the new ACK text in -02, we have a defined concept of when the other side has caught up to the local perspective of a stream state:  When all outstanding STREAM frames through the FIN have been ACK’d, or when a RST_STREAM frame has been ACK’d.  To know that both sides are in sync, you also need to know that your ACKs have been received, so you need to start soliciting ACKs of any ACKs which cover any STREAM or RST_STREAM frames.  You should quickly reach a state where all streams are either closed or idle, and neither side is expecting more ACKs.  At that point, you can discard all state silently.

Perhaps a clean closure looks like this:

  *   App notifies peer (at app layer) of intent to close the connection
     *   Caveat is because in HTTP, where servers only initiate in reaction to requests, an idle client could opt to close without coordination, since the server won’t be sending anything.
     *   Note that this strongly implies every protocol mapping will need a control stream
  *   App calls shutdown() on each side
  *   Transport does the following on shutdown():
     *   Prohibits initiating new streams
     *   Waits for all app-controlled streams to close in the local perspective
        *   Peer could still initiate new streams; we rely on the app to inform them they shouldn’t and reject any new streams
     *   Begins actively soliciting ACKs of ACKs where necessary (by including WINDOW_UPDATE or PING frames)
        *   Where necessary => ACK covers a STREAM or RST_STREAM frame
     *   Discard all state when no ACKs are outstanding

From: QUIC [mailto:[email-masked]] On Behalf Of Patrick McManus
Sent: Friday, March 17, 2017 7:07 AM
To: Martin Duke <[email-masked]>
Cc: Mike Bishop <[email-masked]>; Ted Hardie <[email-masked]>; Ian Swett <[email-masked]>; Ryan Hamilton <[email-masked]>; Lucas Clemente <[email-masked]>; Jana Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Move GOAWAY to HTTP draft

On Fri, Mar 17, 2017 at 1:18 AM, Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
The DRAIN timer in your concept is common to most/all apps and should therefore be in QUIC. To force every application developer to implement a timer to perform a clean close seems pointlessly duplicative.

I think I agree with this - and it would also help to have a sensible baseline here rather than having apps not think through the consequences and just doing 0-idle after last write (which brings up the specter of repeating TCP RST and unintentional data loss as part of supposedly clean shutdown) - that seems like a transport responsibility.

---

From: Martin Thomson <[email-masked]>
Date: Sat, 18 Mar 2017 09:44:04 +1100

On 18 March 2017 at 04:00, Mike Bishop <[email-masked]> wrote:
> I guess I’m still a little stuck on why there’s a timer in the clean case
> anyway.  In the clean case, it should be a deterministic closure based on
> reaching a defined end state, shouldn’t it?  In an error close, you probably
> don’t have enough shared state to have that requirement, so a timer may be
> required.

DRAIN is for the case where you have skew in this agreed end state.
For instance, an endpoint might have send an unneeded retransmission,
but later received an ACK.  The endpoint that sent the ACK reaches a
quiescent state when it sends the ACK and so could close.  But it
could later receive the extra retransmission.  You want to have that
retransmission hit the connection state so that it can be cleanly
dropped, rather than hitting the "this is a new packet" code path,
that's all.

---

From: Martin Duke <[email-masked]>
Date: Tue, 28 Mar 2017 15:10:05 -0700

Yes, the end of any exchange is a pure ACK, which may be lost. So at least
one end has to remain around to send the ACK again if needed.

As for a concluding packet, I agree that in a pure end-to-end world it is
not only unnecessary, but harmful due to power consumption. However, in
#353 we are discussing always sending a Public Reset to clear stateful
middleboxes. For all the concern about power, I suspect most mobile
operators are running these middleboxes and would appreciate the courtesy;
they could even eat this packet before it gets to the phone if they'd like
to optimize. It's how I would recommend operating to our service provider
customers.

I would recommend language as follows:
1) When an endpoint acknowledges a STREAM frame that closes all remaining
streams, it starts the DRAIN timer. If the final FINs pass in flight, this
will result in both sides starting DRAIN.
2) Endpoints SHOULD send a Public Reset with error code "Clean Close" (or
whatever) if: (rule should say one of the following, but not both)
   (2a) When all streams are closed and they do not set the DRAIN timer.
This would allow earlier cleanup of the DRAIN timer state, but causes
simultaneous close cases to not send a reset at all.
   (2b) When the DRAIN timer expires. Guarantees at least one Reset, but
doesn't save us any time on DRAIN.