Subject: Is CONNECTION_CLOSE frame Ack-eliciting？

From: Jiuhai Zhang <[email-masked]>
Date: Wed, 5 Jun 2019 20:03:30 +0800

In https://tools.ietf.org/html/draft-ietf-quic-recovery-20#section-2

Ack-eliciting Frames:  All frames besides ACK or PADDING are
      considered ack-eliciting.

Is CONNECTION_CLOSE frame Ack-eliciting？

Should endpoint responds a ACK frame when receiving a CONNECTION_CLOSE frame？

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Jun 2019 09:48:28 -0400

Technically, yes.  But connections don't send any packets after processing
a CONNECTION_CLOSE frame, so you would never receive and ACK for it.

From: https://tools.ietf.org/html/draft-ietf-quic-transport-20#section-10.1
"The draining state is entered once an endpoint receives a signal that
   its peer is closing or draining.  While otherwise identical to the
   closing state, an endpoint in the draining state MUST NOT send any
   packets.  Retaining packet protection keys is unnecessary once a
   connection is in the draining state."

---

From: Nick Banks <[email-masked]>
Date: Wed, 5 Jun 2019 13:59:47 +0000

Ian,

I have encountered issues both in local automated tests and during interop, where one side gets everything it wants and then cleanly closes the connection. The issues arise because the peer hasn’t received the ACK for that last exchange. For that reason, winquic always encodes its latest ACK state with (always frames before) the CONNECTION_CLOSE frame. This doesn’t increase the size of the packet much, and it’s up to the receiver if they want to process the ACKs or not; but it helps these kind of scenarios.

A simple example test case:

Client: Open one bidirectional stream, send data on it, FIN it, and verify the peer does the same. Then close the connection.
Server: Accept one bidirectional stream, receive all data on it and echo it back. Verify all data is sent successfully.

The “Verify all data is sent successfully” fails if you don’t send ACKs with CONNECTION_CLOSE. So, I personally think it’s a good idea to recommend including ACKs. I know it doesn’t provide any guarantees in the presence of packet loss, so ideally you don’t design an app protocol on top of this, but it does make things a little nicer, IMO.

Thank,
- Nick

From: QUIC <[email-masked]> On Behalf Of Ian Swett
Sent: Wednesday, June 5, 2019 6:48 AM
To: Jiuhai Zhang <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Is CONNECTION_CLOSE frame Ack-eliciting？

Technically, yes.  But connections don't send any packets after processing a CONNECTION_CLOSE frame, so you would never receive and ACK for it.

From: https://tools.ietf..org/html/draft-ietf-quic-transport-20#section-10.1<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-transport-20%23section-10.1&data=02%7C01%7Cnibanks%40microsoft.com%7Ca3e4b3be88064c15350408d6e9bc9185%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953393451363725&sdata=bi9%2FDh3XZuUgM%2B23tC54csAhdE5RQENRTQzgj5%2B%2BPlM%3D&reserved=0>

On Wed, Jun 5, 2019 at 8:04 AM Jiuhai Zhang <[email-masked]<mailto:[email-masked]>> wrote:
In https://tools.ietf.org/html/draft-ietf-quic-recovery-20#section-2<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-20%23section-2&data=02%7C01%7Cnibanks%40microsoft.com%7Ca3e4b3be88064c15350408d6e9bc9185%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953393451363725&sdata=420rgTO4Iud8HbpF8KvvANjFfPFP%2BtwAnjVS7KszGb4%3D&reserved=0>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 5 Jun 2019 07:07:49 -0700

CONNECTION_CLOSE can be response to CONNECTION_CLOSE and that response can
contain other frames. So technically you can ACK a connection close.

Doing so might actually make sense if the original closer take that as a
signal to drain faster.

On 5 June 2019 at 16.00.16, Nick Banks (
nibanks=[email-masked]) wrote:

Ian,

I have encountered issues both in local automated tests and during interop,
where one side gets everything it wants and then cleanly closes the
connection. The issues arise because the peer hasn’t received the ACK for
that last exchange. For that reason, winquic always encodes its latest ACK
state with (always frames before) the CONNECTION_CLOSE frame. This doesn’t
increase the size of the packet much, and it’s up to the receiver if they
want to process the ACKs or not; but it helps these kind of scenarios.

Client: Open one bidirectional stream, send data on it, FIN it, and verify
the peer does the same. Then close the connection.

Server: Accept one bidirectional stream, receive all data on it and echo it
back. Verify all data is sent successfully.

The “Verify all data is sent successfully” fails if you don’t send ACKs
with CONNECTION_CLOSE. So, I personally think it’s a good idea to recommend
including ACKs. I know it doesn’t provide any guarantees in the presence of
packet loss, so ideally you don’t design an app protocol on top of this,
but it does make things a little nicer, IMO.

Thank,

- Nick

*From:* QUIC <[email-masked]> *On Behalf Of * Ian Swett
*Sent:* Wednesday, June 5, 2019 6:48 AM
*To:* Jiuhai Zhang <[email-masked]>
*Cc:* IETF QUIC WG <[email-masked]>
*Subject:* Re: Is CONNECTION_CLOSE frame Ack-eliciting？

From: https://tools.ietf..org/html/draft-ietf-quic-transport-20#section-10.1
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-transport-20%23section-10.1&data=02%7C01%7Cnibanks%40microsoft.com%7Ca3e4b3be88064c15350408d6e9bc9185%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953393451363725&sdata=bi9%2FDh3XZuUgM%2B23tC54csAhdE5RQENRTQzgj5%2B%2BPlM%3D&reserved=0>

In https://tools.ietf.org/html/draft-ietf-quic-recovery-20#section-2
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-20%23section-2&data=02%7C01%7Cnibanks%40microsoft.com%7Ca3e4b3be88064c15350408d6e9bc9185%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953393451363725&sdata=420rgTO4Iud8HbpF8KvvANjFfPFP%2BtwAnjVS7KszGb4%3D&reserved=0>

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Jun 2019 11:58:06 -0400

Nick, I agree that bundling an ACK with a CONNECTION_CLOSE is useful, and
our implementation does that as well.

However, it seems like the existing text indicates you'd never expect to
get a CONNECTION_CLOSE in response to a CONNECTION_CLOSE?  Or am I
misreading the text about draining?

On Wed, Jun 5, 2019 at 10:08 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> contain other frames.. So technically you can ACK a connection close.
>
> I have encountered issues both in local automated tests and during
> interop, where one side gets everything it wants and then cleanly closes
> the connection. The issues arise because the peer hasn’t received the ACK
> for that last exchange. For that reason, winquic always encodes its latest
> ACK state with (always frames before) the CONNECTION_CLOSE frame. This
> doesn’t increase the size of the packet much, and it’s up to the receiver
> if they want to process the ACKs or not; but it helps these kind of
> scenarios.
>
>
>
> Server: Accept one bidirectional stream, receive all data on it and echo
> it back. Verify all data is sent successfully.
>
>
>
> From:
> https://tools.ietf..org/html/draft-ietf-quic-transport-20#section-10.1

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 5 Jun 2019 09:03:11 -0700

10.3: "After receiving a CONNECTION_CLOSE frame, endpoints enter the
draining state. An endpoint that receives a CONNECTION_CLOSE frame MAY send
a single packet containing a CONNECTION_CLOSE frame before entering the
draining state, using a CONNECTION_CLOSE frame and a NO_ERROR code if
appropriate."

On 5 June 2019 at 17.58.19, Ian Swett ([email-masked]) wrote:

On Wed, Jun 5, 2019 at 10:08 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> *From:* QUIC <[email-masked]> *On Behalf Of* Ian Swett

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Jun 2019 12:08:03 -0400

Thanks, I missed that.

On Wed, Jun 5, 2019 at 12:03 PM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> On Wed, Jun 5, 2019 at 10:08 AM Mikkel Fahnøe Jørgensen <
> [email-masked]> wrote:
>
>> CONNECTION_CLOSE can be response to CONNECTION_CLOSE and that response
>> can contain other frames.. So technically you can ACK a connection close.
>>
>> Client: Open one bidirectional stream, send data on it, FIN it, and
>> verify the peer does the same. Then close the connection.
>>
>> Technically, yes.  But connections don't send any packets after
>> processing a CONNECTION_CLOSE frame, so you would never receive and ACK for
>> it.
>>
>>
>>
>> From:

---

From: Marten Seemann <[email-masked]>
Date: Thu, 6 Jun 2019 00:25:45 +0800

It doesn't really make sense to send a CONNECTION_CLOSE frame in response
to a CONNECTION_CLOSE, since in the closing state an endpoint retains "only
enough information to generate a packet containing a CONNECTION_CLOSE frame
and to identify packets as belonging to the connection", i.e. an endpoint
won't even open the packet (but it might reply with another
CONNECTION_CLOSE).
Of course, and endpoint can send as many packets after this as it wants
(there's no way it could be penalized for misbehaving), although I'm not
sure why we're explicitly allowing sending CONNECTION_CLOSE as a response.

I'm not really sure what the benefit is of sending an ACK frame bundled
with a CONNECTION_CLOSE. There's nothing the receiving peer can do in
reaction to receiving that ACK - the connection is already closed.
Furthermore, it would be wrong to conclude that all stream data has been
processed by the application. An ACK for a STREAM frame only means that the
STREAM frame was processed by the QUIC stack, not by the application.

---

From: Nick Banks <[email-masked]>
Date: Wed, 5 Jun 2019 16:38:15 +0000

There is no absolute requirement that you maintain the minimal state to generate the CONNECTION_CLOSE. Personally, I feel a MAY is missing in that sentence. Additionally, I feel that an endpoint SHOULD (not a MUST because it’s unenforceable) respond with an immediate CONNECTION_CLOSE of its own, so the peer can use that as a signal of acknowledgement and immediately go away, without waiting the full draining period. But, I know I’m not in the majority with that opinion.

On the topic of sending ACKs with CONNECTION_CLOSE, I am more referring to bundling ACKs with the initial CONNECTION_CLOSE. I agree, that sending ACKs with the responding CONNECTION_CLOSE doesn’t do much.

- Nick

From: Marten Seemann <[email-masked]>
Sent: Wednesday, June 5, 2019 9:26 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Ian Swett <[email-masked]>; Nick Banks <[email-masked]>; IETF QUIC WG <[email-masked]>; Jiuhai Zhang <[email-masked]>
Subject: Re: Is CONNECTION_CLOSE frame Ack-eliciting？

It doesn't really make sense to send a CONNECTION_CLOSE frame in response to a CONNECTION_CLOSE, since in the closing state an endpoint retains "only enough information to generate a packet containing a CONNECTION_CLOSE frame and to identify packets as belonging to the connection", i.e. an endpoint won't even open the packet (but it might reply with another CONNECTION_CLOSE).
Of course, and endpoint can send as many packets after this as it wants (there's no way it could be penalized for misbehaving), although I'm not sure why we're explicitly allowing sending CONNECTION_CLOSE as a response.

I'm not really sure what the benefit is of sending an ACK frame bundled with a CONNECTION_CLOSE. There's nothing the receiving peer can do in reaction to receiving that ACK - the connection is already closed. Furthermore, it would be wrong to conclude that all stream data has been processed by the application. An ACK for a STREAM frame only means that the STREAM frame was processed by the QUIC stack, not by the application.

On Thu, Jun 6, 2019 at 12:03 AM Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
10.3: "After receiving a CONNECTION_CLOSE frame, endpoints enter the draining state. An endpoint that receives a CONNECTION_CLOSE frame MAY send a single packet containing a CONNECTION_CLOSE frame before entering the draining state, using a CONNECTION_CLOSE frame and a NO_ERROR code if appropriate."

On 5 June 2019 at 17.58.19, Ian Swett ([email-masked]<mailto:[email-masked]>) wrote:
Nick, I agree that bundling an ACK with a CONNECTION_CLOSE is useful, and our implementation does that as well.

However, it seems like the existing text indicates you'd never expect to get a CONNECTION_CLOSE in response to a CONNECTION_CLOSE?  Or am I misreading the text about draining?

On Wed, Jun 5, 2019 at 10:08 AM Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
CONNECTION_CLOSE can be response to CONNECTION_CLOSE and that response can contain other frames.. So technically you can ACK a connection close.

Doing so might actually make sense if the original closer take that as a signal to drain faster.

On 5 June 2019 at 16.00.16, Nick Banks (nibanks=[email-masked]<mailto:nibanks=[email-masked]>) wrote:
Ian,

Thank,
- Nick

From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Ian Swett
Sent: Wednesday, June 5, 2019 6:48 AM
To: Jiuhai Zhang <[email-masked]<mailto:[email-masked]>>
Cc: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Is CONNECTION_CLOSE frame Ack-eliciting？

From: https://tools.ietf..org/html/draft-ietf-quic-transport-20#section-10.1<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-transport-20%23section-10.1&data=02%7C01%7Cnibanks%40microsoft.com%7Cb0b15118b76a4bc6b06e08d6e9d27df3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953487602332765&sdata=IhmvEOZRBJf%2FD7y5e5FZS0sO6yqe9Ztu2nK7%2FUp3voU%3D&reserved=0>

On Wed, Jun 5, 2019 at 8:04 AM Jiuhai Zhang <[email-masked]<mailto:[email-masked]>> wrote:
In https://tools.ietf.org/html/draft-ietf-quic-recovery-20#section-2<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-quic-recovery-20%23section-2&data=02%7C01%7Cnibanks%40microsoft.com%7Cb0b15118b76a4bc6b06e08d6e9d27df3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953487602342762&sdata=tYID8lqmH2DVpOjm07pmSCyKOOBlBDDijlTm8FLQ2G0%3D&reserved=0>

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Jun 2019 12:38:21 -0400

In the past, having an ACK bundled with CONNECTION_CLOSE has been useful to
debug what the peer did and did not receive as of the end of the connection.

Agreed on sending CONNECTION_CLOSE in response to CONNECTION_CLOSE being
mostly useless.  I believe some wanted to allow it because if they received
a CONNECTION_CLOSE in response to their CONNECTION_CLOSE, they could remove
all state for the connection, including the saved CONNECTION_CLOSE packet?

On Wed, Jun 5, 2019 at 12:25 PM Marten Seemann <[email-masked]>
wrote:

> On Thu, Jun 6, 2019 at 12:03 AM Mikkel Fahnøe Jørgensen <

---

From: Christian Huitema <[email-masked]>
Date: Wed, 5 Jun 2019 13:40:30 -0700

On 6/5/2019 9:38 AM, Ian Swett wrote:

> In the past, having an ACK bundled with CONNECTION_CLOSE has been
> useful to debug what the peer did and did not receive as of the end of
> the connection.
>
> Agreed on sending CONNECTION_CLOSE in response to CONNECTION_CLOSE
> being mostly useless.  I believe some wanted to allow it because if
> they received a CONNECTION_CLOSE in response to their
> CONNECTION_CLOSE, they could remove all state for the connection,
> including the saved CONNECTION_CLOSE packet?

Most of the arguments I heard one way or the other are based on power
management. Sending one more message requires waking up the radio, etc.,
which consumes significant power on a mobile device. Replying with a
connection close allows the peer to immediately exit the waiting state,
which might save some power -- but probably not as much as what is saved
on the client with not sending an extra message. If the client does send
a connection close, the server has to keep the state for 3*RTO, which is
not a very big deal.

-- Christian Huitema

---

From: Mike Bishop <[email-masked]>
Date: Wed, 5 Jun 2019 21:13:26 +0000

But IIUC, waking the radio happens when receiving the message in the first place.  Sending vs. receiving isn't that different.  So that's an excellent argument for having a silent close on both sides, but not so compelling with regard to whether a CONNECTION_CLOSE can be used in response to a CONNECTION_CLOSE.

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Christian Huitema
Sent: Wednesday, June 5, 2019 1:41 PM
To: Ian Swett <ianswett=[email-masked]>; Marten Seemann <[email-masked]>
Cc: Nick Banks <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; IETF QUIC WG <[email-masked]>; Jiuhai Zhang <[email-masked]>
Subject: Re: Is CONNECTION_CLOSE frame Ack-eliciting？

> including the saved CONNECTION_CLOSE packet?

Most of the arguments I heard one way or the other are based on power management. Sending one more message requires waking up the radio, etc., which consumes significant power on a mobile device. Replying with a connection close allows the peer to immediately exit the waiting state, which might save some power -- but probably not as much as what is saved on the client with not sending an extra message. If the client does send a connection close, the server has to keep the state for 3*RTO, which is not a very big deal.

-- Christian Huitema

---

From: "Philipp S. Tiesel" <[email-masked]>
Date: Thu, 6 Jun 2019 09:55:35 +0200

> On 5. Jun 2019, at 18:38, Nick Banks <nibanks=[email-masked]> wrote:
> 
> On the topic of sending ACKs with CONNECTION_CLOSE, I am more referring to bundling ACKs with the initialCONNECTION_CLOSE. I agree, that sending ACKs with the responding CONNECTION_CLOSE doesn’t do much.

It can still be useful on short lived connections to update the congestion window cache as well as loss and bandwidth estimates.

AVE!
   Philipp S. Tiesel

-- 
Philipp S. Tiesel 
https://philipp.tiesel.net/

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 06 Jun 2019 10:32:58 +0100

On 06/06/2019, 08:55, Philipp S. Tiesel wrote:
>
>
>> On 5. Jun 2019, at 18:38, Nick Banks 
>> <nibanks=[email-masked] 
>> <mailto:nibanks=[email-masked]>> wrote:
>>
>> There is no absolute requirement that you maintain the minimal state 
>> to generate the CONNECTION_CLOSE. Personally, I feel a MAY is missing 
>> in that sentence. Additionally, I feel that an endpoint SHOULD (not a 
>> MUST because it’s unenforceable) respond with an immediate 
>> CONNECTION_CLOSE of its own, so the peer can use that as a signal of 
>> acknowledgement and immediately go away, without waiting the full 
>> draining period. But, I know I’m not in the majority with that opinion.
>> On the topic of sending ACKs with CONNECTION_CLOSE, I am more 
>> referring to bundling ACKs with the*initial*CONNECTION_CLOSE. I 
>> agree, that sending ACKs with the responding CONNECTION_CLOSE doesn’t 
>> do much.
>
> It can still be useful on short lived connections to update the 
> congestion window cache as well as loss and bandwidth estimates.
>
> AVE!
>    Philipp S. Tiesel
>
> -- 
> Philipp S. Tiesel
I'd agree this can be really useful and there may be more transport 
characteristics that could usefully be cached.

Gorry

---

From: Roland Zink <[email-masked]>
Date: Thu, 6 Jun 2019 15:57:42 +0200

In LTE networks mobile devices RRC (Radio Resource Control) disconnect 
after a few seconds from the mobile network. They only connect back from 
time to time for example when they enter another tracking area. When 
some data needs to be send to it then those device needs first to be 
paged within a tracking area and only after they reconnect the data can 
be send. Often server idle timeouts are longer than the RRC disconnect 
timeout. This means when the server closes the connection and send 
something to do so then this requires all the signalling to bring the 
device online again just for a TCP FIN.

For TCP there are middle boxes which optimize those in mobile networks. 
Some numbers:

Devices stay 5-10% shorter in connected (radio active) state

Number of paging request because of data to be delivered is reduced by 
10-20%

There is an Android App available in the PlayStore named Devmon which 
shows in a graph when such a thing happens. It needs root access and 
tcpdump to determine if a TCP FIN was received, see 
https://play.google.com/store/apps/details?id=com.flashnetworks.netmon. 
<https://play.google.com/store/apps/details?id=com.flashnetworks.netmon&hl=de>

For QUIC the silent connection close after an idle timeout avoids that 
overhead for mobile devices and networks.

Regards,

Roland