Subject: FIN + RST behavior

From: Subodh Iyengar <[email-masked]>
Date: Wed, 22 Mar 2017 22:37:02 +0000

There might be some strange cases around RSTs and FINs which I would love more clarity on.

Let's say that the following actions happen:
1) Client initiates a stream to the server and sends some data
2) Server sends a FIN to the client which the client receives. The client and server are both half closed (remote and local respectively).
3) After some time the client sends a FIN to the server. So it's waiting for the FIN and the ack of all the data before closing the stream.
4) Server sends a RST which races with the client sending the FIN.
5) Client gets a RST

>From the draft I think it's ambiguous on what the client is supposed to do

"An endpoint that receives a RST_STREAM frame (and which has not sent a FIN or a RST_STREAM) MUST immediately respond with a RST_STREAM frame, and MUST NOT send any more data on the stream"

So clearly a client should not send a RST stream in response

However:
"When an endpoint sends a RST_STREAM frame, data outstanding on that stream SHOULD NOT be retransmitted, since subsequent data on this stream is expected to not be delivered by the receiver."

If the client decides to honor both of these requirements I believe it will not retransmit the FIN. So if the FIN gets lost the client will have no idea whether the server has seen the FIN and reduced its
number of streams counting towards the concurrency limit so that the client can open up a new stream. It would need to track the ACK of the ACK of the RST stream that the server sends back.

Am I missing another requirement that prevents this from being an issue or could "not sent a FIN" mean that "not sent a FIN that has been ACKed".

Subodh

---

From: Mike Bishop <[email-masked]>
Date: Thu, 23 Mar 2017 00:44:37 +0000

I don’t see these as in conflict.  If the client receives a RST when it has already sent a FIN, it's not subject to the MUST.  It may, of course, opt to send a RST and abort retransmission.  Or, it may choose to just let the sent data be, but with the obligation to retransmit it.  Its obligation is to deliver the final offset, and all data up to that offset if the offset is in a FIN.

(Note that pulling in DISINTEREST or whatever we call it would hopefully clarify this, since it removes the requirement to generate a RST_STREAM when receiving one.)

Sent from my Windows 10 phone

From: Subodh Iyengar<mailto:[email-masked]>
Sent: Wednesday, March 22, 2017 6:37 PM
To: [email-masked]<mailto:[email-masked]>
Subject: FIN + RST behavior

However:

Subodh

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Mar 2017 11:45:31 +1100

This is just an error.  Both endpoints need to end their stream in one
way or other.  The text about not sending a RST_STREAM needs to take
into account the fact that any FIN or RST_STREAM that was previously
sent ALSO needs to be acknowledged.

A sender needs to wait for acknowledgment before transitioning to new
states.  Receivers just process and acknowledge.

I agree that this is poorly documented.  Here's the short version:

Each endpoint has to end the stream one way or other.  This
establishes the final data offset for the stream, which is necessary
for connection flow control accounting.  A stream can be ended by
either sending a STREAM frame with the FIN flag set, or a RST_STREAM
frame.  A stream isn't ended in the sending direction until a packet
containing one of those two frames has been acknowledged.  An endpoint
MAY choose to switch from sending STREAM frames to sending RST_STREAM
frames at any time.

For the long version, you will have to wait until we get around to
fixing https://github.com/quicwg/base-drafts/issues/413

On 23 March 2017 at 09:37, Subodh Iyengar <[email-masked]> wrote:
> 2) Server sends a FIN to the client which the client receives. The client
> and server are both half closed (remote and local respectively).
> 3) After some time the client sends a FIN to the server. So it's waiting for
> the FIN and the ack of all the data before closing the stream.
> "An endpoint that receives a RST_STREAM frame (and which has not sent a FIN
> or a RST_STREAM) MUST immediately respond with a RST_STREAM frame, and MUST
> NOT send any more data on the stream"
>
> "When an endpoint sends a RST_STREAM frame, data outstanding on that stream
> SHOULD NOT be retransmitted, since subsequent data on this stream is
> expected to not be delivered by the receiver."
>
> If the client decides to honor both of these requirements I believe it will
> not retransmit the FIN. So if the FIN gets lost the client will have no idea
> whether the server has seen the FIN and reduced its
> number of streams counting towards the concurrency limit so that the client
> can open up a new stream. It would need to track the ACK of the ACK of the
> RST stream that the server sends back.
>
> Am I missing another requirement that prevents this from being an issue or
> could "not sent a FIN" mean that "not sent a FIN that has been ACKed".
>
> Subodh
>

---

From: Subodh Iyengar <[email-masked]>
Date: Thu, 23 Mar 2017 04:10:21 +0000

@Mike Bishop<mailto:[email-masked]>
> I don’t see these as in conflict.  If the client receives a RST when it has already sent a FIN, it's not subject to the MUST.  It may, of course, opt to send a RST and abort retransmission.  Or, it…

To confirm that I'm interpreting you correctly I'm going to try and rephrase that a bit: The intent of the draft is that if the client receives the RST stream and it has already sent a FIN, it should ignore the RST, and perform retransmission of all the packets as it usually would, even if the server is no longer interested in it. If it wanted to not retransmit the packets, it should send a RST of it's own.

@Martin Thomson<mailto:[email-masked]>
states.  Receivers just process and acknowledge.
Ya that's definitely a given. The issue here I was asking about was what if there would never be an ACK for the FIN because the client decided to abort the retransmissions.

>  Both endpoints need to end their stream in one
way or other.
That's definitely true but it seems like closing a stream which is not initiated by you has a different behavior with respect to closing streams initiated by you because you need to account for
stream concurrency limit and connection flow control window differently.

> Here's the short version:
Ya this is definitely a very simplistic description of what needs to be done when a connection needs to be closed and doesn't account for how to keep the client + server in sync for the concurrency limit
and connection flow control window which is also very important. My question was actually about the latter.

This stuff gets hairy fast.

For example in a clean close:
1) client sends FIN to server, sever gets FIN
2) Server sends ACK for the client's FIN (client is now half closed local)
2) server sends data and FIN (Server doesn't change state, it waits for the ACK of all the bytes + FIN)
3) client gets all data and FIN and then ACKs all of it.
4) the client ACK for the server FIN gets lost

>From the client's perspective it sent out all the ACKs for the FIN and doesn't know whether they got lost.
How does the client know when the server has released a slot in it's concurrency limit? The server can't immediately release this when it sends out the FIN because it still must keep state until it receives all the
ACKs for the stream. I think the client need to keep track of the ACK of the ACKs of all the server sent bytes in the stream, which would be a bit sad.

Subodh

________________________________
From: Martin Thomson <[email-masked]>
Sent: Wednesday, March 22, 2017 5:45:31 PM
To: Subodh Iyengar
Cc: [email-masked]
Subject: Re: FIN + RST behavior

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Mar 2017 22:04:46 +1100

On 23 March 2017 at 15:10, Subodh Iyengar <[email-masked]> wrote:
> How does the client know when the server has released a slot in it's
> concurrency limit? The server can't immediately release this when it sends
> out the FIN because it still must keep state until it receives all the
> ACKs for the stream. I think the client need to keep track of the ACK of the
> ACKs of all the server sent bytes in the stream, which would be a bit sad.

The client can consider its own resources to be available for use by
the server as soon as it receives the FIN and processes it.  If this
were a server resource, then you are right, it's not that simple
because the client can't assume that the server has released state
yet.

---

From: Subodh Iyengar <[email-masked]>
Date: Thu, 23 Mar 2017 17:09:11 +0000

Ya in the case of the previous example, it is a server resource since it's a client initiated stream.
@Martin Thomson<mailto:[email-masked]> I'm starting to like your proposal of unidirectional streams more and more :)

Subodh

________________________________
From: QUIC <[email-masked]> on behalf of Martin Thomson <[email-masked]>
Sent: Thursday, March 23, 2017 4:04:46 AM
To: Subodh Iyengar
Cc: [email-masked]
Subject: Re: FIN + RST behavior

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 23 Mar 2017 18:27:29 +0000

Concurrency limit is about the amount of local resources a remote peer can consume.  It is not about how much local resources can be consumed by the local application - these limits can be enforced without any help from the protocol.

So concurrency limit is really the number of concurrent in-progress streams that a peer is willing to receive.  This maps well with unidirectional streams, but this is not a requirement.  To be put precisely, the concurrency limit can be defined as "the max number of streams with non-zero unACKed bytes sent by that peer".  This means, peers can negotiate separate concurrency limits.

-          Igor

From: Subodh Iyengar [mailto:[email-masked]]
Sent: Thursday, March 23, 2017 1:09 PM
To: Martin Thomson <[email-masked]>
Cc: [email-masked]
Subject: Re: FIN + RST behavior

Subodh

________________________________
From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Martin Thomson <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, March 23, 2017 4:04:46 AM
To: Subodh Iyengar
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: FIN + RST behavior

---

From: Subodh Iyengar <[email-masked]>
Date: Fri, 24 Mar 2017 04:28:29 +0000

@Lubashev, Igor<mailto:[email-masked]>

<mailto:[email-masked]>>Concurrency limit is about the amount of local resources a remote peer can consume.

Let's say that for every stream there is an initiator (who is subject to stream limit) and a peer.
Aren't the retransmission buffers which the peer has to hold onto until the initiator ACKs them also a part of the local resources here?

An initiator who doesn't ACK any bytes from the server forces a peer to keep state for it. However it could send it's own FIN which is ACKed normally.

My question generally here is that with the current state machine seems that it's difficult for both peers to get a consistent view of what each side thinks that the

open peer streams have. The peer should only release the retransmission buffers when it knows all of the data + FIN has been ACKed, and I think that's when it should release

the stream limit. However the initiator needs to know when this happens so it would need to keep track of the ACKs for the ACKs it sent. Unidirectional streams seem to make this simpler.

>To be put precisely, the concurrency limit can be defined as “the max number of streams with non-zero unACKed bytes sent by that peer”.

I'm not sure I understand your meaning here or maybe I'm confused by what you consider as the peer.
Can't the initiator just send a FIN and get an ACK for the FIN, however never ACK the peer's frames? In this case the initiator would have "zero UNACKed bytes" but the peer would not.

Subodh

________________________________
From: Lubashev, Igor <[email-masked]>
Sent: Thursday, March 23, 2017 11:27:29 AM
To: Subodh Iyengar; Martin Thomson
Cc: [email-masked]
Subject: RE: FIN + RST behavior

Concurrency limit is about the amount of local resources a remote peer can consume.  It is not about how much local resources can be consumed by the local application – these limits can be enforced without any help from the protocol.

So concurrency limit is really the number of concurrent in-progress streams that a peer is willing to receive.  This maps well with unidirectional streams, but this is not a requirement.  To be put precisely, the concurrency limit can be defined as “the max number of streams with non-zero unACKed bytes sent by that peer”.  This means, peers can negotiate separate concurrency limits.

-          Igor

Subodh

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 24 Mar 2017 12:56:41 +0000

I am not thinking about it from the perspective of the initiator of a stream. I am really thinking about a stream having two independent directions. It is the sender of the bytes (on a stream initiated by him or by the peer, does not matter) that must count the stream toward his concurrency limit while there are any unacked sent bytes. (By peer, I mean "an endpoint".)

The management of retransmit buffers is a local concern of the sender. If the buffers grow too large, the local app would not get a chance to write more into them. We are only protecting reassembly buffers here. If we also want to protect a table containing state for active streams, we would need to modify the definition for concurrent streams to "number of streams that this endpoint has sent any data on and still have unacked data or fin, or unacked rst".

- Igor

-----Original Message-----
From: Subodh Iyengar [[email-masked]]
Received: Friday, 24 Mar 2017, 12:28AM
To: Lubashev, Igor [[email-masked]]; Martin Thomson [[email-masked]]
CC: [email-masked] [[email-masked]]
Subject: Re: FIN + RST behavior

Subodh

-          Igor

Subodh

---

From: Subodh Iyengar <[email-masked]>
Date: Sun, 26 Mar 2017 20:27:19 +0000

> We are only protecting reassembly buffers here
Isn't that what the connection flow control window and the stream flow control window is for?

> must count the stream toward his concurrency limit while there are any unacked sent bytes. (By peer, I mean "an endpoint".)

If this is the correct interpretation of how concurrency limit is accounted for in the current draft, then it seems like it provides is no better protection than the flow control window.

The real value of a stream concurrency limit would be to be able to protect the stream state table and to allow senders to protect their write buffers. Sure, senders can slow themselves down, however putting that local limit on yourself is another complexity in addition to the other sender side limits like congestion control, and a tradeoff between throughput and buffer build-up.

It seems like the draft could benefit from is explicit signaling of the available concurrent limit similar to the flow control window which is explicitly signaled. Then we can actually have the peer advertise a concurrent limit whenever it feels like it's ready to receive more streams, which would make this way easier to account for resource usage in the state table and allow the receiver to expand this limit in the future.

Subodh

________________________________
From: Lubashev, Igor <[email-masked]>
Sent: Friday, March 24, 2017 5:56:41 AM
To: [email-masked]; Subodh Iyengar
Cc: [email-masked]
Subject: RE: FIN + RST behavior

- Igor

-----Original Message-----
From: Subodh Iyengar [[email-masked]]
Received: Friday, 24 Mar 2017, 12:28AM
To: Lubashev, Igor [[email-masked]]; Martin Thomson [[email-masked]]
CC: [email-masked] [[email-masked]]
Subject: Re: FIN + RST behavior

Subodh

-          Igor

Subodh

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 27 Mar 2017 04:32:05 +0000

For the record, I was not trying to describe my understanding of the current stream limit, just my thinking of what it should be.

> Isn't that what the connection flow control window and the stream flow control window is for?

You are correct that connection flow control window protects the aggregate amount of data committed to the reassembly buffers.  The max number of concurrent streams controls the number of reassembly buffers (supposedly, there could be significant fixed costs per reassembly buffer).

>> while there are any unacked sent bytes.

Yes, I withdraw that alternative and would go with something like "streams that have sent any bytes and (have unacked bytes or RST_STREAM or if there was no STREAM+FIN sent)".

The size or number of write buffers should not be controlled by some static pre-negotiated number.  This is likely to be something that the sender can (and will want to) control dynamically (just like write buffers are controlled dynamically by the kernel under memory pressure).

-          Igor

From: Subodh Iyengar [mailto:[email-masked]]
Sent: Sunday, March 26, 2017 3:27 PM
To: Lubashev, Igor <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: FIN + RST behavior

Subodh

________________________________
From: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>
Sent: Friday, March 24, 2017 5:56:41 AM
To: [email-masked]<mailto:[email-masked]>; Subodh Iyengar
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: FIN + RST behavior

- Igor

-----Original Message-----
From: Subodh Iyengar [[email-masked]]
Received: Friday, 24 Mar 2017, 12:28AM
To: Lubashev, Igor [[email-masked]]; Martin Thomson [[email-masked]]
CC: [email-masked]<mailto:[email-masked]> [[email-masked]]
Subject: Re: FIN + RST behavior

>Concurrency limit is about the amount of local resources a remote peer can consume.

>To be put precisely, the concurrency limit can be defined as "the max number of streams with non-zero unACKed bytes sent by that peer".

Subodh

From: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>

-          Igor

To: Martin Thomson <[email-masked]<mailto:[email-masked]>>

Subodh