Subject: Re: Stream vs. connection

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jan 2017 17:18:56 +1100

Performing thread-necromancy...

On 11 December 2016 at 09:31, Jim Roskind <[email-masked]> wrote:
> PADDING frames serve different functions, and should not interact with
> limitations/meanings on PING frames.

Is the implication that a PING frame has to be immediately
acknowledged, whereas other frame types can be acknowledged as needed?

PING has two purposes in HTTP/2:
 - check that the link is still usable (it's cheaper than a full handshake)
 - measure end-to-end latency

My understanding was that the use of explicit timestamps in
acknowledgments would have obviated any need to respond aggressively
to pings as far as timing goes.  That only leaves the former use, and
isn't that addressed by PADDING as well?

PING (and PADDING) are one of the ways that implementations can be
abused, so we need to be very crisp about motivating their existence.

PADDING has a fairly simple and elegant design - read zero, ignore the
remainder of the packet after maybe checking it - and a direct
security benefit in terms of traffic analysis resistance.

PING seems less well motivated given that it could be reliably
replicated by a packet containing all zeros (i.e., PADDING).

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 3 Jan 2017 08:32:34 -0800

On Monday, January 2, 2017 10:19 PM, Martin Thomson wrote:
> ...

It is also a very simple way to implement Tail Loss Probe.

> ... 

But then there is the "ACK of ACK" problem. Each of these empty packets consumes a sequence number. The ACK of the empty packet also consumes a sequence number. If the policy is to ACK all pending packets, then QUIC generates an unending ACK ping pong. To stop the ping pong the implementation needs to distinguish between packets that need an ACK, and packets for which the ACK can wait. In the absence of data, PING is an easy way to do that.

-- Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Wed, 4 Jan 2017 09:04:55 +1100

> But then there is the "ACK of ACK" problem. Each of these empty packets consumes a sequence number. The ACK of the empty packet also consumes a sequence number. If the policy is to ACK all pending…

That's a good reason.  So which of the frame types demand
acknowledgment?  clearly not ACK...

STREAM - yes
ACK - no
STOP_WAITING - ?
WINDOW_UPDATE - ?
BLOCKED - ?
RST_STREAM - ?
PADDING - no
PING - always
CONNECTION_CLOSE - ?
GOAWAY - ?

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 3 Jan 2017 14:39:20 -0800

On Tuesday, January 3, 2017 2:05 PM, Martin Thomson wrote:

>> But then there is the "ACK of ACK" problem. Each of these empty packets consumes a sequence
>> number. The ACK of the empty packet also consumes a sequence number. 
>> If the policy is to ACK all pending packets, then QUIC generates an unending ACK ping pong. 
>> To stop the ping pong the implementation needs to distinguish between packets that need 
>> an ACK, and packets for which the ACK can wait. In the absence of data, PING is an easy 
>> way to do that.
>

Hum. Before answering the question, let's note that it would be nice if this was somehow discussed in a revision of draft-ietf-quic-transport.

Now, my personal answer to the question would be, "if a packet contains stream specific frames, it should be acknowledged at the first opportunity." But that leaves issues like connection close or windows update. The real test is, what would the sender do if the packet was known lost? ACK and STOP WAITING don't need to be retransmitted, because they are superseded by the next ACK or STOP WAITING frame. CONNECTION_CLOSE would typically be retransmitted 2 or 3 times, because it is kind of desirable to ensure that both ends have the same view of the connection. Same for GO AWAY. BLOCKED is informational. So, my version of your table would be:

STREAM - yes
ACK - no
STOP_WAITING - no
WINDOW_UPDATE - yes
BLOCKED - no
RST_STREAM - yes
PADDING - no
PING - always
CONNECTION_CLOSE - yes
GOAWAY - yes

-- Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 15:29:57 -0800

> It is also a very simple way to implement Tail Loss Probe.

Additionally, in the current Google implementation, it's also used to
measure reachability to the peer over a new network path. A PING frame is
quite useful, and I'd argue against removing it.

> But then there is the "ACK of ACK" problem. Each of these empty packets
> consumes a sequence number. The ACK of the empty packet also consumes a
> sequence number. If the policy is to ACK all pending packets, then QUIC
> generates an unending ACK ping pong. To stop the ping pong the
> implementation needs to distinguish between packets that need an ACK, and
> packets for which the ACK can wait. In the absence of data, PING is an easy
> way to do that.

Padding is primarily used in the CHLO and SHLO packets, to pad the packets
to a MaxPacketSize. QUIC currently relies on the use of PADDING to detect
whether MaxPacketSize is usable for the rest of this connection... if the
packet size is too large for the network to transmit, the handshake fails,
and the connection does not proceed. An implementation could consider doing
other PMTU size checks at this point, using PADDING to control packet size.

I'd argue against combining them since the acking semantics are different.
PADDING frames by themselves currently do not generate ACKs, while PINGs do.
(This might be what Christian is saying.)

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 15:41:28 -0800

> acknowledgment?  clearly not ACK...

The current strategy is described in the draft, in Section 7 (Packetization
and Reliability):

   A receiver acknowledges receipt of a received packet by sending one
   or more ACK frames containing the packet number of the received
   packet.  To avoid perpetual acking between endpoints, a receiver MUST
   NOT generate an ack in response to every packet containing only ACK
   frames.  However, since it is possible that an endpoint sends only
   packets containing ACK frame (or other non-retransmittable frames),

   the receiving peer MAY send an ACK frame after a reasonable number
   (currently 20) of such packets have been received.

Section 7 lays out what frames MUST and MUST NOT be retransmitted, which
defines which frames are retransmittable. PADDING frames are
non-retransmittable (as explained earlier in Section 7), and therefore do
not evoke a timely ack (since the receiver knows that the sender won't
retrasmit this frame.) PING frames are considered retransmittable, and
therefore evoke an ack, perhaps with a delayed ack timer.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 16:07:17 -0800

Just adding a few thoughts while we're still speaking to dead threads
(since I was on a blackout most of December):

- ACK frame: Here the decision was made that data is ack’ed on a per-packet
>> basis (instead on per frame-basis). I believe this decision is fine (didn’t
>> actually further think about it so far). However, as these information are
>> clearly aligned to packets, I think they should be part of the QUIC packet
>> header. Here I assume that it does not make sense to send multiple ACK
>> frames in one packet because they could always be combined to one (to save
>> space). In this case it would be sufficient to have one flag in the general
>> QUIC packet header that indicates if these information are present or not.
>>
>
> As noted in the Issue Mike opened, there is currently no encrypted QUIC
> packet header, but it could be added.
>

I'm not sure what the value of putting the ack in the packet header. Such a
design would require consuming a flag bit, which would make it visible
outside the encryption envelope that the packet contains an ack (this may
not be important information, but it's certainly a difference from the
current scheme.) Alternatively, you could have a "private flags" field
inside the encrypted packet, but that splits the header into a publicly
visible part and an encrypted part. This can be done (and it was the format
with the QUIC packet header a while ago), but I'm not convinced that this
complexity is warranted.

SCTP has SACK frames, much like QUIC does, which makes it convenient to
evolve this frame separately, independent of the packet format. (See proposal
for NR-SACKs
<https://tools.ietf.org/html/draft-natarajan-tsvwg-sctp-nrsack-08> which
evolves SCTP's SACK frame.)

- STOP_WAITING frame: At the first reading I didn’t really understand what
>> this is needed for because I didn’t fully internalize that packets are not
>> retransmitted (btw. that could be stated more clearly in the intro!). I
>> understand that this is inevitable connection to how ack'ing is done and
>> therefore I really don’t understand why this is a separate frame at all.
>> For me that could simply be one field in the ACK frame basically always
>> indicating the last ACK information received.
>>
>
> As noted in Issue #66(https://github.com/quicwg/base-drafts/issues/66),
> it is not really necessary anymore.  If not, merging into the ack frame as
> you mentioned is a nice simplification, but I'd rather try removing it
> first if possible.
>

I'll note in addition that STOP_WAITING used to be a part of the ACK frame
in QUIC. Bear in mind that the information in ACK and STOP_WAITING are at
different ends of the connection -- the sender of data sends STOP_WAITING,
and the receiver of data sends ACKs.  Combining them made our code more
convoluted, since the treatment of the two pieces of information is
basically done by completely different parts of the code: STOP_WAITING is
for the part of the code that handles received packets, while the ACK
information is for the part of the code that handles sent packets. None of
this is to say that combining them is a bad idea; they're not related to
each other in the packet, and it seemed reasonable to separate them out
into separate frames.

- jana

One more general comment here: I’m also not convince that the ACK scheme
>> chosen (and basically aligned to the TCP ACK scheme is the right choice
>> here). Given that you don’t retransmit, wouldn't a NACK scheme make more
>> sense…? Meaning that you indicate the highest (un-ordered) packet number
>> seen as well as all packet numbers not seen since the packet number
>> announced by the last STOP_WAITING value. Did you consider that previously?
>>
>
> QUIC@Google originally implemented that.  In many ways they're mostly
> equivalent, but it's critical to have the ack contain the smallest packet
> being acked, otherwise the receiver can't unilaterally stop acking old
> packets without receiving a stop waiting that raises the least unacked.
> Once you have a largest acked and a set of nack ranges and a smallest
> acked, you have something that looks more like ack ranges than nack ranges,
> but one could word it as nack ranges.
>
> The new ack block approach is much simpler to implement, and removed quite
> a few edge cases.  I wrote a longer doc on why this change was made, but
> I'm unable to find it at the moment.  I'll try to find it as background
> information.
>
>
>>
>> - PING frame: Having a whole frame for this really, really seems like
>> overhead, while you only need on bit in the packet header and no
>> frames/steam payload data are send at all… Btw. the reaction to this signal
>> is not specified: is the receiver supposed to send a ping back or a
>> different packet or nothing at all?
>>
>
> The overhead is pretty small, since it's just a byte.  The reaction
> specified in 6.8 is: "The receiver of a PING frame simply needs to ACK the
>    packet containing this frame."  Would it be clearer as "only" instead
> of simply?
>
>
>> - CONNECTION_CLOSE and GO_AWAY: These two information bits are clearly
>> per-connection. I also don’t really see a need for a CONNECTION_CLOSE and a
>> PUBLIC_RESET. I think there can be a way to combine these signals given we
>> have a mechanism for integrity checking.
>>
>
> A connection close is encrypted and a public reset is not.  If we have
> non-spoofable public resets and we end up wanting to notify the path of the
> close, I can imagine merging them into one packet, though I would want to
> continue to encrypt connection close reason and details.  In which case,
> we'd still need both.
>
> Mirja
>>
>>
>>
>

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 16:15:54 -0800

On Tue, Jan 3, 2017 at 2:39 PM, Christian Huitema <[email-masked]>
wrote:

> consumes a sequence
> Hum. Before answering the question, let's note that it would be nice if
> this was somehow discussed in a revision of draft-ietf-quic-transport.
>

This is discussed in Section 7 on Packetization and Reliability.

> Now, my personal answer to the question would be, "if a packet contains
> stream specific frames, it should be acknowledged at the first
> opportunity." But that leaves issues like connection close or windows
> update. The real test is, what would the sender do if the packet was known
> lost? ACK and STOP WAITING don't need to be retransmitted, because they are
> superseded by the next ACK or STOP WAITING frame. CONNECTION_CLOSE would
> typically be retransmitted 2 or 3 times, because it is kind of desirable to
> ensure that both ends have the same view of the connection. Same for GO
> AWAY. BLOCKED is informational. So, my version of your table would be:
>
> STREAM - yes
> ACK - no
> STOP_WAITING - no
> WINDOW_UPDATE - yes
> BLOCKED - no
> RST_STREAM - yes
> PADDING - no
> PING - always
> CONNECTION_CLOSE - yes
> GOAWAY - yes
>

I missed BLOCKED in the list in Section 7 (I'll add it), but your
inferences are spot on -- QUIC acks frames that are "retransmittable". In
addition, there's a 20-packet count after which the receiver MAY send an
ack frame to help the peer update RTT as well as clean out sent packet
state (such as packet sent time).

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 3 Jan 2017 16:21:25 -0800

On Tuesday, January 3, 2017 3:41 PM, Jana Iyengar wrote:
> The current strategy is described in the draft, in Section 7 (Packetization and Reliability):
>

Maybe this should be updated to "... MUST NOT generate an ack in response to every packet containing only ACK or STOP WAITING frames." It is fairly natural for applications to generate ACK and STOP WAITING at the same time, so you can easily build a perpetual acking scenario with these two frame types. 

-- Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 16:35:24 -0800

On Tue, Jan 3, 2017 at 4:21 PM, Christian Huitema <[email-masked]>
wrote:

> Maybe this should be updated to "... MUST NOT generate an ack in response
> to every packet containing only ACK or STOP WAITING frames." It is fairly
> natural for applications to generate ACK and STOP WAITING at the same time,
> so you can easily build a perpetual acking scenario with these two frame
> types.

I think we could state more broadly: "... MUST NOT generate an ack in
response to every packet containing only non-retransmittable frames." What
do you think?

---

From: Martin Thomson <[email-masked]>
Date: Wed, 4 Jan 2017 11:36:23 +1100

> response to every packet containing only non-retransmittable frames." What

If we clearly designate frames as retransmittable or not, sure.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 3 Jan 2017 16:39:18 -0800

> If we clearly designate frames as retransmittable or not, sure.

We do in Section 7.

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 4 Jan 2017 01:44:55 +0100

Hi Jana,

thanks for the reply. Please see inline.

> Am 04.01.2017 um 01:07 schrieb Jana Iyengar <[email-masked]>:
> 
> Just adding a few thoughts while we're still speaking to dead threads (since I was on a blackout most of December):
> 
> - ACK frame: Here the decision was made that data is ack’ed on a per-packet basis (instead on per frame-basis). I believe this decision is fine (didn’t actually further think about it so far). However, as these information are clearly aligned to packets, I think they should be part of the QUIC packet header. Here I assume that it does not make sense to send multiple ACK frames in one packet because they could always be combined to one (to save space). In this case it would be sufficient to have one flag in the general QUIC packet header that indicates if these information are present or not.
> 
> As noted in the Issue Mike opened, there is currently no encrypted QUIC packet header, but it could be added. 
> 
> I'm not sure what the value of putting the ack in the packet header. Such a design would require consuming a flag bit, which would make it visible outside the encryption envelope that the packet contains an ack (this may not be important information, but it's certainly a difference from the current scheme.) Alternatively, you could have a "private flags" field inside the encrypted packet, but that splits the header into a publicly visible part and an encrypted part.

Yes, that was the assumption I’ve been taken. I was assuming that there is an explicit encryption boundary rather than an implicit one between packet and stream information.

> This can be done (and it was the format with the QUIC packet header a while ago), but I'm not convinced that this complexity is warranted.

Can you further explain this complexity. Encrypting some header information at the beginning of a packet shouldn’t be more complicated that encrypting a frame, no?

> 
> SCTP has SACK frames, much like QUIC does, which makes it convenient to evolve this frame separately, independent of the packet format. (See proposal for NR-SACKs which evolves SCTP's SACK frame.)

I don’t think this is comparable because SCTP acks chunks while quic acks on a per packet basis; this it why it would be much more natural for me to have these information on a per packet level.

> 
> - STOP_WAITING frame: At the first reading I didn’t really understand what this is needed for because I didn’t fully internalize that packets are not retransmitted (btw. that could be stated more clearly in the intro!). I understand that this is inevitable connection to how ack'ing is done and therefore I really don’t understand why this is a separate frame at all. For me that could simply be one field in the ACK frame basically always indicating the last ACK information received.
> 
> As noted in Issue #66(https://github.com/quicwg/base-drafts/issues/66), it is not really necessary anymore.  If not, merging into the ack frame as you mentioned is a nice simplification, but I'd rather try removing it first if possible.
> 
> I'll note in addition that STOP_WAITING used to be a part of the ACK frame in QUIC. Bear in mind that the information in ACK and STOP_WAITING are at different ends of the connection -- the sender of data sends STOP_WAITING, and the receiver of data sends ACKs.  Combining them made our code more convoluted, since the treatment of the two pieces of information is basically done by completely different parts of the code: STOP_WAITING is for the part of the code that handles received packets, while the ACK information is for the part of the code that handles sent packets. None of this is to say that combining them is a bad idea; they're not related to each other in the packet, and it seemed reasonable to separate them out into separate frames.

This implementation argument is also not really clear to me but I’m assuming that this depends on other implementation choices you’ve probably made. I guess there are multiple ways to implement this. My understanding is that when you receive a packet you have to do certain things: when you receive a packet with data frames, you may decide to send an ack frame. When you receive an ACK frame you should probably update the counter of the highest seen ACK and potentially decide to generate an STOP_WAITING frame. However, the actual generation of these frames can be done on the flight when you send out the packet; or you just copy the latest ack counter in the encrypted part of the header of the next packet you send out if ACK information and STOP_WAITING would part of the packet header.  

Mirja 

> 
> - jana
> 
> One more general comment here: I’m also not convince that the ACK scheme chosen (and basically aligned to the TCP ACK scheme is the right choice here). Given that you don’t retransmit, wouldn't a NACK scheme make more sense…? Meaning that you indicate the highest (un-ordered) packet number seen as well as all packet numbers not seen since the packet number announced by the last STOP_WAITING value. Did you consider that previously?
> 
> QUIC@Google originally implemented that.  In many ways they're mostly equivalent, but it's critical to have the ack contain the smallest packet being acked, otherwise the receiver can't unilaterally stop acking old packets without receiving a stop waiting that raises the least unacked.  Once you have a largest acked and a set of nack ranges and a smallest acked, you have something that looks more like ack ranges than nack ranges, but one could word it as nack ranges.
> 
> The new ack block approach is much simpler to implement, and removed quite a few edge cases.  I wrote a longer doc on why this change was made, but I'm unable to find it at the moment.  I'll try to find it as background information.
>  
> 
> - PING frame: Having a whole frame for this really, really seems like overhead, while you only need on bit in the packet header and no frames/steam payload data are send at all… Btw. the reaction to this signal is not specified: is the receiver supposed to send a ping back or a different packet or nothing at all?
> 
> The overhead is pretty small, since it's just a byte.  The reaction specified in 6.8 is: "The receiver of a PING frame simply needs to ACK the
>    packet containing this frame."  Would it be clearer as "only" instead of simply?
> 
> 
> - CONNECTION_CLOSE and GO_AWAY: These two information bits are clearly per-connection. I also don’t really see a need for a CONNECTION_CLOSE and a PUBLIC_RESET. I think there can be a way to combine these signals given we have a mechanism for integrity checking.
> 
> A connection close is encrypted and a public reset is not.  If we have non-spoofable public resets and we end up wanting to notify the path of the close, I can imagine merging them into one packet, though I would want to continue to encrypt connection close reason and details.  In which case, we'd still need both.
> 
> Mirja
> 
> 
> 
>

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 3 Jan 2017 17:39:41 -0800

On Tuesday, January 3, 2017 4:35 PM, Jana Iyengar wrote:
> On Tue, Jan 3, 2017 at 4:21 PM, Christian Huitema <mailto:[email-masked]> wrote:
> ...
>> Maybe this should be updated to "... MUST NOT generate an ack in response to 
>> every packet containing only ACK or STOP WAITING frames." It is fairly natural for 
>> applications to generate ACK and STOP WAITING at the same time, so you can easily 
>> build a perpetual acking scenario with these two frame types.
>
> I think we could state more broadly: "... MUST NOT generate an ack in response to every 
> packet containing only non-retransmittable frames." What do you think?

Works for me.

-- Christian Huitema