Subject: Proposal: Remove Connection Wide Flow Control from QUIC

From: Nick Banks <[email-masked]>
Date: Fri, 8 Mar 2019 20:55:17 +0000

Hey Folks,

After a lengthy discussion on Slack with a few folks I've become convinced that connection flow control is unnecessary to achieve desired over the wire results, causes unnecessary protocol complexity (synchronizing on stream reset) and creates possible deadlocks if the app doesn't behave correctly (won't drain stream A until it receives what it wants on stream B).

The main reason folks seem to think that connection flow control would be useful is DoS protection; to limit the total memory to allocate for one connection. That doesn't seem right to me. A client can open any number of connections. Imposing an overall limit per connection doesn't really work. On the other hand if you only allocate stream flow control if you have enough over all memory (and are draining the data fast enough) then you can better control your memory usage over all connections.

Also, you still have flow control on the number of streams you allow the peer to create as well. That's another connection-wide flow control value. I think it would be an improvement to the protocol if we removed explicit connection wide flow control. If anyone has any other reasons for keeping it, I'd really like to hear them.

Now, I know this is late in the game and not strictly a necessary change. You can still just ignore connection flow control on the sender side and achieve the same thing; so this might not meet the bar. I just wanted to have this discussion and see what other folks think about this. If most folks agree that it's not necessary, I don't think it should be too big a deal to remove both the text and implementation code.

Thanks,
- Nick

---

From: Ted Hardie <[email-masked]>
Date: Fri, 8 Mar 2019 13:41:41 -0800

Hi Nick,

On Fri, Mar 8, 2019 at 12:55 PM Nick Banks <nibanks=
[email-masked]> wrote:

> Hey Folks,
>
>
>
> After a lengthy discussion on Slack with a few folks I’ve become convinced
> that connection flow control is unnecessary to achieve desired over the
> wire results, causes unnecessary protocol complexity (synchronizing on
> stream reset) and creates possible deadlocks if the app doesn’t behave
> correctly (won’t drain stream A until it receives what it wants on stream
> B).
>
>
>
> The main reason folks seem to think that connection flow control would be
> useful is DoS protection; to limit the total memory to allocate for one
> connection. That doesn’t seem right to me. A client can open any number of
> connections.
>

In a distributed system, though, those new connections will have new
connection IDs (and they might also have different five-tuples because of
NAT, but that's not reliable).  As a result, they may go to different
instances .  That ability to use elastic resources to spin up new instances
means that the threat of the client opening a large number of connections
and the threat of the client consuming unbounded resources in a single
connection may be met very differently.

> Imposing an overall limit per connection doesn’t really work. On the other
> hand if you only allocate stream flow control if you have enough over all
> memory (and are draining the data fast enough) then you can better control
> your memory usage over all connections.
>
>
>
This memory management regime is certainly possible, but it may not be the
only one in use even in a single server. There are a lot of cases where
folks would like "fair" allocation across clients or connections, so that
greedy peers can't starve the others. Tracking that and not allocating
stream resources to greedy peers when it looks to be the case could
substitute for an overall connection-level flow control, but doing so
doesn't look substantially simpler than the current design.

> Also, you still have flow control on the number of streams you allow the
> peer to create as well. That’s another connection-wide flow control value.
> I think it would be an improvement to the protocol if we removed explicit
> connection wide flow control. If anyone has any other reasons for keeping
> it, I’d really like to hear them.
>
>
>
> Now, I know this is late in the game and not strictly a necessary change.
> You can still just ignore connection flow control on the sender side and
> achieve the same thing; so this might not meet the bar. I just wanted to
> have this discussion and see what other folks think about this. If most
> folks agree that it’s not necessary, I don’t think it should be too big a
> deal to remove both the text and implementation code.
>
>
>
I have concerns about it going in at the stage.  Folks who are very used to
TCP's flow control will find the idea that there is no connection-level
flow control problematic, and I think re-calculating all of the engineering
trade-offs will be substantially more work than just "take out the text and
the code."  Done is a feature, and it is one I think we need to prioritize.

I very much appreciate your bringing this to the working group from the
slack, as not all of us tracking that closely,

thanks,

Ted

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 8 Mar 2019 22:01:20 +0000

My preference is to keep things as is. We have a solid extension proposal
for non-stream communication, the Datagram extension frame. This provides
opportunity for many diverse use cases.

Lucas

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 8 Mar 2019 17:06:07 -0500

I think Lucas' point is the killer argument for keeping things as is.

All sorts of things can be introduced in v2 that are not manageable be
stream resources and taking credits just to put them back in is not a good
idea.
Otherwise, if it were only streams, I could see some sense in removing it -
since it can be calculated from stream resources just the same, although I
get Ted’s argument.

Mikkel

On 8 March 2019 at 23.01.43, Lucas Pardue ([email-masked])
wrote:

My preference is to keep things as is. We have a solid extension proposal
for non-stream communication, the Datagram extension frame.. This provides
opportunity for many diverse use cases.

Lucas

---

From: Nick Banks <[email-masked]>
Date: Fri, 8 Mar 2019 22:11:46 +0000

I thought there was already a discussion related to FC and datagrams and it was decided that it was untenable because the datagrams on not reliable. Either way, I personally thing that having something separate to track that kind of flow control would be a better solution anyways.

- Nick

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Sent: Friday, March 8, 2019 2:06 PM
To: Ted Hardie <[email-masked]>; Lucas Pardue <[email-masked]>
Cc: Nick Banks <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

All sorts of things can be introduced in v2 that are not manageable be stream resources and taking credits just to put them back in is not a good idea.
Otherwise, if it were only streams, I could see some sense in removing it - since it can be calculated from stream resources just the same, although I get Ted’s argument.

Mikkel

On 8 March 2019 at 23.01.43, Lucas Pardue ([email-masked]<mailto:[email-masked]>) wrote:
My preference is to keep things as is. We have a solid extension proposal for non-stream communication, the Datagram extension frame.. This provides opportunity for many diverse use cases.

Lucas

---

From: Nick Banks <[email-masked]>
Date: Fri, 8 Mar 2019 22:24:28 +0000

Thank for the responses Ted!

I agree that new connections created by clients\attackers might get load balanced by a different server. I’m just saying that imposing a connection wide memory cap does nothing to prevent a client from causing more memory to be allocated. Perhaps the memory from new connections will get allocated on a different server; perhaps not. But the client is still causing you to allocate more memory either way. The connection FC max isn’t limiting that over all. The maximum number of streams FC value can also be used to a similar effect.

Additionally, you will always need to make use of stream FC. That is the way an app can apply back pressure effectively. Connection FC doesn’t really do that. Generally, the stream FC grows as the app is willing to accept more data. So, if the app can continue to keep up with the data it will continue to accept more. The app does have some responsibility in this to decide if a DoS is happening at their layer. I don’t think that memory management purely at the stream layer is more complicated than additionally having a connection FC.

As to your arguments about folks used to TCP being uncomfortable with no connection FC, I’d argue the opposite. As far as FC is concerned, the closest analogy should be that a TCP connection equals a QUIC stream. In my mind, that’s yet another argument against an additional layer of FC.

I also agree that just taking things out might not be that trivial. But I have multiple folks here at Microsoft that at least feel it’s unnecessary; some that it’s a bad idea all together (and wish it didn’t exist in HTTP/2 either).

Thanks,
- Nick

From: Ted Hardie <[email-masked]>
Sent: Friday, March 8, 2019 1:42 PM
To: Nick Banks <[email-masked]>
Cc: [email-masked]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

Hi Nick,

On Fri, Mar 8, 2019 at 12:55 PM Nick Banks <nibanks=[email-masked]<mailto:[email-masked]>> wrote:
Hey Folks,

After a lengthy discussion on Slack with a few folks I’ve become convinced that connection flow control is unnecessary to achieve desired over the wire results, causes unnecessary protocol complexity (synchronizing on stream reset) and creates possible deadlocks if the app doesn’t behave correctly (won’t drain stream A until it receives what it wants on stream B).

The main reason folks seem to think that connection flow control would be useful is DoS protection; to limit the total memory to allocate for one connection. That doesn’t seem right to me. A client can open any number of connections.

In a distributed system, though, those new connections will have new connection IDs (and they might also have different five-tuples because of NAT, but that's not reliable).  As a result, they may go to different instances .  That ability to use elastic resources to spin up new instances means that the threat of the client opening a large number of connections and the threat of the client consuming unbounded resources in a single connection may be met very differently.

Imposing an overall limit per connection doesn’t really work. On the other hand if you only allocate stream flow control if you have enough over all memory (and are draining the data fast enough) then you can better control your memory usage over all connections.

This memory management regime is certainly possible, but it may not be the only one in use even in a single server. There are a lot of cases where folks would like "fair" allocation across clients or connections, so that greedy peers can't starve the others. Tracking that and not allocating stream resources to greedy peers when it looks to be the case could substitute for an overall connection-level flow control, but doing so doesn't look substantially simpler than the current design.

Also, you still have flow control on the number of streams you allow the peer to create as well. That’s another connection-wide flow control value. I think it would be an improvement to the protocol if we removed explicit connection wide flow control. If anyone has any other reasons for keeping it, I’d really like to hear them.

Now, I know this is late in the game and not strictly a necessary change. You can still just ignore connection flow control on the sender side and achieve the same thing; so this might not meet the bar. I just wanted to have this discussion and see what other folks think about this. If most folks agree that it’s not necessary, I don’t think it should be too big a deal to remove both the text and implementation code.

I have concerns about it going in at the stage.  Folks who are very used to TCP's flow control will find the idea that there is no connection-level flow control problematic, and I think re-calculating all of the engineering trade-offs will be substantially more work than just "take out the text and the code."  Done is a feature, and it is one I think we need to prioritize.

I very much appreciate your bringing this to the working group from the slack, as not all of us tracking that closely,

thanks,

Ted

Thanks,
- Nick

---

From: Roberto Peon <[email-masked]>
Date: Fri, 8 Mar 2019 22:31:05 +0000

Without connection-level flow control you end up doing one of:

  1.  Limiting the amount of sending any one new stream can do to smallish values
  2.  Limiting the number of streams that can be active before ack’d, i.e.
  3.  Delaying things by a RTT as you must now ask permission of the receiver
  4.  Re-sending information that was “lost” because the receiver didn’t have space to receive it.

1-3 were observed problems encountered in the deployment of SPDY. Conn-level flow control was added as a result of that (negative) experience.

The same complexity arguments were brought up then, and we didn’t find any better solutions than the {make a really really big connection-flow control and never worry about it/make a new connection when the connection-flow control window is up/done} solution.
4 is effectively descriptive of congestion control that isn’t working properly.

I do believe that a malicious client could open up many connections to circumvent limits. Malicious clients will do what they do, and servers will respond by doing with they need to do (which could include rate-limiting of packets/connections or variants thereof). While DoS and malicious DoS happens semi-regularly, most of the time non-malicious traffic is the vast majority of traffic.

For that reason, I believe that the more interesting case is a non-malicious client which, whenever possible, is using a single connection. Allowing this single-connection use-case to make forward progress effectively makes the game harder for the malicious clients by making the malicious behavior more easily identifiable and thus easier to filter out.

How would you propose to overcome these issues?
In the case of a mixing of malicious clients and non-malicious clients, how do you bias the system to be more likely to prevent the malicious clients from using resources/bias towards allowing non-malicious clients to make forward progress?

-=R

From: QUIC <[email-masked]> on behalf of Nick Banks <nibanks=[email-masked]>
Date: Friday, March 8, 2019 at 2:12 PM
To: Mikkel Fahnøe Jørgensen <[email-masked]>, Ted Hardie <[email-masked]>, Lucas Pardue <[email-masked]>
Cc: QUIC WG <[email-masked]>
Subject: RE: Proposal: Remove Connection Wide Flow Control from QUIC

- Nick

Mikkel

Lucas

---

From: Ted Hardie <[email-masked]>
Date: Fri, 8 Mar 2019 15:13:47 -0800

Hi Nick,

On Fri, Mar 8, 2019 at 2:24 PM Nick Banks <[email-masked]> wrote:

> I agree that new connections created by clients\attackers might get load
> balanced by a different server. I’m just saying that imposing a connection
> wide memory cap does nothing to prevent a client from causing more memory
> to be allocated. Perhaps the memory from new connections will get allocated
> on a different server; perhaps not. But the client is still causing you to
> allocate more memory either way.
>

I think my perspective is slightly different.  If I'm getting new
connections and the load balancer can re-allocate the flows (or even spin
up new devices in an elastic compute environment), these "greedy" new
connections never need to impact other clients.  They impact my bottom
line, certainly, as that's not free, but to me that's qualitatively
different from memory consumption that does potentially starve other,
well-behaved clients.  The threat model is different.

The connection FC max isn’t limiting that over all. The maximum number of
> streams FC value can also be used to a similar effect.
>
>
>
> Additionally, you will always need to make use of stream FC. That is the
> way an app can apply back pressure effectively. Connection FC doesn’t
> really do that. Generally, the stream FC grows as the app is willing to
> accept more data. So, if the app can continue to keep up with the data it
> will continue to accept more. The app does have some responsibility in this
> to decide if a DoS is happening at their layer. I don’t think that memory
> management purely at the stream layer is more complicated than additionally
> having a connection FC.
>
>
>
> As to your arguments about folks used to TCP being uncomfortable with no
> connection FC, I’d argue the opposite. As far as FC is concerned, the
> closest analogy should be that a TCP connection equals a QUIC stream. In my
> mind, that’s yet another argument against an additional layer of FC.
>
>
I think there will be cases, though, where the same server speaks both
$FOO/QUIC and $FOO/TCP.  In that case, having a connection-level flow
control for one and no connection-based flow control for the other might
seem disquieting.  It's certainly possible that you could architect a
system that would behave appropriately in that situation, and you may be
right that stream-based back pressure will be the natural parallel for some
to draw with TCP connections.  For HTTP 1.1, that does seem intuitive.
I'm a good bit less certain, though, about HTTP/2 since it does have
connection level flow control over TCP.

>
> I also agree that just taking things out might not be that trivial. But I
> have multiple folks here at Microsoft that at least feel it’s unnecessary;
> some that it’s a bad idea all together (and wish it didn’t exist in HTTP/2
> either).
>
>
>

If you were filing an issue for this, I certainly would not close it, but I
think I would mark it v2 or later.

Thanks again for bringing the issue to the list,

Ted

> Thanks,
>
> - Nick
>
>
>
> *From:* Ted Hardie <[email-masked]>
> *Sent:* Friday, March 8, 2019 1:42 PM
> *To:* Nick Banks <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* Re: Proposal: Remove Connection Wide Flow Control from QUIC
>
>
>
> Hi Nick,
>
>
>
> I have concerns about it going in at the stage.  Folks who are very used
> to TCP's flow control will find the idea that there is no connection-level

---

From: Ian Swett <[email-masked]>
Date: Fri, 8 Mar 2019 18:41:02 -0500

I would prefer to keep connection level flow control.  I agree it adds some
complexity, but it solves real problems as stated above.

The deadlock issue is a real concern and adds complexity.  If we wanted to
eliminate flow-control related deadlocks, I think adding DATAGRAM to the
core draft and using it for QPACK instructions(and possibly other HTTP/3
frames) would fix the deadlock issues, and be a simpler and more valuable
change.  But that would restrict a single QPACK instruction to ~1200 bytes,
change transport(where the bar has been raised), and I don't know if that
would create other complexities.

On Fri, Mar 8, 2019 at 6:14 PM Ted Hardie <[email-masked]> wrote:

> Hi Nick,
>
> On Fri, Mar 8, 2019 at 2:24 PM Nick Banks <nibanks@microsoft..com
> <[email-masked]>> wrote:
>
> to draw with TCP connections.  For HTTP 1.1, that does seem intuitive..
> If you were filing an issue for this, I certainly would not close it, but
> I think I would mark it v2 or later.
>
>> After a lengthy discussion on Slack with a few folks I’ve become
>> convinced that connection flow control is unnecessary to achieve desired
>> over the wire results, causes unnecessary protocol complexity
>> (synchronizing on stream reset) and creates possible deadlocks if the app
>> doesn’t behave correctly (won’t drain stream A until it receives what it
>> wants on stream B).
>>
>>
>>
>> Imposing an overall limit per connection doesn’t really work. On the
>> other hand if you only allocate stream flow control if you have enough over
>> all memory (and are draining the data fast enough) then you can better
>> control your memory usage over all connections.
>>
>>
>>
>> This memory management regime is certainly possible, but it may not be
>> the only one in use even in a single server. There are a lot of cases where
>> the code."  Done is a feature, and it is one I think we need to prioritize..
>>
>>
>>

---

From: Nick Banks <[email-masked]>
Date: Fri, 8 Mar 2019 23:48:00 +0000

What is the problem with treating a single QUIC stream like a TCP connection? Whatever DoS mitigation logic you have across all TCP connections for the machine should just match all streams (independent of connection).

- Nick

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
[HxS - 18848 - 16.0.11425.20086]

________________________________
From: Ian Swett <[email-masked]>
Sent: Friday, March 8, 2019 3:41:02 PM
To: Ted Hardie
Cc: Nick Banks; [email-masked]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

I would prefer to keep connection level flow control.  I agree it adds some complexity, but it solves real problems as stated above.

The deadlock issue is a real concern and adds complexity.  If we wanted to eliminate flow-control related deadlocks, I think adding DATAGRAM to the core draft and using it for QPACK instructions(and possibly other HTTP/3 frames) would fix the deadlock issues, and be a simpler and more valuable change.  But that would restrict a single QPACK instruction to ~1200 bytes, change transport(where the bar has been raised), and I don't know if that would create other complexities.

On Fri, Mar 8, 2019 at 6:14 PM Ted Hardie <[email-masked]<mailto:[email-masked]>> wrote:
Hi Nick,

On Fri, Mar 8, 2019 at 2:24 PM Nick Banks <nibanks@microsoft..com<mailto:[email-masked]>> wrote:
Thank for the responses Ted!

I agree that new connections created by clients\attackers might get load balanced by a different server. I’m just saying that imposing a connection wide memory cap does nothing to prevent a client from causing more memory to be allocated. Perhaps the memory from new connections will get allocated on a different server; perhaps not. But the client is still causing you to allocate more memory either way.

I think my perspective is slightly different.  If I'm getting new connections and the load balancer can re-allocate the flows (or even spin up new devices in an elastic compute environment), these "greedy" new connections never need to impact other clients.  They impact my bottom line, certainly, as that's not free, but to me that's qualitatively different from memory consumption that does potentially starve other, well-behaved clients.  The threat model is different.

The connection FC max isn’t limiting that over all. The maximum number of streams FC value can also be used to a similar effect.

I think there will be cases, though, where the same server speaks both $FOO/QUIC and $FOO/TCP.  In that case, having a connection-level flow control for one and no connection-based flow control for the other might seem disquieting.  It's certainly possible that you could architect a system that would behave appropriately in that situation, and you may be right that stream-based back pressure will be the natural parallel for some to draw with TCP connections.  For HTTP 1.1, that does seem intuitive..   I'm a good bit less certain, though, about HTTP/2 since it does have connection level flow control over TCP.

If you were filing an issue for this, I certainly would not close it, but I think I would mark it v2 or later.

Ted

Thanks,
- Nick

From: Ted Hardie <[email-masked]<mailto:[email-masked]>>
Sent: Friday, March 8, 2019 1:42 PM
To: Nick Banks <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

Hi Nick,

I have concerns about it going in at the stage.  Folks who are very used to TCP's flow control will find the idea that there is no connection-level flow control problematic, and I think re-calculating all of the engineering trade-offs will be substantially more work than just "take out the text and the code."  Done is a feature, and it is one I think we need to prioritize..

thanks,

Ted

Thanks,
- Nick

---

From: Ted Hardie <[email-masked]>
Date: Fri, 8 Mar 2019 16:10:52 -0800

Hi Nick,

On Fri, Mar 8, 2019 at 3:48 PM Nick Banks <[email-masked]> wrote:

> What is the problem with treating a single QUIC stream like a TCP
> connection? Whatever DoS mitigation logic you have across all TCP
> connections for the machine should just match all streams (independent of
> connection).
>
>
>

Some of the issues were raised by Roberto earlier, in his description of
how SPDY came to adopt a connection-level flow control. But I think the
bigger point we're making is that this causes a reset in a major part of
how the transport logic works, at a moment after we've raised the bar.
Implementing a change so that the structure of the DoS mitigation/memory
management shifts in this way isn't impossible, as I said before, but it
clearly also can't be accomplished by just cutting the connection-level
flow control out.  That's why I'd rather finish v1 with this approach,
which at least parallels HTTP/2 over TCP, then explore the issue further.

regards,

Ted Hardie

> - Nick
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
> ------------------------------
> *From:* Ian Swett <[email-masked]>
> *Sent:* Friday, March 8, 2019 3:41:02 PM
> *To:* Ted Hardie
> *Cc:* Nick Banks; [email-masked]
> I would prefer to keep connection level flow control.  I agree it adds
> some complexity, but it solves real problems as stated above.
>
>>> I also agree that just taking things out might not be that trivial. But
>>> I have multiple folks here at Microsoft that at least feel it’s
>>> unnecessary; some that it’s a bad idea all together (and wish it didn’t
>>> exist in HTTP/2 either).
>>>
>>>
>>>
>>
>>> The main reason folks seem to think that connection flow control would
>>> be useful is DoS protection; to limit the total memory to allocate for one
>>> Now, I know this is late in the game and not strictly a necessary
>>> change. You can still just ignore connection flow control on the sender
>>> side and achieve the same thing; so this might not meet the bar. I just
>>> wanted to have this discussion and see what other folks think about this.
>>> If most folks agree that it’s not necessary, I don’t think it should be too
>>> big a deal to remove both the text and implementation code.
>>>
>>>
>>>

---

From: Nick Banks <[email-masked]>
Date: Sat, 9 Mar 2019 00:15:01 +0000

Sure. I’m just trying to understand all the points of view. Since I don’t have to use connection FC, if I don’t want to, I’m OK if we don’t remove it (though I still prefer we do).

Can I ask how folks actually decide what value to use for connection wide FC? Do they just use some hard coded value or percentage of overall available memory? This whole thing started (at MS) because we couldn’t figure out what to set connection FC to; eventually deciding it might not be needed at all.

- Nick

________________________________
From: Ted Hardie <[email-masked]>
Sent: Friday, March 8, 2019 4:10:52 PM
To: Nick Banks
Cc: Ian Swett; [email-masked]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

Hi Nick,

On Fri, Mar 8, 2019 at 3:48 PM Nick Banks <[email-masked]<mailto:[email-masked]>> wrote:
What is the problem with treating a single QUIC stream like a TCP connection? Whatever DoS mitigation logic you have across all TCP connections for the machine should just match all streams (independent of connection).

Some of the issues were raised by Roberto earlier, in his description of how SPDY came to adopt a connection-level flow control. But I think the bigger point we're making is that this causes a reset in a major part of how the transport logic works, at a moment after we've raised the bar.  Implementing a change so that the structure of the DoS mitigation/memory management shifts in this way isn't impossible, as I said before, but it clearly also can't be accomplished by just cutting the connection-level flow control out.  That's why I'd rather finish v1 with this approach, which at least parallels HTTP/2 over TCP, then explore the issue further.

regards,

Ted Hardie

- Nick

________________________________
From: Ian Swett <[email-masked]<mailto:[email-masked]>>
Sent: Friday, March 8, 2019 3:41:02 PM
To: Ted Hardie
Cc: Nick Banks; [email-masked]<mailto:[email-masked]>
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

Ted

Thanks,
- Nick

Hi Nick,

thanks,

Ted

Thanks,
- Nick

---

From: Christian Huitema <[email-masked]>
Date: Fri, 8 Mar 2019 16:20:35 -0800

On 3/8/2019 4:10 PM, Ted Hardie wrote:

> Some of the issues were raised by Roberto earlier, in his description
> of how SPDY came to adopt a connection-level flow control. But I think
> the bigger point we're making is that this causes a reset in a major
> part of how the transport logic works, at a moment after we've raised
> the bar.  Implementing a change so that the structure of the DoS
> mitigation/memory management shifts in this way isn't impossible, as I
> said before, but it clearly also can't be accomplished by just cutting
> the connection-level flow control out.  That's why I'd rather finish
> v1 with this approach, which at least parallels HTTP/2 over TCP, then
> explore the issue further.

In addition, applications that don't want to be bothered with managing
FC can just set the value to (2^62 -1) in the transport parameters. 
Either implementations find FC useful and they have a burden to
implement it right, or they don't and they can just set it to infinity.
So I don't think we need to change the spec for V1.

-- Christian Huitema

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 8 Mar 2019 16:22:16 -0800

Connection and stream level flow control were entirely conscious additions
to SPDY; they were not accidental historical artifacts. With HTTP/2 they
have significant deployment in the wild. Given this, and how late in the
process we are, I don't think it would be wise to remove connection level
flow control absent some incredibly compelling issue.

On Fri, Mar 8, 2019 at 12:55 PM Nick Banks <nibanks=
[email-masked]> wrote:

> connections. Imposing an overall limit per connection doesn’t really work.
> On the other hand if you only allocate stream flow control if you have
> enough over all memory (and are draining the data fast enough) then you can
> better control your memory usage over all connections.
>
>
>

---

From: Mike Bishop <[email-masked]>
Date: Sat, 9 Mar 2019 06:24:46 +0000

I’m unclear on what alternative would get us the behavior that’s needed.

You can’t just use the stream flow control windows.  Per-stream flow control is used to enforce fairness amongst streams.  However, if you want to have high parallelism, then SUM(stream flow control) is going to be much larger than the total amount of memory the receiver is actually willing to allocate.  Connection-level flow control bounds the receiver’s total memory allocation regardless of which stream it’s going to.

One could argue that connection-level flow control serves the same purpose as congestion control if the server simply drops incoming packets when it doesn’t have the memory to process more data right now, and it hopes that doesn’t happen very often.  Those packets drop, loss-based congestion control interprets that as a signal to slow down, and the sender backs off.  And that works, albeit imperfectly.  However, you can’t partially acknowledge packets; if you aren’t willing to commit more memory to holding the data being sent, you can’t acknowledge receipt of the control information in those packets either.  That means it’s bad for the network -- you’re generating spurious retransmissions and creating loss where instead you could have inhibited transmission in the first place.

IIRC, Microsoft throws wide the connection flow control doors in HTTP/2, and the world will not end if it opts to do the same in QUIC.  I’m not in favor of removing it from the protocol, because it’s useful to at least some implementations and scenarios.

From: QUIC <[email-masked]> On Behalf Of Ryan Hamilton
Sent: Friday, March 8, 2019 4:22 PM
To: Nick Banks <nibanks=[email-masked]>
Cc: [email-masked]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

Connection and stream level flow control were entirely conscious additions to SPDY; they were not accidental historical artifacts. With HTTP/2 they have significant deployment in the wild. Given this, and how late in the process we are, I don't think it would be wise to remove connection level flow control absent some incredibly compelling issue.

The main reason folks seem to think that connection flow control would be useful is DoS protection; to limit the total memory to allocate for one connection. That doesn’t seem right to me. A client can open any number of connections. Imposing an overall limit per connection doesn’t really work. On the other hand if you only allocate stream flow control if you have enough over all memory (and are draining the data fast enough) then you can better control your memory usage over all connections.

Thanks,
- Nick

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 9 Mar 2019 08:03:45 +0100

Hi,

On Fri, Mar 08, 2019 at 10:24:28PM +0000, Nick Banks wrote:
> balanced by a different server. I'm just saying that imposing a connection
> wide memory cap does nothing to prevent a client from causing more memory to
> be allocated.

That was my first thought about H2 before implementing it. But when I
started to address corner cases, I figured that having multiple limits
gave me more knobs to control the peers behaviour and detect bad ones.

Now for this reason I'm more in favor of keeping the connection FC on
QUIC. It allows servers to adjust the traffic based on available
resources. When it has a lot of connections it can use smaller windows
to preserve memory (or to work within a constrained memory budget).

Sure, a malicious client could circumvent the memory limit by opening
many connections, but in this case the server may use even smaller
connection windows. You'd say that this makes it trivial for a
malicious client to block traffic from many other clients, but in order
to do this it will require this client to open a lot of connections,
and this becomes detectable by the server. This is why it's important
to have multiple limits, to force bad actors to act on another variable
that makes it detectable. Any single setting that can be abused also
has some valid use cases and by itself doesn't constitute a detectable
abuse. But when an abuser has to involve a product of two settings to
abuse resources (e.g. many conns with many streams etc), then you know
something is wrong.

In summary, enforcing one limit makes abuses much easier to detect.
Without the connection flow control, it's very hard to achieve the same
results.

Just my two cents,
Willy

---

From: Ian Swett <[email-masked]>
Date: Sat, 9 Mar 2019 03:30:17 -0500

In the Chromium implementation, we autotune connection flow control with
the same algorithm as stream flow control, but also ensure connection flow
control at least 1.5x the largest stream flow control window, to avoid one
stream starving others.  I haven't observed any issues with this approach,
and the number of BLOCKED frames is very low.

On Fri, Mar 8, 2019 at 7:15 PM Nick Banks <[email-masked]> wrote:

> Sure. I’m just trying to understand all the points of view. Since I don’t
> have to use connection FC, if I don’t want to, I’m OK if we don’t remove it
> (though I still prefer we do).
>
>
>
> Can I ask how folks actually decide what value to use for connection wide
> FC? Do they just use some hard coded value or percentage of overall
> available memory? This whole thing started (at MS) because we couldn’t
> figure out what to set connection FC to; eventually deciding it might not
> be needed at all.
>
>
>
> - Nick
>
>
>
> *Sent:* Friday, March 8, 2019 4:10:52 PM
> *To:* Nick Banks
> *Cc:* Ian Swett; [email-masked]
>> The deadlock issue is a real concern and adds complexity.  If we wanted
>> to eliminate flow-control related deadlocks, I think adding DATAGRAM to the
>>>> I agree that new connections created by clients\attackers might get
>>>> load balanced by a different server. I’m just saying that imposing a
>>>> connection wide memory cap does nothing to prevent a client from causing
>>>> more memory to be allocated. Perhaps the memory from new connections will
>>>> get allocated on a different server; perhaps not. But the client is still
>>>> causing you to allocate more memory either way.
>>>>
>>>
>>> The connection FC max isn’t limiting that over all. The maximum number
>>>> of streams FC value can also be used to a similar effect.
>>>>
>>>>
>>>>
>>>> Additionally, you will always need to make use of stream FC. That is
>>>> the way an app can apply back pressure effectively. Connection FC doesn’t
>>>> As to your arguments about folks used to TCP being uncomfortable with
>>>> no connection FC, I’d argue the opposite. As far as FC is concerned, the
>>>> Also, you still have flow control on the number of streams you allow
>>>> the peer to create as well. That’s another connection-wide flow control
>>>> value. I think it would be an improvement to the protocol if we removed
>>>> explicit connection wide flow control. If anyone has any other reasons for
>>>> keeping it, I’d really like to hear them.
>>>>
>>>>
>>>>
>>>> I have concerns about it going in at the stage.  Folks who are very
>>>> used to TCP's flow control will find the idea that there is no
>>>> connection-level flow control problematic, and I think re-calculating all
>>>> of the engineering trade-offs will be substantially more work than just
>>>> "take out the text and the code."  Done is a feature, and it is one I think
>>>> we need to prioritize..
>>>>
>>>>
>>>>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sat, 9 Mar 2019 15:23:40 +0000

If I had to get rid of a flow control knob, I would be getting rid of a stream flow control before connection flow control.

Connection FC represents a clear constraint on the receiver memory commitment. Stream FC is not about fairness (as it was suggested), because sender is already in the best position to determine the fair connection utilization. Rather, stream FC is a poor-man's communication mechanism from the receiver to the sender about its relative interest in the stream data. And it is a poor mechanism, since the signal is always delayed by at least several RTTs. Scraping stream FC and replacing it with adjustable stream priorities would make for a more responsive protocol at the expense of more complexity (and QUIC already has plenty of complexity).

- Igor

-----Original Message-----
From: Ian Swett [ianswett=[email-masked]]
Received: Saturday, 09 Mar 2019, 3:31AM
To: Nick Banks [[email-masked]]
CC: Ted Hardie [[email-masked]]; [email-masked] [[email-masked]]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

In the Chromium implementation, we autotune connection flow control with the same algorithm as stream flow control, but also ensure connection flow control at least 1.5x the largest stream flow control window, to avoid one stream starving others.  I haven't observed any issues with this approach, and the number of BLOCKED frames is very low.

On Fri, Mar 8, 2019 at 7:15 PM Nick Banks <[email-masked]<mailto:[email-masked]>> wrote:
Sure. I’m just trying to understand all the points of view. Since I don’t have to use connection FC, if I don’t want to, I’m OK if we don’t remove it (though I still prefer we do).

- Nick

Sent from Mail<https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlink_-3FLinkId-3D550986&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=vzNqw5DV3Y0fE0tRyUv8SwPZOhbN7PsaKTkBCrPVbyE&s=gyTT-Ks6yhX_bP2iDhVhnUyraP8IoMR-aVQojUj61wI&e=> for Windows 10
[HxS - 18848 - 16.0.11425.20086]

Cc: Ian Swett; [email-masked]<mailto:[email-masked]>

Hi Nick,

regards,

Ted Hardie

- Nick

Sent from Mail<https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlink_-3FLinkId-3D550986&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=vzNqw5DV3Y0fE0tRyUv8SwPZOhbN7PsaKTkBCrPVbyE&s=gyTT-Ks6yhX_bP2iDhVhnUyraP8IoMR-aVQojUj61wI&e=> for Windows 10
[HxS - 18848 - 16.0.11425.20086]

Ted

Thanks,
- Nick

Hi Nick,

I have concerns about it going in at the stage.  Folks who are very used to TCP's flow control will find the idea that there is no connection-level flow control problematic, and I think re-calculating all of the engineering trade-offs will be substantially more work than just "take out the text and the code."  Done is a feature, and it is one I think we need to prioritize...

thanks,

Ted

Thanks,
- Nick

---

From: Nick Banks <[email-masked]>
Date: Mon, 11 Mar 2019 17:27:01 +0000

Ian,

A clarification on your logic:

If you are using the same logic to tune both windows, doesn’t that mean they would both grow at the same rate then? Do you initialize them to different values? X bytes for streams and 1.5x for the connection? Do you have some different caps on each? If you do, how do you pick those caps?

If you can’t provide all that information, I’d appreciate any info you can share.

Thanks,
- Nick

From: Ian Swett <[email-masked]>
Sent: Saturday, March 9, 2019 12:30 AM
To: Nick Banks <[email-masked]>
Cc: Ted Hardie <[email-masked]>; [email-masked]
Subject: Re: Proposal: Remove Connection Wide Flow Control from QUIC

- Nick

Hi Nick,

regards,

Ted Hardie

- Nick

Ted

Thanks,
- Nick

Hi Nick,

thanks,

Ted

Thanks,
- Nick

---

From: Ian Swett <[email-masked]>
Date: Mon, 11 Mar 2019 23:43:57 -0400

On the server, we are using the same logic to tune both, but we're
attempting to maintain an invariant that the connection flow control window
is always at least 1.5x the largest single stream flow control window.

The reason is to avoid a large transfer blocking all other traffic on the
connection.  There are certainly cases when this connection-wide limit is
insufficient, but the hope is that connection level flow control would
autotune up to mitigate the issue.

I believe the caps are X for stream flow control and 1.5X for connection
flow control, but I don't know what X is off the top of my head, and it may
depend upon application.  I do believe X is always at least 1MB.

In Chrome itself, I believe a large value (8MB?) is used and it's never
increased.  Flow control limits mostly exist to limit server resources, not
client resources.

I can't say what I described above is optimal, but the only time we've seen
flow control limiting performance in the last year or two was on very high
bandwidth uploads on relatively large RTTs.  Most applications don't care
about that use cause, though some do.

On Mon, Mar 11, 2019 at 1:27 PM Nick Banks <[email-masked]> wrote:

> If you are using the same logic to tune both windows, doesn’t that mean
> they would both grow at the same rate then? Do you initialize them to
> different values? X bytes for streams and 1.5x for the connection? Do you
> have some different caps on each? If you do, how do you pick those caps?
>
>
>
> *Sent:* Saturday, March 9, 2019 12:30 AM
> *Cc:* Ted Hardie <[email-masked]>; [email-masked]

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 12 Mar 2019 17:27:54 +1100

And indeed, some widely-deployed HTTP/2 servers do this ("this" being setting INITIAL_WINDOW_SIZE to a very high value). But many do not.

Cheers,

> On 9 Mar 2019, at 11:20 am, Christian Huitema <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/