Subject: Question on flow control for a single file transfer

From: Peter Lepeska <[email-masked]>
Date: Sun, 3 Nov 2013 12:03:54 -0800

Will flow control be used even when an HTTP 2.0 connection is only being
used to transfer a single file?

Today if a peer is sending too fast, the application (web server or
browser) can just not post a receiver buffer for the TCP stack to fill.
Then the TCP stack takes care of informing the sender to slow down.

With HTTP 2.0 I will have two layers both trying to do flow control. I get
it when there are multiple streams active -- flow control at the HTTP 2.0
layer is needed. But is it needed if I'm just using HTTP 2.0 to transfer a
single file?

My apologies if this question was already asked and in general sorry for
the barrage of posts today. Can you tell that I finally had time to read
all the docs on the flight?

Thanks,

Peter

---

From: Martin Thomson <[email-masked]>
Date: Sun, 3 Nov 2013 13:16:10 -0800

> used to transfer a single file?

If you are concerned that flow control will reduce your ability to get
the most out of a connection, turn it off.

In fact, we make that recommendation:
http://http2.github.io/http2-spec/#DisableFlowControl

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sun, 3 Nov 2013 13:25:41 -0800

It's probably understood already, but just to be clear, this is receiver
controlled and directional. Unless you control both endpoints, you must
implement flow control in order to respect the peer's receive windows, even
if you disable your own receive windows. Cheers.

---

From: Peter Lepeska <[email-masked]>
Date: Sun, 3 Nov 2013 15:26:32 -0800

Right. And you don't want to turn it off at the connection level since you
never know if a connection will eventually see multiple simultaneous
streams and so need flow control. That's why I was thinking the sender
should ignore receive window updates unless it is sending more than one
stream.

If the receiver really wants to slow down a single stream connection, then
it will just delay its posting of receive buffers to TCP the way it does
with HTTP 1.x -- this gives flow control back to TCP unless there's more
than one sending stream.

Peter

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sun, 3 Nov 2013 15:33:57 -0800

Er, why should the sender ignore receive window updates? If the receiver
doesn't want to use flow control, it should disable it and not send
WINDOW_UPDATEs. I don't think the sender should ignore receive window
updates.

---

From: Yoav Nir <[email-masked]>
Date: Mon, 4 Nov 2013 02:22:51 +0000

On Nov 3, 2013, at 1:25 PM, William Chan (陈智昌) <[email-masked]<mailto:[email-masked]>> wrote:

It's probably understood already, but just to be clear, this is receiver controlled and directional. Unless you control both endpoints, you must implement flow control in order to respect the peer's receive windows, even if you disable your own receive windows. Cheers.

This discussion started with tools like WGET. If all you're ever sending is one single request equivalent to "GET xxx", you're likely fine not considering server receive window.

For a single file, the data that the client sends to the server never exceeds the default server receive window.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sun, 3 Nov 2013 18:35:27 -0800

I don't feel comfortable encouraging such sloppiness, I worry about future
interop. Respecting a peer's receive window isn't hard. Just do it :)

And even though wget doesn't support upload (to my knowledge, but I'm not
an expert), a command line tool may upload, in which case it should
definitely respect the peer's receive window.
On Nov 3, 2013 6:22 PM, "Yoav Nir" <[email-masked]> wrote:

>
>  On Nov 3, 2013, at 1:25 PM, William Chan (陈智昌) <[email-masked]>
> wrote:
>
>  This discussion started with tools like WGET. If all you're ever sending
> is one single request equivalent to "GET xxx", you're likely fine not
> considering server receive window.
>
>  For a single file, the data that the client sends to the server never
> exceeds the default server receive window.
>
>
>

---

From: Peter Lepeska <[email-masked]>
Date: Sun, 3 Nov 2013 19:44:01 -0800

Sloppiness? I don't get that. The sender's job is to transmit the data as
fast as possible, not to respect the receiver's best guesstimate of
available bandwidth sent ½ RTT ago. In this case, the sender's job is to
keep the TCP buffer full of data so it can send it when it has the
opportunity to.

Respecting the peer's receive window in the single file send case is
harmless at best and detrimental otherwise.

Peter
On Sunday, November 3, 2013, William Chan (陈智昌) wrote:

> On Nov 3, 2013 6:22 PM, "Yoav Nir" <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>>
> wrote:
>
>>
>>  On Nov 3, 2013, at 1:25 PM, William Chan (陈智昌) <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>>
>> wrote:
>>
>>  It's probably understood already, but just to be clear, this is
>> receiver controlled and directional. Unless you control both endpoints, you
>> must implement flow control in order to respect the peer's receive windows,
>> even if you disable your own receive windows. Cheers.
>>
>>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sun, 3 Nov 2013 19:54:12 -0800

http://en.wikipedia.org/wiki/Flow_control_(data) says "In data
communications, flow control is the process of managing the rate of data
transmission between two nodes to prevent a fast sender from overwhelming a
slow receiver."

Guesstimating BDP is only important if the receiver cares about maximizing
throughput. Which hopefully it does, but there's no guarantee. Sometimes
due to resource constraints, the receiver cannot accept that much data, and
it asserts flow control in this case. And senders *need* to respect that.
Otherwise a receiver with any sense, like a highly scalable server, will
terminate the connection since the peer is misbehaving.

On Sun, Nov 3, 2013 at 7:44 PM, Peter Lepeska <[email-masked]> wrote:

>> I don't feel comfortable encouraging such sloppiness, I worry about
>> future interop. Respecting a peer's receive window isn't hard. Just do it :)
>>
>>>  This discussion started with tools like WGET. If all you're ever
>>> sending is one single request equivalent to "GET xxx", you're likely fine
>>> not considering server receive window.
>>>

---

From: Peter Lepeska <[email-masked]>
Date: Sun, 3 Nov 2013 20:04:17 -0800

If a receiver cannot absorb any more data, it will not make a buffer
available to TCP.

Don't forget that in HTTP 1.x we don't do flow control. We leave that to
the transport layer and this works well. Layering flow control on top of
flow control can only result in slower flows. This slowdown is necessary
when two or more streams are being sent at once but let's not take this hit
in the simple case of one stream.

Peter

On Sunday, November 3, 2013, William Chan (陈智昌) wrote:

> On Sun, Nov 3, 2013 at 7:44 PM, Peter Lepeska <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
>>> And even though wget doesn't support upload (to my knowledge, but I'm
>>> not an expert), a command line tool may upload, in which case it should

---

From: Roberto Peon <[email-masked]>
Date: Sun, 3 Nov 2013 20:49:29 -0800

Peter, this isn't true. Servers care about things other than maximizing bdp
to a single connection, and in general care about performance across more
connections.

Speaking as someone who has some implementation experience at scale, when
the receiver asserts flow control or other policy, and the csender fails to
respect it, it will be assumed to be malicious and the connection is far
likelier to be terminated.

Resource utilization constraints exist to constrain utilization. Doing so
can impact bandwidth utilization, true, but sometimes this is exactly what
is intended as a tradeoff for some other property, e.g. maximizing
utilization of a more contended resource.

The fact that this contention is not seen by the client doesn't mean
anything. There is almost alwatlys more than one client or client
connection. I do not want to be forced to override tcp's auto buffer
estimating by imposing a max window in the implementation, as that will
impact latency later, but I also don't want huge buffers in kernel space
soaking up precious memory.
-=R
On Nov 3, 2013 8:06 PM, "Peter Lepeska" <[email-masked]> wrote:

>> Guesstimating BDP is only important if the receiver cares about
>> maximizing throughput. Which hopefully it does, but there's no guarantee.
>> Sometimes due to resource constraints, the receiver cannot accept that much
>> data, and it asserts flow control in this case. And senders *need* to
>> respect that. Otherwise a receiver with any sense, like a highly scalable
>> server, will terminate the connection since the peer is misbehaving.
>>
>>
>> On Sun, Nov 3, 2013 at 7:44 PM, Peter Lepeska <[email-masked]>wrote:
>>
>>> Sloppiness? I don't get that. The sender's job is to transmit the data
>>> as fast as possible, not to respect the receiver's best guesstimate of
>>>>>  For a single file, the data that the client sends to the server
>>>>> never exceeds the default server receive window.
>>>>>
>>>>>
>>>>>
>>

---

From: Peter Lepeska <[email-masked]>
Date: Sun, 3 Nov 2013 21:19:21 -0800

Roberto,

I'm not getting your point. When the sender is the browser (for uploads)
the data is buffered in the TCP stack on the end user's machine, so this
would have no impact on servers at scale. When the sender is the server
(for downloads), there should be no scaleability-related difference if data
is buffered in the user mode application or in the kernel, if it is in
TCP's buffer. Why is there a scalability advantage buffering in the user
mode process (when we respect the remote peer's receive window) as opposed
to the TCP stack in the kernel?

I recognize that I have less implementation experience at scale, but I
still don't understand your argument. Trying to though...

"Speaking as someone who has some implementation experience at scale, when
the receiver asserts flow control or other policy, and the csender fails to
respect it, it will be assumed to be malicious and the connection is far
likelier to be terminated."

We don't have app-level flow control today (in HTTP 1.x implementations) so
why would this be assumed to be malicious? I'm just suggesting we put in
the spec that receiver advertised windows not be respected when there is
only one active stream. If it is the standard and this behavior is in
standards-compliant browsers, why would we assume this to be malicious?

Peter

On Sun, Nov 3, 2013 at 8:49 PM, Roberto Peon <[email-masked]> wrote:

> Peter, this isn't true. Servers care about things other than maximizing
> bdp to a single connection, and in general care about performance across
> more connections.
>
>>>>>>  On Nov 3, 2013, at 1:25 PM, William Chan (陈智昌) <
>>>>>> [email-masked]> wrote:
>>>>>>

---

From: Roberto Peon <[email-masked]>
Date: Sun, 3 Nov 2013 21:50:59 -0800

On Sun, Nov 3, 2013 at 9:19 PM, Peter Lepeska <[email-masked]> wrote:

> the data is buffered in the TCP stack on the end user's machine,
>

Incorrect, it is buffered on both ends of the TCP connection.
For typical TCP implementations this will end up being a BDPs worth or so.
As an example, with a 30ms ping time, and a 10Mb connection, you're talking
~4MB.
If the other side of this is contended, I may need to throttle back. I also
have no need of a full BDP's worth of buffer sitting there doing nothing
for some other user.

> so this would have no impact on servers at scale. When the sender is the
> server (for downloads), there should be no scaleability-related difference
> if data is buffered in the user mode application or in the kernel, if it is
> in TCP's buffer. Why is there a scalability advantage buffering in the user
See above. Kernel buffers are not free, they still count.

> "Speaking as someone who has some implementation experience at scale,
> when the receiver asserts flow control or other policy, and the csender
> fails to respect it, it will be assumed to be malicious and the connection
> is far likelier to be terminated."
>
> We don't have app-level flow control today (in HTTP 1.x implementations)
> so why would this be assumed to be malicious? I'm just suggesting we put in

The standard is not that today, nor do I hope it becomes that, since
predicting the future is.. difficult, and the same rules need be in effect
for all connections, whether or not they end up with two or more streams.

I don't like exceptions, and the one you proposed there includes a
desynchronization race condition (at least until additional complexity is
heaped on top). Yuck.

Regardless, any implementation which uses more server memory than another
is going to the chopping block before any others since that server cannot
verify that the behavior is not malicious.

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 04 Nov 2013 18:55:44 +1300

On 4/11/2013 6:19 p.m., Peter Lepeska wrote:
> Roberto,
>
> I'm not getting your point. When the sender is the browser (for 
> uploads) the data is buffered in the TCP stack on the end user's 
> machine, so this would have no impact on servers at scale. When the 
> sender is the server (for downloads), there should be no 
> scaleability-related difference if data is buffered in the user mode 
> application or in the kernel, if it is in TCP's buffer. Why is there a 
> scalability advantage buffering in the user mode process (when we 
> respect the remote peer's receive window) as opposed to the TCP stack 
> in the kernel?
>
> when the receiver asserts flow control or other policy, and the 
> csender fails to respect it, it will be assumed to be malicious and 
> the connection is far likelier to be terminated."
>
> We don't have app-level flow control today (in HTTP 1.x 
> implementations) so why would this be assumed to be malicious? I'm 
> just suggesting we put in the spec that receiver advertised windows 
> not be respected when there is only one active stream. If it is the 
> standard and this behavior is in standards-compliant browsers, why 
> would we assume this to be malicious?
>
> Peter
>

You are thinking only of the case where a client is directly plugged 
into the origin server.

HTTP contains proxies. Meaning there are at least two TCP buffers in 
series to be traversed.
  * In HTTP/1 this is not a problem since for period when data is being 
relayed end-to-end the proxy is simply pushing client bytes to the 
server and vice versa an can simply pause reading/writing at any point.
  * With HTTP/2 multiplexing this protection disappears. Two or more 
clients requests may be delivered to the same server connection 
simultaneously. If the proxy is not explicitly informed about each 
clients receive limit, it has no way to reliably limit data from the 
server to that client and may hit a deadlock situation where its buffer 
is filled with frames awaiting delivery to the slowest client while the 
faster client is starved.

The more complex case where multiple clients with multiple streams are 
split N:M across several faster server connections can hit contention on 
flow back to the clients. If any one client connection is slower than 
the sum of server connections output destined there, the sheer flood of 
data being delivered to it will cause the same form of server connection 
buffer inside the proxy to fill/block with that clients data and starve 
other clients.

These are strong cases for per-stream WINDOW_UPDATE flow control. Im not 
sure what the use-case for connection-wide flow control is. Perhapse 
that is best left for the TCP level.

Amos

---

From: Yoav Nir <[email-masked]>
Date: Mon, 4 Nov 2013 05:56:48 +0000

Hi, Peter
On Nov 3, 2013, at 9:19 PM, Peter Lepeska <[email-masked]> wrote:

> Roberto,
> 
> I'm not getting your point. When the sender is the browser (for uploads) the data is buffered in the TCP stack on the end user's machine, so this would have no impact on servers at scale. When the sender is the server (for downloads), there should be no scaleability-related difference if data is buffered in the user mode application or in the kernel, if it is in TCP's buffer. Why is there a scalability advantage buffering in the user mode process (when we respect the remote peer's receive window) as opposed to the TCP stack in the kernel? 

I don't have Roberto's experience either, but I do share some experience with people who write intermediaries. Receiver (whether the server for upload, the client for download, or the intermediary in either case) memory is a constrained resource. That is especially true for the server and intermediary. To get TCP to work well, the TCP stack needs to have enough buffer space to avoid the need to slow the client down by dropping packets. If the client uploads at top speed, it could overwhelm the ability of the server software to handle, and end up filling up the TCP buffers on the server. If one client does that, that's OK. If a lot of clients do that, TCP memory runs out, so that limits the number of clients that a single server can support. 

Even for downloads, if the server application does not get a signal from the client application, all it can do is send the whole resource and let its TCP stack buffer the whole thing or block the application, so either way the entire resource is waiting in memory. That is, of course, a naive way to implement a server, and even today servers query TCP to know how much data they can send without blocking and/or overfilling the buffers. So for download it is solvable. But for uploads, there is nothing that the server can do about its TCP buffers filling up faster than it can empty them.  Having such signaling at the HTTP layer gives both sides better control over their use of buffer space (whether that buffer is in kernel or user mode).

Yoav

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 4 Nov 2013 09:05:11 +0000

Some (possibly well-known) comments on TCP flow control - I guess HTTP/2 could run into similar issues and solutions as well...

* Many TCP stacks have complicated heuristics to determine the TCP receive window they advertise to the sender. In addition to free receiver buffer space, other metrics such the estimated path BDP, the socket/application read rate, etc. may be taken into account as well.

* In the worst case, the buffer space needed in a receiver is larger (by factor 2) than what is actually advertised to the sender, if out-of-order data should be stored. In case of TCP, out-of-order data can also be discarded, but most modern stacks buffer such data for efficiency, AFAIK.

* In TCP, it is also not uncommon to oversubscribe memory, i.e., if there are many TCP connections, the sum of all advertised TCP receive windows may exceed the memory that is actually available. This heuristic assumes that not all connections need buffer at the same time. If this assumption turns out to be wrong, the receiver can still drop incoming packets if it cannot allocate buffer for it. This is not different to network congestion, i.e., it will affect throughput, but the data transfer will continue.

* However, a TCP sender must respect the receive window advertised by the receiver. TCP receivers often check this (amongst others, for security reasons).

My 2 cents

Michael

________________________________
Von: [email-masked] [[email-masked]]" im Auftrag von "William Chan (陈智昌) [[email-masked]]
Gesendet: Montag, 4. November 2013 04:54
An: Peter Lepeska
Cc: Yoav Nir; <[email-masked]>; Martin Thomson
Betreff: Re: Question on flow control for a single file transfer

http://en.wikipedia.org/wiki/Flow_control_(data) says "In data communications, flow control is the process of managing the rate of data transmission between two nodes to prevent a fast sender from overwhelming a slow receiver."

Guesstimating BDP is only important if the receiver cares about maximizing throughput. Which hopefully it does, but there's no guarantee. Sometimes due to resource constraints, the receiver cannot accept that much data, and it asserts flow control in this case. And senders *need* to respect that. Otherwise a receiver with any sense, like a highly scalable server, will terminate the connection since the peer is misbehaving.

On Sun, Nov 3, 2013 at 7:44 PM, Peter Lepeska <[email-masked]<mailto:[email-masked]>> wrote:
Sloppiness? I don't get that. The sender's job is to transmit the data as fast as possible, not to respect the receiver's best guesstimate of available bandwidth sent ½ RTT ago. In this case, the sender's job is to keep the TCP buffer full of data so it can send it when it has the opportunity to.

Respecting the peer's receive window in the single file send case is harmless at best and detrimental otherwise.

Peter

I don't feel comfortable encouraging such sloppiness, I worry about future interop. Respecting a peer's receive window isn't hard. Just do it :)

And even though wget doesn't support upload (to my knowledge, but I'm not an expert), a command line tool may upload, in which case it should definitely respect the peer's receive window.

On Nov 3, 2013, at 1:25 PM, William Chan (陈智昌) <[email-masked]> wrote:

---

From: Michael Sweet <[email-masked]>
Date: Mon, 04 Nov 2013 10:50:51 -0500

Another practical example (and I’m aware that most of you don’t deal with this) is a potential future binding of IPP on top of HTTP/2.0.

One common implementation constraint for printers is memory - when the client is sending printer data (potentially 100’s of megabytes per page) it is quite common for the printer to not keep up - print heads, etc. don’t move at network speeds, and if you get a paper jam, run out of ink, etc. then you need to make sure that you leave a little buffer space for a separate query from the client so that it knows that there is a paper jam, the printer is out of ink, etc. and can report that to the user.

Thanks to multiplexing, HTTP/2.0 offers potential “savings” on the number of simultaneous connections a printer will need to manage.  But the flow control support also allows the printer to tell the client to stop/slow down to prevent a deadlock situation and provides opportunities for the client to do other work or just go to sleep until the printer is ready to accept more data.  This translates into a better user experience and potential power savings.

So really, I would advise folks to not limit HTTP/2.0 to the things you currently do with HTTP/1.1.  Sending data as fast as possible (from the client or from the server) is a laudable goal, but usually the client and server and the links that connect them are not equal in capabilities or bandwidth.  Respecting HTTP/2.0 flow control hints may actually improve the perceived performance of a file send since you can avoid log-jamming packets at the TCP level.

> For a single file, the data that the client sends to the server never exceeds the default server receive window.

____________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Michael Sweet <[email-masked]>
Date: Mon, 04 Nov 2013 11:04:57 -0500

Peter,

On Nov 3, 2013, at 11:04 PM, Peter Lepeska <[email-masked]> wrote:
> If a receiver cannot absorb any more data, it will not make a buffer available to TCP. 
> 
> Don't forget that in HTTP 1.x we don't do flow control. We leave that to the transport layer and this works well. Layering flow control on top of flow control can only result in slower flows. This slowdown is necessary when two or more streams are being sent at once but let's not take this hit in the simple case of one stream.

The problem with relying on TCP-based flow control is that you are forcing retransmissions and log-jamming all access to the other end.  If instead you send your file in chunks sized to the receiver’s capabilities then you can either a) do other useful work or b) go to sleep until the receiver tells you it can accept more data.  Add a small amount of rate-tracking code on the sending side and you should be able to keep the receiver window near full.

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 08:05:12 -0800

Amos,

I agree what you said, but again only when there is more than one active
stream. Again, HTTP 2 flow control is harmless at best when there is only
one active stream.

But you don't have to believe me. Just setup a test a browser that does
flow control and add a few % loss and 200 ms latency and see whether you
are able to download large files faster with flow control on or off. The
flow control off case should never lose, assuming the loss/latency are
regular and your test is long enough.

Peter

On Sun, Nov 3, 2013 at 9:55 PM, Amos Jeffries <[email-masked]> wrote:

> You are thinking only of the case where a client is directly plugged into
> the origin server.
>
> relayed end-to-end the proxy is simply pushing client bytes to the server
> and vice versa an can simply pause reading/writing at any point.
> simultaneously. If the proxy is not explicitly informed about each clients
> receive limit, it has no way to reliably limit data from the server to that
> client and may hit a deadlock situation where its buffer is filled with
> frames awaiting delivery to the slowest client while the faster client is
> starved.
>
> flow back to the clients. If any one client connection is slower than the
> sum of server connections output destined there, the sheer flood of data
> being delivered to it will cause the same form of server connection buffer
> inside the proxy to fill/block with that clients data and starve other
> clients.
>
> sure what the use-case for connection-wide flow control is. Perhapse that
> is best left for the TCP level.
>
> Amos
>
>

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 08:10:49 -0800

" Having such signaling at the HTTP layer gives both sides better control
over their use of buffer space (whether that buffer is in kernel or user
mode)."

Why not just decrease the size of the per socket TCP receive buffers?

I don't see why buffering at the HTTP layer gives us better control over
use of buffer space. Can you explain that?

Peter

On Sun, Nov 3, 2013 at 9:56 PM, Yoav Nir <[email-masked]> wrote:

> I don't have Roberto's experience either, but I do share some experience
> with people who write intermediaries. Receiver (whether the server for
> upload, the client for download, or the intermediary in either case) memory
> is a constrained resource. That is especially true for the server and
> intermediary. To get TCP to work well, the TCP stack needs to have enough
> buffer space to avoid the need to slow the client down by dropping packets.
> If the client uploads at top speed, it could overwhelm the ability of the
> server software to handle, and end up filling up the TCP buffers on the
> server. If one client does that, that's OK. If a lot of clients do that,
> TCP memory runs out, so that limits the number of clients that a single
> server can support.
>
> Even for downloads, if the server application does not get a signal from
> the client application, all it can do is send the whole resource and let
> its TCP stack buffer the whole thing or block the application, so either
> way the entire resource is waiting in memory. That is, of course, a naive
> way to implement a server, and even today servers query TCP to know how
> much data they can send without blocking and/or overfilling the buffers. So
> for download it is solvable. But for uploads, there is nothing that the
> server can do about its TCP buffers filling up faster than it can empty
> them.  Having such signaling at the HTTP layer gives both sides better
> control over their use of buffer space (whether that buffer is in kernel or
> user mode).
>
> Yoav
>
>

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 08:15:44 -0800

"Incorrect, it is buffered on both ends of the TCP connection.
for some other user."

If a specific deployment wants to trade off performance for lower
buffering, then flow control for the single stream case makes sense.
Perhaps this should just be a configurable option. In any case, I'd
definitely want the ability to go as fast as 1.x.

On Sun, Nov 3, 2013 at 9:50 PM, Roberto Peon <[email-masked]> wrote:

>
>
>
> On Sun, Nov 3, 2013 at 9:19 PM, Peter Lepeska <[email-masked]>wrote:
>
>> Roberto,
>>
> As an example, with a 30ms ping time, and a 10Mb connection, you're
> talking ~4MB.
> If the other side of this is contended, I may need to throttle back. I
> also have no need of a full BDP's worth of buffer sitting there doing
> nothing for some other user.
>
>
>>> Speaking as someone who has some implementation experience at scale,
>>> is far likelier to be terminated.
>>>
>>> Resource utilization constraints exist to constrain utilization. Doing
>>> so can impact bandwidth utilization, true, but sometimes this is exactly
>>> what is intended as a tradeoff for some other property, e.g. maximizing
>>>> Don't forget that in HTTP 1.x we don't do flow control. We leave that
>>>> to the transport layer and this works well. Layering flow control on top of
>>>>>> Sloppiness? I don't get that. The sender's job is to transmit the
>>>>>> data as fast as possible, not to respect the receiver's best guesstimate of
>>>>>>> And even though wget doesn't support upload (to my knowledge, but
>>>>>>> I'm not an expert), a command line tool may upload, in which case it should

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 08:25:20 -0800

Hi Michael,

We have no choice but to rely on TCP-based flow control. The question is
whether there is anything to be gained by also relying on HTTP level flow
control when there is only one active transfer. As per the emails with Yoav
and William, I get it that there are cases where a server operator may
decide that the benefits of reduced per connection buffering are worth the
loss in performance and in this case HTTP flow control may make sense.

By default though, it should be no slower than 1.x.

Peter

On Mon, Nov 4, 2013 at 8:04 AM, Michael Sweet <[email-masked]> wrote:

> The problem with relying on TCP-based flow control is that you are forcing
> retransmissions and log-jamming all access to the other end.  If instead
> you send your file in chunks sized to the receiver’s capabilities then you
> can either a) do other useful work or b) go to sleep until the receiver
> tells you it can accept more data.  Add a small amount of rate-tracking
> code on the sending side and you should be able to keep the receiver window
> near full.
>
>
>
> Peter
>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 4 Nov 2013 08:45:08 -0800

I feel like a key thing people have missed here is the API.

TCP kernel flow control API: stop calling read(), let kernel buffers build
up. Problem is that you are not allowed to read data that may be sitting in
the kernel buffers, just because you want to assert flow control. But if an
application protocol like HTTP/2 *requires* you to read data from the
socket, then this is broken. You *must* read data from the socket to
process control frames promptly, or else things may break. Not processing
WINDOW_UPDATEs may lead to data transfer deadlocks. Not processing PING
frames may lead to the peer terminating the connection, since they may be
using PING frames as a liveness check. And if you don't respond to the
PING, then the peer will think the connection is dead and tear down the
transport connection.

HTTP/2 API: everything's going to typically be in userspace. You can assert
HTTP/2 level flow control and still process all data available on the
socket. This is a more powerful API. The problem is flow control window
sizing is a hard problem. Don't use it unless you have to :)

On Mon, Nov 4, 2013 at 8:25 AM, Peter Lepeska <[email-masked]> wrote:

>> The problem with relying on TCP-based flow control is that you are
>> forcing retransmissions and log-jamming all access to the other end.  If
>> instead you send your file in chunks sized to the receiver’s capabilities
>> then you can either a) do other useful work or b) go to sleep until the
>> receiver tells you it can accept more data.  Add a small amount of
>> rate-tracking code on the sending side and you should be able to keep the
>> receiver window near full.
>>
>>
>>
>> Peter
>>

---

From: Michael Sweet <[email-masked]>
Date: Mon, 04 Nov 2013 11:45:24 -0500

Peter,

I’m not suggesting that we don’t rely on TCP-based flow control.

I *am* suggesting that we can do better than TCP-based flow control because we know what is happening at the HTTP level (“I’m sending a file”) and can rate-limit to avoid “log jamming” or “turbulence” caused by TCP-based flow control.  A HTTP sender can learn about the bandwidth limits and latency of the receiver by tracking the timing and values in the window updates and adapt to the constraints of the connection and competing traffic. And HTTP also has proxies, even for single file transfers, which have varying capabilities...

There are many real-world examples that can be used to visualize this, but consider rush hour on a busy highway.  In most places in the US and Canada that I’ve lived, driving during rush hour is something you avoid.  Everyone is trying to get home as quickly as possible, and so inevitably you get a traffic jam and stop-and-go traffic.  California, however, uses so-called “metering lights” to limit the rate that new vehicles enter the highway.  This is far from perfect, but in general it does keep traffic creeping along even when there are a million people trying to get home at the same time.  A distance that used to take me multiple hours to get through Washington, DC would take me an hour or less when I lived in the bay area in California.

On Nov 4, 2013, at 11:25 AM, Peter Lepeska <[email-masked]> wrote:

> Hi Michael,
> 
> We have no choice but to rely on TCP-based flow control. The question is whether there is anything to be gained by also relying on HTTP level flow control when there is only one active transfer. As per the emails with Yoav and William, I get it that there are cases where a server operator may decide that the benefits of reduced per connection buffering are worth the loss in performance and in this case HTTP flow control may make sense.
>

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 05 Nov 2013 10:02:24 +1300

> stream. Again, HTTP 2 flow control is harmless at best when there is 

Part of my point was that there is absolutely no way to determine that 
one active stream cases existence all the way along the path. Middleware 
exists (whether it is visible to the endpoints or not) and the "single 
stream" may be sharing any HTTP hop with one or more other streams.

"At best" this single stream will be able to avoid contention in the 
more common cases where it ceases being a single end-to-end stream at 
some middel hop. So no I think the best-case is rather better than you 
are saying.

> regular and your test is long enough.

At what size data frames? and what relative size of TCP and HTTP layer 
buffer sizes? over how many hops?

In the grand scheme of HTTP, single client going to single server, with 
a single stream and nothing in between is a rather rare occurance. Just 
like it is a rather rare and artificial occurance to see only a single 
isolated TCP connection today.

Amos

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 13:15:13 -0800

On Mon, Nov 4, 2013 at 1:02 PM, Amos Jeffries <[email-masked]> wrote:

> Part of my point was that there is absolutely no way to determine that one
> active stream cases existence all the way along the path. Middleware exists
> (whether it is visible to the endpoints or not) and the "single stream" may
> be sharing any HTTP hop with one or more other streams.
>
> "At best" this single stream will be able to avoid contention in the more
> common cases where it ceases being a single end-to-end stream at some
> middel hop. So no I think the best-case is rather better than you are
> saying.

The sender knows if there are other active streams at the time it has data
to send.

> In the grand scheme of HTTP, single client going to single server, with a
> single stream and nothing in between is a rather rare occurance. Just like
> it is a rather rare and artificial occurance to see only a single isolated
> TCP connection today.

Those questions will not impact your results. When there is available
buffering at the TCP layer, HTTP flow control makes one of two decisions --
send now or send later. When flow control is disabled the answer is always
send now. Therefore no HTTP flow control will always be as fast or faster.

Actually, I don't know for sure but I'd bet the single stream case is the
most common from an overall bytes sent perspective due to http streaming
movies from services like Netflix. In any case, there are many uses of HTTP
that involve one at a time file transfers.

Peter

---

From: Mike Bishop <[email-masked]>
Date: Mon, 4 Nov 2013 22:15:04 +0000

No, the sender *doesn't* know if there's a single stream end-to-end.  It knows whether there's a single stream on the first hop, and it doesn't know whether that hop is the only hop.

If a client is directly connected to a server and that's the only request that will ever be made, then you're correct -- you may get some improved performance by disabling flow control, and we provide that option in the spec.  But the server, at least, doesn't know whether it's dealing with a proxy or a direct client.  The client can inform its next hop whether it wants flow control on that hop.  If that next hop is a proxy, it's the proxy's decision whether it wants flow control, and that's an independent choice.

Sent from Windows Mail

From: Peter Lepeska<mailto:[email-masked]>
Sent: ?Monday?, ?November? ?4?, ?2013 ?1?:?17? ?PM
To: Amos Jeffries<mailto:[email-masked]>
Cc: HTTP Working Group<mailto:[email-masked]>

On Mon, Nov 4, 2013 at 1:02 PM, Amos Jeffries <[email-masked]<mailto:[email-masked]>> wrote:
On 2013-11-05 05:05, Peter Lepeska wrote:
Amos,

Part of my point was that there is absolutely no way to determine that one active stream cases existence all the way along the path. Middleware exists (whether it is visible to the endpoints or not) and the "single stream" may be sharing any HTTP hop with one or more other streams.

"At best" this single stream will be able to avoid contention in the more common cases where it ceases being a single end-to-end stream at some middel hop. So no I think the best-case is rather better than you are saying.

The sender knows if there are other active streams at the time it has data to send.

At what size data frames? and what relative size of TCP and HTTP layer buffer sizes? over how many hops?

In the grand scheme of HTTP, single client going to single server, with a single stream and nothing in between is a rather rare occurance. Just like it is a rather rare and artificial occurance to see only a single isolated TCP connection today.

Those questions will not impact your results. When there is available buffering at the TCP layer, HTTP flow control makes one of two decisions -- send now or send later. When flow control is disabled the answer is always send now. Therefore no HTTP flow control will always be as fast or faster.

Actually, I don't know for sure but I'd bet the single stream case is the most common from an overall bytes sent perspective due to http streaming movies from services like Netflix. In any case, there are many uses of HTTP that involve one at a time file transfers.

Peter

Amos

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 14:20:17 -0800

Okay I get the misunderstanding now.

I'm not proposing that the sender determines that there is just one active
stream end to end. It just checks to see if it knows of other active
streams. If it does not, then it ignores flow control. If in fact there are
other active streams end to end, then it will not be in the "ignoring flow
control" state for long.

Peter

On Mon, Nov 4, 2013 at 2:15 PM, Mike Bishop <[email-masked]>wrote:

>  No, the sender *doesn’t* know if there’s a single stream end-to-end.  It
> knows whether there’s a single stream on the first hop, and it doesn’t know
> whether that hop is the only hop.
>
>  If a client is directly connected to a server and that’s the only
> request that will ever be made, then you’re correct -- you may get some
> improved performance by disabling flow control, and we provide that option
> in the spec.  But the server, at least, doesn’t know whether it’s dealing
> with a proxy or a direct client.  The client can inform its next hop
> whether it wants flow control on that hop.  If that next hop is a proxy,
> it’s the proxy’s decision whether it wants flow control, and that’s an
> independent choice.
>
>  Sent from Windows Mail
>
>   *From:* Peter Lepeska <[email-masked]>
> *Sent:* Monday, November 4, 2013 1:17 PM
> *To:* Amos Jeffries <[email-masked]>
> *Cc:* HTTP Working Group <[email-masked]>
>
>
>
>
> On Mon, Nov 4, 2013 at 1:02 PM, Amos Jeffries <[email-masked]>wrote:
>
>  Actually, I don't know for sure but I'd bet the single stream case is
> the most common from an overall bytes sent perspective due to http
> streaming movies from services like Netflix. In any case, there are many
> uses of HTTP that involve one at a time file transfers.
>
>  Peter
>
>>
>>
>> Amos
>>
>>
>>
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 4 Nov 2013 15:58:30 -0800

This thread might be useful fodder for improving understanding of the context of flow control — esp. sections 5.2.1 and 5.2.2.

Cheers,

On 4 Nov 2013, at 2:20 pm, Peter Lepeska <[email-masked]> wrote:

> I'm not proposing that the sender determines that there is just one active stream end to end. It just checks to see if it knows of other active streams. If it does not, then it ignores flow control. If in fact there are other active streams end to end, then it will not be in the "ignoring flow control" state for long.
> 
> Peter
> 
> 
> On Mon, Nov 4, 2013 at 2:15 PM, Mike Bishop <[email-masked]> wrote:
> No, the sender *doesn’t* know if there’s a single stream end-to-end.  It knows whether there’s a single stream on the first hop, and it doesn’t know whether that hop is the only hop.
> 
> If a client is directly connected to a server and that’s the only request that will ever be made, then you’re correct -- you may get some improved performance by disabling flow control, and we provide that option in the spec.  But the server, at least, doesn’t know whether it’s dealing with a proxy or a direct client.  The client can inform its next hop whether it wants flow control on that hop.  If that next hop is a proxy, it’s the proxy’s decision whether it wants flow control, and that’s an independent choice.
> 
> Sent from Windows Mail
> 
> From: Peter Lepeska
> Sent: Monday, November 4, 2013 1:17 PM
> To: Amos Jeffries
> Cc: HTTP Working Group
> 
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Roberto Peon <[email-masked]>
Date: Mon, 4 Nov 2013 16:08:53 -0800

I'll point out again, that the proposal of ignoring flow control at any
time implies that one MUST have a synchronization event when one
transitions from ignoring to adhering to flow control. With the proposal of
ignoring when only having one stream, *which is the common case for the
beginning of browsing*, it would cause de-synchronization, possible
deadlock, etc.

Even if I liked the proposal, which I don't, it would need to be completed
with adequate synchronization for the transition from 1 stream to multiple
streams, and vice versa (what happens when you go from 2-1-2 streams?).

Dealing with this would require additional complexity far in excess of
simply following the flow control semantic as it is defined today.

-=R

On Mon, Nov 4, 2013 at 2:20 PM, Peter Lepeska <[email-masked]> wrote:

>>  No, the sender *doesn’t* know if there’s a single stream end-to-end.
>> It knows whether there’s a single stream on the first hop, and it doesn’t
>> know whether that hop is the only hop.
>>
>>> more common cases where it ceases being a single end-to-end stream at some

---

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Nov 2013 16:13:56 -0800

On 4 November 2013 16:08, Roberto Peon <[email-masked]> wrote:
> I'll point out again, that the proposal of ignoring flow control at any time
> implies that one MUST have a synchronization event when one transitions from
> ignoring to adhering to flow control.

We do not have this mechanism.  Unless it is added, there is no way
that you can restart flow control.  I haven't seen any desire to add
such a mechanism.  The best way I know of to restart flow control is
to send GOAWAY and to start a new connection.

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 16:32:30 -0800

Maybe flow control is off until the receiver sends its first window update.
If the connection is used for a single file transfer, then no flow control
will take place. Also, if the receiver never experiences back pressure
(meaning its never unable to absorb the data at the rate it is received),
then flow control never has to happen.

Peter

On Mon, Nov 4, 2013 at 4:13 PM, Martin Thomson <[email-masked]>wrote:

> time

---

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Nov 2013 16:34:52 -0800

> Maybe flow control is off until the receiver sends its first window update.

I'm afraid that doesn't work.  Once flow control is off, you need a
clear synchronization point upon which to restart it.  We haven't
defined that and my sense is that there is no desire to take on that
complexity.

If you don't want flow control, the SETTINGS frame that you send
before anything else can be used to disable it (for you, you still
have to act responsibly as a packet sender).

---

From: Peter Lepeska <[email-masked]>
Date: Mon, 4 Nov 2013 16:36:03 -0800

Actually not sure this works b/c I think you still have the sync problem in
this case.

On Mon, Nov 4, 2013 at 4:32 PM, Peter Lepeska <[email-masked]> wrote:

> Maybe flow control is off until the receiver sends its first window
> update. If the connection is used for a single file transfer, then no flow
> control will take place. Also, if the receiver never experiences back
> pressure (meaning its never unable to absorb the data at the rate it is
> received), then flow control never has to happen.
>
> Peter
>
>