Subject: [tcpm] A Challenge to TCPM

From: Mike Belshe <[email-masked]>
Date: Fri, 26 Nov 2010 10:10:31 -0800

Hi,

I thought I'd add a slightly different perspective to the discussion about
initial cwnd.  I'm a developer on Chrome & SPDY.

With SPDY, we want to build an application protocol which is efficient on
the transport, good for the network, and faster than HTTP.  But right now,
TCP is holding us back because we're trying to accomplish with one
connection what HTTP does with 10+ connections.

Note:  I don't want to make this thread about SPDY.  I hope the members of
this list agree that the goals of SPDY are good and will help make TCP a
friendly transport for an application protocol like SPDY.

*Background*
Unrelated to TCP, HTTP is not very efficient.  This has been widely
documented and observed for some time.  We can easily demonstrate a more
efficient protocol, which transfers 25-35% fewer packets than HTTP and
reduces bytes transferred by 10-15%.  Hopefully everyone agrees those are
good things for the network.

Unfortunately, any TCP-based protocol which uses a single connection is at a
latency-handicap compared to any protocol that uses multiple connections.
 This is due to IW.  Further, because bandwidth increases have outpaced RTT
decreases, bandwidth is relatively cheap.  So, being efficient on your
bandwidth is less critical than being efficient on your round-trips.

To visualize the effects, please see the attached graph.  (Or click
here<http://6541078575799853287-a-chromium-org-s-sites.googlegroups.com/a/chromium.org/dev/spdy/effect_of_initial_cwnd_on_plt(1).png?attachauth=ANoY7cpm-y5NA-zFvGsmxlK81vUh56WsFe-2l5BOkZxENuopgeUcSmO9-QT9sc784xFZLw0QkSBxmDKgCLxWoVI06vy1u3a7RuS82Nb6YuTHPGeqU_W1c1niKx5W3CzKbzsxnGKlQaectqG8a7cjg5vgU1Gj4cFLtTi7rKiKHF12j_XkANIwSM0saVdoF6c6gtCDg1_8gKIn8QPVSVp9Cwtw1uFTTWw93XFPQkAmEI2FvRbq6IBg1aI%3D&attredirects=0>)
 In these tests, I loaded a replica of the facebook landing page through two
protocols - one using a single connection in a semi-efficient manner (an
early version of spdy) and another using 6 connections (http).  I used a
high-bandwidth network with either 100ms or 200ms of RTT, and then I varied
IW from 2pkts all the way to 128pkts.  I chose this network configuration to
specifically isolate the effects of high-RTT and IW.  As you can see, at low
IW, the single-connection protocol fares poorly compared to the
multi-connection protocol.   For another writeup on the problem, see
http://dev.chromium.org/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf

Today, the fastest way to deliver content is to simply side-step cwnd by use
of multiple connections.

*The Problem*
With SPDY (or whatever protocol comes next), we can definitely reduce the
number of connections out of the browser by *at least* a factor of 6.  With
unfolding of the subdomain tricks that websites have been doing, we can
reduce it by a factor of 10.  This has nothing to do with TCP, except I hope
that folks on this list appreciate that this goodness:  fewer connections -
fewer packets - and fewer bytes.

The only trouble is that *if we abide to the RFCs, fewer connections + fewer
packets + fewer bytes *equates to* slower web pages. *

But I'm not asking that we increase IW by a factor of 10; because it grows
exponentially, if we can increase it to somewhere in the ~16 range, it's
good enough.**

*The Conundrum*
More perplexing is that whenever we talk about increasing IW, we tend to
think of the static throttle that we have today.  If we increase a static
throttle, it affects everything.  So protocols that are already bursting
20-30x more packets than they should (by way of many connections) will burst
even more.

This makes some people want to do nothing...

At the same time, we also know that cwnd has already been increased on the
net at large.   This started with use of multiple connections, multiplied
with the use of multiple subdomains, and is now changing by major sites
simply increasing or ignoring cwnd altogther.  With so many sites
systematically working around cwnd, we know that the current limits can be
increased.  We just don't know how much.

*The Solution?*
Is the obvious solution to stop using a static value?  It doesn't seem like
we're in a one-size-fits-all for IW.

The IW could either be defined to be protocol-specific, or dynamic based on
number of TCP flows between the same two endpoints.  Both of these
approaches would offer the benefit of a controlled, increased cwnd to
reflect the changes in the net over the last 2 decades but also not
completely unthrottle protocols that are already living on the edge of not
having throttles.

Finally, if a dynamic throttle is too much to decide upon, at least updating
the static value seems appropriate.  Jerrry and Nandita have provided good
data for a general-purpose increase to 10.  If inefficient protocols are
allowed to go that high, maybe more efficient protocols could be allowed to
go higher (with data, of course).

*Reality Check*
Finally, regardless of this debate in the IETF, what is actually going to
happen?

a) IETF does not increase the limit (or continues to wait):
As time goes on, more and more evidence emerges to demonstrate that the
current cwnd limits are unnecessarily conservative.  More evidence also
emerges that increasing cwnd makes your site faster.  More sites start
experimenting with their own values of cwnd.  Without a blessed
recommendation from the IETF, sites choose whatever values they want. Ben
Strong offered a writeup today which demonstrates this is already
happening<http://blog.benstrong.com/2010/11/google-and-microsoft-cheat-on-slow.html>
.

b) IETF increases the limit:
With data and a specific recommendation, TCP vendors can quickly move to the
new recommendations, making their products more competitive in the
marketplace, and better for users.  Pressure is alleviated from existing
sites to go renegade, and instead they can rest on the specific findings
from this group.  With a modern recommendation in hand, all of us can
credibly go after any sites which do choose to use dangerously high cwnd
values.  Finally, having an officially increased IW is an enabler for a
future of more efficient application level protocols, which are currently at
a disadvantage to their less-efficient counterparts.

Now, which of these two options carries more risk of internet collapse?
 Vendors implementing whatever-the-heck-their-rookie-system-administrator
decided to configure?  Or an official, controlled and intelligent
recommendation from this group?

Thanks to everyone on this list for their hard work over the years -
although my email is somewhat critical, be aware that I have immense
gratitude and respect for the veterans of this list that have been keeping
the net flowing for so long with such great success.
Mike

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Sat, 27 Nov 2010 12:34:26 -0800

I agree with some of the claims you are making below, and I don't agree with some. I just wanna point out one thing, though. You are largely dismissing the #1 reason why applications are often using more than 1 TCP connection. It is not often about the tput. It is often (more than often actually) about the delay (among a few other reasons), which is critical for short-lived transactions.

You can increase the IW to 100 for all I care but TCP will still have to leave with the HOL blocking issue. IETF can make a recommendation on IW but I have serious doubts about TCP vendors quickly adopting as you suggest. As for the number of TCP connections in parallel, that is not easy to place a policy anyway.

Maybe, trying to fix TCP to make HTTP is not the best strategy here.

-acbegen

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Belshe
> Sent: Friday, November 26, 2010 1:11 PM
> To: [email-masked]
> Subject: [tcpm] A Challenge to TCPM
> 
> Hi,
> 
> I thought I'd add a slightly different perspective to the discussion about initial cwnd.  I'm a developer on Chrome & SPDY.
> 
> With SPDY, we want to build an application protocol which is efficient on the transport, good for the network, and faster than
> HTTP.  But right now, TCP is holding us back because we're trying to accomplish with one connection what HTTP does with
> 10+ connections.
> 
> Note:  I don't want to make this thread about SPDY.  I hope the members of this list agree that the goals of SPDY are good
> and will help make TCP a friendly transport for an application protocol like SPDY.
> 
> Background
> Unrelated to TCP, HTTP is not very efficient.  This has been widely documented and observed for some time.  We can easily
> demonstrate a more efficient protocol, which transfers 25-35% fewer packets than HTTP and reduces bytes transferred by
> 10-15%.  Hopefully everyone agrees those are good things for the network.
> 
> Unfortunately, any TCP-based protocol which uses a single connection is at a latency-handicap compared to any protocol that
> uses multiple connections.  This is due to IW.  Further, because bandwidth increases have outpaced RTT decreases,
> bandwidth is relatively cheap.  So, being efficient on your bandwidth is less critical than being efficient on your round-trips.
> 
> To visualize the effects, please see the attached graph.  (Or click here <http://6541078575799853287-a-chromium-org-s-
> sites.googlegroups.com/a/chromium.org/dev/spdy/effect_of_initial_cwnd_on_plt(1).png?attachauth=ANoY7cpm-y5NA-
> zFvGsmxlK81vUh56WsFe-2l5BOkZxENuopgeUcSmO9-
> QT9sc784xFZLw0QkSBxmDKgCLxWoVI06vy1u3a7RuS82Nb6YuTHPGeqU_W1c1niKx5W3CzKbzsxnGKlQaectqG8a7cjg5vgU1Gj4c
> FLtTi7rKiKHF12j_XkANIwSM0saVdoF6c6gtCDg1_8gKIn8QPVSVp9Cwtw1uFTTWw93XFPQkAmEI2FvRbq6IBg1aI%3D&attredirect
> s=0> )  In these tests, I loaded a replica of the facebook landing page through two protocols - one using a single connection in
> a semi-efficient manner (an early version of spdy) and another using 6 connections (http).  I used a high-bandwidth network
> with either 100ms or 200ms of RTT, and then I varied IW from 2pkts all the way to 128pkts.  I chose this network
> configuration to specifically isolate the effects of high-RTT and IW.  As you can see, at low IW, the single-connection protocol
> fares poorly compared to the multi-connection protocol.   For another writeup on the problem, see
> Today, the fastest way to deliver content is to simply side-step cwnd by use of multiple connections.
> 
> 
> The Problem
> With SPDY (or whatever protocol comes next), we can definitely reduce the number of connections out of the browser by *at
> least* a factor of 6.  With unfolding of the subdomain tricks that websites have been doing, we can reduce it by a factor of
> 10.  This has nothing to do with TCP, except I hope that folks on this list appreciate that this goodness:  fewer connections -
> The only trouble is that if we abide to the RFCs, fewer connections + fewer packets + fewer bytes equates to slower web
> pages.
> 
> But I'm not asking that we increase IW by a factor of 10; because it grows exponentially, if we can increase it to somewhere
> in the ~16 range, it's good enough.
> 
> 
> The Conundrum
> More perplexing is that whenever we talk about increasing IW, we tend to think of the static throttle that we have today.  If
> we increase a static throttle, it affects everything.  So protocols that are already bursting 20-30x more packets than they
> should (by way of many connections) will burst even more.
> 
> At the same time, we also know that cwnd has already been increased on the net at large.   This started with use of multiple
> connections, multiplied with the use of multiple subdomains, and is now changing by major sites simply increasing or
> ignoring cwnd altogther.  With so many sites systematically working around cwnd, we know that the current limits can be
> The Solution?
> Is the obvious solution to stop using a static value?  It doesn't seem like we're in a one-size-fits-all for IW.
> 
> The IW could either be defined to be protocol-specific, or dynamic based on number of TCP flows between the same two
> endpoints.  Both of these approaches would offer the benefit of a controlled, increased cwnd to reflect the changes in the net
> over the last 2 decades but also not completely unthrottle protocols that are already living on the edge of not having
> throttles.
> 
> Finally, if a dynamic throttle is too much to decide upon, at least updating the static value seems appropriate.  Jerrry and
> Nandita have provided good data for a general-purpose increase to 10.  If inefficient protocols are allowed to go that high,
> maybe more efficient protocols could be allowed to go higher (with data, of course).
> 
> 
> Reality Check
> Finally, regardless of this debate in the IETF, what is actually going to happen?
> 
> As time goes on, more and more evidence emerges to demonstrate that the current cwnd limits are unnecessarily
> conservative.  More evidence also emerges that increasing cwnd makes your site faster.  More sites start experimenting with
> their own values of cwnd.  Without a blessed recommendation from the IETF, sites choose whatever values they want. Ben
> Strong offered a writeup today which demonstrates this is already happening <http://blog.benstrong.com/2010/11/google-
> and-microsoft-cheat-on-slow.html> .
> 
> With data and a specific recommendation, TCP vendors can quickly move to the new recommendations, making their
> products more competitive in the marketplace, and better for users.  Pressure is alleviated from existing sites to go
> renegade, and instead they can rest on the specific findings from this group.  With a modern recommendation in hand, all of
> us can credibly go after any sites which do choose to use dangerously high cwnd values.  Finally, having an officially
> increased IW is an enabler for a future of more efficient application level protocols, which are currently at a disadvantage to
> their less-efficient counterparts.
> 
> 
> 
> 
> Now, which of these two options carries more risk of internet collapse?  Vendors implementing whatever-the-heck-their-
> rookie-system-administrator decided to configure?  Or an official, controlled and intelligent recommendation from this
> group?
> 
> 
> Thanks to everyone on this list for their hard work over the years - although my email is somewhat critical, be aware that I
> have immense gratitude and respect for the veterans of this list that have been keeping the net flowing for so long with such
> great success.
> Mike

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 28 Nov 2010 11:39:45 -0000

Hi,

I'm also very interested in fixing head-of-line blocking. However, that
is IMHO orthogonal to reducing small flow completion times. More to the
point, if numerous applications were really concerned about HOL
blocking, you'd expect that they run (idempotent) IO calls over more
than a single TCP session, and go with the earlier response. However,
I'm not aware that such a behavior is widely deployed in the public
internet. Do you have pointers to show how common such protocols are?
(Online Gaming with their proprietary protocols comes to mind. But I
believe many games use UDP).

Also, there already exists another protocol, that can deliver objects to
the receiver, even if some earlier in-sequence data is missing (and
blocks the delivery of an earlier object): SCTP.

Anyway, TCP still has a little bit of room to improve error recovery
latency. Ie. using commonly used extensions in a more synergistic way
can allow loss recovery in ~1RTT under many more circumstances than
today. Improving dupthresh and RTTM algorithms would also reduce the
fraction of RTO loss recovery.

See also:
S. Rewaskar, J. Kaur, and F.D. Smith, "A Performance Study of Loss
Detection/Recovery in Real-world TCP Implementations" 
<http://www.cs.unc.edu/%7Ejasleen/papers/icnp07.pdf>, in Proceedings of
the IEEE International Conference on Network Protocols (ICNP'07),
Beijing, China, Oct 2007.

Richard Scheffenegger

> From: Ali C. Begen (abegen) [mailto:[email-masked]] 
> Sent: Samstag, 27. November 2010 21:34
> To: Mike Belshe; [email-masked]
> Subject: Re: [tcpm] A Challenge to TCPM
> 
> I agree with some of the claims you are making below, and I 
> don't agree with some. I just wanna point out one thing, 
> though. You are largely dismissing the #1 reason why 
> applications are often using more than 1 TCP connection. It 
> is not often about the tput. It is often (more than often 
> actually) about the delay (among a few other reasons), which 
> is critical for short-lived transactions.
> 
> You can increase the IW to 100 for all I care but TCP will 
> still have to leave with the HOL blocking issue. IETF can 
> make a recommendation on IW but I have serious doubts about 
> TCP vendors quickly adopting as you suggest. As for the 
> number of TCP connections in parallel, that is not easy to 
> place a policy anyway.
> 
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Mike Belshe
> > I thought I'd add a slightly different perspective to the 
> discussion about initial cwnd.  I'm a developer on Chrome & SPDY.
> > 
> > With SPDY, we want to build an application protocol which 
> is efficient 
> > on the transport, good for the network, and faster than HTTP.  But 
> > right now, TCP is holding us back because we're trying to 
> accomplish 
> > with one connection what HTTP does with
> > 10+ connections.
> > 
> > Note:  I don't want to make this thread about SPDY.  I hope the 
> > members of this list agree that the goals of SPDY are good 
> > documented and observed for some time.  We can easily demonstrate a 
> > more efficient protocol, which transfers 25-35% fewer 
> packets than HTTP and reduces bytes transferred by 10-15%.  
> Hopefully everyone agrees those are good things for the network.
> > 
> > Unfortunately, any TCP-based protocol which uses a single 
> connection 
> > is at a latency-handicap compared to any protocol that uses 
> multiple 
> > connections.  This is due to IW.  Further, because 
> bandwidth increases have outpaced RTT decreases, bandwidth is 
> relatively cheap.  So, being efficient on your bandwidth is 
> less critical than being efficient on your round-trips.
> > 
> > here <http://6541078575799853287-a-chromium-org-s-
> > 
> > s=0> )  In these tests, I loaded a replica of the facebook landing 
> > page through two protocols - one using a single connection in a 
> > semi-efficient manner (an early version of spdy) and 
> another using 6 
> > connections (http).  I used a high-bandwidth network with 
> either 100ms or 200ms of RTT, and then I varied IW from 2pkts 
> all the way to 128pkts.  I chose this network configuration 
> to specifically isolate the effects of high-RTT and IW.  As 
> you can see, at low IW, the single-connection protocol
> > Today, the fastest way to deliver content is to simply 
> side-step cwnd by use of multiple connections.
> > 
> > 
> > The Problem
> > With SPDY (or whatever protocol comes next), we can 
> definitely reduce 
> > the number of connections out of the browser by *at
> > least* a factor of 6.  With unfolding of the subdomain tricks that 
> > websites have been doing, we can reduce it by a factor of 10.  This 
> > has nothing to do with TCP, except I hope that folks on 
> this list appreciate that this goodness:  fewer connections - 
> > The only trouble is that if we abide to the RFCs, fewer 
> connections + 
> > fewer packets + fewer bytes equates to slower web pages.
> > 
> > But I'm not asking that we increase IW by a factor of 10; 
> because it 
> > grows exponentially, if we can increase it to somewhere in 
> the ~16 range, it's good enough.
> > 
> > 
> > The Conundrum
> > More perplexing is that whenever we talk about increasing 
> IW, we tend 
> > to think of the static throttle that we have today.  If we 
> increase a 
> > static throttle, it affects everything.  So protocols that 
> are already bursting 20-30x more packets than they should (by 
> way of many connections) will burst even more.
> > 
> > At the same time, we also know that cwnd has already been 
> increased on the net at large.   This started with use of multiple
> > connections, multiplied with the use of multiple subdomains, and is 
> > now changing by major sites simply increasing or ignoring cwnd 
> > altogther.  With so many sites systematically working 
> around cwnd, we know that the current limits can be 
> > Is the obvious solution to stop using a static value?  It 
> doesn't seem like we're in a one-size-fits-all for IW.
> > 
> > The IW could either be defined to be protocol-specific, or dynamic 
> > based on number of TCP flows between the same two 
> endpoints.  Both of 
> > these approaches would offer the benefit of a controlled, increased 
> > cwnd to reflect the changes in the net over the last 2 
> decades but also not completely unthrottle protocols that are 
> already living on the edge of not having throttles.
> > 
> > Finally, if a dynamic throttle is too much to decide upon, at least 
> > updating the static value seems appropriate.  Jerrry and 
> Nandita have 
> > provided good data for a general-purpose increase to 10.  
> If inefficient protocols are allowed to go that high, maybe 
> more efficient protocols could be allowed to go higher (with 
> data, of course).
> > 
> > 
> > Reality Check
> > As time goes on, more and more evidence emerges to demonstrate that 
> > the current cwnd limits are unnecessarily conservative.  
> More evidence 
> > also emerges that increasing cwnd makes your site faster.  
> More sites 
> > start experimenting with their own values of cwnd.  Without 
> a blessed 
> > recommendation from the IETF, sites choose whatever values 
> they want. 
> > Ben Strong offered a writeup today which demonstrates this 
> is already 
> > happening <http://blog.benstrong.com/2010/11/google-
> > With data and a specific recommendation, TCP vendors can 
> quickly move 
> > to the new recommendations, making their products more 
> competitive in 
> > the marketplace, and better for users.  Pressure is alleviated from 
> > existing sites to go renegade, and instead they can rest on the 
> > specific findings from this group.  With a modern recommendation in 
> > hand, all of us can credibly go after any sites which do 
> choose to use dangerously high cwnd values.  Finally, having 
> an officially increased IW is an enabler for a future of more 
> efficient application level protocols, which are currently at 
> > Now, which of these two options carries more risk of internet 
> > collapse?  Vendors implementing whatever-the-heck-their- 
> > rookie-system-administrator decided to configure?  Or an 
> official, controlled and intelligent recommendation from this group?
> > 
> > 
> > gratitude and respect for the veterans of this list that 
> have been keeping the net flowing for so long with such great success.
> > Mike
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Sun, 28 Nov 2010 05:07:58 -0800

> From: Scheffenegger, Richard [mailto:[email-masked]]
> Sent: Sunday, November 28, 2010 6:40 AM
> To: Ali C. Begen (abegen); Mike Belshe; [email-masked]
> Subject: RE: [tcpm] A Challenge to TCPM
> 
> 
> 
> Hi,
> 

Let me clarify this a bit.

If your app is not very concerned about the delay but rather tput in general and that is why it is opening multiple connections, then we may call it unfriendly. And probably no matter what we do, they won't change their behavior much (they could use iw of 10, 20 or whatever but they are likely to continue using multiple connections since in general this will favor them or they will "feel" so).

My point was that in the apps where the user is worried about the delay, HOL blocking is a big problem and opening an alternative TCP is usually the temp solution for this. Giving this guy the flexibility of using iw of 10 or 20 does not help.

> internet. Do you have pointers to show how common such protocols are?

One major example is the streaming video (over http). It is already using most of the consumer IP traffic today and it will suck even more next year. And yes, the clients are smart enough to go around the TCP related problems. Some run idempotent connections to different representations of the content and/or from different source while some do even trickier things with multiple TCPs (but I don’t wanna go into the details here).

In these apps, IW is a secondary issue. I am not saying it would not help but there are bigger problems than a conservative IW here.

> believe many games use UDP).

Indeed.
 
> blocks the delivery of an earlier object): SCTP.

Sure but HTTP is not using SCTP today. Should we look into it?
  
> fraction of RTO loss recovery.

Could not agree more but there are already a lot of these things in the newer TCP stacks (like win 7) that have improved a lot over earlier versions. 

BTW, I think you probably got my question below wrong. I suggested maybe HTTP itself needs fixing or maybe using something besides TCP for HTTP will provide better benefits.
 
> Beijing, China, Oct 2007.

Thanks for the pointer.
-acbegen

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 28 Nov 2010 14:20:44 -0500

AFAIR, all SCTP streams in one connection share the same
receive window. If one stream fills the window, it blocks
all others.

SCTP streams were not designed to avoid HOL blocking.

--
Jakob Heitz.

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Scheffenegger, Richard
> Sent: Sunday, November 28, 2010 3:40 AM

---

From: Xiangsong Cui <[email-masked]>
Date: Mon, 29 Nov 2010 10:33:36 +0800

> Sure but HTTP is not using SCTP today. Should we look into it?

Sorry for digging this topic in tcpm list, but I think the guys interested
in this topic also cover tcpm.

I think this is an interesting item.

In my remember, a Bar-BoF (or BoF) named "Negotiation BoF" was ever
mentioned at IETF76 (denied by AD at that time).

There is a draft, http://tools.ietf.org/html/draft-wing-http-new-tech-00, 
and Dan presented this topic at IETF79,
http://www.ietf.org/proceedings/79/slides/tsvwg-9.pptx,

Who can tell me more information about this? I missed the BoF at IETF76.

Thanks and regards,
Xiangsong

---

From: Mike Belshe <[email-masked]>
Date: Sun, 28 Nov 2010 22:46:39 -0800

On Sat, Nov 27, 2010 at 12:34 PM, Ali C. Begen (abegen) <[email-masked]>wrote:

> I agree with some of the claims you are making below, and I don't agree
> with some. I just wanna point out one thing, though. You are largely
> dismissing the #1 reason why applications are often using more than 1 TCP
> connection. It is not often about the tput. It is often (more than often
> actually) about the delay (among a few other reasons), which is critical for
> short-lived transactions.
>

Actually, that was my whole point.  The only way to get the latency down is
to avoid cwnd, and multiple connections are an easy trick to accomplish just
that.

>
> You can increase the IW to 100 for all I care but TCP will still have to
> leave with the HOL blocking issue. IETF can make a recommendation on IW but
> I have serious doubts about TCP vendors quickly adopting as you suggest. As
> for the number of TCP connections in parallel, that is not easy to place a
> policy anyway.
>

That is a separate issue and generally less severe than the cwnd issue.  I
could provide more color about SCTP, HOL and other issues, but I really
don't want to derail this thread to be about anything other than TCP's
initial cwnd setting.

We need to decide if we think an application protocol should be able to be
both a low-latency protocol, and based on TCP.  If you think that should be
possible, then we seriously need to consider changing IW.  The reasons not
to change it, would be if you thought it would kill the net - but, I there
is no evidence of that, especially if we consider which protocols get which
limits.

> Maybe, trying to fix TCP to make HTTP is not the best strategy here.

Are you saying HTTP should move to a new transport?  That seems like a
shame.

Mike

> > I thought I'd add a slightly different perspective to the discussion
> about initial cwnd.  I'm a developer on Chrome & SPDY.
> >
> the transport, good for the network, and faster than
> > HTTP.  But right now, TCP is holding us back because we're trying to
> accomplish with one connection what HTTP does with
> > 10+ connections.
> >
> > Note:  I don't want to make this thread about SPDY.  I hope the members
> of this list agree that the goals of SPDY are good
> documented and observed for some time.  We can easily
> > demonstrate a more efficient protocol, which transfers 25-35% fewer
> packets than HTTP and reduces bytes transferred by
> > Unfortunately, any TCP-based protocol which uses a single connection is
> at a latency-handicap compared to any protocol that
> > uses multiple connections.  This is due to IW.  Further, because
> bandwidth increases have outpaced RTT decreases,
> > bandwidth is relatively cheap.  So, being efficient on your bandwidth is
> > To visualize the effects, please see the attached graph.  (Or click here
> <http://6541078575799853287-a-chromium-org-s-
> >
> > s=0> )  In these tests, I loaded a replica of the facebook landing page
> through two protocols - one using a single connection in
> > a semi-efficient manner (an early version of spdy) and another using 6
> connections (http).  I used a high-bandwidth network
> > configuration to specifically isolate the effects of high-RTT and IW.  As
> number of connections out of the browser by *at
> websites have been doing, we can reduce it by a factor of
> > 10.  This has nothing to do with TCP, except I hope that folks on this
> list appreciate that this goodness:  fewer connections -
> > The only trouble is that if we abide to the RFCs, fewer connections +
> > But I'm not asking that we increase IW by a factor of 10; because it
> grows exponentially, if we can increase it to somewhere
> think of the static throttle that we have today.  If
> > we increase a static throttle, it affects everything.  So protocols that
> are already bursting 20-30x more packets than they
> > At the same time, we also know that cwnd has already been increased on
> the net at large.   This started with use of multiple
> > connections, multiplied with the use of multiple subdomains, and is now
> changing by major sites simply increasing or
> > ignoring cwnd altogther.  With so many sites systematically working
> > Is the obvious solution to stop using a static value?  It doesn't seem
> like we're in a one-size-fits-all for IW.
> >
> > The IW could either be defined to be protocol-specific, or dynamic based
> on number of TCP flows between the same two
> > endpoints.  Both of these approaches would offer the benefit of a
> controlled, increased cwnd to reflect the changes in the net
> > over the last 2 decades but also not completely unthrottle protocols that
> are already living on the edge of not having
> > throttles.
> >
> > Nandita have provided good data for a general-purpose increase to 10.  If
> inefficient protocols are allowed to go that high,
> current cwnd limits are unnecessarily
> > conservative.  More evidence also emerges that increasing cwnd makes your
> site faster.  More sites start experimenting with
> > their own values of cwnd.  Without a blessed recommendation from the
> IETF, sites choose whatever values they want. Ben
> > products more competitive in the marketplace, and better for users.
>  Pressure is alleviated from existing sites to go
> > renegade, and instead they can rest on the specific findings from this
> group.  With a modern recommendation in hand, all of
> > us can credibly go after any sites which do choose to use dangerously
> high cwnd values.  Finally, having an officially
> > increased IW is an enabler for a future of more efficient application
> level protocols, which are currently at a disadvantage to
>  Vendors implementing whatever-the-heck-their-
> > rookie-system-administrator decided to configure?  Or an official,
> controlled and intelligent recommendation from this
> > group?
> >
> >
> although my email is somewhat critical, be aware that I
> > have immense gratitude and respect for the veterans of this list that

---

From: Mike Belshe <[email-masked]>
Date: Sun, 28 Nov 2010 22:51:30 -0800

> SCTP streams were not designed to avoid HOL blocking.

Again - HOL is a whole different set of issues, not directly related to
initial cwnd.  Let's take that to a different thread.  (And yes, SCTP *was*
designed to avoid HOL issues - see packet loss recovery).

But congestion control, of course, is intentional throttling of the
protocol, regardless of what it does with streams.  It's intentionally
blocking for the safety of the net.  SCTP uses the same static limits that
TCP uses.  So if we can't solve cwnd for TCP, SCTP won't be viable either.

Mike

---

From: Qin Wu <[email-masked]>
Date: Mon, 29 Nov 2010 15:31:25 +0800

----- Original Message ----- 
  From: Mike Belshe 
  To: Ali C. Begen (abegen) 
  Cc: [email-masked] 
  Sent: Monday, November 29, 2010 2:46 PM
  Subject: Re: [tcpm] A Challenge to TCPM

  On Sat, Nov 27, 2010 at 12:34 PM, Ali C. Begen (abegen) <[email-masked]> wrote:

  Actually, that was my whole point.  The only way to get the latency down is to avoid cwnd, and multiple connections are an easy trick to accomplish just that.

  [Qin]: I am suspecting the latency also lies on how TCP detect and recover from the loss or congestion control algorithm. Selecting appropriate design parameters for loss detection like minRTO, D rather than cwnd can effectively 
           reduce potencital latency or connection duration. 

  That is a separate issue and generally less severe than the cwnd issue.  I could provide more color about SCTP, HOL and other issues, but I really don't want to derail this thread to be about anything other than TCP's initial cwnd setting.  

  We need to decide if we think an application protocol should be able to be both a low-latency protocol, and based on TCP.  If you think that should be possible, then we seriously need to consider changing IW.  The reasons not to change it, would be if you thought it would kill the net - but, I there is no evidence of that, especially if we consider which protocols get which limits.

  [Qin]: I am not sure changing IW can come up with one low-latency protocol which is based on TCP. I will assume IW changing is orthogonal to RTT or connection duration and will only affect throughput as Ali pointed out. Can you give some data on how IW affect RTT like graph you show in the previous email?

  Are you saying HTTP should move to a new transport?  That seems like a shame.

  Mike

    -acbegen

    > have immense gratitude and respect for the veterans of this list that have been keeping the net flowing for so long with such

------------------------------------------------------------------------------

---

From: Mike Belshe <[email-masked]>
Date: Mon, 29 Nov 2010 01:17:09 -0800

On Sun, Nov 28, 2010 at 11:31 PM, Qin Wu <[email-masked]> wrote:

> *From:* Mike Belshe <[email-masked]>
> *To:* Ali C. Begen (abegen) <[email-masked]>
> *Cc:* [email-masked]
> *Sent:* Monday, November 29, 2010 2:46 PM
> *Subject:* Re: [tcpm] A Challenge to TCPM
>
> [Qin]: I am suspecting the latency also lies on how TCP detect and recover
> from the loss or congestion control algorithm. Selecting appropriate design
> parameters for loss detection like minRTO, D rather than cwnd can
> effectively
There are many factors.  It doesn't all come down to cwnd, of course.   If
you look at the zero-loss case, which is fairly common, cwnd is a major
factor.

> [Qin]: I am not sure changing IW can come up with one low-latency protocol
> which is based on TCP. I will assume IW changing is orthogonal to RTT or
> connection duration and will only affect throughput as Ali pointed out. Can
> you give some data on how IW affect RTT like graph you show in the previous
> email?
>
> There is more to it than just cwnd, I never meant to imply otherwise.  I've
done quite a lot of research and testing here, have isolated cwnd as a
bottleneck, and am attempting to convey that here.

My point is that, any protocol, attempting to send independent, medium-sized
bursts (greater than 4K) but not huge (large enough for TCP to warm up),
will be slower with fewer connections than many.

Ali is incorrect that changing IW will only change connection duration.
Changing IW directly effects the number of round-trips required to download
a resource.  With a typical 100ms RTT, a 5KB resource can be transferred in
100ms less time if it were not for cwnd.   For a 50KB transfer, an increase
of initial cwnd from 2 to 10 reduces the overall time more dramatically -
from ~8 round trips to ~5 round trips, saving 3 round trips * 100ms == 300ms
of total transfer time.

Finally, if you apply how web pages load (an average web page discovers
resources over time, and makes ~49 independent requests totaling ~320KB),
there are additive effects to these cwnd-induced stalls.  The first resource
was stalled by 1 or 2 RTs, then the second resource is stalled by 2-3 RTs,
etc.  And that is why, as the chart shows quite clearly, that a
single-connection protocol for HTTP with current cwnd values is slower than
a multi-connection HTTP - the multi-connection HTTP gets more cwnd.

This chart is pretty clear, I hope.  I've isolated just the one variable to
test- init-cwnd, and demonstrated the effects.  This effect can be seen in
the real world - just open up tcpdump and load a few web pages.  I guarantee
you'll see it! :-)

Mike

>> > With SPDY, we want to build an application protocol which is efficient
>> on the transport, good for the network, and faster than
>> > configuration to specifically isolate the effects of high-RTT and IW.
>>  As you can see, at low IW, the single-connection protocol
>> > With SPDY (or whatever protocol comes next), we can definitely reduce
>> > over the last 2 decades but also not completely unthrottle protocols
>> that are already living on the edge of not having
>> > throttles.
>> >
>> > Nandita have provided good data for a general-purpose increase to 10.
>>  If inefficient protocols are allowed to go that high,
>> > conservative.  More evidence also emerges that increasing cwnd makes
>> your site faster.  More sites start experimenting with
>  ------------------------------
>

---

From: <[email-masked]>
Date: Mon, 29 Nov 2010 09:24:14 +0000

For HTTP over SCTP see discussion in 
http://tools.ietf.org/html/draft-wood-tae-specifying-uri-transports
and particularly
http://tools.ietf.org/html/draft-natarajan-http-over-sctp

This can form part of making HTTP run over more streaming protocols, not just TCP, to adapt it for different/high performane environments:
http://personal.ee.surrey.ac.uk/Personal/L.Wood/publications/internet-drafts/draft-wood-dtnrg-http-dtn-delivery/lloyd-wood-turning-http-into-a-standalone-layer-ietf-75-tsvarea-slides.pdf
talk given to the TSVAREA meeting, IETF 75, Stockholm, 27 July 2009.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xiangsong Cui
Sent: 29 November 2010 02:34
To: 'Ali C. Begen (abegen)'; 'Scheffenegger, Richard'; 'Mike Belshe'; [email-masked]
Cc: [email-masked]
Subject: Re: [tae] [tcpm] A Challenge to TCPM

> Sure but HTTP is not using SCTP today. Should we look into it?

Sorry for digging this topic in tcpm list, but I think the guys interested in this topic also cover tcpm.

In my remember, a Bar-BoF (or BoF) named "Negotiation BoF" was ever mentioned at IETF76 (denied by AD at that time).

Thanks and regards,
Xiangsong

_______________________________________________
tae mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tae

---

From: Xiangsong Cui <[email-masked]>
Date: Mon, 29 Nov 2010 17:43:27 +0800

Thanks for the information!

Regards,
Xiangsong

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, November 29, 2010 5:24 PM
> To: [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: [tae] [tcpm] A Challenge to TCPM
> 
> This can form part of making HTTP run over more streaming protocols, not
just
> TCP, to adapt it for different/high performane environments:
>
http://personal.ee.surrey.ac.uk/Personal/L.Wood/publications/internet-drafts
/
draft-wood-dtnrg-http-dtn-delivery/lloyd-wood-turning-http-into-a-standalone
-l
> ayer-ietf-75-tsvarea-slides.pdf
in this
> topic also cover tcpm.

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Mon, 29 Nov 2010 05:03:24 -0800

...
> 		We need to decide if we think an application protocol should be able to be both a low-latency protocol, and
> based on TCP.  If you think that should be possible, then we seriously need to consider changing IW.  The reasons not to
> change it, would be if you thought it would kill the net - but, I there is no evidence of that, especially if we consider which
> protocols get which limits.
> 
> 		[Qin]: I am not sure changing IW can come up with one low-latency protocol which is based on TCP. I will
> assume IW changing is orthogonal to RTT or connection duration and will only affect throughput as Ali pointed out. Can you
> give some data on how IW affect RTT like graph you show in the previous email?
> 
> There is more to it than just cwnd, I never meant to imply otherwise.  I've done quite a lot of research and testing here, have
> isolated cwnd as a bottleneck, and am attempting to convey that here.
> 
> My point is that, any protocol, attempting to send independent, medium-sized bursts (greater than 4K) but not huge (large
> enough for TCP to warm up), will be slower with fewer connections than many.

Of course, nobody else is saying ow.
 
> Ali is incorrect that changing IW will only change connection duration.   Changing IW directly effects the number of round-
> trips required to download a resource.  With a typical 100ms RTT, a 5KB resource can be transferred in 100ms less time if it
> were not for cwnd.   For a 50KB transfer, an increase of initial cwnd from 2 to 10 reduces the overall time more dramatically
> - from ~8 round trips to ~5 round trips, saving 3 round trips * 100ms == 300ms of total transfer time.

I said "changing IW would only change the conn duration"? I don't recall that but anyway.

Your examples are correct but as I said these are all short-lived transactions. It is no brainer that their transfers are limited by IW. Changing it from 2 to 10, saves you 3 RTTs since you are stuck with SS for the entire connection.

There are many other applications which don't necessarily spend their lifetime entirely in SS periods, though. 

-acbegen
 
> Finally, if you apply how web pages load (an average web page discovers resources over time, and makes ~49 independent
> requests totaling ~320KB), there are additive effects to these cwnd-induced stalls.  The first resource was stalled by 1 or 2
> RTs, then the second resource is stalled by 2-3 RTs, etc.  And that is why, as the chart shows quite clearly, that a single-
> connection protocol for HTTP with current cwnd values is slower than a multi-connection HTTP - the multi-connection HTTP
> gets more cwnd.
> 
> This chart is pretty clear, I hope.  I've isolated just the one variable to test- init-cwnd, and demonstrated the effects.  This
> effect can be seen in the real world - just open up tcpdump and load a few web pages.  I guarantee you'll see it! :-)
> 
> Mike
> 
> 
> 
> 
> 
> 
> 
> 			> Strong offered a writeup today which demonstrates this is already happening
> <http://blog.benstrong.com/2010/11/google-
> ________________________________
> 
> 
>

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Mon, 29 Nov 2010 05:03:27 -0800

> From: Mike Belshe [mailto:[email-masked]]
> Sent: Monday, November 29, 2010 1:47 AM
> 	I agree with some of the claims you are making below, and I don't agree with some. I just wanna point out one thing,
> though. You are largely dismissing the #1 reason why applications are often using more than 1 TCP connection. It is not often
> about the tput. It is often (more than often actually) about the delay (among a few other reasons), which is critical for short-
> lived transactions.
> 
> 
> 

I don't agree with this.
 
> 	You can increase the IW to 100 for all I care but TCP will still have to leave with the HOL blocking issue. IETF can
> make a recommendation on IW but I have serious doubts about TCP vendors quickly adopting as you suggest. As for the
> number of TCP connections in parallel, that is not easy to place a policy anyway.
> 
> 
> 
> That is a separate issue and generally less severe than the cwnd issue.  I could provide more color about SCTP, HOL and other
> issues, but I really don't want to derail this thread to be about anything other than TCP's initial cwnd setting.

I don't wanna hijack this thread, either but saying that increasing IW will fix the delay issues is something I don’t agree with it.

I am not objecting to increase IW. As I said you can increase it to 100 for all I care: if an app still wants low latency in certain conditions, it will open a new TCP connection. You cannot fix that problem and avoid that problem with IW.
 
> We need to decide if we think an application protocol should be able to be both a low-latency protocol, and based on TCP.  If
> you think that should be possible, then we seriously need to consider changing IW.  The reasons not to change it, would be if
> you thought it would kill the net - but, I there is no evidence of that, especially if we consider which protocols get which
> limits.

I don't think it would kill the net. Yet, I don’t think your motivation is justified at least for the apps and scenarios I am interested in, which might be of course different from yours.

> Are you saying HTTP should move to a new transport?  That seems like a shame.

One could try fixing HTTP or try to find a better transport alternative. Neither is a shame. Just because it was designed so, it does not mean that we need to stick with it forever.

-acbegen

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Nov 2010 22:19:33 -0800

I think Ali and Mike are both right. Multiple connections are easy app
work-around for both TCP single-threaded model and slow-start slowness to
improve HTTP performance. Since it's one stone for two birds, sites (by
putting objects on subdomains) and browsers work together to open a lot
connections. Which problem is more severe for HTTP performance? I don't have
any data. But I do know IW10 helps the second problem and maybe a little bit
on the first, and is a simple change in TCP.

Yuchung

On Mon, Nov 29, 2010 at 5:03 AM, Ali C. Begen (abegen) <[email-masked]>wrote:

> >       I agree with some of the claims you are making below, and I don't
> agree with some. I just wanna point out one thing,
> > though. You are largely dismissing the #1 reason why applications are
> often using more than 1 TCP connection. It is not often
> > about the tput. It is often (more than often actually) about the delay
> (among a few other reasons), which is critical for short-
> > lived transactions.
> >
> >
> >
> > Actually, that was my whole point.  The only way to get the latency down
> is to avoid cwnd, and multiple connections are an
> > easy trick to accomplish just that.
>
> I don't agree with this.
>
> >       You can increase the IW to 100 for all I care but TCP will still
> have to leave with the HOL blocking issue. IETF can
> > make a recommendation on IW but I have serious doubts about TCP vendors
> quickly adopting as you suggest. As for the
> > That is a separate issue and generally less severe than the cwnd issue.
>  I could provide more color about SCTP, HOL and other
> > issues, but I really don't want to derail this thread to be about
> anything other than TCP's initial cwnd setting.
>
> I don't wanna hijack this thread, either but saying that increasing IW will
> fix the delay issues is something I don’t agree with it.
>
> I am not objecting to increase IW. As I said you can increase it to 100 for
> all I care: if an app still wants low latency in certain conditions, it will
> open a new TCP connection. You cannot fix that problem and avoid that
> problem with IW.
>
> > We need to decide if we think an application protocol should be able to
> be both a low-latency protocol, and based on TCP.  If
> > you think that should be possible, then we seriously need to consider
> changing IW.  The reasons not to change it, would be if
> > you thought it would kill the net - but, I there is no evidence of that,
> especially if we consider which protocols get which
> > limits.
>
> I don't think it would kill the net. Yet, I don’t think your motivation is
> justified at least for the apps and scenarios I am interested in, which
> might be of course different from yours.
>
> >
> >
> One could try fixing HTTP or try to find a better transport alternative.
> Neither is a shame. Just because it was designed so, it does not mean that
> we need to stick with it forever.
>
> -acbegen
>
> > Mike
> >
> >
> >
> >
> >       -acbegen
> >
> >
> Behalf Of Mike Belshe
> >       > With SPDY, we want to build an application protocol which is
> efficient on the transport, good for the network, and
> > faster than
> >       > HTTP.  But right now, TCP is holding us back because we're trying
> to accomplish with one connection what HTTP
> > does with
> >       > 10+ connections.
> >       >
> >       > Unrelated to TCP, HTTP is not very efficient.  This has been
> widely documented and observed for some time.  We
> > can easily
> >       > demonstrate a more efficient protocol, which transfers 25-35%
> fewer packets than HTTP and reduces bytes
> > transferred by
> connection is at a latency-handicap compared to any
> > protocol that
> >       > bandwidth is relatively cheap.  So, being efficient on your
> >       > To visualize the effects, please see the attached graph.  (Or
> click here <http://6541078575799853287-a-chromium-
> > org-s-
> >       >
> >       > s=0> )  In these tests, I loaded a replica of the facebook
> landing page through two protocols - one using a single
> > connection in
> >
> >       > a semi-efficient manner (an early version of spdy) and another
> using 6 connections (http).  I used a high-bandwidth
> > network
> >       > with either 100ms or 200ms of RTT, and then I varied IW from
> 2pkts all the way to 128pkts.  I chose this network
> >       > configuration to specifically isolate the effects of high-RTT and
> IW.  As you can see, at low IW, the single-connection
> > protocol
> >       > With SPDY (or whatever protocol comes next), we can definitely
> reduce the number of connections out of the
> > browser by *at
> >       > least* a factor of 6.  With unfolding of the subdomain tricks
> that websites have been doing, we can reduce it by a
> > factor of
> >       > 10.  This has nothing to do with TCP, except I hope that folks on
> connections + fewer packets + fewer bytes equates to slower
> > web
> >       > pages.
> >       >
> >       > But I'm not asking that we increase IW by a factor of 10; because
> it grows exponentially, if we can increase it to
> > somewhere
> >       > More perplexing is that whenever we talk about increasing IW, we
> tend to think of the static throttle that we have
> > today.  If
> >       > we increase a static throttle, it affects everything.  So
> protocols that are already bursting 20-30x more packets than
> > they
> >       > connections, multiplied with the use of multiple subdomains, and
> is now changing by major sites simply increasing
> > or
> >       > ignoring cwnd altogther.  With so many sites systematically
> working around cwnd, we know that the current limits
> > can be
> >       > Is the obvious solution to stop using a static value?  It doesn't
> seem like we're in a one-size-fits-all for IW.
> >       >
> >       > The IW could either be defined to be protocol-specific, or
> dynamic based on number of TCP flows between the same
> > two
> >       > over the last 2 decades but also not completely unthrottle
> protocols that are already living on the edge of not
> > having
> >       > throttles.
> >       >
> >       > Finally, if a dynamic throttle is too much to decide upon, at
> least updating the static value seems appropriate.
> > Jerrry and
> >       > Nandita have provided good data for a general-purpose increase to
> 10.  If inefficient protocols are allowed to go
> > that high,
> >       > As time goes on, more and more evidence emerges to demonstrate
> that the current cwnd limits are unnecessarily
> >       > conservative.  More evidence also emerges that increasing cwnd
> makes your site faster.  More sites start
> > experimenting with
> >       > their own values of cwnd.  Without a blessed recommendation from
> the IETF, sites choose whatever values they
> > want. Ben
> >
> happening
> >       > With data and a specific recommendation, TCP vendors can quickly
> move to the new recommendations, making
> > their
> >       > products more competitive in the marketplace, and better for
> users.  Pressure is alleviated from existing sites to go
> >       > renegade, and instead they can rest on the specific findings from
> this group.  With a modern recommendation in
> > hand, all of
> >       > us can credibly go after any sites which do choose to use
> dangerously high cwnd values.  Finally, having an officially
> >       > increased IW is an enabler for a future of more efficient
> application level protocols, which are currently at a
> > disadvantage to
> official, controlled and intelligent recommendation from
> > this
> >       > group?
> >       >
> >       >
> >       > Thanks to everyone on this list for their hard work over the
> years - although my email is somewhat critical, be
> > aware that I
> >       > have immense gratitude and respect for the veterans of this list
> that have been keeping the net flowing for so long
> > with such
> >       > great success.
> >       > Mike
> >
> >
>

---

From: Qin Wu <[email-masked]>
Date: Tue, 30 Nov 2010 16:53:22 +0800

+1
Tuning IW used to improve TCP performance will not work for the application that holds persistent connection.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 30 Nov 2010 09:18:06 -0000

Let's not loose focus...

for restart window (persistent connection), many (most?) TCP stacks
already use their prior knowledge on the specifics of the tcp session,
when deciding which restart window to choose.

IMHO, addressing this by dynamic means (much along the lines with Joe is
a big supporter) is really not much of an issue.

The IW10 proposal addresses some of the issues, when it is simply very
hard to obtain any prior knowledge as to how optimally deal with a new
session. Even the 2nd session to the same target is already making use
of cached information today.

Richard Scheffenegger

> From: Qin Wu [mailto:[email-masked]] 
> Sent: Dienstag, 30. November 2010 09:53
> To: Ali C. Begen (abegen); Mike Belshe
> Cc: [email-masked]
> Tuning IW used to improve TCP performance will not work for 
> the application that holds persistent connection.
> >> Ali is incorrect that changing IW will only change 
> connection duration.   Changing IW directly effects the 
> number of round-
> >> trips required to download a resource.  With a typical 
> 100ms RTT, a 5KB resource can be transferred in 100ms less time if it
> >> were not for cwnd.   For a 50KB transfer, an increase of 
> initial cwnd from 2 to 10 reduces the overall time more dramatically
> >> - from ~8 round trips to ~5 round trips, saving 3 round 
> trips * 100ms == 300ms of total transfer time.
> > 
> > Your examples are correct but as I said these are all 
> short-lived transactions. It is no brainer that their 
> transfers are limited by IW. Changing it from 2 to 10, saves 
> you 3 RTTs since you are stuck with SS for the entire connection.
> > 
> > There are many other applications which don't necessarily 
> spend their lifetime entirely in SS periods, though. 
> > 
> > -acbegen
> > 
> >> Finally, if you apply how web pages load (an average web 
> page discovers resources over time, and makes ~49 independent
> >> requests totaling ~320KB), there are additive effects to 
> these cwnd-induced stalls.  The first resource was stalled by 1 or 2
> >> RTs, then the second resource is stalled by 2-3 RTs, etc.  
> And that is why, as the chart shows quite clearly, that a single-
> >> connection protocol for HTTP with current cwnd values is 
> slower than a multi-connection HTTP - the multi-connection HTTP
> >> gets more cwnd.
> >> 
> >> This chart is pretty clear, I hope.  I've isolated just 
> the one variable to test- init-cwnd, and demonstrated the 
> effects.  This
> >> effect can be seen in the real world - just open up 
> tcpdump and load a few web pages.  I guarantee you'll see it! :-)
> >> 
> >> Mike
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> > From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Mike Belshe
> >> > I thought I'd add a slightly different perspective to 
> the discussion about initial cwnd.  I'm a developer
> >> on Chrome & SPDY.
> >> >
> >> > With SPDY, we want to build an application protocol 
> which is efficient on the transport, good for the
> >> network, and faster than
> >> > HTTP.  But right now, TCP is holding us back because 
> we're trying to accomplish with one connection
> >> what HTTP does with
> >> > 10+ connections.
> >> >
> >> > Note:  I don't want to make this thread about SPDY.  I 
> hope the members of this list agree that the
> >> goals of SPDY are good
> >> > Unrelated to TCP, HTTP is not very efficient.  This has 
> been widely documented and observed for
> >> some time.  We can easily
> >> > demonstrate a more efficient protocol, which transfers 
> 25-35% fewer packets than HTTP and reduces
> >> bytes transferred by
> >> > Unfortunately, any TCP-based protocol which uses a 
> single connection is at a latency-handicap
> >> compared to any protocol that
> >> > uses multiple connections.  This is due to IW.  Further, 
> because bandwidth increases have outpaced
> >> RTT decreases,
> >> > bandwidth is relatively cheap.  So, being efficient on 
> your bandwidth is less critical than being
> >> efficient on your round-trips.
> >> >
> >> 
> sites.googlegroups.com/a/chromium.org/dev/spdy/effect_of_initi
> al_cwnd_on_plt(1).png?attachauth=ANoY7cpm-y5NA-
> FLtTi7rKiKHF12j_XkANIwSM0saVdoF6c6gtCDg1_8gKIn8QPVSVp9Cwtw1uFT
> TWw93XFPQkAmEI2FvRbq6IBg1aI%3D&attredirect
> >> > s=0> )  In these tests, I loaded a replica of the 
> facebook landing page through two protocols - one
> >> using a single connection in
> >> 
> >> > a semi-efficient manner (an early version of spdy) and 
> another using 6 connections (http).  I used a
> >> high-bandwidth network
> >> > with either 100ms or 200ms of RTT, and then I varied IW 
> from 2pkts all the way to 128pkts.  I chose
> >> this network
> >> > configuration to specifically isolate the effects of 
> high-RTT and IW.  As you can see, at low IW, the
> >> single-connection protocol
> definitely reduce the number of connections
> >> out of the browser by *at
> >> > least* a factor of 6.  With unfolding of the subdomain 
> tricks that websites have been doing, we can
> >> reduce it by a factor of
> >> > 10.  This has nothing to do with TCP, except I hope that 
> folks on this list appreciate that this goodness:
> >> fewer connections -
> connections + fewer packets + fewer bytes
> >> equates to slower web
> >> > pages.
> >> >
> >> > But I'm not asking that we increase IW by a factor of 
> 10; because it grows exponentially, if we can
> >> increase it to somewhere
> >> > More perplexing is that whenever we talk about 
> increasing IW, we tend to think of the static throttle
> >> that we have today.  If
> >> > we increase a static throttle, it affects everything.  
> So protocols that are already bursting 20-30x
> >> more packets than they
> >> > At the same time, we also know that cwnd has already 
> been increased on the net at large.   This
> >> started with use of multiple
> >> > connections, multiplied with the use of multiple 
> subdomains, and is now changing by major sites
> >> simply increasing or
> >> > ignoring cwnd altogther.  With so many sites 
> >> > Is the obvious solution to stop using a static value?  
> It doesn't seem like we're in a one-size-fits-all for
> >> IW.
> >> >
> >> > The IW could either be defined to be protocol-specific, 
> or dynamic based on number of TCP flows
> >> between the same two
> >> > endpoints.  Both of these approaches would offer the 
> benefit of a controlled, increased cwnd to
> >> reflect the changes in the net
> >> > over the last 2 decades but also not completely 
> unthrottle protocols that are already living on the
> >> edge of not having
> >> > throttles.
> >> >
> >> > Finally, if a dynamic throttle is too much to decide 
> upon, at least updating the static value seems
> >> appropriate.  Jerrry and
> >> > Nandita have provided good data for a general-purpose 
> increase to 10.  If inefficient protocols are
> >> allowed to go that high,
> >> > As time goes on, more and more evidence emerges to 
> demonstrate that the current cwnd limits are
> >> unnecessarily
> >> > conservative.  More evidence also emerges that 
> increasing cwnd makes your site faster.  More sites
> >> start experimenting with
> >> > their own values of cwnd.  Without a blessed 
> quickly move to the new
> >> recommendations, making their
> >> > products more competitive in the marketplace, and better 
> for users.  Pressure is alleviated from
> >> existing sites to go
> >> > renegade, and instead they can rest on the specific 
> findings from this group.  With a modern
> >> recommendation in hand, all of
> >> > us can credibly go after any sites which do choose to 
> use dangerously high cwnd values.  Finally,
> >> having an officially
> >> > increased IW is an enabler for a future of more 
> efficient application level protocols, which are
> >> currently at a disadvantage to
> >> > Now, which of these two options carries more risk of 
> internet collapse?  Vendors implementing
> >> whatever-the-heck-their-
> official, controlled and intelligent
> >> recommendation from this
> >> > group?
> >> >
> >> >
> >> > Thanks to everyone on this list for their hard work over 
> the years - although my email is somewhat
> >> critical, be aware that I
> >> > have immense gratitude and respect for the veterans of 
> this list that have been keeping the net
> >> flowing for so long with such
> >> > great success.
> >> > Mike
> >> 
> >> 
> >> 
> >> 
> >>

---

From: Qin Wu <[email-masked]>
Date: Tue, 30 Nov 2010 19:46:06 +0800

Hi,
----- Original Message ----- 
From: "Scheffenegger, Richard" <[email-masked]>
To: "Qin Wu" <[email-masked]>; "Ali C. Begen (abegen)" <[email-masked]>; "Mike Belshe" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, November 30, 2010 5:18 PM
Subject: RE: [tcpm] A Challenge to TCPM

Let's not loose focus...

[Qin]: I am not convincing that chaning IW can reduce latency for persistent connection.
I only agree IW10 has its value when Slow start applies in zero-loss case.

[Qin]: No doubt about this.

Richard Scheffenegger

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 07:38:30 -0800

It helps persistent connections during slow-start restart, since IW  MAY 
also be used as the restart window (see RFC3390).

Joe

On 11/30/2010 12:53 AM, Qin Wu wrote:
> From: "Ali C. Begen (abegen)"<[email-masked]>
> To: "Mike Belshe"<[email-masked]>; "Qin Wu"<[email-masked]>
> Cc:<[email-masked]>
>>>> I thought I'd add a slightly different perspective to the discussion about initial cwnd.  I'm a developer
>>> on Chrome&  SPDY.
>>>>