Subject: [tcpm] TCP tuning

From: Lars Eggert <[email-masked]>
Date: Wed, 3 Feb 2010 16:28:58 +0200

Hi,

Jerry Chu has recently started the discussion on whether we need to think about tweaking TCP for the "modern Internet." Just came across another presentation from (AFAICT) another corner of Google that makes similar arguments.

FYI: http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf

This topic seems to be gaining momentum, and the WG should take some time considering if there is work here for it.

Lars

---

From: Adam Langley <[email-masked]>
Date: Wed, 3 Feb 2010 09:41:03 -0500

> This topic seems to be gaining momentum, and the WG should take some time considering if there is work here for it.

The slide format of the presentation possibly isn't the easiest to
read, but I hope that it's understandable. If people have questions,
please do ask.

Mike, Jerry and I are all (mostly) in the same corner of Google
although the above file is regarding SPDY, our new HTTP serialisation:

http://dev.chromium.org/spdy/spdy-whitepaper

Cheers

AGL

-- 
Adam Langley [email-masked] http://www.imperialviolet.org

---

From: Michael Welzl <[email-masked]>
Date: Wed, 3 Feb 2010 15:42:36 +0100

Hi all,

(sorry if this is a bit off topic, as the mentioned presentation is  
mainly
about using a larger initial window)

In the context of the discussion that Jerry brought up on parameter  
tuning,
one question that was raised was the initial RTO during the SYN - SYN/ 
ACK
exchange. I understand that we can't recommend to resend SYN's more
aggressively because this might cause a server to be overloaded with  
SYN's,
but why do we have to wait extremely long until we resend SYN/ACKs?

Back then, when I asked, I got no answer. Both Jerry and I pointed
to our measurement results which show that the effect is nonneglible.

So I'll use this chance to ask again  :-)

Cheers,
Michael

On Feb 3, 2010, at 3:28 PM, Lars Eggert wrote:

> Hi,
>
> Jerry Chu has recently started the discussion on whether we need to  
> think about tweaking TCP for the "modern Internet." Just came across  
> another presentation from (AFAICT) another corner of Google that  
> makes similar arguments.
>
> This topic seems to be gaining momentum, and the WG should take some  
> time considering if there is work here for it.
>
> Lars_______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "SCHARF, Michael" <[email-masked]>
Date: Wed, 3 Feb 2010 16:17:29 +0100

> This topic seems to be gaining momentum, and the WG should 
> take some time considering if there is work here for it.

IMHO there could indeed be room for increasing the initial window. If
most data transfers continue to be smaller than 3 MSS, a larger initial
window would not necessarily cause harm, as it is seldomly used. Still,
this would speed up those data transfers that currently suffer from
Slow-Start.

In the past, I have played a little bit with an initial window e. g. of
10 MSS, and that worked surprisingly well. Also, one could combine a
larger initial window with other mechanisms that would reduce the
aggressiveness of the flow startup. Maybe it is indeed time to think
about the Slow-Start once again.

Michael

---

From: Stefanos Harhalakis <[email-masked]>
Date: Wed, 3 Feb 2010 17:33:21 +0200

> about the Slow-Start once again.

IMHO, such a discussion should also consider the user perspective and 
distinguish  two types of TCP connections: Those that result into user-
observable delay (e.g. HTTP) and those that don't (e.g. SMTP). Such a 
modification could be made configurable by applications: Web servers (and 
browsers) could use a larger initial window while SMTP servers/clients could 
use the current approach.

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 04 Feb 2010 00:13:49 +0800

> about the Slow-Start once again.

FWIW, I've come across traces showing some network equipment
uses an initial window as large as 16 segments (some kind of
turbo mode).  It is probably true many stacks already use a
much larger initial window than recommended in the RFC.  Just
that people have been very quiet about this.

-- 

						K. Poon.
						[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 08:33:02 -0800

> FYI: http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf

I'm curious about the tests described. 200ms seems very high for an RTT.
Even over a cell wireless link (3G), I see latencies that are typically
half that. I wonder what the improvement is for a 50-80ms RTT.

I'm not excited about the prospects of a potential tragedy of the
commons issue (i.e., this works now, but may not work as nicely when
everyone tries this), especially for a mere 50% improvement in latency.

It'd be useful to review the set of proposed mods and claims of impact
for each, though the last time we saw this there seemed to be quite a
few questions about the experiments presented.

Joe

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 04 Feb 2010 01:05:29 +0800

> half that. I wonder what the improvement is for a 50-80ms RTT.

I suspect that it depends on how your view of Internet is.  From where
I am physically located, RTT to my company's internal network is
around 200ms.  It is the normal RTT I live with for doing my day to
day work.  I hope folks won't take a "country-centric" view on Internet
traffic.  People all over the world are communicating with each other.
The Internet is really not that "small..."

-- 

						K. Poon.
						[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 09:12:04 -0800

> The Internet is really not that "small..."

Agreed. The point is to what extent TCP is modified for fractional
benefits for specific communities. TCP is "optimized" to always work,
but not to work especially well in any particular environment.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 18:16:35 +0100

> but not to work especially well in any particular environment.

Yeah... stupid, isn't it?

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 09:19:44 -0800

> Yeah... stupid, isn't it?

I'll trade "always" or even "usually" to "works very well in some
places, but not at all in others".

Making TCP smarter is fine; doing so at the expense of robustness is not.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 18:32:39 +0100

> Making TCP smarter is fine; doing so at the expense of robustness is  

Sorry for being so unspecific - I was just thinking out loud...
my point was: isn't that actually a stupid situation?

Why do we have to worry about TCP over wireless, when so
many users know perfectly well that their access link is WiFi?
Why do we have to worry so much about raising the window
when it's perfectly clear in some cases (at least from the
combined knowledge of the stack + its users) that a certain
connection is only going to face high bandwidth conditions?

In other words, why not harness this knowledge somehow,
and develop a customized transport protocol, or ways of
customizing what we now have?

This is what I was hinting at. Just thoughts... which would be
more appropriate in an IRTF than in an IETF list, in fact - sorry.

To get back to the point, I'm not seriously suggesting to reduce
TCP's robustness and I'm in your camp about the larger
window thing. My personal opinion here is that, however,
the current slow start is indeed questionable, and questioning
it should be done, in large scale thorough experiments. I'm
interested in this and started some such work with a master
student; I might continue if I get another one (or if I can nicely
make that fit as a part of a larger Ph.D. thesis - ideas for
how to do that are welcome!  :-)  ).

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 09:43:08 -0800

Hi, Michael,

> customizing what we now have?

That's been tried in various ways in various IETF efforts. Some try to
find the properties of the immediate link, which are expected for some
link types to dominate the path. Some try to detect path properties.
Some reuse information from past connections.

None of that is all that bad.

...
> how to do that are welcome!  :-)  )

I think revising congestion control to be smarter is already a IRTF
effort (aren't you active in that already?):

http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG

IMO, I would encourage Google to take their work there and get feedback
there first. TCPM is a place where mature, vetted ideas should be
considered, preferably (IMO).

Joe

---

From: Adam Langley <[email-masked]>
Date: Wed, 3 Feb 2010 12:48:20 -0500

> considered, preferably (IMO).

I think an important part of the presentation is that this is already
a very well vetted idea: by using multiple subdomains and concurrent
connections, HTTP has already increased the initcwnd drastically.

AGL

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 09:52:14 -0800

> connections, HTTP has already increased the initcwnd drastically.

By vetted, I would include the experiments and the detailed information
available. That vetting should be checked in the IRTF first; that's a
better place to review technical material for consideration of past and
related work, and verification of methodology.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 10:20:56 -0800

Michael,

On Wed, Feb 3, 2010 at 6:42 AM, Michael Welzl <[email-masked]> wrote:
> Hi all,
>
> (sorry if this is a bit off topic, as the mentioned presentation is mainly
> In the context of the discussion that Jerry brought up on parameter tuning,
> one question that was raised was the initial RTO during the SYN - SYN/ACK
> aggressively because this might cause a server to be overloaded with SYN's,

Actually Mark Allman, Vern Paxon and I have been working on revising
RFC2988 to reduce the initial RTO from 3secs to 1sec. Will submit the
proposal soon!

Best,

Jerry

>> Jerry Chu has recently started the discussion on whether we need to think
>> about tweaking TCP for the "modern Internet." Just came across another
>> presentation from (AFAICT) another corner of Google that makes similar
>> arguments.
>>
>> FYI:
>> http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf
>>
>> This topic seems to be gaining momentum, and the WG should take some time
>> considering if there is work here for it.
>>
> _______________________________________________
> tcpm mailing list
> [email-masked]

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 19:27:07 +0100

> Some reuse information from past connections.

Really? I'd be curious about a few pointers, what exactly
do you have in mind? About reusing information etc., the
congestion manager and your TCB sharing stuff comes to
mind, but for the rest I'm curious

> effort (aren't you active in that already?):

Yep - I just thought out loud on this list because that's
where the discussion happened...

> considered, preferably (IMO).

I absolutely agree.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 19:29:27 +0100

> connections, HTTP has already increased the initcwnd drastically.

I would disagree about "well vetted" (at the level that TCPM
needs to incorporate such a major change in the protocol)
unless I see some very thorough measurements showing
that no harm comes from such a change, in various environments,
with various possible traffic occurrences coming and going...

Talking about this in ICCRG wouldn't be a bad thing at all,
it's really a reasonable first step IMO.

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 10:41:35 -0800

> mind, but for the rest I'm curious

Spencer Dawkins probably recalls better than I; I think there were
numerous BOFs centered around "link indications", including:
	TRIGTRAN BOF
	"LinkUp"
	draft-dawkins-linkup-nevermind
	LCI BOF (Link Characteristic Information for Mobility)
	MPTCP WG
	LEDBAT WG
	RFC-4709

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 11:10:22 -0800

> this would speed up those data transfers that currently suffer from

Actually our data points to the contrary - the average web object and page
size have been rising steadily. E.g., the majority of our search responses
no longer fit in 3 MSS these days.

> 10 MSS, and that worked surprisingly well.

Good to know!

>Also, one could combine a

We are planning to submit an I-D on raising initcwnd to 10MSS before
3/1 to be included in the upcoming IETF meeting.

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 11:18:40 -0800

> that people have been very quiet about this.

Indeed, the percentage of non-compliance can be quite high, as high
as ~15% in one dataset reported by an IMC paper "TCP Revisited: A
Fresh Look at TCP in the Wild"

Jerry

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 11:26:15 -0800

> no longer fit in 3 MSS these days.

This should be relevant only for the first response from a given IP
address; persistent connections should render this moot for subsequent
requests.

Again, the question arises as to how much this is going to achieve a
noticeable impact.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 12:25:59 -0800

Hi Joe,

> considered, preferably (IMO).

Sure we can do that and will love to hear feedback from ICCRG. But our
preference is IETF simply because we consider a simple, modest
increase of the initial window to be more of an engineering project than
a research problem. The former is more about engineering tradeoff between
cost/risk, benefit and TIMELINESS than a research project to find the best,
elegant, risk free solution with little or no time-to-market concern.

We are aware of all the great research work being conducted at ICCRG
on speeding up slow-start. But none of the proposals seems ready for
IETF anytime soon. I'd be happy to be proven wrong here! (Well to be
more correct a modest increase of initial window may have already been
studied at ICCRG. In that case I'd rephrase it to be the closest to
engineering consideration IMHO.)

Jerry

---

From: John Heffner <[email-masked]>
Date: Wed, 3 Feb 2010 15:32:29 -0500

I would like to see tcpm take on the initial window issue, though as
Joe notes we probably do not yet know enough to formally take this on
as a charter item.  (Or, would this be a tsvwg item due to possible
wider impact?)

A couple thoughts:
- A large initial window should probably only be used by mutual
agreement.  Maybe via a TCP option during the handshake?
- A significant increase to the initial window might benefit from
introduction of some form of pacing, since you won't have any ack
clock yet.  This might help reduce losses and delay jitter.

  -John

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 12:35:09 -0800

On Wed, Feb 3, 2010 at 10:39 AM, Biswas, Anumita
<[email-masked]> wrote:
>
>
>> -----Original Message-----
>> From: Jerry Chu [mailto:[email-masked]]
>> Sent: Wednesday, February 03, 2010 10:21 AM
>> To: Michael Welzl
>> Cc: [email-masked] WG
>> Subject: Re: [tcpm] TCP tuning
>>
>>
>> Michael,
>>
>> > mainly about using a larger initial window)
>> >
>> > tuning, one question that was raised was the initial RTO during the
>> > SYN - SYN/ACK exchange. I understand that we can't
>> recommend to resend
>> > SYN's more aggressively because this might cause a server to be
>> > overloaded with SYN's, but why do we have to wait extremely
>> long until
>> > we resend SYN/ACKs?
>> >
>> > Back then, when I asked, I got no answer. Both Jerry and I
>> pointed to
>> > our measurement results which show that the effect is nonneglible.
>>
>> Actually Mark Allman, Vern Paxon and I have been working on
>> revising RFC2988 to reduce the initial RTO from 3secs to
>> 1sec. Will submit the proposal soon!
>>
> This implies that this proposal brings the intial RTO value to be equal to the minimum RTO value of 1 second. Is this true?

Correct.

>
> I am going by RFC 2988, section 2, 2.4. It explicitly states this:
>
> " (2.4) Whenever RTO is computed, if it is less than 1 second then the
>         RTO SHOULD be rounded up to 1 second.
>
>         Traditionally, TCP implementations use coarse grain clocks to
>         measure the RTT and trigger the RTO, which imposes a large
>         minimum value on the RTO.  Research suggests that a large
>         minimum RTO is needed to keep TCP conservative and avoid
>         spurious retransmissions [AP99].  Therefore, this
>         specification requires a large minimum RTO as a conservative
>         approach, while at the same time acknowledging that at some
>         future point, research may show that a smaller minimum RTO is
>         acceptable or superior."
>
> Is it time to reconsider a smaller minimum RTO as well?

Yes this is the 3rd item on my todo list
(see http://www.ietf.org/proceedings/75/slides/tcpm-1.pdf) but is not
nearly as important as the first two IMO.

Jerry

>> take some

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 12:44:02 -0800

Hi John,

> agreement.  Maybe via a TCP option during the handshake?

This may not be needed. A receiver can effectively constrain the
sender's window by advertising a smaller initial receive window.

Obviously this works for only the initial window, not the restart window.
(Oh, on a second thought this can be made to work for the restart
window too.)

> clock yet.  This might help reduce losses and delay jitter.

Agreed. The question is, what is that threshold of window/burst size
beyond which pacing is required. My current guess is > 10.

Jerry

---

From: "Biswas, Anumita" <[email-masked]>
Date: Wed, 3 Feb 2010 10:39:14 -0800

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 12:48:15 -0800

>        LCI BOF (Link Characteristic Information for Mobility)

The following is quoted from RFC3150 (a BCP):

   "Implementors should consider the possibility that a host will be
   directly connected to a low-speed link when choosing default TCP
   receive window sizes."

Jerry

---

From: John Heffner <[email-masked]>
Date: Wed, 3 Feb 2010 15:51:03 -0500

> (Oh, on a second thought this can be made to work for the restart

A good point, though most receivers (Linux 2.4 and later being the
exception) advertise all available window space, not expecting a large
initial burst from the sender.  I think you need a negotiation for
opt-in rather than opt-out.

  -John

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 13:39:21 -0800

> address; persistent connections should render this moot for subsequent

Yes we are well aware of the advantage of maintaining persistent connections.
In fact this is one of the reasons holding us back from resurrecting T/TCP
(in addition to other security issues).

But from our own observation the average HTTP connections just don't
persistent that long for some reason. I don't remember the data or all
the technical reasons on top of my head, but can surely try to get them for
you.

There are other limitations of HTTP 1/1:

1. obviously its effectiveness depends on user's web surfing pattern.
2. web browsers have been going down the slope of opening more and
more simultaneous connections anyway, partly to circumvent HTTP's
serialization, an issue SPDY tries to address.
3. practically web servers have only limited resources. can't hold too many
open connections.
4. a web object will make the TCP slow start algorithm to grow its cwnd
only upto the size of the object. As such, even with persistent connections,
a web object that is larger than any of its predecessors will hit the window
limit, and take more than one round trip time to complete. So persistent
connections might not be as effective as one might think.

> Again, the question arises as to how much this is going to achieve a

Quite significant in our study. Will present our result at IETF/Anaheim.

Jerry

---

From: "Biswas, Anumita" <[email-masked]>
Date: Wed, 3 Feb 2010 13:50:29 -0800

> From: John Heffner [mailto:[email-masked]]
> Sent: Wednesday, February 03, 2010 12:51 PM
> To: Jerry Chu
> Cc: [email-masked] WG

The slow start algorithm tries to probe the network capacity, not the end node's available window space. The network capacity is dependent on switch/router queue lengths, among other things and therefore the initial receive window is not something that can/should be negotiated by end points. 

However, I do have reservations with standardizing an increased initial window assuming that 10 MSS is a good "guess" for what the network can tolerate.

I have seen cases where switches topple over initial receive window sizes as spec'd out by RFC 3390. In fact, RFC 3390 does state that the recommended IW value of 4 segments for 1500MSS can cause more loss in networks with higher segment drop rates.

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 22:59:57 +0100

> 	LCI BOF (Link Characteristic Information for Mobility)

Thanks! I was aware about some (I liked trigtran and was sad to
see it fail) but definitely not all of these. At least LEDBAT doesn't
seem to fit in the list btw

Cheers,
Michael

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 14:01:43 -0800

> opt-in rather than opt-out.

Yes, Linux 2.4 and later constrain its initial receive window per RFC3390,
making it harder for any future change to initcwnd to be deployed (i.e., both
sides' stack need to change). We're trying to fix this. See
http://www.spinics.net/lists/netdev/msg115912.html.

The change will also be useful for those hosts behind slow links for purposes
mentioned before.

Jerry

---

From: Murali Bashyam <[email-masked]>
Date: Wed, 3 Feb 2010 14:03:36 -0800

Any idle period higher than the RTT between HTTP requests tends to reset the cwindow down to the restart window. This hurts persistent connections too. What's the motivation for this, can we have a different slope while reducing the window instead of resetting all the way down?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jerry Chu
Sent: Wednesday, February 03, 2010 1:39 PM
To: Joe Touch
Cc: [email-masked]; SCHARF, Michael
Subject: Re: [tcpm] TCP tuning

Jerry

---

From: Michael Welzl <[email-masked]>
Date: Wed, 03 Feb 2010 23:10:50 +0100

Hi,

> elegant, risk free solution with little or no time-to-market concern.

Without trying to get into a debate on engineering vs. research, I would
say that the research community's perception of TCP research is that,
indeed, such a change is a valid research question. That's not because
of a need to find the "best, elegant, risk free solution", but because
of the need to make sure that it's a solution that's safe to deploy.

This is indeed at odds with needs for timeliness - but your need for
that can't be the basis for lowering the bar about significant changes
to TCP in the IETF.

> engineering consideration IMHO.)

The proposals that have been made in the ICCRG context were not
there because this is the most, or only, appropriate way to do things
in ICCRG - they were there because they were proposed. I would
love to see proposals for more "IETF-ready" approach there.
I don't think that a modest increase of the initial window has been
considered in the ICCRG context, and it would fit very well.

Cheers,
Michael

---

From: John Heffner <[email-masked]>
Date: Wed, 3 Feb 2010 17:11:41 -0500

> Any idle period higher than the RTT between HTTP requests tends to reset the cwindow down to the restart window. This hurts persistent connections too. What's the motivation for this, can we have a…

This is because you lose the ack clock.  Otherwise, when resuming
sending you'll send a line-rate burst of the full BDP, which will
often overflow queues and cause loss and/or large delay spikes.  There
is currently no pacing mechanism in TCP to smooth out such bursts.

  -John

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:20:15 -0800

Jerry Chu wrote:
...
>    directly connected to a low-speed link when choosing default TCP

Yup. "Be conservative in what you do".

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:25:00 -0800

Jerry Chu wrote:
....
> the technical reasons on top of my head, but can surely try to get them for

I think having all of that in one place would be a good start.

> There are other limitations of HTTP 1/1:

Persistent connections predates 1.1, FWIW ;-)

> connections might not be as effective as one might think.

This all depends on many things - e.g., whether the persistent
connection is used for a sequence of requests or whether they are
interleaved using HTTP 'chunks', the size of the chunks, etc.

It's useful as well to keep in mind that HTTP exists because Tim
Berners-Lee thought that FTP would be too inefficient, because 'most of
the time you only get one file from a site', and he didn't want to open
two connections. I.e., one other alternative is to consider FTP's
version of multiplexing as well.

> Quite significant in our study. Will present our result at IETF/Anaheim.

We've seen initial work on this before, and had a number of questions
that remain unanswered. A good start would be to put this all in a tech
report. Presentations at the IETF should not get into deep technical
detail; they ought to summarize work available before the meeting and
provided on the list IMO.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:26:21 -0800

Michael Welzl wrote:
...
> seem to fit in the list btw

There are some people in LEDBAT who raised the issue that how much and
how you background might be related to the path properties. That's the
relationship.

Note that this is just what I could cull from my email archives in a
quick pass; there may be other examples of IETF and other work worth
checking, FWIW.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:31:03 -0800

Murali Bashyam wrote:
> Any idle period higher than the RTT between HTTP requests tends to
> reset the cwindow down to the restart window.

This was first documented in an extension to Jacobson's 1988 Sigcomm
paper, in an appendix. The idea is that if you are idle, you shut the
window down. However, a footnote in that doc (unpublished, FWIW)
suggests that because TCP is used mostly symmetrically, one can use the
"time since last receive" instead of "time since last send". As a
result, the request tends to reset the timers, and the server just
blasts at line rate.

This issue is documented in detail in:

John Heidemann, Katia Obraczka, and Joe Touch, "Modeling the Performance
of HTTP Over Several Transport Protocols", IEEE/ACM Transactions on
Networking, V5, N5, Oct. 1997, pp.616-630.

A correction to TCP's behavior was suggested on this list, and we worked
on it with Sally Floyd for a while ("slow start restart" was the phrase
we used). A number of variations were considered, including 'burst or
lose', 'use it or lose it', etc. We put that on the shelf a while ago,
though we've been interested in 'restarting' it too ;-)

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:42:03 -0800

> as a charter item.  (Or, would this be a tsvwg item due to possible

FWIW, I would expect it would be TSVWG, since it would affect all
"TCP-friendly" protocols too...

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 14:47:28 -0800

Hi Anumita,

> I have seen cases where switches topple over initial receive window sizes as spec'd out by RFC 3390. In fact, RFC 3390 does state that the recommended IW value of 4 segments for 1500MSS can cause m…

Understood the concern. But sometimes we'll need to take active action
in order to break the
deadlock - TCP spec relying on switch/router upgrade while equipment
vendors waiting for
TCP spec to be ratified. In this case a modest increase of initcwnd
may well be the right one
to get both sides into a healthy lockstep toward a faster Internet.
This is because the negative
impact (% of access switch/router with tiny buffer) is limited and
easy (IMHO) to fix while the
gain can be significant for the vast majority. This is a good
incentive to encourage those left
behind to move up the bandwidth curve.

Jerry

---

From: Marco Mellia <[email-masked]>
Date: Wed, 03 Feb 2010 23:56:38 +0100

Just to try summarizing some points:

- Increasing the initial window can benefit short-lived connections.
But it can be dangerous, especially on low speed links, and equipments 
with small buffers. For example, an adsl modem has definitively a small 
buffer, which is typically overloaded by uploading p2p traffic... 
initial bursts of "several packet" (e.g., an email with attachments, a 
POST command with some data in it, etc.) will cause packet losses with 
high probability. At the end, the latency will be higher...
Note that this will happen on any congested link too...

- Reducing the RTO timeout is a possibility to get rid of the initial 
latency due to packet loss during three-way handshake. 3s seems to be 
too conservative these days.
But there are cases in which RTT can be larger than 2s. E.g., 2.5G GSM 
networks, or congested 3G networks. And these kind  of access networks 
are today more and more popular... and more and more congested...

So this calls for a conservative approach.

On the flip side,

- someone is already hacking the TCP rules:

   - google is "retransmitting" the SYN-ACK with a timeout of about 
200ms. This causes spurious retransmission on practically all tcp 
connection on a 2.5G network (I remember a paper from FTW showing this...)

   - windows O.S. (at least XP) is triggering the fast retransmit at the 
third ACK, not at the third DUPLICATE ack
check http://support.microsoft.com/?scid=kb%3Ben-us%3B224829&x=11&y=11 
if you don't believe it! Default value is 2 duplicate packets)

   - large initial windows are commonly seems. 16kB of initial window 
can be easily observed in any network. I suspect this is related to TCP 
Offloading Engines (TOE), that perform segmentation at the NIC, or the 
use of jumbo ethernet frames (9kB) -- the OS sees a "MSS" of 16kB (or 
9kB), the initial windows is 1 segment => 10 (6) packets of 1500B each 
are then generated by the TOE.

This supports the position "at the end, it will work in 95% of the cases".

Researchers did a lot of studies about TCP latency. But are typically 
ignored by IETF.

-- 
Ciao,                    /\/\/\rco

+-----------------------------------+
| Marco Mellia - Assistant Professor|
| Skypeid: mgmellia                 |
| Tel: +39-011-564-4173             |
| Cel: +39-331-6714789              |   /"\  .. . . . . . . . . . . . .
| Politecnico di Torino             |   \ /  . ASCII Ribbon Campaign  .
| Corso Duca degli Abruzzi 24       |    X   .- NO HTML/RTF in e-mail .
| Torino - 10129 - Italy            |   / \  .- NO Word docs in e-mail.
| http://www.telematica.polito.it   |        .. . . . . . . . . . . . .
+-----------------------------------+
The box said "Requires Windows 95 or Better." So I installed Linux.

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:57:45 -0800

Jerry Chu wrote:
>> However, I do have reservations with standardizing an increased
>> initial window assuming that 10 MSS is a good "guess" for what the
>> network can tolerate
>> 
>> I have seen cases where switches topple over initial receive window
>> sizes as spec'd out by RFC 3390. In fact, RFC 3390 does state that the
>> recommended IW value of 4 segments for 1500MSS can cause more loss in
>> networks with higher segment drop rates.
> 
> Understood the concern. But sometimes we'll need to take active
> action in order to break the deadlock - TCP spec relying on
> switch/router upgrade while equipment vendors waiting for TCP spec to
> be ratified.

Let's please be clear that:

1) there is NO *urgent* problem
	We're talking about an optimization. If we do
	nothing, standards-compliant implementations
	will continue to work fine.

2) the benefit and impact need to be considered together, for everyone
	I doubt any logic that assumes both:
	a) we NEED to make a change for some
	AND
	b) this will HURT some others

TCP is designed to *always* work (as much as possible). It's more
conservative than it needs to be most of the time. Some implementors
take advantage of that, and the tragedy of the commons, and get away
with things that the IETF may not want to endorse for widespread use.

We really need to see the details to move on. Keep in mind as well that
if you're modifying TCP, taking the time to document your proposal (no,
an extended set of slides is not documentation) is a good indication of
your overall diligence (and vice versa).

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 14:59:21 -0800

Marco Mellia wrote:
...
>   - large initial windows are commonly seems. 16kB of initial window can
> be easily observed in any network. I suspect this is related to TCP

See RFC2525; some of these may be candidates for 2525-bis.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 15:05:53 -0800

+nandita

> provided on the list IMO.

I'm 100% with you on this, and have been working hard on a detailed report
on our experiments done so far to be published in the next few weeks.

Jerry

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 15:10:28 -0800

Jerry Chu wrote:
...
> on our experiments done so far to be published in the next few weeks.

It would be very useful if that work could be available in advance of
the IETF, e.g., by the 00 cutoff date (to give us time to take a look at
it).

It would also be useful, IMO, if the presentation summarized that work,
and expected those present to be familiar with it, rather than going
through it in detail, FWIW...

Joe

---

From: Jerry Chu <[email-masked]>
Date: Wed, 3 Feb 2010 15:48:49 -0800

> the IETF, e.g., by the 00 cutoff date (to give us time to take a look at

Yes that's our target date (3/1) for both the I-D and the paper detailing our
experimental results. We've been working hard on this for the past 6+
months, with a focus on studying the negative impact, which has been
a challenge both due to its small scale, and the limitation imposed by
passive measurements.

> through it in detail, FWIW...

Ok,

Jerry

---

From: rick jones <[email-masked]>
Date: Wed, 03 Feb 2010 19:43:39 -0800

> address; persistent connections should render this moot for subsequent

Modulo slow-start after idle.  Such as what one might experience  
between viewing the Nth and N+1st search result page.

rick jones

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 19:49:37 -0800

rick jones wrote:
> Modulo slow-start after idle. 

I've already cited the work that explains that this doesn't actually
happen on web exchanges exactly because the client request is small
(doesn't matter that its side was idle), and the client request causes
the server to think it wasn't idle either. That's in the ToN paper I cited.

Yes, if/when that's fixed, it'll be an issue. Not right now, though.

Joe

---

From: Mike Belshe <[email-masked]>
Date: Wed, 3 Feb 2010 20:35:14 -0800

On Wed, Feb 3, 2010 at 2:57 PM, Joe Touch <[email-masked]> wrote:

> >> The slow start algorithm tries to probe the network capacity, not the
> end node's available window space. The network capacity is dependent on
> switch/router queue lengths, among other things and therefore the initial
> receive window is not something that can/should be negotiated by end points.
> >>

>        b) this will HURT some others

The hypothesis is that init-cwnd was effectively raised long ago by
applications (ahem - HTTP) leveraging multiple connections.  I'm interested
to hear what others think of the hypothesis and how we can prove it to be
either true or false.

Assuming the hypothesis is true, it doesn't remove the possibility of (2b)
above, but it morphs it in interesting ways:
    a) The same discussions about increasing initial-cwnd have been around
for years.  So far, TCP changed little, and instead, applications have
adapted.  Like it or not, the same thing will likely happen as we move
forward as long as there are latency benefits to be had.  I know that
doesn't sound good - I'm just thinking pragmatically. :-)

    b) Making changes to initial cwnd today will have a multiplicative
effect in current software.  I can demonstrate that in some cases, initial
cwnd is already 40+ because HTTP is using so many connections.  So
increasing the official initcwnd from 3 to 10 is effectively increasing cwnd
to 100+ in some cases.

    c) It would be sad if, because HTTP worked around this problem by
abusing TCP, that new low-latency protocols would have no choice but to
replicate current broken behavior to be competitive.

Finally, if there is data needed which chromium can assist with gathering, I
volunteer to help make it happen.

Mike

---

From: rick jones <[email-masked]>
Date: Wed, 03 Feb 2010 21:41:28 -0800

> Yes, if/when that's fixed, it'll be an issue. Not right now, though.

I'd not looked far ahead enough in my unread mail queue when I sent my  
message.  Still...  I have run some netperf TCP_RR tests between my  
home system:

Darwin rick-2 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01  
PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386

and netperf.org:

Linux www.netperf.org 2.6.31-15-server #50-Ubuntu SMP Tue Nov 10  
15:50:36 UTC 2009 x86_64 GNU/Linux

To avoid any issues with TSO clouding the trace, I disabled TSO and  
GSO on netperf.org.

On my home system I did ./configure --enable-intervals then built  
netperf, and then while running tcpdump on netperf.org executed:

src/netperf -l 300 -p 12345 -t TCP_RR -H netperf.org -b 1 -w 1s -- -r  
128,16384 -P 23456
TCP REQUEST/RESPONSE TEST from (null) (0.0.0.0) port 23456 AF_INET to  
netperf.org (192.6.38.59) port 23456 AF_INET : interval
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

131070 262140 128      16384   300.00      1.00
16384  87380

Netperf will send a 128 byte request to netperf.org, which will  
respond immediately.  Netperf then waits one second before sending the  
next request etc etc...

The raw and "cooked" traces from that can be seen at:

ftp://ftp.netperf.org/traces

There one can also see "nothinktime" raw and cooked traces for:

[rick-2:~/netperf2_work] perfgeek% src/netperf -l 30 -p 12345 -t  
TCP_RR -H netperf.org  -- -r 128,16384 -P 23456
TCP REQUEST/RESPONSE TEST from (null) (0.0.0.0) port 23456 AF_INET to  
netperf.org (192.6.38.59) port 23456 AF_INET : interval
tcpLocal /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

131070 262140 128      16384   30.00       7.60
16384  87380

It certainly appears that *something* is holding-back the server in  
the "thinktime" case - I don't know if it is slowstart after idle, but  
it is something that does not appear to be holding-back the server in  
the "nothinktime" case.  Hmm... while writing this message, I have  
remembered there is a sysctl in linux to control slow-start after  
idle, so there is also a third trace, one with thinktime, and  
net.ipv4.tcp_slow_start_after_idle = 0.  From the look of that trace,  
and how it compares with the other two, I think that Linux 2.6.31 at  
least is not actually having the arriving request cause it to think it  
wasn't idle?

happy benchmarking,

rick jones

PS - I suppose if I'd been thinking more clearly, I'd have also  
included --enable-historgram in the ./configure command and been able  
to give a histogram of the response times (adding a -v 2 option to the  
global command line options), but I suppose that can be guesstimated  
from the traces.

http://homepage.mac.com/perfgeek

---

From: rick jones <[email-masked]>
Date: Wed, 03 Feb 2010 22:09:00 -0800

On Feb 3, 2010, at 9:41 PM, rick jones wrote:
> included --enable-historgram in the ./configure command and been  
> able to give a histogram of the response times (adding a -v 2 option  
> to the global command line options), but I suppose that can be  
> guesstimated from the traces.

Ok, so perhaps I like running netperf too much :)  Here then are the  
application-level measured response times.  The first netperf is with  
tcp_slow_start_after_idle = 0, the second with it set to 1, I'll leave- 
out the rest of the netperf output as it is repetitive:

tcp_slow_start_after_idle = 0 on server:

Histogram of request/response times
UNIT_USEC     :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_USEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
HUNDRED_USEC  :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
UNIT_MSEC     :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_MSEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
HUNDRED_MSEC  :    0:  294:    4:    0:    2:    0:    0:    0:     
0:    0
UNIT_SEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_SEC       :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
 >100_SECS: 0
HIST_TOTAL:      300

That means 294 times 100ms <= responsetime < 200ms, 4 times it was  
200ms <= responsetime < 300ms etc etc.

tcp_slow_start_after_idle = 1 on server:

UNIT_USEC     :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_USEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
HUNDRED_USEC  :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
UNIT_MSEC     :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_MSEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
HUNDRED_MSEC  :    0:    0:    0:  290:   10:    0:    0:    0:     
0:    0
UNIT_SEC      :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
TEN_SEC       :    0:    0:    0:    0:    0:    0:    0:    0:     
0:    0
 >100_SECS: 0
HIST_TOTAL:      300

Now it was 290 with 300ms <= responsetime < 400ms

Just as a refresher, the request size was 128 bytes, the response size  
16384.

The ICMP Echo RTT is ~120 ms between my home system and netperf.org:

--- netperf.org ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/stddev = 119.523/120.410/121.248/0.493 ms

rick jones
Wisdom teeth are impacted, people are affected by the effects of events

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Feb 2010 22:25:12 -0800

Did you use a netperf option to disable Nagle?

There are also many reasons this could happen; it might be easier to see
in a plot.

Joe

rick jones wrote:
> tcp_slow_start_after_idle = 0, the second with it set to 1, I'll
> leave-out the rest of the netperf output as it is repetitive:
> 
> UNIT_USEC     :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_USEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> HUNDRED_USEC  :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> UNIT_MSEC     :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_MSEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> HUNDRED_MSEC  :    0:  294:    4:    0:    2:    0:    0:    0:    0:    0
> UNIT_SEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_SEC       :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
>>100_SECS: 0
> HIST_TOTAL:      300
> 
> That means 294 times 100ms <= responsetime < 200ms, 4 times it was 200ms
> <= responsetime < 300ms etc etc.
> 
> UNIT_USEC     :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_USEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> HUNDRED_USEC  :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> UNIT_MSEC     :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_MSEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> HUNDRED_MSEC  :    0:    0:    0:  290:   10:    0:    0:    0:    0:    0
> UNIT_SEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
> TEN_SEC       :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
>>100_SECS: 0
> HIST_TOTAL:      300
>

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 04 Feb 2010 09:08:45 +0100

The motivation for this is RFC 2861: TCP Congestion Window Validation

Alex

> is currently no pacing mechanism in TCP to smooth out such bursts.

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220
// email: [email-masked]
// web: http://www.umic-mesh.net
//

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 04 Feb 2010 18:08:57 +0800

> Any idle period higher than the RTT between HTTP requests tends to reset the cwindow down to the restart window. This hurts persistent connections too. What's the motivation for this, can we have a…

I believe the restart cwnd is the same as the initial cwnd in most
stacks.  So are you suggesting that the initial cwnd is not large
enough in the restart case?

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 04 Feb 2010 19:38:01 +0800

> that can't be the basis for lowering the bar about significant changes

Something interesting to "chit-chat" during coffee break...  I
think the history of the current RFC 3390 limit is like the
following.  Please correct me if I am wrong.

The original limit was 1.  But because of an implementation mistake
in BSD, the reality was actually 2.  And IETF accepted the reality.
Then folks realized that during slow start, TCP almost always sent
a burst of 3 segments because of delayed ACK.  So people thought that
around 4380 (3 * 1460 bytes which is a common segment size) was fine
and came up with a formula as in RFC 3390.

Just wondering, if there were no bug, would the initial cwnd be
increased almost 10 years ago?  Note that I am not trying to start
a debate on history nor reality vs research ;-)

-- 

						K. Poon.
						[email-masked]

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 4 Feb 2010 14:35:58 +0100

Jerry,

> Agreed. The question is, what is that threshold of 
> window/burst size beyond which pacing is required. My current 
> guess is > 10.

I have the same impression, at least from what I have seen in simulations.

Yet, one could think of a very simple form of pacing with a single timer that changes the Slow-Start only when it "hurts" - I think that I have mentioned this possibility on this list before:

In this approach, the initial window remains at the value given by RFC 3390. But the sender is allowed to further increase the window after a certain time (say, 100ms), but only 
if the RTT is known to be larger than 100ms (as measured e. g. during handshake), if no ACK has been received until then, and if recently there have been no signs of congestion to this destination.

The advantage of this approach is that TCP uses the larger initial window only if the RTT is larger than 100ms (or whatever threshold is defined). On the one hand, this reduces the performance problems that can be observed for RTTs of the order of 200ms. But, on the other hand, flows with a smaller RTT would use the existing Slow-Start, i. e., the faster startup is only used by a small number of flows and thus be much less disruptive than a general revision of RFC 3390.

A simpler variant of this idea is to allow an increased initial window beyond RFC 3390 if the measured RTT is larger than a certain threshold such as 100ms.

Michael

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 4 Feb 2010 14:59:13 +0100

Jerry,

> Actually our data points to the contrary - the average web 
> object and page size have been rising steadily. E.g., the 
> majority of our search responses no longer fit in 3 MSS these days.

OK. I looked at some traffic traces from the year 2008, which might not
be representative any more.

> > In the past, I have played a little bit with an initial 
> window e. g. 
> > of 10 MSS, and that worked surprisingly well.
> 
> Good to know!
> 

Please note that I used the value of 10 MSS just as an example. I'd
actually expect that doubling the allowed initial window would be
sufficient in order to solve most real-world Slow-Start performance
issues.

Michael

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 4 Feb 2010 15:05:12 +0100 (CET)

On Thu, 4 Feb 2010, SCHARF, Michael wrote:

> Yet, one could think of a very simple form of pacing with a single 
>timer that changes the Slow-Start only when it "hurts" - I think that I 
>have mentioned this possibility on this list before:
>
> In this approach, the initial window remains at the value given by RFC 
>3390. But the sender is allowed to further increase the window after a 
>certain time (say, 100ms), but only
> if the RTT is known to be larger than 100ms (as measured e. g. during 
>handshake), if no ACK has been received until then, and if recently there 
>have been no signs of congestion to this destination.
>
> The advantage of this approach is that TCP uses the larger initial 
>window only if the RTT is larger than 100ms (or whatever threshold is 
>defined). On the one hand, this reduces the performance problems that can 
>be observed for RTTs of the order of 200ms. But, on the other hand, flows 
>with a smaller RTT would use the existing Slow-Start, i. e., the faster 
>startup is only used by a small number of flows and thus be much less 
>disruptive than a general revision of RFC 3390.
>
> A simpler variant of this idea is to allow an increased initial window 
>beyond RFC 3390 if the measured RTT is larger than a certain threshold 
>such as 100ms.

And a possible tweak to this: if we increased the initial window beyond 
RFC3390, measure the number of retransmits that have to be done for those 
segments. If the stack had to retransmit, then subsequent connection 
attempts from the host would need to reduce this "increased" window.

I think this could help to take care of the concern about the short queue 
limits on the access-layer equipment.

cheers,
andrew

---

From: Joe Touch <[email-masked]>
Date: Thu, 04 Feb 2010 06:34:26 -0800

> and came up with a formula as in RFC 3390.

That, and a LOT of simulation analysis. The other alternative was to
declare this a bug and recommend against it.

> a debate on history nor reality vs research ;-)

It's hard to say whether there would have been motivation to look at the
issue so deeply were it not for the bug, but ultimately it was the depth
of analysis that convinced the group to approve the change.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Thu, 4 Feb 2010 07:59:47 -0800

Michael,

> A simpler variant of this idea is to allow an increased initial window beyond RFC 3390 if the measured RTT is larger than a certain threshold such as 100ms.

Thanks for all the interesting ideas. We may need them in the future.

We actually spent the first few months testing a more complex
algorithm, partly based on Joe Touch's caching
idea described in RFC2140. We abandoned that approach later due to a
number of problems, low cache hit
rate due to server side load balancing being one of them. We have
since adopted a two-phase approach. The first
phase follows the KISS principle to simply pick a fixed, larger number
for initial window that has minimal negative
impact for all connections (and I can't believe it can't be bumped up
by more than a couple of MSS safely, given
the huge growth of average bandwidth and the gradual disappearing of
dialup links < 56Kbps. Our test data has
confirmed this, detailed in the upcoming paper). This enables a more
timely deployment and a faster web
experience for most people sooner while we can embark on a second
phase to investigate a more elaborate
algorithm involving pacing and other non-trivial implementation tasks
but can unleash more performance for the
regular users with 1.7Mbps, the average b/w globally reported recently
by Aktimai.

I understand and appreciate the "err on the conservative side"
principle. The robustness and ubiquity of the Internet
largely came out of it. But at some point it needs to be balanced
against the need for improvement. My fear is that
IETF might risk becoming irrelevant if it's too conservative, much
like California's highway speed limit 20 years
ago. It was at 55MPH while hardly anyone drove below that speed. (Of
course what is considered "conservative",
"risky" can be very subjective. The trick to get to some consensus here...)

Jerry

---

From: rick jones <[email-masked]>
Date: Thu, 04 Feb 2010 08:00:42 -0800

> Did you use a netperf option to disable Nagle?

No, there should be no need for that.  For a TCP_RR test netperf  
presents all the data of a request or response to the transport in a  
single send call,  there should be no delays Nagle at all.  Further,  
in this case, there was only ever the one transaction in flight at a  
time, so at the time of each send() the connection was "idle" in the  
Nagle sense.

> There are also many reasons this could happen; it might be easier to  

Anyone should feel free to grab either the binary (.pcap) or  
"cooked" (ascii) traces off of netperf.org and run them through  
whatever post processing tools they like.  I would have shoved them  
into the original email, but didn't want to stuff peoples' inboxes  
unnecessarily.  Still, if you think that copying some examples from  
the ascii into email would help I'm willing to do that.

I'm not the acme of tcpdump trace reading, but it seemed pretty  
conclusive in the traces - with slowstart after idle, the 2.6.31  
server sent the data of each 16K response with what certainly appears  
to be slow-start behaviour, with that sysctl set to disable slowstart  
after idle, no such slow start behaviour in the middle of the trace.   
So, for Linux 2.6.31, as distributed by Ubuntu, it would seem that the  
arrival of the next request is not going to preclude slowstart after  
idle behaviour.

rick jones

>>> able

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 05 Feb 2010 00:28:40 +0800

> if the RTT is known to be larger than 100ms (as measured e. g. during handshake), if no ACK has been received until then, and if recently there have been no signs of congestion to this destination.

For a server, it may just have 1 sample of RTT (the ACK for its 
SYN/ACK).  Is this enough to determine much?  Suppose the ACK
for the SYN/ACK is lost, and the server gets a request (data +
ACK for the SYN/ACK) instead, can it distinguish this case from
the case when a client can bundle data in third leg of 3-way
handshake?

And it seems to defeat much of the benefits of increasing the
initial cwnd if the server needs to wait 100ms before sending
more.

-- 

						K. Poon.
						[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Thu, 04 Feb 2010 08:39:38 -0800

rick jones wrote:
> single send call,  there should be no delays Nagle at all.

I don't know why this still isn't widely known, but there is *no*
correlation between the OS interface and TCP segment emissions per se.
Send calls don't correspond to buffer flushes or anything else. All they
do is load the buffer.

Nagle still needs to be disabled.

> Further, in
> this case, there was only ever the one transaction in flight at a time,
> so at the time of each send() the connection was "idle" in the Nagle sense.

Again, "at the time of each send()" is irrelelvant. What's relevant is
what's in the buffers.

> Anyone should feel free to grab either the binary (.pcap) or "cooked"
> (ascii) traces off of netperf.org and run them through whatever post
> processing tools they like.  I would have shoved them into the original
> email, but didn't want to stuff peoples' inboxes unnecessarily.  Still,
> if you think that copying some examples from the ascii into email would
> help I'm willing to do that.
> 
> conclusive in the traces - with slowstart after idle, the 2.6.31 server
> sent the data of each 16K response with what certainly appears to be
> slow-start behaviour, with that sysctl set to disable slowstart after
> idle, no such slow start behaviour in the middle of the trace.  So, for
> Linux 2.6.31, as distributed by Ubuntu, it would seem that the arrival
> of the next request is not going to preclude slowstart after idle
> behaviour.

Some versions of Linux may have various tricks implemented. None of them
havae been standardized, though. It's worth checking with a few
different OS's.

Joe

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 4 Feb 2010 17:58:30 +0100

> > Yet, one could think of a very simple form of pacing with a 
> single timer that changes the Slow-Start only when it "hurts" 
> - I think that I have mentioned this possibility on this list before:
> >
> > 3390. But the sender is allowed to further increase the 
> window after a certain time (say, 100ms), but only if the RTT 
> is known to be larger than 100ms (as measured e. g. during 
> handshake), if no ACK has been received until then, and if 
> recently there have been no signs of congestion to this destination.
> 
> 
> For a server, it may just have 1 sample of RTT (the ACK for 
> its SYN/ACK).  Is this enough to determine much?  Suppose the 
> ACK for the SYN/ACK is lost, and the server gets a request 
> (data + ACK for the SYN/ACK) instead, can it distinguish this 
> case from the case when a client can bundle data in third leg 
> of 3-way handshake?

Of course, one would not use this mechanism if there is any sign of lost
packets. But, obviously, any heuristic can fail. BTW, another problem is
a highly congested link resulting in a large RTT. In this case, the
algorithm would send additional data to a congested link - but this
would also happen if the initial window would be increased. There is no
free lunch.

> initial cwnd if the server needs to wait 100ms before sending more.

If the RTT is larger than 100ms (or whatever threshold is used), you
would still get a benefit compared to RFC 3390. Maybe that would be just
good enough?

Michael

---

From: rick jones <[email-masked]>
Date: Thu, 04 Feb 2010 10:17:44 -0800

> Again, "at the time of each send()" is irrelelvant. What's relevant is

My experience thusfar with netperf benchmarking has been that the  
single-transaction at a time TCP_RR test has never run into Nagle- 
induced delays on any stack that wasn't broken, and I cannot recall  
the last time I encountered such a broken stack.  Still, if it will  
help remove a crimson fish from the sea, I have placed under:

ftp://ftp.netperf.org/traces/nonagle

another set of traces, this time with the -D option added to netperf  
to cause it to set TCP_NODELAY on either end.  My look at them  
suggests there has been no change in behaviour from the initial set of  
traces without TCP_NODELAY set.

> havae been standardized, though. It's worth checking with a few

The cooked traces with and without thinktime, and HP-UX 11.31 as the  
server are under "hpux_11.31" under the URL above.  They too are with  
TCP_NODELAY set on both ends, and they too seem to show a very slow- 
start after idle behaviour in the presence of client thinktime.  There  
is no obvious ndd option to disable the behaviour so there is no third  
trace there.

rick jones

---

From: Joe Touch <[email-masked]>
Date: Thu, 04 Feb 2010 10:38:08 -0800

rick jones wrote:
...
> single-transaction at a time TCP_RR test has never run into
> Nagle-induced delays on any stack that wasn't broken, and I cannot
> recall the last time I encountered such a broken stack. 

What you have seen is consistency in implementations. Such behavior is
not required by the specs, so it should not be considered "broken", nor
should tests rely on it.

Thanks for removing it as an issue, though. There are many other things
that could be happening, and as I noted the behavior was never really
well spec'd AFAIR.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 04 Feb 2010 10:40:32 -0800

PS - it might be useful to take a look at the following old (expired) ID
on this issue. As I noted, I'd be glad to 'restart' it...

Joe

http://www.isi.edu/touch/pubs/draft-hughes-restart-00.txt

---

From: Jerry Chu <[email-masked]>
Date: Thu, 4 Feb 2010 17:30:07 -0800

On Thu, Feb 4, 2010 at 6:05 AM, Andrew Yourtchenko <[email-masked]> wrote:
>> Yet, one could think of a very simple form of pacing with a single timer
>> that changes the Slow-Start only when it "hurts" - I think that I have
>> mentioned this possibility on this list before:
>>
>> The advantage of this approach is that TCP uses the larger initial window
>> only if the RTT is larger than 100ms (or whatever threshold is defined). On
>> the one hand, this reduces the performance problems that can be observed for
>> RTTs of the order of 200ms. But, on the other hand, flows with a smaller RTT
>> would use the existing Slow-Start, i. e., the faster startup is only used by
>> a small number of flows and thus be much less disruptive than a general
>> revision of RFC 3390.
>>
>> beyond RFC 3390 if the measured RTT is larger than a certain threshold such
>> as 100ms.
>

Yes we have tried something very similar (dubbed "negative cache"), but
discovered its effectiveness depends heavily on a high cache hit, which is
difficult in a large server farm behind a stateless load balancer.

A small tweak included in the upcoming I-D is to back down to rfc3390 on
RW if any burst size (during IW or RW) > 4KB incurs pkt loss. Admittedly
this is a smaller and less effective tweak but it doesn't need to maintain
any history across different connections.

Jerry