Subject: [tcpm] alternate IW proposal

From: Mark Allman <[email-masked]>
Date: Mon, 15 Nov 2010 12:39:38 -0500

Dumb idea:

  http://www.ietf.org/id/draft-allman-tcpm-bump-initcwnd-00.txt

I.e.,

  - Stop waving our hands about where things might break.

  - Be less aggressive than the 10 segment IW proposed by Jerry, et.al.

  - Do something.  And, start observing how it actually works.

  - Get to 10 segments in 3 years.  But, with much less fluffy
    discussion and hand waving.

  - Don't stop at 10 segments, but go all the way to 15 segments over
    the course of the next 9 years (which is how long it has been since
    we last increased the IW).

allman, stirin' the pot

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Mon, 15 Nov 2010 11:45:49 -0800

Folks,

I am not sure my response to this may be construed as "hand waving".
Basically, I have some general process questions to begin with and later
some comments about the document :-

ONE REQUEST : Please read this short email in its entirety and provide
comments, sometimes the case being made is spread across multiple
points, so just please don't pick on one word or sentence and snip the
rest. The hope is, I don't want to create a big thread and start a fiery
discussion on this aspect (mostly process oriented )

- what is considered a minor change versus a major to TCP? Can the
charter be modified to explicitly clarify (best effort would do) this.

- IW change, some may feel that it is a fundamental change, but for some
it is a minor change or a "parameter change". Well, all of these can be
argued as hand waving, IMO.

- Just for the illustration lets divide TCP into multiple segments :-
The API interface, the protocol semantics, TCP options, TCP congestion
and flow control, TCP security, TCP integrity (like checksum etc.,) and
some more..  TCP is pervasive protocol and if you really look at each
and every aspect of this is equally important and fixing those would
generally benefit a lot of applications and hence the "internet". 

- IRTF versus IETF :- When IW=10 was brought up I assumed that it
sounded like IRTF work. Why should we keep asking such questions if the
charter clearly spells out : "what is construed as a minor change to
TCP"? 

Now there seems to be good interest in getting IW=10 adopted and move
on, when folks say, "isn't it aggressive" ? (maybe it is a repeat
question, causing annoyance, but see below), which is a fair question,
IMO, I am baffled to learn that it is construed as hand waving WHEREAS
changes that fall into other categories are generally met with stiff
resistance with some folks conveniently changing hats and proclaim "the
bar is very high for changing TCP". This is double standards, IMO. 

It would be expedient to analyze merits and de-merits and not "hand
wave"..  if there is need for more data, then let's call it out
explicitly. Once data is collected and the issues are resolved the next
revision of the document should acknowledge such a discussion took place
and the consensus was "so and so". This way any new entrant who has same
questions ( a new entrant is not necessarily new to TCP, it is just that
(s)he may not have been following that draft) Makes sense since, if such
issues are considered as roadblocks for forwarding momentum of the
document, then it is very beneficial to state such things in the
document. 

Some earlier documents used to have an appendix which listed the changes
from prior versions and the open issues and consensus of each of them.
Later on it was removed when published as an RFC, when the essence got
captured into the main document. In fact, in some cases multiple email
threads of each open item were started and later posted the resolutions
of each of the items. I think for the current document such a
methodology may be useful. In the current case, most of the questions
are met "we have collected data, go look for it". I clearly want to
state something here : I fully understand that Google has done a good
job on collecting the data, and that needs to be done since they are
proposing an important (for the lack of a better word in my thought
process) change. Isn't it fairly obvious that folks reviewing the
document may not have all the information gathered and can ask some
fundamental questions? Well, if this isn't the expectation, I am
surprised.

Well, when was the data collected? At different times of the day, peak
hours versus no-peak hours?, continents or topologies, how was the
burstiness controlled (the document does acknowledge something to this
effect), also can we get the traceroute of the paths being simulated?
(well, let's not pick on words here, I am not asking for such a thing,
but the data collection needs to list them). The document in its
existing form doesn't answer these questions, but the pointers provided
also don't discuss these stuff exhaustively. That said, I should point
out that for anyone who wants to do a better job in reviewing this
document, needs to study the data being collected and come up with
questions. 

FWIW, I not opposed to this document, I am just trying to question the
"extra protection" this document seems to enjoy, even though this seems
like a important change to TCP and needs more scrutiny. One thing for
sure, this document needs an applicability statement, i/e,. under what
circumstances this IW = 10 can be used, and which cases it may be risky,
in other words if definitely needs to be based on a configurable knob.

Just my few bits.. 

Thanks,
Anantha

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Nov 2010 13:29:22 -0800

On 11/15/2010 11:45 AM, Anantha Ramaiah (ananth) wrote:
...
> charter clearly spells out : "what is construed as a minor change to

I didn't read what happened this way.

AFAIR, we only said "the CCRG ought to look at this too", not that the 
IRTF was the place it should happen once it was reviewed.

As to the hurdle, I think it ought to be high for this, perhaps higher 
than for other changes we've discussed. Some of the TCPM changes don't 
affect anyone except those who use them; this one affects congestion 
everywhere, potentially. I'm not uncomfortable with a fairly high bar on 
such things.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Nov 2010 13:30:09 -0800

FWIW, I like this idea better, with the caveat that the doc needs to 
explain:

	- what does "observing" mean?

	- what is the metric of "breaks"?

Just putting an idea out there, with no metric for repeal or success, is 
asking for trouble.

Joe

On 11/15/2010 9:39 AM, Mark Allman wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Michael Welzl <[email-masked]>
Date: Tue, 16 Nov 2010 10:18:15 +0100

Hi,

I agree that this is a neat idea.

BTW I also agree with the various comments regarding my proposal to  
let applications decide the IW; it was just an idea, for trying to  
find an easy compromise, but you guys convinced me that it was a  
stupid one.

Cheers,
Michael

On Nov 15, 2010, at 10:30 PM, Joe Touch wrote:

> Just putting an idea out there, with no metric for repeal or  
> success, is asking for trouble.
>
> Joe
>

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Nov 2010 15:58:50 -0500

> everywhere, potentially. I'm not uncomfortable with a fairly high bar
> on such things.

I disagree.  I have said this before.  The initial congestion window is
a parameter choice.  As long as we pick a reasonable parameter then TCP
will puzzle its way out (in the cases where that is even needed; a
reasonable number would be a cwnd that the vast majority of connections
would be able to attain---given enough data---anyway).  I've looked at a
lot of evidence that folks have thrown about and it suggests to me that
10 segments is *well* within "reasonable".  This is a detail and it need
not be sweated so much.

allman

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Nov 2010 14:52:12 -0800

On 11/17/2010 12:58 PM, Mark Allman wrote:
> would be able to attain---given enough data---anyway).

That would be true of TCP learned the IW over time, even across various 
connections. I.e., if it started with IW=3, and found that this wasn't a 
problem, then it increased to IW=10 and found no problems. If, however, 
IW=10 resulted in large amounts of drops, then it should back off and 
use a smaller IW for a while.

However, IW doesn't run in AIMD mode. This "parameter" persistently 
affects all subsequent connections, which ends up changing the nature of 
traffic on the network. There have been various discussions of what 
could happen, and we don't really know much better now what will happen 
with IW=10 than we knew when we increased it from 1 to 2 (accidentally, 
due to the bug in the original code), then from 2 to 3 (explicitly).

There are a few differences this time around:

1) the proposed increase changes the parameter in use by 233%, rather 
than by 50%

we don't know what the widespread effect of this will be on whether 
other connections will incur drops, e.g., if this changes how much 
buffering routers need for a given level of drops, etc.

Single-dumbbell experiments are known insufficient to explore this issue.

2) the rationale for the proposal is a particular current use

The increased performance rationale for this is sketchy at best, and 
it's not clear that endorsing this sort of 'millions of individual 
connections' use is good for the network (or whether it's even 
necessary, given appropriate use of Nagle and a reasonable number of 
concurrent connections)

3) we'll end up right back here if we're not careful

Your doc is a good step in avoiding that, but lacks precisely the 
objective metrics we are struggling with now, and without which such 
changes will clearly continue unfettered.

---

I subscribe to the "first do no harm" principle. I have yet to see 
enough evidence this is necessary, and certainly not enough to convince 
me that it's worth even the potential risk to make these changes yet.

Joe

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Nov 2010 22:21:23 -0500

> That would be true of TCP learned the IW over time, 

I disagree.  Its fine if we see this differently.  But, I don't find
your argument correct at all.  I.e., 

  - TCP will react to problems in the initial window.  It will.  It will
    figure itself out of it within a connection.

  - The cross connection dynamic is presuming you're making the same
    mistakes over-and-over.  That isn't clear.  If you make a zillion
    10 packet connections to some host then OK.  I am willing to call
    that pathological.

  - The vast majority of traffic volume is from large flows that
    converge via the usual AIMD.  Therefore, by traffic volume the
    initial window is a quite small effect---especially if we are
    roughly in the area of "reasonable" (i.e., if the 10 packet window
    works the vast majority of the time).  

> This "parameter" persistently affects all subsequent connections,
> which ends up changing the nature of traffic on the network.

Absolutely this will change some of the dynamics.  There is no doubt.
But, there are zillions of things that change the dynamics of the
networks in much larger ways and are perfectly consistent with the
standards.  Add a deep queue.  Add a web cache.  Run BitTorrent.  Shape
traffic.  Do DNS prefetching of a zillion domains because they are all
found on a web page.  Install or bypass a CDN.  Etc.  Etc.  We need to
show some perspective here.

> There have been various discussions of what could happen, and we don't
> really know much better now what will happen with IW=10 than we knew
> when we increased it from 1 to 2 (accidentally, due to the bug in the
> original code), then from 2 to 3 (explicitly).

I disagree.  The data currently on hand is much better than the data we
had [and, I generated a bunch of it] when we moved to 4KB.  I don't
think its even close.

> 1) the proposed increase changes the parameter in use by 233%,
> rather than by 50%
> 

We didn't know that last time either.  This is *nearly impossible* to
understand without just rolling the change out at scale.  I'd for sure
like to know the answer to this question a priori.  But, I don't see how
we get it.

It may be a little bigger change.  The bug that caused the 2 packet
window wasn't somehow ubiquitous and so I am not sure the move to 4KB
wasn't a tripling of the size of the IW---which is roughly what the
proposed change is.

That said, the initial step in my progress---which wasn't called out
well, but I have added text to my private version today---was to 6
segments for a reason.  Slow start by definition tolerates errors of
2x.  So, if we are happy enough with 3 packets then an IW of 6 packets
bounds the error at 2x.  I am not uncomfortable with 3x from what I have
seen.  But, I understand the point.

> necessary, given appropriate use of Nagle and a reasonable number of

[I can't parse this at all.]

> 3) we'll end up right back here if we're not careful

That's not true.  Careful or not we'll end up right back where we
are---because networks evolve and therefore it is natural to evolve
parameters, too.

> changes will clearly continue unfettered.

Right- we talked about this yesterday.  It'd be really nice if we could
plop some objective metrics in there.  I don't quite see it (and, as I
recall, neither do you).  So, the quick suggestion I threw out is
flipping from a path whereby every so often we think hard about this and
ultimately make our best engineering judgment to a path where we make an
informed guess [although, that chart did not come out of thin air] at a
slow and steady progression and we endeavor to try to understand small
changes. 

allman

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 08:31:57 -0800

Hi, Mark,

>      figure itself out of it within a connection.

TCP does NOT react to problems in the IW. It keeps using the same IW for 
every connection, banging its head against the wall continually.

It COULD. That's what I'm wondering might be useful...

e.g.:
	TCP MAY increase its IW if it monitors the impact of that
	increase, and uses an AIMD algorithm to adjust the IW values
	over time.

>      10 packet connections to some host then OK.  I am willing to call

Let me be more clear. The IW isn't a property of the destination, it's a 
property of the path. Either the path can tolerate new connections with 
large IWs, or it cannot.

Some paths from a given source may exhibit problems only for some 
destinations. However, there are some sources that have problems ON 
THEIR END of the network, i.e., that impact most, if not all, of their 
connections.

E.g., a laptop on a low-speed link, or on a disadvantaged net.

>      works the vast majority of the time).

IW traffic is bursty, and bursts tend to synchronize. Further, IW is 
uses more than just for the connection start - it's also used for 
restart, e.g., after an idle period. This all means that increasing the 
IW without feedback could easily lead to problems that we cannot correct 
once deployed.

...
> [I can't parse this at all.]

Let me rephrase:

How much does this help? How much does it reduce latency? Most of the 
info I've seen summarized suggests the benefit is in the 5-20% range. 
That's not even noticeable from a user POV.

So, back to my original point, how about IW using a long-timescale AIMD? 
If we do that, we can avoid needing to monitor the problem en-masse, and 
the system will adapt over time and we won't be back here in another 10 
years...

Joe

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 13:00:51 -0500

Joe:
> TCP does NOT react to problems in the IW. It keeps using the same IW
> for every connection, banging its head against the wall continually.

Joe, please read what I wrote.  Please in particular note the last three
words.  You are not replying to what I said.  My statement is right.
Your reaction that "TCP does NOT react to problems in the IW" is wrong.
(Yes, sometimes it happens.)

(I agree that there is no history across connections, but that is has
nothing to do with what I said above.)

allman

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 13:06:05 -0500

> Let me be more clear. The IW isn't a property of the destination,
> it's a property of the path. 

No it isn't.  The IW is a construct of the sender.  The path DOES NOT
HAVE A WINDOW, LET ALONE AN **INITIAL** WINDOW.  Period.

Congestion is a property of a particular link.  And, congestion of
individual links aggregate into the congestion experienced across a
path.  But, there is no way to, e.g., measure the "IW of a link".  That
doesn't make any sense.

TCP has a system of *inferring* congestion along the network path a
connection traverses (putting aside ECN, which isn't much used).  And,
it has a construct called a "congestion window" to regulate the current
sending rate based on its observations of that path.  When a connection
starts it does not have a history of observations on which to base its
congestion window (/ sending rate).  Yet, it needs some initial rate.
So, we have to have some default here.  None of this is new or novel or
startling.

So, we use some default IW.  That default IW is a probabilistic
engineering *guess* at its root.  We use something that we believe will
work well in the vast majority of the cases and when it doesn't the
results are not somehow catastrophic.

    (It *could* have a history as you have noted (and conducted research
    on).  But, the practical efficacy of that has at least not been
    shown (as far as I know; perhaps I am out of the loop).  E.g., how
    often do two hosts communicate?  And, is it meaningful to cache
    across the timeframe between those communications?  E.g., even if
    you think you're talking to a particular host you may be talking to
    a farm and your connection gets sprayed to the "best" possible
    server at some moment in time (for some secret-sauce definition of
    "best") and that server might not have the state from your last
    interaction.  Etc.  So, I agree that this is something we can think
    further about and it may well be that it is nice for things like
    repeated image gets from some web server.  However, that does not
    get us away from needing some initial setting.  At *best* you have
    reduced the importance of the setting in some cases.)

> restart, e.g., after an idle period.  

I would be perfectly happy with a rule that said the RW had to be
min(IW,max_lossless_cwnd/2).  I.e., basing the RW on observations from
the connection itself seems absolutely reasonable to me.

> How much does this help? How much does it reduce latency? Most of
> the info I've seen summarized suggests the benefit is in the 5-20%
> range. That's not even noticeable from a user POV.

20% seems like a pretty nice chunk of change to me.  That's on the order
of what we got by going to 4KB.  20% of a second is perceptible to
humans.  Add this over a number of objects and it looks like a fine bit
of increase to me.  Certainly this WG has done things for much less
benefit. 

allman

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 11:07:55 -0800

> words.  You are not replying to what I said.  My statement is right.

"Within a connection" is true.

That's not the case I consider important, and not the case I have been 
arguing. Please re-read what I posted.

> (Yes, sometimes it happens.)

TCP uses the same IW over and over. If it's wrong - if every connection 
loses 5 of the first 10 packets sent - TCP *does not learn to do 
anything different for the next **new** connection*.

We agree on this, I think. My point, all along, is that it is *this* 
issue that is of concern.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 11:15:43 -0800

> HAVE A WINDOW, LET ALONE AN **INITIAL** WINDOW.  Period.

Granted. The path doesn't have an IW. The path supports a given IW.

> Congestion is a property of a particular link.

Congestion is a property of a path. It can occur at a link, or across a 
set of links (e.g., if parallelized, or alternated). It can also occur 
on a sequence of links, e.g., if you send 1 Gbps through a path that 
includes links with remaining capacity of 100 Mbps and 10 Mbps, they 
BOTH experience congestion-based loss.

...
>      often do two hosts communicate?

You keep assuming that whether an IW works is a property of a single 
source, or of a pair of endpoints.

I am concerned about cases where the IW interacts with a link that 
affects communication with MANY hosts, and so ends up being a property 
of a set of receivers, or all receivers.

> humans.

Really? You can tell the difference between 1 second and 0.8 seconds? or 
1.2 seconds? Or 400 vs 500 ms?

 > Add this over a number of objects and it looks like a fine bit
> of increase to me.

It doesn't generally add. Separate connections are independent; the only 
way it adds is when you retrieve something in a subsequent connection 
based on the first, at which point you could have used persistent 
connections and avoided the TWHS, which saves an additional 2 RTTs *and* 
allow you to use your current window (which could be larger than the IW).

So basically the only time it adds up is when you should be doing 
something different for a more significant benefit anyway.

Joe

Joe

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 15:28:44 -0500

> source, or of a pair of endpoints.

I ain't so good with the English and only passable at American, but over
here "e.g." means "for example".

Yes, certainly one can think of cases where the congested point is close
to the sender and therefore a cache would want to consider many (or all)
recipients as equivalent for IW treatment.  If you're thinking of
caching this sort of state then that seems like a case to work out, for
sure. 

> of a set of receivers, or all receivers.

This doesn't concern me all that much ...

  - If the congestion point is pretty local then we're very much only
    hurting (if we are hurting) ourselves or those within our
    institution because we don't traverse the shared Internet.

  - If the initial window is problematic across all connections then it
    seems clear that its a bogus IW setting.  If that is the case for a
    whole lot of edges then whatever the IW setting is happens to be a
    really poor choice.  If this edge is somehow an outlier then they
    can manually configure down IW settings.  Or, deal with the small
    amount of fallout this will cause.  As I have said before, a good
    choice for an IW has to be fine for the vast majority of
    situations. 
    
But, regardless, my high order bits on this between connection business
are two-fold:

  (1) It does not divorce us from needing a default initial window when
      there is no previous history to suggest what the IW should be.

  (2) We have no idea how it would or could work.  There have been
      forays into congestion information sharing, but we don't (as far
      as I know) have any understanding of often it'd be useful in
      reality.  Nor do we have any sort of scheme to figure out what
      connections should be considered together when caching (i.e.,
      caching information obtained from connections between two IPs
      seems simple enough, but how do we go beyond that?).  This seems
      like a research project to me---not a viable alternative to
      thinking about the IW.  

      Of course, we could decide the current IW proposals are not viable
      and then go off and do the research and if it turns out to look
      viable then come back and revisit this sharing notion.  All I am
      saying is that this is so under-specified and under-investigated
      at present that it isn't like we can consider IW10
      vs. IW3-with-shared-IW-information at this time.

> Really? You can tell the difference between 1 second and 0.8
> seconds? or 1.2 seconds? 

Yep.  That's what they tell me.  I haven't run the tests, but yeah
that's the right order for human perception to start.

> > Add this over a number of objects and it looks like a fine bit of
> > increase to me.
> 
> It doesn't generally add. Separate connections are independent; the
> only way it adds is when you retrieve something in a subsequent
> connection based on the first, at which point you could have used
> persistent connections and avoided the TWHS, which saves an
> additional 2 RTTs *and* allow you to use your current window (which
> could be larger than the IW).

I'll grant you that it might not add.  

But, it might.  If I want to use 4 parallel connections to download "a
web page" and grabbing components sprayed across a bunch of servers then
it seems pretty clear that connections finishing sooner cause subsequent
connections to start sooner and if those also end sooner then you get
additive benefit.

allman

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 12:40:23 -0800

On 11/18/2010 12:28 PM, Mark Allman wrote:
>      seems clear that its a bogus IW setting.

Or

	- we're sharing an access point at a coffee shop, or one of
	those card-deck sized mobile hotspots (wifi-cell modem)

	- we all live in an apartment building or on the same cable
	Internet segment, or some similar enclave (a cruise ship comes
	to mind)

	- we're all behind some firewall somewhere, such as at a
	big company, or even a country

Ultimately, either the IW setting mostly works or it doesn't. If it 
works, we can keep using it or make it larger. If it doesn't, we should 
back off.

That's an invitation for an automatic system, IMO, not a schedule of 
updates in an RFC.

>        there is no previous history to suggest what the IW should be.

We always need a starting point. If a system *automatically* converges 
over a reasonable time, it doesn't matter much what that is.

If however that's a fixed value, it matters.

>        like a research project to me---not a viable alternative to

I'm thinking very coarsely; if it hurts most of the time when I do this, 
then stop doing it. It can be based on 80% - if 80% of the connections 
lose packets using IW=10, then backoff. If not, then don't. If 80% of 
the connections don't lose packets, then try a larger IW.

You don't need much more than a counter of whether packets are lost 
during an IW and how many connections you have.

Yes, this is an experiment -- BUT SO IS PICKING IW=10, or SETTING A 
SCHEDULE OF IW INCREASES OVER TIME. The point is that it's an adaptive 
solution that should end up doing the right thing without constant 
oversight.

...
> that's the right order for human perception to start.

People can notice things down to about 200ms. That does NOT mean they 
know 1 vs 1.2 seconds, though - and definitely not 400 vs 500ms. We 
looked at this issue in the mid 90s when looking at web push systems.

Joe

---

From: Nandita Dukkipati <[email-masked]>
Date: Thu, 18 Nov 2010 16:55:36 -0800

...
> People can notice things down to about 200ms. That does NOT mean they know
> 1 vs 1.2 seconds, though - and definitely not 400 vs 500ms. We looked at
> this issue in the mid 90s when looking at web push systems.

Fortunately, we don't have to speculate on the impact of delay on user
behaviors. There are more recent (2009, 2010) studies that show that even
small latency increases, such as 100ms, have measurable impacts (obviously
negative) on user experience and perception.

These studies describe experiments and go on to quantify the impact of
delays ranging from 50-2000ms on user behavior, number of clicks, and such
other metrics.
* http://velocityconf.com/velocity2009/public/schedule/detail/8523
* http://googleresearch.blogspot.com/2009/06/speed-matters.html

Other work along similar lines:
http://velocityconf.com/velocity2010/public/schedule/detail/13031
http://velocityconf.com/velocity2009/public/schedule/detail/7709
http://www.gomez.com/pdfs/wp_why_web_performance_matters.pdf

I think we can safely put to rest the question on whether IW10 benefits are
significant enough to warrant it to be standardized.

-Nandita

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 17:21:57 -0800

On 11/18/2010 4:55 PM, Nandita Dukkipati wrote:
...
> behaviors. There are more recent (2009, 2010) studies that show that
> even small latency increases, such as 100ms, have measurable impacts
> (obviously negative) on user experience and perception.
>
> delays ranging from 50-2000ms on user behavior, number of clicks, and
> such other metrics.

If you care that much about latency, keep connections open and reuse 
them. Reusing connections saves 1 RTT - the same amount you will save 
with IW=10. Further, your IW won't matter, so you'll save that too.

Second, you're saving 1 RTT. In most cases on reasonable nets, that's 
well under 100ms. The cases where that's not true are where you probably 
don't want to swamp the buffers (cell nets, bloated home DSL router 
buffers, bloated OS buffers).

Third, the reduction in latency is 1 RTT out of 3 *best case*. For very 
short transfers. For anything longer, it's deep in the noise.

Finally, I claim that this cannot possibly matter to the web. If it did, 
people wouldn't design web pages using bloated images and flash; they'd 
use simple HTML that would load and render faster.

If I have to wait 10-20 seconds after a page shows up just to see the 
first content, can you really argue that 100ms is going to make or break 
anything?

Joe

---

From: Nandita Dukkipati <[email-masked]>
Date: Thu, 18 Nov 2010 18:04:58 -0800

On Thu, Nov 18, 2010 at 5:21 PM, Joe Touch <[email-masked]> wrote:

> If you care that much about latency, keep connections open and reuse them.
> Reusing connections saves 1 RTT - the same amount you will save with IW=10.
> Further, your IW won't matter, so you'll save that too.
>
> Second, you're saving 1 RTT. In most cases on reasonable nets, that's well
> under 100ms. The cases where that's not true are where you probably don't
> want to swamp the buffers (cell nets, bloated home DSL router buffers,
> bloated OS buffers).
>
> people wouldn't design web pages using bloated images and flash; they'd use
> simple HTML that would load and render faster.
>
> If I have to wait 10-20 seconds after a page shows up just to see the first
> content, can you really argue that 100ms is going to make or break anything?
>
>
A couple of points:

1) It seems like you are assuming that IW10 saves 1 RTT over IW3? This is
incorrect. It can save up to 4 RTTs taking delayed ACKs into consideration,
as shown in this attached plot (reposting from Jerry Chu's earlier email in
IW10 thread).

2) The idea of keeping connections persistent is definitely an interesting
one, and requires solving its own set of issues such as scaling the usage of
memory (mainly on servers), as they will now have a large number of mostly
idle connections. Even then, it does not entirely solve the issue at hand if
one decays cwnd over time. Other practical consideration is of course NAT
boxes that auto-timeout connections.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 19 Nov 2010 03:39:38 +0100 (CET)

On Thu, 18 Nov 2010, Joe Touch wrote:

> Third, the reduction in latency is 1 RTT out of 3 *best case*. For very short 
> transfers. For anything longer, it's deep in the noise.
>

Which scientific evidence backs up this claim ?

> simple HTML that would load and render faster.

By the same logic, smoking is not harmful for people - because if it were, noone 
would be smoking.

> content, can you really argue that 100ms is going to make or break anything?

This is not about the sites that take 20 seconds to load. This is about the 
sites that try to squeeze every millisecond for the better user experience.

We should not put everything into the same basket.

thanks,
andrew

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 22:54:14 -0500

Two quick things ...

> them.

This takes a cooperative effort between two hosts and so is not always
possible from the perspective of one host.

> Finally, I claim that this cannot possibly matter to the web. If it
> did, people wouldn't design web pages using bloated images and flash;
> they'd use simple HTML that would load and render faster.

More likely its that people value the design of their web pages over the
performance benefits of stripping to bare bones.  That is a preference
on their part.  But, it doesn't mean they wouldn't also like to see
performance gains, as well.

OK, three things... IW10 can save more than 1RTT.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 18 Nov 2010 23:11:45 -0800

> Further, your IW won't matter, so you'll save that too.

Connection reuse is a great idea but there is many practical limits
unfortunately.

server scale:
Servers try to keep connections forever (at least Google) but idle
connections take up memory. Not to mention certain load-balancer setup
requires per-connection state

client scale:
Browsers close connections after couple minutes, b/c themselves are
opening too many.

NAT boxes:
Even servers and browsers both keep connections, NAT boxes drop them.
It's worse on large carrier-grade NAT.

Even servers do keep connections, they disable slow-start-after-idle,
so after the nice idle period, we can burst response with cwnd likely
larger than 10. It's a commonly known trick to operators in
datacenters/server farms.

> want to swamp the buffers (cell nets, bloated home DSL router buffers,
We save 1RTT for TCP handshake. We save more than 1RTT using IW10

> simple HTML that would load and render faster.
So why do CDN spend billions to deploy proxies/servers near the users
to cut the RTTs?

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Fri, 19 Nov 2010 13:48:46 +0200 (EET)

> Even servers and browsers both keep connections, NAT boxes drop them.

...This is again just one way to case "congestion collapse" if the 
bottleneck is prior to such a NAT box! Though in the most convincing
case it happens for the direction of small requests and not the replies.

-- 
 i.

---

From: Jerry Chu <[email-masked]>
Date: Sat, 27 Nov 2010 20:47:12 -0800

On Thu, Nov 18, 2010 at 12:40 PM, Joe Touch <[email-masked]> wrote:

> Ultimately, either the IW setting mostly works or it doesn't. If it works,
> we can keep using it or make it larger. If it doesn't, we should back off.
>

I'm not sure if it's that simple and I don't know the "IW setting" your
referred to above is on a per-host,
per-dest-IP, or per-whatever basis. From all your objections so far you seem
to be pushing the choice
of IW toward the direction that is reserved for TCP's dynamic CC algorithm,
to figure out an optimal
cwnd to use for a given path right from the start for a connection. But this
is not the job of IW, as
Mark Allman pointed out repeatedly.

So what qualifies as a good IW? It needs not be optimal because it'd require
too much work to get to
the "optimal" value from the beginning (cache from the past
history, offline database...), and it can not
be optimal because the cwnd (together with RTT) as a measure of the
fair-shared bottleneck b/w
includes a temporal component as well, that can only be captured dynamically
through TCP's CC
algorithm.

A good choice of IW IMO can't be too conservative nor too aggressive. It
needs to work well for
the vast majority of connections/paths on the Internet. For the few cases it
may not work well,
TCP will puzzle its way out (as Mark put it). In our opinion IW3 is too
conservative. IW10 will
be a good choice and its benefit far outweighs its penalty.

> We always need a starting point. If a system *automatically* converges over
> a reasonable time, it doesn't matter much what that is.
>
> then stop doing it. It can be based on 80% - if 80% of the connections lose
> packets using IW=10, then backoff. If not, then don't. If 80% of the
> connections don't lose packets, then try a larger IW.
>
> You don't need much more than a counter of whether packets are lost during
> an IW and how many connections you have.
>

I'm all for simplicity and if the very coarse scheme above actually serves
some purpose, i.e., works to
protect some cases that may suffer from a larger IW that I'm all for it. But
does it work?

It may work for some very limited cases where a router with very small
buffer space is near the sender
hosts. But I can't even think of such a case right from the top of my head
as the common case
of users behind slow 56Kbps links is on the receiving, not sending end,
where a different measure like
advertising a small init receive window as mentioned in our draft will work
well. Will the above coarse
measurement works for a web server in general? If our proposed IW has been
completely out of line
(e.g., IW64) it may have worked, but then we'd have found out long ago from
our large scale
experiments.

> solution that should end up doing the right thing without constant

I'd love for someone to show me a simple adaptive solution actually works so
we can incorporate it into
our proposal, get it approved by the community, and move on to tons of other
work. But I doubt it will
be that simple.

Jerry

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 28 Nov 2010 00:01:20 -0500

Why does IW need to be standardized?
How will any host/router on the internet change its behaviour based on the standardized value? That is, any host/router other than the TCP sender.

Has anyone tried an IW based on advertized receive window rather than on MSS?

--
Jakob Heitz.

---

From: Jerry Chu <[email-masked]>
Date: Sat, 27 Nov 2010 22:13:30 -0800

On Thu, Nov 18, 2010 at 8:31 AM, Joe Touch <[email-masked]> wrote:

> destinations. However, there are some sources that have problems ON THEIR
> END of the network, i.e., that impact most, if not all, of their
> connections.
>
> IW traffic is bursty, and bursts tend to synchronize. Further, IW is uses
> more than just for the connection

Not true. IW and RW in recent RFCs have been treated as two separate
variables. Moreover, we actually
propose to reset RW according to RFC3390 if there had been loss during IW10
as follows:
[draft-ietf-tcpm-initcwnd-00.txt]

   Furthermore, to limit any negative effect that a larger initial
   window may have on links with limited bandwidth or buffer space,
   implementations SHOULD fall back to RFC 3390 for the restart window
   (RW), if any packet loss is detected during either the initial
   window, or a restart window, when more than 4KB of data is sent.

The above is a great example of our design principle - anything that is
simple to implement to limit the
potential damage when retaining the bulk of the benefit, we'd love to adopt.

But any caching scheme from the past history that requires lots of memory,
and/or slowly growing
the IW, and/or high cache hit rate in order to work is just a non-starter
IMO. (We even attempted
at some point a "negative" cache scheme to work around the low cache hit
problem but decided
to give up.)

Jerry

start - it's also used for restart, e.g., after an idle period. This all
> means that increasing the IW without feedback could easily lead to problems
> that we cannot correct once deployed.
>
> ...
>
> How much does this help? How much does it reduce latency? Most of the info
> I've seen summarized suggests the benefit is in the 5-20% range. That's not
> even noticeable from a user POV.
>
> So, back to my original point, how about IW using a long-timescale AIMD? If
> we do that, we can avoid needing to monitor the problem en-masse, and the
> system will adapt over time and we won't be back here in another 10 years...
>
>
> Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 28 Nov 2010 06:22:14 -0800

Jerry,

In short:

a) the receive window is an indication of the receiver's ability to 
reorder out-of-order packets; it limits what is sent primarily to avoid 
useless transmissions that the receiver will drop, not as a congestion 
control mechanism

	yes, I find it odd that you are injecting control of the
	receive window to correct for IW issues in your draft

2) NO AMOUNT of in-connection adjustment of CWND has any effect on  the 
impact of a poor choice of IW

	consider that if this were not true, then you would
	not need to propose a new IW value; just as you do not
	want to wait for the connection to converge, I am
	concerned about the network impact until that convergence

I have made the second point repeatedly. It would be useful to know if 
your continued misdirection of the issue is because don't agree with it, 
or some other reason.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Nov 2010 21:49:17 -0800

On Sun, Nov 28, 2010 at 6:22 AM, Joe Touch <[email-masked]> wrote:

>
> Jerry,
>
> In short:
>
> a) the receive window is an indication of the receiver's ability to reorder
> out-of-order packets; it limits what is sent primarily to avoid useless
> transmissions that the receiver will drop, not as a congestion control
> mechanism
>

same argument holds for caching cwnd on a per-host or per prefix basis.

>
> I have made the second point repeatedly. It would be useful to know if your
> continued misdirection of the issue is because don't agree with it, or some
> other reason.
>
>
> Joe
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 06:00:31 -0800

On 11/29/2010 9:49 PM, Yuchung Cheng wrote:
...
>>     2) NO AMOUNT of in-connection adjustment of CWND has any effect on
>>       the impact of a poor choice of IW
>>

There were two points:

1) that no amount of post-start adjusting of CWND affects poor IW choice
	
	auto-adjusting IW corrects for poor IW choice,
	so that argument is valid

2) auto-adjusting IW can still make mistakes

	yes, but over time if it does it will correct.
	fixed IW will not

Fixing the IW over time isn't caching the CWND. The point is that CWND 
and IW are deliberately different. IW should be adjusted only if there 
were issues *during the IW period*, i.e., losses or retransmissions in 
that time.

Perhaps your point was that we shouldn't reuse CWND. Note, however, that 
CWND reuse - as already deployed, FWIW - adjusts over time (as per RFC2140).

The point I have made is that fixed IW does NOT adjust over time, and 
that it repeatedly can make mistakes.

So if you have a counterargument, please explain.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 09:33:02 -0500

> How will any host/router on the internet change its behaviour based on
> the standardized value? That is, any host/router other than the TCP
> sender. 
> 

Only a little bit ...

  Dan Liu, Mark Allman, Shudong Jin, Limin Wang.  Congestion Control
  Without a Startup Phase.  Protocols for Fast, Long Distance Networks
  (PFLDnet) Workshop, February 2007.
  http://www.icir.org/mallman/papers/jumpstart-pfldnet07.pdf

allman

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 07:02:15 -0800

On 11/27/2010 9:01 PM, Jakob Heitz wrote:
> the standardized value? That is, any host/router other than the TCP sender.

FWIW, advertised receive window is an indication of the receiver's 
ability to reorder and accommodate gaps in the transmission; it is not 
related to the congestion window, except that it's not useful to open 
CWND larger than the receive window.

I.e., these values serve completely different purposes. There is no 
rationale for setting CWND to the receive window (i.e., for thinking 
that RWIN should be both the upper and lower bound of CWND).

Joe

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 30 Nov 2010 17:00:20 -0000

But "knowledgeable" clients (which have some - hopefully better than the
server has - understanding of their side of the network can use the rwin
signalled in the 3WHS to cap any excessive IW the server might want to
use...

Ie. if a client "knows" that the first hope interface is a PPP / SLIP
interface (not PPPoE/PPPoA), it might signal a small RWin for the first
few segments received. Or grow signalled RWin by one MSS for each data
segment received and ACKed.

I think such a cooperative signalling would actually help a
dynamic/opportunistic IW adjustment scheme, as "known slow" hosts would
not impact the IW upward adjustments on the sender side (with small
initial RWin, the IW will also be small (smaller than what the sender
might use otherwise), thus it will not observe loss while doing the IW.
Which would not negatively affet the upwards adjustement for "usual"
clients.

Such a dynamic IW adjustment scheme could be a system-wide variable, but
I believe it was Mark that pointed out, that this will only shift the
selection of an initial value one step further out (the kernels boot
init value).

Also, the exact dyamics aren't all that clear yet.

Ie. increase IW by 1 seg, if the balance between total sessions
(previous 24hrs) vs. sessions with loss during IW (previous 24hrs) is
towards the no-loss-during-IW? Reduce by 1 seg otherwise?

Should the metric be absolute time based, or relative to the number of
session setups? (systems with mostly persistent connections won't adjust
IW (Restart window) all that quick, but may not care either...

For the kernel boot time initial value, I would vote for IW10 :)

Richard Scheffenegger

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]] 
> Sent: Dienstag, 30. November 2010 16:02
> To: Jakob Heitz
> Cc: [email-masked]
> Subject: Re: [tcpm] alternate IW proposal
> 
> 
> 
> > Has anyone tried an IW based on advertized receive window 
> rather than 
> > on MSS?
> 
> FWIW, advertised receive window is an indication of the 
> receiver's ability to reorder and accommodate gaps in the 
> transmission; it is not related to the congestion window, 
> except that it's not useful to open CWND larger than the 
> receive window.
> 
> I.e., these values serve completely different purposes. There 
> is no rationale for setting CWND to the receive window (i.e., 
> for thinking that RWIN should be both the upper and lower 
> bound of CWND).
> 
> Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 09:04:02 -0800

Hi, Richard,

FWIW, there have been several attempts to have TCP "know" what its 
first-hop link is, and tune to that. The biggest problem is that this is 
most often incorrect - e.g., at home, my first hop is a 54 Mbps WiFi 
link, but I'm limited by my DSL modem. The same is true for anyone on a 
satellite link (e.g., DirectPC), or anyone using a cellphone as a 
'tethered' modem (via Bluetooth or WiFi).

The RWIN is supposed to indicate the receiver's ability to:
	a) buffer
	b) reorder (related to (a)

RWIN has no relation to CWND, nor should it. CWND is a path property; 
RWIN is an endpoint property.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 30 Nov 2010 13:09:34 -0500

> Behalf Of Joe Touch
> Sent: Tuesday, November 30, 2010 6:01 AM
>

More often than not, auto-adjustment makes things worse.

If the feedback loop gets relevant feedback, the time constants
of the loop are correct and the loop is linear are all
conditions for making it better.
If any of those are wrong, auto-adjustment will make it worse.

Simple example: without limits, it could go to 0 or below.
There are many more not-so-simple examples.

--
Jakob Heitz.

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 10:32:15 -0800

> If any of those are wrong, auto-adjustment will make it worse.

The time constants we're talking about here are faster than the IETF, 
but slower than TCP CC, i.e., O(weeks to months).

The feedback is based on whether you have loss during the IW.

Although your argument about these is true (and somewhat of a platitude 
about control loops), we've done this many times before in TCP and 
arguably got it reasonably right.

> There are many more not-so-simple examples.

It would never go below 3, because 3 is what you have now.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 13:52:36 -0500

> The time constants we're talking about here are faster than the
> IETF, but slower than TCP CC, i.e., O(weeks to months).

O(weeks to months to years to decades)

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 11:08:33 -0800

> O(weeks to months to years to decades)

FWIW, I was defining timescale as the period over which the value is 
re-evaluated and could change, not the period over which it ever does 
change.

But yes, the basic alg we have been discussing off-line has been:

	backoff if loss during IW is dominant
		check O(weeks/months)

	increase the IW by one
		do this every year or two, maybe

Note - the 'increase' part is optional. It is included to incorporate 
Mark's suggestion that the IW should be ramped up slowly over time, 
rather than revisited once a decade in larger jumps.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 14:20:14 -0500

> re-evaluated and could change,

FWIW, I was increasing the timescale to realistic IETF granularities.

allman