Subject: [tcpm] Increasing the Initial Window - Notes

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 10 Nov 2010 16:29:02 +0100

Cause tomorrow Jerry will talk about the IW10 topic my 50 cents in advance.
First some skeptical notes, secondly a alternative approach to increase the
IW.

I am skeptical about this draft because IMHO renounce the conservative principle
towards a aggressive and HTML centric approach (the HTML example is a bad
allegation, but for FTP and friends slow start is fast enough ;-). My costumer,
and probably other networks too, operates multi-hop radio networks with a
bandwidth of less then 100kB/s.

An increased IW of 10 define 10*~MSS byte as the smallest atomic in-flight
value. New data are always pushed into the network in this quantum - a way to
much for the standard TCP CC mechanism to tune into a steady state for network
with a low BDP.

In the discussion I miss the impact of the proposed modification regarding
networks that operates already at a functioning boundary. Are we at a time
where we loss these types of networks? Where are the simulation results that
show that the impact at this kind of networks is acceptable?

Alternative idea: can we shift the idea towards a more robust basis. What about
a increased IW of 10 if we know that the path is capable of an increased IW?

Pseudo Algorithm for a new connection:

	a) No cached CC data available

		IW = 4 (standard mechanism a la RFC 5681, 3390)
		CC mechanism as usual but save CC parameters - especially cwnd - per route

	b) CC data per route is available

		IF (cached_cwnd >= 10)
			IW = 10
		else
			IW = 4

After n minutes, previously saved CC data is marked as invalid, similar to the
existing "back to slow start after n minutes idle" mechanism.

Idea: networks with a low BDP will signal by packet drop or ECN marking that
congestion occur and the BDP is saturated. Why not use actual network
information to setup the IW but preserve the conservative nature of TCP?

Maybe this is an approach for further discussions, I have a curious feeling
that in 2 years the same discussion will start with a IW of 23 ... So maybe we
should drop the static IW approach completely and define IW as
cached_steady_state_cwnd / 2 for cached routes and IW = 4 for startup all other
cases.

Best regards, Hagen

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Wed, 10 Nov 2010 08:04:48 -0800

I think the stuff which you are suggesting below has already been
mentioned in earlier conversations (mailing lists, online during the
earlier presentations). Things like "tomorrow someone will come with IW
= 23 etc.,", using the cached information also has been mentioned. The
IW=10 is a fundamental change to TCP and hence the suggestion of moving
this work item to ICCRG also was mentioned before. Someone also pointed
out "will this work in Africa"?  Also using a TCP option to agree upon
the IW was mentioned (which is probably less useful since you have to
wait for the 3 way HS to complete and also it is not about end hosts
supporting higher IW, it is about the network property as you rightly
mention below)

Actually it would be nicer if the authors mention these things in the
document and respond with what their stance is on each of the questions.
This way anyone who reads the document are well informed of the current
stance. 

$0.02,
-Anantha

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Hagen Paul Pfeifer
Sent: Wednesday, November 10, 2010 7:29 AM
To: Jerry Chu; tmrg; Matt Mathis; tcpm; Lars Eggert
Subject: [tcpm] Increasing the Initial Window - Notes

Cause tomorrow Jerry will talk about the IW10 topic my 50 cents in
advance.
First some skeptical notes, secondly a alternative approach to increase
the
IW.

I am skeptical about this draft because IMHO renounce the conservative
principle
towards a aggressive and HTML centric approach (the HTML example is a
bad
allegation, but for FTP and friends slow start is fast enough ;-). My
costumer,
and probably other networks too, operates multi-hop radio networks with
a
bandwidth of less then 100kB/s.

An increased IW of 10 define 10*~MSS byte as the smallest atomic
in-flight
value. New data are always pushed into the network in this quantum - a
way to
much for the standard TCP CC mechanism to tune into a steady state for
network
with a low BDP.

In the discussion I miss the impact of the proposed modification
regarding
networks that operates already at a functioning boundary. Are we at a
time
where we loss these types of networks? Where are the simulation results
that
show that the impact at this kind of networks is acceptable?

Alternative idea: can we shift the idea towards a more robust basis.
What about
a increased IW of 10 if we know that the path is capable of an increased
IW?

		IW = 4 (standard mechanism a la RFC 5681, 3390)
		CC mechanism as usual but save CC parameters -
especially cwnd - per route

		IF (cached_cwnd >= 10)
			IW = 10
		else
			IW = 4

After n minutes, previously saved CC data is marked as invalid, similar
to the
existing "back to slow start after n minutes idle" mechanism.

Idea: networks with a low BDP will signal by packet drop or ECN marking
that
congestion occur and the BDP is saturated. Why not use actual network
information to setup the IW but preserve the conservative nature of TCP?

Maybe this is an approach for further discussions, I have a curious
feeling
that in 2 years the same discussion will start with a IW of 23 ... So
maybe we
should drop the static IW approach completely and define IW as
cached_steady_state_cwnd / 2 for cached routes and IW = 4 for startup
all other
cases.

Best regards, Hagen
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 10 Nov 2010 17:29:24 +0100

>supporting higher IW, it is about the network property as you rightly

I am aware about "IW >= 23" and the Africa discussion. For me the later is a
strong reason for a refusal of the draft. I am not aware of the discussion of
the _cached cwnd_ approach.

Hagen

---

From: rick jones <[email-masked]>
Date: Wed, 10 Nov 2010 08:33:30 -0800

On Nov 10, 2010, at 7:29 AM, Hagen Paul Pfeifer wrote:

> allegation, but for FTP and friends slow start is fast enough ;-).

Sure, if all one is doing is pushing <=3MSS and >100MSS files around :)

> My costumer,

Do your customers run systems with support for TSO and/or LRO enabled,  
or do they disable that today?  TSO can/may/will cause 10*MSS or  
larger blobs of data to be injected into the network at one time no?   
And LRO can/may induce stretch ACK behaviour not unlike the ACK  
avoidance heuristics of stacks such as those in HP-UX or Solaris. What  
I've seen so far has suggested the goal is to get IW10 "blessed" not  
to make IW10 mandatory.

rick jones
http://homepage.mac.com/perfgeek

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 10 Nov 2010 18:00:20 +0100

* rick jones | 2010-11-10 08:33:30 [-0800]:

>Do your customers run systems with support for TSO and/or LRO
>enabled, or do they disable that today?  TSO can/may/will cause
>10*MSS or larger blobs of data to be injected into the network at one
>time no? 

No, TSO does not change the packet behavior on the wire. At least no CC
related mechanisms. Packet spacing and stuff like that ignored here.

>And LRO can/may induce stretch ACK behaviour not unlike the
>ACK avoidance heuristics of stacks such as those in HP-UX or Solaris.
>What I've seen so far has suggested the goal is to get IW10 "blessed"
>not to make IW10 mandatory.

If the IETF "blessed" IW10 every vendor will enable it ... 

Hagen

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Nov 2010 09:04:43 -0800

On 11/10/2010 8:29 AM, Hagen Paul Pfeifer wrote:
> I am aware about "IW>= 23" and the Africa discussion. For me the later is a

See RFC2140.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Nov 2010 09:06:03 -0800

FWIW, I'd like to add another point to the discussion.

There has been recent concern with "buffer bloat", where home OS's, home 
routers, etc. all have either poorly optimized or incomplete 
implementations of fair queuing and socket buffering.

The basic problem can be seen as follows:
	- run a continuous ping anywhere
	- start a large upload

The latencies go from 10-100 milliseconds up t 0.5-5 seconds. This 
happens because the outbound buffers get stuffed with data.

Although a single connection can cause this to happen (it optimizes for 
rate, not throughput), we know how to handle that - use RED, fair 
queuing, etc.

I'm wondering if increasing the IW can defeat these corrective measures, 
and recreate the problem in a way that has no effective solution.

Has that been considered?

Joe

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 10 Nov 2010 18:20:08 +0100

>See RFC2140.

Thank you Joe, this was the source of the idea. I thought about the Linux
destination cache and to reuse RTAX_CWND from this metric.

Hagen

---

From: rick jones <[email-masked]>
Date: Wed, 10 Nov 2010 09:20:23 -0800

> related mechanisms. Packet spacing and stuff like that ignored here.

Earlier you wrote "New data are always pushed into the network in this  
quantum - a way to
much for the standard TCP CC mechanism to tune into a steady state for  
network
with a low BDP."  I read that as implying a concern about how much  
data any one connection injects into the network at any moment in time.

TSO does not change what any one packet looks like on the wire, but it  
certainly does change how many of them rush the wire at one time, yes  
bounded by cwnd - something those effectively infinite data sources  
like FTP will be pushing as far as it will go.  And depending on  
whether the stack receiving the LRO-stretched ACKs is counting bytes  
ACKed or number of ACKs the LRO can induce similar bursty behaviour.

> If the IETF "blessed" IW10 every vendor will enable it ...

And I presume, make it configurable.  It is already configurable on a  
per-route basis in the Linux stack.

rick jones

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Nov 2010 09:28:08 -0800

> destination cache and to reuse RTAX_CWND from this metric.

Yup - FWIW, I think the Linux cache is basically an implementation of 
what 2140 proposed.

Joe

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 10 Nov 2010 18:40:58 +0100

* rick jones | 2010-11-10 09:20:23 [-0800]:

>Earlier you wrote "New data are always pushed into the network in
>this quantum - a way to much for the standard TCP CC mechanism to tune into a
>steady state for network with a low BDP."  I read that as implying a concern
>about how much data any one connection injects into the network at any moment
>in time.

To clarify this: I especially focus on the start phase of the connection. If
the CC algorithm detects during slow start, congestion avoidance that the
available BDP is larger then the IW it is fine. But if the IW is 10, how
should _new connections_ react, they always will initially transmit 10 packets
- which can be too much - the problem is the initial quantum.

>per-route basis in the Linux stack.

Yes, and you know the statement of davem and other guys if the discussion goes
in this direction. Davem is aware of the impact of this knob and I am not sure
if everybody is happy about it. I remember a discussion with davem about a
HOT-TO-AGGRESSIVELY-TUNE-YOUR-NETWORK-STACK-HOWTO.xml. If these option are
available they can be misused. I am happy that the per route knobs are a
little bit tricky usable. ;-)

Hagen

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 10 Nov 2010 11:15:19 -0800

We have done some prior work to address the Africa instance or in general
IW10 and IW3 fairness issues. But because we didn't present in the last tcpm
but instead iccrg so even we cc'd tcpm of the slides, people may not be
aware:

We did both live and testbed experiments for the Africa case. The testbed
experiment is configured to simulate the links of a particular university in
Africa that an IETF member administrates.
http://code.google.com/speed/protocols/Increasing_TCPs_Initial_Window_IETF78.pdf

Unfortunately, we didn't find much fairness issues. Fred raised concerns
using IW10 on residential low-bandwidth network with many flows and/or
background in the meeting. We went back and worked on this experiment and
Jerry is going to present it in tcpm soon.

Independently Michael Scharf also studied IW3 and IW10 issues in his PhD
research and reached similar conclusion. We presented his slides in the last
ICCRG meeting but due to some copyright issue, please contact him privately
for a copy.

We plan to include all the feedbacks and our responses/experimental results
in the next draft revision, since we want to minimize the revisions and also
collect feedback from this meeting as well.

Lastly, we do not discuss caching IWs b/c essentially that's a duplication
of RFC 2140 and there is no point to write another draft for that.

Thanks,

Yuchung

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 10 Nov 2010 19:29:45 -0000

Hagen,

Regarding cached cwnd: that would assume that you already have some past
expirience with the host (a), but also that your previous interaction
was transporting enough (bulk) data to arrive at some steady state(b)...

>From my understanding, for the suggested use case (minimize delivery
latency of "small" flows, sent out of big server farms), that might not
be possible to achive. Or am I wrong?

For your use case (networks with small BDP), shouldn't the receiver be
able to signal an accordingly small window (in comparison with the
window which is likely being signalled when the user is connected to a
fat pipe)?

(Is there are correlation between configured tcp receive window size,
and maximum bulk transfer speeds within the public internet? Or are
default parameters used without correlation to the actual possible bulk
transfer speeds?)

I have mentioned this earlier (but got swayed a bit) that some cheap
home gateways may have very small buffer sizes, which could be
overwhelmed as well if initcwnd=10 is used. OTOH, your average server
does not reside beind a home gateway box (and if, you may be inclined to
upgrade that box for more reliable service anyway)...

As you say, bursting 10 MSS segments at wirespeed onto a bottleneck
might not be all that bright an idea, especially if the bottleneck is
close to the sender, and runs at a significantly lower speed (ie. 1Gbps
internal, 54Mbps phys public, rate limited to 1Mbps)

Just a thought:

Perhaps pacing or chirping (inverse chirping, to be specific) can be
used to address the issue of burst overloading a small first hop queue;
for internet RTTs, the pacing interval would still be minuscle, but
would allow the queues to drain somewhat...

   __            __ __ __ __  __   __    __     __      __       __
__|S |_/~/_    _|D |D |D |D ||D |_|D |__|D |___|D |____|D |_____|D
|_/~/_   _/~/_   ....
           |SA|
|A|     |A|                    

Obviously, that would need somewhat more fine-grained tcp timers (ie.
1ms, with the pacing/chirping interval increasing by some factor for
each additional segment sent, until a new, higher initcwnd is reached,
or the ACK from the receiver starts the ACK clock...)

That would be similar to the RTO backoff, just with a couple magnitude
smaller timescales, and the sending of new data instead of the resending
of the same data... Also, we wouldn't necessarily have a discussion for
any arbitrary initcwnd soon again ;)

Wasn't there a study to be conducted on the behavior of home gateways
(queue depths, forwarding rates, behavior on unknown tcp options etc)?

I think for Africa / South America, if initcwnd is used on server side
(and the bottleneck bandwidht likely to be approched in multiple, small
steps) there was some research presented since maastricht, not?

Richard Scheffenegger

> From: Hagen Paul Pfeifer [mailto:[email-masked]] 
> Sent: Mittwoch, 10. November 2010 17:29
> To: Anantha Ramaiah (ananth)
> Cc: tmrg; Matt Mathis; tcpm
> Subject: Re: [tcpm] Increasing the Initial Window - Notes
> 
> I am aware about "IW >= 23" and the Africa discussion. For me 
> the later is a strong reason for a refusal of the draft. I am 
> not aware of the discussion of the _cached cwnd_ approach.
> 
> Hagen
>

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 10 Nov 2010 12:04:48 -0800

Hagen,

Please see my earlier response to the caching suggestions and
simulation/experiment questions.

Our draft suggests TCP MAY set the initial window of 10. The host can pick
their best initial cwnd based on their network properties if appropriate or
implement  RFC2140.

If raising IW upper bound is too aggressive and should be rejected as a MAY
clause, I argue IW3 is also too aggressive for some networks too. But RFC
3390 suggests raising IW3 primarily because of web objects size distribution
at that time. Further, we do not propose changing the response function of
TCP, which is the most critical part of maintaining the network stable.

Let's do a reality check: today browsers open tens of flows simultaneously
to work around small TCP IW (http://www.browserscope.org/  the "network
tab). A study from AT&T and U of Mich also show 15% TCP flows observed in
AT&T network already go over IW3. We want to bring this issue to the IETF
community/experts to stop this downward spiral of sneaky unfair latency
"tune-ups".

Cheers,

Yuchung

---

From: Jerry Chu <[email-masked]>
Date: Wed, 10 Nov 2010 14:38:31 -0800

> allegation, but for FTP and friends slow start is fast enough ;-). My costumer,

TCP's slow start being too slow for not just HTTP traffic but also
median size data
transfers has been a well-known issue in the community and active research on
faster startup schemes has been ongoing for years. But you seem to be against
the rest of us and saying this is not a problem at all?

> bandwidth of less then 100kB/s.

Folks from Nokia have been actively investigating the effect of IW10 in a radio
network so I'll defer to their findings. I'd only make two points from
our testbed
study on slow (96Kbps) links:

1. There is no denying that IW10 tends to cause higher packet drop rate during
congestion. (Don't get too turned off by some high packet drop rate in my slides
because some of the testcases were designed to be extreme in order to get to
the worst case, and it looks like the worst case we've found for IW10, it was
also terrible for IW3 (e.g., UCT > 50secs).

2. Regardless of IW10's higher packet drop rate, it's flow completion time has
been consistently better, or at least not worse than IW3's. Some explanation
of it was given in my slides.

3. In a multi-hop multi-congestion-point mobile network the high packet drop
rate may cause more damage due to the waste of bandwidth on the path
leading to the final congestion drop point. Our testbed has only one congestion
point so doesn't exhibit this problem. I think folks from Nokia are concerned
with this problem and I'll defer this to them.

> much for the standard TCP CC mechanism to tune into a steady state for network

Do you have any real numbers to backup your claim here? Numbers from our live
tests at Google's frontends seem to contradict to your statement here.

> show that the impact at this kind of networks is acceptable?

See past email threads and slides.

> cached_steady_state_cwnd / 2 for cached routes and IW = 4 for startup all other

You are late to the party :). All the above and more has been
presented/discussed/
debated in the past already. To make it short, we first tried tweaking Linux dst
cache like you described above but found it not very effective (e.g., cache hit
rate was low...) Then we tried an off-line history database approach
but found it
too complicated. Finally we decided a moderate increase of IW is the right, and
timely solution (until someone else like you could find the optimal or
completely
scalable solution to solve the global warming problem :). Following the KISS
principle, we'd open to any amendment based on info collected during 3WHS
but will be prejudiced against any other states that need to collected and kept
from the past.

I see your name on the attendee list. If you like to chat more you can find me
at CONEX this morning. I'll probably take a front row seat there.

Best,

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 10 Nov 2010 14:47:22 -0800

Anantha,

Thanks for point out the following to Hagen. Yes I must apologize. The
draft has been
overdue for a major revision for a while. We first tried to do it
after summer IETF but
were all busy with other work (and some involves new "Make the Web Faster"
proposal to IETF that we will present in the near future). We will try
to get a major
revision done and published before year end. In the meantime one has
to go back to
the list and slides from the past.

Thanks,

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 10 Nov 2010 15:13:41 -0800

+ yaogong

Joe,

You brought up a good question. We don't have any direct data on the scenario
you described. Also we've used FIFO tail drop so far hence haven't tried any
AQM algorithm on the router. Otherwise our testbed data showed under high
load and some extreme case (e.g., a constantly 5-simultaneous open bursts)
the average qlen of IW10 can get much longer than that of IW3, although under
those loaded conditions IW3 is already bad enough.

To really answer your question, I'd suggest you to go to
http://research.csc.ncsu.edu/netsrv/?q=content/iw10

and tell us how to modify the test setup/parameters/programs to emulate the
exact condition you have in mind. I'm sure our collaboration Yaogong Wang from
NCSU will be happy to run some results for you.

Thanks,

Jerry

On Wed, Nov 10, 2010 at 9:06 AM, Joe Touch <[email-masked]> wrote:
> routers, etc. all have either poorly optimized or incomplete implementations
> of fair queuing and socket buffering.
>
> The latencies go from 10-100 milliseconds up t 0.5-5 seconds. This happens
> because the outbound buffers get stuffed with data.
>
> rate, not throughput), we know how to handle that - use RED, fair queuing,
> etc.
>
> I'm wondering if increasing the IW can defeat these corrective measures, and
> recreate the problem in a way that has no effective solution.
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Nov 2010 15:23:41 -0800

> NCSU will be happy to run some results for you.

Your collaborators need to learn more about the situation of buffer 
bloat - it might be useful to contact Jim Gettys on this, and see what 
he can point you to that would help you understand how you need to 
configure the testbed to consider this issue. I'm following it, but only 
peripherally.

Joe

>
> Thanks,
>
> Jerry
>
> On Wed, Nov 10, 2010 at 9:06 AM, Joe Touch<[email-masked]>  wrote:

---

From: Jerry Chu <[email-masked]>
Date: Wed, 10 Nov 2010 15:48:45 -0800

> - which can be too much - the problem is the initial quantum.

I wonder what factors will argue that the initial quantum can't be changed?
Someone has argued in the past IW10 effectively "disable CC algorithm".
But haven't we already done that years ago when moving up IW from 1 to 3 if
one insisted then that the initial quantum must be 1?

The bottom line is, the right initial quantum is likely to be a
function of b/w for
the most part. So the concern has been on the slow links where the b/w is low.
But for all the practical purpose the slow links to be considered today are
56Kbps, which is 6 times larger than 9.6Kbps, the de-facto slow links back
in the 90' when we increased IW from 1 to 3. Why are we so concerned about
IW10, which is << than 6*3=18?

Our testbed data confirmed that even on 56Kbps link IW10's saving on the
roundtrips seems to more than compensate for the damage caused by higher
retransmission rate and longer queuing delay so its flow completion time
seems always better, or at least no worse than IW3's

Jerry

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Thu, 11 Nov 2010 01:26:50 +0100

>IW10, which is << than 6*3=18?

I think you miss one crucial fact: the last-mile link may be 9.6Kbps but the
overall network links where definitely broader then 9.6Kbps. In the mentioned
scenario _all_ links are 100kB/s.

You are completly right: the ideal IW is a function of the bandwidth of all
links. But we are blind regarding the bandwidth of all links in advance.
Simple overshoting the available bandwith and disable the CC algorithm is not
my favorite ... ;)

HGN

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Thu, 11 Nov 2010 02:24:55 +0100

>community/experts to stop this downward spiral of sneaky unfair latency

Let's do a reality check: we talk about TCP not HTTP. ;-) To clearify it: the
draft address TCP, including FTP and all protocol built on top of TCP. I know
google paradigm is "everything over HTTP" ;) but customers of mine do not use
HTTP at all. TCP based file transfer and UDP based protocols are the lion
share.

Hagen

---

From: Jerry Chu <[email-masked]>
Date: Wed, 10 Nov 2010 18:05:25 -0800

> HTTP at all. TCP based file transfer and UDP based protocols are the lion

Let's assume IW's effect diminishes as flows run longer, so you're probably
not concerned about your flows running IW10 but more concerned about
any collateral damage IW10 might bring to your ftp flows. Do you realize
browsers have silently increased the # of simultaneous connections opened
from 2 to 6 in the past few years, partly to get an effective boost to
IW without
any plan for standardization. Have you been concerned about the equivalent
effect to your customers in the past few years? Browsers may continue to
increase # of simultaneous opens if it helps performance. If they do, are you
going to do something about it?

Jerry

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Thu, 11 Nov 2010 04:20:23 +0200 (EET)

On Wed, 10 Nov 2010, Joe Touch wrote:

> Although a single connection can cause this to happen (it optimizes for rate,
> not throughput), we know how to handle that - use RED, fair queuing, etc.
> 

Please stay tuned, I intend to try to answer this question in my upcoming 
presentation. ...Or if curious enough, you can find my slides already 
among the tcpm material.

-- 
 i.

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Thu, 11 Nov 2010 03:20:56 +0100

* Jerry Chu | 2010-11-10 18:05:25 [-0800]:

>not concerned about your flows running IW10 but more concerned about any
>collateral damage IW10 might bring to your ftp flows. Do you realize browsers
>have silently increased the # of simultaneous connections opened from 2 to 6
>in the past few years, partly to get an effective boost to IW without any
>plan for standardization. Have you been concerned about the equivalent effect
>to your customers in the past few years? Browsers may continue to increase #
>of simultaneous opens if it helps performance. If they do, are you going to
>do something about it?

If the number of browser flow is the problem then do you think the draft which
address TCP fundamentally is the right answer? Not sure, you are right that I
worry about collateral damage for some kind of networks.

IMHO we should act careful with increasing the IW, I am skeptical and all
material that I saw does not appease me. I started to build a multi-hop ns-3
test setup, no dumpbell scenario, a flat multi-hop test setup but it stocked
because of lack of time.

Hagen

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 11 Nov 2010 09:42:51 +0100

> ICCRG meeting but due to some copyright issue, please contact him privately

Just two small notes on this:

First, my studies mainly focused on typical broadband Internet access links (several Mbit/s). I did not comprehensively study networks with rather low bandwidth.

Second, unfortunately, my PhD thesis is still not published. In case that there is interest, I will offer a presentation about my findings once I am able to do so (e. g., in ICCRG or TCPM).

Thanks

Michael

---

From: Michael Welzl <[email-masked]>
Date: Thu, 11 Nov 2010 11:28:56 +0100

This just gave me an idea:

> HTTP at all. TCP based file transfer and UDP based protocols are the  

I've seen this argumentation again and again, going back and forth. So  
it seems that we have a lot of data, and many proponents saying that  
it might be worth trying to use IW=10 for the web... but a lot of  
people who argue against the IW=10 keep pointing out that HTTP isn't  
everything.

Why not make this something that could be enabled on a per-socket  
basis? I mean, I can decide whether my socket does Nagle or not, why  
not let the programmer decide where in the range from 4 to 10 the IW  
should be, with a socket option or something?

Cheers,
Michael

---

From: Jerry Chu <[email-masked]>
Date: Thu, 11 Nov 2010 02:36:46 -0800

On Thu, Nov 11, 2010 at 2:28 AM, Michael Welzl <[email-masked]> wrote:
> I've seen this argumentation again and again, going back and forth. So it seems that we have a lot of data, and many proponents saying that it might be worth trying to use IW=10 for the web... but a lot of people who argue against the IW=10 keep pointing out that HTTP isn't everything.
>
> Why not make this something that could be enabled on a per-socket basis? I mean, I can decide whether my socket does Nagle or not, why not let the programmer decide where in the range from 4 to 10 the IW should be, with a socket option or something?

I certainly won't object to this, if that's all it takes for
standardization. Unfortunately the
point for the non-convinced is they don't want ANY flows to use IW10
for fear of hurting
the performance of their flows.

Jerry

---

From: Michael Welzl <[email-masked]>
Date: Thu, 11 Nov 2010 11:39:31 +0100

On Nov 11, 2010, at 11:36 AM, Jerry Chu wrote:

>> I've seen this argumentation again and again, going back and forth.  
>> So it seems that we have a lot of data, and many proponents saying  
>> that it might be worth trying to use IW=10 for the web... but a lot  
>> of people who argue against the IW=10 keep pointing out that HTTP  
>> isn't everything.
>>
>> basis? I mean, I can decide whether my socket does Nagle or not,  
>> why not let the programmer decide where in the range from 4 to 10  
>> the IW should be, with a socket option or something?
>

... but here, your "browsers open tens of flows" argument totally  
holds. How is a web flow with IW10 worse than a browser opening tens  
of flows? If people only use it for the web, like Google has been  
successfully doing, this seems to be quite safe, based on experience.

I'm curious what the opposition says to this  :)

Cheers,
Michael

---

From: Lars Eggert <[email-masked]>
Date: Thu, 11 Nov 2010 18:44:48 +0800

Hi,

we currently seem to be a bit stuck. The Google folks have gathered a lot of data, but at the moment it's still not quite clear - to me at least - what data-based argument would convince the WG to either permit IW10 and under what considerations, or which argument would convince us to not do it.

I talked to a few folks after the meeting and made this suggestion:

* Google makes the raw data available. Yoshifumi requested this at the mike, and Jerry said he'd see if this was possible.

* We plan a TCPM interim in a few months, with the only topic being IW10.

* Folks (other than Google) who like IW10 crunch down the raw Google data in an attempt to provide a convincing argument to allow it, and under which conditions. (Obviously, folks can also generate their own data, if they like.)

* Folks sceptical about IW10 crunch down the Google data in an attempt to find problematic cases that'd make the WG lean against IW10. (Obviously, folks can also generate their own data, if they like.)

* We hold the interim with these folks presenting, and see what shakes out.

The idea here is that we really need a more diverse set of folks playing with IW10. Thoughts?

Lars

---

From: Lars Eggert <[email-masked]>
Date: Thu, 11 Nov 2010 18:45:49 +0800

> Why not make this something that could be enabled on a per-socket basis? I mean, I can decide whether my socket does Nagle or not, why not let the programmer decide where in the range from 4 to 10…

It will be REALLY hard to explain to folks why they should limit themselves to 10, if we make this a generic socket option.

Lars

---

From: Marco Mellia <[email-masked]>
Date: Thu, 11 Nov 2010 11:49:50 +0100

> I'm curious what the opposition says to this  :)

If today a brower opens 10 flows with a server using IW=1 you gets 10 
packets equivalent IW.
Increasing the IW to 10 leads to an equivalent IW of 100 packets. At 
that point, you DSL modem will definitively start dropping most of them...

-- 

Ciao,                    /\/\/\rco

+-----------------------------------+
| Marco Mellia - Assistant Professor|
| Skypeid: mgmellia                 |
| Tel: +39-011-564-4173             |
| Cel: +39-340-9674888              |   /"\  .. . . . . . . . . . . . .
| Politecnico di Torino             |   \ /  . ASCII Ribbon Campaign  .
| Corso Duca degli Abruzzi 24       |    X   .- NO HTML/RTF in e-mail .
| Torino - 10129 - Italy            |   / \  .- NO Word docs in e-mail.
| http://www.telematica.polito.it   |        .. . . . . . . . . . . . .
+-----------------------------------+
The box said "Requires Windows 95 or Better." So I installed Linux.

---

From: Michael Welzl <[email-masked]>
Date: Thu, 11 Nov 2010 11:53:33 +0100

On Nov 11, 2010, at 11:45 AM, Lars Eggert wrote:

> It will be REALLY hard to explain to folks why they should limit  
> themselves to 10, if we make this a generic socket option.

Provide 4-10 available in the option, not more, based on Google's data?

---

From: Michael Welzl <[email-masked]>
Date: Thu, 11 Nov 2010 11:58:26 +0100

On Nov 11, 2010, at 11:49 AM, Marco Mellia wrote:

>> holds. How is a web flow with IW10 worse than a browser opening  
>> tens of flows? If people only use it for the web, like Google has  
>> been successfully doing, this seems to be quite safe, based on  
>> experience.
>>
> If today a brower opens 10 flows with a server using IW=1 you gets  
> 10 packets equivalent IW.

... which is a good reason to open fewer connections. So that's a  
matter of bringing of bringing out an update of a web client with a  
different behavior for the case that a web server uses a larger IW.  
But how does the web browser know what the web server is doing? I  
guess they could exchange this information with HTTP. Hm. Are we  
getting somewhere here, or is this a dead end?

Cheers,
Michael

---

From: Jerry Chu <[email-masked]>
Date: Thu, 11 Nov 2010 02:59:22 -0800

On Thu, Nov 11, 2010 at 2:49 AM, Marco Mellia <[email-masked]> wrote:
>> ... but here, your "browsers open tens of flows" argument totally holds. How is a web flow with IW10 worse than a browser opening tens of flows? If people only use it for the web, like Google has been successfully doing, this seems to be quite safe, based on experience.
>>
> If today a brower opens 10 flows with a server using IW=1 you gets 10 packets equivalent IW.
> Increasing the IW to 10 leads to an equivalent IW of 100 packets. At that point, you DSL modem will definitively start dropping most of them...

But if we don't start thinking about how to scale these TCP
parameters, tomorrow (ok, maybe
in a couple of years) when browsers decide to open 20 or 30 flows
because they are frustrated
with low TCP parameters like IW, it will effectively be equivalent to
increasing IW to 10 today.
So do we want to do nothing and leave it all to apps, or try to take
some control of some key
TCP parameters back into TCP itself?

Jerry

---

From: Michael Welzl <[email-masked]>
Date: Thu, 11 Nov 2010 12:00:45 +0100

> playing with IW10. Thoughts?

Isn't this what Experimental RFCs encourage?

Cheers,
Michael

---

From: Lars Eggert <[email-masked]>
Date: Thu, 11 Nov 2010 19:07:09 +0800

> Isn't this what Experimental RFCs encourage?

We don't really need an RFC to specify how to set IW to 10 - that should be pretty straightforward to implementors.

We also don't need an RFC for lab experiments with IW10.

My impression is that Google et al. would like to experiment with IW10 *on the Internet*, and the WG is asked whether that type of experimentation is considered to be safe.

Lars

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 11 Nov 2010 12:14:25 -0000

Michael,

I believe that there are numerous other appliactions already deployed,
which could make use of IW10...

Two particular ones I have in mind is NFS & iSCSI - there you also run
into the same issue about short flow completion times - even though the
TCP session itself is much more persistent than a typical http/1.1
session. But every once in a while, you have the same cwnd=initwindow
issue when the session was idle for a brief period.

I don't know the traffic patterns of IMAP close enough, but it would
appear that you also have small objects every once in a while to
transport there (even though the real-time requirements are much
reduced).

Application level signalling would not do it, IMHO. For other
applications, that would mean redeploying millions of commercial OS. In
comparison, redeploying all the web browsers out there is an fairly easy
task.

Although I agree that for the public internet, you rarely run NFS/iSCSI
over it directly ;)

And somehow I think a TCP-level heuristic based on port numbers wouldn't
be nice either...

Richard Scheffenegger

> From: Michael Welzl [mailto:[email-masked]] 
> Sent: Donnerstag, 11. November 2010 11:58
> To: Marco Mellia
> Cc: Mike Belshe; tmrg; Matt Mathis; tcpm
> >>> standardization. Unfortunately the point for the non-convinced is 
> >>> they don't want ANY flows to use IW10 for fear of hurting the 
> >>> performance of their flows.
> >>
> ... which is a good reason to open fewer connections. So 
> that's a matter of bringing of bringing out an update of a 
> web client with a different behavior for the case that a web 
> server uses a larger IW.  
> But how does the web browser know what the web server is 
> doing? I guess they could exchange this information with 
> HTTP. Hm. Are we getting somewhere here, or is this a dead end?
> 
> Cheers,
> Michael
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 11 Nov 2010 12:27:17 -0000

Jerry,

After some thoughts, I have the following summarization for me:

Simulating small pipes is very tricky (concerns about timing effects of simulator, selection of queuing parameters which are realistic...)

Traffic rarely comes in via 1GE, and drops down to a 64kbps bottleneck link. Only in the direction homeuser->webserver this huge bandwidth disparity might be encountered in a single step (but even then I doubt it).

Web server traffic probably runs over a series of shrinking pipes (ie 10G/1G/OC12/OC3/T3/T1/64k) - so in the typical web server environment, even if each link has only small buffers, the server->homeuser direction will less likely show bad effects I would think. Which alignes with the data you presented at ICCSG, correct?

Deploying IW10 with NewReno shows almost the same benefits as with Linux SACK (within 10% of the numbers shown). AFAIK, Linux Newreno is fairly standards-compatible (at least much more than Linux SACK). 

For the record, based on my comments regarding non-web traffic, I'm in support for this proposal, even though I think it should not be deployed just by itself. 

I would like to know if the pacing suggestion would make a difference for the opposition, as that would result in a natural limit of the burstiness and IW (even beyond 10)?

Regards,

Richard Scheffenegger

> From: Jerry Chu [mailto:[email-masked]] 
> Sent: Donnerstag, 11. November 2010 11:59
> To: Marco Mellia
> Cc: tcpm; Mike Belshe; tmrg; Matt Mathis; Michael Welzl
> >> ... but here, your "browsers open tens of flows" argument 
> totally holds. How is a web flow with IW10 worse than a 
> browser opening tens of flows? If people only use it for the 
> web, like Google has been successfully doing, this seems to 
> be quite safe, based on experience.
> >>
> > Increasing the IW to 10 leads to an equivalent IW of 100 
> packets. At that point, you DSL modem will definitively start 
> dropping most of them...
> 
> parameters, tomorrow (ok, maybe in a couple of years) when 
> browsers decide to open 20 or 30 flows because they are 
> frustrated with low TCP parameters like IW, it will 
> effectively be equivalent to increasing IW to 10 today.
> So do we want to do nothing and leave it all to apps, or try 
> to take some control of some key TCP parameters back into TCP itself?
> 
> Jerry
> 
> >
> > --
> >
> > Ciao,                    /\/\/\rco
> >

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Thu, 11 Nov 2010 07:26:09 -0800

Lars,
I have said these things before, again I'll bite.

Firstly, who should answer such a question (safety of choosing a
particular empirical value for IW) :  TCPM or ICCRG?. My book says,
ICCRG recommends such a thing to TCPM.

Secondly, sometimes we say that "the bar is too high for changing TCP",
if we have to go by that golden rule,  IW=10 is very aggressive change
without any knobs to control it, IMO.

-Anantha

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Lars Eggert
Sent: Thursday, November 11, 2010 3:07 AM
To: Michael Welzl
Cc: tmrg; Mike Belshe; Matt Mathis; tcpm
Subject: Re: [tcpm] Increasing the Initial Window - Notes

> Isn't this what Experimental RFCs encourage?

We don't really need an RFC to specify how to set IW to 10 - that should
be pretty straightforward to implementors.

My impression is that Google et al. would like to experiment with IW10
*on the Internet*, and the WG is asked whether that type of
experimentation is considered to be safe.

Lars

---

From: Mark Allman <[email-masked]>
Date: Thu, 11 Nov 2010 12:42:18 -0500

I have inhaled the initial window thread and have some thoughts ...

  + Altering the initial window is not a *fundamental* change to TCP.
    It is a *parameter* tweak.  Certainly parameter tweaks can have
    negative and broad implications, but lets show some perspective
    here.

  + Likewise, changing the initial cwnd **does not remove CC**.  As a
    connection gets feedback from the network path it does what it has
    always done and reacts to its observations.  So, a problematic
    initial window size will in fact get corrected.

  + It is not true that somehow using an initial cwnd of 10 packets
    makes that the minimum cwnd size.  TCP has mechanisms to lower the
    cwnd if that is necessary.

  + Nobody is mandating using a cwnd of 10 packets.  If a host is in a
    particular situation whereby less would be better then it can use
    less (either by tuning its own initial cwnd or by appropriately
    setting the advertised window).

  + Is this HTTP-centric?  Well, maybe.  It is medium transfer
    size-centric.  HTTP is a big component of those transfers at
    present.  But, I don't follow why this is a big deal.  This seems
    like a red herring to me.

  + Adjusting a parameter is not renouncing the conservativeness
    principle.  It is perfectly natural for a parameter like this to
    evolve over time as networks evolve.  Just because there used to be
    excessively low speed networks doesn't mean that somehow we need to
    be bound by that for the entire future.  And, adjusting for
    "inflation" doesn't somehow abdicate the principle of being
    conservative.  Put another way, conservativeness and aggressiveness
    are relative and we are merely trying to find a reasonable tradeoff
    here.

  + Further, there are always *current* networks that lag behind.  At
    some point we have to decide that our engineering is going to be for
    the 99.9% common case and the outliers are going to have to deal
    with it.  People lodged the complaint that there were slow networks
    when we moved from one packet to three.  That will always be the 
    case.  I'm not saying if you're in the minority we forget you here.
    If this change is good for only 51% of the cases then it is not a
    good change.  But, because we have an existence proof of some spot
    that cannot cope with a 10 packet burst does not mean this is a bad
    idea. 

  + Just because IW=10 causes loss doesn't mean it hurts performance.
    Jerry, et.al. have testbed experiments that show this.  The steady
    state loss rate of a TCP is a key component of performance.  But,
    that doesn't mean loss in the initial window dictates long-term
    performance.  If a network can only support a window of three
    packets then the performance is going to be lousy regardless of the
    initial window.  A larger initial window is not likely to change the
    overall performance---as the experiments illustrate.

    (There is the chance that in the case of multiple congested gateways
    that the extra packets will consume scarce resources and therefore
    impact other flows.  But, with excessively low capacity networks it
    is really hard to envision the initial cwnd as being the thing that
    makes the network lousy.)
  
  + Maybe this exacerbates the case of long queues in home networks.
    But, (1) I doubt its a big deal and (2) that isn't exactly TCP's
    problem.  On my DSL line I never build a perceptible queue on the
    downlink.  On the uplink I do.  If I am doing the math right my
    uplink would build a 1sec queue at 10 parallel connections using
    IW=10 and starting at the same time.  But, that seems like a pretty
    thin use case.  I.e., I never start that many connections and pump
    data into them in that fashion.  I might start that many connections
    / second, but I put 300 bytes of HTTP GET in there or the like.  In
    the cases where I can think of that you would start connections at a
    high rate (at least in a burst) and then pump data into them
    (e.g. BitTorrent) you're going to build this queue anyway.  And, it
    just isn't clear to me that we need to somehow make TCP "solve" this
    problem because fundamentally we just need less buffering.

  + Allowing apps to tune the IW via a socket option seems pretty
    pointless to me.  I doubt the application is in any sort of position
    to pick wisely.  If they do anything but turn it to the maximum
    allowed it would be pretty surprising, I think.

While perhaps we can say the onus is on the proposers to convince us, it
seems as though Jerry et.al. have done all the experiments that comments
have pointed to.  If you have a concrete problem with the proposal it'd
be nice to see that in concrete terms---experiments---instead of hand
waves.

FWIW.

allman

---

From: John Heffner <[email-masked]>
Date: Thu, 11 Nov 2010 14:33:10 -0500

I completely agree with everything Mark said.  A couple questions,
though:  Do we have to go through this every ten years?  Do
implementors really need the IETF's blessing on some particular IW
value?  Do we want to put ourselves in that position?  The IETF is not
terribly good at weighing trade-offs, where a change might benefit
some at the expense of others.  Maybe we can issue some broader and
longer-lasting guidance.

Thanks,
  -John

On Thu, Nov 11, 2010 at 12:42 PM, Mark Allman <[email-masked]> wrote:
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>
>

---

From: Mark Allman <[email-masked]>
Date: Thu, 11 Nov 2010 14:33:57 -0500

Lars-

> we currently seem to be a bit stuck. The Google folks have gathered a
> lot of data, but at the moment it's still not quite clear - to me at
> least - what data-based argument would convince the WG to either
> permit IW10 and under what considerations, or which argument would
> convince us to not do it. 

OK.  Hard to argue with that.  But then ...

> * Google makes the raw data available. Yoshifumi requested this at the
>   mike, and Jerry said he'd see if this was possible. 
> 

... this seems a bit weird.  I.e., you can't envision data-based cases
either way and then you propose a whole meeting around trying to build
them. 

> * Folks (other than Google) who like IW10 crunch down the raw Google
>   data in an attempt to provide a convincing argument to allow it, and
>   under which conditions. (Obviously, folks can also generate their
>   own data, if they like.) 
> 
> * Folks sceptical about IW10 crunch down the Google data in an attempt
>   to find problematic cases that'd make the WG lean against
>   IW10. (Obviously, folks can also generate their own data, if they
>   like.) 
> 

It would be great if a more diverse set of folks was playing with this,
but haven't the Google folk answered the questions that have been posed
with their data?  So, what good is it to put that data in other people's
hands?  Do we think Google is botching the analysis or lying to us?  I'd
have framed this differently: it'd be nice to see a more diverse set of
folks playing with empirical data from a diverse set of vantage points.
Just getting more people to mess about in the data that Google has
analyzed in all requested ways (as far as I can tell) doesn't seem
particularly useful.  That is not to say that Google has drawn all
possible insight from their data.  But, it seems they have at least
gotten the high-order bits.  And, my hunch is that giving away data in
any sort of fine-grained way is not going to be easy.

And, further, this is all just fishing.  In other words, if we hand out
the data then we're hoping someone will be able to analyze it and get
some additional insight out of it.  I'd feel much better about this sort
of interim effort if there were at least a strawman set of Big Questions
that folks were going to go after.  And, further, if there are people
who will in fact put in the time to analyze the data in a non-glib
fashion. 

I have an entirely different proposal... Someone should start a list of
questions that give folks pause about IW=10.  Just make this an I-D that
is meant to move the process along and not meant to be published as an
RFC.  Let the proponents---or anyone with handy data that bears on
them---address these issues.  Maybe that resolves the question.  Maybe
it doesn't.  But, at least this (1) moves us beyond fuzzy notions of it
"not feeling right" to move to IW=10 by forcing folks to state possible
issues in concrete terms and (2) it provides a clear path to what the WG
wants to understand before taking some action on this proposal.  The
questions in this mythical I-D can then be annotated with pointers to
answers or possible answers.  I'll (begrudgingly) keep the list if folks
want.

allman

---

From: Jerry Chu <[email-masked]>
Date: Thu, 11 Nov 2010 14:56:59 -0800

On Thu, Nov 11, 2010 at 2:58 AM, Michael Welzl <[email-masked]> wrote:

>>> ... but here, your "browsers open tens of flows" argument totally holds.
>>> How is a web flow with IW10 worse than a browser opening tens of flows? If
>>> people only use it for the web, like Google has been successfully doing,
>>> this seems to be quite safe, based on experience.
>>>
>> Increasing the IW to 10 leads to an equivalent IW of 100 packets. At that
>> point, you DSL modem will definitively start dropping most of them...
>>
>
> ... which is a good reason to open fewer connections. So that's a matter of
> bringing of bringing out an update of a web client with a different behavior
> for the case that a web server uses a larger IW. But how does the web
> browser know what the web server is doing? I guess they could exchange this
> information with HTTP. Hm. Are we getting somewhere here, or is this a dead
> end?
>

One solution to safely migrate to IW10 for browsers opening multiple
connections is to mandate browsers that open X connections, when X is, e.g.,
> 4 to set the initial receive window to 3 (assuming OSes allow initrwnd to
be set on a per connection basis). But I don't know how this can be
enforced...

Jerry

---

From: <[email-masked]>
Date: Fri, 12 Nov 2010 09:48:30 +0000

> If today a brower opens 10 flows with a server using IW=1 you gets 10 packets equivalent IW

And the browser is ignoring RFC 2068 section 8.1:

   Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD maintain AT MOST 2 connections with any
   server or proxy.

---

From: Marco Mellia <[email-masked]>
Date: Fri, 12 Nov 2010 15:03:39 +0100

well...
check this then
http://www.browserscope.org/
in the networking statistics...

Il 12/11/2010 10:48, [email-masked] ha scritto:
> packets equivalent IW

-- 

Ciao,                    /\/\/\rco

+-----------------------------------+
| Marco Mellia - Assistant Professor|
| Skypeid: mgmellia                 |
| Tel: +39-011-090-4173             |
| Cel: +39-331-6714789              |   /"\  .. . . . . . . . . . . . .
| Politecnico di Torino             |   \ /  . ASCII Ribbon Campaign  .
| Corso Duca degli Abruzzi 24       |    X   .- NO HTML/RTF in e-mail .
| Torino - 10129 - Italy            |   / \  .- NO Word docs in e-mail.
| http://www.telematica.polito.it   |        .. . . . . . . . . . . . .
+-----------------------------------+
The box said "Requires Windows 95 or Better." So I installed Linux.

---

From: <[email-masked]>
Date: Fri, 12 Nov 2010 14:05:28 +0000

The point is that implementations don't toe the lne on number of connections per host (6 connections per hostname? Not persistent flows, but actual TCP connections?), and we're not going to move the line to suit them.

The same goes for initial window, I think.

Lloyd Wood
http://tinyurl.com/lloydwood-ccsr
________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Marco Mellia [[email-masked]]
Sent: 12 November 2010 14:03
Cc: [email-masked]; [email-masked]
Subject: Re: [tcpm] Increasing the Initial Window - Notes

well...
check this then

--

Ciao,                    /\/\/\rco

---

From: Marco Mellia <[email-masked]>
Date: Fri, 12 Nov 2010 16:14:06 +0100

Definitively agree with you.

Eventually, the question is if the IEFT should suggest a switch to a 
more aggressive setting when non standard solutions are common practice 
and can possibly interfere (badly) with this change.

I mean, let's say now all linux/windows/bsd servers start switching to 
IW 10, so that 99% of server traffic now arrives with a definitively 
increase burstines. What are the global implication of this switch 
coupled with today common practice of already tweaking the TCP limits 
when IW is more conservative? Say today those tweaks are used by 10% of 
traffic, so they do not interfere too much. What happens tomorrow when 
all traffic becomes more aggressive?

My intuition is that the increase the default IW to 10 would benefit in 
some cases where there is lot of bandwidth, but in some cases it would 
definitively hurt, especially when thinking about mobile terminals, 
which btw seems to me being very important in these days.

This is just my 2c.

Ciao
Marco

-- 

Ciao,                    /\/\/\rco

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 12 Nov 2010 16:54:54 -0000

Hi Marco,

I'm not so sure I follow the argument with the mobile end devices -
especially with that data from browserscope which has been brought up.

Opera Mini 5.1.21566	  	 11 connections/hostname
Nokia 75	            	 12 connections/hostname
Blackberry 8900	       	 12 connections/hostname
Opera Mini 4.2.15227	  	 21 connections/hostname
Sony Ericsson K800i	 	 24 connections/hostname

More of the outliers seem to be browsers on mobile devices... Even
though the majority of desktop browsers runs at 6 connections/host. (My
inititive guess is, that these browsers don't sort objects by size when
downloading, but prefer to present most of a page at the same time -
with TCP head of line blocking subsequent pipelined objects, multiple
objects would be delayed if only few sessions are used. With only one
object per session, but 20+ in parallel, only the objects of directly
impacted flows are delayed).

I think of IW10 more in terms of minimizing latency for small objects
(regardless of application protocol). After all, with IW10 fast recovery
does have a chance to get triggered, whereas with IW3, only Early
Retransmit and RTO are available but no fast retransmit.

My original concerns about IW10 were based on my (obviously unfounded)
assumption, that IW10 will only show better latency for small objects
when the server is also making use of advanced features (sack, early
retransmit, lost retransmit detection, F-RTO, Eifel...).

But the most recent dataset shows, that this goal - minimizing latency -
can be achived even with standard NewReno, and higher packet loss ratios
(which are expected).

It appears to me, that we are now discussing if the undoubtedly
increased packet loss rate (~ +24%) is more troublesome (to the network)
than the faster delivery of objects (to the clients) (~ -13%). And, as
mentioned above, will the increased aggressiveness of flows (because
fast recovery may be leading to more load early on) hurt the network
beyond that...

And as Mark pointed out, this effects are really only transient, for
bulk transfers the behavior is unchanged. And, senders who are adjacent
to links with very small queues would tend to hurt themselves the most -
potentially causing a change to IW3 locally if it improves their sending
performance.

Does that make sense, or am I off track?

Thanks,

Richard Scheffenegger

> From: Marco Mellia [mailto:[email-masked]] 
> Sent: Freitag, 12. November 2010 16:14
> To: [email-masked]
> Eventually, the question is if the IEFT should suggest a 
> switch to a more aggressive setting when non standard 
> solutions are common practice and can possibly interfere 
> (badly) with this change.
> 
> I mean, let's say now all linux/windows/bsd servers start 
> switching to IW 10, so that 99% of server traffic now arrives 
> with a definitively increase burstines. What are the global 
> implication of this switch coupled with today common practice 
> of already tweaking the TCP limits when IW is more 
> conservative? Say today those tweaks are used by 10% of 
> traffic, so they do not interfere too much. What happens 
> tomorrow when all traffic becomes more aggressive?
> 
> My intuition is that the increase the default IW to 10 would 
> benefit in some cases where there is lot of bandwidth, but in 
> some cases it would definitively hurt, especially when 
> thinking about mobile terminals, which btw seems to me being 
> very important in these days.
> 
> This is just my 2c.
> 
> Ciao
> Marco
> 
> > The point is that implementations don't toe the lne on 
> number of connections per host (6 connections per hostname? 
> Not persistent flows, but actual TCP connections?), and we're 
> not going to move the line to suit them.
> >
> > From: [email-masked] [[email-masked]] On 
> Behalf Of Marco 
> > Mellia [[email-masked]]
> >>> 10

---

From: Marco Mellia <[email-masked]>
Date: Sat, 13 Nov 2010 14:37:03 +0100

> fast recovery may be leading to more load early on) hurt the network

Ciao Richard,

My concern related to mobile device is not about them, but about the 
network.
Which is pretty much what you sat above...
Today, 3G networks are already overloaded, with the bottleneck being the 
downlink capacity at the baystation.
If servers move from IW3 to IW10, that would definitively impact that 
bottleneck.
The problem is not what happens on gigabit per second, uncongested paths.
The problem is what happens on low capacity, already congested paths. 
And mobile networks seems to me to be a good example of the latter.

Ciao
Marco

---

From: Marco Mellia <[email-masked]>
Date: Sat, 13 Nov 2010 14:51:14 +0100

> Browsers open lots of connections these days and we always try to cope 
> that in all our experiments.
>
> live:
> To get the best possible apple-to-apple comparison when comparing 
> IW10/IW3 performance on our live frontends, we split the client IPs 
> into two (same size) pools to serve IW10 and IW3 simultaneously. To 
> our own surprise, mobile hosts are among the big winners (even their 
> browsers still open >2 streams per domain).  See slide 19 in our IETF 
> 77 presentation

Not surprised at all... since mobile terminal suffer higher RTT, it is 
quite straight forward to see larger benefit there.
The point is that you cannot see the impairment of other traffic...
It it easy to gain 20% when 20% of flows goes IW10.

But what about the other 80% flows that were still using IW3? Was there 
any impairment for them?
Easy answer: if there is no congestion and there is enough buffer space 
at the bottleneck link, then no impairment is expected.
But if there is already some congestion, and then the bottleneck queue 
is already overloaded, then I expect  lot of impairment (and unfairness).

And what would be the gain if 100% of flows goes IW10?

Are the benefit larger than impairments? Then go for IW10, or even IW16.

Those are questions that your data cannot answer I would say.
Has someone run at least some simulations to see what can happen on a 
congested path?

Ciao
Marco

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 13 Nov 2010 18:13:48 -0000

Marco,

Isn't then what you are really asking for  AQM at the basestation
downlink path?

Obviously, running FIFO queues there is no the brightest of ideas. With
any decent AQM, the impact to any one flow should be proportional to
it's consumed bandwidth. In the degenerate case of IW3 and IW10 initial
transmission burst and an overloaded downlink, shouldn't the basestation
queue drop 10/3 as many packets belonging to IW10 flows, than packets
belonging to IW3 flows, so that each flow is allowed an approximate fair
share of downlink capacity?

And it's been a long time that it's known that "any" AQM scheme will be
beneficial over FIFO tail-drop queuing...

Even then IW10 will be beneficial, as it is easier, and more
packet-conserving, for IW10 flow TCP to recovery that loss in a timely
and work-conserving fashinon...

IIRC, the recent data did contain data points showing the relative
effects of IW3 vs. IW10 vs. bulk transfers over a "shared" bottleneck
link - although only a very limited number of clients being busy at the
same time (4-6 flows in total).

Richard Scheffenegger

> Sent: Samstag, 13. November 2010 14:51
> To: Yuchung Cheng
> Cc: [email-masked] Extensions; [email-masked]
> Not surprised at all... since mobile terminal suffer higher 
> RTT, it is quite straight forward to see larger benefit there.
> But what about the other 80% flows that were still using IW3? 
> Was there any impairment for them?
> Easy answer: if there is no congestion and there is enough 
> buffer space at the bottleneck link, then no impairment is expected.
> But if there is already some congestion, and then the 
> bottleneck queue is already overloaded, then I expect  lot of 
> impairment (and unfairness).
> 
> Has someone run at least some simulations to see what can 
> happen on a congested path?
> 
> Ciao
> Marco

---

From: rick jones <[email-masked]>
Date: Sat, 13 Nov 2010 11:10:48 -0800

On Nov 13, 2010, at 5:37 AM, Marco Mellia wrote:
> Today, 3G networks are already overloaded, with the bottleneck being  
> the downlink capacity at the baystation.

Is there any way to know how much of that overload is what one might  
call "do overs?"  Where a "do over" is not a TCP retransmission, but  
an application-level resending of data because the consumer got  
impatient and re-issued the request.

rick jones

---

From: Lachlan Andrew <[email-masked]>
Date: Sat, 13 Nov 2010 08:14:50 +1100

On 13 November 2010 04:01, Stefan Hirschmann <[email-masked]> wrote:
>
> To be honest. I don't like the idea that an application behaves
> different depending on the used TCP variant. For me, TCP is kernel stuff
> and an application should use it without thinking.

Applications very often check the version of OS or CPU they are
running on, and behave differently accordingly.  Doing this for TCP
wouldn't be setting a precedent.  Of course, we would need a very
clear benefit to justify such an increase in complexity, and it isn't
clear that a factor of 3 in the IW is enough justification.

$0.02
Lachlan

-- 
Lachlan Andrew  Centre for Advanced Internet Architectures (CAIA)
Swinburne University of Technology, Melbourne, Australia
<http://caia.swin.edu.au/cv/landrew>
Ph +61 3 9214 4837

---

From: Jerry Chu <[email-masked]>
Date: Mon, 15 Nov 2010 00:03:24 -0800

On Sat, Nov 13, 2010 at 5:51 AM, Marco Mellia <[email-masked]> wrote:
>> To get the best possible apple-to-apple comparison when comparing IW10/IW3
>> performance on our live frontends, we split the client IPs into two (same
>> size) pools to serve IW10 and IW3 simultaneously. To our own surprise,
>> mobile hosts are among the big winners (even their browsers still open >2
>> streams per domain).  See slide 19 in our IETF 77 presentation
>
> Not surprised at all... since mobile terminal suffer higher RTT, it is quite
> straight forward to see larger benefit there.
> But what about the other 80% flows that were still using IW3? Was there any
> impairment for them?
> Easy answer: if there is no congestion and there is enough buffer space at
> the bottleneck link, then no impairment is expected.
> But if there is already some congestion, and then the bottleneck queue is
> already overloaded, then I expect  lot of impairment (and unfairness).
>

Have you at least checked the title of our most recent presentation in
Beijing to
know it might contain some relevant data for your question?

Best,

Jerry

---

From: Aki Nyrhinen <[email-masked]>
Date: Mon, 15 Nov 2010 20:08:42 +0200

On Thu, 2010-11-11 at 12:27 +0000, Scheffenegger, Richard wrote:

> Traffic rarely comes in via 1GE, and drops down to a 64kbps bottleneck
> link. Only in the direction homeuser->webserver this huge bandwidth
> disparity might be encountered in a single step (but even then I doubt
> it).

> Web server traffic probably runs over a series of shrinking pipes (ie
> 10G/1G/OC12/OC3/T3/T1/64k) - so in the typical web server environment,
> even if each link has only small buffers, the server->homeuser
> direction will less likely show bad effects I would think. Which
> alignes with the data you presented at ICCSG, correct?

i don't know what is common and what is not, but my understanding is
that here, a dsl access network is typically quite fast optical down to
the dslam, which provides dsl access to large number of customers over
their dedicated coppers. on the dslam you have exactly the dslam
manufacturers default magic number of fifo queuing, which is most likely
64KB or 40 packets regardless of the customer sla. this is likely to be
the first place where queuing occurs after the servers uplink, given the
'valley free' characteristic etc.

cable network has a similar story, the amount of cpe served by a single
cmts is large so cmts will sit in a very fast network, so the queuing
takes place there.

for wwan (gprs, edge, 3g, etc.), my experience of queuing delays
suggests 128KB is typical here (worst case queuing delay beyond 10
seconds at 60kbps).

perhaps modem pools aside, i have yet to see the access network were too
little buffering is a problem.

i have been hoping for quite a while that the buffering would go down to
improve interactive performance during bulk transfers but it just does
not seem to happen so i have given up hope. instead, i have started
hoping they keep the defaults because bandwidths are rising. above
10Mbps, 64KB is quite tolerable.

	Aki

---

From: Alexander Zimmermann <[email-masked]>
Date: Mon, 15 Nov 2010 19:14:07 +0100

Hi folks,

three points I really want to say:

* Please stop the discussion about IRTF versus IETF. We (TCPM) had decided to
  discuss this doc in TCPM and we adopted the doc as an WG item. So, the time
  is over and the discussion about that is wasting time.

* We've been going in circles. During the last couples of IETF Meetings, the
  following happened: Jerry presented slides and a bunch of measurements, folks
  came and raised issues a, b, c. Next Meeting, Jerry answered a, b, c and folks
  came with issue d, e, f. Again, next meeting, Jerry ...
  Stop. Please.
  
* I think we have have a rough consensus that IW=10 is something that we want

Alex

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: [email-masked]
// web: http://www.umic-mesh.net
//

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 15 Nov 2010 19:33:31 -0000

+1

Richard Scheffenegger

> From: Alexander Zimmermann 
> [mailto:[email-masked]] 
> Sent: Montag, 15. November 2010 19:14
> To: [email-masked] WG
> // Department of Computer Science, Informatik 4 // RWTH 
> Aachen University // Ahornstr. 55, 52056 Aachen, Germany // 
> phone: (49-241) 80-21422, fax: (49-241) 80-22222 // email: 
> [email-masked] // web: http://www.umic-mesh.net //
> 
>

---

From: Mark Allman <[email-masked]>
Date: Mon, 15 Nov 2010 15:41:09 -0500

> > different depending on the used TCP variant. For me, TCP is kernel
> > stuff and an application should use it without thinking.
> 
> wouldn't be setting a precedent.  

I am not sure that analogy holds.  An app that looks at the OS or the
CPU can get a complete story of what's up.  Checking on TCP seems a
little more tenuous because it isn't really TCP that we're concerned
about, but rather the state of the path relative to the TCP config.
And, its hard to understand the state of the path.

allman

---

From: Lachlan Andrew <[email-masked]>
Date: Mon, 15 Nov 2010 15:57:43 -0800

> And, its hard to understand the state of the path.

I agree the application shouldn't follow the TCP state, but that is
different from adapting to which variant is used.  I thought the issue
was "If we fix problem  X  of TCP (very slow to start), should
applications be able to test whether that fix has been made, and
disable their old work-arounds (opening multiple connections) if it
has?"

Of course, it isn't that simple, because the work-around of opening
many connections may give selfish users additional benefits even if
underlying issues are fixed and so the incentives are wrong.  However,
I don't see a problem in principle with an app adapting to the flavour
of TCP, any more than an app getting a choice of DCCP congestion
control algorithm, or using DCCP if it is available and UDP otherwise.

Cheers,
Lachlan

---

From: Mark Allman <[email-masked]>
Date: Mon, 15 Nov 2010 20:56:31 -0500

> I thought the issue was "If we fix problem X of TCP (very slow to
> start), should applications be able to test whether that fix has been
> made, and disable their old work-arounds (opening multiple
> connections) if it has?"

OK.  So, the client wants to issue small requests to some server and
expects content in return (a la HTTP).  How many connections should it
open?  The server controls the initial cwnd that matters and the client
controls the number of parallel connections.

allman

---

From: Lachlan Andrew <[email-masked]>
Date: Mon, 15 Nov 2010 18:17:18 -0800

> controls the number of parallel connections.

Good point.  That is a compelling argument.  I was only refuting the
argument that for an app to talk to its own kernel about its TCP
variant is some sort of layering violation.  Perhaps I misunderstood
the original poster.

Cheers,
Lachlan

---

From: John Heffner <[email-masked]>
Date: Mon, 15 Nov 2010 22:45:37 -0500

On Thu, Nov 11, 2010 at 5:56 PM, Jerry Chu <[email-masked]> wrote:
>> ... which is a good reason to open fewer connections. So that's a matter
>> of bringing of bringing out an update of a web client with a different
>> behavior for the case that a web server uses a larger IW. But how does the
>> web browser know what the web server is doing? I guess they could exchange
>> this information with HTTP. Hm. Are we getting somewhere here, or is this a
>> dead end?
>

I'm not a Googler, so you probably have better insight into this than
I do, but my impression is that one big reason (if not the *primary*
reason) browsers open multiple connections is being able to submit
requests in parallel.  HTTP pipelining can help some, but (a) most
browsers disable this because of some buggy and widely deployed
servers, and (b) it is still susceptible to head-of-line blocking on
responses.  Might limiting the number of simultaneous connections
based on IW hurt more than it helps?

The solution would seem to be shared congestion state on the server.
Like the "congestion manager" (http://nms.lcs.mit.edu/cm/), or using
multiple channels in SCTP.  Of course, this may not be practical with
a lot of load-balancing systems.

Thanks,
  -John

---

From: Fred Baker <[email-masked]>
Date: Tue, 16 Nov 2010 12:07:21 +0800

> one insisted then that the initial quantum must be 1?

Actually, no. I may be the person who said that this effectively disables congestion control in the average case; my reason for saying that is that per what little data I have, the average TCP transfer is on the order of 15K bytes, which is about ten 1460 byte segments. 

Now, the median is much smaller - on the order of two or three data segments each direction, which is where you get your comment. But the fact that browsers, Bittorrent, and other applications open multiple sessions simultaneously argues that a given endpoint having multiple packets in flight simultaneously doesn't hurt, and research done at the time by Mark Allman and others showed that IW=2..4 wasn't a problem and did in fact reduce the median session to a single RTT for data. But the median session doesn't create a congestion problem.

The question I have been asking is how IW=10 affects parallel sessions on relatively low speed links. My sense is that the sudden arrival of O(15K) bytes at a 56 KBPS line, which is to say the insertion of a tad over two seconds delay in all competing sessions plus an increase in their probability of loss, is likely to cause them all to be disrupted. You told me verbally that in your tests it is net beneficial to the parallel sessions, but in thought experiments I'm not sure how that can be true; I'd be fascinated to see the data that shows that competing voice/video sees no frame loss or induced noise, that competing file transfers slow down but do not see redundant retransmissions, and so on.

So, IW=2..4 demonstrably works pretty well everywhere, but IW=10 seems to me only sensible on MPBS and GBPS, not KBPS, scale links.

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Nov 2010 21:43:43 -0800

On 11/15/2010 7:45 PM, John Heffner wrote:
...
> a lot of load-balancing systems.

Or RFC2140, at least part of which is already in Linux. The 2140 
approach can be aggregated across machines:

    We also note that per-host-pair associations are not the limit of
    these techniques. It is possible that TCBs could be similarly shared
    between hosts on a LAN, because the predominant path can be LAN-LAN,
    rather than host-host.

Joe

---

From: <[email-masked]>
Date: Tue, 16 Nov 2010 08:23:28 +0000

On 16 Nov 2010, at 01:56, Mark Allman wrote:

open?

RFC 2068 section 8.1:

The server controls the initial cwnd that matters and the client
controls the number of parallel connections.

allman

Lloyd Wood
[email-masked]<mailto:[email-masked]>
http://sat-net.com/L.Wood

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Tue, 16 Nov 2010 16:01:19 +0200 (EET)

On Tue, 16 Nov 2010, Fred Baker wrote:

> Actually, no. I may be the person who said that this effectively 
> disables congestion control in the average case; my reason for saying 
> that is that per what little data I have, the average TCP transfer is on 
> the order of 15K bytes, which is about ten 1460 byte segments. 

It only "disables" it if there was no congestion to control in the first 
place.

> Now, the median is much smaller - on the order of two or three data 
> segments each direction, which is where you get your comment. But the 
> fact that browsers, Bittorrent, and other applications open multiple 
> sessions simultaneously argues that a given endpoint having multiple 
> packets in flight simultaneously doesn't hurt, and research done at the 
> time by Mark Allman and others showed that IW=2..4 wasn't a problem and 
> did in fact reduce the median session to a single RTT for data. But the 
> median session doesn't create a congestion problem.
> 
> The question I have been asking is how IW=10 affects parallel sessions 
> on relatively low speed links.

Could you on your behalf answer how does IW=3 affect parallel sessions on 
the same kind of environment? ...I've done research on this (and Jerry has 
too) and I know that the answer is not an intuitive one. Then we changed 
to IW10 in the same environment and observe no (consistent) change. The 
results are driven by problems already with IW3 so IW10 won't make it any 
worse or the effect is just too small to be measured among the other 
troubles that happen.

> My sense is that the sudden arrival of 
> O(15K) bytes at a 56 KBPS line, which is to say the insertion of a tad 
> over two seconds delay in all competing sessions plus an increase in 
> their probability of loss, is likely to cause them all to be disrupted. 

Again, the existing research points to exactly _opposite_ thing to 
happen, ie., IW10 hurts mostly itself, not the others! We'll never get 
anywhere if these same "fuzzy feelings" (like Mark put it) just get 
repeated and the existing _evidence_ overlooked. Jerry has tested a case 
that is very close to what you're specifying and his results don't agree 
what you think would happen. My results also confirm Jerry's observations 
for the low end links though in my case it was a bit higher bw already 
(160kbps).

I personally don't anymore believe we will find a convincing case against 
IW10 from the very lowest end of links, afaict, it doesn't help but 
doesn't really hurt significantly either. ...I understand this is quite 
non-intuitive claim but it is what the performed studies seem to point 
out (and I expect new guys every now and then to jump in with this same 
argument if IW10 is to proceed). ...I'm more concerned about mid-ranged 
links where it could be possible (at least in theory if AQM in use) to 
have both TCP and low-latency requiring audio/video in parallel. 

> You told me verbally that in your tests it is net beneficial to the 
> parallel sessions, but in thought experiments I'm not sure how that can 
> be true; I'd be fascinated to see the data that shows that competing 
> voice/video sees no frame loss or induced noise, that competing file 
> transfers slow down but do not see redundant retransmissions, and so on.

In this I somewhat agree with you. The delay increase was confirmed by my 
measurements, however, I think you also end up too cleverly mixing 
IW3+IW10 and voice/video cases here. It seems that even if delay increases 
and as a result voice/video could possible experience more losses (I'm yet 
to confirm that losses actually happen as I didn't have actual CBR traffic 
running), IW3 TCP in mixed case is not affected in a similar way. Instead, 
IW3 TCP might actually get a boost in its performance against IW10 because 
also IW10 TCP needs to respond to the congestion it caused.

Also, regardless of IW, TCP is well capable of leaving audio/video in 
ruins as it will fill all buffers you give to it, so I find this exercise 
quite pointless anyway unless there is some form of AQM in use.

> So, IW=2..4 demonstrably works pretty well everywhere, but IW=10 seems 
> to me only sensible on MPBS and GBPS, not KBPS, scale links. 

IW3 doesn't work that well if you have parallel connections over 56kbps 
but since the latencies are high anyway, you just won't notice that.

-- 
 i.

---

From: Fred Baker <[email-masked]>
Date: Tue, 16 Nov 2010 22:09:38 +0800

> It only "disables" it if there was no congestion to control in the first 

How do you turn off something that's not on?

I refer to this as in the average case turning off congestion control because the transmission requires ten data packets and they all get sent in a single burst. There is no congestion control, with IW=10, in the case of the average-sized (which is actually gar above the median-sized) file transfer.

> worse or the effect is just too small to be measured among the other 

That may be true. I have asked to see data describing the impact. I haven't seen the data. Maybe I missed something?

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 16 Nov 2010 16:41:50 -0000

Group,

Just one additional data point: Obviously, there are already applications in the wild, that use persistent connections and disable the slow-start-restart mechanism on these sessions. The expectations of the programmers making use of these socket options is to reduce the latency for their appliaction... In effect, these Apps seem use IW=44 (and hurting themselves more than anybody else).

See "Profiling Network Performance in Multi-Tier Data Center Applications", Section 6.1, "cwnd failing to prevent sudden bursts", http://www.cs.princeton.edu/~jrex/papers/snap10.pdf

IMHO this is evidence that some action in this space is absolutely needed (even if it is "only" strong advice to apps programmers), or there will be even more proliferation of clumsy workarounds that are even less well understood and investigated than IW10 already is. Keeping IW3 "forever" may be more problematic than addressing emerging application needs with well-tested transport-layer approaches...

Richard Scheffenegger

> From: Fred Baker [mailto:[email-masked]] 
> Sent: Dienstag, 16. November 2010 15:10
> To: "Ilpo Järvinen"
> Cc: tcpm; tmrg
> Subject: Re: [tcpm] [Tmrg] Increasing the Initial Window - Notes
> 
> 
> >>> But haven't we already done that years ago when moving up 
> IW from 1 
> >>> to 3 if one insisted then that the initial quantum must be 1?
> >> 
> >> that is that per what little data I have, the average TCP 
> transfer is 
> >> on the order of 15K bytes, which is about ten 1460 byte segments.
> > 
> > It only "disables" it if there was no congestion to control in the 
> > first place.
> 
> I refer to this as in the average case turning off congestion 
> control because the transmission requires ten data packets 
> and they all get sent in a single burst. There is no 
> congestion control, with IW=10, in the case of the 
> average-sized (which is actually gar above the median-sized) 
> file transfer.
> 
> > Could you on your behalf answer how does IW=3 affect 
> parallel sessions 
> > on the same kind of environment? ...I've done research on this (and 
> > Jerry has
> > too) and I know that the answer is not an intuitive one. Then we 
> > changed to IW10 in the same environment and observe no (consistent) 
> > change. The results are driven by problems already with IW3 so IW10 
> > won't make it any worse or the effect is just too small to 
> be measured 
> > among the other troubles that happen.
> 
> That may be true. I have asked to see data describing the 
> impact. I haven't seen the data. Maybe I missed something?

---

From: Alexander Zimmermann <[email-masked]>
Date: Wed, 17 Nov 2010 08:58:26 +0100

Hi Richard,

exactly, this is the point!

IMO, two things are crystal clear:

1. We have to touch/increase the IW
2. We have to do it now

Also, IMO he have a rough consensus on these two points.

What I like two see now, or discuss now, is a strategy how we can progress.

Mark proposed something, but what are thoughts from the document author/
other group members/chairs/AD on this point?

Alex

//

---

From: Jerry Chu <[email-masked]>
Date: Wed, 17 Nov 2010 10:13:42 -0800

[sorry I'm trying to catchup while on vacation]

> a lot of load-balancing systems.

The various points you raised above have been covered in our earlier
slides and/or discussed on the list. The following is a short summary
for you:

1. Browsers requires multiple connections to schedule web object downloads
in parallel to get the best user experience, much like a multi-threaded
application requires a multi-core CPU to run well. TCP's small IW is one
important but not the only reason for simultaneous connections. Avoiding
HOLB is another.

2. The # of simultaneous connections by all popular browsers have
continued to increase in the past few years. By increasing IW we hope
to reverse this trend because with a much large IW the browser won't
need that many connections to get the best user experience.

3. The solutions you've mentioned above are well understood on paper
and not deployed for various reasons. Also they would require an even
larger IW to work well.

Best,

Jerry

---

From: David Borman <[email-masked]>
Date: Wed, 17 Nov 2010 13:02:41 -0600

I've been thinking about the whole IW issue, and trying to step back a bit from IW itself.  So I ask myself, what is the main issue with having a large IW?  If the network has the bandwidth, there isn't any practical problem.  But if the network doesn't have the bandwidth, a bunch of TCP connections starting up with a larger IW can lead to congestion.  So then I ask myself, what is wrong with congestion?  The data is going to back up somewhere, so does it matter?  Of course, the answer is that yes, it does matter, because we don't have a lossless network.  When there is congestion in routers, if there isn't enough buffering, the way to deal with that is to drop packets, which in turn leads to retransmissions.

But I don't think it is congestion itself and the dropping of packets that is the problem, the issue is we don't want that congestion to turn into congestion collapse.  So, it seems to me that the issue about whether or not it is safe to increase the IW should be focused more on, if increasing the IW leads to increased congestion, is the network still able to properly respond to that congestion and avoid congestion collapse?  If the answer is "yes", then I think that greatly reduces the risks of increasing the IW.  But if the answer is "no", then we have a bigger problem that is lying in wait and can be triggered by other factors than just increasing the IW.

			-David Borman

---

From: Jerry Chu <[email-masked]>
Date: Wed, 17 Nov 2010 11:27:49 -0800

On Mon, Nov 15, 2010 at 8:07 PM, Fred Baker <[email-masked]> wrote:

> Actually, no. I may be the person who said that this effectively disables
> congestion control in the average case; my reason for saying that is that
> per what little data I have, the average TCP transfer is on the order of 15K
> bytes, which is about ten 1460 byte segments.
>
> segments each direction, which is where you get your comment. But the fact
> that browsers, Bittorrent, and other applications open multiple sessions
> simultaneously argues that a given endpoint having multiple packets in
> flight simultaneously doesn't hurt, and research done at the time by Mark
> Allman and others showed that IW=2..4 wasn't a problem and did in fact
> reduce the median session to a single RTT for data. But the median session
> doesn't create a congestion problem.
>
> The question I have been asking is how IW=10 affects parallel sessions on
> relatively low speed links. My sense is that the sudden arrival of O(15K)
> bytes at a 56 KBPS line, which is to say the insertion of a tad over two
> seconds delay in all competing sessions plus an increase in their
> probability of loss, is likely to cause them all to be disrupted. You told
> me verbally that in your tests it is net beneficial to the parallel
> sessions, but in thought experiments I'm not sure how that can be true; I'd
> be

Perhaps you've missed something in your thought process? In addition to some
points
Ilpo has made in his reply, another plausible explanation that you may not
have
considered is that for the same amount of data to be transfered, IW10 is
simply more
"efficient" in average than IW3. It is true that IW10 causes a larger burst
to be sent,
but it is not like you're going to get the same frequency of 10-pkt bursts
as compared
to 3-pkt bursts. The comparison is more like a single large burst vs three
smaller
bursts. It's not clear to me one can actually prove the single large burst
will often
cause more damage to competing flows than 3 smaller bursts and our testbed
sorta
confirmed this.

Jerry

> fascinated to see the data that shows that competing voice/video sees no
> frame loss or induced noise, that competing file transfers slow down but do
> not see redundant retransmissions, and so on.
>
> So, IW=2..4 demonstrably works pretty well everywhere, but IW=10 seems to
> me only sensible on MPBS and GBPS, not KBPS, scale links.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 17 Nov 2010 11:29:38 -0800

On Tue, Nov 16, 2010 at 6:09 AM, Fred Baker <[email-masked]> wrote:

> I refer to this as in the average case turning off congestion control
> because the transmission requires ten data packets and they all get sent in
> a single burst. There is no congestion control, with IW=10, in the case of
> the average-sized (which is actually gar above the median-sized) file
> transfer.
>
There surely is CC involved even if app only sends < 10 packets and some
packets got dropped.

> seen the data. Maybe I missed something?
The NCSU lab has released the raw test data (result presented in IETF).
http://research.csc.ncsu.edu/netsrv/?q=content/iw10#data

---

From: Jerry Chu <[email-masked]>
Date: Wed, 17 Nov 2010 11:39:33 -0800

> seen the data. Maybe I missed something?

Have you chosen not to see the data, or you've simply forgot we've done some
testbed
study with results presented at TCPM (which you missed due to prior
engagement)?

One of the purposes for this round of our testbed study was to investigate
the questions
you've raised. If you have still issues with the result, e.g., you have a
different set of
parameters in mind than what were used by our tests, please spell them out
and we
can hopefully get the results for you quickly (well I'm still in travel so
it may take a
couple of weeks).

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 17 Nov 2010 12:20:22 -0800

> other group members/chairs/AD on this point?

(Sorry for slow in response due to traveling and limited communication with
my
other colleagues.)

I like the idea in that it is consistent with our belief that IW needs to be
scaled up over time but there are some details/feasibility/execution that
need
to be thought out.

Also 3 years seems a long time. Remember IW10 has been in
active experiments for more than a year at Google and some other places.

More later,

Jerry

> > Just one additional data point: Obviously, there are already applications
> in the wild, that use persistent connections and disable the
> slow-start-restart mechanism on these sessions. The expectations of the
> programmers making use of these socket options is to reduce the latency for
> their appliaction... In effect, these Apps seem use IW=44 (and hurting
> themselves more than anybody else).
> >
> > See "Profiling Network Performance in Multi-Tier Data Center
> Applications", Section 6.1, "cwnd failing to prevent sudden bursts",
> http://www.cs.princeton.edu/~jrex/papers/snap10.pdf
> >
> >
> > IMHO this is evidence that some action in this space is absolutely needed
> (even if it is "only" strong advice to apps programmers), or there will be
> even more proliferation of clumsy workarounds that are even less well
> understood and investigated than IW10 already is. Keeping IW3 "forever" may
> be more problematic than addressing emerging application needs with
> well-tested transport-layer approaches...
> >
> >
> > Richard Scheffenegger
> >
> >

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Nov 2010 15:37:22 -0500

David-

> But I don't think it is congestion itself and the dropping of packets
> that is the problem, the issue is we don't want that congestion to
> turn into congestion collapse.  So, it seems to me that the issue
> about whether or not it is safe to increase the IW should be focused
> more on, if increasing the IW leads to increased congestion, is the
> network still able to properly respond to that congestion and avoid
> congestion collapse?  If the answer is "yes", then I think that
> greatly reduces the risks of increasing the IW.  But if the answer is
> "no", then we have a bigger problem that is lying in wait and can be
> triggered by other factors than just increasing the IW. 

I think the answer is pretty obviously "no".  We are not taking the
reaction to congestion away at all and *that* is the important part in
preventing collapse.  In other words, left alone, TCP will keep reducing
the rate at which it tries to recover (via the window and ultimately via
RTO backoff).  This is what I mean when I have said that changing the
initial cwnd does not eliminate congestion control.

Second, remember that we know that most of the traffic volume is large
flows.  And, initial settings in large flows are pretty meaningless.
Further, there are plenty of flows that aren't even 15KB.  So, there
would have to be some scenario whereby some number of medium-sized flows
that are not taking a large chunk of the bandwidth would collapse the
network.  That seems hard to see.

The only scenario I can come up with that even starts us down the road
towards collapse is the case where a connection uses a 10 segment
initial window and takes some loss and the user gets impatient and hits
reload to give it another shot.  In that case, the packets that made it
through in the first connection *wasted* scarce network resources (i.e.,
made it across the congested link) and yet ultimately accomplished no
useful work.  However, I don't buy this as a very likely scenario
because if you play out sending 10 packets with IW=3 and IW=10 in a
network that can only handle 3 packets the transfer times are pretty
comparable (I did this last night on the TMRG list) with the IW10
connection longer by roughly one RTO.  So, if we're worried about user
impatience with IW10 then we should worry about it with IW3, too.

allman

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 17 Nov 2010 15:41:39 -0500

IMHO:

If you think a small TCP IW is protecting any network from
congestion collapse, you are either living in the past
or under delusions of grandeur.

Today's networks are under greater threat from many protocols
that have no respect for congestion control at all and
would collapse immediately without QOS.

TCP IW protects the TCP session itself.
The problem with TCP is that if it drops too many packets, it
grinds to a halt. That is to be avoided at all costs.
A small IW helps to avoid that.
Of course too small an IW makes it take too long to ramp up.
There is no good way to find the optimum. The best seems to
be history of previous connections.

It probably should not even be standardized.
Remove the IW recommendation from the standard and
let each implementation figure out its own optimum.

--
Jakob Heitz.

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of David Borman
> Sent: Wednesday, November 17, 2010 11:03 AM
> To: Alexander Zimmermann
> Cc: tcpm
> I've been thinking about the whole IW issue, and trying to 
> step back a bit from IW itself.  So I ask myself, what is the 
> main issue with having a large IW?  If the network has the 
> bandwidth, there isn't any practical problem.  But if the 
> network doesn't have the bandwidth, a bunch of TCP 
> connections starting up with a larger IW can lead to 
> congestion.  So then I ask myself, what is wrong with 
> congestion?  The data is going to back up somewhere, so does 
> it matter?  Of course, the answer is that yes, it does 
> matter, because we don't have a lossless network.  When there 
> is congestion in routers, if there isn't enough buffering, 
> the way to deal with that is to drop packets, which in turn 
> leads to retransmissions.
> 
> But I don't think it is congestion itself and the dropping of 
> packets that is the problem, the issue is we don't want that 
> congestion to turn into congestion collapse.  So, it seems to 
> me that the issue about whether or not it is safe to increase 
> the IW should be focused more on, if increasing the IW leads 
> to increased congestion, is the network still able to 
> properly respond to that congestion and avoid congestion 
> collapse?  If the answer is "yes", then I think that greatly 
> reduces the risks of increasing the IW.  But if the answer is 
> "no", then we have a bigger problem that is lying in wait and 
> can be triggered by other factors than just increasing the IW.
> 
> 			-David Borman
> 
> 
> >> Just one additional data point: Obviously, there are 
> already applications in the wild, that use persistent 
> connections and disable the slow-start-restart mechanism on 
> these sessions. The expectations of the programmers making 
> use of these socket options is to reduce the latency for 
> their appliaction... In effect, these Apps seem use IW=44 
> (and hurting themselves more than anybody else).
> >> 
> >> See "Profiling Network Performance in Multi-Tier Data 
> Center Applications", Section 6.1, "cwnd failing to prevent 
> sudden bursts", http://www.cs.princeton.edu/~jrex/papers/snap10.pdf
> >> 
> >> 
> >> IMHO this is evidence that some action in this space is 
> absolutely needed (even if it is "only" strong advice to apps 
> programmers), or there will be even more proliferation of 
> clumsy workarounds that are even less well understood and 
> investigated than IW10 already is. Keeping IW3 "forever" may 
> be more problematic than addressing emerging application 
> needs with well-tested transport-layer approaches...
> >> 
> >> 
> >> Richard Scheffenegger
> >> 
> >>

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Nov 2010 16:15:25 -0500

Hi Fred!

> to me only sensible on MPBS and GBPS, not KBPS, scale links.  

Question: How much do you think we should be able to expect from hosts /
people connected via 56kbps?  There is a control they can use---the
advertised window---to prevent IW=10 from being used on traffic sent
their way.  Is it reasonable to expect outliers such as this to utilize
such controls and not the defaults?

allman

---

From: Fred Baker <[email-masked]>
Date: Thu, 18 Nov 2010 07:04:47 +0800

> such controls and not the defaults?

Not in my experience. They generally don't know much about the network.

---

From: David Borman <[email-masked]>
Date: Thu, 18 Nov 2010 06:46:11 -0600

> or under delusions of grandeur.

That is my point, increasing the TCP IW may increase congestion and packet drops in some scenarios, but it doesn't change how TCP reacts to congestion so there is no reason to think that increasing the IW will lead to congestion collapse.  Congestion by itself isn't so evil that we have to do everything we can to avoid it, it is only when it leads to congestion collapse that it becomes detrimental to the network.

In other words, just saying that increasing the IW can lead to congestion isn't sufficient reason to not allow the IW to be increased.

			-David Borman

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 18 Nov 2010 13:47:49 -0500

You are implying that congestion collapse is something to be avoided.
I agree.
However, I believe that:
It is not TCP's responsibility to avoid congestion collapse.
It is not within TCP's power to avoid congestion collapse.
No sane network operator would rely on the TCP in end hosts to avoid congestion collapse.
No sane server operator would rely on the TCP in clients to avoid server collapse.

Therefore selection of IW should not be influenced by any desire to
avoid congestion or protect the network. The network will protect itself.
The best IW is the amount that the sender thinks he can push through
within the estimated RTO. This will be different in every case.

--
Jakob Heitz.

> From: David Borman [mailto:[email-masked]] 
> Sent: Thursday, November 18, 2010 4:46 AM
> To: Jakob Heitz
> Cc: tcpm
> That is my point, increasing the TCP IW may increase 
> congestion and packet drops in some scenarios, but it doesn't 
> change how TCP reacts to congestion so there is no reason to 
> think that increasing the IW will lead to congestion 
> collapse.  Congestion by itself isn't so evil that we have to 
> do everything we can to avoid it, it is only when it leads to 
> congestion collapse that it becomes detrimental to the network.
> 
> In other words, just saying that increasing the IW can lead 
> to congestion isn't sufficient reason to not allow the IW to 
> be increased.
> 
> 			-David Borman
> >

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 12:00:41 -0800

Hi, all,

On 11/18/2010 4:46 AM, David Borman wrote:
> That is my point, increasing the TCP IW may increase congestion and
> packet drops in some scenarios, but it doesn't change how TCP reacts to
> congestion ...

Actually, it does.

Consider that there are three cases:

A) connections so small the IW doesn't matter (they already fit in IW=3)

B) connections 4-20 or so segments range

C) connections much longer than 20 or so segments

(A) isn't affected by this proposed change at all, so we don't need to 
consider it.

(C) adjusts to the IW using AIMD, so even if we guess wrong, TCP will 
"react to congestion" here.

And then there's (B). (B) doesn't react to congestion. Every new 
connection uses IW=10. Even after repeated attempts - to a given 
address, or, e.g., to every address - end up dropping, e.g., half of 
those segments. (B) will sit there and beat its head against the wall.

That's the congestion-increasing case. Will this cause collapse? I don't 
know. But it's definitely NOT reacting to congestion.

So can we please get off the long-connection case? That's not the issue 
here at all*...

Joe

PS - * even long connections can be affected by the IW, e.g., when they 
do a restart after idle, if they use IW for the restart value 
repeatedly; that's worth considering, but should be considered roughly 
the same as (B) if each busy period is in the 4-20 segments range. I.e., 
we can just talk about (B), IMO.

---

From: David Borman <[email-masked]>
Date: Thu, 18 Nov 2010 14:17:54 -0600

> It is not within TCP's power to avoid congestion collapse.

Wrong.  The whole point of slowstart and exponential backoff in TCP is to keep TCP from causing or contributing to congestion collapse.  This is what happened before TCP had such things, and why we have it now.  It is fully within TCP's power and responsibility to avoid contributing to congestion collapse by implementing proper congestion control algorithms.

> No sane server operator would rely on the TCP in clients to avoid server collapse.

If the TCPs in end hosts (both client and server) did not have congestion control algorithms, we'd surly fall into congestion collapse again.  The network can't prevent that.  Congestion collapse is when no useful data gets through due to congestion, e.g., the TCPs keep blasting out bursts of retransmitted data, but do not make forward progress due to packet loss due to congestion.  The network has no control over what data the end hosts choose to send.

> avoid congestion or protect the network. The network will protect itself.

My point is that because TCPs congestion control algorithms aren't modified by increasing the IW, if the only concern of a larger IW is that it may cause congestion, that argument by itself is not sufficient to defeat increasing the IW.

> within the estimated RTO. This will be different in every case.

Only if it is able to spread out the transmission of those packets over the RTO.  If it blasts out those packets in the first 10% of the RTO, then if there isn't sufficient buffering in the network, you will still have packet loss.  During the first RTO you don't have any returning ACKs to clock out the data.

			-David Borman

---

From: David Borman <[email-masked]>
Date: Thu, 18 Nov 2010 14:32:45 -0600

On Nov 18, 2010, at 2:00 PM, Joe Touch wrote:

> (A) isn't affected by this proposed change at all, so we don't need to consider it.
> 
> (C) adjusts to the IW using AIMD, so even if we guess wrong, TCP will "react to congestion" here.
> 
> And then there's (B). (B) doesn't react to congestion. Every new connection uses IW=10. Even after repeated attempts - to a given address, or, e.g., to every address - end up dropping, e.g., half of those segments. (B) will sit there and beat its head against the wall.

> 
> That's the congestion-increasing case. Will this cause collapse? I don't know. But it's definitely NOT reacting to congestion.

No, (B) will also react to congestion, if the congestion causes packet loss.  For any given flow, if all the packets get through the first time, from it's viewpoint there is no congestion.  If they don't all get through, that connection does normal TCP backoff and retransmits.  It's not really any different than if the connection had started with IW=3, worked it's way up to IW=10, and *then* the network became congested.  If there is congestion and packet loss, those TCP streams affected by the packet loss will all be backing off.  The flows that don't have dropped packets complete and go away.

			-David Borman

> 
> 
> So can we please get off the long-connection case? That's not the issue here at all*...
> 
> Joe
> 
> PS - * even long connections can be affected by the IW, e.g., when they do a restart after idle, if they use IW for the restart value repeatedly; that's worth considering, but should be considered roughly the same as (B) if each busy period is in the 4-20 segments range. I.e., we can just talk about (B), IMO.
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 12:43:25 -0800

On 11/18/2010 12:32 PM, David Borman wrote:
...
> No, (B) will also react to congestion, if the congestion causes
> packet loss. For any given flow, if all the packets get through the
> first time, from it's viewpoint there is no congestion. If they don't
> all get through, that connection does normal TCP backoff and
> retransmits.

To be more clear, here's the case:

	start 1000 connections in a row.

	during the first connection, lose some packets and do
	normal TCP backoff

	so what do the other 999 connections start with?
	ans: 10 packets

The point is that subsequent connections don't do anything different. If 
you have 1000 connections, you're sending a certain amount of data into 
the network without reacting. We're tripling that.

That can easily cause congestion. At which point the *existing* 
connections will backoff, but new connections keep making problems.

Joe

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Thu, 18 Nov 2010 13:05:01 -0800

Why can't you do something like this :

- If any one of the TCP connections egressing out of that interface
(this can determined in TCP layer) is in the TCP retransmit state (or
has experienced congestion in the past xxx secs/mins), then go back to a
lower IW for new TCP connections which are using the same output
interface.. 

- When you want to start a connection, use the connection history (Joe's
RFC)

Well, there may be some gotchas of this scheme, but you can become
conservative when there is some concrete information.

Thanks,
-Anantha

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 13:08:51 -0800

That's basically what I'm suggesting; it can easily be per subnet, per 
interface, or per the entire machine as desired.

The ultimate point is:

	- put something in the end node that notices if/when
	it fails objectively, and fixes it

	- that same thing can allow the IW to increase
	over time if there are no problems

I'll be glad to write this up if people need a more concrete proposal.

Joe

---

From: David Borman <[email-masked]>
Date: Thu, 18 Nov 2010 15:10:26 -0600

On Nov 18, 2010, at 2:43 PM, Joe Touch wrote:

> The point is that subsequent connections don't do anything different. If you have 1000 connections, you're sending a certain amount of data into the network without reacting. We're tripling that.
> 
> That can easily cause congestion. At which point the *existing* connections will backoff, but new connections keep making problems.

If you are doing the 1000 connections serially, then any connection that experiences packet loss will slow down that connection, and delay the creation of all the successive connections, and hence delay the next IW burst.  Seems to me that that is reacting to congestion across connections.

If you are doing 1000 connections in parallel, well, you are now spitting out 3,000 packets with IW=3 and 10,000 packets with IW=10, both of which are probably going to cause congestion.

Remember, from the end-point of a TCP connection, it is not aware of congestion unless there is packet loss or an ECN indication.  There may be congestion on the path, but if the packets gets through, TCP is unaware of the congestion, other than the RTO going up due to delayed packets, and will continue to open up its idea of the congestion window.

But here's a thought:  Any TCP connection with a large IW that has to retransmit or gets an ECN notification during the initial 3-way handshake must clamp down its IW to a small value.  That won't catch every case, but it'll catch some.

			-David

---

From: John Heffner <[email-masked]>
Date: Thu, 18 Nov 2010 16:11:20 -0500

Why is this mechanism required for IW=10 but not for IW=3?

  -John

On Thu, Nov 18, 2010 at 4:08 PM, Joe Touch <[email-masked]> wrote:
>>> you have 1000 connections, you're sending a certain amount of data
>>
>> into
>>>

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 17 Nov 2010 15:09:25 -0500

It probably should not even be standardized.
Let each implementation figure out its own optimum.

--
Jakob Heitz.

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 13:16:27 -0800

Maybe it should have been for IW=3 too, but it didn't occur to me at the 
time.

The benefit is that had we used this mechanism then, we wouldn't be 
asking for a new IW now, though.

Joe

On 11/18/2010 1:11 PM, John Heffner wrote:
> On Thu, Nov 18, 2010 at 4:08 PM, Joe Touch<[email-masked]>  wrote:

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 13:21:24 -0800

On 11/18/2010 1:10 PM, David Borman wrote:
...
> If you are doing the 1000 connections serially, then any connection
> that experiences packet loss will slow down that connection, and delay
> the creation of all the successive connections, and hence delay the next
> IW burst. Seems to me that that is reacting to congestion across
> connections.

That's true if the connections are serial, but I didn't say that. I said
try to start 1000 connections. They'll start in a sequence, but they 
don't get delayed based on what other connections are doing necessarily.

> If you are doing 1000 connections in parallel, well, you are now
> spitting out 3,000 packets with IW=3 and 10,000 packets with IW=10, both
> of which are probably going to cause congestion.

The 10,000 packets cause more congestion, or causes congestion more of 
the time - depending on the network parameters. I.e., it *increases* the 
congestion, regardless.

> Remember, from the end-point of a TCP connection, it is not aware of
> congestion unless there is packet loss or an ECN indication. There may
> be congestion on the path, but if the packets gets through, TCP is
> unaware of the congestion, other than the RTO going up due to delayed
> packets, and will continue to open up its idea of the congestion window.

Agreed.

> But here's a thought: Any TCP connection with a large IW that has to
> retransmit or gets an ECN notification during the initial 3-way
> handshake must clamp down its IW to a small value. That won't catch
> every case, but it'll catch some.

Sure - that's basically what I'm suggesting. We can refine the feedback 
mechanism (packet loss detected via SACK or ACK gaps vs. ECN 
notification vs retransmit). I'm also suggesting that the IW should 
increase over time, too - so that we don't need to revisit this issue 
explicitly.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 18 Nov 2010 17:17:48 -0500

That's good.
I would replace the word "increase" with "increase or decrease".
This would also require an initial value. Let's call it IIW.
I'll propose IIW=10.

--
Jakob Heitz.

> Behalf Of Joe Touch
> Sent: Thursday, November 18, 2010 1:09 PM
> Cc: tcpm; David Borman
> >> The point is that subsequent connections don't do anything 
> different.
> >> If

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 19 Nov 2010 01:47:41 +0100 (CET)

> 	over time if there are no problems

I think this is a very viable idea.

However, I think there is a catch due to which it would need tweaking - there 
should be a signaling mechanism whereby a node on the "conservative" side of 
the connection would be able to signal the node on the "eager" side of the 
connection not to be too aggressive, in case the data flow is mostly from 
from "eager" to "conservative" side.

One might consider the following model for a path:

[peer A] --- [access A] ----- [ backbone ] ---- [access B] --- [peer B]

Where the "access A" and "access B" are the access networks for the peers, and 
the "backbone" is the "big Internet" inbetween.

I am speculating based on experience (please correct me if I am wrong) that the 
congestion would typically be either in [access A] or [access B] cloud.

So in case peer B is a server then storing the losses about the connections on a 
per-something basis would not be productive - because a connection from peer A 
may experience the congestion at access A, whereas connection from peer C would 
not traverse the same access network - so there would not be a congestion.

OTOH if the congestion is in [access B] for both cases then storing the data in 
peer B would indeed make sense.

If the above logic is right, then the peers during the 3whs should communicate 
their memorized values and pick up the minimum of theirs and the communicated by the 
peer.

Therefore both parties know by the end of 3whs the IW value they are going to 
use - and can monitor the initial loss within the connection.

If there happen to be a packet loss during the IW burst - both parties will be 
aware of it.

the result: whether there was a loss or not - will affect the party that had 
the smaller value to begin with - if there was no loss, it means that party can 
use larger value for subsequent connections *if* it had LOSS_THRESH previous 
connections happen without the losses. If there was loss - that party will use smaller 
value for subsequent connections.

I've done a poor man's simulation program (attached) to experiment with this 
idea - it seems more viable in terms of controlling the loss. It assumes a full 
mesh between 10 nodes with nodeX-nodeY backbone capacities potentially unique.

What's more entertaining with this approach is that with sufficiently large 
value of LOSS_THRESH, it relatively converges *regardless* of the initial 
setting for IW.

You can run this code in the following way:

1) Assume access and backbone links can uniformly support IW=10, nonadaptive:

./a.out 10 0 1000 10 11 10 11 | grep round

...
Results from the round: loss: 0, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 0, avg IW size: 10.000000
....

2) Assume access and backbone links can uniformly support only IW=5, nonadaptive:

./a.out 10 0 1000 5 6 5 6 | grep round

Results from the round: loss: 500, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 500, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 500, headroom: 0, avg IW size: 10.000000

3) same as (1), adaptive with the LOSS_THRESH=1000 and IW of 4:

./a.out 4 1 1000 10 11 10 11 | grep round

Results from the round: loss: 2, headroom: 0, avg IW size: 10.020000
Results from the round: loss: 1, headroom: 0, avg IW size: 10.010000
Results from the round: loss: 0, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 0, avg IW size: 10.000000

(loss fluctuates betweeen 0 and 2)

4) same as (2), same adaptive parameters as (3):

./a.out 4 1 1000 5 6 5 6 | grep round

Results from the round: loss: 0, headroom: 0, avg IW size: 5.000000
Results from the round: loss: 1, headroom: 0, avg IW size: 5.010000
Results from the round: loss: 0, headroom: 0, avg IW size: 5.000000
Results from the round: loss: 0, headroom: 0, avg IW size: 5.000000

On to fun stuff with nonuniform distributions:

Assume access links can support 5..8, and backbone 8..15

5) non-adaptive with IW=10:

./a.out 10 0 1000 5 8 8 15 | grep round

Results from the round: loss: 434, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 423, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 415, headroom: 0, avg IW size: 10.000000

6) non-adaptive with IW = 5:

./a.out 5 0 1000 5 8 8 15 | grep round

Results from the round: loss: 0, headroom: 71, avg IW size: 5.000000
Results from the round: loss: 0, headroom: 84, avg IW size: 5.000000
Results from the round: loss: 0, headroom: 66, avg IW size: 5.000000

7) adaptive with IW=10:

./a.out 10 1 1000 5 8 8 15 | grep round

Results from the round: loss: 0, headroom: 12, avg IW size: 5.660000
Results from the round: loss: 0, headroom: 11, avg IW size: 5.670000
Results from the round: loss: 2, headroom: 13, avg IW size: 5.610000
Results from the round: loss: 0, headroom: 26, avg IW size: 5.47000

8) adaptive with IW=5:

./a.out 5 1 1000 5 8 8 15 | grep round

Results from the round: loss: 0, headroom: 11, avg IW size: 5.550000
Results from the round: loss: 0, headroom: 10, avg IW size: 5.670000
Results from the round: loss: 1, headroom: 10, avg IW size: 5.750000
Results from the round: loss: 0, headroom: 3, avg IW size: 5.810000
Results from the round: loss: 2, headroom: 6, avg IW size: 5.800000
Results from the round: loss: 2, headroom: 0, avg IW size: 5.670000

9) (extreme:) - adaptive with IW=100:

./a.out 100 1 1000 5 8 8 15 | grep round

Results from the round: loss: 0, headroom: 8, avg IW size: 5.550000
Results from the round: loss: 0, headroom: 3, avg IW size: 5.600000
Results from the round: loss: 0, headroom: 5, avg IW size: 5.68000

10) "conservative futuristic" - adaptive with IW=5, and access cap between 10 
and 20 and backbone cap between 15 and 35:

./a.out 5 1 1000 10 20 15 35 | grep round

Results from the round: loss: 0, headroom: 54, avg IW size: 14.200000
Results from the round: loss: 0, headroom: 58, avg IW size: 13.750000
Results from the round: loss: 0, headroom: 31, avg IW size: 13.470000
Results from the round: loss: 0, headroom: 53, avg IW size: 13.860000
Results from the round: loss: 0, headroom: 32, avg IW size: 13.590000
Results from the round: loss: 0, headroom: 41, avg IW size: 13.790000
Results from the round: loss: 0, headroom: 62, avg IW size: 14.290000

11) same as (10), "badly converging" - the THRESH being 5:

./a.out 5 1 5 10 20 15 35 | grep round

Results from the round: loss: 20, headroom: 13, avg IW size: 14.140000
Results from the round: loss: 18, headroom: 9, avg IW size: 14.760000
Results from the round: loss: 21, headroom: 4, avg IW size: 14.470000
Results from the round: loss: 18, headroom: 6, avg IW size: 14.420000
Results from the round: loss: 19, headroom: 11, avg IW size: 14.370000
Results from the round: loss: 22, headroom: 11, avg IW size: 14.42000

12) same as (10) with the fixed IW of 10, non-adaptive:

./a.out 10 0 5 10 20 15 35 | grep round

Results from the round: loss: 0, headroom: 398, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 435, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 418, avg IW size: 10.000000
Results from the round: loss: 0, headroom: 422, avg IW size: 10.000000

How to do the signaling during the 3whs is another story - might be either an 
option or some tricks with the TCP window size in 3whs, so far I am not talking 
about it.

Critiques very welcome - as I said, it is a rather non-scientific simulation 
done in half an hour just to test the concept - you'll see some "non-working" 
logic commented out in the code, maybe there is a better approach.

cheers,
andrew

(also, the security side of things on this approach would need to be given a 
thought. There are some interesting attacks lurking with all of the "adaptive" 
logic.)

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 17:30:12 -0800

Hi, Andrew,

On 11/18/2010 4:47 PM, Andrew Yourtchenko wrote:
> However, I think there is a catch due to which it would need tweaking -
> there should be a signaling mechanism whereby a node on the
> "conservative" side of the connection would be able to signal the node
> on the "eager" side of the connection not to be too aggressive, in case
> the data flow is mostly from from "eager" to "conservative" side.

I think the rest of your argument boils down to how the client decides 
to aggregate it's idea of the best IW. That's up to the client to decide 
what it is willing to support - coarse or fine-grained.

An IW=10 as a static proposal is the coarsest possible; anything else we 
could do can't be any worse.

...
> If the above logic is right, then the peers during the 3whs should
> communicate their memorized values and pick up the minimum of theirs and
> the communicated by the peer.

Congestion isn't often bidirectional, so it's unlikely IMO that a peer 
would know what to suggest to its counterpart (if I understand your 
proposal). However, your suggestion is that there could be additional 
input if someone - anyone - knew 'better' what to do. That's fine - 
that's just an out-of-band hint to the algorithm.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2010 19:21:49 -0800

On 11/18/2010 6:26 PM, Andrew Yourtchenko wrote:
> Hi Joe,
...
>> to aggregate it's idea of the best IW. That's up to the client to
>> decide what it is willing to support - coarse or fine-grained.
>
> No. IW of the client does not really matter - it sends the short GET
> request. The server side is the one that may generate congestion by
> having large IW.

It doesn't matter what side you call client or server; they each get an IW.

> So it needs either to remember all the clients somehow, or take a hint.

Right now you assume all remote ends are the same -- IW=3. The proposal 
is to assume they're the same and IW=10. Even if you presume they're the 
same, you can treat them as a group, and if 'most of the time in the 
past' IW=10 caused losses, you can backoff and use IW=5 for future 
connections for a while.

Yes, it's possible to 'learn' this for subsets of remote ends, but 
that's an optimization that can be implemented in whatever granularity 
desired.

> "Average temperature in the hospital" approach (aggregating on the
> server) will not be useful.

Well, that's exactly what we're doing now when we pick a fixed IW 
anyway. So if that's bad, then let's not do it now either ;-)

>> An IW=10 as a static proposal is the coarsest possible; anything else
>> we could do can't be any worse.
>>
>> ...
>> proposal).
> If I connect to http://foo.example/ which sends me an IW=10 and I lose 5
> packets out of that, and I connect to http://bar.example/ which sends me
> an IW=10 and I lose 5 packets out of that, my assumption that if I
> connect to http://baz.example, most chances are that I will also be able
> to get up to 5 packets - so I better give it a hint based on my previous
> knowledge.

If they tell you their IW, you still need to know they actually sent 5. 
You don't know they're lost - they may have never been sent. So IW 
knowledge isn't enough to do anything per se.

Joe

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 19 Nov 2010 03:26:53 +0100 (CET)

Hi Joe,

On Thu, 18 Nov 2010, Joe Touch wrote:

> I think the rest of your argument boils down to how the client decides to 
> aggregate it's idea of the best IW. That's up to the client to decide what it 
> is willing to support - coarse or fine-grained.

No. IW of the client does not really matter - it sends the short GET request. 
The server side is the one that may generate congestion by having large IW.
So it needs either to remember all the clients somehow, or take a hint.

"Average temperature in the hospital" approach (aggregating on the server) will 
not be useful.

> Congestion isn't often bidirectional, so it's unlikely IMO that a peer would 
> know what to suggest to its counterpart (if I understand your proposal).

If I connect to http://foo.example/ which sends me an IW=10 and I lose 5 packets 
out of that, and I connect to http://bar.example/ which sends me an IW=10 and I 
lose 5 packets out of that, my assumption that if I connect to 
http://baz.example, most chances are that I will also be able to get up to 5 
packets - so I better give it a hint based on my previous knowledge.

> However, your suggestion is that there could be additional input if someone - 
> anyone - knew 'better' what to do. That's fine - that's just an out-of-band 
> hint to the algorithm.

well, I've added the proposal you had about adjusting only one side, added an 
extra parameter to the code, and am attaching it.

To save you a compile, just three runs:

1) Static IW=10, access capacity 5..8, backbone capacity 8..15

./a.out 10 0 1000 5 8 8 15 1 | grep round

Results from the round: loss: 423, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 436, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 423, headroom: 0, avg IW size: 10.000000
Results from the round: loss: 426, headroom: 0, avg IW size: 10.000000

2) One-side-adjustment (what I understand is your proposal):

./a.out 10 1 1000 5 8 8 15 1 | grep round

Results from the round: loss: 66, headroom: 10, avg IW size: 6.380000
Results from the round: loss: 52, headroom: 14, avg IW size: 6.180000
Results from the round: loss: 62, headroom: 10, avg IW size: 6.350000
Results from the round: loss: 63, headroom: 9, avg IW size: 6.210000

3) With the hint:

./a.out 10 1 1000 5 8 8 15 0 | grep round

Results from the round: loss: 1, headroom: 32, avg IW size: 5.420000
Results from the round: loss: 0, headroom: 37, avg IW size: 5.350000
Results from the round: loss: 1, headroom: 29, avg IW size: 5.350000
Results from the round: loss: 1, headroom: 41, avg IW size: 5.350000
Results from the round: loss: 0, headroom: 50, avg IW size: 5.290000

So, the "hint" has roughly the same effect on the new proposal as the new 
proposal on the original proposal - of course at the expense of being a bit more 
conservative.

Whether this effect is significant enough and whether the simulation is a good 
enough approximation - I'd be very curious to hear.

cheers,
andrew

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 23:12:13 -0500

> I'll be glad to write this up if people need a more concrete

The problem here is that we still need a default initial window.
Something to use in the absence of this cross connection history.  And,
we don't really know how to develop this history in any sort of concrete
way.  You can write a "concrete proposal", but without some empirical
understanding it'll be more hand wave than concrete.  E.g., what
timescales does the state work at?  What granularity of "recipient"?
What thresholds for increasing the IW?  Decreasing it?  Etc.

All this bears on the prevalence of the opportunity to leverage previous
history in subsequent connections.  It is not hard for me to envision
that all this is going to boil down to being isomorphic with whatever we
choose for the default initial cwnd because we just don't have all that
many opportunities to share the state and/or build up to a appreciably
bigger IW value.

I'd love to see something like this worked out and rigorously
investigated.  But, without some effort it is not a viable alternative
(today).

allman

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Nov 2010 23:24:33 -0500

> The point is that subsequent connections don't do anything

That is absolutely correct---the subsequent connections do not do
anything differently with a static IW.  But, that may well be perfectly
fine.

E.g., the other 999 connections may be going to 999 different
destinations and the shared portion of the path may not be the problem
and those 999 connections may work perfectly.

E.g., the loss in the first connection may have been transient and even
if the connections are going to the same recipient the same process may
not repeat 999 more times.

And, of course, you're right that the first connection might be telling
of things to come.

Your presumption is that there is much opportunity to share state across
connections and that is useful in preventing issues.  But, thus far this
has all been done with cooked up scenarios.  What is the real
opportunity here?

I'd love an adaptive scheme that got us out of the business of doing our
best to make informed guesses about what the IW should be every so
often.  But, before I buy it I'd have to see some evidence that such a
scheme existed that could really in fact get us out of that business.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 18 Nov 2010 23:54:07 -0800

We, authors on IW10, take changing TCP seriously so we proceed with
scientific methods. We did these in live servers and lab tests to
achieve the best possible reality. We took feedbacks from IETF members
to refine our methods after two meetings. In the mean time, Ilpo from
U of Helsinki, and Michael Scharf from Lucent-Alcatel have each done
_independent_ experiments for IW10. None of us find any alarming
fairness issues between IW10/IW3.

For people who are concerned about possible problems of IW10 in
various network scenarios, can they kindly volunteer to help verify
these issues by similar testbed/live experiments? If IW10 turns out to
be a bad idea, we (tcpm/tmrg members) can all learn from these
studies/insights and figure a better solution.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 19 Nov 2010 11:39:40 +0100 (CET)

> It doesn't matter what side you call client or server; they each get an IW.

hm seems like either I still did not describe well enough or I did not 
understand you. Anyway, not a big deal.

> Right now you assume all remote ends are the same -- IW=3. The proposal is to 
> assume they're the same and IW=10. Even if you presume they're the same, you 
> can treat them as a group, and if 'most of the time in the past' IW=10 caused 
> losses, you can backoff and use IW=5 for future connections for a while.
>
> Yes, it's possible to 'learn' this for subsets of remote ends, but that's an 
> optimization that can be implemented in whatever granularity desired.

There can be other options, yes.
However, at least on the surface this one seems to be simple for the impact it 
has. I guess I'll need to write it up in an I-D + do more tests.

> Well, that's exactly what we're doing now when we pick a fixed IW anyway. So 
> if that's bad, then let's not do it now either ;-)

On this you and me are in agreement. This moment is an opportunity to try and 
find something more adaptive.

> If they tell you their IW, you still need to know they actually sent 5. You 
> don't know they're lost - they may have never been sent. So IW knowledge 
> isn't enough to do anything per se.

There will be situations where you can tell, there will be situations where you 
can't.

If you can't tell whether there was no packet loss - do no changes to IW, or do 
them at a fraction of the pace.

cheers,
andrew

---

From: Jerry Chu <[email-masked]>
Date: Fri, 19 Nov 2010 09:13:12 -0800

On Thu, Nov 18, 2010 at 1:16 PM, Joe Touch <[email-masked]> wrote:
> The benefit is that had we used this mechanism then, we wouldn't be asking
> for a new IW now, though.

The problem is that had we used this mechanism we would've quickly found out
it just doesn't work well. It's always easy to hand wave and point to something
never well tested and claim that would've worked better.

Linux has a "dst cache" feature on a per destination IP basis for years.
The problem is that the effectiveness of the dst cache is determined by the
cache hit rate, which is highly dependent on client-server access pattern.
An ftp server serving RFC documents to a handful of clients (e.g., TCP
researchers) may enjoy high cache hit ratio. But consider > 1B users
accessing millions of servers, what kind of cache hit ratio would you expect?

Frustrated by very low cache hits from Linux's dst cache, we went on to
modify dst cache from per dst IP to per dst /24 subnet basis, but it only
showed limited improvement.

Then we discovered out of the small % of cache hits many still suffered
low cached cwnd. We quickly realized simple cwnd cache is not sufficient
- the prior connection must also have sufficient data to grow its cwnd for
the later connection to enjoy a larger IW.

Adding on top of all the above were transient congestions on well
provisioned users causing cwnd to be clamped down, and later connections
to suffer a small IW even though the congestions were long gone (presumably).

BTW,  we've covered this topic in our 1st presentation (see
http://www.ietf.org/proceedings/77/slides/tcpm-4.pdf) and list
discussions
many times. I really hope we don't have to repeat the same topic over
and over again.

Best,

Jerry

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Nov 2010 09:23:02 -0800

Hi, Jerry (et al.),

The logic here is astounding.

We can't use a reactive system because it would make the wrong decision, 
and potentially treat all endpoints the wrong way.

So let's make a static decision that is, essentially, the same as the 
automatic one with particular parameters (A=0, M=1, timescale=infinite, 
aggregation=all). Alternately, for Mark's proposal, that would be (A=+k, 
M=1, timescale=every few years, aggregation=all).

I conclude that:

	- if there is no objective metric for the safety of this
	change, it MUST NOT be deployed

	- if there is an objective metric for its safety, it's surely
	more safe to incorporate that as a test

Even if all the 'auto' system does is backoff to IW=3 if IW=10 causes 
persistent problems, that's MUCH SAFER than anything else.

If, as Mark concludes, an auto system isn't ready for prime time because 
we don't know the impact, then surely we don't have enough info for a 
static configuration of that auto system either.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Fri, 19 Nov 2010 09:24:48 -0800

> But here's a thought:  Any TCP connection with a large IW that has to retransmit or gets an ECN notification during the initial 3-way handshake must clamp down its IW to a small value.  That won't…

Agreed. In fact, RFC3390 already contains a similar clause:

         "If the SYN or SYN/ACK is lost, the initial window used by a
         sender after a correctly transmitted SYN MUST be one segment
         consisting of MSS bytes."

We welcome any contingency that may ease concerns as long as the condition
can be detected during 3WHS. (We have a self-imposed KISS design principle.
See slides 9 of http://www.ietf.org/proceedings/77/slides/tcpm-4.pdf)

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Fri, 19 Nov 2010 09:58:52 -0800

Hi Joe,

On Fri, Nov 19, 2010 at 9:23 AM, Joe Touch <[email-masked]> wrote:
> We can't use a reactive system because it would make the wrong decision, and
> potentially treat all endpoints the wrong way.

We can't use a reactive system like the one you suggested because it just
doesn't work (except in a few limited cases). We are debating on an engineering
issue here and this is IETF so we'll need running code and real deployment
experience. If you can demonstrate your idea in a large server farm environment
without requiring a large amount memory and complexity and actually showing
reasonable performance latency I'll jump on it tomorrow!

I think being conservative to the extreme is like saying "TCP is not
to be touched".
This will likely force application developers frustrated with TCP
performance into
ways that circumvent TCP and cause a lot of more damage dwarfing the one
you've been worried about.

Jerry

> If, as Mark concludes, an auto system isn't ready for prime time because we
> don't know the impact, then surely we don't have enough info for a static
> configuration of that auto system either.
>
> Joe
>
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Nov 2010 10:21:15 -0800

On 11/19/2010 9:58 AM, Jerry Chu wrote:
> Hi Joe,
>
> On Fri, Nov 19, 2010 at 9:23 AM, Joe Touch<[email-masked]>  wrote:
>> Hi, Jerry (et al.),
>>

It's simpler than you're thinking, and doesn't require code in the data 
path:

1) pick an IW s

2) keep stats on whether the IW works
	this can be within every single connection, or done out of
	band using tracers as separate machines

3) lower the IW if it tends to result in retransmits
(possibly even increase it over longer timescales if it doesn't result 
in retransmits)

This can happen on the timescale of days. Weeks even. I'm not talking 
about something that needs to happen in realtime.

The point is to automate the feedback mechanism that Mark and I 
discussed as necessary.

Joe

---

From: Mark Allman <[email-masked]>
Date: Fri, 19 Nov 2010 13:23:09 -0500

Joe-

You are suggesting a mechanism that does two things and trying to argue
that it is strictly more conservative.  One of these aspects seems fine
and the other seems speculative to the point I can't believe you either
don't see it or won't acknowledge it.  Let's take these in turn:

(1) Increasing the IW requires a backstop.

    The concern here is that we could get into a situation where there
    is continual use of a bad initial parameter and that will cause
    some bogosity.

    First, you're not showing this in any sort of concrete way, just
    presuming that it could happen and if it does then its bad enough to
    warrant some mitigation.  We can all dream of pathologies.  But,
    lets just put this aside for the moment and decide that given that
    we cannot fully understand the dynamics of IW10 before quite large
    scale deployment this conservativeness might be a fine design
    principle.

    So, the lets design a backstop.  To flesh out what you said in your
    last email a little, do something like this ...

      - set the default IW to 10 packets

      - if one-third of the connections (regardless of the other
        endpoint) that are able to use IW > 3 lose a packet in the
        initial cwnd then use IW=3 for the next 5min

      - if we use an IW > 3 packets and we lose at least one packet from
        the initial cwnd to some destination X then use IW=3 for
        destination X for the next minute

    The percentage (one-third) and times (1 & 5 minutes) are pulled from
    thin air, maybe they should be something else.  What we can say
    about that scheme is: (a) if we're right about IW=10 then this is a
    no-op on performance, (b) this is a minor bit of complexity and
    state, (c) if you're right that we see persistent issues then this
    will mitigate those issues, (d) this is strictly more conservative
    than a static IW=10 (not "better" necessarily, but more
    conservative) and (e) this arguably errors in the right direction
    when there is an error (i.e., it errors on the side of being less
    aggressive).

(2) We can get out of the business of setting the IW.

    This is the part where your scheme really looks like a solution in
    search of a problem.  You are positing that we can increase the IW
    adaptively by using experience from previous connections.  But, you
    are offering no evidence that we can effectively build that history.
    Jerry offered that they have tried this and found the cache hit rate
    to be quite low.  If this is the case---and his is one data point;
    at least he has that---then we're going to be dependent on the
    default IW and not on the history.

You're conflating two issues that are not intrinsicly coupled.  If you
want a backstop then call for a backstop.  If you want a way to remove
ourselves from the IW decision truck over to the lab and start working.
But, please be clear.

allman

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Nov 2010 10:33:50 -0800

...
...
> ourselves from the IW decision truck over to the lab and start working.

The two are the same argument, as follows:

- either there's an objective metric for 'no harm' or not

	if there is and there's harm, backoff (moderately slowly,
	i.e., over weeks, not minutes)

	if there is and there's no harm, increase over long timescales
	(slowly, in small amounts, over months or years
	- not minutes, fwiw)

if there's no such metric, then we should not be changing anything.

However, if I'm pressed to call the second step "experimental" (which, 
FWIW, is exactly what changing IW=3 to IW=10 in the wild will be - an 
experiment), then fine, I'll be glad if we at least have just the 
backoff part.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Fri, 19 Nov 2010 10:45:30 -0800

On Fri, Nov 19, 2010 at 10:21 AM, Joe Touch <[email-masked]> wrote:
>>> and

Well, it's not as simple as you think because the devil is in the details,
and we had to deal with all the details in order to perform real experiments
before giving up. (See my slides mentioned in previous email.)

> (possibly even increase it over longer timescales if it doesn't result in
> retransmits)
>
> This can happen on the timescale of days. Weeks even. I'm not talking about
> something that needs to happen in realtime.

We've tried both the realtime and the offline schemes (again see my slides).
I've mentioned the problem with the realtime one in my previous email. For
the offline scheme, do you realize how many subnets are their even in just a
region, how much memory it requires, and how much work it takes to try to
import the data into the kernel and make it work efficiently?

Again please show me your "simple" implementation and success experience.

Jerry

>
> The point is to automate the feedback mechanism that Mark and I discussed as
> necessary.
>
> Joe
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Nov 2010 10:48:51 -0800

Hi, Jerry,

The reasons you give for this all being too complex to deploy are great 
reasons we cannot afford to take a risk on IW=10.

If you can't monitor it and tell us whether it works, you're seriously 
asking us to just do it anyway, blindly?

Joe

On 11/19/2010 10:45 AM, Jerry Chu wrote:
> On Fri, Nov 19, 2010 at 10:21 AM, Joe Touch<[email-masked]>  wrote:
>>
>>

---

From: Mark Allman <[email-masked]>
Date: Fri, 19 Nov 2010 14:30:59 -0500

Joe,

I think we can basically agree on the high order bit of a backstop.
But, ...

> 	i.e., over weeks, not minutes)

I can't reconcile this with your example of 1000 connections opened in a
row and if one bumps its head then the rest will and OH MY!  Seemingly
now you're willing to let things bump their head for weeks?  I (a) can't
follow this seemingly moving target you present and (b) think if you're
willing to let 'problems' persist for weeks then it seems you can't
really be all that worried about IW10.

allman

---

From: Aki Nyrhinen <[email-masked]>
Date: Fri, 19 Nov 2010 22:40:41 +0200

All of this makes me wonder.

If we are going to set one initial window for all, we make the
assumption that it works reasonably well for ALL. If we make it dynamic,
why does it suddenly have to be per destination?

Why can't it be just a single variable (or per actual routing table
entry, just to ignore connections to localhost, distinguish local lan
etc) that starts at some value, is incremented once a month and is
decremented if more than whatever number of clients the server owner is
happy to induce IW losses for, experiences them.

The state required to track this is simple enough that it can be
saved/restored at shutdown/boot. Tracking i imagine takes two integers
per possible IW: the number of connection attempts with that particular
IW and another for the experienced losses with that IW. i'm sure clever
people can come up with more compact representations.

I like Joe's proposal and while I agree we don't have any evidence on
how well such a thing would work in practice, I definitely don't think
it is automatically worse than just saying "IW is 10 until the next time
we start fighting over it".

	Aki

---

From: Jerry Chu <[email-masked]>
Date: Fri, 19 Nov 2010 17:38:04 -0800

Hi Joe,

> If you can't monitor it and tell us whether it works, you're seriously

I'm confused - isn't it what we've been trying to do for the past three
IETF meetings, backed with lots of data from both large scale real
experiments and testbed emulation, to tell the world IW10 works?
Or we've just been too subtle? :)

Jerry

>>> about

---

From: Xiangsong Cui <[email-masked]>
Date: Tue, 23 Nov 2010 12:03:51 +0800

Hi,

Firstly I would like to say I like this idea (i.e., increasing initial
window), although I am not sure that is right.

My concern here is, how to evaluate the right IW number? (perhaps, 3 is
good, 10 is better, however, 20 is the right)

I notice that in RFC3390 Increasing TCP's Initial Window, 
there is a statement (in section 7) of 

  "Larger TCP initial windows would not dramatically increase the
   burstiness of TCP traffic in the Internet today, because such traffic
   is already fairly bursty. "

I just wonder, when we increased the IW to 3, how did we get the judgment
that the increase is right? When RFC3390 is implemented, what experience
(positive and negative) do we get?

PS, is a survey on network box (along the path) queue capacity needed?

Thanks and regards,
Xiangsong

> Jerry Chu
> Sent: Saturday, November 20, 2010 9:38 AM
> To: Joe Touch
> Cc: Anantha Ramaiah (ananth); tcpm; David Borman
in
> >> the offline scheme, do you realize how many subnets are their even in
just
> discussed
> >>> as

---

From: Jerry Chu <[email-masked]>
Date: Wed, 24 Nov 2010 17:35:14 -0800

On Thu, Nov 18, 2010 at 4:47 PM, Andrew Yourtchenko <[email-masked]> wrote:
> there should be a signaling mechanism whereby a node on the "conservative"
> side of the connection would be able to signal the node on the "eager" side
> of the connection not to be too aggressive, in case the data flow is mostly
> from from "eager" to "conservative" side.
>
> Where the "access A" and "access B" are the access networks for the peers,
> and the "backbone" is the "big Internet" inbetween.
>
> I am speculating based on experience (please correct me if I am wrong) that
> the congestion would typically be either in [access A] or [access B] cloud.
>
> So in case peer B is a server then storing the losses about the connections
> on a per-something basis would not be productive - because a connection from
> peer A may experience the congestion at access A, whereas connection from
> peer C would not traverse the same access network - so there would not be a
> congestion.

But cwnd in the long run would converge to the fair share bottleneck
b/w regardless
of whether the bottleneck is at the near end or the far end. (Guess I
don't quite
understand your use case.)

>
> OTOH if the congestion is in [access B] for both cases then storing the data
> in peer B would indeed make sense.
>
> communicate their memorized values and pick up the minimum of theirs and the
> communicated by the peer.

There is already a mechanism through which a receiving peer can influence how
much data the send side can burst out. It's called the "advertised
receive window".

>
> Therefore both parties know by the end of 3whs the IW value they are going
> to use - and can monitor the initial loss within the connection.
>
> If there happen to be a packet loss during the IW burst - both parties will
> be aware of it.
>
> the smaller value to begin with - if there was no loss, it means that party
> can use larger value for subsequent connections *if* it had LOSS_THRESH
> previous connections happen without the losses. If there was loss - that
> party will use smaller value for subsequent connections.
>
> idea - it seems more viable in terms of controlling the loss. It assumes a
> full mesh between 10 nodes with nodeX-nodeY backbone capacities potentially
> unique.
>
> 10) "conservative futuristic" - adaptive with IW=5, and access cap between
> 10 and 20 and backbone cap between 15 and 35:
>
> How to do the signaling during the 3whs is another story - might be either
> an option or some tricks with the TCP window size in 3whs, so far I am not
> talking about it.

The additional signal, either explicitly through the previous TCP option
space, or implicit through some window encoding (as you mentioned to me
privately), seems to be adding unnecessary complexity hence violates our
KISS principle IMHO. Just use the advertised receive window described
above should achieve what you want, no?

Jerry

> done in half an hour just to test the concept - you'll see some
> "non-working" logic commented out in the code, maybe there is a better
> approach.
>
> cheers,
> andrew
>
> thought. There are some interesting attacks lurking with all of the
> "adaptive" logic.)
>
>>

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 25 Nov 2010 03:36:03 +0100 (CET)

Hi Jerry,

> understand your use case.)

yes. It's just that at least according to my experimentation that I 
posted, picking the minimum between the two endpoints seems to provide a better 
approximation vs. picking just based on the knowledge of one endpoint vs. just a constant.

Of course it was only a rough sketch FWIW - I just wrote the code to test the 
idea in principle.

> much data the send side can burst out. It's called the "advertised

Yeah - so in this case seems like we should just abandon the IW entirely and 
just use the "advertised receive window" to begin with ?

That seems fine to me except one would need to signal about their ability to 
support new semantic.

Of course, if we use it just as a clamping mechanism then no extra 
signaling needed - but would be cool to not have to revisit this later with the 
new constant for IW.

> above should achieve what you want, no?

It could. The code I played with did not include any specifics on how it is 
communicated. The main idea was to see what happens if two parties can agree on 
an IW during the 3WHS. This can certainly be derived from the advertised window 
size within SYN segments, except one would need a way to figure out which 
semantic to use.

Which implies to me *some* way of signaling. One can set the window value in the 
segments to be an odd number, for example. (again this goes about the case of 
'no upper limit' negotiation. If we talk merely about clamping, the above 
argument is null).

cheers,
andrew