Subject: Re: [tcpm] Increasing TCP's Initial Congestion Window

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 1 Aug 2010 13:14:18 +0100

Hi Nandita, Matt,
 
As mentioned in the session of ICCRG WG (and to not loose this item from
discussion here at TCPM):

I think it would be good, to break out the results you have gathered so
far (flow completion time in high percentiles, across multiple regions)
also into SACK vs non-SACK flows.

Furthermore, as you explained, IW10 (IW16) has only been verified in
your DC with a SACK-enabled sender, which follows Matt's original
intention of using SACK information to recover from lost
retransmissions. No other stack (that I'm familiar with; opensolaris,
{free,open,net}bsd, windows, aix, hpux, is currently doing that, afaik).

My concern here is, that an umbrella recommendation without any strong
remarks about that fact, will led to implementatiors of other stacks to
increase the IW accordingly.

However, the ensuing burst loss will probably cancel out any gains due
to the possibility of fast recovery. (This is why I would like to see
the flow-competion time for non-SACK flows broken out from your
numbers).

Don't understand me wrong here - I feel absolutely positive that SACK
enabled stacks should be encouraged to make use of the information
contained within SACK, to recover fast from lost retransmissions, and
that most stacks should nowadays always use SACK.

However, the Linux Stack is the only one which is currently doing this
(even though there are simple means by which any SACK stack could be
enabled to gain much of the functionality without the fancy scoreboard
implementation of Linux; I think, even the lwIP SACK patch could be made
to do that under a majority of cases).

Perhaps you should write a seperate draft on that aspect, to extend
RFC2018...

Richard Scheffenegger

---

From: Matt Mathis <[email-masked]>
Date: Wed, 4 Aug 2010 16:39:51 -0400

Although I think this is an important point and does need to be
followed up, it is actually inverted relative to the question at hand.

Content providers who are using non-Linux might find some conditions
under which raising IW reduces their performance.  If so they have
grounds to be more cautious about raising IW. In either case, SACK is
(nearly always) work conserving, so the inability to retransmit lost
retransmissions is extremely unlikely to cause additional pain for IW3
users (the fairness question).

If we had done the experimentation with non-Linux, then the
possibility exists that the inability to recover from lost
retransmissions would be protecting IW3 users from our traffic, and
the results couldn't be trusted to apply to Linux.

(I guess this argument does still apply to any other bugs in the Linux
TCP recovery code).

Thanks,
--MM--

-- 
--MM--
The best way to predict the future is to create it.  - Peter Drucker

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 5 Aug 2010 00:17:45 +0100

Hi Matt,

I believe the "inability to retransmit lost retransmissions", which is the current default behavior (except for Linux), DOES permanently cause every receiver pain in the form of lengthy RTO recoveries and long flow completion times. And not necessarily only during slow start and the initial few segments of the flow, but also during steady state, when some other flow's burst causes this effect for some victim flow.

Or, put differently, IW10 may increase that problem (due to a higher probability of lost retransmissions) over IW3 - and you'd be actually penalizing IW10 flows (running into RTO) and the this proposal may see the same fate as ECN deployment.

(Due to some broken algorithms/implementations somewhere else, a very good idea would be blamed).

Therefore I would think, had you done your expirimentation with a "standard" SACK stack, you would have never bothered writing an I-D, as you wouldn't have seen any merit. At least that is what I would have expected as the outcome...

Thus I would like to see some strong recommendations, or an add-on RFC to 2018 /5681, to explicitly recommend lost retransmission detected *before* IW10 is implemented in a stack...

(Perhaps you should have been more insistent back then on how to deal with the wealth of info given by SACK to the sender ;) ).

Richard Scheffenegger

> -----Original Message-----
> From: Matt Mathis [mailto:[email-masked]] 
> Sent: Mittwoch, 4. August 2010 22:40
> To: Scheffenegger, Richard
> Cc: Nandita Dukkipati; [email-masked]; David Borman; Arvind 
> Jain; Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]
> Subject: Re: [tcpm] Increasing TCP's Initial Congestion Window
> 
> Although I think this is an important point and does need to 
> be followed up, it is actually inverted relative to the 
> question at hand.
> 
> Content providers who are using non-Linux might find some 
> conditions under which raising IW reduces their performance.  
> If so they have grounds to be more cautious about raising IW. 
> In either case, SACK is (nearly always) work conserving, so 
> the inability to retransmit lost retransmissions is extremely 
> unlikely to cause additional pain for IW3 users (the fairness 
> question).
> 
> retransmissions would be protecting IW3 users from our 
> traffic, and the results couldn't be trusted to apply to Linux.
> 
> (I guess this argument does still apply to any other bugs in 
> the Linux TCP recovery code).
> 
> Thanks,
> --MM--
> 
> > As mentioned in the session of ICCRG WG (and to not loose this item 
> > from discussion here at TCPM):
> >
> >
> > I think it would be good, to break out the results you have 
> gathered 
> > so far (flow completion time in high percentiles, across multiple 
> > regions) also into SACK vs non-SACK flows.
> >
> > remarks about that fact, will led to implementatiors of 
> other stacks 
> > to increase the IW accordingly.
> >
> > implementation of Linux; I think, even the lwIP SACK patch could be 
> > made to do that under a majority of cases).
> >

---

From: Matt Mathis <[email-masked]>
Date: Wed, 4 Aug 2010 19:59:28 -0400

> I believe the "inability to retransmit lost retransmissions", which is the current default behavior (except for Linux), DOES permanently cause every receiver pain in the form of lengthy RTO recover…
Point taken, although the WG concern is mostly about what happens to
somebody else's flows.   Content providers should always regression
test new OS rollouts, and if by raising IW they hurt themselves (say
because their web page opens 83 concurrent connections), they should
fix something.   Yes, it is possible with a different stack we would
have chosen a different number, and we don't know if the details of
the SACK implementation affect our results much.

> Or, put differently, IW10 may increase that problem (due to a higher probability of lost retransmissions) over IW3 - and you'd be actually penalizing IW10 flows (running into RTO) and the this prop…

It is not the same as ECN at all, because it permits up to IW10 -
users (content providers, OS vendors) can choose their own values.
There should be a sweet spot at 5 or 6 as well (because Fast
Retransmit starts working), which is actually a logical intermediate
step for a large content provider or OS vendor who wants to do
something, but is extra cautious or otherwise not totally confident of
the measurement studies.

> (Perhaps you should have been more insistent back then on how to deal with the wealth of info given by SACK to the sender ;) ).
I actually abandoned the ID for a reason.  Here at Google, I am
considering re-opening it.   2581/5681 actually forbid the correct
answer, which I could not argue when we dropped the FACK ID.   Now I
can make a better argument....  It would get ugly because it conflicts
with many existing standards.   And Fred would not be very happy at
all.

(Search for mathis congestion control easter egg )

Thanks,
--MM--

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 16 Aug 2010 12:31:34 -0700

Hi Richard,

We monitor the "lost retransmissions" reported by Linux
(LINUX_MIB_TCPLOSTRETRANSMIT). They consist of less than <0.5% of all
retransmissions in both IW3 and IW10 flows in our experiments.
Therefore I don't think this feature is critical.

Yuchung

On Wed, Aug 4, 2010 at 4:17 PM, Scheffenegger, Richard <[email-masked]> wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: "Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]" <[email-masked]>
Date: Tue, 17 Aug 2010 10:10:48 -0500

>From: Joe Touch [mailto:[email-masked]]
>Sent: Thursday, July 29, 2010 5:42 AM
>To: Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]
>Cc: Nandita Dukkipati; [email-masked]; David Borman; Arvind Jain; Mathis;
>[email-masked]
>On 7/23/2010 12:37 PM, Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]
>wrote:
>...
>> David and I just talked about this and propose to add a milestone to
>> the charter for late 2011, if the working group agrees, to publish
>> this document on increasing TCP's initial window, based on the large
>> amount of discussion both in the meetings and on the list.  If anyone
>> would object to this, please let us know in the next 2 weeks.
>>
>> One thing I still don't understand, is how we know that the intended
>> status for standards track is appropriate at this point ... from
>> talking to David it sounds like the working group can determine that
>> after deciding to adopt it, and we don't have to have it decided ahead
>> of time.
>
>I don't think that's appropriate process. We need to decide when we
>adopt an item what its intended status will be, IMO.
>
>Otherwise, it's too early to consider adopting it as a work item IMO.
>(that's not to say we can't continue to discuss it, though).

Hi Joe, I checked with Lars on the process question of whether we
can adopt documents with undetermined target status.  He indicated
that it would be reasonable to add a milestone to determine what
the appropriate level to publish the potential document at is, followed
by a milestone to actually publish the document a short time later.

I propose that the first milestone to determine status between
Experimental and Proposed Standard could have a cutoff-date at which
time it would default to the lower option (Experimental) if there
isn't clear consensus and comfort with Proposed Standard at that time.
This assumes there were no problems discovered with deployment at that
level in the meantime.  If so, those problems would probably get
documented as Informational.  Experimental is an appropriate default
since it has nearly been deployed at that level already.

The main question I have, and would like feedback from the authors and
the rest of the working group on, is how far ahead to set that first
milestone to determine status?  Do we need to think about this for a
year (i.e. 3 more meetings)?  Shorter?  Longer?  I would guess after
determining the publication track, we'd only spend a month touching up
the document and last calling it for the 2nd milestone to publish the
document.

Any feedback on this would be appreciated.

--
Wes Eddy
MTI Systems

---

From: Jerry Chu <[email-masked]>
Date: Tue, 17 Aug 2010 23:11:56 -0700

Hi Wes,

Thanks for trying to move this forward quickly. The two-milestone approach
sounds very reasonable to us. Regarding what the appropriate time line
might be for the first milestone, I think this mostly depends on how many
concerns we have successfully addressed vs how many still remaining.
It also depends on how much help folks on this list are willing to provide
to cover the remaining concerns.

To answer the question it may help to recount what we've gotten so far:

1. Results from a number of large scale experiments at Google's data
centers presented at IETF/Anaheim.

http://www.ietf.org/proceedings/77/slides/tcpm-4.pdf

2. Details from the above published in A SIGCOMM CCR paper in July.

http://ccr.sigcomm.org/online/?q=node/621

3. "A Simulation Study on Increasing TCP's IW" presented at
ICCRG/Maastricht by Ilpo Jarvinen on the effect of IW10 over wireless links.

4. Some fairness study by Michael Scharf as part of his PhD thesis that was
sent to the ICCRG list on 7/25/10.

5. Results from another round of experiments plus a testbed study by us in
an attempt to study and address concerns raised at Anaheim.

http://www.ietf.org/proceedings/78/slides/iccrg-3.pdf

Since our presentation was the last one at the ICCRG meeting and the
meeting adjourned shortly after, plus it was the last day of the IETF week,
we did not receive much feedback on how our data address, or even just
partially alleviate the concern on how IW10 flows affect other flows with a
smaller IW, for example. (There was some suggestion related to our testbed.
But unfortunately we have lost our intern back to school so not much has
been done since.)

Thus it will be helpful if people can throw feedbacks/comments/suggestions
at us now on our results so far so we can better sense where we are at
right now and how much further we need to go to get to Proposed Standard.

Thanks,

Jerry

On Tue, Aug 17, 2010 at 8:10 AM, Eddy, Wesley M. (GRC-MS00)[ASRC