Subject: Re: [tcpm] summary of IW thoughts + way forward ? (at least a new subject line to stare at!)

From: "SCHARF, Michael" <[email-masked]>
Date: Wed, 1 Dec 2010 09:28:20 +0100

> This is not a month's long measurement plan.  This is how you 
> run your stack.  From now on.  I.e., it puzzles through what 
> IW is right for your situation (connectivity, mix of traffic, 
> etc.) on a continual month-by-month basis.  (Or, whatever 
> time period we agree on.)  The monitoring is the price of 
> using an IW bigger than RFC5681 allows.

Just a thought about that: Why not to use a limit of the number of
connections, instead of a time period? Or, somethink like: "min(10000
connections, 1 month)" ?

The potential harm of a too large IW somehow scales with the number of
connections that actually use it.

Michael

---

From: Lars Eggert <[email-masked]>
Date: Wed, 1 Dec 2010 14:02:20 +0200

On 2010-11-30, at 21:05, Mark Allman wrote:
> This is not a month's long measurement plan.  This is how you run your
> stack.  From now on.  I.e., it puzzles through what IW is right for your
> situation (connectivity, mix of traffic, etc.) on a continual
> month-by-month basis.  (Or, whatever time period we agree on.)  The
> monitoring is the price of using an IW bigger than RFC5681 allows.

That's an interesting way forward.

I wonder if we could encourage some sort of back channel, so the WG gets some information about what IWs are being settled on over time.

Not-even-half-baked idea: Each month, the stack does a DNS query to iw<number>.<stackid>.tcp.ietf.org and we stat those....

Lars

---

From: Mark Allman <[email-masked]>
Date: Wed, 01 Dec 2010 07:28:29 -0500

> connections that actually use it.

Four answers:

  + I think it'd be fine to make *decrease* decisions every N
    connections.  But, conservativeness makes me think that you
    shouldn't get to increase IW every N connections, but that should be
    based on some amount of time.  (Perhaps if N is not 1K, but 100K
    then this would be basically equivalent.)

  + The whole idea is that we limit the "potential harm" by the way we
    develop IW to begin with.  So, even if IW=X is not working
    particularly well it isn't causing massive harm because IW=X-1
    worked fine the month before.  We can come up with examples where
    this is not the case because a host gets dumped in a new situation
    or something, but in the aggregate this seems like a reasonable way
    to look at "potential harm".

  + As I noted in my email, I think it'd be perfectly fine for a host to
    make a local policy decision that makes it more conservative than
    allowed.  I.e., trying to short-circuit (i.e., before the end of the
    time period) using an IW increase that turns out to not be useful.

  + I don't think we need to over-engineer this to somehow try for
    optimality.  The overall point is to arrive at an initial parameter
    that is reasonable for a given situation.

allman

---

From: <[email-masked]>
Date: Wed, 1 Dec 2010 12:53:20 +0000

>    optimality. 

Agreed, but doesn't it render the entire discussion moot?

---

From: Joe Touch <[email-masked]>
Date: Wed, 01 Dec 2010 06:55:12 -0800

Hi, Lars,

On 12/1/2010 4:02 AM, Lars Eggert wrote:
> I wonder if we could encourage some sort of back channel, so the WG
> gets some information about what IWs are being settled on over time.

I had proposed this early in the discussion, FWIW, but have abandoned 
it...(see below)

> Not-even-half-baked idea: Each month, the stack does a DNS query to
> iw<number>.<stackid>.tcp.ietf.org and we stat those....

There are numerous problems:

- privacy concerns

- opportunity for spoofer poisoning the data (more on that in the doc)

- using a DNS query is a bad idea
	- requires every TCP stack support DNS queries
	- requires a resolution step in addition to the reporting
	- could cause the kernel to block on a timeout

- biased reporting
	if cellphones (e.g.) never do this, we won't know about a
	potentially large percent of endpoints

Much as I would love to know what's going on, we have no precedent for 
doing this in the stack, and I don't want to start with a core protocol.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Thu, 2 Dec 2010 04:59:33 -0800

Mark,

This is an interesting proposal, and if this is what it must take to get to
a WG
consensus, then... so be it. But as I've pointed out before, the optimal IW
is
really a property of the path to the remote peer, not a single global value
per
server.

But for a per destination cwnd cache to work, two conditions must be met:
1) previous connections have sent enough data to grow the cwnd near its
optimal
value, 2) a new connection finds a cache hit in the cwnd cache.

Unfortunately the above two conditions often weren't met in our large scale
tests,
rendering the default IW (i.e., 3) for most of the connections, forcing us
to find
alternative solutions.

One alternative solution we tried briefly to get around the low cwnd cache
hit
problem was to turn the default around to use IW10, unless it's a hit on the
cwnd cache. We dubbed this scheme "negative cache". The rationale is IW10
can only cause damage to a remote peer at most once per server. I wonder
what people think about this scheme.

The per IW global tracking how IW-N has done so far has a flaw in that it
glosses over all the data in a single sum, losing the per-path nature of IW.
Ultimately it seems to be effectively limiting the damage a larger IW may
have to no more than 5% (or whatever threshold) of remote peers. This
may work well for some specific cases, e.g., hosts that are behind slow or
highly multiplex links with insufficient egress buffer. But for the vast
majority
of computing devices on the Internet our large scale data already show that
IW10 potentially hurt << 5% of peers. If our data can be generalized, it
will
mean this active per IW monitoring is mostly moot. E.g., our servers will
likely never fail the IW10 test. Oh, I see it does point to a growth path to
go
beyond IW10. But then have people changed their minds now to tolerate the
potential damage to, e.g., 5% of users?

Also once an IW value is validated for an extended period of time, it should
stick (i.e., no need to continue the validation of the same IW value).

Jerry

On Wed, Dec 1, 2010 at 4:28 AM, Mark Allman <[email-masked]> wrote:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>

---

From: Mark Allman <[email-masked]>
Date: Thu, 02 Dec 2010 09:04:56 -0500

Jerry-

> This is an interesting proposal, and if this is what it must take to
> get to a WG consensus, then... so be it. 

Note, I have seen enough evidence to support IW=10.  For me this isn't a
fallback to "if we can't get 10 we can get something".  For me this is
simply a *better* proposal.

> But as I've pointed out before, the optimal IW is really a property of
> the path to the remote peer, not a single global value per server.

We should forget the notion of an "optimal IW".  As I sketched in the
note at the beginning of this thread,

  o There is not likely a one-size-fits-all set of initial
    parameters for TCP.

  o We have traditionally addressed this heterogeneity by picking
    initial parameters that are "safe" in the overwhelming majority
    of the cases.

I don't think those are radical statements.  To the contrary, I think
they are widely held views.

> The per IW global tracking how IW-N has done so far has a flaw in that
> it glosses over all the data in a single sum, losing the per-path
> nature of IW.

If that is a "flaw" of the proposal I sketched (again, really Joe's idea
at the heart of it) then I claim it is a "flaw" in your proposal, too.
In other words, your proposal (IW=10) and my old RFCs (IW=3) both also
gloss over all situations with a single value and so they too "[lose]
the per-path nature of IW".  I think if you want to flag this as a
"flaw" then you need to tell me how your proposal addresses it.

I think you're just not thinking about this the right way.  In both
proposals we are trying to do *the same thing*: find a reasonable and
general initial starting point for TCP.  You did a lot of work, took a
lot of measurements, etc. to arrive at IW=10 as something you have
posited as reasonable.  The proposed adaption scheme just says we should
not codify *your answer*, but we should codify a *process* that arrives
at a reasonable answer.  And, it is largely the process you have gone
through---i.e., try some new value, measure how it works, decide it is
reasonable.  And, it is largely the process we went through in
developing RFC2414.

> Ultimately it seems to be effectively limiting the damage a larger IW
> may have to no more than 5% (or whatever threshold) of remote
> peers.

*Exactly*.  That is one of the two basic points.  We cannot move on
IW=10 because we have not yet arrived at consensus that it doesn't
somehow "hurt too much" (whatever that means).  So, the idea of adaption
spins things around.  Instead of trying to find that value that is
"safe" for broad use we punt and what we codify is our *tolerance for
problems*.  You can use any IW you want as long as you're within this
tolerance level.

> This may work well for some specific cases, e.g., hosts that are
> behind slow or highly multiplex links with insufficient egress
> buffer. But for the vast majority of computing devices on the Internet
> our large scale data already show that IW10 potentially hurt << 5% of
> peers. If our data can be generalized, it will mean this active per IW
> monitoring is mostly moot. E.g., our servers will likely never fail
> the IW10 test.  Oh, I see it does point to a growth path to go beyond
> IW10. But then have people changed their minds now to tolerate the

And, that's the other point.  Yes, I understand you have gone through
great effort within Google to figure out that for Google IW=10 is fine.
But, who else has?  And, do we want everyone to have to go to such
effort?  And, further, in a few years do you want to go through the
effort again for IW=20?  

What the proposal asks people to do is to stop thinking about this
problem as one of choosing an initial parameter value, but rather think
about it in terms of choosing a process for picking an initial
parameter, as well as choosing a particular tolerance for harm.  Have
people changed their minds?  I don't know.  I have.

> Also once an IW value is validated for an extended period of time, it
> should stick (i.e., no need to continue the validation of the same IW
> value).

Well... perhaps.  I view the validation is O(free) and it isn't hard to
conjure schemes whereby changes to a host's situation could cause the
long-term validated IW to fail (e.g., change of ISP, change of traffic
management policy, etc.).  I understand your world is racks and racks of
hosts that don't change their context, but the world is a heterogeneous
and messy place and this monitoring doesn't seem like a particularly
steep price to me.

allman

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Thu, 2 Dec 2010 16:06:12 +0200 (EET)

> really a property of the path to the remote peer, not a single global value

I don't think anybody questions that, the aim of the proposal is not the 
solve problems that occur for one path, but prevent problems when they 
start to show up for most or significant amount of paths.

There are two independent cases here, one which affect all paths and the 
other which affect an individual path, and this 5% proposal is supposed to 
solve only the first one.

> rendering the default IW (i.e., 3) for most of the connections, forcing us

I don't remember how quickly the entries from the cache were removed in 
your tests but here the time (in cache) is expected to be significantly 
longer in time. Therefore likelyhoods are increased that the particular 
path in use for a single connection has its previous "vote" counted into 
the statistics (of course with a rather negligible share though).

> what people think about this scheme.

So you'd put a flow into this negative cache whenever a problem shows up?
I doubt that a single indication of a loss in IW would really correlate to 
all flows to that particular destination will lose with IW10. ...I also 
doubt that this really works because it is subject to the same low hit 
rate problem and need to replace older entries. ...It would be once per 
some cache recycle time per server at best.

Btw, such a technique would be more conservative and therefore certainly 
allowed if there's green light from the 5% proposal for IW10. So nothing 
prevents you to deploy it if you want in addition to 5% proposal (I doubt 
that provides any real benefits though).

> potential damage to, e.g., 5% of users?

I was thinking that there wouldn't be real and significant problems 
because of losses due to larger IW. ...So I fail to understand your 
position now, why those 5% of users suddently get "damaged"? They might 
get (more) losses but what/where is the damage?

On the other hand, I think that the delay impact is much worse than 
losses, esp. in a case where losses didn't happen, but it is not detected 
by the 5% proposal at all.

I agree with you that for the cases you've measured on Google's servers, 
it is sure to be moot technique, however, I'm not sure if that can be 
generalized to all servers/senders.

> stick (i.e., no need to continue the validation of the same IW value).

I don't agree with you... Also load in the whole Internet would have to 
remain the same if this is to hold... Besides, if the higher IW is 
sustainable, continuing the validation does no harm right?

-- 
 i.

---

From: Jerry Chu <[email-masked]>
Date: Thu, 2 Dec 2010 07:29:47 -0800

Mark,

> "flaw" then you need to tell me how your proposal addresses it.

No, by "flaw" I meant the new proposal is flawed as the old (IW10)
one, and I thought this flaw was why Joe was against the old one
so why the new one is any better now? I'm assuming one reason
why Joe was against IW10 was because it might penalize a small
percentage of users and Joe wanted a solution that worked for all
users. If that's his position then no single, fixed IW beyond some
value will ever work.

Or perhaps you and Joe are now saying penalizing a small percentage
of users is acceptable? But we've already shown in our extensive tests
IW10 only penalize a very small percentage of users.

Or perhaps the active monitoring mechanism you propose is considered
more robust, which I don't disagree? One perhaps the active monitor
can help capturing some special cases, e.g., those hosts behind very
slow link where IW-N with N > 1 is universally bad?

> reasonable.  And, it is largely the process we went through in

Understood. I saw this part later as I said previously "it does point to a
growth path to go beyond IW10".

> people changed their minds?  I don't know.  I have.

Points taken, as long as people feel penalizing 5% of users is an acceptable
cost. I certainly do.

> and messy place and this monitoring doesn't seem like a particularly

Ok.

Jerry

---

From: Mark Allman <[email-masked]>
Date: Thu, 02 Dec 2010 10:45:43 -0500

> Points taken, as long as people feel penalizing 5% of users is an
> acceptable cost. I certainly do.

If there is no "acceptable cost" then the WG should remove this item
From its plate, we should all forget about it and not talk about it any
more.

The adaption proposal codifies this acceptable cost (forget 5%---that
was out of thin air; maybe its 10%, maybe its 2%...I dunno).  It does
not strictly limit the cost, but tries to manage it.  That is, just
because > 95% worked on my machine last month doesn't necessarily mean >
95% will work this month, but last month's data is not for nothin'
either.  

That is a **whole lot** better than what you can say about a static
IW=10 rule.  You can say something like "well, for Google in early 2010
IW=10 worked for > X%" (etc.).  But, how does that extrapolate to my
host?  Or, to mid-2011?  While I think your measurements are suggestive
that IW=10 does in fact generalize to being "safe enough", you clearly
have not convinced the WG yet.  If instead of trying to gather up enough
anecdotes to convince ourselves the specific value is "safe enough" lets
just define what "safe enough" means and leave it at that.

allman

---

From: Jerry Chu <[email-masked]>
Date: Thu, 2 Dec 2010 07:55:03 -0800

Hi Ilpo,

> I don't remember how quickly the entries from the cache were removed in

This is tunable but even if we set the cache to infinity (assuming infinite
amount
of memory) many connections just don't come back to the same server due to
load balancing across large amount of servers.

> the statistics (of course with a rather negligible share though).

In the new scheme a vote will always be counted (not sure what you meant).

> rate problem and need to replace older entries. ...It would be once per

Correct.

> get (more) losses but what/where is the damage?

By damage I meant possibly more losses. Again see my reply to Mark. I'm
saying
why is the new scheme any better than the old (IW10). Guess the answer is
it's
more robust, and providing a dynamic path to higher IW.

> by the 5% proposal at all.

So what can you do about it?

> generalized to all servers/senders.

Understood.

> should

Correct. I'm just thinking to spare some complexity to those simple stack
(e.g.,
running on TinyOS). E.g., at some point IETF can endorse, e.g., IW10
universally.
I guess there is room for refinement but it's not critical right now.

Jerry

---

From: Mark Allman <[email-masked]>
Date: Thu, 02 Dec 2010 11:09:29 -0500

> I'm saying why is the new scheme any better than the old (IW10). Guess
> the answer is it's more robust, and providing a dynamic path to higher
> IW.

Exactly.  If IW=10 is the right answer for your situation then the new
scheme gets you here.  But, it adds robustness if that is not the right
answer.  And, it adds the ability to adapt to larger values in the
future.  To me that is "better".  If you feel that is needless
complexity then we can just disagree.

> Correct. I'm just thinking to spare some complexity to those simple
> stack (e.g., running on TinyOS).

Nothing says a stack *has* to do this.  There is an IW for them.  Its
documented in RFC5681.  Yes, there is a cost for moving to a higher IW.
There is a cost associated with more effective loss recovery via SACK,
too (as but one additional example).  Its a choice whether you want to
pay that cost or not.  (That said, even for TinyOS this seems like a
really trivial cost.)

allman