Subject: [tcpm] summary of IW thoughts + way forward ? (at least a new subject line to stare at!)

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 12:05:00 -0500

Folks-

I *think* this might be a reasonable summary of the recent IW
threads.

  o Everyone understands that *within* a TCP connection TCP will
    find the "right" cwnd as it always has.  So, regardless of IW
    setting (1, 3, 10, 1000, ...) mistaken initial assumptions will
    be detected and normal congestion control actions taken to
    address these problems in the initial parameters within the
    given flow.

  o Mistakes (and in particular, systemic mistakes) in the initial
    parameter choices across connections from some host are not
    mitigated in any fashion with any static IW.  (Certainly a stack
    *can* cache CC state and leverage it in the future, but it is
    not currently *compelled* to do so.)

  o There is not likely a one-size-fits-all set of initial
    parameters for TCP.

  o We have traditionally addressed this heterogeneity by picking
    initial parameters that are "safe" in the overwhelming majority
    of the cases.

  o And, some folks think we have seen enough evidence that IW=10 is
    likely to be safe initial parameter choice for general use in
    the overwhelming majority of the cases in the Internet.

  o However, choosing initial parameters is difficult because its
    hard to understand the implications of such choices both at
    scale and across a broad range of situations.

  o And, so, some folks are uneasy because there are for sure
    pathologies and even more there is a dynamic that is unknown.

  o So, in some sense this leaves us stuck between two sets of
    opinions and without any good way to bring enough data to bear
    on the question without deciding to just let IW=10 happen and
    then observing things to see if we were right or wrong.

  o In an ideal world we'd not be trying to puzzle through tuning
    initial settings every once in a while, but rather there would
    be some more automated way to go about setting them.

  o Another thought that seems to have been accepted to some degree
    is "go slow".  I.e., not jump from 3 to 10, but take smaller
    steps (which I tried to codify in a static fashion in my
    rockheaded I-D).  This lets us learn as we go without as much
    possibility of over-shoot.

Joe and I argued too much.  On and off the list (yeah- we spared you
some of it!).  But, I think we converged (although I am not sure my
esteemed colleague would frame it as such!).  I think the question
comes down to how much error can we tolerate in the IW and for how
long.

So, off-list I framed this to Joe and Aki.  Its basically Joe's
idea, but with a few more details.  As such, Joe will no doubt
correct it. :)  So, do something like this:

  o On Jul/1 set IW=3 [i.e., RFC5681].  Also, start two counters for
    each possible IW, 1--3: total number of connections using the
    given IW and number of connections using that IW that
    experienced loss in the initial window.

    (Remember, just because a connection can use an IW of X doesn't
    mean it will have enough data to do so or that the advertised
    window will allow, hence, counters for all possible values.)

    This costs IW*2 system-wide counters---which seems like a minor
    state burden (e.g., using 8 byte counters would cost 160 whole
    bytes of memory for IW=10).  Plus, its easy enough to retain across
    boots. 

  o Update the counters for every connection.

    Tracking the initial window size and any retransmits from that
    window is a bit of added complexity, but not onerous and
    especially not when compared with the possible benefits of being
    able to use a larger initial cwnd.

  o On Aug/1 (one month later) consider the counters.  

    If each IW setting shows > 95% success (no loss in the initial
    window) then decide we can tolerate the fallout of IW=3 and bump
    IW to 4, clear the counters and move on.

    If not all IW settings show > 95% success then set the IW to the
    maximum IW that both (1) shows 95% success and (2) all lower
    values of IW have 95% success.  Again, clear the counters.

  o Perhaps the mechanism can be augmented with some minimum number
    of connection attempts---either per IW or in the aggregate.
    And, if we don't meet the min then we don't adjust the IW and we
    don't clear the counters, but we wait until the first of the
    next month to re-consider after gathering more evidence.

  o You could always notice a IW of X is not working well before the
    first of the month and reduce the IW.  That would be strictly
    more conservative and so just fine.

  o This scheme *does not* try to achieve congestion control across
    connections.  If IW=X fails now because of something systemic
    (say talking to some badly connected machine on the other side
    of the world) then it will fail in now+5sec, too.  And, in fact,
    for days or weeks to come.  And, if that is only a small
    fraction of the activity then it might continue to fail across
    all of time.  So, we should think of this as empirically
    arriving at an initial parameter we can live with over the long
    haul and not somehow part of short-term congestion control.

  o This scheme bounds the badness.  For some IW=X we can say that:

      - IWs of 1..X-1 worked in over 95% of the cases during the
        previous month and so while we cannot say for sure that all
        these IWs will work this month we have some direct evidence
        in the relevant context that they likely will.

      - IW=X could fail 100% of the time for up to one month.
        However, since X is at most one more segment than what
        worked in the previous month the imposed loss is likely
        minor.  And, further, it'd happen in the second RTT anyway
        given the usual cwnd increase.  (So, this is well within the
        bounds of the usual slow start overshoot error.)

  o So, we can use the time period (one month in this sketch) and
    the success rate (95% in this sketch) to tune our tolerance of
    badness.  (I picked these numbers out of thin air, basically.
    But, to my eye they look pretty OK.)

  o (Everyone doesn't have to decide on the first of the month.  It
    was just easier to type that way.  It could be "one month since
    last considered".)

  o One can always be more conservative.  

    E.g., factor out some particular host with a known issue and use
    the RFC5681 IW (or lower) with that host and use the
    automatically adjusted IW for the remainder of the hosts.

    E.g., not increase IW per the above unless N successive time
    periods suggest the current IW is OK.

    E.g., use a threshold more than 95%.

    Etc.

  o And, if one doesn't want to maintain the state then the IW in
    RFC5681 stands.

I am hopeful Joe will write it up as I think that perhaps it addresses
much of what we have been talking about and perhaps we can then
converge.

allman

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2010 09:12:27 -0800

Hi, all,

Some additional info. I am in the process of writing this up, but also 
trying to participate in a conference today ;-)

Joe

> Folks-
...
>      able to use a larger initial cwnd.

Note - these are simple +1 counters per connection, and deciding if the 
IW fails happens ONLY if there is a retransmission. In the fast-path 
case, there's only ONE additional opcode required.

>      If each IW setting shows>  95% success (no loss in the initial
>      If not all IW settings show>  95% success then set the IW to the
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 12:18:21 -0500

> Some additional info. I am in the process of writing this up, but
> also trying to participate in a conference today ;-)

Excuses, excuses ...

> Note - these are simple +1 counters per connection, and deciding if
> the IW fails happens ONLY if there is a retransmission. In the
> fast-path case, there's only ONE additional opcode required.

(There may be a bit more complexity in that you need to figure out the
size of the IW to index the right counter.  And, you'll need to keep
track of the sequence range of the initial window temporarily to
correctly classify loss.  But, this is all minor implementation detail.)

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 30 Nov 2010 10:58:44 -0800

I am looking forward to see the doc.

Are we asking for volunteers to execute this plan? or we already have some?

> correct it. :)  So, do something like this:
I am looking forward to Joe's writeup. One early question: has anyone sign
up to execute this months long measurement plan?

---

From: Mark Allman <[email-masked]>
Date: Tue, 30 Nov 2010 14:05:04 -0500

> I am looking forward to Joe's writeup. One early question: has anyone
> sign up to execute this months long measurement plan?

This is not a month's long measurement plan.  This is how you run your
stack.  From now on.  I.e., it puzzles through what IW is right for your
situation (connectivity, mix of traffic, etc.) on a continual
month-by-month basis.  (Or, whatever time period we agree on.)  The
monitoring is the price of using an IW bigger than RFC5681 allows.

allman