Subject: Re: [Tsvwg] taking NewReno (RFC 2582) to Proposed Standard

From: Kacheong Poon <[email-masked]>
Date: Fri, 16 May 2003 02:09:11 -0500 (CDT)

Included message from Randall Stewart <[email-masked]>:

>----
>-----10 segments outstanding---->
>  and of course the reciver sends 5 acks (every
>other segment)... but worse case happens and
>you get
>
><----ack5
><----ack4
><----ack3
><----ack2
><----ack1
>
>
>Now the first ack that arrives acknowledges all
>of the 10 segments.. the next 4 are considered
>old and dropped...
>
>Using ABC we would advance our cwnd from 10
>to 11.
>
>But if you apply the max burst you drop to 4.. real
>bummer :-<
>
>However what happens to the network now without
>some sort of burst limit? The sender is now suddenly
>able to spike out 11 segments of data.  And I know
>this can't be a good thing :<

Isn't this the whole point of maxburst?  I guess many
TCP implementations have this feature.  What Sally
pointed out in her mail was that using cwnd to limit
burst was not a good idea.

[ ... ]

>Maybe there is someway we can control the sending without
>killing the cwnd.. but then as Mark Allman as stated.. you get
>two control variables and that seems like it might lead to
>interaction and trouble???

Actually, using cwnd to limit burst is overloading its
semantics and leads to this unwanted interaction.  To
me, cwnd is used to track the "macro" sending rate of TCP.
The fact that TCP cannot send more than cwnd does not mean
that cwnd should also be used for limiting number of segments
sent per ACK, which is "micro" sending rate.

>----

							K. Poon.
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Fri, 16 May 2003 06:38:38 -0500

Kacheong:

I like this idea of a "macro" and "micro" control variables... I don't
know if Mark will like it but it makes sense to me :->

Now, thinking on this.. there is another inter-action that one
must take in to consideration.. can't remember which draft
its in for TCP.. but if one does the "if you don't use the
full cwnd you don't get to increase your cwnd" as we
do in SCTP... this burst limit will interact with that..

aka..

My cwnd says I have room to send 10 segments (as in
my earlier example).. but say a burst limit of 8 stops
me until some other event happens.

Now the "other event" is the ACK for the first two segments
I sent...

The only reason more were not sent is the burst limit not that
the sender did not wish to use the full cwnd...

So I think in this case the sender SHOULD be able to advance
its cwnd by 1... and thus up to 11 .. which would then allow
it to send out 5 more segments...

So I guess the sender needs to track on every send op if the
burst limit was applied.. and if so take that in to account when
determing if the full cwnd was being used or intended to be
used but was stopped by the burst limit...

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 16 May 2003 12:30:53 -0400 (EDT)

> burst was not a good idea.

Yes, but NewReno TCP only uses maxburst after exiting a fast recovery.
SCTP's old implementer's guide (version <= 04) did the same. The change in
the SCTP implementer's guide came about after I brought up the issue that
max burst should always be applied. I didn't (and still don't) see a
reason why max burst should only be applied when recovering from a fast
retransmit. (Randall's example scenario shows that reordering is enough to
need max burst.) Mark Allman suggested, based on Congestion Window
Validation, that max burst and cwnd limitations be unified as they are
currently in the SCTP implementer's guide.

> semantics and leads to this unwanted interaction.

Yes, we thought it was a good way to overload the semantics. I see now
that it is not.

> To me, cwnd is used to track the "macro" sending rate of TCP. The fact
> that TCP cannot send more than cwnd does not mean that cwnd should
> also be used for limiting number of segments sent per ACK, which is
> "micro" sending rate.

Yes, agreed. I think the right thing to do is to apply max burst always.
No matter what, each ack may only trigger a send of at most Max.Burst
packets.

~armando

0--                                                  --0
| Armando L. Caro Jr.      |  Protocol Engineering Lab |
| www.cis.udel.edu/~acaro  |    University of Delaware |
0--                                                  --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 16 May 2003 12:51:23 -0400 (EDT)

> do in SCTP... this burst limit will interact with that..

I don't quite follow you. Are you talking about SCTP or TCP? If SCTP, I
don't think there is a problem. If TCP, I think ABC solves the problem.
Maybe I am missing something...

> the sender did not wish to use the full cwnd...

I follow up to here.

> it to send out 5 more segments...

Isn't the ack information for the first two segments redundant at this
point? Why would the cwnd increase?

> used but was stopped by the burst limit...

I see what you are getting at, but I don't think it is a problem with SCTP
or TCP w/ ABC. The first ack (ack 5) that arrives in your reordering
example acks all the outstanding data and increases the cwnd. The
subsequent acks (ack 4, ack 3, ack 2, ack 1) don't increase the cwnd, but
allow the sender to send more data if max burst prevented the full cwnd to
be used previously.

Does this make sense, or am I missing something?

~armando

0--                                                  --0

---

From: Randall Stewart <[email-masked]>
Date: Fri, 16 May 2003 13:37:58 -0500

Armando;

I see from your response my first example was a
bad one to use for this.. My point is there IS some cases
where you will have a bad interaction...

Heres a different example:

------data-1-------->
------data-2--------->
-----data-3---------->
-----data-4--------->
-----data-5--------->
-----data-6--------->
-----data-7--------->
-----data-8--------->
...
-----data-10------>

(peer is acking every 5th packet.. as Sally points out not
  all implementations ack every other...)

XlostX<---------ack-5-------
<---------ack-10----
Cwnd = 11. (add one with ABC)

------------data-11---->
------------data-12---->
------------data-13---->
-----------data-14----->
<max burst triggers>

               200ms timer expires
  <--------ack-14-------

So here we are... a ack arrives but we are not using
the full cwnd (11) only 4 outstanding... so ABC or SCTP
says you are not using the whole cwnd.. no advance
and you get to send maxburst more, wait for the
200ms timer.. and are now stuck.

I am sure there are other examples one can come up with.. but that
is just a quick one...

R

---

From: Randall Stewart <[email-masked]>
Date: Fri, 16 May 2003 14:28:12 -0500

Armando L. Caro Jr. wrote:
> Randall,
> 
> Ok, I see the problem now, but I am still trying to decide whether that is
> something we should be concerned about. Protection against this type of
> interaction seems tricky to me, and I don't know how common it is for
> receivers to ack more than 2 packets. Maybe their are "smart" devices
> which collapse acks so that the sender sees an ack for more than two
> packets??

I think there are possibly more ways you can get this situation than
just if the peer is acking every 4th packet... even in the every other
packet catagory I think you can hit this...

> 
> I guess what I am getting at is... Unless their is straight forward way to
> avoid this problem, I don't _think_ its worth addressing it. Is it?
> 

I think there is a straight forward solution that is quite simple...
You set a flag when maxburst is imposed and clear it if it
is not in your sending routine.

Then when SACKs arrive.. speaking for SCTP (same would
work for TCP ABC I think)

1) Make sure the cum ack as moved forward .. if not no advance
2) See if the full cwnd was being used OR max burst was in
     effect at the last send. If either of these are true we ARE
     allowed to do the cwnd advancement.

Its a simple flag/boolean and checking it along with are you using
the full cwnd...

R

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 16 May 2003 15:22:19 -0400 (EDT)

Randall,

~armando

0--                                                  --0

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 May 2003 00:08:12 -0400

Folks-

I have a few comments on this bursting discussion.  The caveat is
that these are all pretty general thoughts.  I have not read the
recent newreno draft (yet) and so I do not know what it says
exactly.  But, I have some thoughts on the two anti-bursting schemes
thrown out already and would then like to throw out a third
variant. 

My thoughts:

  * There are some situations when TCP (SCTP) can burst a good
    amount of segments at line rate.

    In some situations this is OK and everything works itself out.

    In some situations this is not OK and the burst causes drops
    (and the resulting loss recovery and cwnd reduction).

    So, bursting is sort of a mixed bag.

  * Do we need some sort of general burst limiting mechanism?

    Beats me.

  * But, for argument's sake lets say that we're going to go down
    that path.  Two scheme have been thrown out:

    (1) maxburst controller

	count = 0
	while ((outstanding_data < cwnd) && (count < maxburst))
	    send_segment ()
	    outstanding_data++
	    count++

    (2) reseting cwnd

	if ((outstanding_data + maxburst) < cwnd)
	    cwnd = outstanding_data + maxburst
	while (outstanding_data < cwnd)
	    send_segment ()
	    outstanding_data++

  * To my way of looking at things, scheme (1) uses two controls for
    how much data can be sent into the network.  That is, we can
    either be limited by cwnd or we can be limited by maxburst.

    And, since there are two controls there are interactions between
    them that must be taken into account -- in specs and code.  And,
    there are things that are currently speced that will break.
    That is, TCP can be running in a state where there is less than
    cwnd on the network because it is limited by maxburst instead of
    cwnd and the specs/code do not envision the case and therefore
    cannot be exepected to do something intelligent.  

    An example of where this breaks things: The limited transmit
    spec says that on dup acks TCP can inject at most cwnd+(2*MSS)
    of new data into the network.  The hope was that this means that
    we can send data on the first two dup acks.  If maxburst has
    limited the amount of outstanding data to something less than
    cwnd then limited transmit now allows the sending of more than
    two new segments on dup ACKs.  That might be OK.  But, I think
    that is clearly at odds with the spirit of RFC 3042.  I think
    there are likely quite a number of examples of unintended
    interactions/consequences.

    But, that can be fixed... we can set a flag, add some
    conditionals, some temporary variables, write some new text and
    plug up these things right up...

  * On the other hand, scheme (2) has only a single controller.  We
    are always limited by cwnd -- and, maxburst is just an input to
    this controller.  On the surface, the above snippets of code may
    look the same, but they are not.  If you take the limited
    transmit case again... LT has no problem under scheme (2)
    because the cwnd is now tracking the amount of outstanding data.
    All of our code and our specs this to be the way things are.
    So, this works nicely.

    However, Sally has noted some nasty examples whereby scheme (2)
    pulls down the cwnd when we'd really like it to not do that.  I
    buy those and am sympathetic.

  * So, next we look at what the long-term differences in the above
    bits of code.  

    Scheme (1) limits how much can be sent to maxburst packets at a
    time.  But, cwnd is retained.  So, conceivably we could "catch
    up" to cwnd over time.  That is, we burst less at first but then
    a bit more for a few ACKs until we end up having
    "outstanding_data == cwnd" again.

    Scheme (2) on the other hand, sacrifices cwnd space
    immediately.  TCP must then build it back up over time per
    normal congestion avoidance.

    My guess is that you could show that scheme (1) always gets back
    to outstanding_data == orig_cwnd faster than scheme (2) (or,
    maybe equally fast in some cases).

  * Now let me take a minute and say that (a) I do not care for the
    downsides of scheme (2) that Sally illustrated and (b) I think
    that scheme (1) is equally bogus because it adds a second
    control for the amount of outstanding data -- and no end of
    headaches will come of it.  Also, I do not buy that because TCP
    sends "micro" and "macro" bursts that we should have two
    controllers for these sending patterns.  It makes no sense to me
    since in the end they control the same thing: the amount of data
    that can be dumped into the network.

  * So, the question I am left puzzling over is: can we combine
    schemes (1) and (2) in a sensible way to avoid the downsides
    that Sally has shown while also avoiding a second control on
    outstanding data?

  * One more observation: If maxburst is 4, we use scheme (1), we
    use delayed ACKs, cwnd = outstanding_data = 10 segments and we
    get an ACK that covers 10 previously unACKed segments (because
    of dropped ACKs) the events will look like this:

	get ACK for 10 segments; send 4 segments
	    (outstanding = 4; cwnd = 10)
	get ACK for 2 segments; send 4 segments
	    (outstanding = 6; cwnd = 10)
	get ACK for 2 segments; send 4 segments
	    (outstanding = 8; cwnd = 10)
	get ACK for 2 segments; send 4 segments
	    (outstanding = 10; cwnd = 10)
	get ACK for 2 segments; send 2 segments
	    (outstanding = 10; cwnd = 10)

    I.e., it's slow start.

  * So, what if we introduce a scheme (3), which is really just a
    riff of scheme (2):

    if ((outstanding_data + maxburst) < cwnd)
        if (ssthresh < cwnd)
	    ssthresh = cwnd
        cwnd = outstanding_data + maxburst
    while (outstanding_data < cwnd)
        send_segment ()
        outstanding_data++

    In other words, we clamp cwnd to limit the burst, but we also
    set ssthresh in such a way that we can slow start to climb out
    of the cwnd reduction.

    It is late, but I think this gives us the good points of (1) and
    (2) -- that is, one controller with the same basic pattern of
    sending given by scheme (1).

Thoughts?

[Also, with appologies to Joe Touch & Amy Hughes --- who probably
 had all this figured out long ago (but, their exact ideas have
 fallen out of my head.]

allman

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 May 2003 09:33:04 -0400

Randall noted that my code was a bit confusing because it lacks
brackets.  I have been programming python lately and so the
independation just Made Sense to me.  In any case, just for clarity,

>     while (outstanding_data < cwnd)

was meant to be:

    {
        {
	    ssthresh = cwnd;
        }
    }
    {
        send_segment ()
        outstanding_data++
    }

If it caused you confusion, sorry!

allman