Subject: [tsvwg] 10 IW also for SCTP

From: <[email-masked]>
Date: Mon, 6 May 2013 13:46:34 +0300

Hi,

Experimental RFC6928 proposes to increase initial window for TCP from the 4 segments from RFC3390
to 10 segments.

>From RC2960 to RFC4960, SCTP was augmented to operate with initial window of 4MTUs following RFC3390.
Indeed RFC4460 says the following:

"RFC 2960 was published with the intention of having the same
   congestion control properties as TCP.  Since the publication of RFC
   2960, TCP's initial congestion window size has been increased via RFC
   3390.  This same update will be needed for SCTP to keep SCTP's
   congestion control properties equivalent to that of TCP."

>From a technical perspective there does not seem to be anything in RFC6928 that is particularly specific to TCP compared with SCTP and based on this observation I suppose that it would be equally relevant to think about introducing an equivalent experimental usage of larger IWs, up to 10MTUs, also for SCTP and to have such usage be appropriately specified at some point in time.

I wonder if this is also how the tsvwg sees this ?

Thanks!

BR, Karen

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 6 May 2013 12:57:20 +0200

> I wonder if this is also how the tsvwg sees this ?
One difference between SCTP and TCP is that SCTP uses a Max.Burst parameter. So you won't send 10 packets
without changing also the Max.Burst parameter. In the FreeBSD implementation you can tune the Max.Burst
parameter and the initial window via sysctls.

Best regards
Michael

---

From: <[email-masked]>
Date: Mon, 6 May 2013 14:02:29 +0300

Hi,

>One difference between SCTP and TCP is that SCTP uses a Max.Burst parameter.
>So you won't send 10 packets without changing also the Max.Burst parameter. In
>the FreeBSD implementation you can tune the Max.Burst parameter and the initial
>window via sysctls.
>
>

But even if Max.Burst parameter is set to 4, can more than 4 full size packets be send prior to return of
SACK. Max.Burst only regulate burst, not CWND.

Do you see any reason that SCTP should not follow TCP experimental usage of 10 IWs ?

BR, Karen

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 6 May 2013 14:15:24 +0200

> SACK. Max.Burst only regulate burst, not CWND.
It depends what you consider a burst. Do include an application calling send() in a loop,
you might want to change cwnd by the formula given in section 6.1, D). So you wouldn't send
more than 4 packets...
> Do you see any reason that SCTP should not follow TCP experimental usage of 10 IWs ?
I think you would need to change Max.Burst at the same time. Then you should get the
same behavior. You can do the experiment...

However, Randy did some testing a while ago and reported that using Max.Burst of 4 really
helps to avoid messages being dropped.

Maybe I'm wrong, but IW of 10 just disables CC for short flows. For long living flows,
I'm not sure if the IW makes a huge difference.

Except for the Max.Burst dependency, I don't see a huge difference between TCP and SCTP.

Best regards
Michael

---

From: <[email-masked]>
Date: Mon, 6 May 2013 15:42:07 +0300

Hi,

>> But even if Max.Burst parameter is set to 4, can more than 4 full size
>> packets be send prior to return of SACK. Max.Burst only regulate burst, not
>CWND.
>It depends what you consider a burst. Do include an application calling send() in a
>loop, you might want to change cwnd by the formula given in section 6.1, D). So
>you wouldn't send more than 4 packets...
>>

I believe that section 6.1 D) proposes a way to implement Max.Burst in an internal routine,
 not a way to limit initial CWND to 4 MTUs. 

It says:

.  The limit MAY be applied by adjusting
      cwnd as follows:

      if((flightsize + Max.Burst*MTU) < cwnd) cwnd = flightsize +
      Max.Burst*MTU

      Or it MAY be applied by strictly limiting the number of packets
      emitted by the output routine.

So perhaps you are saying that FreeBSD implementation would effectively limit the IW to Max.Burst.
If so, this would not be the case for all SCTP implementations.

Or do you mean to say that the intention of Section 6.1 D) also was to keep the IW under Max.Burst*MTU.

>I think you would need to change Max.Burst at the same time. Then you should get
>the same behavior. You can do the experiment...
>
>However, Randy did some testing a while ago and reported that using Max.Burst of
>4 really helps to avoid messages being dropped.
>

That is very interesting as this sounds as to go against the TCP experiments that say that using 10 IWs is not an issue in today's networks.
Or perhaps the situations tested were situations where a high, now unused, CWND had been build up ?

>Maybe I'm wrong, but IW of 10 just disables CC for short flows. For long living
>flows, I'm not sure if the IW makes a huge difference.
>
Yes, I think that this is true. The same is the case for TCP.
This does depend, however, on whether one also augments the restart window and possibly
also the window (CWND/2) used after loss recovery. ( If so,
then for both, one should have to follow the prescriptions of RFC6928 of not to increase the CWND in these situations.)

It can be noted that RFC6928 does not speak about allowing the window after loss recovery to be 
higher then 2MTUs (if CWND/2 <= 2MTU). The same is the case for RFC3390, but
nevertheless RFC4960 implements that the window after loss recovery does not go beneath 4 MTUs.

>Except for the Max.Burst dependency, 

I do not see any such dependency from Section 6.1 D).

I don't see a huge difference between TCP
>and SCTP.

Thanks.

BR, Karen

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 6 May 2013 14:57:10 +0200

>      emitted by the output routine.
Before that RFC 4960 says:
   D) When the time comes for the sender to transmit new DATA chunks,
      the protocol parameter Max.Burst SHOULD be used to limit the
      number of packets sent.

Assume that the application sends a lot of messages in a tight loop. Shouldn't Max.Burst
limit the number of packets sent even if IW allows for more? A way to implement this, is
given by changing cwnd as described above.
> If so, this would not be the case for all SCTP implementations.
I think if an SCTP implementation implements the above SHOULD, IW would be limited to Max.Burst.
> Or do you mean to say that the intention of Section 6.1 D) also was to keep the IW under Max.Burst*MTU.
I just want to stress the point that in the case of SCTP there is Max.Burst which
has to be taken into account.

---

From: <[email-masked]>
Date: Mon, 6 May 2013 16:56:54 +0300

Hi,

>-----Original Message-----
>From: Michael Tuexen [mailto:[email-masked]]
>Sent: Monday, May 06, 2013 2:57 PM
>To: Nielsen Karen
>Cc: [email-masked]
>Subject: Re: [tsvwg] 10 IW also for SCTP
>
>>>> But even if Max.Burst parameter is set to 4, can more than 4 full
>>>> size packets be send prior to return of SACK. Max.Burst only
>>>> regulate burst, not
>>> CWND.
>>> It depends what you consider a burst. Do include an application
>>> calling send() in a loop, you might want to change cwnd by the
>>> formula given in section 6.1, D). So you wouldn't send more than 4 packets...
>>>>
>>
>> I believe that section 6.1 D) proposes a way to implement Max.Burst in
>> an internal routine, not a way to limit initial CWND to 4 MTUs.
>>
>> It says:
>>
Yes, Section 6.1 D) in full reads:

      "When the time comes for the sender to transmit new DATA chunks,
      the protocol parameter Max.Burst SHOULD be used to limit the
      number of packets sent.  The limit MAY be applied by adjusting
      cwnd as follows:

      Or it MAY be applied by strictly limiting the number of packets
      emitted by the output routine."

However "When the time comes for the sender to transmit new DATA chunks" is not very well specified. It certainly does not
clearly say that this assumes arrival of SACK and/or that new arrival of data from application does not initiate new transmit.

>Assume that the application sends a lot of messages in a tight loop. Shouldn't
>Max.Burst limit the number of packets sent even if IW allows for more? 

Well as specified it says that in one output routine, no more than Max.burst *MTU SHOULD be send.
If the intention was that at most Max.burst*MTU SHOULD be sent in between SACK arrival and that it is new SACK arrival that
allow for a new output routine to be initiated (or something like that), then I think this deserves to be spelled out.

A way to
>implement this, is given by changing cwnd as described above.
>>

If the intention of the above SHOULD is as revealed in this thread,
 I would appreciate if an Errata was to be issued.

I think that it would also be good to understand the exact motivation for this limitation.

No I am not certain that IW would be limited to Max.Burst even if the above SHOULD was to be interpreted in this manner.
I suppose that the arrival of SACK would allow for new data transmit (or what else should then trick that a new output operation can be 
started). If such SACK acknowledges the first 2*MTUs, I suppose that one, with an IW of 10IW, could end up having 6*MTUs outstanding now.
With IW of 4*MTU, at most 5 *MTUs could be outstanding at this point in time. 

BR, Karen

>I just want to stress the point that in the case of SCTP there is Max.Burst which has
>to be taken into account.
>>
>>>> Do you see any reason that SCTP should not follow TCP experimental
>>>> usage of
>>> 10 IWs ?
>>> I think you would need to change Max.Burst at the same time. Then you
>>> should get the same behavior. You can do the experiment...
>>>
>> That is very interesting as this sounds as to go against the TCP experiments that
>say that using 10 IWs is not an issue in today's networks.
>>> Maybe I'm wrong, but IW of 10 just disables CC for short flows. For
>>> long living flows, I'm not sure if the IW makes a huge difference.
>>>
>> This does depend, however, on whether one also augments the restart
>> window and possibly also the window (CWND/2) used after loss recovery.
>> ( If so, then for both, one should have to follow the prescriptions of
>> RFC6928 of not to increase the CWND in these situations.)
>>
>> It can be noted that RFC6928 does not speak about allowing the window
>> after loss recovery to be higher then 2MTUs (if CWND/2 <= 2MTU). The
>> same is the case for RFC3390, but nevertheless RFC4960 implements that the
>window after loss recovery does not go beneath 4 MTUs.
>>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 6 May 2013 13:42:24 -0600

Karen,

There was some discussion on this back in 2006 in the ladder that
starts with this message:

 http://www.ietf.org/mail-archive/web/tsvwg/current/msg07054.html

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 6 May 2013 21:54:55 +0200

> clearly say that this assumes arrival of SACK and/or that new arrival of data from application does not initiate new transmit.
True. The intention is to mitigate packet bursts. For the network it doesn't matter how the
burst are generated. So the text is generic.
> allow for a new output routine to be initiated (or something like that), then I think this deserves to be spelled out.
Only limiting to Max.burst in the output routing would avoid bursts in response
to incoming SACks. It would not cover bursts initiated by an application calling
send in bursts. I agree, the text is not very explicit, but taking the goal into
account helps...

This goes back to burst mitigation stuff by Mark Allman. For whatever reason it
didn't make it into an TCP RFC...
> I think that it would also be good to understand the exact motivation for this limitation.
Have a look at
http://www.icsi.berkeley.edu/pubs/networking/measuringinteractions05.pdf
> With IW of 4*MTU, at most 5 *MTUs could be outstanding at this point in time. 
I'm not saying that the IW is limited to value of Max.Burst. I'm saying that
the Max.Burst limitation has to be considered and results in not immediately
being able to send the whole IW.

Best regards
Michael

---

From: Mitchell Erblich <[email-masked]>
Date: Tue, 7 May 2013 02:54:53 -0700

Hello,

	Maybe I am wrong, but it seems that everyone with the emails is only considering MTU / MSS sized segments…
	and that any protocol including SCTP would limit an app bursting segments onto a wire,,,

	In the cases of an application initially limiting, aka pacing TCP flow, such as a fair number of wirespeed echo requests/pings,
	then both protocols SHOULD NOT limit the flow and wrt to Allman's ABC, a large number of smaller segments
	SHOULD then generate the sender equiv of ACK splitting…. grow the initial window faster than normal…

	Thus, IMO, if the 10 or so segments are sub-MTU sized (i.e. 128bytes) and sent with some minimal spacing, I would expect to be able to grow the window faster in SS, mitigate any possible DELACKs and/or loss of ACKs….

	And if at the same time, additionally while segments are placed on the wire, a few could be growing in size, additional growth can be obtained (?unfairly obtained?) while probing for the number of outstanding bytes (segments * avg size per segment) that could cause a drop (or can frags occur below MTU sized segs or … ) …. 

	Mitchell Erblich

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 8 May 2013 00:12:24 +0200

> 	SHOULD then generate the sender equiv of ACK splitting…. grow the initial window faster than normal…
I don't understand how using small messages instead of large messages would grow the cwnd faster than
allowed for SCTP. SCTP does ABC.
> 	Thus, IMO, if the 10 or so segments are sub-MTU sized (i.e. 128bytes) and sent with some minimal spacing, I would expect to be able to grow the window faster in SS, mitigate any possible DELACKs a…
Could you explain how this works for SCTP?
> 	And if at the same time, additionally while segments are placed on the wire, a few could be growing in size, additional growth can be obtained (?unfairly obtained?) while probing for the number of…
Not sure I understand this... SCTP is counting the outstanding bytes. This should cover messages of different sizes.

Best regards
Michael

---

From: <[email-masked]>
Date: Wed, 8 May 2013 16:36:25 +0300

Hi Michael,

I think that it is fair to conclude that it is not exactly clear from RFC4960 (or RFC4460) what is meant by the term

"When the time comes for the sender to transmit new DATA chunks" 

in Section 6.1. D) of RFC4960.

The communication from 

http://www.ietf.org/mail-archive/web/tsvwg/current/msg07095.html

also makes it painfully clear that consensus on the exact meaning has not been reached.

I think what at this stage we can conclude only that the Max.Burst parameter of SCTP may 
impact (limit, but not prevent) the ability of an SCTP implementation to exploit an
Initial Window larger than Max.Burst*MTU. But also that the exact limitation that the Max.Burst 
parameter constitutes in this respect will depend on the SCTP implementation.

As to the topic of deploying larger initial window for SCTP we shall see as what will come from such experiments.
I think that the feedback here was that there are no particular reasons why it should not be equally relevant to do such
for SCTP as for TCP. 

BR, Karen

>Sent: Monday, May 06, 2013 9:55 PM
>To: Nielsen Karen
>Cc: [email-masked]
>>>> I believe that section 6.1 D) proposes a way to implement Max.Burst
>>>> in an internal routine, not a way to limit initial CWND to 4 MTUs.
>>>>
>>>> It says:
>>>>
>> However "When the time comes for the sender to transmit new DATA
>> chunks" is not very well specified. It certainly does not clearly say that this
>assumes arrival of SACK and/or that new arrival of data from application does not
>initiate new transmit.
>True. The intention is to mitigate packet bursts. For the network it doesn't matter
>how the burst are generated. So the text is generic.
>>
>>> Assume that the application sends a lot of messages in a tight loop.
>>> Shouldn't Max.Burst limit the number of packets sent even if IW allows for
>more?
>>
>> If the intention was that at most Max.burst*MTU SHOULD be sent in
>> between SACK arrival and that it is new SACK arrival that allow for a new output
>routine to be initiated (or something like that), then I think this deserves to be
>spelled out.
>Only limiting to Max.burst in the output routing would avoid bursts in response to
>incoming SACks. It would not cover bursts initiated by an application calling send in
>bursts. I agree, the text is not very explicit, but taking the goal into account helps...
>
>>>> So perhaps you are saying that FreeBSD implementation would
>>>> effectively limit
>>> the IW to Max.Burst.
>> If the intention of the above SHOULD is as revealed in this thread, I
>> would appreciate if an Errata was to be issued.
>>
>> No I am not certain that IW would be limited to Max.Burst even if the above
>SHOULD was to be interpreted in this manner.
>> I suppose that the arrival of SACK would allow for new data transmit
>> (or what else should then trick that a new output operation can be started). If such
>SACK acknowledges the first 2*MTUs, I suppose that one, with an IW of 10IW,
>could end up having 6*MTUs outstanding now.
>I'm not saying that the IW is limited to value of Max.Burst. I'm saying that the
>Max.Burst limitation has to be considered and results in not immediately being able
>to send the whole IW.
>
>Best regards
>Michael
>>
>> BR, Karen
>>
>>>> Or do you mean to say that the intention of Section 6.1 D) also was
>>>> to keep the
>>> IW under Max.Burst*MTU.
>>> I just want to stress the point that in the case of SCTP there is
>>> Max.Burst which has to be taken into account.
>>>>
>>>>> I think you would need to change Max.Burst at the same time. Then
>>>>> you should get the same behavior. You can do the experiment...
>>>>>
>>>> Or perhaps the situations tested were situations where a high, now
>>>> unused, CWND
>>> had been build up ?
>>>>
>>>> It can be noted that RFC6928 does not speak about allowing the
>>>> window after loss recovery to be higher then 2MTUs (if CWND/2 <=
>>>> 2MTU). The same is the case for RFC3390, but nevertheless RFC4960
>>>> implements that the

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 8 May 2013 20:12:48 +0200

> in Section 6.1. D) of RFC4960.
I agree. 
> parameter constitutes in this respect will depend on the SCTP implementation.
Agreed.
> I think that the feedback here was that there are no particular reasons why it should not be equally relevant to do such
Agreed.

Best regards
Michael