Subject: [Tsvwg] sctp retrans count

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 11:22:31 +0530

Hi,

In RFC 2960 section 8.1, says ,

   An endpoint shall keep a counter on the total number of consecutive
   retransmissions to its peer (including retransmissions to all the
   destination transport addresses of the peer if it is multi-homed).
   If the value of this counter exceeds the limit indicated in the
   protocol parameter 'Association.Max.Retrans', the endpoint shall
   consider the peer endpoint unreachable and shall stop transmitting
   any more data to it (and thus the association enters the CLOSED
   state).  In addition, the endpoint shall report the failure to the
   upper layer, and optionally report back all outstanding user data
   remaining in its outbound queue. The association is automatically
   closed when the peer endpoint becomes unreachable.

   The counter shall be reset each time a DATA chunk sent to that peer
   endpoint is acknowledged (by the reception of a SACK), or a
   HEARTBEAT-ACK is received from the peer endpoint.

In case where HEART BEAT is enabled for all the paths, is it necessary 
for one to reset the counter for reception of SACK. Or is it that, this 
provision is made for those paths where HEARTBEAT is not enabled.?
Please clarify ...!!

regards,
devayya

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 19 May 2006 01:40:18 -0600

devayya,

devayya wrote:                                                     (Fri, 19 May 2006 11:22:31)
>       The counter shall be reset each time a DATA chunk sent to that peer
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>       endpoint is acknowledged (by the reception of a SACK), or a
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>       HEARTBEAT-ACK is received from the peer endpoint.

I think you answered your own question above: the counter shall be reset each
time a DATA chunk sent to that peer endpoing is acknowledged by the reception
of a SACK.  Whether hearbeat is enabled for any destination has no bearing on
the fact that the counter is reset when a SACK is received that acknowledges
a DATA chunk.  If a heartbeat is acknowledged, it is also reset.

--brian

>    for  one  to  reset  the counter for reception of SACK. Or is it that,
>    this  provision  is  made  for  those  paths  where  HEARTBEAT  is not
>    enabled.?
>    Please clarify ...!!
>    regards,
>    devayya

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 14:06:51 +0530

Brian,

In our present SCTP implementation,  when we reset the restransmission 
count after reception of SACK we faced problem of path status not 
getting updated properly. In case of sctp multi-homing the switch over 
from primary path to secondary path was not happening properly, due to 
path status not getting updated properly.
However when we removed the resetting of retrans count, there was no 
problem, the path status and switch over was happening properly. But 
with this there is violation of the rfc statement. So could you suggest 
some solution as to how do we solve this problem.

Just for your info we are using SCTP stack developed in house as per rfc 
2906, working on Lynx OS.

Thanks and regards,
devaya

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 14:20:43 +0530

Sorry the rfc number is 2960.
Thanks,
devayya

devayya wrote:

> with this there is violation of the rfc statement. So could you 
> suggest some solution as to how do we solve this problem.
>
> Just for your info we are using SCTP stack developed in house as per 
> rfc 2906, working on Lynx OS.
>
> Thanks and regards,
> devaya
>

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 19 May 2006 11:09:55 +0200

Hi devayya,

can describe your problem in more detail?

Best regards
Michael

On May 19, 2006, at 10:50 AM, devayya wrote:

>> In our present SCTP implementation,  when we reset the  
>> restransmission count after reception of SACK we faced problem of  
>> path status not getting updated properly. In case of sctp multi- 
>> homing the switch over from primary path to secondary path was not  
>> happening properly, due to path status not getting updated properly.
>> However when we removed the resetting of retrans count, there was  
>> no problem, the path status and switch over was happening  
>> properly. But with this there is violation of the rfc statement.  
>> So could you suggest some solution as to how do we solve this  
>> problem.
>>
>> Just for your info we are using SCTP stack developed in house as  
>> per rfc 2906, working on Lynx OS.
>>
>> Thanks and regards,
>> devaya
>>
>>> devayya, devayya wrote: (Fri, 19 May 2006 11:22:31)
>>>> Hi, In RFC 2960 section 8.1, says , An endpoint shall keep a  
>>>> counter on the total number of consecutive retransmissions to  
>>>> its peer (including retransmissions to all the destination  
>>>> transport addresses of the peer if it is multi-homed). If the  
>>>> value of this counter exceeds the limit indicated in the  
>>>> consider the peer endpoint unreachable and shall stop  
>>>> transmitting any more data to it (and thus the association  
>>>> enters the CLOSED state). In addition, the endpoint shall report  
>>>> the failure to the upper layer, and optionally report back all  
>>>> outstanding user data remaining in its outbound queue. The  
>>>> association is automatically closed when the peer endpoint  
>>>> becomes unreachable. The counter shall be reset each time a DATA  
>>>> chunk sent to that peer
>>> I think you answered your own question above: the counter shall  
>>> be reset each time a DATA chunk sent to that peer endpoing is  
>>> acknowledged by the reception of a SACK. Whether hearbeat is  
>>> enabled for any destination has no bearing on the fact that the  
>>> counter is reset when a SACK is received that acknowledges a DATA  
>>> chunk. If a heartbeat is acknowledged, it is also reset. --brian
>>>> In case where HEART BEAT is enabled for all the paths, is it  
>>>> necessary for one to reset the counter for reception of SACK. Or  
>>>> is it that, this provision is made for those paths where  
>>>> HEARTBEAT is not enabled.? Please clarify ...!! regards, devayya

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 15:18:18 +0530

Hi Michael,

In our present test set up, we have two paths from originating side and 
two from the destination, for single  association to support SCTP 
multi-homing. When there is no traffic flowing between these paths, and 
when the primary path is disabled, the secondary  path becomes primary 
path.
However when there is traffic flowing(5000 msus/sec), when the primary 
path is disabled it takes some time for the secondary path to become 
primary and carry the traffic. By this time upper layer timer, in our 
case m2pa delay ack timer expires(approx 2 sec), and there is traffic loss.

the default values of sctp protocol parameters as follows,

    3,      /* ASSOCIATION MAX. RETRANSMISSIONS */
    3,      /* PATH MAX. RETRANSMISSIONS */
    8,      /* MAX INITIALIZATION RETRANSMISSIONS */
    20,     /* SCTP HEARTBEAT INTERVAL */
    500,    /* INITIAL RTO */
    200,    /* MINIMUM ALLOWABLE RTO */
    10000,  /* MAXIMUM ALLOWABLE RTO */
    1,      /* RTO ALPHA */
    2,      /* RTO BETA */

However when we removed the resetting of retransmission count when we 
received SACK the switch over from primary to secondary was happening 
more faster and so there was no traffic loss.
Please suggest some solution.

Thanks and regards,
devayya

Michael Tuexen wrote:

>>> no problem, the path status and switch over was happening  properly. 
>>> But with this there is violation of the rfc statement.  So could you 
>>>>> counter on the total number of consecutive retransmissions to  its 
>>>>> peer (including retransmissions to all the destination  transport 
>>>>> addresses of the peer if it is multi-homed). If the  value of this 
>>>>> counter exceeds the limit indicated in the  protocol parameter 
>>>>> 'Association.Max.Retrans', the endpoint shall  consider the peer 
>>>>> endpoint unreachable and shall stop  transmitting any more data to 
>>>>> it (and thus the association  enters the CLOSED state). In 
>>>>> addition, the endpoint shall report  the failure to the upper 
>>>>> layer, and optionally report back all  outstanding user data 
>>>>> closed when the peer endpoint  becomes unreachable. The counter 
>>>>> shall be reset each time a DATA  chunk sent to that peer
>>>>
>>>> I think you answered your own question above: the counter shall  be 
>>>> reset each time a DATA chunk sent to that peer endpoing is

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 15:44:08 +0530

One small correction for the default value, since there are two paths,

Association retransmission count is 6 and path retransmission count 3.

Also, I was just wondering, if the multiple of retransmission timer and 
number of path should it be less than the upper layer delay ack timer, 
in our case m2pa delay ack timer? Please clarify.

Thanks,
devayya

devayya wrote:

> Hi Michael,
>
> In our present test set up, we have two paths from originating side 
> and two from the destination, for single  association to support SCTP 
> multi-homing. When there is no traffic flowing between these paths, 
> and when the primary path is disabled, the secondary  path becomes 
> primary path.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 19 May 2006 04:35:49 -0600

devayya,

Please see comments below...

devayya wrote:                          (Fri, 19 May 2006 15:18:18)
>     3,      /* PATH MAX. RETRANSMISSIONS */

If you have two paths, the association will be brought down when the primary
fails due to retransmissions with the above settings.  In general, assoc_max_
retrans would be better set to N * path_max_retrans, where 'N' is the number
of paths (in your case 6 instead of 3).  In that case the switchover from the
primary to the secondary will occur before the association peg takes the
entire association down.

The recommended values are 10 and 5 respectively, that works well for 2 paths
because 10 == 2 * 5.

>     20,     /* SCTP HEARTBEAT INTERVAL */

This RTO_initial might be too low.

>     200,    /* MINIMUM ALLOWABLE RTO */

Another suggestion: minimum RTO of 200 ms is not good if the other end uses
the recommended SACK delay of 200 ms.  Under light traffic you will have
excessive retransmissions on a clean link.  In general RTO_min should be no
less than 200 ms for SACK delay plus an RTT value that provides 90% confidence
that it will not be exceeded under normal operation.

>     10000,  /* MAXIMUM ALLOWABLE RTO */

Another suggestion: you migh consider reducing the RTO_max as the value of
10 sec is much larger than the delay that is acceptable to M2PA if your T7
is running 2 secs.  Because RTO is doubled on packet loss or retransmission,
you might consider lowering this to 4000 ms (or 4 sec).

> Please suggest some solution.

It think that you may have removed resetting of the path count rather than
resetting of the association count.  That would have the effect that you
cite; however, you are better to double your assoc_max_retrans count instead.

Hope that helps.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 19 May 2006 04:44:23 -0600

devayya,

Well that kill my previous note...  However, if you are not resetting
the path count on SACK you are simply causing the paths to switch
more frequently.  Setting path_max_retrans to 1 might have the same
effect.  Setting your RTO_max down should also help.  If you are
running on a clean link so far, with the values you have, your RTO
will be 200 ms and it will take longer than 200 ms + 400 ms + 800 ms
+ 1600 ms (with path_max_retrans of 3 it will take 4 retransmissions
with a doubling of RTO each time) or more than 3000 ms (3 seconds) to
switch over which is larger than your 2 second M2PA timer.  Simply
reducing path_max_retrans to 2 would reduce the switchover to about
1400 ms which is beneath the 2 second M2PA timer.

--brian

devayya wrote:                             (Fri, 19 May 2006 15:44:08)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 19 May 2006 12:49:03 +0200

Hi devayya,

the problem is not with SCTP, but with the configuration of
the parameters which does not work with your application's
settings.

Do not modify SCTP, but tune the parameters. Please understand
the impact of the SCTP parameter settings and tune them. Brian
gives one possible combination.

Best regards
Michael

On May 19, 2006, at 12:44 PM, Brian F. G. Bidulock wrote:

>>> case m2pa delay ack timer expires(approx 2 sec), and there is  
>>> traffic
>>> loss.
>>>

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 16:53:14 +0530

Brian,

Thanks for that calculations and update.
Does this mean that suppose we decide to change our default SCTP 
parameters to the one recommended in rfc 2960, Should we increase the 
m2pa T7 (delay ack timer) timer value to a higher value?

Presently the m2pa T7 timer value of 2 seconds is the maximum value that 
I ahd mentioned that can be provisioned in our setup. By default T7 
timer is only 1 second.

Thanks and regards,
devayya

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 19 May 2006 05:49:25 -0600

devayya,

M2PA T7 has an impact on MTP3 running above it.  A stock MTP Level 3
cannot accept T7 greater than 2 seconds without serious impact on MTP's
ability to changeover links correctly.  For example, Q.704 T1 and T2 are
1 to 2 seconds.  MTP Level 3 expects that after 2 seconds (or so) all of
the messages for a given link have cleared or the link has failed.
Increasing M2PA T7 to greater than 2 seconds breaks this condition and
would allow old messages to persist in the network.  This could increase
MTP Level 3's duplicated message probability above what is an acceptable
quality of service for MTP management or MTP Users.  Therefore, M2PA T7
should not be increased above 2 seconds without giving consideration for
MTP Level 3 timers and the quality of service expected by MTP Users.

I have seen adding delays into links in the same manner as increasing T7
cause cascading failure and full isolation between STPs.  When
increasing T7, buffer thresholds should be reduced drastically.  For
example, you say that you are running 5000 msus/second.  If you are
running 12-bit sequence numbers (ala Q.703 Annex A) only 4096 MSUs can
be oustanding and more than 1 second ack delay will cause the RTB to
fill and TB occupancy to increase.  If you are running 24-bit sequence
numbers (ala Q.2210), the RTB will not be force to fill due to the
sequence number window, however, say 4 seconds could cause 20000 MSUs to
be waiting for retransmission.  Performing buffer collection and
retransmission on an alternate link of this many MSUs could cause
congestive failure of the alternate link which can cascade to the
failure of all links between two signalling points.  It is better to
keep T7 low and fail links before buffers fill.

M2PA will likely not reliably run with a stock MTP Level 3 over anything
but an engineered newtork.  Care must be taken that M2PA provides the
same link reliability to MTP Level 3 as it expects from other links and
should fail the link otherwise.

--brian

devayya wrote:                                 (Fri, 19 May 2006 16:53:14)
>    Presently  the  m2pa  T7 timer value of 2 seconds is the maximum value
>    that  I ahd mentioned that can be provisioned in our setup. By default
>    T7 timer is only 1 second.
>    Thanks and regards,
>    devayya

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: devayya <[email-masked]>
Date: Fri, 19 May 2006 17:52:16 +0530

Hi,
According to our implementation, during heavy traffic, the messages 
received are buffered including SACK, until they are processed..
Now consider, during the processing of each of the SACK, the primary 
path is made down. Now since the SACK processing is going on and 
simultaneously HEARTBEAT is tried on the already down primary path, each 
time the buffered SACK is processed for the previous primary path, the 
retransmisssion count is reset, which would have otherwise increased to 
1 or 2 by HEARTBEAT.
So untill all the buffered SACK is processed for the already down path 
is done, the path failure of primary is not detected. There will be 
delay in detection of path failure and switch over to secondary. Untill 
this happens the Upper layer would have expired his timer and taken link 
to out of service.
Please clarify....

Thanks and regards,
devayya

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 19 May 2006 06:49:34 -0600

devayya,

I think you need to adjust your buffer sizes more appropriately.  When
you buffer a message for later processing and it has already been assigned
to a Stream (socket), subtract if from the receive window before buffering.
If you have problems ahead of Stream (socket) assignment, you likely need
to drop the size of all your receive buffers.

Other options are: faster processor, more suitable (non-real time) task.
Deadline scheduled tasks are especially poor when faced with busy periods
or event showers.

But now we are talking of topics on which many good textbooks are written.

--brian

devayya wrote:                            (Fri, 19 May 2006 17:52:16)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Sat, 20 May 2006 04:37:44 -0600

devayya wrote:

> simultaneously HEARTBEAT is tried on the already down primary path, 
> each time the buffered SACK is processed for the previous primary 
> path, the retransmisssion count is reset, which would have otherwise 
> increased to 1 or 2 by HEARTBEAT.
> delay in detection of path failure and switch over to secondary. 
> Untill this happens the Upper layer would have expired his timer and 
> taken link to out of service.
> Please clarify....

This looks more like a matter of implementation then clarification.  If 
you want fast failover, fast error recovery, or low message delivery 
latency, buffering SACKs more than absolutely necessary is an unusually 
bad idea.  One may be able to split SACK processing into low and high 
priority phases, as well as aggregate SACKs under load, but certain 
critical consequences of a SACK need to take place as soon as possible, 
*especially* on a multi-path configuration.

 - Mark B.

---

From: devayya <[email-masked]>
Date: Tue, 23 May 2006 15:10:10 +0530

Hi,

The calculation of a_rwnd is dependent on buffer space available in IP 
for the socket. But the a_rwnd controls only the bytes in flight, and 
not headers , SACKs and HB's etc which all consume IP buffer space.
Shouldnt be that these IP buffer space needs to be also considered? If 
not, why not?

One more question is,
If we observe that the SACK cumulative TSN has advanced, then, should we 
reset the path retransmission count and error count for all the paths of 
that association?

Thanks for the clarifications,
Best regards,
devayya

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 23 May 2006 03:59:18 -0600

devayya,

devayya wrote:                                                     (Tue, 23 May 2006 15:10:10)
>    Shouldnt be that these IP buffer space needs to be also considered? If

Those are really OS design issues.

>    One more question is,
>    If  we observe that the SACK cumulative TSN has advanced, then, should
>    we  reset  the  path  retransmission count and error count for all the
>    paths of that association?

No.  Just the one(s) to which the data being acknowledged was sent.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Tue, 23 May 2006 22:54:10 -0600

> Shouldnt be that these IP buffer space needs to be also considered? If 

Yes it should be, and it is a weakness in the specification that it is 
not.  There are two places you can adapt.  On the receiving side, you 
should take whatever your measured free space value is and convert it to 
projected data bytes that you can store.  This typically requires some 
sort of reasonable estimate as to chunk / message size, and an 
implementation specific overhead calculation.  Then advertise the data 
bytes you believe you can handle, a little on the conservative side.

That will work just fine if the message / chunk sizes are typical or 
larger.  However, in order to handle small messages (say 100 bytes) 
without over-projecting overhead on the receiving side, it is a very 
good practice to under-utilize the advertised rwnd on the sending side, 
by subtracting some estimated extra per data chunk or per message 
overhead from the send rwnd on a per chunk/message basis.  If you do not 
do that, you will either have to waste an extraordinary amount of memory 
on the receiving side, or more likely seriously overrun the buffers, 
dropping performance to zilch.

There is one other thing you can do, namely when you are under memory 
pressure, you can collapse data chunks together on the receiving side.  
That almost solves the problem for TCP, with with SCTP you have to 
preserve message boundaries so the technique has serious limitations.

- Mark B.

---

From: devayya <[email-masked]>
Date: Mon, 29 May 2006 12:34:33 +0530

Hi,

RFC 2960 section 8.2 says that :
"When an outstanding TSN is acknowledged or a HEARTBEAT sent to that
   address is acknowledged with a HEARTBEAT ACK, the endpoint shall
   clear the error counter of the destination transport address to which
   the DATA chunk was last sent (or HEARTBEAT was sent).  When the peer
   endpoint is multi-homed and the last chunk sent to it was a
   retransmission to an alternate address, there exists an ambiguity as
   to whether or not the acknowledgement should be credited to the
   address of the last chunk sent.  However, this ambiguity does not
   seem to bear any significant consequence to SCTP behavior.  If this
   ambiguity is undesirable, the transmitter may choose not to clear the
   error counter if the last chunk sent was a retransmission."

So incase where the TSN has advanced and the primary path in case of 
multi-homing, becomes INACTIVE,  can we choose not to reset the path 
retransmission error count for the primary path?

Thanks,
devayya

Mark Butler wrote:

>> So untill all the buffered SACK is processed for the already down 
>> path is done, the path failure of primary is not detected. There will 
>> be delay in detection of path failure and switch over to secondary. 
> This looks more like a matter of implementation then clarification.  
> If you want fast failover, fast error recovery, or low message 
> delivery latency, buffering SACKs more than absolutely necessary is an 
> unusually bad idea.  One may be able to split SACK processing into low 
> and high priority phases, as well as aggregate SACKs under load, but 
> certain critical consequences of a SACK need to take place as soon as 
> possible, *especially* on a multi-path configuration.
>
> - Mark B.
>
>
>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 29 May 2006 01:43:09 -0600

devayya,

devayya wrote:                                                     (Mon, 29 May 2006 12:34:33)
> retransmission error count for the primary path?

No. The passage says that, for messages that were retransmitted to a
different destination, there is an abiguity as to which message the
peer is responding and which count should be reset.  In that abiguous
case the implementation may choose not to reset the count.

Just because a primary path is now INACTIVE does not mean that
acknowledgements for non-retransmitted messages will not be forthcoming.
In fact, it will take such an acknowledgement to ever return the INACTIVE
destinatino to service, so to not reset the count in response to them
could result in permanent inactivity of the destination.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/