Subject: [Tsvwg] Aggressive SCTP RTO tuning

From: Anders Torger <[email-masked]>
Date: Wed, 9 Feb 2005 07:43:52 +0100

I'm using SCTP in a telecom signaling context where failover timers are 
very aggressive - the server has a response time requirement of 10 ms, 
and if there is no response within 2500 ms, the client should fail over 
to a backup server.

Although the network between server and client is low latency and high 
performance and is properly dimensioned to handle signaling traffic 
load, packet loss due to some transient problem may occur, therefore I 
want to tune the SCTP timers the following way:

RTO.Initial = 100 ms
RTO.Min = 100 ms
RTO.Max = 800 ms
MaxRxt = 6

This way a few retransmissions will occur within the failover time 2500 
ms, and if the net has failed and message sending is attempted, SCTP 
will auto-close the connection within ~2500 ms.

Typical defaults for these values seems to be RTO.Initial = 3000, 
RTO.Min = 1000, RTO.Max = 60000 and MaxRxt = 10, thus my timers are 
very aggressive in comparison.

The Linux kernel implementation supports setting the RTO timers as I 
want, and my initial tests shows that it behaves as expected. However, 
the Solaris 10 implementation has the following *minimum* values: 
RTO.Initial = 1000 ms, RTO.Min = 500 ms, RTO.Max = 1000 ms. Thus, the 
Solaris SCTP implementation is not very suitable in my context, it 
seems to work on a (half)second basis rather than millisecond.

My question is if SCTP has been designed to make this type of aggressive 
timers possible, (meaning that the Solaris 10 implementation is bad, 
and I should complain to Sun), or if I should consider another 
transport protocol.

/Anders Torger

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 9 Feb 2005 11:35:12 +0100

Hi Anders,

see my comments in-line.

Best regards
Michael

> and I should complain to Sun), or if I should consider another
SCTP has been designed for transporting signalling traffic and therefore
the protocol works with timers you have given. However, each 
implementation
might have some contraints for these parameters. As you see, the Linux
implementations supports your settings.
So it is not a question regarding the protocol but regarding specific 
implementations.

---

From: Anders Torger <[email-masked]>
Date: Wed, 9 Feb 2005 12:25:57 +0100

Thank you for your response!

I guess what you're saying is - yes, aggressive timers is not a breach 
of the protocol - but implementations are free to put in any limits 
they want. Just like they can limit the number of streams per 
association and so on.

Perhaps the "Stream Control Transmission Protocol (SCTP) Implementer's 
Guide" should contain recommendations whan an implementation should 
minimally support, for example at least 16 streams per association, at 
least 5 peers per association etc, and in that include fairly low RTO 
settings, so it is possible to use SCTP in a (soft-)realtime context. 
An implementation could still break the recommendations, but would then 
not be considered a "fully featured" SCTP stack.

The goal would be to put some pressure on mainstream implementations, 
such as the one in Solaris 10 and Linux, such that SCTP becomes more 
useful, meaning that SCTP users can expect more from a given SCTP 
stack. If not, there is a danger that the established SCTP stacks is 
limited in ways that will harm the protocol's potential and popularity.

/Anders Torger

On Wednesday 09 February 2005 11.35, you wrote:
> > I'm using SCTP in a telecom signaling context where failover timers
> > are very aggressive - the server has a response time requirement of
> > 10 ms, and if there is no response within 2500 ms, the client
> > should fail over to a backup server.
> >
> > Although the network between server and client is low latency and
> > high performance and is properly dimensioned to handle signaling
> > traffic load, packet loss due to some transient problem may occur,
> > therefore I want to tune the SCTP timers the following way:
> >
> > RTO.Initial = 100 ms
> > RTO.Min = 100 ms
> > RTO.Max = 800 ms
> > MaxRxt = 6
> >
> > This way a few retransmissions will occur within the failover time
> > 2500 ms, and if the net has failed and message sending is
> > attempted, SCTP will auto-close the connection within ~2500 ms.
> >
> > The Linux kernel implementation supports setting the RTO timers as
> > I want, and my initial tests shows that it behaves as expected.
> > However, the Solaris 10 implementation has the following *minimum*
> > values: RTO.Initial = 1000 ms, RTO.Min = 500 ms, RTO.Max = 1000 ms.
> > Thus, the Solaris SCTP implementation is not very suitable in my
> > context, it seems to work on a (half)second basis rather than
> > millisecond.
> >
> > timers possible, (meaning that the Solaris 10 implementation is
> > bad, and I should complain to Sun), or if I should consider another
> > transport protocol.
>
> SCTP has been designed for transporting signalling traffic and
> therefore the protocol works with timers you have given. However,
> each implementation
> might have some contraints for these parameters. As you see, the
> Linux implementations supports your settings.

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 9 Feb 2005 13:17:22 +0100

Anders,

the Implementers Guide is for 'bugs' in the RFC 2960.

ETSI has a documents which lists parameters, which
need to be configurable, gives minimum, maximum and
a granularity. Seems pretty much what you want...

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 9 Feb 2005 05:38:40 -0700

Anders,

See also http://www.ietf.org/internet-drafts/draft-ietf-sigtran-signalling-over-sctp-applic-09.txt

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 09 Feb 2005 11:20:04 -0800

Anders Torger wrote:

...

> MaxRxt = 6

...

> and I should complain to Sun), or if I should consider another 

I think Solaris 10's soft limits on the above RTO tunings
are actually good (-:  The above setting is not good for
general use in the Internet and allowing that to happen is
probably a bad idea.  Having said that, there is a way to do
it in Solaris.  Since it is an implementation specific topic,
I think it is not for the tsvwg mailing list.  I will email
you privately.  And for all Solaris SCTP specific questions,
you can use the [email-masked] mailing list.

-- 

						K. Poon.
						[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 9 Feb 2005 13:42:00 -0700

Kacheong,

With a more general view, RTO.Min < 200 ms is a bad idea, unless
one has control also of the maximum SACK delay at both peers.
RTO.Max of 800 ms really does not do much.

If all one needs is faster failure detection put lifetimes on
messages and abort the association if one expires.  So for 2500
ms failure detection, leave the protocol parameters as they are
recommended in RFC2960, set a 2.5 sec lifetime on each message,
and abort the association when a message expires.

--brian

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 09 Feb 2005 13:11:40 -0800

> RTO.Max of 800 ms really does not do much.

I guess it depends on the meaning of "general" (-:  With
a satellite link or even just a trans-continental link, a
RTO.Max of 800ms is probably a little bit low.  It is
not good to have RTO.Max close to or lower than RTT.

-- 

						K. Poon.
						[email-masked]

---

From: Anders Torger <[email-masked]>
Date: Thu, 10 Feb 2005 07:39:56 +0100

I'll look into the SACK delay issue, since it is not guaranteed that I 
can control the SACK delay at the remote peer. Thanks for the tip.

I don't like to use the time-to-live option though, without PR-SCTP I 
don't find it very useful at all (I haven't tested it with PR-SCTP, so 
I'm not sure how it behaves there though), with vanilla SCTP it is just 
a timer which says "if the kernel has not put the message on the wire 
at least once within this time, issue a send failed event".

So if the local computer is not heavily overloaded, and the remote peer 
does not have a full window (which won't happen in the signalling 
scenario, unless there is a serious overload event), there is never a 
send failed event triggered from the time-to-live, despite that the 
message may not be delivered until many seconds after the timer has 
expired.

/Anders Torger

On Wednesday 09 February 2005 21.42, Brian F. G. Bidulock wrote:
> > > My question is if SCTP has been designed to make this type of
> > > aggressive timers possible, (meaning that the Solaris 10
> > > implementation is bad, and I should complain to Sun), or if I
> > > should consider another transport protocol.
> >

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 10 Feb 2005 02:03:31 -0700

Anders,

Well, no... When the first T3-rtx expiry occurs, cwnd will be
set to 1*MTU.  Under normal operation on a fast network, cwnd is
much greater than 1*MTU and new data will back up in queue
quickly.  Although the full cycle to association failure with
recommended values may take several minutes, a lifetime on new
data of 2.5 sec will expire if 2*MTU of new data is sent within
the 2.5 second interval (i.e. you have some traffic).  This has
the nice side effect that it should not drop associations that
are quiescent.

I agree, however, that the lifetime approach is not obvious and
perhaps not the best.  But mucking with SCTP recommended
protocol parameters can be disastrous.  That was the point of
the SCTP applicability statement for SIGTRAN.

There are several other viable approaches that don't involve
changing SCTP recommended protocol parameters:

1) When developing the M2PA protocol under SIGTRAN, we were
   quite concerned about a similar situation.  We needed to
   bring down M2PA links (and not the association) in situations
   where acknowledgments where being delayed by more than about
   1 second (per Q.703 requirements).  To accomplish this, the
   M2PA runs its own acknowledgments and makes its own delay
   calculations.  If your application or UAL takes advantage of
   such an approach it is by far the best way to go.

2) Protocols such as M3UA can use transmit buffer occupancy for
   congestion (which IETF calls flow) control, ultimately ending
   in application congestive failure.  By tuning buffer
   occupancy thresholds, a maximum bound can be set on queuing
   delay.  But the lifetime approach, although less obvious, is
   equivalent.

3) Other applications protocols can use the mechanisms provided
   by the specific SCTP implementation to query the SRTT value
   for destinations  (gee, how about every 2.5 seconds) and if
   the minimum SRTT is greater than some value (say 2.5 seconds)
   the application can take the association down (or simply deem
   it unusable until heartbeats restore a better SRTT).

In your application you could combine all 4 approaches and meet
your needs without changing SCTP primary protocol parameters:

  Have your client run timers on responses and fail over to the
  backup server if a response is not received within 2500 ms.

  Keep track of buffer occupancy toward the server and fail over
  to the backup if it exceeds some tunable value.

  Set lifetimes on your messages of 2500 seconds or much less if
  you'd like and fail over to the backup on lifetime expiry.

  Monitor SRTT for the transport destinations on the association
  and fail over if the minimum SRTT exceeds some maximum (even,
  say, 100 ms).  Remember that SRTT is the actual smoothed RTT
  and is not subjected to either RTO.min or RTO.max.

These four approaches are really similar in that each will
trigger under similar conditions and can be tuned to meet your
requirements.  Each monitors the grade of the connection without
negatively impacting SCTP's ability to respond to network
conditions.

Hope that helps.

(For reference, OpenSS7 SCTP allows you to set these values to 0,
but such values were intended for experimentation and extreme or
tightly controlled network conditions only.  One example is
simulated Ethernet Bonding on a PICMG 2.15 backplane.)

--brian

---

From: Anders Torger <[email-masked]>
Date: Thu, 10 Feb 2005 11:01:31 +0100

Thanks for the very valuable tips!

Actually, I do monitor response times on the upper layer protocol, which 
is a good thing even if SCTP timers are set aggressively, since the 
server (software) could fail in a way such that SCTP is working, but 
responses are not given in a timely manner, and then failover should be 
triggered. The upper layer protocol also have its own flow control to 
protect response times of the server.

However, I want aggressive SCTP timers such that retransmissions occur 
quickly if there should be an MPLS fast reroute (~50 ms downtime) or 
similar transient error on the signalling network.

Ideally, I want the first retransmission to be at say 100 ms, such that 
fast reroutes would not harm response times too much.

The system will work with standard SCTP timers, but I'm looking into 
improving performance during typical failure scenarios.

On Thursday 10 February 2005 10.03, Brian F. G. Bidulock wrote:
> > I'll look into the SACK delay issue, since it is not guaranteed
> > that I can control the SACK delay at the remote peer. Thanks for
> > the tip.
> >
> > I don't like to use the time-to-live option though, without PR-SCTP
> > I don't find it very useful at all (I haven't tested it with
> > PR-SCTP, so I'm not sure how it behaves there though), with vanilla
> > SCTP it is just a timer which says "if the kernel has not put the
> > message on the wire at least once within this time, issue a send
> > failed event".
> >
> > So if the local computer is not heavily overloaded, and the remote
> > peer does not have a full window (which won't happen in the
> > signalling scenario, unless there is a serious overload event),
> > there is never a send failed event triggered from the time-to-live,
> > despite that the message may not be delivered until many seconds
> > after the timer has expired.
> >
> > /Anders Torger
> >

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 10 Feb 2005 05:54:06 -0700

Anders,

Well, that's something different.  If you are looking for fast
retransmission on brief failures, the FR (gee, that's Fast
Retransmission) algorithm will do what needs to be done, again,
without altering RTO.Min.  FR will retransmit much faster than
your 100 ms RTO.Min (on the 4'th gap report), so, again, there
is no need to set RTO.Min so low.

--brian

---

From: Anders Torger <[email-masked]>
Date: Thu, 10 Feb 2005 14:37:44 +0100

Yes, but that assumes that there is continuous traffic, which is not 
certain. However, putting all together it could turn out to be 'good 
enough'. I need to play around some more with the stack and analyze 
some scenarios.

Thanks for the help.

On Thursday 10 February 2005 13.54, Brian F. G. Bidulock wrote:
> > Actually, I do monitor response times on the upper layer protocol,
> > which is a good thing even if SCTP timers are set aggressively,
> > since the server (software) could fail in a way such that SCTP is
> > working, but responses are not given in a timely manner, and then
> > failover should be triggered. The upper layer protocol also have
> > its own flow control to protect response times of the server.
> >
> > However, I want aggressive SCTP timers such that retransmissions
> > occur quickly if there should be an MPLS fast reroute (~50 ms
> > downtime) or similar transient error on the signalling network.
> >
> > Ideally, I want the first retransmission to be at say 100 ms, such
> > that fast reroutes would not harm response times too much.
> >
> > The system will work with standard SCTP timers, but I'm looking
> > into improving performance during typical failure scenarios.