Subject: [Sigtran] M2PA (SIGTRAN) and multi-homing

From: "Robby Pocornie" <[email-masked]>
Date: Wed, 6 Dec 2006 15:12:50 +0100

Hi Brian,
Hi All,

Is it useful to have SCTP multihoming on M2PA links between Signalling
Transfer Points/Gateway (STPG)?

I see in the IETF RFC that multihoming on M2PA has been taken into account
as a possibility.
Is this possibility advisible, without being harmful for the Network ?

I also read that It is possible that transmission of M2PA messages MAY be
delayed by SCTP slow start under certain conditions, like Path failure
(multi-homing) if SCTP switches to a new path.
(<draft-ietf-sigtran-m2pa-09.txt>)

I'm now in doubt, because a supplier of STPG's told me that often customers
do not use multi-homing for M2PA associations because redundancy is already
provided at MTP Layer 3 (Changeover procedures).

The supplier claim that M2PA multi-homing can actually make configuration
more complicated because of timers for RTT (at SCTP level) and COO, CBO at
MTP layer 3.

Do you have any argumentation or practical explanation to this claim?
Who can advise me why to Implement Multihoming OR NOT Implement Multihoming
in a Sigtran M2PA Network connection?

Kind Regards,

Robby Pocornie
Sigtran Network Designer
[email-masked]
_______________________________________________
Sigtran mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/sigtran

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 11 Dec 2006 12:52:21 -0700

Robby,

Robby Pocornie wrote:                                               (Wed, 06 Dec 2006 15:12:50)
>    I  see  in  the  IETF RFC that multihoming on M2PA has been taken into
>    account as a possibility.

SCTP multihoming is not harmful to the network (otherwise it would not have
been included in SCTP).
> 
>    I also read that It is possible that transmission of M2PA messages MAY
>    be  delayed  by  SCTP  slow  start under certain conditions, like Path
>    failure (multi-homing) if SCTP switches to a new path.
>    (<draft-ietf-sigtran-m2pa-09.txt >)

Yes, they may be delayed.  But M2PA runs its own ack timer to ensure that
the link is still viable.

> 
>    I'm  now  in doubt,  because  a supplier  of STPG's told me that often
>    customers  do  not  use  multi-homing  for  M2PA  associations because
>    redundancy is already provided at MTP Layer 3 (Changeover procedures).

That is an engineering decision.  If there exists sufficient link redundancy
at MTP3, multihoming might be unecessary.

> 
>    The   supplier   claim   that   M2PA  multi-homing can  actually  make
>    configuration  more  complicated  because  of  timers for RTT (at SCTP
>    level) and COO, CBO at MTP layer 3.

It is true in practice that if SCTP protocol paramters are not set carefully
that they can interact negatively with MTP3.  This is one reason why we
recommended only 1 M2PA link between hosts, because if multiple M2PA links
existed between hosts and all fail at the same time, MTP3 could thrash
between links.  MTP3 pretty much assumes that links fail independently and
is designed to accomodate only a single link fialure in a network.  In
regular SS7, multiple link failures are extremely rare.

>    Who  can  advise  me why  to  Implement  Multihoming  OR NOT Implement
>    Multihoming in a Sigtran M2PA Network connection?

Unfortunately, there is no golden rule.  These are operational decisions that
must be made with full knowledge of the topologies involved.  If you have
little operational experience of your own upon which to rely, it might be a 
good idea to rely partly upon the operational experience of your vendor.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Robby Pocornie" <[email-masked]>
Date: Tue, 12 Dec 2006 15:23:14 +0100

Brian,

Refer to the Sigtran scenario's specification (e.g. ETSI EG 202 360
v1.1.1(2004-12)),
chapter 4.3...

In case there is only a single M2PA link in the link set, the messages
formerly sent out in parallel over different links are now serialized over
one channel, as M2PA makes no use of the " streams" concept provided by
SCTP...
Advise is: To avoid that messages are not delivered (message loss) in case
of retransmission, more than one link using M2PA should be used to form a
linkset.

That is why I design more than 1 M2PA link between two STP's (for
telecommunication SG, not internet host)

Regards,
Robby Pocornie

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 12 Dec 2006 13:24:06 -0700

Robby,

Robby Pocornie wrote:                           (Tue, 12 Dec 2006 15:23:14)
>    v1.1.1(2004-12)), chapter 4.3...
> 
>    formerly  sent out in parallel over different links are now serialized
>    over  one  channel,  as  M2PA  makes  no use of the " streams" concept
>    provided by SCTP...

Yes, I see that (thanks for the reference).  However, I do not think
that ETSI gave due consideration to the impacts of multiple M2PA links
in a link set.  M2PA runs a slightly modified T7 timer to ensure that no
message is delayed beyond limits.  If, as is stated, an IAM becomes
delayed, if the delay is within T7 no message misses a deadline.  If it
is delayed beyond T7, a changeover will occur to another link in the
link-set.  However, in this scenario, MTP3 will changeover to a link that
follows the same network path as the original link and can be expected
to experience the same problems.  This is radically different from
traditional SS7, where the links in a link set are independent and are
normally engineered for diversity to ensure independence.

Let me give you an example of the bad thing we were trying to avoid with
the recommendation to use only one link in a link-set.  Take 16 M2PA
links in one link set (each using the same network path).

On link 1, an IAM gets delayed T7+.  The link fails.  MTP3 starts
changeover (time controlled for argument's sake).  The traffic through
the network path drops by 1/16th allowing intermediate bottleneck
routers to pass messages now without delay.  MTP3 tries to restore link
1, but proving adds additional burst traffic and link 1 fails alignment
and goes into lockout.  After T2, MTP3 adds the buffered traffic to
link 8.  Because buffered traffic is bursted by MTP3 on link 8, link 8
subsequently fails.  The situation is bad already because the original
IAM (and all the message behind it) has been delayed T7 + T2.  Going
around again, after T2, link 8 adds traffic to link 2 and 9, both of
which will fail because the burst traffic again exceeds the handling
capacity of the intermediate bottleneck.  The original IAM and all
messages behind it are now delay T7 + T2 + T2.  Two more iterations and
all 16 links will fail.  The IAM and all messages behind it are now
delayed T7 + 4 * T2.  MTP3 will then changeover to the other link-set
(after another T2).  The IAM and all messages behind it have been
delayed T7 + 5 * T2 and there could now be about 5 times the normal
changeover burst traffic pent up.

The other link-set, even when it uses a different network path, might not
successfully handle the burst traffic which can lead to another another
16 link failures, and then C-links, and then another cascade failure in
the associated STP in the STP pair, and then network isolation.  Large
transmit buffers and M2PA's extremely large window (maximum RTB size)
contribute to such a cascade failure.  SS7 is vulnerable to this
whenever link failure is load related.

At no point does MTP3 understand that these messages have been delayed
long past their useful lifetime.  With a T7 of 1 second and a T2 of 2
seconds, the initial IAM has been delayed 11 seconds and messages behind
it in the buffers have been delayed up to 12 seconds.  Now, although
ISUP T7 (awaiting ACM timeout) is 20-30 seconds, every other timer in
ISUP is less than 12 seconds.  After the second link-set failure, there
will be 24 seconds delay and after C-link failure, 36 seconds delay
(given 16 C-links).

Consider that if there was only one M2PA link in the link-set, the
initial T7 timer expiry would have resulted in a changeover to the
alternate link set restoring traffic within T7 + T2 and only buffering 3
seconds worth of traffic.

These types of failure scenarios are only possible because MTP3 assumes
that the failure of one link in a link-set is completely independent of
the failure of another link in the link-set and that the failure of a
link is not load related.  That is, that the probability that link 8
will fail given than link 1 has failed is the same as the probability
that link 8 will fail given that link 1 has not failed.  Also, that the
probability that link 8 will fail is independent of the load offered to
link 8.  With multiple M2PA links in a link-set, the first condition no
longer holds true.  Unfortunately, for M2PA the second condition (load
related failure) never holds true.  Therefore, the recommendation as
laid out in the RFC is that there only be one M2PA link in a link-set.

There are two problems with the ETSI passage that you reference.  First,
placing multiple M2PA links in a link-set does not decrease inter-link
changeovers as is stated.  As show above, it increases inter-link
changeovers which is bad, and, worse, can lead to catastrophic cascade
failures.  Second, the messages behind the initial IAM do not get
delayed as implied by the passage.  The messages behind the initial IAM
will be immediately available in the buffers at the receiving SCTP once
the IAM is filled in by Fast Retransmission and will not suffer another
network queueing delay (that is, it is not the same as a Q.703 NACK).
Providing redundant network paths and using SCTP's multi-homing feature,
or CMT (concurrent multi-path transfer) can further reduce delay.

>    Advise  is: To avoid that messages are not delivered (message loss) in
>    case  of  retransmission, more than one link using M2PA should be used
>    to form a linkset.

Bad advise.  You are better to use multi-homing over separate network
paths and CMT to reduce delay in a single M2PA link in a link-set.

>    telecommunication SG, not internet host)

It is your network, but I would not in mine.

BTW, the cascade failure above is not conjecture on my part.  In a
previous life I diagnosed just such a cascade failure that resulted
simply from introducing delay into SS7 B/D links between STPs and also
introducing the characteristic of a load-related link failure.  The
cascade failure (that occurred during the Jan 1998 east coast ice storm)
resulted in 2 hours of network isolation and forty-million dollars in
damages (at least that is what the resulting lawsuit claimed).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Andrew Booth <[email-masked]>
Date: Tue, 12 Dec 2006 16:08:19 -0500

Hi Brian,

The thrust of your reply I think is that, since multiple M2PA links use
the same underlying IP network, you don't get as much redundancy as you
might expect from multiple M2PA links.  In some situations it can cause
problems.

I think in some network topologies there may be advantages to multiple
M2PA links, though a single multi-homed link may provide similar
characteristics.  For instance, suppose that you architect you IP
network into two distinct subnets that use distinct hardware.  Each end
of the M2PA link has 2 NICs, one on each subnet.  Having at least one
M2PA link per subnet seems not unreasonable in this case and shouldn't
be vulnerable to the failure you mentioned.

Having more than one M2PA link per subnet probably doesn't help much in
most cases, since your robustness and throughput are limited by the
underlying IP layer.  Having some additional links may help reduce the
effect of head-of-line blocking.  Having some additional links may help
throughput during SCTP slow-start (if for some reason a burst of traffic
suddenly appears).  Neither of these considerations is likely to be
relevant over a good (managed) IP network.  Additional links increase
the chance of cascading failure and increase overall bandwidth
(slightly).  The above considerations are largely contradictory.  There
are tradeoffs in each case.  Over a good IP network none of them are
likely to be serious.

Robby, regarding your original question, here are some considerations
that weren't mentioned yet.

If you're working over a good IP network, either single or multi-homed
M2PA links should work well.  The differences that I see are mainly
operational.

Consider a network topology with distinct subnets.

- in the multi-homed case, you get to choose your SCTP primary path. 
This is potentially useful if one ISP is cheaper.  Depending on your
SCTP parameters you may also see better throughput and less added
latency if a path fails, as compared to relying on MTP changeover.

- in the single-homed case you get MTP alarms for link failures, link
congestion, etc if one of your IP routes goes down or becomes less
performant.  This gives you some extra diagnostics for free, especially
if you don't get alarms for ISP router failures.

- in the single-homed case you have the option to deactivate a link to
redirect traffic away from the impaired IP network.

Comments welcome,
Andrew

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 12 Dec 2006 14:41:55 -0700

Andrew,

Andrew Booth wrote:                        (Tue, 12 Dec 2006 16:08:19)
> be vulnerable to the failure you mentioned.

Well, yes and no.  Although separate underlying networks might appear to
remove some of the independence between link sets, it only does so at the
hardware level and not necessarily at the system level (i.e. when both
links are still pumped through the same SCTP stack that has a single IP
queue that can act as a common bottleneck).  Many operating systems have
issues with regard to internal IP queue bottlenecks particularly when
high speed interfaces are used such as GigE.  To get independence at the
system level require independent priority queueing through the IP
subsystem, a situation that is not obvious to most.

Also, it does not remove the load related coupling that can lead to
cascade failure (other traffic using the same two subnets).  It is less
costly to over-engineer one network than two.

All in all, if multiple subnets are provided, it is always better to use
the subnets for multi-homing with SCTP (and better still CMT) to provide
a single higher grade M2PA link than two lower grade links.

SCTP multi-homing is designed for the characteristics of the underlying
IP network, whereas MTP3 simply was not.  When SCTP multi-homing is
used, SCTP determines which of the subnets is viable and best suited for
traffic (CMT better than traditional) regardless of any shared
bottleneck that might exist.  If multiple link sets are used, MTP3
requires that no shared bottleneck exists anywhere (network or host),
otherwise cascade failure can result.

Even when using a single link in a link-set, if there are multiple link
sets (C and B/D links in the scenario), a shared IP bottleneck within
the host can lead to cascade failure between link-sets.  To avoid
catastrophic failures in such a scenario would require that it can be
guaranteed that an IP bottleneck internal to the host will never occur,
even when bursting buffers.  (In the SS7 failure that I experienced,
there was a 4 second 1 Erlang burst of traffic over C-links milliseconds
before the second STP went down.)

One way to ensure no shared IP system bottleneck is to provide
independent blades for each M2PA link in the STP: but then they are
separate hosts and there is no more than one M2PA link per host.

So, ideally, one M2PA link per host and if multiple subnets are
available, use SCTP multi-homing and CMT to get better grade of service
on the single link.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/