Subject: RE: [Sigtran] RE: Message loss in SIGTRAN networks

From: "Haresign Lincoln" <[email-masked]>
Date: Mon, 3 Oct 2005 17:26:16 -0400

Tolga,

I agree, it would be interesting to see what other people think and what
their experiences are in real networks that are running load where loss
of an association is bothersome to the end customer.

The network where we saw problems was a private network.  However, it
was a Wide Area Network that spans multiple islands and is doing several
hundred million messages per day.  And this network does have
multihoming.  We are occasionally seeing problems in the network that
can cause loss of associations.  This is rare, but it does happen.  Just
like link failures in the SS7 networks are rare (0.0001% of the time).

Of course the customer SHOULD design their networks to handle the
capacity.  BUT sometimes, all customers choose to do things cheaply
and/or sometimes we are interfacing with other switches that may exhibit
problems and/or the customer may not completely understand how to
engineer their IP networks.  It easy for us sitting far back from the
actual network as engineers designing a protocol to assume that
everything will work fine and everyone will do the right thing.  The
real difficulty in designing a system and/or protocol is to worry about
the 20% cases.  And if you are not going to cover the case where an
association fails, you're missing something in the protocol.  

So, when the association fails, there is message loss (as designed in to
the current RFC).  And I see this as a problem.  Because message loss
can result in hanging calls, tearing down TCAP transactions, and loss of
billing revenue.  Of course the higher layer SS7 protocols have
extensive redundancy and fault tolerance built in to handle these
scenarios and that's a good thing.  We see that historically, the
ITU/ANSI groups have done a very good job here in designing TCAP and
ISUP.  Years of experience.

They also built in a retrieval mechanism for retrieving message and
rerouting to other SS7 links when the E1/T1 connections (that are never
supposed to fail) fail.  So why not do this in M3UA?  You say there is
an expense in implementing this...I agree.  So make it optional.  But in
the end, in my humble opinion, the customers will want this feature.

Regards,
Lincoln

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Tolga Asveren
Sent: Monday, October 03, 2005 4:19 PM
To: [email-masked]
Subject: [Sigtran] RE: Message loss in SIGTRAN networks

Lincoln,

I agree with all of the goals you mentioned. The only point we differ
is, whether SCTP multihoming using multiple networks provides no-single
point of failure semantics or not.

IMO, SCTP multihoming could address router failures, accidently pulling
a cable etc... This obviously won't mean that associations won't fail,
they will, when the peer is down.

Another point is, I personally always thought that SIGTRAN protocols are
going to be run on private networks -with all respect to the "I" in
IETF- which will be engineered for that purpose. Even if SCTP parameters
are adjusted for poor IP network characteristics, I would thint that
most SS7 applications would be more demanding in terms of message
delays.

Would adding additional reliability hurt? IMO, defining new procedures
for reliability will have a very high cost, both in terms of
development/complexity/interoperability and from CPU cycles point of
view with marginal/no added value.

It would be interesting to hear what other people think about this
topic.
IMO, if there are concerns, they need to be addressed, either by
demonstrating a solution from existing procedures or by defining new
procedures. IMO, SCTP multihoming is the solution.

    Thanks,
    Tolga

> -----Original Message-----
> From: Haresign Lincoln [mailto:[email-masked]]
> Sent: Monday, October 03, 2005 4:18 PM
> To: Tolga Asveren; [email-masked]
> Subject: Message loss in SIGTRAN networks
>
>
> Subject changed to reflect topic
>
> Tolga,
>
> In an ideal world with everything working according to the 
> specification, I agree that there will be no message loss.  In the 
> real world, associations fail, fault-tolerant routers fail, customer 
> support people accidently pull cables, execute the wrong commands, 
> etc...  We must deal with poorly designed networks and human error.
>
> If you are running multihoming (optional), you have less of a 
> probability that associations won't fail.  However, they still can 
> fail (and they do fail).  The question is do you want to add 
> additional fault tolerance in to the specifications to cover the 
> inevitable failures that are going to occur in the network.
>
> Regards,
> Lincoln
>
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Tolga Asveren
> Sent: Monday, October 03, 2005 2:57 PM
> To: [email-masked]
> Subject: RE: [Sigtran] recommendations for M3UA/SUA BEATs when running

> overSCTP
>
> Lincoln,
>
> I understand the concerns you mention but I just stated how things are

> supposed to be from M3UA point of view : multihoming + reliable IP 
> networks.
> Yes, things can be broken in the network, so the IP network also needs

> to be designed in a fault-tolerant way -which is possible- and 
> multihoming allows us to tolerate some reasonable long amount of IP 
> down time.
>
> Specifically for the problem you mentioned, lack of multihoming 
> support seems to be the key point, otherwise there wouldn't be any 
> message loss whatsoever. Configuring SCTP parameters properly 
> according to the characteristics of underlying IP network and 
> according tot he needs of the application are also important.
>
> If the basic assumptions of protocol architecture are violated, it is 
> no suprise that certain procedures are broken and/or procedures 
> defined in the protocol fall short of satisfying demands. If there 
> were multihoming support and multiple networks, NIC card failures, 
> cable problems, router failures and alike would be addressed.
>
>
>    Thanks,
>     Tolga
>
> > -----Original Message-----
> > From: Haresign Lincoln [mailto:[email-masked]]
> > Sent: Monday, October 03, 2005 3:00 PM
> > To: Tolga Asveren; [email-masked]
> > Subject: RE: [Sigtran] recommendations for M3UA/SUA BEATs when 
> > running
>
> > overSCTP
> >
> >
> > Tolga,
> >
> > Sorry, but this reasoning doesn't work.  I don't know about you, but

> > our SW has been deployed in some places with fairly unreliable IP
> networks.
> > As a result of this, we have lost some associations.
> >
> > Also, another consideration is that some switches that we have 
> > connected to do not support multihoming.  However, we must provide 
> > fault tolerance.  Therefore, we are running multiple associations to

> > the same switch and loadsharing our traffic across the associations.
> > The associations are on different networks.  If you have a router or

> > IP switch fail between you and the remote switch (and they do fail),

> > then you loose an association and there will be loss of data.
> >
> > Remember, there are many things that can break between you and the 
> > end
>
> > point.
> >
> > With SS7, the peer host can be up, but you can loose an I/O board, 
> > or simply have a cable failure (also true with SIGTRAN).
> >
> > Regards,
> > Lincoln
> >
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] On 
> > Behalf Of Tolga Asveren
> > Sent: Monday, October 03, 2005 2:35 PM
> > To: [email-masked]
> > Subject: RE: [Sigtran] recommendations for M3UA/SUA BEATs when 
> > running
>
> > overSCTP
> >
> > Lincoln,
> >
> > Architecturally, there is no shortcoming of M3UA in terms of message

> > loss for the following reasons:
> > 1- It is assumed that SCTP will provide transport layer/network 
> > interface reliability, i.e. use multihoming and multiple networks.
> > 2- Because of 1), it is assumed that loss of SCTP association 
> > indicates that peer host is down.
> > 3- When peer host is down, there is no way to decide, which messages

> > in retransmission buffers are really received by the peer -please 
> > note
>
> > this is also the case for SS7-.
> >
> >
> > Whether SCTP meets 1) and/or a few changes are necessary in M3UA are

> > a
>
> > different issue but as said, 1) is the main rational behind not 
> > having
>
> > recovery procedures in M3UA.
> >
> >    Tolga
> >
> >
> >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On
> > > Behalf Of Haresign Lincoln
> > > Sent: Monday, October 03, 2005 2:41 PM
> > > To: [email-masked]
> > > Cc: [email-masked]
> > > Subject: RE: [Sigtran] recommendations for M3UA/SUA BEATs when 
> > > running
> >
> > > overSCTP
> > >
> > >
> > > Brian,
> > >
> > > That's really a separate issue.  Whenever we loose an association 
> > > for whatever reason, there exists the possibility of message loss.
> > > I consider this one of the major shortcomings of the current 
> > > specification.  I have reviewed an early draft (Version 0) of your

> > > recommendation for using the COORID to prevent message loss and I 
> > > think it's a good idea.  I haven't seen any other switches that we

> > > have interfaced to using this nor have any customers mentioned
this.
> > > I believe that, as the customers become more educated, they will 
> > > begin
> >
> > > to look for solutions to this problem and I think your 
> > > recommendation is a good place to start.
> > >
> > > Regards,
> > > Lincoln
> > >
> > > -----Original Message-----
> > > From: Brian F. G. Bidulock [mailto:[email-masked]]
> > > Sent: Monday, October 03, 2005 2:29 PM
> > > To: Haresign Lincoln
> > > Cc: Jeff Morriss; [email-masked]
> > > Subject: Re: [Sigtran] recommendations for M3UA/SUA BEATs when 
> > > running
> >
> > > overSCTP
> > >
> > > Lincoln,
> > >
> > > After you abort the association, how are you handling the 
> > > resulting message loss?
> > >
> > > --brian
> > >
>

_______________________________________________
Sigtran mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/sigtran

_______________________________________________
Sigtran mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/sigtran

---

From: "Tolga Asveren" <[email-masked]>
Date: Mon, 3 Oct 2005 17:45:19 -0400

Lincoln,

Despite the risk having hard to follow messages in this thread as well, I
will have a few comments below.

  Thanks,
  Tolga

> Sent: Monday, October 03, 2005 5:26 PM
> Subject: RE: [Sigtran] RE: Message loss in SIGTRAN networks
>
>
> Tolga,
>
[TOLGA]I agree that operational stability is very important.
> ISUP.  Years of experience.
[TOLGA] Association loss is covered by M3UA procedures, but as said before
M3UA assumes SCTP association loss indicates that peer host is down.
Even for a poorly designed network, as long as SCTP parameters are adjusted
accordingly, SCTP association will be alive. Probably the needs of SS7
applications won't be met from time to time but no transport layer
protocol -nor any application layer protocol- can correct this,
delay/message loss etc.. are property of the underlying IP network.
IMO, another important aspect of the problem is, whether there is an
advantage of using multiple SCTP associations instead of multihoming for a
network with low QoS assurance. I think, there isn't. IP packets will follow
the same path and will use the same routers in both of the cases. So,
packets will have the same RTT and the same loss rate.
> the end, in my humble opinion, the customers will want this feature.
[TOLGA]SCTP multihoming also covers NIC type of failures. Afterall, also for
SS7, when the peer host is down, there is no way of recovering messages.
Recovery is only applicable when there is a link failure, similar concept is
true for SCTP. If there is multihoming, failure of a single path is
tolerated with no message loss.

IMHO, considering the maturity of SCTP multihoming and reliance of M3UA on
it, it is a better option to   use it, rather than trying to define other
mechanisms for non-multihomed environments.

Probably the key point is, whether SCTP association failure can be seen as a
sign of peer host failure or not. IMHO with multihoming it can be, but I am
also eager to hear more from operational experiences of different
people -and maybe also from SCTP people-

---

From: "Tolga Asveren" <[email-masked]>
Date: Mon, 3 Oct 2005 18:07:28 -0400

While trying to look to the issue from a different perspective:

Imagine the following scenario:

1)A not so great IP network. It usually delivers what is expected but not
always.
2)SCTP with multihoming.
3)The need that SCTP association failure indicates host failure - we don't
want to be fooled by network errors/failures-
4)The need that host failures are detected in a reasonable amount of time.

a)Because of 3) and 1), we want to set our SCTP parameters a little bit
loose, allowing several retransmissions on each path, before deciding for a
host failure. This would cause host failures to be detected slower.

a) would contradict with 4). We don't want to send messages to a host, which
already has crashed, because all such messages would be lost.

What if:
i. If we allow ASPs to send ASPDN messages on behalf of other ASPs,
addressed by ASPId -the rationale here is that an ASP paired with another
one could -hopefully- detect failure of another one much quicker than SGP
with loose SCTP parameters-? Actually, for Override traffic mode, we
probably don't need to send anything at all, just the previously standby ASP
sends ASPAC and all traffic is diverted to it.

ii. Now a little bit more dirty part of the problem. What to do about failed
SGPs? Use something like SGPId+ASPDN-ACK, SGPId+BEAT, IP Address/SCTP
port+BEAT?

    Thanks,
    Tolga

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 4 Oct 2005 03:31:22 -0600

Tolga,

Tolga Asveren wrote:                           (Mon, 03 Oct 2005 17:45:19)

> M3UA assumes SCTP association loss indicates that peer host is down.

No it doesn't.  In fact the ASP Failure procedures (NOTIFY[ASP Failure]
coupled with ASP Id) were precisely to assist the ASPs in an AS
distinguish between communications loss (partitioning between ASPs) and
ASP nodal failure.  If an ASP both loses communications with another ASP
and receives an NOTIFY[ASP Failure] it can surmise that the ASP has
failed and take appropriate action (e.g. transfering ownership of
transaction of circuit state, performing CORID procedures).  If a
NOTFIY[ASP Failure] is not received from the SG, then the SG is still
communicating with the isolated ASP and the ASP should not transfer
ownership of transaction or circuit state.  An ASP that does not
normally communicate with other ASPs in the AS  (such as in a broadcast
AS) may use simply the NOTIFY[ASP Failure] to determine whether to usurp
transaction/call state.

In fact, from the SG's point of view, there is no difference between
failure to communicate with an ASP and failure of the ASP.  In both
cases the ASP cannot process its assigned load (if any) and in both
cases the ASP need to be marked ASP-DOWN (ASP-INACTIVE in all AS to
which it is a member) and NOTIFY[ASP Failure] with the ASP Identifier
sent to other ASPs in the AS.  In both cases the SG must reassign the
ASP's assigned load to another ASP active in the AS, or move the AS
to AS-PENDING.

Other ASPs active in the AS take a different view.  Whether then can
communicate with an ASP (which they likely can because there are more
closely coupled than ASP and SGP) or whether the ASP has suffered a
Nodal failure is not the point.  The point is whether the ASP is still
communicate with the SG.  If it is, then it can process its assigned
traffic load.  If it is not, then it simply cannot.  This is why the SG
informs all ASPs when it loses communication with another ASP in the AS
by ASP Identifier using the NOTIFY[ASP Failure] message: so that the
other ASPs can assume the isolated ASPs traffic load (in some way).
In the loadshared AS, such a NOTIFY is a clear indication to other ASPs
active in the AS that they may immediatedly receive traffic previously
assigned the failed ASP.  It might behoove the other ASPs to take steps
to make locally available shared transaction/circuit state previously
assigned to the failed ASP in preparation for receiving traffic
associated with those transactions/circuits.

> packets will have the same RTT and the same loss rate.

M3UA (and M2UA and SUA for that matter) do not use multiple associations
for network reliability.  They are used to support M3UA's redundancy
model where multiple ASPs serve and AS and multiple SGPs make up an SG.

Even on a perfect IP network (zero RTT, no message loss, no message
missequencing, infinite bandwidth) real components fail.  Also, even on
a perfect IP network, real components must be taken out of service from
time to time for upgrade or maintenance.

The problem with M3UA (for SS7 replacement) is that even if you have a
perfect IP network messages will be lost due to failing components
despite every effort in ASP or SGP design, and even if the ASP and SGP
components are as reliable as those used in the SS7 network.  Even if
two ASPs completely mirror total M3UA and MTP-User state between them,
when one fails, the SGP must discard messages in transit.

This is because M3UA lacks the protocol mechanisms to interact with the
mirrored ASP and communicate the disposition of messages in transit to
the failed ASP.  Read the introduction the CORID draft again.

One solution would be to Ethernet bond the two ASPs and mirror the
entire SCTP stack state between them.  Then they would appear as one ASP
to M3UA and would communicate over one association.  It would be rather
costly and complex to do so, just to get around M3UA's protocol
defficiencies.

So, even if I have a perfect IP network and ASPs that completely mirror
MTP-User state between them, when one fails, M3UA trashes messages.

Given SS7's strict requirements for message loss, and considering the
impact that would have on Lincoln's SCCP Protocol Class 0 applications
(that rely heavily on MTP grade of service assurances), M3UA is dismally
lacking in this regard.

So I don't think that dismissing this M3UA deficiency as the fault of
the dirty cheap IP network or inappropriate SCTP parameters is a valid
argument.  Even a perfect IP network and a near perfect ASP suffer at
the hands of M3UA and SCTP when switching (or failing over) between
associations.  Certainly failover between associations was what we had
in mind when we put together the redundancy model.  It just doesn't work
well when considering MTP's stringent message loss and duplication
requirements.

Of course this is what prompted me to write the first CORID draft in
January 2002 (before RFC 3332).  But it has fallen on deaf ears.

What is needed, I believe, is an "M3UA Applicability Statement".  One
that says

   It is NOT RECOMMENDED that M3UA be used to replace SS7 transport as
   its design cannot meet the stringent message loss and duplication
   requirements set for the SS7 Message Transfer Part.

and one that addresses the other woefully lacking areas of M3UA:

   - Congestion control.
   - ASP startup.
   - Isolation.
   - ASP Inactive Ack/ASP Down Ack.
   - Use (or misuse) of heartbeats.
   - Timer values for T.pending.
   - When and how it is advisable for M3UA to discard messages.
   - Limitations of load sharing.
   - Traffic distribution to SGPs.

I can probably think of more.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 4 Oct 2005 03:47:11 -0600

Tolga,

Tolga Asveren wrote:                           (Mon, 03 Oct 2005 18:07:28)
> sends ASPAC and all traffic is diverted to it.

We discussed this.  It may help the SGP sooner detect that the ASP is
off the air, but it does not solve any problems.  All messages in
transit to the old ASP must be discarded because the SGP cannot tell
whether they were processed by the failed ASP or not.  (Also because
MTP's duplication requirements are far more stringent than its loss
requirements.)  That is an M3UA protocol deficiency, which happens to be
solved by CORID.

When the ASP sends ASP Down for another ASP in the AS, it does little to
reduce the number of messages in transit to the ASP.   This is because,
under full load, cwnd will already be acheived so no additional messages
will be send until some are acknowledged, such SACK acknowledgements
which can no longer be sent by the failed ASP.

Also, it does not speed the switchover of messages to active ASPs.  This
is because without CORID you cannot changeback immediately or you risk
missequencing of messages at the ASP.  Some time (similar to the
time-control changeover and rerouting procedures of MTP) must be allowed
to transpire before traffic previously destined to the failed ASP can be
redirected to the active ASPs in the AS.  This is particularly important
when failing over between SGP.

> 
> ii. Now a little bit more dirty part of the problem. What to do about
> failed SGPs? Use something like SGPId+ASPDN-ACK, SGPId+BEAT, IP
> Address/SCTP port+BEAT?

See the CORID draft.  Procedures there apply to ASP, SGP and IPSP
(termed SPP).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 4 Oct 2005 08:17:18 -0400

Brian,

> Sent: Tuesday, October 04, 2005 5:31 AM
> To: Tolga Asveren
> Cc: [email-masked]
> Subject: Re: [Sigtran] RE: Message loss in SIGTRAN networks
>
>
> Tolga,
>
[TOLGA]I don't agree that M3UA does not assume that loss of SCTP association
indicates host failure, but I agree that some of the existing
procedures -like the NOTIFY you mentioned- may be helpful for certain cases
to detect peer failure quicker before SCTP association becomes down. OTOH,
like you described, current procedures do not help SGP to detect ASP failure
or ASP to detect SGP failure with the help of another ASP or SGP
respectively.

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 4 Oct 2005 08:45:38 -0400

Brian,

> Sent: Tuesday, October 04, 2005 5:47 AM
> To: Tolga Asveren
> Cc: [email-masked]
[TOLGA]Maybe here and there there has been talk about those topics but I
don't recall that a procedure for them was evaluated.
That proposal -or anything similar- solves the problems. The problem is not
to retransmit messages to another host, because peer is down, so we can't
know which messages are received by the failed host. Please note this is the
same in SS7, when peer is down you can't know about what it received last.
The proposal has the goal to detect host failures quicker than an SCTP
association with loose parameters.
> which can no longer be sent by the failed ASP.
[TOLGA]I agree that this may help sometimes but actually it all depends on
the current value of cwnd, on traffic pattern etc.. For example, if you had
a peak in your traffic -which would increase cwnd- and then returned back to
a regular load traffic, you can have plenty of cwnd to fill -during that
time cwnd may also shrink but as I said this depednds on traffic pattern-.
> when failing over between SGP.
[TOLGA]For this part I agree. It is a good idea to queue messages, which
were supposed to be sent to a failed host for a while before routing them to
another peer.
> > failed SGPs? Use something like SGPId+ASPDN-ACK, SGPId+BEAT, IP
[TOLGA]This has nothing to do with CORID draft. What I am trying to figure
out is how to expedite detection of peer host failures as soon as possible
in a not sot so great IP network with loose SCTP parameters.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 4 Oct 2005 12:13:37 -0600

Tolga,

Tolga Asveren wrote:                                                                 (Tue, 04 Oct 2005 08:45:38)
> don't recall that a procedure for them was evaluated.

Yes we did talk about them and decided they were not that useful.

> same in SS7, when peer is down you can't know about what it received last.

Read CORID, read Q.704 changeover/changeback procedures.

> association with loose parameters.

That accomplishes nothing.

> in a not sot so great IP network with loose SCTP parameters.

But quicker detection accomplishes nothing.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 4 Oct 2005 12:10:27 -0600

Tolga,

Then we disagree.  Perhaps you can point to where your claimed
assumption is written in the RFC.

--brian

Tolga Asveren wrote:                                                                 (Tue, 04 Oct 2005 08:17:18)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 4 Oct 2005 14:02:25 -0400

Brian,

> Behalf Of Brian F. G. Bidulock
> Sent: Tuesday, October 04, 2005 2:14 PM
> To: Tolga Asveren
> Cc: [email-masked]
[TOLGA]IMHO it is better to detect ASAP, because in that way one can limit
number of lost messages -the moment host failure is detetcted, no messages
will be sent on that association, so this is directly proportional with
difference between times of actual host failure and when it is detected.
Let's wait for a draft summarizing issues discussed on this thread -and
maybe a bit more- and then we can work on it to see whether something needs
to be done and if yes what.