Subject: [tsvwg] SCTP: abort after excessive errors?

From: Jeff Morriss <[email-masked]>
Date: Thu, 01 Nov 2012 11:23:54 -0400

Hi list,

I recently had a problem at an interop event where my system was 
responding to message from an SCTP peer with an ABORT.  A little 
analysis showed me the problem:

- the T-bit was set in the ABORT
- (and, after more digging) the application had received notification 
that the association in question had been closed some time before

So my local SCTP had torn down the association but neither I (looking in 
Wireshark but not at the application logs) nor the peer were made aware 
of it.

More analysis in Wireshark showed that my system's SCTP had experienced 
a sufficient number of association errors (lack of responses from the 
peer) to bring down the association (the peer clearly had not 
experienced that many errors since it kept the assoc up).

What confuses me is that I would expect SCTP to send an ABORT when this 
happens; other implementations with which I am familiar do.

In re-reading the RFC, the relevant section doesn't explicitly mention 
sending an ABORT:

~~~
  8.1. Endpoint Failure Detection
  An endpoint shall keep a counter on the total number of consecutive
  retransmissions to its peer (this includes retransmissions to all the
  destination transport addresses of the peer if it is multi-homed),
  including unacknowledged HEARTBEAT chunks. If the value of this
  counter exceeds the limit indicated in the protocol parameter
  ’Association.Max.Retrans’, the endpoint shall consider the peer
  endpoint unreachable and shall stop transmitting any more data to it
  (and thus the association enters the CLOSED state). In addition, the
  endpoint MAY report the failure to the upper layer and optionally
  report back all outstanding user data remaining in its outbound
  queue. The association is automatically closed when the peer
  endpoint becomes unreachable
~~~

But I think that such an association closure is certainly an ungraceful 
one and thus section 9.1 applies:

~~~
9.1.  Abort of an Association

    When an endpoint decides to abort an existing association, it MUST
    send an ABORT chunk to its peer endpoint.  The sender MUST fill in...
~~~

Would you agree?  If so, would a clarification be useful in section 8.1?

Personally I think that an ABORT must be sent in such situations so that 
the peer can be aware of the situation (before a subsequent message or 
heartbeat from the peer finally provokes an ABORT); in my case the peer 
could easily have re-established the association before it came time to 
send the message which my system treated as OOTB.

Regards,
-Jeff

---

From: Randy Stewart <[email-masked]>
Date: Thu, 1 Nov 2012 17:21:29 -0400

Jeff:

No I don't know if I do agree. I don't think you are required to in TCP either. I
do think that *many* implementations (both TCP and SCTP) do this. But lets think
about this, its quite possible the host is unreachable and adding yet another
packet to the network does not gain anything … yes its small … but it is one more
packet.

I think that is why the wording is left loose so that implementations have a choice. Most
do send the ABORT, but not all do. I prefer that this remain an implementation choice… but
I would be glad to hear other opinions..

R
On Nov 1, 2012, at 11:23 AM, Jeff Morriss wrote:

> Hi list,
> 
> I recently had a problem at an interop event where my system was responding to message from an SCTP peer with an ABORT.  A little analysis showed me the problem:
> 
> - (and, after more digging) the application had received notification that the association in question had been closed some time before
> 
> So my local SCTP had torn down the association but neither I (looking in Wireshark but not at the application logs) nor the peer were made aware of it.
> 
> More analysis in Wireshark showed that my system's SCTP had experienced a sufficient number of association errors (lack of responses from the peer) to bring down the association (the peer clearly had not experienced that many errors since it kept the assoc up).
> 
> What confuses me is that I would expect SCTP to send an ABORT when this happens; other implementations with which I am familiar do.
> 
> In re-reading the RFC, the relevant section doesn't explicitly mention sending an ABORT:
> 
> ~~~
> But I think that such an association closure is certainly an ungraceful one and thus section 9.1 applies:
> 
> ~~~
> Personally I think that an ABORT must be sent in such situations so that the peer can be aware of the situation (before a subsequent message or heartbeat from the peer finally provokes an ABORT); in my case the peer could easily have re-established the association before it came time to send the message which my system treated as OOTB.
> 
> Regards,
> -Jeff
> 

-----
Randall Stewart
[email-masked]

---

From: Chris Benson <[email-masked]>
Date: Thu, 1 Nov 2012 14:47:49 -0700 (PDT)

Jeff,

As stated at the beginning of Section 8.1, this error condition
is reached when there has been no response in the last N
retransmission periods, and as you quoted ...

"[this] endpoint shall consider the peer endpoint unreachable"

I am sure it is this logic that suggests that an ABORT message 
won't get there either, though there's probably little harm in
trying.

To address your concern at the lack of notification, the remote 
peer will (!) notice that this side has disappeared, by reaching 
the same condition at least due to unacknowledged SCTP 
HEARTBEATs if no DATA is being transmitted. The local 
application will (!) be informed by the compulsory indication
issed by the local SCTP as described in RFC 4960 Section 10.2. 
E) "COMMUNICATION LOST notification" to the local Upper Layer
Protocol.

With best regards, from Chris Benson.

On Thu, 1 Nov 2012, Jeff Morriss wrote:

>>  Date: Thu, 01 Nov 2012 11:23:54 -0400
>>  From: Jeff Morriss <[email-masked]>
>>  To: "[email-masked]" <[email-masked]>
>>  Subject: [tsvwg] SCTP: abort after excessive errors?
>>  
>>  Hi list,
>>  
>>  I recently had a problem at an interop event where my system was responding
>>  to message from an SCTP peer with an ABORT.  A little analysis showed me the
>>  problem:
>>  
>>  - (and, after more digging) the application had received notification that
>>  the association in question had been closed some time before
>>  
>>  Wireshark but not at the application logs) nor the peer were made aware of
>>  it.
>>  
>>  More analysis in Wireshark showed that my system's SCTP had experienced a
>>  sufficient number of association errors (lack of responses from the peer) to
>>  bring down the association (the peer clearly had not experienced that many
>>  errors since it kept the assoc up).
>>  
>>  ~ ~ ~
>>  ~ ~ ~
>>  ~ ~ ~
>>  ~ ~ ~
>>  Personally I think that an ABORT must be sent in such situations so that the
>>  peer can be aware of the situation (before a subsequent message or heartbeat
>>  from the peer finally provokes an ABORT); in my case the peer could easily
>>  have re-established the association before it came time to send the message
>>  which my system treated as OOTB.
>>  
>>  Regards,
>>  -Jeff
>>

---

From: Jeff Morriss <[email-masked]>
Date: Thu, 01 Nov 2012 18:56:39 -0400

Hi Chris,

In this particular case the peer was not down (and neither was the 
network) there was simply "excessive" packet loss between the two (I 
think the network was overloaded--certainly not by this association). 
My system's SCTP had the *bad luck* to see N consecutive failures 
(retransmissions and/or lack of heartbeat acknowledgements) but the peer 
obviously did not: it kept the association up and sent us another data 
packet long (seconds or 10's or seconds?--nuts, where did that capture 
file go?) after my system tore down the association.

I understand that after N consecutive failures the chances of the ABORT 
making it may not be that great but the peer was able--after it got the 
ABORT we did send--to immediately reestablish the association.

Regards,
-Jeff

---

From: Chris Benson <[email-masked]>
Date: Thu, 1 Nov 2012 16:39:59 -0700 (PDT)

Jeff,

There are current efforts to improve SCTP failover and I haven't
been able to follow them closely. Perhaps others on this list
could suggest if these new practices could show improvement in 
your "almost unusable" network conditions.

I understand your frustration. SCTP doesn't do well in such 
network conditions. If the packets were just being delayed
(not lost forever), then you could set RTO to be longer, but 
then it would take correspondingly longer to notice any
actual loss of a peer. The same applies to any effort to
increase the value of Association.Max.Retrans (N) to handle
truly dropped packets.

As network delays start to exceed RTO, SCTP of course INCREASES 
its network usage (with frequent retransmissions), often making 
bad conditions worse.

To be very specific about sending an ABORT in this circumstance,
Section 8.1 describes an automatic (instant and silent) closure 
of the association. This is not the same as the SCTP meaning of 
ABORTing an association described in Section 9.1.  I think
Randy Stewart observed that an implementation MAY send an
ABORT message in this scenario. Your preference for a MUST
here would only cover your scenario where it could be argued
that the SCTP endpoints are not correctly configured for the 
network condition actually encountered.

SCTP does really well when multi-homed on actually distinct
networks (and when at least one network has a decent chance
of getting a packet through fairly often).

A potential chance of improvement might lie in metering the 
application data to avoid bursts. This might significantly
lessen the chance of suffering N consecutive dropped packets.
In a non-SCTP context, this helped me greatly one time - the 
data metering was the accidental result of modifying UDP packets
en route.  By metering, I mean evening out the time gap
between consecutive packets rather than sending many "at 
once" followed by a significant silence.

Chris Benson.

On Thu, 1 Nov 2012, Jeff Morriss wrote:

>>  Date: Thu, 01 Nov 2012 18:56:39 -0400
>>  To: Chris Benson <[email-masked]>
>>  Cc: "[email-masked]" <[email-masked]>
>>  Subject: Re: [tsvwg] SCTP: abort after excessive errors?
>>  
>>  Hi Chris,
>>  
>>  In this particular case the peer was not down (and neither was the network)
>>  there was simply "excessive" packet loss between the two (I think the
>>  network was overloaded--certainly not by this association). My system's SCTP
>>  had the *bad luck* to see N consecutive failures (retransmissions and/or
>>  lack of heartbeat acknowledgements) but the peer obviously did not: it kept
>>  the association up and sent us another data packet long (seconds or 10's or
>>  seconds?--nuts, where did that capture file go?) after my system tore down
>>  the association.
>>  
>>  > I am sure it is this logic that suggests that an ABORT message won't get
>>  > there either, though there's probably little harm in
>>  > trying.
>>  > 
>>  > To address your concern at the lack of notification, the remote peer will
>>  > (!) notice that this side has disappeared, by reaching the same condition
>>  > at least due to unacknowledged SCTP HEARTBEATs if no DATA is being
>>  > transmitted. The local application will (!) be informed by the compulsory
>>  > indication
>>  > issed by the local SCTP as described in RFC 4960 Section 10.2. E)
>>  > "COMMUNICATION LOST notification" to the local Upper Layer
>>  > Protocol.
>>  > 
>>  > > >  responding
>>  > > >  that
>>  > > >  But I think that such an association closure is certainly an
>>  > > >  ungraceful one
>>  > > >  and thus section 9.1 applies:
>>  > > >  
>>  > > > ~ ~ ~

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 1 Nov 2012 21:58:23 -0600

Randy,

I agree, it doesn't really matter if an ABORT is sent.  In Jeff's
case, the peer (still thinking the association is operational)
should continue to send heartbeats even if it has no data to
send and will either mark the association down if enough heartbeats
get lost, or when the side that has marked the association failed
responds with a T-ABORT to a received heartbeat.

SCTP is not as responsive to error conditions when the user data
is only flowing in one direction.  The peer with nothing to send
can only rely on heartbeats or other data (such as ARP failures
or ICMP) to detect failures.

As you say, the ABORT might no make it through either, but the
heartbeat mechanism will fail the non-sending peer anyway.  If
one needs more responsiveness in this situation, reducing the
heartbeat interval at the non-sending peer is a good way of 
restoring the responsiveness that would be acheived by the sending
peer.

--brian

Randy Stewart wrote:                                      (Thu, 01 Nov 2012 17:21:29)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randy Stewart <[email-masked]>
Date: Fri, 2 Nov 2012 08:48:51 -0400

Chris:

One comment on this ;-)

> once" followed by a significant silence.

SCTP in FreeBSD has had max_burst (defaulting to 4 packets) for years for this
very issue. I did not realize how effective in loaded network conditions this is
until I actually was playing with TCP in FreeBSD (which does not have this yet), and
I put it in.

It was amazing the difference in network conditions even with the same load by
simply having the max-burst set to 4. 

Its not a panacea by any stretch, but in situations like this (overloaded networks) there really
is *no* such thing. You can only make small incremental improvements, network wide, to try
to help the situation..

R

-----
Randall Stewart
[email-masked]

---

From: Jeff Morriss <[email-masked]>
Date: Fri, 02 Nov 2012 17:25:02 -0400

>> increase the value of Association.Max.Retrans (N) to handle

In fact I don't really mind the fact that the association is bouncing: 
it's life when you don't control the network and somebody else's traffic 
is (occasionally) beating the snot out of it.

> is *no* such thing. You can only make small incremental improvements, network wide, to try

Just FWIW: in one case that I found (the "problem" happened many times) 
the path errors were spread out over about 11 seconds (not really a 
burst).  And it really was bad luck that the path errors happened as the 
peer got a data to us during that time, we responded, and the peer 
SACKed that data (of course data from the peer does not clear my 
path/assoc error count).  Then 5 seconds passed before the peer sent us 
another message which we saw as OOTB.

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 2 Nov 2012 19:40:08 -0400

Hi Jeff,

I do understand that you prefer the sending of the ABORT, I actually agree with you.

But I think it is a good way to leave this open (or a "MAY send an ABORT"), since
as Randy indicates, in general it doesn't make sense for the peer to send another packet,
if several before it haven't made it. However, there might be situations, where the
ABORT makes it:
1. Just because it is lucky and doesn't get dropped.
2. Because it is shorter and the other packets were dropped and IP fragmentation
   didn't kick in
3. ...
But the reason I prefer the sending is that you have some indication on what is
going on in tracefiles...

So in summary, I think leaving it open is an acceptable compromise.

Best regards
Michael
On Nov 2, 2012, at 5:25 PM, Jeff Morriss wrote:

>>> could suggest if these new practices could show improvement in your "almost unusable" network conditions.
>>> 
>>> I understand your frustration. SCTP doesn't do well in such network conditions. If the packets were just being delayed
>>> (not lost forever), then you could set RTO to be longer, but then it would take correspondingly longer to notice any
> In fact I don't really mind the fact that the association is bouncing: it's life when you don't control the network and somebody else's traffic is (occasionally) beating the snot out of it.
> 
>>> A potential chance of improvement might lie in metering the application data to avoid bursts. This might significantly
>>> In a non-SCTP context, this helped me greatly one time - the data metering was the accidental result of modifying UDP packets
>>> between consecutive packets rather than sending many "at once" followed by a significant silence.
>>> 
>> simply having the max-burst set to 4. Its not a panacea by any stretch, but in situations like this (overloaded networks) there really
> Just FWIW: in one case that I found (the "problem" happened many times) the path errors were spread out over about 11 seconds (not really a burst).  And it really was bad luck that the path errors happened as the peer got a data to us during that time, we responded, and the peer SACKed that data (of course data from the peer does not clear my path/assoc error count).  Then 5 seconds passed before the peer sent us another message which we saw as OOTB.
>