Subject: Quick Failover in SCTP

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 17 Mar 2010 16:54:39 +0900 (JST)

Hello people,

I realized the schedule of tsvwg meeting at Anaheim is very tight and 
I might not be able to have enough time for the presentation.

So, I just put the slide on http://www.nishida.org/sctp-pf/

Please check if you're interested. 
We believe the motivation of this draft is simple and straightforward.

Your comments and suggestions are highly welcome.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Yoshifumi Nishida <[email-masked]>
Subject: updated draft for path failure detection in SCTP
Date: Wed, 24 Feb 2010 03:43:01 +0900 (JST)
Message-ID: <[email-masked]>

 > 
 > Hello,
 > Preethi and I submitted the updated I-D for path failure detection in SCTP.
 > We believe this proposal is useful and impotant for SCTP. We hope we will 
 > be allowed to give a presentation for this at Anaheim.
 > If you have any commnets, suggestions, please let us know!
 > 
 > Thanks,
 > --
 > Yoshifumi Nishida
 > [email-masked]
 > 
 > 
 > From: [email-masked]
 > Subject: I-D ACTION:draft-nishida-natarajan-sctp-failover-00.txt 
 > Date: Tue, 23 Feb 2010 08:45:01 -0800 (PST)
 > Message-ID: <[email-masked]>
 > 
 >  > A New Internet-Draft is available from the on-line Internet-Drafts 
 >  > directories.
 >  > 
 >  > 
 >  > 	Title		: Quick Failover Algorithm in SCTP
 >  > 	Author(s)	: Y. Nishida, P. Natarajan
 >  > 	Filename	: draft-nishida-natarajan-sctp-failover-00.txt
 >  > 	Pages		: 13
 >  > 	Date		: 2010-2-22
 >  > 	
 >  >    One of the major advantages in SCTP is supporting multi-homing
 >  >    communication.  If an multi-homed end-point has redundant network
 >  >    connections, sctp sessions can have a good chance to survive from
 >  >    network failures by migrating inactive network to active one.
 >  >    However, if we follow the SCTP standard, there can be significant
 >  >    delay for the network migration.  During this migration period, SCTP
 >  >    cannot transmit much data to the destination.  This issue drastically
 >  >    impairs the usability of SCTP in some situations.  This memo
 >  >    describes the issue of SCTP failover mechanism and discuss its
 >  >    solutions which require minimal modification to the current standard.
 >  > 
 >  > A URL for this Internet-Draft is:
 >  > http://www.ietf.org/internet-drafts/draft-nishida-natarajan-sctp-failover-00.txt
 >  > 
 >  > Internet-Drafts are also available by anonymous FTP at:
 >  > ftp://ftp.ietf.org/internet-drafts/
 >  > 
 >  > Below is the data which will enable a MIME compliant mail reader
 >  > implementation to automatically retrieve the ASCII version of the
 >  > Internet-Draft.

---

From: Scott Brim <[email-masked]>
Date: Wed, 17 Mar 2010 10:00:12 -0400

> Your comments and suggestions are highly welcome.

- Under what conditions is it useful to go back to the "primary" path?
  Why not just stay on the new path?

- CMT is always better :-)

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 18 Mar 2010 04:07:13 +0900 (JST)

Hello Scott,
Thanks for your comments.

From: Scott Brim <[email-masked]>
Subject: Re: Quick Failover in SCTP
Date: Wed, 17 Mar 2010 10:00:12 -0400
Message-ID: <[email-masked]>

 > > Your comments and suggestions are highly welcome.

Very good points. 

 >   Why not just stay on the new path?

We don't know if it's useful or not. I guess it depends on the situations.
The point is it's the behavior of the current spec. 
Before we change this logic to stay on the new path, we will need to 
investigate the impact. So, we prefer to unchange this part.
This point is discussed in Section 5.2 in the draft.

 > - CMT is always better :-)

I agree. :) But, it's not standardized yet. We're not sure when it's
coming. Also, some people may not want concurrent data transfer feature, 
but want to have quick failver mechanism. This draft will be useful for 
such people.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Preethi Natarajan" <[email-masked]>
Date: Wed, 17 Mar 2010 13:13:38 -0700

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] 
>On Behalf Of Yoshifumi Nishida
>Sent: Wednesday, March 17, 2010 12:07 PM
>To: [email-masked]
>Cc: [email-masked]
> > > I realized the schedule of tsvwg meeting at Anaheim is 
>very tight and  > > I might not be able to have enough time 
>for the presentation.
> > >
> > > So, I just put the slide on 
>http://www.nishida.org/sctp-pf/  > >  > > Please check if 
>you're interested. 
>Before we change this logic to stay on the new path, we will 
>need to investigate the impact. So, we prefer to unchange this part.
>I agree. :) But, it's not standardized yet. We're not sure 
>when it's coming. Also, some people may not want concurrent 
>data transfer feature, but want to have quick failver 
>mechanism. This draft will be useful for such people.

I just want to clarify this -- CMT extends SCTP so that the sender transmits
data via multiple paths simultaneously. 

CMT too suffers from path failures. In fact, the PF idea was originally
proposed to improve CMT's performnace during path failures. You can find the
reference in the draft (NATARAJAN08), here is the URL for your convenience
-- http://www.cis.udel.edu/nataraja/papers/networking2008.pdf

Thanks,
Preethi

---

From: "Preethi Natarajan" <[email-masked]>
Date: Wed, 17 Mar 2010 16:04:16 -0700

>From: Preethi Natarajan [mailto:[email-masked]] 
>Sent: Wednesday, March 17, 2010 1:14 PM
>To: 'Yoshifumi Nishida'; [email-masked]'
>Cc: [email-masked]'
>Subject: RE: Quick Failover in SCTP
>
>
>CMT too suffers from path failures. In fact, the PF idea was 
>originally proposed to improve CMT's performnace during path 
>failures. You can find the reference in the draft 
>(NATARAJAN08), here is the URL for your convenience -- 
>http://www.cis.udel.edu/nataraja/papers/networking2008.pdf

Just realized the typo in the above URL, sorry about that. 

Please find the corrected URL:
http://www.cis.udel.edu/~nataraja/papers/networking2008.pdf

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 17 Mar 2010 17:16:51 -0600

Yoshifumi,

Please see comments below.

Yoshifumi Nishida wrote:                  (Thu, 18 Mar 2010 04:07:13)
> This point is discussed in Section 5.2 in the draft.

Have you investigated the impact of not staying on the new path?

Consider a unidirectional data flow (one end only sends, the
other end only receives), having the sender switch to the new
path on the first RTO does not alter how the receiver is sending
SACKs.  On the other hand, heartbeats can be more controlled as
to the source address.  Receiving a successful heartbeat ack for
the PF primary path does not give any assurance that the
receiver can return SACKs to the sender, which can result in
another RTO, another switchover to the new path, another
successful heartbeat, etc.  In fact, if the successful heartbeat
sets RTO for the primary back to RTO.min on each success, one
could expect retransmission of 1 RTO's worth of data from the
primary (even thought it was received, just that the SACKs were
lost) on the new path every PFHB.interval + RTO.min.  The
receiver, getting the duplicate DATA chunks on the new path,
must immediately SACK them and they will credit the new path's
congestion window.

The closer PFHB.interval is to RTO.min, or if it is less than
RTO.min, (or the actual RTO of the path if that is greater than
RTO.min), the more data will be delayed by this full RTO and
retransmitted unnecessarily.

Sticking with the new path would not exhibit this syndrome.

> but want to have quick failver mechanism. This draft will be useful for 

I agree with Scott, CMT is always better.

It performs better and is better studied.

One of SCTP's major benefits is independent streams that do not
exhibit head-of-line blocking in simple and avoidable cases.
Under the regular and even the proposed PF approach, path
failure leads to blocking all streams until the path failover is
successful, regardless of the number of paths.  With CMT, only
streams for which data was lost on the failed path are delayed,
and the amount of lost data decreases linearly with the number
of paths.

So, I am having a hard time trying to figure out which people
these are that want an unstudied experimental approach, where
all of their data streams are delayed longer than necessary,
with no relief from adding paths.

Why would anyone not want concurrent data transfer?  Who are
these people that don't want it, what are their real-world use
cases, and why would concurrent data transfer be so unusable to
them?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Preethi Natarajan" <[email-masked]>
Date: Wed, 17 Mar 2010 16:35:09 -0700

>On Behalf Of Brian F. G. Bidulock
>Sent: Wednesday, March 17, 2010 4:17 PM
>To: Yoshifumi Nishida
>Cc: [email-masked]
>path on the first RTO does not alter how the receiver is 
>sending SACKs.  On the other hand, heartbeats can be more 
>controlled as to the source address.  Receiving a successful 
>heartbeat ack for the PF primary path does not give any 
>assurance that the receiver can return SACKs to the sender, 
>which can result in another RTO, another switchover to the new 
>path, another successful heartbeat, etc.  In fact, if the 
>successful heartbeat sets RTO for the primary back to RTO.min 
>on each success, one could expect retransmission of 1 RTO's 
>worth of data from the primary (even thought it was received, 
>just that the SACKs were
>RTO.min, (or the actual RTO of the path if that is greater 
>than RTO.min), the more data will be delayed by this full RTO 
>and retransmitted unnecessarily.
>
>> coming. Also, some people may not want concurrent data transfer 
>> feature, but want to have quick failver mechanism. This 
>draft will be 
>> useful for such people.
>
>One of SCTP's major benefits is independent streams that do 
>not exhibit head-of-line blocking in simple and avoidable cases.
>failure leads to blocking all streams until the path failover 
>is successful, regardless of the number of paths.  

Not true. With the PF approach, data will go over the secondary after the
first timeout, independent of when the primary is marked failed.

> With CMT, 
>only streams for which data was lost on the failed path are 
>delayed, and the amount of lost data decreases linearly with 
>the number of paths.

First, I must be missing something here, but how does an app pin a stream to
a path in CMT? I.e., how can a CMT sender ensure that all data from one
stream go on the same path? 

Second, even CMT suffers during path failures. The PF idea was originally
proposed with Jana Iyengar for CMT. Please see this reference for more
details -- http://www.cis.udel.edu/~nataraja/papers/networking2008.pdf. In
fact, CMT with PF has been implemented in FreeBSD and is turned on whenever
CMT is turned on. Since PF is a fundamental change to SCTP's failure
detection process, we want to make sure that PF is included in SCTP first
before we do the same for CMT.

Thanks,
Preethi

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 17 Mar 2010 18:20:27 -0600

Preethi,

Preethi Natarajan wrote:                    (Wed, 17 Mar 2010 16:35:09)
> first timeout, independent of when the primary is marked failed.

And all streams are blocked until this first timeout.  Under
CMT, fast-retransmit can fill in the data (for the affected
streams) within an RTT rather than a full RTO.   Also, unaffected
streams experience no delay.  Thus CMT can respond in milliseconds
or microseconds for affected streams where, according to the RFC,
PF must block all streams for at least 1 second.

> stream go on the same path? 

It doesn't have to.  Only streams for which ordered data chunks
have actually been lost are blocked.

> before we do the same for CMT.

Not all CMT approaches suffer as do Jana's approach.  (I use
CMT to refer to a class of approaches rather than to refer to
Jana's specific algorithms.)  PF is a first-order approximation
of more sophisticated approaches.

When going to transmit, my current approach weights
routes

 10, for being routable with excessive retransissions

 11, for being routable with non-excessive retransmission,
	but not having an available send window

 12, for being routable, with non-excessive retransmissions,
	and having an available send window

	+1, for no retransmissions
	+1, for no duplicates
	+2, for having a send window that allows a maximum
	    sized packet
	+1, not being in slow start
	-1, for being the same as the last route sent to

The highest weighted route is selected for the current
transmission cycle.

Ties are broken in favor of:

 - routes with the least excessive timeouts
 - routes with the least duplicates
 - routes with the lowest SRTT
 - routes not in slow start
 - routes most recently used
 - routes with the best congestion window
 - routes with the lowest RTO

As you can see, having a timeout is only a small part of the
may criteria that can be used to rate routes.

The above approach was tested by Lockheed Martin and BBN
and demonstrated complete recovery from path failures in
20 milliseconds (on a system with a 10 millisecond RTO
granularity).

In the SIGTRAN space, there are many vendors that have
implemented SCTP stacks will similar criteria that transcend
CMT-PF, as well as algorithms, (no, not Jana's) for suppressing
spurious gap reports at the receiver due to misordered packets,
 detecting SACK loss with duplicates, avoiding parasitic
fast-retransmit from superious gap reports at the sender.

So, when I say CMT, I mean much more than the specific CMT
and CMT-PF algorithms presented in the papers you cite.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 18 Mar 2010 16:22:42 +0900 (JST)

Hello Brian,

Thanks for the comments.

From: "Brian F. G. Bidulock" <[email-masked]>
Subject: Re: Quick Failover in SCTP
Date: Wed, 17 Mar 2010 17:16:51 -0600
Message-ID: <[email-masked]>

 > must immediately SACK them and they will credit the new path's

Sorry. I might miss your point here. According to RFC4960,

   An endpoint SHOULD transmit reply chunks (e.g., SACK, HEARTBEAT ACK,
   etc.) to the same destination transport address from which it
   received the DATA or control chunk to which it is replying. 

What is the situation where receiver can return heartbeat ack, 
but it cannot return SACK? 

 > cases, and why would concurrent data transfer be so unusable to

I hope some of the following answers make sense to you. 
(I'm crossing my fingers.)

 o CMT is not standardized yet. I think it would be better if we 
   don't have to say "please wait until CMT has come"

 o Some people might use secondary path only for emergency back-up
   from some reasons. (cost or policy?)

 o CMT might add another complexity. Small gadgets like mobile phone 
   might prefer simple implenmetaion.

 o CMT will not be very important for non-bulk flows, while quick 
   failover can still be important.

 o In any case, giving choices (use CMT or just SCTP) to users will not 
   be a bad thing. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 18 Mar 2010 06:10:42 -0600

Yoshifumi,

Please see comments below:

Yoshifumi Nishida wrote:                  (Thu, 18 Mar 2010 16:22:42)
> but it cannot return SACK? 

Quite simply where HEARTBEAT arrives with a different source
address from the DATA chunks: then the HEARTBEAT ACK will be
sent to a different destination than the SACKs.

>    don't have to say "please wait until CMT has come"

PF is not only not standardized, on its own it is not even
studied.  At this point it would be better to attempt to
standardize something that has at least been studied.  (But
see below.)

>    from some reasons. (cost or policy?)

And then they might not.

>    might prefer simple implenmetaion.

I found CMT easier to implement and more efficient than the
stock RFC4960 approach.  Multi-homing is itself optional and
adds complexity.  Small gadgets can choose single-homed and
avoid the complexity of multi-homing, PF and CMT all at once.
I don't think this is a case for PF.

>    failover can still be important.

Your assertion is unsupported: for non-bulk flows CMT provides
superior failover and lower latency by keeping destination path
estimates accurate.  PF does not.

>  o In any case, giving choices (use CMT or just SCTP) to users will not 

Choice is good, but standardizing on a narrow implementation
of algorithms that do not affect interoperability is bad.

None of these approaches need affect the on-the-wire protocol
nor interoperability and are in the domain of implementation
decisions (or suggestions) rather than protocol standardization.

The inability to point to a domain of application where PF
meets specific requirements really brings its usefulness into
question.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Preethi Natarajan" <[email-masked]>
Date: Thu, 18 Mar 2010 11:38:37 -0700

>From: Brian F. G. Bidulock [mailto:[email-masked]] 
>Sent: Wednesday, March 17, 2010 5:20 PM
>To: Preethi Natarajan
>Cc: 'Yoshifumi Nishida'; [email-masked]
>> >Under the regular and even the proposed PF approach, path failure 
>> >leads to blocking all streams until the path failover is 
>successful, 
>> >regardless of the number of paths.
>> 
>> Not true. With the PF approach, data will go over the 
>secondary after 
>> the first timeout, independent of when the primary is marked failed.
>
>And all streams are blocked until this first timeout. 

Not true. This will depend on SCTP's retransmission policy -- RTX_ALT or
RTX_SAME. 

> Under 
>streams experience no delay.  Thus CMT can respond in 
>milliseconds or microseconds for affected streams where, 
>according to the RFC, PF must block all streams for at least 1 second.

Yes, depending on SCTP's retransmission policy, SCTP (or SCTP-PF) _may_
block all streams for 1 sec. 

However, what I am wondering about is this -- the draft proposes a quick
failover for SCTP, and your arguments imply that CMT (any form of concurrent
transfer on all paths) must be the solution to improve SCTP's failover
instead of just improving SCTP's failover algorithm. Lets say I am an SCTP
application developer and want to use just SCTP and not CMT, what do I do?
For example, I know that HTTP or video streaming works well over
SCTP/PR-SCTP. The implications of concurrent transfer (data reordering) on
these applications are still unexplored. So I don't want to use CMT yet for
these applications. Does that mean apps shouldn't be taking advantage of
SCTP with a better failover algorithm?

>delayed, 
>> >and the amount of lost data decreases linearly with the number of 
>> >paths.
>> 
>> First, I must be missing something here, but how does an app pin a 
>> stream to a path in CMT? I.e., how can a CMT sender ensure that all 
>> data from one stream go on the same path?
>
>It doesn't have to.  Only streams for which ordered data 
>chunks have actually been lost are blocked.

OK. Thanks for clarifying. I assumed something else and your response now
clears things.

>
>
>> Second, even CMT suffers during path failures. The PF idea was 
>> originally proposed with Jana Iyengar for CMT. Please see this 
>> reference for more details -- 
>> http://www.cis.udel.edu/~nataraja/papers/networking2008.pdf. 
>In fact, 
>> CMT with PF has been implemented in FreeBSD and is turned on 
>whenever 
>> detection process, we want to make sure that PF is included 
>in SCTP first before we do the same for CMT.
>
>Jana's specific algorithms.)  PF is a first-order 
>approximation of more sophisticated approaches.
>
>When going to transmit, my current approach weights routes
>

To me, an SCTP sender becomes a CMT sender when the sender transmits data
simultaneously over all active paths. Given that, what you describe above
seem to be various (re)transmission policies for CMT. Jana's work proposed
similar policies for retransmissions (RTX_CWND, RTX_SSTHRESH, RTX_LOSSRATE
etc) but your policies are applied even for new data transmissions. Is that
the difference? 

>The above approach was tested by Lockheed Martin and BBN and 
>demonstrated complete recovery from path failures in 20 
>milliseconds (on a system with a 10 millisecond RTO granularity).

A sender detects path failure based on PMR value. So, I assume by 'complete
recovery from path failure in 20 ms' you mean fast-retransmit based loss
recovery and not failure detection. Any sender (your CMT, Jana's CMT,
SCTP/SCTP-PF with RTX_ALT) will recover via fast-retransmits, no?  Or, are
you implying that the 20 ms loss-recovery is somehow faster than
fast-retransmit based loss recovery? 

I am trying to understand how the CMT you describe deals with data
transmission _during failure detection_. Lets assume that the CMT sender
transmits data on all _active_ paths. According to 4960, a path is active
until it is marked failed. During failure detection, an SCTP/CMT sender will
transmit new data on this path (even if fast-retransmits go on other paths)
until PMR+1 consecutive timeouts. This data will be lost and can be
recovered only after a timeout. (Note that PF improves performance _during
failure detection_. The advantage with PF is that the sender (SCTP/CMT)
marks a path 'PF' after the first timeout and avoids transmitting _new_ data
on the PF path. So, the impact of path failure is minimized.)

Maybe the CMT sender (in your implementation) does not transmit new data on
all _active_ paths? My co-author and I are curious to know more about your
CMT. Can you point us to drafts/papers/results?

Thanks,
Preethi

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 19 Mar 2010 05:25:36 +0900 (JST)

Hi Brian,

Please let me clarify several points. Our points are:

1: the current sctp has some difficulties in failover 
2: the draft presents 3 remedies for it. PF is one of them.
   we think PF is the best solution in them. But, we just present it so far. 
   PF is merely a simple extention to the current spec.
3: if CMT makes 4960 obsolute, the draft will not be useful.
   But, we're not sure about it so far.
4: if 4960 is still being used, people will need a remedy for failover issue
5: Documenting remedies is the main goal of the draft. Not to make PF standard.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: "Brian F. G. Bidulock" <[email-masked]>
Subject: Re: Quick Failover in SCTP
Date: Thu, 18 Mar 2010 06:10:42 -0600
Message-ID: <[email-masked]>

---

From: Chris Benson <[email-masked]>
Date: Fri, 19 Mar 2010 14:12:12 -0700 (PDT)

Scott,

[This does not directly address Yoshifumi's context,
but may be considered in failover proposals]

>>  - Under what conditions is it useful to go back to the "primary" path?

Well funny you should ask, ....

I know of a real-life situation in which the use of secondary 
path(s) (on leased fibre-optic lines, I think) were "billed"
by usage. The customer had an external financial reason to
prefer an automatic "return to previously designated primary 
path" when it became viable.

For whatever reasons, the SCTP interface definition to the ULP
permits the SCTP-user to SETPRIMARY, regardless of any SRTT 
comparisons. [RFC 4960, Section 10.1, Subsection F)].

Our implementation offered a configurable option to revert
automatically to the originally designated primary when
HEARTBEATs indicated adequately restored SRTT.

One might conjecture other external reasons to want this
besides money, even though I understand your point if all
else were "equal".

Best regards, from Chris Benson.

On Wed, 17 Mar 2010, Scott Brim wrote:

>>  To: Yoshifumi Nishida <[email-masked]>
>>  Cc:  <[email-masked]>