Subject: [tsvwg] path failure detection in SCTP

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 14 Dec 2009 03:15:07 +0900 (JST)

Hello,

I have prepared a very simle I-D for path failure detection in SCTP 
based on the discussion on this ML.

The i-d can be seen on the following URL:
http://tools.ietf.org/search/draft-nishida-sctp-failover-00

This might be a well-known issue in SCTP, but I think it would be better
if we could have a standardized way to address it. It doesn't have to be
a way descrived in the draft. The draft is still rough and premature, 
but I hope it can be a start point to discuss this issue. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Yoshifumi Nishida <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Thu, 13 Aug 2009 17:25:39 +0900 (JST)
Message-ID: <[email-masked]>

 > 
 > From: Randy Stewart <[email-masked]>
 > Subject: Re: [tsvwg] Q about path failure detection in SCTP
 > Date: Wed, 12 Aug 2009 06:07:29 -0700
 > Message-ID: <[email-masked]>
 > 
 >  > > From: Michael Tüxen <[email-masked]>
 >  > > Subject: Re: [tsvwg] Q about path failure detection in SCTP
 >  > > Date: Tue, 11 Aug 2009 21:50:38 +0200
 >  > > Message-ID: <[email-masked]>
 >  > >
 >  > >> On Aug 11, 2009, at 9:27 PM, David Lehmann wrote:
 >  > >>
 >  > >>> Yoshifumi raised a good point.
 >  > >>>
 >  > >>> Waiting for the primary destination to time out doesn't fly with
 >  > >>> telecom
 >  > >>> apps.  Our implementation has being doing something similar for years
 >  > >>> to
 >  > >>> what Yoshifumi has suggested.  Our algorithm checks the primary path.
 >  > >>> If the error count is zero, use it.  Otherwise, look for a path with
 >  > >>> the
 >  > >>> least amount of errors.  As Randall mentioned, the spec says "SHOULD";
 >  > >>> so it is RFC compliant.
 >  > >
 >  > >> This might be a good idea in a telco environment where the different
 >  > >> paths share no common link. But if for example all paths share a
 >  > >> single
 >  > >> link and this one is responsable for the loss observed, switching
 >  > >> to a different path is counter productive.
 >  > >
 >  > > I think it's not very harmful even if all paths share a single link.
 >  > >
 >  > > If all paths share a single link and the shared link is congested,
 >  > > even with normal SCTP, it tries to retransmit data to the non-primary
 >  > > paths.
 >  > > Hence, the data go toward the shared congested link anyhow.
 >  > >
 >  > > The slight difference between the current SCTP and my proposal is
 >  > > heart-beat
 >  > > packets transmitted to the primary path. Yes, this is a negative
 >  > > factor
 >  > > in this approach, but I tend to believe it's mostly acceptable.
 >  > 
 >  > Actually, I believe we do something along these lines
 >  > when a congestion event occurs anyway. This came out
 >  > of Armando's work and was not in violation of the RFC... since
 >  > you can send a HB at any time ;-)
 >  > 
 >  > What I think you really want is what Armando studied in
 >  > his PHd Thesis. And showed to have no real harm.. i.e
 >  > a two level trigger... can't remember his terms for it..
 >  > 
 >  > Lets call it Trigger 1 and Trigger 2..
 >  > 
 >  > When you hit T1 you switch from your active to your
 >  > secondary until a HB is ack'd on the old primary.
 >  > He might have even stayed on the secondary.. I know
 >  > he looked at this. Since when you switch you pretty
 >  > much enter slow start and start ramping up the secondary
 >  > path... and switching back slows you again...
 >  > 
 >  > If you hit T2 with the old primary it is like hitting
 >  > the normal threshold.. i.e. you tell the user its unreachable.
 >  > 
 >  > He played with different values for the values of T1 and T2 and I
 >  > believe
 >  > setting T1 to 1 (which is what you are proposing) was not shown to
 >  > be harmful.
 >  > 
 >  > You really should google his work.. in fact maybe he will chime in
 >  > here since I know he still watches this list...
 >  > 
 >  > Armando? You out there (probably just to busy at BBN ;-D)?
 >  > 
 >  > 
 >  > Anyway, I think doing what you propose is within the bounds of the
 >  > spec
 >  > since what it goes against is in the SHOULD department.
 >  > 
 >  > It might be worth writing up a separate I-D documenting how it
 >  > works and why you can try it. I would think one would want to
 >  > pitch it has experimental.. I would also think one should offer
 >  > to have Armando on board as a co-author if he is interested.
 >  > 
 >  > Of course once the M-TCP folks figure out the right congestion
 >  > control,
 >  > I will simply turn-on by default SCTP-CMT in FreeBSD after we put in
 >  > whatever the CC is they come up with...  that is, in my opinion, the
 >  > only good thing to come out of the M-TCP work ;-)
 > 
 > OK. I'll check Armando's work and other related stuff again and try to prepare I-D 
 > and see how many people have an interest.
 > Thanks for your advice. 
 > --
 > Yoshifumi Nishida
 > [email-masked]

---

From: Alberto Cortés <[email-masked]>
Date: Mon, 14 Dec 2009 09:34:28 +0100

About Section 4.2, 6th parrapraph:

During Potential Failure heartbeats may be sent on the primary
interface each PFHB.Interval instead of HB.Interval + RTO.

Does this mean that HBs are sent periodically instead exponentially delayed?

Is the RTO going to grow with each un-acknowledge heartbeat?

On Sun, Dec 13, 2009 at 7:15 PM, Yoshifumi Nishida
<[email-masked]> wrote:
>  >
>  > From: Randy Stewart <[email-masked]>
>  > Subject: Re: [tsvwg] Q about path failure detection in SCTP
>  > Date: Wed, 12 Aug 2009 06:07:29 -0700
>  > Message-ID: <[email-masked]>
>  >
>  >  > > From: Michael Tüxen <[email-masked]>
>  >  > > Subject: Re: [tsvwg] Q about path failure detection in SCTP
>  >  > > Date: Tue, 11 Aug 2009 21:50:38 +0200
>  >  > > Message-ID: <[email-masked]>
>  >  > >
>  >  > >> On Aug 11, 2009, at 9:27 PM, David Lehmann wrote:
>  >  > >>
>  >  > >>> Yoshifumi raised a good point.
>  >  > >>>
>  >  > >>> Waiting for the primary destination to time out doesn't fly with
>  >  > >>> telecom
>  >  > >>> apps.  Our implementation has being doing something similar for years
>  >  > >>> to
>  >  > >>> what Yoshifumi has suggested.  Our algorithm checks the primary path.
>  >  > >>> If the error count is zero, use it.  Otherwise, look for a path with
>  >  > >>> the
>  >  > >>> least amount of errors.  As Randall mentioned, the spec says "SHOULD";
>  >  > >>> so it is RFC compliant.
>  >  > >
>  >  > >> This might be a good idea in a telco environment where the different
>  >  > >> paths share no common link. But if for example all paths share a
>  >  > >> single
>  >  > >> link and this one is responsable for the loss observed, switching
>  >  > >> to a different path is counter productive.
>  >  > >
>  >  > > I think it's not very harmful even if all paths share a single link.
>  >  > >
>  >  > > If all paths share a single link and the shared link is congested,
>  >  > > even with normal SCTP, it tries to retransmit data to the non-primary
>  >  > > paths.
>  >  > > Hence, the data go toward the shared congested link anyhow.
>  >  > >
>  >  > > The slight difference between the current SCTP and my proposal is
>  >  > > heart-beat
>  >  > > packets transmitted to the primary path. Yes, this is a negative
>  >  > > factor
>  >  > > in this approach, but I tend to believe it's mostly acceptable.
>  >  >
>  >  > Actually, I believe we do something along these lines
>  >  > when a congestion event occurs anyway. This came out
>  >  > of Armando's work and was not in violation of the RFC... since
>  >  > you can send a HB at any time ;-)
>  >  >
>  >  > What I think you really want is what Armando studied in
>  >  > his PHd Thesis. And showed to have no real harm.. i.e
>  >  > a two level trigger... can't remember his terms for it..
>  >  >
>  >  > Lets call it Trigger 1 and Trigger 2..
>  >  >
>  >  > When you hit T1 you switch from your active to your
>  >  > secondary until a HB is ack'd on the old primary.
>  >  > He might have even stayed on the secondary.. I know
>  >  > he looked at this. Since when you switch you pretty
>  >  > much enter slow start and start ramping up the secondary
>  >  > path... and switching back slows you again...
>  >  >
>  >  > If you hit T2 with the old primary it is like hitting
>  >  > the normal threshold.. i.e. you tell the user its unreachable.
>  >  >
>  >  > He played with different values for the values of T1 and T2 and I
>  >  > believe
>  >  > setting T1 to 1 (which is what you are proposing) was not shown to
>  >  > be harmful.
>  >  >
>  >  > You really should google his work.. in fact maybe he will chime in
>  >  > here since I know he still watches this list...
>  >  >
>  >  > Armando? You out there (probably just to busy at BBN ;-D)?
>  >  >
>  >  >
>  >  > Anyway, I think doing what you propose is within the bounds of the
>  >  > spec
>  >  > since what it goes against is in the SHOULD department.
>  >  >
>  >  > It might be worth writing up a separate I-D documenting how it
>  >  > works and why you can try it. I would think one would want to
>  >  > pitch it has experimental.. I would also think one should offer
>  >  > to have Armando on board as a co-author if he is interested.
>  >  >
>  >  > Of course once the M-TCP folks figure out the right congestion
>  >  > control,
>  >  > I will simply turn-on by default SCTP-CMT in FreeBSD after we put in
>  >  > whatever the CC is they come up with...  that is, in my opinion, the
>  >  > only good thing to come out of the M-TCP work ;-)
>  >
>  > OK. I'll check Armando's work and other related stuff again and try to prepare I-D
>  > and see how many people have an interest.
>  > Thanks for your advice.
>  > --
>  > Yoshifumi Nishida
>  > [email-masked]
>

-- 
Alberto Cortés
Telematic Engineering Dept. at UC3M

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 14 Dec 2009 17:56:09 +0900 (JST)

Hello Alberto,
Thanks for the comment. It should be delayed by an exponential backoff algorithm.
In the draft, I put the following sentences.

   PFHB.interval is used to determine the interval of heartbeat packets.  
   It is recommended that a heartbeat packet is sent once per RTO of each 
   destination address plus PFHB.interval with jittering of +/- 50% of the RTO value. 

   If the heartbeat is answered, SCTP marks the path Active again.  
   If unanswered, SCTP increments the error count and use an exponential
   backoff algorithm to increase the RTO.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Alberto Cortés <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Mon, 14 Dec 2009 09:34:28 +0100
Message-ID: <[email-masked]>

---

From: Michael Tüxen <[email-masked]>
Date: Mon, 14 Dec 2009 10:02:53 +0100

>   backoff algorithm to increase the RTO.
This seems to be what Solaris is doing. If the error count is not zero,
they sent a HB every RTO, not every HB.Interval+RTO. RTO uses exponential backoff.

Best regards
Michael

---

From: Alberto Cortés <[email-masked]>
Date: Mon, 14 Dec 2009 16:26:27 +0100

For some time, I have been thinking about stop using retransmission
counting as the path failure detection mechanism. This may be not "a
solution that does not require major modification to the current
standard.", as the recent Nishida draft states in section 1:
Introduction, but I think it's on-topic to the audience of this
thread and may be interesting for the draft.

As you know, retransmissions are (exponentially) delayed because the
estimated RTO must be artificially increased after each retransmission
to avoid using the same (small) RTO again and again after a sudden rise
of the current network RTO.

So retransmissions are (exponentially) slow and some applications need
quick path failure detection. This is a trade off. Many solutions
to slow path failure detection, like the ones on Nishida draft, don't
address this fundamental problem.

I will like to use a N probes instead of 1 probe and its N-1
retransmissions for path failure detection:

 - These probes must not be retransmitted, as they carry no data.

 - These probes don't need to be scheduled exponentially, but in the
  best way to detect the failure (research is needed here).

 - To obey the conservation of packets principle, room may be reserved
  in the congestion window for these probes (small probes preferred).

SCTP Heartbeats can be used as these probes, as they are small and they
must be immediately acknowledged. These probes can be part of the
Potential Failure phase of the SCTP connection, for example.

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 15 Dec 2009 16:04:11 +0900 (JST)

Hello Alberto,

I see your point. But, I tend to think the problem is not exponential backoff 
of the RTO. I mean, if we can start using the secondary path without waiting 
for multiple timeouts on the primary path, we can avoid the delay caused by
slow path failure detection.
The idea of Potential Failure state is based on this concept.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Alberto Cortés <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Mon, 14 Dec 2009 16:26:27 +0100
Message-ID: <[email-masked]>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 15 Dec 2009 16:15:47 +0900 (JST)

Hello Michael,

From: Michael Tüxen <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Mon, 14 Dec 2009 10:02:53 +0100
Message-ID: <[email-masked]>

 > they sent a HB every RTO, not every HB.Interval+RTO. RTO uses exponential backoff.

Thanks for the info.
I guess some implementions are doing similar (but somehow different) ways on this.
I think it would be better if we can have a standardized way or at least have 
a guide line for this.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Mitchell Erblich <[email-masked]>
Date: Tue, 15 Dec 2009 00:21:15 -0800

Lets look at this with some common sense.

In "general", IMO,,,,,

Back-off algorithms should be two phases. In the first phase
we are concerned with minimizing our CPU cycles, minimize
wasted bandwidth, etc. However, the second phase should be
primarily concerned with decreasing the latency of recovery.

Thus, the threshold between these two phases and the max
value of back-off SHOULD in most cases be an important
concern.

Mitchell Erblich
==================

---

From: Randy Stewart <[email-masked]>
Date: Tue, 15 Dec 2009 17:10:33 -0800

Nishida-san:

I would suggest one thing. I would rather see the
potentially failed state allow data transmission to a
path when all paths are in PF... not stop sending to the
path.

This would allow the association to struggle on until it
either hits assoc-max-retran or a HB shows up on one of the
PF paths making that the new active path.

I have long thought about having an interim state such as PF
that one could use to quicken failover.. so in general I am
supportive of the idea.

We also need to be careful about the number of HB's sent. But also
I think sending a HB right away when a path enters PF would be
a good idea too.

R

> but I hope it can be a start point to discuss this issue.

-----
Randall Stewart
[email-masked]

---

From: Xiangsong Cui <[email-masked]>
Date: Wed, 16 Dec 2009 11:05:39 +0800

Hi,

I just take a quick look on draft-nishida-*,
IMHO, it is not a good idea that forbidding data transmission
in a potentially failed path, this is much severer than RFC4960.
You will have to extend the interface with ULP, if all the paths
is potentially failed, how to indicate this status? What should
ULP proceed? maybe more issues would appear.

If you just want to quicken failure detection, why not include
"Reduce RTO.Max" in section 4?
HB in higher frequency is also a good idea.

AFAIK, in telecommunication network RTO.Min and
RTO.Max are both less than RFC4960, may be in ms level.

Regards
Xiangsong

----- Original Message ----- 
From: "Randy Stewart" <[email-masked]>
To: "Yoshifumi Nishida" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, December 16, 2009 9:10 AM
Subject: Re: [tsvwg] path failure detection in SCTP

Nishida-san:

R
On Dec 13, 2009, at 10:15 AM, Yoshifumi Nishida wrote:

>>>> Message-ID: <932B0EC1-B00C-4F0D-9876- [email-masked]>
>>>>
>>>> even with normal SCTP, it tries to retransmit data to the non- primary
>>>> paths.

-----
Randall Stewart
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 16 Dec 2009 15:50:19 +0900 (JST)

Hello Randall, Thanks for the comments.

From: Randy Stewart <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Tue, 15 Dec 2009 17:10:33 -0800
Message-ID: <[email-masked]>

 > PF paths making that the new active path.

Yes, I thought about this, but I didn't choose it. 
However, I don't have strong opinion here, if people think it is too much,
I'm happy to change it.

The reason why I didn't is that the Section 8.2 in RFC4960 states that we can 
increase the error counter each time HEARTBEAT is not acknowledged within one RTO. 

So, I thought as long as we keep sending HB packets, it won't cause any issues. 
It can hit assoc.max.retrans properly and can find the recovery of the path. 

 > that one could use to quicken failover.. so in general I am

That's great. Thanks. :)

 > I think sending a HB right away when a path enters PF would be

In my feeling, we can adjust the number of HBs by tuning HB.Max.Burst.
It is recommented to be 1 and I want to keep this value. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 16 Dec 2009 15:50:36 +0900 (JST)

Hello Xiangsong, Thanks for the comments.

From: Xiangsong Cui <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Wed, 16 Dec 2009 11:05:39 +0800
Message-ID: <[email-masked]>

 > ULP proceed? maybe more issues would appear.

Yes, it's severer than RFC4960. 
Since potential failure is just an interim state, eventually the path becomes
either active or inactive. So, I didn't feel that I need to update the interface
with ULP. 
But, I don't have strong opinion here. If it might cause issues, I will change it.

 > RTO.Max are both less than RFC4960, may be in ms level.

That's a good point. Thank you so much. I will update the draft.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Xiangsong Cui <[email-masked]>
Date: Wed, 16 Dec 2009 16:19:49 +0800

Hi Nishida,

let's take the simplest example, if there is only one path
between the SCTP endpoints, when the timer expires,
what to do? stop data transmission?

Regards
Xiangsong

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 16 Dec 2009 17:45:47 +0900 (JST)

Hello Xiansong,

It might be too severe especially for single homed case though, here is an initial idea. 
It sends HBs to the destination. if it unanswered within RTO, increase the 
error count. (as stated in RFC4960)
If it exceeds path.max.retrans, the path will be marked as inactive. 
The behavior in inactive state should be the same as the current spec.
If it exceeds assoc.max.retrans, the association will be terminated.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Xiangsong Cui <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Wed, 16 Dec 2009 16:19:49 +0800
Message-ID: <[email-masked]>

 > > Yes, it's severer than RFC4960. Since potential failure is just an
 > > interim state, eventually the path becomes
 > > with ULP. But, I don't have strong opinion here. If it might cause
 > > issues, I will change it.

---

From: Xiangsong Cui <[email-masked]>
Date: Wed, 16 Dec 2009 17:08:27 +0800

So you are still using the mechanism specified in RFC4960,
if you really want to detect the failure faster, in this case, 
including the data expiration is a better way, that can be an addend.
Data error count + HB error count  >  HB error count, right?

And, one path is the basic case of multihoming, 
how long is the duration of potential failure state in your assumption?

Regards
Xiangsong

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 16 Dec 2009 18:55:46 +0900 (JST)

From: Xiangsong Cui <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Wed, 16 Dec 2009 17:08:27 +0800
Message-ID: <[email-masked]>

 > Data error count + HB error count  >  HB error count, right?

Yep, I think about this again and tend to think it would be better to allow
to send data packet in PF. It's too severe for single-homed case.

 > And, one path is the basic case of multihoming, how long is the
 > duration of potential failure state in your assumption?

Hmm, I don't have concrete assumptions here since I think it fairly 
depends on network conditions. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Alberto Cortés <[email-masked]>
Date: Thu, 17 Dec 2009 17:56:28 +0100

Nishida-sama, I see your point now: you are not interested in
"detecting path failures quickly" but in "using alternative paths
earlier", it's a subtle difference.

As I see things, in general, both concepts are desirable and non
exclusive of each other. Also "detecting path failures quickly" will
certainly help to "use alternative paths earlier". There are good
reasons to address both concepts together, but there are also good
reasons to address them separately.

My confusion came from:
 - section "4.1. Reduce Path.Max.Retrans", seems to deal with
detecting path failures quickly
 - section "4.2. Introduce Potential Failure Status in Failure
Detection Algorithm" seems to deal with using alternative paths
earlier.

Anyway, thanks for your comments, they are most welcome.

On Tue, Dec 15, 2009 at 8:04 AM, Yoshifumi Nishida
<[email-masked]> wrote:
>  > For some time, I have been thinking about stop using retransmission
>  > counting as the path failure detection mechanism. This may be not "a
>  > solution that does not require major modification to the current
>  > standard.", as the recent Nishida draft states in section 1:
>  > Introduction, but I think it's on-topic to the audience of this
>  > thread and may be interesting for the draft.
>  >
>  > As you know, retransmissions are (exponentially) delayed because the
>  > estimated RTO must be artificially increased after each retransmission
>  > to avoid using the same (small) RTO again and again after a sudden rise
>  > of the current network RTO.
>  >
>  > So retransmissions are (exponentially) slow and some applications need
>  > quick path failure detection. This is a trade off. Many solutions
>  > to slow path failure detection, like the ones on Nishida draft, don't
>  > address this fundamental problem.
>  >
>  > I will like to use a N probes instead of 1 probe and its N-1
>  > retransmissions for path failure detection:
>  >
>  >  - These probes must not be retransmitted, as they carry no data.
>  >
>  >  - These probes don't need to be scheduled exponentially, but in the
>  >   best way to detect the failure (research is needed here).
>  >
>  >  - To obey the conservation of packets principle, room may be reserved
>  >   in the congestion window for these probes (small probes preferred).
>  >
>  > SCTP Heartbeats can be used as these probes, as they are small and they
>  > must be immediately acknowledged. These probes can be part of the
>  > Potential Failure phase of the SCTP connection, for example.
>  >
>  >
>  >
>  > On Mon, Dec 14, 2009 at 10:02 AM, Michael Tüxen
>  > <[email-masked]> wrote:
>  > > On Dec 14, 2009, at 9:56 AM, Yoshifumi Nishida wrote:
>  > >
>  > >>
>  > >> Hello Alberto,
>  > >> Thanks for the comment. It should be delayed by an exponential backoff algorithm.
>  > >> In the draft, I put the following sentences.
>  > >>
>  > >>   PFHB.interval is used to determine the interval of heartbeat packets.
>  > >>   It is recommended that a heartbeat packet is sent once per RTO of each
>  > >>   destination address plus PFHB.interval with jittering of +/- 50% of the RTO value.
>  > >>
>  > >>   If the heartbeat is answered, SCTP marks the path Active again.
>  > >>   If unanswered, SCTP increments the error count and use an exponential
>  > >>   backoff algorithm to increase the RTO.
>  > > This seems to be what Solaris is doing. If the error count is not zero,
>  > > they sent a HB every RTO, not every HB.Interval+RTO. RTO uses exponential backoff.
>  > >
>  > > Best regards
>  > > Michael
>  > >>
>  > >> Thanks,
>  > >> --
>  > >> Yoshifumi Nishida
>  > >> [email-masked]
>  > >>
>  > >>
>  > >> From: Alberto Cortés <[email-masked]>
>  > >> Subject: Re: [tsvwg] path failure detection in SCTP
>  > >> Date: Mon, 14 Dec 2009 09:34:28 +0100
>  > >> Message-ID: <[email-masked]>
>  > >>
>  > >>> About Section 4.2, 6th parrapraph:
>  > >>>
>  > >>> During Potential Failure heartbeats may be sent on the primary
>  > >>> interface each PFHB.Interval instead of HB.Interval + RTO.
>  > >>>
>  > >>> Does this mean that HBs are sent periodically instead exponentially delayed?
>  > >>>
>  > >>> Is the RTO going to grow with each un-acknowledge heartbeat?
>  > >>>
>  > >>>
>  > >>>
>  > >>> On Sun, Dec 13, 2009 at 7:15 PM, Yoshifumi Nishida
>  > >>> <[email-masked]> wrote:
>  > >>>>
>  > >>>> Hello,
>  > >>>>
>  > >>>> I have prepared a very simle I-D for path failure detection in SCTP
>  > >>>> based on the discussion on this ML.
>  > >>>>
>  > >>>> The i-d can be seen on the following URL:
>  > >>>> http://tools.ietf.org/search/draft-nishida-sctp-failover-00
>  > >>>>
>  > >>>> This might be a well-known issue in SCTP, but I think it would be better
>  > >>>> if we could have a standardized way to address it. It doesn't have to be
>  > >>>> a way descrived in the draft. The draft is still rough and premature,
>  > >>>> but I hope it can be a start point to discuss this issue.
>  > >>>>
>  > >>>> Thanks,
>  > >>>> --
>  > >>>> Yoshifumi Nishida
>  > >>>> [email-masked]
>  > >>>>
>  > >>>>
>  > >>>> From: Yoshifumi Nishida <[email-masked]>
>  > >>>> Subject: Re: [tsvwg] Q about path failure detection in SCTP
>  > >>>> Date: Thu, 13 Aug 2009 17:25:39 +0900 (JST)
>  > >>>> Message-ID: <[email-masked]>
>  > >>>>
>  > >>>>  >
>  > >>>>  > From: Randy Stewart <[email-masked]>
>  > >>>>  > Subject: Re: [tsvwg] Q about path failure detection in SCTP
>  > >>>>  > Date: Wed, 12 Aug 2009 06:07:29 -0700
>  > >>>>  > Message-ID: <[email-masked]>
>  > >>>>  >
>  > >>>>  >  > > From: Michael Tüxen <[email-masked]>
>  > >>>>  >  > > Subject: Re: [tsvwg] Q about path failure detection in SCTP
>  > >>>>  >  > > Date: Tue, 11 Aug 2009 21:50:38 +0200
>  > >>>>  >  > > Message-ID: <[email-masked]>
>  > >>>>  >  > >
>  > >>>>  >  > >> On Aug 11, 2009, at 9:27 PM, David Lehmann wrote:
>  > >>>>  >  > >>
>  > >>>>  >  > >>> Yoshifumi raised a good point.
>  > >>>>  >  > >>>
>  > >>>>  >  > >>> Waiting for the primary destination to time out doesn't fly with
>  > >>>>  >  > >>> telecom
>  > >>>>  >  > >>> apps.  Our implementation has being doing something similar for years
>  > >>>>  >  > >>> to
>  > >>>>  >  > >>> what Yoshifumi has suggested.  Our algorithm checks the primary path.
>  > >>>>  >  > >>> If the error count is zero, use it.  Otherwise, look for a path with
>  > >>>>  >  > >>> the
>  > >>>>  >  > >>> least amount of errors.  As Randall mentioned, the spec says "SHOULD";
>  > >>>>  >  > >>> so it is RFC compliant.
>  > >>>>  >  > >
>  > >>>>  >  > >> This might be a good idea in a telco environment where the different
>  > >>>>  >  > >> paths share no common link. But if for example all paths share a
>  > >>>>  >  > >> single
>  > >>>>  >  > >> link and this one is responsable for the loss observed, switching
>  > >>>>  >  > >> to a different path is counter productive.
>  > >>>>  >  > >
>  > >>>>  >  > > I think it's not very harmful even if all paths share a single link.
>  > >>>>  >  > >
>  > >>>>  >  > > If all paths share a single link and the shared link is congested,
>  > >>>>  >  > > even with normal SCTP, it tries to retransmit data to the non-primary
>  > >>>>  >  > > paths.
>  > >>>>  >  > > Hence, the data go toward the shared congested link anyhow.
>  > >>>>  >  > >
>  > >>>>  >  > > The slight difference between the current SCTP and my proposal is
>  > >>>>  >  > > heart-beat
>  > >>>>  >  > > packets transmitted to the primary path. Yes, this is a negative
>  > >>>>  >  > > factor
>  > >>>>  >  > > in this approach, but I tend to believe it's mostly acceptable.
>  > >>>>  >  >
>  > >>>>  >  > Actually, I believe we do something along these lines
>  > >>>>  >  > when a congestion event occurs anyway. This came out
>  > >>>>  >  > of Armando's work and was not in violation of the RFC... since
>  > >>>>  >  > you can send a HB at any time ;-)
>  > >>>>  >  >
>  > >>>>  >  > What I think you really want is what Armando studied in
>  > >>>>  >  > his PHd Thesis. And showed to have no real harm.. i.e
>  > >>>>  >  > a two level trigger... can't remember his terms for it..
>  > >>>>  >  >
>  > >>>>  >  > Lets call it Trigger 1 and Trigger 2..
>  > >>>>  >  >
>  > >>>>  >  > When you hit T1 you switch from your active to your
>  > >>>>  >  > secondary until a HB is ack'd on the old primary.
>  > >>>>  >  > He might have even stayed on the secondary.. I know
>  > >>>>  >  > he looked at this. Since when you switch you pretty
>  > >>>>  >  > much enter slow start and start ramping up the secondary
>  > >>>>  >  > path... and switching back slows you again...
>  > >>>>  >  >
>  > >>>>  >  > If you hit T2 with the old primary it is like hitting
>  > >>>>  >  > the normal threshold.. i.e. you tell the user its unreachable.
>  > >>>>  >  >
>  > >>>>  >  > He played with different values for the values of T1 and T2 and I
>  > >>>>  >  > believe
>  > >>>>  >  > setting T1 to 1 (which is what you are proposing) was not shown to
>  > >>>>  >  > be harmful.
>  > >>>>  >  >
>  > >>>>  >  > You really should google his work.. in fact maybe he will chime in
>  > >>>>  >  > here since I know he still watches this list...
>  > >>>>  >  >
>  > >>>>  >  > Armando? You out there (probably just to busy at BBN ;-D)?
>  > >>>>  >  >
>  > >>>>  >  >
>  > >>>>  >  > Anyway, I think doing what you propose is within the bounds of the
>  > >>>>  >  > spec
>  > >>>>  >  > since what it goes against is in the SHOULD department.
>  > >>>>  >  >
>  > >>>>  >  > It might be worth writing up a separate I-D documenting how it
>  > >>>>  >  > works and why you can try it. I would think one would want to
>  > >>>>  >  > pitch it has experimental.. I would also think one should offer
>  > >>>>  >  > to have Armando on board as a co-author if he is interested.
>  > >>>>  >  >
>  > >>>>  >  > Of course once the M-TCP folks figure out the right congestion
>  > >>>>  >  > control,
>  > >>>>  >  > I will simply turn-on by default SCTP-CMT in FreeBSD after we put in
>  > >>>>  >  > whatever the CC is they come up with...  that is, in my opinion, the
>  > >>>>  >  > only good thing to come out of the M-TCP work ;-)
>  > >>>>  >
>  > >>>>  > OK. I'll check Armando's work and other related stuff again and try to prepare I-D
>  > >>>>  > and see how many people have an interest.
>  > >>>>  > Thanks for your advice.
>  > >>>>  > --
>  > >>>>  > Yoshifumi Nishida
>  > >>>>  > [email-masked]
>  > >>>>
>  > >>>
>  > >>>
>  > >>>
>  > >>> --
>  > >>> Alberto Cortés
>  > >>> Telematic Engineering Dept. at UC3M
>  > >>
>  > >
>  > >
>  >
>  >
>  >
>  > --
>  > Alberto Cortés
>  > Telematic Engineering Dept. at UC3M
>

---

From: "Preethi Natarajan" <[email-masked]>
Date: Thu, 17 Dec 2009 13:39:37 -0800

This is an interesting proposal, using most of the concepts from the CMT-PF
work. In fact, Randall and I had a discussion about coming up with a PF
state for SCTP sometime last year. 

I have a general comment about the differences in PF-based failure detection
procedure between SCTP vs. CMT. This comment is also relevant to Section 5.1
in the ID, which has the following paragraph:

   "Second, even if all communication paths between end-nodes share the
   same bottleneck, the proposed method does not make situations worse.
   In case of congestion, the current standard tries to transmit data
   packets to the primary during failover, while the proposed method
   tries to explore other destinations.  In any case, the same amount of
   data packets sent to the same bottleneck. "

Since a CMT sender transmits data on all paths, the sender has a more
accurate view of the path characteristics (RTT, RTO, cwnd) than the SCTP
case. In SCTP (and CMT), precise RTT/RTO estimates are critical for faster
failure detection. For example, consider a situation where primary &
secondary share the same bottleneck and this bottleneck has failed. Unless
the sender's RTO estimate for the secondary is as good as the primary,
failure detection would take longer with the PF proposal. I think the
proposal must list this issue. Depending on whether or not this situation is
a common occurrence (which would depend on the environments considered), the
ID might even have to address this issue. 

Comments most welcome,

Preethi

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] 
>On Behalf Of Yoshifumi Nishida
>Sent: Sunday, December 13, 2009 10:15 AM
>To: [email-masked]
>Subject: [tsvwg] path failure detection in SCTP
>
>
>Hello,
>
>I have prepared a very simle I-D for path failure detection in 
>SCTP based on the discussion on this ML.
>
>This might be a well-known issue in SCTP, but I think it would 
>be better if we could have a standardized way to address it. 
>It doesn't have to be a way descrived in the draft. The draft 
>is still rough and premature, but I hope it can be a start 
>point to discuss this issue. 
>
>Thanks,
>--
>Yoshifumi Nishida
>[email-masked]
>
>
> > From: Randy Stewart <[email-masked]>  > Subject: Re: 
>[tsvwg] Q about path failure detection in SCTP  > Date: Wed, 
>12 Aug 2009 06:07:29 -0700  > Message-ID: 
><[email-masked]>
> >
> >  > > Subject: Re: [tsvwg] Q about path failure detection in 
>SCTP  >  > > Date: Tue, 11 Aug 2009 21:50:38 +0200  >  > > 
> >  > >>> Waiting for the primary destination to time out 
>doesn't fly with  >  > >>> telecom  >  > >>> apps.  Our 
>implementation has being doing something similar for years  >  
>> >>> to  >  > >>> what Yoshifumi has suggested.  Our 
>algorithm checks the primary path.
> >  > >>> If the error count is zero, use it.  Otherwise, look 
>for a path with  >  > >>> the  >  > >>> least amount of 
>errors.  As Randall mentioned, the spec says "SHOULD";  >  > 
>>>> so it is RFC compliant.
> >  > >
> >  > >> This might be a good idea in a telco environment 
>where the different  >  > >> paths share no common link. But 
>if for example all paths share a  >  > >> single  >  > >> link 
>and this one is responsable for the loss observed, switching  
> >  > > If all paths share a single link and the shared link 
>is congested,  >  > > even with normal SCTP, it tries to 
>retransmit data to the non-primary  >  > > paths.
> >  > > The slight difference between the current SCTP and my 
>proposal is  >  > > heart-beat  >  > > packets transmitted to 
>the primary path. Yes, this is a negative  >  > > factor  >  > 
> >  > Actually, I believe we do something along these lines  > 
> > when a congestion event occurs anyway. This came out  >  > 
> >  > you can send a HB at any time ;-)  >  >  >  > What I 
>think you really want is what Armando studied in  >  > his PHd 
>Thesis. And showed to have no real harm.. i.e  >  > a two 
>level trigger... can't remember his terms for it..
> >  >
> >  > When you hit T1 you switch from your active to your  >  
> >  > He might have even stayed on the secondary.. I know  >  
>> he looked at this. Since when you switch you pretty  >  > 
>much enter slow start and start ramping up the secondary  >  > 
> >  > If you hit T2 with the old primary it is like hitting  > 
> >  > He played with different values for the values of T1 and 
>T2 and I  >  > believe  >  > setting T1 to 1 (which is what 
>you are proposing) was not shown to  >  > be harmful.
> >  >
> >  > You really should google his work.. in fact maybe he 
>will chime in  >  > here since I know he still watches this list...
> >  >
> >  > Anyway, I think doing what you propose is within the 
>bounds of the  >  > spec  >  > since what it goes against is 
>in the SHOULD department.
> >  >
> >  > It might be worth writing up a separate I-D documenting 
>how it  >  > works and why you can try it. I would think one 
>would want to  >  > pitch it has experimental.. I would also 
>think one should offer  >  > to have Armando on board as a 
>co-author if he is interested.
> >  >
> >  > Of course once the M-TCP folks figure out the right 
>congestion  >  > control,  >  > I will simply turn-on by 
>default SCTP-CMT in FreeBSD after we put in  >  > whatever the 
>CC is they come up with...  that is, in my opinion, the  >  > 
>only good thing to come out of the M-TCP work ;-)  >  > OK. 
>I'll check Armando's work and other related stuff again and 
>try to prepare I-D  > and see how many people have an interest.
> > Thanks for your advice. 
> > --
> > Yoshifumi Nishida
> > [email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 18 Dec 2009 20:14:47 +0900 (JST)

Hello Alberto,

From: Alberto Cortés <[email-masked]>
Subject: Re: [tsvwg] path failure detection in SCTP
Date: Thu, 17 Dec 2009 17:56:28 +0100
Message-ID: <[email-masked]>

 > reasons to address them separately.

Yes, I agree with you that we have two different concepts here and 
I think we had better work on these individually.

 > Detection Algorithm" seems to deal with using alternative paths

Thanks for the comment. I will update the draft to clarify these more precisely. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 18 Dec 2009 20:17:19 +0900 (JST)

Hello Preethi,

From: "Preethi Natarajan" <[email-masked]>
Subject: RE: [tsvwg] path failure detection in SCTP
Date: Thu, 17 Dec 2009 13:39:37 -0800
Message-ID: <[email-masked]>

 > state for SCTP sometime last year. 

Thanks! I also appreciate you agreed with me to be a co-author of the draft. 

 > ID might even have to address this issue. 

I also think it would be better if the sender's RTO estimate is good.
But is there big difference bewteen RTT measurement by HBs and RTT measurement 
by data packets? 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 18 Dec 2009 13:19:50 +0100

> But is there big difference bewteen RTT measurement by HBs and RTT measurement 
It depends... HB based measurements are accurate, DATA chunk based measurements
can contain the SACK delay timer. On a bulk transfer this does not matter,
but in other cases it does. One way around that is to set the I-bit...

Best regards
Michael

---

From: Scott Brim <[email-masked]>
Date: Fri, 18 Dec 2009 08:54:44 -0500

> But is there big difference bewteen RTT measurement by HBs and RTT measurement 

Data packets may, and may not, traverse intermediate routers in the same
way as non-data packets.  It's better to measure behavior with real data
packets.

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 18 Dec 2009 15:07:10 +0100

> way as non-data packets.  It's better to measure behavior with real data
Do you think a router analyses the chunks contained in an SCTP packet
to base any decision on it?

Best regards
Michael

---

From: Randy Stewart <[email-masked]>
Date: Wed, 23 Dec 2009 11:57:02 -0800

> to base any decision on it?

I would highly doubt that this occurs currently... It may happen
in the future though... I am sure Scott was thinking in terms
of TCP... which is easy to detect if data is attached to the
packet. With SCTP it is not so easy since the router would have
to do a deep inspection going through the TLV's to see if a non-control
chunk is present...

I would think its pretty safe to assume no router does this currently  
just
because of the extra work it would involve ;-)

R

-----
Randall Stewart
[email-masked]