Subject: [tsvwg] QUESTION on Return of SACKs and HB ACKs

From: <[email-masked]>
Date: Tue, 7 May 2013 13:05:43 +0300

Hi,

RFC4960 in Section 6.4 opens up for the possibility that SACKs may be returned on a different path (or IP Path)
that the reflected path (or IP Path) of the incoming Data.

This, as said in RFC4960, can be the situation when a SACK acknowledges Data TSNs that have come in on different paths as well as it
 may (MAY) be a beneficial reaction if evidence is available that indicates the unavailability of the reflected data path. Such unavailability
evidence could come from arrival of duplicate Data chunks (as pointed to in RFC4960) possibly such evidence could also be established
by other means (like e.g., if HBs indicates the reflected Path to be unavailable). The latter aspect are not mentioned in RFC4960.

I am wondering, if anybody (IETF SCTP community and/or RFC4960 Authors) has any view on the validity (or the contrary) of
not returning SACKs on the reflected IP Path ?

Obviously it is very important for the path selection for return of SACKs to be as
stable and consistent as feasible, as one otherwise risks to jeopardize the validity of Data RTT measurements of peer, and using the reflected IP path
is one way to achieve this. But having said this, then I wonder if there otherwise are considered to be significant problems with the
return of SACKs via paths there are different from the reflected IP Path of Data. Issues that makes is so that an SCTP implementation should best possible prevent situations where data and SACK do not follow the same path (in opposite directions).

Possibly also the situation with return of SACKs acknowledging Data chunks that have come in on multiple paths will be even more relevant for CMT-SCTP.
This may open up for either demanding for SACKs to be returned split according to the incoming path of Data or for the recommendation of RFC4960 to use reflected path of Data to be broaden up to recommend for consistent choice of return path of SACKs.

Finally following the thoughts above for SACK return, then one may also think about the corresponding situation with the return of HB ACKs.

RFC4960 seems to assume for HB ACKs always to be returned on the reflected path of the HB as it says, Section 3.3.6, but notably the only Keyword
that applies to return of HB ACK is the SHOULD of Section 6.4 :

A HEARTBEAT ACK is always
   sent to the source IP address of the IP datagram containing the
   HEARTBEAT chunk to which this ack is responding.

Also here one can wonder as to whether it is best to have the HB ACK returned in the path over which it is most likely to reach peer or whether
it must (?) be returned on the reflect path of the incoming HB. I am not sure if the RFC4960 SHOULD here should be read mostly as a MUST or mostly as a
SHOULD unless other things make it beneficiary to use a different path ?

A BIG Thanks in Advance !

BR, Karen

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 8 May 2013 12:01:52 -0700

Hi Karen,

I'm trying to see what kind of benefits we can get by this. Do you
have some use cases?

Thanks,
--
Yoshifumi

On Tue, May 7, 2013 at 3:05 AM,  <[email-masked]> wrote:
> Hi,
>
>
>
> RFC4960 in Section 6.4 opens up for the possibility that SACKs may be
> returned on a different path (or IP Path)
>
> This, as said in RFC4960, can be the situation when a SACK acknowledges Data
> TSNs that have come in on different paths as well as it
>
>  may (MAY) be a beneficial reaction if evidence is available that indicates
> the unavailability of the reflected data path. Such unavailability
>
> evidence could come from arrival of duplicate Data chunks (as pointed to in
> RFC4960) possibly such evidence could also be established
>
> by other means (like e.g., if HBs indicates the reflected Path to be
> unavailable). The latter aspect are not mentioned in RFC4960.
>
>
>
> I am wondering, if anybody (IETF SCTP community and/or RFC4960 Authors) has
> any view on the validity (or the contrary) of
>
> stable and consistent as feasible, as one otherwise risks to jeopardize the
> validity of Data RTT measurements of peer, and using the reflected IP path
>
> is one way to achieve this. But having said this, then I wonder if there
> otherwise are considered to be significant problems with the
>
> return of SACKs via paths there are different from the reflected IP Path of
> Data. Issues that makes is so that an SCTP implementation should best
> possible prevent situations where data and SACK do not follow the same path
> (in opposite directions).
>
>
>
> Possibly also the situation with return of SACKs acknowledging Data chunks
> that have come in on multiple paths will be even more relevant for CMT-SCTP.
>
> This may open up for either demanding for SACKs to be returned split
> according to the incoming path of Data or for the recommendation of RFC4960
> to use reflected path of Data to be broaden up to recommend for consistent
> choice of return path of SACKs.
>
>
>
> Finally following the thoughts above for SACK return, then one may also
> think about the corresponding situation with the return of HB ACKs.
>
>
>
> RFC4960 seems to assume for HB ACKs always to be returned on the reflected
> path of the HB as it says, Section 3.3.6, but notably the only Keyword
>
> Also here one can wonder as to whether it is best to have the HB ACK
> returned in the path over which it is most likely to reach peer or whether
>
> it must (?) be returned on the reflect path of the incoming HB. I am not
> sure if the RFC4960 SHOULD here should be read mostly as a MUST or mostly as
> a
>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 9 May 2013 16:19:27 -0600

Karen,

First off, I would like to discourage you from using the term
"path": SCTP has no concept of a path (largely because IP underneath
it doesn't either).  The only real choice SCTP has over how a packet
gets to the peer is by setting the destination IP address.

So consider whether packets containing response chunks should be
sent to the destination address that was provided as a source
address in the corresponding request chunk, rather than paths.

Indeed, by sending a response packet to the source address of the
packet containing the request may not take the reverse path through
the network that was traversed by the request packet at all.  In
fact, it might not even be the normal case for all but the most
trivial of networks.

Please see more comment inline...

[email-masked] wrote:    (Tue, 07 May 2013 13:05:43)
> 
> I am wondering, if anybody (IETF SCTP community and/or RFC4960
> Authors) has any view on the validity (or the contrary) of not
> returning SACKs on the reflected IP Path ?

IMO there no reason to send SACK and DATA to different destinations.
Bundling SACK and DATA are essential for efficiency at high
throughputs.  The "best" destination should be used for both, where
the "best" destination is the primary DATA destination for non-CMT.

> Possibly also the situation with return of SACKs acknowledging
> Data chunks that have come in on multiple paths will be even more
> relevant for CMT-SCTP.

CMT needs to perform some sort of gap-ack merging to suppress
spurious fast retransmit.  It can be done both at the sender and at
the receiver.  SACKs that make it out of this logic at the sender
should still be sent to the "best" destination and bundled with DATA
wherever possible.  The "best" destination in this case is
determined by the CMT distribution logic.

> Finally following the thoughts above for SACK return, then one may
> also think about the corresponding situation with the return of HB
> ACKs.

> Also here one can wonder as to whether it is best to have the HB
> ACK returned in the path over which it is most likely to reach
> peer or whether it must (?) be returned on the reflect path of the
> incoming HB. I am not sure if the RFC4960 SHOULD here should be
> read mostly as a MUST or mostly as a SHOULD unless other things
> make it beneficiary to use a different path ?

HB are a little different.  COOKIE-ACK, HEARTBEAT-ACK, SHUTDOWN-ACK,
SHUTDOWN-COMPLETE, ASCONF-ACK, should IMO be sent to the source
address contained in the corresponding request chunk, so long as the
destination is considered viable.  Otherwise, it should send to the
"best" destination.

These approaches work quite well in practice.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: <[email-masked]>
Date: Sun, 19 May 2013 17:17:25 +0300

Hi,

Sorry for the wait.
I am coping in Randy and Michael as this touches the interpretation of a SHOULD in the RFC and
it will be very valuable to understand to which extend the solution that we are discussing is seen as being 
in compliance with the RFC, or if not, if it is considered a reasonable, acceptable and perhaps even a recommended deviation from the RFC.
... Thanks in advance .. 

>Bundling SACK and DATA are essential for efficiency at high throughputs.  The
>"best" destination should be used for both, where the "best" destination is the
>primary DATA destination for non-CMT.

Agree completely. Apart from the mere throughput considerations, I would also like to point to
that return of SACK to the Primary DATA destination also should be better from a robustness perspective as
the SACK returner are likely to have fresh and reliable information on the "availability" of the Primary DATA destination (availability = 
data or HB's send to the Primary DATA destination are acknowledge to be received)
While availability of other destination addresses may not be fresh (depending on Path monitoring and HB intervals).
Generally in simple scenarios where one can trust there to be either "bi-directional" connectivity or no connectivity at all
in between end-point addresses, then the receipt of DATA on a path of course is fresh information that the return path works, but this
is not generally the situation in today's network - at least not on a permanent basis. I.e., in failure situations,
connectivity may only work in one direction until the network has recovered (whether by fault recovery or by route migration).

The return of SACK  to the Primary DATA destination will provide more robust delivery of the SACK. If the SACK is not bundled
with forward DATA, e.g., if no data is in transit in this direction, one could then argue for that implementations 
could bundle HBs with solely returned SACKs (though surely one should implement appropriate rate limiting on such HBs)
thereby making sure to keep the path availability information fresh (If the HBs consistently fail, the Primary DATA destination should be changed).

>
>> Possibly also the situation with return of SACKs acknowledging Data
>> chunks that have come in on multiple paths will be even more relevant
>> for CMT-SCTP.
>
>CMT needs to perform some sort of gap-ack merging to suppress spurious fast
>retransmit.  It can be done both at the sender and at the receiver.  SACKs that make
>it out of this logic at the sender should still be sent to the "best" destination and
>bundled with DATA wherever possible.  The "best" destination in this case is
Yes - Thanks.
>> returned in the path over which it is most likely to reach peer or
>> whether it must (?) be returned on the reflect path of the incoming
>> HB. I am not sure if the RFC4960 SHOULD here should be read mostly as
>> a MUST or mostly as a SHOULD unless other things make it beneficiary
>> to use a different path ?
>
>address contained in the corresponding request chunk, so long as the destination is
>considered viable.  Otherwise, it should send to the "best" destination.
>
Sounds good IMO.

>These approaches work quite well in practice.

It could be very interesting to have such C P or B C P be written up, I think.

Thanks a lot.

BR, Karen

---

From: <[email-masked]>
Date: Sun, 19 May 2013 17:49:58 +0300

Hi,

The motivation and/or use-cases behind the discussion is the following:

* The return path of an incoming DATA path may not be working. I.e., there can be connectivity from
source address of incoming DATA to destination address of incoming DATA, but not the other way around.

* The Primary Path of the SACK returner is generally the path which is considered best in the direction that the SACK should take. Would one
then not think that it generally would be best to return also the SACKs on this path.
Indeed if the SACK returner is itself transmitting DATA (which then would be transmitted on the Primary Path) then:
* fresh path availability information is available on the Primary Path as driven by the DATA SACKs
* from throughput perspectives (including packet processing costs in sender, receiver and intermediate routers) it is beneficiary to
bundle the SACK with the DATA, opposed to have them be sent as two different packets.

One argument that motivates SACK return on the "reflected" path of the DATA is to keep the SRTT stable. But as long
as the SACK consistently is returned on the Primary Path the fluctuations that could come from Primary Path changes should be infrequent in time.
The reasoning is that such infrequent fluctuations in SRTT would be an acceptable price to pay to have a more robust SACK return.

BR, Karen

>-----Original Message-----
>From: Yoshifumi Nishida [mailto:[email-masked]]
>Sent: Wednesday, May 08, 2013 9:02 PM
>To: Nielsen Karen
>Cc: tsvwg
>Subject: Re: [tsvwg] QUESTION on Return of SACKs and HB ACKs
>
>Hi Karen,
>
>I'm trying to see what kind of benefits we can get by this. Do you have some use
>cases?
>
>Thanks,
>--
>Yoshifumi
>
>> This, as said in RFC4960, can be the situation when a SACK
>> acknowledges Data TSNs that have come in on different paths as well as
>> it
>>
>>  may (MAY) be a beneficial reaction if evidence is available that
>> indicates the unavailability of the reflected data path. Such
>> unavailability
>>
>> Authors) has any view on the validity (or the contrary) of
>>
>> stable and consistent as feasible, as one otherwise risks to
>> jeopardize the validity of Data RTT measurements of peer, and using
>> the reflected IP path
>>
>> is one way to achieve this. But having said this, then I wonder if
>> there otherwise are considered to be significant problems with the
>>
>> return of SACKs via paths there are different from the reflected IP
>> Path of Data. Issues that makes is so that an SCTP implementation
>> should best possible prevent situations where data and SACK do not
>> follow the same path (in opposite directions).
>>
>>
>>
>> chunks that have come in on multiple paths will be even more relevant for CMT-
>SCTP.
>>
>> according to the incoming path of Data or for the recommendation of
>> RFC4960 to use reflected path of Data to be broaden up to recommend
>> for consistent choice of return path of SACKs.
>>
>>
>>
>> also think about the corresponding situation with the return of HB ACKs.
>>
>>
>>
>> RFC4960 seems to assume for HB ACKs always to be returned on the
>> reflected path of the HB as it says, Section 3.3.6, but notably the
>> only Keyword
>>
>> whether
>> it must (?) be returned on the reflect path of the incoming HB. I am
>> not sure if the RFC4960 SHOULD here should be read mostly as a MUST or
>> mostly as a
>>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 24 May 2013 03:23:57 -0700

Hi Karen,

Thanks for your response. I think there'll be situations where this
approach will be beneficial.
I can see that when the destination is inactive, it would be better to
send SACK or HB ACK to the different path.
I guess this is possible with the current spec by ignoring SHOULD.

However, I'm not very sure if bundling SACK for CMT is always
beneficial. It's sometimes better to have path diversity.
If the return path is congested, it will affect all data transfer.
When to try this might be a tricky question.

Thanks,
--
Yoshifumi

---

From: <[email-masked]>
Date: Fri, 24 May 2013 14:05:36 +0300

Hi Yoshifumi,

>
>Thanks for your response. I think there'll be situations where this approach will be
>beneficial.
>I can see that when the destination is inactive, it would be better to send SACK or
>HB ACK to the different path.

Yes, I agree. Though I find it unfortunate that one need to go to the length of ignoring a SHOULD
for something which indeed sounds like a beneficial approach. 
There is also a chicken egg issue here in that the destination will only be stepped to inactive if data (or HBs)
if flowing towards it. That's why I think that effectively one will have to take one of the following approaches:

* Send SACK on return path as long as the destination is not inactive. Bundle HB with SACK if no data is in transit.
* Always send SACK on primary path. Bundle HB with SACK if no data is in transit.

(Some rate limiting shall be implemented on the HB bundling).

>However, I'm not very sure if bundling SACK for CMT is always beneficial. It's
>sometimes better to have path diversity.

Yes. But doesn't this again boil down to choosing a good path for return of SACKs, and with CMT then multiple good return paths for SACK,
when/if multiple paths are available. 

Thanks a lot.
BR, Karen

>> * The return path of an incoming DATA path may not be working. I.e.,
>> there can be connectivity from source address of incoming DATA to destination
>address of incoming DATA, but not the other way around.
>>
>> * The Primary Path of the SACK returner is generally the path which is
>> considered best in the direction that the SACK should take. Would one then not
>think that it generally would be best to return also the SACKs on this path.
>> * from throughput perspectives (including packet processing costs in
>> sender, receiver and intermediate routers) it is beneficiary to bundle the SACK with
>the DATA, opposed to have them be sent as two different packets.
>>
>> One argument that motivates SACK return on the "reflected" path of the
>> DATA is to keep the SRTT stable. But as long as the SACK consistently is
>returned on the Primary Path the fluctuations that could come from Primary Path
>changes should be infrequent in time.
>> The reasoning is that such infrequent fluctuations in SRTT would be an acceptable
>price to pay to have a more robust SACK return.
>>
>> BR, Karen
>>
>>>> RFC4960 in Section 6.4 opens up for the possibility that SACKs may
>>>> be returned on a different path (or IP Path)
>>>>
>>>> acknowledges Data TSNs that have come in on different paths as well
>>>> as it
>>>>
>>>> not sure if the RFC4960 SHOULD here should be read mostly as a MUST
>>>> or mostly as a
>>>>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 26 May 2013 03:32:46 -0600

Karen,

Please see comments below...

[email-masked] wrote:           (Sun, 19 May 2013 17:17:25)
> Agree completely. Apart from the mere throughput considerations, I
> would also like to point to that return of SACK to the Primary DATA
> destination also should be better from a robustness perspective as the
> SACK returner are likely to have fresh and reliable information on the
> "availability" of the Primary DATA destination (availability = data or
> HB's send to the Primary DATA destination are acknowledge to be
> received) While availability of other destination addresses may not be
> fresh (depending on Path monitoring and HB intervals).  Generally in
> simple scenarios where one can trust there to be either
> "bi-directional" connectivity or no connectivity at all in between
> end-point addresses, then the receipt of DATA on a path of course is
> fresh information that the return path works, but this is not
> generally the situation in today's network - at least not on a
> permanent basis. I.e., in failure situations, connectivity may only
> work in one direction until the network has recovered (whether by
> fault recovery or by route migration).
> 
> The return of SACK  to the Primary DATA destination will provide more
> robust delivery of the SACK. If the SACK is not bundled with forward
> DATA, e.g., if no data is in transit in this direction, one could then
> argue for that implementations could bundle HBs with solely returned
> SACKs (though surely one should implement appropriate rate limiting on
> such HBs) thereby making sure to keep the path availability
> information fresh (If the HBs consistently fail, the Primary DATA
> destination should be changed).

Yes and no.

When sending SACKs only to a destination, RTT calculations are not
performed and eventually a HB/ACK exchange must be performed in
accordance with the RFC.  The SACK sender has one other indication
of failure and that is the receipt of duplications and the sending
of dup acks.  Sending of SACKs only is a typically situation when
the data flow is unidirectional (or primarily unidirectional, or
unidirectional over a significant interval).  What I do is keep
track of duplicates for destinations by remembering where the
first SACK was sent for a TSN.  If I receive a duplicate TSN for
a SACK that was sent, I immediately trigger a heartbeat procedure
for the destination (which will result in a HB being sent if one
is not already outstanding or if data is not outstanding to the
destination: i.e. no RTO timer is running for the destination).
This cuts down non-CMT failure detection to an RTO instead of
nominally 1/2 of the heartbeat interval.

I do CMT, so when I receive duplicates that pass the ack-splitting
logic, the destination is lowered in priority with respect to other
destinations due to the dups.  For CMT the triggering of heartbeat
serves to restore the priority of the destination very quickly (in
1 RTT--RTT not RTO) in the event of a SACK loss anomaly.  Otherwise
it would take a nominal 1/2 HB interval to restore the priority of
the destination.

These procedures tested out as providing failure recovery of 1/2
the granularity of the system clock, even under purely unidirectional
data flows!

I would recommend against sending HB with SACK under any kind of
basis, even rate limited, as there is no point in increasing the
traffic to a destination that is under suspicion of failure.  Also,
exponential back-off of HB sending is essential.  Therefore, I
send one, and only if one is not already outstanding and only if
no data is outstanding that would do the job otherwise.

> It could be very interesting to have such C P or B C P be written up,

Do we write BCPs for PSs?

Michael wrote some good RFCs and drafts on related subjects.

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: <[email-masked]>
Date: Tue, 28 May 2013 17:53:52 +0300

Hi Brian,

>Yes and no.
>
>When sending SACKs only to a destination, RTT calculations are not performed and
>eventually a HB/ACK exchange must be performed in accordance with the RFC.
>The SACK sender has one other indication of failure and that is the receipt of
>duplications and the sending of dup acks.  Sending of SACKs only is a typically
>situation when the data flow is unidirectional (or primarily unidirectional, or
>unidirectional over a significant interval).  What I do is keep track of duplicates for
>destinations by remembering where the first SACK was sent for a TSN.  If I receive a
>duplicate TSN for a SACK that was sent, I immediately trigger a heartbeat procedure
>for the destination (which will result in a HB being sent if one is not already
>outstanding or if data is not outstanding to the
>This cuts down non-CMT failure detection to an RTO instead of nominally 1/2 of the
>heartbeat interval.
>

Yes, I think that your "reactive" approach has advantages compared to the "proactive" approach of
having all sole SACKs be bundled with HBs (provided, of course, no data or HB is outstanding 
to the destination).

With the "proactive" approach (HB sent with each sole SACK, HB bundling rate limited to max one HB per RTO, No HB if data or other HB is outstanding)
then one will adjust return path of SACK faster than with your proposed "reactive" approach.
Namely after 1½ RTO in average. On the other hand a loss of one isolated HB bundled with SACK will, with the "proactive" approach,
result in change of return path of SACK even if it was just a single isolated loss that resulted in no problems
from a data transmit perspective. I.e., no duplicate data. Whereas then with your approach, actions are only taken
when arrival of duplicate data hints at there being problems with loss of SACKs.

I would still argue that the immediate HB sent once a duplicate data packet is received should be bundled with the 
SACK that also needs to be returned immediately. This at least for non-CMT SCTP.

I would further stress that one should discourage that the arrival of duplicate data in any way is treated as reliable indication that
SACKs have been lost. Such would be vulnerable to packet duplications in the network as well as it would be very vulnerable to 
malicious nodes that are able to insert duplicates in the network. Thus in non-CMT SCTP I would say that change of SACK
return path should not occur unless loss of connectivity is confirmed by loss of the HB bundled with the immediately returned SACK.
I wonder if you agree with this ?

I suppose that your immediately returned HB also need to be rate limited. It should rather not be so that every duplicate data packet is allowed to
result in transmit of HB (assuming that the HB subsequently is successfully acked). Rate limiting is required also for the reactive approach.

I think that remembering SACK return path on a per TSN basis sounds relatively expensive, but for non-CMT SCTP it should suffice to remember
the only and only Primary Path,  knowledge which already is maintained, and use this path also for SACKs. For CMT-SCTP I  recognize that
more state will have to be maintained.

>I do CMT, so when I receive duplicates that pass the ack-splitting logic, the
>destination is lowered in priority with respect to other destinations due to the dups.
>For CMT the triggering of heartbeat serves to restore the priority of the destination
>very quickly (in
>1 RTT--RTT not RTO) in the event of a SACK loss anomaly.  Otherwise it would take
>a nominal 1/2 HB interval to restore the priority of the destination.
>
>These procedures tested out as providing failure recovery of 1/2 the granularity of
>the system clock, even under purely unidirectional data flows!
>
>I would recommend against sending HB with SACK under any kind of basis, even
>rate limited, as there is no point in increasing the traffic to a destination that is under
>suspicion of failure. 

Actually I think that both approaches does exactly this - namely they send HBs to destinations that are under suspicion of failure.
The difference is that in the "proactive" approach HBs are sent by default, whereas in your approach the HBs are sent only when
there is reason to suspect problems.

I agree with the benefit of eliminating some of the unnecessary HBs at the cost of 
detection problems only a little later on. Further I think that your approach is more robust as
one have two sources of problems: the duplicate data packet PLUS a subsequently lost HB (and SACK).

 Also, exponential back-off of HB sending is essential.
>Therefore, I send one, and only if one is not already outstanding and only if no data
>is outstanding that would do the job otherwise.
>

Sure.

>Do we write BCPs for PSs?

I don't know where the line is drawn exactly.
Is the right approach then to create an RFC4960bis or an separate work item that 
clarifies this part of RFC4960 ?

Part of what we are discussing here are additional details that may not need to be specified (I don't know) ?
But it seems that we agree that the SHOULD in section 6.4 of RFC4960 in regard to return of SACKs is something which one 
should generally not feel constrained by.
Do you agree ?

Thanks a lot !

BR, Karen

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 30 May 2013 04:20:38 -0600

Karen.Nielsen,

[email-masked] wrote:                     (Tue, 28 May 2013 17:53:52)
> 
> With the "proactive" approach (HB sent with each sole SACK, HB
> bundling rate limited to max one HB per RTO, No HB if data or other HB
> is outstanding) then one will adjust return path of SACK faster than
> with your proposed "reactive" approach.  Namely after 1½ RTO in
> average. On the other hand a loss of one isolated HB bundled with SACK
> will, with the "proactive" approach, result in change of return path
> of SACK even if it was just a single isolated loss that resulted in no
> problems from a data transmit perspective. I.e., no duplicate data.
> Whereas then with your approach, actions are only taken when arrival
> of duplicate data hints at there being problems with loss of SACKs.

No--for a number of reasons:

1. Senders are not required to bundle
2. SACKs that are not dup-acks can be (and normally are) delayed up to 200ms.
3. HB interval is required to be settable in ms and is often set to
   less than 200ms on high performance networks (i.e. SIGTRAN).
4. SACK rates depend on DATA flow in the reverse direction.  If data in
   the reverse direction stops, SACKs stop.
5. HB are sent according to a HB interval which is statistically varied
   to avoid having many hosts sending HB's synchronously and simultaneously.
   If HB are bundled with SACK and data is flowing continuously in the
   reverse direction (common situation), HB will be sent ever 200ms if
   data flow is slow, and every two packet emission times when data is
   fast.  In either case, HBs can be sent synchronously, defeating the
   entire statistical interval varying feature of the HB mechanism.

You cannot support your 1-1/2 RTO figure.  In fact the nominal recovery
due to that approach alone can be >100ms, making it instead an "inactive"
approach. ;)

> I would still argue that the immediate HB sent once a duplicate data
> packet is received should be bundled with the SACK that also needs to
> be returned immediately. This at least for non-CMT SCTP.

Do not bundle them together.  The HB is going to the destination that is
suspect of losing SACKs.  Why send a SACK there?

The SACK needs to go to the destination to which you are going to send
SACKs and any pending DATA when it comes time to send (and there is
still the incoming queue to process).

> I would further stress that one should discourage that the arrival of
> duplicate data in any way is treated as reliable indication that SACKs
> have been lost. Such would be vulnerable to packet duplications in the
> network as well as it would be very vulnerable to malicious nodes that
> are able to insert duplicates in the network.

Any MIM attacker that can duplicate packets can do way worse damage with
far less effort than merely attempting to make the sender think his
SACKs might not be getting through.  Also, true network packet duplication
can be curtailed at the network layer without even involving SCTP.
Further on a CMT implementations, the duplicates must pass the multiple
queue ack-splitting logic before being considered non-spurious and an
attacker would be very hard pressed to time it perfectly.

Nevertheless, the HB I send acts to "restore" the destination in one
RTT.  In my CMT algorithm, dups are weighted quite low as they are a
weak signal.

> Thus in non-CMT SCTP I would say that change of SACK return path
> should not occur unless loss of connectivity is confirmed by loss of
> the HB bundled with the immediately returned SACK.  I wonder if you
> agree with this ?

No.  Operational experience shows that, for non-CMT, SACKs should be
sent on the primary path.  When the primary path changes, the
destination of the SACKs changes.  Using loss of a HB bundled with a
SACK (which you shouldn't do anyway), is an inadequate reason to
change the primary path, even for unidirectional traffic flows, and
could easily lead to thrashing.

The same is true for CMT.  Receipt of duplicates is a poor indicator
of path availability and can really only support a weak suspicion,
worthy of no more than a Red October ping ("one ping only").  In CMT
however, it can serve to alter the order of paths selected for
transmission when all other factors are equal between the paths and can
support failover times on the order of the granularity of the system
clock, particularly in the face of single path failures under
unidirectional traffic flows, where duplicates are the ONLY indication
of path failure other than the normal heartbeat mechanism (which, even
at 25ms is too slow).

The factor that makes it an even weaker indicator is CMT is the fact
that SACKs are cumulative.  Therefore, particularly when sending SACKs
on multiple paths, it is difficult to determine with any certainty which
destination lost the SACK (or SACKs) that caused the duplication.  There
are some ways I get around that, but there are far too complicated to
discuss adequately in this note.

> I suppose that your immediately returned HB also need to be rate
> limited. It should rather not be so that every duplicate data packet
> is allowed to result in transmit of HB (assuming that the HB
> subsequently is successfully acked). Rate limiting is required also
> for the reactive approach.

It is effectively rate limited.  I only send them when I receive a
duplicate, no delayed SACK is pending (handles RTO < SACKdelay), no
data in flight to the destination.  I am required to send a SACK
package even when the last two conditions do not hold, so the rate
cannot be of concern.

> I think that remembering SACK return path on a per TSN basis sounds
> relatively expensive, but for non-CMT SCTP it should suffice to
> remember the only and only Primary Path,  knowledge which already is
> maintained, and use this path also for SACKs. For CMT-SCTP I
> recognize that more state will have to be maintained.

Actually no.  The destination I send the heartbeat to is the current
preferred CMT destination or the current primary non-CMT destination
(the one to which DATA and SACKs are sent) for less than full saturation
traffic flows, and particularly unidirectional traffic flows, this
is the likely destination to which the SACK was sent).  The procedure
is intended on handling unidirectional traffic flows where there is no
other (fast) indicator of destination failure.

BTW, I send the HB immediately, and do not bundle the SACK.  This is
because the SACK should be bundled with DATA in any event, and as a
result of the duplicate pegging, is likely to be sent to a different
destination as the HB: remember?--the destination to which the HB is
going is already suspected of not passing SACKs, why would one send it
to the suspect destination?

> Part of what we are discussing here are additional details that may
> not need to be specified (I don't know) ?  But it seems that we agree
> that the SHOULD in section 6.4 of RFC4960 in regard to return of SACKs
> is something which one should generally not feel constrained by.
> Do you agree ?

I think that operational experience shows that it should be changed
to say that SACKs should be bundled with DATA (when bundling is
supported) and sent to the primary destination.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/