Subject: Re: [tsvwg] QUESTION on Return of SACKs and HB ACKs

From: <[email-masked]>
Date: Fri, 31 May 2013 13:54:27 +0300

Hi Brian,

>-----Original Message-----
>From: Brian F. G. Bidulock [mailto:[email-masked]]
>Sent: Thursday, May 30, 2013 12:21 PM
>To: Nielsen Karen
>Cc: [email-masked]; [email-masked]; [email-masked]
>Subject: Re: [tsvwg] QUESTION on Return of SACKs and HB ACKs
>
>Karen.Nielsen,
>
>Please see comments below...
>
>[email-masked] wrote:                     (Tue, 28 May 2013 17:53:52)
>>
>> With the "proactive" approach (HB sent with each sole SACK, HB
>> bundling rate limited to max one HB per RTO, No HB if data or other HB
>> is outstanding) then one will adjust return path of SACK faster than
>> with your proposed "reactive" approach.  Namely after 1½ RTO in
>> average. On the other hand a loss of one isolated HB bundled with SACK
>> will, with the "proactive" approach, result in change of return path
>> of SACK even if it was just a single isolated loss that resulted in no
>> problems from a data transmit perspective. I.e., no duplicate data.
>> Whereas then with your approach, actions are only taken when arrival
>> of duplicate data hints at there being problems with loss of SACKs.
>
>No--for a number of reasons:
>
>1. Senders are not required to bundle
[Karen] Not sure what bundling that you refer to here.
>2. SACKs that are not dup-acks can be (and normally are) delayed up to 200ms.

[Karen] Yes. I am sorry for the sloppiness and the resulting waste of time.
 In the SIGTRAN networks setting that I am aware of, however, a set of RTO_MIN below delay_ack (or conversely a set of delay_ack larger than RTO_MIN) is considered a mis-configuration. Now one may argue that here I am comparing apples and pears as, in theory at least, one could have different setting of delay_ack and RTO_MIN on the two sides and then have RTO_MIN on one side comply with delay_ack setting on other side while the delay_ack and the RTO_MIN on the same side do not match. Such situations perhaps needs to be considered for full generality.

The 1½RTO referenced was intended to indicate the average time it takes to detect that the path used for sending SACKs is not working (or at least there was a loss) when no data is flowing. It is right that this evaluation is sloppy as it did not take into consideration delay-ack nor did it take data tails (data stop) into consideration.
If we take data tail into consideration and delay ack, I suppose that we are speaking, worst case, something like MAX(RTO_data_sender, delay_ack_SACK_sender, RTO_SACK_sender)+RTO_SACK-sender before we discover (by HB timeout of HB bundled with SACK) that SACK Path is not working.
In the SIGTRAN network settings that I know of this would be below 2*RTO worst case for data tail and ~1½RTO in average otherwise. 

The norm in the SIGTRAN networks that I am familiar with is not 200ms for delay_ack. It is lower and it is definitively kept below RTO_MIN.
Further RTO_MIN/RTO_MAX << HB interval. 

When data is flowing, I do not think we need to discuss anything. We have already agreed that SACK should go
on data path and then data will serve to detect if the data path (and thus the path used for SACK) is working.

>3. HB interval is required to be settable in ms and is often set to
>   less than 200ms on high performance networks (i.e. SIGTRAN).

[Karen] I wasn't aware that such aggressive HB intervals really was used. The SIGTRAN networks that I am familiar with 
do not use such aggressive settings for the HB interval. Here typically RTO << HB interval. I concur that this changes the picture quite a lot.
I have not considered the situation where HB interval and RTO are comparable.

>4. SACK rates depend on DATA flow in the reverse direction.  If data in
>   the reverse direction stops, SACKs stop.
[Karen] Yes. 
>5. HB are sent according to a HB interval which is statistically varied
>   to avoid having many hosts sending HB's synchronously and simultaneously.

[Karen] Are you referring to RTO jitter or to something else entirely ?

>   If HB are bundled with SACK and data is flowing continuously in the
>   reverse direction (common situation), HB will be sent ever 200ms if
>   data flow is slow, and every two packet emission times when data is
>   fast.  

[Karen] Depending on delay_ack setting. The default values that I hear of are
in the area of 50-100 msecs for delay_ack.  I also know that some use 200msecs.

If delay_ack is 200msecs, RTO will be above 200 msecs. Rate-limiting HBs to RTO would
mean that not every SACK would carry a HB. 

In either case, HBs can be sent synchronously, defeating the
>   entire statistical interval varying feature of the HB mechanism.
>

[Karen] I agree that I have not considered such aggressive HB intervals at all.
I am not sure why the varying of HBs bundled with a SACK (and driven by a SACK) is of concern.

>You cannot support your 1-1/2 RTO figure.  In fact the nominal recovery due to that
>approach alone can be >100ms, making it instead an "inactive"
>approach. ;)

[Karen] 1½ RTO, not ½-1 RTO, but that's beyond the point of this discussion.
Yes I would expect detection to happen after 100msec. You expect to have retransmissions of data arrive faster 
than after 100 msecs. Perhaps as a result of fast retransmission, but then some SACKs are going back. 

I 

>
>> I would still argue that the immediate HB sent once a duplicate data
>> packet is received should be bundled with the SACK that also needs to
>> be returned immediately. This at least for non-CMT SCTP.
>
>Do not bundle them together.  The HB is going to the destination that is suspect of
>losing SACKs.  Why send a SACK there?
>

[Karen]

We send SACK on assumed transfer data path (primary path).

With your approach, I thought not to change neither SACK path nor data transfer path (primary path),
as a result of the arrival of a duplicate data packet. But to only change both when 
a HB had confirmed that the SACK return path (aka the primary path) had issues.

I understand now that you take a more "active", ;-), approach and change 
SACK path (an data transfer path) immediately based on the hint from the duplicate data packet.

I was concerned that this would be too aggressive (and vulnerable to DoS),
but you may have result that prove otherwise and the benefits may be so
great that they outweigh potential DoS considerations.

>The SACK needs to go to the destination to which you are going to send SACKs
>and any pending DATA when it comes time to send (and there is still the incoming
>queue to process).
>
>> I would further stress that one should discourage that the arrival of
>> duplicate data in any way is treated as reliable indication that SACKs
>> have been lost. Such would be vulnerable to packet duplications in the
>> network as well as it would be very vulnerable to malicious nodes that
>> are able to insert duplicates in the network.
>
>Any MIM attacker that can duplicate packets can do way worse damage with far
>less effort than merely attempting to make the sender think his SACKs might not be
>getting through.  Also, true network packet duplication can be curtailed at the
>network layer without even involving SCTP.
>Further on a CMT implementations, the duplicates must pass the multiple queue
>ack-splitting logic before being considered non-spurious and an attacker would be
>very hard pressed to time it perfectly.
>
>Nevertheless, the HB I send acts to "restore" the destination in one RTT.  In my CMT
>algorithm, dups are weighted quite low as they are a weak signal.
>

[Karen] Is there no reason to be concerned with the RTT fluctuations that temporary change of SACK path 
could cause in other end ? 

>> Thus in non-CMT SCTP I would say that change of SACK return path
>> should not occur unless loss of connectivity is confirmed by loss of
>> the HB bundled with the immediately returned SACK.  I wonder if you
>> agree with this ?
>
>No.  Operational experience shows that, for non-CMT, SACKs should be sent on the
>primary path.  When the primary path changes, the destination of the SACKs
>changes.  Using loss of a HB bundled with a SACK (which you shouldn't do
>anyway), is an inadequate reason to change the primary path, even for unidirectional
>traffic flows, and could easily lead to thrashing.
>
[Karen] I concur, and did concur, that it is better to use the HBs only for confirmation of 
problems that are hinted at from the arrival of duplicate data.

What happens once the duplicate data arrives is the discussion. 

>The same is true for CMT.  Receipt of duplicates is a poor indicator of path
>availability and can really only support a weak suspicion, worthy of no more than a
>Red October ping ("one ping only").  In CMT however, it can serve to alter the order
>of paths selected for transmission when all other factors are equal between the
>paths and can support failover times on the order of the granularity of the system
>clock, particularly in the face of single path failures under unidirectional traffic
>flows, where duplicates are the ONLY indication of path failure other than the normal
>heartbeat mechanism (which, even at 25ms is too slow).

[Karen] You have an timeout on the HBs that is below 25msecs ?

>
>The factor that makes it an even weaker indicator is CMT is the fact that SACKs are
>cumulative.  Therefore, particularly when sending SACKs on multiple paths, it is
>difficult to determine with any certainty which destination lost the SACK (or SACKs)
>that caused the duplication.  There are some ways I get around that, but there are far
>too complicated to discuss adequately in this note.

[Karen] yes I can see this getting very tricky with CMT. 

>
>> I suppose that your immediately returned HB also need to be rate
>> limited. It should rather not be so that every duplicate data packet
>> is allowed to result in transmit of HB (assuming that the HB
>> subsequently is successfully acked). Rate limiting is required also
>> for the reactive approach.
>
>It is effectively rate limited.  I only send them when I receive a duplicate, no delayed
>SACK is pending (handles RTO < SACKdelay), no data in flight to the destination.  I
>am required to send a SACK package even when the last two conditions do not
>hold, so the rate cannot be of concern.
>
>> I think that remembering SACK return path on a per TSN basis sounds
>> relatively expensive, but for non-CMT SCTP it should suffice to
>> remember the only and only Primary Path,  knowledge which already is
>> maintained, and use this path also for SACKs. For CMT-SCTP I recognize
>> that more state will have to be maintained.
>
>Actually no.  The destination I send the heartbeat to is the current preferred CMT
>destination 

[Karen]
I thought perhaps that you had multiple such preferred CMT destinations to choose among.

or the current primary non-CMT destination (the one to which DATA and
>SACKs are sent) for less than full saturation traffic flows, and particularly
>unidirectional traffic flows, this is the likely destination to which the SACK was sent).
>The procedure is intended on handling unidirectional traffic flows where there is no
>other (fast) indicator of destination failure.
>
>BTW, I send the HB immediately, and do not bundle the SACK.  This is because the
>SACK should be bundled with DATA in any event, and as a result of the duplicate
>pegging, is likely to be sent to a different destination as the HB: remember?--the
>destination to which the HB is going is already suspected of not passing SACKs,
>why would one send it to the suspect destination?
>

[Karen] I am not sure that it is wise to have hints from the arrival of a duplicate packet change
the data transfer path. If there is DATA going out with the SACK, DATA will much better serve to
tell about whether the destination works or not. DATA, given Fast Retransmission,
 is not as prone to single losses as SACK or HBs are. 
If there is no data going out (in the direction of the SACK) then one is left with working with the information
gained from the HB and the duplicate packets that arrive. 

Whether one react on the duplicate packet alone or whether one awaits potential back-up confirmation
of failure from the HB is what divided us above. Agree ?

I am still not 100% sure what is right (or best), but then we have not done the experiments.

>> Part of what we are discussing here are additional details that may
>> not need to be specified (I don't know) ?  But it seems that we agree
>> that the SHOULD in section 6.4 of RFC4960 in regard to return of SACKs
>> is something which one should generally not feel constrained by.
>> Do you agree ?
>
>I think that operational experience shows that it should be changed to say that
>SACKs should be bundled with DATA (when bundling is
>supported) and sent to the primary destination.
>

Great - Thanks !

BR, Karen
>--brian
>--
>Brian F. G. Bidulock
>[email-masked]
>http://www.openss7.org/