Subject: Re: [tcpm] Tech review of draft-grimes-tcpm-tcpsce-01.txt

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Mar 2020 00:03:23 +0000

Jonathan,

Reminder: In Nov'19,  I carefully unpicked the descriptions of all the 
ideas in your draft, asking questions on the list where necessary, and 
provided a detailed review of whether it does what it says it aims to do 
(setting aside whether I agree with what it aims to do).
https://mailarchive.ietf.org/arch/msg/tcpm/Mbdoh-GZCO_fc6e-84ORCKk75-w/
The draft hasn't been updated since.

The draft contains four tentative ideas for different semantics for the 
proposed feedback. There isn't any mechanism for negotiating which 
scheme to use, so I assume the intent is to eventually choose one.

The draft said more assessment was needed of each. It rejected one idea 
itself. I analyzed the other three, and concluded that they all had 
major problems. I spent some time structuring my review to make it easy 
to follow and included an exec summary and table assessing each idea 
against the goals given in the draft.

You were going to "examine my arguments in detail".

Cheers

Bob

On 16/11/2019 00:59, Bob Briscoe wrote:
> Jonathan,
>
> I did ask you not to reply until you've read the review. The reasoning 
> against what you've said below is in the review.
>
> What you say (which you wouldn't say if you'd read the review) is only 
> true for long-term stable traffic. But the Internet is hardly ever 
> like that.
>
> We need to see the delay from SCE at higher percentiles, which 
> determines whether interactive applications will experience an 
> improvement. If you don't make a significant improvement in 
> /consistency/ of delay, there's no point burning codepoints. To 
> measure that requires hundred's of thousands of data samples. Not just 
> a few time series plots and plots of averaged Q delay. We have 
> provided tools to extract all this, and we shall be making them easier 
> to use during the Hackathon.
>
> Briefly repeating what I've already said (sigh):
> To ensure delay is /consistently/ very low under dynamic conditions 
> (e.g. for real-time interactive applications), accuracy and timeliness 
> in the feedback loop is critical.
>
> With inaccurate feedback the set point that the congestion control is 
> aiming for is incorrect, so the rate it aims for is either too high 
> (->qdelay) or too low (->under-utilization). Yes, given enough time 
> (under stable conditions), as it heads towards the wrong point, 
> congestion will rise higher (or fall lower) than the error, and it 
> will head closer to the correct point, but still aiming for the wrong 
> target (the right target but with the latest feedback errors (+/-) 
> added).
>
> In addition, the review explains why all the schemes in the tcpsce 
> draft give an inherently more biased error the higher the level of 
> congestion. Biased error is much more problematic.
>
>
>
> Bob
>
> PS. Pls do not reply again without having taken the trouble to read 
> what I have spent a lot of time explaining.
>
>
> On 15/11/2019 23:28, Jonathan Morton wrote:
>>> On 16 Nov, 2019, at 1:59 am, Bob Briscoe <[email-masked]> wrote:
>>>
>>> Why go through all the years of pain of altering the TCP wire 
>>> protocol for an accurate feedback scheme that isn't accurate?
>> Without delving into the rest of the detail right now, I'd like to 
>> address this point immediately.
>>
>> Congestion control is, ultimately, a system with a control loop under 
>> negative feedback.  When designing such a control loop, one is 
>> generally concerned with whether it has sufficient gain to converge 
>> on the appropriate setpoint, and is acceptably stable.  The nature of 
>> the feedback path feeds into that analysis, but as long as the result 
>> meets acceptability criteria, a simpler control system is often 
>> preferable.  One fundamental property of a negative-feedback system, 
>> though, is that the system returns towards the setpoint after a 
>> disturbance, even that disturbance occurred *in* the feedback path.
>>
>> Consequently, SCE can tolerate as much as 100% relative error in the 
>> SCE feedback path.  The case of being 100% low - no ESCE feedback 
>> received at all - corresponds to falling back to RFC-3168 compliant 
>> behaviour, ie. the MD response to CE acts as a ceiling on the 
>> positive excursion of the control output (cwnd).  The case of being 
>> 100% high - twice as much ESCE feedback received as SCE marks were 
>> applied on the forward path - will cause a low excursion which will 
>> drain the affected queue, after which SCE marking will stop, and the 
>> system will then recover.
>>
>> Consequently we made the design decision that *reliable* accurate 
>> feedback was not necessary, nor was the associated complexity 
>> desirable.  We also observed that a single additional bit per 
>> received segment was always adequate to convey *accurate* feedback, 
>> and that usually fewer bits were needed, which fits well with using 
>> one spare bit per Delayed ACK packet - which is what we do here.  
>> Several such bits are available in the TCP header for such uses, and 
>> we decided to recommend the use of one which had recently been freed 
>> up and whose previous use was closely related.
>>
>> Without yet having examined your other arguments in detail, I suspect 
>> you are misunderstanding how some of our proposed feedback methods 
>> work in practice, perhaps stemming from the above fallacy.  
>> Understand that we have actually implemented each of these methods in 
>> running code, and can demonstrate their operation at the Hackathon.
>>
>> See you there.
>>
>>   - Jonathan Morton
>>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 13 Mar 2020 03:38:19 +0200

> On 13 Mar, 2020, at 2:03 am, Bob Briscoe <[email-masked]> wrote:
> 
> Reminder: In Nov'19,  I carefully unpicked the descriptions of all the ideas in your draft, asking questions on the list where necessary, and provided a detailed review of whether it does what it says it aims to do (setting aside whether I agree with what it aims to do).
> The draft said more assessment was needed of each. It rejected one idea itself. I analyzed the other three, and concluded that they all had major problems. I spent some time structuring my review to make it easy to follow and included an exec summary and table assessing each idea against the goals given in the draft.
> 

Which we did, and concluded there wasn't a way (at that time) of answering your points in detail without sparking a massive and counterproductive flamewar.  So I'm not going to do that, but I have managed to extract a few salient points which I *can* address.

> The draft contains four tentative ideas for different semantics for the proposed feedback. There isn't any mechanism for negotiating which scheme to use, so I assume the intent is to eventually choose one.

Your assumption would be incorrect.

SCE is designed to accommodate flexibility in several areas, including the precise algorithms used to control the send rate, marking schedule, and feedback.  This draft, of course, covers the feedback.  Any algorithm which succeeds in reflecting the received SCE marks as ESCE flags with reasonable accuracy is acceptable.  The draft describes several such algorithms and tries to characterise their relative advantages, including the extent to which they maintain a desired Delayed Ack ratio.

The scheme actually implemented in our Linux-based reference code is the fourth algorithm, labelled Advanced.  This precisely matches segments' marks with their acks' flags, while still permitting Delayed Acks to function at both low and high marking ratios.  Really, this is the ideal.  The other schemes were proposed as being potentially easier to implement in some TCP stacks, with lesser performance.  (We might delete the Dithered scheme from a future version.)

Ack thinning, as you describe it, is something that happens on the reverse path, after the ack stream has left the receiver.  SCE's feedback scheme in TCP is unreliable, so naive ack thinning (that is, discarding acks without considering the potential loss of information in flags and/or options) can indeed lead to a temporary bias in the feedback signal as seen at the sender.  Acks can also be lost for other reasons, such as AQM action in the event of congestion, and this would have a similar effect in principle.

The important point is that SCE can tolerate this because the control loop remains closed, even if the feedback signal becomes systematically biased.  The AQM at the bottleneck is part of that control loop; it will adjust the marking rate as much as is needed to control the queue.  So even if the signal is so heavily biased by thinning that it appears to saturate at 100% or 0%, the system will be driven to a state where that bias is reduced, or even becomes impossible because it has reached 0% or 100% respectively.

This is part of what we mean when we say that SCE can tolerate a 100% relative error in the feedback.  The other part is that even if ESCE feedback fails entirely, we have a built-in failsafe in the form of CE marks and the reliable ECE/CWR feedback mechanism, as defined in RFC-3168.  This would cause a Multiplicative Decrease, which might technically be overkill for the actual congestion situation, but is conservative and safe.

So the result is a system of minimal complexity that can be implemented easily, requires minimal resources (one spare header bit) on the wire, and achieves closed-loop control even under less than ideal network conditions.

This is engineering, not an ivory tower.

As a final point, there is in fact a way to perform Ack Thinning which preserves nearly all of the information in the lost acks, by being selective about which acks are actually dropped.  This does require more careful inspection of consecutive acks, to see if the information in the first is adequately reflected in the information present in the second.  An example implementation of this is found in net/sched/sch_cake.c in our reference codebase, and completely avoids the bias effects that you postulate.

 - Jonathan Morton