Subject: Re: [tcpm] question regarding draft-gomez-tcpm-delack-suppr-reqs

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Fri, 1 May 2020 10:56:31 +0200

Dear Yuchung,

Thanks for your questions and comments.

Please find below my inline responses:

> thanks for the talk!
>
> https://tools.ietf.org/html/draft-gomez-tcpm-delack-suppr-reqs-01#section-3.1
> "Note that, while Appropriate Byte Counting (ABC) [RFC3465] might be used
>    to address this problem, it remains as an experimental mechanism, not
>    fully included in RFC 5681, which specifies standard TCP congestion
>    control.  "
>
> It's not clear what "fully included" means here. AFAICT, RFC5681
> includes the essence of ABC explicitly.
>
> https://tools.ietf.org/html/rfc5681#section-3.1
>
> " During slow start, a TCP increments cwnd by at most SMSS bytes for
>    each ACK received that cumulatively acknowledges new data.  Slow
>    start ends when cwnd exceeds ssthresh (or, optionally, when it
>    reaches it, as noted above) or when congestion is observed.  While
>    traditionally TCP implementations have increased cwnd by precisely
>    SMSS bytes upon receipt of an ACK covering new data, we RECOMMEND
>    that TCP implementations increase cwnd, per:
>
>       cwnd += min (N, SMSS)                      (2)
>
>    where N is the number of previously unacknowledged bytes acknowledged
>    in the incoming ACK.  This adjustment is part of Appropriate Byte
>    Counting [RFC3465] and provides robustness against misbehaving
>    receivers that may attempt to induce a sender to artificially inflate
>    cwnd using a mechanism known as "ACK Division" [SCWA99].  ACK
>    Division consists of a receiver sending multiple ACKs for a single
>    TCP data segment, each acknowledging only a portion of its data.  A
>    TCP that increments cwnd by SMSS for each such ACK will
>    inappropriately inflate the amount of data injected into the network."

Note that the problem pointed out is that, as stated in the RFC 5681 text
above:

   "During slow start, a TCP increments cwnd by at most SMSS bytes for
    each ACK received that cumulatively acknowledges new data"

It may be desirable to support, during slow start, a cwnd increase by more
than SMSS per each ACK acknowledging new data.

In fact, couple of paragraphs later, RFC 5681 explicitly indicates the
following:

   "We note that [RFC3465] allows for cwnd increases of more than SMSS
   bytes for incoming acknowledgments during slow start on an
   experimental basis; however, such behavior is not allowed as part of
   the standard."

That is what we meant by "not fully included in RFC 5681" in our document.

> As a side note, Linux stack has implemented the principle of abc in
> its congestion control (including slow start)

Thanks for the information.

Yes, in fact, I second the WG members who stated on the chat, during the
interim, that it is surprising that ABC remains as an experimental
document.

I realize that, at an implementation level, the problem pointed out above
is being addressed in many platforms.

> regarding delayed ACK harming slow-start or latency, I am not too sure
> either:
>
> 1) The ACK is delayed but not the data
>
> 2) The delay of the ACK can indeed delay cwnd growth during slowstart.
>
> But the ACK is delayed because the packet is likely a sub-MSS data,
> which is again likely caused by application limit (i.e. the
> application has sent all its data like a HTTP response). In this case
> the delay of the cwnd growth does not hurt the performance since the
> application has no data left to send.
>
>    The only penalty occurs when the application has more data to send
> but cwnd is still gated by the arrival of the delayed ACK. However, in
> many request - response protocol, either the (non-delayed) ACKs
> triggered by prior full-MSS data inflight or the delayed ACKs would
> have returned to grow the cwnd to sending immediately.
>
> I can construct scenarios where delayed ACK significantly hurts
> application. But I don't think the scenario is too common. I will
> suggest more tests to quantify the effect of delayed ACK effect on
> slow-start.

Yes, I tend to agree.

In fact, the two main problems due to Delayed ACKs that we identified so
far in the area of slow start are:

- The fact that ABC might not always be supported (due to its
"Experimental" status).

- Precluding sender behaviors such as chirping.

Once again, thanks for your valuable feedback!

Cheers,

Carles

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 1 May 2020 17:46:35 +0100

Gorry,

On 30/04/2020 09:36, Gorry Fairhurst wrote:
> See some more explanation in line.
>
> On 29/04/2020 22:58, Bob Briscoe wrote:
>> Gorry,
>>
>> On 29/04/2020 18:54, Gorry Fairhurst wrote:
>>
>>> - DAASS is a simple fix, that appears to help a lot when there is 
>>> data waiting for the arrival of the delayed ACK. 
>>
>> Any heuristic at the receiver to detect the end of a pattern of 
>> start-up such as slow-start, ossifies that pattern into the Internet. 
>> Sender control of the DelACK ratio removes that ossification.
>>
> I initially thought the same, DAASS tried to fix multiple problems. 
> However, as regards DAASS  reducing the waiti for the ACK to a rounds 
> of initial cwnd growth, I think the key understanding is the sender 
> use of byte accounting (for stretch-ACKs) not individual ACKs, (that 
> might imply a need for some burst-mitigation for stretch ACKs, because 
> you reduce ACK clocking).
>
> I like what I understand Chrome's QUIC does:  In the first 100 
> received packets, send an ACK for each packet (could be each pair of 
> packets).  This rapidly grows a cwnd of about 140 packets (assuming 
> exponetial growth). Once the cwnd is >100 then an ACK covering 10 
> packets/segments becomes a small proportion of the cwnd (the 
> additional delay from per-packet ACKs is small). The receiver can 
> moves to a larger default ACK ratio.

[BB] If it's likely that we need to move beyond slow-start behaviour in 
future, building heuristics around slow-start is the worst possible step 
we could take. That's the ossification I'm talking about.

When we implemented paced chirping, it was really hard not to confuse 
the heuristics in Linux receivers. Linux receivers are built to detect 
the end of today's slow-start behaviour. Our sender wasn't using 
anything like slow start behaviour, which triggered their heuristics 
much too early.

Magically increasing the delack ratio after 100 packets is another 
behaviour we must not encourage.

Sender control of the DelAck ratio can be really simple. Then we have a 
simple solution that will last for ever.
We must not dig the heuristics and magic numbers hole any deeper.

>
>>
>>> ABC is important also (well what I think people understand by ABC - 
>>> rather than the RFC on this).
>>
>> Yes.
>>
>>> * High bit rate environments and short data segments
>>>
>>> If you really don’t want delayed ACKs, the protocol using TCP ought 
>>> to disable nagle. Or you would perhaps use DAASS... I’m not sure 
>>> which problem you speak about?
>>>
>>>
>>> * Beyond classic ACK transmission behavior
>>> I do agree that Delayed ACKs preclude using sender behaviors 
>>> intended to quickly and non-intrusively probe for available capacity 
>>> during slow start. This is important to me, but I am very wary about 
>>> the idea of sending more ACKs to do this, and we need to be careful 
>>> to gain sufficient information about what the receiver has seen - 
>>> not just to get a bunch of separate (each cumulative) ACKs. This 
>>> really doesn’t give much fidelity to the sender. I’d be happy to 
>>> talk more about what might be better!
>>
>> Indeed, ACK frames in QUIC that give the arrival times of data 
>> packets are an alternative - as long as they are not stretched beyond 
>> the time at which they would have been useful.
>>
> Interesting, I think.
>> But we are more limited for space in TCP. As long as a TCP server is 
>> serving bulk and short flows, it can put more effort into receiving 
>> more ACKs during flow startup, and save its processing power by using 
>> less frequent ACKs for the majority of its data - transmitted in 
>> congestion avoidance.
>>
>>
> Also seems true.
>>> * IoT scenarios
>>> I can see the argument of an ACK energy cost for the IoT device. At 
>>> least, I would assume IoT devices can be tuned, and the apps they 
>>> talk to can be tuned. Appropriate guidance helps!
>>
>> Tuning is for a scenario, not for widely differing scenarios. The 
>> message of this draft is that a connection sometimes needs to be able 
>> to adapt its ACK ratio at run-time. Not every IoT device is always 
>> deployed for just one scenario. Neither is its peer. To me this says 
>> adaptation mustn't be complex (and doesn't have to be).
>>
> Heterogeniety of the network segments is tricky. Some optimisations 
> could be handled by link-specific methods - header compression perhaps 
> is an example, other things need to be adapted end-to-end. However, 
> end-to-end transport has to work for all types of path (links) - and 
> often does not know the path.

[BB] Sender control of the receiver's DelAck ratio would be always 
better and never worse in these cases. It might not solve the lack of 
path knowledge. But wherever path knowledge can be gathered, sender 
control of the receiver's DelAck ratio would then be useful/necessary.

Then, any sender can implement "link-specific methods" unilaterally (or 
not if it chooses). Sndr-rcvr co-ordination is an important piece that's 
missing.

You seem to be generating confusion. There is still good reason to add a 
generally useful missing capability (sender control of receiver DelACK 
ratio), whether or not other capabilities might sometimes be missing in 
certain odd scenarios.

Do one thing and do it well.

>>>
>>> * Bursty Apps
>>> I think you can add varying workloads as something important. By 
>>> which I mean apps that do transactional stuff, or are controlled by 
>>> applications where the data transmission need varies.When we looked 
>>> at CWV and the ideas that the applications control the traffic 
>>> patterns in one group of applications, we also noted that these 
>>> applications change the way they use the network. That’s important 
>>> for a timely restart to growing the cwnd (etc). Such applications 
>>> can also be very sensitive to network delay.  These are probably 
>>> also of interest!
>>
>> Yes.
>>
>>>
>>> So....  the reason for hesitancy overall is I see this as tricky 
>>> space to get correct.
>>
>> But trickiness is the point of starting this requirements document, 
>> isn't it?
>> Please can you articulate better why you seem negative about this 
>> exercise?
>>>
>>> There are many places where fewer ACKs are good - if you have per 
>>> ACK interrupt costs - if the capacity consumed by ACKs is important 
>>> - the cost of sending in the link technology is high - etc. This is 
>>> complicated for TCP (and much less so for QUIC - because QUIC has a 
>>> notion of pacing; QUIC’s loss recovery is different; and QUIC’s ACKs 
>>> are not easily thinned, at least at present). Any TCP method has to 
>>> live with networks that experience pain from more ACKs, but also may 
>>> deploy (various) mitigations.
>>
>> Yes. But reading between the lines I hear you say "Don't start this 
>> exercise".
>> Pls explicitly articulate your implicit message.
>>
> The slide asked about adopting this work - my words were against doing 
> that at that time with that revision of this document.
>
> I miss seeing presentations with results and showing the thinking. I  
> wonder if we need a specific agenda slot to focus on such topics. I 
> expect we'll learn a lot from some data from a variety of 
> problem-spaces/use-cases, especially if the timeslot is sufficient to 
> include thinking and experience from people outside of TCPM. (That's 
> another thought - and probably should be another thread).

[BB] I certainly agree with the need to base this on experience.
However, all we have at the moment is experience of protocols /without/ 
sender control of the DelAck behaviour of the receiver. That's useful 
experience, but only to demonstrate the /problem/.

It would be good to hear some empirical experience of protocols /with/ 
sender control.

>
>>>
>>> There are also cases where you’d like more information than a 
>>> delayed ACK provides (e.g. chirping and similar probing methods) and 
>>> there are application pathologies that would love an ACK at the end 
>>> of a packet burst (be that 1 or 100s of packets). Any method has to 
>>> live with a wide variety of paths and applications.
>>>
>>> Happy to discuss more. And I suspect other people may also respond 
>>> on list. We have lots of data for QUIC and TCP - but most of our 
>>> data focusses on testbeds or on actual broadband satellite services.
>>

[BB] As above. Data about the existing Internet without the proposed 
capability is useful to characterize the problem. But need to be aware 
of its limitations at characterizing the solution-space proposed here.

Bob

>> Thanks for elaborating.
>> Bob
> Gorry
>>> Gorry

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 1 May 2020 09:48:57 -0700

> than SMSS per each ACK acknowledging new data.
I agree. We in fact implemented Linux to increase cwnd by the exact
amount sequence (in unit of packets) so slow-start increase function is
regardless of ack-thinning effect but purely based on amount
delivered. This has an important effect on
Wireless as the other ack talk mentions. It also helps in case of ACK
losses or reordering.

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Sat, 2 May 2020 10:34:15 +0200

Hi Yuchung,

<snip>

> Wireless as the other ack talk mentions. It also helps in case of ACK

Agreed!

By the way, in the next draft update, we will add that, despite its
"Experimental" status, ABC is supported in many implementations.

<snip>

Thanks,

Carles

---

From: Vidhi Goel <[email-masked]>
Date: Sat, 02 May 2020 20:21:15 -0700

Hi Carles,

> "Experimental" status, ABC is supported in many implementations.

Some time ago, I had brought up this same problem with ABC (RFC 3465) that increasing by at most one MSS during slow start makes the growth some what non-exponential in cases where ACK ratio < 1 (for example, receivers doing GRO or ACK compression, ACK losses etc). To allow an exponential growth, we should certainly increase congestion window by the amount of bytes ACKed (what Linux does).

Vidhi 

> On May 2, 2020, at 1:34 AM, Carles Gomez Montenegro <[email-masked]> wrote:
> 
> Hi Yuchung,
> 
> <snip>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 4 May 2020 10:57:15 -0700

> Some time ago, I had brought up this same problem with ABC (RFC 3465) that increasing by at most one MSS during slow start makes the growth some what non-exponential in cases where ACK ratio < 1 (f…

+1
I do want to mention that there's a legit concern of always
increasing based on #bytes: slow start (or generally congestion
control) can be more bursty with stretched-ACKs. IMHO idle-restart
bursts are just as bad and probably even more frequent with many TCP
senders disable slow-start after-idle and don't bother to pace. The
best form to de-burst is through (good) pacers like Linux's fq/pacing.
In other words congestion control and burst control should not be
juggled together via only congestion window changes.