Subject: Re: [tcpm] More motivating scenarios for tcpm-ack-pull

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Sun, 15 Mar 2020 10:38:43 +0100

Hi Jonathan,

(Sorry for the late response.)

Thanks a lot for your detailed response, which helped shape our latest
draft [1] in several ways.

While we focus on analyzing the problem space and its requirements, it
will be good to keep an eye on potential solutions as well, including the
one you describe below. In particular, I found the idea of spacing AKP
flags (or something similar) in order to control the desired ACK ratio to
be very interesting!

Thanks again!

Cheers,

Carles

[1] https://tools.ietf.org/html/draft-gomez-tcpm-delack-suppr-reqs-00.
Comments are very much welcome!

> I note that the semantics of the AKP flag differ subtly from that of PSH,
> in that the former says "I want an ack immediately", while the latter says
> "I have temporarily run out of data to send, so the receiving
> application's buffer should be flushed now".  It would be just as easy to
> modify existing receivers to treat PSH as an immediate ack request as to
> implement handling of AKP, so this simple use doesn't seem very compelling
> for allocating a scarce bit.
>
> One potentially interesting application that was suggested to me, and
> which could legitimately motivate the use of a new reserved flag bit, was
> to provide a hint as to the correct Delayed Ack ratio, for use as a
> congestion control mechanism for acks.  Presently an ack stream can be
> anti-responsive, in that dropping acks from a heavily congested reverse
> path can actually increase the ack arrival rate, by improving the
> throughput of an uncongested forward path that was limited by
> ack-clocking.  AckCC exists as an RFC to address this problem - and would
> tend to benefit from Generalised ECN - but its reliance on a TCP option is
> less than optimal, given the traversal problems often cited.
>
> A way that such a hint could work is by spacing AKP flags at the desired
> ack ratio.  A receiver observing AKP flags could then infer that the usual
> Delayed Ack rule requiring an ack for every two MSS received is suspended,
> and thus only send acks when AKP is set, the Delayed Ack timer expires, or
> some other forcing condition occurs.
>
> It might be wise to restore normal Delayed Ack rules if the AKP flags stop
> indicating a reasonable ratio, for example if the Delayed Ack timer
> expires twice without an AKP flag arriving.  A maximum ratio could be
> specified as a sanity check, eg. 1:64, at which the receiver generates an
> ack even if an AKP hasn't arrived yet.
>
> The above hinting semantics would remain compatible with other use cases
> mentioned, as AKP still forces an ack.  However this is also a theoretical
> disadvantage because the segments marked with AKP generally won't coincide
> with the trailing end of bursts caused, for example, by link-layer
> aggregation or IRQ "optimisation" in NICs.  Those trailing segments are
> where you would optimally want to trigger an ack, but as currently
> specified, AKP removes the receiver's discretion to synchronise a sparse
> Delayed Ack ratio with those segments.
>
> This could be addressed by pairing AKP with PSH.  Both flags set together
> would require an immediate ack of that particular segment, with the
> implication both that an ack is expected promptly and that no more data is
> immediately forthcoming.  PSH by itself would continue to permit a degree
> of latitude with respect to ack timing, and merely indicates that prompt
> delivery to the application is required.  AKP by itself then requests a
> prompt ack, but one that may reflect following segments that are already
> in receive buffers and will be processed immediately after this one, as
> well as the hinting described above.
>
> I think that's a reasonable amount of new functionality to introduce for
> the cost of one bit.  The above suggestion addresses both the original
> problem and another long-standing one, with minimal newly allocated
> resources and relatively little risk of traversal problems.  And
> (importantly) TCP doesn't break if receivers ignore or fail to understand
> it.
>
>  - Jonathan Morton

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Sun, 15 Mar 2020 10:47:21 +0100

Hi Bob,

Thanks, once again, for your kind suggestions!

We agreed with your proposal below, and we published the initial version
of a new I-D [1] that has the aim that you described below (i.e.
requirements analysis plus discussion of suggested solutions).

We hope the document can be useful to analyze the problem space in a more
comprehensive approach.

Should you have further comments, please do not hesitate to let us know!

Cheers,

Carles

[1] https://tools.ietf.org/html/draft-gomez-tcpm-delack-suppr-reqs-00

> Carles,
>
> It strikes me that there is a lot of interest in doing something like
> ACK-pull, but also a lot of disagreement in where to draw the line
> between what to do and what not to do.
>
> Can I suggest that a way to resolve this would be to start with an
> informational requirements draft, that I am sure would be adopted as WG
> business immediately given the interest. Whereas any particular choice
> of mechanism at this stage would probably not get the necessary
> consensus to be adopted.
>
> The requirements draft could discuss solutions, not as a "on this we all
> agree" protocol, but in terms of their pros and cons, e.g. which
> mechanisms could satisfy which requirements, so they can be judged
> relative to each mechanism's complexity / traversibility / usage of bits
> / etc.
>
> I will warn you that a requirements draft could itself add years to the
> process (the accurate ECN requirements did [RFC7560]). But not if we
> agree to allow a solution draft to start in parallel once the
> requirements are starting to converge, rather than waiting for the
> requirements to go thru the full RFC process.
>
>
> You might prefer to continue evangelising ack-pull as a sufficient
> solution, so I don't want to say a requirements draft is the only way
> forward. But you saw my comment earlier - that once you've been through
> all the years of effort of getting anything changed in TCP, you
> sometimes feel that you wish you'd been more ambitious than just one bit
> at the start.
>
>
> Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 15 Mar 2020 16:26:43 +0000

Carles & Jon,

Yes, if the tcpm WG was prepared to consider adopting this, I'd support it.

A few comments:

I'm not sure I would limit it to "suppression" of delayed ACKs, and I 
don't just mean it should include releasing suppression. In many 
circumstances the sender would be happy with less frequent ACKs than the 
receiver is generating. E.g. with high performance data transfers, 1/16 
would often be fine when there are hundreds of packets in flight. But, 
because the receiver can't be sure whether the sender would suffer one 
of the downsides to delaying ACKs, it maintains a conservatively low 
delayed ACK ratio.

So perhaps a better scope (and title) would be "Sender control of 
Delayed ACKs in TCP"?

I've added 'in TCP', because I think it's useful to make the scope 
concrete. But I think this doc could also be useful for other transport 
protocols, particularly QUIC.

S3.1 & S3.2 Slow start & High bit rate environments and short data segments

    However, use of Delayed ACKs reduces the amount
    of ACKs received by the sender, thus reducing the rate of cwnd
    growth, increasing transfer time and reducing throughput, when
    compared with sending an ACK for each incoming data segment.

I think ABC (appropriate byte counting) allows the sender to 
unilaterally address that concern without needing ACK Pull [RFC3465].

The reference to 'more rapidly get up to speed during slow-start without 
overshoot' is a bit opaque. Rather than "describing by reference", it 
needs to describe the lack of ability to send patterns of packets (e.g. 
chirps) and be able to time the gaps between the ACKs. At minimum, the 
citation should refer to Appendix B.4, which was the mechanism proposed 
to do ACK pull at the time.

S.3.4 Beyond classic ACK transmission behavior

It ought to mention that many link technologies (Satellite, DOCSIS, LTE, 
WiFi, ...?) do TCP ACK thinning 'cos TCP doesn't do it itself, but it 
improves performance (when the ACK stream becomes the bottleneck for the 
forward path, because many reverse paths have pathetic bandwidth, esp. 
if a parallel upload is going on).

See "ACK Scaling and Performance on AsymmetricPaths"
https://erg.abdn.ac.uk/~downloads/ackscaling.pdf

4.2. Per-segment granularity

Between per-segment and permanent, there's per-connection suppression. 
I'm not saying that would be any use, I'm just saying it doesn't have to 
be permanent.

4.4 Support for enabling generic ACK ratios

This reminds me of a subtle additional requirement. When you move from 
ACK pull to controlling the ACK ratio, there's an important distinction 
between a sender's packet stopping the receiver delaying the ACK for 
/that/ packet (as in ACK pull), vs. setting the receiver's Delayed ACK 
policy for /that and subsequent/ packets.

This raises the question, should the mechanism use soft state (repeated 
in every packet), or connection state (remembered by the receiver)? 
Which is hinted at in your section on "safe return to normal operation".x

4.10+ Extra requirement: Who's in control?

The receiver cannot be expected to control ACKs in any way the sender 
wants, which it will not always be able to honour anyway. So the 
semantics have to be of a hint, not a request or a command. Then there's 
the question of whether the receiver just does what it wants, or whether 
it ought to tell the sender it's not doing what it was asked.

It might seem that the the receiver silently not doing what it's told is 
the simple case. However,  it adds complexity to anything trying to rely 
on the behaviour (e.g. the slow start examples).

A good first draft.
Thank you.

Bob

On 15/03/2020 09:47, Carles Gomez Montenegro wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 15 Mar 2020 19:38:13 +0200

> On 15 Mar, 2020, at 11:38 am, Carles Gomez Montenegro <[email-masked]> wrote:
> 

I'm glad you found my comment useful.

I think one of the big takeaways from draft-gomez-tcpm-delack-suppr-reqs is that the sender has the most information about when, and to what extent, Delayed Acks are helpful, but the receiver is the point at which they must be implemented.  So some means to convey this information from sender to receiver is desirable.  The precise form of that is still open to debate.

And there are well-known circumstances where the receiver already has to interrupt or suspend Delayed Ack processing - when it notices a packet loss, or receives an ECN mark, for example.  I assume these would not change materially on a transport supporting Ack Pull or similar.

 - Jonathan Morton

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Thu, 26 Mar 2020 15:06:04 +0100

Hi Bob,

Thanks a lot once again for all your comments!

As you may have seen, we just published a -01, with the aim to address
your comments.

> protocols, particularly QUIC.

Agreed!

We have updated accordingly the document (including the title and abstract).

> unilaterally address that concern without needing ACK Pull [RFC3465].

Thanks for the comment. Version -01 now reflects that ABC could be used to
address the concern, while also mentioning that it remains as an
experimental mechanism, not fully included in RFC 5681.

> to do ACK pull at the time.

Agreed. We have added a more explicit description in -01, and we also
refer to Appendix B.4.

> https://erg.abdn.ac.uk/~downloads/ackscaling.pdf

Agreed. Added in -01. And thanks for the pointer!

> I'm not saying that would be any use, I'm just saying it doesn't have to

Agreed! Added in -01.

> Which is hinted at in your section on "safe return to normal operation".x

We have added text in section 4.4, with the aim to incorporate your
comments above.

> the simple case. However,Â  it adds complexity to anything trying to rely

We have added the new requirement, and have edited its content, based on
your input.

> Thank you.

Once again, thanks a lot for your comments!

Best regards,

Carles

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Thu, 26 Mar 2020 15:23:37 +0100

Hi Jonathan,

First of all, sorry for the late reply.

Thanks a lot for your comments. Please find below our inline responses:

> I think one of the big takeaways from draft-gomez-tcpm-delack-suppr-reqs
> is that the sender has the most information about when, and to what
> extent, Delayed Acks are helpful, but the receiver is the point at which
> they must be implemented. So some means to convey this information from
> sender to receiver is desirable.  The precise form of that is still open
> to debate.

Thanks for your conclusions and assessment.

We agree that the form to convey this information from sender to receiver
is open to debate. We hope that section 5 of our document can contribute
to the discussion!

> And there are well-known circumstances where the receiver already has to
> interrupt or suspend Delayed Ack processing - when it notices a packet
> loss, or receives an ECN mark, for example.  I assume these would not
> change materially on a transport supporting Ack Pull or similar.

In fact, there is a new subsection (4.10), suggested by Bob, that mentions
that the receiver might sometimes be unable to honour the ACK ratio or ACK
behavior desired by the sender (this relates also to your first paragraph
above). In a subsequent revision, we'll be more specific about
circumstances such as the ones you mention above.

Cheers,

Carles

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 27 Mar 2020 16:22:11 +0000

Carles,

> Agreed. Added in -01. And thanks for the pointer!

[BB]
                 Examples of technologies where deployments have
                 been reported to do ACK thinning include satellite 
links, DOCSIS
                 cable networks, mobile cellular networks, among others.

For DOCSIS at least I know that it's not just a case of "deployments 
have been reported". TCP ACK thinning is in the DOCSIS spec [DOCSIS3.1].

Also, I've just been talking with a colleague, who tells me that in WiFi 
(sorry, don't know how widely this applies), if there are multiple TCP 
ACKs for the same flow in the transmit queue, it will remove them all 
except the last when it transmits the queue. Also [Kim06] includes 
pseudocode for a model of WiFi ACK thinning.

  [DOCSIS3.1] CableLabs, "MAC and Upper Layer Protocols Interface 
(MULPI) Specification, CM-SP-MULPIv3.1", Data-Over-Cable Service 
Interface Specifications DOCSIS(R) 3.1, 
<https://specification-search.cablelabs.com/CM-SP-MULPIv3.1>.

[Kim06] Hyogon Kim, Hyogon Kim, Heejo Lee, Heejo Lee, Sangmin Shin and 
Inhye Kang, "On the Cross-Layer Impact of TCP ACK Thinning on IEEE 
802.11 Wireless MAC Dynamics"     DOI: 10.1109/VTCF.2006.463 In Proc. 
64th IEEE Vehicular Technology Conference, VTC 25-28 (September 2006)

Nit:
s/may allow alleviating congestion on that path/
  /may allow congestion on that path to be alleviated/

> We have added text in section 4.4, with the aim to incorporate your

[BB]

                 The desired generic ACK ratio is intended to be in 
force for the
                 current data segment, and for subsequent data segments 
(at least,
                 during a time interval of a duration that may depend on 
several
                 factors).

I didn't mean to say "that packet and subsequent packets" should be the 
only requirement. The original ACK pull pattern where the pull is just 
for "that packet" is also a valid possibility. They are just different 
models. If the mechanism ends up using a field in the main TCP header, 
then presumably it will be sent on every packet, and therefore there's 
no need to say what the behaviour should be in the absence of any 
request on subsequent packets.

> We have added the new requirement, and have edited its content, based on

[BB] Thanks for picking up all my comments so willingly.

It occurs to me that there's probably not much point in a message from 
the receiver saying whether it is complying with the request or not. The 
sender can detect which ACKs are arriving, so it knows the truth, 
whatever the receiver says it's doing. It sort-of might be useful to 
know what the receiver says it's trying to do, to know how the network 
might be intervening. But, if ground truth doesn't match what the 
receiver says it's doing, the sender doesn't know whether the receiver 
is lying, or whether the network is thinning ACKs.

> Once again, thanks a lot for your comments!

[BB] I suggest you ask the chairs how to go about asking for an adoption 
call, given that ought to be able to happen on the ML given we might not 
have a f2f meeting for some time.

Cheers

Bob

---

From: Rahul Arvind Jadhav <[email-masked]>
Date: Mon, 30 Mar 2020 09:33:39 +0000

Hello Bob,
Thanks for the pointer.

[BB]
                Examples of technologies where deployments have
                been reported to do ACK thinning include satellite links, DOCSIS
                cable networks, mobile cellular networks, among others.

For DOCSIS at least I know that it's not just a case of "deployments have been reported". TCP ACK thinning is in the DOCSIS spec [DOCSIS3.1].

[RJ] I couldn’t find any refs towards TCP ACK thinning in the mentioned DOCSIS spec. Not sure if I am missing something? The only sentence I found, “The CM SHOULD support a mechanism by which TCP ACK frames are prioritized or filtered in order to increase TCP session throughput” in section 9.1.2.3.2 but not sure if this is what you were referring?
I am interested in understanding more about what DOCSIS attempts  in the context of ACK thinning. Any details are appreciated.
Thanks in advance.

- Rahul

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 31 Mar 2020 08:55:24 +0100

Rahul,

On 30/03/2020 10:33, Rahul Arvind Jadhav wrote:
> [RJ] I couldn’t find any refs towards TCP ACK thinning in the 
> mentioned DOCSIS spec. Not sure if I am missing something? The only 
> sentence I found, “The CM SHOULD support a mechanism by which TCP ACK 
> frames are prioritized or filtered in order to increase TCP session 
> throughput” in section 9.1.2.3.2 but not sure if this is what you were 
> referring?
>
> I am interested in understanding more about what DOCSIS attempts  in 
> the context of ACK thinning. Any details are appreciated.
>
> Thanks in advance.
>

[BB] That's the sentence I meant.

I'm trying to find out what actual practice is with different 
implementations (initially by asking, but ultimately this can be measured).

Bob