Subject: [tcpm] More motivating scenarios for tcpm-ack-pull

From: Bob Briscoe <[email-masked]>
Date: Tue, 5 Nov 2019 23:13:54 +0000

Carles, Jon, (re-sending, this time without accidentally omitting tcpm, 
also see couple of addenda tagged [Bob adds:])

I'm generally supportive of a mechanism to suppress delayed ACKs from 
the receiver (in any transport protocol including TCP). So thank you.

You might want to borrow at least some of the text for additional 
motivating scenarios for such a facility from the first two posts on a 
thread for a similar facility in QuiC:
https://github.com/quicwg/base-drafts/issues/1978

Also pasted at the end.

Some is not relevant to your specific ack-pull scheme, 'cos it's 
proposing a more general facility in QUIC for the sender to the receiver 
alters its ACK ratio. (theoretically there are more bits available in 
QUIC, but this particular request has been put on hold while someone 
goes and finds them ;)

      Another motivating example (rather niche)

Coincidentally, Just this morning I published a tech report that had to 
use the hack of overlapping a byte from the previous segment to force a 
quick ack:
TCP Prague Fall-back on Detection of a Classic ECN AQM 
<https://arxiv.org/pdf/1911.00710.pdf#page9> (the link takes you to Fig 
2, which illustrates the hack)
I'd rather not have to do these sorts of hacks - it really messes with 
the segmentation code.

      Middlebox traversal of bit 6

I recall that someone did a study of traversal of each of the reserved 
flags (Marcelo Bagnulo maybe?). I don't think it was that good [Bob 
adds: I mean traversal - not the study!].
A good search engine might find it ;)

      More alternative approaches:

1/ At one stage, we tried to include a bit for Delayed Ack control in 
another protocol called Accurate ECN. We were persuaded to take it out, 
because it was mission creep (outside the scope of the protocol we were 
working on):
https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#appendix-B.4

2/ Have you considered using the Urgent Pointer in a novel way? I.e. a 
non-zero Urgent Pointer field, even tho the URG flag is zero.
[Bob adds:] From memory, traversal was pretty good - much better than 
bits 4-6. But I think Windows treated it as a potential attack. I 
believe Fernando Gont did a study on how 'invalid' TCP header fields are 
handled.

You could assign, say, 3 bits of the urgent pointer for a variable 
ack_exp that is log base 2 of the ack ratio the sender would like. Then 
the sender can request the receiver uses ack_ratio` = 2^ack_exp, as in 
the QUIC proposal below,
With ack_exp = 0, you get ack_ratio = 2^0 = 1, which has the same effect 
as ack-pull. But you also have a more general facility for high speed 
machines to widen out the ack ratio.

If this works, you could open up a registry for the other 13 bits. So 
instead of using up a precious bit, you generate 13 more ;)

See:
https://tools.ietf.org/agenda/90/slides/slides-90-tcpm-10.pdf#page=5

I think there were problems with some implementations and middleboxes 
(bound to be). I've checked the minutes of the IETF meeting where that 
slide was presented 
<https://www.ietf.org/proceedings/90/minutes/minutes-90-tcpm>, but 
there's no push-back in there. I suggest you delve back into the 
archives of the tcpm mailing list around the time of that meeting to 
find out if there were any show-stoppers.

Cheers

Bob

Paste from QUIC thread:
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

      Problem/goals:

 1. Getting up to speed fast without inducing much queue is one of the
    main areas where latency reductions are needed (for short and long
    flows). The sender needs frequent ACKing during this phase in many
    approaches (hystart, etc). A Linux TCP receiver starts with
    ack_ratio 1 and uses heuristics to identify the end of the sender's
    slow-start (or a re-start after idle). This has ossified a certain
    slow-start behaviour into all TCP senders (whether Linux or not). We
    are trying to new sender behaviours (paced chirping being an
    example), but we need a way for the sender to suppress delayed ACKs.
 2. At the other end of the scale, for long-running flows, many current
    CC approaches (e.g. BBR) use pacing not ACK-clocking so they can use
    very few ACKs per RTT. And fewer means less cache ejections for GRO.
    But the receiver doesn't know what CC the sender is using, so it
    doesn't know how many or few is too many or too few.
 3. Some middleboxes and the majority of link technologies (e.g. DOCSIS,
    LTE, Satellite) thin TCP ACKs when they detect the upstream is
    filled with TCP ACK stream(s), which are unresponsive. Otherwise the
    ACKs constrain downstream throughput (and any upstream data either
    in the flow itself, or in others). We don't want these links to
    attempt to guess which are the QUIC ACKs and try to thin them. QUIC
    can and should do this itself. QUIC has all the machinery for the
    sender CC to detect ACK congestion, but not the protocol to tell the
    receiver to do the thinning. This protocol addition would provide a
    sufficient hook for hosts to unilaterally add this to their CC
    behaviour.

      Some scenarios where the sender's preferred ack_ratio changes
      through the connection:

 1.

    A sender CC that wants the receiver to turn off DelAcks during
    flow-start (e.g. it's using hybrid slow-start as in Cubic and wants
    to get delay measurements more frequently) sets ack_exp=0 during
    flow-start (ack_ratio=1), then increases ack_exp during congestion
    avoidance. If it goes idle, then re-starts, it would set ack_exp=0
    again.

    Note on heuristics: A Linux TCP receiver currently uses a heuristic
    to determine when the sender has exited slow-start. However,
    heuristics -> ossification. A Linux receiver's heuristic only works
    with the current pattern of slow-start. In TCP, when we tried to
    improve the pattern on the sender (paced chirping), the heuristic on
    the receiver killed us.

 2.

    Imagine a paced sender has hardware generic receive offload (GRO),
    so for a long-running flow it doesn't want a high rate of QUIC ACKs
    that are opaque to GRO. Let's say it would prefer at least 8 ACKs
    per RTT. Again, it starts with ack_exp=0, but in congestion
    avoidance it would use:

    ack_ratio <= cwnd_in_packets/8

Upshot: By remote controlling the receiver, the server offloads nearly 
all the ACKs from large downloads, but still focuses its ACK-receiving 
resources on getting each client up to speed.

Note that calculation of ack_ratio lends itself to fast integer arithmetic.

Bob

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Jon Crowcroft <[email-masked]>
Date: Wed, 06 Nov 2019 07:36:18 +0000

thanks v. much - especially for more motivating cases/text

we're strongly inclinded (due to context) to keep our particular approach as simple as possible ...(but
no simpler...) - that said, sharing common mechanism is also, of course, good if it can be done...

> I'm generally supportive of a mechanism to suppress delayed ACKs from the
> receiver (in any transport protocol including TCP). So thank you.
> 
> 
> 
> Some is not relevant to your specific ack-pull scheme, 'cos it's proposing
> a more general facility in QUIC for the sender to the receiver alters its
> ACK ratio. (theoretically there are more bits available in QUIC, but this
> particular request has been put on hold while someone goes and finds them
> ;)
> 
> 
> 
> use
> the hack of overlapping a byte from the previous segment to force a quick
> ack: TCP Prague Fall-back on Detection of a Classic ECN AQM <https://
> arxiv.org/pdf/1911.00710.pdf#page9> (the link takes you to Fig 2, which
> illustrates the hack) I'd rather not have to do these sorts of hacks - it
> really messes with the segmentation code.
> 
> 
> 
> flags (Marcelo Bagnulo maybe?). I don't think it was that good [Bob adds: 
> I
> mean traversal - not the study!].
> 
> non-zero Urgent Pointer field, even tho the URG flag is zero. [Bob adds:]
> From memory, traversal was pretty good - much better than bits 4-6. But I
> think Windows treated it as a potential attack. I believe Fernando Gont 
> did
> a study on how 'invalid' TCP header fields are handled.
> 
> 
> 
> You could assign, say, 3 bits of the urgent pointer for a variable ack_exp
> that is log base 2 of the ack ratio the sender would like. Then the sender
> can request the receiver uses ack_ratio` = 2^ack_exp, as in the QUIC
> proposal below, With ack_exp = 0, you get ack_ratio = 2^0 = 1, which has
> the same effect as ack-pull. But you also have a more general facility for
> high speed machines to widen out the ack ratio.
> 
> 
> 
> slide was presented <https://www.ietf.org/proceedings/90/minutes/
> minutes-90-tcpm>, but there's no push-back in there. I suggest you delve
> back into the archives of the tcpm mailing list around the time of that
> meeting to find out if there were any show-stoppers.
> 
> 
> Cheers
> 
> 
> 
> Bob
> 
> 
> Paste from QUIC thread:
> 
> Upshot: By remote controlling the receiver, the server offloads nearly all
> the ACKs from large downloads, but still focuses its ACK-receiving

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 6 Nov 2019 09:18:41 +0000

Jon,

 From the experience of recent years, I've changed my understanding of 
what is the appropriate level of 'simplicity' for each change to TCP.

It takes a huge amount of effort over probably a decade for any change 
to become useful (in terms of both standardization effort, and 
subsequent network traversal effort by all the implementers). So it 
makes little sense to judge simplicity solely on  the simplicity of the 
design. Once you take the whole process into account, it becomes worth 
introducing something perhaps slightly more complex that will be more 
useful to more use-cases. IMHO.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Mon, 18 Nov 2019 04:45:47 +0100

Hi Bob,

Once again, thanks a lot for all the very useful feedback!

Please find below some inline responses.

> QUIC, but this particular request has been put on hold while someone

Thanks a lot. This is indeed very useful!

As you explain, the more general facility for QUIC allows to suppress
Delayed ACKs and also allows to modify the ACK ratio in a more general
way.

Regarding the Delayed ACK suppression discussion, the additional
motivation is definitely important.

> I'd rather not have to do these sorts of hacks - it really messes with

This is very useful feedback as well, and a nice coincidence!

Regarding the hack, sending an old byte may be even worse when there is
not a next data segment readily available for sending, which would lead to
sending a whole packet just to carry an old byte.

> A good search engine might find it ;)

I wasn't able to find the study, but this is an important consideration.

One might think that if new functionality is defined and standardized,
related middlebox traversal of it should improve over time. Perhaps that
might be optimistic...

> https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#appendix-B.4

I was unaware of this, thanks!

This is actually very aligned with what we are proposing...

> instead of using up a precious bit, you generate 13 more ;)

Nice! :)

Actually, we had not considered such an idea.

In your proposal above, if a segment has the URG flag set to 1, would then
the 3 ack_exp bits still be used as ack_exp bits, thus reducing the size
of the urgent pointer in practice to 13 bits?

> find out if there were any show-stoppers.

Thanks for the pointers!

Sure, we'll check the tcpm mailing list archive.

Cheers,

Carles

---

From: Jon Crowcroft <[email-masked]>
Date: Mon, 18 Nov 2019 08:21:02 +0100

Mark Handley said a lot of the middlebox corner case avoidance stuff is in
the tcpmp RFC...have got time to look just yet...