Subject: [tcpm] A review for draft-ietf-tcpm-rack-09

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 18 Aug 2020 14:42:40 +0100

I have read draft-ietf-tcpm-rack-09 in preparation for TCPM to publish 
this document, and do have some comments. I note that this new rev is 
much easier to read this time, so thanks for the significant work to 
produce a clear spec, and I see also that you have already addressed my 
major concerns - thanks. (I'll separately catch a set of minor editorial 
notes on this rev).

(1) I don’t understand this clause:
“  3.  The RACK reordering window SHOULD leverage that to adaptively 
estimate the duration of reordering events, if the receiver uses  
Duplicate Selective Acknowledgement (DSACK) [RFC2883].”
- What does “leverage that” actually mean?
- Might it mean something like: can be increased if the sender receives 
a Duplicate Selective Acknowledgement (DSACK) [RFC2883], that suggests 
the window is too small, and has resulted in spurious retransmission." … 
or does it mean something different?

(2) I did not understand this part in Section 3.3:
“However, the fact that the initial reordering window is low, and the
    reordering window's adaptive growth is bounded, means that there will
    continue to be a cost to reordering to disincentivize excessive
    network reordering over highly disjoint paths.  For such networks
    there are good alternative solutions, such as MPTCP.“
- Is this intended to read as /reordering that disincentivizes excessive/
- If that was intended, the spec appears to be for a single path, MPTCP 
would still view this path with reordering as a single path, so I do not 
understand how MPTCP helps unless the awareness of the disjoint paths is 
somehow known at the endpoints? Please explain.

- (You’d need a REF for MPTCP if you keep this senetnce and the 
explanation).

(3)
  Question in Section 7.4.2.
/If the TLP
    sender does not receive such an indication, then it SHOULD assume
    that either the original data segment or the TLP retransmission were
    lost, for congestion control purposes./
- Why is this not a MUST?
- Under what conditions would it be safe to ignore a SHOULD?

(4) in Section 6.2. I was also left with a potential concern about 
“min_RTT” with respect to “a simple global minimum of all RTT 
measurements from the connection”.  This method results in a significant 
path change from low to high RTT exhibiting an invalid RTT. Even a 
simple windowed min-filtered estimate - mentioned here - would avoid 
this effect on reordering detection.

- I’d prefer adding a short sentence  so people know why the updates to 
min_RTT might be useful.

(5) In section 8.1, I expected more discussion of the potential 
disadvantages (even if the authors seem to suggest these are not 
significant) :

* I still was left without understanding whether there is an impact from 
a path with a varying RTT (perhaps one that uses a link layer 
retransmission or access technology). My feeling is that if there is 
excessive variation the method could go wrong when the RTT increases, 
but I wonder if this is usually captured by inflating the SRTT and that 
more excessive variation is not common. I think this case should be 
discussed briefly.

* As I read it, RACK can also send more retransmissions after there has 
been loss, where it is making spurious retransmissions - from timer 
events, or as a result of multiple segments from a single TSO segment 
being retransmitted. While it seems to be argued these are not 
significant, it could none the less a disadvantage in some scenarios, 
and should be captured in section 8.1.

* Also there seem cases where RACK allows TCP to continue to increase 
the congestion window upon receiving ACKs after loss, making the sender 
more aggressive. This is noted in 8.3, but I think should be listed in 
the disadvantages in 8.1.

(6) Section 8.4 talks about ACK loss or a delayed ACK without a DSACK, 
but does not mention Stretch ACKs, which have also been experienced, and 
need some form of mention!

(7) Why isn't TCP-NCR (RFC4653) at least mentioned and discussed in the 
intro?

Gorry

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 18 Aug 2020 13:28:24 -0700

> major concerns - thanks. (I'll separately catch a set of minor editorial
Thank you for the review!

> or does it mean something different?
It means exactly what you said :-)

Here is the latest rev based on Theresa' suggestion:
"The RACK reordering window SHOULD adaptively increase if the sender
receives the Duplicate Selective Acknowledgement (DSACK) [RFC2883],
suggesting the window is too small which causes spurious retransmission."

Let me know if that's better?

> - (You’d need a REF for MPTCP if you keep this senetnce and the
Yes that's the intention. Will removing the last sentence (about MPTCP)
help?

> - Under what conditions would it be safe to ignore a SHOULD?
We use SHOULD because the ACK could be lost instead of the data, so if the
sender has some mechanism to detect ACK losses well, it may be safe not to
assume data was lost.

But I agree this is over-thinking the corner cases so how about

If the TLP sender does not receive such an indication, then it *MUST*
assume that either the original data segment, or the TLP retransmission
were lost*, or their ACKs* are lost for congestion control purposes.

> - I’d prefer adding a short sentence  so people know why the updates to
Good point. (some) windowed-filter is definitely prefered. how about:

The sender SHOULD track a windowed min-filtered estimate of recent RTT
measurements to adapt migrating to significant longer paths, compared to a
simple global minimum of all RTT measurements.

> more excessive variation is not common. I think this case should be
AFAIU the scenario you are referring is when the link-layer retransmitted
packets took a much longer > current SRTT to be delivered. Then RACK
reordering window (and its SRTT-bound) requires DSACK and new RTT
measurement to raise to capture this variation. Depending on the RTT
variation, it may take multiple round trips to adapt to. The idea is for
short flows, RACK may not be able to cope with it (and we welcome any
simple idea to improve that). For long flows, RACK should eventually adapt
to these high varying RTT. Hence we've tested RACK with high varying degree
of reordering (in time) in synthetic benchmarks. In production experiments,
we have not found such high variation too common tho: the wireless radio
does not alter its transmission rate too often or change its retry count
dynamically. But of course, we try to avoid any mention of "empirical case
frequency" :-)

It that addresses your question, we can add some words in the reordering
design rationale about this.

> and should be captured in section 8.1.
The first case spurious RACK timer expiration is possible. So we can append
to the end of 8.1 with
"Another disadvantage is the reordering timer may expire prematurely (like
any other retransmission timer) to cause higher spurious retransmission
especially if DSACK is not supported".

However RACK should not cause extra spurious retransmission with TSO (vs
non-TSO) as a loss detection algorithm. Perhaps some text in TSO was not
clear on this?

> the disadvantages in 8.1.
Section 8.3 clearly states RACK can make C.C. more aggressive. Disadvantage
or not depends on how one looks at  it. IMO really beyond the scope of this
doc.

> need some form of mention!
There are two kinds of stretched ACKs I knew of: (a) ACKs delayed by the
receiver in hope to accumulate more.  (b) ACKs "compressed" or decimated by
the receiver or middle-boxes. It's the former (a) that affects RACK so we
use delayed ACK in the general delayed stretched ACK form.

> (7) Why isn't TCP-NCR (RFC4653) at least mentioned and discussed in the
It is briefly mentioned in
https://tools.ietf.org/html/draft-ietf-tcpm-rack-09#section-2.2 and
discussed a bit more in
https://tools.ietf.org/html/draft-ietf-tcpm-rack-09#section-8.2

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 19 Aug 2020 08:26:55 +0100

See below.

On 18/08/2020 21:28, Yuchung Cheng wrote:
>
>
> On Tue, Aug 18, 2020 at 6:42 AM Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
WFM.
> Yes that's the intention. Will removing the last sentence (about 
> MPTCP) help?
>
That would be an easy fix that would WFM.

> We use SHOULD because the ACK could be lost instead of the data, so if 
> the sender has some mechanism to detect ACK losses well, it may be 
> safe not to assume data was lost.
>
> assume that either the original data segment, or the TLP 
> retransmission were lost*, or their ACKs* are lost for congestion 
> control purposes.
>
I think the new text reads as safer to me.

> The sender SHOULD tracka windowed min-filtered estimate of recent RTT 
> measurements to adapt migrating to significant longer paths, compared 
> to a simple global minimum of all RTT measurements.
>
>
I like that proposed text. I don't think teh method has to be 
sophisticated, but some method seems the right thing to do.
> AFAIU the scenario you are referring is when the link-layer 
> retransmitted packets took a much longer > current SRTT to be delivered.
Yes.
> Then RACK reordering window (and its SRTT-bound) requires DSACK and 
> new RTT measurement to raise to capture this variation. Depending on 
> the RTT variation, it may take multiple round trips to adapt to. The 
> idea is for short flows, RACK may not be able to cope with it (and we 
> welcome any simple idea to improve that).
Understood.
> For long flows, RACK should eventually adapt to these high varying 
> RTT. Hence we've tested RACK with high varying degree of reordering 
> (in time) in synthetic benchmarks. In production experiments, we have 
> not found such high variation too common tho: the wireless radio does 
> not alter its transmission rate too often or change its retry count 
> dynamically. But of course, we try to avoid any mention of "empirical 
> case frequency" :-)
>
> It that addresses your question, we can add some words in the 
> reordering design rationale about this.
>
I think a few words would help - just so that if the corner case is 
encountered people understand there was a tradeoff.
> The first case spurious RACK timer expiration is possible. So we can 
> append to the end of 8.1 with
> "Another disadvantage is the reordering timer may expire prematurely 
> (like any other retransmission timer) to cause higher spurious 
> retransmission especially if DSACK is not supported".
>
That would be fine for me.
> However RACK should not cause extra spurious retransmission with TSO 
> (vs non-TSO) as a loss detection algorithm. Perhaps some text in TSO 
> was not clear on this?
>
>
> Section 8.3 clearly states RACK can make C.C. more aggressive. 
> Disadvantage or not depends on how one looks at it. IMO really beyond 
> the scope of this doc.

:-)

Indeed, how you view this will depend on what is regarded as best. I 
think it's kind of obvious there might be CC implications, maybe there 
is no need to discuss that.

> There are two kinds of stretched ACKs I knew of: (a) ACKs delayed by 
> the receiver in hope to accumulate more.  (b) ACKs "compressed" or 
> decimated by the receiver or middle-boxes.
> It's the former (a) that affects RACK so we use delayed ACK in the 
> general delayed stretched ACK form.
So, on this I think any stretch ACK caused by ACK delay, loss in the 
network, or intentional dropping has the same effect of extending the 
period it takes for the endpoint to observe the ACK. The typical stretch 
ACK intervals seem to me to be a few packets, so I am not sure typically 
this creates an issue for RACK, but if ACK delay is mentioned I don't 
understand why stretch ACKs from other cases are also not equally mentioned?
> https://tools.ietf.org/html/draft-ietf-tcpm-rack-09#section-8.2
The existing discussion is fine (thanks), I did not see the RFC listed 
in the References!
>     Gorry
Best wishes,

Gorry

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 19 Aug 2020 18:17:59 -0700

On Wed, Aug 19, 2020 at 12:27 AM Gorry Fairhurst <[email-masked]>
wrote:

> packets took a much longer > current SRTT to be delivered.
>
> Yes.
>
> Then RACK reordering window (and its SRTT-bound) requires DSACK and new
> RTT measurement to raise to capture this variation. Depending on the RTT
> simple idea to improve that).
>
> Understood.
>
> For long flows, RACK should eventually adapt to these high varying RTT.
> Hence we've tested RACK with high varying degree of reordering (in time) in
> synthetic benchmarks. In production experiments, we have not found such
> high variation too common tho: the wireless radio does not alter its
> transmission rate too often or change its retry count dynamically. But of
> course, we try to avoid any mention of "empirical case frequency" :-)
>

Sure -- how about adding this right after the reordering window rules in
the 'reordering window adaptation section'

...
2. The RACK reordering window SHOULD adaptively increase if the sender
receives the Duplicate Selective Acknowledgement (DSACK) [RFC2883],
suggesting the window is too small which causes spurious retransmission.

3. The RACK reordering window MUST be bounded and this bound SHOULD be SRTT.

<new> *The rule 2 and 3 combined are required to adapt reordering caused by
extended link layer recovery time described earlier. Then the reordering
window (and its SRTT-bound) requires DSACK and new RTT measurement to
increase. Depending on the RTT variation, it may take multiple round trips
to adapt. *</new>For short flows, the low initial reordering window is key
to recover quickly by risking spurious retransmissions. The rationale is
that spurious retransmissions for short flows are not expected to produce
excessive network traffic additionally. For long flows the design tolerates
reordering within a round trip. This handles reordering caused by path
divergence in small time scales (reordering within the round-trip time of
the shortest path).

> Disadvantage or not depends on how one looks at  it. IMO really beyond the
> scope of this doc.
>
>
> :-)
>
> Indeed, how you view this will depend on what is regarded as best. I think
> it's kind of obvious there might be CC implications, maybe there is no need
> to discuss that.
>
>
> the receiver or middle-boxes.
>
> It's the former (a) that affects RACK so we use delayed ACK in the general
> delayed stretched ACK form.
>
>
ok thanks for clarification. I propose we change to

Delayed *or stretched ACKs *complicate the detection of repairs done by
TLP, since with such ACKs the sender *takes longer time to receive fewer
ACKs *than would normally be expected.

> The existing discussion is fine (thanks), I did not see the RFC listed in
> the References!
>
Thanks for catching that. Will fix it. and double check if there're missing
refs. Probably another bug in our google-doc to rfc xml conversion script...

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 20 Aug 2020 08:28:31 +0100

Please see again below, it looks like this answered my questions.

On 20/08/2020 02:17, Yuchung Cheng wrote:
>
>
> On Wed, Aug 19, 2020 at 12:27 AM Gorry Fairhurst <[email-masked] 
>>     On Tue, Aug 18, 2020 at 6:42 AM Gorry Fairhurst
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>     "The RACK reordering window SHOULD adaptively increase if the
>>     sender receives the Duplicate Selective Acknowledgement (DSACK)
>>     [RFC2883], suggesting the window is too small which causes
>>     spurious retransmission."
>>
>>     We use SHOULD because the ACK could be lost instead of the data,
>>     so if the sender has some mechanism to detect ACK losses well, it
>>     may be safe not to assume data was lost.
>>
>>     If the TLP sender does not receive such an indication, then it
>>     *MUST* assume that either the original data segment, or the TLP
>>     The sender SHOULD tracka windowed min-filtered estimate of recent
>>     RTT measurements to adapt migrating to significant longer paths,
>>     compared to a simple global minimum of all RTT measurements.
>>
>>
>>     Then RACK reordering window (and its SRTT-bound) requires DSACK
>>     and new RTT measurement to raise to capture this variation.
>>     Depending on the RTT variation, it may take multiple round trips
>>     to adapt to. The idea is for short flows, RACK may not be able to
>>     cope with it (and we welcome any simple idea to improve that).
>     Understood.
>>     For long flows, RACK should eventually adapt to these high
>>     varying RTT. Hence we've tested RACK with high varying degree of
>>     reordering (in time) in synthetic benchmarks. In production
>>     experiments, we have not found such high variation too common
>>     tho: the wireless radio does not alter its transmission rate too
>>     often or change its retry count dynamically. But of course, we
>>     try to avoid any mention of "empirical case frequency" :-)
>>
>     I think a few words would help - just so that if the corner case
>     is encountered people understand there was a tradeoff.
>
>
> Sure -- how about adding this right after the reordering window rules 
> in the 'reordering window adaptation section'
>
> ...
> <new> *The rule 2 and 3 combined are required to adapt reordering 
> caused by extended link layer recovery time described earlier. Then 
> the reordering window (and its SRTT-bound) requires DSACK and new RTT 
> measurement to increase. Depending on the RTT variation, it may take 
> multiple round trips to adapt. *</new>

> For short flows, the low initial reordering window is key to recover 
> quickly by risking spurious retransmissions. The rationale is that 
> spurious retransmissions for short flows are not expected to produce 
> excessive network traffic additionally. For long flows the design 
> tolerates reordering within a round trip. This handles reordering 
> caused by path divergence in small time scales (reordering within the 
> round-trip time of the shortest path).
>
I think you have experience here, so I'd expect you to write the correct 
thing. From my side this looks like it addresses my comment.
>>     The first case spurious RACK timer expiration is possible. So we
>>     can append to the end of 8.1 with
>>     "Another disadvantage is the reordering timer may expire
>>     prematurely (like any other retransmission timer) to cause higher
>>     spurious retransmission especially if DSACK is not supported".
>>
>>     However RACK should not cause extra spurious retransmission with
>>     TSO (vs non-TSO) as a loss detection algorithm. Perhaps some text
>>     in TSO was not clear on this?
>>
>>
>>     Disadvantage or not depends on how one looks at  it. IMO really
>>     beyond the scope of this doc.
>
>     :-)
>
>     Indeed, how you view this will depend on what is regarded as best.
>     I think it's kind of obvious there might be CC implications, maybe
>     there is no need to discuss that.
>
>>
>>     There are two kinds of stretched ACKs I knew of: (a) ACKs delayed
>>     by the receiver in hope to accumulate more.  (b) ACKs
>>     "compressed" or decimated by the receiver or middle-boxes.
>>     It's the former (a) that affects RACK so we use delayed ACK in
>>     the general delayed stretched ACK form.
>     So, on this I think any stretch ACK caused by ACK delay, loss in
>     the network, or intentional dropping has the same effect of
>     extending the period it takes for the endpoint to observe the ACK.
>     The typical stretch ACK intervals seem to me to be a few packets,
>     so I am not sure typically this creates an issue for RACK, but if
>     ACK delay is mentioned I don't understand why stretch ACKs from
>     other cases are also not equally mentioned?
>
> Delayed *or stretched ACKs *complicate the detection of repairs done 
> by TLP, since with such ACKs the sender *takes longer time to receive 
> fewer ACKs *than would normally be expected.
This seems to address my comment :-).
>     The existing discussion is fine (thanks), I did not see the RFC
>     listed in the References!
>
> Thanks for catching that. Will fix it. and double check if there're 
> missing refs. Probably another bug in our google-doc to rfc xml 
> conversion script...
>
>>
>>         Gorry
>>
>>
>     Best wishes,
>
>     Gorry
>
Thanks,

Gorry

-- 
G. Fairhurst, School of Engineering