Subject: Re: [tcpm] Last Call: <draft-ietf-tcpm-rack-13.txt> (The RACK-TLPloss detection algorithm for TCP) to Proposed Standard

From: Markku Kojo <[email-masked]>
Date: Fri, 4 Dec 2020 15:01:50 +0200 (EET)

Hi all,

I know this is a bit late but I didn't have time earlier to take look at
this draft.

Given that this RFC to be is standards track and RECOMMENDED to replace
current DupAck-based loss detection, it is important that the spec is
clear on its advice to those implementing it. Current text seems to
lack important advice w.r.t congestion control, and even though
the spec tries to decouple loss detection from congestion control
and does not intend to modify existing standard congestion control
some of the examples advice incorrect congestion control actions.
Therefore, I think it is worth to correct the mistakes and take
yet another look at a few implications of this specification.

Sec. 3.4 (and elsewhere when discussing recovering a dropped
retransmission):

It is very useful that RACK-TLP allows for recovering dropped rexmits. 
However, it seems that the spec ignores the fact that loss of a 
retransmission is a loss in a successive window that requires reacting 
to congestion twice as per RFC 5681. This advice must be included in 
the specification because with RACK-TLP recovery of dropped rexmit 
takes place during the fast recovery which is very different 
from the other standard algorithms and therefore easy to miss 
when implementing this spec.

Sec 9.3:

In Section 9.3 it is stated that the only modification to the existing 
congestion control algorithms is that one outstanding loss probe 
can be sent even if the congestion window is fully used. This is 
fine, but the spec lacks the advice that if a new data segment is sent 
this extra segment MUST NOT be included when calculating the new value 
of ssthresh as per the equation (4) of RFC 5681. Such segment is an 
extra segment not allowed by cwnd, so it must be excluded from 
FlightSize, if the TLP probe detects loss or if there is no ack 
and RTO is needed to trigger loss recovery.

In these cases the temporary over-commit is not accounted for as DupAck 
does not decrease FlightSize and in case of an RTO the next ACK comes too 
late. This is similar to the rule in RFC 5681 and RFC 6675 that prohibits
including the segments transmitted via Limitid Transmit in the 
calculation of ssthresh.

In Section 9.3 a few example scenarios are used to illustriate the
intended operation of RACK-TLP.

  In the first example a sender has a congestion window (cwnd) of 20
  segments on a SACK-enabled connection.  It sends 10 data segments
  and all of them are lost.

The text claims that without RACK-TLP the ending cwnd would be 4 segments 
due to congestion window validation. This is incorrect.
As per RFC 7661 the sender MUST exit the non-validated phase upon an 
RTO. Therefore the ending cwnd would be 5 segments (or 5 1/2 segments if 
the TCP sender uses the equation (4) of RFC 5681).

The operation with RACK-TLP would inevitably result in congestion 
collapse if RACK-TLP behaves as described in the example because 
it restores the previous cwnd of 10 segments after the fast recovery 
and would not react to congestion at all! I think this is not the 
intended behavior by this spec but a mistake in the example. 
The ssthresh calculated in the beginning of loss recovery should 
be 5 segments as per RFC 6675 (and RFC 5681).

Furthermore, it seems that this example with RACK-TLP refers to using 
PRR_SSRB which effectively implements regular slow start in this 
case(?). From congestion control point of view this is correct because 
the entire flight of data as well as ack clock was lost.

However, as correctly discussed in Sec 2, congestion window must be reset 
to 1 MSS when an entire flight of data is and Ack clock is lost. But how 
can an implementor know what to do if she/he is not implementing the 
experimental PRR algrorithm? This spec articulates specifying an 
alternative for DupAck counting, indicating that TLP is used to trigger 
Fast Retransmit & Fast Recovery only, not a loss recovery in slow start. 
This means that without an additional advise an implementation of this 
spec would just halve the cwnd and ssthresh and send a potentially very 
large burst of segments in the beginning of the Fast Recovery because 
there is no ack clock. So, this spec begs for an advise (MUST) when to 
slow start and reset cwnd and when not, or at least a discussion of 
this problem and some sort of advise what to do and what to avoid.
And, maybe a recommendation to implement it with PRR?

Another question relates to the use of TLP and adjusting timer(s) upon 
timeout. In the same example discussed above, it is clear that PTO 
that fires TLP is just a more aggressive retransmit timer with 
an alternative data segment to (re)transmit.

Therefore, as per RFC 2914 (BCP 41), Sec 9.1, when PTO expires, it is in 
effect a retransmission timout and the timer(s) must be backed-off.
This is not adviced in this specification. Whether it is the TCP RTO 
or PTO that should be backed-off is an open question.  Otherwise, 
if the congestion is persistent and further transmission are also lost, 
RACK-TLP would not react to congestion properly but would keep 
retransmitting with "constant" timer value because new RTT estimate 
cannot be obtained.
On a buffer bloated and heavily congested bottleneck this would easily 
result in sending at least one unnecessary retransmission per one 
delivered segment which is not advisable (e.g., when there are a huge 
number of applications sharing a constrained bottleneck and these 
applications are sending only one (or a few) segments and then 
waiting for an reply from the peer before sending another request).

Additional notes:

Sec 2.2:

Example 2:
"Lost retransmissions cause a  resort to RTO recovery, since
  DUPACK-counting does not detect the loss of the retransmissions.
  Then the slow start after RTO recovery could cause burst losses
  again that severely degrades performance [POLICER16]."

RTO reovery is done in slow start. The last sentence is confusing as 
there is no (new) slow-start after RTO recovery (or more precisely 
slow start continues until cwnd > ssthresh). Do you mean: if/when slow 
start still continues after RTO Recovery has repaired lost segments, 
it may cause burst losses again?

Example 3:
  "If the reordering degree is beyond DupThresh, the DUPACK-
   counting can cause a spurious fast recovery and unnecessary
   congestion window reduction.  To mitigate the issue, [RFC4653]
   adjusts DupThresh to half of the inflight size to tolerate the
   higher degree of reordering.  However if more than half of the
   inflight is lost, then the sender has to resort to RTO recovery."

This seems to be somewhat incorrect description of TCP-NCR specified in 
RFC 4653. TCP-NCR uses Extended Limited Transmit that keeps on sending 
new data segments on DupAcks that makes it likely to avoid an RTO in 
the given example scenario, if not too many of the the new data 
segments triggered by Extended Limited Transmit are lost.

Sec. 3.5:

  "For example, consider a simple case where one
  segment was sent with an RTO of 1 second, and then the application
  writes more data, causing a second and third segment to be sent right
  before the RTO of the first segment expires.  Suppose only the first
  segment is lost.  Without RACK, upon RTO expiration the sender marks
  all three segments as lost and retransmits the first segment.  When
  the sender receives the ACK that selectively acknowledges the second
  segment, the sender spuriously retransmits the third segment."

This seems incorrect. When the sender receives the ACK that selectively 
acknowledges the second segment, it is a DupAck as per RFC 6675 and does 
not increase cwnd and cwnd remains as 1 MSS and pipe is 1 MSS. So, the 
rexmit of the third segment is not allowad until the cumulative ACK of 
the first segment arrives.

Best regards,

/Markku

On Mon, 16 Nov 2020, The IESG wrote:

>
> The IESG has received a request from the TCP Maintenance and Minor Extensions
> WG (tcpm) to consider the following document: - 'The RACK-TLP loss detection
> algorithm for TCP'
>  <draft-ietf-tcpm-rack-13.txt> as Proposed Standard
>
> The IESG plans to make a decision in the next few weeks, and solicits final
> comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2020-11-30. Exceptionally, comments may
> be sent to [email-masked] instead. In either case, please retain the beginning
> of the Subject line to allow automated sorting.
>
> Abstract
>
>
>   This document presents the RACK-TLP loss detection algorithm for TCP.
>   RACK-TLP uses per-segment transmit timestamps and selective
>   acknowledgements (SACK) and has two parts: RACK ("Recent
>   ACKnowledgment") starts fast recovery quickly using time-based
>   inferences derived from ACK feedback.  TLP ("Tail Loss Probe")
>   leverages RACK and sends a probe packet to trigger ACK feedback to
>   avoid retransmission timeout (RTO) events.  Compared to the widely
>   used DUPACK threshold approach, RACK-TLP detects losses more
>   efficiently when there are application-limited flights of data, lost
>   retransmissions, or data packet reordering events.  It is intended to
>   be an alternative to the DUPACK threshold approach.
>
>
>
>
> The file can be obtained via
> https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/
>
>
>
> No IPR declarations have been submitted directly on this I-D.
>
>
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 4 Dec 2020 15:56:55 -0800

> yet another look at a few implications of this specification.
As you noted, the intention is to decouple the two as much as possible

Unlike the 20 years ago where TCP loss detection and congestion
control are essentially glued in one piece, the decoupling of the two
(including modularizing congestion controls in implementations) has
helped fueled many great inventions of new congestion controls.
Codifying so-called-default C.C. reactions in the loss detection is a
step backward that the authors try their best to avoid.To keep the
document less "abstract / unclear" as many WGLC reviewers commented,
we use examples to illustrate that includes CC actions. But the
details of these CC actions are likely to become obsolete as CC
continues to advance hopefully.

> when implementing this spec.

per RFC5681 sec 4.3 https://tools.ietf.org/html/rfc5681#section-4.3
"Loss in two successive windows of data, or the loss of a
  retransmission, should be taken as two indications of congestion and,
   therefore, cwnd (and ssthresh) MUST be lowered twice in this case."

RACK-TLP is a loss detection algorithm. RFC5681 is crystal clear on
this so I am not sure what clause you suggest to add to RACK-TLP.
> and RTO is needed to trigger loss recovery.

Why exclude TLP (or any data) from FlightSize? The congestion control
needs precise accounting of the flight size to react to congestion
properly.

> be 5 segments as per RFC 6675 (and RFC 5681).
To clarify, would this text look more clear?

'an ending cwnd set to the slow start threshold of 5 segments (half of
the original congestion window of 10 segments)'

> And, maybe a recommendation to implement it with PRR?

It's wise to decouple loss detection (RACK-TLP) vs congestion/burst
control (when to slow-start). The use of PRR is just an example to
illustrate and not meant for a recommendation.

Section 3 has a lengthy section to elaborate the key point of RACK-TLP
is to maximize the chance of fast recovery. How C.C. governs the
transmission dynamics after losses are detected are out of scope of
this document in our authors' opinions.

> waiting for an reply from the peer before sending another request).

Thanks for pointing to the RFC.  After TLP, RTO timers will
exp-backoff (as usual) for stability reasons mentioned in sec 9.3
(didn't find 9.1 relevant). In your scenario, you presuppose the
retransmission is unnecessary so obviously TLP is not good. Consider
what happens without TLP where all the senders fire RTO spuriously and
blow up the network. It is equally unfortunate behavior. "bdp
insufficient of many flows" is a congestion control problem

> it may cause burst losses again?
I mean the slow start after (the start of) RTO recovery. HTH

> segments triggered by Extended Limited Transmit are lost.
sorry I don't see how the text is wrong describing RFC4653,
specifically the algorithm in adjusting ssthresh

> the first segment arrives.
I don't see where RFC6675 forbids growing cwnd. Even if it does, I
don't think it's a good thing (in RTO-slow-start) as DUPACK clearly
indicates a delivery has been made.

---

From: Ian Swett <[email-masked]>
Date: Fri, 4 Dec 2020 20:13:51 -0500

+1 to decoupling loss detection from congestion control.  It ensures RACK
does not need to not be updated every time a new congestion controller is
standardized.

---

From: Markku Kojo <[email-masked]>
Date: Mon, 7 Dec 2020 18:06:03 +0200 (EET)

Hi Yuchung,

thanks for your reply. My major point is that IMHO this RACK-TLP 
specification should give the necessary advice w.r.t congestion control 
in cases when such advice is not available in the RFC series or is not 
that easy to interpreted correctly from the existing RFCs.

Even though the CC principles were available in RFC series I believe you 
agree with me that getting detailed CC actions correct is sometimes quite 
hard, especially if one needs to interpret and apply them from another 
spec than what one is implementing. Please see more details inline below.

In addition, we need to remember that this document is not meant only for 
TCP experts following the tcpm list and having deep understanding of 
congestion control but also those implementing TCP, for the very first 
time, for various appliances, for example. They do not have first hand 
experience in implementing TCP congestion control and deserve clear 
advice what to do.

On Fri, 4 Dec 2020, Yuchung Cheng wrote:

> step backward that the authors try their best to avoid.

While I fully agree with the general principle of decoupling loss 
detection from congestion control when it is possible without leaving 
open questions, I find it hard to get congestion control right with this 
spec in certain cases I raised just by following the current standards 
track CC specifications. The reason for this is that RACK-TLP introduces 
new ways to detect loss (i.e., not present in any earlier standard track 
RFC) and the current CC specifications do not provide correct CC actions 
for such cases as I try to point out below.

> To keep the

Agreed. But I would appreciate if the CC actions in the examples would 
correctly follow what is specified in the the current CC RFCs. And, I 
would suggest explicitly citing the RFC(s) that each of the examples is 
illustriating so that there is no doubt which CC variant the example 
is valid with. Then there is no problem with the correctness of the 
example either even if the cited RFC becomes later obsoleted.

> this so I am not sure what clause you suggest to add to RACK-TLP.

Right, this is the CC *principle* in RFC 5681 I am refering to but I am 
afraid it is not enough to help one to correctly implement such lowering 
of cwnd (and ssthresh) twice when a loss of a retransmission is detected 
during Fast Recovery. Nor do RFCs clearly advice *when* this reduction 
must take place.

Congestion control principles tell us that congestion must be reacted 
immediately when detected. But at the same time, standards track CC 
specifications react to congestion only once during Fast Recovery 
because the losses in the current window, which Fast Recovery repairs, 
occured during the same RTT. That is, the current CC specifications do 
not handle lost rexmits during Fast Recovery but, instead, the correct CC 
reaction to a loss of a rexmit is automatically achieved by those 
specifications via RTO when cwnd is explicitly reset upon RTO.

Furthermore, the problem I am trying to point out is that there is no 
correct rule/formula available in the standards track RFCs that would 
give the correct way to reduce cwnd when the loss of rexmit is detected 
with RACK-TLP.

I suggest everyone reading this message stops reading at this point 
for a while before continuing reading and figures out themselves what 
they think would be the correct equation to use in the standards RFC 
series to find the new halved value for cwnd when RACK-TLP detects a loss 
of a rexmit during Fast Recovery. I would appreciate a comment on the 
tcpm list from those who think they found the correct answer immediately 
and easily, and what was the formula to use.

...

I think the best advice one may find by looking at RFC 6675 (and RFC 
5681) is to set

  ssthresh = cwnd = (FlightSize / 2) (RFC 6675, Sec 5, algorithm step 4.2)

Now, let's modify the example in Sec 3.4 of the draft:

1. Send P0, P1, P2, ..., P20
   [Assume P1, ..., P20 dropped by network]

2.   Receive P0, ACK P0
3a.  2RTTs after (2), TLP timer fires
3b.  TLP: retransmits P20
...
5a.  Receive SACK for P20
5b.  RACK: marks P1, ..., P20 lost
      set cwnd=ssthresh=FlightSize/2=10
5c.  Retransmit P1, P2 (and some more depending on how CC implemented)
      [P1 retransmission dropped by network]

      Receive SACK P2 & P3 
7a.  RACK: marks P1 retransmission lost
      As per RFC 6675: set cwnd=ssthresh=FlightSize/2=20/2=10
7b.  Retransmit P1
      ...

So, is the new value of the cwnd (10MSS) correct and halved twice? If not, 
where is the correct formula to do it?

Before RFC 5681 was published we had a discussion on FlighSize and that 
during Fast Recovery it does not reflect the amount of segments in 
flight correctly but may also have a too large value. It was decided not 
to try to correct it because it only has an impact when RTO fires during 
Fast Recovery and in such a case cwnd is reset to 1 MSS. Having too large 
ssthresh for RTO recovery in some cases was not considered that bad 
because a TCP sender anyway takes to most conservative CC action with the 
cwnd and would slow start from cwnd = 1 MSS. But now when RACK-TLP enables 
detecting loss of a rexmit during Fast Recovery we have an unresolved 
problem.

> needs precise accounting of the flight size to react to congestion

Because FlightSize does not always reflect the correct amount of data 
allowed by cwnd. When a TCP sender is not already in loss recovery and 
it detects loss, this loss indicates the congestion point for the TCP 
sender, i.e., how much data it can have outstanding. It is this amount of 
data that it must use in calculating the new value of cwnd (and 
ssthresh), so it must not include any data sent beyond the congestion 
point. When TLP sends a new data segment, it is beyond the congestion 
point and must not be included. Same holds for the segments sent via 
Limited Transmit: they are allowed to be send out by the packet 
conservation rule (DupAck indicates a pkt has left the network, but does 
not allow increasing cwnd), i.e., the actual amount of data in flight 
remains the same.

> the original congestion window of 10 segments)'

This is correct, but please replace:

  (half of the original congestion window of 10 segments)
-->
  (half of> the original FlightSize of 10 segments)

cwnd in the example was 20 segments.

Please also correct the ending cwnd for "without RACK" scenario.
I poined out wrong equation number IN RFC 5681 and INCORRECT cwnd value, 
my apologies. It MAENT equation (3) AND that results in ending cwnd of 5 
and 2/5 MSS (not 5 and 1/2 MSS).
NB: and if a TCP sender implements entering CA when cwnd > ssthresh, then 
ending cwnd would be 6 and 1/6 MSS).

> illustrate and not meant for a recommendation.

I understand the use of PRR was just an example, but my point is that if 
one wants to implement RACK-TLP but does not intend to implement PRR but 
RFC 6675 then we do not have a rule in RFC 6675 to correctly implement 
CC for the case when an entire flight is lost and loss is detected with 
TLP. Congestion control principle for this is clear and also stated in 
this draft but IMHO it is not enough to ensure correct implementation.

To my understanding we only have implementation experience for RACK-TLP 
only togerher with PRR, which has the necessary rule to handle this kind 
of scenario correctly.

So, my question is how can one implement CC correctly without PRR such 
a scenario where entire inflight is lost?
Which rule and where in the RFC series gives the necessary guidance to 
reset cwnd and slow start when TCP detets loss of an entire flight?

> (didn't find 9.1 relevant).

My apologies for refering to the wrong section of RFC 2914, Yes, I meant 
Sec 9.3.

> In your scenario, you presuppose the

If (without TLP) RTO is spurious, it may result in unnecessary 
retransmissions. But we have F-RTO (RFC 5682) and Eifel (RFC 3522) to 
detect and resolve it without TLP, so I don't find it as a problem.

To clarify more, what I am concerned about. Think about a scenario where a 
(narrow) bottleneck becomes heavily congested by a huge number of 
competing senders such that the available capacity per sender is less 
than 1 segment (or << 1 MSS).
This is a situation that network first enters before congestion collapse 
gets realized. So, it is extremely important that all CC and timer 
mechanisms handle it properly. Regular TCP handles it via RFC 6298 by 
backing off RTO expotentially and keeping this backed-off RTO until an 
new ACK is received for new data. This saves RACK-TLP from full congestion 
collapse. But consider what happens: even though RTO is backed off, each 
time a TCP sender manages to get one segment through (with cwnd = 1 MSS) 
it always first arms PTO with more or less constant value of 2*SRTT. If 
the bottleneck is buffer bloated the actual RTT easily exceeds 2*SRTT and 
TLP becomes spurious. After a spurious TLP, RTO expires (maybe more than 
once before exponential back-off of RTO results in large enough value) 
and a new RTT sample is not received. So, SRTT remains unchanged and even 
if sometimes a new sample is received, SRTT gets very slowly adjusted. As 
a result, each TCP sender would keep on sending a spurious TLP for each 
new segment resulting in at least 50% of the packets being unnecessary 
rexmitted and the utilization of the bottleneck is < 50%. This would not 
be a full congestion collapse but has unwanted symptoms towards 
congestion collapse (Note: there is no clear line for the level of 
reduction in delivery of useful data is considered as congestion 
collapse).

> I mean the slow start after (the start of) RTO recovery. HTH

Tnx. I'd appreciate if the text could be clarified to reflect this more 
accurately. Maybe something along the lines(?):

  "Then the RTO recovery in slow start could cause burst
  losses again that severely degrades performance [POLICER16]."

> specifically the algorithm in adjusting ssthresh

To my understanding RFC4653 initializes DupThresh to half of the inflight 
size in the beginning of the Extended Limited Transmit. Then on each 
DupAck it adjusts (recalculates) DupThresh again such that ideally a cwnd 
worth of DupAcks are received before packet loss is declared (or 
reordering detected). So, if I am not incorrect, loss of a half of the 
inflight does not necessarily result in RTO recovery with TCP-NCR.

> indicates a delivery has been made.

SACKed sequences with DUpAcks indicate that those sequences were 
delivered but it does not tell when they were sent. The basic principle 
of slow start is to reliably determine the available network capacity 
during slow start. Therefore, slow start must ensure it uses only 
segments sent during the slow start to increase cwnd. Otherwise, a TCP 
sender may encounter exactly the problem of unnecessary retransmission 
envisioned in this example of RACK-TCP draft (and increase cwnd on not 
valid Acks).

RFC 6675 does re-specify DupAck with SACK option but it does not include 
the rule for slow start. Slow start is specified in RFC 5681. It is 
crystal clear in allowing increase in cwnd only on cumulative 
Acks, i.e., forbidding to increase cwnd on DupAcks (RFC 5681, Sec 3,
page 6:

   During slow start, a TCP increments cwnd by at most SMSS bytes for
   each ACK received that cumulatively acknowledges new data.

Maybe this example in the RACK-TLP draft was inspired by a incorrect 
implementation of SACK-based loss recovery?

FYI: when we were finalizing RFC 6675 I suggested including also an 
algorithm for RTO recovery with SACK in RFC 6675. The reason was exactly 
that it might be not easy to gather info from multiple documents and 
hence help the implementor to have all necessary advice in a single 
document. This unfortunately did not get realized though.

BR,

/Markku

---

From: Michael Welzl <[email-masked]>
Date: Mon, 7 Dec 2020 18:47:38 +0100

Hi all,

thanks for this discussion, and Markku for this long email, I found it very interesting to read.  Sorry for top posting but:

Sent from my iPhone
:-)

....   so, just a comment about the backoff upon a lost rexmit in recovery: I‘d say that halving again is the right thing to do as it‘s certainly in the spirit of the cong.avoid & control paper which, iirc, motivates halving ssthresh with an exponential backoff. this just allows to truly backoff exponentially instead of halving once, followed by cwnd=1.

regarding how to do it because flightsize may be wrong: perhaps i‘m missing something, i‘m not writing this with rfcs and rack-tlp open on the side....  but: surely rack-tlp also still has the „pipe“ variable?  wouldn‘t cwnd = ssthresh = pipe / 2 be the right thing to do?

cheers,
michael

> On 7 Dec 2020, at 17:07, Markku Kojo <kojo=[email-masked]> wrote:
> 
> Hi Yuchung,
> 
> thanks for your reply. My major point is that IMHO this RACK-TLP specification should give the necessary advice w.r.t congestion control in cases when such advice is not available in the RFC series or is not that easy to interpreted correctly from the existing RFCs.
> 
> Even though the CC principles were available in RFC series I believe you agree with me that getting detailed CC actions correct is sometimes quite hard, especially if one needs to interpret and apply them from another spec than what one is implementing. Please see more details inline below.
> 
> In addition, we need to remember that this document is not meant only for TCP experts following the tcpm list and having deep understanding of congestion control but also those implementing TCP, for the very first time, for various appliances, for example. They do not have first hand experience in implementing TCP congestion control and deserve clear advice what to do.
> 
> While I fully agree with the general principle of decoupling loss detection from congestion control when it is possible without leaving open questions, I find it hard to get congestion control right with this spec in certain cases I raised just by following the current standards track CC specifications. The reason for this is that RACK-TLP introduces new ways to detect loss (i.e., not present in any earlier standard track RFC) and the current CC specifications do not provide correct CC actions for such cases as I try to point out below.
> 
>> To keep the
> Agreed. But I would appreciate if the CC actions in the examples would correctly follow what is specified in the the current CC RFCs. And, I would suggest explicitly citing the RFC(s) that each of the examples is illustriating so that there is no doubt which CC variant the example is valid with. Then there is no problem with the correctness of the example either even if the cited RFC becomes later obsoleted.
> 
>>> 
> Right, this is the CC *principle* in RFC 5681 I am refering to but I am afraid it is not enough to help one to correctly implement such lowering of cwnd (and ssthresh) twice when a loss of a retransmission is detected during Fast Recovery. Nor do RFCs clearly advice *when* this reduction must take place.
> 
> Congestion control principles tell us that congestion must be reacted immediately when detected. But at the same time, standards track CC specifications react to congestion only once during Fast Recovery because the losses in the current window, which Fast Recovery repairs, occured during the same RTT. That is, the current CC specifications do not handle lost rexmits during Fast Recovery but, instead, the correct CC reaction to a loss of a rexmit is automatically achieved by those specifications via RTO when cwnd is explicitly reset upon RTO.
> 
> Furthermore, the problem I am trying to point out is that there is no correct rule/formula available in the standards track RFCs that would give the correct way to reduce cwnd when the loss of rexmit is detected with RACK-TLP.
> 
> I suggest everyone reading this message stops reading at this point for a while before continuing reading and figures out themselves what they think would be the correct equation to use in the standards RFC series to find the new halved value for cwnd when RACK-TLP detects a loss of a rexmit during Fast Recovery. I would appreciate a comment on the tcpm list from those who think they found the correct answer immediately and easily, and what was the formula to use.
> 
> ...
> 
> I think the best advice one may find by looking at RFC 6675 (and RFC 5681) is to set
> 
>     Receive SACK P2 & P3 7a.  RACK: marks P1 retransmission lost
> So, is the new value of the cwnd (10MSS) correct and halved twice? If not, where is the correct formula to do it?
> 
> Before RFC 5681 was published we had a discussion on FlighSize and that during Fast Recovery it does not reflect the amount of segments in flight correctly but may also have a too large value. It was decided not to try to correct it because it only has an impact when RTO fires during Fast Recovery and in such a case cwnd is reset to 1 MSS. Having too large ssthresh for RTO recovery in some cases was not considered that bad because a TCP sender anyway takes to most conservative CC action with the cwnd and would slow start from cwnd = 1 MSS. But now when RACK-TLP enables detecting loss of a rexmit during Fast Recovery we have an unresolved problem.
> 
> 
>>> Sec 9.3:
>>> 
> Because FlightSize does not always reflect the correct amount of data allowed by cwnd. When a TCP sender is not already in loss recovery and it detects loss, this loss indicates the congestion point for the TCP sender, i.e., how much data it can have outstanding. It is this amount of data that it must use in calculating the new value of cwnd (and ssthresh), so it must not include any data sent beyond the congestion point. When TLP sends a new data segment, it is beyond the congestion point and must not be included. Same holds for the segments sent via Limited Transmit: they are allowed to be send out by the packet conservation rule (DupAck indicates a pkt has left the network, but does not allow increasing cwnd), i.e., the actual amount of data in flight remains the same.
> 
> I poined out wrong equation number IN RFC 5681 and INCORRECT cwnd value, my apologies. It MAENT equation (3) AND that results in ending cwnd of 5 and 2/5 MSS (not 5 and 1/2 MSS).
> NB: and if a TCP sender implements entering CA when cwnd > ssthresh, then ending cwnd would be 6 and 1/6 MSS).
> 
>>> 
> I understand the use of PRR was just an example, but my point is that if one wants to implement RACK-TLP but does not intend to implement PRR but RFC 6675 then we do not have a rule in RFC 6675 to correctly implement CC for the case when an entire flight is lost and loss is detected with TLP. Congestion control principle for this is clear and also stated in this draft but IMHO it is not enough to ensure correct implementation.
> 
> To my understanding we only have implementation experience for RACK-TLP only togerher with PRR, which has the necessary rule to handle this kind of scenario correctly.
> 
> So, my question is how can one implement CC correctly without PRR such a scenario where entire inflight is lost?
> Which rule and where in the RFC series gives the necessary guidance to reset cwnd and slow start when TCP detets loss of an entire flight?
> 
> My apologies for refering to the wrong section of RFC 2914, Yes, I meant Sec 9.3.
> 
> If (without TLP) RTO is spurious, it may result in unnecessary retransmissions. But we have F-RTO (RFC 5682) and Eifel (RFC 3522) to detect and resolve it without TLP, so I don't find it as a problem.
> 
> To clarify more, what I am concerned about. Think about a scenario where a (narrow) bottleneck becomes heavily congested by a huge number of competing senders such that the available capacity per sender is less than 1 segment (or << 1 MSS).
> This is a situation that network first enters before congestion collapse gets realized. So, it is extremely important that all CC and timer mechanisms handle it properly. Regular TCP handles it via RFC 6298 by backing off RTO expotentially and keeping this backed-off RTO until an new ACK is received for new data. This saves RACK-TLP from full congestion collapse. But consider what happens: even though RTO is backed off, each time a TCP sender manages to get one segment through (with cwnd = 1 MSS) it always first arms PTO with more or less constant value of 2*SRTT. If the bottleneck is buffer bloated the actual RTT easily exceeds 2*SRTT and TLP becomes spurious. After a spurious TLP, RTO expires (maybe more than once before exponential back-off of RTO results in large enough value) and a new RTT sample is not received. So, SRTT remains unchanged and even if sometimes a new sample is received, SRTT gets very slowly adjusted. As a result, each TCP sender would keep on sending a spurious TLP for each new segment resulting in at least 50% of the packets being unnecessary rexmitted and the utilization of the bottleneck is < 50%. This would not be a full congestion collapse but has unwanted symptoms towards congestion collapse (Note: there is no clear line for the level of reduction in delivery of useful data is considered as congestion collapse).
> 
> 
>>> 
>>> Additional notes:
>>> 
>>> Sec 2.2:
>>> 
>>> Example 2:
> Tnx. I'd appreciate if the text could be clarified to reflect this more accurately. Maybe something along the lines(?):
> 
> To my understanding RFC4653 initializes DupThresh to half of the inflight size in the beginning of the Extended Limited Transmit. Then on each DupAck it adjusts (recalculates) DupThresh again such that ideally a cwnd worth of DupAcks are received before packet loss is declared (or reordering detected). So, if I am not incorrect, loss of a half of the inflight does not necessarily result in RTO recovery with TCP-NCR.
> 
>>> 
>>> Sec. 3.5:
>>> 
> SACKed sequences with DUpAcks indicate that those sequences were delivered but it does not tell when they were sent. The basic principle of slow start is to reliably determine the available network capacity during slow start. Therefore, slow start must ensure it uses only segments sent during the slow start to increase cwnd. Otherwise, a TCP sender may encounter exactly the problem of unnecessary retransmission envisioned in this example of RACK-TCP draft (and increase cwnd on not valid Acks).
> 
> RFC 6675 does re-specify DupAck with SACK option but it does not include the rule for slow start. Slow start is specified in RFC 5681. It is crystal clear in allowing increase in cwnd only on cumulative Acks, i.e., forbidding to increase cwnd on DupAcks (RFC 5681, Sec 3,
> page 6:
> 
> Maybe this example in the RACK-TLP draft was inspired by a incorrect implementation of SACK-based loss recovery?
> 
> FYI: when we were finalizing RFC 6675 I suggested including also an algorithm for RTO recovery with SACK in RFC 6675. The reason was exactly that it might be not easy to gather info from multiple documents and hence help the implementor to have all necessary advice in a single document. This unfortunately did not get realized though.
> 
> BR,
> 
> /Markku
> 
>>> 
>>> Best regards,
>>> 
>>> /Markku
>>> 
>>> 
>>> 
> -- 
> last-call mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/last-call

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 7 Dec 2020 12:09:23 -0800

>       As per RFC 6675: set cwnd=ssthresh=FlightSize/2=20/2=10

To account for your points, IMO clearly stating the existing CC RFC
interactions w/o mandating any C.C. actions are the best way to move
forward. Here are the text diff I proposed:

"Figure 1, above, illustrates  ...
 Notice a subtle interaction with existing congestion control actions
on event 7a. It essentially starts another new episode of congestion
due to the detection of lost retransmission. Per RFC5681 (section 4.3)
that loss in two successive windows of data, or the loss of a
retransmission, should be taken as two indications of congestion as a
principle. But RFC6675 that introduces the pipe concept does not
specify such a second window reduction. This document reminds RACK-TLP
implementation to carefully consider the new multiple congestion
episode cases in the corresponding congestion control."

and to emphasize in section 9.3 Interaction with congestion control

"9.3.  Interaction with congestion control

RACK-TLP intentionally decouples loss detection ... this appropriate.
As mentioned in Figure 1 caption, RFC5681 mandates a principle that
Loss in two successive windows of data, or the loss of a
retransmission, should be taken as two indications of congestion, and
therefore reacted separately. However implementation of RFC6675 pipe
algorithm may not directly account for this newly detected congestion
events properly. Therefore the documents reminds RACK-TLP
implementation to carefully consider these implications in its
corresponding congestion control.

..."

>   (half of> the original FlightSize of 10 segments)

sure will do

> reset cwnd and slow start when TCP detets loss of an entire flight?

I think we're going in loops. To move forward it'd help if you suggest
some text you like to change.

> reduction in delivery of useful data is considered as congestion

AFAIK you are saying: under extreme congestion shared by many short
flows, RACK-TLP can cause more packet losses because of the more
aggressive PTO timer. I agree and can add this to the "section 9.3".

What authors disagree is that RTO must be back-off on the first
instance if TLP is not acked. While your suggestion helps the
congestion case, it may also hurt the recovery in other cases when the
TLP is dropped due to light/burst/transient congestion. Arguing which
scenarios matter more subjectively is not productive.

So the question we need to look at is if RACK-TLP 2RTT-PTO + regular
RTO-backoff is going to cause major stability issues in extreme
congestion. My understanding based on my officemate Van Jacobson's
explanation is, as long as there's eventual exponential backoff, we'll
avoid the repeated shelling the network. As a matter of fact some
major TCP (!= Linux) implementation has implemented linear backoff of
first N RTOs before exp-backoff.

> inflight does not necessarily result in RTO recovery with TCP-NCR.
Could you suggest the text you'd like on NCR description.

> document. This unfortunately did not get realized though.

I am honestly lost quibbling these RFC6675 implementations and feeling
pedantic to these standards largely serve as guiding principles
instead of line-by-line code. Is cwnd one packet bigger or smaller in
this example making any different in advancing the Internet's
capability to do better loss detection. I do not think so.

At this point please suggest a text you like to change.

---

From: Markku Kojo <[email-masked]>
Date: Tue, 8 Dec 2020 20:06:43 +0200 (EET)

Hi Michael,

I am top posting, too (as this is a quite short answer to one point only).

Thanks for the suggestion. I was also thinking of using pipe among first 
options. Unfortunately, it is not that usable either, if you wish the 
outcome to be "halving again" as you  envisioned.

In the example scenario, RACK-TLP first declares segments P1-19 lost and 
segment P20 SACKed (my quick example marked segment P20 incorrectly lost, 
sorry). When the loss of the segment P1 is detected, P2&3 are marked SACKed.
This means that at this point pipe is empty, so it would result in 
resetting cwnd (= 1), which is safe and the same as when detecting the 
lost rexmit with RTO, but not what you are aiming for (halving again).

Cheers,

/Markku

---

From: Markku Kojo <[email-masked]>
Date: Tue, 8 Dec 2020 21:21:00 +0200 (EET)

Hi,

please see inline.

> specify such a second window reduction. This document reminds RACK-TLP

This is not quite correct characterization of RFC 6675.
RFC 6675 does not repeat all guidelines of RFC 5681. RFC 6675 DOES specify 
the two indications of congestion implicitly by clearly stating in the 
intro that it follows the guidelines set in RFC  5681.
And RFC 5681 articulates a set of MUSTs which ALL alternative loss 
recovery algorithms MUST follow in order to become RFCs.

(*) RFC 5681, Sec 4.3 (pp. 12-13):

  "While this document does not standardize any of the
   specific algorithms that may improve fast retransmit/fast recovery,
   ...
   That is, ...
   Loss in two successive windows of data, or the loss of a
   retransmission, should be taken as two indications of congestion and,
   therefore, cwnd (and ssthresh) MUST be lowered twice in this case."

> episode cases in the corresponding congestion control."

I am sorry to say that this is very fuzzy and leaves an implementer 
all alone what to do.

More inportantly, AFAIK we have no discussion nor consensus in the IETF 
on what is the correct CC action when lost rexmit is detected. 
Should one reset cwnd like current CCs do, or would "halving again" be 
fine, or something else? IMHO this requires experimentation to decide.

I assumed there is an implementation of RACK-TLP with corresponding CC 
actions and experimental results to devise what are the implications of 
the selected CC action(s) with various levels of congestion. But it seems 
we do not have such implementation nor experimental evidence? Am I wrong?

I sincerely apologize that this got raised this late in the prosess and 
I know how irritating it may be. I like the idea of RACK-TLP and by no 
means my intention is not to hold up the process but the lack of evidence 
makes me quite concerned. In particular, when this document says that 
the current loss detection SHOULD be replaced in all implementations with 
RACK-TLP.

> some text you like to change.

Unfortunately I do not have an immediate solution. I assumed there is an 
implementation and you could enlighten what was the solution there and 
experimental results showing how it seems to work. If I have understood 
it correctly there is a solution/implementation and experience of RACK-TLP 
that works together with PRR but no solution/implementation nor experience 
how it works without PRR. Am I correct?

Given my understanding of PRR, the problem of an entire flight being 
dropped is quite nicely solved with it. However, implementing RACK-TLP 
without PRR begs for a solution. Here is my current understanding:

(1) RACK part is what can be called to "replace" DupAck counting
(2) TLP part is effectively a new retransmission timeout to detect
     a loss of an entire flight of data (i.e., it is only invoked
     when a tail of the current flight becomes lost which equals to
     the case of losing an entire flight since all segments before
     the tail loss will get cumulatively Acked and hence these segments
     are not anymore a part of the current flight at the time the loss
     is detected. And we an assume application limited TCP sender.
(3) Current CC principles require resetting cwnd in such a case
     and entering slow start (and so effectively does PRR though it is
     not explicitly stated in RFC 6937). Slow start avoids a big burst
     if the lost flight is big.
(4) RACK-TLP would possibly like to allow that cwnd is set to a half
     of the lost flight and not to slow start. This means that the bigger
     the lost flight is, the bigger is the burst that gets transmitted at
     the beginning of the recovery which is bad. So, this approach would
     need at least a rule/advice for burst avoidance (slow start, pacing,
     ...).
(5) When the lost flight is small (<= 4 segments) there is no difference
     in recovery efficiency between (3) and (4). If the lost flight is
     > 5 segments, then (4) takes less RTTs to complete the recovery
     but generates a burst. Note, even if pacing over one SRTT would be
     used, it is still a burst.

Now, it would be useful to have experimental data to know how the size of 
the lost flight is distributed. Is it typically just a few segments as 
illustriated in the examples of the draft or is it often larger?

My advice would be to add a rule that cwnd MUST be reset to 1 MSS 
and the sender MUST enter slow start if TLP detects loss of an entire 
flight. This would be safe. Otherwise, without experimental evidence from 
a wide range of different network conditions and workloads it feels 
unsafe to allow more aggressive approach.

> aggressive PTO timer. I agree and can add this to the "section 9.3".

No, that was not what I meant. This happens simply when enough flows are 
competing on the same bottleneck. It may be long flows or maybe 
applications with more or less continuous request-reply exhanges.
E.g., if the bottleneck bit rate is 1 Mbits/s, RTT is 500msecs and PMTU 
is 1500B, then a bit over 40 simultaneous flows would mean that the 
bottleneck becomes fully utilized with the equal share of roughly 1 MSS 
per flow. With a quite typical bottleneck buffer size roughly equaling to 
BDP, about 80+ flows would fill up the buffer as well and increase
RTT >= 1 secs.  A buffer bloated bottleneck buffer would mean even larger 
RTT and allow more flows sharing the bottleneck without loss.

If the number of TCP flows is > 90 (or >> 90) RACK-TLP would ends up 
(almost) always unnecessarily retransmitting each new segment once (PTO 
being 1 sec or around 1 sec). RTO back-off after a few rounds saves each 
TCP sender from additional unnecessary rexmits but still ~ 50% of the 
deivered packets are not making any useful progress.

If the bottleneck buffer is small, it would result in more losses as you 
suggest but would not be that much of problem because majority of the 
unnecessary rexmits would not get delivered over the bottleneck link. 
Instead, they wuold get dropped at the bottleneck. Unnecessary rexmits 
would just create extra load on the poth before the bottleneck (which of 
course is not a non-problem either).

> scenarios matter more subjectively is not productive.

When RACK-TLP is required to be implemented by all TCP stacks, it 
is extremely important that it always works in a safe way and congestion 
is always the primary concern to get properly handled. Without arguing 
more, I just want to point out that TCP must work reasonably for all 
Internet users. That means it must not generate a situation where even a 
small minority of the Internet users often or almost always encounter 
severe problems with their connectivity.

For example, also users in developing countries where possibly an entire 
village shares just one mobile (cellular, maybe 3G possibly only GPRS) 
connection for their Internet access and pays per amount of data should 
get reasonable TCP behavior. In such a case, I cannot agree with 
engineering that results in almost always getting only half of the 
already scarce bandwidth whle paying double price fot the useful data.

But thinking about a way forward. Karn's algorithm would require backing 
off PTO even if you get Ack of PTO. Relaxing this does not sound me that 
bad at all, because there is often a pause before TLP and a TCP sender 
gets feedback, so apparently conditions are not that bad and loss 
recovery gets triggered (hopefully in slow start).
If TLP is not acked, RTO is needed and recovery is completed in using RTO 
recovery in slow start. Now, if the RTO recovery is successful (no 
losses during RTO recovery), it should be quite likely that the TCP 
sender is able to successfully send also one new segment, because it 
enters CA when it is sending at the half of the previous rate. So, once 
you get Ack for the new segment, the TLP back off can be removed and it 
is unlikely that the TLP back off slowed down next loss detection.
On the other hand, if the RTO recovery after an unsuccessful TLP is not 
successful (more losses are detected), it is quite likely that congestion 
has not been resolved. So, it is important to be conservative and have a 
backed-off PTO (or even turn it off) to avoid (further) unnecessary 
rexmits. If PTO is not backed of, I'd envision PTO mainly failing to get 
an Ack in such a case and thereby it not being that useful.

This certainly would require experimental data in a heavily congested 
setting to really figure out the actual impact of different alternatives.

> avoid the repeated shelling the network.

Right. But the problem is that with TLP as specified we do not have full 
exponential back off. It lacks Karn's clamped retransmit backoff (as it 
is called in Van Jacobson's seminal paper) which requires keeping 
backed-off timer of a retransmitted segment for the next (new) segment and 
ensures that there is eventual exponential backoff.
Backing off just RTO is not enough, because "fixed" PTO for each new 
segment breaks this "back-off chain", that is, the exponential backoff 
is not continueed until there is evidence that congestion has been 
resolved (a cumulative Ack arrives for new data). But as I said, backing 
off RTO saves RACK-TLP from a full congestion collapse. Still, wasting 
50% of the available network capacity in certain usage scenarios does not 
sound acceptable for me.

> As a matter of fact some

But that's quite ok as long as there is eventual exponential backoff, 
including Karn's clamped retransmit backoff. Linear back-off in the 
beginning just makes resolving (heavy) congestion a bit slower.

> Could you suggest the text you'd like on NCR description.

I'm not an expert nor closely acquainted with NCR. There might be many 
different packet loss patterns that may affect the behavior. So, my 
advice is to simply drop the last sentence starting with "However ...", 
because it seems incorrect and replace the second but last sentence:

   To mitigate the issue, [RFC4653]
   adjusts DupThresh to half of the inflight size to tolerate the
   higher degree of reordering.

-->

   To mitigate the issue, TCP-NCR [RFC4653]
   increases the DupThresh from the current fixed value of three duplicate
   ACKs [RFC5681] to approximately a congestion window of data having left
   the network.

> capability to do better loss detection. I do not think so.

Sorry, I don't understand what was quibbling here. I believe I did not 
argue anything about cwnd size with this example at hand in Sec. 3.5.
My point is that it describes incorrect slow start behavior. Correctly 
implemented slow start does not have the problem illustriated in the 
example.

> At this point please suggest a text you like to change.

That is simple. Please remove the description of the behavior without 
RACK. Or correct it: the behavior would be exactly the same as with RACK, 
but the reason for not unnecessarily retransmitting third segment can be 
described to be different.
And, please correct also the description with RACK. With RACK, third 
segment does not get unnecessarly rexmitted for the reason already 
indicated and also because cwnd=1 MSS. And, no new segments are allowed 
either by cwnd=1.

Please remove also the last paragraph of the Sec. 3.5, it being also 
incorrect description of behavior.

BR,

/Markku

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 12 Dec 2020 13:04:24 -0500

Hi Markku,

Thanks for your many detailed and thoughtful comments. Please see
below in-line for our comments based on discussions among RACK-TLP
draft authors.

> we do not have such implementation nor experimental evidence? Am I wrong?

[RACK-TLP-team:]

We do have such implementations of RACK-TLP with corresponding CC
actions, and experimental results were reported in RACK-TLP
presentations and earlier revisions of the RACK-TLP draft. With
respect to implementations, there is the Linux TCP stack, which has
been using RACK-TLP as the default loss recovery algorithm since Linux
v4.18 in August 2018. The exact commit is:

  b38a51fec1c1 tcp: disable RFC6675 loss detection

With Linux, the default sender behavior since that commit has been
governed by CUBIC, PRR, and RACK-TLP.

RACK-TLP is used by all Google TCP traffic, including internal RPC
traffic and external YouTube and google.com traffic over the public
Internet. At Google we have years of high-traffic-volume experience
with RACK-TLP with three different CC schemes: CUBIC+PRR, BBRv1, and
BBRv2.

My understanding is that Netflix FreeBSD TCP traffic uses RACK-TLP,
and Microsoft Windows TCP traffic has also been using RACK-TLP.

> how it works without PRR. Am I correct?

[RACK-TLP-team:]

The implementation solutions and experiments were mentioned above. Our
team's experience is with RACK-TLP in combination with either
CUBIC+PRR, BBRv1, and BBRv2. We are not sure if there are other TCP
implementations that implement RACK-TLP without PRR.

> unsafe to allow more aggressive approach.

[RACK-TLP-team:]

Our team's position is that the RACK-TLP draft documents a loss
detection algorithm, and normative declarations about congestion
control behavior (like your "cwnd MUST be reset to 1 MSS", above)
should be in a separate document focused on congestion control.

> course is not a non-problem either).

[RACK-TLP-team:]

If we are correctly understanding the scenario you are outlining, the
PTO would likely not be 1 sec in this case. In this scenario, the RTT
is around 1sec, so the PTO would be around 2*srtt, which would be
around 2 secs. With a 1sec RTT, and even considering delayed ACKs, it
seems very unlikely that a 2 sec PTO would cause a significant number
of unnecessary TLP probes.

Perhaps your postulated PTO of 1 sec is coming from the pseudocode
path in TLP_calc_PTO() where, if SRTT is not available, the PTO is 1
second? However, this case where there is no SRTT only happens if (a)
the connection is *not* using TCP timestamps (enabled for most
connections on the public Internet, due to default-enabled support in
Linux/Android and iOS/MacOS), and (b) the connection has *never* had a
single RTT sample from a non-retransmitted packet. To meet both of
those conditions is quite rare in practice. Is this the path that is
causing your concerns about a PTO of 1 sec?

Furthermore, in cases like this where the number of flows is roughly
equal to or greater than the BDP of the path, Reno/[RFC5681] and CUBIC
will spend the majority of their time in loss recovery (the minimum
ssthresh from [RFC5681] being 2*SMSS). And the RACK-TLP ID specifies
that TLP probes are not sent while in loss recovery.

> 50% of the available network capacity in certain usage scenarios does not

[RACK-TLP-team:]

To address concerns related to Karn's algorithm, perhaps it would be
sufficient to specify that after a RACK-TLP sender sends a TLP probe
that is a retransmitted segment, it temporarily disables further TLP
retransmissions until the sender obtains a valid RTT sample (i.e. from
a non-retransmitted segment, or a segment with [RFC7323] TCP
timestamps)?

>    ACKs [RFC5681] to approximately a congestion window of data having left

[RACK-TLP-team:]

Sure, we will revise accordingly.

> indicated and also because cwnd=1 MSS. And, no new segments are allowed

[RACK-TLP-team:]

Thanks. Perhaps it makes sense to consider a slightly different RTO
scenario to illustrate the point about how RACK RTO behavior can avoid
spurious retransmissions.

Here is the proposed new text:

"3.5.  An Example of RACK-TLP in Action: RTO

   In addition to enhancing fast recovery, RACK improves the accuracy of
   RTO recovery by reducing spurious retransmissions.

   Without RACK, upon RTO timer expiration the sender marks all the
   unacknowledged segments lost.  This approach can lead to spurious
   retransmissions.  For example, consider a simple case where one
   segment was sent with an RTO of 1 second, and then the application
   writes more data, causing a second and third segment to be sent right
   before the RTO of the first segment expires.  Suppose none of the
   segments were lost.  Without RACK, if there is a spurious RTO then
   the sender marks all three segments as lost and retransmits the
   first segment. If the ACK for the original copy of the first segment
   arrives right after the spurious RTO retransmission, then the
   sender continues slow start and spuriously retransmits the second
   and third segments, since it (erroneously) presumed they are lost.

   With RACK, upon RTO timer expiration the only segment automatically
   marked lost is the first segment (since it was sent an RTO ago); for
   all the other segments RACK only marks the segment lost if at least
   one round trip has elapsed since the segment was transmitted.
   Consider the previous example scenario, this time with RACK.  With
   RACK, when the RTO expires the sender only marks the first segment as
   lost, and retransmits that segment.  The other two very recently sent
   segments are not marked lost, because they were sent less than one
   round trip ago and there were no ACKs providing evidence that they
   were lost. Upon receiving the ACK for the RTO retransmission the
   RACK sender would not yet retransmit the second or third segment. but
   rather would rearm the RTO timer and wait for a new RTO interval to elapse
   before marking the second or third segments as lost.
"

best regards,

neal

---

From: Markku Kojo <[email-masked]>
Date: Tue, 15 Dec 2020 20:56:50 +0200 (EET)

Hi Neal, all,

(my apologies for this message getting so long but I didn't want to drop 
any parts of it in case someone wants to get the entire discussion in one 
piece. Also, apologies for repeating some of my concerns inline below; 
Those who are clear with the problems may well opt skipping those parts.)

Thanks Neal for the reply. I believe we are slowly converging with many 
of my comments but two of the important issues still remain open. That is, 
where can an implementer find advice for correct congestion control 
actions with RACK-TLP, when:
(1) a loss of rexmitted segment is detected
(2) an entire flight of data gets dropped (and detected),
     that is, when there is no feedback available and a timeout
     is needed to detect the loss

I fully understand the intent of the RACK-TLP draft not to include 
congestion control actions in this document (or the intent to minimise 
such advise). However, when new ways to detect loss are introduced and 
published (as this document would do), IMHO it is imperative to have such 
congestion control actions readily available for an implementer. AFAIK 
we do not have in the RFC series correct actions available for these two 
cases.

I do not have a strong opinion whether the necessary actions should be in 
this document or in another document, as long as we have them. This is 
more or less a process decision, I think.

Please see more details inline.

I also investigated a bit more on RACK-TLP as specified in the draft and 
it seems that there is a problem with the pseudocode as currently 
presented (some sequence numbers seem to be off by one, resulting in 
wrong outcome in some/many? cases). Please see inline in the end of my 
reply.

> presentations and earlier revisions of the RACK-TLP draft. With

Yes, that was my understanding what is implemented. Given there is this 
implementation that reacts appropriately to congestion when RACK-TLP 
detects loss of a retransmitted segment, could you please illuminate the 
congestion control actions taken in the implementation with which there 
is experimental experience.

If I understood it correctly, the RACK-TLP in question is implemented 
with PRR. Let's assume another implementer wants to add RACK-TLP with PRR 
in the stack she/he is working on. I believe that the implementer would 
think that detecting a loss of a rexmit during fast recovery would start 
a new congestion episode in PRR (and that's what Yuchung also indicated 
in the new proposed text in his previeous reply shown above). With 
that advice and reading RFC 6937, an implementer would reinitialize PRR 
state for the new congestion episode. So how should the implementer 
calculate new values for ssthresh and RecoverFS? Should she/he use what 
is currently described in RFC 6937, Sec 3 (to keep discussion simple 
assume the implementer wants to have an IETF Standards Track comformant 
implementation, so use CongCtrlAlg() = FlightSize/2):

  ssthresh = CongCtrlAlg()  // Target cwnd after recovery
  prr_delivered = 0         // Total bytes delivered during recovery
  prr_out = 0               // Total bytes sent during recovery
  RecoverFS = snd.nxt-snd.una // FlightSize at the start of recovery

Or should one use something else? AFAIK depending on the loss pattern 
neither FlightSize nor "snd.nxt-snd.una" would give correct values for 
the intended behavior. Depending on the loss pattern, at the time 
of detecting a loss of rexmit FlighSize may be ~ 50% larger than in the 
beginning of fast recovery. This would result in increasing ssthresh 
(send rate), instead of decreasing it. It seems that something else in 
PRR algo would also need to be modified to get sndcnt correct but I 
cannot figure it out right now.

If one wants to implement RACK-TLP without PRR, that is, with RFC 6675, 
we have the same probelm with FlightSize in determining a new value for 
cwnd and ssthresh.

My point is that using what we currently have out there in the RFC 
series gives incorrect CC advice. And, having correct CC actions in this 
case is far from trivial so IMHO high quality standards specifications 
should not leave an implementer of RACK-TLP alone to figure it out.

In addition, if a new episode of fast recovery is started for CC purposes 
how would that play together with RACK-TLP loss detection? In Sec 7.1 the 
text says initialize (reset) TLP.end_seq and TLP.is_retrans when 
initiating fast recovery. But detecting a lost rexmit seemingly should 
not reinitialize TLP because PTO is disabled during fast recovery as per 
Sec 7.2. If CC actions are in another document and loss detection in this 
document, it would be of high importance to clarify when each of the 
fast recovery episodies (CC vs. loss detection) are supposed to end: is 
the end condition the same or different?

> and Microsoft Windows TCP traffic has also been using RACK-TLP.

And I wonder what would be the CC actions taken in each of these 
implementations when a loss of rexmited segment is detected?

> implementations that implement RACK-TLP without PRR.

So, in addition to the lack of experimental evidence using RACK-TLP 
without PRR we have a specification problem for which we have no answer:

The RACK-TLP text suggests that if PTO fires and TLP detects a loss, the 
TCP sender enters fast recovery. But

PTO = timeout that expires if there is no feedback (ACKs) from
       the network, i.e., entire flight of data is lost.

TLP = a single segment that is sent when timer expires. What is
       different compared to RTO is that PTO may expire faster
       and a different segment is (re)transmitted, allowing to
       detect more than one lost segment with a single Ack w/ SACK
       triggered by TLP (with regular RTO recovery only the next
       unacknowledged segment can reliably be considered lost)

>From congestion control point of view this means that PTO is no different 
from RTO. The congestion control principle as stated in Sec 2.2 of this 
document tells us that a loss of entire flight deserves a cautious 
response, i.e., reset cwnd and slow start.

When RACK-TLP is implemented with PRR we have appropriate CC actions 
specified in RFC 6937 that effectively reset cwnd and force the TCP sender 
to slow start, even though it tecnically enters fast recovery. This is 
safe.

However, if an implementer wants to implement RACK-TLP without PRR, 
she/he has to consult RFC 6675 which gives an incorrect CC response for 
this case, resulting in an unsafe implementation. The problem is not in 
the fast recovery algorithm in RFC 6675 because it was not intended to be 
initiated with TLP-kind of loss detection.

So the question is how to ensure an implementer has the necessary advice 
to implement correct congestion control actions for RACK-TLP once this 
document is published. The lack of correct advice becomes even more 
prominent as this document requires (all SACK capable) TCPs to implement 
RACK-TLP.

> should be in a separate document focused on congestion control.

I do not disagree with you what becomes to the role of congestion control 
in the RACK-TLP draft. It may well be in another document but IMHO such 
document must be available at the time this docment gets published.
How can we otherwise ensure correct implementations?

So this is more of a process question than critique towards the draft.

> of unnecessary TLP probes.

Well. Roughly the first initiated 40 flows will have SRTT ~ 0.5 sec. If 
next ~ 40 flows arrive within an RTT or so they will get RTT samples 
roughly between 0.5 - 1.x secs. The rest 80 - N hundreds flows will not 
get an RTT sample and will use initial/default PTO = 1 sec (unless using 
timestamps). The larger the buffer bloat the worse things get.
Once ~200 flows are competing over the bottleneck, RTT will be well 
beyond 2 secs and grows when more flows join, guaranteeing spurious TLP 
for each new segment (unless timestamps are in use).

> causing your concerns about a PTO of 1 sec?

Sure. Having timestamps helps a lot in case the bottleneck buffer is 
buffer bloated (though RTT sample will not include delayed Ack effect for 
rexmits that are immediately acked). But RACK-TLP does not require 
timestamps and there are a lot of TCP stacks out there in addition to 
Linux/Android and iOS/MacOS. And more are coming with IoT expansion, 
for example. In addition, it is quite suprising how old (mobile) devices 
many people are using, some of which are old enough and have not been 
eligible for an OS upgrade for many years.

But my quick example scenario was just to illustriate the problem, maybe 
typical in some areas where the only Internet access is over a 
geostationary satellite connection and people quite likely not using 
mobile/cellular devices. And, I selected a buffer-bloated scenario 
because that generates unnecessary rexmits which are a prerequisite for 
classical congestion collapse.

It may well be in some environments that these hundreds of competing 
flows are application limited (request-reply continuously) and get 
started over a longer period of time, each getting RTT sample(s) over a 
lightly loaded bottleneck and then becoming idle or sending data only 
sporadically. Then later they all become active more or less at the same 
time (maybe by a common trigger). Base RTT may then be much lower than 
500 ms as long as there is enough buffer bloat.

Furthermore, timestamps are not that helpful in case of a heavily 
congested small buffer bottleneck. Having approximately accurate SRTT 
does not alone help in reducing the sending rate below 1 segment/RTT in 
heavily congested scenarios. Exponential backoff of timer is a must.

I believe that either of us know what's really out there employing 
Internet for communication purposes. It does not matter whether it is a 
notable percentage or not as long there is a group of users who are 
encountering the envisioned problem.
It may very well be a small minority, but if this kind of a problem 
escalates for this group very often (maybe almost always), it means we 
are disabling use of Internet technology for them.

> ssthresh from [RFC5681] being 2*SMSS).

Sure there is surprisingly lot to improve with TCP when the window needs 
to be kept lower than 2 MSS or when the available share would be even 
less than 1 MSS/RTT. RFC 5681 would exponentially back-off on every RTO, 
leaving opportunity for other flows to make progress during the 
backed-off interval. Once the sender is lucky and gets an Ack for a new 
segment, min ssthresh of 2*MSS unfortunately allows it to accelerate to 
2*MSS/RTT and maybe more until RTO expires again. We cannot avoid some 
drops or unnecessary rexmits. Actually, in such a heavily congested 
scenario the combination of Karn's exponential back-off and slow start 
results in Multiplicative Increase Multiplicative Decrease (MIMD) 
behaviorwhich does not converge that well to fairness. But it avoids 
congestion collapse. It is kind a tradeoff between safe and stable 
behavior during heavy congestion and being able to ramp up fast when the 
congestion fades away. Even though the current behavior is not ideal 
for fairness, we should not make it even worse.

Currently only way to avoid spending most of the time in loss 
recovery seems to be ECN that would make a TCP sender to become rate 
controlled once cwnd hits the minimum cwnd of 1 MSS(*). But there is a 
lot to optimise also with ECN in this kind of scenario ...

(*) to avoid anyone repeating the misconception that minimum cwnd is
     2 MSS per equation (4) in RFC 5681: equation (4) sets ssthresh,
     not cwnd. RFC 3168 clearly states that cwnd must be halved on
     an arrival of ECE; this holds until cwnd = 1 MSS. After this,
     the sending rate of new segments is (RTO) timer controlled and
     the timer avlue exponentially backed-off if more ECEs arrive.
     At least two major OSs implement this incorrectly unless now
     already fixed.

> And the RACK-TLP ID specifies

Maybe I missed it but I couldn't find an end condition of RTO recovery 
for RACK-TLP purposes in the document. Regular RTO does not have such end 
condition because it is not needed. Congestion control and loss recovery 
are quite decoupled during  RTO recovery; slow start follows its own 
rules and ends when cwnd > ssthresh and loss recovery has its own 
rules to select segments to rexmit and just moves on sending new data 
once all lost segments have been retransmitted. Some implementations may 
have such variable/condition but it didn't help me.

Without timestamps, if TLP is not backed-off, a TLP is likely to be sent 
for each new segment (assuming certain conditions). Each not backed-off 
rexmit is potentially stealing an opportunity from another flow to get 
its segment being delivered, thereby contributing to unfair use of 
bottleneck resources.

> a non-retransmitted segment, or a segment with [RFC7323] TCP

As a basic rule: disabling TLP probes for the period of loss recovery as 
currently specified sounds like a correct approach provided that the end 
condition

  "until the sender obtains a valid RTT sample"

is added (this would be without timestamps, see some considerations with 
timestamps later below).

Then some justification and maybe some exeptions can also be specified 
(all these without timestamps):

If fast recovery is not entered via detecting loss using TLP, the 
sender does not need to wait for a valid RTT sample. It should be enough 
that RecoveryPoint is reached, i.e., use normal exit condition for fast 
recovery. However, I am not sure whether detecting a loss of a rexmit 
during fast recovery might change this?

I think it does not matter whether a TLP probe is a retransmitted segment
or not:

(a) if there is no ack after TLP probe and RTO expires it should be clear 
that TLP must be disabled until an ack for a new previously not rexmitted 
segment is received, regardless of the probe segment sent by TLP.
This is Karn's algorithm.

(b) if TLP triggers an Ack that is used to detect a loss of an entire 
flight of data, this may later during loss recovery result in an RTO 
which definitely requires disabling TLP as per the basic rule. There 
might be some exeptions when the basic rule is not necessary but it is 
hard for me to figure one right now, except

(c) if TLP rexmit detects and repairs a single loss, then CC is anyway 
invoked and Ack is received. If the arriving Ack is triggered by the TLP 
probe segment, it indicates that the conditions are likely not that bad 
(RTO not needed). So it might be ok, to rearm PTO for the next time 
without any backoff (without disbling it). However,

(d) if TLP was spurious rexmit, then PTO value obviously was too low 
(effective RTT may have increased suddenly and Acks got just delayed). 
This is possibly detected only one RTT later when the DupAck of the TLP 
rexmit arrives. TLP may have got rearmed by that time again with too 
low value, so it should be rearmed with doubled value to avoid another 
spurious TLP PTO (or disabled).

Then with timestamps. I think the simple rule for disabling TLP "until 
the sender obtains a valid RTT sample" is not necessarily giving the 
desired outcome in all cases. For example, if a TLP probe is send and 
then a late ack arrives for a segment sent before the TLP probe and gives 
a valid RTT sample with help of timestamps. This would allow a new TLP 
even though the sender has not got an Ack for a new segment sent after 
the TLP probe?

So, also with timestamps the requirement must be that the sender obtains 
a valid RTT sample for a segment sent after the TLP probe. And, at least, 
if RTO recovery is entered, then the Ack must be for a new segment.
Cannot figure out now all potential exceptions when a valid RTT sample 
with timestamps would allow re-enabling TLP earlier. Too complex 
algorithm with too many possible scenarios to investigate right now ;)

A single, simple rule would be advisable here. Otherwise, this may end up 
being too complex to get it right.

> Sure, we will revise accordingly.

Tnx.

>   and third segments, since it (erroneously) presumed they are lost.

That seems fine. Would it be useful to add an additional note saying that 
if F-RTO is in use, then further unnecessary retransmissions can be 
avoided?

>   before marking the second or third segments as lost.

This behavior with RACK-TLP does not quite match with my reading of 
the draft in all parts, so I have some questions.

First, wouldn't the PTO timer be first running for the first segment 
and when the sender sends the 2nd and 3rd segment it would rearm PTO for 
the 3rd segment? My understanding based on Sec 8 is that RTO cannot be 
armed until PTO fires? And when the PTO fires, the sender would 
retransmit the 3rd segment? Maybe I am missing something?

Anyways, an RTO timer may expire in this scenario (e.g., if there is no 
Ack for TLP).
Let's assume the RTO timer expires as described. When the first Ack after 
the rexmit of the 1st segment arrives, isn't that the late ACK for the 
original copy of the first segment, not the ACK for the RTO 
retransmission? Anyways, when this first Ack arrives after the RTO, the 
second and third segment are not marked as lost as described, which 
nicely prevents unnecessary retransmissions of those segments.
However, if the 2nd and 3rd segments were lost, wouldn't rearming RTO 
with exponentially backed-off timer value (2*RTO) make the recovery of 
the 2nd and 3rd segment very slow? After 2*RTO the timer expires and 
the 2nd segment is rexmitted and only after one additional RTT when the 
ack for the 2nd segment arrives, the sender rexmits the 3rd segment?
Wouldn't it be much more efficient to disable RACK loss detection during 
an RTO recovery and employ F-RTO? Maybe I am missing something again?

Then to the problem with the pseudocode I mentioned in the beginning of 
this message. If I am not incorrect it seems to me that setting a value 
to TLP.end_seq in TLP_send_probe() is off by one byte? Maybe something 
similar is wrong somewhere else too, didn't check. Please apologise me if 
I missed something.

Let's use an example with RACK-TLP recovery:

Send P1, ..., P9, P10
[P1,.., P9 delayed, P10 dropped]
/* I believe P1-9 delayed is not important here */

After 2*SRTT PTO expires
P10 rexmitted by TLP
  [ TLP_send_probe(): set TLP.end_seq = SND.NXT (= SEG.P11) ]

  [after TLP, the  application possibly provides more data to send]

Ack for original P1 arrives
(P11 sent, if more data by application)
Ack for original P2 arrives
(P12 sent, if more data by application)
...
Ack for original P9 arrives
(P19 sent, if more data by application)

Ack for P10 triggered by TLP rexmit of P10 arrives
  [TLP_process_ack(ACK):
   ...
   If ACK's ack. number > TLP.end_seq
    returns False
    since ACK's ack. number = SEQ.ACK = TLP.end_seq = SEG.P11]

   Loss of P10 is undetected and no reaction to congestion]

My understanding of draft text is that a loss of P10 is not concealed.
Please advise.

Best regards,

/Markku

---

From: Martin Duke <[email-masked]>
Date: Tue, 15 Dec 2020 15:53:58 -0800

Hi Markku,

Thanks for the comments. The authors will incorporate many of your
suggestions after the IESG review.

There's one thing I don't understand in your comments:

" That is,

(1) a loss of rexmitted segment is detected
(2) an entire flight of data gets dropped (and detected),
     that is, when there is no feedback available and a timeout
     is needed to detect the loss "

Section 9.3 is the discussion about CC, and is clear that the implementer
should use either 5681 or 6937.

You went through the 6937 case in detail.

If 5681, it's pretty clear to me that in (1) this is a new loss detection
after acknowledgment of new data, and therefore requires a second halving
of cwnd. For (2), the RTO timer is still operative so the RTO recovery
rules would still follow.

In other words, I am not seeing a case that requires new congestion control
concepts except as discussed in 9.3.

What am I missing?

---

From: Markku Kojo <[email-masked]>
Date: Wed, 16 Dec 2020 21:39:30 +0200 (EET)

Hi Martin,

On Tue, 15 Dec 2020, Martin Duke wrote:

> Section 9.3 is the discussion about CC, and is clear that the
> implementer should use either 5681 or 6937.

Just a cite nit: RFC 5681 provides basic CC concepts and some useful CC 
guidelines but given that RACK-TLP MUST implement SACK the algorithm in 
RFC 5681 is not that useful and an implementer quite likely follows 
mainly the algorithm in RFC 6675 (and not RFC 6937 at all if not 
implementing PRR).
And RFC 6675 is not mentioned in Sec 9.3, though it is listed in the 
Sec. 4 (Requirements).

> You went through the 6937 case in detail.

Yes, but without correct CC actions.

> If 5681, it's pretty clear to me that in (1) this is a new loss
> detection after acknowledgment of new data, and therefore requires a
> second halving of cwnd.

Hmm, not sure what you mean by "this is a new loss detection after 
acknowledgment of new data"?
But anyway, RFC 5681 gives the general principle to reduce cwnd and 
ssthresh twice if a retransmission is lost but IMHO (and I believe many 
who have designed new loss recovery and CC algorithms or implemented them 
agree) that it is hard to get things right if only congestion control 
principles are available and no algorithm.
That's why ALL mechanisms that we have include a quite detailed algorithm 
with all necessary variables and actions for loss recovery and/or CC 
purposes (and often also pseudocode). Like this document does for loss 
detection.

So the problem is that we do not have a detailed enough algorithm or 
rule that tells exactly what to do when a loss of rexmit is detected.
Even worse, the algorithms in RFC 5681 and RFC 6675 refer to 
equation (4) of RFC 5681 to reduce ssthresh and cwnd when a loss 
requiring a congestion control action is detected:

  (cwnd =) ssthresh = FlightSize / 2)

And RFC 5681 gives a warning not to halve cwnd in the equation but 
FlightSize.

That is, this equation is what an implementer intuitively would use 
when reading the relevant RFCs but it gives a wrong result for 
outstanding data when in fast recovery (when the sender is in 
congestion avoidance and the equation (4) is used to halve cwnd, it 
gives a correct result).
More precisely, during fast recovery FlightSize is inflated when new 
data is sent and reduced when segments are cumulatively Acked. 
What the outcome is depends on the loss pattern. In the worst case, 
FlightSize is signficantly larger than in the beginning of the fast 
recovery when FlightSize was (correctly) used to determine the halved 
value for cwnd and ssthresh, i.e., equation (4) may result in 
*increasing* cwnd upon detecting a loss of a rexmitted segment, instead 
of further halving it.

A clever implementer might have no problem to have it right with some 
thinking but I am afraid that there will be incorrect implementations 
with what is currently specified. Not all implementers have spent 
signicicant fraction of their career in solving TCP peculiarities.

> For (2), the RTO timer is still operative so
> the RTO recovery rules would still follow.

In short:
When with a non-RACK-TLP implementation timer (RTO) expires: cwnd=1 MSS, 
and slow start is entered.
When with a RACK_TLP implementation timer (PTO) expires, 
normal fast recovery is entered (unless implementing 
also PRR). So no RTO recovery as explicitly stated in Sec. 7.4.1.

This means that this document explicitly modifies standard TCP congestion 
control when there are no acks coming and the retransmission timer 
expires

from: RTO=SRTT+4*RTTVAR (RTO used for arming the timer)
       1. RTO timer expires
       2. cwnd=1 MSS; ssthresh=FlightSize/2; rexmit one segment
       3. Ack of rexmit sent in step 2 arrives
       4. cwnd = cwnd+1 MSS; send two segments
       ...

to:   PTO=min(2*SRTT,RTO) (PRO used for arming the timer)
       1. PTO times expires
       2. (cwnd=1 MSS); (re)xmit one segment
       3. Ack of (re)xmit sent in srep 2 arrives
       4. cwnd = ssthresh = FlightSize/2; send N=cwnd segments

For example, if FlightSize is 100 segments when timer expires,
congestion control is the same in steps 1-3, but in step 4 the 
current standard congestion control allows transmitting 2 segments, 
while RACK-TLP would allow blasting 50 segments.

Question is: what is the justification to modify standard TCP 
congestion control to use fast recovery instead of slow start for a 
case where timeout is needed to detect loss because there is no 
feedback and ack clock is lost? The draft does not give any 
justification. This clearly is in conflict with items (0) and (1) 
in BCP 133 (RFC 5033).

Furthermore, there is no implementation nor experimental experience 
evaluating this change. The implementation with experimental experience 
uses PRR (RFC 6937) which is an Experimental specification including a 
novel "trick" that directs PRR fast recovery to effectively use slow 
start in this case at hand.

> In other words, I am not seeing a case that requires new congestion
> control concepts except as discussed in 9.3.

See above. The change in standard congestion control for (2).
The draft intends not to change congestion control but effectively it 
does without any operational evidence.

What's also is missing and would be very useful:

- For (1), a hint for an implementer saying that because RACK-TLP is
   able to detect a loss of a rexmit unlike any other loss detection
   algorithm, the sender MUST react twice to congestion (and cite
   RFC 5681). And cite a document where necessary correct actions
   are described.

- For (1), advise that an implementer needs to keep track when it
   detects a loss of a retransmitted segment. Current algorithms
   in the draft detect a loss of retransmitted segment exactly in
   the same way as loss of any other segment. There seems to be
   nothing to track when a retransmission of a retransmitted segment
   takes place. Therefore, the algorithms should have additional
   actions to correctly track when such a loss is detected.

- For (1), discussion on how many times a loss of a retransmission
   of the same segment may occur and be detected. Seems that it
   may be possible to drop a rexmitted segment more than once and
   detect it also several times?  What are the implications?

- If previous is possible, then the algorithm possibly also
   may detect a loss of a new segment that was sent during fast
   recovery? This is also loss in two successive windows of data,
   and cwnd MUST be lowered twice. This discussion and necessary
   actions to track it are missing, if such scenario is possible.

> What am I missing?

Hope the above helps.

/Markku

>       >>>>> per RFC5681 sec 4.3
>       https://tools.ietf.org/html/rfc5681#section-4.3
>       >>     > 5 segments, then (4) takes less RTTs to complete
>       the recovery

---

From: Martin Duke <[email-masked]>
Date: Wed, 16 Dec 2020 12:46:12 -0800

Hi Markku,

There is a ton here, but I'll try to address the top points. Hopefully they
obviate the rest.

1.
[Markku]
"Hmm, not sure what you mean by "this is a new loss detection after
acknowledgment of new data"?
But anyway, RFC 5681 gives the general principle to reduce cwnd and
ssthresh twice if a retransmission is lost but IMHO (and I believe many
who have designed new loss recovery and CC algorithms or implemented them
agree) that it is hard to get things right if only congestion control
principles are available and no algorithm."

[Martin]
So 6675 Sec 5 is quite explicit that there is only one cwnd reduction per
fast recovery episode, which ends once new data has been acknowledged. By
definition, if a retransmission is lost it is because newer data has been
acknowledged, so it's a new recovery episode. Meanwhile, during the Fast
Recovery period the incoming acks implicitly remove data from the network
and therefore keep flightsize low.

We can continue to go around on our interpretation of these documents, but
fundamentally if there is ambiguity in 5681/6675 we should bis those RFCs
rather than expand the scope of RACK.

2.
[Markku]
" In short:
also PRR). So no RTO recovery as explicitly stated in Sec. 7.4.1."

[Martin]
There may be a misunderstanding here. PTO is not the same as RTO, and both
mechanisms exist! The loss response to a PTO is to send a probe; the RTO
response is as with conventional TCP. In Section 7.3:

"

After attempting to send a loss probe, regardless of whether a loss
   probe was sent, the sender MUST re-arm the RTO timer, not the PTO
   timer, if FlightSize is not zero.  This ensures RTO recovery remains
   the last resort if TLP fails.

"

So a pure RTO response exists in the case of persistent congestion that
causes losses of probes or their ACKs.

Martin

On Wed, Dec 16, 2020 at 11:39 AM Markku Kojo <[email-masked]> wrote:

> <snipping the rest>

---

From: Markku Kojo <[email-masked]>
Date: Thu, 17 Dec 2020 02:11:06 +0200 (EET)

Hi Martin,

See inline.

On Wed, 16 Dec 2020, Martin Duke wrote:

> Hi Markku,
> 
> There is a ton here, but I'll try to address the top points. Hopefully
> they obviate the rest.

Sorry for being verbose. I tried to be clear but you actually removed my 
key issues/questions ;)

> So 6675 Sec 5 is quite explicit that there is only one cwnd reduction
> per fast recovery episode, which ends once new data has been
> acknowledged.

To be more precise: fast recovery ends when the current window becomes 
cumulatively acknowledged, that is,

(4.1) RecoveryPoint (= HighData at the beginning) becomes acknowledged

I believe we agree and you meant this although new data below 
RecoveryPoint may become cumulatively acknowledged already earlier 
during the fast recovery. Reno loss recovery in RFC 5681 ends, when 
(any) new data has been acknowledged.

> By definition, if a retransmission is lost it is because
> newer data has been acknowledged, so it's a new recovery episode.

Not sure where you have this definition? Newer than what are you 
referring to?

But, yes, if a retransmission is lost with RFC 6675 algorithm, 
it requires RTO to be detected and definitely starts a new recovery 
episode. That is, a new recovery episode is enforced by step (1.a) of 
NextSeg () which prevents retransmission if a segment that has already 
been retransmitted. If RACK-TLP is used for detecting loss with RFC 6675 
things get different in many ways, because it may detect loss of a 
retransmission. It would pretty much require an entire redesign 
of the algorith. For example, calculation of pipe does not consider 
segments that have been retransmitted more than once.

> Meanwhile, during the Fast Recovery period the incoming acks implicitly
> remove data from the network and therefore keep flightsize low.

Incorrect. FlightSize != pipe. Only cumulative acks remove data from 
FlightSize and new data transmitted during fast recovery inflate 
FlightSize. How FlightSize evolves depends on loss pattern as I said. 
It is also possible that FlightSize is low, it may err in both 
directions. A simple example can be used as a proof for the case where 
cwnd increases if a loss of retransmission is detected and repaired:

RFC 6675 recovery with RACK-TLP loss detection:
(contains some inaccuracies because it has not been defined how
lost rexmits are calculated into pipe)

cwnd=20; packets P1,...,P20 in flight = current window of data
[P1 dropped and rexmit of P1 will also be dropped]

DupAck w/SACK for P2 arrives
[loss of P1 detected after one RTT from original xmit of P1]
[cwnd=ssthresh=10]
P1 is rexmitted (and it logically starts next window of data)

DupAcks w/ SACK for original P3..11 arrive
DupAck w/ SACK for original P12 arrives
[cwnd-pipe = 10-9 >=1]
send P21
DupAck w/SACK for P13 arrives
send P22
...
DupAck w/SACK for P20 arrives
send P29
[FlightSize=29]

(Ack for rexmit of P1 would arrive here unless it got dropped)

DupAck w/SACK for P21 arrives
[loss of rexmit P1 detected after one RTT from rexmit of P1]

SET cwnd = ssthresh = FlightSize/2= 29/2 = 14,5

CWND INCREASES when it should be at most 5 after halving it twice!!!

> We can continue to go around on our interpretation of these documents,
> but fundamentally if there is ambiguity in 5681/6675 we should bis
> those RFCs rather than expand the scope of RACK.

As I said earlier, I am not opposing bis, though 5681bis wuold not 
be needed, I think.

But let me repeat: if we publish RACK-TLP now without necessary warnings 
or with a correct congesion control algorithm someone will try to 
implement RACK-TLP with RFC 6675 and it will be a total mesh. The 
behavior will be unpredictable and quite likely unsafe congestion 
control behavior.

> There may be a misunderstanding here. PTO is not the same as RTO, and
> both mechanisms exist! The loss response to a PTO is to send a probe;
> the RTO response is as with conventional TCP. In Section 7.3:

No, I don't think I misunderstood. If you call timeout with 
another name, it is still timeout. And congestion control does not 
consider which segments to send (SND.UNA vs. probe w/ higher sequence 
number), only how much is sent.

You ignored my major point where I decoupled congestion control from loss 
detection and loss recovery and compared RFC 5681 behavior to RACK-TLP 
behavior in exactly the same scenario where an entire flight is lost and 
timer expires.

Please comment why congestion control behavior is allowed to be radically 
different in these two implementations?

RFC 5681 & RFC 6298 timeout:

        RTO=SRTT+4*RTTVAR (RTO used for arming the timer)
       1. RTO timer expires
       2. cwnd=1 MSS; ssthresh=FlightSize/2; rexmit one segment
       3. Ack of rexmit sent in step 2 arrives
       4. cwnd = cwnd+1 MSS; send two segments
       ...

RACK-TLP timeout:

        PTO=min(2*SRTT,RTO) (PTO used for arming the timer)
       1. PTO times expires
       2. (cwnd=1 MSS); (re)xmit one segment
       3. Ack of (re)xmit sent in srep 2 arrives
       4. cwnd = ssthresh = FlightSize/2; send N=cwnd segments

If FlightSize is 100 segments when timer expires, congestion control is 
the same in steps 1-3, but in step 4 the standard congestion control 
allows transmitting 2 segments, while RACK-TLP would allow 
blasting 50 segments.

>    the last resort if TLP fails.

This does not prevent the above RACK-TLP behavior from getting realized.

> causes losses of probes or their ACKs.

Yes, RTO response exists BUT only after RACK-TLP at least once blasts the 
network. It may well be that with smaller windows RACK-TLP is successful 
during its TLP initiated overly aggressive "fast recovery" and never 
enters RTO recovery because it may detect and repair also loss of 
rexmits. That is, it continues at too high rate even if lost rexmits 
indicate that congestion persists in successive windows of data. And 
worse, it is successful because it pushes away other compatible TCP 
flows by being too aggressive and unfair.

Even a single shot burst every time there is significant loss 
event is not acceptable, not to mention continuous aggressiveness, and 
this is exactly what RFC 2914 and RFC 5033 explicitly address and warn 
about.

Are we ignoring these BCPs that have IETF consensus?

And the other important question I'd like to have an answer:

What is the justification to modify standard TCP congestion control to 
use fast recovery instead of slow start for a case where timeout is 
needed to detect the packet losses because there is no feedback and ack 
clock is lost? RACK-TLP explicitly instructs to do so in Sec. 7.4.1.

As I noted: based on what is written in the draft it does not intend to 
change congestion control but effectively it does.

/Markku

---

From: Martin Duke <[email-masked]>
Date: Wed, 16 Dec 2020 18:55:03 -0800

(1) Flightsize: in RFC 6675 <https://tools.ietf.org/html/rfc6675>. Section
5, Step 4.2:

       (4.2) ssthresh = cwnd = (FlightSize / 2)

             The congestion window (cwnd) and slow start threshold
             (ssthresh) are reduced to half of FlightSize per [RFC5681
<https://tools.ietf.org/html/rfc5681>].
             Additionally, note that [RFC5681
<https://tools.ietf.org/html/rfc5681>] requires that any
             *segments sent as part of the Limited Transmit mechanism not
             be counted in FlightSize *for the purpose of the above
             equation.

IIUC the segments P21..P29 in your example were sent because of Limited
Transmit, and so don't count. The flightsize for the purposes of (4.2) is
therefore 20 after both losses, and the cwnd does not go up on the second
loss.

(2)
" Even a single shot burst every time there is significant loss
event is not acceptable, not to mention continuous aggressiveness, and
this is exactly what RFC 2914 and RFC 5033 explicitly address and warn
about."

"Significant loss event" is the key phrase here. The intent of TLP/PTO is
to equalize the treatment of a small packet loss whether it happened in the
middle of a burst or the end. Why should an isolated loss be treated
differently based on its position in the burst? This is just a logical
extension of fast retransmit, which also modified the RTO paradigm. The
working group consensus is that this is a feature, not a bug; you're
welcome to feel otherwise but I suspect you're in the rough here.

Regards
Martin

---

From: Martin Duke <[email-masked]>
Date: Wed, 16 Dec 2020 23:26:59 -0800

I spent a little longer looking at the specs more carefully, and I
explained (1) incorrectly in my last two messages. P21..29 are not Limited
Transmit packets.

However, unless I'm missing something else, 6675 is clear that the recovery
period does not end until the cumulative ack advances, meaning that
detecting the lost retransmission of P1 does not trigger another MD
directly.

Thanks for this exercise! It's refreshed my memory of these details after
working on slightly different QUIC algorithms a long time.

On Wed, Dec 16, 2020, 18:55 Martin Duke <[email-masked]> wrote:

> (1) Flightsize: in RFC 6675 <https://tools.ietf.org/html/rfc6675>.
> Section 5, Step 4.2:
>
>              (ssthresh) are reduced to half of FlightSize per [RFC5681 <https://tools.ietf.org/html/rfc5681>].
>              Additionally, note that [RFC5681 <https://tools.ietf.org/html/rfc5681>] requires that any

---

From: Markku Kojo <[email-masked]>
Date: Thu, 17 Dec 2020 10:17:43 +0200 (EET)

Hi Martin,

On Wed, 16 Dec 2020, Martin Duke wrote:

> (1) Flightsize: in RFC 6675. Section 5, Step 4.2:
>
>              (ssthresh) are reduced to half of FlightSize per [RFC5681].
>              Additionally, note that [RFC5681] requires that any
>              segments sent as part of the Limited Transmit mechanism not
>              be counted in FlightSize for the purpose of the above
>              equation.
> 
> IIUC the segments P21..P29 in your example were sent because of Limited Transmit,
> and so don't count. The flightsize for the purposes of (4.2) is therefore 20 after
> both losses, and the cwnd does not go up on the second loss.

Incorrect. (And you corrected this in your later reply)

> "Significant loss event" is the key phrase here. The intent of TLP/PTO is to
> equalize the treatment of a small packet loss whether it happened in the middle of
> a burst or the end. Why should an isolated loss be treated differently based on
> its position in the burst? This is just a logical extension of fast retransmit,
> which also modified the RTO paradigm. The working group consensus is that this is
> a feature, not a bug; you're welcome to feel otherwise but I suspect you're in the
> rough here.

An isolated loss in the end of flight and repairing it is handled 
as a separate case in the draft. It is easy to have a less conservative 
congestion control response for it, and I would be fine with that.

The problem we are discussing is the "significant loss event" where 
even a huge number of packets are lost, potentially indicating severe 
congestion. The draft accidentally handles it the same as a small packet 
loss. And that is wrong.

Why draft handles it accidentally wrong: because RACK-TLP has never 
been implemented with the standard congestion control (RFC 6675) but 
with PRR and the experimental evidence presented to the working group was 
based on the implementation with PRR. One may say that the WG was 
*accidentally* misguided.

And as I have stated many times: PRR modifies RFC 6675 congestion control 
and includes necessary actions to handle a significant loss event 
properly. But current standards track congestion control does not. And 
this draft explicitly modifies standards track congestion control but 
conceals this significant modification in Sec. 9.3 when discussing the 
interaction with congestion control.

Best regards,

/Markku

>       >       > Hi Markku,
>       >       >
>       >       > Thanks for the comments. The authors will incorporate
>       >       many of your
>       >       > suggestions after the IESG review.
>       >       >
>       >       > There's one thing I don't understand in your comments:
>       >       >
>       >       > " That is,
>       >       > where can an implementer find advice for correct
>       >       congestion control
>       >       > actions with RACK-TLP, when:
>       >       >
>       >       > (1) a loss of rexmitted segment is detected
>       >       > (2) an entire flight of data gets dropped (and detected),
>       >       >      that is, when there is no feedback available and a
>       >       timeout
>       >       >      is needed to detect the loss "
>       >       >
>       >       > Section 9.3 is the discussion about CC, and is clear that
>       >       the
>       >       > implementer should use either 5681 or 6937.
>       >
>       >       > You went through the 6937 case in detail.
>       >
>       >       > If 5681, it's pretty clear to me that in (1) this is a
>       >       new loss
>       >       > detection after acknowledgment of new data, and therefore
>       >       requires a
>       >       > second halving of cwnd.
>       >
>       >       > For (2), the RTO timer is still operative so
>       >       > the RTO recovery rules would still follow.
>       >
>       >       In short:
>       >       > In other words, I am not seeing a case that requires new
>       >       congestion
>       >       > control concepts except as discussed in 9.3.
>       >
>       >       > What am I missing?

---

From: Markku Kojo <[email-masked]>
Date: Thu, 17 Dec 2020 10:46:53 +0200 (EET)

Hi,

On Wed, 16 Dec 2020, Martin Duke wrote:

> I spent a little longer looking at the specs more carefully, and I explained (1)
> incorrectly in my last two messages. P21..29 are not Limited Transmit packets. 

Correct. Just normal the rule that allows sending new data during fast 
recovery.

> However, unless I'm missing something else, 6675 is clear that the recovery period
> does not end until the cumulative ack advances, meaning that detecting the lost
> retransmission of P1 does not trigger another MD directly.

As I have said earlier, RFC 6675 does not repeat all congestion control 
principles from RFC 5681. It definitely honors the CC principle that
requires to treat a loss of a retransmission as a new congestion 
indication and another MD. I believe I am obligated to know this as a 
co-author of RFC 6675. ;)

RFC 6675 explicitly indicates that it follows RFC 5681 by stating in the 
abstract:

" ... conforms to the spirit of the current congestion control
  specification (RFC 5681 ..."

And in the intro:

   "The algorithm specified in this document is a straightforward
    SACK-based loss recovery strategy that follows the  guidelines
    set in [RFC5681] ..."

I don't think there is anything unclear in this.

RFC 6675 and all other standard congestion controls (RFC 5581 and RFC 
6582) handle a loss of a retransmission by "enforcing" RTO to detect it. 
And RTO guarantees MD. RACK-TLP changes the loss detection in this case 
and therefore the standard congestion control algorithms do not have 
actions to handle it corrrectly. That is the point.

BR,

/Markku

> Thanks for this exercise! It's refreshed my memory of these details after working
> on slightly different QUIC algorithms a long time.
> 
> equalize the treatment of a small packet loss whether it happened in the
> working group consensus is that this is a feature, not a bug; you're welcome
> to feel otherwise but I suspect you're in the rough here.
> 
> Regards
> Martin
> 
> 
>       > who have designed new loss recovery and CC algorithms or
>       implemented
>       > them
>       > When with a non-RACK-TLP implementation timer (RTO) expires:
>       cwnd=1
>       > MSS,
>       > On Wed, Dec 16, 2020 at 11:39 AM Markku Kojo
>       <[email-masked]>
>       > wrote:
>       >       Hi Martin,
>       >
>       >       guidelines but given that RACK-TLP MUST implement SACK
>       the
>       >       algorithm in
>       >       RFC 5681 is not that useful and an implementer quite
>       likely
>       >       follows
>       >       mainly the algorithm in RFC 6675 (and not RFC 6937 at
>       all
>       >       if not
>       >       implementing PRR).
>       >       But anyway, RFC 5681 gives the general principle to
>       reduce
>       >       cwnd and
>       >       ssthresh twice if a retransmission is lost but IMHO (and
>       I
>       >       believe many
>       >       with all necessary variables and actions for loss
>       recovery
>       >       and/or CC
>       >       rule that tells exactly what to do when a loss of rexmit
>       is
>       >       detected.
>       >       Even worse, the algorithms in RFC 5681 and RFC 6675
>       refer
>       >       to
>       >       equation (4) of RFC 5681 to reduce ssthresh and cwnd
>       when a
>       >       loss
>       >       That is, this equation is what an implementer
>       intuitively
>       >       would use
>       >       when reading the relevant RFCs but it gives a wrong
>       result
>       >       for
>       >       outstanding data when in fast recovery (when the sender
>       is
>       >       in
>       >       congestion avoidance and the equation (4) is used to
>       halve
>       >       cwnd, it
>       >       gives a correct result).
>       >       More precisely, during fast recovery FlightSize is
>       inflated
>       >       when new
>       >       FlightSize is signficantly larger than in the beginning
>       of
>       >       the fast
>       >       recovery when FlightSize was (correctly) used to
>       determine
>       >       the halved
>       >       value for cwnd and ssthresh, i.e., equation (4) may
>       result
>       >       in
>       >       A clever implementer might have no problem to have it
>       right
>       >       with some
>       >       with what is currently specified. Not all implementers
>       have
>       >       spent
>       >       also PRR). So no RTO recovery as explicitly stated in
>       Sec.
>       >       7.4.1.
>       >
>       >       This means that this document explicitly modifies
>       standard
>       >       TCP congestion
>       >       congestion control is the same in steps 1-3, but in step
>       4
>       >       the
>       >       current standard congestion control allows transmitting
>       2
>       >       segments,
>       >       Question is: what is the justification to modify
>       standard
>       >       TCP
>       >       case where timeout is needed to detect loss because
>       there
>       >       is no
>       >       justification. This clearly is in conflict with items
>       (0)
>       >       and (1)
>       >       in BCP 133 (RFC 5033).
>       >
>       >       uses PRR (RFC 6937) which is an Experimental
>       specification
>       >       including a
>       >       novel "trick" that directs PRR fast recovery to
>       effectively
>       >       use slow
>       >          able to detect a loss of a rexmit unlike any other
>       loss
>       >       detection
>       >          RFC 5681). And cite a document where necessary
>       correct
>       >       actions
>       >          are described.
>       >
>       >       - For (1), advise that an implementer needs to keep
>       track
>       >       when it
>       >          the same way as loss of any other segment. There
>       seems
>       >       to be
>       >          may detect a loss of a new segment that was sent
>       during
>       >       fast
>       >          recovery? This is also loss in two successive windows
>       of
>       >       data,

---

From: Markku Kojo <[email-masked]>
Date: Thu, 17 Dec 2020 10:56:38 +0200 (EET)

Hi,

Just one clarification inline.

On Thu, 17 Dec 2020, Markku Kojo wrote:

>> Transmit,
>> middle of
> An isolated loss in the end of flight and repairing it is handled as a 
> separate case in the draft. It is easy to have a less conservative congestion 
> control response for it, and I would be fine with that.
>
> The problem we are discussing is the "significant loss event" where even a 
> huge number of packets are lost, potentially indicating severe congestion. 
> The draft accidentally handles it the same as a small packet loss. And that 
> is wrong.
>
> Why draft handles it accidentally wrong: because RACK-TLP has never been 
> implemented with the standard congestion control (RFC 6675) but with PRR and 
> the experimental evidence presented to the working group was based on the 
> implementation with PRR. One may say that the WG was *accidentally* 
> misguided.
>
> And as I have stated many times: PRR modifies RFC 6675 congestion control and 
> includes necessary actions to handle a significant loss event properly. But 
> current standards track congestion control does not.

To clarify: current standards track congestion control does not 
handle such a significant loss event properly *if* it is detected by TLP 
and would initiate fast recovery as specified in this draft.

/Markku

> And this draft 
> explicitly modifies standards track congestion control but conceals this 
> significant modification in Sec. 9.3 when discussing the interaction with 
> congestion control.
>
> Best regards,
>
> /Markku
>
>> Regards
>> Martin
>> 
>> 
>>       >       implemented them

---

From: Martin Duke <[email-masked]>
Date: Thu, 17 Dec 2020 07:30:15 -0800

Hi Markku,

Thanks, now I understand your objections.

Martin

On Thu, Dec 17, 2020 at 12:46 AM Markku Kojo <[email-masked]> wrote:

> explained (1)
> welcome
> >       >       >      that is, when there is no feedback available and
> >       a
> >       >       timeout
> >       >       > Section 9.3 is the discussion about CC, and is clear
> >       that
> >       >       the
> >       >       > detection after acknowledgment of new data, and
> >       therefore
> >       >       requires a
> >       >       > second halving of cwnd.
> >       >
> >       >       > In other words, I am not seeing a case that requires
> >       new
> >       >       congestion

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 17 Dec 2020 18:51:46 +0000

I agree that we should have a note in this RFC about congestion control action upon detecting lost retransmission(s).

From: tcpm <[email-masked]> On Behalf Of Martin Duke
Sent: Thursday, December 17, 2020 7:30 AM
To: Markku Kojo <[email-masked]>
Cc: [email-masked] Extensions <[email-masked]>; [email-masked]; Michael Tuexen <[email-masked]>; [email-masked]; Last Call <[email-masked]>; tcpm-chairs <[email-masked]>
Subject: [EXTERNAL] Re: [tcpm] Last Call: <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to Proposed Standard

Hi Markku,

Martin

On Thu, Dec 17, 2020 at 12:46 AM Markku Kojo <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

And in the intro:

BR,

/Markku

> On Wed, Dec 16, 2020, 18:55 Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
> On Wed, Dec 16, 2020 at 4:11 PM Markku Kojo <[email-masked]<mailto:[email-masked]>> wrote:
>       Hi Martin,
>
>       See inline.
>
>       <[email-masked]<mailto:[email-masked]>>
>       > wrote:
>       >       Hi Martin,
>       >

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 17 Dec 2020 11:35:59 -0800

How about

RACK-TLP intentionally decouples loss detection ...
As mentioned in Figure 1 caption, RFC5681 mandates a principle that
Loss in two successive windows of data, or the loss of a
retransmission, should be taken as two indications of congestion, and
therefore reacted separately. However implementation of RFC6675 pipe
algorithm may not directly account for this newly detected congestion
events properly. PRR [RFCxxxx] is RECOMMENDED for the specific
congestion control actions taken upon the losses detected by RACK-TLP"

To Makku's request for "what's the justification to enter fast recovery".
Consider this example w/o RACK-TLP

T0: Send 100 segments but application-limited. All are lost.
T-2RTT: app writes so another 3 segments are sent. Made to the destination
and triggered 3 DUPACKs
T-3RTT: 3 DUPACK arrives. start fast recovery and subsequent cc reactions
to burst ~50 packets with Reno

In this case any ACK occured before RTO is (generally) considered
clock-acked, and how I understand Van's initial design.  This behavior
existed decades before RACK-TLP. RACK-TLP essentially changes the
"3-segments by app" to "1-segment by tcp".

On Thu, Dec 17, 2020 at 10:52 AM Praveen Balasubramanian <
[email-masked]> wrote:

> I agree that we should have a note in this RFC about congestion control
> action upon detecting lost retransmission(s).
>
>
>
> *From:* tcpm <[email-masked]> *On Behalf Of * Martin Duke
> *Sent:* Thursday, December 17, 2020 7:30 AM
> *To:* Markku Kojo <[email-masked]>
> *Cc:* [email-masked] Extensions <[email-masked]>;
> [email-masked]; Michael Tuexen <[email-masked]>;
> [email-masked]; Last Call <[email-masked]>;
> tcpm-chairs <[email-masked]>
> *Subject:* [EXTERNAL] Re: [tcpm] Last Call:
> <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to
> Proposed Standard
>
>
>
> Hi Markku,
>
>
>

---

From: Martin Duke <[email-masked]>
Date: Thu, 17 Dec 2020 13:58:59 -0800

This is good, thanks.

I might also add the sentence "In the absence of PRR, when TCP RACK detects
a lost retransmission it MUST trigger a second congestion response" or
something to that effect. What do you think?

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 17 Dec 2020 14:54:59 -0800

On Thu, Dec 17, 2020 at 1:59 PM Martin Duke <[email-masked]> wrote:

> This is good, thanks.
>
> I might also add the sentence "In the absence of PRR, when TCP RACK
> detects a lost retransmission it MUST trigger a second congestion response"
> or something to that effect. What do you think?
>
Sure I'd revise a bit to be even more detailed and consistent with the
principle in RFC5681

"In the absence of PRR, when RACK-TLP detects a lost retransmission the
congestion control MUST trigger an additional congestion response per the
aforementioned principle in RFC5681.
If multiple retransmissions were lost in a window, the congestion control
specified in RFC5681 only reacts once per window, but some other congestion
control may want to respond multiple times differently. The congestion
control implementer is advised to carefully consider this subtle situation."

>> T-2RTT: app writes so another 3 segments are sent. Made to the
>> destination and triggered 3 DUPACKs

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 18 Dec 2020 09:45:28 -0500

> from: RTO=SRTT+4*RTTVAR (RTO used for arming the timer)

It's also worth mentioning this aspect of [RFC6298]:

   (2.4) Whenever RTO is computed, if it is less than 1 second, then the
         RTO SHOULD be rounded up to 1 second.

>        2. (cwnd=1 MSS); (re)xmit one segment

It may be worthwhile to point out here that the RACK-TLP draft does not
specify setting cwnd to 1 at this point, and the Linux TCP implementation
from our team does not do this. The rationale is that at this point there
is no solid evidence that anything has been lost, and setting cwnd to 1 at
this point would make the algorithm more timid than the preceding
approaches, for no good reason.

>        4. cwnd = ssthresh = FlightSize/2; send N=cwnd segments

That step (4) assumes a particular congestion control implementation that
is different than what we would recommend.

> justification. This clearly is in conflict with items (0) and (1)

The draft pointedly does not modify standard TCP congestion control.

RACK-TLP does not specify using fast recovery instead of slow start for a
case where timeout is needed to detect loss because there is no  feedback
and the ACK clock is lost. Rather, RACK-TLP only triggers fast recovery if
there *is* ACK feedback providing an ACK clock and strong evidence of a
packet loss.

The main aspect of triggering loss recovery that is new is the approach of
allowing a sender to transmit one additional "probe" segment in flight
after 2*SRTT. Once this is accepted, the rest of the recovery process
essentially follows from principles already generally accepted in the IETF
TCP community.

Put another way, it seems to me that if one is to object to TLP-triggered
fast recovery, then the objection must be mounted specifically against the
permission granted to the sender to transmit one additional "probe" segment
in flight after 2*SRTT. Once that permission is granted, there is nothing
really new about TLP-triggered fast recovery.

> start in this case at hand.

What do you think of Yuchung's latest suggestion for new text in "9.3.
Interaction with congestion control" suggested by Yuchung Thursday
afternoon (Dec 17), which explicitly recommends PRR? As mentioned earlier
in this thread, there is considerable implementation and experimental
experience with RACK-TLP plus PRR since the Linux TCP stack has been using
RACK-TLP with PRR as the default loss recovery algorithm since Linux v4.18
in August 2018. The exact commit is:

best,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 17 Dec 2020 20:35:41 -0500

> "3-segments by app" to "1-segment by tcp".

To amplify Yuchung's nice example, and try to restate it in more general
terms:

As far as I'm aware, TLP probes do not introduce materially new congestion
control behaviors, beyond what can happen with [RFC5681] and [RFC6675].

This is because a TLP probe serves much the same probing function that an
application write() could have served, if the application had been so lucky
as to time its write at the appropriate time (i.e. delay the write() of the
last segment in the flight to be 2*SRTT after the preceding segment).

Thus, for any scenario that one constructs where a TLP probe initiates a
fast recovery episode, it is possible to construct, for a TCP
implementation implementing just [RFC5681] and [RFC6675], an application
write() pattern where the on-the-wire behavior is nearly identical to the
TLP-initiated recovery.

For folks concerned about a scenario with FlightSize of 100 segments, and a
sender entering fast recovery and blasting 50 segments, the same behavior
could happen with the existing RFCs [RFC5681] and [RFC6675], which allow
that. And implementers who are worried about such bursts (a very reasonable
thing to be worried about) should probably be implementing pacing and PRR,
or something like that. But that was already true before RACK-TLP.

best,
neal

---

From: Markku Kojo <[email-masked]>
Date: Fri, 18 Dec 2020 22:41:21 +0200 (EET)

Hi,

On Thu, 17 Dec 2020, Yuchung Cheng wrote:

> events properly. PRR [RFCxxxx] is RECOMMENDED for the specificcongestion control actions taken
> upon the losses detected by RACK-TLP"

I believe you intend to add such a mention in Figure 1. So ACK for it.

I think

  "... implementation of RFC6675 pipe algorithm may not directly
   account ..."

is too mild expression, i.e., "may not" does not quite capture the 
problem. The problem in my view is that RFC6675 algorithm simply breaks, 
i.e., it possibly requires significant modification in many aspects, but 
I cannot immediately say what all falls apart other than determining the 
new value for cwnd & ssthresh with FlightSize and the pipe calculation. 
Also, NextSeg () seemingly does not work for lost retransmissions, etc.

I think the draft should say that RFC 6675 algorithm does not handle such 
a second congestion response correctly. Therefore it MUST not be used but 
a modified algorithm that carefully addresses such a case is needed.

Unless I am not mistaken, PRR algorithm does not work without any 
modification either (for reducing window again for a lost retransmission). 
At least FlightSize cannot be used for deterniming new value for ssthresh 
when a loss of a retransmission is detected in the middle of fast 
recovery. I cannot tell for sure what else possibly breaks. If it is a 
trivial change in PRR, I just don't see it immediately. But if it is, I 
think it would be useful to explicitly point it out.

Another important issue with detecting lost retransmissions is tracking 
when this occurs. Currently RACK-TLP algorithm (pseudocode) detects a loss 
of a retransmitted segment in the same way as a loss of an original 
transmission. In order to take an appropriate congestion response when the 
loss of a retransmitted segment occurs, it would be good if the RACK-TLP 
algorithm would track when this occurs and indicate it. In particular, I 
think it would be very important when the congestion control actions are 
in another document.
Otherwise such a congestion control document would first need to modify 
the RACK-TLP algorithm to add the necessary variables and actions.

FYI: When we decoupled spurious retransmission detection and congestion 
control response, the detection RFCs (EIFEL [RFC 3522] and F-RTO [RFC 
5682] included the necessary variable for indicating that spurious 
retransmission was detected and the algorithms set the variable.

> To Makku's request for "what's the justification to enter fast recovery". Consider this example
> w/o RACK-TLP
> 
> T-2RTT: app writes so another 3 segments are sent. Made to the destination and triggered 3
> DUPACKs
> T-3RTT: 3 DUPACK arrives. start fast recovery and subsequent cc reactions to burst ~50 packets
> with Reno 
> 
> In this case any ACK occured before RTO is (generally) considered clock-acked, and how I
> understand Van's initial design.  This behavior existed decades before RACK-TLP. RACK-TLP
> essentially changes the "3-segments by app" to "1-segment by tcp". 

Ack, this kind of scenario is a problem with RFC 6675 loss detection and 
recovery, but not with Reno loss recovery (RFC 5681) nor with Newreno 
loss recovery (RFC 6582).

Sure congestion control response for this scenario should be fixed with 
RFC 6675. PRR is one such fix.

But now we are discussing pure RFC 6675 (without PRR cc modification).
My point is that we should not make things (congestion control behavior) 
worse than in the current standard track algorithms even though there is 
this problem with the current algorithm. I try to explain why I think 
things may get worse if TLP probe is used to trigger fast recovery in 
*all* scenarios where it detects a loss. So we compare RACK-TLP loss 
detection to pure RFC 6675 DupAck detection when a TCP sender uses 
standard RFC 6675 loss recovery and congestion control in both cases.

(0) In the aforementioned scenario the same problem occurs also with
     RACK-TLP loss detection.
     No diffrence, both should be fixed/mitigated.

Differences:

(1) RFC 6675 has this burst problem only in the aforementioned scenario.

     RACK-TLP has also additional scenarios when this problem may
     occur:

     (a) Always when the sender is cwnd limited and the entire
         flight is lost. This case includes large transfers
         where cwnd is often large, resulting in large bursts.

     (b) Always when the sender is rwnd limited and the entire
         flight is lost.

     (c) Always when the sender remains application limited and
         the entire flight is lost. That is, when the sending
         application does not provide more data to send within
         a period of one RTO.

      This means that the burst problem is likely to occur more
      often with RACK-TLP because there are more occassions when
      a problematic scenario may emerge.
      This is an important difference that is likely to create more
      congestion with RACK-TLP detection than with DupAck-based
      detection, if TLP-detected loss always invokes fast recovery
      without necessary safeguards in the fast recovery algorithm

(2)  A combined effect of entering fast recovery when the
      entire flight is lost and capability to detect lost
      retransmissions:

      Important to note first: if there was severe congestion that
      continues it is likely that some (many) segments are
      dropped from the burst of retransmitted segments.

      In this case:

        RFC 6675 sender (re)transmits more segments ack-clocked at
        most for one RTT at half of the previous rate and then waits
        until RTO expires --> cwnd=1

        RACK-TLP sender continues (re)transmitting segments ack-clocked
        at the half of the previous rate maybe for *several RTTs* until
        all segments repaired or RTO expires.
        If all losses are repaired and RACK-TLP does not take another
        congestion response when a lost retransmission is detected,
        it continues all the way without further cwnd reduction.

      Thereby RACK-TLP sender would be significantly more aggessive that
      RFC 6675 sender.

      If RACK-TLP takes another congestion response when a lost
      retransmission is detected, it is still more aggressive but
      much more reasonable as it halves cwnd again.

(3) Maybe not that important:

     Three DupAcks provide some ack clocking feedback that can be
     used to mitigate pure RFC 6675 behavior. For example, the sender
     may derive the ack clock from these three DupAcks and use it to
     clock out the retransmissions (e.g., using at least doubled
     spacing compared to what is inferfed from DupAcks). Segment
     offloading (GRO) may interfere though?

     A single DupAck for RACK-TLP does not provide any ack clock
     information for this purposes.

Best regards,

/Markku

> On Thu, Dec 17, 2020 at 10:52 AM Praveen Balasubramanian <[email-masked]> wrote:
>
>       Cc: [email-masked] Extensions <[email-masked]>; [email-masked]; Michael
>       Tuexen <[email-masked]>; [email-masked]; Last Call
>       <[email-masked]>; tcpm-chairs <[email-masked]>
>       Subject: [EXTERNAL] Re: [tcpm] Last Call:
>       <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to Proposed
>       Standard
> 
>  
> 
> Hi Markku,
> 
>  
> 
>       period
>       working
>       >       > There is a ton here, but I'll try to address the top points.
>       >       Hopefully
>       >       > they obviate the rest.
>       >
>       >       > 1.
>       >       > [Markku]
>       >       > "Hmm, not sure what you mean by "this is a new loss detection
>       >       after
>       >       > acknowledgment of new data"?
>       >       > But anyway, RFC 5681 gives the general principle to reduce
>       >       cwnd and
>       >       > ssthresh twice if a retransmission is lost but IMHO (and I
>       >       believe many
>       >       > who have designed new loss recovery and CC algorithms or
>       >       implemented
>       >       > them
>       >       > agree) that it is hard to get things right if only congestion
>       >       control
>       >       > principles are available and no algorithm."
>       >       >
>       >       > [Martin]
>       >       > So 6675 Sec 5 is quite explicit that there is only one cwnd
>       >       reduction
>       >       > per fast recovery episode, which ends once new data has been
>       >       > acknowledged.
>       >
>       >       > By definition, if a retransmission is lost it is because
>       >       > newer data has been acknowledged, so it's a new recovery
>       >       episode.
>       >
>       >       > Meanwhile, during the Fast Recovery period the incoming acks
>       >       implicitly
>       >       > remove data from the network and therefore keep flightsize
>       >       low.
>       >
>       >       > We can continue to go around on our interpretation of these
>       >       documents,
>       >       > but fundamentally if there is ambiguity in 5681/6675 we should
>       >       bis
>       >       > those RFCs rather than expand the scope of RACK.
>       >
>       >       > 2.
>       >       > [Markku]
>       >       > " In short:
>       >       > When with a non-RACK-TLP implementation timer (RTO) expires:
>       >       cwnd=1
>       >       > MSS,
>       >       > and slow start is entered.
>       >       > When with a RACK_TLP implementation timer (PTO) expires,
>       >       > normal fast recovery is entered (unless implementing
>       >       > also PRR). So no RTO recovery as explicitly stated in Sec.
>       >       7.4.1."
>       >       >
>       >       > [Martin]
>       >       > There may be a misunderstanding here. PTO is not the same as
>       >       RTO, and
>       >       > both mechanisms exist! The loss response to a PTO is to send a
>       >       probe;
>       >       > the RTO response is as with conventional TCP. In Section 7.3:
>       >
>       >       > After attempting to send a loss probe, regardless of whether a
>       >       loss
>       >       >    probe was sent, the sender MUST re-arm the RTO timer, not
>       >       the PTO
>       >       >    timer, if FlightSize is not zero.  This ensures RTO
>       >       recovery remains
>       >       >    the last resort if TLP fails.
>       >       > "
>       >
>       >       > So a pure RTO response exists in the case of persistent
>       >       congestion that
>       >       > causes losses of probes or their ACKs.
>       >
>       >       > Martin
>       >       > On Wed, Dec 16, 2020 at 11:39 AM Markku Kojo
>       >       <[email-masked]>
>       >       > wrote:
>       >       >       Hi Martin,
>       >       >
>       >       >       On Tue, 15 Dec 2020, Martin Duke wrote:
>       >       >
>       >       >       > Hi Markku,
>       >       >       >
>       >       >       > Thanks for the comments. The authors will incorporate
>       >       >       many of your
>       >       >       > suggestions after the IESG review.
>       >       >       >
>       >       >       > There's one thing I don't understand in your comments:
>       >       >       >
>       >       >       > " That is,
>       >       >       > where can an implementer find advice for correct
>       >       >       congestion control
>       >       >       > actions with RACK-TLP, when:
>       >       >       >
>       >       >       > (1) a loss of rexmitted segment is detected
>       >       >       > (2) an entire flight of data gets dropped (and
>       >       detected),
>       >       >       >      that is, when there is no feedback available and
>       >       a
>       >       >       timeout
>       >       >       >      is needed to detect the loss "
>       >       >       >
>       >       >       > Section 9.3 is the discussion about CC, and is clear
>       >       that
>       >       >       the
>       >       >       > implementer should use either 5681 or 6937.
>       >       >
>       >       >       Just a cite nit: RFC 5681 provides basic CC concepts and
>       >       >       some useful CC
>       >       >       guidelines but given that RACK-TLP MUST implement SACK
>       >       the
>       >       >       algorithm in
>       >       >       RFC 5681 is not that useful and an implementer quite
>       >       likely
>       >       >       follows
>       >       >       mainly the algorithm in RFC 6675 (and not RFC 6937 at
>       >       all
>       >       >       if not
>       >       >       implementing PRR).
>       >       >       And RFC 6675 is not mentioned in Sec 9.3, though it is
>       >       >       listed in the
>       >       >       Sec. 4 (Requirements).
>       >       >
>       >       >       > You went through the 6937 case in detail.
>       >       >
>       >       >       Yes, but without correct CC actions.
>       >       >
>       >       >       > If 5681, it's pretty clear to me that in (1) this is a
>       >       >       new loss
>       >       >       > detection after acknowledgment of new data, and
>       >       therefore
>       >       >       requires a
>       >       >       > second halving of cwnd.
>       >       >
>       >       >       Hmm, not sure what you mean by "this is a new loss
>       >       >       detection after
>       >       >       acknowledgment of new data"?
>       >       >       But anyway, RFC 5681 gives the general principle to
>       >       reduce
>       >       >       cwnd and
>       >       >       ssthresh twice if a retransmission is lost but IMHO (and
>       >       I
>       >       >       believe many
>       >       >       who have designed new loss recovery and CC algorithms or
>       >       >       implemented them
>       >       >       agree) that it is hard to get things right if only
>       >       >       congestion control
>       >       >       principles are available and no algorithm.
>       >       >       That's why ALL mechanisms that we have include a quite
>       >       >       detailed algorithm
>       >       >       with all necessary variables and actions for loss
>       >       recovery
>       >       >       and/or CC
>       >       >       purposes (and often also pseudocode). Like this document
>       >       >       does for loss
>       >       >       detection.
>       >       >
>       >       >       So the problem is that we do not have a detailed enough
>       >       >       algorithm or
>       >       >       rule that tells exactly what to do when a loss of rexmit
>       >       is
>       >       >       detected.
>       >       >       Even worse, the algorithms in RFC 5681 and RFC 6675
>       >       refer
>       >       >       to
>       >       >       equation (4) of RFC 5681 to reduce ssthresh and cwnd
>       >       when a
>       >       >       loss
>       >       >       requiring a congestion control action is detected:
>       >       >
>       >       >         (cwnd =) ssthresh = FlightSize / 2)
>       >       >
>       >       >       And RFC 5681 gives a warning not to halve cwnd in the
>       >       >       equation but
>       >       >       FlightSize.
>       >       >
>       >       >       That is, this equation is what an implementer
>       >       intuitively
>       >       >       would use
>       >       >       when reading the relevant RFCs but it gives a wrong
>       >       result
>       >       >       for
>       >       >       outstanding data when in fast recovery (when the sender
>       >       is
>       >       >       in
>       >       >       congestion avoidance and the equation (4) is used to
>       >       halve
>       >       >       cwnd, it
>       >       >       gives a correct result).
>       >       >       More precisely, during fast recovery FlightSize is
>       >       inflated
>       >       >       when new
>       >       >       data is sent and reduced when segments are cumulatively
>       >       >       Acked.
>       >       >       What the outcome is depends on the loss pattern. In the
>       >       >       worst case,
>       >       >       FlightSize is signficantly larger than in the beginning
>       >       of
>       >       >       the fast
>       >       >       recovery when FlightSize was (correctly) used to
>       >       determine
>       >       >       the halved
>       >       >       value for cwnd and ssthresh, i.e., equation (4) may
>       >       result
>       >       >       in
>       >       >       *increasing* cwnd upon detecting a loss of a rexmitted
>       >       >       segment, instead
>       >       >       of further halving it.
>       >       >
>       >       >       A clever implementer might have no problem to have it
>       >       right
>       >       >       with some
>       >       >       thinking but I am afraid that there will be incorrect
>       >       >       implementations
>       >       >       with what is currently specified. Not all implementers
>       >       have
>       >       >       spent
>       >       >       signicicant fraction of their career in solving TCP
>       >       >       peculiarities.
>       >       >
>       >       >       > For (2), the RTO timer is still operative so
>       >       >       > the RTO recovery rules would still follow.
>       >       >
>       >       >       In short:
>       >       >       When with a non-RACK-TLP implementation timer (RTO)
>       >       >       expires: cwnd=1 MSS,
>       >       >       and slow start is entered.
>       >       >       When with a RACK_TLP implementation timer (PTO) expires,
>       >       >       normal fast recovery is entered (unless implementing
>       >       >       also PRR). So no RTO recovery as explicitly stated in
>       >       Sec.
>       >       >       7.4.1.
>       >       >
>       >       >       This means that this document explicitly modifies
>       >       standard
>       >       >       TCP congestion
>       >       >       control when there are no acks coming and the
>       >       >       retransmission timer
>       >       >       expires
>       >       >
>       >       >       from: RTO=SRTT+4*RTTVAR (RTO used for arming the timer)
>       >       >              1. RTO timer expires
>       >       >              2. cwnd=1 MSS; ssthresh=FlightSize/2; rexmit one
>       >       >       segment
>       >       >              3. Ack of rexmit sent in step 2 arrives
>       >       >              4. cwnd = cwnd+1 MSS; send two segments
>       >       >              ...
>       >       >
>       >       >       to:   PTO=min(2*SRTT,RTO) (PRO used for arming the
>       >       timer)
>       >       >              1. PTO times expires
>       >       >              2. (cwnd=1 MSS); (re)xmit one segment
>       >       >              3. Ack of (re)xmit sent in srep 2 arrives
>       >       >              4. cwnd = ssthresh = FlightSize/2; send N=cwnd
>       >       >       segments
>       >       >
>       >       >       For example, if FlightSize is 100 segments when timer
>       >       >       expires,
>       >       >       congestion control is the same in steps 1-3, but in step
>       >       4
>       >       >       the
>       >       >       current standard congestion control allows transmitting
>       >       2
>       >       >       segments,
>       >       >       while RACK-TLP would allow blasting 50 segments.
>       >       >
>       >       >       Question is: what is the justification to modify
>       >       standard
>       >       >       TCP
>       >       >       congestion control to use fast recovery instead of slow
>       >       >       start for a
>       >       >       case where timeout is needed to detect loss because
>       >       there
>       >       >       is no
>       >       >       feedback and ack clock is lost? The draft does not give
>       >       any
>       >       >       justification. This clearly is in conflict with items
>       >       (0)
>       >       >       and (1)
>       >       >       in BCP 133 (RFC 5033).
>       >       >
>       >       >       Furthermore, there is no implementation nor experimental
>       >       >       experience
>       >       >       evaluating this change. The implementation with
>       >       >       experimental experience
>       >       >       uses PRR (RFC 6937) which is an Experimental
>       >       specification
>       >       >       including a
>       >       >       novel "trick" that directs PRR fast recovery to
>       >       effectively
>       >       >       use slow
>       >       >       start in this case at hand.
>       >       >
>       >       >
>       >       >       > In other words, I am not seeing a case that requires
>       >       new
>       >       >       congestion
>       >       >       > control concepts except as discussed in 9.3.
>       >       >
>       >       >       See above. The change in standard congestion control for
>       >       >       (2).
>       >       >       The draft intends not to change congestion control but
>       >       >       effectively it
>       >       >       does without any operational evidence.
>       >       >
>       >       >       What's also is missing and would be very useful:
>       >       >
>       >       >       - For (1), a hint for an implementer saying that because
>       >       >       RACK-TLP is
>       >       >          able to detect a loss of a rexmit unlike any other
>       >       loss
>       >       >       detection
>       >       >          algorithm, the sender MUST react twice to congestion
>       >       >       (and cite
>       >       >          RFC 5681). And cite a document where necessary
>       >       correct
>       >       >       actions
>       >       >          are described.
>       >       >
>       >       >       - For (1), advise that an implementer needs to keep
>       >       track
>       >       >       when it
>       >       >          detects a loss of a retransmitted segment. Current
>       >       >       algorithms
>       >       >          in the draft detect a loss of retransmitted segment
>       >       >       exactly in
>       >       >          the same way as loss of any other segment. There
>       >       seems
>       >       >       to be
>       >       >          nothing to track when a retransmission of a
>       >       >       retransmitted segment
>       >       >          takes place. Therefore, the algorithms should have
>       >       >       additional
>       >       >          actions to correctly track when such a loss is
>       >       detected.
>       >       >
>       >       >       - For (1), discussion on how many times a loss of a
>       >       >       retransmission
>       >       >          of the same segment may occur and be detected. Seems
>       >       >       that it
>       >       >          may be possible to drop a rexmitted segment more than
>       >       >       once and
>       >       >          detect it also several times?  What are the
>       >       >       implications?
>       >       >
>       >       >       - If previous is possible, then the algorithm possibly
>       >       also
>       >       >          may detect a loss of a new segment that was sent
>       >       during
>       >       >       fast
>       >       >          recovery? This is also loss in two successive windows
>       >       of
>       >       >       data,
>       >       >          and cwnd MUST be lowered twice. This discussion and
>       >       >       necessary
>       >       >          actions to track it are missing, if such scenario is
>       >       >       possible.
>       >       >
>       >       >       > What am I missing?
>       >       >
>       >       >       Hope the above helps.
>       >       >
>       >       >       /Markku
>       >       >
>       >       >
>       >       > <snipping the rest>
>       >       >
>       >       >
>       >
>       >
>       >
> 
> 
>

---

From: Markku Kojo <[email-masked]>
Date: Fri, 18 Dec 2020 23:31:49 +0200 (EET)

Hi,

On Fri, 18 Dec 2020, Praveen Balasubramanian wrote:

> 
> Looks good to me. I would omit “, but some other congestion control may want to respond
> multiple times differently”.

Agreed to omit. The lost retransmissions of segments that were first 
retransmitted during the same RTT are considered to be in the same window 
of data, so it is enough to react once per RTT.

Current standard congestion controls that are able to recover from 
multiple losses per window may however take several RTTs to recover.
NewReno takes 1 RTT per lost segment. RFC 6675 ideally requires just one 
RTT but depending on the loss pattern the recovery in practice often takes 
several RTTs if there is a significant number of losses in one window of 
data. RACK makes detection more efficient than DupAck-based detection, so 
it is more likely that less RTTs are need with RACK, I believe.

Anyway, any upcoming congestion control spec should carefully consider 
how to handle the case where several lost rexmits were detected and they 
were sent in different RTTs. The congestion control principle is that the 
sender should react once per each subsequent RTT (window) in which a loss 
is detected. This would make things much more complicated though.

Best regards,

/Markku

> From: Yuchung Cheng <[email-masked]>
> Sent: Thursday, December 17, 2020 2:55 PM
> To: Martin Duke <[email-masked]>
> Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: [EXTERNAL] Re: [tcpm] Last Call:
> <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to Proposed Standard
> 
>  
> 
>  
> 
>  
> 
> I might also add the sentence "In the absence of PRR, when TCP RACK detects a lost
> retransmission it MUST trigger a second congestion response" or something to that
> effect. What do you think?
> 
> Sure I'd revise a bit to be even more detailed and consistent with the principle in RFC5681
> 
>  
> 
> "In the absence of PRR, when RACK-TLP detects a lost retransmission the congestion control
> MUST trigger an additional congestion response per the aforementioned principle in RFC5681.
> 
> If multiple retransmissions were lost in a window, the congestion control specified in
> RFC5681 only reacts once per window, but some other congestion control may want to respond
> multiple times differently. The congestion control implementer is advised to
> carefully consider this subtle situation."
> 
>  
> 
>  
>
>        
>
> T-2RTT: app writes so another 3 segments are sent. Made to the destination and
> triggered 3 DUPACKs
> 
> T-3RTT: 3 DUPACK arrives. start fast recovery and subsequent cc reactions to
> burst ~50 packets with Reno 
> 
>  
> 
> In this case any ACK occured before RTO is (generally) considered clock-acked,
> and how I understand Van's initial design.  This behavior existed decades before
> RACK-TLP. RACK-TLP essentially changes the "3-segments by app" to "1-segment by
> tcp". 
> 
>  
> 
>       I agree that we should have a note in this RFC about congestion
>       control action upon detecting lost retransmission(s).
>
>        
>
>       Cc: [email-masked] Extensions <[email-masked]>;
>       [email-masked]; Michael Tuexen
>       <[email-masked]>; [email-masked]; Last Call
>       <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP)
>       to Proposed Standard
> 
>  
> 
> Hi Markku,
> 
>  
> 
>       >       Sorry for being verbose. I tried to be clear but you
>       actually
>       >       removed my
>       >       key issues/questions ;)
>       >
>       >       > 1.
>       >       > [Markku]
>       >       To be more precise: fast recovery ends when the current
>       window
>       >       becomes
>       >       (4.1) RecoveryPoint (= HighData at the beginning)
>       becomes
>       >       acknowledged
>       >
>       >       RecoveryPoint may become cumulatively acknowledged
>       already
>       >       earlier
>       >       during the fast recovery. Reno loss recovery in RFC
>       5681 ends,
>       >       when
>       >       it requires RTO to be detected and definitely starts a
>       new
>       >       recovery
>       >       episode. That is, a new recovery episode is enforced by
>       step
>       >       (1.a) of
>       >       NextSeg () which prevents retransmission if a segment
>       that has
>       >       already
>       >       been retransmitted. If RACK-TLP is used for detecting
>       loss with
>       >       RFC 6675
>       >       things get different in many ways, because it may
>       detect loss of
>       >       a
>       >       of the algorith. For example, calculation of pipe does
>       not
>       >       consider
>       >       Incorrect. FlightSize != pipe. Only cumulative acks
>       remove data
>       >       from
>       >       FlightSize. How FlightSize evolves depends on loss
>       pattern as I
>       >       said.
>       >       directions. A simple example can be used as a proof for
>       the case
>       >       where
>       >       cwnd increases if a loss of retransmission is detected
>       and
>       >       repaired:
>       >
>       >       CWND INCREASES when it should be at most 5 after
>       halving it
>       >       twice!!!
>       >
>       >       As I said earlier, I am not opposing bis, though
>       5681bis wuold
>       >       not
>       >       be needed, I think.
>       >
>       >       But let me repeat: if we publish RACK-TLP now without
>       necessary
>       >       warnings
>       >       or with a correct congesion control algorithm someone
>       will try
>       >       to
>       >       implement RACK-TLP with RFC 6675 and it will be a total
>       mesh.
>       >       The
>       expires:
>       >       another name, it is still timeout. And congestion
>       control does
>       >       not
>       >       consider which segments to send (SND.UNA vs. probe w/
>       higher
>       >       sequence
>       >       You ignored my major point where I decoupled congestion
>       control
>       >       from loss
>       >       detection and loss recovery and compared RFC 5681
>       behavior to
>       >       RACK-TLP
>       >       behavior in exactly the same scenario where an entire
>       flight is
>       >       lost and
>       >       timer expires.
>       >
>       >       Please comment why congestion control behavior is
>       allowed to be
>       >       radically
>       >       If FlightSize is 100 segments when timer expires,
>       congestion
>       >       control is
>       >       the same in steps 1-3, but in step 4 the standard
>       congestion
>       >       control
>       >       This does not prevent the above RACK-TLP behavior from
>       getting
>       >       realized.
>       >
>       >       Yes, RTO response exists BUT only after RACK-TLP at
>       least once
>       >       blasts the
>       >       network. It may well be that with smaller windows
>       RACK-TLP is
>       >       successful
>       >       during its TLP initiated overly aggressive "fast
>       recovery" and
>       >       never
>       >       enters RTO recovery because it may detect and repair
>       also loss
>       >       of
>       >       rexmits. That is, it continues at too high rate even if
>       lost
>       >       rexmits
>       >       indicate that congestion persists in successive windows
>       of data.
>       >       And
>       >       worse, it is successful because it pushes away other
>       compatible
>       >       TCP
>       >       this is exactly what RFC 2914 and RFC 5033 explicitly
>       address
>       >       and warn
>       >       about.
>       >
>       >       What is the justification to modify standard TCP
>       congestion
>       >       control to
>       >       use fast recovery instead of slow start for a case
>       where timeout
>       >       is
>       >       needed to detect the packet losses because there is no
>       feedback
>       >       and ack
>       >       clock is lost? RACK-TLP explicitly instructs to do so
>       in Sec.
>       >       7.4.1.
>       >
>       >       As I noted: based on what is written in the draft it
>       does not
>       >       intend to

---

From: Markku Kojo <[email-masked]>
Date: Sat, 19 Dec 2020 02:41:31 +0200 (EET)

Hi Neal,

> It's also worth mentioning this aspect of [RFC6298]:

Sure.

> It may be worthwhile to point out here that the RACK-TLP draft does not specify setting cwnd
> to 1 at this point, and the Linux TCP implementation from our team does not do this. The

Yes, that's why I put it in parenthesis. In my view the RACK-TLP 
draft implicitly limits cwnd to one segment by allowing just one TLP 
probe segment.

> rationale is that at this point there is no solid evidence that anything has been lost, and
> setting cwnd to 1 at this point would make the algorithm more timid than the preceding

Sure, no need to set cwnd at this point.

A good reason could be: No feedback, Ack clock lost? But, of course, 
it is too early even though after the arrival of ack the sender may well 
modify cwnd again. Like it now does, if it decides it was loss other than 
probe segment.
   
> That step (4) assumes a particular congestion control implementation that is different than
> what we would recommend.

Ok. I just used the Standards Track formula as does the RACK-TLP draft in 
its examples. And because RACK-TLP draft states it does not modify 
current congestion control.

> RACK-TLP does not specify using fast recovery instead of slow start for a  case where timeout
> is needed to detect loss because there is no  feedback and the ACK clock is lost. Rather,
> RACK-TLP only triggers fast recovery if there *is* ACK feedback providing an ACK clock and
> strong evidence of a packet loss.

So here our views diverge. In the above steps I decoupled congestion 
control from what segments are sent (rexmit and xmit are mentioned there 
just as comments to check what is going on, they can be freely removed).
Congestion control governs how many segments can be sent.

In my view, when there is no feedback RACK TLP uses timeout (PTO) to help 
make progress. Without the timeout it cannot make progress. Just like 
an RFC 5681 sender, it cannot make progress until timeout expires. 
So this should be taken as the criteria to (effectively) enter slow start, 
once loss becomes detected.

Or, at least I don't see any difference why different timeout value would 
change the congestion control.

When timeout expires RACK-TLP sends one segment (just like an RFC 5681 
sender when RTO expires). The only difference is that RFC 5681 sender 
selects a different segment (first unacknowledged segment) to retransmit 
"blindly" in order to get feedback and start ACK clock. RACK-TLP sends 
"blindly" the last segment from the retransmission queue (or a new 
segment). Selecting a different segment for transmission upon timeout 
does not change anything, in my view. In both cases it is a "blind" 
selection; the sender does not know what was lost. And in both cases the 
ACK for this one segment provides feedback about what potentially has 
been lost. There the only difference is that the segment that RACK-TLP 
selected to transmit is a better choice when SACK option is use because 
it provides more information.

If there is some difference in that the ACK for RACK-TLP provides 
stronger evidence for packet loss (and what was lost), then it should be 
also ok to modify the current standard TCP congestion control such that 
upon RTO timeout the sender does not select the first unacknowledged 
segment for blind retransmission but the last segment in the 
retransmission queue (or maybe a new segment). With SACK this would 
provide exactly the same information as TLP probe does. And, upon arrival 
of the first ACK, RTO recovery would use similar rules as in RACK-TLP to 
better decide whether it was spurious RTO or loss and move from slow 
start to fast recovery and set cwnd=ssthresh.

I really don't see how this change in "blindly" retrasmitted first segment 
in slow start would allow modifying congestion control for RTO recovery.

> The main aspect of triggering loss recovery that is new is the approach of allowing a sender
> to transmit one additional "probe" segment in flight after 2*SRTT. Once this is accepted, the
> rest of the recovery process essentially follows from principles already generally accepted
> in the IETF TCP community.

Could you please see above and explain (or provide a pointer to an RFC) 
what are those "principles already generally accepted in the IETF TCP 
community". That would help me to understand your point.

> Put another way, it seems to me that if one is to object to TLP-triggered fast recovery, then
> the objection must be mounted specifically against the permission granted to the sender to
> transmit one additional "probe" segment in flight after 2*SRTT. Once that permission is
> granted, there is nothing really new about TLP-triggered fast recovery.

I am sorry but I still fail to see what is the preceding evidence that 
makes this not new. A pointer could help.

In my view the probe is not anything to object as long as it is not 
considered as a cwnd increase in the later cwnd&ssthresh calculation 
(a minor detail, but someone might later suggest first two then 4 and so 
on probe segments with the justufication that it is just one more than 
earlier).

> What do you think of Yuchung's latest suggestion for new text in "9.3.  Interaction with
> congestion control" suggested by Yuchung Thursday afternoon (Dec 17), which explicitly
> recommends PRR? As mentioned earlier in this thread, there is considerable implementation and
> experimental experience with RACK-TLP plus PRR since the Linux TCP stack has been using
> RACK-TLP with PRR as the default loss recovery algorithm since Linux v4.18 in August 2018.

As I have already indicated, in my view PRR does not have the problem we 
are discussing here because PRR-SSRB makes fast recovery to behave like 
slow start. And PRR-CRB is even more conservative. So it would be a safe 
choice for this problem unlike the current RFC 6675 algorithm.

In other words, I only object allowing the use of RACK-TLP with the 
RFC 6675 congestion control algorithm unmodified because it does not have 
a safeguard like PRR. This does not mean that RACK-TLP document would 
need to include the necessary modifications to the RFC 6675 algorithm.

I don't know processwise but PRR possibly cannot be used as normative 
requirement because it is currently Experimental? Not quite sure though.

Best regards,

/Markku

> The exact commit is:

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 18 Dec 2020 19:35:45 +0000

Looks good to me. I would omit “, but some other congestion control may want to respond multiple times differently”.

From: Yuchung Cheng <[email-masked]>
Sent: Thursday, December 17, 2020 2:55 PM
To: Martin Duke <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [EXTERNAL] Re: [tcpm] Last Call: <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to Proposed Standard

On Thu, Dec 17, 2020 at 1:59 PM Martin Duke <[email-masked]<mailto:[email-masked]>> wrote:
This is good, thanks.

I might also add the sentence "In the absence of PRR, when TCP RACK detects a lost retransmission it MUST trigger a second congestion response" or something to that effect. What do you think?
Sure I'd revise a bit to be even more detailed and consistent with the principle in RFC5681

"In the absence of PRR, when RACK-TLP detects a lost retransmission the congestion control MUST trigger an additional congestion response per the aforementioned principle in RFC5681.
If multiple retransmissions were lost in a window, the congestion control specified in RFC5681 only reacts once per window, but some other congestion control may want to respond multiple times differently. The congestion control implementer is advised to carefully consider this subtle situation."

On Thu, Dec 17, 2020 at 11:36 AM Yuchung Cheng <[email-masked]<mailto:[email-masked]>> wrote:
How about

To Makku's request for "what's the justification to enter fast recovery". Consider this example w/o RACK-TLP

T0: Send 100 segments but application-limited. All are lost.
T-2RTT: app writes so another 3 segments are sent. Made to the destination and triggered 3 DUPACKs
T-3RTT: 3 DUPACK arrives. start fast recovery and subsequent cc reactions to burst ~50 packets with Reno

In this case any ACK occured before RTO is (generally) considered clock-acked, and how I understand Van's initial design.  This behavior existed decades before RACK-TLP. RACK-TLP essentially changes the "3-segments by app" to "1-segment by tcp".

On Thu, Dec 17, 2020 at 10:52 AM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
I agree that we should have a note in this RFC about congestion control action upon detecting lost retransmission(s).

From: tcpm <[email-masked]<mailto:[email-masked]>> On Behalf Of Martin Duke
Sent: Thursday, December 17, 2020 7:30 AM
To: Markku Kojo <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]> Extensions <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Michael Tuexen <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Last Call <[email-masked]<mailto:[email-masked]>>; tcpm-chairs <[email-masked]<mailto:[email-masked]>>
Subject: [EXTERNAL] Re: [tcpm] Last Call: <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithmforTCP) to Proposed Standard

Hi Markku,

Martin

And in the intro:

BR,

/Markku

---

From: Markku Kojo <[email-masked]>
Date: Fri, 18 Dec 2020 23:56:26 +0200 (EET)

Hi Neal,

On Thu, 17 Dec 2020, Neal Cardwell wrote:

> events properly. PRR [RFCxxxx] is RECOMMENDED for the specificcongestion control
> actions taken upon the losses detected by RACK-TLP"
> 
> In this case any ACK occured before RTO is (generally) considered clock-acked, and
> how I understand Van's initial design.  This behavior existed decades before RACK-TLP.
> RACK-TLP essentially changes the "3-segments by app" to "1-segment by tcp". 
> 
> 
> To amplify Yuchung's nice example, and try to restate it in more general terms:
> 
> As far as I'm aware, TLP probes do not introduce materially new congestion control behaviors,
> beyond what can happen with [RFC5681] and [RFC6675].

Please see my previous reply. No new congestion control behaviors if I 
understand what you mean by that but there are more scenarios in which 
loss of a full window is detected without RTO.

> This is because a TLP probe serves much the same probing function that an application write()
> could have served, if the application had been so lucky as to time its write at the
> appropriate time (i.e. delay the write() of the last segment in the flight to be 2*SRTT after
> the preceding segment).
> 
> Thus, for any scenario that one constructs where a TLP probe initiates a fast recovery
> episode, it is possible to construct, for a TCP implementation implementing just [RFC5681]
> and [RFC6675], an application write() pattern where the on-the-wire behavior is nearly
> identical to the TLP-initiated recovery.

Except when the sender is  congestion window limited. Receive window 
limited is another case where application cannot write new data that 
would get transmitted but a TLP probe could send the highest-sequence 
segment?

The draft is a bit unclear on this but I believe this was the intent 
(draft says: "If such a segment is not available, ..." but in receive 
window limited case such a new segment would be available but it cannot 
be transmitted)

> For folks concerned about a scenario with FlightSize of 100 segments, and a sender entering
> fast recovery and blasting 50 segments, the same behavior could happen with the existing RFCs
> [RFC5681] and [RFC6675], which allow that. And implementers who are worried about such bursts
> (a very reasonable thing to be worried about) should probably be implementing pacing and PRR,

I would rather say it is a bug in RFC6675. Nobody paid sufficient 
attention to such a scenario.

Yes, PRR handles it fine, practically directing the sender to slow start 
and start ack clock by limiting the number segments sent per incoming 
Ack. Pure pacing over one RTT would not be enough in my opinion because it 
still sends too many segments per RTT.

Best regards,

/Markku