Subject: [tcpm] draft-ietf-tcpm-rack-05 review

From: Martin Duke <[email-masked]>
Date: Tue, 3 Sep 2019 15:30:49 -0700

Late in the day, but I have a few nits on this draft.

(1) The definition of RACK.rtt in Sec 5.2 does not match the Step 2
pseudocode in Sec 6.2:

"RACK.rtt" is the RTT of the most recently transmitted packet that
   has been delivered (either cumulatively acknowledged or selectively
   acknowledged) on the connection.

6.2 :

If the ACK is not ignored as invalid, update the RACK.rtt to be the
   RTT sample calculated using this ACK, and continue.

...

RACK.rtt = rtt
           If RACK_sent_after(Packet.xmit_ts, Packet.end_seq
                              RACK.xmit_ts, RACK.end_seq):

The definition implies that RACK.rtt should only be send if
RACK_sent_after(), but IIUC this is not what the the text and pseudocode
say.

(2) Sec 7.1

RACK always marks the entire
   TSO blob lost...

unless, of course, some of them are sacked?

(3) sec 7.3
We have evaluated using the smoothed RTT (SRTT from
   [RFC6298 <https://tools.ietf.org/html/rfc6298>] RTT estimation) or
the most recently measured RTT
   (RACK.rtt) using an experiment similar to that in the Performance
   Evaluation section.  They do not make any significant difference in
   terms of total recovery latency.

If there is truly no difference, then why not use SRTT as the standard?
Every TCP implementation has to store this, while min_rtt is unneeded
for many (most?) congestion controls.

Alternatively, you could strengthen this paragraph to not sound like
it makes no difference.

Thanks for your work on this.

Martin

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 4 Sep 2019 17:54:32 -0700

On Tue, Sep 3, 2019 at 3:31 PM Martin Duke <[email-masked]> wrote:
> (1) The definition of RACK.rtt in Sec 5.2 does not match the Step 2 pseudocode in Sec 6.2:
>
> The definition implies that RACK.rtt should only be send if RACK_sent_after(), but IIUC this is not what the the text and pseudocode say.
Yes for catching this! we'll update that in our next rev.

> unless, of course, some of them are sacked?
Yes, the draft could be more specific. If the TSO blob is partially
sacked, Linux splits it and marks one sacked and the other lost.

>    [RFC6298] RTT estimation) or the most recently measured RTT
> Every TCP implementation has to store this, while min_rtt is unneeded for many (most?) congestion controls.
>
> Alternatively, you could strengthen this paragraph to not sound like it makes no difference..
That's a good point -- our experiment at Google servers indeed didn't
show much difference. But I think it's still better to use min_RTT
than SRTT. On buffer-bloat friendly C.C. the SRTT could be orders of
magnitude longer than the actual paths' RTT -- in my opinion factoring
this network queuing delay in reordering window is not a good idea. So
how about adding

'While the experiment does not show a difference between min RTT and
SRTT, SRTT is less desirable to size the reordering window as it
includes network congestion or delayed ACKs effects."

> Thanks for your work on this.
Thanks for the review!
>
> Martin
>
>
>
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Martin Duke <[email-masked]>
Date: Wed, 4 Sep 2019 19:44:44 -0700

I think the proposed text is fine.

On Wed, Sep 4, 2019 at 5:55 PM Yuchung Cheng <[email-masked]> wrote:

> <snip>

---

From: Emmanuel Lochin <[email-masked]>
Date: Fri, 6 Sep 2019 12:24:00 +0200

Hi Yuchung, all,

I have a question concerning the GNU/Linux kernel implementation of RACK.
There are 3 sysctl options to enable RACK (0x1 0x2 and 0x4) but none to 
disable it (or may be 0x0 is not documented). As tcp_input.c implements 
this boolean function:

static bool tcp_is_rack(const struct sock *sk)
{
         return sock_net(sk)->ipv4.sysctl_tcp_recovery & 
TCP_RACK_LOSS_DETECTION;
}

can I consider that I can disable RACK by typing sysctl -w 
net.ipv4.tcp_recovery=0 ?
I ask this question because I've seen several update done inside the 
kernel to implement it and I want to be sure that I can really go back 
to 3DUPACK scheme this way.

Cheers,

Emmanuel

Le 05/09/2019 à 04:44, Martin Duke a écrit :
> On Wed, Sep 4, 2019 at 5:55 PM Yuchung Cheng <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Tue, Sep 3, 2019 at 3:31 PM Martin Duke
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
> <snip>
>
>     > (3) sec 7.3
>     >
>     > Every TCP implementation has to store this, while min_rtt is
>     unneeded for many (most?) congestion controls.
>     >

-- 
Emmanuel LOCHIN
Professeur ISAE
ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -
http://www.isae-supaero.fr
Tel +33 5 61 33 84 85 - Fax (+33) 5 61 33 83 45
Web : http://personnel.isae.fr/emmanuel-lochin/

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 6 Sep 2019 15:32:32 -0700

Hi Emmanuel,

Yes setting the sysctl to 0 will completely disable RACK. Although by
default it supports DUPTHRESH-based detection (as a simple extension)
beside time-based detection. You can disable that extension w/ 0x4. But if
you want absolutely old-style pure DUPTHRESH-based then you can set that to
0.

On Fri, Sep 6, 2019 at 3:24 AM Emmanuel Lochin <
[email-masked]> wrote:

> kernel to implement it and I want to be sure that I can really go back to
> 3DUPACK scheme this way.
>
> Cheers,
>
> Emmanuel
>
> tcpm mailing [email-masked]://www.ietf.org/mailman/listinfo/tcpm
>
>
> --
> Emmanuel LOCHIN
> Professeur ISAE
> 10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -http://www.isae-supaero.fr

---

From: Emmanuel Lochin <[email-masked]>
Date: Sat, 7 Sep 2019 13:03:26 +0200

Hi Yuchung,

Thanks for your answer, I've asked this question as we observed a 
decrease of the performance of TCP transfer in some reordering scenario 
when RACK is enabled. In brief, the problem has been solved by disabling 
RACK.
I am involved in a CNES project (French space agency) where SATCOM 
systems are used (GEO or LEO) and in particular I work with Nicolas Kuhn 
from CNES (in copy of this email) .

To assess whether or not the problem effectively came from RACK, we have 
started a bunch of experiments using Mininet or simply using the 
loopback interface. The results presented below are done with Mininet 
except the last one done in localhost

The x-axis reports the size of the flows, basically we send 5 TCP flows 
in the same time of a given size and we repeat the experiments hundred 
of times over a Netem queue. We then compute the following statistics 
over these 500 flows for each flow length.

There is no artificial losses introduced with Netem, we only use it only 
to reorder packets. The nominal RTT is 500ms (equiv. to a GEO system) 
where some packets are reordered every 5, 10, 15 or 20 packets sent. 
These reordered packets have then an RTT=250ms. As you can see, in all 
cases when RACK is enabled, the completion time (time difference between 
the first packet received at destination and the last one) is always 
worse with RACK. The kernel is 4.15.0-60-generic.

We thought, may be, it was due to the reordering pattern which is too 
deterministic. We thus applied several other random patterns.
Below we present the Netem config documented in the TC manpage for 
simplicity (i.e. reorder 25% 50% gap 5 and 10)

But results obtained where the same.

In all these previous experiments the reorder delay is 50% of the RTT, 
and indeed, the RTT is important.
If we decrease it to 100ms, when the reorder delay corresponds to 90% of 
the RTT (meaning that RTT=100ms and reordered packets get 90ms), in that 
case, RACK performs much more better as shown below.

Using the loopback interface, even with low RTT (i.e. 100ms), and when 
the reordering delay is half the RTT (50ms), RACK still does not perform 
always better than 3DUPACK.

We really believe that RACK is a good idea, but it seems that the kernel 
implementation does not lead to what we expected. Do you have 
experienced such behavior or do you have measurements that thwart mine ?

Thanks

Emmanuel

Le 07/09/2019 à 00:32, Yuchung Cheng a écrit :
> beside time-based detection. You can disable that extension w/ 0x4. 
> But if you want absolutely old-style pure DUPTHRESH-based then you can 
> set that to 0.
>
>
> On Fri, Sep 6, 2019 at 3:24 AM Emmanuel Lochin 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Yuchung, all,
>
>     There are 3 sysctl options to enable RACK (0x1 0x2 and 0x4) but
>     none to disable it (or may be 0x0 is not documented). As
>     tcp_input.c implements this boolean function:
>
>     I ask this question because I've seen several update done inside
>     the kernel to implement it and I want to be sure that I can really
>     go back to 3DUPACK scheme this way.
>
>     Cheers,
>
>     Emmanuel
>
>>         > Every TCP implementation has to store this, while min_rtt
>>         is unneeded for many (most?) congestion controls.
>>         >
>>     [email-masked]  <mailto:[email-masked]>
>     Web :http://personnel.isae.fr/emmanuel-lochin/
>

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 11 Sep 2019 12:19:16 -0700

Re-sending in plainetext to avoid tcpm filter

Hi Emmanuel,

Sorry for the late reply and thanks for testing RACK. Some questions:
1) what's the congestion control? I assume it's the default (i.e. Cubic)

2) what's the value of sysctl net.ipv4.tcp_no_metrics_save? I assume
it's default 0 meaning TCP save some metrics to be reused by future
flows towards same IP

3) what's the bandwidth of the bottleneck, or is BDP generally beyond
the 5000 packets in your test

If my assumptions above are correct, I speculate that the performance
gap is caused by spurious early exit TCP Cubic slow-start. This is
caused by RACK not catching the some reordering well initially with
its init RTT/4 reordering window. Many RACK parameters are tuned based
on Google's traffic where nominal RTT is much lower than 500ms.

In contrast, the Linux dupack implementation (referred by 3DUPACK in
your graph) has two features:
a) it dynamically extends the dupthresh to a maximum of 300 packets
(sysctl net.ipv4.tcp_max_reordering), hence makes TCP very resilient
if only OOO is reordering not drops.
b) the high dupthresh is cached and reused for new connection toward
the same dst ip by (2).
Thus it's possible all subsequent flows in one test are starting with
dupthresh to the highest possible, making it bullet-proof to any
reordering by never entering fast recovery falsely.

To check my theory, could you provide some tcpdump of each test (RACK
vs non-RACK)? only header captures are needed.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 11 Sep 2019 12:16:47 -0700

Hi Emmanuel,

2) what's the value of sysctl net.ipv4.tcp_no_metrics_save? I assume it's
default 0 meaning TCP save some metrics to be reused by future flows
towards same IP

3) what's the bandwidth of the bottleneck, or is BDP generally beyond the
5000 packets in your test

If my assumptions above are correct, I speculate that the performance gap
is caused by spurious early exit TCP Cubic slow-start. This is caused by
RACK not catching the some reordering well initially with its init RTT/4
reordering window. Many RACK parameters are tuned based on Google's traffic
where nominal RTT is much lower than 500ms.

In contrast, the Linux dupack implementation (referred by 3DUPACK in your
graph) has two features:
a) it dynamically extends the dupthresh to a maximum of 300 packets (sysctl
net.ipv4.tcp_max_reordering), hence makes TCP very resilient if only OOO is
reordering not drops.
b) the high dupthresh is cached and reused for new connection toward the
same dst ip by (2).
Thus it's possible all subsequent flows in one test are starting with
dupthresh to the highest possible, making it bullet-proof to any reordering
by never entering fast recovery falsely.

To check my theory, could you provide some tcpdump of each test (RACK vs
non-RACK)? only header captures are needed.

On Sat, Sep 7, 2019 at 4:03 AM Emmanuel Lochin <
[email-masked]> wrote:

> Hi Yuchung,
>
> Thanks for your answer, I've asked this question as we observed a decrease
> of the performance of TCP transfer in some reordering scenario when RACK is
> enabled. In brief, the problem has been solved by disabling RACK.
> I am involved in a CNES project (French space agency) where SATCOM systems
> are used (GEO or LEO) and in particular I work with Nicolas Kuhn from CNES
> (in copy of this email) .
>
> started a bunch of experiments using Mininet or simply using the loopback
> interface. The results presented below are done with Mininet except the
> last one done in localhost
>
> The x-axis reports the size of the flows, basically we send 5 TCP flows in
> the same time of a given size and we repeat the experiments hundred of
> times over a Netem queue. We then compute the following statistics over
> these 500 flows for each flow length.
>
> to reorder packets. The nominal RTT is 500ms (equiv. to a GEO system) where
> some packets are reordered every 5, 10, 15 or 20 packets sent. These
> reordered packets have then an RTT=250ms. As you can see, in all cases when
> RACK is enabled, the completion time (time difference between the first
> packet received at destination and the last one) is always worse with RACK.
> The kernel is 4.15.0-60-generic.
>
>
>
> In all these previous experiments the reorder delay is 50% of the RTT, and
> indeed, the RTT is important.
> Using the loopback interface, even with low RTT (i.e. 100ms), and when the
> reordering delay is half the RTT (50ms), RACK still does not perform always
> better than 3DUPACK.
>
>
> implementation does not lead to what we expected. Do you have experienced
> such behavior or do you have measurements that thwart mine ?
>
> Thanks
>
> Emmanuel
>
>
>
>>> it makes no difference..
>> Tel +33 5 61 33 84 85 <+33%205%2061%2033%2084%2085> - Fax (+33) 5 61 33 83 45 <+33%205%2061%2033%2083%2045>
> Tel +33 5 61 33 84 85 <+33%205%2061%2033%2084%2085> - Fax (+33) 5 61 33 83 45 <+33%205%2061%2033%2083%2045>