Subject: [tsvwg] l4s and sce testing on vpns

From: Dave Taht <[email-masked]>
Date: Sun, 21 Feb 2021 17:53:09 -0800

The last time I paid attention to all this, I'd called for more
testing. While a minority opinion at the time, in particular, I'd
wanted a survey of the scope and extent of this vpn decapsulation bug
in linux, to see how serious it was, and to see if it was in any other
operating systems or hardware, particularly firewalling
products like checkpoint, etc. I felt it would take a very, very long
time to flush it out of the field, regardless

Most recently endianess was fixed in this commit:

2867e1eac61016f59b3d730e3f7aa488e186e917

Author: Toke Høiland-Jørgensen <[email-masked]>
Date:   Mon Nov 30 19:37:05 2020 +0100

    inet_ecn: Fix endianness of checksum update when setting ECT(1)

    When adding support for propagating ECT(1) marking in IP headers it seems I
    suffered from endianness-confusion in the checksum update calculation: In
    fact the ECN field is in the *lower* bits of the first 16-bit word of the
    IP header when calculating in network byte order. This means that the
    addition performed to update the checksum field was wrong; let's fix that.

    Fixes: b723748750ec ("tunnel: Propagate ECT(1) when decapsulating as recomme
nded by RFC6040")
    Reported-by: Jonathan Morton <[email-masked]>
    Tested-by: Pete Heist <[email-masked]>
    Signed-off-by: Toke Høiland-Jørgensen <[email-masked]>
    Link: https://lore.kernel.org/r/[email-masked]
    Signed-off-by: Jakub Kicinski <[email-masked]>

Has anyone done a study/survey of the effects of this fix and the
original attempt? Was a detection method for L4S derived for it?

There were more than a few other things I'd wanted tested since last
time. I've read over the SCE and L4S related reports as of, august, or
so of
last year, is there new data?

Have the L4S and SCE codebases been updated to modern kernels so they
can be easily tested again?

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

[email-masked] <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 22 Feb 2021 05:10:34 +0200

> On 22 Feb, 2021, at 3:53 am, Dave Taht <[email-masked]> wrote:
> 

The SCE kernel has been updated to the 5.10 series, which is quite new and is also an LTS series.  That's actually what had us detect the bug you mentioned.  As part of forward porting, we switched over to Toke's new functions instead of some close equivalents we'd rolled ourselves.

Last we checked, the L4S kernel has not been updated recently.  I'm sure we'll hear about it from interested parties if that's incorrect.

> original attempt? Was a detection method for L4S derived for it?

SCE is among a very small and experimental cadre of projects which exercise code related to ECT(1), which probably explains why the bug wasn't noticed for half a year.  We actually had to manually reconfigure Wireshark to verify header checksums as part of troubleshooting, since it doesn't do that by default.

What we noticed was that packets that were marked ECT(1) by a middlebox running the faulty code would be treated as dropped by the receiver, because they had the wrong checksum.  Needless to say, this had a marked effect on throughput and caused AIMD sawtooth behaviour with retransmissions, very unlike what SCE would normally do, and instantly recognisable in a time-series plot of the type Flent produces.  It didn't stop the flows from functioning, however.  When the checksum code was corrected, normal SCE behaviour resumed.

Since L4S sets ECT(1) at origin instead of modifying an existing packet, it probably wouldn't have exercised this bug in a straight connection (though if it had, the effect would have been much more severe and immediately noticeable), because it would calculate the checksum ab-initio over the whole header, not incrementally.  However it might have shown up if the L4S flow were run through a tunnel, as Toke added these functions as part of an overhaul of tunnel decapsulation.

> so of last year, is there new data?

We published some test data for November, and that included some investigation into VPN effects.  As we expected, flow-aware AQMs treat tunnels as a single flow regardless of how many flows are actually in them, and the effect when that includes a mixture of L4S and conventional flows is predictable from previous demonstrations of such mixtures in a single queue AQM.  We take it to mean that single-queue AQM effects are not limited to deployments of native single-queue AQMs, but can also affect FQ and AF AQMs in the presence of tunnelled traffic.

IIRC, we also demonstrated then that Not-ECT conventional flows are affected to the same extent as conventional ECN flows.  This is important, because it shows that users suffering from starvation due to L4S cannot avoid the problem by simply turning off ECN for themselves.  They must either remove the AQM or the L4S traffic.

 - Jonathan Morton