Subject: [tsvwg] I-D Action: draft-ietf-tsvwg-ecn-l4s-id-17.txt

From: [email-masked]
Date: Fri, 21 May 2021 02:09:17 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Transport Area Working Group WG of the IETF.

        Title           : Explicit Congestion Notification (ECN) Protocol for Very Low Queuing Delay (L4S)
        Authors         : Koen De Schepper
                          Bob Briscoe
	Filename        : draft-ietf-tsvwg-ecn-l4s-id-17.txt
	Pages           : 57
	Date            : 2021-05-21

Abstract:
   This specification defines the protocol to be used for a new network
   service called low latency, low loss and scalable throughput (L4S).
   L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
   layer that is similar to the original (or 'Classic') ECN approach,
   except as specified within.  L4S uses 'scalable' congestion control,
   which induces much more frequent control signals from the network and
   it responds to them with much more fine-grained adjustments, so that
   very low (typically sub-millisecond on average) and consistently low
   queuing delay becomes possible for L4S traffic without compromising
   link utilization.  Thus even capacity-seeking (TCP-like) traffic can
   have high bandwidth and very low delay at the same time, even during
   periods of high traffic load.

   The L4S identifier defined in this document distinguishes L4S from
   'Classic' (e.g. TCP-Reno-friendly) traffic.  It gives an incremental
   migration path so that suitably modified network bottlenecks can
   distinguish and isolate existing traffic that still follows the
   Classic behaviour, to prevent it degrading the low queuing delay and
   low loss of L4S traffic.  This specification defines the rules that
   L4S transports and network elements need to follow with the intention
   that L4S flows neither harm each other's performance nor that of
   Classic traffic.  Examples of new active queue management (AQM)
   marking algorithms and examples of new transports (whether TCP-like
   or real-time) are specified separately.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 21 May 2021 10:21:02 +0100

Chairs, list,

We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17 attempting to 
address all the discussion since the last posting just before the 
interim. In particular:
* review comments on a careful read from Gorry and the chairs
* the VPN anti-replay problem
* added an out-of-band test for an RFC3168 ECN AQM in a shared queue.

There are a couple of outstanding discussions, which I'm sure will 
continue on the list, e.g. the role of RFC4774 and whether to remove any 
of Appx C. But it was considered better to get the queued up changes 
out, to re-base the discussions.

This is quite an extensive set of changes, so pls check and pass any 
comments to the list.

Thanks for everyone who is contributing, and particularly to the chairs 
for continuing to referee this all. We've added appropriate thanks in 
the Acks section.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 21 May 2021 23:26:40 +0200

Bob, chairs,

section 6.2 with its, "use two SAs, one for ECT(1) and one for the rest" seems a bit limited since it ignores that VPNs might propagate both DSCPs and ECN bits between the layers, so IMHO a better approach might be to recommend to treat DSCP+ECN bits as one aggregate byte (let's cal it TOS ;) ) as the extra ECT(1)-SA seems to be required for all SAs that already exist to deal with multiple supported DSCPs. So in a sense the recommendation would be to double the number of SAs.

Also:
"and the current draft of DTLS 1.3 says "The receiver	
 	   SHOULD pick a window large enough to handle any plausible reordering,	
 	   which depends on the data rate."  However, in practice, the size of	
 	   the VPN's anti-replay window is not always scaled appropriately."

L4S on a 10 ms path under load can introduce re-ordering in the range of 50 ms (roughly twice the difference between the L- and C-queue delay targets), re-ordering tolerance 5 times of the path RTT seems to be a bit on the high side to expect, no?

Regards
	Sebastian

> On May 21, 2021, at 11:21, Bob Briscoe <[email-masked]> wrote:
> 
> Chairs, list,
> 
> We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17 attempting to address all the discussion since the last posting just before the interim. In particular:
> There are a couple of outstanding discussions, which I'm sure will continue on the list, e.g. the role of RFC4774 and whether to remove any of Appx C. But it was considered better to get the queued up changes out, to re-base the discussions.
> 
> This is quite an extensive set of changes, so pls check and pass any comments to the list.
> 
> Thanks for everyone who is contributing, and particularly to the chairs for continuing to referee this all. We've added appropriate thanks in the Acks section.
> 
> 
> Bob
> 
>

---

From: "Black, David" <[email-masked]>
Date: Fri, 21 May 2021 22:02:03 +0000

> section 6.2 with its, "use two SAs, one for ECT(1) and one for the rest" seems a bit limited

The situation is more severe than that, and it's problematic if there's any non-L4S TCP traffic that uses ECN involved.  The two SAs are actually for 2 ECN values each:

   To avoid this limitation, a VPN ingress participating in the L4S
   experiment SHOULD map packets onto two parallel SAs indexed by the
   lowest significant bit of the IP-ECN field.

That indexing puts Not-ECT [00b] and ECT(0) [10b] packets into one SA, and puts ECT(1) [01b] and CE [11b] packets into another.  Non-L4S TCP uses ECT(0) and CE, hence a flow that carries any congestion indications is split across those two SAs.  ECMP based on SPI (deployed technology) that puts those parallel SAs on different network paths is not assured to preserve packet order across those SAs, and packet reordering is not a good thing to do to a non-L4S TCP flow.

Thanks, --David

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, May 21, 2021 5:27 PM
To: Bob Briscoe
Cc: Gorry Fairhurst; Black, David; Wesley Eddy; [email-masked]
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[EXTERNAL EMAIL] 

Bob, chairs,

Also:

Regards
	Sebastian

>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$ [datatracker[.]ietf[.]org]
>> 
>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$ [datatracker[.]ietf[.]org]
>> 
>> https://urldefense.com/v3/__https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$ [ietf[.]org]
>> 
>> 
>> https://urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$ [ftp[.]ietf[.]org]
>> 
>> 
> 
> -- 
> Bob Briscoe                               https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$ [bobbriscoe[.]net]
>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 23 May 2021 21:54:56 +0100

Sebastian, inline [BB]

> section 6.2 with its, "use two SAs, one for ECT(1) and one for the rest" seems a bit limited since it ignores that VPNs might propagate both DSCPs and ECN bits between the layers, so IMHO a better…

[BB] Yes, we ought to reword it to say that the VPN ingress should /at 
least/ use two SAs indexed on the LSB of the ECN field, and, if it is 
also classifying on DSCPs, it could also consider classifying any low 
latency DSCP(s) with the L4S packets. To avoid the anti-replay problem, 
there would only need to be one SA configured per each degree of queuing 
delay, not one for every ECN x DSCP combination.

We'll have to see how common multiple combinations are in practice. As 
ecn-l4s-id says, L4S with just best efforts...

    "is expected to be the most common and useful
    arrangement.  But, more generally, an operator might choose to
    control bandwidth allocation through a hierarchy of Diffserv PHBs"

So the ECN field could be the only field that gives a delay delta. 
However different networks will have their own view on which technology 
they want to use for low latency. So VPNs will probably need to cater 
for both DSCPs and the ECN field being used for low delay in different 
networks.

> L4S on a 10 ms path under load can introduce re-ordering in the range of 50 ms (roughly twice the difference between the L- and C-queue delay targets), re-ordering tolerance 5 times of the path RTT…

[BB] IMO, the above text that I quoted from the DTLS spec. is 
reasonable, both practically (see below) and in terms of taking 
responsibility for the problem. Beyond its window, the anti-replay 
function presumes a packet is guilty of a replay attack with no 
evidence, purely because it chooses not to hold that amount of evidence. 
Therefore it's proper that it holds a sufficient window of evidence for 
any plausible reordering.

BTW, the C-queue target has never been 25ms. I noticed JM said that 
incorrectly as well recently.
* A default C queue delay target of 15ms has always been recommended in 
aqm-dualq-coupled. Under a heavy load of short and long flow arrivals in 
both the L&C queues, that results in PI2 Qdelay of about 25ms at the 
99%ile or 30ms at the 99.9%ile. We have been considering whether to 
change the default target to 10ms for some time, but not done so yet.
* Low Latency DOCSIS specifies a default C queue delay target of 10ms.

So a replay window allowing for 30ms of packets at the interface rate 
would probably be sufficient.
At 1Gb/s (say) using 1500B packets, that's a replay window of 2500 packets.

Quoting Pete Heist's info here 
https://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled 
:

    "Modern Linux kernels have a default maximum replay window size of
    4096 (|XFRMA_REPLAY_ESN_MAX| in xfrm.h
    <https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/xfrm.h>).
    Wireguard uses a hardcoded value of 8192 with no option for runtime
    configuration, increased from 2048 in May 2020 by this commit
    <https://git.zx2c4.com/wireguard-linux/commit/drivers/net/wireguard?id=c78a0b4a78839d572d8a80f6a62221c0d7843135>."

Regards

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 24 May 2021 00:06:11 +0300

> On 23 May, 2021, at 11:54 pm, Bob Briscoe <[email-masked]> wrote:
> 
> BTW, the C-queue target has never been 25ms. I noticed JM said that incorrectly as well recently.

It looks an awful lot like 25ms in some tests we did earlier this week.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 23 May 2021 23:00:07 +0100

David,
See inline, tagged [BB]...

> That indexing puts Not-ECT [00b] and ECT(0) [10b] packets into one SA, and puts ECT(1) [01b] and CE [11b] packets into another.  Non-L4S TCP uses ECT(0) and CE, hence a flow that carries any conges…

[BB] Yes, this would be a potential problem.

This particular path divergence occurs iff an ECT0 flow is CE-marked 
/prior/ to the VPN ingress:
* In general where the VPN ingress is on the end-system (transport mode) 
that doesn't happen.
* There is however a chance that CE could be introduced before a 'bump 
in the wire' VPN ingress (tunnel mode).

For a VPN participating in the L4S experiment with two SAs, if there was 
any CE before the ingress, we did think carefully about which way to 
classify it:
* If CE goes with ECT0, then if a subsequent L4S node gives ECT0 more 
queue delay than CE, the VPN anti-replay function could discard some 
ECT0 packets.
* If CE goes with ECT1, it avoids anti-replay discard completely. We had 
tried to think of possible problems with splitting Classic flows across 
SAs,  but we hadn't thought of your point where ECMP on the SPI makes CE 
and ECT0 from the same flow taking different paths.

I believe it is still best to classify CE with ECT1, based on the 
following reasoning. I think the scenario would occur with very low 
probability {Note 1}, and if it did, the CE might either arrive a little 
earlier or a little later than data packets around them.
* If earlier, the benefit of ECN would not be lost, but there would be 
an extremely small chance (p_s * p_r * p_N) of an occasional spurious 
re-xmt {Note 2}.
* If later, the chance that a delayed CE would be mistaken for a drop 
and trigger a spurious re-xmt and a congestion response would probably 
be higher, but still very small (p_s * p_r * p_l) {Note 3}. That would 
lose the benefit of ECN but, at least for Classic flows, a single CE is 
meant to trigger a large congestion response as if it was a loss anyway.

I don't want to belittle the point you've made, because this is 
certainly a new problem. However, I do think the probabilities need to 
be put in perspective.

I suspect the ECMP problem with any ECN (3168 or L4S) is likely to be a 
greater concern. You might recall that João Taveira from Fastly pointed 
this out during a tsvwg discussion on ECN back in 2017. This link should 
jump to 14:25 mins into a 2017 NANOG talk from Lorenzo Saino of Fastly 
about how they had to disable ECN negotiation when Apple clients started 
to request ECN in 2015 - because a number of different ECMP vendors 
still hash on the ToS byte for ECMP and load balancing, so the data 
would have been routed to a different server from the handshake:
https://youtu.be/ciClZdwHelU?t=805

Regards

Bob

{Note 1}: Reasoning for scenario occurring with low probability (we'll 
call it p_s):
Usually, an institution provides a tunnel-mode VPN between physically 
secured networks (e.g. between their intranet and an employee's home 
laptop). AQM deployment would be unusual in the interior of corporate 
and institutional intranets, because the bottleneck is more likely 
elsewhere (e.g. in the access link between the site and the Internet). 
However, there are bound to be some cases of tunnel-mode VPNs where the 
traffic arriving could have already been ECN-marked (for instance 
Jonathan Morton's example of gamers providing a VPN end-point on the 
public Internet to hide their own IP address).
Then, to experience this problem, there also has to be some load 
balancing or ECMP after the VPN ingress but before the egress. I'm sure 
that will occur sometime somewhere (let's say with probability p_r). The 
likelihood of the scenario occurring will then be p_s * p_r.

{Note 2}:
* N packets in a row within a Classic flow have to be CE-marked for 
early CE packets to result in a spurious re-xmt, let's say that happens 
with probability p_N. As already explained in Appx B of ecn-l4s-id, N 
was traditionally 3, but RACK is making it larger. And the main sources 
of ECN marking for Classic flows are FQ_CoDel and CAKE, both of which 
take great care not to mark multiple packets in a row. So p_N is going 
to be tiny.
And the probability of a spurious re-xmt with early reordering will be 
the vanishingly small product (p_s * p_r * p_N).

{Note 3}:
* Let's say p_l (for late) is the probability that the reordering 
between the two SAs is sufficient to make each CE packet arrive late 
enough  to be deemed a loss. p_l is unlikely to be as small as p_N, but 
the overall probability of this occurring is still reduced because the 
probability that CE marking precedes that VPN (p_s), and that there's 
routing keyed on flow IDs and SPIs (p_r). So the overall probability of 
a spurious rexmt due to late reordering is (p_s * p_r * p_l).

Bob

> section 6.2 with its, "use two SAs, one for ECT(1) and one for the rest" seems a bit limited since it ignores that VPNs might propagate both DSCPs and ECN bits between the layers, so IMHO a better…

[BB] Yes, we ought to reword it to say that the VPN ingress should use 
two SAs indexed on the LSB of the ECN field, and, if it was also 
classifying on DSCPs, it could also consider classifying any low latency 
DSCP(s) with the L4S packets. To avoid the anti-replay problem, there 
would only need to be one SA configured per each degree of queuing 
delay, not one for every ECN x DSCP combination.

> L4S on a 10 ms path under load can introduce re-ordering in the range of 50 ms (roughly twice the difference between the L- and C-queue delay targets), re-ordering tolerance 5 times of the path RTT…

[BB] The above text that I quoted from the DTLS spec. is reasonable, 
both practically (see below) and in terms of taking responsibility for 
the problem. Beyond its window, the anti-replay function presumes a 
packet is guilty of a replay attack with no evidence, purely because it 
chooses not to hold that amount of evidence. Therefore it's proper that 
it holds a sufficient window of evidence for any plausible reordering.

aqm-dualq-coupled. That results in PI2 Qdelay of about 25ms at the 

would be sufficient.

Regards

Bob

---

From: Pete Heist <[email-masked]>
Date: Mon, 24 May 2021 09:54:59 +0200

Hi Bob/Sebastian,

On Sun, 2021-05-23 at 21:54 +0100, Bob Briscoe wrote:
> [BB] Yes, we ought to reword it to say that the VPN ingress should
> /at least/ use two SAs indexed on the LSB of the ECN field, and, if
> it is also classifying on DSCPs, it could also consider classifying
> any low latency DSCP(s) with the L4S packets. To avoid the anti-
> replay problem, there would only need to be one SA configured per
> each degree of queuing delay, not one for every ECN x DSCP
> combination. 
> 
> We'll have to see how common multiple combinations are in practice.
> As ecn-l4s-id says, L4S with just best efforts...
> However different networks will have their own view on which
> technology they want to use for low latency. So VPNs will probably
> need to cater for both DSCPs and the ECN field being used for low
> delay in different networks.
> 
> 
> > 
> > Also:
> evidence, purely because it chooses not to hold that amount of
> evidence. Therefore it's proper that it holds a sufficient window of
> evidence for any plausible reordering.
> 
> * A default C queue delay target of 15ms has always been recommended
> in aqm-dualq-coupled. Under a heavy load of short and long flow
> arrivals in both the L&C queues, that results in PI2 Qdelay of about
> 25ms at the 99%ile or 30ms at the 99.9%ile. We have been considering
> whether to change the default target to 10ms for some time, but not
> done so yet.
> Quoting Pete Heist's info
> herehttps://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled
> :
> 
> > 4096 (XFRMA_REPLAY_ESN_MAX in xfrm.h). Wireguard uses a hardcoded
> > value of 8192 with no option for runtime configuration, increased
> > from 2048 in May 2020 by this commit."

Just noting that I updated the above text to change "default maximum"
to "fixed maximum" as that's the hard limit. To allow for 30ms, that
would place a tunnel bandwidth limit at ~1.6Gbps until drops occur that
can't be avoided without increasing this xfrm limit.

Regarding the sojourn times, Flent plots TCP RTT using periodic sampled
values from running the 'ss' utility, which gets it from the tcpi_rtt
field of the tcp_info struct. tcpi_rtt is set to tcp_sock->srtt_us >> 3
in tcp.c, and srtt_us is the "smoothed round trip time << 3 in usecs",
according to tcp.h.

Are we sure that we have a good handle on what the peak differences in
sojourn times between C and L can be?

Regards,
Pete

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 24 May 2021 12:54:22 +0200

HI Bob,

> On May 23, 2021, at 22:54, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian, inline [BB]
> 
> [BB] Yes, we ought to reword it to say that the VPN ingress should /at least/ use two SAs indexed on the LSB of the ECN field, and, if it is also classifying on DSCPs, it could also consider classifying any low latency DSCP(s) with the L4S packets.

	[SM] That sounds quite risky, the whole "trick" behind the multiple SAs is to have independent replay-window for each fate-shared group, and a mapping multiple DSCPs and ECT(1)/CE into the same SA requires that no differential re-ordering can happen between these sets. I would probably rather advise to instantiate one SA per each TOS byte code point... sure that is 256 SAs, but it will also squash the issue comprehensively (modulo the fact that a flow's CE packets can be reordered in regards to their non-CE packets, but that is already like that for non replay-protected traffic).

> To avoid the anti-replay problem, there would only need to be one SA configured per each degree of queuing delay, not one for every ECN x DSCP combination. 

	[SM] True, but the number of queueing delay sets is a network function, that for a longer path the endpoints can't reasonably know or discover, so I do not think that this observation is actionable.

> 
> We'll have to see how common multiple combinations are in practice. As ecn-l4s-id says, L4S with just best efforts...

	[SM] Yes, but that is not really discoverable short of trying and then see whether replay errors accumulate, which seems less than ideal for a recommendation IMHO.

> However different networks will have their own view on which technology they want to use for low latency. So VPNs will probably need to cater for both DSCPs and the ECN field being used for low delay in different networks.

	[SM] We seem to agree.

> [BB] IMO, the above text that I quoted from the DTLS spec. is reasonable, both practically (see below) and in terms of taking responsibility for the problem. Beyond its window, the anti-replay function presumes a packet is guilty of a replay attack with no evidence, purely because it chooses not to hold that amount of evidence. Therefore it's proper that it holds a sufficient window of evidence for any plausible reordering.

	[SM] Exactly, my argument is that L45S additional re-ordering is not plausible... or reasonable... but I digress

> BTW, the C-queue target has never been 25ms. I noticed JM said that incorrectly as well recently.

	[SM] Fair enough (it seems I ms-attributed the https://github.com/L4STeam/linux/commit/a2ef76f8da1c9d1b13fa941f55607f3e60d4112e prague_rtt_target with the typical_rtt), but that only changes the numbers slightly the issue is still he same...

> * A default C queue delay target of 15ms has always been recommended in aqm-dualq-coupled.

	[SM] Not quite "always", https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled-00 stated:
"target = 20ms"
But from https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled-01 on it has been and still is set to 15ms...

Which reminds me, I still would like to see the stability analysis and optimization work that lead to that number.

> Under a heavy load of short and long flow arrivals in both the L&C queues, that results in PI2 Qdelay of about 25ms at the 99%ile or 30ms at the 99.9%ile. We have been considering whether to change the default target to 10ms for some time, but not done so yet.

	[SM] Again, it would be nice to see robust analysis behind this parameter selection. 

> So a replay window allowing for 30ms of packets at the interface rate would probably be sufficient.

	[SM] I would respectfully propose to actually run a few experiments before recommending any specific number here. "Probably sufficient" seems not the best basis for recommendations for setting replay-window sizes. 

> At 1Gb/s (say) using 1500B packets, that's a replay window of 2500 packets.

	[SM] It seems sub-optimal to give a fixed packet number recommendation for replay-windows, a formula allowing each user to set the replay-window based on e.g. the known speed of the ingress interface seems more useful (assuming that the interface speed will give an upper bound for achievable rates) as well as expected packet sizes.
I can guarantee that there is not going to be a one-number-fits-all situations solution to sizing replay-windows...

> 
> Quoting Pete Heist's info here https://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled :
> "Modern Linux kernels have a default maximum replay window size of 4096 (XFRMA_REPLAY_ESN_MAX in xfrm.h). Wireguard uses a hardcoded value of 8192 with no option for runtime configuration, increased from 2048 in May 2020 by this commit."

	[SM] I happen to know this, but ipsec's default is still either 32 or 64... I also understand that that was a "solution" that was tested against the worst case re-ordering for a given rate and packetsize assumption, and in 10 years when everybody as 100 Gbps links at home (wishful thinking) that number will need re-visiting.

Regards
	Sebastian

>>> On 21/05/2021 10:09, 
>>> [email-masked]
>>>  wrote:
>>> http://bobbriscoe.net/
> http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 24 May 2021 14:08:18 +0300

> On 24 May, 2021, at 1:54 pm, Sebastian Moeller <[email-masked]> wrote:
> 
> 	[SM] Fair enough (it seems I ms-attributed the https://github.com/L4STeam/linux/commit/a2ef76f8da1c9d1b13fa941f55607f3e60d4112eprague_rtt_target with the typical_rtt), but that only changes the numbers slightly the issue is still he same...
> 
> 

Checking the code in sch_dualpi2.c, I find that the default target delay is indeed 15ms.  However, this still results in a *median* delay of 25ms when used with CUBIC.  Not 99+%ile, but MEDIAN.  It's not my place to figure out how that happens, precisely, just to observe that it does.  So there is basically 25ms of delay *difference* between the L and C queues, and that is what I based my analysis on.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 24 May 2021 13:11:48 +0100

Pete,

On 24/05/2021 08:54, Pete Heist wrote:
>> any low latency DSCP(s) with the L4S packets. To avoid the 
>> anti-replay problem, there would only need to be one SA configured 
>> per each degree of queuing delay, not one for every ECN x DSCP 
>> combination.
>>
> would place a tunnel bandwidth limit at ~1.6Gbps until drops occur 
> that can't be avoided without increasing this xfrm limit.

[BB] Yes, everyone will have to keep scaling the replay window as link 
rates continue to increase. But we all know that older deployments hang 
around. That's why in the latest ecn-l4s-id draft we recommended 
splitting the traffic into two SAs, to side-step the problem altogether. 
But it's still important that implementations scale the anti-replay 
window for a reasonable degree of reordering as well.

>
> Regarding the sojourn times, Flent plots TCP RTT using periodic 
> sampled values from running the 'ss' utility, which gets it from the 
> tcpi_rtt field of the tcp_info struct. tcpi_rtt is set to 
> tcp_sock->srtt_us >> 3 in tcp.c, and srtt_us is the "smoothed round 
> trip time << 3 in usecs", according to tcp.h.

[BB] As I have said for 6 years now, by using the *smoothed* RTT, Flent 
is already filtering out important queue delay variability metrics. And, 
I've also said that, if you want the higher percentiles of delay 
measurements, *sampling* becomes increasingly inaccurate to conclude 
anything about queuing delay percentiles for any reasonable experiment 
duration. For certain you cannot say anything at all about the peak if 
you're only taking samples.

Higher percentile queuing delay is not only important for assessing 
anti-replay. It's important for real-time applications, because they 
have to choose a buffering time before play-out, given any packet that 
arrives after play-out is useless (equivalent to a loss). E.g. a 
real-time app that waits for the 99.9%-ile delay will effectively 
discard 0.1% of packets itself.

That is why, 6 years ago now, we developed a way to measure higher 
percentiles of queuing delay. An AQM measures the sojourn of every 
packet anyway, so we encoded that qDelay into the IPID field of the IP 
header (v4 only), so that it could be decoded at the receiver. We could 
have just logged the measurements at the AQM, but we wanted to have the 
AQM delay data in synch with the e2e measurement data, so we could 
verify that they match, e.g. when debugging.

The code for this is open-source, and available from 
https://github.com/L4STeam/l4sdemo .

>
> Are we sure that we have a good handle on what the *peak* differences 
> in sojourn times between C and L can be?

It's much more useful to measure high percentiles than the peak, which 
might be just a weird outlier.

You will have seen the plots we produce of the higher queuing delay 
percentiles. For instance this one is for the worst-case test we do with 
an extremely demanding traffic pattern in each queue (L and C): 300 web 
flows arriving per second as well as a long-running flow:
https://datatracker.ietf.org/meeting/106/materials/slides-106-tsvwg-sessb-31-tcp-prague-status-of-implementation-and-evaluation-00#page=3

The plot linked above shows 34ms at the 99.9%-ile. I said 30ms based on 
the 99.9%-iles of the most recent test runs we've done (we run all the 
tests after every code change).

Even measuring every packet, you can see that the plot becomes less 
regular by the 99.999%- ile, because we only run the experiments long 
enough to have about an order of magnitude more packets than the highest 
percentile we're looking for (e.g. 10^6 packets for a 5-nines 
percentile). If you were sampling, you would have to run an experiment 
for days to get reliable percentiles like this.

Bob

>
> Regards,
> Pete
>
>> Regards
>>
>>
>>
>> Bob
>>
>>
>>> Regards
>>> 	Sebastian
>>>
>>>> On May 21, 2021, at 11:21, Bob Briscoe<[email-masked]>  wrote:
>>>> Chairs, list,
>>>> On 21/05/2021 10:09,[email-masked]  wrote:
>>>>
>>>> Bob Briscoehttp://bobbriscoe.net/
>>
>

---

From: "Black, David" <[email-masked]>
Date: Tue, 25 May 2021 20:22:21 +0000

Two comments:

[1] This VPN discussion appears to be assuming a remote access VPN, so it does not align well with site-to-site VPNs:

> This particular path divergence occurs iff an ECT0 flow is CE-marked /prior/ to the VPN ingress:
> * In general where the VPN ingress is on the end-system (transport mode) that doesn't happen.
> * There is however a chance that CE could be introduced before a 'bump in the wire' VPN ingress (tunnel mode).

For site-to-site VPNs, the last line above is a significant understatement.  A scenario that helps to understand this is ROBO, specifically a sizeable remote office or branch office such as a large retail store, where the remote or branch VPN gateway is a potential bottleneck node for outbound traffic courtesy of fan-in from multiple end systems and the LAN having more bandwidth than the (purchased) VPN bandwidth.

[2] I think the whole approach of parallel SAs is not a good idea – it would be better to recommend larger replay windows, perhaps much larger, and/or use DSCPs to direct traffic to parallel SAs (which would disambiguate CE across L4S and non-L4S traffic).

Thanks, --David

From: Bob Briscoe <[email-masked]>
Sent: Sunday, May 23, 2021 6:00 PM
To: Black, David; Gorry Fairhurst; Wesley Eddy
Cc: [email-masked]
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[EXTERNAL EMAIL]
David,

This particular path divergence occurs iff an ECT0 flow is CE-marked /prior/ to the VPN ingress:
* In general where the VPN ingress is on the end-system (transport mode) that doesn't happen.
* There is however a chance that CE could be introduced before a 'bump in the wire' VPN ingress (tunnel mode).

For a VPN participating in the L4S experiment with two SAs, if there was any CE before the ingress, we did think carefully about which way to classify it:
* If CE goes with ECT0, then if a subsequent L4S node gives ECT0 more queue delay than CE, the VPN anti-replay function could discard some ECT0 packets.
* If CE goes with ECT1, it avoids anti-replay discard completely. We had tried to think of possible problems with splitting Classic flows across SAs,  but we hadn't thought of your point where ECMP on the SPI makes CE and ECT0 from the same flow taking different paths.

I believe it is still best to classify CE with ECT1, based on the following reasoning. I think the scenario would occur with very low probability {Note 1}, and if it did, the CE might either arrive a little earlier or a little later than data packets around them.
* If earlier, the benefit of ECN would not be lost, but there would be an extremely small chance (p_s * p_r * p_N) of an occasional spurious re-xmt {Note 2}.
* If later, the chance that a delayed CE would be mistaken for a drop and trigger a spurious re-xmt and a congestion response would probably be higher, but still very small (p_s * p_r * p_l) {Note 3}. That would lose the benefit of ECN but, at least for Classic flows, a single CE is meant to trigger a large congestion response as if it was a loss anyway.

I don't want to belittle the point you've made, because this is certainly a new problem. However, I do think the probabilities need to be put in perspective.

I suspect the ECMP problem with any ECN (3168 or L4S) is likely to be a greater concern. You might recall that João Taveira from Fastly pointed this out during a tsvwg discussion on ECN back in 2017. This link should jump to 14:25 mins into a 2017 NANOG talk from Lorenzo Saino of Fastly about how they had to disable ECN negotiation when Apple clients started to request ECN in 2015 - because a number of different ECMP vendors still hash on the ToS byte for ECMP and load balancing, so the data would have been routed to a different server from the handshake:
    https://youtu.be/ciClZdwHelU?t=805 [youtu.be]<https://urldefense.com/v3/__https:/youtu.be/ciClZdwHelU?t=805__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtmUFhVbc$>

Regards

Bob

{Note 1}: Reasoning for scenario occurring with low probability (we'll call it p_s):
Usually, an institution provides a tunnel-mode VPN between physically secured networks (e.g. between their intranet and an employee's home laptop). AQM deployment would be unusual in the interior of corporate and institutional intranets, because the bottleneck is more likely elsewhere (e.g. in the access link between the site and the Internet). However, there are bound to be some cases of tunnel-mode VPNs where the traffic arriving could have already been ECN-marked (for instance Jonathan Morton's example of gamers providing a VPN end-point on the public Internet to hide their own IP address).
Then, to experience this problem, there also has to be some load balancing or ECMP after the VPN ingress but before the egress. I'm sure that will occur sometime somewhere (let's say with probability p_r). The likelihood of the scenario occurring will then be p_s * p_r.

{Note 2}:
* N packets in a row within a Classic flow have to be CE-marked for early CE packets to result in a spurious re-xmt, let's say that happens with probability p_N. As already explained in Appx B of ecn-l4s-id, N was traditionally 3, but RACK is making it larger. And the main sources of ECN marking for Classic flows are FQ_CoDel and CAKE, both of which take great care not to mark multiple packets in a row. So p_N is going to be tiny.
And the probability of a spurious re-xmt with early reordering will be the vanishingly small product (p_s * p_r * p_N).

{Note 3}:
* Let's say p_l (for late) is the probability that the reordering between the two SAs is sufficient to make each CE packet arrive late enough  to be deemed a loss. p_l is unlikely to be as small as p_N, but the overall probability of this occurring is still reduced because the probability that CE marking precedes that VPN (p_s), and that there's routing keyed on flow IDs and SPIs (p_r). So the overall probability of a spurious rexmt due to late reordering is (p_s * p_r * p_l).

Bob

Thanks, --David

-----Original Message-----

From: Sebastian Moeller <[email-masked]><mailto:[email-masked]>

To: Bob Briscoe

Cc: Gorry Fairhurst; Black, David; Wesley Eddy; [email-masked]<mailto:[email-masked]>

[EXTERNAL EMAIL]

Bob, chairs,

[BB] Yes, we ought to reword it to say that the VPN ingress should use two SAs indexed on the LSB of the ECN field, and, if it was also classifying on DSCPs, it could also consider classifying any low latency DSCP(s) with the L4S packets. To avoid the anti-replay problem, there would only need to be one SA configured per each degree of queuing delay, not one for every ECN x DSCP combination.

Also:

[BB] The above text that I quoted from the DTLS spec. is reasonable, both practically (see below) and in terms of taking responsibility for the problem. Beyond its window, the anti-replay function presumes a packet is guilty of a replay attack with no evidence, purely because it chooses not to hold that amount of evidence. Therefore it's proper that it holds a sufficient window of evidence for any plausible reordering.

BTW, the C-queue target has never been 25ms. I noticed JM said that incorrectly as well recently.
* A default C queue delay target of 15ms has always been recommended in aqm-dualq-coupled. That results in PI2 Qdelay of about 25ms at the 99%ile or 30ms at the 99.9%ile. We have been considering whether to change the default target to 10ms for some time, but not done so yet.
* Low Latency DOCSIS specifies a default C queue delay target of 10ms.

So a replay window allowing for 30ms of packets at the interface rate would be sufficient.
At 1Gb/s (say) using 1500B packets, that's a replay window of 2500 packets.

Quoting Pete Heist's info here https://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled [github.com]<https://urldefense.com/v3/__https:/github.com/heistp/l4s-tests/*dropped-packets-for-tunnels-with-replay-protection-enabled__;Iw!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8Hywtt-9Grg1$> :
"Modern Linux kernels have a default maximum replay window size of 4096 (XFRMA_REPLAY_ESN_MAX in xfrm.h [elixir.bootlin.com]<https://urldefense.com/v3/__https:/elixir.bootlin.com/linux/latest/source/include/uapi/linux/xfrm.h__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8Hywtu2jvbJy$>). Wireguard uses a hardcoded value of 8192 with no option for runtime configuration, increased from 2048 in May 2020 by this commit [git.zx2c4.com]<https://urldefense.com/v3/__https:/git.zx2c4.com/wireguard-linux/commit/drivers/net/wireguard?id=c78a0b4a78839d572d8a80f6a62221c0d7843135__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtrpY39m8$>."
Regards

Bob

Regards

  Sebastian

On May 21, 2021, at 11:21, Bob Briscoe <[email-masked]><mailto:[email-masked]> wrote:

Chairs, list,

Bob

On 21/05/2021 10:09, [email-masked]<mailto:[email-masked]> wrote:

                          Bob Briscoe

 Pages           : 57

 Date            : 2021-05-21

Abstract:

https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$> [datatracker[.]ietf[.]org]

https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$> [datatracker[.]ietf[.]org]

https://urldefense.com/v3/__https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$> [ietf[.]org]

https://urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$<https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$> [ftp[.]ietf[.]org]

--

Bob Briscoe                               https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$> [bobbriscoe[.]net]

--

Bob Briscoe                               http://bobbriscoe.net/ [bobbriscoe.net]<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtoRgTac-$>

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 26 May 2021 15:22:18 +0000

Hi Jonathan,

 > Checking the code in sch_dualpi2.c, I find that the default target delay
 > is indeed 15ms.  However, this still results in a *median* delay of 25ms
 > when used with CUBIC.  Not 99+%ile, but MEDIAN.  It's not my place to
 > figure out how that happens, precisely, just to observe that it does.  So
 > there is basically 25ms of delay *difference* between the L and C queues,
 > and that is what I based my analysis on.

Given the open-source code has not been touched in over a year, this seems
surprising. Could you point to where you see such behavior? Given the large
amount of results over various repository, I have probably missed the
experiment you are referring to here.

A glance at your one-year old results does not show this (e.g. [1]).
The secondary Y axis on your more recent results seems off so is not
helping here (e.g., [2]). However, the reported tcp_metrics after
experiments look fine, e.g., 34ms on a 20ms base rtt [3], 21ms on a
10ms base rtt [4]. But I am not sure I'd trust the metric cache seeing
how many teardown logs only report the base RTT as srtt.

Best,
Olivier

[1] http://sce.dnsmgr.net/results/ect1-2020-04-23-final/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-dualpi2-20ms_tcp_delivery_with_rtt.svg
[2] http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-dualpi2-50Mbit-20ms-20ms_tcp_delivery_with_rtt.svg
[3] http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s1-rttfair/batch-l4s-s1-rttfair-ns-cubic-vs-cubic-dualpi2-50Mbit-20ms-20ms.teardown.log
[4] http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s1-rttfair/batch-l4s-s1-rttfair-ns-cubic-vs-cubic-dualpi2-50Mbit-10ms-10ms.teardown.log

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 26 May 2021 18:16:22 +0100

David,

On 25/05/2021 21:22, Black, David wrote:
>
> Two comments:
>
> [1] This VPN discussion appears to be assuming a remote access VPN, so 
> it does not align well with site-to-site VPNs:
>
> > * There is however a chance that CE could be introduced before a 
> 'bump in the wire' VPN ingress (tunnel mode).
>
> For site-to-site VPNs, the last line above is a significant 
> understatement.  A scenario that helps to understand this is ROBO, 
> specifically a sizeable remote office or branch office such as a large 
> retail store, where the remote or branch VPN gateway is a potential 
> bottleneck node for outbound traffic courtesy of fan-in from multiple 
> end systems and the LAN having more bandwidth than the (purchased) VPN 
> bandwidth.
>

[BB] Wouldn't that particular bottleneck apply marking at the egress of 
the VPN gateway after encap?

> [2] I think the whole approach of parallel SAs is not a good idea – it 
> would be better to recommend larger replay windows, perhaps much 
> larger, and/or use DSCPs to direct traffic to parallel SAs (which 
> would disambiguate CE across L4S and non-L4S traffic).
>

[BB] You haven't given any reasons.

Bob

> Thanks, --David
>
> *From:* Bob Briscoe <[email-masked]>
> *Sent:* Sunday, May 23, 2021 6:00 PM
> *To:* Black, David; Gorry Fairhurst; Wesley Eddy
> *Cc:* [email-masked]
> *Subject:* Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17
>
> [EXTERNAL EMAIL]
>
> David,
> For a VPN participating in the L4S experiment with two SAs, if there 
> was any CE before the ingress, we did think carefully about which way 
> to classify it:
> * If CE goes with ECT1, it avoids anti-replay discard completely. We 
> had tried to think of possible problems with splitting Classic flows 
> across SAs,  but we hadn't thought of your point where ECMP on the SPI 
> makes CE and ECT0 from the same flow taking different paths.
>
> probability {Note 1}, and if it did, the CE might either arrive a 
> little earlier or a little later than data packets around them.
> lose the benefit of ECN but, at least for Classic flows, a single CE 
> is meant to trigger a large congestion response as if it was a loss 
> anyway.
>
> I suspect the ECMP problem with any ECN (3168 or L4S) is likely to be 
> a greater concern. You might recall that João Taveira from Fastly 
> pointed this out during a tsvwg discussion on ECN back in 2017. This 
> link should jump to 14:25 mins into a 2017 NANOG talk from Lorenzo 
> Saino of Fastly about how they had to disable ECN negotiation when 
> Apple clients started to request ECN in 2015 - because a number of 
> different ECMP vendors still hash on the ToS byte for ECMP and load 
> balancing, so the data would have been routed to a different server 
> from the handshake:
> https://youtu.be/ciClZdwHelU?t=805 [youtu.be] 
> <https://urldefense.com/v3/__https:/youtu.be/ciClZdwHelU?t=805__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtmUFhVbc$>
>
> Regards
>
>
>
> Bob
>
> However, there are bound to be some cases of tunnel-mode VPNs where 
> the traffic arriving could have already been ECN-marked (for instance 
> balancing or ECMP after the VPN ingress but before the egress. I'm 
> sure that will occur sometime somewhere (let's say with probability 
> p_r). The likelihood of the scenario occurring will then be p_s * p_r.
>
> {Note 2}:
> early CE packets to result in a spurious re-xmt, let's say that 
> happens with probability p_N. As already explained in Appx B of 
> ecn-l4s-id, N was traditionally 3, but RACK is making it larger. And 
> the main sources of ECN marking for Classic flows are FQ_CoDel and 
> CAKE, both of which take great care not to mark multiple packets in a 
> row. So p_N is going to be tiny.
> enough  to be deemed a loss. p_l is unlikely to be as small as p_N, 
> but the overall probability of this occurring is still reduced because 
> the probability that CE marking precedes that VPN (p_s), and that 
> there's routing keyed on flow IDs and SPIs (p_r). So the overall 
> probability of a spurious rexmt due to late reordering is (p_s * p_r * 
> p_l).
>
>
>
> Bob
>
>
>     Thanks, --David
>
>     From: Sebastian Moeller<[email-masked]>  <mailto:[email-masked]>  
>
>     Cc: Gorry Fairhurst; Black, David; Wesley Eddy;[email-masked]  <mailto:[email-masked]>
>
> classifying on DSCPs, it could also consider classifying any low 
> latency DSCP(s) with the L4S packets. To avoid the anti-replay 
> problem, there would only need to be one SA configured per each degree 
> of queuing delay, not one for every ECN x DSCP combination.
>
>
>     Also:
>
> packet is guilty of a replay attack with no evidence, purely because 
> it chooses not to hold that amount of evidence. Therefore it's proper 
> that it holds a sufficient window of evidence for any plausible 
> reordering.
>
> in aqm-dualq-coupled. That results in PI2 Qdelay of about 25ms at the 
> [github.com] 
> <https://urldefense.com/v3/__https:/github.com/heistp/l4s-tests/*dropped-packets-for-tunnels-with-replay-protection-enabled__;Iw!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8Hywtt-9Grg1$> 
> :
>
>     4096 (|XFRMA_REPLAY_ESN_MAX| in xfrm.h [elixir.bootlin.com]
>     <https://urldefense.com/v3/__https:/elixir.bootlin.com/linux/latest/source/include/uapi/linux/xfrm.h__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8Hywtu2jvbJy$>).
>     Wireguard uses a hardcoded value of 8192 with no option for
>     runtime configuration, increased from 2048 in May 2020 by this
>     commit [git.zx2c4.com]
>     <https://urldefense.com/v3/__https:/git.zx2c4.com/wireguard-linux/commit/drivers/net/wireguard?id=c78a0b4a78839d572d8a80f6a62221c0d7843135__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtrpY39m8$>."
>
> Regards
>
>
>
> Bob
>
>
>     Regards
>
>        Sebastian
>
>         On May 21, 2021, at 11:21, Bob Briscoe<[email-masked]>  <mailto:[email-masked]>  wrote:
>
>         Chairs, list,
>
>         On 21/05/2021 10:09,[email-masked]  <mailto:[email-masked]>  wrote:
>
>             https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$>  [datatracker[.]ietf[.]org]
>
>             https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$>  [datatracker[.]ietf[.]org]
>
>             https://urldefense.com/v3/__https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$  <https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$>  [ietf[.]org]
>
>             https://urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$  <https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$>  [ftp[.]ietf[.]org]
>
>         -- 
>
>         Bob Briscoehttps://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$>  [bobbriscoe[.]net]
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/ [bobbriscoe.net]  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!y0AavGemYqNsy2Urub76LQ2eIuIinYWTadbh2ddv0r80YQabjQCK8HywtoRgTac-$>

---

From: "Black, David" <[email-masked]>
Date: Wed, 26 May 2021 19:34:12 +0000

Inline … --David

From: Bob Briscoe <[email-masked]>
Sent: Wednesday, May 26, 2021 1:16 PM
To: Black, David; Gorry Fairhurst; Wesley Eddy
Cc: [email-masked]
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[EXTERNAL EMAIL]
David,

[BB] Wouldn't that particular bottleneck apply marking at the egress of the VPN gateway after encap?
[David>] That depends on the details of how the VPN is integrated with the network, e.g., if queue is upstream of encap.

[David>] Sure:

  *   See RFC 7657 for general discussion about why variable QoS treatment within a single reliable transport protocol session is not a good idea – that discussion is couched in Diffserv terms, but it's applicable to L4S ECN-based service differentiation.
  *   Asking VPNs to classify traffic based on ECN field values is also asking for the possibility of discard of traffic marked with some of those values, which is not a good place to go given the desire for the ECN field to remain end-to-end.
  *   Using DSCPs is a "reduce to previous case" approach that doesn't involve modification of any security specs and aligns with existing VPN implementations that are cognizant of DSCPs.
  *   Larger replay windows are a lower-effort approach that ought to apply to deployed VPNs whose replay window is configurable.  In practice, this is the place to start if L4S causes VPN problems.

Bob

Thanks, --David

From: Bob Briscoe <[email-masked]><mailto:[email-masked]>
Sent: Sunday, May 23, 2021 6:00 PM
To: Black, David; Gorry Fairhurst; Wesley Eddy
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[EXTERNAL EMAIL]
David,

Regards

Bob

{Note 2}:

{Note 3}:

Bob

Thanks, --David

-----Original Message-----

To: Bob Briscoe

[EXTERNAL EMAIL]

Bob, chairs,

Also:

Bob

Regards

  Sebastian

Chairs, list,

Bob

                          Bob Briscoe

 Pages           : 57

 Date            : 2021-05-21

Abstract:

--

--

--

Bob Briscoe                               http://bobbriscoe.net/ [bobbriscoe.net]<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!z5o0nAykh9F3vkOXqwkB6EByr7rXTsxWZsk8cnefqdJrD1IFy5L0W8qh71SgIKm1$>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 27 May 2021 08:46:47 +0000

I agree that larger (adaptive) replay window is the main solution, and removes all other needs. The adaptive window could easily grow based on the gap that an early packet is creating, allowing the next packets to fill the gap (optionally shrinking the window afterwards back to gradually smaller values).

As an alternative, another smarter reordering-safe mechanism would help as well. I think a symmetric window that also drops (or “keeps aside without moving the window”) sporadic early packets, will perform better… It would drop (or delay) only the expedited CE(s).

This all avoids the need for different SA’s and/or DCSPs…

Koen.

From: tsvwg <[email-masked]> On Behalf Of Black, David
Sent: Wednesday, May 26, 2021 9:34 PM
To: Bob Briscoe <[email-masked]>; Gorry Fairhurst <[email-masked]>; Wesley Eddy <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

Inline … --David

From: Bob Briscoe <[email-masked]<mailto:[email-masked]>>

[EXTERNAL EMAIL]
David,

[David>] Sure:

Bob

Thanks, --David

[EXTERNAL EMAIL]
David,

Regards

Bob

{Note 2}:

{Note 3}:

Bob

Thanks, --David

-----Original Message-----

To: Bob Briscoe

[EXTERNAL EMAIL]

Bob, chairs,

Also:

Bob

Regards

  Sebastian

Chairs, list,

Bob

                          Bob Briscoe

 Pages           : 57

 Date            : 2021-05-21

Abstract:

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 27 May 2021 11:50:47 +0100

David, see [BB]

On 26/05/2021 20:34, Black, David wrote:
> Inline …--David
> *Sent:* Wednesday, May 26, 2021 1:16 PM
>     [1] This VPN discussion appears to be assuming a remote access
>     VPN, so it does not align well with site-to-site VPNs:
>
>     specifically a sizeable remote office or branch office such as a
>     large retail store, where the remote or branch VPN gateway is a
>     potential bottleneck node for outbound traffic courtesy of fan-in
>     from multiple end systems and the LAN having more bandwidth than
>     the (purchased) VPN bandwidth.
>
>
> /[David>] That depends on the details of how the VPN is integrated 
> with the network, e.g., if queue is upstream of encap./
>

[BB2] And the details are...
...that the queue into the off-site link is always after encapsulation. 
You have to queue the packets /after/ you've encrypted and encapsulated 
them, otherwise you don't know how much space they will take up in the 
queue. And you can't apply active queue management if you don't know how 
much is in the queue.

> //
>
>     [2] I think the whole approach of parallel SAs is not a good idea
>     – it would be better to recommend larger replay windows, perhaps
>     much larger, and/or use DSCPs to direct traffic to parallel SAs
>     (which would disambiguate CE across L4S and non-L4S traffic).
>
>
> *//*
> /[David>] Sure:/
>   * /See RFC 7657 for general discussion about why variable QoS
>     treatment within a single reliable transport protocol session is
>     not a good idea – that discussion is couched in Diffserv terms,
>     but it's applicable to L4S ECN-based service differentiation./
>

[BB2] Yes, I know and agree with that. Your argument here though is not 
against separating Classic from L4S flows into separate SAs. I assume 
it's solely about whether to risk separating packets that are already 
CE-marked into another SA because they might have started life as ECT0...

The original L4S codepoint scheme was chosen on the basis that CE 
ambiguity led to spurious retransmits in a vanishingly small proportion 
of cases. So, an 'absolutist' argument against any risk at all of 
reordering doesn't really take us forward. The question here is not 
whether there'll be /some/ reordering. We know there will. The question 
is how likely it is. So the argument needs to be against my points about 
low likelihood.

You've now pointed out that routing per SPI can lead to either early /or 
late/ reordering of CE packets, which is a new problem. In {Note 3} 
(still quoted below) I characterized the probability of late reordering 
/occurring/ as likely to be higher than without a VPN. But the 
/magnitude/ of the late reordering is nothing like as great as the max 
difference between Classic and L4S. That's significant given transports 
are rapidly becoming tolerant to a greater degree of reordering due to 
RACK in TCP and QUIC. So the question is whether there is a flaw in 
{Note 3}.

>
>   * /Asking VPNs to classify traffic based on ECN field values is also
>     asking for the possibility of discard of traffic marked with some
>     of those values, which is not a good place to go given the desire
>     for the ECN field to remain end-to-end./
>

[BB2] There must be something in your head that's not in mine here. Can 
you explain more fully where this discard comes from, and where the risk 
of losing the e2e property of the ECN field comes from?

>
>   * /Using DSCPs is a "reduce to previous case" approach that doesn't
>     involve modification of any security specs and aligns with
>     existing VPN implementations that are cognizant of DSCPs./
>

[BB2] That's true. Which is why the draft suggests using DSCPs if the 
VPN hasn't been or can't be modified yet to classify on ECN. But DSCPs 
are, by definition, per domain and not reliable. So the ultimate 
solution to make the problem go away with certainty would be to classify 
on ECN.

>
>   * /Larger replay windows are a lower-effort approach that ought to
>     apply to deployed VPNs whose replay window is configurable.  In
>     practice, this is the place to start if L4S causes VPN problems./
>

[BB2] I don't disagree with making sure the replay window is large 
enough as well. But the IPsec and DTLS specs already say that, and it 
clearly doesn't always happen. As someone already said (Sebastian, 
Jonathan?), relying on a variable that gets updated only every so often, 
means we'll get problems every so often.

In summary, the answer is a combination of the three solutions (ECN, 
DSCP and sufficient window). But classification by ECN is the robust 
solution and therefore the one stated as a SHOULD. But that depends on 
my argument about the low likelihood of reordering.

Bob

>     <mailto:[email-masked]>
>     *Cc:* [email-masked] <mailto:[email-masked]>
>     For a VPN participating in the L4S experiment with two SAs, if
>     there was any CE before the ingress, we did think carefully about
>     which way to classify it:
>     * If CE goes with ECT0, then if a subsequent L4S node gives ECT0
>     more queue delay than CE, the VPN anti-replay function could
>     discard some ECT0 packets.
>     * If CE goes with ECT1, it avoids anti-replay discard completely.
>     We had tried to think of possible problems with splitting Classic
>     flows across SAs,  but we hadn't thought of your point where ECMP
>     on the SPI makes CE and ECT0 from the same flow taking different
>     paths.
>
>     following reasoning. I think the scenario would occur with very
>     low probability {Note 1}, and if it did, the CE might either
>     arrive a little earlier or a little later than data packets around
>     them.
>     * If earlier, the benefit of ECN would not be lost, but there
>     would be an extremely small chance (p_s * p_r * p_N) of an
>     occasional spurious re-xmt {Note 2}.
>     * If later, the chance that a delayed CE would be mistaken for a
>     drop and trigger a spurious re-xmt and a congestion response would
>     probably be higher, but still very small (p_s * p_r * p_l) {Note
>     3}. That would lose the benefit of ECN but, at least for Classic
>     flows, a single CE is meant to trigger a large congestion response
>     as if it was a loss anyway.
>
>     certainly a new problem. However, I do think the probabilities
>     need to be put in perspective.
>
>     I suspect the ECMP problem with any ECN (3168 or L4S) is likely to
>     be a greater concern. You might recall that João Taveira from
>     Fastly pointed this out during a tsvwg discussion on ECN back in
>     2017. This link should jump to 14:25 mins into a 2017 NANOG talk
>     from Lorenzo Saino of Fastly about how they had to disable ECN
>     negotiation when Apple clients started to request ECN in 2015 -
>     because a number of different ECMP vendors still hash on the ToS
>     byte for ECMP and load balancing, so the data would have been
>     routed to a different server from the handshake:
>     Usually, an institution provides a tunnel-mode VPN between
>     physically secured networks (e.g. between their intranet and an
>     employee's home laptop). AQM deployment would be unusual in the
>     interior of corporate and institutional intranets, because the
>     bottleneck is more likely elsewhere (e.g. in the access link
>     between the site and the Internet). However, there are bound to be
>     some cases of tunnel-mode VPNs where the traffic arriving could
>     have already been ECN-marked (for instance Jonathan Morton's
>     example of gamers providing a VPN end-point on the public Internet
>     to hide their own IP address).
>     sure that will occur sometime somewhere (let's say with
>     probability p_r). The likelihood of the scenario occurring will
>     then be p_s * p_r.
>
>     {Note 2}:
>     * N packets in a row within a Classic flow have to be CE-marked
>     for early CE packets to result in a spurious re-xmt, let's say
>     that happens with probability p_N. As already explained in Appx B
>     of ecn-l4s-id, N was traditionally 3, but RACK is making it
>     larger. And the main sources of ECN marking for Classic flows are
>     FQ_CoDel and CAKE, both of which take great care not to mark
>     multiple packets in a row. So p_N is going to be tiny.
>     And the probability of a spurious re-xmt with early reordering
>     will be the vanishingly small product (p_s * p_r * p_N).
>
>     {Note 3}:
>     between the two SAs is sufficient to make each CE packet arrive
>     late enough  to be deemed a loss. p_l is unlikely to be as small
>     as p_N, but the overall probability of this occurring is still
>     reduced because the probability that CE marking precedes that VPN
>     (p_s), and that there's routing keyed on flow IDs and SPIs (p_r).
>     So the overall probability of a spurious rexmt due to late
>     reordering is (p_s * p_r * p_l).
>
>
>
>     Bob
>
>
>
>           
>
>         Thanks, --David
>
>           
>
>     use two SAs indexed on the LSB of the ECN field, and, if it was
>     also classifying on DSCPs, it could also consider classifying any
>     low latency DSCP(s) with the L4S packets. To avoid the anti-replay
>     problem, there would only need to be one SA configured per each
>     degree of queuing delay, not one for every ECN x DSCP combination.
>
>
>
>           
>
>           
>
>         Also:
>
>     [BB] The above text that I quoted from the DTLS spec. is
>     evidence. Therefore it's proper that it holds a sufficient window
>     of evidence for any plausible reordering.
>
>     * A default C queue delay target of 15ms has always been
>     recommended in aqm-dualq-coupled. That results in PI2 Qdelay of
>     about 25ms at the 99%ile or 30ms at the 99.9%ile. We have been
>     considering whether to change the default target to 10ms for some
>     time, but not done so yet.
>         "Modern Linux kernels have a default maximum replay window
>         size of 4096 (|XFRMA_REPLAY_ESN_MAX| in xfrm.h
>         [elixir.bootlin.com]
> Bob Briscoehttp://bobbriscoe.net/ [bobbriscoe.net]  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!z5o0nAykh9F3vkOXqwkB6EByr7rXTsxWZsk8cnefqdJrD1IFy5L0W8qh71SgIKm1$>

---

From: Pete Heist <[email-masked]>
Date: Fri, 28 May 2021 12:03:20 +0200

Hi Koen,

On Thu, 2021-05-27 at 08:46 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> I agree that larger (adaptive) replay window is the main solution,
> and removes all other needs. The adaptive window could easily grow
> based on the gap that an early packet is creating, allowing the next
> packets to fill the gap (optionally shrinking the window afterwards
> back to gradually smaller values).
>  
> As an alternative, another smarter reordering-safe mechanism would
> help as well. I think a symmetric window that also drops (or “keeps
> aside without moving the window”) sporadic early packets, will
> perform better… It would drop (or delay) only the expedited CE(s).

I'm just double checking, we're on the same page that the reordering
that affects tunnel anti-replay mechanisms doesn't require CE marks,
right? It isn't just expedited CEs that can advance the window past the
tunnel's valid traffic, but any traffic marked ECT(1), legitimately or
not. David wrote a crystal clear explanation earlier, in which CE marks
are only mentioned in order to clear up that prior misconception:

https://mailarchive.ietf.org/arch/msg/tsvwg/PEVCuDJfbrel74ud8kNJtmVwhHA/

Regards,
Pete

> To: Bob Briscoe <[email-masked]>; Gorry Fairhurst
> <[email-masked]>; Wesley Eddy <[email-masked]>
> Cc: [email-masked]
> > [1] This VPN discussion appears to be assuming a remote access VPN,
> > so it does not align well with site-to-site VPNs:
> >  
> [David>] That depends on the details of how the VPN is integrated
> with the network, e.g., if queue is upstream of encap.
>  
> > [2] I think the whole approach of parallel SAs is not a good idea –
> > it would be better to recommend larger replay windows, perhaps much
> * See RFC 7657 for general discussion about why variable QoS
> treatment
> within a single reliable transport protocol session is not a good
> idea – that discussion is couched in Diffserv terms, but it's
> applicable to L4S ECN-based service differentiation.
> * Asking VPNs to classify traffic based on ECN field values is also
> asking for the possibility of discard of traffic marked with some of
> those values, which is not a good place to go given the desire for
> the ECN field to remain end-to-end.
> * Using DSCPs is a "reduce to previous case" approach that doesn't
> involve modification of any security specs and aligns with existing
> VPN implementations that are cognizant of DSCPs.
> * Larger replay windows are a lower-effort approach that ought to
> practice, this is the place to start if L4S causes VPN problems.
> 
> 
> Bob
> 
> >  
> > Thanks, --David
> >  
> > > The situation is more severe than that, and it's problematic if
> there's any non-L4S TCP traffic that uses ECN involved.  The two SAs
> are actually for 2 ECN values each:
> > >  
> > > That indexing puts Not-ECT [00b] and ECT(0) [10b] packets into
> one SA, and puts ECT(1) [01b] and CE [11b] packets into another. 
> Non-L4S TCP uses ECT(0) and CE, hence a flow that carries any
> congestion indications is split across those two SAs.  ECMP based on
> SPI (deployed technology) that puts those parallel SAs on different
> network paths is not assured to preserve packet order across those
> SAs, and packet reordering is not a good thing to do to a non-L4S TCP
> flow.
> > 
> > * If earlier, the benefit of ECN would not be lost, but there would
> > be an extremely small chance (p_s * p_r * p_N) of an occasional
> > spurious re-xmt {Note 2}.
> > certainly a new problem. However, I do think the probabilities need
> > to be put in perspective.
> > 
> > ecn-l4s-id, N was traditionally 3, but RACK is making it larger.
> > And the main sources of ECN marking for Classic flows are FQ_CoDel
> > and CAKE, both of which take great care not to mark multiple
> > packets in a row. So p_N is going to be tiny.
> > And the probability of a spurious re-xmt with early reordering will
> > be the vanishingly small product (p_s * p_r * p_N).
> > 
> > {Note 3}:
> > late enough  to be deemed a loss. p_l is unlikely to be as small as
> > p_N, but the overall probability of this occurring is still reduced
> > because the probability that CE marking precedes that VPN (p_s),
> > and that there's routing keyed on flow IDs and SPIs (p_r). So the
> > overall probability of a spurious rexmt due to late reordering is
> > (p_s * p_r * p_l).
> > 
> > 
> > 
> > Bob
> > 
> > 
> > >  
> > > Thanks, --David
> > >  
> > > section 6.2 with its, "use two SAs, one for ECT(1) and one for
> the rest" seems a bit limited since it ignores that VPNs might
> propagate both DSCPs and ECN bits between the layers, so IMHO a
> better approach might be to recommend to treat DSCP+ECN bits as one
> aggregate byte (let's cal it TOS ;) ) as the extra ECT(1)-SA seems to
> be required for all SAs that already exist to deal with multiple
> supported DSCPs. So in a sense the recommendation would be to double
> the number of SAs.
> > 
> > > L4S on a 10 ms path under load can introduce re-ordering in the
> range of 50 ms (roughly twice the difference between the L- and C-
> queue delay targets), re-ordering tolerance 5 times of the path RTT
> seems to be a bit on the high side to expect, no?
> > 
> > Quoting Pete Heist's info here https://github.com/heistp/l4s-
> > tests/#dropped-packets-for-tunnels-with-replay-protection-enabled
> > [github.com] :
> > > "Modern Linux kernels have a default maximum replay window size
> > > of 4096 (XFRMA_REPLAY_ESN_MAX inxfrm.h [elixir.bootlin.com]).
> > > runtime configuration, increased from 2048 in May 2020 bythis
> > > commit [git.zx2c4.com]."
> > Regards
> > 
> > 
> > 
> > Bob
> > 
> > 
> > >  
> > >  
> > >  
> > >  
> > >  
> > > Regards
> > >   Sebastian
> > >  
> > >  
> > > > We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17
> attempting to address all the discussion since the last posting just
> before the interim. In particular:
> > > > There are a couple of outstanding discussions, which I'm sure
> will continue on the list, e.g. the role of RFC4774 and whether to
> remove any of Appx C. But it was considered better to get the queued
> up changes out, to re-base the discussions.
> > > >  
> > > > Thanks for everyone who is contributing, and particularly to
> the chairs for continuing to referee this all. We've added
> appropriate thanks in the Acks section.
> > > >  
> > > >  
> > > > Bob
> > > >  
> > > >  
> > > > >         Title           : Explicit Congestion Notification
> (ECN) Protocol for Very Low Queuing Delay (L4S)
> https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$
> [bobbriscoe[.]net]
> > > >  
> > >  
> > 
> > 
> > -- 
> [bobbriscoe.net]
> [bobbriscoe.net]

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 28 May 2021 13:16:02 +0200

Hi Bob,

only addressing one point, so I shortened the email a bit to make reading easier, prefixed [SM]

> On May 27, 2021, at 12:50, Bob Briscoe <[email-masked]> wrote:
> 
> David, see [BB]
> 

[...]

>> 	• Larger replay windows are a lower-effort approach that ought to apply to deployed VPNs whose replay window is configurable.  In practice, this is the place to start if L4S causes VPN problems.
> 
> [BB2] I don't disagree with making sure the replay window is large enough as well. But the IPsec and DTLS specs already say that, and it clearly doesn't always happen.

	[SM] Do you have proof that the default replay-window sizes actually cause much issues on over-the-internet VPNs without the additional systematic re-ordering L4S introduces? If yes, please post a reference/link. thanks. 

> As someone already said (Sebastian, Jonathan?), relying on a variable that gets updated only every so often, means we'll get problems every so often.

	[SM] Well, that just means in first order that we need to recommend a decent rule how to  appropriately size replay-windows for each individual VPN, based on packet-rate and RTT. The initial recommendation could be quite simple:
packet-rate [packets/ms] * expected-worst RTT [ms]
where packet rate can be approximated by:
ingress interface rate [b/ms] / average expected packet size [b/packet]
while expected-worst RTT needs to be set based on knowledge of the spatial/network distance of end points (I base this on the recommendation in the RACK RFC to limit the temporal re-ordering window to one RTT).
I understand that for short RTTs this rule faces the issue, that the L4S' induced re-ordering can be larger than the path RTT, but I assume most VPNs will be configured for longer RTTs anyway.

Personally, I would very much argue that L4S should try to keep its re-ordering as limited as possible, by keeping CE in the C-queue, after all the L4S argument goes that only one bottleneck along a path will be congested, and hence for the rest of the path queueing delay will be low already*... The treat all CE-packts ot the L-queue feels like an optimization anyway that should only be introduced after the L4S-experiment has been successful... I would imagine introducing such small tweaks would be appropriate when turning an experimental RC into a stadards track RFC...

Regards
	Sebastian

*) I am not 100% convinced that that argument actually holds water, but as long as L4S proponents use this argument, they need to accept it as valid.

---

From: "Black, David" <[email-masked]>
Date: Fri, 28 May 2021 21:39:56 +0000

[1] Location of VPN queue:

> ...that the queue into the off-site link is always after encapsulation. You have to queue the packets /after/ you've encrypted and encapsulated them, otherwise you don't know how much space they will take up in the queue.
Nope – encap overhead is generally fixed, can be determined by looking at packet without investing effort to do the encap.

[2-1] RFC 7657 and reordering

>> [David>]
>> 1. See RFC 7657 for general discussion about why variable QoS treatment within a single reliable transport protocol session is not a good idea –
>> that discussion is couched in Diffserv terms, but it's applicable to L4S ECN-based service differentiation.

> [BB2] Yes, I know and agree with that. Your argument here though is not against separating Classic from L4S flows into separate SAs.
> I assume it's solely about whether to risk separating packets that are already CE-marked into another SA because they might have started life as ECT0...
Correct – it's about splitting a flow across SAs that use different legs of an ECMP setup.

> The original L4S codepoint scheme was chosen on the basis that CE ambiguity led to spurious retransmits in a vanishingly small proportion of cases.
> So, an 'absolutist' argument against any risk at all of reordering doesn't really take us forward. The question here is not whether there'll be /some/ reordering.
> We know there will. The question is how likely it is. So the argument needs to be against my points about low likelihood.

I strongly disagree with that line of reasoning.  It is architecturally cleaner to not create the problem in the first place, as that completely avoids discussion about how low "low likelihood" is.

I'm generally skeptical of any "low likelihood" argument at the scale of the Internet courtesy of the law of large numbers (https://en.wikipedia.org/wiki/Law_of_large_numbers), doubly skeptical of probability arguments that contain no actual probabilities, and triply skeptical of all such arguments in security context, as if the "low likelihood" situation provides significant advantage to the attacker, it becomes certain to happen – there's some truly impressive exploit code for use after free OS kernel vulnerabilities that turns a "low likelihood" vulnerability involving data structure reuse (e.g., from a heap) into a near certainty.

[2-2] RFC 7657 and reordering

>> 2. Asking VPNs to classify traffic based on ECN field values is also asking for the possibility of discard of traffic marked with some of those values,
>> which is not a good place to go given the desire for the ECN field to remain end-to-end.

> [BB2] There must be something in your head that's not in mine here. Can you explain more fully where this discard comes from, and where the risk of losing the e2e property of the ECN field comes from?

Yes, this L4S two-SA proposal is effectively adding the ECN field values (well, two pairs of them) to the IPsec Security Policy Database, one of whose policy actions is DISCARD.  See RFC 4301.

Beyond that, I think we mostly agree on the facts for [2-3] (DSCPs) and [2-4] (Larger replay windows), but disagree about the implications for use of ECN codepoints for SA selection.

Finally, could I suggest that the L4S team stop trying to redesign IPsec, please?  This is at least the third time I've had to correct a factually incorrect statement (the incorrect statement this time is the assertion that a bottleneck queue cannot precede VPN endpoint encapsulation).

Thanks, --David

From: Bob Briscoe <[email-masked]>
Sent: Thursday, May 27, 2021 6:51 AM
To: Black, David; Gorry Fairhurst; Wesley Eddy
Cc: [email-masked]
Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[EXTERNAL EMAIL]
David, see [BB]

[EXTERNAL EMAIL]
David,

[BB2] And the details are...
...that the queue into the off-site link is always after encapsulation. You have to queue the packets /after/ you've encrypted and encapsulated them, otherwise you don't know how much space they will take up in the queue. And you can't apply active queue management if you don't know how much is in the queue.

[David>] Sure:

  1.  See RFC 7657 for general discussion about why variable QoS treatment within a single reliable transport protocol session is not a good idea – that discussion is couched in Diffserv terms, but it's applicable to L4S ECN-based service differentiation.

[BB2] Yes, I know and agree with that. Your argument here though is not against separating Classic from L4S flows into separate SAs. I assume it's solely about whether to risk separating packets that are already CE-marked into another SA because they might have started life as ECT0...

The original L4S codepoint scheme was chosen on the basis that CE ambiguity led to spurious retransmits in a vanishingly small proportion of cases. So, an 'absolutist' argument against any risk at all of reordering doesn't really take us forward. The question here is not whether there'll be /some/ reordering. We know there will. The question is how likely it is. So the argument needs to be against my points about low likelihood.

You've now pointed out that routing per SPI can lead to either early /or late/ reordering of CE packets, which is a new problem. In {Note 3} (still quoted below) I characterized the probability of late reordering /occurring/ as likely to be higher than without a VPN. But the /magnitude/ of the late reordering is nothing like as great as the max difference between Classic and L4S. That's significant given transports are rapidly becoming tolerant to a greater degree of reordering due to RACK in TCP and QUIC. So the question is whether there is a flaw in {Note 3}.

  1.  Asking VPNs to classify traffic based on ECN field values is also asking for the possibility of discard of traffic marked with some of those values, which is not a good place to go given the desire for the ECN field to remain end-to-end.

[BB2] There must be something in your head that's not in mine here. Can you explain more fully where this discard comes from, and where the risk of losing the e2e property of the ECN field comes from?

  1.  Using DSCPs is a "reduce to previous case" approach that doesn't involve modification of any security specs and aligns with existing VPN implementations that are cognizant of DSCPs.

[BB2] That's true. Which is why the draft suggests using DSCPs if the VPN hasn't been or can't be modified yet to classify on ECN. But DSCPs are, by definition, per domain and not reliable. So the ultimate solution to make the problem go away with certainty would be to classify on ECN.

  1.  Larger replay windows are a lower-effort approach that ought to apply to deployed VPNs whose replay window is configurable.  In practice, this is the place to start if L4S causes VPN problems.

[BB2] I don't disagree with making sure the replay window is large enough as well. But the IPsec and DTLS specs already say that, and it clearly doesn't always happen. As someone already said (Sebastian, Jonathan?), relying on a variable that gets updated only every so often, means we'll get problems every so often.

In summary, the answer is a combination of the three solutions (ECN, DSCP and sufficient window). But classification by ECN is the robust solution and therefore the one stated as a SHOULD. But that depends on my argument about the low likelihood of reordering.

Bob

Bob

Thanks, --David

[EXTERNAL EMAIL]
David,

Regards

Bob

{Note 2}:

{Note 3}:

Bob

Thanks, --David

-----Original Message-----

To: Bob Briscoe

[EXTERNAL EMAIL]

Bob, chairs,

Also:

Bob

Regards

  Sebastian

Chairs, list,

Bob

                          Bob Briscoe

 Pages           : 57

 Date            : 2021-05-21

Abstract:

--

--

--

--

Bob Briscoe                               http://bobbriscoe.net/ [bobbriscoe.net]<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!LpKI!2g5T-SpmXBuwGfJ3NFRkXkTF6gk-jxsS1NvGZtabfcyOsBgAFf9nppE3SNQVHCtR$>