Subject: Re: [tsvwg] L4S dual-queue re-ordering and VPNs

From: Greg White <[email-masked]>
Date: Mon, 3 May 2021 17:35:53 +0000

I'm not familiar with the replay attack mitigations used by VPNs, so can't comment on whether this would indeed be an issue for some VPN implementations.  A quick search revealed (https://www.wireguard.com/protocol/ ) that Wireguard apparently has a window of about 2000 packets, so perhaps it isn't an immediate issue for that VPN software?

But, if it is an issue for a particular algorithm, perhaps another solution to address condition b would be to use a different "head of window" for ECT1 packets compared to ECT(0)/NotECT packets?  

In your 100 Gbps case, I guess you are assuming that A) the bottleneck between the two tunnel endpoints is 100 Gbps, B) a single VPN tunnel is consuming the entirety of that 100 Gbps link, and C) that there is a PI2 AQM targeting 20ms of buffering delay in that 100 Gbps link?  If so, I'm not sure that I agree that this is likely in the near term.  But, in any case, it seems to me that protocols that need to be robust to out-of-order delivery would need to consider being robust to re-ordering in time units anyway, and so would naturally need to scale that functionality as packet rates increase.

I'm happy to include text in the L4Sops draft on this if the WG agrees it is useful to include it, and someone provides text that would fit the bill.

-Greg

On 5/3/21, 1:44 AM, "tsvwg on behalf of Sebastian Moeller" <[email-masked] on behalf of [email-masked]> wrote:

    Dear All,

    we had a few discussions in the past about L4S' dual queue design and the consequences of packets of a single flow being accidentally steered into the wrong queue.
    So far we mostly discussed the consequence of steering all packets marked CE into the LL-queue (and https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.1 Risk of reordering Classic CE packets: only discusses this point); there the argument is, that this condition should be rare and should also be relative benign, as an occasional packet to early should not trigger the 3 DupACK mechanism. While I would liked to see hard data confirming the two hypothesis, let's accept that argument for the time being.

    BUT, there is a traffic class that is actually sensitive to packets arriving out-of-order and too early: VPNs. Most VPNs try to secure against replay attacks by maintaining a replay window and only accept packets that fall within that window. Now, as far as I can see, most replay window algorithms use a bounded window and use the highest received sequence number to set the "head" of the window and hence will trigger replay attack mitigation, if the too-early-packets move the replay window forward such that "in-order-packets" from the shorter queue fall behind the replay window.

    Wireguard is an example of a modern VPN affected by this issue, since it supports ECN and propagates ECN bits between inner and outer headers on en- and decapsulation. 

    I can see two conditions that trigger this:
    a) the arguably relatively rare case of an already CE-marked packet hitting an L4S AQM (but we have no real number on the likelihood of that happening)
    b) the arguably more and more common situation (if L4S actually succeeds in the field) of an ECT(1) sub-flow zipping past ECT(0)/NotECT sub-flows (all within the same tunnel outer flow)

    I note that neither single-queue rfc3168 or FQ AQMs (rfc3168 or not) are affected by that issue since they do not cause similar re-ordering.

    QUESTIONS @ALL:

    1)  Are we all happy with that and do we consider this to be acceptable collateral damage?

    2) If yes, should the L4S OPs draft contain text to recommend end-points how to cope with that new situation? 
    	If yes, how? Available options are IMHO to eschew the use of ECN on tunnels, or to recommend increased replay window sizes, but with a Gigabit link and L4S classic target of around 20ms, we would need to recommend a repay window of:
    >= ((1000^3 [b/s]) / (1538 [B/packet] * 8 [b/B])) * (20[ms]/1000[ms]) = 1625.48764629 [packets]
    or with a power of two algorithm 2048, which is quite a bit larger than the old default of 64...
    	But what if the L4s AQM is located on a back-bone link with considerably higher bandwidth, like 10 Gbps or even 100 Gbps? IMHO a replay window of 1625 * 100 = 162500 seems a bit excessive

    Also the following text in https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-A.1.7

    "  Should work in tunnels:  Unlike Diffserv, ECN is defined to always
          work across tunnels.  This scheme works within a tunnel that
          propagates the ECN field in any of the variant ways it has been
          defined, from the year 2001 [RFC3168] onwards.  However, it is
          likely that some tunnels still do not implement ECN propagation at
          all."

    Seems like it could need additions to reflect the just described new issue.

    Best Regards
    	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 5 May 2021 23:20:53 +0200

Hi Greg,

thanks for your response, more below prefixed [SM].

> On May 3, 2021, at 19:35, Greg White <[email-masked]> wrote:
> 
> I'm not familiar with the replay attack mitigations used by VPNs, so can't comment on whether this would indeed be an issue for some VPN implementations.

[SM] I believe this to be an issue for at least those VPNs that use UDP and defend against replay attacks (including ipsec, wireguard, OpenVPN). All more or less seem to use the same approach with a limited accounting window to allow out-of-order delivery of packets. The head of the window typically seems to be advanced to the packet with the highest "sequence" number, hence all of these are sensitive for the kind of packet re-ordering the L4S ecn id draft argues was benign...

>  A quick search revealed (https://www.wireguard.com/protocol/ ) that Wireguard apparently has a window of about 2000 packets, so perhaps it isn't an immediate issue for that VPN software?

[SM] Current Linux kernels seem to use a window of ~8K packets, while OpnenVPN defaults to 64 packets, Linux ipsec seems to default to either 32 or 64. 8K should be reasonably safe, but 64 seems less safe.

> But, if it is an issue for a particular algorithm, perhaps another solution to address condition b would be to use a different "head of window" for ECT1 packets compared to ECT(0)/NotECT packets?  

[SM] Without arguing whether that might or might not be a good idea, it is not what is done today, so all deployed end-points will treat all packets the same but at least wireguard and linux ipsec will propagate ECN vaule at en- and decapsulation, so are probably affected by the issue.

> In your 100 Gbps case, I guess you are assuming that A) the bottleneck between the two tunnel endpoints is 100 Gbps, B) a single VPN tunnel is consuming the entirety of that 100 Gbps link, and C) that there is a PI2 AQM targeting 20ms of buffering delay in that 100 Gbps link?  If so, I'm not sure that I agree that this is likely in the near term.

[SM] Yes, the back-of-an-envelop worst case estimate is not terribly concerning, I agree, but the point remains that a fixed 20ms delay target will potentially cause the issue with increasing link speeds...

>  But, in any case, it seems to me that protocols that need to be robust to out-of-order delivery would need to consider being robust to re-ordering in time units anyway, and so would naturally need to scale that functionality as packet rates increase.

[SM] The thing is these methods aim to avoid Mallory fudging with the secure connection between Alice and Bob (not our's), and need to track packet by packet, that is not easily solved efficiently with a simple time-out (at least not as far as I can seem but I do not claim expertise in cryptology or security engineering). But I am certain, if you have a decent new algorithm to enhance RFC2401 and/or RFC6479 the crypto community might be delighted to hear them. ;)

> I'm happy to include text in the L4Sops draft on this if the WG agrees it is useful to include it, and someone provides text that would fit the bill.

[SM] I wonder whether a section on L4S-OPs a la, "make sure to configure a sufficiently large replay window to allow for ~20ms reordering" would be enough, or  wether the whole discussion would not also be needed in https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.1 widening the re-ordering scope from the existing "Risk of reordering Classic CE packets" subpoint 3.?

Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Thu, 06 May 2021 17:31:26 +0200

Hi,

To test this out, I added a scenario to the l4s-tests repo of two flow
competition in an IPsec tunnel through DualPI2:

https://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled

It shows that non-L4S traffic through the C queue can see drops when
packets arrive outside the replay window- essentially the concern that
Sebastian raised. The effect here is reduced throughput for a CUBIC
flow in competition with Prague, depending on the replay window in use.

For this 100Mbps case, I tried a range of 5 values for replay-window
(0, 32, 64, 128, 256). It looks like it's good to account for peak
sojourn times through the C queue- perhaps 50ms? If so, a replay window
of 512 packets might be better here. As I understand it, the default
for IPsec is typically 32 or 64 packets. So, some existing deployed
tunnels may need to be reconfigured, depending on their bandwidths.

The default and compiled-in value for Wireguard appears to now be 8192
packets(?), so there I wouldn't expect this to be a problem until the
tunneled traffic is at least ~2Gbps, but that's just a guess, by
estimating what bandwidth can exceed 8192 packets in 50ms.

Thanks Sebastian for bringing this up...

Pete

On Wed, 2021-05-05 at 23:20 +0200, Sebastian Moeller wrote:
> > I'm not familiar with the replay attack mitigations used by VPNs, so
> > can't comment on whether this would indeed be an issue for some VPN
> > implementations.
> 
> [SM] I believe this to be an issue for at least those VPNs that use UDP
> and defend against replay attacks (including ipsec, wireguard,
> OpenVPN). All more or less seem to use the same approach with a limited
> accounting window to allow out-of-order delivery of packets. The head
> of the window typically seems to be advanced to the packet with the
> highest "sequence" number, hence all of these are sensitive for the
> kind of packet re-ordering the L4S ecn id draft argues was benign...
> 
> 
> >  A quick search revealed (https://www.wireguard.com/protocol/ ) that
> > Wireguard apparently has a window of about 2000 packets, so perhaps
> > it isn't an immediate issue for that VPN software?
> 
> [SM] Current Linux kernels seem to use a window of ~8K packets, while
> OpnenVPN defaults to 64 packets, Linux ipsec seems to default to either
> 32 or 64. 8K should be reasonably safe, but 64 seems less safe.
> 
> > But, if it is an issue for a particular algorithm, perhaps another
> > solution to address condition b would be to use a different "head of
> > window" for ECT1 packets compared to ECT(0)/NotECT packets?  
> 
> [SM] Without arguing whether that might or might not be a good idea, it
> is not what is done today, so all deployed end-points will treat all
> packets the same but at least wireguard and linux ipsec will propagate
> ECN vaule at en- and decapsulation, so are probably affected by the
> issue.
> 
> > In your 100 Gbps case, I guess you are assuming that A) the
> > bottleneck between the two tunnel endpoints is 100 Gbps, B) a single
> > VPN tunnel is consuming the entirety of that 100 Gbps link, and C)
> > that there is a PI2 AQM targeting 20ms of buffering delay in that 100
> > Gbps link?  If so, I'm not sure that I agree that this is likely in
> > the near term.
> 
> [SM] Yes, the back-of-an-envelop worst case estimate is not terribly
> concerning, I agree, but the point remains that a fixed 20ms delay
> target will potentially cause the issue with increasing link speeds...
> 
> 
> >  But, in any case, it seems to me that protocols that need to be
> > robust to out-of-order delivery would need to consider being robust
> > to re-ordering in time units anyway, and so would naturally need to
> > scale that functionality as packet rates increase.
> 
> [SM] The thing is these methods aim to avoid Mallory fudging with the
> secure connection between Alice and Bob (not our's), and need to track
> packet by packet, that is not easily solved efficiently with a simple
> time-out (at least not as far as I can seem but I do not claim
> expertise in cryptology or security engineering). But I am certain, if
> you have a decent new algorithm to enhance RFC2401 and/or RFC6479 the
> crypto community might be delighted to hear them. ;)
> 
> > I'm happy to include text in the L4Sops draft on this if the WG
> > agrees it is useful to include it, and someone provides text that
> > would fit the bill.
> 
> [SM] I wonder whether a section on L4S-OPs a la, "make sure to
> configure a sufficiently large replay window to allow for ~20ms
> reordering" would be enough, or  wether the whole discussion would not
> also be needed in
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.1
>  widening the re-ordering scope from the existing "Risk of reordering
> Classic CE packets" subpoint 3.?
> 
> Regards
>         Sebastian
> 
> 
> > 
> > -Greg
> > 
> > 
> > On 5/3/21, 1:44 AM, "tsvwg on behalf of Sebastian Moeller"
> > <[email-masked] on behalf of [email-masked]> wrote:
> > 
> >    Dear All,
> > 
> >    we had a few discussions in the past about L4S' dual queue design
> > and the consequences of packets of a single flow being accidentally
> > steered into the wrong queue.
> >    So far we mostly discussed the consequence of steering all packets
> > marked CE into the LL-queue (and
> > https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.1
> >  Risk of reordering Classic CE packets: only discusses this point);
> > there the argument is, that this condition should be rare and should
> > also be relative benign, as an occasional packet to early should not
> > trigger the 3 DupACK mechanism. While I would liked to see hard data
> > confirming the two hypothesis, let's accept that argument for the
> > time being.
> > 
> >    BUT, there is a traffic class that is actually sensitive to
> > packets arriving out-of-order and too early: VPNs. Most VPNs try to
> > secure against replay attacks by maintaining a replay window and only
> > accept packets that fall within that window. Now, as far as I can
> > see, most replay window algorithms use a bounded window and use the
> > highest received sequence number to set the "head" of the window and
> > hence will trigger replay attack mitigation, if the too-early-packets
> > move the replay window forward such that "in-order-packets" from the
> > shorter queue fall behind the replay window.
> > 
> >    Wireguard is an example of a modern VPN affected by this issue,
> > since it supports ECN and propagates ECN bits between inner and outer
> > headers on en- and decapsulation. 
> > 
> >    a) the arguably relatively rare case of an already CE-marked
> > packet hitting an L4S AQM (but we have no real number on the
> > likelihood of that happening)
> >    b) the arguably more and more common situation (if L4S actually
> > succeeds in the field) of an ECT(1) sub-flow zipping past
> > ECT(0)/NotECT sub-flows (all within the same tunnel outer flow)
> > 
> >    I note that neither single-queue rfc3168 or FQ AQMs (rfc3168 or
> > not) are affected by that issue since they do not cause similar re-
> > ordering.
> > 
> > 
> >    QUESTIONS @ALL:
> > 
> >    2) If yes, should the L4S OPs draft contain text to recommend end-
> > points how to cope with that new situation? 
> >         If yes, how? Available options are IMHO to eschew the use of
> > ECN on tunnels, or to recommend increased replay window sizes, but
> > with a Gigabit link and L4S classic target of around 20ms, we would
> > need to recommend a repay window of:
> > > = ((1000^3 [b/s]) / (1538 [B/packet] * 8 [b/B])) *
> > > (20[ms]/1000[ms]) = 1625.48764629 [packets]
> >         But what if the L4s AQM is located on a back-bone link with
> > considerably higher bandwidth, like 10 Gbps or even 100 Gbps? IMHO a
> > replay window of 1625 * 100 = 162500 seems a bit excessive
> > 
> > 
> >    Also the following text in
> > https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-A.1.7
> >

---

From: "Black, David" <[email-masked]>
Date: Sat, 8 May 2021 02:01:10 +0000

[posting as an individual, not a WG chair]
Linking together a couple of related points:

> [SM] Current Linux kernels seem to use a window of ~8K packets, while OpenVPN defaults to 64 packets, Linux ipsec seems to default to either 32 or 64. 8K should be reasonably safe, but 64 seems less safe.

Common VPN design practice here appears to be picking a plausible default size (which can be reconfigured and change from release to release) for the accounting window to detect replay, hence this:

> >  But, in any case, it seems to me that protocols that need to be robust to out-of-order delivery would need to consider being robust
> > to re-ordering in time units anyway, and so would naturally need to scale that functionality as packet rates increase.

may not happen in a smooth fashion.  As Sebastian writes:

> [SM] The thing is these methods aim to avoid Mallory fudging with the secure connection between Alice and Bob (not our's), and need to track packet by packet, that is not easily solved efficiently with a simple time-out

That's correct, and use of a simple time-out by itself is prohibited for obvious security reasons.  For more details on a specific example, see Section 3.4.3 of RFC 4303 (ESP), which specifies the ESP anti-replay mechanism (could be used as a reference in writing text on how L4S interacts with anti-replay)  ... and the observant reader will notice that this section is a likely source of the anti-replay 32 and 64 packet values for Linux IPsec: https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3 .

Thanks, --David

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
Sent: Wednesday, May 5, 2021 5:21 PM
To: Greg White
Cc: TSVWG
Subject: Re: [tsvwg] L4S dual-queue re-ordering and VPNs

[EXTERNAL EMAIL] 

Hi Greg,

>  A quick search revealed (https://urldefense.com/v3/__https://www.wireguard.com/protocol/__;!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOP789qBv$ [wireguard[.]com] ) that Wireguard apparently has a window of about 2000 packets, so perhaps it isn't an immediate issue for that VPN software?

> I'm happy to include text in the L4Sops draft on this if the WG agrees it is useful to include it, and someone provides text that would fit the bill.

[SM] I wonder whether a section on L4S-OPs a la, "make sure to configure a sufficiently large replay window to allow for ~20ms reordering" would be enough, or  wether the whole discussion would not also be needed in https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-B.1__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOFX4wc3G$ [datatracker[.]ietf[.]org] widening the re-ordering scope from the existing "Risk of reordering Classic CE packets" subpoint 3.?

Regards
	Sebastian

>    So far we mostly discussed the consequence of steering all packets marked CE into the LL-queue (and https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-B.1__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOFX4wc3G$ [datatracker[.]ietf[.]org] Risk of reordering Classic CE packets: only discusses this point); there the argument is, that this condition should be rare and should also be relative benign, as an occasional packet to early should not trigger the 3 DupACK mechanism. While I would liked to see hard data confirming the two hypothesis, let's accept that argument for the time being.
> 
>    Also the following text in https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-A.1.7__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOJfaO_VT$ [datatracker[.]ietf[.]org]
>

---

From: Pete Heist <[email-masked]>
Date: Sat, 08 May 2021 08:45:30 +0200

I've added some additional tests at 10 and 20Mbps, and re-worked the
writeup to include a table of the results:

I noticed that this issue seems to affect tunnels with replay window
sizes of 32 and 64 packets regardless of the bottleneck bandwidth,
likely because the peak C sojourn times can also increase as the
bandwidth decreases. IMO, this seems like a safety concern from the
standpoint that the deployment of DualPI2 can cause harm to
conventional traffic, in IPsec tunnels using common defaults in
particular, beyond that which is caused by DualPI2 itself.

It may be fixed by increasing the window size or disabling replay
protection, but it may not be easy for admins or users to identify the
source of this problem when it occurs, or know who to contact about it.

Pete

On Sat, 2021-05-08 at 02:01 +0000, Black, David wrote:
> > OpenVPN defaults to 64 packets, Linux ipsec seems to default to
> > either 32 or 64. 8K should be reasonably safe, but 64 seems less
> > safe.
> 
> Common VPN design practice here appears to be picking a plausible
> default size (which can be reconfigured and change from release to
> release) for the accounting window to detect replay, hence this:
> 
> > secure connection between Alice and Bob (not our's), and need to
> > track packet by packet, that is not easily solved efficiently with a
> > simple time-out
> 
> That's correct, and use of a simple time-out by itself is prohibited
> for obvious security reasons.  For more details on a specific example,
> see Section 3.4.3 of RFC 4303 (ESP), which specifies the ESP anti-
> replay mechanism (could be used as a reference in writing text on how
> L4S interacts with anti-replay)  ... and the observant reader will
> notice that this section is a likely source of the anti-replay 32 and
> 64 packet values for Linux IPsec:
> https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3 .
> 
> Thanks, --David
> 
> >  A quick search revealed
> > (https://urldefense.com/v3/__https://www.wireguard.com/protocol/__;!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOP789qBv$
> >  [wireguard[.]com] ) that Wireguard apparently has a window of about
> > 2000 packets, so perhaps it isn't an immediate issue for that VPN
> > software?
> 
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-B.1__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOFX4wc3G$
>  [datatracker[.]ietf[.]org] widening the re-ordering scope from the
> existing "Risk of reordering Classic CE packets" subpoint 3.?
> 
> Regards
>         Sebastian
> 
> 
> > 
> > -Greg
> > 
> > 
> > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-B.1__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOFX4wc3G$
> >  [datatracker[.]ietf[.]org] Risk of reordering Classic CE packets:
> > only discusses this point); there the argument is, that this
> > condition should be rare and should also be relative benign, as an
> > occasional packet to early should not trigger the 3 DupACK mechanism.
> > While I would liked to see hard data confirming the two hypothesis,
> > let's accept that argument for the time being.
> > 
> > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-A.1.7__;Iw!!LpKI!0R_YA5wY-HgCAeBd-ajbFbEamek2Wo9ESyoFSJ6whDL8_0kmFhysbbCeOJfaO_VT$
> >  [datatracker[.]ietf[.]org]
> >

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 8 May 2021 17:22:12 +0300

> On 8 May, 2021, at 9:45 am, Pete Heist <[email-masked]> wrote:
> 

It turns out that there's also a Denial of Service attack enabled by this phenomenon - one which does not rely on any legitimate L4S endpoints being deployed or used.  It is a property of the passive deployment of DualQ.

We assume a VPN tunnel upon which Sequence Verification is in use (RFC-4303 §3.4.3), over a path that includes a DualQ-Coupled qdisc (draft-ietf-tsvwg-aqm-dualq-coupled), abbreviated DualQ.  We further assume that an attacker is able to send traffic with chosen ECN codepoints into the input end of this tunnel.  We outline two attack scenarios, the second of which further assumes that the attacker sends traffic directly over the DualQ path, not via the tunnel, which requires that the attacker knows more information than the first attack scenario.  The threat is the ability to deny any useful traffic being conveyed by the VPN tunnel, at a significantly lower cost than a conventional DDoS.

In the first attack scenario, the attacker generates two distinct streams of traffic, named Lambda and Omega, both of which are directly into the input end of the tunnel.  The Lambda stream is bulk data with an ECN codepoint that filters into the C queue - either Not-ECT or ECT(0).  The Omega stream is a low-volume stream of small packets with an ECN codepoint that filters into the L queue - either ECT(1) or CE.  It is not necessary for either stream to be accepted by a host at the receiving end, only by the endpoints of the VPN itself.

The purpose of the Lambda stream is to keep the C queue of the DualQ full, and it is sufficient to send MTU-sized packets in only slightly greater volume than the capacity of the link.  We will assume for analysis that DualQ maintains the C queue at its AQM setpoint, which is 25ms by default.

The purpose of the Omega stream is to bypass the Lambda stream by taking the "fast lane" through the DualQ instance, and thereby control the leading edge of the Sequence Verification window of the VPN receiver.  If the leading edge of the window can be kept so far ahead of the traffic arriving via the C queue that the latter always falls behind the trailing edge of the window, then the amount of useful traffic conveyed by the VPN in the same direction falls to nil - a successful Denial of Service.

The required Sequence Verification window to deny this attack scenario full effectiveness is the VPN packet throughput of the C queue multiplied by the difference in queue sojourn times between the C and L queues, plus the interval between packets delivered in the Omega stream.  Since the default SV window in some popular VPNs is 32 or 64, this attack can be fairly easy to implement.

Another mitigation that the VPN could apply is to limit the throughput of the VPN, such that it cannot by itself be used to saturate the DualQ link and thus keep the C queue full.  We address this with the second attack scenario, which bypasses such a mitigation.

In the second attack scenario, the Lambda stream is sent directly to the DualQ link rather than through the VPN.  The Omega stream is still sent through the VPN.  The properties of the two streams are mostly similar otherwise to those of the first attack scenario, but the volume of the Omega stream must be increased.

The second attack scenario will succeed if the number of Omega packets delivered during any interval of time equal to the difference between the C and L queue sojourn times is greater than the Sequence Verification window.  This is also easy to achieve if the Omega packets are small and the SV window is at the usual defaults of 32 or 64.

The cause of this attack enablement is that DualQ can reorder packets from the same flow under fairly precise control of an attacker.  A straight FIFO, or a flow-aware qdisc that treats a VPN as a single flow (such as RFC-8290 fq_codel or Cake), delivers packets from any given flow in-order, and is thus immune from this attack.  Adding DualQ-like semantics to any of these carries the risk of enabling the attack as described above.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 8 May 2021 15:26:22 +0100

Thank you, Sebastian, for picking up this inter-area inconsistency, and 
thanks, Pete, for the test data.

I would have thought that the delay-delta between two AQMs (as in the 
DualQ) will in general be much less than the delay-delta between a low 
delay and best efforts Diffserv behaviour, where the best efforts 
generally has no AQM at all, and is therefore prone to the delay of the 
whole buffer, which may be bloated.

So, wherever a VPN includes flows using different DSCPs, and there is a 
Diffserv-enabled bottleneck between the ends of the VPN, the VPN's 
replay window will need to cater for considerably more than 50ms 
delay-delta within the VPN. More like at least 200ms, and possibly 1-2s 
in some cases of bloat.

David himself has written about the reordering problem when a WebRTC 
application encapsulates TCP, SCTP and RTP flows with different DSCPs 
within UDP [RFC7657]. Indeed, datagram transport layer security (DTLS) 
is a common encapsulation for WebRTC flows. And DTLS also recommends a 
default replay window of 64 (see 
https://tools.ietf.org/html/rfc6347#section-4.1.2.6 ).

- I'm not trying to say low replay windows won't affect the DualQ - 
l4sops and aqm-dualq-coupled should certainly recommend a large enough 
replay window.

- I'm just saying that there are other established technologies that 
reduce queuing delay for a subset of traffic, and from current insanely 
high levels, so they will be a longer pole in the tent than the DualQ. 
Then, as long as the replay window of VPNs is large enough for those 
established technologies, it will be large enough for the DualQ experiment.

In the context of the IETF, irrespective of the L4S experiment, the IETF 
needs to fix this bigger inconsistency between the standards tracks of 
its transport and security areas. I'll leave David to escalate this to 
the ADs if appropriate. Because Pete's right - it may not be easy for 
admins to identify the cause of this problem, and admins and security 
implementers don't tend to reach out for advice in transport RFCs.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 8 May 2021 16:05:09 +0100

Jonathan,

As just pointed out (my email crossed yours), existing established 
Diffserv technology is also prone to this problem, and it's lower 
hanging fruit for an attacker, because the delay-delta is typically larger.

The solution to all these interactions between a poorly configured 
replay protection window and queues with different delay differences is 
simply to configure a replay window appropriate to the packet rate of 
the VPN tunnel, taking account of expected delay differences between queues.

Whatever, VPNs rarely provide open access for traffic from unauthorized 
sources to enter the VPN.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 8 May 2021 18:25:21 +0300

> On 8 May, 2021, at 6:05 pm, Bob Briscoe <[email-masked]> wrote:
> 
> As just pointed out (my email crossed yours), existing established Diffserv technology is also prone to this problem, and it's lower hanging fruit for an attacker, because the delay-delta is typically larger.

Perhaps - but usable Diffserv deployments at the last-mile are not very common.  L4S proposes to put a DualQ instance in front of every cable internet subscriber, making it much easier for an attacker to predict how he needs to conduct his attack than the equivalent with Diffserv.  If the PHB is implemented at a backbone node rather than a last-mile head-end, the bandwidth required to saturate DualQ is also much less than to saturate the Diffserv queue.

Also, if ECT(0) traffic is what gets stuffed down the C queue, how much can the sojourn time grow before packets start getting dropped, and is that situation persistent under an unresponsive load?  The difference in queue depth might *not* actually be much different from the Diffserv case.

> The solution to all these interactions between a poorly configured replay protection window and queues with different delay differences is simply to configure a replay window appropriate to the packet rate of the VPN tunnel, taking account of expected delay differences between queues.

Maybe so, but the small replay window size is the environment that exists today.  Perhaps it will improve in future, but I think it has to be increased by several orders of magnitude to make this attack infeasible.  The Wireguard implementation in Linux has a hard ceiling at 8192 packets - is that enough?

> Whatever, VPNs rarely provide open access for traffic from unauthorized sources to enter the VPN.

One common application of VPNs is to hide players' home IP addresses from other players in multiplayer games.  They do this to prevent a type of griefing whereby an opposing player launches a DDoS attack to effectively disconnect them from the game, or lag them enough to get an easy kill.  This is possible because fast-paced games tend to send data directly between players to reduce latency, rather than going through a central server which is a better choice for slower-paced games.

If a player uses a VPN to connect to the game, then the remote end of the VPN is the address given to the other players.  A DoS attack that could be launched into such a VPN endpoint would be highly attractive to a griefer.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 8 May 2021 18:04:57 +0100

Jonathan,

On 08/05/2021 16:25, Jonathan Morton wrote:
> Perhaps - but usable Diffserv deployments at the last-mile are not very common.

[BB] WMM (Wifi MultiMedia).

> L4S proposes to put a DualQ instance in front of every cable internet subscriber, making it much easier for an attacker to predict how he needs to conduct his attack than the equivalent with Diffserv.  If the PHB is implemented at a backbone node rather than a last-mile head-end, the bandwidth required to saturate DualQ is also much less than to saturate the Diffserv queue.
>
> Maybe so, but the small replay window size is the environment that exists today.  Perhaps it will improve in future, but I think it has to be increased by several orders of magnitude to make this attack infeasible.

[BB] I understood this already when Pete said it. My point was that, you 
seem to want this to be a criticism of L4S, rather than of poorly 
configured VPNs. Surely you are not saying the IETF should halt any 
advances that reduce delay, like Diffserv and L4S, just because the 
replay windows of VPNs are poorly configured today?

> The Wireguard implementation in Linux has a hard ceiling at 8192 packets - is that enough?

[BB] A hard ceiling in /soft/ware?

> If a player uses a VPN to connect to the game, then the remote end of the VPN is the address given to the other players.  A DoS attack that could be launched into such a VPN endpoint would be highl…

[BB] So the player has a strong incentive to configure the replay window 
of her VPN properly.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 8 May 2021 20:09:32 +0200

Hi Bob, list, 

see [SM] below

> On May 8, 2021, at 16:26, Bob Briscoe <[email-masked]> wrote:
> 
> Thank you, Sebastian, for picking up this inter-area inconsistency, and thanks, Pete, for the test data.
> 
> I would have thought that the delay-delta between two AQMs (as in the DualQ) will in general be much less than the delay-delta between a low delay and best efforts Diffserv behaviour, where the best efforts generally has no AQM at all, and is therefore prone to the delay of the whole buffer, which may be bloated.

	[SM] Well possible, but not that relevant, as far as I understand the Linux source code wireguard and ipsev/xfrm only propagate the ECN bits between inner and outer layers and do not propagate the other 6 dscp bits (of the former TOS byte). In fact the issue I see is that L4S introduces a novel re-ordering condition that has not exited before and that hence is on nobody's radar.

> So, wherever a VPN includes flows using different DSCPs, and there is a Diffserv-enabled bottleneck between the ends of the VPN, the VPN's replay window will need to cater for considerably more than 50ms delay-delta within the VPN. More like at least 200ms, and possibly 1-2s in some cases of bloat.

	[SM] Sure, but neither ipsec nor wireguard do that, outer DSCP for wireguard seems to be fixed to 0... 

> David himself has written about the reordering problem when a WebRTC application encapsulates TCP, SCTP and RTP flows with different DSCPs within UDP [RFC7657]. Indeed, datagram transport layer security (DTLS) is a common encapsulation for WebRTC flows. And DTLS also recommends a default replay window of 64 (see https://tools.ietf.org/html/rfc6347#section-4.1.2.6 ).

	[SM] Interesting case but different from the typical end-user uses a VPN case in which the issue will potentially crop up. VPNs (full or split tunnel) from end users to VPN providers or into the office are quite common nowadays and I bet most will not propagate dscps to the outer layer, so let's treat these as an independent category.

> - I'm not trying to say low replay windows won't affect the DualQ -

	[SM] The other way around, DualQ requires an increases replay-window, it is L4S that is the root cause of the configuration change here, since it constitutes a new mechanism for re-ordering.

> l4sops and aqm-dualq-coupled should certainly recommend a large enough replay window.

	[SM] Well, that is a stop-gap measure, really. IMHO the IETF recommends to propagate ECN bits, so the IETF should make sure that this recommended behavior does not cause unexpected negative side-effects. Telling VPN users to change because of someone else's experiment is a bit lame. Also it will be hard to actually get to the affected users and tell them not to worry about replay attacks, but simply enlarge the replay-window instead...
	Also, if I might add, it demonstrates why FQ actually is a pretty decent solution in general, sure equitable sharing is by no means guaranteed to be the optimal solution, but at the same time, given the limited information at the AQM it might be the least worst it can do...

> - I'm just saying that there are other established technologies that reduce queuing delay for a subset of traffic, and from current insanely high levels, so they will be a longer pole in the tent than the DualQ. Then, as long as the replay window of VPNs is large enough for those established technologies, it will be large enough for the DualQ experiment.

	[SM] How that? Pete demonstrated that recommended default values of 32 or 64 packets are already enough to see the issue, but the same 32/64 packets seem to work reasonably well for the re-ordering one might encounter on the existing internet. As I indicated during the re-ordering of CE discussion some time ago, declaring a specific level of re-ordering benign requires a set of assumptions, and these might or might not hold... in the VPN with replay-protection case (which seems to be the recommended mode) these assumptions do not seem to hold.

> In the context of the IETF, irrespective of the L4S experiment, the IETF needs to fix this bigger inconsistency between the standards tracks of its transport and security areas. I'll leave David to escalate this to the ADs if appropriate. Because Pete's right - it may not be easy for admins to identify the cause of this problem, and admins and security implementers don't tend to reach out for advice in transport RFCs.

	[SM] And no wordsmithing in any RFC is guaranteed to reach the current operators/users of such tunnels any time soon.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 8 May 2021 20:22:02 +0200

Hi Bob,

> On May 8, 2021, at 19:04, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
> 
> [BB] I understood this already when Pete said it. My point was that, you seem to want this to be a criticism of L4S, rather than of poorly configured VPNs.

	[SM] Well, L4S novel re-ordering behaviour can turn a VPN configuration that worked just fine into one that throws spurious replay attack warnings. IMHO that indicates that the VPN was properly configured for its envoronment before, so the party changing the environment, L4S in this case, carries at least as much responsibility as the VPN. 
	Side-note, it gets predictable, that all new L4S failure modes are immediately interpreted as someone else's fault.

> Surely you are not saying the IETF should halt any advances that reduce delay, like Diffserv and L4S, just because the replay windows of VPNs are poorly configured today?
> 

	[SM] It is a define, so ATM you need to compile a new kernel, so for many end-user this is effectively a hard ceiling...

> [BB] So the player has a strong incentive to configure the replay window of her VPN properly.

	[SM] Or to bleach ECT(1) on tunnel egress to head off such behavior... Really I consider your evaluation of responsibility in this issue a bit offensive. L4S breaks existing known-working configurations, and all we say, is fix your VPN, really?

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 8 May 2021 21:32:00 +0300

> [BB] WMM (Wifi MultiMedia).

That's not the last mile - that's within customer premises.  I would also question how often a DSCP would survive on the outer header of a VPN packet long enough to influence this mechanism.  But I would also note the increase in the use of fq_codel in wifi AP stacks, which would tend to reduce any differential delay to a greater degree than the C queue in DualQ does.  Those, taken together, might be significant reasons why problems have not been noticed up to now.

For the particular use case I mentioned, a player connecting over wifi would be ridiculed due to the inconsistent latency that wifi offers, especially in congested radio environments.

>> L4S proposes to put a DualQ instance in front of every cable internet subscriber, making it much easier for an attacker to predict how he needs to conduct his attack than the equivalent with Diffserv.

...

> My point was that, you seem to want this to be a criticism of L4S, rather than of poorly configured VPNs.

As often seems to happen, you seem to want to deflect any possible criticism of L4S onto some other piece of technology that it might happen to interact poorly with, so that you don't need to change L4S, only all those other things need to change or just plain go away.  You don't like FQ, or Codel, or fq_codel, or the continuing deployment of RFC-3168 compliant middleboxes in general, and you *certainly* don't like me personally.  I've got used to that.

The operative phrase here is that you're entitled to your own opinions, but not your own facts.

> Surely you are not saying the IETF should halt any advances that reduce delay, like Diffserv and L4S, just because the replay windows of VPNs are poorly configured today?

Configuring the replay window size is a game of compromises.  If it's too small, then a variety of effects can result in legitimate packets falling outside the window.  If it's too large, then there is presumably a greater risk of the replay attacks that it's designed to protect against being inadvertently permitted.  I don't know enough about the details of that to judge how serious that particular risk is - but the fact that many VPNs default to values near the minimum recommended in RFC-4303 is probably not a gross error of judgement.

I do note that RFC-4303 significantly post-dates most of the Diffserv specifications, so it notionally had the opportunity to consider the implications.  One way that VPNs could sidestep this particular problem in a Diffserv context is to fix a single DSCP on all outer headers, ensuring that they'll go through the same queue - and at least some of them actually do this.

But they can't really do the equivalent of this to the ECN field in an L4S context, because that would trigger the CE ambiguity problem without also sending the L4S traffic through the L queue.  Not unless they put Not-ECT on the outer header unconditionally - but that *would* be a step backwards.

I hope you can see the difference here.

> [BB] A hard ceiling in /soft/ware?

It's a value that you can't exceed merely by userspace configuration, only by patching the kernel.  Hence there will be significant inertia in raising it, if it needs to be raised.  Hence my question - *is* it large enough?  And following on from that, what are the downsides of choosing such a large value?

> [BB] So the player has a strong incentive to configure the replay window of her VPN properly.

For normal operation, the default replay window is big enough.  For network paths that deliver packets more-or-less in-order, the default replay window is big enough.  It's only when an attack arises on a vulnerable path - a path *made* vulnerable by the installation of DualQ - that an adjustment becomes necessary.

So where is the cause, exactly?  And where should the cure be applied?

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 8 May 2021 23:35:20 +0200

Hi Bob,

I a still considering the VPN-tunnel from home-network scenario below ([SM]), since I cnsider that to be an important use case where replay-windows seem to be actively used today, by quite a lot of users.

> On May 8, 2021, at 17:05, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
> 

[SM] This is a good point, see https://datatracker.ietf.org/doc/html/rfc4301#page-50:

"If different classes of traffic (distinguished by Differentiated
   Services Code Point (DSCP) bits [NiBlBaBL98], [Gro02]) are sent on
   the same SA, and if the receiver is employing the optional
   anti-replay feature available in both AH and ESP, this could result
   in inappropriate discarding of lower priority packets due to the
   windowing mechanism used by this feature.  Therefore, a sender SHOULD
   put traffic of different classes, but with the same selector values,
   on different SAs to support Quality of Service (QoS) appropriately.
   To permit this, the IPsec implementation MUST permit establishment
   and maintenance of multiple SAs between a given sender and receiver,
   with the same selectors.  Distribution of traffic among these
   parallel SAs to support QoS is locally determined by the sender and
   is not negotiated by IKE.  The receiver MUST process the packets from
   the different SAs without prejudice.  These requirements apply to
   both transport and tunnel mode SAs.  In the case of tunnel mode SAs,
   the DSCP values in question appear in the inner IP header.  In
   transport mode, the DSCP value might change en route, but this should
   not cause problems with respect to IPsec processing since the value
   is not employed for SA selection and MUST NOT be checked as part of
   SA/packet validation.  However, if significant re-ordering of packets
   occurs in an SA, e.g., as a result of changes to DSCP values en
   route, this may trigger packet discarding by a receiver due to
   application of the anti-replay mechanism."

In short, this potential for re-ordering was recognized and the recommendation seems to be to not mix different DSCPs in the same "tunnel/SA"...
Additionally rfc4301 also seems to allow not exposing inner DSCP codepoints on encapsulation at all. 
Also note how the recommendation is NOT just increase the replay-window size until the issue goes away.

So for ipsec the RFCs seem to already tackle the issue of exposed DSCPs "accidentally" resulting in packet reordering through differential prioritization.

Which brings us back to the new re-ordering cause that L4S introduces... 
Extrapolating from rfc4301 there are multiple options:

A) set up two SAs, one for ECT(1) and one for ECT(0) and steer packets into the appropriate one 
	-> this is incomplete since it hits the old CE ambiguity issue that dctcp/L4S introduce. We could fix this by giving L4S a proper 1bit ID nstead of the currently proposed 0.5 bit approach.

B) re-mark the ECN bits to 00 on encapsulation
	-> this foregoes all ECN benefits, but should not cause reordering.

C) heuristically increase the replay-window until the error messages subside...
	-> this is tricky, because the amount of re-ordering an L4S AQM will introduce depends on its local latency targets for the C (and less for the L queue) which can not be unambiguously discovered from endpoints. Currently the recommended value seems 25 ms, but a) that recommendation is not based on numeric analysis, and b) it is one of an L4S AQMs "free" configuration parameters, so it can not be relied to always be 25ms*.
And it invites the question of the cost benefit ratio of large replay windows...

Best Regards
	Sebastian

*) This is essentially the same issue TCP Pragues RTT-debias code has, to undo the DualQ class unfairness, TCP Prague should be configured such that the cwin growing function behaves as if the experienced RTT was true RTT + (C-queue latency target - L-queue latency target), but only the first of these three values can be empirically be deduced...

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 8 May 2021 23:12:47 +0100

Sebastian,

On 08/05/2021 19:09, Sebastian Moeller wrote:
> 	[SM] Well possible, but not that relevant, as far as I understand the Linux source code wireguard and ipsev/xfrm only propagate the ECN bits between inner and outer layers and do not propagate the other 6 dscp bits (of the former TOS byte).

[BB] For a patch that adds DSCP propagation to the outer of a wireguard 
tunnel, see 
https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html

Also, there are many more VPNs on the Internet than the software you 
happen to use.

Throughout the world there's a thriving market in VPN services for home 
office working, corporate satellite offices, inter-business 
collaborations, and so on. These VPNs propagate the DSCP to the outer, 
so they can use Diffserv service like this one in the published spec of 
the 'Wholesale Broadband Connect' service my previous employer (BT) 
offers in the UK. See section 5.8 for the DSCPs supported:
https://www.bt.com/bt-plc/assets/documents/sinet/sins/downloads/472v2p11.pdf
This is just one ISP's wholesale product sold to retail ISPs, who then 
build on top the services they sell to their customers, such as VoIP, 
corporate VPNs, etc.

> In fact the issue I see is that L4S introduces a novel re-ordering condition that has not exited before and that hence is on nobody's radar.

[BB] When all the flows are within a VPN, the reordering within a flow 
in the DualQ is indistinguishable from reordering between flows, as far 
as the VPN sequencing is concerned. Reordering between flows within a 
VPN is not a new phenomenon at all - it's expected.

> 	[SM] Interesting case but different from the typical end-user uses a VPN case in which the issue will potentially crop up. VPNs (full or split tunnel) from end users to VPN providers or into the o…

[BB] You just lost your bet. Not a good idea to bet that no-one uses a 
technology that is being sold all round the world, especially not on the 
IETF list for that technology.

> 	[SM] The other way around, DualQ requires an increases replay-window, it is L4S that is the root cause of the configuration change here, since it constitutes a new mechanism for re-ordering.

[BB] Here we go again. Let's find a problem that already exists, show 
L4S suffers from it, then blame L4S. This is tiresome.

> 	[SM] How that? Pete demonstrated that recommended default values of 32 or 64 packets are already enough to see the issue, but the same 32/64 packets seem to work reasonably well for the re-ordering one might encounter on the existing internet.

[BB] How do you know? As Pete said, this is the sort of problem that it 
would be very difficult to diagnose.

A quick search found this Cisco troubleshooting guide for replay drops.
https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html
You'll see this text in the problem statement:
"Certain QoS features, such as Low Latency Queueing (LLQ), can cause 
IPSec packet delivery to become out-of-order and dropped by the 
receiving endpoint due to a replay check failure."

If a VoIP flow within a VPN went through a low latency queue, and a BE 
flow within the VPN didn't, the BE flow would experience replay drops as 
it pushed up its the delay in its own queue. That would sort-of act like 
an AQM - pretty cool! (except no burst tolerance) ...
...I'm distracting myself. My point was going to be that the drops would 
always be from the higher delay queue. Normally no-one would feel a need 
to check why there were some packet drops from a BE flow. So this could 
well be going on unnoticed right under our noses (speculation).

> As I indicated during the re-ordering of CE discussion some time ago, declaring a specific level of re-ordering benign requires a set of assumptions, and these might or might not hold... in the VPN with replay-protection case (which seems to be the recommended mode) these assumptions do not seem to hold.

[BB] I'm saying the number 64 for a reply window is insane at today's 
packet rates (it's perfectly sane for implementation efficiency, but 
perfectly insane in terms of sensitivity to reordering).

> 	[SM] And no wordsmithing in any RFC is guaranteed to reach the current operators/users of such tunnels any time soon.

[BB] I wasn't meaning wordsmithing. There's a technical conflict to be 
resolved.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 9 May 2021 01:59:28 +0200

Hi Bob,

> On May 9, 2021, at 00:12, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] For a patch that adds DSCP propagation to the outer of a wireguard tunnel, see https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html

[SM] But see actual wireguard implementation in the Linux kernel (https://elixir.bootlin.com/linux/v5.12.2/source/drivers/net/wireguard/send.c):

skb_queue_walk(&packets, skb) {
		/* 0 for no outer TOS: no leak. TODO: at some later point, we
		 * might consider using flowi->tos as outer instead.
		 */
		
		PACKET_CB(skb)->ds = ip_tunnel_ecn_encap(0, ip_hdr(skb), skb);
		PACKET_CB(skb)->nonce =
		
		atomic64_inc_return(&keypair->sending_counter) - 1;

		if (unlikely(PACKET_CB(skb)->nonce >= REJECT_AFTER_MESSAGES))			
		goto out_invalid;
}

See the comment and the first zero argument to ip_tunnel_ecn_encap? That is where current wireguard irgnores the DSCP and enforces DSCP 0. I have actually looked into the source while trying to understand this issue.

> Also, there are many more VPNs on the Internet than the software you happen to use.

	[SM] True, and nothing I claimed to be otherwise. I do mention though that for end-users wireguard  and OpenVPN are common and attractive VPN implementations since they are supported by commercial VPN endpoints (which makes them a bit more attractive than pure ipsec) and the clients are free to use.

> Throughout the world there's a thriving market in VPN services for home office working, corporate satellite offices, inter-business collaborations, and so on. These VPNs propagate the DSCP to the outer,

	[SM] What fraction of them does that, and what fraction allows to disable that?

> so they can use Diffserv service like this one in the published spec of the 'Wholesale Broadband Connect' service my previous employer (BT) offers in the UK. See section 5.8 for the DSCPs supported:
> This is just one ISP's wholesale product sold to retail ISPs, who then build on top the services they sell to their customers, such as VoIP, corporate VPNs, etc.

	[SM] This is interesting data, but orthogonal to my point and report. If such VPNs do this, they need to solve the re-ordering issue. BTW, the document does not go into any details of the VPN, so no way to figure out whether it actually protects against replay attacks.

> [BB] When all the flows are within a VPN, the reordering within a flow in the DualQ is indistinguishable from reordering between flows, as far as the VPN sequencing is concerned. Reordering between flows within a VPN is not a new phenomenon at all - it's expected.

	[SM] Encrypted traffic is in now way different from other internet traffic, hence some re-ordering along a network path has to be accepted, true; network operators would prefer that "some" to be larger, end-users/protocols would prefer that "some" to be smaller. But the observed up to ~50ms speed-up that ECT(1) flows see over ECT(0) in the DualQ AQM (with default targets) is just far outside of the expected "some" for VPNs. It is unfortunate (for L4S), that replay windows are sensitive for expedited early packets, while ACK streams seem more affected by delayed packets*. 

*) that is the theory, there has been little data to actually conform that hypothesis 

> [BB] You just lost your bet. Not a good idea to bet that no-one uses a technology that is being sold all round the world, especially not on the IETF list for that technology.

	[SM] So you have numbers of relative usage of the different VPNs from end users home networks? If so please post a link to the source of that knowledge.
 My hypothesis is that most end users will use either OpenVPN or wireguard which both do not seem to propagate DSCP bits around by default... 

> [BB] Here we go again. Let's find a problem that already exists, show L4S suffers from it, then blame L4S. This is tiresome.

	[SM] That is not how I see the issue. While this is not a conceptually novel problem L4S created, it is an issue that L4S makes significantly worse compared to the status quo. Not my responsibility that L4S has many of these problematic spots (RTT-bias, equitable sharing between C and L queues, lack of even a single fully requirement fulfilling protocol, rfc3168 incompatibility, the list goes on).

> [BB] How do you know? As Pete said, this is the sort of problem that it would be very difficult to diagnose.

	[SM] For example wireguard will throw an error:

net_dbg_ratelimited("%s: Packet has invalid nonce %llu (max %llu)\n",
	peer->device->dev->name,
	PACKET_CB(skb)->nonce,				    
	keypair->receiving_counter.counter);

logging the reception of a "stale" packet (as will OpenVPN). So the "noticing something is afoot" part is not that subtle, IMHO getting to the "where and why" the error was thrown is difficult to diagnose. 

> "Certain QoS features, such as Low Latency Queueing (LLQ), can cause IPSec packet delivery to become out-of-order and dropped by the receiving endpoint due to a replay check failure."
> 
> If a VoIP flow within a VPN went through a low latency queue, and a BE flow within the VPN didn't, the BE flow would experience replay drops as it pushed up its the delay in its own queue. That would sort-of act like an AQM - pretty cool! (except no burst tolerance) ...
> ...I'm distracting myself. My point was going to be that the drops would always be from the higher delay queue. Normally no-one would feel a need to check why there were some packet drops from a BE flow. So this could well be going on unnoticed right under our noses (speculation).
> 
> [BB] I'm saying the number 64 for a reply window is insane at today's packet rates (it's perfectly sane for implementation efficiency, but perfectly insane in terms of sensitivity to reordering).

	[SM] Mmmh, that is the point, Pete's data shows the issue already at 10 Mbps, which is not a high packet rate, see http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s9-tunnel-reordering/l4s-s9-tunnel-reordering-ns-ipsec-replay-win-32-dualpi2-10Mbit-20ms_tcp_delivery_with_rtt.svg. At 10 Mbps a 64 packet window covers up to* (32*1538*8) / (10 * 1000^2) * 1000 = 39.37 ms. It takes a bit of reordering along the path to exceed that replay window... 

*) A fixed packet window obviously will have a duration at bottleneck rate equivalent that depends on the packet size distribution inside the current window.

> [BB] I wasn't meaning wordsmithing. There's a technical conflict to be resolved.

	[SM] Well, one could try to get better recommendations for how to set the replay window into other ipsec/replay-window RFCs.
That in turn leads to the question how these would look?
For L4S Pete's proposal of something like 
(2 * C-queue latency target [ms]) / ((average packet size [bit]) / (bottleneck rate [bit/ms]))
or for 25 ms and 10 mbps:
(2 * 25) / ((1538*8) / (10 * 1000^2) * 1000)  = 40.6371911573
might do, at least for dealing with L4S, IFF the C-queues latency target would not be a free configuration parameter of the L4S AQM, that can be set to arbitrary values...
But I might be over seeing something here, so please explain your solution to this conundrum.

And that still leaves the question how to let operators of existing replay-protected tunnels now that they are supposed to fix somebody else's experiment by adapting to new configurations.

Regards
	Sebastian

---

From: "Black, David" <[email-masked]>
Date: Sun, 9 May 2021 02:51:36 +0000

Top posting to try to add some insight.

There are multiple different sorts of networking technology that use the virtual private network (VPN) term.  The specific technology under discussion here is secure VPNs that use IPsec, TLS or DTLS, and specifically secure VPNs that provide an anti-replay service.  The BT WBC offering that Bob provided a reference to may be a VPN, but it is not a secure VPN (e.g., the only mention of IPsec in the referenced document occurs in cautionary text about the impact of tunneling on MTU).

As Sebastian has pointed out, IPsec (see RFC 4301) was designed based on Diffserv and ECN as they existed at that time (e.g., RFC 4301 cites RFCs 2474, 2475 and 3168), and RFC 4301 covers the interaction of Diffserv QoS differences with anti-replay, including providing mechanisms to deal with that interaction, so I'm not seeing a significant inconsistency between Diffserv and IPsec.  FWIW, I would not expect to, as I worked on this area of RFC 4301, e.g., as the author of RFC 2983.

I observe that appropriate L4S usage of a Guard DSCP would result in L4S fitting cleanly into the RFC 4301 provisions for reordering caused by Diffserv, and on that basis I'll politely decline Bob's implicit invitation to revise RFC 4301 (IPsec Architecture) in order to support the L4S experiment's choice of identifier.

Thanks, --David

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
Sent: Saturday, May 8, 2021 7:59 PM
To: Bob Briscoe
Cc: TSVWG
Subject: Re: [tsvwg] L4S dual-queue re-ordering and VPNs

[EXTERNAL EMAIL] 

Hi Bob,

> [BB] For a patch that adds DSCP propagation to the outer of a wireguard tunnel, see https://urldefense.com/v3/__https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD_94Muae$ [lists[.]zx2c4[.]com]

[SM] But see actual wireguard implementation in the Linux kernel (https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.12.2/source/drivers/net/wireguard/send.c__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD6hnvgVD$ [elixir[.]bootlin[.]com]):

		

> https://urldefense.com/v3/__https://www.bt.com/bt-plc/assets/documents/sinet/sins/downloads/472v2p11.pdf__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD0Rynjgg$ [bt[.]com]

>>> David himself has written about the reordering problem when a WebRTC application encapsulates TCP, SCTP and RTP flows with different DSCPs within UDP [RFC7657]. Indeed, datagram transport layer security (DTLS) is a common encapsulation for WebRTC flows. And DTLS also recommends a default replay window of 64 (see https://urldefense.com/v3/__https://tools.ietf.org/html/rfc6347*section-4.1.2.6__;Iw!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZDy4nJfZz$ [tools[.]ietf[.]org] ).

> https://urldefense.com/v3/__https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD9D3Q2Lg$ [cisco[.]com]

	[SM] Mmmh, that is the point, Pete's data shows the issue already at 10 Mbps, which is not a high packet rate, see https://urldefense.com/v3/__http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s9-tunnel-reordering/l4s-s9-tunnel-reordering-ns-ipsec-replay-win-32-dualpi2-10Mbit-20ms_tcp_delivery_with_rtt.svg__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD1vzqfng$ [sce[.]dnsmgr[.]net]. At 10 Mbps a 64 packet window covers up to* (32*1538*8) / (10 * 1000^2) * 1000 = 39.37 ms. It takes a bit of reordering along the path to exceed that replay window... 

Regards
	Sebastian

>>>> https://urldefense.com/v3/__https://github.com/heistp/l4s-tests/*dropped-packets-for-tunnels-with-replay-protection-enabled__;Iw!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD7pb7Elu$ [github[.]com]
>>>> 
>>>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc4303*section-3.4.3__;Iw!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZD4c2bPtT$ [datatracker[.]ietf[.]org] .
>>>>> 
>>>>> Thanks, --David
>>>>> 
>>> Bob Briscoe                               https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZDwqJJoHk$ [bobbriscoe[.]net]
>>> 
> 
> -- 
> Bob Briscoe                               https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!3aRj0rF8_yeDW2vWWy5DqZcqd7I2qrSnPx7OyvWWWZh_Bu8BIMkGkTEZDwqJJoHk$ [bobbriscoe[.]net]

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 9 May 2021 15:28:05 +0100

Jonathan,

>>> Perhaps - but usable Diffserv deployments at the last-mile are not very common.

[BB] See email to Sebastian for the global market in business services 
that is built on Diffserv deployments in the last mile. But you are 
right that my point about authorization to use VPNs applies to last-mile 
Diffserv as well - Diffserv's rarely open access. That's why I pointed 
to WMM.

> That's not the last mile - that's within customer premises.

[BB] Is that relevant? I mean, in the context of this issue (I know it's 
not literally what you said).

[BB] All the reasons you give for WMM not being relevant below seem to 
be clutching at straws.

[BB] I found the same text that Sebastian found in the IPsec 
architecture ([RFC4301] section 4.1) saying that different security 
associations 'SHOULD' be used for each Diffserv codepoint. That would 
help rule out Diffserv's vulnerability, but I'm not sure whether IPsec 
implementations follow that advice - anyone know?

> I would also question how often a DSCP would survive on the outer header of a VPN packet long enough to influence this mechanism.

[BB]
* Upstream: Diffserv (typically an abstraction of Diffserv) is 
invariably the interface that OSs present to applications in order to 
use WMM.
* Downstream: ISPs typically supply an access point, so they will often 
hook into WMM from their QoS regime using Diffserv as the go-between.

In the past I'm sure Greg White has given the results of his informal 
study of just how common different DSCPs are on frames snooped from busy 
WiFi environments (downstream and upstream).

> But I would also note the increase in the use of fq_codel in wifi AP stacks, which would tend to reduce any differential delay to a greater degree than the C queue in DualQ does.  Those, taken together, might be significant reasons why problems have not been noticed up to now.

[BB] One would need to imagine that nearly the whole WMM world has 
nearly always used FQ-CoDel for this to be a reason.

> For the particular use case I mentioned, a player connecting over wifi would be ridiculed due to the inconsistent latency that wifi offers, especially in congested radio environments.

[BB] It's the attack traffic that needs to use the lower delay WMM 
access category, not (necessarily) the user.

> As often seems to happen, you seem to want to deflect any possible criticism of L4S onto some other piece of technology that it might happen to interact poorly with, so that you don't need to change L4S, only all those other things need to change or just plain go away.

[BB] If L4S is the problem, I'll say. But I'm trying to establish 
whether this problem already exists, and whether solutions to it have 
been proposed.

The approach a researcher takes when they think they have found a new 
attack is to generalize it, and then see if it's already been proposed 
and studied. So far I haven't found any research that uses a 
pre-existing delay-delta on the path like your attack. The following two 
papers are about related attacks exploiting the IPsec replay window, but 
the attacker needs to be on-path:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.183.2766&rep=rep1&type=pdf
     https://arxiv.org/pdf/0910.3511.pdf

> You don't like FQ, or Codel, or fq_codel, or the continuing deployment of RFC-3168 compliant middleboxes in general, and you *certainly* don't like me personally.  I've got used to that.
>

[BB] A mailing list is for establishing what the facts are.

I should know some facts in this space, because I've been involved in 
the access network QoS business of operators for decades (and I helped 
set up a start-up before operators were doing it). But I also know that 
there's always a lot I don't know. So the normal etiquette is to be 
circumspect about the breadth of your own knowledge, because none of us 
know how much we don't know.

> Configuring the replay window size is a game of compromises.  If it's too small, then a variety of effects can result in legitimate packets falling outside the window.  If it's too large, then ther…

[BB] I also didn't know much about the replay window tradeoff until 
yesterday, but I've been reading. It seems to be a lot to do with 
implementation efficiency. The number 64 for the replay window seems to 
come from the variable you would use to hold the bit map of the replay 
window on a 64-bit machine.

 From IPSec [RFC4302] Appx B.1.:

    (No requirement is established for minimum or
    recommended sizes for this window, beyond the 32- and 64-packet
    values already established for 32-bit sequence number windows.
    However, it is suggested that an implementer scale these values
    consistent with the interface speed supported by an implementation
    that makes use of the ESN option.

I found papers proposing techniques to efficiently use larger bit-maps, 
including RFC6479 (the IPR declaration offers non-assertion with 
reciprocity).

The papers I cited earlier attempted to reason about a recommended 
window size. The unstated rule for preventing replay attacks seems to be 
"as small as possible without creating vulnerabilities to other 
attacks". The paper proposes a replay window needs to be "as large as 
the e2e bandwidth-delay product at the rate supported by the VPN over a 
max delay path". But I don't see any justification for that.

>
> I do note that RFC-4303 significantly post-dates most of the Diffserv specifications, so it notionally had the opportunity to consider the implications.

[BB] That assumes the problem was known.
The 1998 version of IPsec  had the same replay prevention window 
approach as the current (2005) version in RFC4301. But until the 2009 
papers linked above (and your proposed attack), IPsec's anti-replay 
protocol seems to have only been considered useful for /preventing/ DoS 
attacks.

> One way that VPNs could sidestep this particular problem in a Diffserv context is to fix a single DSCP on all outer headers, ensuring that they'll go through the same queue - and at least some of them actually do this.

[BB] "You can't do what you want to do" isn't really a solution. When 
the DSCP is actually being used for different levels of QoS, one needs a 
Diffserv tunnel's uniform mode [RFC2983], i.e. propagation and from outer.

> It's a value that you can't exceed merely by userspace configuration, only by patching the kernel.  Hence there will be significant inertia in raising it, if it needs to be raised.  Hence my questi…

[BB] Yes, those questions arose for me too. See sketchy summary of the 
only paper I can find about this, above.

> For normal operation, the default replay window is big enough.

[BB] Can you justify that statement?

Never mind Diffserv or DualQ. Just the background reordering degree on 
some Internet paths is greater than 64.
     https://www.ietf.org/proceedings/90/slides/slides-90-tcpm-9.pdf
Where this is due to bonded links, even if the reordering degree was 
half a dozen, an attacker could tip it over 64 by sending a stream of 
tiny back-to-back datagrams into the VPN (29B instead of 1500B packets 
could inflate the reordering degree by up to x50). It wouldn't matter if 
the tinygrams were spread across both/all channels; IPsec's replay 
protection would still start deleting genuine packets.

As a feel for the order of magnitude, at only 100Mb/s, 64 injected 
back-to-back min sized datagrams (29B) occupy only 150μs.

> For network paths that deliver packets more-or-less in-order, the default replay window is big enough.  It's only when an attack arises on a vulnerable path - a path *made* vulnerable by the installation of DualQ - that an adjustment becomes necessary.
>

[BB] Still trying to find that answer.

Replay attacks have to be mounted by on-path attackers, and they are 
application-specific but the attacker is blind to what's in the packets. 
Plus e2e security protocols take care about replay attacks themselves 
(salting passwords, etc).
So it seems that it's just useful for a VPN to have /a/ replay window in 
case there might be a replay attack that e2e protocols have not 
protected against.

IPsec replay protection seems like an extra pair of underpants just in case.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 9 May 2021 16:58:54 +0100

David,

On 09/05/2021 03:51, Black, David wrote:
> There are multiple different sorts of networking technology that use the virtual private network (VPN) term.  The specific technology under discussion here is secure VPNs that use IPsec, TLS or DTLS, and specifically secure VPNs that provide an anti-replay service.

[BB] Correct.

> The BT WBC offering that Bob provided a reference to may be a VPN, but it is not a secure VPN (e.g., the only mention of IPsec in the referenced document occurs in cautionary text about the impact of tunneling on MTU).

[BB] I thought I'd made it clear, sorry (Sebastian also misunderstood so 
it must be me). I only offered BT's WBC (and the similar products of 
other operators) as proof that operators offer Diffserv service on the 
access link.

Then Retail ISPs and other companies use it as a component part of their 
service packages like VoIP, or corporate VPNs, or VPNs between 
collaborative companies, or whatever.

Most users wouldn't be aware if their VPN or VoIP was using Diffserv 
underneath or not. But if you've ever wondered why the VoIP that certain 
ISPs offer doesn't suffer from latency under load, whereas an 
Over-The-Top one does, it might be 'cos the former is built over 
Diffserv in the access network.

I can't immediately find an example of a corporate VPN product that uses 
the above Diffserv service.

PlusNet is one example of a UK retail ISP that uses Diffserv (years ago 
I tried to get them into AQM, but they prefer their way). Here's their 
page about it:
https://www.plus.net/help/broadband/about-traffic-prioritisation/ . 
You'll see they detect VPN traffic in the network and apply a DSCP to 
the VPN (rather than the VPN asking for it). So this is /not/ an example 
of a case where there might be different DSCPs within a single VPN.

> As Sebastian has pointed out, IPsec (see RFC 4301) was designed based on Diffserv and ECN as they existed at that time (e.g., RFC 4301 cites RFCs 2474, 2475 and 3168), and RFC 4301 covers the inter…

[BB] All correct.

> I observe that appropriate L4S usage of a Guard DSCP would result in L4S fitting cleanly into the RFC 4301 provisions for reordering caused by Diffserv, and on that basis I'll politely decline Bob'…

[BB] No revision of 4301 is needed (except I'd point out that RFC6040 
already updated it). What did you think my implicit invitation was? It 
was so implicit I must have missed it.

The issue is whether the non-mandatory advice in 4301 is being followed:
* scale the replay window with bit-rate,
* separate DSCPs per SA.

On this last one, did you see my question earlier in the thread asking 
whether someone knows whether the 'SHOULD' in RFC4301 recommending one 
SA per DSCP is generally adhered to? I imagine implementers might cut 
corners on that SHOULD, due to the complexity of setting up multiple SAs.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 9 May 2021 17:50:12 +0100

Sebastian,

Hopefully some of my responses to David's response to you were useful. A 
little more added below, [BB]...

> See the comment and the first zero argument to ip_tunnel_ecn_encap? That is where current wireguard irgnores the DSCP and enforces DSCP 0. I have actually looked into the source while trying to und…

I understood when you said it's not in the mainline. My intent in 
pointing to that patch was to show that other products/distros that do 
propagate DSCPs have been based on wg.

> 	[SM] This is interesting data, but orthogonal to my point and report. If such VPNs do this, they need to solve the re-ordering issue.

[BB] That was my point - it's not just an L4S issue.

> BTW, the document does not go into any details of the VPN, so no way to figure out whether it actually protects against replay attacks.

[BB] Hope response to David covers this.

> 	[SM] Encrypted traffic is in now way different from other internet traffic, hence some re-ordering along a network path has to be accepted, true; network operators would prefer that "some" to be l…

[BB] Agreed.

>   My hypothesis is that most end users will use either OpenVPN or wireguard which both do not seem to propagate DSCP bits around by default...

[BB] I only know what I know from talking with customers and potential 
customers, esp. in the finance sector, which was where I got called in 
more, due to latency questions. I don't know of any market surveys.

> 	[SM] That is not how I see the issue. While this is not a conceptually novel problem L4S created, it is an issue that L4S makes significantly worse compared to the status quo.

[BB] I thought you agreed it would likely be worse with Diffserv (which 
is status quo).

> Not my responsibility that L4S has many of these problematic spots (RTT-bias, equitable sharing between C and L queues, lack of even a single fully requirement fulfilling protocol, rfc3168 incompatibility, the list goes on).
>
>

[BB] See last email to Jonathan, about VPN replay protection being 
second line of defence behind replay protection in e2e security protocols.

But I agree that telling VPN users to change for an experiment is lame. 
Fortunately, I doubt anyone will notice if they leave their replay 
window as it is. Although this is not a fifth order low probability as 
in the spurious retransmits problem, it is a third order low 
probability, with the same first two points as for spurious retransmits:
1) multiple bottlenecks on a path are rare
2) the first bottleneck has to be a classic AQM and the second an L4S DualQ
3) running a VPN with replay protection enabled, while coinciding with 
both 1 & 2.

I should maybe add a point zero that Jonathan often makes about FQ AQMs:
0) bottlenecks tend to be congested only transiently

> logging the reception of a "stale" packet (as will OpenVPN). So the "noticing something is afoot" part is not that subtle, IMHO getting to the "where and why" the error was thrown is difficult to d…

[BB] I meant how do you know "the same 32/64 packets seem to work 
reasonably well for the re-ordering one might encounter on the existing 
internet"?

Log messages abound for all sorts of bumps and farts. Few admins spend 
their time looking for problems to solve.

> 	[SM] Mmmh, that is the point, Pete's data shows the issue already at 10 Mbps, which is not a high packet rate, see http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s9-tunnel-reordering/l4s-s9-tunnel-reordering-ns-ipsec-replay-win-32-dualpi2-10Mbit-20ms_tcp_delivery_with_rtt.svg.

[BB] I mean 64 is insane today (without L4S). See email to Jonathan with 
pointer to reordering degree on the Internet today, and example where 
even 150μs jitter between two bonded links could be used by an attacker 
to trigger VPN drops at just 100Mb/s.

> At 10 Mbps a 64 packet window covers up to* (32*1538*8) / (10 * 1000^2) * 1000 = 39.37 ms. It takes a bit of reordering along the path to exceed that replay window...
>
>

Why 32 in your formula, not 64?

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 9 May 2021 23:07:39 +0200

Hi Bob,

more below, prefixed [SM2].

> On May 9, 2021, at 18:50, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> Hopefully some of my responses to David's response to you were useful. A little more added below, [BB]...
> 
> I understood when you said it's not in the mainline. My intent in pointing to that patch was to show that other products/distros that do propagate DSCPs have been based on wg.

	[SM2] Your link points to a post in the wireguard mailing list with exactly zero indication that a) that patch went anywhere, and that b) it might have made it into other distributions. so what additional information makes you believe that these other products/distros actually exist, that are based on a patched wireguard? 

> [BB] That was my point - it's not just an L4S issue.

	[SM2] Sure, but as before with RTT-bias, L4S adds its own significant increase in relevant re-ordering even IFF dscps do not come into play, and that is an L4S issue.

> [BB] Hope response to David covers this.

	[SM2] Honestly, this feels like a distraction, introducing another class of "VPNs" to distract from the issue that "early re-ordering" is not as benign as you made it sound...

> [BB] I only know what I know from talking with customers and potential customers, esp. in the finance sector, which was where I got called in more, due to latency questions. I don't know of any market surveys.

	[SM2] Well see my condition "VPNs (full or split tunnel) from end users to VPN providers or into the office", I am talking about end-users here, I clearly should have made that clearer. And in my experience ISPs either do not accepts arbitrary DSCPs from mere end-users, at best they keep the dscp but give no differential treatment, at worst they re-mark as they see fit... (my experience is limited, so this is anecdotal only, I claim no generality here)

> [BB] I thought you agreed it would likely be worse with Diffserv (which is status quo)

	[SM2] Not what I intended to convey. In my setting of end-users using an encryppted VPN from their home link, and either OpenVPN or nowadays more often wireguard, DSCP propagation is not happening by default and hence can not be a significant cause of concern, I hope that clarifies my position.

> [BB] See last email to Jonathan, about VPN replay protection being second line of defence behind replay protection in e2e security protocols.
> 

	[SM2] Unexpected, but good!

> Fortunately, I doubt anyone will notice if they leave their replay window as it is.
> Although this is not a fifth order low probability as in the spurious retransmits problem, it is a third order low probability, with the same first two points as for spurious retransmits:
> 3) running a VPN with replay protection enabled, while coinciding with both 1 & 2.

	[SM2] We might be talking past each other, but Pete's scenario is much simpler, one ECT(1) "greedy" flow and one ECT(0) "greedy" flow through an ipsec tunnel with replay protection pass through an L4S AQM, and depending on the replay-window size the ECT(0) flow gets less than expected throughput, while the ECT(1) flow gets more.
Specifically this means that:
1) the number of bottlenecks is pretty irrelevant, one is sufficient, it does need to be DualQ though.
2) is irrelevant since one L4S AQM at one bottleneck is sufficient
3) is required for the new failure mode, but the relevant 1) & 2) are considerably weaker constraints than in your list.

You might be talking about a different problem here, but I am still concerned about simple functionality under normal expected conditions, I am not looking into attempts at malice here.

> 0) bottlenecks tend to be congested only transiently

	[SM2] Which is enough to cause the issue, and the classic queue is expected to see periodic peaks (the data transmitted into the network between the AQM emitting a CE/drop and the sender learning about that)

> [BB] I meant how do you know "the same 32/64 packets seem to work reasonably well for the re-ordering one might encounter on the existing internet"?

	[SM2] My hypothesis here is that networks tend to be designed to give reasonable performance with TCP, which already is re-ordering sensitive, but I have no hard data to back up that hypothesis yet. Why 32/64, here I agree with you that was convenient for some implementation to simply interpret a 32 or 64 integer as bitfield, but it seems that works reasonably well (not perfect though).

> Log messages abound for all sorts of bumps and farts. Few admins spend their time looking for problems to solve.

	[SM2] Might be true, but people I know tend to take error message of security/privacy applications more serious than other notifications. If you do not bother whether encryption actually works you are probably much better of with plain encapsulation without encryption, less work.CPU-load. 

> [BB] I mean 64 is insane today (without L4S). See email to Jonathan with pointer to reordering degree on the Internet today, and example where even 150μs jitter between two bonded links could be used by an attacker to trigger VPN drops at just 100Mb/s.

	[SM] Not sure this is the right venue to discuss recommendations for default replay-window sizes, the relevant audience simply is not here. This is the right venue though to argue that L4S should do its best not to increase re-ordering gratuitously... in the end applications are only tolerant to re-ordering to a certain degree, so we should think in terms of an acceptable re-ordering budget, and I do not think we should invest that budget to help out L4S side-effects.

> Why 32 in your formula, not 64?

	[SM2] Good point, sorry. I initially posted a link to the replay-win-64 graph and had calculations for 64 packets, when I changed that to 32 I failed to update all references. The reason for using the 32 is simply that the issue at hand is more subtle in the 64 packet case (and gets confounded with L4S default biased sharing, almost always giving ECT(1) flows an extra helping of capacity, but since the effect of L4S on re-ordering is an independent issue, let's ignore that here).

Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 10 May 2021 16:00:45 +0300

> On 10 May, 2021, at 12:07 am, Sebastian Moeller <[email-masked]> wrote:
> 

And, more to the point, the much more serious DoS scenario involves an attacker deliberately inducing *persistent* congestion at the DualQ node.  That renders any argument that it "only happens occasionally" completely moot; the attacker can choose when those occasions occur.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 18 May 2021 09:06:51 +0100

Sebastian,

I think this thread has converged on agreement on every point - so no 
need to reply.
See [BB2] for a couple of clarifications.

> 	[SM2] Well see my condition "VPNs (full or split tunnel) from end users to VPN providers or into the office", I am talking about end-users here, I clearly should have made that clearer. And in my…

[BB2] You're in general correct. I already pointed that out a couple of 
posts ago in this thread: 
https://mailarchive.ietf.org/arch/msg/tsvwg/qtEJqivzZUDQeyLehdlLbI3leF8/

> 	[SM2] Not what I intended to convey. In my setting of end-users using an encryppted VPN from their home link, and either OpenVPN or nowadays more often wireguard, DSCP propagation is not happening…

[BB2] This is a view of a world populated by techies who choose their 
own VPN software. I accept that the VPN products that you select from 
don't propagate the DSCP. But we also need to think of all the 
commercial VPN products that IT departments choose and install on behalf 
of their end-users so they can access the work intranet from home or 
while on the road. That's all I'm trying to explain here.

Regards

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 18 May 2021 11:39:20 +0200

Hi Bob,

> On May 18, 2021, at 10:06, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> I think this thread has converged on agreement on every point - so no need to reply.

	[SM] I just want to emphasize one point below, see [SM3], I took the liberty to reduce the cited text.

>>> [...]
> [BB2] This is a view of a world populated by techies who choose their own VPN software. I accept that the VPN products that you select from don't propagate the DSCP. But we also need to think of all the commercial VPN products that IT departments choose and install on behalf of their end-users so they can access the work intranet from home or while on the road. That's all I'm trying to explain here.

	[SM3] I think that getting change into commercial VPN products to change is a much easier task than to get in into all end-user run VPNs. For one there is only a limited number of relevant VPN vendors/developers that would need to "bake" changes into their product and/or their documentation and that might even be able to send all of their customers information about expected changes (via update notes or maintenance contacts). Also if an IT department exists that is in charge of operating a VPN, IMHO they can be reasonably expected to understand/research how to implement such a change or find different work-arounds. 
	Normal end-users will have to do all of this themselves... without professional help, this is why I want to focus on that perspective of the issue (and I do not want to imply that getting change into commercial VPNs is going to be easy, just a lot easier than without).

Best Regards
	Sebastian