Subject: [tsvwg] L4S & VPN anti-replay interaction: Explanation

From: "Black, David" <[email-masked]>
Date: Tue, 11 May 2021 14:56:18 +0000

This message is the explanation of the interaction of L4S and VPN anti-replay that I promised to send during the interim meeting yesterday.  I'm writing (explaining) as an individual, not as a WG chair.

This explanation is for IPsec VPNs, although it applies in principle to any VPN that propagates the ECN field to outer IP headers.

For IPsec, the primary RFCs involved are RFC 4301 (IPsec architecture, https://datatracker.ietf.org/doc/rfc4301/) and RFC 4303 (ESP - Encapsulating Security Payload, https://datatracker.ietf.org/doc/rfc4303/).  RFC 4302 (AH - Authentication Header, https://datatracker.ietf.org/doc/rfc4302/) was mentioned in discussion, but that RFC is not generally applicable to VPNs because AH does not encrypt its payload.  As a result, this explanation focuses on ESP anti-replay functionality, as specified in Section 3.4.3 of RFC 4303 (https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3).  This explanation also assumes a tunnel-mode IPsec VPN, where the outer IP header is applied and removed by the VPN sender and receiver.  Tunnel-mode is a common configuration, as it separates VPN IP addressing (in inner IP headers) from Internet IP addressing (in outer IP headers).

Here's a simplified summary of how ESP anti-replay works:

  *   The VPN sender numbers each packet with a sequence number that is incremented for each packet.
  *   The VPN receiver uses a sliding sequence number window to scoreboard (track) which packets have been received and reject duplicates.
     *   There's a bit per packet within the window that indicates whether that packet has been received.
     *   The scoreboard is initialized with all bits cleared to '0'.  Setting a bit to '1' indicates that the corresponding packet has been received.
  *   Receipt of a packet whose sequence number is within the sliding window causes the VPN receiver to check the bit for that sequence number:
     *   If that bit is already set to '1', the packet is a duplicate and is discarded.
     *   If that bit is cleared to '0', that bit is set to '1' and the packet is processed and forwarded.
  *   Receipt of a packet whose sequence number is larger than the range covered by the sliding window causes the window to advance to include that sequence number (clearing new scoreboard bits to '0'), and then proceeds as above because the packet's sequence number is within the window after the advance.
     *   The packet sequence number is included in ESP crypto computations, so rogue packets injected by an attacker who does not know the relevant VPN key(s) are discarded without advancing the window.
  *   Receipt of a packet with a sequence number less than the range covered by the window always causes the packet to be discarded.
  *   Here's the entire paragraph on scoreboard window size (last paragraph in Section 3.4.3 of RFC 4303):

           A minimum window size of 32 packets MUST be supported when 32-bit

           sequence numbers are employed; a window size of 64 is preferred and

           SHOULD be employed as the default.  Another window size (larger than

           the minimum) MAY be chosen by the receiver.  (The receiver does NOT

           notify the sender of the window size.)  The receive window size

           should be increased for higher-speed environments, irrespective of

           assurance issues.  Values for minimum and recommended receive window

           sizes for very high-speed (e.g., multi-gigabit/second) devices are

           not specified by this standard.

  *   Prior discussion on this list indicates that the 32 and 64 window size values are present in some running code, at least as defaults.

L4S can exhibit undesirable interactions with ESP anti-replay based only on ECT(0) and ECT(1) in the outer headers as follows:

  1.  Simultaneously send L4S flows (ECT(1) in ECN field) and Classic flows (ECT(0) in ECN field) through the same VPN.
  2.  The VPN sender assigns in-order sequence numbers to the packets independent of which type of flow they belong to.
  3.  The VPN sender propagates the ECN field in each packet to the outer IP header, as required by the tables in Sections 5.1.2.1 and 5.1.2.2 of RFC 4301.
  4.  One or more network node bottlenecks between the VPN sender and receiver apply L4S treatment to the L4S ECT(1) packets, causing them to advance past some Classic ECT(0) packets.

At the VPN receiver, the received L4S packets drive advancing the sliding window to higher sequence numbers.  If the sequence numbers in the arriving L4S packets get ahead of the sequence numbers in the arriving Classic packets by the size of the sliding window or more, the arriving Classic packets are dropped *even though they are not duplicates* (!).

The problem arises at step 1 - mixing L4S and non-L4S traffic in the same VPN.  IPsec has addressed the analogous problem for Diffserv by requiring (MUST) support of multiple VPN sessions (which IPsec calls Security Associations, SAs) between the same endpoints, and strongly recommending (SHOULD) that traffic in different Diffserv classes use different SAs (RFC 4301, Section 4.1, paragraph spanning pp.13-14).

At least two incorrect assertions have been made about IPsec anti-replay in list discussion and/or in the interim meeting:

  *   The undesirable anti-replay interaction is neither specific to nor caused by L4S.
     *   That is incorrect because RFC 4301 has addressed the anti-replay interaction for Diffserv.  In contrast, L4S is new wrt RFC 4301.
  *   The undesirable anti-replay interaction only occurs if CE marks are present in some of the packets involved.
     *   This is incorrect because ECT(0) and ECT(1) values in the ECN field suffice to create the problem, see above.

I'll close by noting that if L4S flows carried a DSCP that differed from Classic flows, that would enable a "reduce to previous case" line of reasoning (yes, I used to be a mathematician) that reuses the IPsec provisions for Diffserv to address this anti-replay interaction with L4S.  That reuse won't prevent all reordering-caused anti-replay problems that could occur, but it would take advantage of existing/known techniques for dealing with them.

Thanks for reading this far, --David

David L. Black, Sr. Distinguished Engineer, Technology & Standards
Infrastructure Solutions Group, Dell Technologies
mobile +1 978-394-7754 [email-masked]<mailto:[email-masked]>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 11 May 2021 18:26:59 +0300

> On 11 May, 2021, at 5:56 pm, Black, David <[email-masked]> wrote:
> 

The practical effects of this should also be described to give a full picture.

With normal, congestion-controlled traffic, the above condition results only during relatively brief periods when the C queue builds to some level (which may or may not be up to the AQM threshold).  The packets dropped for arriving outside the replay window cause a reduction in cwnd for that flow, which tends to empty the C queue.  The ultimate result is a reduction in throughput for traffic in the VPN using the C queue.

However, this behaviour can also be exploited by an attacker using unresponsive traffic to the C queue and a relatively small flow to the L queue, at least the latter being directed through the VPN.  This is capable of causing *all* VPN traffic using the C queue to be discarded due to violating the replay window, because the C queue is kept full permanently by the unresponsive traffic.  This is a Denial of Service attack which warrants a "Serious" risk severity rating.

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Tue, 11 May 2021 08:48:29 -0700

Dear David:

This was an extremely clear discussion of what can go wrong with
reordering in a vpn tunnel.
I, like many other linux devs, have switched to wireguard and not
looked back, as we made site-site vpns like ipsec and that work
beautifully with fq_codel also on that bottleneck link, and userspace
vpns like openvpn have severe bufferbloat problems whether run

I've long meant to get on top of openvpn to have it manage its
internal queues better.

In reading your description I ultimately have 3 points to make.

1) I was reminded of how hard it was to diagnose and fix a similar but
different crypto/reordering interaction problem over wifi we
encountered in early versions of the "ending the anomaly" codebase,
which I'd written up, as amusingly as possible, over here:

http://blog.cerowrt.org/post/crypto_fq_bug/

I am curious what other tools, besides tsde as used there, exist to
detect persistent re-ordering issues, as a plugin to wireshark, or so
on?

2)

I don't know a lot about userspace p2p vpns nowadays, but after a
difficult encounter with
one corp that shall remain nameless's universal per employee vpn tech,
I plan to go take a hard look at

http://www.infradead.org/openconnect/ to see what they do as to
encapsulating dscp and ecn markings. Not having sources to many other
ostensibly secure vpn technologies kind of bothers me... I guess I
should go look hard at ssh also.

3)

Exposing different diffserv codepoints in a single flow to a wifi
stack will end up dropping
it into different wifi AC classes, which have dramatically different
media access properties.
The maximum number of packets that can fit into each of the 4 classes
each is 64k or 42 packets on 802.11n, I think it is 96 or higher (and
4Mbytes) on 802.11ac, and I'm about to go review the now final ax spec
to see if they were paying attention to how vpns might behave over it.

---

From: "Black, David" <[email-masked]>
Date: Tue, 11 May 2021 18:21:39 +0000

Dave,

Thanks - I have one clarification on the third item in your message:

> 3) Exposing different diffserv codepoints in a single flow ...

For clarity, that's not part of the scenario here.  Each transport flow would use a single DSCP, which differs for L4S vs. Classic flows so that a Diffserv classifier can direct L4S and Classic traffic to different IPsec SAs.  FWIW, Transport protocols such as TCP have a history of not working well with multiple DSCPs in the same flow - see RFC 7657 for details (https://datatracker.ietf.org/doc/rfc7657/).

In contrast to the ECN field, a VPN sender for an IPsec tunnel-mode VPN is allowed to change the DSCP as part of tunnel encapsulation - an example described in earlier list discussion always clears the outer DSCP to zero.

Thanks, --David

-----Original Message-----
From: Dave Taht <[email-masked]> 
Sent: Tuesday, May 11, 2021 11:48 AM
To: Black, David
Cc: [email-masked]
Subject: Re: [tsvwg] L4S & VPN anti-replay interaction: Explanation

[EXTERNAL EMAIL] 

Dear David:

2)

3)

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 18 May 2021 00:51:24 +0100

David,

On 11/05/2021 15:56, Black, David wrote:
>
> This message is the explanation of the interaction of L4S and VPN 
> anti-replay that I promised to send during the interim meeting 
> yesterday.  I'm writing (explaining) as an individual, not as a WG chair.
>
> This explanation is for IPsec VPNs, although it applies in principle 
> to any VPN that propagates the ECN field to outer IP headers.
>
> For IPsec, the primary RFCs involved are RFC 4301 (IPsec architecture, 
> https://datatracker.ietf.org/doc/rfc4301/ 
> <https://datatracker.ietf.org/doc/rfc4301/>) and RFC 4303 (ESP – 
> Encapsulating Security Payload, 
> https://datatracker.ietf.org/doc/rfc4303/ 
> <https://datatracker.ietf.org/doc/rfc4303/>).  RFC 4302 (AH – 
> Authentication Header, https://datatracker.ietf.org/doc/rfc4302/ 
> <https://datatracker.ietf.org/doc/rfc4302/>) was mentioned in 
> discussion, but that RFC is not generally applicable to VPNs because 
> AH does not encrypt its payload. As a result, this explanation focuses 
> on ESP anti-replay functionality, as specified in Section 3.4.3 of RFC 
> 4303 (https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3 
> <https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3>). This 
> explanation also assumes a tunnel-mode IPsec VPN, where the outer IP 
> header is applied and removed by the VPN sender and receiver.  
> Tunnel-mode is a common configuration, as it separates VPN IP 
> addressing (in inner IP headers) from Internet IP addressing (in outer 
> IP headers).
>

[BB] The reason I focused on AH is that replay attacks are a lot harder 
to mount when blind (i.e. when traffic is encrypted). But I accept that 
replay protection is normally used on encrypted VPNs, whether or not the 
replay protection is actually necessary. Then of course the ESP VPN is 
prone to this interaction. But AH exhibits the more difficult tradeoff, 
because it's more critical to keep replay protection enabled.

>   * The VPN receiver uses a sliding sequence number window to
>     scoreboard (track) which packets have been received and reject
>     duplicates.
>       o There's a bit per packet within the window that indicates
>         whether that packet has been received.
>       o The scoreboard is initialized with all bits cleared to '0'.
>         Setting a bit to '1' indicates that the corresponding packet
>         has been received.
>   * Receipt of a packet whose sequence number is within the sliding
>     window causes the VPN receiver to check the bit for that sequence
>     number:
>       o If that bit is already set to '1', the packet is a duplicate
>         and is discarded.
>       o If that bit is cleared to '0', that bit is set to '1' and the
>         packet is processed and forwarded.
>   * Receipt of a packet whose sequence number is larger than the range
>     covered by the sliding window causes the window to advance to
>     include that sequence number (clearing new scoreboard bits to
>     '0'), and then proceeds as above because the packet's sequence
>     number is within the window after the advance.
>       o The packet sequence number is included in ESP crypto
>         computations, so rogue packets injected by an attacker who
>         does not know the relevant VPN key(s) are discarded without
>         advancing the window.
>   * Receipt of a packet with a sequence number less than the range
>     covered by the window always causes the packet to be discarded.
>   * Prior discussion on this list indicates that the 32 and 64 window
>     size values are present in some running code, at least as defaults.
>
> L4S can exhibit undesirable interactions with ESP anti-replay based 
> only on ECT(0) and ECT(1) in the outer headers as follows:
>
>  1. Simultaneously send L4S flows (ECT(1) in ECN field) and Classic
>     flows (ECT(0) in ECN field) through the same VPN.
>  2. The VPN sender assigns in-order sequence numbers to the packets
>     independent of which type of flow they belong to.
>  3. The VPN sender propagates the ECN field in each packet to the
>     outer IP header, as required by the tables in Sections 5.1.2.1 and
>     5.1.2.2 of RFC 4301.
>  4. One or more network node bottlenecks between the VPN sender and
>     receiver apply L4S treatment to the L4S ECT(1) packets, causing
>     them to advance past some Classic ECT(0) packets.
>
> At the VPN receiver, the received L4S packets drive advancing the 
> sliding window to higher sequence numbers.  If the sequence numbers in 
> the arriving L4S packets get ahead of the sequence numbers in the 
> arriving Classic packets by the size of the sliding window or more, 
> the arriving Classic packets are dropped *even though they are not 
> duplicates* (!).
>
> The problem arises at step 1 – mixing L4S and non-L4S traffic in the 
> same VPN.  IPsec has addressed the analogous problem for Diffserv by 
> requiring (MUST) support of multiple VPN sessions (which IPsec calls 
> Security Associations, SAs) between the same endpoints, and strongly 
> recommending (SHOULD) that traffic in different Diffserv classes use 
> different SAs (RFC 4301, Section 4.1, paragraph spanning pp.13-14).
>

[BB] In the thread where Sebastian first pointed out this interaction, I 
asked if you knew how prevalent it is for IPsec implementations to 
follow that SHOULD. Indeed, are you aware of any implementations at all 
that follow it?

You might have noticed the Cisco troubleshooting guide I found about how 
to remedy Diffserv interacting with replay-protection, which implies 
that the different DSCPs do not always have separate SAs:
https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html

It occurs to me that multiple application sessions can come and go 
within a VPN. So it might be some time after the set up of a VPN when an 
application is launched that starts to use different DSCPs within the 
VPN. The VPN would not have known this in advance, so then it would have 
to set up a new SA on the fly. Then it would have to either buffer or 
discard the new Diffserv packets while it did the new key exchange.

Seems unlikely that an implementer would have gone to all this trouble, 
rather than just using one SA, which is allowed by the SHOULD in the spec.
It strikes me that this SHOULD in RFC4301 might be easy to say but 
infeasible to implement.

On a host with an L4S CC enabled, a VPN would never need more than two 
SAs for the two types of ECN. So it would be more feasible to ensure 
that a VPN sets up two SAs in advance (for ECT1 or NotECT&ECT0). But 
that would require an L4S-specific patch to the VPN code. That would be 
do-able but probably only reasonable to expect it to be officially 
maintained in a VPN if L4S one day became standards track. And by then 
ECT0 might be on the decrease anyway.

> At least two incorrect assertions have been made about IPsec 
> anti-replay in list discussion and/or in the interim meeting:
>
>       o That is incorrect because RFC 4301 has addressed the
>         anti-replay interaction for Diffserv.  In contrast, L4S is new
>         wrt RFC 4301.
>

[BB] Correct, RFC4301 doesn't ask for ECT1 to be in a separate SA 'cos 
(obviously) L4S didn't exist when 4301 was written.

But that first sentence ("RFC 4301 has addressed the anti-replay 
interaction for Diffserv") might not be the whole story, if the SHOULD 
is infeasible to implement (see above).

>      o
>
>
>   * The undesirable anti-replay interaction only occurs if CE marks
>     are present in some of the packets involved.
>       o This is incorrect because ECT(0) and ECT(1) values in the ECN
>         field suffice to create the problem, see above.
>

[BB] Correct (if ECT0 and ECT1 are in the same SA).

I was thinking of the corner-case where the VPN ingress places ECT1 into 
a separate SA, but there are multiple bottlenecks between the two ends 
of the VPN as in the unlikely CE reordering problem with L4S discussed 
before:
1. VPN ingress
2. Classic ECN AQM
3. L4S DualQ AQM
4. VPN egress

If both AQMs are simultaneously bottlenecked (unusual) then, any ECT0 
packets that the Classic AQM marks as CE will be classified into the L4S 
queue of the DualQ, along with the packets in the ECT1 SA. Then replay 
protection in the VPN egress could drop some Classic traffic.

> I'll close by noting that if L4S flows carried a DSCP that differed 
> from Classic flows, that would enable a "reduce to previous case" line 
> of reasoning (yes, I used to be a mathematician) that reuses the IPsec 
> provisions for Diffserv to address this anti-replay interaction with 
> L4S.  That reuse won't prevent all reordering-caused anti-replay 
> problems that could occur, but it would take advantage of 
> existing/known techniques for dealing with them.
>

[BB] That would only make a difference if the VPN actually implemented 
separating DSCPs into separate SAs (which I questioned earlier).
It might be as feasible for the VPN to separate ECT1 into a separate SA, 
as discussed earlier.

Bob

> *David L. Black, *Sr. Distinguished Engineer, Technology & Standards
>
> Infrastructure Solutions Group,*Dell Technologies*
>
> mobile +1 978-394-7754 [email-masked] <mailto:[email-masked]>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 18 May 2021 05:26:16 +0300

> On 18 May, 2021, at 2:51 am, Bob Briscoe <[email-masked]> wrote:
> 
>> The problem arises at step 1 – mixing L4S and non-L4S traffic in the same VPN.  IPsec has addressed the analogous problem for Diffserv by requiring (MUST) support of multiple VPN sessions (which IPsec calls Security Associations, SAs) between the same endpoints, and strongly recommending (SHOULD) that traffic in different Diffserv classes use different SAs (RFC 4301, Section 4.1, paragraph spanning pp.13-14).
> 
> [BB] In the thread where Sebastian first pointed out this interaction, I asked if you knew how prevalent it is for IPsec implementations to follow that SHOULD. Indeed, are you aware of any implementations at all that follow it?
> 
> You might have noticed the Cisco troubleshooting guide I found about how to remedy Diffserv interacting with replay-protection, which implies that the different DSCPs do not always have separate SAs:
> It occurs to me that multiple application sessions can come and go within a VPN. So it might be some time after the set up of a VPN when an application is launched that starts to use different DSCPs within the VPN. The VPN would not have known this in advance, so then it would have to set up a new SA on the fly. Then it would have to either buffer or discard the new Diffserv packets while it did the new key exchange. 

The easy way out of this, for tunnel-mode SAs, is to maintain the same DSCP on all outer headers belonging to the same SA.  This permits it to establish fewer SAs between a host pair (typically 1) than the number of DSCPs that might theoretically be encountered (always 64).  Since middleboxes on the tunnel path will only see the outer header, they won't then reorder packets based purely on the DSCP, and no systematic DoS attack against the replay window can be mounted via the DSCP field.

I'd appreciate data on whether IPsec implementations take this approach in practice.

L4S does not have recourse to this approach, because the ECT(1) codepoint is the only identifier informing middleboxes that high-fidelty CE marking is expected rather than RFC-3168 marking.  The only thing that *could* safely be done here is to revert the ECN field to Not-ECT on tunnelled L4S packets.  That seems like rather a retrograde step, and the vulnerability would still exist with respect to non-participating users who are still using an existing RFC-3168 compliant VPN, that preserves the ECN field of the inner packet on the outer header.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 18 May 2021 09:58:45 +0200

Hi Bob,

> On May 18, 2021, at 01:51, Bob Briscoe <[email-masked]> wrote:
> 
> David,
> 
>> For IPsec, the primary RFCs involved are RFC 4301 (IPsec architecture, https://datatracker.ietf.org/doc/rfc4301/) and RFC 4303 (ESP – Encapsulating Security Payload, https://datatracker.ietf.org/doc/rfc4303/).  RFC 4302 (AH – Authentication Header, https://datatracker.ietf.org/doc/rfc4302/) was mentioned in discussion, but that RFC is not generally applicable to VPNs because AH does not encrypt its payload.  As a result, this explanation focuses on ESP anti-replay functionality, as specified in Section 3.4.3 of RFC 4303 (https://datatracker.ietf.org/doc/html/rfc4303#section-3.4.3).  This explanation also assumes a tunnel-mode IPsec VPN, where the outer IP header is applied and removed by the VPN sender and receiver.  Tunnel-mode is a common configuration, as it separates VPN IP addressing (in inner IP headers) from Internet IP addressing (in outer IP headers).
> 
> [BB] The reason I focused on AH is that replay attacks are a lot harder to mount when blind (i.e. when traffic is encrypted). But I accept that replay protection is normally used on encrypted VPNs, whether or not the replay protection is actually necessary. Then of course the ESP VPN is prone to this interaction. But AH exhibits the more difficult tradeoff, because it's more critical to keep replay protection enabled.

	[SM] Do you have references for cryptography papers supporting that view that replay protection is more important for AH than ESP? 

>> 	• The VPN sender numbers each packet with a sequence number that is incremented for each packet.
>> 	• The VPN receiver uses a sliding sequence number window to scoreboard (track) which packets have been received and reject duplicates.
>> 		• There's a bit per packet within the window that indicates whether that packet has been received.
>> 		• The scoreboard is initialized with all bits cleared to '0'.  Setting a bit to '1' indicates that the corresponding packet has been received.
>> 	• Receipt of a packet whose sequence number is within the sliding window causes the VPN receiver to check the bit for that sequence number:
>> 		• If that bit is already set to '1', the packet is a duplicate and is discarded.
>> 		• If that bit is cleared to '0', that bit is set to '1' and the packet is processed and forwarded.
>> 	• Receipt of a packet whose sequence number is larger than the range covered by the sliding window causes the window to advance to include that sequence number (clearing new scoreboard bits to '0'), and then proceeds as above because the packet's sequence number is within the window after the advance.
>> 		• The packet sequence number is included in ESP crypto computations, so rogue packets injected by an attacker who does not know the relevant VPN key(s) are discarded without advancing the window.
>> 	• Receipt of a packet with a sequence number less than the range covered by the window always causes the packet to be discarded.
>> 	• Here's the entire paragraph on scoreboard window size (last paragraph in Section 3.4.3 of RFC 4303):
>> 	• Prior discussion on this list indicates that the 32 and 64 window size values are present in some running code, at least as defaults.
>>  
>> 	• Simultaneously send L4S flows (ECT(1) in ECN field) and Classic flows (ECT(0) in ECN field) through the same VPN.
>> 	• The VPN sender assigns in-order sequence numbers to the packets independent of which type of flow they belong to.
>> 	• The VPN sender propagates the ECN field in each packet to the outer IP header, as required by the tables in Sections 5.1.2.1 and 5.1.2.2 of RFC 4301.
>> 	• One or more network node bottlenecks between the VPN sender and receiver apply L4S treatment to the L4S ECT(1) packets, causing them to advance past some Classic ECT(0) packets.
>>  

	[SM] Have a look at https://www.strongswan.org/testing/testresults/ikev2/net2net-psk-dscp/ which gives an example setup for Linux/strongswan exactly fr this use-case. So I woud say the implementation supports and documents that. Now, how many deployed instances actually use that (say compared to the number of instances that opted not to propagate DSCPs at encapsulation) I do not know.

> https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html

	[SM] Well sure, because that is a policy decision each network deploying IPSEC has to make, Cisco has not way of knowing what people actually configured...

> It occurs to me that multiple application sessions can come and go within a VPN. So it might be some time after the set up of a VPN when an application is launched that starts to use different DSCP…

	[SM] The ipsec RFC talks about DSCP bypass and an explicit mapping table of inner DSCPs to outer DSCPs,... anybody taking DSCP propagation seriously will certainly figure out not to leak arbitrary DSCPs, if she/he already went to the effort to configure different SAs for some dscps, no?

> Seems unlikely that an implementer would have gone to all this trouble, rather than just using one SA, which is allowed by the SHOULD in the spec.
> It strikes me that this SHOULD in RFC4301 might be easy to say but infeasible to implement.

	[SM] Look at the strongswan example above, that does not seem to be witchcraft, but rather mundane iptables/nftables marking/filtering. I see no reason why this could not be extended for all 64 possible DSCPs if one really would like to prepare for all eventualities... but realsitically it shoukd suffice to cover those DSCPs the end network actually intends to use and simply re-mark the other ones sanely...

> On a host with an L4S CC enabled, a VPN would never need more than two SAs for the two types of ECN.

	[SM] Unless there is also DSCP based priority forwarding in polace along the path, like WiFi WMM, because dscp and L4S re-ordering are not mutually excusive along an arbitrary network path, no?

> So it would be more feasible to ensure that a VPN sets up two SAs in advance (for ECT1 or NotECT&ECT0). But that would require an L4S-specific patch to the VPN code. That would be do-able but probably only reasonable to expect it to be officially maintained in a VPN if L4S one day became standards track. And by then ECT0 might be on the decrease anyway.

	[SM] Yes that is one way to solve that, unfortunately this again requires bystanders to fix up L4S side-effects...

>> 	• The undesirable anti-replay interaction is neither specific to nor caused by L4S.
>> 		• That is incorrect because RFC 4301 has addressed the anti-replay interaction for Diffserv.  In contrast, L4S is new wrt RFC 4301.
> 
> [BB] Correct, RFC4301 doesn't ask for ECT1 to be in a separate SA 'cos (obviously) L4S didn't exist when 4301 was written.
> 
> But that first sentence ("RFC 4301 has addressed the anti-replay interaction for Diffserv") might not be the whole story, if the SHOULD is infeasible to implement (see above).

	[SM] While it would seem convenient for your argument, it seems that at least for strongswan there is evidence that the dscp replay issue got sufficient attention, that a reference configuration was created and added to the test suite. 

>> 
>> 	• The undesirable anti-replay interaction only occurs if CE marks are present in some of the packets involved.
>> 		• This is incorrect because ECT(0) and ECT(1) values in the ECN field suffice to create the problem, see above.
> 

	[SM] Which they will be by default.

> I was thinking of the corner-case where the VPN ingress places ECT1 into a separate SA, but there are multiple bottlenecks between the two ends of the VPN as in the unlikely CE reordering problem with L4S discussed before:
> 1. VPN ingress
> 2. Classic ECN AQM
> 3. L4S DualQ AQM
> 4. VPN egress
> 
> If both AQMs are simultaneously bottlenecked (unusual) then, any ECT0 packets that the Classic AQM marks as CE will be classified into the L4S queue of the DualQ, along with the packets in the ECT1 SA. Then replay protection in the VPN egress could drop some Classic traffic.

	[SM] Yes, that seems to be correct, IMHO this is also quite problematic, but this will only trigger if (one) AQM (along the path) wants the classic traffic to slow down anyway, so is not as bad as the case with competing ECT(1) and ECT(0) traffic (as that will be quite normal for a long time, assuming L4S actually gains some traction).

> [BB] That would only make a difference if the VPN actually implemented separating DSCPs into separate SAs (which I questioned earlier).

	[SM] As above, for Linux xfrm/ipsec it is incumbent upon the party setting up an ipsec tunnel to implement the desired policies, but the cited example indicates IMHO that the separate SA modus can be expected to not exist in the real word, as convenient as that might be.

> It might be as feasible for the VPN to separate ECT1 into a separate SA, as discussed earlier.

	[SM] I am sure all operators of ipsec tunnels are going to absolutely adore L4S once they figure out that L4S is the root cause that they need to change all of their configurations...

Best Regards
	Sebastian

>> mobile +1 978-394-7754 [email-masked]
>>  
> 
> --

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 18 May 2021 12:33:28 +0000

Hi,

I think I have read through all the relevant discussion of this issue. I have to
agree with Bob Briscoe that this is a general issue for tunneling protocols that
have several properties:

 - Any form of replay protection with a window shorter than produced reordering 
   (e.g. IPsec) or have a reordering restoring functionality (e.g. L2TP).
 - Marks through ECN and/or DSCP.
 - Aggregate multiple sub-flows

These tunnels will exhibit issues either resulting in packet loss (replay
protection) or additional delay (to cancel out the reordering) when the tunnel
flow are going through some type of queue that will cause reordering, i.e. when
sufficiently loaded to have any queue buildup to reorder around. 

Any forwarding impacting technologhy that would cause subflows to be subject to
improved performance compared to other flows will trip over this issue. That is
clear based on the significant discussion of this related to diffserv in IPsec
RFC 4301, and Section 4.1 in RFC 2983 (
https://datatracker.ietf.org/doc/html/rfc2983#section-4.1).

From my perspective we can't halt progress towards improved performance based on
this general problem. We should mitigate and inform about the issue. However, I
think part of the burden here long term will need to be put on the tunnels that
exhibit the above properties. They need to track development in network
technologies to stay current. It is clear that tunnels that handles this by
correctly classifying the subflows before aggregation and put them in tunnels
with same type of forwarding performance will not suffer. The alternative
solution is to avoid the mark through seperation and loose the potential benefit
if encountering a queue where differentation could have been done. But that
requires the pipe style handling on egress to correctly preserve the ECN
information for L4S, just as it does for DSCP field. And thirdly, if ones replay
protection is reasonably scaled with experienced jitter and tunnel throughput
rates this would also not be a significant issue. 

Thus my opinion is that in the L4S context we need to document this impact. Link
to solutions or mitigations that can be applied and potential push for updates
on important affected specifications, like IPsec. 

Cheers

Magnus Westerlund

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 May 2021 14:01:26 +0000

I'm sorry Magnus, but I'm going to be blunt.

For IPsec tunnels, there appears to be both "rough consensus" (RFC 4301) and "running code" (e.g., strongSwan VPN) to address this problem for DSCP-caused reordering.

The message below appears to regard both as irrelevant.  I don't understand why - would you please explain?

Thanks, --David

-----Original Message-----
From: Magnus Westerlund <[email-masked]> 
Sent: Tuesday, May 18, 2021 8:33 AM
To: [email-masked]; Black, David
Subject: Re: [tsvwg] L4S & VPN anti-replay interaction: Explanation

Hi,

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 18 May 2021 15:16:24 +0000

On Tue, 2021-05-18 at 14:01 +0000, Black, David wrote:
> For IPsec tunnels, there appears to be both "rough consensus" (RFC 4301) and
> "running code" (e.g., strongSwan VPN) to address this problem for DSCP-caused
> reordering.
> 

Okay, I must have expressed myself poorly. That you should arrive at the above
interpretation was definitely not my intention. 

My intention was to state that the same approach of addressing the issue for L4S
ECN as for DSCP appear to be appropriate. However, to me that appear to mean
that L4S would document the issue and point to similar approaches of handling or
mitiagation like for DSCP. To my undestanding, the better approach of using two
SAs after L4S aware subflow classifyier in tunnel ingress, alternatively to
downgrade the pipe segment the tunnel represents to not enable L4S treatment and
recover the L4S on egress from the inner ECN markings. But, it does imply that
tunnel implementations have some work to do. I recognize that guard DSCP would
simplify that process to likely just a configuration change for implementation
already capable of using sets of DSCP to SA mappings. 

I think a third way of dealing with this issue is adaptive scaling of the replay
protection window in the egress side of the tunnels. This maybe anyway is needed
based on the increase bandwidth, the changes in what lower layers are and how
they behave and on what level they preserve order. 

I think the core of my thinking is that this type of tunnels that has the
properties that I expressed are going to be impacted by any change in forwarding
behaviors that impacts packet priority or path when being forwarded through the
network. L4S do redefine in some sense what the flow selectors are when
forwarding packets. This, has to no surprise some cross area impact. The WG have
been discussing these impacts and there cost and severity for a while now. I
think describing this one and its handling appear fairly straightforward. So I
suggest that we continue on writing up the issue and its handling. The severity
and impact discussion can go on in parallel. I do want to arrive at the point
where the L4S related specifications are in good enough state that we can
actually see where the IETF wide consensus on this really stands. 

Cheers

Magnus

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 May 2021 21:54:55 +0000

Magnus,

Thanks for the prompt explanation.

> ECN as for DSCP appear to be appropriate. However, to me that appears to mean
> mitigation like for DSCP. To my understanding, the better approach of using two
> SAs after L4S aware subflow classifier in tunnel ingress, alternatively to

I generally agree with the above paragraph.  The crucial takeaway from my vantage
point is that IPsec has addressed this problem for Diffserv DSCPs.  FWIW,  the RFC 2983
text that I wrote likely contributed to the RFC 4301 provisions that address this problem,
so this is a familiar area.

I do hope that there will be no further attempts to argue that Diffserv interaction with
IPsec anti-replay is an unaddressed problem.

Thanks, --David

-----Original Message-----
From: Magnus Westerlund <[email-masked]> 
Sent: Tuesday, May 18, 2021 11:16 AM
To: [email-masked]; Black, David
Subject: Re: [tsvwg] L4S & VPN anti-replay interaction: Explanation

Cheers

Magnus

> RFC 4301, and Section 4.1 in RFC 2983 (https://datatracker.ietf.org/doc/html/rfc2983#section-4.1).
>