Subject: [tsvwg] Case for ECN-capable TCP control packets: draft-bagnulo-tsvwg-generalized-ecn

From: Bob Briscoe <[email-masked]>
Date: Fri, 15 Jul 2016 19:31:05 +0100

David, [re-sending to include tsvwg & tcpm, which is what I had intended 
to do first time.]

Here's a heads-up for tsvwg, plus cross-posting to tcpm as suggested,
And for a fast read, there's a summary of every argument below.
Pls bash the arguments.

Summary:
The ECN spec [RFC 3168] says various TCP control packets must not be ECN 
capable.
This draft lays out each argument given in the RFCs, and articulates a 
rebuttal against each one:
draft-bagnulo-tsvwg-generalized-ecn 
<https://tools.ietf.org/html/draft-bagnulo-tsvwg-generalized-ecn>
Subject to some more experiments, we think this knocks down every reason 
given, so that all these packets could safely be ECN-capable.

Pls read the draft for more details (preferably before responding).

Enumeration of arguments about each type of control packet follows, 
starting with some arguments common to many:
___________________________________________________________________________________________

*Common arguments against ECT*

 1. Unreliable congestion notification delivery
 2. DoS Attacks

Common rebuttals (respectively):

 1. No worse than undetectable loss of Not-ECT control packet, and
    better performance
 2. Mandating Not-ECT does not stop attackers using ECT for flooding.
    Nonetheless, it would allow firewalls to discard control packets not
    meant to be ECT., however this would negate the benefit of ECT for
    compliant transports and seems unnecessary for the following reason.
    RFC3168 (Sec.7) and RFC7567 (Sec.4.2.1) say an AQM MUST turn off ECN
    support if under persistent overload. This makes it hard for
    flooding packets to gain from ECT, but more experiments needed to
    see how much might be gained by an attacker flying "just under the
    radar".

*SYN/ACK**
*ECT on SYN/ACK already justified in RFC5562

*SYN**
*Arguments against ECT:

 1. Unknown ECN capability at the responder (may discard ECT SYN or RST
    connection)
 2. Loss of congestion notification due to lack of support for feeding
    back CE on SYN at the responder.
 3. DoS attacks.

Rebuttals (respectively):

 1. No RFC mandates these responder behaviours, but 0.6% - 0.8% of Alexa
    top 1M Web sites (or their network paths) exhibit this. Could
    retransmit the SYN without ECT, and cache this knowledge to avoid 1
    RTT delay in future
 2. Support for CE feedback in SYN needed in the TCP wire protocol.
    Solution proposed in AccECN
    <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> draft: if
    SYN/ACK shows lack of support for feeding back CE on SYN (i.e. no
    support for AccECN) IW MUST be reduced conservatively as if CE on
    SYN (no need to be as conservative as drop of SYN - cannot be severe
    congestion as packet got through). Could cache this knowledge.
 3. See common DoS Attack rebuttal

*Pure ACK**
*Arguments against ECT:

 1. Unreliable congestion notification delivery
 2. No means to feed back congestion notifications (no ACKs of ACKs)

Rebuttals (respectively):

 1. See common unreliability rebuttal
 2. No worse than drop of Pure ACK, and better performance

*Retransmission**
*Arguments against ECT:

 1. Unreliable congestion notification delivery
 2. DoS attacks
 3. Over-reacting to congestion.

Rebuttals (respectively):

 1. See common unreliability rebuttal
 2. See common DoS Attack rebuttal, complemented by recommendation to
    ignore CE on out of window packets
 3. Correct to react twice to congestion in different RTTs

*Window Probe**
*
Arguments against ECT:

 1. Unreliable congestion notification delivery

Rebuttal:

 1. See common unreliability rebuttal

*FIN**
*
No arguments against using ECT in RFCs.
To be discussed in next rev of draft.

*RST**
*
No arguments against using ECT in RFCs
To be discussed in next rev of draft.

Cheers

Bob & Marcelo

On 11/07/16 21:28, Black, David wrote:
> Marcelo and Bob,
>
> Interesting draft - I need to read it in more detail ... in my "copious spare time" this week ;-).
>
> While it is of interest to TSVWG, as general ECN work happens here, I suspect
> that as a proposed modification to TCP it would be in the domain of the TCPM WG,
> as was the case for RFC 5562.
>
> Thanks, --David
> ----------------------------------------------------
> David L. Black, Distinguished Engineer
> EMC, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953     Cell: +1 (978) 394-7754
> [email-masked]         
> ---------------------------------------------------
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

---

From: Michael Welzl <[email-masked]>
Date: Fri, 15 Jul 2016 21:25:01 +0200

Hi,

I think this is interesting stuff!  Definitely good to work on, I’m interested to help out if needed.

Cheers,
Michael

> On 15. jul. 2016, at 20.31, Bob Briscoe <[email-masked]> wrote:
> 
> David, [re-sending to include tsvwg & tcpm, which is what I had intended to do first time.]
> 
> The ECN spec [RFC 3168] says various TCP control packets must not be ECN capable.
> This draft lays out each argument given in the RFCs, and articulates a rebuttal against each one:
> draft-bagnulo-tsvwg-generalized-ecn <https://tools.ietf.org/html/draft-bagnulo-tsvwg-generalized-ecn>
> Subject to some more experiments, we think this knocks down every reason given, so that all these packets could safely be ECN-capable.
> 
> Enumeration of arguments about each type of control packet follows, starting with some arguments common to many:
> Common arguments against ECT
> Unreliable congestion notification delivery
> DoS Attacks
> No worse than undetectable loss of Not-ECT control packet, and better performance
> Mandating Not-ECT does not stop attackers using ECT for flooding. Nonetheless, it would allow firewalls to discard control packets not meant to be ECT., however this would negate the benefit of ECT for compliant transports and seems unnecessary for the following reason. 
> RFC3168 (Sec.7) and RFC7567 (Sec.4.2.1) say an AQM MUST turn off ECN support if under persistent overload. This makes it hard for flooding packets to gain from ECT, but more experiments needed to see how much might be gained by an attacker flying "just under the radar".
> 
> SYN/ACK
> ECT on SYN/ACK already justified in RFC5562
> 
> SYN
> Arguments against ECT:
> Unknown ECN capability at the responder (may discard ECT SYN or RST connection)
> Loss of congestion notification due to lack of support for feeding back CE on SYN at the responder.
> DoS attacks.
> No RFC mandates these responder behaviours, but 0.6% - 0.8% of Alexa top 1M Web sites (or their network paths) exhibit this. Could retransmit the SYN without ECT, and cache this knowledge to avoid 1 RTT delay in future
> Support for CE feedback in SYN needed in the TCP wire protocol. Solution proposed in AccECN <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> draft: if SYN/ACK shows lack of support for feeding back CE on SYN (i.e. no support for AccECN) IW MUST be reduced conservatively as if CE on SYN (no need to be as conservative as drop of SYN - cannot be severe congestion as packet got through). Could cache this knowledge.
> See common DoS Attack rebuttal
> 
> Pure ACK
> Arguments against ECT:
> Unreliable congestion notification delivery
> No means to feed back congestion notifications (no ACKs of ACKs)
> See common unreliability rebuttal
> No worse than drop of Pure ACK, and better performance
> 
> Retransmission
> Arguments against ECT:
> Unreliable congestion notification delivery
> DoS attacks
> Over-reacting to congestion.
> See common unreliability rebuttal
> See common DoS Attack rebuttal, complemented by recommendation to ignore CE on out of window packets
> Correct to react twice to congestion in different RTTs
> 
> Window Probe
> 
> Arguments against ECT:
> Unreliable congestion notification delivery
> Rebuttal:
> See common unreliability rebuttal
> FIN
> 
> RST
>> [email-masked] <mailto:[email-masked]>        
>> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>_______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Brian Trammell <[email-masked]>
Date: Sat, 16 Jul 2016 11:14:21 +0200

hi Bob, all,

Interesting stuff.

There is one argument against ECN-marked control packets not appearing in 3168: the presence of ECT or CE marked TCP control packets is currently an indication of ECN signaling malfunction in the Internet, either due to buggy implementations or deployed boxes who still believe the ToS byte is the ToS byte.

In our measurement we haven't seen a lot of this (10^-5 - 10^-6 or so), but it seems relatively stable as long as we've been looking at it.

Allowing TCP control packets to use ECN marking will cause confusion between correct signaling and incorrect signaling, which makes this measurement methodology no longer available.

It's a minor problem, admittedly, but it will blind us to the ability to continue monitoring the de-deployemnt of these broken middleboxes and endpoints.

Cheers,

Brian

> On 15 Jul 2016, at 20:31, Bob Briscoe <[email-masked]> wrote:
> 
> 	• Unreliable congestion notification delivery
> 	• DoS Attacks
> 	• No worse than undetectable loss of Not-ECT control packet, and better performance
> 	• Mandating Not-ECT does not stop attackers using ECT for flooding. Nonetheless, it would allow firewalls to discard control packets not meant to be ECT., however this would negate the benefit of ECT for compliant transports and seems unnecessary for the following reason.
> 	• Unknown ECN capability at the responder (may discard ECT SYN or RST connection)
> 	• Loss of congestion notification due to lack of support for feeding back CE on SYN at the responder.
> 	• DoS attacks.
> 	• No RFC mandates these responder behaviours, but 0.6% - 0.8% of Alexa top 1M Web sites (or their network paths) exhibit this. Could retransmit the SYN without ECT, and cache this knowledge to avoid 1 RTT delay in future
> 	• Support for CE feedback in SYN needed in the TCP wire protocol. Solution proposed in AccECN draft: if SYN/ACK shows lack of support for feeding back CE on SYN (i.e. no support for AccECN) IW MUST be reduced conservatively as if CE on SYN (no need to be as conservative as drop of SYN - cannot be severe congestion as packet got through). Could cache this knowledge.
> 	• See common DoS Attack rebuttal
> 
> Pure ACK
> Arguments against ECT:
> 	• Unreliable congestion notification delivery
> 	• No means to feed back congestion notifications (no ACKs of ACKs)
> 	• See common unreliability rebuttal
> 	• No worse than drop of Pure ACK, and better performance
> 
> Retransmission
> Arguments against ECT:
> 	• Unreliable congestion notification delivery
> 	• DoS attacks
> 	• Over-reacting to congestion.
> 	• See common unreliability rebuttal
> 	• See common DoS Attack rebuttal, complemented by recommendation to ignore CE on out of window packets
> 	• Correct to react twice to congestion in different RTTs
> 
> Window Probe
> 
> Arguments against ECT:
> 	• Unreliable congestion notification delivery
> Rebuttal:
> 	• See common unreliability rebuttal
> FIN
>

---

From: Aaron Falk <[email-masked]>
Date: Sat, 16 Jul 2016 12:04:20 +0200

> It's a minor problem, admittedly, but it will blind us to the ability to continue monitoring the de-deployemnt of these broken middleboxes and endpoints.

So, it isn’t a feature, it’s a bug?

---

From: David Schinazi <[email-masked]>
Date: Sat, 16 Jul 2016 12:08:13 +0200

Bob,

This is interesting and promising work.
It looks like it's scheduled in tcpm if time permits; let's hope time does permit!

David

> On Jul 15, 2016, at 20:31, Bob Briscoe <[email-masked]> wrote:
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 16 Jul 2016 16:13:49 +0000

This is certainly interesting work and very relevant.

Marking ECT on SYN makes sense and data shows that for schemes like DCTCP, not marking SYN puts new flows establishment at a disadvantage especially when schemes like Dual Q AQM are in play. FIN occupies a byte in sequence number space so marking ECT on packet with FIN (with or without data) seems reasonable. Marking RST is probably harmless. I don’t have a strong opinion about window probes and retransmissions but the consistency argument should hold unless there is a demonstrable DoS attack.

My main concern is with pure ACKs. If the receiver is a dumb reflector by sending ECE upon receiving any control packet marked with CE then it could interfere with “accurate ECN” like schemes. For example if the application has bi-directional traffic then how would the sender distinguish a ECE was in response to a mark on a data packet versus an ACK packet? I am of the opinion that the receiver should not echo ECN marks on pure ACK packets. Any other ideas?

I think bringing this up for discussion in working group would be very valuable.

Thanks

From: tcpm [mailto:[email-masked]] On Behalf Of Bob Briscoe
Sent: Friday, July 15, 2016 11:31 AM
To: Black, David <[email-masked]>; tsvwg IETF list <[email-masked]>; tcpm IETF list <[email-masked]>
Cc: [email-masked]
Subject: [tcpm] Case for ECN-capable TCP control packets: draft-bagnulo-tsvwg-generalized-ecn

Summary:
The ECN spec [RFC 3168] says various TCP control packets must not be ECN capable.
This draft lays out each argument given in the RFCs, and articulates a rebuttal against each one:
draft-bagnulo-tsvwg-generalized-ecn<https://tools.ietf.org/html/draft-bagnulo-tsvwg-generalized-ecn>
Subject to some more experiments, we think this knocks down every reason given, so that all these packets could safely be ECN-capable.

  1.  No worse than undetectable loss of Not-ECT control packet, and better performance
  2.  Mandating Not-ECT does not stop attackers using ECT for flooding. Nonetheless, it would allow firewalls to discard control packets not meant to be ECT., however this would negate the benefit of ECT for compliant transports and seems unnecessary for the following reason.
RFC3168 (Sec.7) and RFC7567 (Sec.4.2.1) say an AQM MUST turn off ECN support if under persistent overload. This makes it hard for flooding packets to gain from ECT, but more experiments needed to see how much might be gained by an attacker flying "just under the radar".

SYN/ACK

SYN
Arguments against ECT:

  1.  Unknown ECN capability at the responder (may discard ECT SYN or RST connection)
  2.  Loss of congestion notification due to lack of support for feeding back CE on SYN at the responder.
  3.  DoS attacks.
Rebuttals (respectively):

  1.  No RFC mandates these responder behaviours, but 0.6% - 0.8% of Alexa top 1M Web sites (or their network paths) exhibit this. Could retransmit the SYN without ECT, and cache this knowledge to avoid 1 RTT delay in future
  2.  Support for CE feedback in SYN needed in the TCP wire protocol. Solution proposed in AccECN<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> draft: if SYN/ACK shows lack of support for feeding back CE on SYN (i.e. no support for AccECN) IW MUST be reduced conservatively as if CE on SYN (no need to be as conservative as drop of SYN - cannot be severe congestion as packet got through). Could cache this knowledge.
  3.  See common DoS Attack rebuttal

Pure ACK
Arguments against ECT:

Retransmission
Arguments against ECT:

  1.  See common unreliability rebuttal
  2.  See common DoS Attack rebuttal, complemented by recommendation to ignore CE on out of window packets
  3.  Correct to react twice to congestion in different RTTs

Window Probe

Arguments against ECT:

RST

Cheers

Bob & Marcelo

Marcelo and Bob,

Thanks, --David

[email-masked]<mailto:[email-masked]>

--

Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 16 Jul 2016 23:40:55 +0100

Praveen,

On 16/07/16 17:13, Praveen Balasubramanian wrote:
> Marking ECT on SYN makes sense and data shows that for schemes like 
> DCTCP, not marking SYN puts new flows establishment at a disadvantage 
> especially when schemes like Dual Q AQM are in play.
>
Not just for DCTCP.
For just normal public Internet, if the loss probability is say 0.2%, 
then 1 in 500 flows will lose the SYN and stall for a timeout when it 
starts. Given there are so many small flows, that's surprisingly often.

Nothing to do with this draft, but I've written-up some tips for 
configuring the AQM in a DC to mitigate this problem if you are not 
using ECT SYNs. See "{Note 1}" at the end.

> FIN occupies a byte in sequence number space so marking ECT on packet 
> with FIN (with or without data) seems reasonable. Marking RST is 
> probably harmless. I don’t have a strong opinion about window probes 
> and retransmissions but the consistency argument should hold unless 
> there is a demonstrable DoS attack.
>
See draft for argument about DoS.
>
> My main concern is with pure ACKs. If the receiver is a dumb reflector 
> by sending ECE upon receiving any control packet marked with CE then 
> it could interfere with “accurate ECN” like schemes. For example if 
> the application has bi-directional traffic then how would the sender 
> distinguish a ECE was in response to a mark on a data packet versus an 
> ACK packet? I am of the opinion that the receiver should not echo ECN 
> marks on pure ACK packets. Any other ideas?
>
In AccECN at the moment, we have a dumb receiver with two feedback 
methods so the sender can choose not to react to CE on Pure ACKs (or it 
can react - it has the info to do either). If the sender were to set ECT 
on pure ACKs:
i) the count of CE marked packets it would get back from the receiver in 
the ACE field would include pure ACKS
ii) the count of CE-marked bytes it would get back from the receiver in 
the (optional) ECEB field would not count marked pure ACKS (because they 
contain no bytes).

You have told me already that you don't want to use the TCP option part 
of AccECN (part (ii) above). Well, then what can I say - you can't have 
the features of the optional part you choose not to use :|
If you can work out a way to do this that suits you better, without 
adding complexity, that would save me having to!

If you don't set ECT on pure ACKs you lose the benefit of near-zero loss 
of ACKs (quantified in the draft - marginal - you just get occasional 
pauses in the clocking out of packets, then a little catch-up burst when 
the next ACK arrives).

> I think bringing this up for discussion in working group would be very 

Thanks for sharing your thoughts.

Bob

{Note 1}:
For DCTCP in your own DC, surely you can use ECT SYNs (and ECT on DNS?).

If not (why?), you can improve the situation (depending on the 
capabilities of your switches) by configuring WRED on each switch with 
two AQMs using the ECN field as the classifier between them (one for 
Not-ECT, the other for any other ECN codepoint). You can still config 
both REDs as for DCTCP with a step at K packets. And config the EWMA 
parameter of the ECN RED to zero as normal for DCTCP, but config the 
other (drop) with a higher EWMA parameter so it uses a moving average of 
the queue to determine drop. Then
a) it won't be so jumpy for drop, avoiding losing so many SYNs and DNS 
datagrams; but
b) it will still have the same threshold for marking vs drop in the 
long-run, ensuring roughly equal throughput for any long-running non-ECN 
flows.

> Thanks
>
> *From:*tcpm [mailto:[email-masked]] *On Behalf Of *Bob Briscoe
> *Sent:* Friday, July 15, 2016 11:31 AM
> *To:* Black, David <[email-masked]>; tsvwg IETF list 
> <[email-masked]>; tcpm IETF list <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* [tcpm] Case for ECN-capable TCP control packets: 
> Subject to some more experiments, we think this knocks down every 
> reason given, so that all these packets could safely be ECN-capable.
>
>     Nonetheless, it would allow firewalls to discard control packets
>     not meant to be ECT., however this would negate the benefit of ECT
>     for compliant transports and seems unnecessary for the following
>     reason.
>     RFC3168 (Sec.7) and RFC7567 (Sec.4.2.1) say an AQM MUST turn off
>     ECN support if under persistent overload. This makes it hard for
> *SYN/ACK
> *SYN
>  1. No RFC mandates these responder behaviours, but 0.6% - 0.8% of
>     Alexa top 1M Web sites (or their network paths) exhibit this.
>     Could retransmit the SYN without ECT, and cache this knowledge to
>     avoid 1 RTT delay in future
>     <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> draft:
>     if SYN/ACK shows lack of support for feeding back CE on SYN (i.e.
>     no support for AccECN) IW MUST be reduced conservatively as if CE
>     on SYN (no need to be as conservative as drop of SYN - cannot be
>     severe congestion as packet got through). Could cache this knowledge.
> *Pure ACK
> *Retransmission
> *Window Probe
> *FIN
> *RST