Subject: Re: [tsvwg] Mirja's comments on the circuit breaker draft

From: "Black, David" <[email-masked]>
Date: Mon, 16 Nov 2015 21:31:26 +0000

Plus a few additional editorial comments from Mirja:

There are a few occasions where the draft uses "UDP traffic" as a synonym
for non-responsive/non-congestion controlled traffic which is not always
the case. That should also be clarified, e.g.:

- in 1.: "including many application of the User Datagram Protocol (UDP)"
[this text is correct, but the comment is on emphasis]

- in 5.3: "used to control uni-directional UDP traffic" (uni-directional
is probably intended to say that there is no feedback channel, however,
there could also be an out-of band feedback channel, in which case,
the UDP flow is still unidirectional.

Also in general the doc could maybe try make it even more clear when
it talks about non-responsive traffic, e.g.

- in 1.1.: "to protect from network traffic of a single flow or related
groups of flows" could be "to protect from network traffic of a single
non-responsive flow or related group of non-responsive flows"

Thanks,
--David

> -----Original Message-----
> From: Black, David
> Sent: Monday, November 16, 2015 4:17 PM
> To: [email-masked]
> Cc: [email-masked]; Black, David
> Subject: Mirja's comments on the circuit breaker draft
> 
> <WG chair hat OFF>
> 
> Mirja Kuehlewind reviewed the circuit breaker draft during the Yokohama
> meeting week - I have no idea how she found that time ;-).
> 
> As she's offline now, I'm passing along her comments - this is my summary
> of an earlier discussion w/her ('>'d text) followed by her elaboration,
> which I'm prepared to defend/explain in her absence.
> 
> There are a few points that could use some attention:
> 
> ----------
> 
> > The notion of "persistent congestion" is not well-defined - a single
> > TCP connection running over a link that constrains its throughput will
> > drop packets on an ongoing basis.  There seems to be an operational
> > characteristic to this - it's abnormal congestion caused by traffic
> > that is non-responsive to network conditions (e.g., drops, ECN
> > congestion indications) ... and in that description, both 'abnormal'
> > and 'non-responsive' have some dependency on the network and the
> > nature of the traffic that it normally carries.
> 
> Yes, persistent congestion is not only not well defined, it's really the
> wrong word, because congestion can be persistent under normal operation.
> The level of congestion (over time) is actually important.
> 
> Further the draft uses persistent congestion nearly as equivalent to
> congestion collapse. However, a congestion collapse happens if the demand
> is increasing/network is always fully loaded while the goodput decreases
> to zero (just because the demand is permanently increasing this situation
> will not resolve on its own -> collapse). So there is the notion that the
> congestion level over time is actually increasing (until no goodput is
> possible anymore). So the use of the term congestion collapse is also
> not fully correct I believe.
> 
> > We discussed a measurement concern - a key goal here is to avoid
> > tripping a circuit breaker in response to a brief traffic spike or
> > two.
> 
> Yes
> 
> >
> > The concern about shutting down traffic as a consequence of loss of
> > circuit breaker control path connectivity is already in the process
> > of being addressed.
> 
> Actually as a side note, the draft says a couple of time that traffic
> or flows should be "shut down" or the aggregate rate should be reduced.
> Maybe if some hint about how much it should be reduced at once might
> also be useful. Not thinking about a number, but say something like,
> if the time-scale when those actions are take is rather large you
> might reduce more, while at a lower time-scale you'd could star
> with a very small reduction?
> 
> ----------
> 
> </WG chair hat OFF>
> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Distinguished Engineer
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
>

---

From: [email-masked]
Date: Tue, 17 Nov 2015 16:46:01 -0000

I’ve tried to answer the comments below by preparing a new revision, but
here include responses in-line.

> [this text is correct, but the comment is on emphasis]
I added text that explained the benefits of performing this in a transport
or application to the fast circuit breaker section (addressing a previous
round of comments). I think now this is better placed at the very start of
the document - try to use "CC".

> the UDP flow is still unidirectional.
I removed the reference to UDP at this point, and rephrased this, I hope
the new text is clearer (and I don't think it can now be easily misread).

> non-responsive flow or related group of non-responsive flows"
I changed this.

>> which I'm prepared to defend/explain in her absence.
OK, that's good.

>> congestion collapse.
>> However, a congestion collapse happens if the
>> demand
Agree.  In the latest rev, I had already removed all but one mention of 
"congestion collapse", to only refer to a previous RFC in the intro, and
even there be super clear this is not a solution, but a mitigation.

We need a term to say what triggers the reaction. I chose "persistent
congestion" to be something I hadn't seen much used, but I am OK to
change, because I know that with ECN (and also other cases) you can expect
to see some incipient congestion under normal operation - so I can see
saying only "persistent" is not sufficient, it also needs to be above a
threshold.

We suggest using some sort of threshold to determine that we experience
the normal operational value for some longish time.

Therefore I suggest "persistent excessive congestion”?

I've looked at this change, and while it is longer, it's probably clearer
(which has to be good). I'd be happy to apply this throughout.

>> > tripping a circuit breaker in response to a brief traffic spike or
I think this issue  - not to react quickly -  is now clear. The rewritten
requirements seek to avoid any potential confusion.

>> with a very small reduction?

I'm not sure I agree with an argument that seems to imply that one could
use a circuit breaker to gradually reduce traffic  for a short control
loop - To me, this is suggesting some sort of coarse-grain congestion
control, but I'd rather not go there. I suggest any from of congestion
control is likely to be better than a CB (I made this clearer) - A CB
*necessarily* must operate over a longer time interval. I'd prefer to
avoid providing suggestions to tune for a small RTT and a more gradual
response. In that case, I'd prefer the method used a design that resembles
CC.

However, I think it is also quite possible that greater clarity in the new
requirements section (based on Boib’s comments) would have avoided this
comment.

If this helps to resolve these issues - I'd like to submit a rev with all
the new updates.

Gorry

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 17 Nov 2015 11:29:58 -0600

As the AD who's AD evaluating this document ...

> response. In that case, I'd prefer the method used a design that resembles

I thought the "decisively"/"order of magnitude" language in

   o  Once triggered, the Circuit Breaker MUST react decisively by
      disabling or significantly reducing traffic at the source (e.g.,
      ingress).  A reaction that results in a reduction SHOULD result in
      reducing the traffic by at least an order of magnitude, each time
      the Circuit Breaker is triggered.  This response needs to be much
      more severe than that of a Congestion Controller algorithm (such
      as TCP's congestion control [RFC5681
<https://tools.ietf.org/html/rfc5681>] or TCP-Friendly Rate
      Control, TFRC [RFC5348 <https://tools.ietf.org/html/rfc5348>]),
because the Circuit Breaker reacts to
      more persistent congestion and operates over longer timescales
      (i.e., the overload condition will have persisted for a longer

      time before the Circuit Breaker is triggered).

made a pretty clear distinction between "we are twiddling with this using
transport-level mechanisms to see if we can make things better" and "we are
dropping a hammer on this, because we've been twiddling with this using
transport-level mechanisms for some time, and twiddling hasn't made things
better".

Is the working group backing away from that, and going to a more gradual
continuum?

Curiously yours,

Spencer

---

From: "Black, David" <[email-masked]>
Date: Tue, 17 Nov 2015 23:50:05 +0000

<WG chair hat OFF>

> Is the working group backing away from that, and going to a more gradual continuum?

I don’t think so - I don’t view Mirja’s original comment as asking for
gradual reduction or some form of congestion control (e.g., the current
“react decisively” language seems fine).

Rather, there may be a tradeoff in designing a circuit breaker
between the reaction time (including time covered by the measurements
that are used to trip the circuit breaker) and the level of impact of
the circuit breaker.  The draft already recommends “at least an order
of magnitude” reduction and notes that “operates over longer timescales”
by comparison to congestion control mechanisms which is a “hint about how
much it should be reduced at once.”  Beyond that, as one moves from
reduction to cessation, lengthening that time-scale may be appropriate.

I think not making a change here is ok, as there is already text that
addresses her primary comment (hint about ...) and the rest of the
text was a weak suggestion (Maybe ...).

</WG chair hat OFF>

Thanks,
--David

From: Spencer Dawkins at IETF [mailto:[email-masked]]
Sent: Tuesday, November 17, 2015 12:30 PM
To: [email-masked]
Cc: Black, David; [email-masked]
Subject: Re: [tsvwg] Mirja's comments on the circuit breaker draft

      as TCP's congestion control [RFC5681<https://tools.ietf.org/html/rfc5681>] or TCP-Friendly Rate

      Control, TFRC [RFC5348<https://tools.ietf.org/html/rfc5348>]), because the Circuit Breaker reacts to

made a pretty clear distinction between "we are twiddling with this using transport-level mechanisms to see if we can make things better" and "we are dropping a hammer on this, because we've been twiddling with this using transport-level mechanisms for some time, and twiddling hasn't made things better".

Is the working group backing away from that, and going to a more gradual continuum?

Curiously yours,

Spencer

---

From: "Black, David" <[email-masked]>
Date: Wed, 18 Nov 2015 00:01:39 +0000

Gorry,

<WG chair hat OFF>

I've responded to the "use a circuit breaker to gradually reduce traffic"
concern in a separate email to Spencer - IMHO, more has been read into
that comment than is actually there, and (with thanks to Spencer for
quoting the relevant text), no text change is needed to address it.

Everything else below is fine, including "persistent excessive congestion."
WRT that term, what constitutes "excessive" may be the network operator's
call, which suggests that at least the trigger threshold, and perhaps also
the measurement interval on which triggering is based, SHOULD be operator-
configurable (with reasonable defaults defined).
 
</WG chair hat OFF>

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, November 17, 2015 11:46 AM
> To: Black, David
> Cc: [email-masked]
> I've tried to answer the comments below by preparing a new revision, but
> Therefore I suggest "persistent excessive congestion"?
> 
> requirements section (based on Boib's comments) would have avoided this
> comment.
> 
> >> ----------
> >>
> >> </WG chair hat OFF>
> >>
> >> Thanks,
> >> --David

---

From: [email-masked]
Date: Wed, 18 Nov 2015 07:31:18 -0000

OK.

In this case, I'll upload a new revision to address these comments and the
final part of the Apps review, and see if anyone else notes anything.

Gorry

---

From: Dave Taht <[email-masked]>
Date: Wed, 18 Nov 2015 08:53:26 +0100

I have largely missed this discussion. To what extent is this
mechanism intended to deal with DDOS attacks? To what extent does it
resemble existing mechanisms used by cloudflare et al.?

==
Dave Täht
Let's go make home routers and wifi faster! With better software!
https://www.gofundme.com/savewifi

---

From: [email-masked]
Date: Wed, 18 Nov 2015 09:29:48 -0000

This doesn't target DDOS - the original motivations came from work on
tunnels, and also on transport mechanisms that had no CC, but needed to
control their traffic envelope.

Gorry

> Dave TÃ¤ht