Subject: Re: [tsvwg] TSVWG WGLC: draft-ietf-tsvwg-circuit-breaker-02

From: Colin Perkins <[email-masked]>
Date: Thu, 3 Sep 2015 17:09:20 +0100

On 17 Aug 2015, at 03:29, Black, David <[email-masked]> wrote:
> This email announces a TSVWG Working Group Last Call (WGLC) on:
> 
> 	Network Transport Circuit Breakers
> 	draft-ietf-tsvwg-circuit-breaker-02 
> 
> Due to the timing (many people take time off during August), this WGLC will
> run for longer than usual - it will be for just over 3 weeks, ending at
> midnight US Eastern (daylight) Time on Tuesday, September 8 (this date &
> time have been chosen to be after the US Labor Day holiday on Monday,
> September 7).  Comments should be sent to the [email-masked] list,
> although purely editorial comments may be sent directly to the author
> (Gorry Fairhurst <[email-masked]>).

I've reviewed this draft. In general, it seems well written, and the proposals it makes are appropriate. I support publication, but do have some detailed comments:

- Section 1, 4th paragraph ("There are important differences..."): I'd agree that the congestion control algorithms used in the Internet do operate using timescales on the order of the RTT, but it's not clear that they "need to" operate on such a timescale (e.g., one could envisage a system with slowly response congestion control that operates over longer time scales). Changing "need to operate" to "operates" would address this.

- Section 3.1, 1st paragraph: suggest adding an example of an end-host circuit breaker.

- Section 3.1, list item 1: perhaps add "for example" to "could be every few seconds", since not all circuit breakers operate on that timescale.

- Section 3.1, list item 4: "accurate measurement intervals are not typically important, since isolated loss events need to be disregarded" - to clarify, is it the measurement interval that doesn't have to be accurate, or the measurement that is conducted during the interval? It might be useful to be explicit about the degree of inaccuracy that's acceptable. 

- Section 3.1, list item 5: "needs to be robust to single or spurious events triggering a reaction" might be clearer written "need to be robust so that single or spurious events do not trigger a reaction"?

- Section 3.1, list item 6: to match the style of the other list items, suggest changing "The CB also triggers when..." to "A feedback mechanism that triggers when...". Also, change "sender and receiver measurements" to "ingress and egress measurements" for consistency with Figure 1.

- Section 4, 1st bullet: "indicating a congested period is designed" - perhaps "needs to be designed" would be clearer, since it's imposing requirements on a future protocol?

- Section 4, 2nd bullet: this suggests the minimum measurement period "must" ("MUST"?) be many RTTs. That's not always feasible for the RTP circuit breaker, where the measurement period can sometime be significantly less than the RTT, depending on the session bandwidth, RTT, and configuration. Having the circuit breaker measure over small intervals, but average several measurements to get a longer reaction period would seem acceptable, yet is prohibited by this text. 

- Section 4, 6th bullet: "significantly reduced" - what counts as significant? I suggest being more explicit, for example "reduced by at least a factor of ten". 

- Section 4, 8th bullet: "MUST NOT be immediate" is a reasonable start, but what is acceptable? Would it make sense to define a minimum restart period? The RTP circuit breaker says don't restart until you have some indication that the congestion might have dissipated, instead, since it was difficult to specify a time.

- Section 4.1: would an entirely received driven circuit breaker that drops multicast group membership be suitable? Not clear from the text if this is acceptable. 

- Section 5.1.1, 2nd paragraph: "RTCP Reception Report (RR or XRR) packets" is incorrect, and should be "RTCP reception report blocks, as contained in SR or RR packets".

- Section 6.1, 2nd paragraph: might be useful to state that persistent congestion could occur if a failure causes the traffic to be routed over a non-pre-provisioned path, for example?

- Section 6.2: not clear whether this applies to all CC traffic, or just tunnelled CC traffic. The section heading is general, but the text seems specific to tunnels.

- The draft contains lower-case RFC 2119 keywords on the following lines:

80:   6.  Examples where circuit breakers may not be needed.  . . . . .  11
147:   may react to a single packet loss/marking and reduce the transmission
153:   In contrast, CBs are recommended for non-congestion-controlled
156:   circuit-breakers may or may not be desirable.
159:   is experiencing congestion and must be designed to trigger robustly
182:   the required protection to prevent persistent congestion being
202:      functions that may be present in a managed service to implement a
241:   and a egress point.  In some cases, the ingress and egress may be
285:       measures the supported load and may utilise other signals to
290:       the CB Measurement function.  This may use several methods
355:      measurement period needs to be.  The minimum time must be
368:      triggering period.  For example, a CB may combine the results of
399:      will usually be required to restore the flow.  If an automated
424:   control protocol may be invoked to shed the network load when the
429:   There are multiple types of CB that may be defined for use in
439:   since it may prematurely trigger (e.g., when multiple congestion-
480:       falls below some required minimum acceptable quality.
484:   A slow-trip CB may be implemented in an endpoint or network device.
494:   tunnel may need to cross a customer access router.
525:   infrastructure, then it may be expected that the Pseudo-Wire (PW)
532:   recommended measurement period is 1 second, and the trigger operates
536:   If such a condition is detected, a SAToP PW should shut down
555:   the trigger while engineering a solution to the problem.  This may
568:6.  Examples where circuit breakers may not be needed.
570:   A CB is not required for a single CC-controlled flow using TCP, SCTP,
582:   argued that this event ought never to happen - but it may also be
600:   However, the answer to the question may not be obvious, because the
604:   reduce the rate of a flow, an aggregate of many short flows may
608:   headers may not be possible).  The important thing to note is that if
653:   protection, it may be trivial for an attacker to inject packets with
679:   traffic may be a side-effect of a congested network, but also may
682:   Each design of a circuit breaker must evaluate whether the particular

To avoid any potential confusion, it would be good to review these and either change them to upper case if the normative term was intended, or rewrite them to avoid the RFC 2119 term.

- Editorial: I think the draft would be clearer if the abbreviations CB and CC were expanded throughout.

> As part of this WGLC, comments are requested on the appropriate RFC status
> for this draft - this circuit breaker draft has been written with BCP (Best
> Current Practice) RFC status in mind, but another status (e.g., Proposed
> Standard) may be appropriate.  Reviews and comments should assume BCP as
> the intended status, although that may change as a result of the WGLC
> (in consultation with our ADs).

The draft seems to describe requirements for how to implement a circuit breaker. As a BCP on that topic, it seems fine.

Section 1, 5th paragraph says "CBs are recommended for non-congestion controlled Internet flows and for traffic aggregates". It's not clear if this is intended to impose a requirement on implementations or future standards work, or not. If not, perhaps it ought to be rephrased as "CBs are intended for..."; if it is intended to be a requirement, then it needs to use a clear RFC 2119 term ("RECOMMENDED"?), and be in the main body of the document, after the RFC 2119 terminology is introduced, rather than being in the Introduction.

Cheers,
Colin

-- 
Colin Perkins
https://csperkins.org/

---

From: [email-masked]
Date: Sat, 5 Sep 2015 11:12:53 +0100

These are good comments Colin, thanks for the detailed read. I'm going to
implement fixes to these immediately, but first author comments below.

Gorry

> I've reviewed this draft. In general, it seems well written, and the
> proposals it makes are appropriate. I support publication, but do have
> some detailed comments:
>
> - Section 1, 4th paragraph ("There are important differences..."): I'd
> agree that the congestion control algorithms used in the Internet do
> operate using timescales on the order of the RTT, but it's not clear that
> they "need to" operate on such a timescale (e.g., one could envisage a
> system with slowly response congestion control that operates over longer
> time scales). Changing "need to operate" to "operates" would address this.
>
I think it is essential, since the reaction is intended to be significant,
(more than CC) and the traffic shares with other CC-controlled flows that
may have very different RTTs and CC behaviours.

As an example, for TCP, it's unwise for RTO to be equal to RTT - which
would otherwise cause ssthresh to be rest and congestion path state to be
loss. So the IETF doesn't recommend setting RTO aggressively small.

However, this clearly requires proper explanation and I shall rework the
text and see if this seems acceptable to you.

> - Section 3.1, 1st paragraph: suggest adding an example of an end-host
Done. Added figure and updated explanation.
Also added note to address issue from David Black on alternative trigger
placement.

> - Section 3.1, list item 1: perhaps add "for example" to "could be every
> few seconds", since not all circuit breakers operate on that timescale.
>
Done.

> - Section 3.1, list item 4: "accurate measurement intervals are not
> typically important, since isolated loss events need to be disregarded" -
> to clarify, is it the measurement interval that doesn't have to be
> accurate, or the measurement that is conducted during the interval? It
> might be useful to be explicit about the degree of inaccuracy that's
> acceptable.
>
I added text, but don't know how to pin-down numbers. I also tried to hint
that rates and ratios are better than values when making decisions. if
people have ideas of good guidance that would be excellent.

> - Section 3.1, list item 5: "needs to be robust to single or spurious
> events triggering a reaction" might be clearer written "need to be robust
> so that single or spurious events do not trigger a reaction"?
>
Done.

> - Section 3.1, list item 6: to match the style of the other list items,
> suggest changing "The CB also triggers when..." to "A feedback mechanism
> that triggers when...". Also, change "sender and receiver measurements" to
> "ingress and egress measurements" for consistency with Figure 1.
>
Done.

> - Section 4, 1st bullet: "indicating a congested period is designed" -
> perhaps "needs to be designed" would be clearer, since it's imposing
> requirements on a future protocol?
>
Done.

> - Section 4, 2nd bullet: this suggests the minimum measurement period
> "must" ("MUST"?) be many RTTs. That's not always feasible for the RTP
> circuit breaker, where the measurement period can sometime be
> significantly less than the RTT, depending on the session bandwidth, RTT,
> and configuration. Having the circuit breaker measure over small
> intervals, but average several measurements to get a longer reaction
> period would seem acceptable, yet is prohibited by this text.
>
Yes, it's a must for the minperiod - but text updated, and noted that you
can utilise multiple measures to achieve a sufficiently long measurement
period. Please text new text when sent.

> - Section 4, 6th bullet: "significantly reduced" - what counts as
> significant? I suggest being more explicit, for example "reduced by at
> least a factor of ten".
>
Done.

> - Section 4, 8th bullet: "MUST NOT be immediate" is a reasonable start,
> but what is acceptable? Would it make sense to define a minimum restart
> period? The RTP circuit breaker says don't restart until you have some
> indication that the congestion might have dissipated, instead, since it
> was difficult to specify a time.
>
Added text.

> - Section 4.1: would an entirely received driven circuit breaker that
> drops multicast group membership be suitable? Not clear from the text if
> this is acceptable.
>
Done. Added a multicast example, because it is different and I hope a
useful concrete example.

> - Section 5.1.1, 2nd paragraph: "RTCP Reception Report (RR or XRR)
> packets" is incorrect, and should be "RTCP reception report blocks, as
> contained in SR or RR packets".
>
Done.

> - Section 6.1, 2nd paragraph: might be useful to state that persistent
> congestion could occur if a failure causes the traffic to be routed over a
> non-pre-provisioned path, for example?
>
Done.

> - Section 6.2: not clear whether this applies to all CC traffic, or just
> tunnelled CC traffic. The section heading is general, but the text seems
> specific to tunnels.
>
Text updated to indicate tunnels.

> To avoid any potential confusion, it would be good to review these and
> either change them to upper case if the normative term was intended, or
> rewrite them to avoid the RFC 2119 term.
>
All done.

> and CC were expanded throughout.
Done.

> The draft seems to describe requirements for how to implement a circuit
> breaker. As a BCP on that topic, it seems fine.
>
> Section 1, 5th paragraph says "CBs are recommended for non-congestion
> controlled Internet flows and for traffic aggregates". It's not clear if
> this is intended to impose a requirement on implementations or future
> standards work, or not. If not, perhaps it ought to be rephrased as "CBs
> are intended for..."; if it is intended to be a requirement, then it needs
> to use a clear RFC 2119 term ("RECOMMENDED"?), and be in the main body of
> the document, after the RFC 2119 terminology is introduced, rather than
> being in the Introduction.
>
Added a section on the actual recommendation, following above. This will
need working group agreement!

> Colin
Thanks again.

---

From: Colin Perkins <[email-masked]>
Date: Tue, 8 Sep 2015 14:05:56 +0100

Hi Gorry,

Thanks for the update - this looks to address my comments. Some responses inline, based also on https://erg.abdn.ac.uk/users/gorry/ietf/circuit-breaker-02-03-diff.pdf

Cheers,
Colin

> On 5 Sep 2015, at 11:12, [email-masked] wrote:
> 

I’m not sure I entirely agree, but I don’t disagree with the text in the draft, so I think this is fine.

> Also added note to address issue from David Black on alternative trigger

There’s a typo in Figure 1 (“tarffic”), otherwise looks okay.

> people have ideas of good guidance that would be excellent.

The new text is clearer, I think. Perhaps also note that high precision measurement isn’t needed since the goal is to determine if excess congestion is occurring, not to vary the response based on the precise degree of congestion?

>> "ingress and egress measurements" for consistency with Figure 1.

I’m not sure I follow “either the receive or ingress and egress measurements”. Should this be “either the ingress or egress measurements”, or have I misunderstood?

> period. Please text new text when sent.

This is much clearer, and consistent with how the RTP circuit breaker works.

>> was difficult to specify a time.

Good, although I wonder if rather than “SHOULD NOT perform an automated reset when there is evidence of continued congestion” we want a more active “SHOULD NOT perform an automated reset unless it has evidence the path characteristics have changed to reduce congestion”?

> Done. Added a multicast example, because it is different and I hope a

I think this is helpful.

Heading 4.1.2 has a typo “prosvisioned”

> need working group agreement!

Looks good to me.

-- 
Colin Perkins
https://csperkins.org/

---

From: Lucy yong <[email-masked]>
Date: Tue, 8 Sep 2015 16:33:22 +0000

IMO: This draft is about the framework and requirements for the circuit breaker. It specifies the circuit breaker components and design requirement. It does not specify a protocol to implement it. Therefore, it seems proper as an informational draft. The BCP is for an implementation draft, isn't it?

Lucy

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Black, David
Sent: Sunday, August 16, 2015 9:29 PM
To: [email-masked]
Subject: [tsvwg] TSVWG WGLC: draft-ietf-tsvwg-circuit-breaker-02
Importance: High

Due to the timing (many people take time off during August), this WGLC will run for longer than usual - it will be for just over 3 weeks, ending at midnight US Eastern (daylight) Time on Tuesday, September 8 (this date & time have been chosen to be after the US Labor Day holiday on Monday, September 7).  Comments should be sent to the [email-masked] list, although purely editorial comments may be sent directly to the author (Gorry Fairhurst <[email-masked]>).

As part of this WGLC, comments are requested on the appropriate RFC status for this draft - this circuit breaker draft has been written with BCP (Best Current Practice) RFC status in mind, but another status (e.g., Proposed
Standard) may be appropriate.  Reviews and comments should assume BCP as the intended status, although that may change as a result of the WGLC (in consultation with our ADs).

As I mentioned in the Prague tsvwg meeting, there is an expired WG draft (formerly in the pwe3 WG, now the responsibility of the pals WG) on congestion considerations for TDM pseudowires (PWs) - that draft contains some discussion of a managed circuit breaker for TDM PWs.  If that draft reappears before the end of this tsvwg LC on the circuit breaker draft, that reappearance will be taken as a tsvwg WGLC comment requesting changes/additions to section 5.3.1 of the tsvwg circuit breaker draft, which currently discusses another managed circuit breaker for SAToP PWs.

To set a good example, I have reviewed the draft (it's well-written, and a relatively easy read) and have a few LC comments as a WG chair to start this
WGLC:

Section 3.1, end of 1st paragraph:

   A CB is used to control traffic
   passing through a subset of these routers, acting between an ingress
   and a egress point.  In some cases, the ingress and egress may be
   within one or both network endpoints, in other cases they will be
   within a network device.  For example, one expected use would be at
   the ingress and egress of a tunnel service.

Editorial: "tunnel service" -> "service"

Editorial: Figure 1 shows measure and trigger at ingress - they could also be at ingress or in network operations/management infrastructure, e.g., with trigger event communicated to ingress to take action.  The text following Figure 1 looks ok, so this is a request for text to indicate that Figure 1 is not fully general in this regard.

Technical: p.7:

   o  A CB MUST define a measurement period over which the receiver
      measures the level of congestion.

"congestion" -> "congestion or loss"  Congestion may not be directly measurable in all cases, even when ECN is used, as ECN nodes may drop packets, and packets can be dropped for non-congestive reasons.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Black, David" <[email-masked]>
Date: Sun, 13 Sep 2015 20:18:48 +0000

Hi Lucy,

> The BCP is for an implementation draft, isn't it?

No.  According to RFC 2026, Section 5 (second sentence in
first paragraph), a BCP should be:

	- "a statement of principle"; or
	- "what is believed to be the best way to perform some
		operations or IETF process function".

Do you think this fits one or both of those categories?

I do think your comment on implementation is apt, though, as it
suggests that Informational status would be more appropriate
than Proposed Standard status if this draft does not become a BCP.

Thanks,
--David

> From: tsvwg [mailto:[email-masked]] On Behalf Of Lucy yong
> Sent: Tuesday, September 08, 2015 12:33 PM
> To: [email-masked]
> Subject: Re: [tsvwg] TSVWG WGLC: draft-ietf-tsvwg-circuit-breaker-02
> 
> IMO: This draft is about the framework and requirements for the circuit
> breaker. It specifies the circuit breaker components and design requirement.
> It does not specify a protocol to implement it. Therefore, it seems proper as
> an informational draft. The BCP is for an implementation draft, isn't it?
> 
> Lucy
> 
> midnight US Eastern (daylight) Time on Tuesday, September 8 (this date & time
> have been chosen to be after the US Labor Day holiday on Monday, September 7).
> Comments should be sent to the [email-masked] list, although purely editorial
> comments may be sent directly to the author (Gorry Fairhurst
> <[email-masked]>).
> 
> As part of this WGLC, comments are requested on the appropriate RFC status for
> this draft - this circuit breaker draft has been written with BCP (Best
> Standard) may be appropriate.  Reviews and comments should assume BCP as the
> intended status, although that may change as a result of the WGLC (in
> consultation with our ADs).
> 
> As I mentioned in the Prague tsvwg meeting, there is an expired WG draft
> (formerly in the pwe3 WG, now the responsibility of the pals WG) on congestion
> considerations for TDM pseudowires (PWs) - that draft contains some discussion
> of a managed circuit breaker for TDM PWs.  If that draft reappears before the
> end of this tsvwg LC on the circuit breaker draft, that reappearance will be
> taken as a tsvwg WGLC comment requesting changes/additions to section 5.3.1 of
> the tsvwg circuit breaker draft, which currently discusses another managed
> circuit breaker for SAToP PWs.
> 
> To set a good example, I have reviewed the draft (it's well-written, and a
> relatively easy read) and have a few LC comments as a WG chair to start this
> WGLC:
> 
> Editorial: Figure 1 shows measure and trigger at ingress - they could also be
> at ingress or in network operations/management infrastructure, e.g., with
> trigger event communicated to ingress to take action.  The text following
> Figure 1 looks ok, so this is a request for text to indicate that Figure 1 is
> not fully general in this regard.
> 
> Technical: p.7:
> 
> "congestion" -> "congestion or loss"  Congestion may not be directly
> measurable in all cases, even when ECN is used, as ECN nodes may drop packets,
> and packets can be dropped for non-congestive reasons.
> 
> Thanks,
> --David