Subject: Re: [Tsv-art] Tsvart last call review of draft-ietf-detnet-architecture-08

From: "Black, David" <[email-masked]>
Date: Sun, 4 Nov 2018 17:21:12 +0000

Trying to help with these Transport issues.  I generally agree with Michael Sharf's
first two concerns.  Like Michael, I will defer to the IESG on appropriate use of
RFC 7426.

In each case, I've extracted a key piece of text to try to make the point crisp ...

[1] Minimum required level of DetNet support.

> Does "some of the DetNet capabilities
> (not necessarily all)" include e.g. vanilla traffic engineering? For instance, if a
> network node can ensure that the QoS requirements are met by leveraging
> existing standardized TE without any additional DetNet awareness, is this
> enough for deploying DetNet? Or does this node have to implement additional
> DetNet-specific functionality? Specifically, please explain if an existing DetNet-
> unaware router on the path taken by DetNet traffic can be used to forward
> DetNet traffic, or if it is mandatory that any router forwarding DetNet traffic
> indeed implements new capabilities specific to DetNet.

It's the former, as indicated by this text in Section 4.2.2 (I'm working from the -09 draft):

    In general, if a DetNet flow passes through one or more DetNet-
   unaware network nodes between two DetNet nodes providing the DetNet
   transport sub-layer for that flow, there is a potential for
   disruption or failure of the DetNet QoS.  A network administrator
   needs to ensure that the DetNet-unaware network nodes are configured
   to minimize the chances of packet loss and delay, and provision
   enough extra buffer space in the DetNet transit node following the
   DetNet-unaware network nodes to absorb the induced latency
   variations.

Perhaps a forward reference to Section 4.2.2 from the new text in the Introduction would help.

[2] Protecting the network from detnet problems, failures, etc.

> > * It is surprising that there is hardly any discussion on network robustness
> > and safety; this probably also relates to security. For instance,
> > misconfiguration or errors of functions performing packet replication could
> > severely and permanently congest a network and cause harm.
[... snip ...]
> NEW:
> Robust real-time systems require to reduce the number of possible
> failures. Filters and policers should be used in a DetNet network to
> detect if DetNet packets are received on the wrong interface, or at
> the wrong time, or in too great volume. Furthermore, filters and
> policers can take action to discard offending packets or flows, or
> trigger shutting down the offending DetNet flow, or shutting down
> the offending interface.
> 
> [ms] That does not address my concern. As I wrote already, I believe this
> document has to discuss the applicability of circuit breakers according to RFC
> 8084.

I concur with Michael's implied concern that the "should" in the NEW text
is woefully inadequate.  The provisions to prevent escaped non-elastic traffic
from causing damage elsewhere need to be mandatory particularly if DetNet
and non-DetNet networks are interconnected.  The above
text on filters doesn't appear to cover escape from the DetNet network,
and needs to do so.  Deployment of filters at DetNet nodes that drop all
DetNet traffic that tries to escape, and that drop all misdirected DetNet
traffic should address most of this problem.

Once that is done, I think a discussion of circuit breakers is needed, in the
form of Detnet service mechanisms that can shut down (cease transmission
of) a flow that is being sent entirely into the bit-bucket by a downstream filter,
due to misconfiguration, failure, etc.

Thanks, --David

> -----Original Message-----
> From: Tsv-art [mailto:[email-masked]] On Behalf Of Scharf, Michael
> Sent: Monday, October 22, 2018 5:44 PM
> To: 'János Farkas'
> Cc: [email-masked]; DetNet WG; [email-masked]
> Subject: Re: [Tsv-art] Tsvart last call review of draft-ietf-detnet-architecture-08
> 
> 
> [EXTERNAL EMAIL]
> Please report any suspicious attachments, links, or requests for sensitive
> information.
> 
> 
> Hi Janos,
> 
> Thanks for the reply. My comments are inline [ms].
> 
> I have omitted all proposed changes that work for me.
> 
> Michael
> 
> 
> -----Original Message-----
> From: János Farkas <[email-masked]>
> Sent: Wednesday, October 17, 2018 10:56 AM
> To: Scharf, Michael <[email-masked]>
> Cc: [email-masked]; [email-masked]; DetNet WG
> <[email-masked]>
> Subject: Re: Tsvart last call review of draft-ietf-detnet-architecture-08
> 
> Hi Michael,
> 
> Thank you very much for the thorough review and good comments!
> 
> Please find below in-line responses and how we plan to update the draft to
> resolve your comments.
> 
> Best regards,
> Janos
> 
> 
> On 9/29/2018 12:24 AM, Michael Scharf wrote:
> > Reviewer: Michael Scharf
> > Review result: Ready with Issues
> >
> > The document "Deterministic Networking Architecture"
> > (draft-ietf-detnet-architecture-08) defines an overall framework for
> > Deterministic Networking.
> >
> > As TSV-ART reviewer, I believe that this document has issues as
> > detailed below.
> >
> > Michael
> >
> > Major issues:
> 
> [...]
> 
> > DetNet
> > basically requires support in (almost) all network devices
> > transporting DetNet traffic. That assumption should be explicitly
> > spelt out early in the document, e.g., in the introduction.
> 
> Actually, some form of DetNet support is required from each node that
> transports DetNet traffic. For instance, DetNet flows have to be recognized in
> order not to spoil their QoS at the minimum.
> 
> [ms] I look for an clear explanation in the architecture document of what "some
> form of DetNet support" exactly means. For instance, is standard MPLS TE
> without any DetNet support good enough for an LSR on the IP/MPLS path used
> by DetNet traffic, or not? As an example, when I read the following paragraph in
> draft-ietf-detnet-dp-sol-mpls-01 ...
> 
>    Transit nodes are normal MPLS Label Switching Routers (LSRs).  They
>    are generally unaware of the special requirements of DetNet flows,
>    although they need to provide traffic engineering services and proper
>    QoS to the LSPs associated with DetNet flows to enhance the prospect
>    of the LSPs meeting the DetNet service requirements.  Some
>    implementations of transit nodes may be DetNet aware, but such nodes
>    just support the DetNet transport layer.
> 
> ... I could maybe assume that actually there can be "transit nodes" that do not
> require *any* sort of DetNet support or DetNet awareness. Is that correct? I
> think the architecture document has to be clear on such fundamental questions.
> 
> The Introduction could be extended to clarify, e.g., the third paragraph:
> 
> OLD:
> A goal of DetNet is a converged network in all respects. That is, the presence of
> DetNet flows does not preclude non-DetNet flows, and the benefits offered
> DetNet flows should not, except in extreme cases, prevent existing QoS
> mechanisms from operating in a normal fashion, subject to the bandwidth
> required for the DetNet flows. A single source-destination pair can trade both
> DetNet and non-DetNet flows. End systems and applications need not
> instantiate special interfaces for DetNet flows. Networks are not restricted to
> certain topologies; connectivity is not restricted. Any application that generates
> a data flow that can be usefully characterized as having a maximum bandwidth
> should be able to take advantage of DetNet, as long as the necessary resources
> can be reserved. Reservations can be made by the application itself, via network
> management, by an application’s controller, or by other means, e.g., a dynamic
> control plane (e.g., [RFC2205]).
> 
> NEW:
> A goal of DetNet is a converged network in all respects. That is, the presence of
> DetNet flows does not preclude non-DetNet flows, and the benefits offered
> DetNet flows should not, except in extreme cases, prevent existing QoS
> mechanisms from operating in a normal fashion, subject to the bandwidth
> required for the DetNet flows. A single source-destination pair can trade both
> DetNet and non-DetNet flows. End systems and applications need not
> instantiate special interfaces for DetNet flows. Networks are not restricted to
> certain topologies; connectivity is not restricted. Any application that generates
> a data flow that can be usefully characterized as having a maximum bandwidth
> should be able to take advantage of DetNet, as long as the necessary resources
> can be reserved. Reservations can be made by the application itself, via network
> management, by an application’s controller, or by other means, e.g., a dynamic
> control plane (e.g., [RFC2205]). Network nodes supporting DetNet flows have to
> implement some of the DetNet capabilities (not necessarily all) in order to treat
> DetNet flows such that their QoS requirements are met.
> 
> [ms] To me, the last sentence is not clear. Does "some of the DetNet capabilities
> (not necessarily all)" include e.g. vanilla traffic engineering? For instance, if a
> network node can ensure that the QoS requirements are met by leveraging
> existing standardized TE without any additional DetNet awareness, is this
> enough for deploying DetNet? Or does this node have to implement additional
> DetNet-specific functionality? Specifically, please explain if an existing DetNet-
> unaware router on the path taken by DetNet traffic can be used to forward
> DetNet traffic, or if it is mandatory that any router forwarding DetNet traffic
> indeed implements new capabilities specific to DetNet. In the latter case, I think
> the document would need to discuss incremental deployment strategies as well.
> 
> > * It is surprising that there is hardly any discussion on network robustness
> > and safety; this probably also relates to security. For instance,
> > misconfiguration or errors of functions performing packet replication could
> > severely and permantly congest a network and cause harm. How does the DetNet
> > architecture ensure that a network stays fully operational e.g. if the topology
> > changes or there are equipment failures? Probably this can be solved by
> > implementations (e.g., dynamic control plane), but why are corresponding
> > requirements not spelt out? Section 3.3.2 speculates that filters and policers
> > can help, and that may be true, but that probably still assumes consistently
> > and correctly configured (and well-behaving) devices. And Section 3.3.2 is
> > vague and mentions a "infinite variety of possible failures" without stating
> > any requirements or recommendations. There may be further solutions, such as
> > circuit breakers and the like. Why are such topics not discussed?
> Security issues and considerations are addressed by the DetNet Security
> draft: https://datatracker.ietf.org/doc/draft-ietf-detnet-security.
> Reference to the  DetNet Security draft will be added.
> 
> There will be a document dedicated to Operations, Administration and
> Maintenance (OAM), which will address operational, misconfiguration, and
> failure detection issues.
> 
> The topology changes have to be solved by the control plane, either
> centralized or distributed.
> 
> The filters and policers described in Section 3.3.2 also provide
> robustness by eliminating/mitigating traffic coming from malfunctioning
> devices.
> 
> RFC2475 is recommended for traffic policing functions to increase
> robustness.
> 
> The text  in Section 3.3.2 could be made clearer, e.g., by updating the
> first paragraph to:
> 
> OLD:
> One key to building robust real-time systems is to reduce the
> infinite variety of possible failures to a number that can be
> analyzed with reasonable confidence. DetNet aids in the process by
> allowing for filters and policers to detect DetNet packets received
> on the wrong interface, or at the wrong time, or in too great a
> volume, and to then take actions such as discarding the offending
> packet, shutting down the offending DetNet flow, or shutting down the
> offending interface.
> 
> NEW:
> Robust real-time systems require to reduce the number of possible
> failures. Filters and policers should be used in a DetNet network to
> detect if DetNet packets are received on the wrong interface, or at
> the wrong time, or in too great volume. Furthermore, filters and
> policers can take action to discard offending packets or flows, or
> trigger shutting down the offending DetNet flow, or shutting down
> the offending interface.
> 
> [ms] That does not address my concern. As I wrote already, I believe this
> document has to discuss the applicability of circuit breakers according to RFC
> 8084.
> 
> > * Somewhat related, the document only looks at impact of failures to the QoS of
> > DetNet traffic. What is missing is a discussion how to protect non-DetNet parts
> > of a network from any harm caused by DetNet mechanisms. Solutions to this
> > probably exist. But why is the impact on non-DetNet traffic (e.g., in case of
> > topology changes or failures of DetNet functions) not discussed at all in the document?
> Actually the regulation of DetNet traffic by polices and filters
> protects both other DetNet traffic and non-DetNet traffic.
> 
> Section 3.3.1 could be extended to make it clear, e.g., by extending the
> last paragraph:
> 
> OLD:
> Ideally, the net effect of the presence of DetNet flows in a network
> on the non-DetNet packets is primarily a reduction in the available
> bandwidth.
> 
> NEW:
> Traffic policing functions (e.g. [RFC2475]) are used to avoid the
> starvation of non-DetNet traffic. Thus, the net effect of the presence
> of DetNet flows in a network on non-DetNet flows is primarily a
> reduction in the available bandwidth.
> 
> [ms] I believe a stronger wording is needed, e.g. along the lines of "a DetNet
> deployment must avoid starvation of non-DetNet traffic ...".
> 
> [...]
> 
> > * Section 4.4 refers to RFC 7426, which is an informational RFC on IRTF stream,
> > and the document uses the concepts introduced there (e.g., "planes"). This is
> > very confusing. First, an IETF Proposed Standard should probably refer to
> > documents having IETF consensus. An example would be RFC 7491, albeit there is
> > other related work as well, e.g., in the TEAS WG. Second, Section 4.4 is by and
> > large decoupled from the rest of the document and not specific to DetNet.
> > Neither do other sections of the document refer to the concepts introduced in
> > Section 4.4, nor does Section 4.4 use the DetNet terminology or discuss
> > applicability to DetNet. Section 4.4 even mentions explicitly at the end that
> > it discusses aspects that are orthogonal to the DetNet architecture. It is not
> > at all clear why Section 4.4 is in this document. Section 4.4 could be removed
> > from the document without impacting the rest of the document.
> The document should point out to traffic engineering and centralized
> control plane aspects, so it is better to keep Section 4.4.
> 
> RFC 7426 provides the full picture of SDN architecture, which needs to
> be referred from the DetNet architecture. No other RFC provides such
> detailed picture. Therefore, we need to keep the reference to RFC 7426,
> which is an informative reference.
> 
> [ms] I'll leave that up to the IESG to decide whether an IETF proposed standard
> should build an architecture based on research work instead of, say, documents
> with IETF consensus. As a side note, I believe the understanding in industry what
> "SDN architecture" really means and how it is implemented may have evolved
> since publication of IRTF RFC 7426, which was published in Jan. 2015. For
> instance, IRTF RFC 7426 does not discuss the implications of segment routing,
> BGP-LS, and other recent IETF techniques. And, based on what I read in the
> DetNet architecture, I believe DetNet could actually be implemented without
> any "SDN architecture", e.g. by distributed traffic engineering. As a result, I don't
> even understand the need to discuss "SDN".
> 
> [ms] For the record: As TSV-ART reviewer, I question whether IRTF RFC 7426 is
> indeed an appropriate reference, I have doubts that IRTF RFC 7426 is really up-
> to-date, and I am concerned about the content quality of section 4.4, e.g.,
> regarding the lack of alignment with the rest of the document. The IESG will
> certainly have a better understanding that than me how to deal with such topics.
> 
> The related work in TEAS is already referred to.
> 
> [...]
> 
> > Minor issues:
> >
> > * Terminology "DetNet transport layer"
> >
> > The term "transport layer" has a well-defined meaning in the IETF, e.g.
> > originating from RFC 1122. While "transport" and e.g. "transport network" is
> > used in the IETF for different technologies in different areas, I think the
> > term "transport layer" is typically understood to refer to transport
> > protocols such as TCP and UDP. As such, I personally find the term "DetNet
> > transport layer" misleading and confusing. The confusion is easy to see e.g.
> > in Figure 4, where UDP (which is a transport protocol as per RFC 1122) sits
> > on top of "transport".
> >
> > Based on the document it also may be solution/implementation specific
> whether
> > the "DetNet transport layer" is actually a separate protocol layer compared
> > to the "DetNet service layer". Thus it is not clear to me why the word
> > "layer" has to be used, specifically in combination "transport layer".
> >
> > To me as, the word "transport layer" (and "transport protocol") should be
> > used for protocols defined in TSV area, consistent with RFC 1122. But this is
> > probably a question to be sorted out by the IESG.
> 
> "transport" is used here as in "packet transport networks" not as OSI L4
> transport.
> 
> [ms] At the risk of stating the obvious: The relevant section of RFC 1122 defining
> "transport layer" is entitled by "The Internet Architecture". And, according e.g.
> to RFC 5921, "packet transport network" is a term defined by ITU-T. In an IETF
> document, I believe IETF terminology should be used.
> 
> I suggest to use the terms "DetNet transport sub-layer" and "DetNet
> service sub-layer" throughout the document, which would hopefully avoid
> confusion.
> 
> [ms] "DetNet transport sub-layer" somewhat solves my concern of avoiding the
> expression "transport layer". To really avoid confusion, I believe it would be
> better to avoid the ambiguous term "transport" altogether.  But I will be fine if
> all DetNet documents consistently avoid the term "transport layer".
> 
> Michael
> _______________________________________________
> Tsv-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tsv-art

---

From: Lou Berger <[email-masked]>
Date: Thu, 8 Nov 2018 14:45:12 +0700

Hi,

See below for in-line responses

On 11/5/2018 12:21 AM, Black, David wrote:
> ...

> due to misconfiguration, failure, etc.

I think we have a bit of a disconnect of what DetNet service delivers.

DetNet does not itself impact any Transport protocol  (e.g., UDP, TCP) 
so itself does not *produce* elastic or non-elastic traffic.

DetNet operates at Layer 3 and basically provides detnet flows with a 
service equivalent to IntServ's Guaranteed Quality of Service.  As such, 
DetNet flows basically operate as if the network is uncongested.

WRT escaped traffic, as I mentioned in today's session, I really think 
this is no difference between escaped DetNet traffic and traffic 
delivered over an uncongested network to a peer (congested or 
uncongested) network.

Based on all this, I'm not sure how or why a  discussion of circuit 
breakers is needed.

Lou

> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

---

From: "Scharf, Michael" <[email-masked]>
Date: Thu, 8 Nov 2018 18:11:54 +0000

Inline

> From: Tsv-art [mailto:[email-masked]] On Behalf Of Lou Berger
> Sent: Thursday, November 8, 2018 8:45 AM
> To: Black, David; Scharf, Michael; 'János Farkas'
> Subject: Re: [Tsv-art] [Detnet] Tsvart last call review of draft-ietf-
> detnet-architecture-08
> 
> Hi,
> 

First, the question whether DetNet *produces* elastic or non-elastic traffic does not matter for the applicability of circuit breakers. Section 3 of RFC 8084 states:

   CBs are RECOMMENDED for IETF protocols and tunnels that carry non-
   congestion-controlled Internet flows and for traffic aggregates.
   This includes traffic sent using a network tunnel.  Designers of
   other protocols and tunnel encapsulations also ought to consider the
   use of these techniques as a last resort to protect traffic that
   shares the network path being used.

Section 3.2.1.1 of draft-ietf-detnet-architecture-09 explains explicitly that DetNet can carry non-congestion-controlled traffic:

   Note that a DetNet flow cannot be throttled, i.e., its
   transmission rate cannot be reduced via explicit congestion
   notification.

So, clearly the condition of carrying non-congestion-controlled flows or traffic aggregates applies to the DetNet architecture. And, as a result, the RECOMMENDED in RFC 8084 is in principle applicable to DetNet. Also, note that the RECOMMENDED in RFC 8084 is not limited to tunnels.

As pointed out by David, the key question is whether, and how, the DetNet architecture prevents that non-congestion controlled traffic escapes from a controlled environment into the Internet. As also explained in RFC 8084, this question in particular matters for failure scenarios.

As TSV-ART reviewer, I insist in a discussion of this problem in draft-ietf-detnet-architecture. And I believe that a reference to RFC 8084 is needed in draft-ietf-detnet-architecture.

Note that the wording in RFC 8084 is RECOMMENDED and not MUST, i.e., it is possible that circuit breakers may not be needed for certain solutions. As an example, MPLS traffic may not easily escape from a managed MPLS network. 

Second, in my TSV-ART review I have explicitly mentioned the DetNet Packet Replication Function (PRF). Replicating non-elastic traffic may create additional challenges if the replicated packets can somehow escape into the Internet, e.g., in failure cases. For instance, as far as I understand draft-ietf-detnet-architecture, I assume the PRF can replicate non-congestion-controlled traffic and forward it along different paths. If one of these paths escapes to the Internet, to the Internet the PRF could basically be a producer of non-congestion controlled traffic, and that traffic could cause significant harm in the Internet. Also, note that in that case it may not necessarily help if the traffic carried by DetNet is elastic, e.g., if DetNet carries TCP traffic. For instance, if other paths used by the PRF within the DetNet domain would deliver all packets, the congestion control in TCP would not never be triggered as each packet gets indeed delivered over one of the paths. So, an unelastic traffic stream would escale into the Internet and never back off. Unless I miss something, DetNet needs counter-means to prevent this sort of issues.

As TSV-ART reviewer, I believe draft-ietf-detnet-architecture needs a dedicated discussion whether the PRF can cause harm and, if so, how to avoid such harm.

> DetNet flows basically operate as if the network is uncongested.

RFC 8084 exactly addresses that case. The question is not the normal operation of the network, but abnormal cases.
 
> delivered over an uncongested network to a peer (congested or

That comes down to how the DetNet network deals with abnormal situations at the edges, e.g., for the problems mentioned in RFC 8084:

   o  anomalous traffic that exceeds the provisioned capacity (or whose
      traffic characteristics exceed the threshold configured for the
      CB);

   o  traffic generated by an application at a time when the provisioned
      network capacity is being utilized for other purposes;

   o  routing changes that cause additional traffic to start using the
      path monitored by the CB;

   o  misconfiguration of a service/network device where the capacity
      available is insufficient to support the current traffic
      aggregate;

   o  misconfiguration of an admission controller or traffic policer
      that allows more traffic than expected across the path monitored
      by the CB.

According to the last point, misconfiguration of filters/policies is one of the challenges to look at.
 
> Based on all this, I'm not sure how or why a  discussion of circuit

RFC 8084 represents IETF community consensus. And in my reading the DetNet architecture falls into the type of solutions discussed in RFC 8084.

Thanks

Michael

---

From: "Black, David" <[email-masked]>
Date: Fri, 9 Nov 2018 04:33:04 +0000

So, I'm going to agree with Michael on the problem, but suggest a different approach for tackling it, with apologies for the length of this message necessitated by the need to "show your work."

> > DetNet flows basically operate as if the network is uncongested.

And hence DetNet flows have no requirement or incentive to react to congestion, even though some DetNet traffic will no doubt react to congestion.

> > delivered over an uncongested network to a peer (congested or

Unfortunately, that argument leads to a contradiction, namely a conclusion that congestion control is not necessary for the Internet, which is known to be wrong.  Having arrived at a result that mathematicians would call "proof by contradiction," one of the assumptions that lead to this point has to be wrong, but which one?

The "clear, simple and wrong" (Mencken) approach to resolving this would be to focus on the inelastic traffic and require that all DetNet traffic MUST be congestion controlled.  I'm not advocating this, and neither is Michael.

A more subtle approach (with credit to Deb Brungard for an insightful discussion after RTGAREA yesterday) is to recognize that DetNet is a Controlled Environment, to which different rules apply than the Internet and networks in general.  For the definition of Controlled Environment, see Section 3.6 of RFC 8085 (https://tools.ietf.org/html/rfc8085#section-3.6).  For now, please ignore the material in that section on safety, robustness and corruption, as inapplicable to this discussion.  FWIW, that material was motivated by the incessant attempts to optimize out UDP checksums without serious consideration of the resulting packet corruption risks and the potential consequences thereof.  Viewing DetNet as a Controlled Environment, this discussion is then primarily about what to require/do at the boundary between the DetNet Controlled Environment and other networks.  I've been there, done that for other Controlled Environments ... and have the scars ... from the worked examples in RFC 7510 (MPLS/UDP) and RFC 8086's TMCE (Traffic Management Controlled Environment) applicability scenario for GRE/UDP.  Those two RFCs may be helpful background reading.

The good news here is that we are not as far from agreeing on a solution as might appear from the length of this discussion.   Going back to a couple of chunks of text that Janos proposed earlier:

	NEW:
	This document provides the overall Architecture for Deterministic
	Networking (DetNet), which provides a capability for the delivery of
	data flows with extremely low packet loss rates and bounded end-to-
	end delivery latency within a network domain. DetNet is for networks
	that are under a single administrative control or within a closed group
	of administrative control; these include campus-wide networks and
	private WANs. DetNet is not for large groups of domains such as the
	Internet.

That reads to me as a clear statement that a DetNet network is a Controlled Environment, so I would hope that we're in agreement on that.

	NEW:

That proposed text actually contains two circuit breakers, "shutting down the offending DetNet flow" or "shutting down the offending interface."  So far, so good, but what's missing is text on deployment requirements of the mechanisms described in that text, particularly at the edges of DetNet networks.

Text that requires deployment of "filters and policers ... [that] discard offending [DetNet] packets and flows" at the boundaries between DetNet and non-DetNet networks would go a long way towards resolving this discussion.  I would suggest expanding "offending" into text that includes all unauthorized DetNet traffic and all unexpected DetNet traffic.  Once that's done, the "circuit breaker" discussion becomes more tractable, as the discarding of all DetNet traffic that would otherwise escape the DetNet network reduces the "circuit breaker" discussion to the existing DetNet material measures that need to be taken inside a DetNet network to protect delivery of DetNet service.

Thanks, --David

> From: Scharf, Michael [mailto:[email-masked]]
> Sent: Thursday, November 8, 2018 1:12 PM
> To: Lou Berger; Black, David; 'János Farkas'
> Subject: RE: [Tsv-art] [Detnet] Tsvart last call review of draft-ietf-detnet-
> architecture-08
> 
> 
> [EXTERNAL EMAIL]
> First, the question whether DetNet *produces* elastic or non-elastic traffic does
> not matter for the applicability of circuit breakers. Section 3 of RFC 8084 states:
> 
> Section 3.2.1.1 of draft-ietf-detnet-architecture-09 explains explicitly that
> DetNet can carry non-congestion-controlled traffic:
> 
> So, clearly the condition of carrying non-congestion-controlled flows or traffic
> aggregates applies to the DetNet architecture. And, as a result, the
> RECOMMENDED in RFC 8084 is in principle applicable to DetNet. Also, note that
> the RECOMMENDED in RFC 8084 is not limited to tunnels.
> 
> As pointed out by David, the key question is whether, and how, the DetNet
> architecture prevents that non-congestion controlled traffic escapes from a
> controlled environment into the Internet. As also explained in RFC 8084, this
> question in particular matters for failure scenarios.
> 
> As TSV-ART reviewer, I insist in a discussion of this problem in draft-ietf-detnet-
> architecture. And I believe that a reference to RFC 8084 is needed in draft-ietf-
> detnet-architecture.
> 
> Note that the wording in RFC 8084 is RECOMMENDED and not MUST, i.e., it is
> possible that circuit breakers may not be needed for certain solutions. As an
> example, MPLS traffic may not easily escape from a managed MPLS network.
> 
> 
> Second, in my TSV-ART review I have explicitly mentioned the DetNet Packet
> Replication Function (PRF). Replicating non-elastic traffic may create additional
> challenges if the replicated packets can somehow escape into the Internet, e.g.,
> in failure cases. For instance, as far as I understand draft-ietf-detnet-
> architecture, I assume the PRF can replicate non-congestion-controlled traffic
> and forward it along different paths. If one of these paths escapes to the
> Internet, to the Internet the PRF could basically be a producer of non-congestion
> controlled traffic, and that traffic could cause significant harm in the Internet.
> Also, note that in that case it may not necessarily help if the traffic carried by
> DetNet is elastic, e.g., if DetNet carries TCP traffic. For instance, if other paths
> used by the PRF within the DetNet domain would deliver all packets, the
> congestion control in TCP would not never be triggered as each packet gets
> indeed delivered over one of the paths. So, an unelastic traffic stream would
> escale into the Internet and never back off. Unless I miss something, DetNet
> needs counter-means to prevent this sort of issues.
> 
> As TSV-ART reviewer, I believe draft-ietf-detnet-architecture needs a dedicated
> discussion whether the PRF can cause harm and, if so, how to avoid such harm.
> 
> That comes down to how the DetNet network deals with abnormal situations at
> the edges, e.g., for the problems mentioned in RFC 8084:
> 
> RFC 8084 represents IETF community consensus. And in my reading the DetNet
> architecture falls into the type of solutions discussed in RFC 8084.
> 
> Thanks
> 
> Michael
> 
> 
> >
> > Lou
> >
> > >
> > > Thanks, --David
> > >

---

From: "Black, David" <[email-masked]>
Date: Fri, 9 Nov 2018 05:08:16 +0000

Acronym expansion correction:

> I've been there, done that for other
> Controlled Environments ... and have the scars ... from the worked examples in
> RFC 7510 (MPLS/UDP) and RFC 8086's TMCE (Traffic Management Controlled
> Environment) applicability scenario for GRE/UDP.  Those two RFCs may be
> helpful background reading.

TMCE = Traffic-Managed Controlled Environment

The spelling corrector "helped" with the original ;-)

Thanks, --David

> From: Black, David
> Sent: Thursday, November 8, 2018 11:33 PM
> To: 'Scharf, Michael'; 'Lou Berger'; 'János Farkas'
> Cc: [email-masked]'; 'DetNet WG'; [email-masked]';
> Black, David
> So, I'm going to agree with Michael on the problem, but suggest a different
> approach for tackling it, with apologies for the length of this message
> necessitated by the need to "show your work."
> 
> And hence DetNet flows have no requirement or incentive to react to
> congestion, even though some DetNet traffic will no doubt react to congestion.
> 
> Unfortunately, that argument leads to a contradiction, namely a conclusion that
> congestion control is not necessary for the Internet, which is known to be
> wrong.  Having arrived at a result that mathematicians would call "proof by
> contradiction," one of the assumptions that lead to this point has to be wrong,
> but which one?
> 
> The "clear, simple and wrong" (Mencken) approach to resolving this would be to
> focus on the inelastic traffic and require that all DetNet traffic MUST be
> congestion controlled.  I'm not advocating this, and neither is Michael.
> 
> A more subtle approach (with credit to Deb Brungard for an insightful discussion
> after RTGAREA yesterday) is to recognize that DetNet is a Controlled
> Environment, to which different rules apply than the Internet and networks in
> general.  For the definition of Controlled Environment, see Section 3.6 of RFC
> 8085 (https://tools.ietf.org/html/rfc8085#section-3.6).  For now, please ignore
> the material in that section on safety, robustness and corruption, as inapplicable
> to this discussion.  FWIW, that material was motivated by the incessant
> attempts to optimize out UDP checksums without serious consideration of the
> resulting packet corruption risks and the potential consequences thereof.
> Viewing DetNet as a Controlled Environment, this discussion is then primarily
> about what to require/do at the boundary between the DetNet Controlled
> Environment and other networks.  I've been there, done that for other
> Controlled Environments ... and have the scars ... from the worked examples in
> RFC 7510 (MPLS/UDP) and RFC 8086's TMCE (Traffic Management Controlled
> Environment) applicability scenario for GRE/UDP.  Those two RFCs may be
> helpful background reading.
> 
> The good news here is that we are not as far from agreeing on a solution as
> might appear from the length of this discussion.   Going back to a couple of
> chunks of text that Janos proposed earlier:
> 
> 	NEW:
> That reads to me as a clear statement that a DetNet network is a Controlled
> Environment, so I would hope that we're in agreement on that.
> 
> 	NEW:
> That proposed text actually contains two circuit breakers, "shutting down the
> offending DetNet flow" or "shutting down the offending interface."  So far, so
> good, but what's missing is text on deployment requirements of the mechanisms
> described in that text, particularly at the edges of DetNet networks.
> 
> Text that requires deployment of "filters and policers ... [that] discard offending
> [DetNet] packets and flows" at the boundaries between DetNet and non-DetNet
> networks would go a long way towards resolving this discussion.  I would
> suggest expanding "offending" into text that includes all unauthorized DetNet
> traffic and all unexpected DetNet traffic.  Once that's done, the "circuit breaker"
> discussion becomes more tractable, as the discarding of all DetNet traffic that
> would otherwise escape the DetNet network reduces the "circuit breaker"
> discussion to the existing DetNet material measures that need to be taken inside
> a DetNet network to protect delivery of DetNet service.
> 
> Thanks, --David
> 
> 
> > RFC 8084 exactly addresses that case. The question is not the normal
> operation
> > of the network, but abnormal cases.
> >

---

From: Lou Berger <[email-masked]>
Date: Fri, 9 Nov 2018 13:37:48 +0700

David,

     I think we're at the point were some specific proposed language 
would be helpful at this point.  (FYI - I'm responding to Michael's 
message, just need some time to finish..)

What language would you like to see in addition to the text you quoted 
below.

See below for more specific and secondary responses.

>> congestion, even though some DetNet traffic will no doubt react to congestion.

This may be true, but this is an issue for users of DetNet services, not 
DetNet.  This is an important distinction and one we can reflect in the 
architecture document -- i.e., that users/applications/transport 
protocols using detnet need to address this and for those uses 
https://tools.ietf.org/html/rfc8084#section-3.2.2 applies.

>> contradiction," one of the assumptions that lead to this point has to be wrong,

Sure, but the current internet architecture, for good or bad, puts the 
burden for congestion control at the transport protocol layer not the 
network layer - and DetNet is about control and service delivery at the 
network layer.

>> congestion controlled.  I'm not advocating this, and neither is Michael.

There is certainly no issues with adding a statement that the use of 
DetNet doesn't change the internet architecture or the need to ensure 
that congestion control is appropriately and 
https://tools.ietf.org/html/rfc8084#section-5.3 applies.  I'll note that 
rfc8084 doesn't place any requirements on general IP traffic that use 
https://tools.ietf.org/html/rfc8084#section-3.2.2

>> helpful background reading.

Again, these are users of DetNet -- DetNet really is nothing more than 
IntServ (for IP data plane) an MPLS-TE (for MPLS data plane) .  Keeping 
this in mind, what is the right 8085 or other language that applies and 
should be added.

>> a DetNet network to protect delivery of DetNet service.

Here's where I think some specific proposed language would be helpful.

Thanks,

Lou

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 9 Nov 2018 06:38:30 +0000

Inline

> From: Black, David [mailto:[email-masked]]
> Sent: Friday, November 9, 2018 5:33 AM
> To: Scharf, Michael; Lou Berger; 'János Farkas'
> Cc: [email-masked]; DetNet WG; draft-ietf-detnet-
> [email-masked]; Black, David
> Subject: RE: [Tsv-art] [Detnet] Tsvart last call review of draft-ietf-
> detnet-architecture-08
> 
> So, I'm going to agree with Michael on the problem, but suggest a
> different approach for tackling it, with apologies for the length of
> this message necessitated by the need to "show your work."
> 

Indeed.

And, as noted in my earlier way, depending on how the PRF is realized DetNet may even shield DetNet traffic from packet losses due to congestion, i.e., even traffic that would be able to react to congestion may not react due to DetNet. If DetNet is able to mask congestion from the carried traffic, special care is needed inside DetNet.
 
> Unfortunately, that argument leads to a contradiction, namely a
> conclusion that congestion control is not necessary for the Internet,
> which is known to be wrong.  Having arrived at a result that
> mathematicians would call "proof by contradiction," one of the
> assumptions that lead to this point has to be wrong, but which one?
> 
> The "clear, simple and wrong" (Mencken) approach to resolving this
> would be to focus on the inelastic traffic and require that all DetNet
> traffic MUST be congestion controlled.  I'm not advocating this, and
> neither is Michael.
> 
> A more subtle approach (with credit to Deb Brungard for an insightful
> discussion after RTGAREA yesterday) is to recognize that DetNet is a
> Controlled Environment, to which different rules apply than the
> Internet and networks in general.  For the definition of Controlled
> Environment, see Section 3.6 of RFC 8085
> (https://tools.ietf.org/html/rfc8085#section-3.6).  For now, please
> ignore the material in that section on safety, robustness and
> corruption, as inapplicable to this discussion.  FWIW, that material
> was motivated by the incessant attempts to optimize out UDP checksums
> without serious consideration of the resulting packet corruption risks
> and the potential consequences thereof.  Viewing DetNet as a Controlled
> Environment, this discussion is then primarily about what to require/do
> at the boundary between the DetNet Controlled Environment and other
> networks.  I've been there, done that for other Controlled Environments
> ... and have the scars ... from the worked examples in RFC 7510
> (MPLS/UDP) and RFC 8086's TMCE (Traffic Management Controlled
> The good news here is that we are not as far from agreeing on a
> solution as might appear from the length of this discussion.   Going
> back to a couple of chunks of text that Janos proposed earlier:
> 
> 	NEW:
> That reads to me as a clear statement that a DetNet network is a
> Controlled Environment, so I would hope that we're in agreement on
> that.
> 
> 	NEW:
> That proposed text actually contains two circuit breakers, "shutting
> down the offending DetNet flow" or "shutting down the offending
> interface."  So far, so good, but what's missing is text on deployment
> requirements of the mechanisms described in that text, particularly at
> the edges of DetNet networks.

Yep, strict deployment requirements are needed.

> Text that requires deployment of "filters and policers ... [that]
> discard offending [DetNet] packets and flows" at the boundaries between
> DetNet and non-DetNet networks would go a long way towards resolving
> this discussion.  I would suggest expanding "offending" into text that
> includes all unauthorized DetNet traffic and all unexpected DetNet
> traffic.  Once that's done, the "circuit breaker" discussion becomes
> more tractable, as the discarding of all DetNet traffic that would
> otherwise escape the DetNet network reduces the "circuit breaker"
> discussion to the existing DetNet material measures that need to be
> taken inside a DetNet network to protect delivery of DetNet service.

As already mentioned, it is possible that some solutions don't need any "circuit breaker" as they achieve sufficient robustness by some other means. I only ask for text that explains how to avoid harm from non-DetNet networks, not for a specific solution.

Michael

> > > >>> * It is surprising that there is hardly any discussion on
> network
> > > robustness
> > > >> [ms] That does not address my concern. As I wrote already, I
> believe
> > > this
> > > > I concur with Michael's implied concern that the "should" in the
> NEW
> > > text
> > > > from causing damage elsewhere need to be mandatory particularly
> if
> > > DetNet
> > > > and needs to do so.  Deployment of filters at DetNet nodes that
> drop
> > > all
> > > > DetNet traffic that tries to escape, and that drop all
> misdirected
> > > DetNet
> > According to the last point, misconfiguration of filters/policies is
> one of the
> > challenges to look at.
> >
> > > >> From: Tsv-art [mailto:[email-masked]] On Behalf Of
> Scharf,
> > > Michael
> > > >> Subject: Re: [Tsv-art] Tsvart last call review of draft-ietf-
> detnet-
> > > architecture-08
> > > >>
> > > >>
> > > >> [EXTERNAL EMAIL]
> > > >> Cc: [email-masked]; draft-ietf-detnet-
> [email-masked];
> > > DetNet WG
> > > >> <[email-masked]>
> > > >> Please find below in-line responses and how we plan to update
> the
> > > draft to
> > > >> resolve your comments.
> > > >>
> > > >> Best regards,
> > > >> Janos
> > > >>
> > > >>
> > > >>> (draft-ietf-detnet-architecture-08) defines an overall
> framework
> > > for
> > > >> [ms] I look for an clear explanation in the architecture
> document of
> > > what "some
> > > >> by DetNet traffic, or not? As an example, when I read the
> following
> > > paragraph in
> > > >>     Transit nodes are normal MPLS Label Switching Routers
> (LSRs).
> > > They
> > > >>     although they need to provide traffic engineering services
> and
> > > proper
> > > >>     implementations of transit nodes may be DetNet aware, but
> such
> > > nodes
> > > >> ... I could maybe assume that actually there can be "transit
> nodes"
> > > that do not
> > > >> require *any* sort of DetNet support or DetNet awareness. Is
> that
> > > correct? I
> > > >> think the architecture document has to be clear on such
> fundamental
> > > questions.
> > > >>
> > > >> A goal of DetNet is a converged network in all respects. That
> is,
> > > the presence of
> > > >> DetNet flows does not preclude non-DetNet flows, and the
> benefits
> > > offered
> > > >> DetNet flows should not, except in extreme cases, prevent
> existing
> > > QoS
> > > >> required for the DetNet flows. A single source-destination pair
> can
> > > trade both
> > > >> instantiate special interfaces for DetNet flows. Networks are
> not
> > > restricted to
> > > >> certain topologies; connectivity is not restricted. Any
> application
> > > that generates
> > > >> a data flow that can be usefully characterized as having a
> maximum
> > > bandwidth
> > > >> should be able to take advantage of DetNet, as long as the
> necessary
> > > resources
> > > >> can be reserved. Reservations can be made by the application
> itself,
> > > via network
> > > >> management, by an application’s controller, or by other means,
> e.g.,
> > > a dynamic
> > > >> A goal of DetNet is a converged network in all respects. That
> is,
> > > the presence of
> > > >> DetNet flows does not preclude non-DetNet flows, and the
> benefits
> > > offered
> > > >> DetNet flows should not, except in extreme cases, prevent
> existing
> > > QoS
> > > >> required for the DetNet flows. A single source-destination pair
> can
> > > trade both
> > > >> instantiate special interfaces for DetNet flows. Networks are
> not
> > > restricted to
> > > >> certain topologies; connectivity is not restricted. Any
> application
> > > that generates
> > > >> a data flow that can be usefully characterized as having a
> maximum
> > > bandwidth
> > > >> should be able to take advantage of DetNet, as long as the
> necessary
> > > resources
> > > >> can be reserved. Reservations can be made by the application
> itself,
> > > via network
> > > >> management, by an application’s controller, or by other means,
> e.g.,
> > > a dynamic
> > > >> implement some of the DetNet capabilities (not necessarily all)
> in
> > > order to treat
> > > >> [ms] To me, the last sentence is not clear. Does "some of the
> DetNet
> > > capabilities
> > > >> (not necessarily all)" include e.g. vanilla traffic engineering?
> For
> > > instance, if a
> > > >> existing standardized TE without any additional DetNet
> awareness, is
> > > this
> > > >> DetNet-specific functionality? Specifically, please explain if
> an
> > > existing DetNet-
> > > >> unaware router on the path taken by DetNet traffic can be used
> to
> > > forward
> > > >> indeed implements new capabilities specific to DetNet. In the
> latter
> > > case, I think
> > > >> the document would need to discuss incremental deployment
> strategies
> > > as well.
> > > >>
> > > >>> * It is surprising that there is hardly any discussion on
> network
> > > robustness
> > > >>> severely and permantly congest a network and cause harm. How
> does
> > > the DetNet
> > > >>> architecture ensure that a network stays fully operational e.g.
> if
> > > the topology
> > > >>> requirements not spelt out? Section 3.3.2 speculates that
> filters
> > > and policers
> > > >>> and correctly configured (and well-behaving) devices. And
> Section
> > > 3.3.2 is
> > > >> Maintenance (OAM), which will address operational,
> misconfiguration,
> > > and
> > > >> The text  in Section 3.3.2 could be made clearer, e.g., by
> updating
> > > the
> > > >> first paragraph to:
> > > >>
> > > >> OLD:
> > > >> packet, shutting down the offending DetNet flow, or shutting
> down
> > > the
> > > >> offending interface.
> > > >>
> > > >> NEW:
> > > >> [ms] That does not address my concern. As I wrote already, I
> believe
> > > this
> > > >>> * Somewhat related, the document only looks at impact of
> failures
> > > to the QoS of
> > > >>> DetNet traffic. What is missing is a discussion how to protect
> non-
> > > DetNet parts
> > > >>> of a network from any harm caused by DetNet mechanisms.
> Solutions
> > > to this
> > > >>> probably exist. But why is the impact on non-DetNet traffic
> (e.g.,
> > > in case of
> > > >>> topology changes or failures of DetNet functions) not discussed
> at
> > > all in the document?
> > > >> Section 3.3.1 could be extended to make it clear, e.g., by
> extending
> > > the
> > > >> last paragraph:
> > > >>
> > > >> OLD:
> > > >> [ms] I believe a stronger wording is needed, e.g. along the
> lines of
> > > "a DetNet
> > > >>> * Section 4.4 refers to RFC 7426, which is an informational RFC
> on
> > > IRTF stream,
> > > >>> very confusing. First, an IETF Proposed Standard should
> probably
> > > refer to
> > > >>> other related work as well, e.g., in the TEAS WG. Second,
> Section
> > > 4.4 is by and
> > > >>> large decoupled from the rest of the document and not specific
> to
> > > DetNet.
> > > >>> applicability to DetNet. Section 4.4 even mentions explicitly
> at
> > > the end that
> > > >>> at all clear why Section 4.4 is in this document. Section 4.4
> could
> > > be removed
> > > >> RFC 7426 provides the full picture of SDN architecture, which
> needs
> > > to
> > > >> detailed picture. Therefore, we need to keep the reference to
> RFC
> > > 7426,
> > > >> should build an architecture based on research work instead of,
> say,
> > > documents
> > > >> with IETF consensus. As a side note, I believe the understanding
> in
> > > industry what
> > > >> "SDN architecture" really means and how it is implemented may
> have
> > > evolved
> > > >> instance, IRTF RFC 7426 does not discuss the implications of
> segment
> > > routing,
> > > >> BGP-LS, and other recent IETF techniques. And, based on what I
> read
> > > in the
> > > >> DetNet architecture, I believe DetNet could actually be
> implemented
> > > without
> > > >> any "SDN architecture", e.g. by distributed traffic engineering.
> As
> > > a result, I don't
> > > >> [ms] For the record: As TSV-ART reviewer, I question whether
> IRTF
> > > RFC 7426 is
> > > >> indeed an appropriate reference, I have doubts that IRTF RFC
> 7426 is
> > > really up-
> > > >> regarding the lack of alignment with the rest of the document.
> The
> > > IESG will
> > > >> certainly have a better understanding that than me how to deal
> with
> > > such topics.
> > > >>
> > > >>> The term "transport layer" has a well-defined meaning in the
> IETF,
> > > e.g.
> > > >>> originating from RFC 1122. While "transport" and e.g.
> "transport
> > > network" is
> > > >>> used in the IETF for different technologies in different areas,
> I
> > > think the
> > > >>> protocols such as TCP and UDP. As such, I personally find the
> term
> > > "DetNet
> > > >>> transport layer" misleading and confusing. The confusion is
> easy to
> > > see e.g.
> > > >>> in Figure 4, where UDP (which is a transport protocol as per
> RFC
> > > 1122) sits
> > > >>> on top of "transport".
> > > >>>
> > > >>> the "DetNet transport layer" is actually a separate protocol
> layer
> > > compared
> > > >>> to the "DetNet service layer". Thus it is not clear to me why
> the
> > > word
> > > >>> used for protocols defined in TSV area, consistent with RFC
> 1122.
> > > But this is
> > > >> "transport" is used here as in "packet transport networks" not
> as
> > > OSI L4
> > > >> transport.
> > > >>
> > > >> [ms] At the risk of stating the obvious: The relevant section of
> RFC
> > > 1122 defining
> > > >> "transport layer" is entitled by "The Internet Architecture".
> And,
> > > according e.g.
> > > >> to RFC 5921, "packet transport network" is a term defined by
> ITU-T.
> > > In an IETF
> > > >> service sub-layer" throughout the document, which would
> hopefully
> > > avoid
> > > >> confusion.
> > > >>
> > > >> expression "transport layer". To really avoid confusion, I
> believe
> > > it would be
> > > >> better to avoid the ambiguous term "transport" altogether.  But
> I
> > > will be fine if

---

From: Lou Berger <[email-masked]>
Date: Fri, 9 Nov 2018 13:56:06 +0700

Hi Michael,

First off let me thank you for all your work on this document and 
apologize for not yet responding to you previous message -- I'll get there.

> As already mentioned, it is possible that some solutions don't need any "circuit breaker" as they achieve sufficient robustness by some other means. I only ask for text that explains how to avoid h…

Can you suggest some specific text to address your concern?

Thanks,

Lou

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Nov 2018 04:56:37 +0100

> And, as noted in my earlier way, depending on how the PRF is realized DetNet may even shield DetNet traffic from packet losses due to congestion, i.e., even traffic that would be able to react to c…

I don't think we should complicate the matters by talking about DetNet
flows that may react to congestion. I don't think this is in scope of
the current architecture.

AFAIK, all DetNet flows have a known envelope, and DetNet flows should
be shutdown (filtered/blocked) by the DetNet service whenever they exceed
the envelope (*).

Within the envelope of DetNet flows, there is no useful support
for congestion loss, because congestion constitutes a misconfiguration
or misbehavior of the DetNet service. 

PEF/POF could actually act as a circuit breaker using the maximum
permissible loss of the envelope as a trigger to make the PRF stop
the flow. That would be a good feature for the PREOF.

Cheers
    Toerless

(*) I am not sure if DetNet intends to make such a strong
statement, but from what i heard, this is usually done in TSN
applications:

If you have an interlocked system and a sender exceeds its
envelope, that indicates a malfunctioning sender, and there is
no way to design a safe&secure system with randomnly
malfunctioning components. Shutting down a component by
stopping its traffic is a supported system behavior resulting
typically in bringing up redundant components or shutting down
parts of the system. Unknown component behavior results in
unpredictable system behavior, often catastrophic.

> > [email-masked]
> > > > [email-masked]
> > > > [email-masked]
> > > > >> management, by an application???s controller, or by other means,
> > e.g.,
> > > > a dynamic
> > > > >> management, by an application???s controller, or by other means,
> > e.g.,
> > > > a dynamic

---

From: "Scharf, Michael" <[email-masked]>
Date: Thu, 15 Nov 2018 13:18:09 +0000

My comments are about the question what happens if DetNet packets get dropped despite that this should _never_ happen in normal operation. There can be different reasons for packet drops in a managed network. One root cause would be that the DetNet traffic gets routed erroneously over a DetNet unaware node somewhere in the Internet and then causes harm there.

For inelastic traffic, if such packet loss persists, that flow (at least on the path where losses occur) may need to be blocked for safety reasons. I assume that is what you suggest in your last sentence?

Michael

-----Original Message-----
From: Toerless Eckert [mailto:[email-masked]] 
Sent: Thursday, November 15, 2018 4:57 AM
To: Scharf, Michael
Cc: Black, David; Lou Berger; 'J?nos Farkas'; [email-masked]; DetNet WG; [email-masked]
Subject: Re: [Detnet] [Tsv-art] Tsvart last call review of draft-ietf-detnet-architecture-08

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Nov 2018 23:53:58 +0100

> My comments are about the question what happens if DetNet packets get dropped despite that this should _never_ happen in normal operation. There can be different reasons for packet drops in a manag…

Understood. 

> For inelastic traffic, if such packet loss persists, that flow (at least on the path where losses occur) may need to be blocked for safety reasons. I assume that is what you suggest in your last se…

Yes. Of course, i can not speak for the authors, and i don't think so
far this type of detail for the  PREOF functions have been required,
but i certainly think it would make the PEF/POF a lot more useful if it
would include diagnostics about packet loss - and that could then be
used to trigger some control/management-plane circuit breaker. 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 16 Nov 2018 15:08:47 +0000

> > My comments are about the question what happens if DetNet packets get
> dropped despite that this should _never_ happen in normal operation.
> There can be different reasons for packet drops in a managed network.
> One root cause would be that the DetNet traffic gets routed erroneously
> over a DetNet unaware node somewhere in the Internet and then causes
> harm there.
> 
> Understood.
> 
> > For inelastic traffic, if such packet loss persists, that flow (at
> least on the path where losses occur) may need to be blocked for safety
> reasons. I assume that is what you suggest in your last sentence?
> 

Thanks a lot for the clarification. Such a safety mechanism would address a lot of my concerns.

Michael

> > Cc: Black, David; Lou Berger; 'J?nos Farkas'; [email-masked];
> DetNet WG; [email-masked]
> > > And, as noted in my earlier way, depending on how the PRF is
> realized DetNet may even shield DetNet traffic from packet losses due
> to congestion, i.e., even traffic that would be able to react to
> congestion may not react due to DetNet. If DetNet is able to mask
> congestion from the carried traffic, special care is needed inside
> DetNet.
> >
> > > > That quoted comment is followed by a "reduce to previous case"
> argument
> > > > about networking in general:
> > > >
> > > > > > WRT escaped traffic, as I mentioned in today's session, I
> really
> > > > think
> > > > 	Networking (DetNet), which provides a capability for the
> delivery
> > > > of
> > > > 	data flows with extremely low packet loss rates and bounded
> end-
> > > > to-
> > > > 	that are under a single administrative control or within a
> closed
> > > > group
> > > > 	private WANs. DetNet is not for large groups of domains
> such as
> > > > the
> > > > 	Internet.
> > > >
> > > > 	failures. Filters and policers should be used in a DetNet
> network
> > > > to
> > > > 	detect if DetNet packets are received on the wrong
> interface, or
> > > > at
> > > > 	policers can take action to discard offending packets or
> flows,
> > > > or
> > > As already mentioned, it is possible that some solutions don't need
> any "circuit breaker" as they achieve sufficient robustness by some
> other means. I only ask for text that explains how to avoid harm from
> non-DetNet networks, not for a specific solution.
> > >
> > > Michael
> > >
> > >
> > > >
> > > > Thanks, --David
> > > >
> > > >
> > > > > Subject: RE: [Tsv-art] [Detnet] Tsvart last call review of
> draft-
> > > > ietf-detnet-
> > > > > architecture-08
> > > > >
> > > > >
> > > > > [EXTERNAL EMAIL]
> > > > > Please report any suspicious attachments, links, or requests
> for
> > > > sensitive
> > > > > information.
> > > > >
> > > > >
> > > > > Inline
> > > > >
> > > > Berger
> > > > > > Subject: Re: [Tsv-art] [Detnet] Tsvart last call review of
> draft-
> > > > ietf-
> > > > > > detnet-architecture-08
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > >> detect if DetNet packets are received on the wrong
> interface, or
> > > > at
> > > > > > >> policers can take action to discard offending packets or
> flows,
> > > > or
> > > > > > >> trigger shutting down the offending DetNet flow, or
> shutting
> > > > down
> > > > > > >> the offending interface.
> > > > > > >>
> > > > > > >> document has to discuss the applicability of circuit
> breakers
> > > > > > according to RFC
> > > > > > >> 8084.
> > > > > > > is woefully inadequate.  The provisions to prevent escaped
> non-
> > > > > > elastic traffic
> > > > > > > text on filters doesn't appear to cover escape from the
> DetNet
> > > > > > network,
> > > > > > > Once that is done, I think a discussion of circuit breakers
> is
> > > > > > needed, in the
> > > > > > > of) a flow that is being sent entirely into the bit-bucket
> by a
> > > > > > downstream filter,
> > > > > > DetNet does not itself impact any Transport protocol  (e.g.,
> UDP,
> > > > TCP)
> > > > > First, the question whether DetNet *produces* elastic or non-
> elastic
> > > > traffic does
> > > > > not matter for the applicability of circuit breakers. Section 3
> of
> > > > RFC 8084 states:
> > > > >
> > > > >    other protocols and tunnel encapsulations also ought to
> consider
> > > > the
> > > > > So, clearly the condition of carrying non-congestion-controlled
> flows
> > > > or traffic
> > > > > RECOMMENDED in RFC 8084 is in principle applicable to DetNet.
> Also,
> > > > note that
> > > > > As pointed out by David, the key question is whether, and how,
> the
> > > > DetNet
> > > > > architecture prevents that non-congestion controlled traffic
> escapes
> > > > from a
> > > > > controlled environment into the Internet. As also explained in
> RFC
> > > > 8084, this
> > > > > As TSV-ART reviewer, I insist in a discussion of this problem
> in
> > > > draft-ietf-detnet-
> > > > > architecture. And I believe that a reference to RFC 8084 is
> needed in
> > > > draft-ietf-
> > > > > detnet-architecture.
> > > > >
> > > > > Note that the wording in RFC 8084 is RECOMMENDED and not MUST,
> i.e.,
> > > > it is
> > > > > Second, in my TSV-ART review I have explicitly mentioned the
> DetNet
> > > > Packet
> > > > > challenges if the replicated packets can somehow escape into
> the
> > > > Internet, e.g.,
> > > > > in failure cases. For instance, as far as I understand draft-
> ietf-
> > > > detnet-
> > > > > and forward it along different paths. If one of these paths
> escapes
> > > > to the
> > > > > Internet, to the Internet the PRF could basically be a producer
> of
> > > > non-congestion
> > > > > controlled traffic, and that traffic could cause significant
> harm in
> > > > the Internet.
> > > > > DetNet is elastic, e.g., if DetNet carries TCP traffic. For
> instance,
> > > > if other paths
> > > > > used by the PRF within the DetNet domain would deliver all
> packets,
> > > > the
> > > > > congestion control in TCP would not never be triggered as each
> packet
> > > > gets
> > > > > indeed delivered over one of the paths. So, an unelastic
> traffic
> > > > stream would
> > > > > escale into the Internet and never back off. Unless I miss
> something,
> > > > DetNet
> > > > > As TSV-ART reviewer, I believe draft-ietf-detnet-architecture
> needs a
> > > > dedicated
> > > > > discussion whether the PRF can cause harm and, if so, how to
> avoid
> > > > such harm.
> > > > >
> > > > > > DetNet operates at Layer 3 and basically provides detnet
> flows with
> > > > a
> > > > > > service equivalent to IntServ's Guaranteed Quality of
> Service.  As
> > > > > > such,
> > > > > > WRT escaped traffic, as I mentioned in today's session, I
> really
> > > > think
> > > > >    o  anomalous traffic that exceeds the provisioned capacity
> (or
> > > > whose
> > > > >    o  routing changes that cause additional traffic to start
> using
> > > > the
> > > > > RFC 8084 represents IETF community consensus. And in my reading
> the
> > > > DetNet
> > > > > > >> Please report any suspicious attachments, links, or
> requests for
> > > > > > sensitive
> > > > > > >> information.
> > > > > > >>
> > > > > > >>
> > > > > > >> Hi Janos,
> > > > > > >>
> > > > > > >> Actually, some form of DetNet support is required from
> each node
> > > > > > that
> > > > > > >> transports DetNet traffic. For instance, DetNet flows have
> to be
> > > > > > recognized in
> > > > > > >> form of DetNet support" exactly means. For instance, is
> standard
> > > > > > MPLS TE
> > > > > > >> without any DetNet support good enough for an LSR on the
> IP/MPLS
> > > > > > path used
> > > > > > >>     are generally unaware of the special requirements of
> DetNet
> > > > > > flows,
> > > > > > >>     QoS to the LSPs associated with DetNet flows to
> enhance the
> > > > > > prospect
> > > > > > >> The Introduction could be extended to clarify, e.g., the
> third
> > > > > > paragraph:
> > > > > > >>
> > > > > > >> OLD:
> > > > > > >> mechanisms from operating in a normal fashion, subject to
> the
> > > > > > bandwidth
> > > > > > >> DetNet and non-DetNet flows. End systems and applications
> need
> > > > not
> > > > > > >> mechanisms from operating in a normal fashion, subject to
> the
> > > > > > bandwidth
> > > > > > >> DetNet and non-DetNet flows. End systems and applications
> need
> > > > not
> > > > > > >> control plane (e.g., [RFC2205]). Network nodes supporting
> DetNet
> > > > > > flows have to
> > > > > > >> network node can ensure that the QoS requirements are met
> by
> > > > > > leveraging
> > > > > > >> enough for deploying DetNet? Or does this node have to
> implement
> > > > > > additional
> > > > > > >> DetNet traffic, or if it is mandatory that any router
> forwarding
> > > > > > DetNet traffic
> > > > > > >>> changes or there are equipment failures? Probably this
> can be
> > > > > > solved by
> > > > > > >>> implementations (e.g., dynamic control plane), but why
> are
> > > > > > corresponding
> > > > > > >>> can help, and that may be true, but that probably still
> assumes
> > > > > > consistently
> > > > > > >>> vague and mentions a "infinite variety of possible
> failures"
> > > > > > without stating
> > > > > > >> Security issues and considerations are addressed by the
> DetNet
> > > > > > Security
> > > > > > >> There will be a document dedicated to Operations,
> Administration
> > > > and
> > > > > > >> The topology changes have to be solved by the control
> plane,
> > > > either
> > > > > > >> analyzed with reasonable confidence. DetNet aids in the
> process
> > > > by
> > > > > > >> detect if DetNet packets are received on the wrong
> interface, or
> > > > at
> > > > > > >> policers can take action to discard offending packets or
> flows,
> > > > or
> > > > > > >> trigger shutting down the offending DetNet flow, or
> shutting
> > > > down
> > > > > > >> the offending interface.
> > > > > > >>
> > > > > > >> document has to discuss the applicability of circuit
> breakers
> > > > > > according to RFC
> > > > > > >> 8084.
> > > > > > >>
> > > > > > >> Ideally, the net effect of the presence of DetNet flows in
> a
> > > > network
> > > > > > >> Traffic policing functions (e.g. [RFC2475]) are used to
> avoid
> > > > the
> > > > > > >> starvation of non-DetNet traffic. Thus, the net effect of
> the
> > > > > > presence
> > > > > > >>> and the document uses the concepts introduced there
> (e.g.,
> > > > > > "planes"). This is
> > > > > > >>> documents having IETF consensus. An example would be RFC
> 7491,
> > > > > > albeit there is
> > > > > > >>> Neither do other sections of the document refer to the
> concepts
> > > > > > introduced in
> > > > > > >>> Section 4.4, nor does Section 4.4 use the DetNet
> terminology or
> > > > > > discuss
> > > > > > >> be referred from the DetNet architecture. No other RFC
> provides
> > > > such
> > > > > > >> [ms] I'll leave that up to the IESG to decide whether an
> IETF
> > > > > > proposed standard
> > > > > > >> since publication of IRTF RFC 7426, which was published in
> Jan.
> > > > > > 2015. For
> > > > > > >> to-date, and I am concerned about the content quality of
> section
> > > > > > 4.4, e.g.,
> > > > > > >>> term "transport layer" is typically understood to refer
> to
> > > > > > transport
> > > > > > >>> Based on the document it also may be
> solution/implementation
> > > > > > specific
> > > > > > >> whether
> > > > > > >>> "layer" has to be used, specifically in combination
> "transport
> > > > > > layer".
> > > > > > >>>
> > > > > > >>> To me as, the word "transport layer" (and "transport
> protocol")
> > > > > > should be
> > > > > > >> I suggest to use the terms "DetNet transport sub-layer"
> and
> > > > "DetNet
> > > > > > >> [ms] "DetNet transport sub-layer" somewhat solves my
> concern of
> > > > > > avoiding the
> > > > > > >> all DetNet documents consistently avoid the term
> "transport
> > > > layer".
> > > > > > >>
> > > > > > >> Michael

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Nov 2018 20:23:52 +0100

Why does DetNet over IP not simply mandate proper use of DiffServ ?

IMHO, all inelastic IP traffic MUST use DSCP appropriate for inelastic
traffic (DetNet or not). If i then have a malfunctioning originator
of inelastic traffic, say DetNet Network A, and it leaks the traffic to
Internet provider B, then B can simply filter out that traffic by DSCP.

Don't remember seeing this mentioned in the arch spec.

If an endpoint is originating inelastic traffic with DSCP 0, thats not
DetNet. That's a broken endpoint.

Cheers
    Toerless

On Fri, Nov 16, 2018 at 03:08:47PM +0000, Scharf, Michael wrote:
> > shutting down
> > > > > network to
> > > > > network to

-- 
---
[email-masked]