Subject: Re: [Detnet] Tsvart last call review of draft-ietf-detnet-architecture-08

From: Lou Berger <[email-masked]>
Date: Mon, 19 Nov 2018 10:34:58 -0500

Michael,

I wanted to take a step back from the multiple discussions that were 
spawned by your review -- from a doc shepherd perspective, and see where 
we are.   I know that the authors have sent a -09 version that addresses 
some, but not all issues.

 From the exchanges I've seen, I think the key remaining issues are 
related to:

(a) possibly introduction of congestion in the general internet if 
packets were somehow to escape a detnet domain.  The source of this 
congestion would be inelastic traffic using DetNet or due to congestion 
loss that is masked by PREOF.

(b) The use of the term 'transport' in DetNet to refer to what is 
basically a Traffic Engineered sub-network layer, such as is provided 
with MPLS-TE or Optical Transport Networks.

Do you have any other issues that that are critical to be addressed 
before this work moves forward?  If so which?

Thank you,

Lou

On 9/28/2018 6:24 PM, Michael Scharf wrote:
> Reviewer: Michael Scharf
> Review result: Ready with Issues
>
> The document "Deterministic Networking Architecture"
> (draft-ietf-detnet-architecture-08) defines an overall framework for
> Deterministic Networking.
>
> As TSV-ART reviewer, I believe that this document has issues as detailed below.
>
> Michael
>
> Major issues:
>
> * It seems that DetNet cannot easily be deployed in the Internet without
> additional means. Thus, for a baseline document, one could expect some
> explanation on the requirements of deploying DetNet in a network. DetNet
> basically requires support in (almost) all network devices transporting DetNet
> traffic. That assumption should be explicitly spelt out early in the document,
> e.g., in the introduction. There also needs to be an explicit discussion of the
> implications if not the whole network is aware of or supports DetNet. There is
> some text in Section 4.2.2 and Section 4.3.3, but I believe additional explicit
> discussion is needed at a prominant place. For instance, can use of DetNet do
> harm to parts of a network not supporting DetNet? As a side note, when TCPM
> published RFC 8257, the following disclaimer was added: "DCTCP, as described in
> this specification, is applicable to deployments in controlled environments
> like data centers, but it must not be deployed over the public Internet without
> additional measures." I wonder if a similar disclaimer is needed for DetNet. If
> there is an implicit assumption that DetNet will  be used in homogenous
> environments with mostly DetNet-aware devices within the same organization,
> such an assumption should be made explicit.
>
> * It is surprising that there is hardly any discussion on network robustness
> and safety; this probably also relates to security. For instance,
> misconfiguration or errors of functions performing packet replication could
> severely and permantly congest a network and cause harm. How does the DetNet
> architecture ensure that a network stays fully operational e.g. if the topology
> changes or there are equipment failures? Probably this can be solved by
> implementations (e.g., dynamic control plane), but why are corresponding
> requirements not spelt out? Section 3.3.2 speculates that filters and policers
> can help, and that may be true, but that probably still assumes consistently
> and correctly configured (and well-behaving) devices. And Section 3.3.2 is
> vague and mentions a "infinite variety of possible failures" without stating
> any requirements or recommendations. There may be further solutions, such as
> circuit breakers and the like. Why are such topics not discussed?
>
> * Somewhat related, the document only looks at impact of failures to the QoS of
> DetNet traffic. What is missing is a discussion how to protect non-DetNet parts
> of a network from any harm caused by DetNet mechanisms. Solutions to this
> probably exist. But why is the impact on non-DetNet traffic (e.g., in case of
> topology changes or failures of DetNet functions) not discussed at all in the
> document?
>
> * Regarding security, an architecture like DetNet probably requires that only
> authenticated and authorized end systems have access to the data plane. The
> security considerations only briefly mention the control aspect ("the
> authentication and authorization of the controlling systems").
>
> * For an architecture document, the lack of clarity and consistency regarding
> terminology is concerning. This specifically applies to the case of incomplete
> networks (as per Section 4.2.2 and 4.3.3) that include "DetNet-unaware nodes".
> The document introduces terms such as "DetNet intermediate nodes" but then
> repeatedly uses generic terms such as "node" or "hop" that may include
> DetNet-unaware nodes. For instance, for incomplete networks, a sentence such as
> "The primary means by which DetNet achieves its QoS assurances is to reduce, or
> even completely eliminate, congestion within a node as a cause of packet loss"
> seems to only apply to "DetNet transit nodes" but not "DetNet-unaware nodes".
> Similar ambiguity exist for other use of the terms "hop" and "node", which may
> or may not include DetNet-unaware nodes. It is unclear why the document does
> not consistently use the terminology introduced in Section 2.1 in all sections
> and clearly distinguishes cases with and without DetNet support.
>
> * Section 4.4 refers to RFC 7426, which is an informational RFC on IRTF stream,
> and the document uses the concepts introduced there (e.g., "planes"). This is
> very confusing. First, an IETF Proposed Standard should probably refer to
> documents having IETF consensus. An example would be RFC 7491, albeit there is
> other related work as well, e.g., in the TEAS WG. Second, Section 4.4 is by and
> large decoupled from the rest of the document and not specific to DetNet.
> Neither do other sections of the document refer to the concepts introduced in
> Section 4.4, nor does Section 4.4 use the DetNet terminology or discuss
> applicability to DetNet. Section 4.4 even mentions explicitly at the end that
> it discusses aspects that are orthogonal to the DetNet architecture. It is not
> at all clear why Section 4.4 is in this document. Section 4.4 could be removed
> from the document without impacting the rest of the document.
>
> Minor issues:
>
> * Terminology "DetNet transport layer"
>
>    The term "transport layer" has a well-defined meaning in the IETF, e.g.
>    originating from RFC 1122. While "transport" and e.g. "transport network" is
>    used in the IETF for different technologies in different areas, I think the
>    term "transport layer" is typically understood to refer to transport
>    protocols such as TCP and UDP. As such, I personally find the term "DetNet
>    transport layer" misleading and confusing. The confusion is easy to see e.g.
>    in Figure 4, where UDP (which is a transport protocol as per RFC 1122) sits
>    on top of "transport".
>
>    Based on the document it also may be solution/implementation specific whether
>    the "DetNet transport layer" is actually a separate protocol layer compared
>    to the "DetNet service layer". Thus it is not clear to me why the word
>    "layer" has to be used, specifically in combination "transport layer".
>
>    To me as, the word "transport layer" (and "transport protocol") should be
>    used for protocols defined in TSV area, consistent with RFC 1122. But this is
>    probably a question to be sorted out by the IESG.
>
> * Page 9
>
>     A DetNet node may have other resources requiring allocation and/or
>     scheduling,
>
>    This is just one of several examples for inconsistent use of terminology.
>    What is a "DetNet node"? That term is not introduced in Section 2.1
>
> * Page 14
>
>     A DetNet network supports the dedication of a high proportion (e.g.
>     75%) of the network bandwidth to DetNet flows.
>
>    The 75% value is not reasoned. What prevents using 99% of the bandwidth for
>    DetNet traffic?
>
> * Page 15: Figure 2
>
>    If the term "transport layer" cannot be avoided, the labels in this figure
>    should at least be expanded to "DetNet transport layer".
>
> * Page 18: Figure 4
>
>    As already mentioned earlier, Figure 4 is confusing. UDP is a transport
>    protocol. If the term "transport" cannot be avoided, the labels in this
>    figure should at least be expanded to "DetNet transport".
>
> * Page 23
>
>     If the source transmits less data than this limit
>     allows, the unused resource such as link bandwidth can be made
>     available by the system to non-DetNet packets.
>
>    Could there be additional requirements on the use of unused resources by
>    non-DetNet packets, e.g., regarding preemption? I am just wondering... If
>    that was possible, a statement like "... can be made available by the system
>    to non-DetNet packets as long as all guarantees are fulfilled" would be on
>    the safe side, no?
>
> * Page 27:
>
>     DetNet achieves congestion protection and bounded delivery latency by
>     reserving bandwidth and buffer resources at every hop along the path
>     of the DetNet flow.
>
>    Why does this sentence use the word "hop"? As far as I understand, in DetNet
>    bandwidth and buffer resources are reserved in each DetNet intermediate node.
>    If there were hops over IP routers not being DetNet intermediate nodes, no
>    resources would be reserved there. As per Section 4.3.3, it is possible to
>    deploy DetNet this way. And obviously there can be resource bottlenecks below
>    IP, on devices that are not routers... So does "hop" here refer to IP router
>    hops or also to devices not processing IP (or IP/MPLS)?
>
> * Page 27:
>
>     Standard queuing and transmission selection algorithms allow a
>     central controller to compute the latency contribution of each
>     transit node to the end-to-end latency, ...
>
>    The text does not explain why a _central_ controller is needed for this
>    computation. Why would a distributed control plane not be able to realize
>    this computation. Isn't this implementation-specific?
>
> * Page 32
>
>    To somebody who is not deeply familiar with DetNet, it is impossible to parse
>    the description of the examples in Section 4.7.3. For instance, "VID +
>    multicast MAC address" is not introduced. I think this example must be
>    expaned with additional context and explanation to be useful to readers.
>
> * Page 34
>
>     There are three classes of information that a central controller or
>     distributed control plane needs to know that can only be obtained
>     from the end systems and/or nodes in the network.
>
>    Wouldn't it be sufficient to state "Provisioning of DetNet requires knowledge
>    about ...". Does it matter in this context whether the provisioning is done
>    by a central controller or a distributed control plane? For instance, could
>    the same paragraph also apply to a network that uses _multiple_ central
>    controllers, or hybrid combinations of central controllers and distributed
>    control planes? In general, an architecture document should be agnostic to
>    implementation aspects unless there is a specific need. In this specific
>    case, I fail to see a need to discuss the realization of the control plane of
>    a network.
>
> Editorial nits:
>
> * Page 9:
>
>     The low-level mechanisms described in Section 4.5 provide the
>     necessary regulation of transmissions by an end system or
>     intermediate node to provide congestion protection.  The allocation
>     of the bandwidth and buffers for a DetNet flow requires provisioning
>     A DetNet node may have other resources requiring allocation and/or
>     scheduling, that might otherwise be over-subscribed and trigger the
>     rejection of a reservation.
>
>    Probably a full stop is missing after "provisioning".
>
> * Page 11: "... along separate (disjoint non-SRLG) paths ..."
>
>    I find this confusing. I would understand e.g. "along separate
>    (SRLG-disjoint) paths".
>
> * Page 34:
>
>     When using a peer-
>     to-peer control plane, some of this information may be required by a
>     system's neighbors in the network.
>
>    Would "acquired" be a better term?
>
> * Page 34:
>
>     o  The identity of the system's neighbors, and the characteristics of
>        the link(s) between the systems, including the length (in
>        nanoseconds) of the link(s).
>
>    "Latency" or "delay" would probably be a better terms if the value is
>    measured in nanoseconds.
>
> * Page 35:
>
>     DetNet is provides a Quality of Service (QoS), and as such, does not
>     directly raise any new privacy considerations.
>
>    Broken sentence
>
> * Please expand acronyms on first use (e.g., OTN)
>
>

---

From: "Scharf, Michael" <[email-masked]>
Date: Mon, 19 Nov 2018 22:15:22 +0000

Lou,

> -- 

These are two major issues that need to be addressed. Note that it may not be sufficient just to add a section on operational and deployment considerations. Also the existing text in the document will need to get aligned to normative guidance on how to avoid a congestion collapse.

In -09, one example would be Section 3.1. "Primary goals defining the DetNet QoS"

   Congestion protection operates by allocating resources along the path
   of a DetNet flow, e.g., buffer space or link bandwidth.  Congestion
   protection greatly reduces, or even eliminates entirely, packet loss
   due to output packet congestion within the network, but it can only
   be supplied to a DetNet flow that is limited at the source to a
   maximum packet size and transmission rate.  Note that congestion
   protection provided via congestion detection and notification is
   explicitly excluded from consideration in DetNet, as it serves a
   different set of applications.

At least the last sentence would contradict a better discussion of congestion in the document. For instance, it could just be removed. In any case, the current wording in the last sentence is not correct, as the IETF term for what is described in the last sentence is "congestion control".

Another example would be  Section 3.2.1.1. "Eliminate congestion loss"
 
   The primary means by which DetNet achieves its QoS assurances is to
   reduce, or even completely eliminate, congestion within a DetNet node
   as a cause of packet loss.  This can be achieved only by the
   provision of sufficient buffer storage at each node through the
   network to ensure that no packets are dropped due to a lack of buffer
   storage.  Note that a DetNet flow cannot be throttled, i.e., its
   transmission rate cannot be reduced via explicit congestion
   notification.

This section IMHO has to include a discussion of what happens in the (not expected) case that packets get dropped or that ECN marks are received. It is understood that this would not happen in normal operation of a DetNet network, but I believe just considering the error-free operation of a DetNet network is not sufficient for this document. At least for the risk of traffic that may escape from a DetNet network is inherently not sufficient to assume that the DetNet network is always error-free.

As a result, addressing my concerns will most likely require editing several parts of the document.

In addition, I'd like to emphasize that my review comment "It is surprising that there is hardly any discussion on network robustness and safety" covers more than just inelastic traffic that escapes from a DetNet network and masking of packet loss. Given that DetNet traffic may be extremely critical traffic, I really wonder why the document doesn't emphasize more the required robustness against failures *inside* the DetNet network as well as counter-measures. But this is something the WG needs to decide. As TSV-ART reviewer, I will be fine if the document clearly describes how the impact of failures will be isolated inside the DetNet network and will not put the general Internet at risk.

> with MPLS-TE or Optical Transport Networks.

In the Internet architecture, the term 'transport' refers to Internet transport protocols. I doubt that the document can avoid discussing the implications of and interactions with Internet transport protocols such as UDP or TCP. As a result, I disagree that the document can use the term 'transport' to refer to traffic engineered sub-network layers.

From a TSV-ART point of view, the document can either only use the term "transport" for Internet transport protocols and use another term for sub-network layers (as handled in the *routing* area of the IETF), or the document has to clearly distinguish between the Internet transport layer and other uses of the term "transport" and explain the overlap. I believe the former would be less confusing, but I will leave it up to the TSV ADs to discuss terminology overlap in the IESG. As TSV-ART reviewer I insist that the document uses the terms "transport layer" and "transport protocol" only when referring to the Internet transport layer.

> before this work moves forward?  If so which?

Regarding Section 4.4 I have already deferred the discussion to the IESG. The TSV-ART review comment is that the IESG needs to carefully look at the concepts, terminology, and references in section 4.4.

Regarding my other comments, I acknowledge that -09 is a step forward. But given the cross-dependencies e.g. regarding terminology and definitions, I will need to read the text completely once there is a proposal how to address my review. As noted in my review, I believe the document must use terminology clearly and consistently. As example, a statement in -09 such as "Network nodes supporting DetNet flows have to implement some of the DetNet capabilities (not necessarily all) in order to treat DetNet flows such that their QoS requirements are met" is IMHO too vague. But in such cases it depends whether there is precise normative guidance elsewhere. And this requires looking at the text as a whole.

Best regards

Michael

---

From: Lou Berger <[email-masked]>
Date: Tue, 20 Nov 2018 07:11:10 -0500

Michael,

I think we're getting somewhere and identifying where we have 
disconnects and what may (and what may not) need to change in the 
document.  My takeaways are:

- The document needs a good 'scrub' of the congestion related references 
to ensure that the document only makes statements on what is actually 
done within a DetNet and the relationship with transport protocols that 
use detnet (which are in fact outside the scope of the DetNet WG).  I'll 
work with the authors and WG on this -- I see this change as important, 
but editorial in nature.

- We have a perception issue with at least one member of the TSV area on 
the meaning and more importantly, implication, of the term "DetNet 
*Transport* sub-layer".  While I don't disagree that a good portion of 
the IETF thinks transport protocol (UDP/TCP) when they hear "transport" 
there are plenty others, particularly in the routing area, who 
understand that "transport" can refer to Transport Networks.  And 
Transport Network is a well understood general industry term. The IETF 
even has a bunch of RFCs that relate to Transport networks.  This said, 
I think it reasonable to go back to the DetNet WG and discuss changing 
the name of the "DetNet Transport sub-layer"  to avoid the word 
"transport".  -- BTW we made a parallel change in the TEAS WG when 
producing RFC8453.

See below for detail response in-line.

> This section IMHO has to include a discussion of what happens in the (not expected) case that packets get dropped or that ECN marks are received. It is understood that this would not happen in norm…

I think these are examples of text that needs to be cleanup up and to 
delineate what is done with a DetNet.

> In addition, I'd like to emphasize that my review comment "It is surprising that there is hardly any discussion on network robustness and safety"

I have no idea what you mean by safety here.  Can you elaborate.

> covers more than just inelastic traffic that escapes from a DetNet network and masking of packet loss. Given that DetNet traffic may be extremely critical traffic, I really wonder why the document doesn't emphasize more the required robustness against failures *inside* the DetNet network as well as counter-measures. But this is something the WG needs to decide. As TSV-ART reviewer, I will be fine if the document clearly describes how the impact of failures will be isolated inside the DetNet network and will not put the general Internet at risk.

I agree - I think, the document should be clear on it's scope and 
relationship to general internet usage.

> In the Internet architecture, the term 'transport' refers to Internet transport protocols. I doubt that the document can avoid discussing the implications of and interactions with Internet transpor…

I think this is covered by my comment above.

>  From a TSV-ART point of view, the document can either only use the term "transport" for Internet transport protocols and use another term for sub-network layers (as handled in the *routing* area o…

I'm personally okay with a name change and even willing to push this 
discussion within the WG, but as said above, "Transport Network" is a 
generally understood industry term that is also used in RFCs -- so we'll 
have to see what where WG consensus ends up.

> Regarding my other comments, I acknowledge that -09 is a step forward. But given the cross-dependencies e.g. regarding terminology and definitions, I will need to read the text completely once ther…

I think the next steps lie with me and the WG.  We'll let you know once 
there is a new version.  Of course, you can also contribute to the WG 
discussion on the topic.

Thanks,

Lou

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

---

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 20 Nov 2018 15:02:28 +0000

Lou,

Just regarding my comment that seems unclear...

[snip]

>I have no idea what you mean by safety here.  Can you elaborate.

As far as I understand, DetNet introduces network functions that at least partly novel, such as PREOF.  And as PREOF explicitly allows solutions beyond 1+1 protection ("network coding" is mentioned as example). I believe that any new function typically raises operational issues.

Out-of-my-head, here are some examples of potential challenges that seem not well addressed in -09:

- PREOF (and specifically) PEF as a single point of failure: Is there any need to worry about that? As far as I can see, the document discusses failures of paths between POF and PEF, but not failures of the functions themselves. Wouldn't it make sense to discuss basic requirements such as implementing these functions redundantly etc.? And if they are implemented redundantly, does this impact the architecture, e.g., result in additional interfaces that may need standardization? 

- Overload of DetNet components: As another example for PREOF, how would a DetNet deployment ensure that PEF never gets overloaded? Buffering and resequencing of packets requires processing. Techniques to allocate buffer space alone may not be enough to ensure that PEF never gets overloaded. Is this a problem or not? And even if this problem may perhaps not really exist for 1+1, is there no need to worry about other, possibly more complex PREOF schemes that may require complex packet processing (or even coding)? Are there scaling issues in PREOF? And could this impact signaling, e.g., require control plane protocol extensions?

More generally, if the intention of DetNet is to carry mission-critical or safety-critical traffic that cannot tolerate any packet loss, let alone any downtime, wouldn't words such as "reliability", "dependability", "redundancy", "high availability", etc. be relevant in a description of the architecture? (Some of these terms are actually used in this document, mostly in references to non-IETF work.)

And this may not only matter for the data plane. Assuming that a lot of DetNet traffic requires extremely high reliability, are there architectural requirements on components outside the data plane, e.g., regarding redundancy? Obviously, control plane and management plane functions are often built today already as high availability solutions, but if the intention is to _never_ fail, isn't there a risk that DetNet may require solutions "better-than-current-IP" outside the data plane as well?

This list of topics may not be comprehensive. I just want to illustrate some examples for what could also matter when running a network _really_ deterministically, beyond the well-known congestion issue. 

However, just to be precise: I don't ask for additional text to address what is written in this e-mail.

Michael

---

From: "Black, David" <[email-masked]>
Date: Tue, 20 Nov 2018 16:19:38 +0000

> "transport".  -- BTW we made a parallel change in the TEAS WG when

Well, "Transport Network" is not ambiguous, so another option would be to
change the name of the "DetNet Transport sub-layer"  to "DetNet Transport
Network sub-layer" and also change all uses of "transport to "transport
network" or "L4 transport protocol" or something else along one of those
two lines.  The unqualified use of the word "transport" is the source of
confusion here.

Thanks, --David

> -----Original Message-----
> From: Tsv-art [mailto:[email-masked]] On Behalf Of Lou Berger
> Sent: Tuesday, November 20, 2018 7:11 AM
> To: Scharf, Michael
> Cc: [email-masked]; [email-masked]; draft-ietf-detnet-
> [email-masked]
> Subject: Re: [Tsv-art] [Detnet] Tsvart last call review of draft-ietf-detnet-
> architecture-08
> 
> 
> [EXTERNAL EMAIL]
> 
> Michael,
> 
> > These are two major issues that need to be addressed. Note that it may not be
> sufficient just to add a section on operational and deployment considerations.
> Also the existing text in the document will need to get aligned to normative
> guidance on how to avoid a congestion collapse.
> >
> > At least the last sentence would contradict a better discussion of congestion in
> the document. For instance, it could just be removed. In any case, the current
> wording in the last sentence is not correct, as the IETF term for what is
> described in the last sentence is "congestion control".
> >
> > This section IMHO has to include a discussion of what happens in the (not
> expected) case that packets get dropped or that ECN marks are received. It is
> understood that this would not happen in normal operation of a DetNet
> network, but I believe just considering the error-free operation of a DetNet
> network is not sufficient for this document. At least for the risk of traffic that
> may escape from a DetNet network is inherently not sufficient to assume that
> the DetNet network is always error-free.
> 
> > In addition, I'd like to emphasize that my review comment "It is surprising that
> there is hardly any discussion on network robustness and safety"
> 
> > covers more than just inelastic traffic that escapes from a DetNet network and
> masking of packet loss. Given that DetNet traffic may be extremely critical
> traffic, I really wonder why the document doesn't emphasize more the required
> robustness against failures *inside* the DetNet network as well as counter-
> measures. But this is something the WG needs to decide. As TSV-ART reviewer, I
> will be fine if the document clearly describes how the impact of failures will be
> isolated inside the DetNet network and will not put the general Internet at risk.
> 
> > In the Internet architecture, the term 'transport' refers to Internet transport
> protocols. I doubt that the document can avoid discussing the implications of
> and interactions with Internet transport protocols such as UDP or TCP. As a
> result, I disagree that the document can use the term 'transport' to refer to
> traffic engineered sub-network layers.
> 
> >  From a TSV-ART point of view, the document can either only use the term
> "transport" for Internet transport protocols and use another term for sub-
> network layers (as handled in the *routing* area of the IETF), or the document
> has to clearly distinguish between the Internet transport layer and other uses of
> the term "transport" and explain the overlap. I believe the former would be less
> confusing, but I will leave it up to the TSV ADs to discuss terminology overlap in
> the IESG. As TSV-ART reviewer I insist that the document uses the terms
> "transport layer" and "transport protocol" only when referring to the Internet
> transport layer.
> 
> > Regarding Section 4.4 I have already deferred the discussion to the IESG. The
> TSV-ART review comment is that the IESG needs to carefully look at the
> concepts, terminology, and references in section 4.4.
> >
> > Regarding my other comments, I acknowledge that -09 is a step forward. But
> given the cross-dependencies e.g. regarding terminology and definitions, I will
> need to read the text completely once there is a proposal how to address my
> review. As noted in my review, I believe the document must use terminology
> clearly and consistently. As example, a statement in -09 such as "Network nodes
> supporting DetNet flows have to implement some of the DetNet capabilities
> (not necessarily all) in order to treat DetNet flows such that their QoS
> requirements are met" is IMHO too vague. But in such cases it depends whether
> there is precise normative guidance elsewhere. And this requires looking at the
> text as a whole.
> 
> Tsv-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tsv-art

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 20 Nov 2018 16:21:44 +0000

Hello Lou:

About ' discuss changing the name of the "DetNet Transport sub-layer"  to avoid the word "transport".  '

For one I'd like to make that call. The unfortunate name collision has started to hurt us quite a bit already and people are getting confused on very active exchanges we have on the mailing list. 

I tend to agree that for the general IETF "transport" generally means "L4". Even point one in your email uses "transport" that way I guess. Sadly many alternate names are highly overloaded already (think "carrier" for instance, or "bus"). I like the term "train" because of the association with a schedule, but that's just me.

Same goes actually for the complex path that we build. That complex path can be an elongated DODAG with multiple PREOF points. Usual terms like "circuit" or "path" fail to capture that complexity. 6TiSCH found the term "Track" to refer to it.

Would you push that discussion to the ML?

Take care,

Pascal 

> From: Lou Berger <[email-masked]>
> Sent: mardi 20 novembre 2018 13:11
> To: Scharf, Michael <[email-masked]>
> Subject: Re: [Detnet] Tsvart last call review of draft-ietf-detnet-architecture-08
> 
> Michael,
> 
> I think we're getting somewhere and identifying where we have disconnects
> and what may (and what may not) need to change in the document.  My
> takeaways are:
> 
> - The document needs a good 'scrub' of the congestion related references to
> ensure that the document only makes statements on what is actually done
> within a DetNet and the relationship with transport protocols that use detnet
> (which are in fact outside the scope of the DetNet WG).  I'll work with the
> authors and WG on this -- I see this change as important, but editorial in
> nature.
> 
> - We have a perception issue with at least one member of the TSV area on the
> meaning and more importantly, implication, of the term "DetNet
> *Transport* sub-layer".  While I don't disagree that a good portion of the IETF
> thinks transport protocol (UDP/TCP) when they hear "transport"
> there are plenty others, particularly in the routing area, who understand that
> "transport" can refer to Transport Networks.  And Transport Network is a well
> understood general industry term. The IETF even has a bunch of RFCs that
> relate to Transport networks.  This said, I think it reasonable to go back to the
> DetNet WG and discuss changing the name of the "DetNet Transport sub-
> layer"  to avoid the word "transport".  -- BTW we made a parallel change in
> the TEAS WG when producing RFC8453.
> 
> >> spawned by your review -- from a doc shepherd perspective, and see
> >> where we are.   I know that the authors have sent a -09 version that
> >> addresses some, but not all issues.
> >>
> >> congestion would be inelastic traffic using DetNet or due to
> >> congestion loss that is masked by PREOF.
> > These are two major issues that need to be addressed. Note that it may not
> be sufficient just to add a section on operational and deployment
> considerations. Also the existing text in the document will need to get aligned
> to normative guidance on how to avoid a congestion collapse.
> >
> > At least the last sentence would contradict a better discussion of congestion
> in the document. For instance, it could just be removed. In any case, the
> current wording in the last sentence is not correct, as the IETF term for what is
> > In addition, I'd like to emphasize that my review comment "It is surprising
> that there is hardly any discussion on network robustness and safety"
> 
> > covers more than just inelastic traffic that escapes from a DetNet network
> and masking of packet loss. Given that DetNet traffic may be extremely critical
> traffic, I really wonder why the document doesn't emphasize more the
> required robustness against failures *inside* the DetNet network as well as
> counter-measures. But this is something the WG needs to decide. As TSV-ART
> reviewer, I will be fine if the document clearly describes how the impact of
> failures will be isolated inside the DetNet network and will not put the general
> Internet at risk.
> 
> has to clearly distinguish between the Internet transport layer and other uses
> of the term "transport" and explain the overlap. I believe the former would be
> less confusing, but I will leave it up to the TSV ADs to discuss terminology
> overlap in the IESG. As TSV-ART reviewer I insist that the document uses the
> terms "transport layer" and "transport protocol" only when referring to the
> Internet transport layer.
> 
> clearly and consistently. As example, a statement in -09 such as "Network
> nodes supporting DetNet flows have to implement some of the DetNet
> capabilities (not necessarily all) in order to treat DetNet flows such that their
> QoS requirements are met" is IMHO too vague. But in such cases it depends
> whether there is precise normative guidance elsewhere. And this requires
> looking at the text as a whole.
>

---

From: Lou Berger <[email-masked]>
Date: Tue, 20 Nov 2018 13:07:35 -0500

Michael,

Thanks for the clarification.

On 11/20/2018 10:02 AM, Scharf, Michael wrote:
> As far as I understand, DetNet introduces network functions that at least partly novel, such as PREOF.  And as PREOF explicitly allows solutions beyond 1+1 protection ("network coding" is mentioned as example).

at the network layer, protection and restoration is not new. There is 
LFAs. FRR, 1+1, 1:n, shared mesh etc all defined in RFCs - mostly, but 
not solely for MPLS.  It's also not new transport protocol layer either 
as there is work (have to check if RFCs) on network coding and 1+1 -- 
but this is outside the scope of Detnet.

> I believe that any new function typically raises operational issues.
>

No more so than exists in the current  RFCs related to the topic.

> - Overload of DetNet components: As another example for PREOF, how would a DetNet deployment ensure that PEF never gets overloaded?

Umm, how does an RFC ever dictate that a router/node/host isn't 
overloaded?  In the normal internet case, I think this is way outside 
the scope of an RFC.  For TE networks, this is what 
configuration/management/control is all about - and sure we can add a 
sentence on that.

> Buffering and resequencing of packets requires processing. Techniques to allocate buffer space alone may not be enough to ensure that PEF never gets overloaded. Is this a problem or not? And even if this problem may perhaps not really exist for 1+1, is there no need to worry about other, possibly more complex PREOF schemes that may require complex packet processing (or even coding)? Are there scaling issues in PREOF? And could this impact signaling, e.g., require control plane protocol extensions?

yes, there is configuration/management/control-- this can be mentioned.

> More generally, if the intention of DetNet is to carry mission-critical or safety-critical traffic that cannot tolerate any packet loss, let alone any downtime, wouldn't words such as "reliability"…

The IETF generally just defines the behavior on the wire in standards 
track RFCs and let others define requirement system performance levels.  
I don't think we should be changing this in DetNet.

> And this may not only matter for the data plane. Assuming that a lot of DetNet traffic requires extremely high reliability, are there architectural requirements on components outside the data plane…

There are plenty of high availability (and different level of 
availability) implementations of systems out there.  The only think the 
IETF gets involved with specifying are those aspects that impact 
interoperability between systems.  (E.g., all the graceful restart 
related extensions in control plane protocols). Again, I don't think 
it's for DetNet to move beyond this.

> However, just to be precise: I don't ask for additional text to address what is written in this e-mail.

Thanks - it sounds like we're ending at the same place even if we didn't 
follow the same path to get here.

Thanks again for comments/review.  I do think we'll have a better 
document once the other points are addressed.

Lou

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 20 Nov 2018 18:27:28 +0000

I support this change;

Pascal

> Sent: mardi 20 novembre 2018 19:19
> To: Pascal Thubert (pthubert) <[email-masked]>; Scharf, Michael
> <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: Transport sub-layer name change (Was Re: [Detnet] Tsvart last call
> review of draft-ietf-detnet-architecture-08)
> 
> ALL,
> 
> There is a desire to replace the word "Transport" from the DetNet Transport
> sub-layer to avoid confusion with L$ Transport protocols.
> 
> In the TEAS WG we had a similar discussion and we replaced "Transport"
> with "Traffic Engineered (TE) ".
> 
> While a bit more verbose, what do people think about this change?
> 
> To be clear, the suggestion is:
> 
> OLD
> 
>                     .
>                     .
>       +----------------------------+
>       |  DetNet Service sub-layer  | PW, UDP, GRE
>       +----------------------------+
>       | DetNet Transport sub-layer | IPv6, IPv4, MPLS TE LSPs, MPLS SR
>       +----------------------------+
>                     .
>                     .
> 
>                   Figure 4: DetNet adaptation to data plane
> 
> NEW
> 
>                     .
>                     .
>       +----------------------------+
>       |  DetNet Service sub-layer  | PW, UDP, GRE
>       +----------------------------+
>       |      DetNet TE sub-layer   | IPv6, IPv4, MPLS TE LSPs, MPLS SR
>       +----------------------------+
>                     .
>                     .
> 
>                   Figure 4: DetNet adaptation to data plane
> 
> Lou
> 
> On 11/20/2018 11:21 AM, Pascal Thubert (pthubert) wrote:
> > Hello Lou:
> >
> >
> > For one I'd like to make that call. The unfortunate name collision has started
> to hurt us quite a bit already and people are getting confused on very active
> exchanges we have on the mailing list.
> >
> > I tend to agree that for the general IETF "transport" generally means "L4".
> Even point one in your email uses "transport" that way I guess. Sadly many
> alternate names are highly overloaded already (think "carrier" for instance, or
> "bus"). I like the term "train" because of the association with a schedule, but
> that's just me.
> >
> > Same goes actually for the complex path that we build. That complex path
> can be an elongated DODAG with multiple PREOF points. Usual terms like
> "circuit" or "path" fail to capture that complexity. 6TiSCH found the term
> "Track" to refer to it.
> >
> >> - The document needs a good 'scrub' of the congestion related
> >> references to ensure that the document only makes statements on what
> >> is actually done within a DetNet and the relationship with transport
> >> protocols that use detnet (which are in fact outside the scope of the
> >> DetNet WG).  I'll work with the authors and WG on this -- I see this
> >> change as important, but editorial in nature.
> >>
> >> - We have a perception issue with at least one member of the TSV area
> >> on the meaning and more importantly, implication, of the term "DetNet
> >> *Transport* sub-layer".  While I don't disagree that a good portion
> >> of the IETF thinks transport protocol (UDP/TCP) when they hear "transport"
> >> Transport Network is a well understood general industry term. The
> >> IETF even has a bunch of RFCs that relate to Transport networks.
> >> This said, I think it reasonable to go back to the DetNet WG and
> >> discuss changing the name of the "DetNet Transport sub- layer"  to
> >> avoid the word "transport".  -- BTW we made a parallel change in the TEAS
> WG when producing RFC8453.
> >>
> >>>> I wanted to take a step back from the multiple discussions that
> >>>> were spawned by your review -- from a doc shepherd perspective, and
> >>>> see where we are.   I know that the authors have sent a -09 version
> >>>> that addresses some, but not all issues.
> >>>>
> >>>>    From the exchanges I've seen, I think the key remaining issues
> >>>> are related to:
> >>>>
> >> considerations. Also the existing text in the document will need to
> >> get aligned to normative guidance on how to avoid a congestion collapse.
> >>> In -09, one example would be Section 3.1. "Primary goals defining
> >>> the
> >> DetNet QoS"
> >> in the document. For instance, it could just be removed. In any case,
> >> the current wording in the last sentence is not correct, as the IETF
> >> term for what is described in the last sentence is "congestion control".
> >> expected) case that packets get dropped or that ECN marks are
> >> received. It is understood that this would not happen in normal
> >> operation of a DetNet network, but I believe just considering the
> >> error-free operation of a DetNet network is not sufficient for this
> >> document. At least for the risk of traffic that may escape from a
> >> DetNet network is inherently not sufficient to assume that the DetNet
> network is always error-free.
> >>
> >>> As a result, addressing my concerns will most likely require editing
> >>> several
> >> parts of the document.
> >> and masking of packet loss. Given that DetNet traffic may be
> >> extremely critical traffic, I really wonder why the document doesn't
> >> emphasize more the required robustness against failures *inside* the
> >> DetNet network as well as counter-measures. But this is something the
> >> WG needs to decide. As TSV-ART reviewer, I will be fine if the
> >> document clearly describes how the impact of failures will be
> >>>> basically a Traffic Engineered sub-network layer, such as is
> >>>> provided with MPLS-TE or Optical Transport Networks.
> >> protocols. I doubt that the document can avoid discussing the
> >> implications of and interactions with Internet transport protocols
> >> such as UDP or TCP. As a result, I disagree that the document can use
> >> the term 'transport' to refer to traffic engineered sub-network layers.
> >>
> >> "transport" for Internet transport protocols and use another term for
> >> sub- network layers (as handled in the *routing* area of the IETF),
> >> or the document has to clearly distinguish between the Internet
> >> transport layer and other uses of the term "transport" and explain
> >> the overlap. I believe the former would be less confusing, but I will
> >> leave it up to the TSV ADs to discuss terminology overlap in the
> >> IESG. As TSV-ART reviewer I insist that the document uses the terms
> >> "transport layer" and "transport protocol" only when referring to the
> >> generally understood industry term that is also used in RFCs -- so
> >> we'll have to see what where WG consensus ends up.
> >>
> >> TSV-ART review comment is that the IESG needs to carefully look at
> >> the concepts, terminology, and references in section 4.4.
> >> given the cross-dependencies e.g. regarding terminology and
> >> definitions, I will need to read the text completely once there is a
> >> proposal how to address my review. As noted in my review, I believe
> >> the document must use terminology clearly and consistently. As
> >> example, a statement in -09 such as "Network nodes supporting DetNet
> >> flows have to implement some of the DetNet capabilities (not
> >> necessarily all) in order to treat DetNet flows such that their QoS
> >> requirements are met" is IMHO too vague. But in such cases it depends
> >> I think the next steps lie with me and the WG.  We'll let you know
> >> once there is a new version.  Of course, you can also contribute to
> >> the WG discussion on the topic.
> >>
> >> Thanks,
> >>
> >> Lou
> >>
> >>> Best regards
> >>>
> >>> Michael
> >>>
> >>>
> >>>
> >>>> Thank you,
> >>>>
> >>>> Lou
> >>>>
> >>>>> (draft-ietf-detnet-architecture-08) defines an overall framework
> >>>>> for Deterministic Networking.
> >>>>>
> >>>>> traffic. That assumption should be explicitly spelt out early in
> >>>>> the
> >>>> document,
> >>>>> discussion is needed at a prominant place. For instance, can use
> >>>>> of
> >>>> DetNet do
> >>>>> additional measures." I wonder if a similar disclaimer is needed
> >>>>> for
> >>>> DetNet. If
> >>>>> misconfiguration or errors of functions performing packet
> >>>>> replication
> >>>> could
> >>>>> severely and permantly congest a network and cause harm. How does
> >>>>> the
> >>>> DetNet
> >>>>> changes or there are equipment failures? Probably this can be
> >>>>> solved
> >>>> by
> >>>>> requirements not spelt out? Section 3.3.2 speculates that filters
> >>>>> and
> >>>> policers
> >>>>> vague and mentions a "infinite variety of possible failures"
> >>>>> without
> >>>> stating
> >>>>> any requirements or recommendations. There may be further
> >>>>> solutions,
> >>>> such as
> >>>>> * Somewhat related, the document only looks at impact of failures
> >>>>> to
> >>>> the QoS of
> >>>>> DetNet traffic. What is missing is a discussion how to protect
> >>>>> non-
> >>>> DetNet parts
> >>>>> of a network from any harm caused by DetNet mechanisms. Solutions
> >>>>> to
> >>>> this
> >>>>> probably exist. But why is the impact on non-DetNet traffic (e.g.,
> >>>>> in
> >>>> case of
> >>>>> * Regarding security, an architecture like DetNet probably
> >>>>> requires
> >>>> that only
> >>>>> security considerations only briefly mention the control aspect
> >>>>> ("the authentication and authorization of the controlling systems").
> >>>>>
> >>>>> * For an architecture document, the lack of clarity and
> >>>>> consistency
> >>>> regarding
> >>>>> terminology is concerning. This specifically applies to the case
> >>>>> of
> >>>> incomplete
> >>>>> The document introduces terms such as "DetNet intermediate nodes"
> >>>>> but
> >>>> then
> >>>>> "The primary means by which DetNet achieves its QoS assurances is
> >>>>> to
> >>>> reduce, or
> >>>>> seems to only apply to "DetNet transit nodes" but not
> >>>>> "DetNet-unaware
> >>>> nodes".
> >>>>> Similar ambiguity exist for other use of the terms "hop" and
> >>>>> "node",
> >>>> which may
> >>>>> not consistently use the terminology introduced in Section 2.1 in
> >>>>> all
> >>>> sections
> >>>>> documents having IETF consensus. An example would be RFC 7491,
> >>>>> albeit
> >>>> there is
> >>>>> other related work as well, e.g., in the TEAS WG. Second, Section
> >>>>> 4.4
> >>>> is by and
> >>>>> applicability to DetNet. Section 4.4 even mentions explicitly at
> >>>>> the
> >>>> end that
> >>>>> at all clear why Section 4.4 is in this document. Section 4.4
> >>>>> could
> >>>> be removed
> >>>>>      The term "transport layer" has a well-defined meaning in the
> >>>>> IETF,
> >>>> e.g.
> >>>>>      originating from RFC 1122. While "transport" and e.g.
> >>>>> "transport
> >>>> network" is
> >>>>>      used in the IETF for different technologies in different
> >>>>> areas, I
> >>>> think the
> >>>>>      protocols such as TCP and UDP. As such, I personally find the
> >>>>> term
> >>>> "DetNet
> >>>>>      transport layer" misleading and confusing. The confusion is
> >>>>> easy
> >>>> to see e.g.
> >>>>>      in Figure 4, where UDP (which is a transport protocol as per
> >>>>> RFC
> >>>> 1122) sits
> >>>>>      on top of "transport".
> >>>>>
> >>>>>      the "DetNet transport layer" is actually a separate protocol
> >>>>> layer
> >>>> compared
> >>>>>      to the "DetNet service layer". Thus it is not clear to me why
> >>>>> the
> >>>> word
> >>>>>      "layer" has to be used, specifically in combination
> >>>>> "transport
> >>>> layer".
> >>>>>      To me as, the word "transport layer" (and "transport
> >>>>> protocol")
> >>>> should be
> >>>>>      What is a "DetNet node"? That term is not introduced in
> >>>>> Section
> >>>> 2.1
> >>>>> * Page 14
> >>>>>
> >>>>>       A DetNet network supports the dedication of a high
> >>>>> proportion
> >>>> (e.g.
> >>>>>      If the term "transport layer" cannot be avoided, the labels
> >>>>> in
> >>>> this figure
> >>>>>      protocol. If the term "transport" cannot be avoided, the
> >>>>> labels in
> >>>> this
> >>>>>      that was possible, a statement like "... can be made
> >>>>> available by
> >>>> the system
> >>>>>       reserving bandwidth and buffer resources at every hop along
> >>>>> the
> >>>> path
> >>>>>       of the DetNet flow.
> >>>>>
> >>>>>      Why does this sentence use the word "hop"? As far as I
> >>>>> understand,
> >>>> in DetNet
> >>>>>      If there were hops over IP routers not being DetNet
> >>>>> intermediate
> >>>> nodes, no
> >>>>>      resources would be reserved there. As per Section 4.3.3, it
> >>>>> is
> >>>> possible to
> >>>>>      IP, on devices that are not routers... So does "hop" here
> >>>>> refer to
> >>>> IP router
> >>>>>      The text does not explain why a _central_ controller is
> >>>>> needed for
> >>>> this
> >>>>>      computation. Why would a distributed control plane not be
> >>>>> able to
> >>>> realize
> >>>>>      the description of the examples in Section 4.7.3. For
> >>>>> instance,
> >>>> "VID +
> >>>>>      multicast MAC address" is not introduced. I think this
> >>>>> example
> >>>> must be
> >>>>>      expaned with additional context and explanation to be useful
> >>>>> to
> >>>> readers.
> >>>>> * Page 34
> >>>>>
> >>>>>       There are three classes of information that a central
> >>>>> controller
> >>>> or
> >>>>>      the same paragraph also apply to a network that uses
> >>>>> _multiple_
> >>>> central
> >>>>>      controllers, or hybrid combinations of central controllers
> >>>>> and
> >>>> distributed
> >>>>>      control planes? In general, an architecture document should
> >>>>> be
> >>>> agnostic to
> >>>>>      implementation aspects unless there is a specific need. In
> >>>>> this
> >>>> specific
> >>>>>       scheduling, that might otherwise be over-subscribed and
> >>>>> trigger
> >>>> the
> >>>>>       to-peer control plane, some of this information may be
> >>>>> required
> >>>> by a
> >>>>>      "Latency" or "delay" would probably be a better terms if the
> >>>>> value
> >>>> is
> >>>>>      measured in nanoseconds.
> >>>>>
> >>>>> * Page 35:
> >>>>>
> >>>>>       DetNet is provides a Quality of Service (QoS), and as such,
> >>>>> does
> >>>> not

---

From: "Grossman, Ethan A." <[email-masked]>
Date: Tue, 20 Nov 2018 18:28:47 +0000

I like it. 
Ethan.

-----Original Message-----
From: Pascal Thubert (pthubert) <[email-masked]> 
Sent: Tuesday, November 20, 2018 10:27 AM
To: Lou Berger <[email-masked]>; Scharf, Michael <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: RE: Transport sub-layer name change (Was Re: [Detnet] Tsvart last call review of draft-ietf-detnet-architecture-08)

I support this change;

Pascal

> Subject: Transport sub-layer name change (Was Re: [Detnet] Tsvart last 
> call review of draft-ietf-detnet-architecture-08)
> 
> ALL,
> 
> There is a desire to replace the word "Transport" from the DetNet 
> Transport sub-layer to avoid confusion with L$ Transport protocols.
> 
> > About ' discuss changing the name of the "DetNet Transport 
> > sub-layer"  to
> avoid the word "transport".  '
> >
> to hurt us quite a bit already and people are getting confused on very 
> active exchanges we have on the mailing list.
> >
> Even point one in your email uses "transport" that way I guess. Sadly 
> many alternate names are highly overloaded already (think "carrier" 
> for instance, or "bus"). I like the term "train" because of the 
> association with a schedule, but that's just me.
> >
> "circuit" or "path" fail to capture that complexity. 6TiSCH found the 
> term "Track" to refer to it.
> >
> >> references to ensure that the document only makes statements on 
> >> what is actually done within a DetNet and the relationship with 
> >> transport protocols that use detnet (which are in fact outside the 
> >> scope of the DetNet WG).  I'll work with the authors and WG on this 
> >> -- I see this change as important, but editorial in nature.
> >>
> >> - We have a perception issue with at least one member of the TSV 
> >> area on the meaning and more importantly, implication, of the term 
> >> "DetNet
> >>>> were spawned by your review -- from a doc shepherd perspective, 
> >>>> and see where we are.   I know that the authors have sent a -09 
> >>>> version that addresses some, but not all issues.
> >>>>
> >>>> (a) possibly introduction of congestion in the general internet 
> >>>> if packets were somehow to escape a detnet domain.  The source of 
> >>>> this congestion would be inelastic traffic using DetNet or due to 
> >> in the document. For instance, it could just be removed. In any 
> >> case, the current wording in the last sentence is not correct, as 
> >> the IETF term for what is described in the last sentence is "congestion control".
> >> I think these are examples of text that needs to be cleanup up and 
> >> to delineate what is done with a DetNet.
> >>
> >>
> >>> As a result, addressing my concerns will most likely require 
> >>> editing several
> >> parts of the document.
> >> extremely critical traffic, I really wonder why the document 
> >> doesn't emphasize more the required robustness against failures 
> >> *inside* the DetNet network as well as counter-measures. But this 
> >> is something the WG needs to decide. As TSV-ART reviewer, I will be 
> >> fine if the document clearly describes how the impact of failures 
> >> will be isolated inside the DetNet network and will not put the 
> >> general Internet at
> risk.
> >>
> >> such as UDP or TCP. As a result, I disagree that the document can 
> >> use the term 'transport' to refer to traffic engineered sub-network layers.
> >>
> >> the overlap. I believe the former would be less confusing, but I 
> >> will leave it up to the TSV ADs to discuss terminology overlap in 
> >> the IESG. As TSV-ART reviewer I insist that the document uses the 
> >> I'm personally okay with a name change and even willing to push 
> >> this discussion within the WG, but as said above, "Transport 
> >> Network" is a generally understood industry term that is also used 
> >> in RFCs -- so we'll have to see what where WG consensus ends up.
> >>
> >>>> Do you have any other issues that that are critical to be 
> >>>> addressed before this work moves forward?  If so which?
> >> definitions, I will need to read the text completely once there is 
> >> a proposal how to address my review. As noted in my review, I 
> >> believe the document must use terminology clearly and consistently. 
> >> As example, a statement in -09 such as "Network nodes supporting 
> >> DetNet flows have to implement some of the DetNet capabilities (not 
> >> requirements are met" is IMHO too vague. But in such cases it 
> >> depends whether there is precise normative guidance elsewhere. And 
> >> this requires
> >>>>> additional means. Thus, for a baseline document, one could 
> >>>>> expect
> >>>> some
> >>>>> harm to parts of a network not supporting DetNet? As a side 
> >>>>> note,
> >>>> when TCPM
> >>>>> published RFC 8257, the following disclaimer was added: "DCTCP, 
> >>>>> as
> >>>> described in
> >>>>> * It is surprising that there is hardly any discussion on 
> >>>>> network
> >>>> robustness
> >>>>> and safety; this probably also relates to security. For 
> >>>>> instance, misconfiguration or errors of functions performing 
> >>>>> packet replication
> >>>> could
> >>>>> severely and permantly congest a network and cause harm. How 
> >>>>> does the
> >>>> DetNet
> >>>>> architecture ensure that a network stays fully operational e.g. 
> >>>>> if
> >>>> the topology
> >>>>> requirements not spelt out? Section 3.3.2 speculates that 
> >>>>> filters and
> >>>> policers
> >>>>> and correctly configured (and well-behaving) devices. And 
> >>>>> Section
> >>>> 3.3.2 is
> >>>>> * Somewhat related, the document only looks at impact of 
> >>>>> failures to
> >>>> the QoS of
> >>>>> of a network from any harm caused by DetNet mechanisms. 
> >>>>> Solutions to
> >>>> this
> >>>>> probably exist. But why is the impact on non-DetNet traffic 
> >>>>> (e.g., in
> >>>> case of
> >>>>> topology changes or failures of DetNet functions) not discussed 
> >>>>> at
> >>>> all in the
> >>>>> document?
> >>>>>
> >>>>> "The primary means by which DetNet achieves its QoS assurances 
> >>>>> is to
> >>>> reduce, or
> >>>>> even completely eliminate, congestion within a node as a cause 
> >>>>> of
> >>>> packet loss"
> >>>>> not consistently use the terminology introduced in Section 2.1 
> >>>>> in all
> >>>> sections
> >>>>> * Section 4.4 refers to RFC 7426, which is an informational RFC 
> >>>>> on
> >>>> IRTF stream,
> >>>>> large decoupled from the rest of the document and not specific 
> >>>>> to
> >>>> DetNet.
> >>>>>      The term "transport layer" has a well-defined meaning in 
> >>>>> the IETF,
> >>>> e.g.
> >>>>>      protocols such as TCP and UDP. As such, I personally find 
> >>>>> the term
> >>>> "DetNet
> >>>>>      in Figure 4, where UDP (which is a transport protocol as 
> >>>>> per RFC
> >>>> 1122) sits
> >>>>>      on top of "transport".
> >>>>>
> >>>>>      Based on the document it also may be 
> >>>>> solution/implementation
> >>>> specific whether
> >>>>>      the "DetNet transport layer" is actually a separate 
> >>>>> protocol layer
> >>>> compared
> >>>>>      to the "DetNet service layer". Thus it is not clear to me 
> >>>>> why the
> >>>> word
> >>>>>       A DetNet node may have other resources requiring 
> >>>>> allocation
> >>>> and/or
> >>>>>       scheduling,
> >>>>>
> >>>>>      This is just one of several examples for inconsistent use 
> >>>>> of
> >>>> terminology.
> >>>>>      The 75% value is not reasoned. What prevents using 99% of 
> >>>>> the
> >>>> bandwidth for
> >>>>>      DetNet traffic?
> >>>>>
> >>>>> * Page 15: Figure 2
> >>>>>
> >>>>>      As already mentioned earlier, Figure 4 is confusing. UDP is 
> >>>>> a
> >>>> transport
> >>>>>       reserving bandwidth and buffer resources at every hop 
> >>>>> along the
> >>>> path
> >>>>>       of the DetNet flow.
> >>>>>
> >>>>>      expaned with additional context and explanation to be 
> >>>>> useful to
> >>>> readers.
> >>>>> * Page 34
> >>>>>
> >>>>>      case, I fail to see a need to discuss the realization of 
> >>>>> the
> >>>> control plane of
> >>>>>      a network.
> >>>>>
> >>>>> Editorial nits:
> >>>>>
> >>>>> * Page 9:
> >>>>>
> >>>>>       A DetNet node may have other resources requiring 
> >>>>> allocation
> >>>> and/or
> >>>>>      "Latency" or "delay" would probably be a better terms if 
> >>>>> the value
> >>>> is
> >>>>>      measured in nanoseconds.
> >>>>>
> >>>>> * Page 35:
> >>>>>
> >>>>>       DetNet is provides a Quality of Service (QoS), and as 
> >>>>> such, does
> >>>> not

---

From: Lou Berger <[email-masked]>
Date: Tue, 20 Nov 2018 13:18:44 -0500

ALL,

OLD

                    .
                    .

NEW

                    .
                    .

Lou

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 20 Nov 2018 15:11:42 -0500

This terminology is certainly appropriate for TEAS. For DetNet, this seems
to make the assertion that the DetNet underlay is always
traffic-engineered, even if IPv4 or IPv6 (which is certainly possible using
TE extensions for the IGPs).

As long as people are OK with this assertion, then I'm OK with it as well.
That should be made clear in the architecture spec where the term "DetNet
TE sub-layer" is introduced/defined.

Cheers,
Andy

On Tue, Nov 20, 2018 at 1:29 PM Grossman, Ethan A. <[email-masked]> wrote:

> Subject: RE: Transport sub-layer name change (Was Re: [Detnet] Tsvart last

---

From: Lou Berger <[email-masked]>
Date: Tue, 20 Nov 2018 16:10:58 -0500

On 11/20/2018 3:11 PM, Andrew G. Malis wrote:
> This terminology is certainly appropriate for TEAS. For DetNet, this 
> seems to make the assertion that the DetNet underlay is always 
> traffic-engineered, even if IPv4 or IPv6 (which is certainly possible 
> using TE extensions for the IGPs).

This is certainly a fair point and one that does lead me to have a 
slight reservation about the change, that said, it seems the benefit out 
ways the downsides.

>
> As long as people are OK with this assertion, then I'm OK with it as 
> well. That should be made clear in the architecture spec where the 
> term "DetNet TE sub-layer" is introduced/defined.

agreed.

Lou

>
> Cheers,
> Andy
>
> On Tue, Nov 20, 2018 at 1:29 PM Grossman, Ethan A. <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I like it.
>     Ethan.
>
>     From: Pascal Thubert (pthubert) <[email-masked]
>     <mailto:[email-masked]>>
>     To: Lou Berger <[email-masked] <mailto:[email-masked]>>;
>     Scharf, Michael <[email-masked]
>     <mailto:[email-masked]>>
>     Cc: [email-masked] <mailto:[email-masked]>;
>     [email-masked]
>     <mailto:[email-masked]>
>     Subject: RE: Transport sub-layer name change (Was Re: [Detnet]
>     Tsvart last call review of draft-ietf-detnet-architecture-08)
>
>     I support this change;
>
>     Pascal
>
>     > From: Lou Berger <[email-masked] <mailto:[email-masked]>>
>     > To: Pascal Thubert (pthubert) <[email-masked]
>     <mailto:[email-masked]>>; Scharf, Michael
>     > <[email-masked]
>     <mailto:[email-masked]>>
>     > Cc: [email-masked] <mailto:[email-masked]>;
>     [email-masked]
>     <mailto:[email-masked]>

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 20 Nov 2018 22:34:22 +0100

IMHO, "Traffic Engineering" will be worse than "Transport":

- Makes more important people confused

  IMHO, the mayor group of people (rightfully) confused by the term "Transport"
  in the DetNet context are IETF apps and transport area people, and
  quite frankly, thats not the primary reader constituency of DetNet
  documents. Network operators / vendors / TSN members / TSN solutoin
  devenlopers, aka: the primary reader consitutency for DetNet documents will
  IMHO be less confused by "Transport" than by "Traffic Engineering".

- Creates even more inconsistent confusion

  IMHO, Differnt confused people will even more likely assume through 
  name recognition a lot more different functions into "Traffic
  Engineering". (TE means RSVP-TE, oh no, TE means network planning,
  TE can not mean QoS, TE is jut admission control, no TE needs to be
  per-hop, yada yada yada).

I am not arguing for keeping Transport, i am just more worried about
"Traffic Engineering" than "Transport". I think a uniue new term without
name recognition and resulting confusion would be best. 

Maybe form a new term from key words like the following, eg.:

 DetNet Flow SubNetwork Encapsulation and Adaption Layer (DF-SEAL)

Cheers
    Toerless

On Tue, Nov 20, 2018 at 04:10:58PM -0500, Lou Berger wrote:
> This is certainly a fair point and one that does lead me to have a slight
> reservation about the change, that said, it seems the benefit out ways the
> downsides.
> 
> > 
> > well. That should be made clear in the architecture spec where the term
> > "DetNet TE sub-layer" is introduced/defined.
> 
> agreed.
> 
> Lou
> 
> > 
> > Cheers,
> > Andy
> >

---

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 20 Nov 2018 21:39:16 +0000

Just to add some additional references. I have made my point already and I will not further follow-up.

> From: Lou Berger [mailto:[email-masked]]
> Sent: Tuesday, November 20, 2018 7:08 PM
> To: Scharf, Michael
> Subject: Re: [Tsv-art] [Detnet] Tsvart last call review of draft-ietf-
> detnet-architecture-08
> 
> Michael,
> 
> >>> In addition, I'd like to emphasize that my review comment "It is
> surprising that there is hardly any discussion on network robustness
> and safety"
> > As far as I understand, DetNet introduces network functions that at
> least partly novel, such as PREOF.  And as PREOF explicitly allows
> solutions beyond 1+1 protection ("network coding" is mentioned as
> example).
> 

From draft-ietf-detnet-architecture-09:

  3.2.2.3.  Packet encoding for service protection

   There are methods for using multiple paths to provide service
   protection that involve encoding the information in a packet
   belonging to a DetNet flow into multiple transmission units,
   combining information from multiple packets into any given
   transmission unit.  Such techniques, also known as "network coding",
   can be used as a DetNet service protection technique.

So, "network coding" is explicitly *in scope* of draft-ietf-detnet-architecture-09. 

And my comments are about the architectural impact of that section.

It is not clear to me why one would need an IRTF research group (NWCRG) if "network coding" was just a straightforward extension of existing MPLS or OTN protection and restoration schemes.

> > - PREOF (and specifically) PEF as a single point of failure: Is there
> any need to worry about that? As far as I can see, the document
> discusses failures of paths between POF and PEF, but not failures of
> the functions themselves. Wouldn't it make sense to discuss basic
> requirements such as implementing these functions redundantly etc.? And
> if they are implemented redundantly, does this impact the architecture,
> e.g., result in additional interfaces that may need standardization?
> 
> > - Overload of DetNet components: As another example for PREOF, how
> would a DetNet deployment ensure that PEF never gets overloaded?
> 

See RFCs 5390, 6357, 7068, 7339, 7683, etc. for examples where overload control is standardized. These are typical solutions in environments where five-nine availability is needed. I won't argue that these RFCs directly relate to DetNet, as these RFCs are not about routing technology. But, nonetheless, it is surprising that the working group does not worry at all about overload of newly defined functions such as PREOF, while the working group at the same time allows possibly compute-intensive solutions for that. That comes down to the assumption that compute power is never a bottleneck...

But I'll not follow-up further on that. I assume that is also a topic where other IETF areas have better expertise.

Michael

> > Buffering and resequencing of packets requires processing. Techniques
> to allocate buffer space alone may not be enough to ensure that PEF
> never gets overloaded. Is this a problem or not? And even if this
> problem may perhaps not really exist for 1+1, is there no need to worry
> about other, possibly more complex PREOF schemes that may require
> complex packet processing (or even coding)? Are there scaling issues in
> PREOF? And could this impact signaling, e.g., require control plane
> protocol extensions?
> 
> > More generally, if the intention of DetNet is to carry mission-
> critical or safety-critical traffic that cannot tolerate any packet
> loss, let alone any downtime, wouldn't words such as "reliability",
> "dependability", "redundancy", "high availability", etc. be relevant in
> a description of the architecture? (Some of these terms are actually
> used in this document, mostly in references to non-IETF work.)
> 
> > And this may not only matter for the data plane. Assuming that a lot
> of DetNet traffic requires extremely high reliability, are there
> architectural requirements on components outside the data plane, e.g.,
> regarding redundancy? Obviously, control plane and management plane
> functions are often built today already as high availability solutions,
> but if the intention is to _never_ fail, isn't there a risk that DetNet
> may require solutions "better-than-current-IP" outside the data plane
> as well?
> 
> > This list of topics may not be comprehensive. I just want to
> illustrate some examples for what could also matter when running a
> network _really_ deterministically, beyond the well-known congestion
> issue.
> >

---

From: Lou Berger <[email-masked]>
Date: Tue, 20 Nov 2018 17:34:09 -0500

Hi Toreless,

> IMHO, "Traffic Engineering" will be worse than "Transport":

FWIW David Black - suggested that "Transport Network" as another 
alternative.

>    devenlopers,

> aka: the primary reader consitutency for DetNet documents will
I personally think we have enough terms and should go with "DetNet TE 
Sub-Layer", "DetNet TE Network Sub-Layer", or  "DetNet Transport Network 
Sub-Layer"

Lou

---

From: Mach Chen <[email-masked]>
Date: Wed, 21 Nov 2018 07:58:17 +0000

Hi,

I think "DetNet TE Sub-Layer" is good name, because one of the goals of DetNet is to provide bounded latency, to achieve that, dedicated resource (bandwidth, buffer) has to be reserved, explicit routing may be required, these are typical behaviors of Traffic Engineering. 

I used to think that "DetNet TE Tunnel Sub-Layer" is an appropriate name, given that DetNet likes an overlay model, especially for MPLS-based encapsulation. Where the Service Layer is over the "Transport Layer", it just likes the VPN model. And in the world of VPN, they are normally called Service Layer and Tunnel Layer.  If we only have MPLS-based encapsulation, I think "DetNet TE Tunnel Sub-layer" may be the perfect name. But as Andy pointed out, we have IP-based encapsulation that does not have the service layer, there is no such an overlay model.  Therefore, I think "DetNet TE Sub-Layer" is an appropriate name for both IP and MPLS-based encapsulations. 

Best regards,
Mach 

> From: detnet [mailto:[email-masked]] On Behalf Of Lou Berger
> Sent: Wednesday, November 21, 2018 6:34 AM
> To: Toerless Eckert <[email-masked]>
> Cc: Pascal Thubert (pthubert) <[email-masked]>; Andrew G. Malis
> <[email-masked]>; [email-masked]; draft-ietf-detnet-
> [email-masked]; Grossman, Ethan A. <[email-masked]>; detnet
> WG <[email-masked]>
> Subject: Re: [Detnet] Transport sub-layer name change (Was Re: Tsvart last
> > "Traffic Engineering" than "Transport". I think a uniue new term
> > without name recognition and resulting confusion would be best.
> >
> I personally think we have enough terms and should go with "DetNet TE Sub-
> Layer", "DetNet TE Network Sub-Layer", or  "DetNet Transport Network Sub-
> Layer"
> 
> Lou
> 
> > Cheers
> >      Toerless
> >
> >>> traffic-engineered, even if IPv4 or IPv6 (which is certainly
> >>> possible using TE extensions for the IGPs).
> >> slight reservation about the change, that said, it seems the benefit
> >> out ways the downsides.
> >>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 21 Nov 2018 09:24:43 +0000

Sorry Toerless I cannot buy " quite frankly, thats not the primary reader constituency of DetNet". 

We already face huge confusion in the current discussion in the ML. We do discuss Transport on the ML if only to decide what needs to be done in-house vs. elsewhere (in Transport Area) for a complete solution. And then we'll have our docs reviewed by the IESG for every publication request.

We could keep the word transport if we hide it well in an acronym that no one ever expands in the discussions (like MPLS). With David's suggestion that would be DTN, DetNet Transport Network. Sadly DTN is already taken at the IETF. Maybe DTNS with Service included : )

Otherwise using the word "transport" for an alternate meaning than layer-4 is really harmful to the discussions and publications.
Note that we decided to work only on centralized solutions, with a controller that allocates resources along a "path". So the "path" we build is effectively TE. This is why I supported Lou's suggestion.

All the best;

Pascal

> From: Toerless Eckert <[email-masked]>
> Sent: mardi 20 novembre 2018 22:34
> To: Lou Berger <[email-masked]>
> Cc: Andrew G. Malis <[email-masked]>; Grossman, Ethan A.
> <[email-masked]>; [email-masked]; Pascal
> Thubert (pthubert) <[email-masked]>; detnet WG <[email-masked]>;
> [email-masked]
> I am not arguing for keeping Transport, i am just more worried about "Traffic
> Engineering" than "Transport". I think a uniue new term without name
> recognition and resulting confusion would be best.
>

---

From: Loa Andersson <[email-masked]>
Date: Wed, 21 Nov 2018 19:13:30 +0800

Folks,

I think we have encountered this before, at these times we either talked
about "Layer 2 Transport (L2 Transport)" or "Transport Network", fwiw
that would work here also.

But otoh I could live with "DetNet TE Sub-Layer", the small caveat is
that that implies we are talking about a Sub-Layer of DetNet, rather
than an independent transport network.

/Loa

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert
Bronze Dragon Consulting             phone: +46 739 81 21 64

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 21 Nov 2018 17:56:08 +0100

Thanks, Lou, inline

> Sub-Layer", "DetNet TE Network Sub-Layer", or  "DetNet Transport Network

AFAIK, TE classically does include protection, such as in RSVP-TE. Yet
in DetNet, protection is specifically part of the service sub-layer,
and arguably you probably won't want to reuse pre-existing protection
mechanism of the transport/TE sub-layer without a lot of further design
considerations if you're using the DetNet Service sub-layer
protection, so that TE/transport sub-layer is not really doing full TE.

Toerless

---

From: Lou Berger <[email-masked]>
Date: Wed, 21 Nov 2018 12:18:50 -0500

> protection, so that TE/transport sub-layer is not really doing full TE.

If a TE/Transport Network sub-layer includes sufficient protection 
mechanisms, I'm not sure why we would not make use of it.  This is 
exactly what is planned for IP over) TSN for example. Now you can 
certainly argue that TSN is below the DetNet TE layer, but I think the 
point still holds.

BTW I think DetNet's service/TE split is very similar to what we have in 
RFC5921, but in a less MPLS-centric form.

Lou

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 23 Nov 2018 01:05:04 +0100

On Wed, Nov 21, 2018 at 12:18:50PM -0500, Lou Berger wrote:
> mechanisms, I'm not sure why we would not make use of it.  This is exactly
> what is planned for IP over) TSN for example. Now you can certainly argue
> that TSN is below the DetNet TE layer, but I think the point still holds.

Sure, and TSN dual-path redundancy is sufficient for DetNet, whereas
pre-existing RSVP-TE FRR is arguably not. But that same
"TE" protection is also available at the DetNet service layer. Hence the
confusion that would occur if we would call the "transport" layer now the
"TE" layer (IMHO). Aka: DetNet TE functions are not a sole property of
the DetNet "transport" layer.

> RFC5921, but in a less MPLS-centric form.

Not sure which specific terms/layers in rfc5921 you're referring to.
It does extensively ue the word "transport" though ;-))

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Norman Finn <[email-masked]>
Date: Tue, 27 Nov 2018 22:24:04 +0000

I don't think it's ideal, but I have no better suggestion.  OK.

-- Norm

________________________________________
From: Lou Berger [[email-masked]]
Sent: Tuesday, November 20, 2018 10:18 AM
To: Pascal Thubert (pthubert); Scharf, Michael
Cc: [email-masked]; [email-masked]
Subject: Transport sub-layer name change (Was Re: [Detnet] Tsvart last call review of draft-ietf-detnet-architecture-08)

ALL,

OLD

                    .
                    .

NEW

                    .
                    .

Lou

---

From: Norman Finn <[email-masked]>
Date: Tue, 27 Nov 2018 22:31:44 +0000

Typically, the resources for DetNet flows are reserved before the flow starts.  In the case of PREOF, the functions are at least configured before being used.  I would think that, if insufficient capability exists, the reservation/configuration would be refused.  We're talking an awful lot of 9s here, to justify this feature, so doing otherwise would be a poor implementation choice.  Do we need to say something about that, or figure that implementors will do the right thing?

-- Norm

________________________________________
From: detnet [[email-masked]] on behalf of Lou Berger [[email-masked]]
Sent: Tuesday, November 20, 2018 10:07 AM
To: Scharf, Michael
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] [Tsv-art] Tsvart last call review of draft-ietf-detnet-architecture-08

Michael,

Lou

---

From: "Scharf, Michael" <[email-masked]>
Date: Wed, 28 Nov 2018 07:02:36 +0000

Just to repeat: I won't mandate text on this specific issue.

But there could be an implicit assumption that PREOF compute resources in the node fast path will _always_ be available. And it is entirely unclear to me if that would be a realistic assumption if the router fast path has to do complex coding on each packet, e.g. for "network coding" as explicitly mentioned in the architecture document. There could be tradeoffs on power, cost, etc. when doing such complex PREOF coding functions for every single packet. And it could be more complex than existing MPLS protection and restoration schemes.

So, one could e.g. wonder if a DetNet network could run into the situation that a DetNet node as enough bandwidth and buffer, but not enough "network coding" coding/decoding capacity, or if a DetNet node would have to announce that it can do, say, 1+1 protection with PREOF, but there are not enough transcoder resources left for doing PREOF "network coding". Then there could be architectural implications, e.g., on protocols.

Assuming that "implementers will do the right thing" would be reasonable if per-packet coding/decoding actions in PREOF are always cheap. Otherwise, one may have to consider transcoders as a potentially scarce resource that could also impact flow placement decisions - maybe somewhat similar to wavelength converters in optical networks. 

I don't know how to implement "network coding" in the fast path of a router that runs at Tbit/s. If running code has already shown that doing "network coding" at the line speed of today's routers is easily doable, this specific point may not be relevant. Clearly I am not an expert for router fast path designs.

Michael

> From: Norman Finn [mailto:[email-masked]]
> Sent: Tuesday, November 27, 2018 11:32 PM
> To: Lou Berger; Scharf, Michael
> Subject: RE: [Detnet] [Tsv-art] Tsvart last call review of draft-ietf-
> detnet-architecture-08
> 
> Typically, the resources for DetNet flows are reserved before the flow
> starts.  In the case of PREOF, the functions are at least configured
> before being used.  I would think that, if insufficient capability
> exists, the reservation/configuration would be refused.  We're talking
> an awful lot of 9s here, to justify this feature, so doing otherwise
> would be a poor implementation choice.  Do we need to say something
> about that, or figure that implementors will do the right thing?
> 
> -- Norm
>