Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Wed, 9 Mar 2016 23:29:04 +0000

In our plan from November, we talked about adopting an architecture draft.
There is a draft for one, and time comment and revise it before the
deadline for April.  Is it so good there is nothing more to say?  :)  :)

https://www.ietf.org/internet-drafts/draft-finn-detnet-architecture-03.txt

— Norm

---

From: "Schmitt, Juergen" <[email-masked]>
Date: Thu, 10 Mar 2016 15:14:39 +0000

Hi Norm,

First of all thank you for the great work in this document.
I have just collected some notes which may be interesting to talk about.

In the following I copied parts of the clause in question with may proposal for change:

Snippet: ==================================================================================
2.  Terminology
...

   reservation
           A trail of configuration from source to destination(s)
           through relay systems associated with a DetNet flow, required
           to deliver the benefits of DetNet.

END-Snippet: ===============================================================================

I don’t see that the flow of configuration is just from source to destination. I would argue in some of  today’s RSVP-Models the opposite is true.
Many models are imaginable.  So I wouldn’t mention any direction here. 

Snippet: ==================================================================================
3. Providing the DetNet Quality of Service

DetNet Quality of Service is expressed in terms of:

o Minimum and maximum end-to-end latency from source to destination;

o Probability of loss of a packet, assuming the normal operation of
the relay systems and links;

o Probability of loss of a packet in the event of the failure of a
relay system or link.

o Delay variation <-- NEW

END-Snippet: ===============================================================================

Delay variation is in my opinion also a very important criteria for Quality of Service.
There are use-cases where it is more important, that the Packet arrives Just-In-Time, than the raw 
End-to-end latency.

Just a comment to clause 6.4. Peer-to-peer reservation protocol:

I think this is a interesting model. It is surely debatable within the network. But as an Interface for endstations (Talkers and Listeners) this can be very useful. 
Not to forget about small customer networks which need also a simple to administer network with end-to-end QoS and Ad-Hoc reservation. 

Best regards,
Jürgen

-----Ursprüngliche Nachricht-----
Von: detnet [mailto:[email-masked]] Im Auftrag von Norman Finn (nfinn)
Gesendet: Donnerstag, 10. März 2016 00:29
An: [email-masked]
Betreff: [Detnet] draft-finn-detnet-architecture-03 comments??

In our plan from November, we talked about adopting an architecture draft.
There is a draft for one, and time comment and revise it before the deadline for April.  Is it so good there is nothing more to say?  :)  :)

— Norm

_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Lou Berger <[email-masked]>
Date: Tue, 15 Mar 2016 17:56:23 -0400

Hi Norm/Authors,
    I reviewed your document and have some comments.  These comments can
be addressed now or once the document becomes a WG document (your choice):

- First let me say this is a great start and very much appreciated!

Section 2:
- general point: I think it would be good to separate into their own
subsection, e.g., 2.1,  terms that are listed for the sole purpose of
translating between IEEE 802 and DetNet terminology, e.g., listener and
relay, and keeping terms used throughout the document in the main section.

OLD
   end system
           Commonly called a "host" in IETF documents, and an "end
           station" is IEEE 802 documents.  End systems of interest to
           this document are either sources or destinations.
NEW
   end system
           Commonly called a "host" or "node" in IETF documents, and an "end
           station" is IEEE 802 documents.  End systems of interest to
           this document are either sources or destinations. Note that a
system that
           takes non-DetNet aware traffic and transmits it via a DetNet
flow is also an
        end system.  (For comparison, a Label Edge Router (LER) would be
        an MPLS "end system".)

Question: Do we need a term of the DetNet equivalent of an LER

To relay system, add "transit node" to the list.

Add:

  DetNet node
        A DetNet aware end system or relay system.
        "DetNet" may be omitted in some text.

  DetNet domain
        The portion of a network that is DetNet aware.  It includes end
systems,
        and other DetNet nodes.

  link
    A connection between two DetNet nodes.  It may be composed of a
physical link
    or a sub-network technology that can provide appropriate traffic
delivery for DetNet
    flows.

Section 3.

add to (c).  "This function is also know as 1+1 protection, e.g., see
[RFC6372]."

Section 3.1

    this section should mention allocation and scheduling or resources
that might otherwise be over subscribed and trigger congestion loss.

Section 3.2

Might be worth mentioning that pinned paths are commonly used in MPLS TE
LSPs.

Section 3.3

- This section should reference that "Seamless Redundancy as defined in
this section is also known as 1+1 hitless protection."

- Adding sequence number isn't and architectural requirement, but rather
one *example* way to implement Seemless redundancy.

- section 4.1
s/provisioning by relay systems and end systems sharing peer-to-peer
protocols/via control plane protocols (running on relay systems and end
systems)

- Section 4.2.3
s/The Network Plane/The Network Plane, which is also commonly referred
to as the data plane,

- Section 4.2.3, second paragraph
    to me UNIs and NNIs are as much about the control plane as the data
plane (see RFC5921 for examples and figures).  I'm not sure if this is
what you intended or not, but scope should be clarified. Either way 5921
is probably a better reference than 3209.

Section 4.5
s/not interfere excessively/coexist with
s/QoS/Class of Service (e.g., DiffServ)

Section 4.7

- Add after the 1st sentence. "It may serve as the foundation for the
DetNet model which will be defined by the working group."

Section 4.8
s/a central controller/a central controller or decentralized control plane
 
Section 4.9.2
s/[RFC5127]/[RFC3209] or [RFC3473]

also at end of section add:
"The integration/interaction of the DetNet control layer an underlying
IEEE 802.1 sub-network control layer will need to be defined."

Section 4.11
s/Layer-2 tunnels/Layer-2 connections
or
s/Layer-2 tunnels/Layer-2 VPNs

Section 5.

I think there's an additional point to be made, here's some suggested text:

All DetNet enabled systems and nodes will be interconnected by
sub-networks, i.e., Layer 2 technologies. These sub-networks will
provide DetNet compatible service for support of DetNet traffic.
Examples of sub-networks include 802.1TSN and a point-to-point OTN link.
Of course, multi-layer DetNet systems may be possible too, where one
DetNet appears as a sub-network, and provides service to, a higher layer
DetNet system.

That's it for now. 

So do you want to do an update before a WG adoption call or would you
prefer to address my comments in a -01 rev of the WG document?

Thanks,
Lou

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 15 Mar 2016 17:57:44 -0700

Thanks Lou. If I hear no complaints I'll edit these comments into the 
draft while I am rearranging the entire document to reflect the new 
structure.

- Jouni

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 15 Mar 2016 17:59:49 -0700

Ah bummer.. forget this mail. My brain was entirely in a different 
context ;-) (and it is early here).

- Jouni

---

From: zhayiyong <[email-masked]>
Date: Wed, 16 Mar 2016 07:32:29 +0000

Hi Norm,

Thank you for your effort on architecture draft. I believe it is a good work that covers almost every aspects of a working system of DetNet. I have some questions as follows.
Section 3:
Three candidate techniques are listed. Does that mean the WG is focused how to use these techniques to guarantee for example delay and loss? For pinned path, are we going to define routing mechanism to setup redundant paths or just utilize existing management protocols to manage? Maybe first step we can focus on a fixed path scenario. 

Section 4:
4.1: 
There are a lot of models been discussed here. In my opinion, some of them are close related that can be represented using one model, some are dependent on specific technique being used which is not easy to model. So maybe all the models can be classified as: service model to define the northbound interface for service provisioning; device model/configuration model to define the southbound interface; information model and data model of DetNet flow to represent the DetNet flow that is used by control plane. Thus, YANG can be used to model service model, device/configuration model and DetNet flow data model. Info model is to guide the design of data model and does not have to be YANG. 
4.2:
To me, an centralized SDN controller based architecture is introduced. But in open question peer-to-peer protocol is still mentioned in case of distributed architecture. Maybe we can use controller-based architecture but still adopt some distributed protocols. For example, using central controller for some job that needs global information such as PCE and routing, while using distributed protocols to reserve resource so the DetNet flow does not have to wait the communication between controller and NEs.
4.5:
Resource sharing is only allowed between DetNet flow and normal flow. All DetNet flows are scheduled with resource that cannot be shared among DetNet flows. And correct me if I am wrong, burstness is not taking into account as it suppose shaping can solve this. And one more question can be schedulability problem, with given flows with delay requirements and given resource, can the requirements be meet?  

Section 6:
For the open question, we would like to do more on flow identification and peer-to-peer reservation protocols. 

Cheers,
Yiyong Zha

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 7:29 AM
To: [email-masked]
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

---

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Thu, 17 Mar 2016 02:39:48 +0000

Lou,

Thank you for the comments.  They look very useful.

I don’t see any open-ended comments, here, and they make sense.  I’ll
incorporate at least most of your comments (and a few others — see other
emails) today, and upload a new version for adoption before the deadline.
Exactly which and how many comments depend on how much interaction we
managed to have before Monday.

A few comments/questions for you in-line, below.  Assume remarked comments
are fine with me.

— Norm

-----Original Message-----
From: Lou Berger <[email-masked]>
Date: Wednesday, March 16, 2016 at 05:56 AM
To: Norman Finn <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Detnet] draft-finn-detnet-architecture-03 comments??

>Question: Do we need a term of the DetNet equivalent of an LER

Interesting question.  I don’t think I’d put that in, without some more
discussion.  We have the concept in P802.1CB of the end system DetNet
functions as a lump that can be on either end of the wire connecting the
end system to the first relay system.  So, is a relay system with 1000
ordinary relay system DetNet ports and one lump an LER?  If the one port
configured for that capability is down at the moment?  Well, yes, but it’s
also, and mostly, a relay system.  I need to get a better feel for what an
LER is/isn’t before I add that to the document, so I don’t say something
stupid.  I don’t know if there is time before the deadline.

>s/The Network Plane/The Network Plane, which is also commonly referred

I don’t think that’s what Pascal meant, here.  As I read it, the intention
here is that the Network plane includes nailed-down paths that DetNet
flows might use.  It would also include the RSVP-TE that might be used to
nail that path down.

>is probably a better reference than 3209.

The Network Plane includes both control and data aspects.  The usage here
is that UNI and NNI are definitely control plane.  That’s how I’m used to
using those terms over the years in ATM, ITU-T, and IEEE.  Is that
different in IETF?

---

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Thu, 17 Mar 2016 03:02:58 +0000

Ah!  I think I see the problem in 4.2.3 — The text doesn’t mention that
the data plane is part of the network plane.

Pascal?

— Norm

-----Original Message-----
From: Norman Finn <[email-masked]>
Date: Thursday, March 17, 2016 at 10:39 AM
To: Lou Berger <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: Pascal Thubert <[email-masked]>
Subject: Re: [Detnet] draft-finn-detnet-architecture-03 comments??

>>>https://www.ietf.org/internet-drafts/draft-finn-detnet-architecture-03.t
>>>x
>>>t
>>>
>>>
>>> — Norm
>>>

---

From: Lou Berger <[email-masked]>
Date: Thu, 17 Mar 2016 09:17:58 -0400

Norm,
   Thanks for the response.  See below.

On 3/16/2016 10:39 PM, Norman Finn (nfinn) wrote:
> end system to the first relay system. 
lump; cool, a new networking term -- a rare commodity!  (In MPLS speak,
I believe this is an LER.  In MPLS-TP and LxVPN contexts, this is a PE
when the CE doesn't talk MPLS).

>  So, is a relay system with 1000
That's fine.  The document isn't even a -00 WG draft so there is time to
work this out.

> flows might use.  It would also include the RSVP-TE that might be used to

Humm, I included RSVP-TE and other control plane protocols as part of
the controller plane.  The differences to me there is off-path
(logically centralized) vs on-path (fully distributed) control. See mote
on this below.

> using those terms over the years in ATM, ITU-T, and IEEE.  Is that

I think I just don't understand these new terms.  I get management,
control, data and even policy planes.  I don't understand the
distinction being drawn by these new terms.  Looking at Figure 2, I
clearly see (and understand) the Application, Control, Management
planes.  Where things get fuzzy is in the 'Network Device' box (which is
not labeled as a plane -- a good thing).  In it I see the Forwarding
Plane, and the Operational Plane.  How about this: change "Network
Device" to "Data Plane",  "Forwarding Plane" to "Forwarding",
"Operational Plane" to "OAM" .  ( per the charter OAM = Operations,
Administration, and Maintenance)  And then after the figure add some
text along the lines of:
    The presented figure is logical view, and actual protocols and
    implementation platforms are not shown.  For example the Control
    Plane may be implemented via: a fully distributed control plane
    running on network devices, e.g., as found in some MPLS
    implementations; a fully centralized "SDN" controller; or a hybrid
    approach where some control functions are distributed to the network
    devices and others reside in a more centralized controller, e.g.,
    stateful PCE. 

I understand that this is a departure from the pure SDN model presented
in 7426, but DetNet isn't so narrowly scoped so it's architecture
document shouldn't be either.

The rest of the section will then need to be tweaked to line up with
these changes.

Thanks,
Lou

---

From: Balázs Varga A <[email-masked]>
Date: Thu, 17 Mar 2016 21:01:01 +0000

Hi Norm,

some comments/questions for clarification:

1, PACKET LOSS

Chapter 3 defines two packet loss term:

- Probability of loss of a packet, assuming the normal operation of

- Probability of loss of a packet in the event of the failure of a

relay system or link.

I have sympathy with the approach to distinguish the root cause of

packet loss, however I think the loss caused by failure is not an

appropriate parameter here. Packet loss have to be measured by

definition over a "set of packets" what is not easy to define for

example in case of re-routing that last for 200msec. Also the DetNet

applications may suffer differently from such an outage after a

failure. A "Maximum outage time" caused by failure events would be a

better parameter. That would also help network designer to decide

about the need for seamless redundancy for a DetNet flow. A network

tuned for "fast convergence" may require only for some very critical

DetNet flow the usage of seamless redundancy.

Proposed change:

Replace text on second packet loss parameter: "Maximum outage time in

the event of the failure of a relay system or link."

2, SEAMLESS REDUNDANCY (1+N)

Chapter 3.3 states names the motivation for seamless redundancy "...

causes of packet loss are random media and/or memory faults, and

equipment failures.", but describes only link/node failure scenarios.

Text describes only 1+1 protection and Lou has also commented on this

" Section 3.3 - This section should reference that "Seamless

Redundancy as defined in this section is also known as 1+1 hitless

protection." "

I have found reference to 1+1 protection misleading. Protection

against multiple failures or seamless redundancy motivated by random

media faults (e.g., bit errors) may require 1+N protection. Of course

1+N protection may be an issue for high BW DetNet flows, but it would

describe the usage of seamless redundancy better.

Proposed change:

Generalize text/example to describe 1+N seamless redundancy.

3, END SYSTEMS (L2 vs. L3)

Chapter 4.2.3. The Network Plane states: "... in the end systems,

which are typically IP hosts". I think this is rather use-case

specific. IEEE discussions and also the use-case draft list several

scenarios, where DetNet end systems are L2 nodes, as the node

developer does not wanted to pay the tax of the IP protocol stack

implementation and DetNet flow is encapsulated directly in Ethernet

frames.

I would argue to keep the DetNet scenarios/use-cases/architecture as

general as possible and we should be prepared to transport both L2

flows and L3 flows over a DetNet domain.

Proposed change:

Generalize text in the terminology section:

"end system - Commonly called a "host" in IETF documents, and an "end

this document are either sources or destinations. End systems may be

L2 or L3 hosts."

4, DETNET PROTOCOL STACK

I have found the Figure and description in "4.7. Protocol Stack

Model" somewhat confusing.

4.A, For the Sequence generation/recovery layer text states "Supplies

the sequence number for Seamless Redundancy ...". I think sequence

number is not used exclusively for Seamless Redundancy. Seq.number

is needed for (i) loss detection, (ii) deal with out-of-order-

delivery and (iii) duplicate detection. Therefore even if seamless

redundancy is not needed by a DetNet flow (e.g., due to its

parameters) transport requires the seq.number.

Proposed change:

List all motivations for seq.num:

"Supplies the sequence number for (i) loss detection, (ii) deal with

out-of-order-delivery and (iii) duplicate detection (e.g., for

Seamless Redundancy) ..."

4.B, "discards duplicate packets coming up the stack" is stated for

both layer "Sequence generation/recovery" and "Individual flow

checking". Why is that duplicate discard implemented in two layers?

Proposed change: Any clarification ???

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list

[email-masked]<mailto:[email-masked]>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 17 Mar 2016 21:54:59 +0000

I agree with Lou:

The UNI is control plane. The trouble I had here is that is it an exchange between the network node and the end system, not over the southband interface like the flow setup. How should be characterize this?

Cheers,

Pascal

> From: Norman Finn (nfinn)
> Sent: jeudi 17 mars 2016 04:03
> To: Lou Berger <[email-masked]>; [email-masked]
> Cc: Pascal Thubert (pthubert) <[email-masked]>
> Ah!  I think I see the problem in 4.2.3 — The text doesn’t mention that the
> data plane is part of the network plane.
> 
> Pascal?
> 
> — Norm
> 
> >A few comments/questions for you in-line, below.  Assume remarked
> >comments are fine with me.
> >
> >— Norm
> >
> >>    I reviewed your document and have some comments.  These comments
> >>can be addressed now or once the document becomes a WG document
> (your
> >>choice):
> >>
> >>translating between IEEE 802 and DetNet terminology, e.g., listener
> >>and relay, and keeping terms used throughout the document in the main
> >>section.
> >>
> >>OLD
> >>   end system
> >>           this document are either sources or destinations. Note that
> >>a system that
> >>           takes non-DetNet aware traffic and transmits it via a
> >>DetNet flow is also an
> >functions as a lump that can be on either end of the wire connecting
> >the end system to the first relay system.  So, is a relay system with
> >1000 ordinary relay system DetNet ports and one lump an LER?  If the
> >one port configured for that capability is down at the moment?  Well,
> >yes, but it’s also, and mostly, a relay system.  I need to get a better
> >feel for what an LER is/isn’t before I add that to the document, so I
> >don’t say something stupid.  I don’t know if there is time before the
> deadline.
> >
> >>        The portion of a network that is DetNet aware.  It includes
> >>end systems,
> >>        and other DetNet nodes.
> >>
> >>  link
> >>Might be worth mentioning that pinned paths are commonly used in MPLS
> >>TE LSPs.
> >>
> >>Section 3.3
> >>
> >>- This section should reference that "Seamless Redundancy as defined
> >>in this section is also known as 1+1 hitless protection."
> >>
> >>- Adding sequence number isn't and architectural requirement, but
> >>rather one *example* way to implement Seemless redundancy.
> >>
> >>
> >>- section 4.1
> >I don’t think that’s what Pascal meant, here.  As I read it, the
> >intention here is that the Network plane includes nailed-down paths
> >that DetNet flows might use.  It would also include the RSVP-TE that
> >might be used to nail that path down.
> >
> >>    to me UNIs and NNIs are as much about the control plane as the
> >>data plane (see RFC5921 for examples and figures).  I'm not sure if
> >>this is what you intended or not, but scope should be clarified.
> >>Either way 5921 is probably a better reference than 3209.
> >
> >The Network Plane includes both control and data aspects.  The usage
> >here is that UNI and NNI are definitely control plane.  That’s how I’m
> >used to using those terms over the years in ATM, ITU-T, and IEEE.  Is
> >that different in IETF?
> >
> >>Section 4.5
> >>s/not interfere excessively/coexist with s/QoS/Class of Service (e.g.,
> >>DiffServ)
> >>
> >>Section 4.7
> >>
> >>DetNet appears as a sub-network, and provides service to, a higher
> >>layer DetNet system.
> >>
> >>That's it for now.
> >>

---

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Fri, 18 Mar 2016 20:51:49 +0000

Some thought provoking comments, Balázs; they’ll improve the document.  Detailed responses in-line.

— Norm

From: Balázs Varga A <[email-masked]<mailto:[email-masked]>>
Date: Friday, March 18, 2016 at 05:01 AM
To: Norman Finn <[email-masked]<mailto:[email-masked]>>, DetNet WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: draft-finn-detnet-architecture-03 comments??

Hi Norm,

1, PACKET LOSS

relay system or link.

Proposed change:

This gets a bit tricky.  In case you’re not using the “Packet replication and deletion” = “Seamless redundancy” = “1+1 hitless redundancy”, which is the usual case of a stream of packets over the Internet (or a bridged LAN, for that matter), what you say is dead right, of course.  But if you are using that scheme, the case is a little different.  Let me explain the situation, then look at what we can do to fix the document.

With the “Packet replication and deletion”, you’re not switched between streams based on OAM; you’re receiving every packet, looking at the sequence information, and passing on the ones you haven’t seen, possibly out of order because the paths are of different lengths.  (Yes, that requires keeping a history, in the most general case.  That’s another discussion.)  When using this technique, you’re going to use pinned paths, so network response time and convergence time after a failure is not a factor.  (To a first approximation.  More to come.)  Assuming that you are also using resource reservation to eliminate congestion loss, you have a pretty low probability of loss.  Ignoring big failures, it’s the bit error rate squared.  (!)  Of course, on a wired network, the actual total error rate is likely to be dominated by the failure rate of the branch points, source, and destination, which are not included in the list.

If a link fails on one of the paths, there is no convergence time issue, there is no switchover time issue.  But, now that one path has failed, the packet loss probability is now the probability of a loss on the second, still good, path.  This is the sense in which “Probability of loss of a packet in the event of the failure of a relay system or link” seems to me to be just as valid as the first bullet, although the dependencies are not stated in the clearest way.  (As evidenced by this long explanation.)  Connectivity loss time doesn’t kick in until you talk about the probability of loss of the branch points, or of two failures, one on each path.

Now, we could have three points, restated somewhat:

- Probability of loss of a packet, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet in the event of a connectivity failure along one path.
- Maximum outage time in the event of a connectivity failure involving both paths.

This is a possibility.  But, in the scenarios envisioned so far, this gets us into the question, “When one path fails, do you wait for it to get fixed, or do you try to automatically create a new path?”  There are many answers to this question.  Some would like to see ISIS wait a while, then create a new path.  Some (like me) think that the right thing is to let a central controller and/or human decide whether to create a new 2nd pinned path based on knowledge of, for example, whether the outage was simply of mater of the ring being broken, momentarily, to add a new hop.  The justification is that the remaining zero congestion loss path buys you some reaction time.

So, let’s fix the document.  Given the above discussion, which has not been seen by the reader at this point in the draft, I’d say the following:

(- Minimum and maximum end-to-end latency from source to destination;)
- Probability of loss of a packet, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet during a connectivity failure along one path, if packet replication and elimination [section 3.3] is employed;
- Characteristics of recovery times from path failures.

Note the change of wording in the two bullets.  I think that this makes the bullets right.  But, given this discussion, I think the right answer is to start section 3 with just two bullets, max latency and very low packet loss probability, then explain the mechanism, and only then put the above three bullets.

Also, as long as we’re at it — In the paragraph following the 3 bullets, it talks about maxima, rather than averages.  Detent provides worst-case latency, but packet loss remains probabilistic, not a worst-case thing, so that discussion needs to be modified, too.

I think adding this sentence to the end of the first paragraph will help:

Both causes of packet loss can be greatly reduced by sending the same packets over multiple paths.

protection." "

Proposed change:

First of all, I want to get rid of the term “seamless redundancy” except when explicitly talking about IEC 62439-3 HSR.  So, let’s use the term “Packet replication and elimination” until we have a better term.

Secondly, you’re right about the reference.  I’ve been working on this section, also.  I’ll keep the reference, if only to contrast 1+1 stream switching with selecting individual packets from two streams.  And, I’ll mention that it can more than two paths are possible.

frames.

Proposed change:

L2 or L3 hosts."

Lou made some comments on the end system definition, also; it’s getting too big.  I’ll add the L2/L3 difference, but in fewer words.  Also, we’ll say in 4.2.3 that end systems can be L2 or L3.

4, DETNET PROTOCOL STACK

Proposed change:

Lou also commented on this area, because sometimes the sequence number is already there in the transport protocol.  Your idea is good.  I’ll combine them.

A word of explanation is needed in the description of individual flow checking.  I hesitated to include it at all.  Basically, it is a function related to 4.6 fault mitigation; it’s nice to catch faults as soon as possible and defend agains misbehaving relay systems before the separate flows come together.  Also, operating on one stream at a time, and then operating on the combined pair, is thought by some to improve the reliability.  It’s in the 802.1 diagrams, today.

However, your comment makes me reread the section, and now I think that it would be better to omit the “individual” layer, and at most, mention that the sequence generation/recovery can usefully operate on a single stream.

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list

---

From: Balázs Varga A <[email-masked]>
Date: Sun, 20 Mar 2016 18:38:07 +0000

Hi Norm,

Many thanks for the detailed explanations. Now I see your points much better. Some further thoughts on packet loss are below.
Your feedback/proposed resolution regarding all other comments are fine with me.

Regarding the packet loss I see two things we may have to fix differently:
1, General definition of SLA parameters for DetNet --> draft-use-case?
This is a more general (and tricky, as you wrote :--) ) discussion. DetNet drafts refer to deterministic delay and extreme low
packet loss. Delay is a more easy thing to define, as a maximum value seems to be appropriate for all of the DetNet use-cases.
However defining loss is more challenging. There are a group of loss related parameters defined already by various SDOs:
Packet Loss Ratio (PLR), Maximum Outage Time, Severely Errored Seconds (SES), etc-etc. They are defined to cover the
data loss related requirements of different applications.

For PSNs we use most the PLR, which is calculated over a population of packets. How to define the population? Further, packet
loss events also have a distribution over that population. What is better: a burst of losses or evenly distributed losses? I think
the answers to such questions have to be provided by the applications requiring DetNet transport.

There can be several reasons causing the loss that has triggered this mailings and I agree we have to list them and consider
them all. My first thought is to deal with the SLA definition problem in the use-case draft as it is aimed to describe all relevant
scenarios/applications. We need a common set of SLA parameters and it can be constructed based on the DetNet scenarios
/ applications. If done, the architecture draft have to consider how to achieve all those SLA parameters.

2, Resolving text in the current version of the architecture draft
I like your proposal to start section 3 with just two bullets:
> I think the right answer is to start section 3 with just two bullets,
> max latency and very low packet loss probability, then explain the
> mechanism, and only then put the above three bullets.

I think your extended list of loss related parameters points towards the right direction to resolve our issue here.
Let’s analyze the story from a somewhat different viewpoint. There are 4 reasons of packet losses:
1, buffer overflow
2, media errors (BER: Bit Error Ratio)
3, link failure
4, node failure

The first is something to be solved with “Zero Congestion Loss” tools. Done.

Packet loss probability caused by all the others (2, 3 and 4) are connection specific and depends on topology, distance, etc.
Therefore we cannot define a single packet loss value for the whole DetNet network. What we can say is what packet loss
value can be achieved by using different tools for a given connection.

For simplicity let’s take as an example a p2p connection between two UNIs. Between the UNIs several path may exist (or
only a single one) depending on the network topology. What is our toolset to deal with the cause of the packet loss?
- media error (BER): we can use “Packet replication and elimination”
- link failure: we can use “Re-routing” and “Packet replication and elimination”
- node failure: we can use “Re-routing” and “Packet replication and elimination”

So we have in practice two tools: “Re-routing” and “Packet replication and elimination”.

Re-routing impacts only packet loss caused by failures and can be characterized by convergence time (determined by the
control plane). Of course actual convergence time is failure scenarios specific but we can give an upper bound on it based
on the control plane protocol what is network wide specific. Done.

But the impact of “Packet replication and elimination” is connection (UNIs) & path specific. For each path we can calculate
a packet loss value for normal operation (no failures). Loss is determined by media errors. Therefore we have a set of
packet loss values!!! If “Packet replication and elimination” is used with 1+N path, than the expected packet loss value
can be calculated based on the “error correlation of the paths” and the packet loss value sets of the paths.

So, the complete list for a given connection (in our example between the two UNIs) using 1+N protection:
- Minimum and maximum end-to-end latency from source to destination;
- N x Probability of loss of a packet along a given path, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet if “packet replication and elimination (1+N)” [section 3.3] is employed;
- Characteristics of recovery times from path failures.

The list if no “packet replication and elimination” is used:
- Minimum and maximum end-to-end latency from source to destination;
- Probability of loss of a packet along a given path, assuming the normal operation of the relay systems and links;
- Characteristics of recovery times from path failures.

Let’s see whether that is fine with all the use-cases or we have to extend the list further.

Thanks & Cheers
Bala’zs

From: Norman Finn (nfinn) [mailto:[email-masked]]
Sent: Friday, March 18, 2016 9:52 PM
To: Balázs Varga A; DetNet WG
Subject: Re: draft-finn-detnet-architecture-03 comments??

— Norm

Hi Norm,

1, PACKET LOSS

relay system or link.

Proposed change:

protection." "

Proposed change:

frames.

Proposed change:

L2 or L3 hosts."

4, DETNET PROTOCOL STACK

Proposed change:

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 21 Mar 2016 07:01:38 +0000

Hello Norm

The discussion about 1+1 assumes equilatent streams so that copies a and b of a packet arrive roughly at the same time.

If they are not, one copy mat eventually make it to destination but the end point will sense a glitch. That will be OK for some transports and less so for others.

I think the tension is really between the services of resilience and in order delivery, the former deriving from 1+1 and the latter from the sequencing.

I could propose text but it will be short before submission which is cutoff today.

Regards,

Pascal

Regards,

Pascal
Le 18 mars 2016 à 21:52, Norman Finn (nfinn) <[email-masked]<mailto:[email-masked]>> a écrit :

— Norm

Hi Norm,

1, PACKET LOSS

relay system or link.

Proposed change:

protection." "

Proposed change:

frames.

Proposed change:

L2 or L3 hosts."

4, DETNET PROTOCOL STACK

Proposed change:

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list

---

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Mon, 21 Mar 2016 16:00:01 +0000

Great stuff Balázs.  It remains to see how much of this stuff I can get into the document before the deadline.  Not all of it.  But at some point, it needs to be there.

— Norm

From: Balázs Varga A <[email-masked]<mailto:[email-masked]>>
Date: Sunday, March 20, 2016 at 11:38 AM
To: Norman Finn <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: draft-finn-detnet-architecture-03 comments??

Hi Norm,

Thanks & Cheers
Bala’zs

— Norm

Hi Norm,

1, PACKET LOSS

relay system or link.

Proposed change:

protection." "

Proposed change:

frames.

Proposed change:

L2 or L3 hosts."

4, DETNET PROTOCOL STACK

Proposed change:

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list