Subject: Re: [Detnet] draft-finn-detnet-architecture-03 comments??

From: "Patrick Wetterwald (pwetterw)" <[email-masked]>
Date: Mon, 21 Mar 2016 09:13:17 +0000

Dear all,

This mail raises a question. Do we consider re-routing as part of deterministic networking?

Future users of Detnet (the ones I talked to, mainly in the smart grid area) do not want to consider re-routing as an option for a Deterministic networking.

Personally, I would categorise it as out of the current scope of Detnet.

Thanks,

Patrick

From: detnet <[email-masked]<mailto:[email-masked]>> on behalf of Balázs Varga A <[email-masked]<mailto:[email-masked]>>
Date: Sunday 20 March 2016 at 19:38
To: "Norman Finn (nfinn)" <[email-masked]<mailto:[email-masked]>>, DetNet WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Detnet] draft-finn-detnet-architecture-03 comments??

Hi Norm,

Many thanks for the detailed explanations. Now I see your points much better. Some further thoughts on packet loss are below.
Your feedback/proposed resolution regarding all other comments are fine with me.

Regarding the packet loss I see two things we may have to fix differently:
1, General definition of SLA parameters for DetNet --> draft-use-case?
This is a more general (and tricky, as you wrote :--) ) discussion. DetNet drafts refer to deterministic delay and extreme low
packet loss. Delay is a more easy thing to define, as a maximum value seems to be appropriate for all of the DetNet use-cases.
However defining loss is more challenging. There are a group of loss related parameters defined already by various SDOs:
Packet Loss Ratio (PLR), Maximum Outage Time, Severely Errored Seconds (SES), etc-etc. They are defined to cover the
data loss related requirements of different applications.

For PSNs we use most the PLR, which is calculated over a population of packets. How to define the population? Further, packet
loss events also have a distribution over that population. What is better: a burst of losses or evenly distributed losses? I think
the answers to such questions have to be provided by the applications requiring DetNet transport.

There can be several reasons causing the loss that has triggered this mailings and I agree we have to list them and consider
them all. My first thought is to deal with the SLA definition problem in the use-case draft as it is aimed to describe all relevant
scenarios/applications. We need a common set of SLA parameters and it can be constructed based on the DetNet scenarios
/ applications. If done, the architecture draft have to consider how to achieve all those SLA parameters.

2, Resolving text in the current version of the architecture draft
I like your proposal to start section 3 with just two bullets:
> I think the right answer is to start section 3 with just two bullets,
> max latency and very low packet loss probability, then explain the
> mechanism, and only then put the above three bullets.

I think your extended list of loss related parameters points towards the right direction to resolve our issue here.
Let’s analyze the story from a somewhat different viewpoint. There are 4 reasons of packet losses:
1, buffer overflow
2, media errors (BER: Bit Error Ratio)
3, link failure
4, node failure

The first is something to be solved with “Zero Congestion Loss” tools. Done.

Packet loss probability caused by all the others (2, 3 and 4) are connection specific and depends on topology, distance, etc.
Therefore we cannot define a single packet loss value for the whole DetNet network. What we can say is what packet loss
value can be achieved by using different tools for a given connection.

For simplicity let’s take as an example a p2p connection between two UNIs. Between the UNIs several path may exist (or
only a single one) depending on the network topology. What is our toolset to deal with the cause of the packet loss?
- media error (BER): we can use “Packet replication and elimination”
- link failure: we can use “Re-routing” and “Packet replication and elimination”
- node failure: we can use “Re-routing” and “Packet replication and elimination”

So we have in practice two tools: “Re-routing” and “Packet replication and elimination”.

Re-routing impacts only packet loss caused by failures and can be characterized by convergence time (determined by the
control plane). Of course actual convergence time is failure scenarios specific but we can give an upper bound on it based
on the control plane protocol what is network wide specific. Done.

But the impact of “Packet replication and elimination” is connection (UNIs) & path specific. For each path we can calculate
a packet loss value for normal operation (no failures). Loss is determined by media errors. Therefore we have a set of
packet loss values!!! If “Packet replication and elimination” is used with 1+N path, than the expected packet loss value
can be calculated based on the “error correlation of the paths” and the packet loss value sets of the paths.

So, the complete list for a given connection (in our example between the two UNIs) using 1+N protection:
- Minimum and maximum end-to-end latency from source to destination;
- N x Probability of loss of a packet along a given path, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet if “packet replication and elimination (1+N)” [section 3.3] is employed;
- Characteristics of recovery times from path failures.

The list if no “packet replication and elimination” is used:
- Minimum and maximum end-to-end latency from source to destination;
- Probability of loss of a packet along a given path, assuming the normal operation of the relay systems and links;
- Characteristics of recovery times from path failures.

Let’s see whether that is fine with all the use-cases or we have to extend the list further.

Thanks & Cheers
Bala’zs

From: Norman Finn (nfinn) [mailto:[email-masked]]
Sent: Friday, March 18, 2016 9:52 PM
To: Balázs Varga A; DetNet WG
Subject: Re: draft-finn-detnet-architecture-03 comments??

Some thought provoking comments, Balázs; they’ll improve the document.  Detailed responses in-line.

— Norm

From: Balázs Varga A <[email-masked]<mailto:[email-masked]>>
Date: Friday, March 18, 2016 at 05:01 AM
To: Norman Finn <[email-masked]<mailto:[email-masked]>>, DetNet WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: draft-finn-detnet-architecture-03 comments??

Hi Norm,

some comments/questions for clarification:

1, PACKET LOSS

Chapter 3 defines two packet loss term:

- Probability of loss of a packet, assuming the normal operation of

the relay systems and links;

- Probability of loss of a packet in the event of the failure of a

relay system or link.

I have sympathy with the approach to distinguish the root cause of

packet loss, however I think the loss caused by failure is not an

appropriate parameter here. Packet loss have to be measured by

definition over a "set of packets" what is not easy to define for

example in case of re-routing that last for 200msec. Also the DetNet

applications may suffer differently from such an outage after a

failure. A "Maximum outage time" caused by failure events would be a

better parameter. That would also help network designer to decide

about the need for seamless redundancy for a DetNet flow. A network

tuned for "fast convergence" may require only for some very critical

DetNet flow the usage of seamless redundancy.

Proposed change:

Replace text on second packet loss parameter: "Maximum outage time in

the event of the failure of a relay system or link."

This gets a bit tricky.  In case you’re not using the “Packet replication and deletion” = “Seamless redundancy” = “1+1 hitless redundancy”, which is the usual case of a stream of packets over the Internet (or a bridged LAN, for that matter), what you say is dead right, of course.  But if you are using that scheme, the case is a little different.  Let me explain the situation, then look at what we can do to fix the document.

With the “Packet replication and deletion”, you’re not switched between streams based on OAM; you’re receiving every packet, looking at the sequence information, and passing on the ones you haven’t seen, possibly out of order because the paths are of different lengths.  (Yes, that requires keeping a history, in the most general case.  That’s another discussion.)  When using this technique, you’re going to use pinned paths, so network response time and convergence time after a failure is not a factor.  (To a first approximation.  More to come.)  Assuming that you are also using resource reservation to eliminate congestion loss, you have a pretty low probability of loss.  Ignoring big failures, it’s the bit error rate squared.  (!)  Of course, on a wired network, the actual total error rate is likely to be dominated by the failure rate of the branch points, source, and destination, which are not included in the list.

If a link fails on one of the paths, there is no convergence time issue, there is no switchover time issue.  But, now that one path has failed, the packet loss probability is now the probability of a loss on the second, still good, path.  This is the sense in which “Probability of loss of a packet in the event of the failure of a relay system or link” seems to me to be just as valid as the first bullet, although the dependencies are not stated in the clearest way.  (As evidenced by this long explanation.)  Connectivity loss time doesn’t kick in until you talk about the probability of loss of the branch points, or of two failures, one on each path.

Now, we could have three points, restated somewhat:

- Probability of loss of a packet, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet in the event of a connectivity failure along one path.
- Maximum outage time in the event of a connectivity failure involving both paths.

This is a possibility.  But, in the scenarios envisioned so far, this gets us into the question, “When one path fails, do you wait for it to get fixed, or do you try to automatically create a new path?”  There are many answers to this question.  Some would like to see ISIS wait a while, then create a new path.  Some (like me) think that the right thing is to let a central controller and/or human decide whether to create a new 2nd pinned path based on knowledge of, for example, whether the outage was simply of mater of the ring being broken, momentarily, to add a new hop.  The justification is that the remaining zero congestion loss path buys you some reaction time.

So, let’s fix the document.  Given the above discussion, which has not been seen by the reader at this point in the draft, I’d say the following:

(- Minimum and maximum end-to-end latency from source to destination;)
- Probability of loss of a packet, assuming the normal operation of the relay systems and links;
- Probability of loss of a packet during a connectivity failure along one path, if packet replication and elimination [section 3.3] is employed;
- Characteristics of recovery times from path failures.

Note the change of wording in the two bullets.  I think that this makes the bullets right.  But, given this discussion, I think the right answer is to start section 3 with just two bullets, max latency and very low packet loss probability, then explain the mechanism, and only then put the above three bullets.

Also, as long as we’re at it — In the paragraph following the 3 bullets, it talks about maxima, rather than averages.  Detent provides worst-case latency, but packet loss remains probabilistic, not a worst-case thing, so that discussion needs to be modified, too.

2, SEAMLESS REDUNDANCY (1+N)

Chapter 3.3 states names the motivation for seamless redundancy "...

causes of packet loss are random media and/or memory faults, and

equipment failures.", but describes only link/node failure scenarios.

I think adding this sentence to the end of the first paragraph will help:

Both causes of packet loss can be greatly reduced by sending the same packets over multiple paths.

Text describes only 1+1 protection and Lou has also commented on this

" Section 3.3 - This section should reference that "Seamless

Redundancy as defined in this section is also known as 1+1 hitless

protection." "

I have found reference to 1+1 protection misleading. Protection

against multiple failures or seamless redundancy motivated by random

media faults (e.g., bit errors) may require 1+N protection. Of course

1+N protection may be an issue for high BW DetNet flows, but it would

describe the usage of seamless redundancy better.

Proposed change:

Generalize text/example to describe 1+N seamless redundancy.

First of all, I want to get rid of the term “seamless redundancy” except when explicitly talking about IEC 62439-3 HSR.  So, let’s use the term “Packet replication and elimination” until we have a better term.

Secondly, you’re right about the reference.  I’ve been working on this section, also.  I’ll keep the reference, if only to contrast 1+1 stream switching with selecting individual packets from two streams.  And, I’ll mention that it can more than two paths are possible.

3, END SYSTEMS (L2 vs. L3)

Chapter 4.2.3. The Network Plane states: "... in the end systems,

which are typically IP hosts". I think this is rather use-case

specific. IEEE discussions and also the use-case draft list several

scenarios, where DetNet end systems are L2 nodes, as the node

developer does not wanted to pay the tax of the IP protocol stack

implementation and DetNet flow is encapsulated directly in Ethernet

frames.

I would argue to keep the DetNet scenarios/use-cases/architecture as

general as possible and we should be prepared to transport both L2

flows and L3 flows over a DetNet domain.

Proposed change:

Generalize text in the terminology section:

"end system - Commonly called a "host" in IETF documents, and an "end

station" is IEEE 802 documents. End systems of interest to

this document are either sources or destinations. End systems may be

L2 or L3 hosts."

Lou made some comments on the end system definition, also; it’s getting too big.  I’ll add the L2/L3 difference, but in fewer words.  Also, we’ll say in 4.2.3 that end systems can be L2 or L3.

4, DETNET PROTOCOL STACK

I have found the Figure and description in "4.7. Protocol Stack

Model" somewhat confusing.

4.A, For the Sequence generation/recovery layer text states "Supplies

the sequence number for Seamless Redundancy ...". I think sequence

number is not used exclusively for Seamless Redundancy. Seq.number

is needed for (i) loss detection, (ii) deal with out-of-order-

delivery and (iii) duplicate detection. Therefore even if seamless

redundancy is not needed by a DetNet flow (e.g., due to its

parameters) transport requires the seq.number.

Proposed change:

List all motivations for seq.num:

"Supplies the sequence number for (i) loss detection, (ii) deal with

out-of-order-delivery and (iii) duplicate detection (e.g., for

Seamless Redundancy) ..."

Lou also commented on this area, because sometimes the sequence number is already there in the transport protocol.  Your idea is good.  I’ll combine them.

4.B, "discards duplicate packets coming up the stack" is stated for

both layer "Sequence generation/recovery" and "Individual flow

checking". Why is that duplicate discard implemented in two layers?

Proposed change: Any clarification ???

A word of explanation is needed in the description of individual flow checking.  I hesitated to include it at all.  Basically, it is a function related to 4.6 fault mitigation; it’s nice to catch faults as soon as possible and defend agains misbehaving relay systems before the separate flows come together.  Also, operating on one stream at a time, and then operating on the combined pair, is thought by some to improve the reliability.  It’s in the 802.1 diagrams, today.

However, your comment makes me reread the section, and now I think that it would be better to omit the “individual” layer, and at most, mention that the sequence generation/recovery can usefully operate on a single stream.

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

In our plan from November, we talked about adopting an architecture draft.

There is a draft for one, and time comment and revise it before the deadline for April.  Is it so good there is nothing more to say?  :)  :)

https://www.ietf.org/internet-drafts/draft-finn-detnet-architecture-03.txt

— Norm

_______________________________________________

detnet mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/detnet

---

From: "Norman Finn (nfinn)" <[email-masked]>
Date: Mon, 21 Mar 2016 15:51:12 +0000

I would agree; it’s a bit too deep for round 1.

— Norm

From: "Patrick Wetterwald (pwetterw)" <[email-masked]<mailto:[email-masked]>>
Date: Monday, March 21, 2016 at 02:13 AM
To: Balázs Varga A <[email-masked]<mailto:[email-masked]>>, Norman Finn <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Detnet] draft-finn-detnet-architecture-03 comments??

Dear all,

Thanks,

Patrick

Hi Norm,

Thanks & Cheers
Bala’zs

— Norm

Hi Norm,

1, PACKET LOSS

relay system or link.

Proposed change:

protection." "

Proposed change:

frames.

Proposed change:

L2 or L3 hosts."

4, DETNET PROTOCOL STACK

Proposed change:

Thanks & Cheers

Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn (nfinn)
Sent: Thursday, March 10, 2016 12:29 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] draft-finn-detnet-architecture-03 comments??

— Norm

detnet mailing list