Subject: [Detnet-dp-dt] quick notes from call 2/14/15

From: Jouni Korhonen <[email-masked]>
Date: Wed, 15 Feb 2017 11:37:27 -0800

Present: Jouni, Norm, Loa, Balazs, Janos, Carlos, Yuanlong.

Agenda:
- Was meant to be about CoS and QoS. However, we ended up discussion all time about proper layering and encapsulation of PW stack.

Discussion:
- Sequence number. We made a >>decision<< to settle down to 16 bits. This is the most “compatible” approach. At high speeds it has been argued 16 bits is not enough. In practical implementations even 16 bits of sequence number won’t be maintained. It is typically a much smaller window of the entire sequence number space (think about numbers.. 1M PWs times the seqnum bit vector etc..)
- PWs and the label stack. See the latest slides from Loa and specifically the slide 9 (attached). I did add one slide to this deck, the slide 14.
- T-labels are typically per hop. L-labels are between DetNet aware S/P-PEs and essentially form an overlay over the underlying network. d-pw labels are end to end (at the moment.. to be discussed) between the T-PEs or in general between the DetNet aware end point that understand the detnet data plane.
- currently all d-pw (detnet PW labels) experience FRER if that functionality is enabled. d-pw labels are tied to sequence numbers (the detnet CW).
- L-labels seen beneficial allowing the autoconfiguration of FRER i.e., build the overlay over the network and do not care configuring the PWs. This mimics one 802.1CB feature (see .1CB sub-clause 7.11).
- L-labels also allow simple label swap in a detnet S-PE i.e., no FRER would be applied. 
- This setup seems plausible. There was concerns overloading L-labels with some of the PW decision making in the fast path and thus possibly causing a lookup that needs to be done over two labels (L & d-pw). Essentially it is the L-label that signals whether the FRER gets applied.
- General consensus towards the PW instance “facing egress ports” in a detnet S-PE would do the elimination.  This discussion needs to be completed. An interesting use case was described by Norm (see slide 14). An example: packet arrives from A towards B (flow X), it always gets replicated towards C, but towards D only if the same packet (from flow Y) has not earlier arrived from C. Whether this works ok with the current S-PE and PW instance doing elimination concepts needs to be verified.
- Discussion to be completed whether there are cases where L-labels can be left out i.e., only use T-labels and d-pw labels. 
- Balazs said to provide a matrix/table of different permutations for labels & replications & eliminations.

We’ll have a call next Tuesday 2/21/17.

--
Jouni Korhonen, Broadcom Ltd.
M: +1-408-391-7160

---

From: Jouni Korhonen <[email-masked]>
Date: Fri, 17 Feb 2017 10:50:36 -0800

Folks,

See the attached slideset for some thoughts on implementing FRER in S-DetNet-PEs and Norm’s ladder redundancy case. Note, S-DetNet-PEs and T-DetNet-PEs are not off the shelf x-PEs, since they do understand DetNet quirks in addition to existing x-PE functionality.

The solution still assumes global d-pw space witin an administrative domain due the easier handling of SN counters and bit vectors for elimination on each x-PE. I have rightfully been pointed out that having a common reserved label space among all x-PEs for d-pw (detnet) purposes can be hard to achieve, especially in a multivendor environment (there are similar issues on segment routing as well). This is mostly a management/implementation issue, not a technical hardship IMHO.

If we were to agree to go towards a solution where each x-PE knows the d-pw space of their peers (at L-label level) the d-pw could be different on each PW for the same e2e detnet flow. The d-pw label would in this case be an x-PE DetNet-label-base + index. The index here would actually identify the e2e detnet flow.

Thoughts? Comments? Flames?

- Jouni

> On Feb 15, 2017, at 11:37 AM, Jouni Korhonen <[email-masked]> wrote:
> 
> <detnet-replication-jik.pptx>

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 13:37:33 +0800

Jouni,

You st me off thinking in another direction. I have a concern that I 
tried to describe in two slides I've added in front of your. The 
solution involves adding  DetNet Id label to the stack.

The issues arises when two pair of nodes agree on the same d-pw label
value.

Though I need to re-read draft-ietf-mpls-flow-ident to see if there is a
more efficient way of doing it.

I also have a question on what now is slide 5. WHy do we send a packet 
out over the same interface it came in over?

/Loa

On 2017-02-18 02:50, Jouni Korhonen wrote:
> _______________________________________________
> Detnet-dp-dt mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet-dp-dt
>

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 22:49:21 -0800

Additional thoughts from my side.

> On 17 Feb 2017, at 10:50, Jouni Korhonen <[email-masked]> wrote:
> 
> Folks,
> 

Assuming that after replication each outgoing PWs would have also different d-pw label makes the solution less hardware friendly. That would mean yet another mapping table and effectively doubling the label space needed for each PW.

With a single outgoing d-pw one could be effectively reusing (to most part) 1+1 redundancy if the existing implementation already has one.

- Jouni

> <detnet-frer-jik_v3.pptx>
> --

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 15:04:11 +0800

Jouni,

> With a single outgoing d-pw one could be effectively reusing (to most part) 1+1 redundancy if the existing implementation already has one.
What is the scope of "single" here? Do you mean that packet being 
replicated will have the same d-pw label? I thought that was necessary
otherwise we can identify duplicates.

If you mean that each node only uses one d-pw, I don't think that can
work.

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 23:11:34 -0800

Loa,

> On 18 Feb 2017, at 23:04, Loa Andersson <[email-masked]> wrote:
> 
> Jouni,
> 
> 
> 
> What is the scope of "single" here? Do you mean that packet being replicated will have the same d-pw label? I thought that was necessary

As in the slidesset the detnet/administrative domain. All x-PE devices would use the same d-pw. What I am saying above is that if we were using different d-pw labels stuff gets complicated even when replicating.

- JOuni

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 15:17:14 +0800

Jouni,

> As in the slidesset the detnet/administrative domain. All x-PE devices would use the same d-pw. What I am saying above is that if we were using different d-pw labels stuff gets complicated even whe…
I don't follow - let us say we have one Ethernet and one TDM PW crossing
the domain, id e use the same d-pw label, how do we know which one we
do Ethernet NSP and which one we do TDM NSP on??

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 23:17:55 -0800

Loa,

> On 18 Feb 2017, at 21:37, Loa Andersson <[email-masked]> wrote:
> 
> Jouni,
> 
> You st me off thinking in another direction. I have a concern that I tried to describe in two slides I've added in front of your. The solution involves adding  DetNet Id label to the stack.

No slides?

> The issues arises when two pair of nodes agree on the same d-pw label

Is this an issue with an _existing_ control plane? 

> I also have a question on what now is slide 5. WHy do we send a packet out over the same interface it came in over?

That was an automatic byproduct of the replication rule as it does not care which interface the packet came in. Also I think the ladder redundancy case needed to have packets going two directions and this type of “echoing” would do that. The content of a packet on A->B->C would be the same as coming from C->B->C.

- Jouni

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 15:21:55 +0800

slides

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 23:22:46 -0800

Loa,

> On 18 Feb 2017, at 23:17, Loa Andersson <[email-masked]> wrote:
> 
> Jouni,
> 
> 

Of course each e2e detnet flow have their own d-pw. Otherwise you cannot make a difference between them. Being the same means an e2e detnet flow would have a single d-pw label throughout the detnet/administrative domain in each x-PE.

- Jouni

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 15:26:37 +0800

On 2017-02-19 15:22, Jouni Korhonen wrote:
<snip>
> Of course each e2e detnet flow have their own d-pw. Otherwise you cannot make a difference between them. Being the same means an e2e detnet flow would have a single d-pw label throughout the detnet…
OK . so what you are saying is that replicated packets will have the 
same d-pw? I agree to that!

/Loa
> - Jouni
<snip>
--

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 23:27:03 -0800

Loa,

On slide 1 why would F and A assign the same d-pw label to different detnet flows? That does not obviously work.

Or do you mean the detnet flow was already replicated before it reached F or A?

- Jouni

> On 18 Feb 2017, at 23:21, Loa Andersson <[email-masked]> wrote:
> 
> slides
> 
> /Loa
> 
> <detnet-frer-loa.pptx>

---

From: Jouni Korhonen <[email-masked]>
Date: Sat, 18 Feb 2017 23:29:18 -0800

Loa,

> On 18 Feb 2017, at 23:26, Loa Andersson <[email-masked]> wrote:
> 
> 
> 
> OK . so what you are saying is that replicated packets will have the same d-pw? I agree to that!

Yes. 

- JOuni

---

From: Loa Andersson <[email-masked]>
Date: Sun, 19 Feb 2017 15:31:36 +0800

Jouni,

The we have the problem I tried to describe in the slides I just sent?
Or at lest I think so. I would be very happy to be wrong.

/Loa

--

---

From: Jiangyuanlong <[email-masked]>
Date: Tue, 21 Feb 2017 09:30:39 +0000

Hi all,

The intention of this slide is to provide an optional scheme that can support Detnet with a single layer of PW label. 
For the time being the slide demonstrates that it is possible in the data plane.
Your comments are very welcome.

Best regards,
Yuanlong

-----Original Message-----
From: Detnet-dp-dt [mailto:[email-masked]] On Behalf Of Jouni Korhonen
Sent: Sunday, February 19, 2017 3:29 PM
To: Loa Andersson
Cc: [email-masked]
Subject: Re: [Detnet-dp-dt] FRER example.. maybe a way forward? was Re: quick notes from call 2/14/15

Loa,

>>> I don't follow - let us say we have one Ethernet and one TDM PW 
>>> crossing the domain, id e use the same d-pw label, how do we know 
>>> which one we do Ethernet NSP and which one we do TDM NSP on??
>> 

Yes. 

- JOuni

---

From: Loa Andersson <[email-masked]>
Date: Wed, 22 Feb 2017 11:53:53 +0800

Yuanlong,

Very interesting, and I think we need to spend some time on this. I 
think you are right bringing up the ingress and egress single point of
failure. The drawback is that the CE's gets more complicated.

This is still MPLS, but I miss the label stacks to understand the
processing.

And I have one specific question on slide 8. CE1 sends two identical 
packets to PE1 and PE2, each PE asdd a sequence number, right?
After that PE1 sends it packet to the out port for PW1 and D-PW1 and
PE2 send its packet to the out port for PW2 and D-PW2.

Before the packets are send out on PW1 and PW2, filtering (elimination)
takes place. The sequence number is used for that? How do we make sure
that the sequence numbers generated by PE1 and PE2 are the same?

Also the filtering functions on PE1 will handle a pair of packets 
intended for PW1. One of the packets come from the AC in port and the
other from the P-PW1 in port, what is used coordinate the packets?

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 21 Feb 2017 20:02:13 -0800

A quick question. What is the difference between PW and D-PW in your slides?

- Jouni

> On Feb 21, 2017, at 1:30 AM, Jiangyuanlong <[email-masked]> wrote:
> 
> Hi all,
> 
> <detnet-one-PW.ppt>

---

From: Jiangyuanlong <[email-masked]>
Date: Wed, 22 Feb 2017 05:50:46 +0000

PW is normal PW, maybe the same as d-pw named by Loa; D-PW is a special PW set up between a pair of T-PEs (or S-PEs), not sure if it is similar to I-label as described by Balazs?
In my slide, d-pw will not be the same from one side of T-PE to the other side of T-PE, but change from one PW segment to another PW segment, just like in the MS-PW architecture.
But the label values of these PW segment can be allocated the same value (D-PW must be allocated a different value thus help to distinguish traffic from the normal PW).

Thanks,
Yuanlong

-----Original Message-----
From: Jouni Korhonen [mailto:[email-masked]] 
Sent: Wednesday, February 22, 2017 12:02 PM
To: Jiangyuanlong
Cc: Loa Andersson; [email-masked]
Subject: Re: An optional scheme that can support Detnet with a single layer of PW label

- Jouni

---

From: Jiangyuanlong <[email-masked]>
Date: Wed, 22 Feb 2017 05:56:42 +0000

Loa, please see my comments in line.

Thanks,
Yuanlong

-----Original Message-----
From: Detnet-dp-dt [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Wednesday, February 22, 2017 11:54 AM
To: Jiangyuanlong; Jouni Korhonen
Cc: [email-masked]
Subject: Re: [Detnet-dp-dt] An optional scheme that can support Detnet with a single layer of PW label

Yuanlong,

Very interesting, and I think we need to spend some time on this. I think you are right bringing up the ingress and egress single point of failure. The drawback is that the CE's gets more complicated.
[YJ] Thanks;)

This is still MPLS, but I miss the label stacks to understand the processing.

And I have one specific question on slide 8. CE1 sends two identical packets to PE1 and PE2, each PE asdd a sequence number, right?
After that PE1 sends it packet to the out port for PW1 and D-PW1 and
PE2 send its packet to the out port for PW2 and D-PW2.

Before the packets are send out on PW1 and PW2, filtering (elimination) takes place. The sequence number is used for that? How do we make sure that the sequence numbers generated by PE1 and PE2 are the same?
[YJ] Yes. If CE is 802.1CB compatible, we can use its sequence number directly. Otherwise, a Sequence Generator is needed on a PE attached with AC, how to generate the same seq on two PEs may pose a challenge. Maybe a hash module which takes a whole frame as an input can merely work.

Also the filtering functions on PE1 will handle a pair of packets intended for PW1. One of the packets come from the AC in port and the other from the P-PW1 in port, what is used coordinate the packets?
[YJ] there is no coordination, but the encapsulation will be the same (all in Ethernet for Ethernet frame).

/Loa

--

---

From: Jiangyuanlong <[email-masked]>
Date: Thu, 23 Feb 2017 07:58:14 +0000

Hi loa,

Your slides provide a good start point, it also made me think about the "Eliminate first and then replicate" scheme. 
In slide 4 & 5, you said:
"Eliminate duplicates in PW instance (indexed by d-pw).
Replicate d-pw after duplicate elimination towards C and D using L5 and L2."

Assume that we have an S-PE router with a switch fabric inside (given the high throughput in the typical aggregation and metro network, switch fabric is a reasonable piece imbedded in a router).
Assume that Eliminate module and Replicate module are separate modules. 
For a DetNet flow with a 10M bps, let us calculate what is the requirement on the switch fabric.
We have several options to implement Eliminate & Replicate in the S-PE1 (please see the attached slides):
1. "Eliminate first and then replicate", if we put both Eliminate module and Replicate module in some central processing unit near the switch fabric, then 4 times of the DetNet flow (i.e., 40Mbps) need to pass through the switch fabric. When d-pw1 is broken, the traffic from d-pw3 is looped back to S-PE2.
2. "Eliminate first and then replicate", if we put both Eliminate module and Replicate module in the port connecting to the other S-PE, then optimally only 2 times of the DetNet flow (i.e., 20Mbps) need to pass through the switch fabric. When d-pw1 is broken, the traffic from d-pw3 is also looped back to S-PE2.
3. "Replicate first and then Eliminate", we can put Replicate module in the ingress of pw1 and put Eliminate module in the egress to PW2, then 3 times of the DetNet flow (i.e., 30Mbps) need to pass through the switch fabric.
When d-pw1 is broken, no traffic from d-pw3 is looped back to S-PE2.

In summary, "Eliminate & replicate" scheme will influence the bandwidth consumed in the switch fabric and on d-pw3.

Thanks,
Yuanlong

-----Original Message-----
From: Detnet-dp-dt [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Sunday, February 19, 2017 3:22 PM
To: Jouni Korhonen
Cc: [email-masked]
Subject: Re: [Detnet-dp-dt] FRER example.. maybe a way forward? was Re: quick notes from call 2/14/15

slides

/Loa

On 2017-02-19 15:17, Jouni Korhonen wrote:
>> Though I need to re-read draft-ietf-mpls-flow-ident to see if there 
>> is a more efficient way of doing it.
>>
>>>> - Sequence number. We made a >>decision<< to settle down to 16 
>>>> bits. This is the most “compatible” approach. At high speeds it has 
>>>> been argued 16 bits is not enough. In practical implementations 
>>>> even 16 bits of sequence number won’t be maintained. It is 
>>>> typically a much smaller window of the entire sequence number space 
>>>> (think about numbers.. 1M PWs times the seqnum bit vector etc..)

--

---

From: Loa Andersson <[email-masked]>
Date: Thu, 23 Feb 2017 18:23:35 +0800

Yuanlong,

I still not sure about this, it looks very much like the element 
functional blocks that ITU-T SG15 used to do, but that IETF never
done. IETF leave this mostly up the different implementations.

Having said that, I realize that for an implementor this is important,
but just now I don't see how to fold it on to an IETF standard.

/Loa

--

---

From: Jiangyuanlong <[email-masked]>
Date: Thu, 23 Feb 2017 10:51:19 +0000

Hi Loa,

This discussion is not meant to include switch fabric in any IETF standard, but to show how it influences the choice of "Eliminate & replicate" scheme.
My opinion is Detnet DT can follow the PWE3 architecture, while some texts may be needed to explain how the "Eliminate & replicate" scheme works.

Cheers,
Yuanlong

-----Original Message-----
From: Loa Andersson [mailto:[email-masked]] 
Sent: Thursday, February 23, 2017 6:24 PM
To: Jiangyuanlong; Jouni Korhonen
Cc: [email-masked]
Subject: Re: [Detnet-dp-dt] FRER example.. maybe a way forward? was Re: quick notes from call 2/14/15

Yuanlong,

I still not sure about this, it looks very much like the element functional blocks that ITU-T SG15 used to do, but that IETF never done. IETF leave this mostly up the different implementations.

Having said that, I realize that for an implementor this is important, but just now I don't see how to fold it on to an IETF standard.

/Loa

On 2017-02-23 15:58, Jiangyuanlong wrote:
>>> The issues arises when two pair of nodes agree on the same d-pw 
>>> label value.
>>
>>>>> bits. This is the most “compatible” approach. At high speeds it 
>>>>> has been argued 16 bits is not enough. In practical 
>>>>> implementations even 16 bits of sequence number won’t be 
>>>>> maintained. It is typically a much smaller window of the entire 
>>>>> sequence number space (think about numbers.. 1M PWs times the 
>>>>> seqnum bit vector etc..)

--

---

From: Jiangyuanlong <[email-masked]>
Date: Sat, 25 Feb 2017 07:46:52 +0000

Hi Jouni,

Some further comments based on our recent discussions.

Thanks,
Yuanlong

From: Detnet-dp-dt [mailto:[email-masked]] On Behalf Of Jouni Korhonen
Sent: Thursday, February 16, 2017 3:37 AM
To: [email-masked]
Subject: [Detnet-dp-dt] quick notes from call 2/14/15

Agenda:

Discussion:
- Sequence number. We made a >>decision<< to settle down to 16 bits. This is the most “compatible” approach. At high speeds it has been argued 16 bits is not enough. In practical implementations even 16 bits of sequence number won’t be maintained. It is typically a much smaller window of the entire sequence number space (think about numbers.. 1M PWs times the seqnum bit vector etc..)
[YJ] Good to be compatible with the generic CW. We need to be careful to avoid sequence number wrap up. 32Mbps detnet flow is enough to overflow the sequence in a second (if 64 bytes/packet is assumed). If the "eliminate first and then replicate" mechanism is used, detnet traffic from a S-PE may be looped back to itself, thus there is also a risk of sequence overrun. "replicate first and then eliminate" will be more safe IMO.

- PWs and the label stack. See the latest slides from Loa and specifically the slide 9 (attached). I did add one slide to this deck, the slide 14.
- T-labels are typically per hop. L-labels are between DetNet aware S/P-PEs and essentially form an overlay over the underlying network. d-pw labels are end to end (at the moment.. to be discussed) between the T-PEs or in general between the DetNet aware end point that understand the detnet data plane.
[YJ] LSP is actually hop by hop allocated a label, though each hop can be overlaid in a tunnel itself. Thus T-labels and L-labels can be regarded as a single LSP label without loss of generality. PHP can be enabled optionally for this LSP.

- currently all d-pw (detnet PW labels) experience FRER if that functionality is enabled. d-pw labels are tied to sequence numbers (the detnet CW).
[YJ] Yes, d-pw label always go together with detnet CW.

- L-labels seen beneficial allowing the autoconfiguration of FRER i.e., build the overlay over the network and do not care configuring the PWs. This mimics one 802.1CB feature (see .1CB sub-clause 7.11).
[YJ] we don't need the L-label if MS-PW approach is taken, since each segment of PW can mimic the 802.1CB member stream.

- L-labels also allow simple label swap in a detnet S-PE i.e., no FRER would be applied. 
[YJ] we don't need the L-label. PW swap is the only action if no FRER is applied, this is the traditional MS-PW mechanism.

- This setup seems plausible. There was concerns overloading L-labels with some of the PW decision making in the fast path and thus possibly causing a lookup that needs to be done over two labels (L & d-pw). Essentially it is the L-label that signals whether the FRER gets applied.
[YJ] Using an L-label to trigger an S-PE looking into the CW inside a PW will be less clean compared with using a PW label directly.