Subject: [Detnet-dp-dt] detnet LSPs and PWs

From: Loa Andersson <[email-masked]>
Date: Mon, 13 Feb 2017 09:31:33 +0800

Folks,

The mail from Yuanlong mmade me go back and check the PW architecture 
and consult with Andy Malis and Stewart Bryant. So I have one more thing
that we should discuss on "Tuesday".

What Yaunlong said was: "IMHO, multiple layers of PW is a break from
the PWE3 architecture, and all DP/CP/MP things will become more
complicated."

It is correct that multi-layer pw's is problematic, though Andy said
that "if you have a good use case, you can do it".

The problem is that there is a native service processing (NSP) at the 
end of the PW. Multi-layer PWs will only do NSP at one level. I think
we should replace the MS-PW with an LSP. I've added one slide (9) and
change slide 8,9 and 11 in the earlier package. The other slides arere
for reference

I want Andy and Stewart to have a chance to review this prior to that
we commit too hard to it. Copied them.

/Loa
-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: Jouni Korhonen <[email-masked]>
Date: Sun, 12 Feb 2017 17:49:44 -0800

Thanks Loa bringing this up. I came to similar issue when writing down the draft text for T-PE, S-PE and forwarder descriptions. I did not think about having the d-pw go all way though (yet.. although that was part of the original Yokohama ‘proposal' in IETF 94 ;)

The excerpt from the Editor’s node in github text:

  <t>
   [Ed. Thinking of the DetNet label.. MAYBE we should just define a DetNet
   Control Word that keeps inside the e2e flow identification information what
   we now use the Detnet label for or change the ordering of the DetNet CW and
   the DetNet label. This would IMHO be a bit more cleaner approach - following
   the RFC5087 - and call it e.g., DetNet Adaptation Layer.  The current with
   DetNet label between the PW label and the CW does not seem to fit nicely
   into the existing PW architecture. On the other hand DetNet label could be used
   within the pipeline as an internal next-hop indicator.. I welcome discussion
   here..]
  </t>

- JOuni

--
Jouni Korhonen, Broadcom Ltd.
M: +1-408-391-7160

> On Feb 12, 2017, at 5:31 PM, Loa Andersson <[email-masked]> wrote:
> 
> Folks,
> 
> The mail from Yuanlong mmade me go back and check the PW architecture and consult with Andy Malis and Stewart Bryant. So I have one more thing
> The problem is that there is a native service processing (NSP) at the end of the PW. Multi-layer PWs will only do NSP at one level. I think
> <detnet-replication.pptx>_______________________________________________
> Detnet-dp-dt mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet-dp-dt

---

From: "Andrew G. Malis" <[email-masked]>
Date: Sun, 12 Feb 2017 23:11:39 -0500

Loa et al,

To be clear, there’s currently no definition of PWs encapsulated in PWs,
and while it might be conceptually possible, such as an Ethernet PW carried
within a SONET/SDH PW, I couldn’t imagine a use case for doing it as it’s
very inefficient, and I asked Loa if he had one. And if you were to do so,
each PW in the hierarchy would need NSP functionality and real or emulated
CE access circuits at the endpoints. Also, thinking about it some more, you
couldn’t have both PWs in the same label stack, since a PW emulates a
physical circuit. So there would need to be a separate label stack (and
MPLS LSP) inside the emulated circuit for the outer PW. By definition, PW
labels terminate a label stack.

As I haven’t been following Detnet at all, I don’t have the context for
what you’re trying to accomplish. That said, I’ll take a look at the slides
and let you know if I have any comments.

Cheers,
Andy

On Sun, Feb 12, 2017 at 8:31 PM, Loa Andersson <[email-masked]> wrote:

> Folks,
>
> The mail from Yuanlong mmade me go back and check the PW architecture and
> consult with Andy Malis and Stewart Bryant. So I have one more thing
> The problem is that there is a native service processing (NSP) at the end
> of the PW. Multi-layer PWs will only do NSP at one level. I think

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Feb 2017 10:55:55 +0000

Hi,

I was given some background information on your thoughts on detnet-PW, 
and pass on my thoughts in response.

I think NSP issue  is a red herring. NSP can be NULL.

An S-PE does no NSP, although in any case I suspect that you may need 
some processing function at the detnet-S-PE - see below.

The underlying DETNET PW is an SS-PW in the diagram I was shown, in that 
the PW label is the same end to end, although of course it need not be, 
you could have an equivalence label set and run pure MS-PW. Indeed when 
you have multiple administrations you would like them to be different 
for administrative purposes (that is why we designed MS-PW like that).

So if you create an equivalence relationship in the egress-PE, i.e. two 
entries in the global label table pointing to the same duplicate 
suppressor (sequencer), then you could use regular MS-PW for this.

If the S-PEs are in the same administrative domain in both ingress and 
egress, you can also use a single label value on egress and on egress 
since you can give them the same label mappings, i.e. they have 
identical swap instructions programmed into the L-FIB.

We don't have NSP at the S-PE's in the current PW architecture, in the 
data-plane it is essentially a simple MPLS LSR, swapping PW labels and 
forwarding the packet on a new LSP. What you will almost certainly want 
to do is to have the ability to replicate at nodes at the S-PEs, and 
that is new functionality.

An approach I would look at is as follows:

Create a new detnet-T-PE. On ingress this adds the sequence number, 
replicates and adds the PW label, which as I said above MAY be next hop 
detnet-S-PE dependent. Then it delivers the copies to the detnet-S-PEs 
over the LSPs. Now if you have an ECMP path between the detnet-T-PE and 
a detnet-S-PE, or you have SR or RSVP-TE available you can also deliver 
multiple copies to the detnet-S-PE and take advantage of the variability 
of transit time in the MPLS underlay.

Now you create a new detnet-S-PE that operates as follows. On it's 
ingress side it looks for the first packet at a given sequence number on 
this PW (or PW set) and suppresses all future packets on that sequence 
number on that PW or PW-set. It then replicates the packet if required, 
swaps the PW label (note that it may also use multiple outgoing labels) 
and send the packet over the egress LSP set.

At the egress T-PE it looks at the sequence number on this PW (or 
PW-set), trims all duplicates, applies any required egress processing 
and send the packet on it's way.

In summary on ingress a detnet-T-PE replicates to multiple S-PEs using 
the PW label the detnet-S-PE expects and potentially sends the packet 
over multiple paths to the detnet-S-PEs. At egress a detnet-T-PE looks 
at the sequence numbers across the detnet-PW set and selects the first 
of the sequence number suppressing all others, and sends the underlying 
packet on its way. A detnet-S-PE is a back to back detnet-egress-T-PE 
and a detnet-ingress-T-PE with a PW label swap in the middle and no 
other PW processing.

Now for the elephant in the corner of all of the schemes I have seen. If 
you have multiple paths to an X-PE, packets will likely arrive on 
different line cards. Sequence number co-ordination amongst different 
line cards, and at high speed even amongst different ports on the same 
line card is a hard problem. Indeed depending on the pipeline design on 
the line card, ANY sequence number processing can be hard. You could 
mitigate this (at the cost of availability) by requiring a common 
ingress port at any detnet X-PE. This would normally require an RSVP-TE 
or SR underlay.

- Stewart

On 13/02/2017 04:11, Andrew G. Malis wrote:
> Loa et al,
>
> To be clear, there’s currently no definition of PWs encapsulated in 
> PWs, and while it might be conceptually possible, such as an Ethernet 
> PW carried within a SONET/SDH PW, I couldn’t imagine a use case for 
> doing it as it’s very inefficient, and I asked Loa if he had one. And 
> if you were to do so, each PW in the hierarchy would need NSP 
> functionality and real or emulated CE access circuits at the 
> endpoints. Also, thinking about it some more, you couldn’t have both 
> PWs in the same label stack, since a PW emulates a physical circuit. 
> So there would need to be a separate label stack (and MPLS LSP) inside 
> the emulated circuit for the outer PW. By definition, PW labels 
> terminate a label stack.
>
> As I haven’t been following Detnet at all, I don’t have the context 
> for what you’re trying to accomplish. That said, I’ll take a look at 
> the slides and let you know if I have any comments.
>
> Cheers,
> Andy
>
>
> On Sun, Feb 12, 2017 at 8:31 PM, Loa Andersson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Folks,
>
>     The mail from Yuanlong mmade me go back and check the PW
>     architecture and consult with Andy Malis and Stewart Bryant. So I
>     have one more thing
>     The problem is that there is a native service processing (NSP) at
>     the end of the PW. Multi-layer PWs will only do NSP at one level.
>     I think
>     <mailto:[email-masked]>
>     Senior MPLS Expert [email-masked] <mailto:[email-masked]>
>     <tel:%2B46%20739%2081%2021%2064>
>
>

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 13 Feb 2017 06:58:13 -0500

On Mon, Feb 13, 2017 at 5:55 AM, Stewart Bryant <[email-masked]>
wrote:

> you have multiple paths to an X-PE, packets will likely arrive on different
> line cards. Sequence number co-ordination amongst different line cards, and
> at high speed even amongst different ports on the same line card is a hard
> problem. Indeed depending on the pipeline design on the line card, ANY
> sequence number processing can be hard. You could mitigate this (at the
> cost of availability) by requiring a common ingress port at any detnet
> X-PE. This would normally require an RSVP-TE or SR underlay.
>
I would like to second Stewart’s comment above. Back in the early days of
PWs, some vendors implemented PWs without a sequence number because they
couldn’t do sequence number processing across multiple parallel forwarding
engines even on the same physical port, as the forwarding engines didn’t
have common shared memory. Of course, their customers had to not use ECMP,
at least for those PWs. I think most vendors have gotten past that these
days, but you can never tell. But keep these possible limitations in mind
as you design Detnet.

Cheers,
Andy

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Feb 2017 12:56:45 +0000

Talking of corner residing elephants how to make sequence number 
processing atomic at the egress detnet-T-PE?

This is not a problem at the S-PEs because it does not matter if two 
copies get through but it is critical at the egress T-PE.

Also what do you plan to do if a later packet overtakes an earlier one? 
Presumably declare all the late delivery packets as "lost" rather than 
attempt to re-order.

- Stewart

On 13/02/2017 10:55, Stewart Bryant wrote:
> The underlying DETNET PW is an SS-PW in the diagram I was shown, in 
> that the PW label is the same end to end, although of course it need 
> not be, you could have an equivalence label set and run pure MS-PW. 
> Indeed when you have multiple administrations you would like them to 
> be different for administrative purposes (that is why we designed 
> MS-PW like that).
>
> So if you create an equivalence relationship in the egress-PE, i.e. 
> two entries in the global label table pointing to the same duplicate 
> forwarding the packet on a new LSP. What you will almost certainly 
> want to do is to have the ability to replicate at nodes at the S-PEs, 
> and that is new functionality.
>
> replicates and adds the PW label, which as I said above MAY be next 
> hop detnet-S-PE dependent. Then it delivers the copies to the 
> detnet-S-PEs over the LSPs. Now if you have an ECMP path between the 
> detnet-T-PE and a detnet-S-PE, or you have SR or RSVP-TE available you 
> can also deliver multiple copies to the detnet-S-PE and take advantage 
> of the variability of transit time in the MPLS underlay.
>
> ingress side it looks for the first packet at a given sequence number 
> on this PW (or PW set) and suppresses all future packets on that 
> sequence number on that PW or PW-set. It then replicates the packet if 
> required, swaps the PW label (note that it may also use multiple 
> outgoing labels) and send the packet over the egress LSP set.
>
> of the sequence number suppressing all others, and sends the 
> underlying packet on its way. A detnet-S-PE is a back to back 
> detnet-egress-T-PE and a detnet-ingress-T-PE with a PW label swap in 
> the middle and no other PW processing.
>
> Now for the elephant in the corner of all of the schemes I have seen. 
> If you have multiple paths to an X-PE, packets will likely arrive on 
> line card is a hard problem. Indeed depending on the pipeline design 
> on the line card, ANY sequence number processing can be hard. You 
> could mitigate this (at the cost of availability) by requiring a 
> common ingress port at any detnet X-PE. This would normally require an 
> RSVP-TE or SR underlay.
>
> - Stewart
>
>
>
>> So there would need to be a separate label stack (and MPLS LSP) 
>> inside the emulated circuit for the outer PW. By definition, PW

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Feb 2017 13:02:28 +0000

On 13/02/2017 11:58, Andrew G. Malis wrote:
>
> On Mon, Feb 13, 2017 at 5:55 AM, Stewart Bryant 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Now for the elephant in the corner of all of the schemes I have
>     seen. If you have multiple paths to an X-PE, packets will likely
>     arrive on different line cards. Sequence number co-ordination
>     amongst different line cards, and at high speed even amongst
>     different ports on the same line card is a hard problem. Indeed
>     depending on the pipeline design on the line card, ANY sequence
>     number processing can be hard. You could mitigate this (at the
>     cost of availability) by requiring a common ingress port at any
>     detnet X-PE. This would normally require an RSVP-TE or SR underlay.
>
> I would like to second Stewart’s comment above. Back in the early days 
> of PWs, some vendors implemented PWs without a sequence number because 
> they couldn’t do sequence number processing across multiple parallel 
> forwarding engines even on the same physical port, as the forwarding 
> engines didn’t have common shared memory. Of course, their customers 
> had to not use ECMP, at least for those PWs. I think most vendors have 
> gotten past that these days, but you can never tell. But keep these 
> possible limitations in mind as you design Detnet.
>
> Cheers,
> Andy
>

Andy it is not common shared memory but the pipelining and the intercard 
communication that stops it working.

As you know we prohibit ECMP on most PWs via the control word, with the 
exception of the Ethernet PW where it is optional and in place for 
historic reasons. However an Ethernet PW can re-order and as such does 
not fully comply with the Ethernet spec.

The problem is still live - which is why we need the packet marking 
scheme for packet loss measurement.

- Stewart

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 13 Feb 2017 09:35:10 -0500

Stewart,

In at least one implementation I’m familiar with, lack of common shared
memory between parallel microprogrammed forwarding engines for the same
physical port was the issue. I can see lack of intercard or interport
communications also being a problem.

We added the control word and sequence number to Ethernet PWs later on, and
I believe most implementations today do use it to reorder and comply with
the Ethernet spec.

Cheers,
Andy

On Mon, Feb 13, 2017 at 8:02 AM, Stewart Bryant <[email-masked]>
wrote:

> exception of the Ethernet PW where it is optional and in place for historic
> reasons. However an Ethernet PW can re-order and as such does not fully
> comply with the Ethernet spec.
>
> The problem is still live - which is why we need the packet marking scheme
> for packet loss measurement.
>
> - Stewart
>

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Feb 2017 15:23:18 +0000

Andy

I think we need to ask around about this, because I have never run into 
a mainstream box that supported s/n, much less re-order. The normal 
process was to strip off the PW label and CW only checking for the ACH, 
and then forward the packet straight out of the interface.

I just did a quick look at a couple of config manuals and there seemed 
to be no CW related options other than include it or not.

As I said, implementing other than that is actually very hard in a high 
speed interface.

Clearly we need to check.

Stewart

On 13/02/2017 14:35, Andrew G. Malis wrote:
> Stewart,
>
> In at least one implementation I’m familiar with, lack of common 
> shared memory between parallel microprogrammed forwarding engines for 
> the same physical port was the issue. I can see lack of intercard or 
> interport communications also being a problem.
>
> We added the control word and sequence number to Ethernet PWs later 
> on, and I believe most implementations today do use it to reorder and 
> On Mon, Feb 13, 2017 at 8:02 AM, Stewart Bryant 
>>         Now for the elephant in the corner of all of the schemes I
>>         have seen. If you have multiple paths to an X-PE, packets
>>         will likely arrive on different line cards. Sequence number
>>         co-ordination amongst different line cards, and at high speed
>>         even amongst different ports on the same line card is a hard
>>         problem. Indeed depending on the pipeline design on the line
>>         card, ANY sequence number processing can be hard. You could
>>         mitigate this (at the cost of availability) by requiring a
>>         common ingress port at any detnet X-PE. This would normally
>>         require an RSVP-TE or SR underlay.
>>
>>     I would like to second Stewart’s comment above. Back in the early
>>     days of PWs, some vendors implemented PWs without a sequence
>>     number because they couldn’t do sequence number processing across
>>     multiple parallel forwarding engines even on the same physical
>>     port, as the forwarding engines didn’t have common shared memory.
>>     Of course, their customers had to not use ECMP, at least for
>>     those PWs. I think most vendors have gotten past that these days,
>>     but you can never tell. But keep these possible limitations in
>>     mind as you design Detnet.
>>
>>     Cheers,
>>     Andy
>>
>
>     Andy it is not common shared memory but the pipelining and the
>     intercard communication that stops it working.
>
>     As you know we prohibit ECMP on most PWs via the control word,
>     with the exception of the Ethernet PW where it is optional and in
>     place for historic reasons. However an Ethernet PW can re-order
>     and as such does not fully comply with the Ethernet spec.
>
>     The problem is still live - which is why we need the packet
>     marking scheme for packet loss measurement.
>
>     - Stewart
>
>

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 14 Feb 2017 13:18:38 -0800

Loa,

In the updated slide 9 the node B is receiving both L1 and d-pw labels after the php, right?

- Jouni

---

From: Loa Andersson <[email-masked]>
Date: Wed, 15 Feb 2017 12:45:27 +0800

Folks,

Actually I think (except for the nomenclature, which I think we should
adopt) of what Stewart says is there in the new slides.

There are a few "if" that I don't agree to a design (maybe it may be
possible to collapse the design), e.g. assuming one single domain.

I think there are some design decision that need to be there.

- replication and elimination of packets at places/nodes that the
   operator can control, i.e. not every node should do it
- with an "S-detnet-PE" in the path (for replication/elimination) we
   need three levels of labels. The implication is that if we don't do
   replication/elimination other than at the T-detnet-PWs then we only
   need two layers.
- what we called "d-pw" is not a new pw, but a pw from the PWE3/PALS
   inventory

Now that last bullet is a problem, the sequence number is 16 bits in
PWE3/PALS and we know that is to small. Maybe we have to define a detnet
PW after all. A PW that carries sequence number + another PW.

I've been  thinking hard about the equivalence relationship, but is 
still not sure it will work in the generic case, equivalence
relationship tells node D that it should treat L1 from B and L2 from C
the same, which is fine, but it does still not tell us that the packet
came from A originally.

BTW - I think we need some type of sign-off from Stewart/Andy before we
commit to hard.

Also the problem of comparing sequence numbers arriving over
different physical interface seems to be a eal problem.

/Loa

On 2017-02-13 18:55, Stewart Bryant wrote:
>>     <mailto:[email-masked]>
> _______________________________________________

--

---

From: Loa Andersson <[email-masked]>
Date: Wed, 15 Feb 2017 12:56:19 +0800

Jouni.

> In the updated slide 9 the node B is receiving both L1 and d-pw labels after the php, right?

Yes, but I thought that it was in the slides I sent. Can you share the
updated slides?

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 14 Feb 2017 21:00:16 -0800

Loa,

> On 14 Feb 2017, at 20:45, Loa Andersson <[email-masked]> wrote:
> 
> Folks,
> 

It was never assumed every node would do it - or even ever S-PE on path. 

>  replication/elimination other than at the T-detnet-PWs then we only

Could you elaborate more.. 

> - what we called "d-pw" is not a new pw, but a pw from the PWE3/PALS

Ok.

> PW after all. A PW that carries sequence number + another PW.

Erm why? One can define a new control word. The merge + replication + elimination alone done on PW instance is big enough change to add other tweaks there, like bigger seqnums.

> I've been  thinking hard about the equivalence relationship, but is still not sure it will work in the generic case, equivalence

I do not see this as an issue if separate port are attached to e.g., same switch soc.  How separate physical ports end up to the same PW instance is an implementation issue. There are non-trivial issues if the separate physical ports are in different blades or say in different switch socs.

- Jouni

---

From: Norman Finn <[email-masked]>
Date: Wed, 15 Feb 2017 05:45:27 +0000

There is certainly a problem processing sequence numbers at very high speed.  Unless/until the hardware is able to do that, this technique probably won't be used at really high speed.  You can't do what you can't do.  In the meantime, it can be very useful for lower speeds.

Discarding out-of-order packets doesn't makes the whole exercise pointless, but does decrease its utility.  There are two major classes of use cases known:

 1. "Intermittent flows": Where the difference in latency between paths is smaller than the interval between transmissions.  This is a typical case in machine control applications.  In this case, the long-path repeated packet is received before the next short-path packet.  Discarding out-of-order only has the usual issues with reset transmitters.

 2. "Bulk flows":  Where there are several packets more in flight along the slow path than the fast path.  This is a typical case in super high-definition video applications.  Tossing out-of-order packets makes the packet replication/elimination pointless; The object is to receive every packet, even if out-of-order.

-- Norm

________________________________
From: Detnet-dp-dt [[email-masked]] on behalf of Stewart Bryant [[email-masked]]
Sent: Monday, February 13, 2017 4:56 AM
To: Andrew G. Malis; Loa Andersson
Cc: [email-masked]
Subject: Re: [Detnet-dp-dt] detnet LSPs and PWs

Talking of corner residing elephants how to make sequence number processing atomic at the egress detnet-T-PE?

This is not a problem at the S-PEs because it does not matter if two copies get through but it is critical at the egress T-PE.

Also what do you plan to do if a later packet overtakes an earlier one? Presumably declare all the late delivery packets as "lost" rather than attempt to re-order.

- Stewart

Hi,

I was given some background information on your thoughts on detnet-PW, and pass on my thoughts in response.

An S-PE does no NSP, although in any case I suspect that you may need some processing function at the detnet-S-PE - see below.

The underlying DETNET PW is an SS-PW in the diagram I was shown, in that the PW label is the same end to end, although of course it need not be, you could have an equivalence label set and run pure MS-PW. Indeed when you have multiple administrations you would like them to be different for administrative purposes (that is why we designed MS-PW like that).

So if you create an equivalence relationship in the egress-PE, i.e. two entries in the global label table pointing to the same duplicate suppressor (sequencer), then you could use regular MS-PW for this.

If the S-PEs are in the same administrative domain in both ingress and egress, you can also use a single label value on egress and on egress since you can give them the same label mappings, i.e. they have identical swap instructions programmed into the L-FIB.

We don't have NSP at the S-PE's in the current PW architecture, in the data-plane it is essentially a simple MPLS LSR, swapping PW labels and forwarding the packet on a new LSP. What you will almost certainly want to do is to have the ability to replicate at nodes at the S-PEs, and that is new functionality.

Create a new detnet-T-PE. On ingress this adds the sequence number, replicates and adds the PW label, which as I said above MAY be next hop detnet-S-PE dependent. Then it delivers the copies to the detnet-S-PEs over the LSPs. Now if you have an ECMP path between the detnet-T-PE and a detnet-S-PE, or you have SR or RSVP-TE available you can also deliver multiple copies to the detnet-S-PE and take advantage of the variability of transit time in the MPLS underlay.

Now you create a new detnet-S-PE that operates as follows. On it's ingress side it looks for the first packet at a given sequence number on this PW (or PW set) and suppresses all future packets on that sequence number on that PW or PW-set. It then replicates the packet if required, swaps the PW label (note that it may also use multiple outgoing labels) and send the packet over the egress LSP set.

At the egress T-PE it looks at the sequence number on this PW (or PW-set), trims all duplicates, applies any required egress processing and send the packet on it's way.

In summary on ingress a detnet-T-PE replicates to multiple S-PEs using the PW label the detnet-S-PE expects and potentially sends the packet over multiple paths to the detnet-S-PEs. At egress a detnet-T-PE looks at the sequence numbers across the detnet-PW set and selects the first of the sequence number suppressing all others, and sends the underlying packet on its way. A detnet-S-PE is a back to back detnet-egress-T-PE and a detnet-ingress-T-PE with a PW label swap in the middle and no other PW processing.

Now for the elephant in the corner of all of the schemes I have seen. If you have multiple paths to an X-PE, packets will likely arrive on different line cards. Sequence number co-ordination amongst different line cards, and at high speed even amongst different ports on the same line card is a hard problem. Indeed depending on the pipeline design on the line card, ANY sequence number processing can be hard. You could mitigate this (at the cost of availability) by requiring a common ingress port at any detnet X-PE. This would normally require an RSVP-TE or SR underlay.

- Stewart

To be clear, there’s currently no definition of PWs encapsulated in PWs, and while it might be conceptually possible, such as an Ethernet PW carried within a SONET/SDH PW, I couldn’t imagine a use case for doing it as it’s very inefficient, and I asked Loa if he had one. And if you were to do so, each PW in the hierarchy would need NSP functionality and real or emulated CE access circuits at the endpoints. Also, thinking about it some more, you couldn’t have both PWs in the same label stack, since a PW emulates a physical circuit. So there would need to be a separate label stack (and MPLS LSP) inside the emulated circuit for the outer PW. By definition, PW labels terminate a label stack.

As I haven’t been following Detnet at all, I don’t have the context for what you’re trying to accomplish. That said, I’ll take a look at the slides and let you know if I have any comments.

Cheers,
Andy

On Sun, Feb 12, 2017 at 8:31 PM, Loa Andersson <[email-masked]<mailto:[email-masked]>> wrote:
Folks,

/Loa
--

Loa Andersson                        email: [email-masked]<mailto:[email-masked]>
Senior MPLS Expert                          [email-masked]<mailto:[email-masked]>
Huawei Technologies (consultant)     phone: +46 739 81 21 64<tel:%2B46%20739%2081%2021%2064>

---

From: Loa Andersson <[email-masked]>
Date: Wed, 15 Feb 2017 13:50:37 +0800

Jouni,

inline pl.ease

> It was never assumed every node would do it - or even ever S-PE on path.

True, but we have spelled it out, and if this is a data plane decision, 
how do you prevent a node capable of doing it to do it?
> Could you elaborate more..

which part? the egress "T-detnet-PE" will know by the label where the
packet came from
T-label says it came from the immediate neighbor
L-label says it came from its S-detnet-PE neighbor
d-pw label says it came from the T-detnet-PE peer

> Erm why? One can define a new control word. The merge + replication + elimination alone done on PW instance is big enough change to add other tweaks there, like bigger seqnums.
I think we need to discuss this, I'm not sure we can change the control
word in an existing PW, or if we want to do that it will have
implications outside detnet.

> I do not see this as an issue if separate port are attached to e.g., same switch soc.  How separate physical ports end up to the same PW instance is an implementation issue. There are non-trivial i…
So you are saying, it is sometimes a problem, sometimes not?

/Loa

--

---

From: Jouni Korhonen <[email-masked]>
Date: Tue, 14 Feb 2017 21:58:29 -0800

Hi Loa,

> On 14 Feb 2017, at 21:50, Loa Andersson <[email-masked]> wrote:
> 
> Jouni,
> 
> inline pl.ease
> 
> 
> True, but we have spelled it out, and if this is a data plane decision, how do you prevent a node capable of doing it to do it?

By configuration. It is not an automatic function, but a purposely made decision.

> d-pw label says it came from the T-detnet-PE peer

ok. gotcha.

> implications outside detnet.

Now I do not follow which “existing PW” in this context. Do you mean that you would terminate a PW with DetNet flavor to a T/S-PE that has no idea of DetNet? I doubt that would work.

> So you are saying, it is sometimes a problem, sometimes not?

One can always come up with a node architecture that woudl have issues here at high or moderate speeds. That should not prevent us defining a solution that can  perform just fine at multi-tera speeds.

- Jouni

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Feb 2017 08:49:31 +0000

>   operator can control, i.e. not every node should do it

Sure, a regular S-PE would work like a regular S-PE and simply
forward all packets on the PW to the x-PE it was configured
to forward to.

>   replication/elimination other than at the T-detnet-PWs then we only

I am not convinced that you need three labels. As I explained you can
do it with a PW over an LSP if you create a new PW or PW set if needed.

Apart from Ethernet, what types of packet do you anticipate carrying?

> - what we called "d-pw" is not a new pw, but a pw from the PWE3/PALS

I am not sure that makes sense.

> PW after all. A PW that carries sequence number + another PW.

So, you really don't want to use the existing Ethernet s/n in this 
anyway. It
has a horrible skip zero count. What I would think about is to follow
the CW with a 32bit counter. There are PWs that have an 8 octet CW.
You need to retain the first nibble design and the length design (to allow
for short packets), so I would use something like:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |0 0 0 0| Flags |FRG|  Length   | Reserved                      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                     Sequence Number                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Then do OAM with the normal 0001 GACH

> the same, which is fine, but it does still not tell us that the packet

You know from the PW label which PW it is. PWs are connection oriented.
In a MS-PW you may not know that A is A since you only really know about
the last hop S-PE.

However I am not sure I understand the underlying problem.

> different physical interface seems to be a eal problem.

Yes, that is what worries me most.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Feb 2017 09:02:41 +0000

On 15/02/2017 05:45, Norman Finn wrote:
> There is certainly a problem processing sequence numbers at very high 
> speed.  Unless/until the hardware is able to do that, this technique 
> probably won't be used at really high speed.  You can't do what you 
> can't do. In the meantime, it can be very useful for lower speeds.

If you can force the traffic to arrive at the same physical interface, 
for example by SR or RSVP-TE you are better off. I am not sure how this 
scales because you have to have the s/n in RAM that the forwarder can 
get to quickly, but this would be OK at the egress T-PE since you are 
bounded on the number of PWs, although I suppose that could be # ports * 
# VLANs

Depending on the PE architecture we might be able to do egress 
processing, although that is not a common feature in the platforms I 
have looked at in the past.

>
> Discarding out-of-order packets doesn't makes the whole exercise 
> pointless, but does decrease its utility.  There are two major classes 
> of use cases known:
>
>  1. "Intermittent flows": Where the difference in latency between 
> paths is smaller than the interval between transmissions.  This is a 
> typical case in machine control applications.  In this case, the 
> long-path repeated packet is received before the next short-path 
> packet.  Discarding out-of-order only has the usual issues with reset 
> transmitters.
>
>  2. "Bulk flows":  Where there are several packets more in flight 
> along the slow path than the fast path.  This is a typical case in 
> super high-definition video applications. Tossing out-of-order packets 
> makes the packet replication/elimination pointless; The object is to 
> receive every packet, even if out-of-order.

So in case 2, it is possibly better  to deliver up all the packets and 
let the device do the discard. In video there is usually a sequence 
number system that would permit that, assuming it was nit confused by 
the abnormally high number of duplicates.

Stewart

>
> -- Norm
>
> ------------------------------------------------------------------------
> *From:* Detnet-dp-dt [[email-masked]] on behalf of 
> Stewart Bryant [[email-masked]]
> *Sent:* Monday, February 13, 2017 4:56 AM
> *To:* Andrew G. Malis; Loa Andersson
> *Cc:* [email-masked]
> *Subject:* Re: [Detnet-dp-dt] detnet LSPs and PWs
>
> Also what do you plan to do if a later packet overtakes an earlier 
> one? Presumably declare all the late delivery packets as "lost" rather 
> than attempt to re-order.
>
> - Stewart
>
>
>> I was given some background information on your thoughts on 
>> detnet-PW, and pass on my thoughts in response.
>>
>> If the S-PEs are in the same administrative domain in both ingress 
>> and egress, you can also use a single label value on egress and on 
>> egress since you can give them the same label mappings, i.e. they 
>> have identical swap instructions programmed into the L-FIB.
>>
>> We don't have NSP at the S-PE's in the current PW architecture, in 
>> the data-plane it is essentially a simple MPLS LSR, swapping PW 
>> labels and forwarding the packet on a new LSP. What you will almost 
>> certainly want to do is to have the ability to replicate at nodes at 
>> the S-PEs, and that is new functionality.
>>
>> detnet-T-PE and a detnet-S-PE, or you have SR or RSVP-TE available 
>> you can also deliver multiple copies to the detnet-S-PE and take 
>> advantage of the variability of transit time in the MPLS underlay.
>>
>> sequence number on that PW or PW-set. It then replicates the packet 
>> if required, swaps the PW label (note that it may also use multiple 
>> In summary on ingress a detnet-T-PE replicates to multiple S-PEs 
>> using the PW label the detnet-S-PE expects and potentially sends the 
>> packet over multiple paths to the detnet-S-PEs. At egress a 
>> detnet-T-PE looks at the sequence numbers across the detnet-PW set 
>> and selects the first of the sequence number suppressing all others, 
>> and sends the underlying packet on its way. A detnet-S-PE is a back 
>> to back detnet-egress-T-PE and a detnet-ingress-T-PE with a PW label 
>> swap in the middle and no other PW processing.
>>
>> line cards, and at high speed even amongst different ports on the 
>> same line card is a hard problem. Indeed depending on the pipeline 
>> design on the line card, ANY sequence number processing can be hard. 
>> You could mitigate this (at the cost of availability) by requiring a 
>> common ingress port at any detnet X-PE. This would normally require 
>> an RSVP-TE or SR underlay.
>>
>> - Stewart
>>
>>
>>
>>> PWs, and while it might be conceptually possible, such as an 
>>> Ethernet PW carried within a SONET/SDH PW, I couldn’t imagine a use 
>>> case for doing it as it’s very inefficient, and I asked Loa if he 
>>> had one. And if you were to do so, each PW in the hierarchy would 
>>> need NSP functionality and real or emulated CE access circuits at 
>>> the endpoints. Also, thinking about it some more, you couldn’t have 
>>> both PWs in the same label stack, since a PW emulates a physical 
>>> circuit. So there would need to be a separate label stack (and MPLS 
>>> LSP) inside the emulated circuit for the outer PW. By definition, PW 
>>>     architecture and consult with Andy Malis and Stewart Bryant. So
>>>     I have one more thing
>>>     The problem is that there is a native service processing (NSP)
>>>     at the end of the PW. Multi-layer PWs will only do NSP at one
>>>     level. I think
>>>     Loa Andersson                        email:
>>>     [email-masked] <mailto:[email-masked]>

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Feb 2017 09:22:34 +0000

On 15/02/2017 05:50, Loa Andersson wrote:
> True, but we have spelled it out, and if this is a data plane 
> decision, how do you prevent a node capable of doing it to do it?

It would be a property of the PW at the x-PE.

There is no PW that can do replication at the moment. We abandoned 
P-MP-MS PWs as not having a use case, and as I remember P-MP-SS-PWs rely 
on the underlay to do the replication.

> d-pw label says it came from the T-detnet-PE peer

I guess the reason you need this is because you do not want to use a PW 
group.

You need to think about whether it is better to include and process more 
labels
or to design a PW group, where several labels refer to different paths 
for the
same PW.

As I said earlier, PWs are connections, so given the PW you can identify
the origin.

>> Erm why? One can define a new control word. The merge + replication + 
>> elimination alone done on PW instance is big enough change to add 
>> other tweaks there, like bigger seqnums.
>>

So is your assumption that the T-PEs are unchangable? If not it is 
better to design a new PW with the exact properties you need. At the 
moment I cannot see how you could use an unmodified T-PE.

Also how many base types do you expect to need?

>> I do not see this as an issue if separate port are attached to e.g., 
>> same switch soc.  How separate physical ports end up to the same PW 
>> instance is an implementation issue. There are non-trivial issues if 
>> the separate physical ports are in different blades or say in 
>> different switch socs.
>>

*IF* you can channel the PW through the same processor, and that has 
fast enough access to the s/n registry, then this works, but it does so 
at a cost of scaling and reliability.

The n/w operators will not be happy with this, and we probably need OAM 
to verify it is happening, but if the core facing egress T-PE ports that 
service the LSP set that service a particular set of PWs all go through 
the same forwarder, then it can be made to work. However this is at a 
cost of availability, generality, scaling and opex.

If that meets you requirements, I am happy, I am just doing due 
diligence here.

- Stewart

>>>> that
>>>> two
>>>> want
>>>> hop
>>>> underlying
>>>> seen. If
>>>>> inside

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Feb 2017 09:23:47 +0000

I think this lines up with what I just said, so we agree.

S

---

From: Jouni Korhonen <[email-masked]>
Date: Wed, 15 Feb 2017 10:01:11 -0800

> *IF* you can channel the PW through the same processor, and that has fast enough access to the s/n registry, then this works, but it does so at a cost of scaling and reliability.

When we are talking real high speed implementations, I do not necessarily see a need for a GPP to be involved. Scaling I do not sign, reliability yes. If the node/blade goes down with the SN dbs, they are gone. But on the other hand, in that case the PWs in general are gone terminating the failing node and need to be re-established.

> The n/w operators will not be happy with this, and we probably need OAM to verify it is happening, but if the core facing egress T-PE ports that service 

I do not understand the OAM verifying comment.

> the LSP set that service a particular set of PWs all go through the same forwarder, then it can be made to work. However this is at a cost of availability, generality, scaling and opex.

Eventually a detnet flow that experienced replication always need to go through a single point (e.g. a S-PE or T-OE forwarder) in order to merge detnet flows and eliminate duplicates properly. One can have multiple T-PE egress nodes (for reliability for example) but then it will the detnet end station that needs to do the final merge and elimination.

- Jouni

> If that meets you requirements, I am happy, I am just doing due diligence here.
> 
> - Stewart

---

From: Jouni Korhonen <[email-masked]>
Date: Wed, 15 Feb 2017 10:02:25 -0800

Great!

- Jouni

> On Feb 15, 2017, at 1:23 AM, Stewart Bryant <[email-masked]> wrote:
> 
>

---

From: "Andrew G. Malis" <[email-masked]>
Date: Wed, 15 Feb 2017 13:48:19 -0500

Loa,

As I said earlier, I really don’t have any of the Detnet background, but
interpolating from what I have seen lately, is the general idea to send out
multiple replicas of a packet along different ECMP paths, and then use the
first one to arrive, dropping later duplicates?

Do you expect that a 16-bit sequence number would wrap around before all of
the duplicates have arrived? Note that this was not the original purpose of
the PW sequence number, we originally expected that all packets would take
the same route through the network, but there could be occasional out of
order packets if routing should change, for example if there were a node or
link outage followed by an MPLS fast reroute of the underlying traffic
tunnel. The sequence number would be used to reorder such out of order
packets. The PW generic control word was designed to prevent automatic ECMP
in routers by making sure that the first nibble of the PW payload isn’t a 4
or 6, thus looking like IPv4 or IPv6 to an interior router that’s just
doing ECMP.

However, there are some cases where ECMP can be used with a PW, see RFC
6391 for further details.

There is no requirement that all PWs use the same control word format, for
example, the TDM PWs have their own control word different from the generic
one. So if a Detnet PW were to be defined, it could, for example, have a
sequence number of whatever length is required for the application.

I doubt that you really want "A PW that carries sequence number + another
PW”, that sounds very inefficient.

MPLS traffic tunnels can also deliberately use ECMP, see RFC 6790.

Cheers,
Andy

On Tue, Feb 14, 2017 at 11:45 PM, Loa Andersson <[email-masked]> wrote:

> I've been  thinking hard about the equivalence relationship, but is still
> not sure it will work in the generic case, equivalence

---

From: Loa Andersson <[email-masked]>
Date: Fri, 17 Feb 2017 18:37:48 +0800

Andy,

My background is mostly MPLS, while DetNet is also strongly influenced 
by 802. I hope that someone from that side of the house can chime in.

On 2017-02-16 02:48, Andrew G. Malis wrote:
> interpolating from what I have seen lately, is the general idea to send
> out multiple replicas of a packet along different ECMP paths, and then
> use the first one to arrive, dropping later duplicates?

In general - yes! Though equal cost is not a strict requirement, but we 
will likely look at routing info to find sufficient paths.
>
> Do you expect that a 16-bit sequence number would wrap around before all
> of the duplicates have arrived?

We discussed that at the last dt-meeting, and Norm put up figures that
convinced the people that early said 16 bits is not enough, that it will
be.
  Note that this was not the original
> purpose of the PW sequence number, we originally expected that all
> packets would take the same route through the network, but there could
> be occasional out of order packets if routing should change, for example
> if there were a node or link outage followed by an MPLS fast reroute of
> the underlying traffic tunnel. The sequence number would be used to
> reorder such out of order packets. The PW generic control word was
> designed to prevent automatic ECMP in routers by making sure that the
> first nibble of the PW payload isn’t a 4 or 6, thus looking like IPv4 or
> IPv6 to an interior router that’s just doing ECMP.
>
> There is no requirement that all PWs use the same control word format,
> for example, the TDM PWs have their own control word different from the
> generic one. So if a Detnet PW were to be defined, it could, for
> example, have a sequence number of whatever length is required for the
> application.

Yes - but wouldn't that require that we had a DetNet variant for each
of the existing PWs, right?
>
> I doubt that you really want "A PW that carries sequence number +
> another PW”, that sounds very inefficient.

Yes - that is my concern too, but cam we force all PWs to carry a
CW with a sequence number? I think we have converged on the MPLS label
stack on top of the PW, but still need to carefully discuss the PW
part. I've included slides that I think reflect where the dt is just
now. Even if I think the dt agrees that "PW in PW" is not efficient,
we still are not sure exactly how to do it.

The PW in DetNet my carry any of the traffic types for which you have
defined PWs, If we can count on a sequence number to be there, that
would solve most of our problems.
> MPLS traffic tunnels can also deliberately use ECMP, see RFC 6790.

Yes - but is not exactly ECMP we are after, even if it from a high
level can look quite like ECMP.

/Loa
>
> Cheers,
> Andy
>
>
> On Tue, Feb 14, 2017 at 11:45 PM, Loa Andersson <[email-masked]
>         using
>             Ethernet
>             <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>
>                 Folks,
>
>                 <mailto:[email-masked]
>             <mailto:[email-masked]>>
>                 <mailto:[email-masked] <mailto:[email-masked]>>
>                 Huawei Technologies (consultant)     phone: +46 739 81
>             21 64 <tel:%2B46%20739%2081%2021%2064>
>         [email-masked] <mailto:[email-masked]>
>         <https://www.ietf.org/mailman/listinfo/detnet-dp-dt>
>
>
>     --
>
>

--

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 17 Feb 2017 06:29:06 -0500

Loa,

You asked: "Yes - but wouldn't that require that we had a DetNet variant
for each of the existing PWs, right?”

That depends - what kind of packets does Detnet plan to carry? If it’s only
IP, then you need to define a new PW type for Detnet that carries IP
packets preceded by a new Detnet-specific control word. It gets more
complicated if the goal is to also carry Ethernet frames, or TDM, or FR
packets, etc. Then you would need to replicate each of those PW types with
the new control word.

Also part of a Detnet PW definition would be multiple parallel PW paths to
be carried by multiple distinct traffic tunnels, and the NSPs, which on the
sending side would replicate packets and on the receiving side to do the
duplicate elimination (and reordering?).

Cheers,
Andy

On Fri, Feb 17, 2017 at 5:37 AM, Loa Andersson <[email-masked]> wrote:

> Andy,
>
> My background is mostly MPLS, while DetNet is also strongly influenced by
> 802. I hope that someone from that side of the house can chime in.
>

---

From: Loa Andersson <[email-masked]>
Date: Fri, 17 Feb 2017 19:54:06 +0800

Andy,

On 2017-02-17 19:29, Andrew G. Malis wrote:
> That depends - what kind of packets does Detnet plan to carry? If it’s
> only IP, then you need to define a new PW type for Detnet that carries
> IP packets preceded by a new Detnet-specific control word. It gets more
> packets, etc. Then you would need to replicate each of those PW types
> with the new control word.
>
> Also part of a Detnet PW definition would be multiple parallel PW paths
> to be carried by multiple distinct traffic tunnels, and the NSPs, which
> on the sending side would replicate packets and on the receiving side to
> do the duplicate elimination (and reordering?).

I don't understand that last sentence, if I have a tunnel with enough BW
and good enough QoS, why can't I carry two different types of PWs over
that tunnel?

The NSP is PW not per tunnel, right?

/Loa

>
> Cheers,
> Andy
>
> On Fri, Feb 17, 2017 at 5:37 AM, Loa Andersson <[email-masked]
>     My background is mostly MPLS, while DetNet is also strongly
>     influenced by 802. I hope that someone from that side of the house
>     can chime in.
>
>     In general - yes! Though equal cost is not a strict requirement, but
>     we will likely look at routing info to find sufficient paths.
>
>
>                 An S-PE does no NSP, although in any case I suspect that
>         you may
>                 need
>                 The underlying DETNET PW is an SS-PW in the diagram I
>         was shown,
>                 in that
>                 the PW label is the same end to end, although of course
>         it need
>                 not be,
>                 for administrative purposes (that is why we designed
>         MS-PW like
>                 that).
>
>                 So if you create an equivalence relationship in the
>         egress-PE,
>                 i.e. two
>                 egress, you can also use a single label value on egress
>         and on
>                 egress
>                 We don't have NSP at the S-PE's in the current PW
>         architecture,
>                 in the
>                 replicates and adds the PW label, which as I said above
>         MAY be
>                 next hop
>                 a detnet-S-PE, or you have SR or RSVP-TE available you
>         can also
>                 deliver
>                 ingress side it looks for the first packet at a given
>         sequence
>                 number on
>                 this PW (or PW set) and suppresses all future packets on
>         that
>                 sequence
>                 number on that PW or PW-set. It then replicates the
>         packet if
>                 required,
>                 swaps the PW label (note that it may also use multiple
>         outgoing
>                 labels)
>                 the PW label the detnet-S-PE expects and potentially
>         sends the
>                 packet
>                 over multiple paths to the detnet-S-PEs. At egress a
>         detnet-T-PE
>                 looks
>                 at the sequence numbers across the detnet-PW set and
>         selects the
>                 first
>                 line cards, and at high speed even amongst different
>         ports on
>                 the same
>                 line card is a hard problem. Indeed depending on the
>         pipeline
>                 design on
>                 ingress port at any detnet X-PE. This would normally
>         require an
>                 RSVP-TE
>                 or SR underlay.
>
>                 - Stewart
>
>
>
>                     PW carried within a SONET/SDH PW, I couldn’t imagine
>         a use
>                     case for
>                     doing it as it’s very inefficient, and I asked Loa
>         if he had
>                     one. And
>                     endpoints. Also, thinking about it some more, you
>         couldn’t
>                     have both
>                     PWs in the same label stack, since a PW emulates a
>         physical
>                     circuit.
>                     So there would need to be a separate label stack
>         (and MPLS
>                     LSP) inside
>                     As I haven’t been following Detnet at all, I don’t
>         have the
>                     context
>                     for what you’re trying to accomplish. That said,
>         I’ll take a
>                     look at
>                     On Sun, Feb 12, 2017 at 8:31 PM, Loa Andersson
>         <[email-masked] <mailto:[email-masked]>
>                     <mailto:[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>
>                         Folks,
>
>                         What Yaunlong said was: "IMHO, multiple layers
>         of PW is
>                     a break from
>                         The problem is that there is a native service
>         processing
>                     (NSP) at
>                         the end of the PW. Multi-layer PWs will only do
>         NSP at
>                     one level.
>                         I think
>                         we should replace the MS-PW with an LSP. I've
>         added one
>                     slide (9) and
>                         I want Andy and Stewart to have a chance to
>         review this
>                     prior to that
>         <mailto:[email-masked] <mailto:[email-masked]>>
>         <mailto:[email-masked]>>>
>                         Senior MPLS Expert
>                         <mailto:[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked] <mailto:[email-masked]>>>
>         <mailto:[email-masked] <mailto:[email-masked]>>
>                 <https://www.ietf.org/mailman/listinfo/detnet-dp-dt
>         <https://www.ietf.org/mailman/listinfo/detnet-dp-dt>>
>
>
>             --
>
>
>             Loa Andersson                        email:
>             <mailto:[email-masked] <mailto:[email-masked]>>
>         <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>

--

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 17 Feb 2017 07:56:52 -0500

Loa.

Yes, it’s per PW.

Cheers,
Andy

---

From: Loa Andersson <[email-masked]>
Date: Fri, 17 Feb 2017 21:02:38 +0800

Andy,

So different types of PW in the same tunnel would be OK?

/Loa

--

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 17 Feb 2017 13:55:32 +0000

Yes

S

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 17 Feb 2017 12:59:13 -0500

Loa,

Yup, you can multiplex multiple PW types in the same MPLS traffic tunnel,
as long as the tunnel as the desired characteristics to carry them (BW,
etc.).

Cheers,
Andy