Subject: [Detnet] IP dataplane

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 10 Nov 2021 13:48:29 +0000

Thanks to both Lou and David for rewording very correctly what I was trying to say at the meeting today.

Effectively, the question is whether we need for a pure IP dataplane that can carry the DetNet information for Forwarding and Service sublayer.

Arguments on the table:

- Not all DetNet environments employ MPLS and pseudowires as a basic tool. Forcing such concepts beyond their domain may hinder adoption in pure IP (IT) and industrial (OT) spaces. A native IP solution is desirable. Note that IPv4 can be encapsulated in IPv6 so arguably we can live with IPv6 signaling.
- Hardware operation (common ASICs) need the information very early in the packet. Digging after UDP may not be feasible in all cases. The DetNet information cannot be missed and should be very early after the IP header (before UDP if present).
- An IP solution is bound to other rules than MPLS, e.g., use of EH and encapsulation vs. stack of label. The properties of the solution might be different and IP may possibly express richer semantics. But as of now, the IP data plane is more limited than the MPLS one.  
- There's a need with IPv6 to encapsulate when playing with merging, re-sourcing (for duplication and network coding), altering the destination (to an intermediate elimination node) or changing packet header information (e.g., sequencing); such encapsulation will hinder with the visibility of deep information (UDP and application data), which cannot be used for DetNet processing
- the elimination or decapsulation node may not be the destination, but it still needs to understand the DetNet signaling; the DetNet signaling should be 100% L3, fully independent of L4 and above, and should not imply that the transport is USP.
- the 5-6 tuple points on upper layer information for a flow. DetNet may aggregate flows (several times leading to multiple reencapsulations) disaggregate flows (decapsulations and separation) and OAM packets. We want the information that signals the dataplane processing independent of which application flow / already merged and encasulated application flows / and or OAM is transported.
- it would make sense to build a solution that integrates well with IPv6 state of the art, especially SRv6, and very possibly HbH which is getting traction, see the discussion at 6MAN / v6ops on Friday.

Bottom line: I disagree with adopting a document that would seem to indicate that we're done with IP. On the contrary I believer that there's a rich ground that we need to plow to deliver the best for IP networks.

What do others think?

Keep safe,

Pascal Thubert

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 10 Nov 2021 19:53:19 +0100

Hi Pascal

Thanks for the email, it touches on a lot of good points. Let me give some opinions from my side

a) I too would like to see that we do get all of the detnet services for a pure
   IP data plane, and not only some short-term IP over MPLS workarounds.

b) I have no strong opinions yet what this means for the document in question.
   Maybe it would suffice to more clearly describe the scope of any current
   "workaround" or "incomplete" IP dataplane documents as such - to make readers
   understand that these solutions may not be good for best-long-term deployments.
   Just an idea.

c) As i said in my slot today, it would help to have more audio discussion time.
   Hopefully we can start such "design-team" meetings..

d) I have not really considered the 'E'limination or othre DetNet service layer
   functions since our old BIER-TE extension work. I think it would be great
   to have some writeup of reference example use cases where this is done, so that
   one had a more concrete reference when designing an appropriate solution.

e) Right now i am roughly thinking that for bounded latency we would want a 
   Hop-By-Hop header with sequence number and appropriate bounded latency paramer(s).
   If we had such a header, we could perform elimination of course also from
   fields in the packet header and not need to rewrite IP header addresses along
   the path. Lets see what fridays 6man meetin brings up avbout the future of HBH.

f) Aggregation/disaggregation is i think a problem we can punt up from IP layer
   to overall DetNet layer. Aka: We have no architectural design or solution of
   this for MPLS AFAIK. But we would need reference scenarios to work against them.

Cheers
    Toerless

On Wed, Nov 10, 2021 at 01:48:29PM +0000, Pascal Thubert (pthubert) wrote:
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

-- 
---
[email-masked]

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 11 Nov 2021 09:05:21 +0000

Hello Toerless

Please see below

Hi Pascal

Count me in; RAW seems to be asking for that sort of team up to progress with its architecture. We are definitely missing the IETF corridors…

True… This should be mentioned in the RAW architecture where it’s a perfect fit. A todo on my plate. When the need is made clear to all then I believe we can progress the solution.

We already spent quite a bit of effort on that. Would you kindly comment on https://sandbox.ietf.org/doc/html/draft-pthubert-detnet-ipv6-hbh-06 ?

IPv6 offers the capability to aggregate by encapsulating multiple streams in one tunnel.
The key for DetNet is that the outer tunnel must signal the needed information for the aggregation including a new sequencing information and packet treatment, independent of the inner signaling when tunnels, OAM or application flows are merged.
The trick is to determine how wider the aggregation tunnel must be vs the sum of inner flows to adapt the bounded latency, a problem that might be related to the wide area discussion.

Enjoy the rest of the week and remember the 6MAM v6ops on Friday!

Keep safe,

Pascal

Cheers
   Toerless

Arguments on the table:

What do others think?

Keep safe,

Pascal Thubert

--
---
[email-masked]

---

From: "Yangfan(Fan,IP Standards)" <[email-masked]>
Date: Thu, 11 Nov 2021 19:11:41 +0000

Hi Pascal and Toerless,

I am fully concurred with your discussions. 
The reasons Pascal lists in previous emails elaborate why IP+UDP encapsulation is not feasible, especially from the hardware perspective. These are sound technical issues which every vendor would face. 
I would also agree the IP+UDP+MPLS DetNet header encapsulation is a workaround solution. If it is already implied in previous RFCs, I question the necessity of producing the separate RFC if so.
Regarding the IPv6 extension header discussions in v6ops/6man, DetNet should catch up and take the advantages of new mechanisms. As the hardware evolves, DetNet shouldn’t be achieved only based on the legacy techniques. Take d-ACH format for OAM as an example, DetNet OAM extension is also keeping aligned with PALS discussion. 

draft-pthubert-detnet-ipv6-hbh-06 introduces a generic and flexible format to solve multiple DetNet problems, e.g. PREOF, OAM, and DN flow identification on forwarding layer etc. It can be richly extended because of the characteristics of IPv6 extensions headers, which is greatly interested by many companies right now. If the adoption call is going to poll to draft-varga-detnet-ip-preof, it is also reasonable to adopt draft-pthubert-detnet-ipv6-hbh, as they represent different underlying encapsulations to support IP PREOF function. 

Best regards,
Fan 

-----邮件原件-----
发件人: detnet [mailto:[email-masked]] 代表 Toerless Eckert
发送时间: 2021年11月11日 2:53
收件人: Pascal Thubert (pthubert) <pthubert=[email-masked]>
抄送: [email-masked]
主题: Re: [Detnet] IP dataplane

Hi Pascal

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 11 Nov 2021 23:05:51 +0100

> We already spent quite a bit of effort on that. Would you kindly comment on https://sandbox.ietf.org/doc/html/draft-pthubert-detnet-ipv6-hbh-06 ?

wht is the teal with snadbox.ietf.org, it seems not to work for me,
"rendering fails". Is this something for collaborative commenting/editing,
or is there a github too ?

As i mentioned in DetNet, i think it would be great if we could try to see
if/how much unification across different data-planes we can get. With
that thought in mind, it might be useful to split any work (like this
document) into 

a) an encoding idependent part describing use-cases/scenarios,
   AND information model. Where information model describes
   which variables we want, and where/how each of them needs to be
   examined.

b) encoding spec. Which may be as lame as an IPv6 HbH heder, or multiple,
   or maybe an extension header that can be shared across IPv6 and future
   MPLS.

> The key for DetNet is that the outer tunnel must signal the needed information for the aggregation including a new sequencing information and packet treatment, independent of the inner signaling wh…

Sure, but how would that be fundamentally different from aggregation in MPLS ?
Which AFAIK, we have also not yet tackled in DetNet.

> The trick is to determine how wider the aggregation tunnel must be vs the sum of inner flows to adapt the bounded latency, a problem that might be related to the wide area discussion.

Sure, but thats not an encap issue, unless you are thinking only about
the added packet header impact. 

And we also have the option to agregate even without tunneling. Just
think about all the traffic to one destination, and you just instantice
a per-destination traffic shaper on every hop, but let the controller
calculate the correct shapper parameter for the aggregate at that hop.
Not good for SP networks, but maybe for non-SP networks.

Aka: I am not saying that we shouldn't figure out the information model
for aggregation, but its a big new building block.

> Enjoy the rest of the week and remember the 6MAM v6ops on Friday!

Indeed!

You to.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 12 Nov 2021 13:01:58 +0000

Hello Toerless

> > We already spent quite a bit of effort on that. Would you kindly
> comment on https://sandbox.ietf.org/doc/html/draft-pthubert-detnet-
> ipv6-hbh-06 ?

Not sure where that comes from, please use https://datatracker.ietf.org/doc/draft-pthubert-detnet-ipv6-hbh/

> "rendering fails". Is this something for collaborative
> commenting/editing, or is there a github too ?

Yes, that's the link that was memorized in my browser and it seemed to work. 

> 
> As i mentioned in DetNet, i think it would be great if we could try to
> see if/how much unification across different data-planes we can get.
> With that thought in mind, it might be useful to split any work (like
> this
> document) into
> 

Makes sense to me; do you think the architecture did not dig deep enough?

> > The key for DetNet is that the outer tunnel must signal the needed
> information for the aggregation including a new sequencing information
> and packet treatment, independent of the inner signaling when tunnels,
> OAM or application flows are merged.
> 

Fundamentally the same thing. But we need to stuff all the DetNet info early in the packet, and minimize the cost of encapsulation? This rules out having to add UDP and stuff and point to small HbH EHs.

> 
> > The trick is to determine how wider the aggregation tunnel must be vs
> the sum of inner flows to adapt the bounded latency, a problem that
> might be related to the wide area discussion.
> 

We agree

> Not good for SP networks, but maybe for non-SP networks.

Think about a large pipe between 2 sites where you funnel multiple streams.
That pipe will encaps between egress of site one and ingress of site 2. 
It can be very stable and share dynamically by shorter lived DetNet streams.

> for aggregation, but its a big new building block.

For now I just mean we need a native IPv6 solution based on EHs, so we can use encapsulation for aggregation. Do we agree?

Keep safe,

Pascal

> > Effectively, the question is whether we need for a pure IP dataplane
> that can carry the DetNet information for Forwarding and Service
> sublayer.
> >
> > Arguments on the table:
> >
> > - Not all DetNet environments employ MPLS and pseudowires as a basic
> tool. Forcing such concepts beyond their domain may hinder adoption in
> pure IP (IT) and industrial (OT) spaces. A native IP solution is
> desirable. Note that IPv4 can be encapsulated in IPv6 so arguably we
> can live with IPv6 signaling.
> > - Hardware operation (common ASICs) need the information very early
> in the packet. Digging after UDP may not be feasible in all cases. The
> DetNet information cannot be missed and should be very early after the
> IP header (before UDP if present).
> > - An IP solution is bound to other rules than MPLS, e.g., use of EH
> and encapsulation vs. stack of label. The properties of the solution
> might be different and IP may possibly express richer semantics. But as
> of now, the IP data plane is more limited than the MPLS one.
> > - There's a need with IPv6 to encapsulate when playing with merging,
> > re-sourcing (for duplication and network coding), altering the
> > destination (to an intermediate elimination node) or changing packet
> > header information (e.g., sequencing); such encapsulation will hinder
> > with the visibility of deep information (UDP and application data),
> > which cannot be used for DetNet processing
> > - the elimination or decapsulation node may not be the destination,
> but it still needs to understand the DetNet signaling; the DetNet
> signaling should be 100% L3, fully independent of L4 and above, and
> should not imply that the transport is USP.
> > - the 5-6 tuple points on upper layer information for a flow. DetNet
> may aggregate flows (several times leading to multiple
> reencapsulations) disaggregate flows (decapsulations and separation)
> and OAM packets. We want the information that signals the dataplane
> processing independent of which application flow / already merged and
> encasulated application flows / and or OAM is transported.
> > - it would make sense to build a solution that integrates well with
> IPv6 state of the art, especially SRv6, and very possibly HbH which is
> getting traction, see the discussion at 6MAN / v6ops on Friday.
> >
> > Bottom line: I disagree with adopting a document that would seem to
> indicate that we're done with IP. On the contrary I believer that
> there's a rich ground that we need to plow to deliver the best for IP
> networks.
> >
> > What do others think?
> >
> > Keep safe,
> >
> > Pascal Thubert
> >

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 12 Nov 2021 18:56:14 -0800

Hi Toerless,
RE: aggregation in MPLS question.
The aggregation in MPLS can be achieved using a hierarchical LSP. RFC 6107
<https://www.rfc-editor.org/rfc/rfc6107.html> describes how GMPLS can be
used to signal dynamic hierarchical LSPs. As Pascal suggested for DetNet
flow aggregation over IP, the aggregate must be seen and treated as a new
DetNet flow.
What are your thoughts?

Regards,
Greg

On Thu, Nov 11, 2021 at 2:06 PM Toerless Eckert <[email-masked]> wrote:

> > We already spent quite a bit of effort on that. Would you kindly comment
> on https://sandbox.ietf.org/doc/html/draft-pthubert-detnet-ipv6-hbh-06 ?
>
> information for the aggregation including a new sequencing information and
> packet treatment, independent of the inner signaling when tunnels, OAM or
> application flows are merged.
>
> the sum of inner flows to adapt the bounded latency, a problem that might
> be related to the wide area discussion.
>
> that can carry the DetNet information for Forwarding and Service sublayer.
> >
> > Arguments on the table:
> >
> tool. Forcing such concepts beyond their domain may hinder adoption in pure
> IP (IT) and industrial (OT) spaces. A native IP solution is desirable. Note
> that IPv4 can be encapsulated in IPv6 so arguably we can live with IPv6
> signaling.
> > - Hardware operation (common ASICs) need the information very early in
> the packet. Digging after UDP may not be feasible in all cases. The DetNet
> information cannot be missed and should be very early after the IP header
> (before UDP if present).
> > - An IP solution is bound to other rules than MPLS, e.g., use of EH and
> encapsulation vs. stack of label. The properties of the solution might be
> different and IP may possibly express richer semantics. But as of now, the
> IP data plane is more limited than the MPLS one.
> re-sourcing (for duplication and network coding), altering the destination
> (to an intermediate elimination node) or changing packet header information
> (e.g., sequencing); such encapsulation will hinder with the visibility of
> deep information (UDP and application data), which cannot be used for
> DetNet processing
> > - the elimination or decapsulation node may not be the destination, but
> it still needs to understand the DetNet signaling; the DetNet signaling
> should be 100% L3, fully independent of L4 and above, and should not imply
> that the transport is USP.
> > - the 5-6 tuple points on upper layer information for a flow. DetNet may
> aggregate flows (several times leading to multiple reencapsulations)
> disaggregate flows (decapsulations and separation) and OAM packets. We want
> the information that signals the dataplane processing independent of which
> application flow / already merged and encasulated application flows / and
> or OAM is transported.
> > - it would make sense to build a solution that integrates well with IPv6
> state of the art, especially SRv6, and very possibly HbH which is getting
> traction, see the discussion at 6MAN / v6ops on Friday.
> >
> indicate that we're done with IP. On the contrary I believer that there's a
> rich ground that we need to plow to deliver the best for IP networks.
> >
> > What do others think?
> >
> > Keep safe,
> >
> > Pascal Thubert
> >

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 12 Nov 2021 19:45:45 -0800

Hi Toerless,
upon thinking it further, the Packet PW
<https://datatracker.ietf.org/doc/html/rfc6658> might offer a more suitable
solution for the DetNet MPLS aggregation than the hierarchical LSP. Yes,
the Packet PW requires an extra internal Ethernet header.

Regards,
Greg

On Fri, Nov 12, 2021 at 6:56 PM Greg Mirsky <[email-masked]> wrote:

>> comment on
>> https://sandbox.ietf.org/doc/html/draft-pthubert-detnet-ipv6-hbh-06 ?
>>
>> may aggregate flows (several times leading to multiple reencapsulations)