Subject: [Rtg-ooam-dt] comments on draft-ooamdt-rtgwg-ooam-requirement-00

From: Alia Atlas <[email-masked]>
Date: Thu, 9 Jun 2016 12:35:20 -0400

Hi,

I am doing a read through of the work of the DT so far.
I have a few comments on draft-ooamdt-rtgwg-ooam-requirement-00.

I have cc'd a couple people who may have perspective and opinions on issues
around OAM and running an overlay network.   For instance, one of the
impediments to TRILL has been solid OAM  and associated implementations.
I'm sure that there are others who could also chime in.

First, a great deal of it reads like generic OAM requirements for any layer.
What I would like to see as concepts:

With overlay networks, it should be understood that there is NOT a static
assumption of layers.  For instance, an NSH header might be inside VXLAN-GPE
which might be inside a BIER header which is inside MPLS or IP.   Ethernet
can
play as well.

One of the large challenges is understanding and seeing the topology of
overlays and dependencies along the path.  For instance, consider existing
MPLS ping, where the
set of labels on the packet and their meaning at the responding LSR, can be
returned.
Consider RFC 5837 - where a traceroute can return details about the
incoming interface, information about the Ethernet member of the LAG, and
so on.

So - there should be a concrete way to drill down BETWEEN layers.  It might
be optional, but I should be able to do the equivalent of a traceroute and
get down to the specific layer-2 link that is being used.  Maybe I can get
a report of the different next-hops available if the TTL equiv hadn't timed
out?

I WANT you to think OUTSIDE the standard existing OAM box!  What are the
headaches of operating an overlay network?  What are the pain points that
make these things hard to deploy?

What about policy considerations where one layer of the network is owned by
a provider?

What about MTU discovery and how that relies on things like traceroute
concepts and getting back a packet too large?  Could an OAM packet return
information about the max header added along the path?

Have you brainstormed with operators who are running these overlays - or
similar proprietary ones?  What are the issues?

Now - to more specific details:

 REQ#1:  The listed requirements MUST be supported with any type of
            transport layer over which the overlay network can be
            realized

What if one overlay has an OAM packet inside another overlay header?  Is
the outer header the only one examined?  Is it ok to do this?

What restrictions might practically exist in terms of underlay to cause
this to be an issue? I.e. what issues to be solve arise from this
requirement?

What do you mean by "transport layer"?  It might be more helpful to define
terminology in terms of overlay/underlay or service/transport or...

  "REQ#2:  It MUST be possible to initialize Overlay OAM session from
            any node in the overlay network.

   REQ#3:  It SHOULD be possible to initialize an Overlay OAM session
            from a centralized controller."

First - this is assuming a session - what aspects do you have tied up with
this?
Second - does this handle problems like the standard traceroute & ECMP with
hashing -
where a midpoint node can do traceroute but it takes a different path
because the source
address to respond to is hashed?

Is it the case that the address to respond to is in the packet?  How is it
integrity-protected?

Frankly, most of the requirements in 4.1 and 4.2 read like a standard OAM -
within a particular layer.  Sure - we need these basics, but there's
nothing novel here.  Why not just refer to common OAM requirements from
some of the MPLS work or elsewhere?
Focus on what is needed for OVERLAY/UNDERLAY interactions.

Regards,
Alia

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Thu, 9 Jun 2016 17:38:01 +0000

HI Alia,

Thanks for the time and sharing the valid comments. We will check this out and will reply back.

Thanks,
Nagendra

From: Rtg-ooam-dt <[email-masked]<mailto:[email-masked]>> on behalf of Alia Atlas <[email-masked]<mailto:[email-masked]>>
Date: Thursday, June 9, 2016 at 12:35 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Tom Herbert <[email-masked]<mailto:[email-masked]>>, Jesse Gross <[email-masked]<mailto:[email-masked]>>, Jon Hudson <[email-masked]<mailto:[email-masked]>>
Subject: [Rtg-ooam-dt] comments on draft-ooamdt-rtgwg-ooam-requirement-00

Hi,

I have cc'd a couple people who may have perspective and opinions on issues around OAM and running an overlay network.   For instance, one of the impediments to TRILL has been solid OAM  and associated implementations.  I'm sure that there are others who could also chime in.

With overlay networks, it should be understood that there is NOT a static
assumption of layers.  For instance, an NSH header might be inside VXLAN-GPE
which might be inside a BIER header which is inside MPLS or IP.   Ethernet can
play as well.

One of the large challenges is understanding and seeing the topology of overlays and dependencies along the path.  For instance, consider existing MPLS ping, where the
set of labels on the packet and their meaning at the responding LSR, can be returned.
Consider RFC 5837 - where a traceroute can return details about the incoming interface, information about the Ethernet member of the LAG, and so on.

So - there should be a concrete way to drill down BETWEEN layers.  It might be optional, but I should be able to do the equivalent of a traceroute and get down to the specific layer-2 link that is being used.  Maybe I can get a report of the different next-hops available if the TTL equiv hadn't timed out?

I WANT you to think OUTSIDE the standard existing OAM box!  What are the headaches of operating an overlay network?  What are the pain points that make these things hard to deploy?

What about policy considerations where one layer of the network is owned by a provider?

What about MTU discovery and how that relies on things like traceroute concepts and getting back a packet too large?  Could an OAM packet return information about the max header added along the path?

Have you brainstormed with operators who are running these overlays - or similar proprietary ones?  What are the issues?

What if one overlay has an OAM packet inside another overlay header?  Is the outer header the only one examined?  Is it ok to do this?

What restrictions might practically exist in terms of underlay to cause this to be an issue? I.e. what issues to be solve arise from this requirement?

What do you mean by "transport layer"?  It might be more helpful to define terminology in terms of overlay/underlay or service/transport or...

First - this is assuming a session - what aspects do you have tied up with this?
Second - does this handle problems like the standard traceroute & ECMP with hashing -
where a midpoint node can do traceroute but it takes a different path because the source
address to respond to is hashed?

Is it the case that the address to respond to is in the packet?  How is it integrity-protected?

Frankly, most of the requirements in 4.1 and 4.2 read like a standard OAM - within a particular layer.  Sure - we need these basics, but there's nothing novel here.  Why not just refer to common OAM requirements from some of the MPLS work or elsewhere?
Focus on what is needed for OVERLAY/UNDERLAY interactions.

Regards,
Alia

---

From: Ignas Bagdonas <[email-masked]>
Date: Sat, 18 Jun 2016 02:15:06 +0100

Hi Alia & the team,

Some assorted comments from operator's perspective.

On 09/06/2016 17:35, Alia Atlas wrote:
> I have cc'd a couple people who may have perspective and opinions on 
> issues around OAM and running an overlay network.   For instance, one 
> of the impediments to TRILL has been solid OAM  and associated 
> implementations.  I'm sure that there are others who could also chime in.

TRILL indeed has nice OAM mechanism, however, it is too tailored for 
TRILL specific needs - the payload there is L2 only and TRILL itself is 
not necessary and overlay as such.

> which might be inside a BIER header which is inside MPLS or IP.   

The static aspect could be interpreted twofold here - the mapping of 
layers (the stack of dataplane encapsulations) is static over the 
lifetime of the flow between the communications endpoints, or the stack 
could change (protection events involving rerouting the overlay over 
different topology having a different dataplane stackup). The second one 
is not necessary practical today, this is more of an overall network 
architecture question. The complexity of what it could bring into the 
operational aspects of the different layers involved seems to be 
substantial.

> One of the large challenges is understanding and seeing the topology 
> of overlays and dependencies along the path. For instance, consider 
> existing MPLS ping, where the
> incoming interface, information about the Ethernet member of the LAG, 
> and so on.

Traceroute based approaches work reasonably well for IP based payloads 
and not in the overlay context (generalizing this also applies to IP 
based tunnels). Trying to extend IP based tools to be aware of lower 
packet transport layers does not seem to be robust enough. Maybe it is 
time for IP OAM mechanism from scratch that is able to accommodate 
current requirements and represents current deployments?

Looking from the operator's side, I am supportive of defining something 
(protocol and corresponding methodology) new from grounds up, not trying 
to extend existing tools and mechanisms. It seems that a replacement for 
ICMP is needed, a protocol to coordinate (at least get a reliable 
information about MTU on lower layers) MTU across different layers is 
needed, a mechanism to exchange topology (maybe reuse I2RS, although 
there are design decisions made on modelling side there that could be 
hard to deploy practically), a mechanism to interface into the network 
OAM in general for all types and not a separate interface per 
protocol/mechanism. Extensions tend to be fragmenting the range of 
available vendors/platforms to choose, also operational tools support 
gets complex this way. Doing the one protocol/mechanism from the 
beginning and spending time to address what is not necessary right at 
the moment, if covering what is really needed from operations side, 
likely will get better traction for support than yet another extension 
to BFD/LSP ping/etc.

> So - there should be a concrete way to drill down BETWEEN layers.  It 
> might be optional, but I should be able to do the equivalent of a 
> traceroute and get down to the specific layer-2 link that is being 
> used.  Maybe I can get a report of the different next-hops available 
> if the TTL equiv hadn't timed out?

In my view it should not be optional - if not needed, simply unused, but 
the problem today lies in the absence of such cross-layer interworking 
or visibility mechanism.

This is closely related to topology in different layers. There could be 
points defined in corresponding layers that interconnect between the 
layers (it is not practical to expect cross-layer interworking happening 
on more than a few separate places) that allow for (limited) topology 
visibility and corresponding OAM mechanisms' state.

> I WANT you to think OUTSIDE the standard existing OAM box!  What are 
> the headaches of operating an overlay network?  What are the pain 
> points that make these things hard to deploy?

Awareness of existing operational tools and reliance on them are one of 
the blocking factors for bringing in mechanisms that would be able to 
address the crosslayer OAM requirements in forms of extensions. Defining 
a protocol to do something is trivial compared to the complexity of 
integrating the result of that protocol into the operation tools and 
systems. If such protocol is done, realistically it will take 
substantial time (decade is reasonable to expect, as MPLS OAM has shown) 
to get it into widespread deployment due to integration aspects of the 
tools and systems. This would mean that such mechanism must be 
extensible and done right in the beginning for allowing such extensions 
to be minimally disruptive.

> What about policy considerations where one layer of the network is 

This is a question of a place in the network. For controlled 
environments it can be solved, and the solution is far more 
administrative than technical. I would not be certain whether concept of 
cross-layter OAM has practical meaning outside of a controlled 
environment, and even less so in the context of global internet. It 
could have, but that gets into the scope of discussion whether it is 
worth doing in the first place. Possibly except of MTU related aspects.

> concepts and getting back a packet too large?  Could an OAM packet 
> return information about the max header added along the path?

This seems to be the domain of MTU handling, and while overlays could 
add in complexity into it, the question again is about the scope of the 
problem space. Controlled environments do not have issues with MTU, if 
something is an issue then that is most likley the result of some 
exception and it will get dealt with by means other than cross-layer 
OAM. MTU aspects could be one of the properties of links and nodes that 
could be signalled over such (crosslayer)OAM mechanism. However the line 
needs to be drawn what gets to be considered as an OAM signalling, and 
what remains topology and routing control. Routing protocol should nbe 
aware of OAM, but not rely on it, and not replace it.

> Have you brainstormed with operators who are running these overlays - 
> or similar proprietary ones?  What are the issues?

The issues would fall into several main categories:

- Ability to map overlay nodes and links to underlay nodes and links. 
SRLG awareness of the overlay.

- Ability to map underlay nodes and links to (groups of) overlay nodes 
and links. For the event in the underlay, which overlays are affected?

- Ability to interwork or at least get information about protection 
events and topology changes between underlay and overlay. A really 
painful aspect of OAM intelligence in different layers without 
coordination is the incongruence of decisions made on different layers. 
Although we happily had this since days of ATM. :-)

- Ability to have a single interface into network OAM (the generic broad 
term of OAM, regardless of layers) that would allow for a single 
interface being implemented from the tools side, both across different 
layers and between applications and network itself.

Not every use case requires overlays, and not every operator's products 
are realized with overlays. The layering of different dataplanes and 
encapsulations happens universally everywhere, the differences lie in 
depths and varieties of encapsulations used, therefore cross-layer OAM 
seems to be generically applicable. All of this can and is addressable 
today via a multitude of means, both technical and administrative, but 
that is really complex and contains too much of inhouse development.

> What if one overlay has an OAM packet inside another overlay header?  
> Is the outer header the only one examined? Is it ok to do this?
>
> What restrictions might practically exist in terms of underlay to 
> cause this to be an issue? I.e. what issues to be solve arise from 
> this requirement?
>
> What do you mean by "transport layer"?  It might be more helpful to 
> define terminology in terms of overlay/underlay or service/transport or...
>

An important functionality is ability to initiate proxy operations on 
behalf of shadow/dedicated OAM service nodes.

> Frankly, most of the requirements in 4.1 and 4.2 read like a standard 
> OAM - within a particular layer.  Sure - we need these basics, but 
> there's nothing novel here.  Why not just refer to common OAM 
> requirements from some of the MPLS work or elsewhere?
> _______________________________________________
> Rtg-ooam-dt mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtg-ooam-dt

Ignas