Subject: Re: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 2 Jan 2017 11:38:39 +0000

Hi Nobo,

happy new year. Thanks much for your comments. Please see inline… (“…FB” prefix)

From: OPSAWG [mailto:[email-masked]] On Behalf Of Nobo Akiya
Sent: Dienstag, 27. Dezember 2016 05:45
To: [email-masked]
Subject: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

Hi Authors,

I'm happy to see interesting work in the OAM area :)

Comments on draft-brockners-inband-oam-requirements-02:

>    o  Low-power Networks: Include application level OAM information

>      (e.g., battery charge level, cache or buffer fill level) into data

>      traffic to avoid sending extra OAM traffic which incur an extra

>      cost on the devices.  Using the battery charge level as example,

>      one could avoid sending extra OAM packets just to communicate

>      battery health, and as such would save battery on sensors.

Are you placing above use case in this document because you the protocol is capable of doing this, or do you consider it to be a hard requirement for in-band OAM?

...FB: The above mentioned use-case is to be understood as just one deployment example. We came across this use-case when discussing the concept of in-situ OAM with folks who work on low power networks. As such, it is not a “hard requirement”, but it would be good to keep these deployment cases in mind.

Comments on draft-brockners-inband-oam-data-02:

> queue depth:  4-octet unsigned integer field.  This field indicates

>      the length of the egress interface queue of the interface where

>      the packet is forwarded out of.

Many interfaces can have multiple queues (e.g., multiple unicast queues and multiple multicast queues). Won't be need an queue ID to indicate which "queue" above is refering to?

...FB: In-situ OAM uses its own name space for most of the data records, including the identifiers for “interfaces”. This means that you can use the interface identifier pretty flexibly, i.e. it can apply to a physical interface or sub-interface, a logical interface, or even a queue. We’ll make this clearer in the next rev of the drafts.

>      node_id:  3-octet unsigned integer.  Node identifier field to

>         uniquely identify a node within in-situ OAM domain.  The

>         procedure to allocate, manage and map the node_ids is beyond

>         the scope of this document.

...

>       node_id:  7-octet unsigned integer.  Node identifier field to

>         uniquely identify a node within in-situ OAM domain.  The

>         procedure to allocate, manage and map the node_ids is beyond

>         the scope of this document.

Why is node_id defined as different size?

Same with if_id (4 bytes vs. 8 bytes).

...FB: In the initial version of the drafts we only had a single size defined. It turned out that for specific use-cases, e.g. a 4 bytes interface_id isn’t sufficient (think about your above mentions case where every queue would need an id, or think of a mobile gateway with millions of interfaces/sessions). Similar considerations apply for large networks, where a 3 byte node_id wasn’t seen as sufficient.
As a consequence, there are now two options for several of the data records – a compact and a wide format.

>    Maximum Length:  8-bit unsigned integer.  This field specifies the

>      maximum length of the node data list in octets.  Given that the

>      sender knows the minimum path MTU, the sender can set the maximum

>      of node data bytes allowed before exceeding the MTU.  Thus, a

>      simple comparison between "Opt data Len" and "Max Length" allows

>      to decide whether or not data could be added.

How does the sender know this? PMTUD is a discovery of MTU on a path (i.e., point-to-point) and not the entire network domain. Meaning PMTUD is not completely accurate, as a path a packet can take can change at anytime. Does a centralized controller/orchestrator educate every sender?

...FB: In-situ OAM is considered a “per domain” feature, where the supported max MTU is known. Consequently, one can derive the maximum amount of in-situ OAM that can be added.

>    OAM-trace-type:  16-bit identifier of a particular trace element

>      variant.

Offset will depend on which bits are set. Typically it is much easier to implement if one option type having fixed data at fixed locations. I'm sure "smart programmable chip" can handle this, but I feel that we are trying to be too flexible here. Thoughts?

...FB: Good point. This is why the draft offer two options: “pre-allocated trace option” and “incremental trace option”. The pre-allocated option is mostly suited for software dataplanes, whereas the incremental trace option applies well to hardware implementations, where one would like to drop-in data always at the very same location within the packet, as opposed to doing a costly lookup of a pointer first.

General comments:

- When a packet traverses a LAG, it would be good to clarify the expected if_id to be used.

...FB: Good point. Per your above note on queues – we’ll further clarify how the interface_id namespace could be used.

- When a packet traverses a virtualized environment, it would be good to clarify the expected behavior of each physical/virtual node.- I feel that much of "error" cases are lacking in the document. It would be good to clarify how each node need to behave when there are errors. For example, if a node receives an In Situ OAM packet with "overflow" bit already set but can still add more data, do you expect this node to also skip or do you expect this node add the data and clear the "overflow" bit?

...FB: Again a good point. We’ll further elaborate on failure cases in the next revision. On the particular overflow bit: In-situ OAM is considered a per-domain feature. If an node indicates that no more in-situ OAM data can be fitted, no other node should clear this bit. Or in other terms, all nodes should have the same understanding of the maximum amount of in-situ OAM data that could be fitted. If this isn’t the case, this hints at a misconfiguration.

Thanks again, Frank

Regards,

Nobo

---

From: David Mozes <[email-masked]>
Date: Tue, 3 Jan 2017 13:25:43 +0000

Happy new year .
Nobo as coauthor thanks for the comments and the interesting.
It looks that a lot of work need to be done on the draft.
That one more reason why the working group need to adapt this draft.

Thx
David

From: OPSAWG [mailto:[email-masked]] On Behalf Of Frank Brockners (fbrockne)
Sent: Monday, January 02, 2017 1:39 PM
To: Nobo Akiya <[email-masked]>; [email-masked]
Subject: Re: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

Hi Nobo,

From: OPSAWG [mailto:[email-masked]] On Behalf Of Nobo Akiya
Sent: Dienstag, 27. Dezember 2016 05:45
To: [email-masked]<mailto:[email-masked]>
Subject: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

Hi Authors,

...

General comments:

Thanks again, Frank

Regards,

Nobo

---

From: Nobo Akiya <[email-masked]>
Date: Fri, 6 Jan 2017 16:57:50 +0900

Hi Frank, David,

Happy new year to you guys as well.

And thank you for responding to my comments.
Your response has clarified much of the questions that I had.
For the rest, I'll wait for an updated version.

One more question.

>       The in-situ OAM data records can be transported by a

>   variety of transport protocols, including NSH, Segment Routing,
>   VXLAN-GPE, IPv6, IPv4.  Encapsulation details for these different
>   transport protocols are outside the scope of this document.

Where exactly does in-situ OAM data fit in the packet?

Above implies that it's out of the scope of this document ... but can
you clarify for my understanding?

Let's take a simple IPv4 as an example.

(1) Is in-situ OAM data placed in between the IPv4 header and the
original user data/payload?

(2) Or is in-situ OAM data placed after the entire user data/payload?

If (1), then:
  (a) in-situ OAM will need a protocol field to describe what follows the
in-situ OAM data.
  (b) it will have an side effect of messing up a lot the load balancing in
transit nodes.
    (i) many devices often peek deeper in the packet beyond the outer L3
header in order to better load balance. However, these devices may not
understand the in-situ OAM data, thus the result will be poor load
balancing on packets having in-situ OAM data.
    (ii) if some packets of a same flow has in-situ OAM data and other
packets of the same flow does not have in-situ OAM data, then the load
balancing result of these packets can be different (i.e., packets can take
different path, potential out of order).

If (2), then how does the receiver know that in-situ OAM data follows the
user data/payload in received packets?

Regards,
Nobo

On Tue, Jan 3, 2017 at 10:25 PM, David Mozes <[email-masked]> wrote:

> *From:* OPSAWG [mailto:[email-masked]] *On Behalf Of *Frank
> Brockners (fbrockne)
> *Sent:* Monday, January 02, 2017 1:39 PM
> *To:* Nobo Akiya <[email-masked]>; [email-masked]
> *Subject:* Re: [OPSAWG] Comments on draft-brockners-inband-oam-* documents
>
>
>
> Hi Nobo,
>
>
>
> *From:* OPSAWG [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Nobo Akiya
> *Sent:* Dienstag, 27. Dezember 2016 05:45
> *To:* [email-masked]
> *Subject:* [OPSAWG] Comments on draft-brockners-inband-oam-* documents
>
>
>
> Hi Authors,
>
>
>
> Are you placing above use case in this document because you the protocol
> is capable of doing this, or do you consider it to be a hard requirement
> for in-band OAM?
>
>
>
> ...FB: The above mentioned use-case is to be understood as just one
> deployment example. We came across this use-case when discussing the
> concept of in-situ OAM with folks who work on low power networks. As such,
> it is not a “hard requirement”, but it would be good to keep these
> deployment cases in mind.
>
>
>
> Many interfaces can have multiple queues (e.g., multiple unicast queues
> and multiple multicast queues). Won't be need an queue ID to indicate which
> "queue" above is refering to?
>
>
>
> ...FB: In-situ OAM uses its own name space for most of the data records,
> including the identifiers for “interfaces”. This means that you can use the
> interface identifier pretty flexibly, i.e. it can apply to a physical
> interface or sub-interface, a logical interface, or even a queue. We’ll
> make this clearer in the next rev of the drafts.
>
>
>
> ...FB: In the initial version of the drafts we only had a single size
> defined. It turned out that for specific use-cases, e.g. a 4 bytes
> interface_id isn’t sufficient (think about your above mentions case where
> every queue would need an id, or think of a mobile gateway with millions of
> interfaces/sessions). Similar considerations apply for large networks,
> where a 3 byte node_id wasn’t seen as sufficient.
> How does the sender know this? PMTUD is a discovery of MTU on a path
> (i.e., point-to-point) and not the entire network domain. Meaning PMTUD is
> not completely accurate, as a path a packet can take can change at anytime.
> Does a centralized controller/orchestrator educate every sender?
>
> ...FB: In-situ OAM is considered a “per domain” feature, where the
> supported max MTU is known. Consequently, one can derive the maximum amount
> of in-situ OAM that can be added.
>
>
>
> Offset will depend on which bits are set. Typically it is much easier to
> implement if one option type having fixed data at fixed locations. I'm sure
> "smart programmable chip" can handle this, but I feel that we are trying to
> be too flexible here. Thoughts?
>
> ...FB: Good point. This is why the draft offer two options: “pre-allocated
> trace option” and “incremental trace option”. The pre-allocated option is
> mostly suited for software dataplanes, whereas the incremental trace option
> applies well to hardware implementations, where one would like to drop-in
> data always at the very same location within the packet, as opposed to
> doing a costly lookup of a pointer first.
>
>
>
> General comments:
>
>
>
> ...FB: Good point. Per your above note on queues – we’ll further clarify
> how the interface_id namespace could be used.
>
> - When a packet traverses a virtualized environment, it would be good to
> clarify the expected behavior of each physical/virtual node.- I feel that
> much of "error" cases are lacking in the document. It would be good to
> clarify how each node need to behave when there are errors. For example, if
> a node receives an In Situ OAM packet with "overflow" bit already set but
> can still add more data, do you expect this node to also skip or do you
> expect this node add the data and clear the "overflow" bit?
>
> ...FB: Again a good point. We’ll further elaborate on failure cases in the
> next revision. On the particular overflow bit: In-situ OAM is considered a
> per-domain feature. If an node indicates that no more in-situ OAM data can
> be fitted, no other node should clear this bit. Or in other terms, all
> nodes should have the same understanding of the maximum amount of in-situ
> OAM data that could be fitted. If this isn’t the case, this hints at a
> misconfiguration.
>
>
>
> Thanks again, Frank
>
>
>
> Regards,
>
> Nobo
>

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 6 Jan 2017 10:43:29 +0000

Hi Nobo,

thanks. With regards to transport, https://tools.ietf.org/html/draft-brockners-inband-oam-transport-02 details how in-situ OAM data records could be transported for a variety of transports. Those are just suggestions for now and of course need to be worked in the various working groups.
We’ve not concluded on how to best do things for v4 – and you raise very valid questions and we greatly appreciate your thoughts. An additional approach – you can call it #3 - would be to double-encap the original packet (i.e. new-ip-hdr, new-udp-hdr, ioam, orig-ip-hdr, payload) where new-src-ip would be the address of the encapsulating node and new-dest-ip would be the same as orig-dest-ip, but this approach has downsides as well.

Regards, Frank

From: Nobo Akiya [mailto:[email-masked]]
Sent: Freitag, 6. Januar 2017 08:58
To: David Mozes <[email-masked]>
Cc: Frank Brockners (fbrockne) <[email-masked]>; [email-masked]
Subject: Re: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

Hi Frank, David,

One more question.

Above implies that it's out of the scope of this document ... but can you clarify for my understanding?

(1) Is in-situ OAM data placed in between the IPv4 header and the original user data/payload?

If (1), then:
  (a) in-situ OAM will need a protocol field to describe what follows the in-situ OAM data.
  (b) it will have an side effect of messing up a lot the load balancing in transit nodes.
    (i) many devices often peek deeper in the packet beyond the outer L3 header in order to better load balance. However, these devices may not understand the in-situ OAM data, thus the result will be poor load balancing on packets having in-situ OAM data.
    (ii) if some packets of a same flow has in-situ OAM data and other packets of the same flow does not have in-situ OAM data, then the load balancing result of these packets can be different (i.e., packets can take different path, potential out of order).

If (2), then how does the receiver know that in-situ OAM data follows the user data/payload in received packets?

Regards,
Nobo

On Tue, Jan 3, 2017 at 10:25 PM, David Mozes <[email-masked]<mailto:[email-masked]>> wrote:
Happy new year .
Nobo as coauthor thanks for the comments and the interesting.
It looks that a lot of work need to be done on the draft.
That one more reason why the working group need to adapt this draft.

Thx
David

From: OPSAWG [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Frank Brockners (fbrockne)
Sent: Monday, January 02, 2017 1:39 PM
To: Nobo Akiya <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [OPSAWG] Comments on draft-brockners-inband-oam-* documents

Hi Nobo,

Hi Authors,

...

General comments:

Thanks again, Frank

Regards,

Nobo