Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

From: Tianran Zhou <[email-masked]>
Date: Mon, 11 Mar 2019 02:39:44 +0000

Hi WG,

As you may have seen, the authors have posted an update to draft-song-opsawg-ntf-03 to address discussions in Bangkok and after.
https://datatracker.ietf.org/doc/draft-song-opsawg-ntf/
https://mailarchive.ietf.org/arch/msg/opsawg/g338UPfVAtOhVhdDzhJcR2nS76E

In Bangkok there seemed to some interest in working on this topic and the chairs believe it is in scope for this working group.

This email starts a poll for adoption. 
If you support adopting this document please say so, and please give an indication of why you think it is important. Also please say if you will be willing to review and help the draft.
If you do not support adopting this document as a starting point for work on this topic, please say why.
This poll will run until 9am in Prague on Monday 25th March.

Regards,
Tianran, OPSAWG Co-Chair

---

From: li zhenqiang <[email-masked]>
Date: Tue, 12 Mar 2019 09:05:40 +0000

Dear All,

Support the adoption as a coauthor. As stated in the abstract, this document intents to provide a framework and taxonomy to set a common ground for the collection of related work and provide guidance for future technique and standard developments.

Best Regards,
Zhenqiang Li
________________________________
[email-masked]

From: Tianran Zhou<mailto:[email-masked]>
Date: 2019-03-11 10:39
To: [email-masked]<mailto:[email-masked]>
CC: [email-masked]<mailto:[email-masked]>
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf
Hi WG,

Regards,
Tianran, OPSAWG Co-Chair

_______________________________________________
OPSAWG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/opsawg

---

From: Giuseppe Fioccola <[email-masked]>
Date: Tue, 12 Mar 2019 09:21:33 +0000

Hi Tianran, All,
Yes, I support its adoption. I also helped with some sections of the document.

Best Regards,

Giuseppe

From: OPSAWG [mailto:[email-masked]] On Behalf Of li zhenqiang
Sent: Tuesday, 12 March 2019 10:06
To: Tianran Zhou <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Dear All,

Best Regards,
Zhenqiang Li
________________________________
[email-masked]<mailto:[email-masked]>

Regards,
Tianran, OPSAWG Co-Chair

_______________________________________________
OPSAWG mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/opsawg

---

From: Lizhenbin <[email-masked]>
Date: Tue, 12 Mar 2019 14:02:23 +0000

Hi All,
I strongly support the adoption. There is much Telemetry work is done or being done in IETF, but very distributed. It seems a little mess and even cause much confusion. I believe IETF should take the responsibilty to define a systematic framework for the Telemetry work. When discussed with many experts from operators about the NTF work, they thought it is very helpful and useful. Though there is much work to be done, the WG adoption will be an important starting point.

Best Regards,
Zhenbin (Robin)

--------------------------------------------------
李振斌 Robin
M: +86-13651017745<tel:+86-13651017745>/+968-91797068<tel:+968-91797068>
E: [email-masked]<mailto:[email-masked]>
产品与解决方案-网络标准专利部
Products & Solutions-Network Standard & Patent Dept
发件人：Tianran Zhou <[email-masked]>
收件人：[email-masked] <[email-masked]>
抄 送：[email-masked] <[email-masked]>
时间：2019-03-11 10:41:39
主 题：[OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

Regards,
Tianran, OPSAWG Co-Chair

---

From: "Aijun Wang" <[email-masked]>
Date: Wed, 13 Mar 2019 09:33:17 +0800

Support for the adoption.
Based on this framework, we can easily judge, compare and select the
emerging telemetry technologies in various scenarios. It also gives the
common ground for future discussion.

Best Regards.

Aijun Wang
Network R&D and Operation Support Department
China Telecom Corporation Limited Beijing Research Institute,Beijing, China.

-----邮件原件-----
发件人: OPSAWG [mailto:[email-masked]] 代表 Tianran Zhou
发送时间: 2019年3月11日 10:40
收件人: [email-masked]
抄送: [email-masked]
主题: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

As you may have seen, the authors have posted an update to
draft-song-opsawg-ntf-03 to address discussions in Bangkok and after.
https://datatracker.ietf.org/doc/draft-song-opsawg-ntf/
https://mailarchive.ietf.org/arch/msg/opsawg/g338UPfVAtOhVhdDzhJcR2nS76E

In Bangkok there seemed to some interest in working on this topic and the
chairs believe it is in scope for this working group.

This email starts a poll for adoption. 
If you support adopting this document please say so, and please give an
indication of why you think it is important. Also please say if you will be
willing to review and help the draft.
If you do not support adopting this document as a starting point for work on
this topic, please say why.
This poll will run until 9am in Prague on Monday 25th March.

Regards,
Tianran, OPSAWG Co-Chair

---

From: Huaimo Chen <[email-masked]>
Date: Wed, 13 Mar 2019 02:12:58 +0000

Support.

Best Regards,
Huaimo

Regards,
Tianran, OPSAWG Co-Chair

---

From: "Chongfeng Xie" <[email-masked]>
Date: Wed, 13 Mar 2019 10:34:46 +0800

I support the adoption of this draft, for it provides a comprehensive illustration of the framework and uses cases of telementry, and also gives some guidance for the future development, which will be helpful to push forward the actual utilization of telementry in future network management. 
Chongfeng 

From: Tianran Zhou
Date: 2019-03-11 10:39
To: [email-masked]
CC: [email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf
Hi WG,

 
 
Regards,
Tianran, OPSAWG Co-Chair

---

From: oliverxu(许锋) <[email-masked]>
Date: Wed, 13 Mar 2019 02:49:20 +0000

Hi WG,
   Support adoption
   We Tencent are very interested about  E2E Control Plane/Data Plan OAM Solution for DCI Backbone Network.
   this document clarifies existing approaches with pros and cons, it helps in when refining and designing new OAM solutions.

Oliver

    
    -----Original Message-----
    From: OPSAWG [mailto:[email-masked]] On Behalf Of Tianran Zhou
    Sent: Monday, March 11, 2019 10:40 AM
    To: [email-masked]
    Cc: [email-masked]
    Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf
    
    Hi WG,

    
    
    Regards,
    Tianran, OPSAWG Co-Chair

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 13 Mar 2019 18:16:18 +0100

> This poll will run until 9am in Prague on Monday 25th March.

I think there are two main questions to answer:

a) Does the IETF need a Network Telemetry Framework?
b) Is this document a suitable starting point for such a framework?

I am not sure about a) and I assume that may need more WG discussion
time to find out and concerning b) I believe the current document
needs major work and hence I would not recommend to adopt it yet
even if there is a need for a).

For the details of my reasoning, see the longer text
below. Disclaimer: I have not physically participated in the last IETF
meetings, so I only comment on what I read today and what I recall
from the mailing list.  It may not be useful to start a discussion on
each of the points I am making, I suggest you rather take this as
general input to think about in preparation of a possible discussion
in Prague.

Details follow for those interested in them...

a) Does the IETF need a Network Telemetry Framework? Which problems does
   such an architectural framework help to solve?

   For me, an architectural framework needs to serve a clear purpose.
   Possible clear purposes can be:

   - The architectural framework guides future work in the IETF.
   - The architectural framework defines concepts and terminology
     that help simplifying communication.

   The abstract and the introduction kind of indicate that this is the
   goal of this document. Section 3 goes into more details why an
   architectural framework is needed. The main arguments (leaving
   out things where I failed to see the argument):

   - A telemetry framework can help to normalize the technique developments.
   - Applications require network telemetry to be elastic.
   - Efficient data fusion is critical.
   - The lack of coherence (of the work in the IETF) makes it difficult
     to assemble a comprehensive network telemetry system and causes
     repetitive and redundant work.

   The goal seems to be to guide future work in the IETF. But what is
   the work to be guided in the IETF? The IPFIX work has concluded,
   YANG push is likely soon reaching conclusion in NETCONF, gRPC is
   not really happening within the IETF.

   From recent mailing list statements, it seems the work is much more
   geared towards helping operators to deal with the many different
   technologies being proposed. This may indeed be a problem but then
   I wonder to what extend the IETF can provide an architectural
   framework that solves this problem. The market ultimately decides
   what people use. If there is a need to summarize what is out there,
   then this may not be an architectural framework.

b) Is this document a suitable starting point for such a framework?

   I believe the document is far from being a suitable starting point.
   I am picking out some statements to make my point but the list of
   issues I have is not complete (and I do not intend to make it
   complete):

     One difference between the network telemetry and the network OAM
     is that the network telemetry assumes machines as data consumer,
     while the conventional network OAM usually assumes human
     operators.  Hence, the network telemetry can directly trigger the
     automated network operation, but the conventional OAM tools only
     help human operators to monitor and diagnose the networks and
     guide manual network operations.  The difference leads to very
     different techniques.

   I believe there is already quite some automation in place today but
   yes there are still things done by humans. I doubt that an
   architectural framework will change the level of automation since
   this is largely in the hands of the network operators and outside
   the work the IETF can actually do. From what I have heard, the
   level of automation is already quite advanced for some operators.

     Although the network telemetry techniques are just emerging and
     subject to continuous evolution, several characteristics of
     network telemetry have been well accepted.

   Not sure what 'just emerging means' but IPFIX was published 12
   years ago after ~10 years of experience with NetFlow. Papers
   discussing push vs. pull in network management appeared some 20
   years ago (e.g., doi:10.1109/INM.1999.770671).

   Section 4.1 talks about simple vs custom vs event-triggered vs
   streaming data and then it provides a figure that frankly does not
   tell me anything. First, what the authors call 'custom data' seems
   be what I would call 'aggregated' data or 'derived data' and how
   data is accessed is pretty much orthogonal to the kind of data
   itself. This is even stated but in the text but the figure entirely
   misses that point. (I also fail to see why one would reference
   finite state machines when talking about events. Technologies to
   detect threshold crossings are 20+ years old and some events just
   occur naturally.)

   While I understand that telemetry data may differ between
   data/control/management planes, I fail to see why this
   _necessarily_ impacts their encoding and transport. Or let me state
   it differently, a well designed telemetry transport solution should
   be working efficiently with any of these kinds of data. It would
   help to explain what exactly the different requirements originating
   from the different planes are instead of leaving the details of the
   requirements pretty much undefined.

   Section 5 talks about four levels of network telemetry. The
   statement that with "the help of a clearly defined network
   telemetry framework, we can assemble the technologies to support
   level 2 and make solid steps towards level 3" seems a bit wishful
   thinking. The IETF can perhaps define protocols that allow network
   operators to continuously customize the telemetry data in real time
   to reflect the network operation's visibility requirements (and the
   YANG push work might even provide that functionality), but the idea
   that the IETF can do anything to move the human out of the loop
   seems rather questionable. I think how someone runs a network is
   fundamentally not the IETF's business. The IETF is not going to
   build an "intelligent network operation engine" I think.

   The fact that RESTCONF is entirely ignored is surprising given that
   it does support different encodings and integrates well with the
   HTTP ecosystem etc.

c) Suggestions

   I believe this work has a much better chance to be useful if the
   document would start with defining what network telemetry is (and
   what it is not) and then collecting concrete requirements a network
   telemetry system should satisfy.  Once there are concrete (and
   agreeable) requirements (or objectives), you can perhaps justify an
   architectural framework and look at the question to what extend the
   state of the art aligns with the requirements of such an
   architectural framework. By putting an emphasis on a clear
   definitions of requirements and how these requirements lead to
   certain architectural concerns, the authors will be forced to think
   clearly about what makes network telemetry different from network
   management as we know it for a long time.

   There are some useful requirements hidden in the appendix. Lift
   them up and derive the architectural framework from the
   requirements.  Make sure that requirements are well defined (for
   example, 'convenient data subscription' is likely in the eye of the
   beholder since what I find convenient may differ from what you find
   convenient). And the requirements likely have to include security
   and data privacy requirements these days.
   
d) Further comments on the content

   This document sometimes reads as a whitepaper. It does not help me
   much that this document talks about AI, intent, autonomous,
   ... networks as long as these terms have not a clear definition.
   (The NMRG tried to define what intent is but has not succeeded so
   far - the devil is in the details and the industry has several
   different ideas what intent is.)

   The use cases discussed in 2.1 are considered part of network
   management for say ~20-30 years. So what is new?

   I would add NetFlow (which became IPFIX) to the list of
   technologies that have a long history. I think NetFlow appeared in
   the mid 1990s. Also note that syslog is not poll-based and that the
   SNMP MIB is an abstraction; the data a MIB exposes clearly
   originates from many different components that you mention in the
   second bullet of 2.2.)

   Syslog meanwhile can carry structured content, I am not sure how
   widely this is used. Perhaps syslog has been so successful because
   it does not require defined data models but this is likely a
   different discussion. ;-)

   I see that sFLow is popping up as conventional technology but not
   NetFlow/IPFIX. Why?

   Terminology: The notion AI = machine learning clearly will make a
   certain number of people very unhappy. If you list terms that are
   not defined in the document itself, there should be a reference to
   where these terms are defined. And I rather have clear definitions
   and not statements like 'something with characteristics defined
   somewhere in this document'. What about, say RFC 6291? Point to
   existing definitions. YANG clearly is not just a data modeling
   language for NETCONF anymore.

   I fail to understand Fig. 3. The idea that a MIB only covers
   objects of the management plane is somewhat surprising. If event
   reporting is part of the game, then, well, SNMP may still be part
   of deployed solutions. And syslog in practice often runs over UDP
   but UDP is not listed as a transport where syslog is mentioned.
   There are a number of these technical inconsistencies that need
   to be fixed.

   It seems some of the authors kind of use this document to refer to
   some of their other proposed works and this actually diminishes my
   excitement about this document. I suggest you limit yourself to
   standards or chartered work entertained by WGs or work that has
   significant industry traction. If this document comes across as an
   attempt to push the authors own ideas, it looses a bit of its
   credibility.

   There are a whole bunch of 'transmission' MIB modules covering lots
   of statistics about the different data transmission systems used by
   the data plane. I find it surprising that they are all pretty much
   ignored. Using simple counters for data plane data collection is to
   my understanding a requirement simply originating from the fact
   that things need to be done at line rate. Some of the data export
   will move from SNMP to other protocols, there is already MIB data
   export over IPFIX and you can translate MIB modules into YANG
   modules and use the YANG ecosystem using standardized
   mechanisms. The point I am making is that there are a ton of data
   models relevant for the data plane and they will continue to have
   value, whether the data is streamed is just a different way of
   accessing to the data and it does not imply that all data models
   have to be reinvented.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Haoyu song <[email-masked]>
Date: Wed, 13 Mar 2019 20:07:05 +0000

Hi Juergen,

Thank you very much for your very detailed comments. Your feedback shows exactly we need to engage with the entire community to improve this document.

Your two questions are very good. Here's my high level response first:

a) Does the IETF need a network telemetry framework?

I strongly believe so. Given the facts that the telemetry is very active and important topic, the area is already crowded, yet many misunderstandings and confusions exist, 
we do need such an informational document. Many world largest operators and OTTs have shared the same opinion.  
In fact, the comments and opinions you provided exactly show that so many things about telemetry need to be clearly clarified and well defined.

This is a working document. We don't expect at this point every statement in this document is accurate and reaches consensus. 
Yet the document has gone through many revisions which absorbed all the comments and suggestions we gathered from several working group (including OPSAWG, RTGWG, and NMRG).
We agree there are still a lot of work to do (e.g., add new content, address inconsistencies, fix mistakes, clarify concepts, etc.) but at same time it has also laid out the foundation and major pieces of the framework.  
So I believe this is qualified as a "suitable starting point".

More response please see inline.

Best regards,
Haoyu

-----Original Message-----
From: OPSAWG [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
Sent: Wednesday, March 13, 2019 10:16 AM
To: Tianran Zhou <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

	
I am not sure about a) and I assume that may need more WG discussion time to find out and concerning b) I believe the current document needs major work and hence I would not recommend to adopt it yet even if there is a need for a).

For the details of my reasoning, see the longer text below. Disclaimer: I have not physically participated in the last IETF meetings, so I only comment on what I read today and what I recall from the mailing list.  It may not be useful to start a discussion on each of the points I am making, I suggest you rather take this as general input to think about in preparation of a possible discussion in Prague.

HS>  Some works are still going on (e.g., data plane telemetry, YANG PUSH)
  and we can't conclude that we've done and no new things (or 
  updates to the existing things) are needed in the future. 
  We also need to study how to consolidate and integrate the existing
  technologies. Therefore, a framework is needed to guide the work. 

HS> The standard development, product design, and network deployment need to 
  be streamlined. A formal guideline is not only educational but also practical.

HS> Some of the following points are very good and meaningful. In future revisions, 
   we should definitely address them after thorough discussions within the WG.
   But on the other hand, these points are easy to address in future revision. 
   Saying its "far from being a suitable starting point" just based on
   these points is an overstatement IMHO.  The main contribution of this work is 
   to classify the network  telemetry techniques and components, show its position in 
   network architecture, and clarify its relationship with OAM.  

HS> Yes I should clarify what I mean is the conventional OAM technique.
   And there is not a black and white line. Everything is evolving.
   Here we don't mean to guide how network operators automate their network.
   We just focus on the standard part of it and make sure the standard
   development are well aligned with the network automation goal.
   IETF, as the SDO for many telemetry protocols, should assume such responsibility.   

HS> Correct. Some existing techniques are already qualified as telemetry. The fact that the term
   of telemetry emerges recently doesn't mean the techniques don't exist. We'll
  rephrase the statement and make the clarification. 

HS> The figure shows the different level of data complexity and the capability 
   to handle each type of data results in different techniques. The appendix
   shows some examples. Some events are much more complex than simple
   threshold crossing. FSM is a way to model them.  

HS>  For example, if one needs to stream data from forwarding chip directly, 
  it would be not possible to use heavy weighted protocols such as gRPC.
  The framework  clarifies this point: data from different plane may 
  need different techniques. There's no one size fits all solution. Probably 
  we should try to consolidate the existing techniques but we have to keep
  this in mind. 

HS> This section aims to trigger more discussions. Note this is just a working
  document. Nothing is final. We just listed everything we think might be useful. 

HS> Will add reference to it. 

c) Suggestions

HS>Very good suggestion. While the authors have tried to cover all the above mentioned points,
   surely we need to continue our work to make the arguments more logical and forcible . That's why we want to
   make it a WG document and involve experts like to contribute the work and make it better.  

HS> Putting the  review in appendix is a discussion result from previous meetings. Techniques are evolving,
   some may become obsolete and irrelevant.  So the doc is better to be organized this way.
   But you are right, requirements can be derived from these real examples. 
   We have a section to talk about the security and privacy requirements.  Again, more work needs to be done
   on this.

HS> Some use cases are new (like intent verification); some are old but
  still not very well addressed. Network telemetry offers new ways to 
  do that.

HS> We usually only provide some example techniques. We acknowledged that
   we didn't provide an exhaustive list of existing techniques. But sure,
   we can add it in the future revisions.  

HS> Strictly speaking, sFlow can be seen as telemetry. Will clarify it.

HS> Fair point.

HS> These are just example to illustrate the main point. It shows the diversity and
   difference. 

HS> Currently it covers a survey. And some works are also evolving.
   When we work it toward an RFC, only tangible and well-accepted work
   in IETF will be covered. 

HS> Good comment. We don't mean to reinvent anything in this doc.
 But if some coverage is missing or insufficient, we'll definitely improve it
 in future revisions. I just want to restate that this is just a starting point
 and we agree there's still a big room to improve the document. 
 Your review and suggestions are very needed, timely, and highly appreciated!

/js

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 14 Mar 2019 11:06:40 -0000

Hi Tianran, working group,

Tl;dr -- I support adoption

I read Juergen's very substantive comments, and I probably need to go back
and re-read them. Juergen is a beacon for how IETF participants should
contribute constructively and in detail.

His detailed comments and suggestions for improvement to the document seem
to me to raise very valid points. This is exactly how the working group
process should work so that individual participants can guide and contribute
to Internet-Drafts. I believe that Juergen's comments on the document would
be best addressed by bringing the document into the working group and
developing improvements under the care of the working group.

However, in answering "Does the IETF need a Network Telemetry Framework?"
Juergen provides a number of additional pointers to where and how the
document could be improved to make its scope more useful. I think I agree
with his guidance about making the document better, but these are not points
that address the specific question.

My view is that it is the job of area-wide or broad-scoped working groups to
guide other work in the IETF by providing definitions, requirements,
frameworks, and architectures. Telemetry is clearly an area of increasing
interest and relevance, and I think it is for the OPSAWG to provide the lead
on this topic.

I think that this document could serve as a place to collect the
definitions, requirements, a framework, and an architecture for telemetry. I
am not hung up on its current name as a "framework" and believe that that is
a good enough umbrella under which to place all of the necessary text.

I believe that the OPSAWG is the right place to do this work and that this
document provides a good enough starting point. I'd like to see it adopted
and then updated particularly in line with Juergen's comments.

Thanks,
Adrian

-----Original Message-----
From: OPSAWG <[email-masked]> On Behalf Of Tianran Zhou
Sent: 11 March 2019 02:40
To: [email-masked]
Cc: [email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

Regards,
Tianran, OPSAWG Co-Chair

---

From: <[email-masked]>
Date: Sun, 17 Mar 2019 10:09:00 +0000

Hi Tianran and co-authors,

I support the adoption from a network operator perspective. Thank you very much that you driving this important topic at IETF.

I agree with the authors that IETF needs to ensure that all activities covered by Network Telemetry has to be coordinated to allow metric correlation and Big Data integration to gain visibility into our networks. Without visibility into our networks, networks and automation won't reach maturity levels we know from other industries.

I like to add the following input:

Section 3, "The Necessity of a Network Telemetry Framework", paragraph "Network visibility presents multiple viewpoints"

--

In section 4.2 "Data Objects" you lay out nicely the 3 different areas metrics can be collected. Control-Plane, Forwarding-Plane and Management (device, physical topology). It is important to describe that if metrics are correlated between these 3 areas, metrics can be viewed from these 3 different viewpoints. Examples:

Traffic Impact:                                                From forwarding-plane perspective we can verify if traffic impact was caused due to a control-plane or a management plane change or not.

Service Provisioning:                                    From a control-plane perspective we can verify if service provisioning was successful and undesired traffic or device impact was caused.

Physical Topology Modification:               From a device/physical topology perspective, we can verify when new devices are introduced or existing are changed or upgraded, if control-plane worked, forwarding path was adjusted to redundant device and no traffic impact is visible from a customer/service perspective.

For a automated closed loop operation this is essential. Same as an autopilot in an modern airplane works, it needs sensors covering multiple aspects. When change is performed, it has to verify if desired outcome is achieved.

These metric correlation between control-plane, forwarding-plane and management (device, physical topology) can only be achieved when:

1.           Each area has key fields which contain the same values

2.           These time series metrics have same timestamps

In those areas, IETF and this NTF draft needs to play an important role by not only point out which key metrics exists (find the right amount is key), it also has to ensure that within RFC's covering metric collection, enrichment and correlation, this aspects are fostered.

Example1: With "if-index" field in ietf-interfaces.yang  model (A YANG Data Model for Interface Management, RFC 7223) we are able to correlate to IPFIX (RFC 7011) entity 10 (ingressInterface) and 14 (egressInterface).

Example2: With "ip address" field in ietf-ip.yang model (A YANG Data Model for IP Management, RFC 8344) we are able to correlate BGP next-hop attribute in BMP (BGP Monitoring Protocol, RFC 7854).

Example3: With IPFIX entity 90 (mplsVpnRouteDistinguisher) we are able to correlate to BGP route-distinguisher in BMP (BGP Monitoring Protocol, RFC 7854).

If you look at the last example and the following IETF draft which wants to enrich forwarding-plane with BGP metrics on the flow exporter, we have to ask ourselves if IPFIX entity 90 is a key field and should be part of any RFC which correlates between BGP control-plane and forwarding-plane.

Export BGP community information in IP Flow Information Export (IPFIX)

https://tools.ietf.org/html/draft-ietf-opsawg-ipfix-bgp-community-12

Another aspect which is currently not covered is schema conversion. Network metrics have schema definitions (YANG, IPFIX template, BMP message format) which are currently not supported in Big Data environment. Widely used in Hadoop environments is AVRO (https://en.wikipedia.org/wiki/Apache_Avro). We need to describe and allow that schema conversion is possible.

Further, IETF should take the step towards Big Data and standardize a schema language which can be used at Big Data and network metric schema's can be easily converted to. Without proper schema conversion, collected metrics can't be automatically processed at scale within Big Data environments.

Kind regards

Thomas Graf

____________________________________________________________________________

Network Engineer

Datacenter Functions

Telefon +41-58-223 84 01

Mobile  +41-79-728 80 12

[email-masked]

____________________________________________________________________________

Swisscom (Schweiz) AG

IT, Network & Infrastructure

Datacenter Functions

Binzring 17

8045 Zürich

www.swisscom.com

Postadresse:

Binzring 17

8045 Zürich

-----Original Message-----
From: OPSAWG <[email-masked]> On Behalf Of Tianran Zhou
Sent: Monday, March 11, 2019 3:40 AM
To: [email-masked]
Cc: [email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

Regards,

Tianran, OPSAWG Co-Chair

OPSAWG mailing list

---

From: Haoyu song <[email-masked]>
Date: Mon, 18 Mar 2019 17:49:22 +0000

Hi Thomas,

Thank you very much for your support! I'm glad to see that the vision of the draft is well aligned with yours.
You pointed out a very important issue (metrics correlation) that definitely needs to be addressed by IETF. We'll articulate this based on your comments.
Schema conversion is also a real issue and solid requirement. My only concern is if this is in the working scope of IETF. Maybe for the completeness of the work, we should also mention it at least. Or we should think about how to make the conversion easier.

We are looking forward to your further help  and contribution!

Best regards,
Haoyu

From: OPSAWG [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Sunday, March 17, 2019 3:09 AM
To: Tianran Zhou <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

I agree with the authors that IETF needs to ensure that all activities covered by Network Telemetry has to be coordinated to allow metric correlation and Big Data integration to gain visibility into our networks. Without visibility into our networks, networks and automation won't reach maturity levels we know from other industries..

--

Another aspect which is currently not covered is schema conversion.. Network metrics have schema definitions (YANG, IPFIX template, BMP message format) which are currently not supported in Big Data environment. Widely used in Hadoop environments is AVRO (https://en.wikipedia.org/wiki/Apache_Avro). We need to describe and allow that schema conversion is possible.

Kind regards

Thomas Graf

Network Engineer

Datacenter Functions

Telefon +41-58-223 84 01

Mobile  +41-79-728 80 12

[email-masked]<mailto:[email-masked]>

Swisscom (Schweiz) AG

Datacenter Functions

Binzring 17

8045 Zürich

www.swisscom.com<http://www.swisscom.com>

Postadresse:

Binzring 17

8045 Zürich

-----Original Message-----
From: OPSAWG <[email-masked]<mailto:[email-masked]>> On Behalf Of Tianran Zhou
Sent: Monday, March 11, 2019 3:40 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

Regards,

Tianran, OPSAWG Co-Chair

OPSAWG mailing list

---

From: <[email-masked]>
Date: Mon, 18 Mar 2019 18:53:37 -0000

Hi All, 

Yes, I fully support the adoption of the I-D. 

The OPSAWG is a great home for the I-D. Thomas, Juergen and Adrian, and
others have all made excellent observations and suggestions to develop the
document. It is refreshing to see polls where it is not just a binary yes/no
flag. :-)

A few general observations:

o Having a clear definition of "network telemetry" is good, and this
document goes a long way to help clarify the role of telemetry and
differences from traditional OAM in current networks. For completeness it
may be worth including references to other formal SDOs that also have
network telemetry definitions;

o It may be useful to expand on the Data Objects (section 4.2) and add a
column in figure 3 that highlights the various types of data
(event/metric/route) and query functions. You could even have a short
sub-section that highlights the various combinations of management
functions, roles, queries and data types:

Fault Management (event data)
Performance Management (metric data)
Billing Management (event data and metric data)
Capacity Management (metric data)
Security Management (metric data and route data)

o The security section in the I-D contains some useful high-level discussion
text. The section needs further development, and I will be glad to make more
detailed contributions. 

BR, Dan. 
	
From: OPSAWG <[email-masked]> On Behalf Of Haoyu song
Sent: 18 March 2019 17:49
To: [email-masked]; Tianran Zhou <[email-masked]>;
[email-masked]
Cc: [email-masked]
Subject: Re: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi Thomas,

Thank you very much for your support! I’m glad to see that the vision of the
draft is well aligned with yours.
You pointed out a very important issue (metrics correlation) that definitely
needs to be addressed by IETF. We’ll articulate this based on your comments.
Schema conversion is also a real issue and solid requirement. My only
concern is if this is in the working scope of IETF. Maybe for the
completeness of the work, we should also mention it at least. Or we should
think about how to make the conversion easier. 

Best regards,
Haoyu

-----Original Message-----
From: OPSAWG <mailto:[email-masked]> On Behalf Of Tianran Zhou
Sent: Monday, March 11, 2019 3:40 AM
To: mailto:[email-masked]
Cc: mailto:[email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

this topic, please say why..

Regards,
Tianran, OPSAWG Co-Chair

mailto:[email-masked]

---

From: Haoyu song <[email-masked]>
Date: Mon, 18 Mar 2019 21:35:46 +0000

Hi Daniel,

Thank you very much for the support! Your comments and suggestions are all noted and we look forward your contributions. 

We couldn't find a tight definition of network telemetry anywhere else although the term has been widely used. That's why I think we should take the responsibility. Of course if anybody is aware of that, please feel free to let us know.
Our current definition is intended to be loose enough to cover a wide range of techniques yet to catch the essence of the term. We believe the WG can help to make it well defined and widely accepted. 

Best regards,
Haoyu 

-----Original Message-----
From: Daniel King [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 18, 2019 11:54 AM
To: [email-masked]; Haoyu song <[email-masked]>
Cc: [email-masked]
Subject: RE: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi All, 

The OPSAWG is a great home for the I-D. Thomas, Juergen and Adrian, and others have all made excellent observations and suggestions to develop the document. It is refreshing to see polls where it is not just a binary yes/no flag. :-)

o Having a clear definition of "network telemetry" is good, and this document goes a long way to help clarify the role of telemetry and differences from traditional OAM in current networks. For completeness it may be worth including references to other formal SDOs that also have network telemetry definitions;

o It may be useful to expand on the Data Objects (section 4.2) and add a column in figure 3 that highlights the various types of data
(event/metric/route) and query functions. You could even have a short sub-section that highlights the various combinations of management functions, roles, queries and data types:

Fault Management (event data)
Performance Management (metric data)
Billing Management (event data and metric data) Capacity Management (metric data) Security Management (metric data and route data)

o The security section in the I-D contains some useful high-level discussion text. The section needs further development, and I will be glad to make more detailed contributions. 

BR, Dan. 
	
To: [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]

Hi Thomas,

Best regards,
Haoyu

-----Original Message-----
From: OPSAWG <mailto:[email-masked]> On Behalf Of Tianran Zhou
Sent: Monday, March 11, 2019 3:40 AM
To: mailto:[email-masked]
Cc: mailto:[email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

This email starts a poll for adoption. 
If you support adopting this document please say so, and please give an indication of why you think it is important. Also please say if you will be willing to review and help the draft.
If you do not support adopting this document as a starting point for work on this topic, please say why..
This poll will run until 9am in Prague on Monday 25th March.

Regards,
Tianran, OPSAWG Co-Chair

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 19 Mar 2019 15:26:52 +0800

Support WG adoption.
The draft is beneficial for us to deploy telemety.
Thanks.

HUANAN CHEN
 
-----Original Message-----
From: OPSAWG [mailto:[email-masked]] On Behalf Of Tianran Zhou
Sent: Monday, March 11, 2019 10:40 AM
To: [email-masked]
Cc: [email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf
 
Hi WG,

 
 
Regards,
Tianran, OPSAWG Co-Chair

---

From: "Ciavaglia, Laurent (Nokia - FR/Paris-Saclay)" <[email-masked]>
Date: Sun, 24 Mar 2019 22:55:32 +0000

Hello,

I'm a co-author of the draft.
I support its adoption as WG document.
The draft is a good starting point for further work from the WG, including the relevant comments made on missing/incomplete  aspects.
Various forms of telemetry are important components of modern networks, their operation and management. Clarifying concepts and reference elements, and providing guidelines for designers, implementers and users is a valuable result/realization.

Best regards, Laurent.
________________________________
From: OPSAWG <[email-masked]> on behalf of Tianran Zhou <[email-masked]>
Sent: Monday, March 11, 2019 3:39:44 AM
To: [email-masked]
Cc: [email-masked]
Subject: [OPSAWG] WG adoption poll for draft-song-opsawg-ntf

Hi WG,

Regards,
Tianran, OPSAWG Co-Chair