Subject: [OPSAWG] Specifying protocols in draft-claise-opsawg-collected-data-manifest

From: Jean Quilbeuf <[email-masked]>
Date: Wed, 9 Feb 2022 12:32:36 +0000

Dear All,
We are wondering whether to add information about protocols in the data manifest draft https://datatracker.ietf.org/doc/draft-claise-opsawg-collected-data-manifest/ 

Details are here https://github.com/JeanQuilbeufHuawei/draft-collected-data-manifest/issues/9  (that git repo contains a pre-01 version of the draft).

The data manifest should contain the information that a device able to stream telemetry can gather to allow a posteriori understanding of values stored in a time series database for instance. In that context, knowing the protocol would enable to understand whether some metrics can be missed (for instance if UDP push is used) and explain some gaps in the time serie.

1 - In the model, do we want to encode that fact as a Boolean (such as unreliable_subscription) that would be attached to a particular MDT subscription or do we want to specify the protocol used for subscription and let the consumer of the model draw the conclusion themselves?

2 - Another point is about the encoding used, do we need to specify it in the data-manifest or do we leave this as a responsibility of the collector/database system? 

For the second question, not that he collector/database system still  has the responsibility to modify the data manifest if that encoding is changed.

Any suggestions?

Best,
Jean

---

From: Tianran Zhou <[email-masked]>
Date: Wed, 9 Feb 2022 13:04:05 +0000

Hi Jean,
I am a little confused about this manifest?
Can we just read from the device about the configuration? We can get all the running information.
Best,
Tianran

________________________________

Sent from WeLink
发件人： Jean Quilbeuf<jean.quilbeuf=[email-masked]<mailto:jean.quilbeuf=[email-masked]>>
收件人： opsawg<[email-masked]<mailto:[email-masked]>>
主题： [OPSAWG] Specifying protocols in draft-claise-opsawg-collected-data-manifest
时间： 2022-02-09 20:33:06

Dear All,

Any suggestions?

Best,
Jean
_______________________________________________
OPSAWG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/opsawg

---

From: Jean Quilbeuf <[email-masked]>
Date: Wed, 9 Feb 2022 14:29:21 +0000

Hi Tiaran,
The value of this draft is not to expose new information via YANG but rather to define what needs to be kept as metadata (or data manifest) to ensure that the data can still be interpreted correctly even if the device is not present anymore, has been updated  or has a new configuration.

The proposed model does not necessarily have to be implemented on the device, it could be generated by the collector by collecting in parallel all the information already present, although having it available directly on the device would simplify the collector operation.

Best,
Jean

From: OPSAWG [mailto:[email-masked]] On Behalf Of Tianran Zhou
Sent: Wednesday 9 February 2022 13:04
To: Jean Quilbeuf <jean.quilbeuf=[email-masked]>; opsawg <[email-masked]>
Subject: Re: [OPSAWG] Specifying protocols in draft-claise-opsawg-collected-data-manifest

Hi Jean,

Sent from WeLink

Dear All,

Any suggestions?

Best,
Jean
_______________________________________________
OPSAWG mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/opsawg

---

From: Benoit Claise <[email-masked]>
Date: Wed, 9 Feb 2022 15:42:04 +0100

Hi Tianran,

> Can we just read from the device about the configuration? We can get all the running information.
I'm not sure whether this is a generic question or whether your question 
relates to the message below.
Let me answer the generic question. We don't always want to read the 
devices information from the closed-loop automation systems or the 
database. Once known ...

    "This data manifest instance file MUST be streamed all with the data
    and stored along with the collected data.  In case the data are moved
    to different place (typically a database), the data manifest MUST
    follow the collected data.  This can render the data unusable if that
    context is lost, for instance when the data is stored without the
    relevent information."

Also the context might be lost, so  not available any longer from the 
device Ex: how was it metered?

This is what we were trying to say with this generic sentence in the 
charter: "This can render the data unusable if that context is lost, for 
instance when the data is stored without the relevent information."

There is some more justifications in the introduction (section 2).

Regards, Benoit
> Best,
> Tianran
>
>
>
>
> ------------------------------------------------------------------------
>
> Sent from WeLink
> *发件人： *Jean Quilbeuf<jean.quilbeuf=[email-masked]>
> *收件人： *opsawg<[email-masked]>
> *主题： *[OPSAWG] Specifying protocols in 
> draft-claise-opsawg-collected-data-manifest
> *时间： *2022-02-09 20:33:06
>
> Dear All,
> We are wondering whether to add information about protocols in the 
> data manifest draft 
> https://datatracker.ietf.org/doc/draft-claise-opsawg-collected-data-manifest/ 
>
>
> Details are here 
> https://github.com/JeanQuilbeufHuawei/draft-collected-data-manifest/issues/9 
> (that git repo contains a pre-01 version of the draft).
>
> The data manifest should contain the information that a device able to 
> stream telemetry can gather to allow a posteriori understanding of 
> values stored in a time series database for instance. In that context, 
> knowing the protocol would enable to understand whether some metrics 
> can be missed (for instance if UDP push is used) and explain some gaps 
> in the time serie.
>
> 1 - In the model, do we want to encode that fact as a Boolean (such as 
> unreliable_subscription) that would be attached to a particular 
> MDT subscription or do we want to specify the protocol used for 
> subscription and let the consumer of the model draw the conclusion 
> themselves?
>
> 2 - Another point is about the encoding used, do we need to specify it 
> in the data-manifest or do we leave this as a responsibility of the 
> collector/database system?
>
> For the second question, not that he collector/database system still  
> has the responsibility to modify the data manifest if that encoding is 
> changed.
>
> Any suggestions?
>
> Best,
> Jean

---

From: Tianran Zhou <[email-masked]>
Date: Thu, 10 Feb 2022 01:46:03 +0000

Thanks Jean and Benoit for your clarification.
I think this manifest information is useful.
One more justification IMO might be useful.
When controller and collector are on separate servers, which is a very common deployment, collector may not have access to the device configuration.

T.
From: OPSAWG [mailto:[email-masked]] On Behalf Of Benoit Claise
Sent: Wednesday, February 9, 2022 10:42 PM
To: Tianran Zhou <zhoutianran=[email-masked]>; Jean Quilbeuf <jean.quilbeuf=[email-masked]>; opsawg <[email-masked]>
Subject: Re: [OPSAWG] Specifying protocols in draft-claise-opsawg-collected-data-manifest

Hi Tianran,
On 2/9/2022 2:04 PM, Tianran Zhou wrote:
Hi Jean,
I am a little confused about this manifest?
Can we just read from the device about the configuration? We can get all the running information.
I'm not sure whether this is a generic question or whether your question relates to the message below.
Let me answer the generic question. We don't always want to read the devices information from the closed-loop automation systems or the database. Once known ...

   relevent information."
Also the context might be lost, so  not available any longer from the device Ex: how was it metered?

This is what we were trying to say with this generic sentence in the charter: "This can render the data unusable if that context is lost, for instance when the data is stored without the relevent information."

Regards, Benoit

Best,
Tianran

Sent from WeLink

Dear All,

Any suggestions?

Best,
Jean

OPSAWG mailing list

---

From: <[email-masked]>
Date: Mon, 14 Feb 2022 14:12:20 +0000

Hi Benoît, Jean,

FWIW, you may find some comments/suggestions at:

  *   pdf: https://github.com/boucadair/IETF-Drafts-Reviews/raw/master/draft-claise-opsawg-collected-data-manifest-00-rev%20Med.pdf
  *   doc: https://github.com/boucadair/IETF-Drafts-Reviews/raw/master/draft-claise-opsawg-collected-data-manifest-00-rev%20Med.doc

Some of the context information is not specific to telemetry, but can be applied in other contexts to assess the reliability of collected data. Generalizing the concept would make sense.

Cheers,
Med

De : OPSAWG <[email-masked]> De la part de Benoit Claise
Envoyé : mercredi 9 février 2022 15:42
À : Tianran Zhou <zhoutianran=[email-masked]>; Jean Quilbeuf <jean.quilbeuf=[email-masked]>; opsawg <[email-masked]>
Objet : Re: [OPSAWG] Specifying protocols in draft-claise-opsawg-collected-data-manifest

Hi Tianran,

   relevent information."

Regards, Benoit

Best,
Tianran

Sent from WeLink

Dear All,

Any suggestions?

Best,
Jean

OPSAWG mailing list

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.