Subject: [6tisch] FW: New Version Notification for draft-karaagac-6tisch-int-00.txt

From: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>
Date: Tue, 14 Jan 2020 09:43:28 +0000

Dear all,
We have submitted a new draft describing an efficient and timely monitoring solution for 6TiSCH Networks. We believe that this mechanism can become a very powerful tool for 6TiSCH Networks with the contribution of the 6TiSCH community.

All kinds of feedback and comments are very welcome. Thanks a lot!

Kind regards,
Abdulkadir

-----Original Message-----
From: [email-masked] <[email-masked]> 
Sent: Saturday, January 11, 2020 23:21
To: Jeroen Hoebeke (UGent-imec) <[email-masked]>; Abdulkadir Karaağaç (UGent-imec) <[email-masked]>
Subject: New Version Notification for draft-karaagac-6tisch-int-00.txt

A new version of I-D, draft-karaagac-6tisch-int-00.txt has been successfully submitted by Abdulkadir Karaagac and posted to the IETF repository.

Name:		draft-karaagac-6tisch-int
Revision:	00
Title:		In-band Network Telemetry for 6TiSCH Networks
Document date:	2020-01-11
Group:		Individual Submission
Pages:		14
URL:            https://www.ietf.org/internet-drafts/draft-karaagac-6tisch-int-00.txt
Status:         https://datatracker.ietf.org/doc/draft-karaagac-6tisch-int/
Htmlized:       https://tools.ietf.org/html/draft-karaagac-6tisch-int-00
Htmlized:       https://datatracker.ietf.org/doc/html/draft-karaagac-6tisch-int

Abstract:
   This document describes In-band Network Telemetry for 6TiSCH
   Networks, offering a flexible monitoring solution with minimal
   resource consumption and communication overhead while supporting a
   wide range of monitoring operations and strategies for dealing with
   various network scenarios and use cases.  It enables 6TiSCH networks
   to collect per-packet and per-hop monitoring information by
   piggybacking telemetry information onto the data packets by
   exploiting the remaining space in the IEEE 802.15.4e frames, thus not
   impacting network behavior and performance.  This document also
   discusses the data fields and associated data types for 6TiSCH INT
   mechanism.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 14 Jan 2020 13:11:00 +0000

Dear authors

This draft strikes me as very mature for a 00, and a very useful piece of work as well.
It looks like you have implemented and tried it out already, correct?

A few comments and questions:

* Since you are using IEs Fig 1 could represent the insertion before the IPv6 payload.
* There could be a Fig describing the typical format of the inserted IE at the end of 2.2
* A discussion on the chances of loss that are augmented as the frame are enlarged could be useful in the intro
* A discussion of the INT header used as a hidden channel could be useful in the security section
* It seems that the data can only be reported on packets that are going outwards. You may want to indicate that in a classical non-storing network with the root acting as backbone router, all packets fly through the 6BBR and can be instrumented. 
* Because of the Q bit it seems that the INT IE is also used in packets going down: the source routing header may make it complicated to instrument such packets. More description on the operation and size of inserted header in downwards packets could be useful. Maybe we could make the query a different IE altogether and save the Q bit for the future. 
* If the telemetry relates to a transmission by a child, then the node ID is not enough to identify the link since the child has multiple parents.
* Apparently in HbH mode 1 only the first nodes may insert data if the network is deep and the room limited. 
* In HbH mode 2, how do we balance the use of the space so that all hops have an equal chance to report their information? Is that what 2.3.1 is all about? 
* In section 2.3 
** I believe that if nodes use different strategies some may starve. Like for the objective function, there should be a single strategy, that may have to be tuned for the shape of the network. 
** in 2.3.1 if all nodes source traffic and not only the deepest ones, actually the nodes near the root may have more opportunities, it really depends on the shape of the network.
** in 2.3.2 we seem to assume that only nodes deep in the network source packets. In reality, say that there can be 3 values stored, then the number of chances is proportional to the number of children and grandchildren, and nodes deep in the network may have less of these.
** it could be good that a node that refrains to insert redundant information may still indicate that already sent information (sequence) is unchanged. Along that line of thought how do we handle loss? A same sequence sent several times?
* Security section should indicate that the payload IE is modified at each hop and there is no guarantee that the original information is preserved

Great work!

Pascal

> From: 6tisch <[email-masked]> On Behalf Of Abdulkadir Karaagaç
> (UGent-imec)
> Sent: mardi 14 janvier 2020 10:43
> To: [email-masked]
> Subject: [6tisch] FW: New Version Notification for draft-karaagac-6tisch-int-
> 00.txt
> 
> Dear all,
> We have submitted a new draft describing an efficient and timely monitoring
> solution for 6TiSCH Networks. We believe that this mechanism can become a
> very powerful tool for 6TiSCH Networks with the contribution of the 6TiSCH
> community.
> 
> To: Jeroen Hoebeke (UGent-imec) <[email-masked]>; Abdulkadir
> Karaağaç (UGent-imec) <[email-masked]>
> A new version of I-D, draft-karaagac-6tisch-int-00.txt has been successfully
> submitted by Abdulkadir Karaagac and posted to the IETF repository.
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 
> _______________________________________________
> 6tisch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6tisch

---

From: <[email-masked]>
Date: Tue, 14 Jan 2020 13:56:50 +0000

Hi,

I have comments, some of which Pascal's email covers, though.

- L3 seems a better place to implement such a mechanism than L2. L2 has no idea whether the packet (frame) under process is going toward the border router nor not. 
- we need two Node IDs, source (sender) Node ID and destination (receiver) Node ID, to express a link
- we may want 64-bit Node ID
- Do you have an implementation of a scheduling algorithm using this proposal? I cannot immediately see how useful the default set of data defined in Table 1 is... Why isn't link PDR listed there...? I may miss something...

pascal> It looks like you have implemented and tried it out already, correct?

I'm interested in this part ;-)

Best,
Yatch

> From: 6tisch [mailto:[email-masked]] On Behalf Of Pascal Thubert
> (pthubert)
> Sent: Tuesday, January 14, 2020 10:11 PM
> To: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>;
> [email-masked]
> Subject: Re: [6tisch] New Version Notification for
> draft-karaagac-6tisch-int-00.txt
> 
> Dear authors
> 
> * It seems that the data can only be reported on packets that are going
> outwards. You may want to indicate that in a classical non-storing network
> with the root acting as backbone router, all packets fly through the 6BBR and
> can be instrumented.
> * Because of the Q bit it seems that the INT IE is also used in packets going
> down: the source routing header may make it complicated to instrument such
> packets. More description on the operation and size of inserted header in
> downwards packets could be useful. Maybe we could make the query a
> different IE altogether and save the Q bit for the future.
> * If the telemetry relates to a transmission by a child, then the node ID is not
> enough to identify the link since the child has multiple parents.
> * In HbH mode 2, how do we balance the use of the space so that all hops have
> an equal chance to report their information? Is that what 2.3.1 is all about?
> * In section 2.3
> ** I believe that if nodes use different strategies some may starve. Like for the
> objective function, there should be a single strategy, that may have to be tuned
> for the shape of the network.
> ** in 2.3.1 if all nodes source traffic and not only the deepest ones, actually the
> nodes near the root may have more opportunities, it really depends on the
> shape of the network.
> ** in 2.3.2 we seem to assume that only nodes deep in the network source
> packets. In reality, say that there can be 3 values stored, then the number of
> chances is proportional to the number of children and grandchildren, and
> nodes deep in the network may have less of these.
> ** it could be good that a node that refrains to insert redundant information may
> still indicate that already sent information (sequence) is unchanged. Along
> that line of thought how do we handle loss? A same sequence sent several
> times?
> * Security section should indicate that the payload IE is modified at each hop
> and there is no guarantee that the original information is preserved
> 
> Great work!
> 
> Pascal
> 
> 
> 
> 
> 
> > Subject: [6tisch] FW: New Version Notification for
> > draft-karaagac-6tisch-int- 00.txt
> >
> > Dear all,
> > We have submitted a new draft describing an efficient and timely
> > monitoring solution for 6TiSCH Networks. We believe that this
> > mechanism can become a very powerful tool for 6TiSCH Networks with the
> > contribution of the 6TiSCH community.
> >
> > A new version of I-D, draft-karaagac-6tisch-int-00.txt has been
> > successfully submitted by Abdulkadir Karaagac and posted to the IETF
> repository.
> >
> > URL:
> https://www.ietf.org/internet-drafts/draft-karaagac-6tisch-int-
> > 00.txt
> > Htmlized:
> https://datatracker.ietf.org/doc/html/draft-karaagac-6tisch-int
> >
> >
> > Abstract:
> > Please note that it may take a couple of minutes from the time of
> > submission until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
> >

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 14 Jan 2020 14:17:13 +0000

Hello Yatch : )

> - L3 seems a better place to implement such a mechanism than L2. L2 has no idea whether the packet (frame) under process is going toward the border router nor not. 

True, at L3 we could sign end to end. 

On the other hand it is hard to inject headers on the fly at L3 l, you have to do IP in IP. The IE technique avoids that problem.

In non storing all packets reach the root so I guess the proposal has a wide domain of applicability.

Take care,

Pascal

---

From: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>
Date: Tue, 14 Jan 2020 14:18:43 +0000

Dear Pascal, all,
Indeed, we have already implemented and validated the described mechanism in a real network. Then we thought that this work might be also interesting for the 6TiSCH community and it might be really beneficial to receive feedback from you all. Therefore, we decided to turn this work to an Internet Draft.

Thank you very much for your nice comments and very useful feedback. A new version of the draft will shortly follow which will be adapted based on your comments. 

In addition, I tried to respond your comments and questions below:

* Since you are using IEs Fig 1 could represent the insertion before the IPv6 payload.
-> I agree. We will update the figure in a way that represents the real position of the IEs. Maybe [Header+IEs+Payload]

* There could be a Fig describing the typical format of the inserted IE at the end of 2.2
-> We will try to add a descriptive figure for a typical INT IE format.

-> Agreed!

* A discussion of the INT header used as a hidden channel could be useful in the security section.
-> Agreed!

-> Agreed!

* Because of the Q bit it seems that the INT IE is also used in packets going down: the source routing header may make it complicated to instrument such packets. More description on the operation and size of inserted header in downwards packets could be useful. Maybe we could make the query a different IE altogether and save the Q bit for the future. 
-> Indeed, although that part is not that mature, we also aimed a support for the downwards packets. I guess further consideration is needed in this regard.

* If the telemetry relates to a transmission by a child, then the node ID is not enough to identify the link since the child has multiple parents.
-> If the parent node is also able to add INT data (including node ID) to the same packet, then it will be possible to identify the link from these node IDs. If not, then as you said, the node id will not be sufficient for determining the links. Then, maybe the nodes can leverage on the previous or following telemetry information and try to infer the link?

* Apparently in HbH mode 1 only the first nodes may insert data if the network is deep and the room limited. 
-> It depends on the INT insertion strategy. Actually, a probabilistic approach (as described in 2.3.2) can enable middle nodes to insert data to some extend.

* In HbH mode 2, how do we balance the use of the space so that all hops have an equal chance to report their information? Is that what 2.3.1 is all about? 
-> Actually, we tried to aim a balance by using a probabilistic INT strategy in 2.3.2. 

* In section 2.3
** I believe that if nodes use different strategies some may starve. Like for the objective function, there should be a single strategy, that may have to be tuned for the shape of the network.  
-> I agree that more flexible and tunable strategies are required for larger networks. But I am not sure about the use of network-wide single strategy.

** in 2.3.1 if all nodes source traffic and not only the deepest ones, actually the nodes near the root may have more opportunities, it really depends on the shape of the network.
** in 2.3.2 we seem to assume that only nodes deep in the network source packets. In reality, say that there can be 3 values stored, then the number of chances is proportional to the number of children and grandchildren, and nodes deep in the network may have less of these.
-> If you consider the whole network and all available traffic, these comments correct. But we are trying to balance/compare the INT insertion opportunities of different nodes for a single flow.

** it could be good that a node that refrains to insert redundant information may still indicate that already sent information (sequence) is unchanged. Along that line of thought how do we handle loss? A same sequence sent several times?
-> Actually, we think that each INT entry carries telemetry information for that particular packet. If that packet is lost, so the INT entries are too. 

* Security section should indicate that the payload IE is modified at each hop and there is no guarantee that the original information is preserved.
-> Agreed!

Kind regards,
Abdulkadir

PS: It is possible to reach further details about our work in our paper named "In-Band Network Telemetry in Industrial Wireless Sensor Networks".

-----Original Message-----
From: Pascal Thubert (pthubert) <[email-masked]> 
Sent: Tuesday, January 14, 2020 14:11
To: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>; [email-masked]
Subject: RE: New Version Notification for draft-karaagac-6tisch-int-00.txt

Dear authors

Great work!

Pascal

> successfully submitted by Abdulkadir Karaagac and posted to the IETF repository.
>

---

From: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>
Date: Tue, 14 Jan 2020 14:55:33 +0000

Dear Yatch,
Thanks for the comments and your interest! 

- L3 seems a better place to implement such a mechanism than L2. L2 has no idea whether the packet (frame) under process is going toward the border router nor not. 
-> In addition to what Pascal pointed out, we aim to minimize the resource usage and the impact on the forwarding behavior (especially for critical data). For instance, an entry insertion in the L3 may easily lead to 6lowpan fragmentation which causes extra packets. But here at L2, we avoid it any means.

- we need two Node IDs, source (sender) Node ID and destination (receiver) Node ID, to express a link
-> Maybe, we should consider more packet-oriented. Maybe, it is not harmful if we don’t know which link is exactly used for some packets.
-> Or maybe we can already infer the link information from the Node-IDs of the following INT entries.

- we may want 64-bit Node ID
-> If you consider INT data (including Node IDs) insertion in every hop towards the BBR, 64bit Node-Ids will be too costly. The frames will become full easily which will decrease the INT insertion frequency. That is why we considered 8-bit IDs in the first hand.

- Do you have an implementation of a scheduling algorithm using this proposal? I cannot immediately see how useful the default set of data defined in Table 1 is... Why isn't link PDR listed there...? I may miss something...
-> No we don’t have a scheduling algorithm using this yet. But we implemented and validated this mechanism as an efficient monitoring solution that could serve to any kind of scheduling mechanism.
->  Maybe our paper ("In-Band Network Telemetry in Industrial WirelessSensor Networks") can be useful to see what could be achieved with the given set of data. And we totally agree that other metrics could/should be added. 

Kind regards,
Abdulkadir

-----Original Message-----
From: Pascal Thubert (pthubert) <[email-masked]> 
Sent: Tuesday, January 14, 2020 15:17
To: [email-masked]
Cc: Abdulkadir Karaağaç (UGent-imec) <[email-masked]>; [email-masked]
Subject: Re: [6tisch] New Version Notification for draft-karaagac-6tisch-int-00.txt

Hello Yatch : )

Take care,

Pascal 
>> * A discussion on the chances of loss that are augmented as the frame 
>> are enlarged could be useful in the intro
>> * It seems that the data can only be reported on packets that are 
>> going outwards. You may want to indicate that in a classical 
>> non-storing network with the root acting as backbone router, all 
>> packets fly through the 6BBR and can be instrumented.
>> down: the source routing header may make it complicated to instrument 
>> such packets. More description on the operation and size of inserted 
>> header in downwards packets could be useful. Maybe we could make the 
>> query a different IE altogether and save the Q bit for the future.
>> * If the telemetry relates to a transmission by a child, then the 
>> node ID is not enough to identify the link since the child has multiple parents.
>> * Apparently in HbH mode 1 only the first nodes may insert data if 
>> the network is deep and the room limited.
>> * In HbH mode 2, how do we balance the use of the space so that all 
>> hops have an equal chance to report their information? Is that what 2.3.1 is all about?
>> * In section 2.3
>> ** I believe that if nodes use different strategies some may starve. 
>> Like for the objective function, there should be a single strategy, 
>> that may have to be tuned for the shape of the network.
>> ** in 2.3.1 if all nodes source traffic and not only the deepest 
>> ones, actually the nodes near the root may have more opportunities, 
>> it really depends on the shape of the network.
>> ** in 2.3.2 we seem to assume that only nodes deep in the network 
>> source packets. In reality, say that there can be 3 values stored, 
>> then the number of chances is proportional to the number of children 
>> and grandchildren, and nodes deep in the network may have less of these.
>> ** it could be good that a node that refrains to insert redundant 
>> information may still indicate that already sent information 
>> (sequence) is unchanged. Along that line of thought how do we handle 
>> loss? A same sequence sent several times?
>> * Security section should indicate that the payload IE is modified at 
>> each hop and there is no guarantee that the original information is 
>> preserved
>> 
>> Great work!
>> 
>> Pascal
>> 
>> 
>> 
>> 
>> 
>>> mechanism can become a very powerful tool for 6TiSCH Networks with 
>>> the contribution of the 6TiSCH community.
>>>

---

From: <[email-masked]>
Date: Wed, 15 Jan 2020 01:08:33 +0000

Hello Pascal, Abdulkadir,

pascal> On the other hand it is hard to inject headers on the fly at L3 l, you have to do IP
pascal> in IP. The IE technique avoids that problem.

Or, we may be able to use the hop-by-hop option?

abdulkadir> Indeed, we have already implemented and validated the described mechanism in a real network.

How did you determine if an outgoing packet is an upward one at L2 in your implementation?

It was just a comment, anyway; I think this kind of discussion is too detailed at this stage. I will read the paper later.

Thanks!
Yatch