Subject: [6lo-fragmentation-dt] Performance report for fragment forwarding

From: Rahul Jadhav <[email-masked]>
Date: Tue, 18 Sep 2018 21:35:13 +0530

<< turns out my earlier mail didn't reach the ML; trying again >>

Hello all,

We experimented with Fragment Forwarding and tried to understand the
performance implications vis-a-vis per-hop reassembly.

Following is the detailed report:
https://github.com/nyrahul/ietf-data/blob/master/6lo-fragfwd-perf-report.md

To summarize, we found that fragment forwarding has some practical issues
when it comes to forwarding efficiency or PDR "on single channel 802.15.4".
While similar concerns were been raised previously during IETF meetings, we
tried to validate it with data.

Please let us know if any comments.

Regards,
Rahul

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 18 Sep 2018 17:13:34 +0000

Yes, I documented that issue when I presented to the 6lo WG. The only protection is to introduce delay between fragments so they get out of interference domain before you send the next one. The next problem is incast in the network, when you start multiple flows that cross/converge at a same intermediate node. My draft allows for flow control by asking an ack every so many fragment…
Take care,
Pascal

From: 6lo-fragmentation-dt <[email-masked]> On Behalf Of Rahul Jadhav
Sent: mardi 18 septembre 2018 18:05
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: [6lo-fragmentation-dt] Performance report for fragment forwarding

Hello all,

We experimented with Fragment Forwarding and tried to understand the performance implications vis-a-vis per-hop reassembly.

To summarize, we found that fragment forwarding has some practical issues when it comes to forwarding efficiency or PDR "on single channel 802.15.4". While similar concerns were been raised previously during IETF meetings, we tried to validate it with data.

Regards,
Rahul

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Sep 2018 21:59:05 +0200

Hi Rahul,

the memory issues in forwarders discussed during IETF101 are real, and they would need to enter the calculations about PDR differences.

Is a max-retry of 3 something that people are actually choosing? Sounds low to me.

More importantly, it seems we neglected to discuss pacing in both draft-watteyne-6lo-minimal-fragment-02.txt and draft-ietf-lwig-6lowpan-virtual-reassembly-00.txt.  Pacing is essential for fragment forwarding (as it is for any application that sends more than one datagram in a row).  If the original sender does not pace, there will always be a collision between the forwarding of fragment N and the origination of fragment N+1.

Unfortunately, pacing is another knob that needs to be tuned, and I’m not aware of good research that tells us what  the right setting for that knob is.  Ideally, we’d want it to be self-tuning.

Grüße, Carsten

> On 18. Sep 2018, at 18:05, Rahul Jadhav <[email-masked]> wrote:
> 
> _______________________________________________
> 6lo-fragmentation-dt mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6lo-fragmentation-dt

---

From: Rahul Jadhav <[email-masked]>
Date: Wed, 19 Sep 2018 09:55:16 +0530

Thanks Pascal, Carsten for the comments.

Pascal, Introducing delay is easy but it has further complications with
regards to buffering requirement. Holding the fragment with a delay while
receiving more fragments from the downstream would mean keeping additional
buffers.
Carsten also mentioned a pacing mechanism .. while it might improve fwding
efficiency it will add to buffer requirement. Also such a scheme might be
non-trivial to be implemented.

Also Carsten, the PDR we reported takes into consideration all factors,,
buffering as well (we use contiki and have single 1280B buffer on each
node). While the transmission scheme we chose results in much less impact
on buffering, it is much closer to our traffic pattern expectation.

Regarding keeping higher mac-retry, ... We have chosen mac-retry of 3 after
some experimentation (considering expected node densities and tx
frequency). increasing mac-retry might not necessarily help, in fact it may
backfire in terms of both PDR as well as mean latency. Would you still
suggest to give it a try and what mac-retry do you think makes sense ?

Based on your comment, we will document the fragments dropped on nodes in
both scenarios, just to be clear.

Thanks,
Rahul

On Wed, 19 Sep 2018 at 01:29, Carsten Bormann <[email-masked]> wrote:

> Hi Rahul,
>
> the memory issues in forwarders discussed during IETF101 are real, and
> they would need to enter the calculations about PDR differences.
>
> More importantly, it seems we neglected to discuss pacing in both
> draft-watteyne-6lo-minimal-fragment-02.txt and
> draft-ietf-lwig-6lowpan-virtual-reassembly-00.txt.  Pacing is essential for
> fragment forwarding (as it is for any application that sends more than one
> datagram in a row).  If the original sender does not pace, there will
> always be a collision between the forwarding of fragment N and the
> origination of fragment N+1.
>
> Unfortunately, pacing is another knob that needs to be tuned, and I’m not
> aware of good research that tells us what  the right setting for that knob
> is.  Ideally, we’d want it to be self-tuning.
>
> Grüße, Carsten
>
>
> > To summarize, we found that fragment forwarding has some practical
> issues when it comes to forwarding efficiency or PDR "on single channel
> 802.15.4". While similar concerns were been raised previously during IETF
> meetings, we tried to validate it with data.
> >

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Sep 2018 06:55:28 +0200

Hi Rahul,

> Carsten also mentioned a pacing mechanism .. while it might improve fwding efficiency it will add to buffer requirement. Also such a scheme might be non-trivial to be implemented.

Pacing is something the origin of the datagram does; I wasn’t suggesting the forwarders to add that (which indeed essentially puts the datagram into buffers there).
Clearly, pacing is counterproductive when forwarders do full reassembly, as it increases the time the partially assembled datagram is vulnerable to fragments of other datagrams hijacking the buffer — you want to capture the channel as much as possible then (to the level that you’d want to send following fragments with a higher MAC priority).  Of course, as the datagram travels through the network, it will become more and more smeared over time.

It would be interesting to see how your numbers change with some pacing at the origin; it should be easy to hack that in if you are not trying to do this in a general way.

> Regarding keeping higher mac-retry, ... We have chosen mac-retry of 3 after some experimentation (considering expected node densities and tx frequency). increasing mac-retry might not necessarily help, in fact it may backfire in terms of both PDR as well as mean latency. Would you still suggest to give it a try and what mac-retry do you think makes sense ?

Yeah, yet another tunable that in a robust protocol really should be self-tuning.
A MAC-layer retransmission is not useful if the layers above have already timed out.
So the latency introduced by the backoff mechanism is really the determining factor here.

Grüße, Carsten

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 19 Sep 2018 05:38:19 +0000

Hello Rahul

Pacing is another word for introducing delay between fragments and yes thus locks the buffers for a longer time. Having to do retries because a fragment interferes with the next also introduces delay, and possibly longer if you keep at it.

The key message in my talk was that this problem simply goes away with 6TiSCH. I think that should be the key message of your writing...

Take care,

Pascal

Le 19 sept. 2018 à 06:25, Rahul Jadhav <[email-masked]<mailto:[email-masked]>> a écrit :

Pascal, Introducing delay is easy but it has further complications with regards to buffering requirement. Holding the fragment with a delay while receiving more fragments from the downstream would mean keeping additional buffers.
Carsten also mentioned a pacing mechanism .. while it might improve fwding efficiency it will add to buffer requirement. Also such a scheme might be non-trivial to be implemented.

Also Carsten, the PDR we reported takes into consideration all factors,, buffering as well (we use contiki and have single 1280B buffer on each node). While the transmission scheme we chose results in much less impact on buffering, it is much closer to our traffic pattern expectation.

Based on your comment, we will document the fragments dropped on nodes in both scenarios, just to be clear.

Thanks,
Rahul

On Wed, 19 Sep 2018 at 01:29, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
Hi Rahul,

Grüße, Carsten

> On 18. Sep 2018, at 18:05, Rahul Jadhav <[email-masked]<mailto:[email-masked]>> wrote:
>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
6lo-fragmentation-dt mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/6lo-fragmentation-dt

---

From: Rahul Jadhav <[email-masked]>
Date: Wed, 19 Sep 2018 11:28:52 +0530

Doing pacing on origin-node only should be easy to change/hack and might
improve the forwarding efficiency. But this may also shoot up the mean
latency.
We ll try this anways (add randomized delay between 10ms to 500ms) to check
the impact on PDR. Thanks for this suggestion.

On Wed, 19 Sep 2018 at 10:25, Carsten Bormann <[email-masked]> wrote:

> Hi Rahul,
>
> > Carsten also mentioned a pacing mechanism .. while it might improve
> fwding efficiency it will add to buffer requirement. Also such a scheme
> might be non-trivial to be implemented.
>
> Pacing is something the origin of the datagram does; I wasn’t suggesting
> the forwarders to add that (which indeed essentially puts the datagram into
> buffers there).
> Clearly, pacing is counterproductive when forwarders do full reassembly,
> as it increases the time the partially assembled datagram is vulnerable to
> fragments of other datagrams hijacking the buffer — you want to capture the
> channel as much as possible then (to the level that you’d want to send
> following fragments with a higher MAC priority).  Of course, as the
> datagram travels through the network, it will become more and more smeared
> over time.
>
> It would be interesting to see how your numbers change with some pacing at
> the origin; it should be easy to hack that in if you are not trying to do
> this in a general way.
>
> > Regarding keeping higher mac-retry, ... We have chosen mac-retry of 3
> after some experimentation (considering expected node densities and tx

---

From: Rahul Jadhav <[email-masked]>
Date: Wed, 19 Sep 2018 11:31:09 +0530

Agreed. 6TiSCH with nodes using non-shared cells for transmission will be
the best deployment scenario for fragment-forwarding.
Unfortunately our testbest currently does not support TSCH and thus we wont
be able to report the numbers.

On Wed, 19 Sep 2018 at 11:08, Pascal Thubert (pthubert) <[email-masked]>
wrote:

> Hello Rahul
>
> Pacing is another word for introducing delay between fragments and yes
> thus locks the buffers for a longer time. Having to do retries because a
> fragment interferes with the next also introduces delay, and possibly
> longer if you keep at it.
>
> The key message in my talk was that this problem simply goes away with
> 6TiSCH. I think that should be the key message of your writing...
>
> Take care,
>
> Pascal
>
> Le 19 sept. 2018 à 06:25, Rahul Jadhav <[email-masked]> a écrit :
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 19 Sep 2018 14:56:30 +0000

Hello Rahul :

If you could instrument the intermediate nodes, I’d like to see if for reason of retries a fragment may catch up with another one. This can be tested by making a hop in the middle very far and unreliable.

Also, even if you pace at the source by inserting gaps, fragments may still pile up at the ingress of the slowest link on the way and your initial problem will be recreated there. This is only visible if you effectively have a slower hop, but is a very classical effect in window based networks, the full window ends up piling at ingress of the slowest link. You may recreate that by injecting additional traffic via a node on your path.

To avoid that you would need to pace at the source to a rate that is slower what the slowest hop would do if it was the source. Really hard to do. So ideally, each node would ensure that there is a minimum gap between fragments, which should be no-op everywhere along a path but at the slowest hop where fragments will pile up.

Finally: you can limit the amount of queueing at the slowest hop if you use the recoverable frags draft, using windows of a few fragments for flow control.

Bottom line: the more you test with one frequency only the more you’ll find it’s a bad idea : )

All the best;

Pascal

From: 6lo-fragmentation-dt <[email-masked]> On Behalf Of Rahul Jadhav
Sent: mercredi 19 septembre 2018 07:59
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

Doing pacing on origin-node only should be easy to change/hack and might improve the forwarding efficiency. But this may also shoot up the mean latency.
We ll try this anways (add randomized delay between 10ms to 500ms) to check the impact on PDR. Thanks for this suggestion.

On Wed, 19 Sep 2018 at 10:25, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
Hi Rahul,

Grüße, Carsten

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 20 Sep 2018 17:02:18 +0530

Hello Pascal,

Please find my comments inline..

Regards,
Rahul

On Wed, 19 Sep 2018 at 20:26, Pascal Thubert (pthubert) <[email-masked]>
wrote:

> Hello Rahul :
>
>
>
> If you could instrument the intermediate nodes, I’d like to see if for
> reason of retries a fragment may catch up with another one. This can be
> tested by making a hop in the middle very far and unreliable.
>

[RJ] Sorry but I m not sure what you mean here.

>
>
> Also, even if you pace at the source by inserting gaps, fragments may
> still pile up at the ingress of the slowest link on the way and your
> initial problem will be recreated there. This is only visible if you
> effectively have a slower hop, but is a very classical effect in window
> based networks, the full window ends up piling at ingress of the slowest
> link. You may recreate that by injecting additional traffic via a node on
> your path.
>
>
>
[RJ] Agreed that a bottleneck link may still create a problem. But if you
see the network in our report, there is no such bottleneck node/link ...
the paths are sufficiently balanced.

> To avoid that you would need to pace at the source to a rate that is
> slower what the slowest hop would do if it was the source. Really hard to
> do. So ideally, each node would ensure that there is a minimum gap between
> fragments, which should be no-op everywhere along a path but at the slowest
> hop where fragments will pile up.
>
>
>
> Finally: you can limit the amount of queueing at the slowest hop if you
> use the recoverable frags draft, using windows of a few fragments for flow
> control.
>
[RJ] Amount of queueing can be limited in our case by using L2 feedback as
well i.e. if the fragment is lost (and the sender knows this based on L2
feedback) then we can as well stop transmitting all subsequent fragments.
For 802.15.4 the recovery procedure can be applied based on L2 feedback
without any explicit ACK.
But right now we arent using any of these advanced techniques.

Our aim for this experiment was to check how much of an impact was it when
fragment forwarding was used as mentioned in the VRB draft in single
channel mode of operation.

> Bottom line: the more you test with one frequency only the more you’ll
[RJ] Yes we have realized (it the hard way) that using fragment forwarding
in single channel mode is a very bad idea.

>
>
> All the best;
>
>
>
> Pascal
>
>
>
> *From:* 6lo-fragmentation-dt <[email-masked]> *On
> Behalf Of *Rahul Jadhav
> *Sent:* mercredi 19 septembre 2018 07:59
> *To:* Carsten Bormann <[email-masked]>
> *Cc:* [email-masked]; [email-masked];
> [email-masked]
> *Subject:* Re: [6lo-fragmentation-dt] Performance report for fragment
> forwarding
>
>
>
> We ll try this anways (add randomized delay between 10ms to 500ms) to
> check the impact on PDR. Thanks for this suggestion.
>
>
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 20 Sep 2018 12:16:58 +0000

Hello Rahul :

6TiSCH protects against fragments interfering with one another, but it does not protect against a Hop1 sending frag N+1 to Hop2 while Hop2 is sending frag N to Hop3. Even 6TiSCH needs to pace its traffic to be forwarded by a same next hop, and that’s true for all packets not just fragments. Do you have any form of listen before talk in your simulation?

If you do not delay the transmissions of intermediate fragments, your test certainly proves that the system cannot work satisfactorily. This much was predictable on paper. Say it take 10ms for a transmit + ack, and most transmissions succeed within 3-4 tries, it makes sense to do your testing injecting 50 to 100ms between fragment transmissions. Every node along the path must validate that fragments are thus spaced. Could you please relaunch your test with a rule like that?

For the rest please see below:

[PT>] The idea is to make a link less reliable then others in order to cause retries so a virtually slower link. You’ll see that packets pile up there even if paced at the source. That’s what I mean by other packets catching up.

[RJ] Agreed that a bottleneck link may still create a problem. But if you see the network in our report, there is no such bottleneck node/link ... the paths are sufficiently balanced.

[PT>] So the space must be inserted at the bottleneck too. In theory, a transmitter needs to ensure that every packet to a same next hop is sufficiently spaced with the next packet, unless the next hop is the destination, so the next hop can forward.
To avoid that you would need to pace at the source to a rate that is slower what the slowest hop would do if it was the source. Really hard to do. So ideally, each node would ensure that there is a minimum gap between fragments, which should be no-op everywhere along a path but at the slowest hop where fragments will pile up.

Finally: you can limit the amount of queueing at the slowest hop if you use the recoverable frags draft, using windows of a few fragments for flow control.
[RJ] Amount of queueing can be limited in our case by using L2 feedback as well i.e. if the fragment is lost (and the sender knows this based on L2 feedback) then we can as well stop transmitting all subsequent fragments. For 802.15.4 the recovery procedure can be applied based on L2 feedback without any explicit ACK.
But right now we arent using any of these advanced techniques.

Our aim for this experiment was to check how much of an impact was it when fragment forwarding was used as mentioned in the VRB draft in single channel mode of operation.

[PT>] Without any pacing, the simulation is not so realistic and though interesting, conclusions there would be premature. I’d be very interested in seeing more tests varying the inter frame space and see how that impacts the delivery.

Bottom line: the more you test with one frequency only the more you’ll find it’s a bad idea : )
[RJ] Yes we have realized (it the hard way) that using fragment forwarding in single channel mode is a very bad idea.
[PT>] Note that the same would happen with a series of packets in a row. I do not think that the fact these are fragments have much to do with it.

Take care,

Pascal

All the best;

Pascal

From: 6lo-fragmentation-dt <[email-masked]<mailto:[email-masked]>> On Behalf Of Rahul Jadhav
Sent: mercredi 19 septembre 2018 07:59
To: Carsten Bormann <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

Grüße, Carsten

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 20 Sep 2018 18:17:46 +0530

All in all, some of the concerns mentioned in this mail thread should be
reflected in the draft as well. A naive implementation or implementers
without any background on mac layer may seriously goof up the performance
if they simply go by the draft currently.

Regards,
Rahul

>
> 6TiSCH protects against fragments interfering with one another, but it
> does not protect against a Hop1 sending frag N+1 to Hop2 while Hop2 is
> sending frag N to Hop3. Even 6TiSCH needs to pace its traffic to be
> forwarded by a same next hop, and that’s true for all packets not just
> fragments. Do you have any form of listen before talk in your simulation?
>
>
[RJ] Yes, there is carrier sensing before transmission in my simulation.

>
>
> If you do not delay the transmissions of intermediate fragments, your test
> certainly proves that the system cannot work satisfactorily. This much was
> predictable on paper. Say it take 10ms for a transmit + ack, and most
> transmissions succeed within 3-4 tries, it makes sense to do your testing
> injecting 50 to 100ms between fragment transmissions. Every node along the
> path must validate that fragments are thus spaced. Could you please
> relaunch your test with a rule like that?
>
>
>
[RJ] Yes we ll try this and get the data again. Carsten also suggested the
same before in this mail chain.

> *[PT>] The idea is to make a link less reliable then others in order to
> cause retries so a virtually slower link. You’ll see that packets pile up
> there even if paced at the source. That’s what I mean by other packets
> catching up.*
>

[RJ] Ok got it. Thanks

> *[PT>] So the space must be inserted at the bottleneck too. In theory, a
> transmitter needs to ensure that every packet to a same next hop is
> sufficiently spaced with the next packet, unless the next hop is the
> destination, so the next hop can forward.*
>
> *[PT>] Without any pacing, the simulation is not so realistic and though
> interesting, conclusions there would be premature. I’d be very interested
> in seeing more tests varying the inter frame space and see how that impacts
> the delivery.*
>

[RJ] Yes we ll try this.

> *[PT>] Note that the same would happen with a series of packets in a row.
> I do not think that the fact these are fragments have much to do with it.*
>

[RJ] With series of packets, do you mean per-hop reassembly? Well, per-hop
reassembly does not suffer from this issue. But it suffers from buffering
issue. So if the traffic is sufficiently sparse then using per-hop
reassembly in this scenario makes more sense.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 20 Sep 2018 12:59:59 +0000

Hello Rahul

From: Rahul Jadhav <[email-masked]>
Sent: jeudi 20 septembre 2018 14:48
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

All in all, some of the concerns mentioned in this mail thread should be reflected in the draft as well. A naive implementation or implementers without any background on mac layer may seriously goof up the performance if they simply go by the draft currently.
[PT>] makes sense. Note that a burst of IP packets for a same destination have the same issue. The problem has to do with transmitting frames at L2 not that they are individual packets or multiple fragments at upper layer. If the upper layer makes 5 IP packets to avoid fragmentation of a larger packet in 5 fragments, and sends the 5 IP packets together, you’ll get the exact same result as if using fragments – unless I miss something always interesting to dig deeper..

[RJ] Yes, there is carrier sensing before transmission in my simulation.
[PT>] Then other strategies are possible, like a longer CCA for the first sender.

[RJ] Yes we ll try this and get the data again. Carsten also suggested the same before in this mail chain.
[PT>] very cool, looking forward to seeing the change. Great research!

[RJ] Ok got it. Thanks

[RJ] Yes we ll try this.

[RJ] With series of packets, do you mean per-hop reassembly? Well, per-hop reassembly does not suffer from this issue. But it suffers from buffering issue. So if the traffic is sufficiently sparse then using per-hop reassembly in this scenario makes more sense.
[PT>] No just different IP packet same destinations to be sent in a row, no fragmenting. Unless you do ECMP, they will be routed along a same path and have the same fate as fragments. This is why my expectation is that the the problem you point out is not limited to fragments.

 [PT>] : )
[PT>]
Pascal

Take care,

Pascal

All the best;

Pascal

Grüße, Carsten