Subject: Re: [6lo] Mirja Kühlewind's Discuss on draft-ietf-6lo-fragment-recovery-13: (with DISCUSS and COMMENT)

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 6 Mar 2020 18:57:48 +0000

Hello Mirja

A great many thanks for your really deep review, this is both really appreciated and  incredibly useful.

If that's OK with you let's make a round to clear the DISCUSSes separately like I did for Benjamin's review.

Also considering the breadth of the discuss alone, I'd rather publish the proposed changes so you and Benjamin can review the changes I made for your DISCUSSes.

Please find the proposed changes discussed below in https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-14 

> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Thanks for this well written document, however, I have a couple points
> below that need further clarification, all mostly related to congestion
> control. From an editorial point of view most of this is discussed either in the
> intro text of section 6, then some part in 7.1, and some in the appendix C. I
> would really recommend you to instead have a separate section that much
> clearer states what should be done by default (probably no dynamically
> window but a small fixed window with maybe size of 1) and what could be
> don as further optimisation, and also to discuss the parameter/variables
> there before the algorithms are discussed.
> 

A size of 1 is probably not acceptable for the general LLN case, considering the cost of an ack. I'd rather leave that to config.
Otherwise, I agree. What about adding a subsection in section 4 as follows (including changes that cover your comments below):

"
4.3.  Flow Control

   The inter-frame gap is the only protection that [FRAG-FWD] imposes by
   default.  This document enables to group fragments in windows and
   request intermediate acknowledgements so the number of in-flight
   fragments can be bounded.  This document also adds an ECN mechanism
   that can be used to adapt the size of the window, the size of the
   fragments, and/or the inter-frame gap to protect the network.

   This specification enables the source endpoint to apply a flow
   control mechanism to tune those parameters, but the mechanism itself
   is out of scope.  In most cases, the expectation is that most
   datagrams will represent only a few fragments, and that only the last
   fragment will be acknowledged.  A basic implementation of the source
   endpoint is NOT REQUIRED to variate the size of the window, the
   duration of the inter-frame gap or the size of a fragment in the
   middle of the transmission of a datagram, and it MAY ignore the ECN
   signal or simply reset the window to 1 (see Appendix C for more) till
   the end of this datagram upon detecting a congestion.

   The size of the fragments is typically computed from the Link MTU to
   maximize the size of the resulting frames.  The size of the window
   and the duration of the inter-frame gap SHOULD be configurable, to
   roughly adapt the size of the window to the number of hops in an
   average path, and to follow the general recommendations in
   [FRAG-FWD], respectively.
"

> And a bit of a provoking question: wouldn't it be easier to just use a reliable
> transport protocol on top?

Just that the classical transports I'm aware of will :
- not support the interframe gap and that's the basic requirement in 6lo 
- not be capable to variate the interframe gap nor the fragment size
- use the return path excessively for acks. 6lo is very much about saving energy and bandwidth
- be generally Overkill/too complex for a LLN node, see the text we just added

Also this spec enables a flow control mechanism but that mechanism is out of scope. 
It is internal to the sending endpoint and does not affect the interoperability that this specification enables.
A bit like there's an ECN in IP but the behavior belongs to the various transport protocols. 
Just that on top of signaling ECN we provide tools that the flow control mechanism may play with, e.g., window size

> If this mechanism is intended to be used over a
> short path with a few hops only (in a local network), I think this should be
> stated more clearly at the beginning of the document. 

This is very true and implicit since we are talking about a contiguous 6LoWPAN route-over mesh.
Propose to tweak the last sentence in the introduction to
"
   This specification provides a method to forward fragments over
   typically a few hops in a route-over 6LoWPAN mesh, and a selective
   acknowledgment to recover individual fragments between 6LoWPAN
   endpoints.  The method is designed to limit congestion loss in the
   network and addresses the requirements that are detailed in
   Appendix B.

"

> In the appendix you state
> this: " In addition, deploying such a mechanism requires
>    that the end-to-end transport is aware of the delivery properties of
>    the underlying LLN,..."
> But I'm not sure what you mean...? Can you further explain?

"Requires" might be exaggerated since TCP was shown to work fine in LLNs. 
But things like the default RTO of 1s is really unsafe, for an endpoint in the internet that communicates to the LLN device (e.g., using HTTP instead of COAP).
But it's  probably better to just remove that text. Instead it is probably good to mention the extra acks on the return path.
For one thing, though, we do not want to discard packets that traversed the LLN to indicate congestion to the source. Ideally things like slow start should be really smooth, and the window size should remain very small to cope with the memory available in the LN node without the need to drop a packet in the LLN.

The end of appendix A becomes:

"

   Mechanisms such as TCP or application-layer segmentation could be
   used to support end-to-end reliable transport.  One option to support
   bulk data transfer over a frame-size-constrained LLN is to set the
   Maximum Segment Size to fit within the link maximum frame size.
   Doing so, however, can add significant header overhead to each
   802.15.4 frame and cause extraneous acknowledgements across the LLN
   compared to the method in this specification.

"

> 
> 1) Sec 6:
> "Upon exhaustion of the retries the
>    sender may either abort the transmission of the datagram or retry the
>    datagram from the first fragment with an 'X' flag set in order to
>    reestablish a path and discover which fragments were received over
>    the old path in the acknowledgment bitmap. "
> I'm not sure about this "or". Why should the first fragment be more
> successful than any other which requests an ACK? Also if you really want to
> keep this condition, you need to specify it better. How often do you retry? I
> guess you need to set the PTO again...? Further the RTO should also
> implement an exponential back-off.

The first fragment draws a new path so it may avoid the problem though there is no guarantee. 
Once the new path is established, the next fragments will follow it and the segments of the old path that are no more used time out.

Proposed updated text:

"
   This automatic repeat request (ARQ) process MUST be protected by a
   Retransmission Time Out (RTO) timer, and the fragment that carries
   the 'X' flag MAY be retried upon a time out for a configurable number
   of times (see Section 7.1) with an exponential backoff.  Upon
   exhaustion of the retries the sender may either abort the
   transmission of the datagram or resend the first fragment with an 'X'
   flag set in order to establish a new path for the datagram and obtain
   the list of fragments that were received over the old path in the
   acknowledgment bitmap.

"

> 
> 2) sec 6.3:
> "Upon an acknowledgment with a NULL bitmap, the sender endpoint
>    MUST abort the transmission of the fragmented datagram with one
>    exception: In the particular case of the first fragment, it MAY
>    decide to retry via an alternate next hop instead."
> What's mean with "In the particular case of the first fragment"? And does
> this mean it should retry only with the first fragment or the whole
> transmission. 
> However, if this signal is from the receiving endpoint why should that
> endpoint change it mind only if a different path is used? If the assumption is
> that this NULL bitmap is sent by an intermediate node? However, then it
> would make sense to  rather signal this information explicitly (e.g. using a
> flag).

This is also linked to the fact that the first fragment draws the path as in the case above. As you figured, the expectation is that a node in the middle experiences an issue and cannot do the FF operation for that datagram, so it aborts with a NULL bitmap.

Yes, the problem could be in the receiving endpoint in which case rerouting does not help. But then, it is probably temporary, e.g., if the receiving endpoint has a single reassembly buffer, which is quite common, and is already receiving a datagram from another source. There is a variety of use cases and which is most probable depends on the use case. 

So let the source endpoint decide. 

> 3) Sec 7.1 (and to some extend sec 6)
> "   OptWindowSize:  The OptWindowSize is the value for the Window_Size
>       that the sender should use to start with.  It is greater than or
>       equal to MinWindowSize.  It is less than or equal to
>       MaxWindowSize.  The Window_Size should be maintained below the
>       number of hops in the path of the fragment to avoid stacking
>       fragments at the bottleneck on the path.  If an inter-frame gap is
>       used to avoid interference between fragments then the Window_Size
>       should be at most on the order of the estimation of the trip time
>       divided by the inter-frame gap."
> This needs normative language and more explanation. 

Well, this was not intended to be normative but just a rule of a thumb. Some (many I expect) people will want to ack only the last fragment and that's a tradeoff between the cost of the ack back and the chances of congestion loss. 

> I recommend to even
> say that if no congestion control (as discussed in the appendix) is applied, the
> Window MUST be set to 1. 

This makes full sense in other cases but is too expensive here. The default that people go for is a single ack in the end. Note that this spec compares to the art of RFC 4944 where all the fragments are pushed to the network without any feedback as hot potatoes. Apparently that did noes work too well in cases, thus this work. But people still love the fact that there's no traffic back and that is why this original work (https://tools.ietf.org/html/draft-thubert-6lowpan-simple-fragment-recovery-01) was split 3 docs, this,  the overarching minimal-fragments and the LWIG draft that forwards fragments in the RFC 4944 and no ack.

> Further, the assumption that the window can or
> should be set to (at maximum) the number of hop does seem correctly to
> me. No matter how many hops there are packets are only queued at the
> bottleneck (the link where the current rate is smaller than the sending rate)
> and it depends on the sending rate of the bottleneck link how many packets
> need to be queued. This is completely independent of the number of hops.

The rationale here is due to the inter-frame gap. In normal conditions, it ensures that a fragment progresses before the next comes in. So there's at most one fragment per hop and there's no point having a window bigger that. There's less than that actually, because frag 0 reaches node 2 before node 0 can send frag 1 to node 1, so we could divide the recommendation by 2. But then we need to keep the network busy while the ack comes back. 

I agree that this is not providing the optimal window but more like a reasonable upper bound. If there was no interframe gap I'd say (please correct me) that the lower bound to keep the bottleneck busy (in fragments not bytes) is (Bottleneck Speed / Bottleneck MTU) * RTT.  The average low power network is mostly sleeping, and does not  experience congestion in normal operation. So there is usually no bottleneck. If  the LLN is homogeneous we get a lower bound of  (PHY Speed / MTU) * RTT. But then, some people could be conservative and use a window of 1 as you recommend.

All in all it appears that the text creates more confusion then help, and dives into the sender flow control which is out of scope.
I'd rather remove recommendation on the runtime window size at all.

So we'd get:
"
   OptWindowSize:  The OptWindowSize is the value for the Window_Size
      that the sender should use to start with.  It is greater than or
      equal to MinWindowSize.  It is less than or equal to
      MaxWindowSize.  A rule of a thumb for OptWindowSize could be an
      estimation of the trip time divided by the inter-frame gap to keep
      the network busy.
"

> Further, even if that would be true, as long as this document does not discuss
> also away to estimate or know the number of hops, this advise would
> unfortunately be useless... 

Yes, better remove it

> Further I don't think pointing to rfc6298 for RTT
> calculation is sufficient (as done in the appendix). rfc6298 assume frequent
> ACKs and a reasonably large window, which is both not the case here. All in

TCP has been successfully used on LLNs, though, so it cannot be that bad a recommendation.
Note that there's probably less fragments than hops, so there's probably not a chance to even measure RTT before all fragments are out, and if tehre is, not many chances to update the initial reading till the datagram is fully sent. I agree there may be better ways so we need to remove the RECOMMEND. What about:

"
                                                        For the lack of a more adapted
   technique, the method detailed in "Computing TCP's Retransmission
   Timer" [RFC6298] may be used for that computation. 

"

Earlier in 6.0 I also suggest to change:

"
                                                                                                      The sender
   protects the transmission over the LLN mesh with a retry timer that
   is configured for a use case and may be adapted dynamically, e.g.,
   according to the method detailed in [RFC6298].  

"

> all, any window adjustments itself are not described at all. What should be
> done when a congestion marking is received? How does the window need to
> be adjusted based on an RTO? When should the window be increased again?
> And how much?

This is out of scope.

The goal of the draft is to specify what goes over the air to recover fragments. The flow control operation is an internal decision to the sender endpoint, and can be adapted to the use case an interoperation issue with the other endpoints. We do not have enough experience to enforce something, and there can be very different use cases and variations, so we only provide non-normative hints.

We want to allow implementations to try their own stuff, including slow start and fast recovery for a  device that can afford it in a use case that justifies it. I hope we'll see future spec(s) that specify flow control mechanisms, but as I said earlier, this is out of scope here, we just provide the controls.

Following your earlier recommendation, we could suggest in case of a ECN we set  W=1 and stay there as a rule of a thumb for that datagram in the absence of a more intelligent / adapted flow control operation

To clarify let me change
"
4.  Extending draft-ietf-6lo-minimal-fragment

   This specification implements the generic FF technique defined in
   "LLN Minimal Fragment Forwarding" [FRAG-FWD], provides end-to-end
   fragment recovery and mechanisms that can be used for flow control.

"

Also see the new section 4.3

> 
> 4) Sec 7.1.: Inline with the TSV-ART review (Thanks Collin!), the parameters
> need more guidance. Especially for he number of retries it should be possible
> to recommend a default value (e.g. 3) and it would be good to also give an
> upper limits (MUST NOT be larger than X). Similar for the window size: there
> should be also at least a default value (see comment above). And further the
> RTO needs further explanation about how to find a reasonable value. If the
> RTO is configured (and not estimated dynamically) e.g. it could be set to 3x
> the maximum expected RTT in the respective network. And it would be even
> better to provide a minimum default (initial) value. Not that TCP is also
> designed to work on a large variety of timescales and a minimum initial value
> of 1s is seen as safe for all Internet scenarios. It's really important to also
> provide some recommendations like this here.

Makes sense. 

The number of retries is really bounded by the upper layer protocol. 

It is actually the time allowed to transfer the datagram that must remain below whatever the upper layer protocol expects.
We could give a rule of a thumb and yes your 3 looks good. 

The window size can be anything, I expect many will only ask for an ack the last fragment. 
But by construction that is bounded by the bitmap to 32. 

All in all we get:

"
   MinWindowSize:  The minimum value of Window_Size that the sender can
      use.  A value of 1 is RECOMMENDED.

   OptWindowSize:  The OptWindowSize is the value for the Window_Size
      that the sender should use to start with.  It is greater than or
      equal to MinWindowSize.  It is less than or equal to
      MaxWindowSize.  A rule of a thumb for OptWindowSize could be an
      estimation of the one-way trip time divided by the inter-frame
      gap.  If the acknowledgement back is too costly, it is possible to
      set this to 32, meaning that only the last Fragment is
      acknowledged in the first round.

   MaxWindowSize:  The maximum value of Window_Size that the sender can
      use.  The value MUST be strictly less than 33.

   An implementation may perform its estimate of the RTO or use a
   configured one.  The ARQ process is controlled by the following
   parameters:

   MinARQTimeOut:  The minimum amount of time a node should wait for an
      RFRAG Acknowledgment before it takes the next action.  It MUST be
      more than the maximum expected round-trip time in the respective
      network.

   OptARQTimeOut:  The initial value of the RTO, which is the amount of
      time that a sender should wait for an RFRAG Acknowledgment before
      it takes the next action.  It is greater than or equal to
      MinARQTimeOut.  It is less than or equal to MaxARQTimeOut.  See
      Appendix C for recommendations on computing the round-trip time.
      By default a value of 3 times the maximum expected round-trip
      time in the respective network is RECOMMENDED.

   MaxARQTimeOut:  The maximum amount of time a node should wait for the
      RFRAG Acknowledgment before it takes the next action.  It must
      cover the longest expected round-trip time, and be several times
      less than the time-out that covers the recomposition buffer at the
      receiver, which is typically on the order of the minute.  An upper
      bound can be estimated to ensure that the datagram is either fully
      transmitted or dropped before an upper layer decides to retry it.

   MaxFragRetries:  The maximum number of retries for a particular
      fragment.  A default value of 3 is RECOMMENDED.  An upper bound
      can be estimated to ensure that the datagram is either fully
      transmitted or dropped before an upper layer decides to retry it.

   MaxDatagramRetries:  The maximum number of retries from scratch for a
      particular datagram.  A default value of 1 is RECOMMENDED.  An
      upper bound can be estimated to ensure that the datagram is either
      fully transmitted or dropped before an upper layer decides to
      retry it.

"

> 
> 5) Sec 7.2:
> "The management system should monitor the number of retries and of ECN
>    settings that can be observed from the perspective of both the sender
>    and the receiver, and may tune the optimum size of Fragment_Size and
>    of Window_Size, OptFragmentSize, and OptWindowSize, respectively, at
>    the sender."
> This does not see seem correct, as OptFragmentSize and OptWindowSize are
> the initial values which are configured and therefore should not be changed
> dynamically. Only Fragment_Size and Window_Size are changes. Further the
> network should also normatively state somewhere that Fragment_Size and
> Window_Size MUST not grow above the configured max value. That seems
> obvious but it's better to be explicit and use normative language respectively.

This is meant to change the starting values to be applied to the next datagrams.
Note that talking to the management system can take a very long time. 
We do not expect the kind of reactivity that would affect the current datagram.

Proposed changes:
"
7.1.  Protocol Parameters

   The management system SHOULD be capable of providing the parameters
   listed in this section and an implementation MUST abide by those
   parameters and in particular never exceed the minimum and maximum
   configured boundaries.

"

And

"
7.2.  Observing the network

   The management system should monitor the number of retries and of ECN
   settings that can be observed from the perspective of both the sender
   and the receiver with regards to the other endpoint.  It may then
   tune the optimum size of Fragment_Size and of Window_Size,
   OptFragmentSize, and OptWindowSize, respectively, at the sender
   towards a particular receiver, applicable to the next datagrams.
"

> 6) Further sec 7.2 says:
> "The inter-frame gap is another tool that can be
>    used to increase the spacing between fragments of the same datagram
>    and reduce the ratio of time when a particular intermediate node
>    holds a fragment of that datagram."
> However, inter-frame gap is a configuration parameter and this is the first
> time that adapting it dynamically is mentioned here. If you want to adapt it
> dynamically you need to add more information.

This is now discussed in 4.3 . But not in great details but then the flow control mechanism is out of scope.

Again many thanks!

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 9 Mar 2020 17:56:28 +0000

Hello Mirja:

Again many thanks for your very deep review. 

I published -14 to propose some changes that address your DISCUSS comments as well as Benjamin's
Please see https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-14 . 
Then I published  https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-15 with proposals to fix Benjamin's COMMENTS.
Since it is cut off, I'm publishing https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-16 with proposals to fix your COMMENTS in a row without waiting for the answer.

Please find a first round of discussion on your COMMENTS below:

> COMMENT:
> 1) side comment regarding section 2.2: Note that there is also a working group
> draft defining PMTUD for datagram transports: draft-ietf-tsvwg-datagram-
> plpmtud

Yes, I guess 6LoWPAN could learn and adapt. The problem is below IP for sub 1280 interfaces over a number of mesh hops, but the method could be the same or similar.

> 
> 2) Sec 6:
> "Note that acknowledgments might consume precious resources so the use  of
> unsolicited acknowledgments should be configurable and not enabled
>    by default."
> Maybe SHOULD?

done

> 
> 3) Sec 6: Maybe use normative language here?
> OLD
> "Fragments are sent in a round-robin fashion: the sender sends all the
>    fragments for a first time before it retries any lost fragment; lost
>    fragments are retried in sequence, oldest first.  This mechanism
>    enables the receiver to acknowledge fragments that were delayed in
>    the network before they are retried."
> NEW
> "Fragments MUST be sent in a round-robin fashion: the sender MUST send all
> the
>    fragments for a first time before it retries any lost fragment; lost
>    fragments MUST be retried in sequence, oldest first.  This mechanism
>    enables the receiver to acknowledge fragments that were delayed in
>    the network before they are retried."
> 

Applied

> 4) Sec 6:
> "When a single frequency is used by contiguous hops, the sender should
>    insert a delay between the frames (e.g., carrying fragments) that are
>    sent to the same next hop.
> Maybe SHOULD?
> 

done

> 5) sec 6.3:
> "Until the timer
>    elapses, fragments of that datagram may still be received, e.g. if
>    the RFRAG-ACK was lost on the way back and the source retried the
>    last fragment.  In that case, the router forwards the fragment
>    according to the state in the VRB."
> Why should a router forward the segment, rather than re-sending/re-
> generating the full ACK knowing that all segments have been successfully
> received?

I guess that there was a reason; but I cannot fathom it at this point.
Let me change the text.
"

   Either way, if the RFRAG-ACK indicates that the fragment was entirely
   received (FULL bitmap), it arms a short timer, and upon timeout, the
   VRB and all the associated state are destroyed.  Until the timer
   elapses, fragments of that datagram may still be received, e.g. if
   the RFRAG-ACK was lost on the path back and the source retried the
   last fragment.  In that case, the router generates an RFRAG-ACK with
   a FULL bitmap back to the fragmenting endpoint if an acknowledgement
   was requested, else it silently drops the fragment.

"

> 6) sec 8:
> As currently described an off-path attacker could abort the transmission if the
> datagram_tag is known. I think it should be mention somewhere that a null
> bitmap should only be accepted and forwarded if received on the right
> interface. Further it might make sense to not erase state immediately but wait
> to see if any further fragments are received from the sender. In any case this
> attack should at least be mentioned in section 8.

You are correct, I made a change in 15 that now includes the interface in the lookup for another comment from Benjamin. That caused me to change minimal fragments as well. With that change, the attacker would need to be on the same interface as a hop of the path, which means on-path really, otherwise it cannot cause the MPLS forwarding. 

The text quoted above has the timer that maintains a short-lived state for the retries in flight.

Some new text was added to section 8 in particular:

"

   In addition to the threats detailed therein, an attacker that is on-
   path can prematurely end the transmission of a datagram by sending a
   RFRAG Acknowledgment to the fragmenting endpoint.  It can also cause
   extra transmissions of fragments by resetting bits in the RFRAG
   Acknowledgment bitmap, and of RFRAG Acknowledgments by forcing the
   Ack-Request bit in fragments that it forwards.  As indicated in
   [FRAG-FWD], Secure joining and the Link-Layer security are REQUIRED
   to protect against those attacks.

"

Basically the only protection we have for 6LoWPAN compression and fragmentation as a whole is the lower layer security.

> 
> 7) Appendix B:
> "The recovery mechanism must support highly
>       fragmented packets, with a maximum of 32 fragments per packet."
> Where does the 32 come from and shouldn't this be stated normatively in
> body of the document?

It was computed from LoWPAN (IEEE std 802.15.4) to ensure we fit above 2K bytes in a payload that could be in the order or above 70 bytes, with a margin. We used that requirement to dimension the bitmap. I'm not sure about normative, this is the information that we used to design the protocol but the protocol coud live without that text at all.

> 
> Nit: you use both "time out" and "time-out". I recommend to change the two
> occasions of "time out" to "time-out".
> 

Fixed

Again many thanks!

P

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 19 Mar 2020 10:30:46 +0100

Hi Pascal,

Thanks for you updates. Sorry for my late reply. I unfortunately have some more comments. Please see below.

> On 6. Mar 2020, at 19:57, Pascal Thubert (pthubert) <[email-masked]> wrote:
> 
> Hello Mirja
> 
> “
Thanks for adding this. However, as I said a couple of times in my discuss there must be more guidance. This is not only about flow control but also about congestion control and it is not okay to declare congestion control out of scope. If you only do fragmentation but no retransmission, you don’t need to care about congestion control (but only flow control) as you don’t increase the actual network load by this. However, if you retransmit you are sending more data than the original sender (that hopefully is congestion controlled) and therefore you increase the load on the network and you MUST implement your own congestion control or some fixed rate limiting for that additional load. Saying this is out of scope and you want to do experimentation is not acceptable for a Proposed Standard.

To be clear the request of this discuss is to give a normative recommendation for the default value of the window size and/or inter-frame gap.

Further note, as you allow to adapt both the window and the inter-frame gap dynamically, you actually implement two control mechanisms here. I actually recommend to only use the inter-frame gap and don’t have window here. You say a couple of times in your reply below, that the window determines the ask-rate, however, it is not clear to me because the ack rate should be a parameter at the receiver and not at the sender (maybe I don’t remember things correctly because it’s a while back since I read the draft and I couple find anything about this in the draft quickly). If the window size however does define the ack rate, then maybe you should rename that parameter respectively.

However, if there is really a need for a window, I still recommend to talk less about adapting this value dynamically and make clear that having a fixed value is the recommended default. Therefore I recommend to remove the parameter MinWindowSize and MaxWindowSize because these should actually not be parameters than can be configured but are actual bounds. If someone decides to implement dynamic window adaption, they can decide to re-introduce these parameter again and make them configurable but it doesn’t need to be part of this spec.

So it could be something like:

"Window_Size:  Window_Size MUST be at least 1 and less than 33. If the inter-frame gap is selected large enough to not overload the path and the one-way delay is known, the Window_Size SHOULD be set to the one-way trip time divided by the inter-frame gap.  Otherwise a small value of X SHOULD be configured. Note that the Window_Size determines the ack rate. If the window_size is set this to 32, this means only the last Fragment is acknowledged in the first round. If it is set to a smaller value, more acks are generated but the load on the forward path will be lower. Window_Size MAY be adapted dynamically to reduce load on the forward path in case of congestion.”

Still you also need to say more about how to set and dynamically adapt the inter-frame gap because that is probably the real limiting fact to avoid network overload.

Also below you remove the recommendation for using the number of hops as window size but here you added it again. This is just incorrect. There is no dependency between the number of hops and the window size: If there is no bottleneck on the path, you can just send with line rate at the sender. If there is a bottleneck on the path and you send at a higher rate than the bottleneck than soon or later the buffer at that hop will fill up completely. So the window size depends only on the bottleneck rate and end-to-end delay (BDP) (which let’s you calculate the number of packet in flight) plus the buffer size at the bottleneck. The number of hops is irrelevant.

Mirja

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 19 Mar 2020 14:57:15 +0000

Hello Mirja

> more comments. Please see below.

More comments => more thanks, you'll have to live with it 😊

As usual I published for your convenience so you can observe the global effect of your review.
Please see https://tools.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-18.txt 
(and as usual also I found a typo there that I fixed, this time the duplication of "reduce the")

Please see below the discussion:

> >> ---------------------------------------------------------------------

> Thanks for adding this. However, as I said a couple of times in my discuss there
> must be more guidance. This is not only about flow control but also about
> congestion control and it is not okay to declare congestion control out of
> scope. If you only do fragmentation but no retransmission, you don’t need to
> care about congestion control (but only flow control) as you don’t increase the
> actual network load by this. However, if you retransmit you are sending more
> data than the original sender (that hopefully is congestion controlled) and
> therefore you increase the load on the network and you MUST implement your
> own congestion control or some fixed rate limiting for that additional load.
> Saying this is out of scope and you want to do experimentation is not
> acceptable for a Proposed Standard.

I agree. I should be a lot more careful with my wording. 

Understanding that the flow control is pacing from the receiver to protect the receiver and congestion control is protecting the network (ECN etc...); stricto sensu  we are not doing any flow control since the receiver is expected to have a buffer for the whole datagram and he does not need to be protected. All we do is congestion control.

=> I should change "flow control" to "congestion control" throughout, and add text about the above, is that correct?

Note that all classical wireless interfaces perform ARQ. On one hop, you get the same effect of multiplying the traffic in the air vs. what the transport see. The factor can be high, e.g., 64. On a mesh, we get the additional effect of multiple flows converging on a same node.

> 
> To be clear the request of this discuss is to give a normative recommendation
> for the default value of the window size and/or inter-frame gap.

Yes, and since there is no great expectation that ECN will be implemented, that must be reasonable.
Also we want to agree on the proposed mechanism to drop the window to 1 in case of congestion notification, or is that behind us already?

> 
> Further note, as you allow to adapt both the window and the inter-frame gap
> dynamically, you actually implement two control mechanisms here. I actually
> recommend to only use the inter-frame gap and don’t have window here. You
> say a couple of times in your reply below, that the window determines the ask-
> rate, however, it is not clear to me because the ack rate should be a parameter
> at the receiver and not at the sender (maybe I don’t remember things correctly
> because it’s a while back since I read the draft and I couple find anything about
> this in the draft quickly). If the window size however does define the ack rate,
> then maybe you should rename that parameter respectively.

The ack is not for flow control (as we do not have it) but in support of ARQ. The possibility to use it for congestion control is a desirable side effect. The fragmenting endpoint FSM may want to wait and see how things went for the fragments that it already sent. E.g., there's the case where the fragmenting endpoint would use an ack on the first fragment for  a number of reasons such as check that a path is available, that the MTU is OK or assess an initial RTT. It may maximize the number of fragments in flight for congestion control. But whether to do any of that is left to implementation (so far).

> 
> However, if there is really a need for a window, I still recommend to talk less
> about adapting this value dynamically and make clear that having a fixed value
> is the recommended default. Therefore I recommend to remove the parameter
> MinWindowSize and MaxWindowSize because these should actually not be
> parameters than can be configured but are actual bounds. If someone decides
> to implement dynamic window adaption, they can decide to re-introduce these
> parameter again and make them configurable but it doesn’t need to be part of
> this spec.

I can see that, yes. I still like the idea to drop to 1 in case of ECN. 
Do you suggest to remove that as well?
If so, should we augment the inter-frame gap in case of ECN? 
That may be better though not simpler to specify or implement.

> "Window_Size:  Window_Size MUST be at least 1 and less than 33. If the inter-
> frame gap is selected large enough to not overload the path and the one-way

I see the IFG as more efficient for flow control than for congestion control: Increasing IFG slows down the packets but as long as the result is faster than the bottleneck, it does not help much does it? So I'm still  unsatisfied on how to characterize an IFG that does not overload a path. But I'm not sure we can do better. I moved that piece to the IFG definition if that's OK?

> delay is known, the Window_Size SHOULD be set to the one-way trip time
> divided by the inter-frame gap.  Otherwise a small value of X SHOULD be
> configured. Note that the Window_Size determines the ack rate. If the
> window_size is set this to 32, this means only the last Fragment is
> acknowledged in the first round. If it is set to a smaller value, more acks are
> generated but the load on the forward path will be lower. Window_Size MAY
> be adapted dynamically to reduce load on the forward path in case of
> congestion.”

The assertion that the load on the forwarding path will be lower is usually incorrect in a typical  LLN, since the radios are half duplex. In the example of 6TiSCH, an rfrag_ack consumes the exact same bandwidth as a fragment (one time slot). Also the path of the rfrag_ack is the reverse LSP, so it goes across the exact same links.

The last sentence is already present above in the text above it, all quoted below, so I'm also trying to avoid duplication.

> Still you also need to say more about how to set and dynamically adapt the
> inter-frame gap because that is probably the real limiting fact to avoid network
> overload.
> 

Yes, I see that tuning IFG impacts the rate and can help alleviate the congestion once you pass below the rate the bottleneck can give you. I've done some adaptive CIR long ago in IBM FR switches and it can be made to work, and though it was a lot of fun, it's not any easier than window-based flow control. And it really depends on the relay doing the right thing, e.g., reacting quickly on growing queue latency and applying fair sharing.

> Also below you remove the recommendation for using the number of hops as
> window size but here you added it again. This is just incorrect. There is no
> dependency between the number of hops and the window size: If there is no
> bottleneck on the path, you can just send with line rate at the sender. 

The rationale was: If there is no bottleneck on the path and the window is less than the number of hops then the sender will be blocked and the maximum throughput cannot be achieved.. If the rfrag_ack is as slow as the frag, which is reasonable in an LLN, we're talking about a window of twice the number of hops to keep the fragments going.

I saw the number of hops as a starting point, but I'm (really) happy to stick to RTT/IFG which makes more sense considering the focus that you seem to recommend placing on IFG (and I agree with that too).
 
>                                                                                                     If there
> is a bottleneck on the path and you send at a higher rate than the bottleneck
> than soon or later the buffer at that hop will fill up completely. So the window
> size depends only on the bottleneck rate and end-to-end delay (BDP) (which
> let’s you calculate the number of packet in flight) plus the buffer size at the
> bottleneck. The number of hops is irrelevant.

Yes, I understand that model. It was easier to apply some 25 years ago.
So far the links in a LLN are usually the same and the PHYs are usually the same so it's still usable.
But that is bound to change rapidly as even LLN radios are going to be agile WRT PHY rate. Meaning that the rate at the bottleneck will become hard to fathom and will change (rapidly) over time (same as Wi-Fi).

All in all we'd get:

"
   An implementation must control the rate at which it sends packets
   over the same path to allow the next hop to forward a packet before
   it gets the next.  In a wireless network that uses the same frequency
   along a path, more time must be inserted to avoid hidden terminal
   issues between fragments (more in Section 4.2).  An implementation
   should consider the generic recommendations from the IETF in the
   matter of congestion control and rate management in [RFC5033].  An
   implementation may perform a congestion control by using a dynamic
   value of the window size (Window_Size), adapting the fragment size
   (Fragment_Size), and may reduce the load by inserting an inter-frame
   gap that is longer than necessary.  In a large network where nodes
   contend for the bandwidth, a larger Fragment_Size consumes less
   bandwidth but also reduces fluidity and incurs higher chances of loss
   in transmission.

   This is controlled by the following parameters:

   inter-frame gap:  Indicates the minimum amount of time between
      transmissions.  The inter-frame gap protects the propagation of
      one transmission before the next one is triggered and creates a
      duty cycle that controls the ratio of air time and memory in
      intermediate nodes that a particular datagram will use.  The
      inter-frame gap controls the rate at which fragments are sent and
      SHOULD be selected large enough to protect the network.

<snip>

   Window_Size:  The Window_Size MUST be at least 1 and less than 33.

      *  If the round-trip time is known, the Window_Size SHOULD be set
         to the round-trip time divided by the time per fragment, that
         is the time to transmit a fragment plus the inter-frame gap.

      Otherwise:

      *  Setting the window_size to 32 is to be understood as only the
         last Fragment is acknowledged in each round.  This is the
         RECOMMENDED value in a half-duplex LLN where the fragment
         acknowledgment consumes roughly the same bandwidth on the same
         links as the fragments themselves

      *  If it is set to a smaller value, more acks are generated.  In a
         full-duplex network, the load on the forward path will be
         lower, and a small value of 3 SHOULD be configured.
"

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 19 Mar 2020 18:41:32 +0100

Hi Pascal,

Inline.

> On 19. Mar 2020, at 15:57, Pascal Thubert (pthubert) <pthubert=[email-masked]> wrote:
> 
> Hello Mirja
> 
>>>> -

Yes, thanks!

> Note that all classical wireless interfaces perform ARQ. On one hop, you get the same effect of multiplying the traffic in the air vs. what the transport see. The factor can be high, e.g., 64. On a…

Yes but with only “one hop”/the network you are connected to directly, and there is usually also some kind of back-off mechanism that reacts to congestion/collision/contention on that layer.

> Also we want to agree on the proposed mechanism to drop the window to 1 in case of congestion notification, or is that behind us already?

Dropping to 1 on CE mark is fine. However, when do you increase the window again? If you want to say something here, you have to specify that as well.

> That may be better though not simpler to specify or implement.

That’s an option as well. Again when you reduce something you might as well need to specify when to increase it again and that means you are specifying basically a congestion control scheme.

> I see the IFG as more efficient for flow control than for congestion control: Increasing IFG slows down the packets but as long as the result is faster than the bottleneck, it does not help much do…

So how do you currently set the IFG? Both IFG and window_size can be used for both flow as well as congestion control, it only depends who generated the signal that is used to adapt the value, either the endpoint/receiver or the network/nodes on the path.

Using a window would be a window-based congestion control. Using the IFG would be a rate-based congestion control. But the principle is the same.

> The assertion that the load on the forwarding path will be lower is usually incorrect in a typical  LLN, since the radios are half duplex. In the example of 6TiSCH, an rfrag_ack consumes the exact…

Okay. Yes makes sense.
> The rationale was: If there is no bottleneck on the path and the window is less than the number of hops then the sender will be blocked and the maximum throughput cannot be achieved.. If the rfrag_…

I see that the idea was rather to get the frames flowing (that avoiding overload) under the assumption that there is no bottleneck on the path. However, in this case you don’t really need the window at all and using the IFG should be enough.

>   over the same path to allow the next hop to forward a packet before

What does “same" relate to here?

>   matter of congestion control and rate management in [RFC5033].  An

Maybe RFC8085 is the better reference?

>   gap that is longer than necessary.

This is a bit the part that I don’t understand fully. Why do you need three different ways to enable congestion control instead of just having one.

You already have the IFG. What's the benefits of having a window based rate limit in addition?

>  In a large network where nodes

I think you need to provide some (normative) recommendation for the default configuration of the IFG. If that is specified in draft-ietf-6lo-minimal-fragment a pointer would be good and more explanation.

>         lower, and a small value of 3 SHOULD be configured.

If having the window is still useful, this is fine I think.

Mirja

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 19 Mar 2020 19:12:09 +0000

Hello Mirja

>>>>> -----------------------------------------
> 
>> 

If we keep things really simple it would not. Note that this applies to a single data gram and that’s usually just a few fragments.

We could double at each round trip but by the time it takes effect the datagram will be done...

> That’s an option as well. Again when you reduce something you might as well need to specify when to increase it again and that means you are specifying basically a congestion control scheme.

My goal for this doc was to keep it dead simple, build it so we have the necessary basis with ecn and windowing, and then play with it and learn from it. only then we can do a valuable spec.

can we keep it at what the doc has now?
> So how do you currently set the IFG? Both IFG and window_size can be used for both flow as well as congestion control, it only depends who generated the signal that is used to adapt the value, eith…

This is specified in minimal fragments. The IFG ensures that the previous fragment is beyond interference range. In a single frequency mesh that is multiple hops away.

> Using a window would be a window-based congestion control. Using the IFG would be a rate-based congestion control. But the principle is the same.
I’d love to chat about that at the next IETF to get your view. On paper the rate based does not guarantee the amount of bytes that this node will pack at the bottleneck. What I found implementing it years ago was that it is sensitive to when and how the congested node sets ECN. I ended up adjusting only once per RTT...

> I see that the idea was rather to get the frames flowing (that avoiding overload) under the assumption that there is no bottleneck on the path. However, in this case you don’t really need the windo…

Yep it is actually more constrained since IFG usually covers transmission over multiple hops. You found that I removed hop count throughout this time (hopefully) and followed your recommendation.

> What does “same" relate to here?

Same next hop (in TSCH) and possibly multiple hops but usually it does not know (requires link state which we usually do not have)

I can change over the same path to via te same next hop to keep it simple?
> Maybe RFC8085 is the better reference?

Happy to change 

> This is a bit the part that I don’t understand fully. Why do you need three different ways to enable congestion control instead of just having one.

To enable experimenting. The text above is true all this is possible. But the only thing that we mandate is resetting the window.

> You already have the IFG. What's the benefits of having a window based rate limit in addition?

Tuning IFG is complex. Implementation may prefer window over rate based. Once we have experience we’ll build the necessary specs.

> I think you need to provide some (normative) recommendation for the default configuration of the IFG. If that is specified in draft-ietf-6lo-minimal-fragment a pointer would be good and more explan…

It is and it I will. There’s already such reference but I can probably do better.

> If having the window is still useful, this is fine I think.

I don’t know but we’ll learn. Please let me know if we agree on the above and I’ll update tomorrow (it’s family time now in CET).

Take care,

Pascal

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 20 Mar 2020 14:48:40 +0100

Hi Pascal,

Okay! :-)

About the use of ECN, I agree as you say there should only be a few fragments and not increasing might be okay. However, you would need to clarify that the window is reset for each new datagram, I guess, right? Also I don’t think you necessarily need to reduce to 1 on CE marking but maybe halve the window or something. Or you leave this open like “If an E flag is received the window SHOULD be reduced, at least by 1 and at max to 1. Halving the window for each E flag received, could be a good compromise but needs further experimentation.”…

I wonder if it would be good to say a bit more about the recommended values for the window size, as I think 32 will usually in most network not limit transmission (and the limiting value will be IFG) while with a size of 3, that's very conservative to not overload the network (and will be slow than the limits induced by IFG). Is my understanding correct?

Mirja

> On 19. Mar 2020, at 20:12, Pascal Thubert (pthubert) <[email-masked]> wrote:
> 
> Hello Mirja
> 
> 
> 
>> 
>>> 
>>>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 20 Mar 2020 14:45:12 +0000

Hello Mirja:

Continuing from yesterday night.

I published 18 and then 19 for this discussion. 

So the total diff is https://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://tools.ietf.org/id/draft-ietf-6lo-fragment-recovery-17.txt&url2=https://tools.ietf.org/id/draft-ietf-6lo-fragment-recovery-19.txt 

Based on the discussion below, 7.1 would become:

"
7.1.  Protocol Parameters

   An implementation should consider the generic recommendations from
   the IETF in the matter of congestion control and rate management for
   IP datagrams in [RFC8085].  An implementation may perform a
   congestion control by using a dynamic value of the window size
   (Window_Size), adapting the fragment size (Fragment_Size), and may
   reduce the load by inserting an inter-frame gap that is longer than
   necessary.  In a large network where nodes contend for the bandwidth,
   a larger Fragment_Size consumes less bandwidth but also reduces
   fluidity and incurs higher chances of loss in transmission.

   inter-frame gap:  The inter-frame gap indicates the minimum amount of
      time between transmissions.  The inter-frame gap controls the rate
      at which fragments are sent, the ratio of air time, and the amount
      of memory in intermediate nodes that a particular datagram will
      use.  It can be used as a flow control, a congestion control, and/
      or a collision control measure.  It MUST be set at a minimum to a
      value that protects the propagation of one transmission against
      collision with next [FRAG-FWD].  In a wireless network that uses
      the same frequency along a path, this may represent the time for a
      frame to progress over multiple hops (more in Section 4.2).  It
      SHOULD be augmented beyond this as necessary to protect the
      network against congestion.
"

Where section 4.2 is where we indicate that the IFG is 
"inserted  between transmissions to the same next hop"

> -----Original Message-----
> From: Pascal Thubert (pthubert)
> Sent: jeudi 19 mars 2020 20:12
> To: Mirja Kuehlewind <[email-masked]>
> Cc: Pascal Thubert (pthubert) <pthubert=[email-masked]>; 6lo-
> [email-masked]; The IESG <[email-masked]>; [email-masked]; draft-ietf-6lo-
> [email-masked]; Carles Gomez <[email-masked]>; Martin
> Duke <[email-masked]>; Benjamin Kaduk <[email-masked]>
> Subject: Re: Mirja Kühlewind's Discuss on draft-ietf-6lo-fragment-recovery-13:
> (with DISCUSS and COMMENT)
> 
> Hello Mirja
> 
> 
> 
> >
> >>
> >>
> >> Note that all classical wireless interfaces perform ARQ. On one hop, you get
> the same effect of multiplying the traffic in the air vs. what the transport see.
> The factor can be high, e.g., 64. On a mesh, we get the additional effect of
> multiple flows converging on a same node.
> >
> > Yes but with only “one hop”/the network you are connected to directly, and
> there is usually also some kind of back-off mechanism that reacts to
> congestion/collision/contention on that layer.
> >
> >
> >>
> >>>
> >>> To be clear the request of this discuss is to give a normative
> >>> recommendation for the default value of the window size and/or inter-
> frame gap.
> >>
> >> Also we want to agree on the proposed mechanism to drop the window to 1
> in case of congestion notification, or is that behind us already?
> >
> > Dropping to 1 on CE mark is fine. However, when do you increase the window
> again? If you want to say something here, you have to specify that as well.
> 
> 
> If we keep things really simple it would not. Note that this applies to a single
> data gram and that’s usually just a few fragments.
> 
> >>> Further note, as you allow to adapt both the window and the
> >>> inter-frame gap dynamically, you actually implement two control
> >>> mechanisms here. I actually recommend to only use the inter-frame
> >>> gap and don’t have window here. You say a couple of times in your
> >>> reply below, that the window determines the ask- rate, however, it
> >>> is not clear to me because the ack rate should be a parameter at the
> >>> receiver and not at the sender (maybe I don’t remember things
> >>> correctly because it’s a while back since I read the draft and I
> >>> couple find anything about this in the draft quickly). If the window size
> however does define the ack rate, then maybe you should rename that
> parameter respectively.
> >>
> >> The ack is not for flow control (as we do not have it) but in support of ARQ.
> The possibility to use it for congestion control is a desirable side effect. The
> fragmenting endpoint FSM may want to wait and see how things went for the
> fragments that it already sent. E.g., there's the case where the fragmenting
> endpoint would use an ack on the first fragment for  a number of reasons such
> as check that a path is available, that the MTU is OK or assess an initial RTT. It
> may maximize the number of fragments in flight for congestion control. But
> whether to do any of that is left to implementation (so far).
> >>
> >>
> >>>
> >>> However, if there is really a need for a window, I still recommend
> >>> to talk less about adapting this value dynamically and make clear
> >>> that having a fixed value is the recommended default. Therefore I
> >>> recommend to remove the parameter MinWindowSize and
> MaxWindowSize
> >>> because these should actually not be parameters than can be
> >>> configured but are actual bounds. If someone decides to implement
> >>> dynamic window adaption, they can decide to re-introduce these
> >>> parameter again and make them configurable but it doesn’t need to be
> part of this spec.
> >>
> > That’s an option as well. Again when you reduce something you might as well
> need to specify when to increase it again and that means you are specifying
> basically a congestion control scheme.
> 
> My goal for this doc was to keep it dead simple, build it so we have the
> necessary basis with ecn and windowing, and then play with it and learn from
> it. only then we can do a valuable spec.
> 
> >>> "Window_Size:  Window_Size MUST be at least 1 and less than 33. If
> >>> the inter- frame gap is selected large enough to not overload the
> >>> path and the one-way
> >>
> >> I see the IFG as more efficient for flow control than for congestion control:
> Increasing IFG slows down the packets but as long as the result is faster than
> the bottleneck, it does not help much does it? So I'm still  unsatisfied on how to
> characterize an IFG that does not overload a path. But I'm not sure we can do
> better. I moved that piece to the IFG definition if that's OK?
> >
> > So how do you currently set the IFG? Both IFG and window_size can be used
> for both flow as well as congestion control, it only depends who generated the
> signal that is used to adapt the value, either the endpoint/receiver or the
> network/nodes on the path.
> >
> 
> This is specified in minimal fragments. The IFG ensures that the previous
> fragment is beyond interference range. In a single frequency mesh that is
> multiple hops away.
> 
> > Using a window would be a window-based congestion control. Using the IFG
> would be a rate-based congestion control. But the principle is the same.
> >
> >
> I’d love to chat about that at the next IETF to get your view. On paper the rate
> based does not guarantee the amount of bytes that this node will pack at the
> bottleneck. What I found implementing it years ago was that it is sensitive to
> when and how the congested node sets ECN. I ended up adjusting only once
> per RTT...
> 
> >>
> >>> delay is known, the Window_Size SHOULD be set to the one-way trip
> >>> time divided by the inter-frame gap.  Otherwise a small value of X
> >>> SHOULD be configured. Note that the Window_Size determines the ack
> >>> rate. If the window_size is set this to 32, this means only the last
> >>> Fragment is acknowledged in the first round. If it is set to a
> >>> smaller value, more acks are generated but the load on the forward
> >>> path will be lower. Window_Size MAY be adapted dynamically to reduce
> >>> load on the forward path in case of congestion.”
> >>
> >> The assertion that the load on the forwarding path will be lower is usually
> incorrect in a typical  LLN, since the radios are half duplex. In the example of
> 6TiSCH, an rfrag_ack consumes the exact same bandwidth as a fragment (one
> time slot). Also the path of the rfrag_ack is the reverse LSP, so it goes across
> the exact same links.
> >
> > Okay. Yes makes sense.
> >>
> >> The last sentence is already present above in the text above it, all quoted
> below, so I'm also trying to avoid duplication.
> >>
> >>> Still you also need to say more about how to set and dynamically
> >>> adapt the inter-frame gap because that is probably the real limiting
> >>> fact to avoid network overload.
> >>>
> >>
> >> Yes, I see that tuning IFG impacts the rate and can help alleviate the
> congestion once you pass below the rate the bottleneck can give you. I've done
> some adaptive CIR long ago in IBM FR switches and it can be made to work,
> and though it was a lot of fun, it's not any easier than window-based flow
> control. And it really depends on the relay doing the right thing, e.g., reacting
> quickly on growing queue latency and applying fair sharing.
> >>
> >>> Also below you remove the recommendation for using the number of
> >>> hops as window size but here you added it again. This is just
> >>> incorrect. There is no dependency between the number of hops and the
> >>> window size: If there is no bottleneck on the path, you can just send with
> line rate at the sender.
> >>
> >> The rationale was: If there is no bottleneck on the path and the window is
> less than the number of hops then the sender will be blocked and the
> maximum throughput cannot be achieved.. If the rfrag_ack is as slow as the
> frag, which is reasonable in an LLN, we're talking about a window of twice the
> number of hops to keep the fragments going.
> >
> > I see that the idea was rather to get the frames flowing (that avoiding
> overload) under the assumption that there is no bottleneck on the path.
> However, in this case you don’t really need the window at all and using the IFG
> should be enough.
> 
> 
> Yep it is actually more constrained since IFG usually covers transmission over
> multiple hops. You found that I removed hop count throughout this time
> (hopefully) and followed your recommendation.
> 
> >
> >>
> >> I saw the number of hops as a starting point, but I'm (really) happy to stick
> to RTT/IFG which makes more sense considering the focus that you seem to
> recommend placing on IFG (and I agree with that too).
> >>
> >>>
> >>> If there is a bottleneck on the path and you send at a higher rate
> >>> than the bottleneck than soon or later the buffer at that hop will
> >>> fill up completely. So the window size depends only on the
> >>> bottleneck rate and end-to-end delay (BDP) (which let’s you
> >>> calculate the number of packet in flight) plus the buffer size at the
> >> But that is bound to change rapidly as even LLN radios are going to be agile
> WRT PHY rate. Meaning that the rate at the bottleneck will become hard to
> fathom and will change (rapidly) over time (same as Wi-Fi).
> >>
> >> All in all we'd get:
> >>
> >> "
> Same next hop (in TSCH) and possibly multiple hops but usually it does not
> know (requires link state which we usually do not have)
> 
> >>  it gets the next.  In a wireless network that uses the same
> >> frequency  along a path, more time must be inserted to avoid hidden
> >> terminal  issues between fragments (more in Section 4.2).  An
> >> implementation  should consider the generic recommendations from the
> >> IETF in the  matter of congestion control and rate management in
> >> [RFC5033].  An
> >
> > This is a bit the part that I don’t understand fully. Why do you need three
> different ways to enable congestion control instead of just having one.
> 
> To enable experimenting. The text above is true all this is possible. But the only
> thing that we mandate is resetting the window.
> 
> 
> >
> Tuning IFG is complex. Implementation may prefer window over rate based.
> Once we have experience we’ll build the necessary specs.
> 
> 
> > I think you need to provide some (normative) recommendation for the
> default configuration of the IFG. If that is specified in draft-ietf-6lo-minimal-
> fragment a pointer would be good and more explanation.
> >
> 
> I don’t know but we’ll learn. Please let me know if we agree on the above and
> I’ll update tomorrow (it’s family time now in CET).
> 
> Take care,
> 
> Pascal
> 
> > Mirja
> >
> >
> >
> >>
> >

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 20 Mar 2020 16:15:05 +0000

Hello Mirja

I missed this while working on yesterday's message.

Let's see below:

> Okay! :-)
> 
> About the use of ECN, I agree as you say there should only be a few fragments
> and not increasing might be okay. However, you would need to clarify that the
> window is reset for each new datagram, I guess, right? 

Agreed. Let's change appendix C, more below;

> Also I don’t think you
> necessarily need to reduce to 1 on CE marking but maybe halve the window or
> something. Or you leave this open like “If an E flag is received the window
> SHOULD be reduced, at least by 1 and at max to 1. Halving the window for
> each E flag received, could be a good compromise but needs further
> experimentation.”…

Thanks for this text! Appendix C now becomes:
"
   Congestion on the forward path can also be indicated by an Explicit
   Congestion Notification (ECN) mechanism.  Though whether and how ECN
   [RFC3168] is carried out over the LoWPAN is out of scope, this draft
   provides a way for the destination endpoint to echo an ECN indication
   back to the fragmenting endpoint in an acknowledgment message as
   represented in Figure 4 in Section 5.2.  While the support of echoing
   the ECN at the reassembling endpoint is mandatory, this specification
   only provides a minimalistic behaviour on the fragmenting endpoint.
   If an E flag is received the window SHOULD be reduced, at least by 1
   and at max to 1.  Halving the window for each E flag received, could
   be a good compromise but needs further experimentation.  A very
   simple implementation may just reset the window to 1 so the fragments
   are sent and acknowledged one by one.  Note that the action on ECN
   only applies till the end of the datagram and the next datagram uses
   the configured Window_Size again.

"

I'd like to fully converge before I publish again

> I wonder if it would be good to say a bit more about the recommended values
> for the window size, as I think 32 will usually in most network not limit
> transmission (and the limiting value will be IFG) while with a size of 3, that's
> very conservative to not overload the network (and will be slow than the limits
> induced by IFG). Is my understanding correct?

I'd believe so:

Note that the exact use of the ack and the window_size is left to implementation. The optimistic one could send all the fragments up to window_size and ask for an ack only on the last one, wait for the bitmap, which means a gap of RTT/2, and resend the losses.
Then again we want to leave room for learning. The pessimistic implementation could set the bit on the first fragment to check that the path works and open the window only upon receiving the ack. It could then set an ack bit again on the second fragment and then use the window as a credit to send up to window_size before it is blocked. In that case, if the ack comes back before the window starves, the gating factor is the IFG. If the ack does not arrive in time, the window_size is the gating factor.

If the sender uses the window_size as a credit, the window of 3 will be the gating factor that starves the sender and causes gaps longer than IFG as soon as you have more than 3 hops in a TSCH network and 5-6 hops in a single frequency mesh. I guess that's what you mean by slower. The more hops the more it will hurt.

I initially used nb_of_hops as a rule of a thumb. That was an approximation of RTT/(XMIT+IFG) in the case one fragment needs to progress only to the second hop before the next can be sent (case of TSCH). 

The assumptions behind are that an rfrag_ack takes the same path and the same time as the rfrag, which is mostly correct. So if we spread the fragments with IFG, keeping busy one hop out of 2 on both directions means aligning the window_size of the nb of hops. As you pointed out this is not a congestion control measure, it is just the window that is not "slower" than IFG, per your expression. The text that uses RTT/(XMIT+IFG) being equivalent I'm happy either way. Because knowing the RTT is the same problem as knowing the number of hops so we do not need to indicate both. 

3 is when you h	ave no idea of either RTT or nb of hops. it is conservative and will starve the TSCH sender if we have more than 3 hops. It also protects the intermediate node that can be very constrained, to at most 3 fragments per datagram that it relays. Note that I'm not aware of any full duplex radio available in LLNs (though there are prototypes for 5G). So this is really a corner case.

As written 32 indicates the last fragment of the datagram, which is usually way less than 32. When used, as you understood well, the only protection is IFG, and I'm must say I'm quite happy with the progress we made on that text together.

Please let me know, should I change something more?

Pascal

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 20 Mar 2020 18:11:39 +0100

Hi Pascal,

The proposed text below and in your previous email look good! Thanks!

I would appreciate and recommend to add also a shorter version of your good explanation below to make the reader better understand which factor is the limiting one when using a certain parameter setting and respectively which factor could be tuned if any dynamic adaptation is implemented.

Thanks!
Mirja

P.S.: Off for today but will check for an update tomorrow in order to clear my discuss!

> On 20. Mar 2020, at 17:15, Pascal Thubert (pthubert) <pthubert=[email-masked]> wrote:
> 
> Hello Mirja
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 20 Mar 2020 18:25:34 +0000

Hello Mirja:

I published 20 with this addition.

https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-20

I guess we're close to be done by now. In that case, I must thanks you again and deeply.

Be safe,

Pascal

> From: Mirja Kuehlewind <[email-masked]>
> Sent: vendredi 20 mars 2020 18:12
> To: Pascal Thubert (pthubert) <[email-masked]>
> Cc: [email-masked]; The IESG <[email-masked]>; [email-masked]; draft-ietf-6lo-
> I would appreciate and recommend to add also a shorter version of your good
> explanation below to make the reader better understand which factor is the
> limiting one when using a certain parameter setting and respectively which
> factor could be tuned if any dynamic adaptation is implemented.
> 
> Thanks!
> Mirja
> 
> > On 20. Mar 2020, at 17:15, Pascal Thubert (pthubert)
> <pthubert=[email-masked]> wrote:
> >
> > Hello Mirja
> >
> >> About the use of ECN, I agree as you say there should only be a few
> >> fragments and not increasing might be okay. However, you would need
> >> to clarify that the window is reset for each new datagram, I guess, right?
> >
> >> necessarily need to reduce to 1 on CE marking but maybe halve the
> >> window or something. Or you leave this open like “If an E flag is
> >> received the window SHOULD be reduced, at least by 1 and at max to 1.
> >> Halving the window for each E flag received, could be a good
> >> compromise but needs further experimentation.”…
> >
> >> I wonder if it would be good to say a bit more about the recommended
> >> values for the window size, as I think 32 will usually in most
> >> network not limit transmission (and the limiting value will be IFG)
> >> while with a size of 3, that's very conservative to not overload the
> >> network (and will be slow than the limits induced by IFG). Is my
> understanding correct?
> >
> > I'd believe so:
> >
> > Note that the exact use of the ack and the window_size is left to
> implementation. The optimistic one could send all the fragments up to
> window_size and ask for an ack only on the last one, wait for the bitmap, which
> means a gap of RTT/2, and resend the losses.
> > Then again we want to leave room for learning. The pessimistic
> implementation could set the bit on the first fragment to check that the path
> works and open the window only upon receiving the ack. It could then set an
> ack bit again on the second fragment and then use the window as a credit to
> send up to window_size before it is blocked. In that case, if the ack comes back
> before the window starves, the gating factor is the IFG. If the ack does not
> arrive in time, the window_size is the gating factor.
> >
> > If the sender uses the window_size as a credit, the window of 3 will be the
> gating factor that starves the sender and causes gaps longer than IFG as soon
> as you have more than 3 hops in a TSCH network and 5-6 hops in a single
> frequency mesh. I guess that's what you mean by slower. The more hops the
> more it will hurt.
> >
> > I initially used nb_of_hops as a rule of a thumb. That was an approximation of
> RTT/(XMIT+IFG) in the case one fragment needs to progress only to the second
> hop before the next can be sent (case of TSCH).
> >
> > The assumptions behind are that an rfrag_ack takes the same path and the
> same time as the rfrag, which is mostly correct. So if we spread the fragments
> with IFG, keeping busy one hop out of 2 on both directions means aligning the
> window_size of the nb of hops. As you pointed out this is not a congestion
> control measure, it is just the window that is not "slower" than IFG, per your
> expression. The text that uses RTT/(XMIT+IFG) being equivalent I'm happy
> either way. Because knowing the RTT is the same problem as knowing the
> number of hops so we do not need to indicate both.
> >
> > 3 is when you h	ave no idea of either RTT or nb of hops. it is
> conservative and will starve the TSCH sender if we have more than 3 hops. It
> also protects the intermediate node that can be very constrained, to at most 3
> fragments per datagram that it relays. Note that I'm not aware of any full
> duplex radio available in LLNs (though there are prototypes for 5G). So this is
> really a corner case.
> >
> > As written 32 indicates the last fragment of the datagram, which is usually
> way less than 32. When used, as you understood well, the only protection is
> IFG, and I'm must say I'm quite happy with the progress we made on that text
> together.
> >

---

From: Mirja Kuehlewind <[email-masked]>
Date: Sun, 22 Mar 2020 19:09:44 +0100

Hi Pascal,

Thanks for this additional text. I just cleared my discuss.

A couple of nits you/Suresh maybe want to add as RFC editor note:

OLD
Note that the action upon detecting a congestion only applies till the end of the datagram
NEW
Note that the action upon detecting congestion only applies till the end of the datagram.
MAYBE EVEN
Note that any action that has been performed upon detection of congestion only applies for the transmission of one datagram

OLD
an inter-frame gap can be as a flow control or a congestion control measure
NEW
The inter-frame gap can be used as a flow control or a congestion control measure

OLD
a conservative Window_Size of, say, 3, will be the gating factor that starves the sender
MAYBE
a conservative Window_Size of, say, 3, will be the gating factor that limits the transmission rate of the sender

Mirja

> On 20. Mar 2020, at 19:25, Pascal Thubert (pthubert) <pthubert=[email-masked]> wrote:
> 
> Hello Mirja:
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 23 Mar 2020 13:54:46 +0000

Hello Again Mirja : )

> Thanks for this additional text. I just cleared my discuss.
Very cool; again and again, many thanks for your deep reviews.

> A couple of nits you/Suresh maybe want to add as RFC editor note:

Happy to republish:           https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-21 

> NEW Note that the action upon detecting congestion only
> applies till the end of the datagram.
> MAYBE EVEN
> Note that any action that has been performed upon detection of congestion
> only applies for the transmission of one datagram

Picked the latter

> NEW The inter-frame gap can be used as a flow control or a congestion
> control measure

done

> MAYBE a conservative Window_Size of, say, 3, will be the gating
> factor that limits the transmission rate of the sender
> 

Again and again, many thanks;
all the best with the IAB,

Pascal

---

From: Suresh Krishnan <[email-masked]>
Date: Mon, 23 Mar 2020 18:03:10 -0400

Hi Pascal,
  I went through the latest rev and found an off-by-one error on the Fragment size.

* Section 5.1.

   Fragment_Size:  10-bit unsigned integer; the size of this fragment in
      a unit that depends on the Link-Layer technology.  Unless
      overridden by a more specific specification, that unit is the
      byte, which allows fragments up to 1024 bytes.

Shouldn’t this be fragments upto *1023* bytes instead of 1024 bytes?

Thanks
Suresh

> On Mar 23, 2020, at 9:54 AM, Pascal Thubert (pthubert) <pthubert=[email-masked]> wrote:
> 
> Hello Again Mirja : )
> 
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 24 Mar 2020 06:19:27 +0000

Aïe, true!

Can we leave it to the RFC editor?

Be safe!

Pascal

> Le 23 mars 2020 à 23:03, Suresh Krishnan <[email-masked]> a écrit :
> 
> Hi Pascal,

---

From: Suresh Krishnan <[email-masked]>
Date: Tue, 24 Mar 2020 09:39:59 -0400

Hi Pascal,

> On Mar 24, 2020, at 2:19 AM, Pascal Thubert (pthubert) <[email-masked]> wrote:
> 
> Aïe, true!
> 

Sounds good. I have made the change with an RFC Editor Note.

Regards
Suresh

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 24 Mar 2020 15:56:10 +0000

: )

Keep safe, Suresh!

> From: Suresh Krishnan <[email-masked]>
> Sent: mardi 24 mars 2020 14:40
> Cc: Pascal Thubert (pthubert) <pthubert=[email-masked]>; Mirja
> Kuehlewind <[email-masked]>; [email-masked]; The IESG
> <[email-masked]>; [email-masked]; [email-masked];
> Carles Gomez <[email-masked]>; Martin Duke
> <[email-masked]>; Benjamin Kaduk <[email-masked]>