Subject: Re: [Last-Call] Last Call: <draft-ietf-lwig-tcp-constrained-node-networks-10.txt> (TCP Usage Guidance in the Internet of Things (IoT)) to Informational RFC

From: Ted Lemon <[email-masked]>
Date: Wed, 16 Sep 2020 10:01:48 -0400

To a first approximation, this document is harmful garbage. It recommends larger mss. Probably not based on opex.

> On Sep 16, 2020, at 9:55 AM, The IESG <[email-masked]> wrote:
> 
> 
> The IESG has received a request from the Light-Weight Implementation Guidance
> WG (lwig) to consider the following document: - 'TCP Usage Guidance in the
> Internet of Things (IoT)'
>  <draft-ietf-lwig-tcp-constrained-node-networks-10.txt> as Informational RFC
> 
> The IESG plans to make a decision in the next few weeks, and solicits final
> comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2020-09-30. Exceptionally, comments may
> be sent to [email-masked] instead. In either case, please retain the beginning
> of the Subject line to allow automated sorting.
> 
> Abstract
> 
> 
>   This document provides guidance on how to implement and use the
>   Transmission Control Protocol (TCP) in Constrained-Node Networks
>   (CNNs), which are a characterstic of the Internet of Things (IoT).
>   Such environments require a lightweight TCP implementation and may
>   not make use of optional functionality.  This document explains a
>   number of known and deployed techniques to simplify a TCP stack as
>   well as corresponding tradeoffs.  The objective is to help embedded
>   developers with decisions on which TCP features to use.
> 
> 
> 
> 
> The file can be obtained via
> https://datatracker.ietf.org/doc/draft-ietf-lwig-tcp-constrained-node-networks/
> 
> 
> 
> No IPR declarations have been submitted directly on this I-D.
> 
> 
> 
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: Ted Lemon <[email-masked]>
Date: Wed, 16 Sep 2020 10:05:43 -0400

> To a first approximation, this document is harmful garbage. It recommends larger mss. Probably not based on opex.

Wow, my first reply-all screwup in ages, and it was pretty bad. Really sorry about this—I did not intend to be this transparent on a public mailing list, and I hope my first impression is wrong. But with that said, that is my first impression, and I will be giving the document further review. It’s very disappointing to me that lwig has come out with this without, as far as I can tell, any reference to existing research that shows that MSS of greater than perhaps five lowpan frames is quite harmful.

Could the authors common on what operational experience this document is based on?

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Fri, 18 Sep 2020 14:14:56 +0200

Hi Ted,

This document aims at offering guidance on the use of TCP in IoT
environments or use cases, based on contributions from a really long list
of folks from the LWIG and TCPM working groups, with diverse backgrounds
and expertise (from academia, industry, implementers, etc.). If
interested, you may want to see the Acknowledgments section.

Could you please provide any pointers to "existing research that
shows that MSS of greater than perhaps five lowpan frames is quite
harmful." ?

Also, we believe that the statement that the document "recommends larger
mss" is not true or accurate. In 4.1.1 we rather discuss the benefits of
not exceeding an MTU of 1280 bytes, and we give guidelines that may help
in this regard.

There is also the last paragraph of 4.1.1, which includes:

   "using larger MSS (to a suitable extent) may be beneficial, especially
    when transferring large payloads, as it reduces the number of packets
   (and packet headers) required for a given payload"

but it purposefully includes "to a suitable extent", which is also not
quantified, and it is given in the context of transferring large payloads.
And the "may" is also very relevant here.

We believe that what that last paragraph states is actually quite obvious,
implicitly assuming that the application of that paragraph will be done in
a careful, cautious way...

Best regards,

Carles (on behalf of all coauthors)

>> To a first approximation, this document is harmful garbage. It
>> recommends larger mss. Probably not based on opex.
>
> Wow, my first reply-all screwup in ages, and it was pretty bad. Really
> sorry about this—I did not intend to be this transparent on a public
> mailing list, and I hope my first impression is wrong. But with that said,
> that is my first impression, and I will be giving the document further
> review. It’s very disappointing to me that lwig has come out with this
> without, as far as I can tell, any reference to existing research that
> shows that MSS of greater than perhaps five lowpan frames is quite
> harmful.
>
> Lwip mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lwip
>

---

From: Ted Lemon <[email-masked]>
Date: Fri, 18 Sep 2020 09:46:27 -0400

On Sep 18, 2020, at 8:14 AM, Carles Gomez Montenegro <[email-masked]> wrote:
> harmful.”
> ?

So, first of all, I really owe you an apology for both comments—this is just great evidence for why we should never say something about someone else’s work that we are not prepared for them to read. My reaction was the result of going to the thing I cared most about in the document, finding what it said to be incorrect based on what I think is true, and assuming that this would be representative of the rest of the document. There was actually no reason for me to make this assumption; what I was actually looking for from the person to whom I intended to send the message was reassurance that this wasn’t as bad as it seemed. My subsequent comment on the list acknowledging this faux pas actually made it worse because I was embarrassed and hadn’t gotten past the self-justification stage to the simple retraction stage. Sigh.

The specific paper I am thinking of is one titled Performant TCP for Low-Power Wireless Networks, by Sam Kumar, Michael P Andersen, Hyung-Sin Kim, and David E. Culler at UC Berkeley (https://www.usenix.org/conference/nsdi20/presentation/kumar <https://www.usenix.org/conference/nsdi20/presentation/kumar>). Reviewing the paper, what it says is not inconsistent with what you’ve said. It appears to be the case that an mss of five frames tends to perform better than an mss of fewer frames, and that for example an mss of one frame performs poorly. Performance appears to increase up to five frames. So in a sense this is supporting the notion that even more frames would be better, but this was not studied.

The reason I’m concerned about this is that it’s my understanding that generally on LLNs fragments are acknowledged at the packet level, not at the fragment level. This means that if five fragments are transmitted and one dropped, all five have to be retransmitted. This assumption may actually not be true—I haven’t tested it. It’s based on what others have told me about how this works. If this assumption isn’t true, then my primary concern goes away. The concern I have is that as packet loss rates rise, the likelihood of any given IP packet making it across an 802.15.4 mesh intact (with no fragments lost) drops. Because every fragment has to be retransmitted, this can result in a lot of extra traffic being sent, which can further increase packet loss.

So if that’s true, it makes sense to keep the mss short, preferably shorter than 1280. 1280 would be thirteen fragments on 802.15.4, if my math is right. I think ideally we’d want to keep the MSS down nearer to 500 bytes.

It may be that my reasoning is completely wrong here, but the reason for my reaction to this document is that this is my present understanding of the problem, and hence the recommendation of a 1280 byte mss seems wrong. If I misread that, I apologize. As I said, I need to give the document a closer read.  Generally speaking I would really like to see the old canard that TCP can’t work on LLNs put to pasture, so I want this work to succeed. Again, I’m sorry for the unkind comment.

---

From: Michael Richardson <[email-masked]>
Date: Sat, 19 Sep 2020 17:49:49 -0400

Ted Lemon <[email-masked]> wrote:
    > The reason I’m concerned about this is that it’s my understanding that
    > generally on LLNs fragments are acknowledged at the packet level, not
    > at the fragment level. This means that if five fragments are
    > transmitted and one dropped, all five have to be retransmitted. This
    > assumption may actually not be true—I haven’t tested it. It’s based on
    > what others have told me about how this works. If this assumption isn’t
    > true, then my primary concern goes away. The concern I have is that as
    > packet loss rates rise, the likelihood of any given IP packet making it
    > across an 802.15.4 mesh intact (with no fragments lost) drops. Because
    > every fragment has to be retransmitted, this can result in a lot of
    > extra traffic being sent, which can further increase packet loss.

There is always a terminology issue, because things got overloaded in my opinion.
RFC4944 was never clear enough for my liking, and overloads "fragment"

So let me be clear:
   frame: a thing transmitted at layer 2
   fragment: a part of an IP packet that has undergone fragmentation
   fraglet:  a sometimes used term for "fragmentation" at layer 2.
   segment: a part of a TCP transmission that can be ACKnowledged
   packet: a IP packet as we know it.

LLNs usually take packets which are 1280 or less and turn them into fraglets
("fragments") for transmission through the LLN.  Many LLN technologies
including 802.15.4 provide to ACKs for each transmission, and there is a
retransmission a few times, but then it stops and goes on.  Does this result
in an immediate loss of the entire packet?  Yes, often.

RFC4944 says that each node re-assembles the fraglets to a packet at each
hop, and then tries again.  But, we have proposals for fraglet forwarding.

IP can, of course, fragment a packet into many IP fragments, each of which
can be transmitted on their own, to be assembled by the end points, not the
intermediate nodes.

TCP works best if you never create IP fragments, so setting the segment size
to what can efficiently get through the LLN is good.

I'm sorry to repeat, but I think we've done ourselves a disservice by not
being more precise.

    > So if that’s true, it makes sense to keep the mss short, preferably
    > shorter than 1280. 1280 would be thirteen fragments on 802.15.4, if my
    > math is right. I think ideally we’d want to keep the MSS down nearer to
    > 500 bytes.

It sounds right that an *MSS* of 1280 would be 13 fraglets on 802.15.4.
I think that if you set the MTU to 1280, you get an MSS smaller than 1280 though.

     > It may be that my reasoning is completely wrong here, but the reason
     > for my reaction to this document is that this is my present
     > understanding of the problem, and hence the recommendation of a 1280
     > byte mss seems wrong. If I misread that, I apologize. As I said, I
     > need to give the document a closer read.  Generally speaking I would
     > really like to see the old canard that TCP can’t work on LLNs put to
     > pasture, so I want this work to succeed. Again, I’m sorry for the
     > unkind comment.

The fraglet forwarding proposal would more reliably forward larger IP
packets, so a larger MSS might actually be more efficient use of energy.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Sun, 20 Sep 2020 08:28:30 +0200

Hi Ted,

Thanks for your response. Please find below my inline responses:

> So, first of all, I really owe you an apology for both comments—this is
> just great evidence for why we should never say something about someone
> else’s work that we are not prepared for them to read. My reaction was
> the result of going to the thing I cared most about in the document,
> finding what it said to be incorrect based on what I think is true, and
> assuming that this would be representative of the rest of the document.
> There was actually no reason for me to make this assumption; what I was
> actually looking for from the person to whom I intended to send the
> message was reassurance that this wasn’t as bad as it seemed. My
> subsequent comment on the list acknowledging this faux pas actually made
> it worse because I was embarrassed and hadn’t gotten past the
> self-justification stage to the simple retraction stage. Sigh.

No worries, and thanks for your detailed response.

> The specific paper I am thinking of is one titled Performant TCP for
> Low-Power Wireless Networks, by Sam Kumar, Michael P Andersen, Hyung-Sin
> Kim, and David E. Culler at UC Berkeley
> (https://www.usenix.org/conference/nsdi20/presentation/kumar
> <https://www.usenix.org/conference/nsdi20/presentation/kumar>). Reviewing
> the paper, what it says is not inconsistent with what you’ve said. It
> appears to be the case that an mss of five frames tends to perform better
> than an mss of fewer frames, and that for example an mss of one frame
> performs poorly. Performance appears to increase up to five frames. So in
> a sense this is supporting the notion that even more frames would be
> better, but this was not studied.

Thanks for the insight on the paper you mention. It offers interesting
details, and also experimental results consistent with our text, at least
for some MSS range. It would have been great to see results for even
greater MSS than the ones considered in the paper.

> generally on LLNs fragments are acknowledged at the packet level, not at
> the fragment level. This means that if five fragments are transmitted and
> one dropped, all five have to be retransmitted. This assumption may
> actually not be true—I haven’t tested it. It’s based on what others
> have told me about how this works. If this assumption isn’t true, then
> my primary concern goes away. The concern I have is that as packet loss
> rates rise, the likelihood of any given IP packet making it across an
> 802.15.4 mesh intact (with no fragments lost) drops. Because every
> fragment has to be retransmitted, this can result in a lot of extra
> traffic being sent, which can further increase packet loss.
>
>

In the document, our aim is to be comprehensive regarding underlying IoT
technologies. Some of these support a sufficiently large L2 MTU, and
therefore fragmentation is not needed. Many other IoT technologies support
their own (ARQ-enhanced) fragmentation mechanism at the link layer.
However, there are indeed examples where fragmentation does not support
additional functionality for reliable fragment delivery, such as the
original 6LoWPAN fragmentation defined in RFC 4944 over IEEE 802.15.4. In
such cases, a relatively large MSS may certainly create the issue that you
mentioned.

Would the following proposed new text (that would replace the last
paragraph of 4.1.1) address your concern?

PROPOSED:

   Using larger MSS (to a suitable extent) may be beneficial in some
   scenarios, especially when transferring large payloads, as it reduces the
   number of packets (and packet headers) required for a given payload.
   However, the characteristics of the constrained network need to be
   considered. In particular, in a network where unreliable fragment
   delivery is used, the amount of data that TCP unnecessarily
   retransmits due to fragment loss increases with the MSS. This happens
   because the loss of a fragment leads to the loss of the whole fragmented
   packet being transmitted. Unnecessary data retransmission is particularly
   harmful in CNNs due to the resource constraints of such environments.
   Note that, while the original 6LoWPAN fragmentation mechanism [RFC 4944]
   does not offer reliable fragment delivery, fragment recovery
   functionality for 6LoWPAN or 6Lo environments is being standardized as of
   the writing [draft-ietf-6lo-fragment-recovery].

> It may be that my reasoning is completely wrong here, but the reason for
> my reaction to this document is that this is my present understanding of
> the problem, and hence the recommendation of a 1280 byte mss seems wrong.
> If I misread that, I apologize. As I said, I need to give the document a
> closer read.  Generally speaking I would really like to see the old canard
> that TCP can’t work on LLNs put to pasture, so I want this work to
> succeed. Again, I’m sorry for the unkind comment.

Thanks for your words. Yes, breaking the myth that 'TCP is not suitable
for IoT' is one of the main objectives of this document. Let's hope we can
contribute to that!

Cheers,

Carles

---

From: Ted Lemon <[email-masked]>
Date: Fri, 25 Sep 2020 13:52:14 -0400

> greater MSS than the ones considered in the paper.

I agree. My intuition is that there is likely a peak number of segments past which adding more segments reduces performance; I suspect that this is probably close to the numbers the Berkeley paper arrived at, but we have no data. It would be good to have more data before making general assertions about what mss IoT devices should use, particularly given that some IoT transports may do per-fragment retransmission, while others won’t, and as you say, some IoT transports have reasonably large MTUs, while others don’t.
>   the writing [draft-ietf-6lo-fragment-recovery].

I think this is okay, although you don’t mention that given a constant per-frame error rate, the more frames you send, the higher the actual error rate will be, that this increases exponentially as the number of fragments increases, and further that, as you mention, in CNNs, retransmission traffic can swamp successful transmissions leading to worse and worse throughput as MSS increases.

I don’t think it’s sufficient to mention this in a few sentences in a single paragraph. This needs to be part of a more detailed analysis.

> for IoT' is one of the main objectives of this document. Let's hope we can

100% agree. Thanks!