Subject: [tcpm] Questions about TLP

From: Yi Huang <[email-masked]>
Date: Thu, 2 May 2019 05:47:56 +0000

Hi RACK/TLP authors,

I have the following questions (page numbers refer to draft-ietf-tcpm-rack-05):

1.In page 16, it says "Finally, if the time at which an RTO would fire (here denoted "TCP_RTO_expire") is sooner than the computed time for the PTO, then a probe is scheduled to be sent at that earlier time." What does this TCP_RTO_expire actually mean? Does it mean there is another RTO timer (possibly started some time in the past) running along with PTO or just RTT + 4*RTTVar + Now()? Also, why would another probe be sent at RTO expiration time instead of treating it like RTO and collapsing the cwnd?

2.In page 18 section 6.6, a new variable TLPRxtOut is defined and it is stated that TLPRxtOut is used to guarantee that there is only one outstanding TLP retransmission. However, it is not clear to me when TLPRxtOut should be really used. Should it be used in 6.5.1 Step. 1 (checking whether we should and can schedule a TLP) or in TLP_send_probe()?

Thanks,

Yi

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 1 May 2019 23:33:04 -0700

On Wed, May 1, 2019 at 10:48 PM Yi Huang
<huanyi=[email-masked]> wrote:
> 1.In page 16, it says “Finally, if the time at which an RTO would fire (here denoted "TCP_RTO_expire") is sooner than the computed time for the PTO, then a probe is scheduled to be sent at that earlier time.” What does this TCP_RTO_expire actually mean? Does it mean there is another RTO timer (possibly started some time in the past) running along with PTO or just RTT + 4*RTTVar + Now()? Also, why would another probe be sent at RTO expiration time instead of treating it like RTO and collapsing the cwnd?

TCP_RTO_expire() should return a timeout value for regular RTO, i.e.
SRTT + 4*RTTVAR + Now()
We use TCP_RTO_expire() because many implementations including Linux
do not use the exact RFC formula

The reason it's not treated as a regular RTO is to avoid resetting
congestion window to 1. The rationale is if the probe was sent within
min(PTO, RTO) and was delivered successfully, there's no need to reset
congestion window and re-start slow-start, similar to the rationale of
Reno's fast recovery reducing window to ssthresh instead of 1. We have
found this strategy benefits wireless connections, as the chance of
spurious RTO is high due to the delay variation.

>
>
>
> 2.In page 18 section 6.6, a new variable TLPRxtOut is defined and it is stated that TLPRxtOut is used to guarantee that there is only one outstanding TLP retransmission.. However, it is not clear to me when TLPRxtOut should be really used. Should it be used in 6.5.1 Step. 1 (checking whether we should and can schedule a TLP) or in TLP_send_probe()?

This is in

6.6.2.  Recording loss probe states

   Senders MUST only send a TLP loss probe retransmission if TLPRxtOut
   is false.  This ensures that at any given time a connection has at
   most one outstanding TLP retransmission.  This allows the sender to

but I agree it'd be more clear to include this in TLP_send_probe()
pseudo code. I'd incorporate in the next rev.
>
>
>
> Thanks,
>
>
>
> Yi
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 2 May 2019 10:25:07 -0700

On Thu, May 2, 2019 at 10:09 AM Matt Olson <[email-masked]> wrote:
>
> Also, section 6.5.1 Step 1 is presented as a complete set of conditions for scheduling a PTO, but it is missing a bullet point for TLPRxtOut.
Thanks for checking: but checking TLPRxtOut is not necessary and is
not preferred when scheduling a probe -- the goal is to prevent more
than one probe inflight, so checking right before TLPRxtOut sending the next one
allows the best coverage of application-limited writes (burst-idle-burst-...).

>
> -----Original Message-----
> From: Yuchung Cheng <[email-masked]>
> Sent: Wednesday, May 1, 2019 11:33 PM
> To: Yi Huang <huanyi=[email-masked]>
> Cc: [email-masked]; Matt Olson <[email-masked]>
> Subject: Re: [tcpm] Questions about TLP
>
> On Wed, May 1, 2019 at 10:48 PM Yi Huang <huanyi=[email-masked]> wrote:
> >
> > Hi RACK/TLP authors,
> >
> >
> >
> We use TCP_RTO_expire() because many implementations including Linux do not use the exact RFC formula
>
> The reason it's not treated as a regular RTO is to avoid resetting congestion window to 1. The rationale is if the probe was sent within min(PTO, RTO) and was delivered successfully, there's no need to reset congestion window and re-start slow-start, similar to the rationale of Reno's fast recovery reducing window to ssthresh instead of 1. We have found this strategy benefits wireless connections, as the chance of spurious RTO is high due to the delay variation.
>
> >
> >
> >
> but I agree it'd be more clear to include this in TLP_send_probe() pseudo code. I'd incorporate in the next rev.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Yi
> >
> >
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=01%7C01%7Cmaolson%40micr
> > osoft.com%7C66db44672e3d4101d72908d6cec82001%7C72f988bf86f141af91ab2d7
> > cd011db47%7C1&amp;sdata=n8rp5Emowm459iSKolhIP1hFXBs8pZjsGG2iy3OBAno%3D
> > &amp;reserved=0

---

From: Yi Huang <[email-masked]>
Date: Thu, 2 May 2019 18:47:04 +0000

If TLPRxtOut is checked in TLP_send_probe(), a new PTO can cancel an already running RTO timer armed by sending TLP previously. If TLP_send_probe() decides not to send the probe for the new PTO, what should the sender do next? Rearm RTO or treat it as RTO timeout? The draft states we must arm an RTO after transmitting TLP but in this case, no probe will be sent.

Also, in page 17, the draft says "This is important to avoid TLP loops if an application writes periodically at an interval less than PTO." but if the app writes periodically at an interval less than PTO, PTO will just be pushed out and not fire until the sender cannot send anything due to window limit. Then in this case, TLP loops do not seem to exist if TLP loops means back-to-back TLP probes. Am I missing anything here?

Thanks,

Yi

-----Original Message-----
From: Yuchung Cheng <ycheng=[email-masked]> 
Sent: Thursday, May 2, 2019 10:25 AM
To: Matt Olson <[email-masked]>
Cc: Yi Huang <[email-masked]>; [email-masked]
Subject: Re: [tcpm] Questions about TLP

> Also, section 6.5.1 Step 1 is presented as a complete set of conditions for scheduling a PTO, but it is missing a bullet point for TLPRxtOut.
Thanks for checking: but checking TLPRxtOut is not necessary and is not preferred when scheduling a probe -- the goal is to prevent more than one probe inflight, so checking right before TLPRxtOut sending the next one allows the best coverage of application-limited writes (burst-idle-burst-....).

> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww..

---

From: Matt Olson <[email-masked]>
Date: Thu, 2 May 2019 17:09:26 +0000

-----Original Message-----
From: Yuchung Cheng <[email-masked]> 
Sent: Wednesday, May 1, 2019 11:33 PM
To: Yi Huang <huanyi=[email-masked]>
Cc: [email-masked]; Matt Olson <[email-masked]>
Subject: Re: [tcpm] Questions about TLP

This is in

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 6 May 2019 14:28:17 -0700

Sorry for the late reply. Forget to hit send after drafting the reply...

Thanks for spotting that. TCP_send_probe() should always re-arm the
RTO regardless of a TLP is sent or not, if there's unacknowledged
data. Proposed rev in

"Note that the sender MUST arm an RTO timer and not the PTO timer *at the
end of TLP_send_probe() as long as FlightSize is not zero, regardless of
sending a probe or not*. This ensures that the sender does not send
repeated, back-to-back TLP probes. *Also checking TLPRxtOut prior to
sending the loss probe* is important to avoid TLP loops if an application
writes periodically at an interval less than PTO."

*From: *Yi Huang <huanyi=[email-masked]>
*Date: *Thu, May 2, 2019 at 11:47 AM
*To: *Yuchung Cheng, Matt Olson
*Cc: [email-masked]

If TLPRxtOut is checked in TLP_send_probe(), a new PTO can cancel an
> already running RTO timer armed by sending TLP previously. If
> TLP_send_probe() decides not to send the probe for the new PTO, what should
> the sender do next? Rearm RTO or treat it as RTO timeout? The draft states
> we must arm an RTO after transmitting TLP but in this case, no probe will
> be sent.
>
> Also, in page 17, the draft says "This is important to avoid TLP loops if
> an application writes periodically at an interval less than PTO." but if
> the app writes periodically at an interval less than PTO, PTO will just be
> pushed out and not fire until the sender cannot send anything due to window
> limit. Then in this case, TLP loops do not seem to exist if TLP loops means
> back-to-back TLP probes. Am I missing anything here?
>
> Thanks,
>
> Yi
>
> > Also, section 6.5.1 Step 1 is presented as a complete set of conditions
> for scheduling a PTO, but it is missing a bullet point for TLPRxtOut.
> Thanks for checking: but checking TLPRxtOut is not necessary and is not
> preferred when scheduling a probe -- the goal is to prevent more than one
> probe inflight, so checking right before TLPRxtOut sending the next one
> allows the best coverage of application-limited writes
> (burst-idle-burst-....).
>
>
>
>
> >
> > > 1.In page 16, it says “Finally, if the time at which an RTO would fire
> (here denoted "TCP_RTO_expire") is sooner than the computed time for the
> PTO, then a probe is scheduled to be sent at that earlier time.” What does
> this TCP_RTO_expire actually mean? Does it mean there is another RTO timer
> (possibly started some time in the past) running along with PTO or just RTT
> + 4*RTTVar + Now()? Also, why would another probe be sent at RTO expiration
> time instead of treating it like RTO and collapsing the cwnd?
> >
> > > 2.In page 18 section 6.6, a new variable TLPRxtOut is defined and it
> is stated that TLPRxtOut is used to guarantee that there is only one
> outstanding TLP retransmission.. However, it is not clear to me when
> TLPRxtOut should be really used. Should it be used in 6.5.1 Step. 1
> (checking whether we should and can schedule a TLP) or in TLP_send_probe()?
> >
> > This is in
> >

---

From: Yi Huang <[email-masked]>
Date: Mon, 6 May 2019 22:20:41 +0000

Thanks for the clarification. The proposed rev of the statement is much easier to understand.

Yi

From: Yuchung Cheng <ycheng=[email-masked]>
Sent: Monday, May 6, 2019 2:28 PM
To: Yi Huang <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; Matt Olson <[email-masked]>; [email-masked]; Neal Cardwell <[email-masked]>
Subject: Re: [tcpm] Questions about TLP

"Note that the sender MUST arm an RTO timer and not the PTO timer at the end of TLP_send_probe() as long as FlightSize is not zero, regardless of sending a probe or not. This ensures that the sender does not send repeated, back-to-back TLP probes. Also checking TLPRxtOut prior to sending the loss probe is important to avoid TLP loops if an application writes periodically at an interval less than PTO."

From: Yi Huang <huanyi=[email-masked]<mailto:[email-masked]>>
Date: Thu, May 2, 2019 at 11:47 AM
To: Yuchung Cheng, Matt Olson
Cc: [email-masked]<mailto:[email-masked]>
If TLPRxtOut is checked in TLP_send_probe(), a new PTO can cancel an already running RTO timer armed by sending TLP previously. If TLP_send_probe() decides not to send the probe for the new PTO, what should the sender do next? Rearm RTO or treat it as RTO timeout? The draft states we must arm an RTO after transmitting TLP but in this case, no probe will be sent.

Thanks,

Yi

-----Original Message-----
From: Yuchung Cheng <ycheng=[email-masked]<mailto:[email-masked]>>
Sent: Thursday, May 2, 2019 10:25 AM
To: Matt Olson <[email-masked]<mailto:[email-masked]>>
Cc: Yi Huang <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tcpm] Questions about TLP

> From: Yuchung Cheng <[email-masked]<mailto:[email-masked]>>
> To: Yi Huang <huanyi=[email-masked]<mailto:[email-masked]..org>>
> Cc: [email-masked]<mailto:[email-masked]>; Matt Olson <[email-masked]<mailto:[email-masked]>>
> On Wed, May 1, 2019 at 10:48 PM Yi Huang <huanyi=[email-masked]<mailto:[email-masked]>> wrote:
> >
> > Hi RACK/TLP authors,
> >
> >
> >
> > [email-masked]<mailto:[email-masked]>
> > ietf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=01%7C01%7Chuanyi%40microsoft.com%7Ca0763e85f54f443b4c5508d6d269de1e%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=RzcKX7o5C4MDfaouTd1DPvPtHVoArS3NUUP236eId2Y%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=01%7C01%7Cmaolson%40mi
> > cr
> > osoft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fosoft.com&data=01%7C01%7Chuanyi%40microsoft.com%7Ca0763e85f54f443b4c5508d6d269de1e%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=hwiY%2B%2BymtCri2QxlpzfzKUQ1Bj2ck2JqkYTbfpaPpvE%3D&reserved=0>%7C66db44672e3d4101d72908d6cec82001%7C72f988bf86f141af91ab2
> > d7
_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=01%7C01%7Chuanyi%40microsoft.com%7Ca0763e85f54f443b4c5508d6d269de1e%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=CwvkXq3qwbh2LqwCpkAFIJ21Hkl3mproI2wA%2BPHvXus%3D&reserved=0>

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 7 May 2019 09:52:07 -0700

Thanks for the review as well!

*From: *Yi Huang <huanyi=[email-masked]>
*Date: *Thu, May 2, 2019, 11:47 AM
*To: *Yuchung Cheng, Matt Olson
*Cc: [email-masked]

---

From: Yi Huang <[email-masked]>
Date: Thu, 9 May 2019 00:19:37 +0000

Hi Yuchung,

I have one more question regarding to TLP and the proposed rev. So during a post-TLP RTO, if the application writes something before this RTO fires, the sender will still try to schedule PTO, which should cancel the already running RTO. When this new PTO fires, since TLPRxtOut is true due the previous TLP sent, the sender will not send another TLP but rearm RTO timer. Is this true? If so and an app always writes data in such a pattern and no ACKs are coming back due to network failure, RTO will be postponed infinitely until the window limit is reached? My main concern is that we could have detected the network failure very early. It seems like the current algorithm might cause delays in tearing down the connection depending on app send behavior.

A sequence to help illustrate the problem I mentioned and assume no ACKs will be received due to network failure during so.
App writes->arm PTO->PTO fires->TLP->arm RTO->app writes->arm PTO->PTO fires->arm RTO->app writes->arm PTO->PTO fires->arm RTO…

Also, the proposed rev says “Also checking TLPRxtOut prior to sending the loss probe is important to avoid TLP loops if an application writes periodically at an interval less than PTO.”. If an app writes periodically at an interval less than PTO, the sender will just periodically postpone PTO until window limit is reached since each time the app writes data, a new PTO will be scheduled and cancel the existing one. I don’t quite understand how this app behavior is related to repeated back-to-back TLPs problem caused by arming PTO after sending TLP. Btw, “TLP loops” looks like a brand new term used only in this paragraph.

Thanks,

Yi

From: Yuchung Cheng <ycheng=[email-masked]>
Sent: Tuesday, May 7, 2019 9:52 AM
To: Yi Huang <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; Matt Olson <[email-masked]>; [email-masked]
Subject: Re: [tcpm] Questions about TLP

From: Yi Huang <huanyi=[email-masked]<mailto:40microsoft.com@dmarc..ietf.org>>
Date: Thu, May 2, 2019, 11:47 AM
To: Yuchung Cheng, Matt Olson
Cc: [email-masked]<mailto:[email-masked]>
If TLPRxtOut is checked in TLP_send_probe(), a new PTO can cancel an already running RTO timer armed by sending TLP previously. If TLP_send_probe() decides not to send the probe for the new PTO, what should the sender do next? Rearm RTO or treat it as RTO timeout? The draft states we must arm an RTO after transmitting TLP but in this case, no probe will be sent.

Thanks,

Yi

-----Original Message-----
From: Yuchung Cheng <ycheng=[email-masked]<mailto:[email-masked]>>
Sent: Thursday, May 2, 2019 10:25 AM
To: Matt Olson <[email-masked]<mailto:[email-masked]>>
Cc: Yi Huang <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tcpm] Questions about TLP

> > ietf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=y8%2F%2BCOxN%2FjAJiSPuZyIEgFkFZ9l7S%2FojnhmfwDRhk6U%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=01%7C01%7Cmaolson%40mi
> > cr
> > osoft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fosoft.com&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=NtY%2BZy379V4V7C1dlTRyjsmBtvGzDSNeFmcbZpRIT0s%3D&reserved=0>%7C66db44672e3d4101d72908d6cec82001%7C72f988bf86f141af91ab2
> > d7
_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=95VwD1Wy13YO%2FksNzS4A9OjyfPEyuQKdGcKn9goi5sY%3D&reserved=0>

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 8 May 2019 18:28:43 -0700

*From: *Yi Huang <[email-masked]>
*Date: *Wed, May 8, 2019 at 5:19 PM
*To: *Yuchung Cheng
*Cc: *Yuchung Cheng, Matt Olson, [email-masked]

Hi Yuchung,
>
>
>
> I have one more question regarding to TLP and the proposed rev. So during
> a post-TLP RTO, if the application writes something before this RTO fires,
> the sender will still try to schedule PTO, which should cancel the already
> running RTO. When this new PTO fires, since TLPRxtOut is true due the
> previous TLP sent, the sender will not send another TLP but rearm RTO
> timer. Is this true?
>
yes

> If so and an app always writes data in such a pattern and no ACKs are
> coming back due to network failure, RTO will be postponed infinitely until
> the window limit is reached? My main concern is that we could have detected
> the network failure very early. It seems like the current algorithm might
> cause delays in tearing down the connection depending on app send behavior.
>
>
>
> A sequence to help illustrate the problem I mentioned and assume no ACKs
> will be received due to network failure during so.
>
> App writes->arm PTO->PTO fires->TLP->*arm RTO*->app writes->arm PTO->PTO
> fires->*arm RTO*->app writes->arm PTO->PTO fires->*arm RTO*…
>
Good question. In the second "arm PTO" of your example, the last two lines
in TLP_timeout() pseudo code may set PTO timer to expire at the original
RTO (TCP_RTO_expire()). The timer expiration won't be extended forever but
rather set to expire at max(PTO, original_RTO) of the first app writes.

>
> Also, the proposed rev says “*Also checking TLPRxtOut prior to sending
> the loss probe* is important to avoid TLP loops if an application writes
> periodically at an interval less than PTO.”. If an app writes periodically
> at an interval less than PTO, the sender will just periodically postpone
> PTO until window limit is reached since each time the app writes data, a
> new PTO will be scheduled and cancel the existing one. I don’t quite
> understand how this app behavior is related to repeated back-to-back TLPs
> problem caused by arming PTO after sending TLP. Btw, “TLP loops” looks like
> a brand new term used only in this paragraph.
>
>
>
> Thanks,
>
>
>
> Yi
>
>
>
>
>
> *From:* Yuchung Cheng <ycheng=[email-masked]>
> *Sent:* Tuesday, May 7, 2019 9:52 AM
> *To:* Yi Huang <[email-masked]>
> *Cc:* Yuchung Cheng <[email-masked]>; Matt Olson <[email-masked]>;
> [email-masked]
> *Subject:* Re: [tcpm] Questions about TLP
>
>
>
> *From: *Yi Huang <huanyi=[email-masked]
> <40microsoft.com@dmarc..ietf.org>>
> > To: Yi Huang <huanyi=[email-masked]
> <[email-masked]..org>>
> > > ietf.org
> <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=y8%2F%2BCOxN%2FjAJiSPuZyIEgFkFZ9l7S%2FojnhmfwDRhk6U%3D&reserved=0>
> %2Fmailman%2Flistinfo%2Ftcpm&amp;data=01%7C01%7Cmaolson%40mi
> > > cr
> > > osoft.com
> <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fosoft.com&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=NtY%2BZy379V4V7C1dlTRyjsmBtvGzDSNeFmcbZpRIT0s%3D&reserved=0>
> %7C66db44672e3d4101d72908d6cec82001%7C72f988bf86f141af91ab2
> > > d7
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=01%7C01%7Chuanyi%40microsoft.com%7Cbbf1ad4a524c4821031b08d6d30c646f%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=95VwD1Wy13YO%2FksNzS4A9OjyfPEyuQKdGcKn9goi5sY%3D&reserved=0>
>
>