Subject: [tcpm] Hystart and delay jitter

From: Christian Huitema <[email-masked]>
Date: Fri, 12 Mar 2021 13:28:55 -0800

Back in November 2019, when adding Cubic and Hystart to my 
implementation of QUIC, I noticed that Hystart was sensitive to delay 
jitter. Hystart detects the buildup of queues by monitoring the RTT. 
Some links experience delay jitter, caused for example by access 
protocol for shared radio links or possibly by link-local ARQ protocols. 
The delay jitter can cause Hystart to make the wrong decision, in two ways:

1) Delay jitter during a previous period could cause some packets to be 
delivered "faster than usual", causing Hystart to under-estimate the min 
RTT for that period.

2) Delay jitter during the currently measured period can cause packets 
to be delivered "slower than usual",  causing Hystart to over-estimate 
the min RTT for that period.

The combination of these two issues may cause Hystart to make the wrong 
decisions, and exit slow start at levels well below link capacity.

The draft-ietf-tcpm-hystartplusplus-01 does have some protection against 
the second issue, because currentRoundMinRTT is computed on at least 
N_RTT_SAMPLE. If that number is large enough, computing the min over N 
samples should filter out "slower than usual" anomalies. However, the 
draft does not include a protection against "faster than usual" 
anomalies happening in the previous period. In my implementation, I 
protected against that by computing a "min of max" function: compute a 
rolling "MAX over N_RTT_SAMPLE", then compute the MIN value of that 
during the reference period, and use that to set the reference value 
"lastRoundMinRTT".

I think it would be good to add a discussion of the effect of jitter to 
the hystart++ draft. In addition, we may also want to mention 
timestamps. The jitter on RTT may be caused by jitter on either 
direction of transmission -- data path or ACK path. The effect of jitter 
on the ACK path can be minimized if time stamps can be used to monitor 
the variation of one-way delays. This is not discussed in the current 
draft. Maybe it should be.

-- Christian Huitema

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 12 Mar 2021 17:18:10 -0500

> decisions, and exit slow start at levels well below link capacity.

Yes, we have found in both our production experience and controlled
experiments that the Hystart-Delay algorithm is very susceptible to
spurious triggering from jitter, particularly in LTE and wifi paths.

We discussed this a bit in Spring 2017 in the comparison of BBR's
bandwidth-based mechanism for exiting startup, vs Hystart-Delay's
delay-based mechanism:
  https://www.ietf.org/proceedings/98/slides/slides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf#page=8

> draft. Maybe it should be.

A discussion of the effect of jitter sounds like a great idea.

neal

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 18 Mar 2021 22:37:17 +0000

Thanks Christian for the suggestions! We will include a discussion on jitter in the next draft update and also experiment with the min of max function. 

-----Original Message-----
From: tcpm <[email-masked]> On Behalf Of Neal Cardwell
Sent: Friday, March 12, 2021 2:18 PM
To: Christian Huitema <[email-masked]>
Cc: [email-masked] Extensions <[email-masked]>
Subject: [EXTERNAL] Re: [tcpm] Hystart and delay jitter

On Fri, Mar 12, 2021 at 4:29 PM Christian Huitema <[email-masked]> wrote:
> 1) Delay jitter during a previous period could cause some packets to 
> be delivered "faster than usual", causing Hystart to under-estimate 
> The combination of these two issues may cause Hystart to make the 
> wrong decisions, and exit slow start at levels well below link capacity.

Yes, we have found in both our production experience and controlled experiments that the Hystart-Delay algorithm is very susceptible to spurious triggering from jitter, particularly in LTE and wifi paths.

We discussed this a bit in Spring 2017 in the comparison of BBR's bandwidth-based mechanism for exiting startup, vs Hystart-Delay's delay-based mechanism:
  https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fproceedings%2F98%2Fslides%2Fslides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf%23page%3D8&amp;data=04%7C01%7Cpravb%40microsoft.com%7Cbc3399870c0d4105299408d8e5a4cbaa%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637511843745826011%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=6SQWDb1mFH0cgrayEwiRYeccYvP0dv%2BAJ1eokyJF%2BKs%3D&amp;reserved=0

> The draft-ietf-tcpm-hystartplusplus-01 does have some protection 
> against the second issue, because currentRoundMinRTT is computed on at 
> least N_RTT_SAMPLE. If that number is large enough, computing the min 
> over N samples should filter out "slower than usual" anomalies. 
> However, the draft does not include a protection against "faster than usual"
> I think it would be good to add a discussion of the effect of jitter 
> to the hystart++ draft. In addition, we may also want to mention 
> direction of transmission -- data path or ACK path. The effect of 
> jitter on the ACK path can be minimized if time stamps can be used to 
> monitor the variation of one-way delays. This is not discussed in the 
> current draft. Maybe it should be.

neal

_______________________________________________
tcpm mailing list
[email-masked]
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=04%7C01%7Cpravb%40microsoft.com%7Cbc3399870c0d4105299408d8e5a4cbaa%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637511843745826011%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oqte6wCaTBUvrwgXVRMcD13YgtspMJyIWMMLlwNuYdY%3D&amp;reserved=0

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 19 Mar 2021 09:16:13 +0000

Hi

My finding is that it is a bit problematic to rely on RTT estimates for path
capacity especially with cellular access . It was summarized in section 4 in
https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-02 and also
presented in
https://www.ietf.org/proceedings/96/slides/slides-96-iccrg-1.pdf

Regards
Ingemar

> From: tcpm <[email-masked]> On Behalf Of Praveen
> Balasubramanian
> Sent: den 18 mars 2021 23:37
> To: ncardwell=[email-masked]; Christian Huitema
> <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [tcpm] [EXTERNAL] Re: Hystart and delay jitter
> 
> Thanks Christian for the suggestions! We will include a discussion on
jitter in
> the next draft update and also experiment with the min of max function.
> 
spurious
> triggering from jitter, particularly in LTE and wifi paths.
> 
bandwidth-
> based mechanism for exiting startup, vs Hystart-Delay's delay-based
> mechanism:
>   https://protect2.fireeye.com/v1/url?k=6c35f204-33aecb47-6c35b29f-
> 861fcb972bfc-c8f94340e007b7bd&q=1&e=fd13218f-2e70-4b39-a879-
> e9b53c01447d&u=https%3A%2F%2Fnam06.safelinks.protection.outlook.com
> %2F%3Furl%3Dhttps%253A%252F%252Fwww.ietf.org%252Fproceedings%25
> 2F98%252Fslides%252Fslides-98-iccrg-an-update-on-bbr-congestion-control-
> 00.pdf%2523page%253D8%26amp%3Bdata%3D04%257C01%257Cpravb%2540
> microsoft.com%257Cbc3399870c0d4105299408d8e5a4cbaa%257C72f988bf86f
> 141af91ab2d7cd011db47%257C1%257C0%257C637511843745826011%257CUn
> known%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT
> iI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3D6SQWDb1
> mFH0cgrayEwiRYeccYvP0dv%252BAJ1eokyJF%252BKs%253D%26amp%3Brese
> rved%3D0
> 
> https://protect2.fireeye.com/v1/url?k=032901f3-5cb238b0-03294168-
> 861fcb972bfc-7a6ccb41bd90a846&q=1&e=fd13218f-2e70-4b39-a879-
> e9b53c01447d&u=https%3A%2F%2Fnam06.safelinks.protection.outlook.com
> %2F%3Furl%3Dhttps%253A%252F%252Fwww.ietf.org%252Fmailman%252Flis
> tinfo%252Ftcpm%26amp%3Bdata%3D04%257C01%257Cpravb%2540microsof
> t.com%257Cbc3399870c0d4105299408d8e5a4cbaa%257C72f988bf86f141af91a
> b2d7cd011db47%257C1%257C0%257C637511843745826011%257CUnknown%
> 257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3Doqte6wCaTBUvrw
> gXVRMcD13YgtspMJyIWMMLlwNuYdY%253D%26amp%3Breserved%3D0
> 
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 22 Mar 2021 15:12:15 +0000

Yes, we also reported on some problems/bugs in HyStart in maprg two meetings back:

https://datatracker.ietf.org/meeting/interim-2020-maprg-01/materials/slides-interim-2020-maprg-01-sessa-behavior-of-tcp-cubic-in-low-latency-mobile-radio-networks-00

This problem, where Slow Start is left wrongly and much too early, is particularly bad in combination with the TCP friendly region in Cubic. However, the actually problems lies in HyStart which is not covered in the Cubic RFC/bis-draft.

On 19.03.21, 10:16, "tcpm on behalf of Ingemar Johansson S" <[email-masked] on behalf of ingemar.s.johansson=[email-masked]> wrote:

    Hi

    Regards
    Ingemar

    jitter in
    spurious
    bandwidth-

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 22 Mar 2021 15:17:05 +0000

Sorry, I was confused and thought this was a comment on the Cubic work. Would be really good to mention and discuss this in HyStart++ draft. Happy to discuss more if needed!

On 22.03.21, 16:12, "tcpm on behalf of Mirja Kuehlewind" <[email-masked] on behalf of mirja.kuehlewind=[email-masked]> wrote:

        Hi

        Regards
        Ingemar

        jitter in
        spurious
        bandwidth-

---

From: Christian Huitema <[email-masked]>
Date: Mon, 22 Mar 2021 09:18:56 -0700

Back to Ingemar's point about delay measurement. In wireless networks, 
we should expect conditions to change when the device moves from cell to 
cell, when the number of devices in the cell changes, or even when the 
client stays in the same cell and moves closer or farther to the base 
station. In Wi-Fi network, I remember presentations showing that the 
quality of the radio signal could change radically if the device just 
moved a couple feet, maybe because it would then catch different 
reflections on walls of the signals sent by the base station. Conditions 
could also change if the device is rotated and the alignment of the 
device's antenna with the base station changes. Or if a big two-legged 
of water moves between device and base station, whether it is a big 
two-legged body or a smaller four-legged body with whiskers.

The congestion control algorithms that we currently use are not 
specifically designed for such changes in network conditions. They make 
hypotheses about min-rtt, max queue sizes, relation between rtt and 
congestion, or relative stability of the bottleneck bandwidth. As 
Ingemar points out, these hypotheses are not true in wireless networks.

If I look at simulations in classic papers, I almost always see networks 
made of static links, with fixed bandwidth and fixed delays. Such 
simulations are not going to point out the issues with wireless 
networks. Are there available link models that incorporate the typical 
behaviors of wireless networks? Better simulations would make the 
problems obvious, and researchers would probably invent some clever way 
of dealing with that. And if such simulation models were widely known, 
peer-reviewers would have something to say about papers based on fixed 
link simulations.

Ingemar, Mirja, can you point us to such to simulation models for 
wireless segments?

-- Christian Huitema

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 25 Mar 2021 07:40:26 +0000

Hi Christian, Mirja, Praveen, Neil + others

We use an a proprietary 5G simulator in our work that simulates the protocol stack down to the physical layer. In addition we of course have the real gear set up in a number of labs where we can try out different combinations of UE (phones) and base stations.

I am not sure about the capabilities of other open source simulators like NS3, NS3 appears to have an LTE module at least. I don't know how well it models reality however.

More inline 

/Ingemar

> From: Christian Huitema <[email-masked]>
> Sent: den 22 mars 2021 17:19
> To: Mirja Kuehlewind <[email-masked]>; Ingemar
> Johansson S <[email-masked]>; Praveen
> Balasubramanian <[email-masked]>;
> ncardwell=[email-masked] <[email-masked]>
> Cc: [email-masked]
> Back to Ingemar's point about delay measurement. In wireless networks, we
> should expect conditions to change when the device moves from cell to cell,
> when the number of devices in the cell changes, or even when the client
> stays in the same cell and moves closer or farther to the base station. In Wi-Fi
> network, I remember presentations showing that the quality of the radio
> signal could change radically if the device just moved a couple feet, maybe
> because it would then catch different reflections on walls of the signals sent
> by the base station. Conditions could also change if the device is rotated and
> the alignment of the device's antenna with the base station changes. Or if a
> big two-legged of water moves between device and base station, whether it
> is a big two-legged body or a smaller four-legged body with whiskers.
> 
> The congestion control algorithms that we currently use are not specifically
> designed for such changes in network conditions. They make hypotheses
> about min-rtt, max queue sizes, relation between rtt and congestion, or
> relative stability of the bottleneck bandwidth. As Ingemar points out, these
> hypotheses are not true in wireless networks.
[IJ] Agree, one easy to do mistake is to do a first 3-WHS ping measurement and try to assess path characteristics. DRX (a power saving feature in cell phones) can inflate this initial RTT. And it can well look completely different, seen from the server or client side. DRX can also kick in occasionally if traffic is sparse, which may make RTT spike even though it is not actually congestion. The latter is one good argument for ECN/L4S.

> 
> If I look at simulations in classic papers, I almost always see networks made of
> static links, with fixed bandwidth and fixed delays. Such simulations are not
> going to point out the issues with wireless networks. Are there available link
> models that incorporate the typical behaviors of wireless networks? Better
> simulations would make the problems obvious, and researchers would
> probably invent some clever way of dealing with that. And if such simulation
> models were widely known, peer-reviewers would have something to say
> about papers based on fixed link simulations.
> 
> Ingemar, Mirja, can you point us to such to simulation models for wireless
> segments?
> 
> -- Christian Huitema
> 
> > Sorry, I was confused and thought this was a comment on the Cubic work.
> Would be really good to mention and discuss this in HyStart++ draft. Happy to
> discuss more if needed!
> >
> >
> > On 22.03.21, 16:12, "tcpm on behalf of Mirja Kuehlewind" <tcpm-
> [email-masked] on behalf of
> mirja.kuehlewind=[email-masked]> wrote:
> >
> > https://datatracker.ietf.org/meeting/interim-2020-maprg-01/materials/s
> > lides-interim-2020-maprg-01-sessa-behavior-of-tcp-cubic-in-low-latency
> > -mobile-radio-networks-00
> >
> >      This problem, where Slow Start is left wrongly and much too early, is
> particularly bad in combination with the TCP friendly region in Cubic.
> However, the actually problems lies in HyStart which is not covered in the
> Cubic RFC/bis-draft.
> >
> >
> >      On 19.03.21, 10:16, "tcpm on behalf of Ingemar Johansson S" <tcpm-
> [email-masked] on behalf of
> ingemar.s.johansson=[email-masked]> wrote:
> >
> >          Hi
> >
> >          > On Fri, Mar 12, 2021 at 4:29 PM Christian Huitema
> <[email-masked]>
> >          > wrote:
> >          > >
> >          > > The delay jitter can cause Hystart to make the wrong decision, in
> two
> >          ways:
> >          > >
> >          > > However, the draft does not include a protection against "faster
> than
> >          > usual"