Subject: [tsvwg] tests of delay induced in dualpi2 L queue by videoconferencing traffic

From: Pete Heist <[email-masked]>
Date: Sun, 08 Aug 2021 11:54:13 +0200

Continuing with our results obtained during IETF 111, we ran a test
that placed videoconferencing traffic (Skype) through the L4S dualpi2 L
queue, then measured the delay through L with an irtt measurement flow
(uses UDP), and made a CDF plot:

https://github.com/heistp/l4s-tests/#between-flow-induced-delay

The results suggest that the bursty traffic typical of
videoconferencing applications, which would seem to be good candidates
for placement in L, can induce delays that are far greater than the sub
1ms ultra-low latency goal.

Pete

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 8 Aug 2021 11:10:40 +0000

Hi Pete
Your results are expected given that I believe that Skype does not pace 
traffic. This means that large video frames (I-frames) are bursted out at a 
high rate.
There are ways to mitigate this :
* Implement packet pacing, packet pacing is an integral part of BBRv1 and 
BBRv2, Prague and SCReAM. This does not solve the whole problem though as the 
queue is now moved to the sender queue instead, so the following is needed as 
well:
**  Reduce the size of the I-frame or transmit them less frequently. In the RC 
Car prototype that I have experimented with, the I-frames are quantized harder 
at low bitrates, in addition the interval between I-frames is stretched and 
essentially I-frames are only transmitted when packet loss occurs. Another 
solution is to use periodic intra refresh if that is supported by the video 
encoder.
** Give a larger headroom to accommodate larger video frames. SCReAM, when 
used in L4S mode, inherently pushes down the bitrate to give sufficient 
headroom for large video frames. I doubt that Skype gives this headroom as it 
is not L4S capable AFAIK.

With that said..
1) The results you depict gives something of a worst case with a relatively 
narrow bottleneck and only one flow. The situation would look different if you 
would for instance have 10 flows, of them one being the skype flow, over a 
100Mbps bottleneck.
2) It is possible in a 5G context, to devise L4S capable bearers that can 
absorb larger bursts of traffic without large queue buildup, for example with 
the use of delay based schedulers as exemplified in the work by Davide 
Brunello (see https://ieeexplore.ieee.org/abstract/document/9448764 ).

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Pete Heist
> Sent: den 8 augusti 2021 11:54
> To: tsvwg IETF list <[email-masked]>
> Subject: [tsvwg] tests of delay induced in dualpi2 L queue by
> videoconferencing traffic
>
> Continuing with our results obtained during IETF 111, we ran a test that
> placed videoconferencing traffic (Skype) through the L4S dualpi2 L queue,
> then measured the delay through L with an irtt measurement flow (uses
> UDP), and made a CDF plot:
>
> https://protect2.fireeye.com/v1/url?k=b34e0fd2-ecd536d7-b34e4f49-
> 86d2114eab2f-c6e9a3fcca8935cf&q=1&e=17dccacc-9757-460a-8171-
> 7be10a55b8f6&u=https%3A%2F%2Fgithub.com%2Fheistp%2Fl4s-
> tests%2F%23between-flow-induced-delay
>
> The results suggest that the bursty traffic typical of videoconferencing
> applications, which would seem to be good candidates for placement in L,
> can induce delays that are far greater than the sub 1ms ultra-low latency
> goal.
>
> Pete
>

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 8 Aug 2021 16:40:52 +0300

> On 8 Aug, 2021, at 2:10 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

Nevertheless, I believe Skype video-call traffic would qualify for NQB treatment, and could therefore legitimately end up in the L queue by that means.  It never exceeds a reasonably low send rate on average (about 2.5 Mbps), and adapts the encoding rate to keep the induced latency below about one frame time, thereby keeping the queue empty on average.  Existing technology copes splendidly with this traffic without needing special treatment, as seen on the same plot where fq_codel traces are also present.

> solution is to use periodic intra refresh if that is supported by the video 

The above measures would add significant complexity to the application, as workarounds for what appears to be a shortcoming in the network-side technology.

> is not L4S capable AFAIK.

The tightest bottleneck here gives double the capacity actually required on average.  Two other traces on the same plot show what happens when four and eight times the necessary capacity, respectively, is provided.  Even with 8x capacity, the 99.9th percentile shows 11ms induced delay, which corresponds to about 30KB of queued data (less than some individual TSO bursts).  I will remind you that fq_codel, shown for the same traffic and link capacities on the same plot, handles this scenario much more gracefully.

This is not unexpected, and actually doesn't affect application performance, as anything under about 30ms is less than one video frame, and is thus nearly imperceptible to the Skype users.  Our concern is on the effects on other traffic going through the same bottleneck, which Pete is presently preparing a write-up of.

> would for instance have 10 flows, of them one being the skype flow, over a 

I'm sure it *would* look different when you provide vastly more capacity, and a lot of other traffic to bully the interloper out of the way.  Unfortunately, you can't argue away a bad result by pointing to a better one obtained under more favourable conditions.

> Brunello (see https://ieeexplore.ieee.org/abstract/document/9448764 ).

"Possible to devise" doesn't cut it here.  Show us a working specification and running code.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 8 Aug 2021 14:57:59 +0100

Thanks Pete, 

It’s interesting to see results that look at latency.

> On 8 Aug 2021, at 12:11, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi Pete
The need for burst mitigation is certainly something that we discussed in QUIC also,

I think Burst mitigation - e.g. pacing, is something that will indeed be quite .important in reducing latency, and this is certainly something that I think we should advocate in any transport going forward.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 08 Aug 2021 16:18:18 +0200

Gorry,

While I agree that striving for less bursty traffic is a worthy goal, taking that as rationale to deploy exquisitely burst intolerant network technology like current L4S seems pushing away from Postel's principle, no? 
Also this seems like this increases the attac surface of L4S, as it seems rather trivial and hard to detect/recognize as adversarial to inject bursty traffic into an end user's L4S'd access link. So what do we actually gain from these new burst sensitivity L4S' network behavior introduces?

Obviously not robust and reliable low latency performance nor link utilisation, because both seem to suffer under bursty traffic conditions. And yet we are in the WGLC process for L4S.

Regards
       Sebastian

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Pete Heist <[email-masked]>
Date: Sun, 08 Aug 2021 19:13:08 +0200

Hi Ingemar,

Thanks for the reply. Testing Skype traffic in L may be kind of a
conversation starter on this topic (Jon's remark about NQB aside, as
then it would be more than that), but it does give some idea of where
existing videoconferencing applications are in terms of bursts, i.e. a
ways off from sub 1ms, depending on bandwidth. It was also a lead-in to
the utilization test result, posted in a separate thread.

>From my perspective, I'm skeptical that we'll reliably see the ultra-
low L queue delays in the real world that we can sometimes see in the
lab, between the bursts added by the application, by the network, and
by mis-marked traffic.

Pete

On Sun, 2021-08-08 at 11:10 +0000, Ingemar Johansson S wrote:
> Brunello (see
> https://ieeexplore.ieee.org/abstract/document/9448764 ).
> 
> > that
> > videoconferencing

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 08 Aug 2021 19:38:06 +0200

Hi Pete,

I seem to recall that I read that tgere is a positivre correlation Between path length and RTT variance, which I think might indicate that bursts are more likely to occur the linger a path though the real internet is. This puts relevance behind my long standing request to show data demonstrating L4S delivering on its promises over longer RTT real internet paths. And preferably before mass deployment so that the results if those measurements/experiments can be potentially  addressed in L4S' design and implementation. Which, as far as I can tell is just plain common sense engineering practise, what am I missing?

Regards
        Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 9 Aug 2021 09:00:03 +0000

Hi Jonathan
Please see inline [IJ]

> From: Jonathan Morton <[email-masked]>
> Sent: den 8 augusti 2021 15:41
> To: Ingemar Johansson S <[email-masked]>
> Cc: Pete Heist <[email-masked]>; tsvwg IETF list <[email-masked]>
> Subject: Re: [tsvwg] tests of delay induced in dualpi2 L queue by
> > On 8 Aug, 2021, at 2:10 pm, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > Your results are expected given that I believe that Skype does not
> > pace traffic. This means that large video frames (I-frames) are
> > bursted out at a high rate.
> 
> Nevertheless, I believe Skype video-call traffic would qualify for NQB
> treatment, and could therefore legitimately end up in the L queue by that
> means.  It never exceeds a reasonably low send rate on average (about 2.5
> Mbps), and adapts the encoding rate to keep the induced latency below
> about one frame time, thereby keeping the queue empty on average.
> Existing technology copes splendidly with this traffic without needing
special
> treatment, as seen on the same plot where fq_codel traces are also
present.

[IJ] Empty (or low) queue on average does not help much as these queue delay
spikes means that a de-jitterbuffer on the receiving side need to be
enlarged to avoid choppy playout. As regards to the difference between the
dualPi2 and fq_codel, isn't the difference here that fq_codel treats new
flows differently ?. As skype video frequently gives an empty queue that
becomes prioritized. 

> > * Implement packet pacing, packet pacing is an integral part of BBRv1
> > and BBRv2, Prague and SCReAM. This does not solve the whole problem
> > though as the queue is now moved to the sender queue instead, so the
> > following is needed as
> > well:
> > **  Reduce the size of the I-frame or transmit them less frequently.
> > In the RC Car prototype that I have experimented with, the I-frames
> > are quantized harder at low bitrates, in addition the interval between
> > I-frames is stretched and essentially I-frames are only transmitted
> > when packet loss occurs. Another solution is to use periodic intra
> > refresh if that is supported by the video encoder.
> 
> The above measures would add significant complexity to the application, as
> workarounds for what appears to be a shortcoming in the network-side
> technology.
[IJ] What makes you believe that. Adjustment of qp_min and qp_max that is
supported by nvv4l2h264enc (NVIDIA Jetson Nano) . Periodic Intra Refresh is
not widely supported yet (I believe) but I believe that it more because the
market has not asked for it,

> 
> > ** Give a larger headroom to accommodate larger video frames. SCReAM,
> > when used in L4S mode, inherently pushes down the bitrate to give
> > sufficient headroom for large video frames. I doubt that Skype gives
> > this headroom as it is not L4S capable AFAIK.
> 
> The tightest bottleneck here gives double the capacity actually required
on
> average.  Two other traces on the same plot show what happens when four
> and eight times the necessary capacity, respectively, is provided.  Even
with
> 8x capacity, the 99.9th percentile shows 11ms induced delay, which
> corresponds to about 30KB of queued data (less than some individual TSO
> bursts).  I will remind you that fq_codel, shown for the same traffic and
link
> capacities on the same plot, handles this scenario much more gracefully.
> 
> This is not unexpected, and actually doesn't affect application
performance,
> as anything under about 30ms is less than one video frame, and is thus
nearly
> imperceptible to the Skype users.  Our concern is on the effects on other
> traffic going through the same bottleneck, which Pete is presently
preparing
> a write-up of.
> 
> > With that said..
> > 1) The results you depict gives something of a worst case with a
> > relatively narrow bottleneck and only one flow. The situation would
> > look different if you would for instance have 10 flows, of them one
> > being the skype flow, over a 100Mbps bottleneck.
> 
> I'm sure it *would* look different when you provide vastly more capacity,
> and a lot of other traffic to bully the interloper out of the way.
> Unfortunately, you can't argue away a bad result by pointing to a better
one
> obtained under more favourable conditions.
> 
> > 2) It is possible in a 5G context, to devise L4S capable bearers that
> > can absorb larger bursts of traffic without large queue buildup, for
> > example with the use of delay based schedulers as exemplified in the
> > work by Davide Brunello (see
> "Possible to devise" doesn't cut it here.  Show us a working specification
and
> running code.
[IJ] Running code : SCReAM for instance (turn off packet pacing). But for
quite obvious reasons I wont disclose radio scheduler code.

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 9 Aug 2021 09:18:43 +0000

Hi Pete
Please see inline

/Ingemar

> Sent: den 8 augusti 2021 19:13
> conversation
> starter on this topic (Jon's remark about NQB aside, as then it would be 
> more
> than that), but it does give some idea of where existing videoconferencing
> applications are in terms of bursts, i.e. a ways off from sub 1ms, depending
> on bandwidth. It was also a lead-in to the utilization test result, posted 
> in a
> separate thread.
>
> low L queue delays in the real world that we can sometimes see in the lab,
> between the bursts added by the application, by the network, and by mis-
> marked traffic.

[IJ] It is important to address all sources of excessive queue latency to make 
L4S deliver. For video coders the large "peak to average ratio" is a concern. 
I believe that your Skype experiment was good was it brings this concern on 
the table but I don't believe that this is how it is implemented 5 years from 
now, especially of the application is to be L4S capable.

> > In the RC
> > Car prototype that I have experimented with, the I-frames are
> > quantized harder at low bitrates, in addition the interval between
> > when packet loss occurs.
> > Another
> > solution is to use periodic intra refresh if that is supported by the
> > video encoder.
> > relatively
> > can