Subject: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Mar 2020 08:02:52 +0000

Hi

I recently updated the readme on the SCReAM github with a comparison with
SCReAM in three different settings

1.	No ECN
2.	CoDel ECN
3.	L4S

https://github.com/EricssonResearch/scream#ecn-explicit-congestion-notificat
ion

Even though it is more than a magnitude difference in queue delay between
CoDel-ECN and L4S, it is fair to say that these simple simulations should of
course be seen as just a snapshot. 

We hope to present some more simulations with 5G access, and not just simple
bottlenecks with one flow, after the summer. 

Meanwhile, the SCReAM code on github is freely available for anyone who wish
to make more experiments.

/Ingemar

================================

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

RESEARCHER

GFTL ER NAP NCM Netw Proto & E2E Perf

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked]

www.ericsson.com

  Reality, is the only thing… That’s real!
      James Halliday, Ready Player One

=================================

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 10 Mar 2020 11:43:23 +0200

> On 10 Mar, 2020, at 10:02 am, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> I recently updated the readme on the SCReAM github with a comparison with SCReAM in three different settings
> 	• No ECN
> 	• CoDel ECN
> 	• L4S
> https://github.com/EricssonResearch/scream#ecn-explicit-congestion-notification
>  
> Even though it is more than a magnitude difference in queue delay between CoDel-ECN and L4S, it is fair to say that these simple simulations should of course be seen as just a snapshot. 
> We hope to present some more simulations with 5G access, and not just simple bottlenecks with one flow, after the summer. 

Looking at the throughput numbers, I also notice the L4S case has less than half the throughput of the Codel ECN case.  That seems like a significant tradeoff for the lower delay numbers.  In fact, even though the throughput *before* the capacity reduction is less than the capacity *after* it, the throughput is still halved when the capacity reduces.

This seems like something you might still want to look into.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Mar 2020 10:45:28 +0100

Hi Ingemar,

thanks for posting this interesting piece of data!

> On Mar 10, 2020, at 09:02, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi
>  
> Even though it is more than a magnitude difference in queue delay between CoDel-ECN and L4S,

	[SM] So, in this simulations of a 20ms path, SCReAM over L4S gives ~10 times less queueing delay, but also only ~2 less bandwidth compared to SCReAM over codel. You describe this as "L4S reduces the delay considerably more" and "L4S gives a somewhat lower media rate". I wonder how many end-users would tradeoff these 25ms in queueing delay against the decrease in video quality from halving the bitrate?
Could you repeat the Codel test with interval set to 20 and target to 1ms, please? 

If that improves things considerably it would argue for embedding the current best RTT estimate into SCReAM packets, so an AQM could tailor its signaling better to individual flow properties (and yes, that will require a flow-aware AQM).

> it is fair to say that these simple simulations should of course be seen as just a snapshot. 

	[SM] Fair enough.

> We hope to present some more simulations with 5G access, and not just simple bottlenecks with one flow, after the summer. 

	[Looking] forward to that.

>  
> Meanwhile, the SCReAM code on github is freely available for anyone who wish to make more experiments.
>  
> /Ingemar

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Mar 2020 09:59:05 +0000

Hi

The throughput reduction with  L4S is a natural consequence of the irregular frame size from the video encoder. 
If one look at the congestion window size and inspect the code to see how the pacing rate is computed, one can see that the pacing rate is ~70Mbps  during the first 50s *. This means that SCreAM with L4S adjust to be able to transmit the large frames without causing large queue buildup in the network (or in the RTP queue). This means that the average rate goes down to give the necessary headroom. This is exactly the behavior that I seek for.

/Ingemar

* CWND = 180kByte, RTT = 20ms --> 180000*8/0.02 = 72Mbps

> -----Original Message-----
> From: Jonathan Morton <[email-masked]>
> Sent: den 10 mars 2020 10:43
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>
> Cc: [email-masked]; Ingemar Johansson S
> <[email-masked]>; [email-masked]
> Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S
> 
> > On 10 Mar, 2020, at 10:02 am, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > We hope to present some more simulations with 5G access, and not just
> simple bottlenecks with one flow, after the summer.
> 
> Looking at the throughput numbers, I also notice the L4S case has less than half
> the throughput of the Codel ECN case.  That seems like a significant tradeoff for
> the lower delay numbers.  In fact, even though the throughput *before* the
> capacity reduction is less than the capacity *after* it, the throughput is still
> halved when the capacity reduces.
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Mar 2020 10:07:36 +0000

Hi

For the future studies we will only focus on L4S as the scope is to study the performance gain that L4S give for instance for AR/VR, gaming and remote control applications.
Flow aware AQMs with RTT estimates as metadata in the packets is outside the scope as it would require packet inspection, which is not feasible if queues build up on the RLC layer in the 3GPP stack.

/Ingemar

> From: Sebastian Moeller <[email-masked]>
> Sent: den 10 mars 2020 10:45
> To: Ingemar Johansson S
> > On Mar 10, 2020, at 09:02, Ingemar Johansson S
> > https://protect2.fireeye.com/v1/url?k=63019d27-3f884737-6301ddbc-0cc47
> > ad93e2a-489fa99c3277fb8a&q=1&e=5aab95a7-4aab-4a64-99a5-
> 5b55606e303b&u=
> > https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream%23ecn-explicit-
> co
> > ngestion-notification
> >
> 	[SM] So, in this simulations of a 20ms path, SCReAM over L4S gives ~10
> times less queueing delay, but also only ~2 less bandwidth compared to SCReAM
> over codel. You describe this as "L4S reduces the delay considerably more" and
> "L4S gives a somewhat lower media rate". I wonder how many end-users would
> tradeoff these 25ms in queueing delay against the decrease in video quality from
> halving the bitrate?
> If that improves things considerably it would argue for embedding the current
> best RTT estimate into SCReAM packets, so an AQM could tailor its signaling
> better to individual flow properties (and yes, that will require a flow-aware
> AQM).
> 
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Mar 2020 11:27:57 +0100

Hi Ingemar,

> On Mar 10, 2020, at 11:07, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] How are you going to "study the performance gain that L4S give[s]" if you do not compare it with the best of class alternatives? I am truly puzzled.

> Flow aware AQMs with RTT estimates as metadata in the packets is outside the scope as it would require packet inspection, which is not feasible if queues build up on the RLC layer in the 3GPP stack.

	[SM] Fair enough. What is this comparison intended to show us then?

As far as I can see you paired an application designed for 1/p-type congestion feed-back with an 1/sqrt(p)-type AQM that was also set for sub-optimal RTT and latency target for the test path. And lo and behold, the application does "better*" for the 1/p-type AQM (with lower latency target; I assume that  L4S ramp-marker (Th_low=2ms, Th_high=10ms) was carefully selected to match what SCReAM expects). IMHO that simply demonstrates, that in communication it pays if sender and receiver of a symbol (CE here) assign the same meaning to it.

But that can not be it, sohat am I missing here?

Best Regards
	Sebastian

*) Assuming one buys into your definition of better, in which (instantaneous) queueing delay is valued over video quality. From a network operators perspective that seems a valid position

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Mar 2020 12:37:36 +0000

Hi

The SCReAM code is freely available on https://github.com/EricssonResearch/scream for anybody interested to run their own experiment with whatever AQM/ECN configuration. 
Please note that SCReAM is configured in an L4S mode when the network AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN however, SCReAM runs in normal ECN mode with a beta of 0.8 (=20% reduction on CWND for each congestion event)

I tried also with other different ramp markers (1ms/10ms), (2ms/10ms),(5ms/15ms). There are slight variations  in throughput and latency but not dramatic.
And truth to be told, the ECN behavior is better tuned in the code than the L4S behavior. 
There is room for improvement as regards to the L4S behavior (for instance faster ramp-up) and it may well be the case that SCReAM is completely scrapped in favor of new designs. 
But the bottomline, the L4S thresholds and L4S code is not carefully picked to show a good performance.

/Ingemar

> Sent: den 10 mars 2020 11:28
> To: Ingemar Johansson S <[email-masked]>
> Cc: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; [email-masked];
> [email-masked]
> > On Mar 10, 2020, at 11:07, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Hi
> >
> > For the future studies we will only focus on L4S as the scope is to study the
> performance gain that L4S give for instance for AR/VR, gaming and remote
> control applications.
> 
> 	[SM] How are you going to "study the performance gain that L4S
> give[s]" if you do not compare it with the best of class alternatives? I am truly
> puzzled.
> 
> > Flow aware AQMs with RTT estimates as metadata in the packets is outside
> the scope as it would require packet inspection, which is not feasible if queues
> build up on the RLC layer in the 3GPP stack.
> 
> As far as I can see you paired an application designed for 1/p-type congestion
> feed-back with an 1/sqrt(p)-type AQM that was also set for sub-optimal RTT and
> latency target for the test path. And lo and behold, the application does
> "better*" for the 1/p-type AQM (with lower latency target; I assume that  L4S
> ramp-marker (Th_low=2ms, Th_high=10ms) was carefully selected to match
> what SCReAM expects). IMHO that simply demonstrates, that in communication
> it pays if sender and receiver of a symbol (CE here) assign the same meaning to
> it.
> 
> *) Assuming one buys into your definition of better, in which (instantaneous)
> queueing delay is valued over video quality. From a network operators
> perspective that seems a valid position
> 
> 
> 
> 
> >
> > /Ingemar
> >
> >> 	[SM] So, in this simulations of a 20ms path, SCReAM over L4S gives
> >> ~10 times less queueing delay, but also only ~2 less bandwidth
> >> compared to SCReAM over codel. You describe this as "L4S reduces the
> >> delay considerably more" and "L4S gives a somewhat lower media rate".
> >> I wonder how many end-users would tradeoff these 25ms in queueing
> >> delay against the decrease in video quality from halving the bitrate?
> >> If that improves things considerably it would argue for embedding the
> >> current best RTT estimate into SCReAM packets, so an AQM could tailor
> >> its signaling better to individual flow properties (and yes, that
> >> will require a flow-aware AQM).
> >>
> >>
> >>
> >>> it is fair to say that these simple simulations should of course be
> >>> seen as just a
> >> snapshot.
> >>
> >> 	[SM] Fair enough.
> >>
> >>> Meanwhile, the SCReAM code on github is freely available for anyone
> >>> who
> >> wish to make more experiments.
> >>>
> >>> /Ingemar
> >>> GFTL ER NAP NCM Netw Proto & E2E Perf Labratoriegränd 11
> >>> 971 28, Luleå, Sweden
> >>> Phone +46-1071 43042
> >>> SMS/MMS +46-73 078 3289

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Mar 2020 16:29:24 +0100

Hi Ingemar,

> On Mar 10, 2020, at 13:37, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] Well, if you are not interested in figuring out what is happening with with rfc3168 ECN ans SCReAM that is your prerogative... 

> Please note that SCReAM is configured in an L4S mode when the network AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN however, SCReAM runs in normal ECN mode with a beta of 0.8 (=20% reducti…

	[SM] Okay, that is a far cry from Reno's 50% though, have you tested different reduction percentages or how did you arrive at 20%?
But is it really 0.8? ScreamTx.h has the following:

// CWND scale factor upon ECN-CE event
static const float kEcnCeBeta = 0.9f;

I am probably missing something somewhere...

> 
> I tried also with other different ramp markers (1ms/10ms), (2ms/10ms),(5ms/15ms).

	[SM] That means for L4S, or do you mean you tried different values for Codel's target ans ce_threshold parameters?

> There are slight variations  in throughput and latency but not dramatic.

	[SM] Well, these are also pretty slight variations in numerical values, at least if compared to codel's default interval of 100ms...

> And truth to be told, the ECN behavior is better tuned in the code than the L4S behavior. 

	[SM] For a defintion of better that ignores that you seem to be much happier with the results L4S gives you.

> But the bottomline, the L4S thresholds and L4S code is not carefully picked to show a good performance.

	[SM] So, you tried different L4S parameters (and found the to have similar performance) but for rfc3168 ECN and Codel you only tried the default parameters?

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Mar 2020 21:04:51 +0000

Hi
Attached is a simulation of the same simple bottleneck case with target=1ms and interval=20ms. 
Two ECN beta values : 0.8 and 0.6. A larger backoff (0.6) does actually not improve things as the CWND reduction leads to that packets from the video coder are queued up in the RTP queue. 
As you can see there is still a way to go to reach the L4S delays. 

Perhaps the SCReAM's response to CoDel - ECN marks can be optimized further, don't know, but I already spent a considerable time to try and get where the code is now, and I spent a lot more time on this than I spent on the response to the L4S signal. 
My impression is that it is the fractional congestion signal with L4S that makes it easier to get a good algorithm behavior (and I definitely believe that there is room for improvement)

And as said earlier, the SCReAM code is freely available to experiment with and occasionally it is quite fun (https://www.youtube.com/watch?v=eU1crtEvMv4 ). 

/Ingemar

> Sent: den 10 mars 2020 16:29
> > On Mar 10, 2020, at 13:37, Ingemar Johansson S
> > The SCReAM code is freely available on
> https://protect2.fireeye.com/v1/url?k=3fa4aff0-6370a99d-3fa4ef6b-
> 867011091b6c-c4d35656f5c9b268&q=1&e=16e2b451-36c7-4814-af9d-
> 4acb21ac792b&u=https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream
> for anybody interested to run their own experiment with whatever AQM/ECN
> configuration.
> 
> 	[SM] Well, if you are not interested in figuring out what is happening
> with with rfc3168 ECN ans SCReAM that is your prerogative...
> 
> 
> > Please note that SCReAM is configured in an L4S mode when the network
> > AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN however, SCReAM
> > runs in normal ECN mode with a beta of 0.8 (=20% reduction on CWND for
> > each congestion event)
> 
> 	[SM] Okay, that is a far cry from Reno's 50% though, have you tested
> different reduction percentages or how did you arrive at 20%?
> 	[SM] That means for L4S, or do you mean you tried different values for
> Codel's target ans ce_threshold parameters?
> 
> 
> 	[SM] Well, these are also pretty slight variations in numerical values, at
> least if compared to codel's default interval of 100ms...
> 
> > There is room for improvement as regards to the L4S behavior (for instance
> faster ramp-up) and it may well be the case that SCReAM is completely scrapped
> in favor of new designs.
> 	[SM] So, you tried different L4S parameters (and found the to have
> similar performance) but for rfc3168 ECN and Codel you only tried the default
> parameters?
> 
> Best Regards
> 	Sebastian
> 
> >
> > /Ingemar
> >
> >> performance gain that L4S give for instance for AR/VR, gaming and
> >> remote control applications.
> >>
> >> give[s]" if you do not compare it with the best of class
> >> alternatives? I am truly puzzled.
> >>
> >> the scope as it would require packet inspection, which is not
> >> feasible if queues build up on the RLC layer in the 3GPP stack.
> >>
> >> As far as I can see you paired an application designed for 1/p-type
> >> congestion feed-back with an 1/sqrt(p)-type AQM that was also set for
> >> sub-optimal RTT and latency target for the test path. And lo and
> >> behold, the application does "better*" for the 1/p-type AQM (with
> >> lower latency target; I assume that  L4S ramp-marker (Th_low=2ms,
> >> Th_high=10ms) was carefully selected to match what SCReAM expects).
> >> IMHO that simply demonstrates, that in communication it pays if
> >> sender and receiver of a symbol (CE here) assign the same meaning to it.
> >>
> >> *) Assuming one buys into your definition of better, in which
> >> (instantaneous) queueing delay is valued over video quality. From a
> >> network operators perspective that seems a valid position
> >>
> >>
> >>
> >>
> >>>
> >>> /Ingemar
> >>>
> >>>>> https://protect2.fireeye.com/v1/url?k=63019d27-3f884737-6301ddbc-0
> >>>>> cc
> >>>>> 47
> >>>> compared to SCReAM over codel. You describe this as "L4S reduces
> >>>> the delay considerably more" and "L4S gives a somewhat lower media
> rate".
> >>>> If that improves things considerably it would argue for embedding
> >>>> the current best RTT estimate into SCReAM packets, so an AQM could
> >>>> tailor its signaling better to individual flow properties (and yes,
> >>>> that will require a flow-aware AQM).
> >>>>
> >>>>
> >>>>
> >>>>> it is fair to say that these simple simulations should of course
> >>>>> be seen as just a
> >>>> snapshot.
> >>>>
> >>>> 	[SM] Fair enough.
> >>>>
> >>>>> Meanwhile, the SCReAM code on github is freely available for
> >>>>> anyone who

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 12 Mar 2020 12:51:55 +0100

Hi Ingemar,

Thanks for following up with interesting data.

> On Mar 10, 2020, at 22:04, Ingemar Johansson S <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi

	[SM] I see. But I also note that the way you describe scream's operation as being a bad match for rfc3168's one mark per RTT operations mode, you really want to track the instantaneous congestion state, so more instantaneous signaling is a better match for your application. 

> 
> Perhaps the SCReAM's response to CoDel - ECN marks

	[SM] Question, in your test-bed/simulations, is there a linux host with a codel qdisc, or is this using seperate simulation code like ns-3?

> can be optimized further, don't know, but I already spent a considerable time to try and get where the code is now, and I spent a lot more time on this than I spent on the response to the L4S signal. 

	[SM] Yes, as far as I can see scream really wants high resolution (low-magnitude) congestion information and hence seems like a perfect match for the higher signaling rates of 1/p-type congestion signaling, like L4S or (dare I say it SCE).

> My impression is that it is the fractional congestion signal with L4S that makes it easier to get a good algorithm behavior (and I definitely believe that there is room for improvement)

	[SM] I would naively guess that is is both the higher signaling rate and the magnitude that helps, your ECN experiments with 90, 80, and 60% backoff indicate that it does not seem to be the magnitude alone.

> 
> And as said earlier, the SCReAM code is freely available to experiment with and occasionally it is quite fun (https://www.youtube.com/watch?v=eU1crtEvMv4 <https://www.youtube.com/watch?v=eU1crtEvMv4> ). 

	[SM] So to run tests like yours, what else is needed in addition to the contents of the scream repository (and matlab/octave to generate the plots)?

Best Regards
	Sebastian

>> From: Sebastian Moeller <[email-masked] <mailto:[email-masked]>>

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 13 Mar 2020 20:32:59 +0000

Hi

The recent simulations are with SCReAM integrated in our proprietary 5G system simulator. It is however the same code as that available on github. I believe that the example code and the presentation there will give sufficient guidelines on how to integrate it into NS-3.

Regards

/Ingemar 

From: Sebastian Moeller <[email-masked]> 
Sent: den 12 mars 2020 12:52
To: Ingemar Johansson S <[email-masked]>
Cc: Ingemar Johansson S <ingemar.s.johansson=[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

Hi Ingemar,

On Mar 10, 2020, at 22:04, Ingemar Johansson S <[email-masked] <mailto:[email-masked]> > wrote:

Hi

Best Regards

             Sebastian

/Ingemar

-----Original Message-----
From: Sebastian Moeller <[email-masked] <mailto:[email-masked]> >
Sent: den 10 mars 2020 16:29
To: Ingemar Johansson S <[email-masked] <mailto:[email-masked]> >
Cc: Ingemar Johansson S
<ingemar.s.johansson=[email-masked] <mailto:ingemar.s.johansson=[email-masked]> >; [email-masked] <mailto:[email-masked]> ;
[email-masked] <mailto:[email-masked]> 
Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

Hi Ingemar,

<[email-masked] <mailto:[email-masked]> > wrote:

Hi

I tried also with other different ramp markers (1ms/10ms),

(2ms/10ms),(5ms/15ms).

And truth to be told, the ECN behavior is better tuned in the code than the L4S

behavior.

          [SM] For a defintion of better that ignores that you seem to be much
happier with the results L4S gives you.

But the bottomline, the L4S thresholds and L4S code is not carefully picked to

show a good performance.

Best Regards
          Sebastian

/Ingemar

-----Original Message-----
From: Sebastian Moeller <[email-masked] <mailto:[email-masked]> >
Sent: den 10 mars 2020 11:28
To: Ingemar Johansson S <[email-masked] <mailto:[email-masked]> >
Cc: Ingemar Johansson S
<ingemar.s.johansson=[email-masked] <mailto:ingemar.s.johansson=[email-masked]> >; [email-masked] <mailto:[email-masked]> ;
[email-masked] <mailto:[email-masked]> 
Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

Hi Ingemar,

<[email-masked] <mailto:[email-masked]> > wrote:

Hi

For the future studies we will only focus on L4S as the scope is to
study the

Flow aware AQMs with RTT estimates as metadata in the packets is
outside

Best Regards
          Sebastian

/Ingemar

-----Original Message-----
From: Sebastian Moeller <[email-masked] <mailto:[email-masked]> >
Sent: den 10 mars 2020 10:45
To: Ingemar Johansson S
<ingemar.s.johansson=[email-masked] <mailto:ingemar.s.johansson=[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> ; Ingemar Johansson S
<[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

Hi Ingemar,

<ingemar.s.johansson=[email-masked] <mailto:ingemar.s.johansson=[email-masked]> > wrote:

Hi

I recently updated the readme on the SCReAM github with a
comparison with

• No ECN
• CoDel ECN
• L4S

5b55606e303b&u=

https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream%23ecn-

explicit-

co

ngestion-notification

Even though it is more than a magnitude difference in queue delay
between CoDel-ECN and L4S,

rate".

I wonder how many end-users would tradeoff these 25ms in queueing
delay against the decrease in video quality from halving the bitrate?
Could you repeat the Codel test with interval set to 20 and target
to 1ms, please?

snapshot.

[SM] Fair enough.

We hope to present some more simulations with 5G access, and not
just

/Ingemar

Ericsson Research
RESEARCHER
GFTL ER NAP NCM Netw Proto & E2E Perf Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked] <mailto:[email-masked]> 
www.ericsson.com <http://www.ericsson.com>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Mar 2020 01:01:27 +0000

Ingemar,

> Please note that SCReAM is configured in an L4S mode when the network AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN however, SCReAM runs in normal ECN mode with a beta of 0.8 (=20% reducti…

[BB] In your previous work, in order to get early L4S ECN marking, you 
used a virtual queue in the network draining at a little less than the 
rate given by the scheduler. Am I right that you're not doing that here?

In the L4S case you show, when the video is running at about half the 
rate in the CoDel case (which is presumably closer to that available 
from the scheduler), one would have thought it would get very little ECN 
marking. But I notice it's getting frequent little spikes of marking. Is 
that due to I-frames causing bursts or something?

What if it wasn't getting those spikes of marks (e.g. using sthg like 
x264 to spread the I-frames over time)?

A random thought: Could the video use the Classic queue as a scavenger 
service to fill in extra coding layer(s) whenever bandwidth is available?

Bob

> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Mar 2020 01:12:12 +0000

Sebastian

>> thanks for posting this interesting piece of data!

[BB] Yes, thanks, @Ingemar...

>>> https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream%23ecn-explicit-congestion-notification
>>>

[BB] This does seem more harsh than I would have imagined - so a useful 
data point; thx Ingemar. Nonetheless, this is the end-system taking 
bandwidth away from itself in order to give itself lower latency in the 
presence of varying network capacity. Nothing to stop other applications 
making different tradeoffs.

It is worth thinking about the complexity of the policy control and 
signalling system needed for Ingemar's video to express the tradeoffs it 
is making here;... if it had to get an FQ scheduler to allow these 
tradeoffs instead. The scheduler would not necessarily have to make all 
the tradeoffs itself - for instance the app could underutilize it's 
'fair' share, but it would need to be able to take more than its 'fair' 
share during periods of lower capacity.

In the list of SCE issues,
https://github.com/heistp/sce-l4s-bakeoff/blob/master/README.md#list-of-sce-issues
it says "Another argument is the perception that FQ can do harm to 
periodic bursty flows, however we have not yet shown this to be the case 
with hard evidence". I don't recognize that argument, but I would if it 
had said "...can do harm to flows needing variable throughput or smooth 
throughput in the presence of variable available capacity". If Ingemar 
had run a TCP flow in parallel here, I think that would go some way 
towards the hard evidence sought here?

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Mar 2020 01:32:47 +0000

Sebastian,

I suspect it is the 20ms interval that is the main cause the extra 
queuing delay with CoDel.

The 20:1 ratio between interval and target was really designed around 
loss. The need to filter out brief excursions in the queue is only 
necessary when you're using impairments (losses) as signals. If you're 
using explicit signals, you can signal unfiltered queue variations, 
because there is no harm having higher ECN marking levels. For that 
matter, you might as well use Eric Dumazet's simple patch to CoDel that 
adds immediate ECN marking at a shallow threshold, bypassing all the 
CoDel machinery, which is really tailored for loss.

Of course, unless you have FQ (which you can't at the RLC layer), you 
would have to work out some other way to ensure coexistence with 
existing congestion controls. That is left as an exercise for the reader 
- I wouldn't want to be seen to be pushing any particular solution. But 
hopefully, through scenarios like this, you might start to understand 
the set of requirements that led to one solution rather than another.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 17 Mar 2020 06:19:06 +0200

> On 10 Mar, 2020, at 11:04 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

I had a look through this thread again, and I remembered something which may be very relevant to put these results into context.  I think you are making at least some of these tests with a 50fps video stream from a typical PAL-format camcorder, in which case the interval between frames is already 20ms.  The top of the delay graph in the tightened Codel traces is only 50ms, and in the originals is 100ms.

I estimate the queue delay in the tightened Codel traces is approximately 10ms most of the time - only half a frame.  Is this actually noticeable to the user?  Perhaps this performance is already good enough.

I think this level of delay mostly results from the behaviour of your pacing algorithm under bursty demand.  Codel deliberately tolerates a degree of burstiness, in the name of maintaining reasonable throughput on the networks and traffic patterns found in the real world.  As long as the queue empties below the target delay within the interval, Codel will not intervene - and at the tightened settings, the interval is approximately the same as the video framerate, which seems entirely appropriate.

Something you could try is to partially simulate an SCE deployment, by setting SCReAM into L4S mode and employing a Codel AQM set to, say, 25ms interval, 2.5ms target.  I think that might have interesting results.  This actually represents the SCE portion of an AQM optimised for a 100ms path, but it's a configuration that's quite well tested and should cope well with a bursty traffic load.  There are some differences between the L4S response algorithm and the ones we use in SCE, but I think you should get good results with this combination.

I've looked briefly at the code, and I think true SCE support could also be implemented without difficulty; you already have sufficient ECN codepoint feedback from the receiver.  You would need to apply the L4S response (or something broadly equivalent) when noting ECT(1) codepoints, and the conventional Multiplicative Decrease response when noting CE codepoints.  Obviously, this also means ECT(0) must be emitted at origin, not ECT(1).

I'd be happy to advise further on work in this direction, if you think it's promising.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Mar 2020 06:40:12 +0000

Hi Bob

Please see inline 

/Ingemar

From: Bob Briscoe <[email-masked]> 
Sent: den 17 mars 2020 02:01
To: Ingemar Johansson S <ingemar.s.johansson=[email-masked]>
Cc: Sebastian Moeller <[email-masked]>; Ingemar Johansson S <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [iccrg] [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S

Ingemar,

Hi
 
The SCReAM code is freely available on https://github.com/EricssonResearch/scream <https://protect2.fireeye.com/v1/url?k=70bef122-2c35fa1d-70beb1b9-868f25761b72-27b51935b33618c8&q=1&e=6939be47-9753-40b0-819b-9c42d21fc6e4&u=https%3A%2F%2Fgithub.com%2FEricssonResearch%2Fscream>  for anybody interested to run their own experiment with whatever AQM/ECN configuration. 
Please note that SCReAM is configured in an L4S mode when the network AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN however, SCReAM runs in normal ECN mode with a beta of 0.8 (=20% reduction on CWND for each congestion event)

[BB] In your previous work, in order to get early L4S ECN marking, you used a virtual queue in the network draining at a little less than the rate given by the scheduler. Am I right that you're not doing that here? 

[IJ] That is correct and yes, virtual queues are definitely a possibility to achieve zero (or near zero) queue delay.

In the L4S case you show, when the video is running at about half the rate in the CoDel case (which is presumably closer to that available from the scheduler), one would have thought it would get very little ECN marking. But I notice it's getting frequent little spikes of marking. Is that due to I-frames causing bursts or something?

What if it wasn't getting those spikes of marks (e.g. using sthg like x264 to spread the I-frames over time)?

[IJ] If periodic intra refresh was used instead then the “peak to average ratio” (for lack of a better term) in frame sizes would decrease and SCReAM would increase its target bitrate. Periodic intra refresh is however not always feasible, for instance scene changes (in gaming to take one example) will require IDR frames. One question mark here is what impact an occasional large I-frame will have then?. In the simple bottleneck case here, there is no more capacity available and one will inevitably get a delay spike. In a e.g 5G system there is a possibility to use different scheduling algorithms that makes it possible for flows to temporarily scavenge on the available capacity. I don’t however have the data to show that yet.   

A random thought: Could the video use the Classic queue as a scavenger service to fill in extra coding layer(s) whenever bandwidth is available?

[IJ] Depends on the delay budget, I guess. 

Bob

 
 
/Ingemar

-----Original Message-----
From: Sebastian Moeller  <mailto:[email-masked]> <[email-masked]>
Sent: den 10 mars 2020 11:28
To: Ingemar Johansson S  <mailto:[email-masked]> <[email-masked]>
Cc: Ingemar Johansson S
 <mailto:ingemar.s.johansson=[email-masked]> <ingemar.s.johansson=[email-masked]>; [email-masked] <mailto:[email-masked]> ;
 
Hi Ingemar,

 <mailto:[email-masked]> <[email-masked]> wrote:

Hi

 

 

 
Best Regards
   Sebastian

 

/Ingemar

-----Original Message-----
From: Sebastian Moeller  <mailto:[email-masked]> <[email-masked]>
Sent: den 10 mars 2020 10:45
To: Ingemar Johansson S
 <mailto:ingemar.s.johansson=[email-masked]> <ingemar.s.johansson=[email-masked]>
Cc: [email-masked] <mailto:[email-masked]> ; Ingemar Johansson S
 <mailto:[email-masked]> <[email-masked]>; [email-masked] <mailto:[email-masked]> 
Subject: Re: [tsvwg] SCReAM (RFC8298) with CoDel-ECN and L4S
 
Hi Ingemar,

 <mailto:ingemar.s.johansson=[email-masked]> <ingemar.s.johansson=[email-masked]> wrote:

Hi
 
I recently updated the readme on the SCReAM github with a comparison
with

       • No ECN
       • CoDel ECN
       • L4S
https://protect2.fireeye.com/v1/url?k=63019d27-3f884737-6301ddbc-0cc
47
ad93e2a-489fa99c3277fb8a&q=1&e=5aab95a7-4aab-4a64-99a5-

5b55606e303b&u=

explicit-

co

ngestion-notification

 
Could you repeat the Codel test with interval set to 20 and target to
1ms, please?

 

snapshot.
 
      [SM] Fair enough.

 

/Ingemar
 
Ericsson Research
RESEARCHER

_______________________________________________
iccrg mailing list
[email-masked] <mailto:[email-masked]> 
https://www.irtf.org/mailman/listinfo/iccrg <https://protect2.fireeye.com/v1/url?k=4fa6ca14-132dc12b-4fa68a8f-868f25761b72-1432ef259e224229&q=1&e=6939be47-9753-40b0-819b-9c42d21fc6e4&u=https%3A%2F%2Fwww.irtf.org%2Fmailman%2Flistinfo%2Ficcrg> 

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/ <https://protect2.fireeye.com/v1/url?k=4b6f9fdb-17e494e4-4b6fdf40-868f25761b72-0cf56bc001a75473&q=1&e=6939be47-9753-40b0-819b-9c42d21fc6e4&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Mar 2020 07:02:30 +0000

Hi Jonathan
Please see inline
/Ingemar

> Sent: den 17 mars 2020 05:19
> To: Ingemar Johansson S
> Cc: Sebastian Moeller <[email-masked]>; Ingemar Johansson S
> <[email-masked]>; [email-masked]; [email-masked]
> > On 10 Mar, 2020, at 11:04 pm, Ingemar Johansson S
> > Attached is a simulation of the same simple bottleneck case with
target=1ms
> and interval=20ms.
> > Two ECN beta values : 0.8 and 0.6. A larger backoff (0.6) does actually
not
> improve things as the CWND reduction leads to that packets from the video
> coder are queued up in the RTP queue.
> > Perhaps the SCReAM's response to CoDel - ECN marks can be optimized
> further, don't know, but I already spent a considerable time to try and
get where
> the code is now, and I spent a lot more time on this than I spent on the
response
> to the L4S signal.
> > My impression is that it is the fractional congestion signal with L4S
that makes
> it easier to get a good algorithm behavior (and I definitely believe that
there is
> room for improvement)
> 
> I had a look through this thread again, and I remembered something which
may
> be very relevant to put these results into context.  I think you are
making at least
> some of these tests with a 50fps video stream from a typical PAL-format
> camcorder, in which case the interval between frames is already 20ms.  The
top
> of the delay graph in the tightened Codel traces is only 50ms, and in the
originals
> is 100ms.
[IJ] The frame rate in this example is 60fps, which means a 16.6ms frame
interval. I am not convinced though that the frame rate plays so much role
here as the packet pacing make these 16.6ms frames become transmitted in
~10ms or so (depends in the computed pacing rate in SCReAM)

> 
> I estimate the queue delay in the tightened Codel traces is approximately
10ms
> most of the time - only half a frame.  Is this actually noticeable to the
user?
> Perhaps this performance is already good enough.
> 
> I think this level of delay mostly results from the behaviour of your
pacing
> algorithm under bursty demand.  Codel deliberately tolerates a degree of
> burstiness, in the name of maintaining reasonable throughput on the
networks
> and traffic patterns found in the real world.  As long as the queue
empties below
> the target delay within the interval, Codel will not intervene - and at
the
> tightened settings, the interval is approximately the same as the video
> framerate, which seems entirely appropriate.
> 
> Something you could try is to partially simulate an SCE deployment, by
setting
> SCReAM into L4S mode and employing a Codel AQM set to, say, 25ms interval,
> 2.5ms target.  I think that might have interesting results.  This actually
represents
> the SCE portion of an AQM optimised for a 100ms path, but it's a
configuration
> that's quite well tested and should cope well with a bursty traffic load.
There
> are some differences between the L4S response algorithm and the ones we
use
> in SCE, but I think you should get good results with this combination.
[IJ] Yes, possible that it may be as you say, but doesn't this reveal a flaw
in CoDel if you need to adapt thresholds based on assumptions on application
layer behavior?. 
I cannot for instance rule out that somebody in the future (or perhaps
already now?) implements a gaming engine that updates part of the view with
240fps and other, less important parts with 30fps. QUIC would allow all that
media over one connection but in different streams. For the network it would
just be a bunch of packets passing through.

> 
> I've looked briefly at the code, and I think true SCE support could also
be
> implemented without difficulty; you already have sufficient ECN codepoint
> feedback from the receiver.  You would need to apply the L4S response (or
> something broadly equivalent) when noting ECT(1) codepoints, and the
> conventional Multiplicative Decrease response when noting CE codepoints.
> Obviously, this also means ECT(0) must be emitted at origin, not ECT(1).
> 
> I'd be happy to advise further on work in this direction, if you think
it's
> promising.
[IJ] Surely possible. But like said before, I am not going there. I don't
see that it is worth spending effort on this as it should be apparent by now
that SCE would not give better performance than L4S . I'd rather spend my
time on improving SCReAM, or any other congestion control algorithm for low
latency steaming media to utilize the potential with L4S better. But the
SCReAM code is of course available for anybody who wants to with it in any
desired direction.

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 17 Mar 2020 10:12:16 +0200

> On 17 Mar, 2020, at 9:02 am, Ingemar Johansson S <[email-masked]> wrote:
> 
> The frame rate in this example is 60fps, which means a 16.6ms frame

Hmm, takes 10ms to transmit the frame, results in 10ms peak queuing latency, on a 20ms baseline path.  Seems like an awfully big coincidence to me.  All joking aside…

> Yes, possible that it may be as you say, but doesn't this reveal a flaw
> layer behavior?

No, because that's not what we're making assumptions about when choosing Codel parameters.  It just so happens that your baseline path RTT and framerate are conveniently similar, but the typical path RTT is what Codel is supposed to be tuned for.  The default parameters, with a 100ms interval, are intended for use on general Internet paths with conventional AIMD traffic.

The 25/2.5ms parameters are what we are already using for high-fidelity congestion control in much of our SCE testing.  They are not tuned for your particular application.  I'm just curious how it would react, with the L4S-type response, if given that style of signalling.  No code changes are needed to try this out.

> just be a bunch of packets passing through.

I imagine such a stream would have more frequent cycling and relatively smaller peaks of traffic.  This would make the problem slightly easier, not harder, from a queuing point of view.

> I don't see that it is worth spending effort on this as it should be apparent by now that SCE would not give better performance than L4S.

When the traffic is not bursty and does not exhibit a significant sawtooth structure, the difference between controlling the peak or the trough of the traffic is minor.  However, SCReAM presently does not fit that description.

To a great extent, L4S' aggressive marking strategy ends up controlling the peaks of the traffic, while Codel tends to control the troughs.  When applied to SCReAM traffic, the latter results in noticeably higher throughput - which I presume corresponds to better image quality - than the former.  It does do while still controlling the added delay to less than one frame time, except during the exceptional event of the sharp reduction in capacity.

SCE is designed to be safe to use over existing Internet paths.  L4S, as presently implemented, is not.  That's a point which might not show up on a raw performance comparison, in which traffic is run only through nodes which fully implement the new specification.  It is of course possible to employ L4S' aggressive signalling schedule using the SCE method.  This gets you the same performance as L4S on a conforming network, but also keeps you compatible with conventional networks by retaining the standard meaning of a CE mark.

So I'm not certain what metrics you're using which say L4S outperforms SCE here, but since we are using Codel for SCE marking, I'd say the above characteristics are at least worthy of further study.  Of course I can't insist on it, but I think it would save some time and effort in the long run.

Separately, I also think that if you were to make the pacing rate and target encoding rate more similar (as the encoder seems to track the latter pretty closely), that would improve SCReAM's performance on both L4S and SCE - for different reasons.  Both stem from the fact that your data-in-flight number currently fluctuates a lot on short timescales, and on average is much less than the cwnd.

In the case of L4S, you would get a straightforward increase in throughput and thus image quality from what you presently have.

In the case of Codel-based marking, there would be less noise in the queue-depth signal, resulting in a better estimate of the true BDP being represented in the cwnd, and better control of the network side of the delay equation. As I understand it, SCReAM is designed so that data still in the application-side buffer can be discarded to accelerate recovery when network conditions abruptly change, and this would facilitate that.  The result should be roughly equal throughput but improved delay characteristics.

But until it's tested, those are just my educated guesses.  I think SCReAM-like traffic would be a valuable addition to our test suite, but it will take time to figure out how to integrate it - since it's written in a different language than we normally use.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Mar 2020 10:38:39 +0000

Hi
Inline..
Ingemar

> Sent: den 17 mars 2020 09:12
> <ingemar.s.johansson=[email-masked]>; Sebastian Moeller
> <[email-masked]>; [email-masked]; [email-masked]
> > On 17 Mar, 2020, at 9:02 am, Ingemar Johansson S
> > interval. I am not convinced though that the frame rate plays so much
> > role here as the packet pacing make these 16.6ms frames become
> > transmitted in ~10ms or so (depends in the computed pacing rate in
> > SCReAM)
> 
> Hmm, takes 10ms to transmit the frame, results in 10ms peak queuing latency,
> on a 20ms baseline path.  Seems like an awfully big coincidence to me.  All joking
> aside…
[IJ] ~10ms means roughly 10ms, could be 11.23342344354654667ms or 13.23423434423ms or 9.23423432434234ms ......

> 
> > Yes, possible that it may be as you say, but doesn't this reveal a
> > flaw in CoDel if you need to adapt thresholds based on assumptions on
> > application layer behavior?
> 
> No, because that's not what we're making assumptions about when choosing
> Codel parameters.  It just so happens that your baseline path RTT and framerate
> are conveniently similar, but the typical path RTT is what Codel is supposed to be
> tuned for.  The default parameters, with a 100ms interval, are intended for use
> on general Internet paths with conventional AIMD traffic.
> 
> The 25/2.5ms parameters are what we are already using for high-fidelity
> congestion control in much of our SCE testing.  They are not tuned for your
> particular application.  I'm just curious how it would react, with the L4S-type
> response, if given that style of signalling.  No code changes are needed to try
> this out.
> 
> > already now?) implements a gaming engine that updates part of the view
> > with 240fps and other, less important parts with 30fps. QUIC would
> > allow all that media over one connection but in different streams. For
> > the network it would just be a bunch of packets passing through.
> 
> I imagine such a stream would have more frequent cycling and relatively smaller
> peaks of traffic.  This would make the problem slightly easier, not harder, from a
> queuing point of view.
> 
> > I don't see that it is worth spending effort on this as it should be apparent by
> now that SCE would not give better performance than L4S.
> 
> 
> When the traffic is not bursty and does not exhibit a significant sawtooth
> structure, the difference between controlling the peak or the trough of the
> traffic is minor.  However, SCReAM presently does not fit that description.
> 
> To a great extent, L4S' aggressive marking strategy ends up controlling the
> peaks of the traffic, while Codel tends to control the troughs.  When applied to
> SCReAM traffic, the latter results in noticeably higher throughput - which I
> presume corresponds to better image quality - than the former.  It does do while
> still controlling the added delay to less than one frame time, except during the
> exceptional event of the sharp reduction in capacity.
> 
> SCE is designed to be safe to use over existing Internet paths.  L4S, as presently
> implemented, is not.  That's a point which might not show up on a raw
> performance comparison, in which traffic is run only through nodes which fully
> implement the new specification.  It is of course possible to employ L4S'
> aggressive signalling schedule using the SCE method.  This gets you the same
> performance as L4S on a conforming network, but also keeps you compatible
> with conventional networks by retaining the standard meaning of a CE mark.
> 
> So I'm not certain what metrics you're using which say L4S outperforms SCE
> here, but since we are using Codel for SCE marking, I'd say the above
> characteristics are at least worthy of further study.  Of course I can't insist on it,
> but I think it would save some time and effort in the long run.
> 
> Separately, I also think that if you were to make the pacing rate and target
> encoding rate more similar (as the encoder seems to track the latter pretty
> closely), that would improve SCReAM's performance on both L4S and SCE - for
> different reasons.  Both stem from the fact that your data-in-flight number
> currently fluctuates a lot on short timescales, and on average is much less than
> the cwnd.
> 
> In the case of L4S, you would get a straightforward increase in throughput and
> thus image quality from what you presently have.
> 
> In the case of Codel-based marking, there would be less noise in the queue-
> depth signal, resulting in a better estimate of the true BDP being represented in
> the cwnd, and better control of the network side of the delay equation. As I
> understand it, SCReAM is designed so that data still in the application-side buffer
> can be discarded to accelerate recovery when network conditions abruptly
> change, and this would facilitate that.  The result should be roughly equal
> throughput but improved delay characteristics.
> 
> But until it's tested, those are just my educated guesses.  I think SCReAM-like
> traffic would be a valuable addition to our test suite, but it will take time to
> figure out how to integrate it - since it's written in a different language than we
> normally use.
[IJ] Please go ahead and experiment with SCReAM and SCE if you feel it's worth it. Personally I will not waste my precious brain cycles on SCE.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Mar 2020 14:15:46 +0100

Hi Bob,

> On Mar 17, 2020, at 02:32, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> I suspect it is the 20ms interval that is the main cause the extra queuing delay with CoDel.

	[SM] That and the fact that rfc3168 asserts ECE until it receives the CWR from the sender and that on Ingemar's test path takes 20ms, so the temporal fifelity of rfc3168 ECN is simply too low for SCReAM's feed-back fidelity requirements.

> 
> The 20:1 ratio between interval and target was really designed around loss.

	[SM] I do not think that this actually makes a big difference for rfc3168 ECN, CE will be asserted just as a drop would be for NON-ECN packets, and it will take a while for a drop being registered by the receiver and send back to the sender as well. Note how the 5% rule has no bearing on that.

> The need to filter out brief excursions in the queue is only necessary when you're using impairments (losses) as signals.

	[SM] I disagree, (limited) burst tolerance is not only helpful if loss is involved.

> If you're using explicit signals, you can signal unfiltered queue variations, because there is no harm having higher ECN marking levels.

	[SM] Sure, but I strongly believe the bigger issue here is that L4S CE feedback simply has higher temporal fidelity not being limited by the 1RTT ECE->CWR dance.

> For that matter, you might as well use Eric Dumazet's simple patch to CoDel that adds immediate ECN marking at a shallow threshold, bypassing all the CoDel machinery, which is really tailored for loss.

	[SM] Note how I asked Ingemar about whether he tested codel's ce_threshold... But I am still wondering whether Ingemar's test where done with SCReAM's own limited Codel implementation or with a full fledged Linux AQM node in the path (looking at the SCReAM repository makes me believe its own Codel being very vrudimentary and not implementing ce_threshold and as it appears not even the square law drop interval calculation, @Ingemar, I am sure I am reading something wrong here, so please help me see what I am missing*).

*) I am 99.999% sure I am missing something ;)

> 
> Of course, unless you have FQ (which you can't at the RLC layer)

	[SM] Then use feedback the queueing information from the RLC layer back up to where at least IP packets are handled/seen, akin to say the wifi airtime fairness approach?

> , you would have to work out some other way to ensure coexistence with existing congestion controls. That is left as an exercise for the reader - I wouldn't want to be seen to be pushing any particular solution. But hopefully, through scenarios like this, you might start to understand the set of requirements that led to one solution rather than another.

	[SM] Well, I believe that neither 1/p type signaling, nor higer temporal feed-back fidelity are really under discussion, we all want those, the question is more what is the best and safest/backward compatiblest way of using those, but I digress (this is not the thread for that discussion ;) )

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Mar 2020 13:44:56 +0000

Hi

It is true that the "CoDel" implementation in the test code is a botched version. The CoDel algorithm implemented in our system simulator that was used to produce these examples is however _the_ CoDel

/Ingemar

> Sent: den 17 mars 2020 14:16
> To: Bob Briscoe <[email-masked]>
> Cc: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Ingemar Johansson S
> 	[SM] That and the fact that rfc3168 asserts ECE until it receives the CWR
> from the sender and that on Ingemar's test path takes 20ms, so the temporal
> fifelity of rfc3168 ECN is simply too low for SCReAM's feed-back fidelity
> requirements.
> 
> 
> >
> 	[SM] I do not think that this actually makes a big difference for rfc3168
> ECN, CE will be asserted just as a drop would be for NON-ECN packets, and it
> will take a while for a drop being registered by the receiver and send back to the
> sender as well. Note how the 5% rule has no bearing on that.
> 
> > The need to filter out brief excursions in the queue is only necessary when
> you're using impairments (losses) as signals.
> 
> > If you're using explicit signals, you can signal unfiltered queue variations,
> because there is no harm having higher ECN marking levels.
> 
> 	[SM] Sure, but I strongly believe the bigger issue here is that L4S CE
> feedback simply has higher temporal fidelity not being limited by the 1RTT ECE-
> >CWR dance.
> 
> 
> > For that matter, you might as well use Eric Dumazet's simple patch to CoDel
> that adds immediate ECN marking at a shallow threshold, bypassing all the
> 	[SM] Note how I asked Ingemar about whether he tested codel's
> ce_threshold... But I am still wondering whether Ingemar's test where done with
> SCReAM's own limited Codel implementation or with a full fledged Linux AQM
> node in the path (looking at the SCReAM repository makes me believe its own
> Codel being very vrudimentary and not implementing ce_threshold and as it
> appears not even the square law drop interval calculation, @Ingemar, I am sure
> I am reading something wrong here, so please help me see what I am missing*).
> 
> 
> 	[SM] Then use feedback the queueing information from the RLC layer
> back up to where at least IP packets are handled/seen, akin to say the wifi
> airtime fairness approach?
> 
> > , you would have to work out some other way to ensure coexistence with
> existing congestion controls. That is left as an exercise for the reader - I
> wouldn't want to be seen to be pushing any particular solution. But hopefully,
> through scenarios like this, you might start to understand the set of
> requirements that led to one solution rather than another.
> 
> 	[SM] Well, I believe that neither 1/p type signaling, nor higer temporal
> feed-back fidelity are really under discussion, we all want those, the question is
> more what is the best and safest/backward compatiblest way of using those, but
> I digress (this is not the thread for that discussion ;) )
> 
> Best Regards
> 	Sebastian
> 
> 
> >
> >
> > Bob
> >
> >> Two ECN beta values : 0.8 and 0.6. A larger backoff (0.6) does actually not
> further, don't know, but I already spent a considerable time to try and get where
> the code is now, and I spent a lot more time on this than I spent on the response
> to the L4S signal.
> >> that makes it easier to get a good algorithm behavior (and I
> >> definitely believe that there is room for improvement)
> >>
> >> And as said earlier, the SCReAM code is freely available to experiment with
> and occasionally it is quite fun
> (https://www.youtube.com/watch?v=eU1crtEvMv4 ).
> >>
> >>
> >> /Ingemar
> >>
> >>> for anybody interested to run their own experiment with whatever
> >>> AQM/ECN configuration.
> >>>
> >>> 	[SM] Well, if you are not interested in figuring out what is
> >>> happening with with rfc3168 ECN ans SCReAM that is your prerogative...
> >>>
> >>>
> >>>> Please note that SCReAM is configured in an L4S mode when the
> >>>> network AQM does L4S marking (mimicking ECT(1)). For CoDel-ECN
> >>>> however, SCReAM runs in normal ECN mode with a beta of 0.8 (=20%
> >>>> reduction on CWND for each congestion event)
> >>> 	[SM] Okay, that is a far cry from Reno's 50% though, have you
> >>> tested different reduction percentages or how did you arrive at 20%?
> >>> // CWND scale factor upon ECN-CE event static const float kEcnCeBeta
> >>> = 0.9f;
> >>>
> >>> 	[SM] That means for L4S, or do you mean you tried different values
> >>> for Codel's target ans ce_threshold parameters?
> >>>
> >>>
> >>> 	[SM] Well, these are also pretty slight variations in numerical
> >>> values, at least if compared to codel's default interval of 100ms...
> >>>
> >>> 	[SM] For a defintion of better that ignores that you seem to be
> >>> much happier with the results L4S gives you.
> >>>
> >>> faster ramp-up) and it may well be the case that SCReAM is
> >>> completely scrapped in favor of new designs.
> >>> similar performance) but for rfc3168 ECN and Codel you only tried
> >>> the default parameters?
> >>>
> >>> Best Regards
> >>> 	Sebastian
> >>>
> >>>> /Ingemar
> >>>>
> >>>>> <ingemar.s.johansson=[email-masked]>;
> >>>>> [email-masked]; [email-masked]
> >>>>> As far as I can see you paired an application designed for
> >>>>> 1/p-type congestion feed-back with an 1/sqrt(p)-type AQM that was
> >>>>> also set for sub-optimal RTT and latency target for the test path.
> >>>>> And lo and behold, the application does "better*" for the 1/p-type
> >>>>> AQM (with lower latency target; I assume that  L4S ramp-marker
> >>>>> (Th_low=2ms,
> >>>>> (instantaneous) queueing delay is valued over video quality. From
> >>>>> a network operators perspective that seems a valid position
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> /Ingemar
> >>>>>>
> >>>>>>> I wonder how many end-users would tradeoff these 25ms in
> >>>>>>> queueing delay against the decrease in video quality from halving the
> bitrate?
> >>>>>>> If that improves things considerably it would argue for
> >>>>>>> embedding the current best RTT estimate into SCReAM packets, so
> >>>>>>> an AQM could tailor its signaling better to individual flow
> >>>>>>> properties (and yes, that will require a flow-aware AQM).
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> it is fair to say that these simple simulations should of
> >>>>>>>> course be seen as just a
> >>>>>>> snapshot.
> >>>>>>>
> >>>>>>> 	[SM] Fair enough.
> >>>>>>>
> >>>>>>>> ================================ Ingemar Johansson  M.Sc.
> >>>>>>>> Master Researcher
> >>>>>>>>
> >>>>>>>> Ericsson Research
> >>>>>>>> RESEARCHER
> >>>>>>>> [email-masked] www.ericsson.com
> >>>>>>>>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Mar 2020 15:39:48 +0100

Hi Ingemar,

> On Mar 17, 2020, at 14:44, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] Great! See, I was missing something ;)

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Mar 2020 15:54:49 +0100

Hi Bob,

> On Mar 17, 2020, at 02:12, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian
> 
> [BB] This does seem more harsh than I would have imagined - so a useful data point; thx Ingemar. Nonetheless, this is the end-system taking bandwidth away from itself in order to give itself lower latency in the presence of varying network capacity. Nothing to stop other applications making different tradeoffs.
> 
> It is worth thinking about the complexity of the policy control and signalling system needed for Ingemar's video to express the tradeoffs it is making here;... if it had to get an FQ scheduler to allow these tradeoffs instead. The scheduler would not necessarily have to make all the tradeoffs itself - for instance the app could underutilize it's 'fair' share, but it would need to be able to take more than its 'fair' share during periods of lower capacity.

	[SM] I fail to see it that way, in Codel for example, it is the burst tolerance that allows exactly that kind of bandwidth trading with one self (send more now, make up for it a bit later by sending less). In fq_codel it is the FQ component that decides when a flow/bucket/tin is eligible to send something, and inside each flow/bucket/tin is a codel instance that decides what/when to mark/drop. But as Ingemar's example shows, his application gets significantly more bandwidth with Codel than with L4S, it would be interesting to see a plot of video quality (measured at the receiver). Any thing longer than a "burst" will get into tricky accounting territory if one wants to actually enforce long-term adherence to a set bandwidth share.

> it says "Another argument is the perception that FQ can do harm to periodic bursty flows, however we have not yet shown this to be the case with hard evidence". I don't recognize that argument, but I would if it had said "...can do harm to flows needing variable throughput or smooth throughput in the presence of variable available capacity". If Ingemar had run a TCP flow in parallel here, I think that would go some way towards the hard evidence sought here?

	[SM] Only if we assume that the video stream would be more important to the link's user, what if the TCP flow's speedy completion would actually be more important to the user? 
	That is, to be blunt, where I believe you fail to see the forrest for the trees, the AQM has no chance of being able to optimally split bandwidth between eligible flows without additional information to base its optimization upon. So either you supply that information (say explicit DSCP marking within a well-managed DSCP-domain) or you ned to accept that all you can do is aim for good enough and "do no harm". 
IMHO FQ strikes a decent balance here, it might rarely be optimal, but it also is equally rarely pessimal (while any unequal sharing mechanism will result in optimal and pessimal sharing, depending on whom you ask). In addition FQ allows much simpler prediction of what to expect from a link und saturating load. 
Of course this is just an opinion, and everybody here is entitled to their own opinion on this matter...

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Mar 2020 16:07:27 +0100

HI Ingemar,

> On Mar 17, 2020, at 08:02, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Jonathan
> Please see inline
> /Ingemar
> [...]
>> 
> that SCE would not give better performance than L4S .

	[SM] My prediction is, that both will give you more or less equal performance, because both offer the kind of fast feedback that you designed SCReAM to require. I do understand, that you do not want to change your test vehicle for yet another variant of ECN and that is fine and fair. But no matter which of the higher-temporal-fidelity, lower amplitude ECN schemes is left standing once the dust settles, any will work well for SCReAM (or rather better than rfc3168 ECN).

> I'd rather spend my
> latency steaming media to utilize the potential with L4S better.

	[SM] That is obviously your prerogative, but it also IMHO makes SCReAM not really a helpful data point in the discussion about how to deploy 1/p-style congestion control to the internet, as it does not help to choose between the contenders. Unless I missed something, the discussion is less about "if" we should deploy 1/p fast congestion signaling and more about "how" and which variant.

> But the

	[SM] Thanks for the open project!

Best Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Mar 2020 18:45:45 +0000

Sebastian,

> 	[SM] I fail to see it that way, in Codel for example, it is the burst tolerance that allows exactly that kind of bandwidth trading with one self (send more now, make up for it a bit later by sendi…

[BB] These views are from one side of a long-standing philosophical 
debate, in which neither side holds a monopoly on the truth.

Focusing on the "significantly more bandwidth" side without also saying 
"significantly less latency" misses Ingemar's point. The app could have 
chosen either but it chose what it chose.

If an app chooses quality that leads to a worse quality score, that 
probably means that the weighting of the factors in the quality scoring 
is flawed. Shouldn't the quality score reflect what is important for the 
application?

The ideas you've expressed here such as:

  * the network decides your maximum burst tolerance
  * the network decides that applications can have no more flexibility
    than a burst tolerance
  * the network decides how much users should prefer bandwidth over
    latency, and
  * the network decides one objective video quality metric

...are from the Bell-head world. I'm not saying Bell-head is wrong. I'm 
saying it's pointless trying to insist that a Bell-head idea is more 
correct than a Net-head idea. They are different philosophies.

I prefer to shift the debate to how to design a Net-head solution with a 
configurable degree of Bell-headedness. That is what the combination of 
the DualQ plus per-flow Queue Protection is. If you wind the queue 
protection up tight, you emulate per-flow scheduling. If you loosen it 
off (or disable it, or don't deploy it), you get application freedom.

This is also why it's important to enable both DualQ and FQ. Neither are 
objectively correct. Let the market decide.

> 	That is, to be blunt, where I believe you fail to see the forrest for the trees, the AQM has no chance of being able to optimally split bandwidth between eligible flows without additional informat…

[BB] Outside the forest that your mind is in, there is a wider set of 
forests where:
* in your forest, FQ attempts to optimally split bandwidth
* in other forests, the DualQ Coupled AQM doesn't even attempt to 
optimally split bandwidth, depending instead on "the application knows 
that best" (collectively)
* and in yet other forests (DualQ with configurable per-flow queue 
protection and policing), there can be various points on a spectrum 
between the two

L4S signalling was designed to support all these forests. But according 
to your ironic insult, one of the architects of this ecosystem of 
forests only sees trees. There are more forests than the one you are in.

> Of course this is just an opinion, and everybody here is entitled to their own opinion on this matter...

[BB] It is neither appropriate nor necessary for the IETF to decide to 
assign a codepoint in the IP header that only works well in your forest 
(FQ). The whole point of L4S was to broaden the ways extremely low 
latency could be provided (more forests): from FQ to DualQ with 
QProt+DualQ in between.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Mar 2020 09:18:20 +0100

Hi Bob,

> On Mar 17, 2020, at 19:45, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
[...]
> [BB] These views are from one side of a long-standing philosophical debate, in which neither side holds a monopoly on the truth.

	[SM] We are not discussing a philosophical point here, but the sheer practicality that long-term  average fairness as you seem to propose as a desirable property above, will drag in way more severe flow accounting issues than the more traditional FQ plus burst-tolerant AQM solutions. You are not even willing to concede that per flow state tracking is an obvious good idea (e.g. to monitor whether a flow's behavior honors certain requirements), so why are you proposing your idea of "anything goes" as a method to implement long-term fairness? I

> 
> Focusing on the "significantly more bandwidth" side without also saying "significantly less latency" misses Ingemar's point. The app could have chosen either but it chose what it chose.
> 
> If an app chooses quality that leads to a worse quality score, that probably means that the weighting of the factors in the quality scoring is flawed.

	[SM] Isn't that shooting the messenger? I will assume that this is supposed to be a jest.

> Shouldn't the quality score reflect what is important for the application?

	[SM] In the case of of a capacity-seeking real-time video streaming application, the point is, most users will expect the application to deliver the best audio-visual video quality that the latency/bandwidth budget allows. If an application decides to deliver sub-par quality, users will switch applications (unless there are redeeming qualities of said application)... And video quality is not rated by what is convenient for encoders/decoders, but by psychophysics experiments with "eyeballs", so no the video  the quality score does not and should not depend on what is important for the application (unless that is best video quality possible under the given constraints). In the reverse applications are compared on what quality they deliver at what costs. 
	Bob, it is arguments like the above that make me feel like you are trying to debate me and are trying to "win" points, instead of a discussion. We can keep doing that, but it will keep leading to situations like this where you end up "painting yourself into a corner".

> 	• the network decides your maximum burst tolerance

	[SM] That is a hard reality and not a mere idea, the existing network puts up quite a number of hard to negotiate with constraints... L4S will simply replace one burst buffer with two probably shallower ones, but it is still going to be the network that controls this. If you overload the network, something will drop, so yes the network limits the "maximum burst tolerance". 

> 	• the network decides that applications can have no more flexibility than a burst tolerance

	[SM] But can they? I would say i a strict FQ system, applications are free to do whatever they want until they reach their instantaneous share limit, just the same as without FQ, the only thing that changes where the "hard" limit is.

> 	• the network decides how much users should prefer bandwidth over latency, and 

	[SM] Sure, but the dual queue coupled AQM's 1ms target for the LL-queue and 15ms target for the non-LL queue seem exactly such  network decisions. Except in the limit the LL-queue will give a paced sender both lower queuing delay and bandwidth superiority over the non-LL queue, giving flows even less choice.

> 	• the network decides one objective video quality metric 

	[SM] SCReAM's goal could be summarized as trying to model the instantaneous network conditions to always send the best achievable video quality. In tracking the networks congestion with high fidelity it can select the best video quality with acceptable side-effects. But note how tracking the network congestion still means that the network decides the video quality...

> ...are from the Bell-head world. I'm not saying Bell-head is wrong. I'm saying it's pointless trying to insist that a Bell-head idea is more correct than a Net-head idea. They are different philosophies.

	[SM] I fail to see a philosophical discussion here, sorry. 

> 
> I prefer to shift the debate

	[SM] I would prefer to have a discussion instead of a debate.

> to how to design a Net-head solution with a configurable degree of Bell-headedness. That is what the combination of the DualQ plus per-flow Queue Protection is. If you wind the queue protection up tight, you emulate per-flow scheduling. If you loosen it off (or disable it, or don't deploy it), you get application freedom. 

	[SM] Or anarchy... But wait, "anything goes" is what we have right now, and people are not all content with the status quo...

> 
> This is also why it's important to enable both DualQ and FQ. Neither are objectively correct. Let the market decide. 

	[SM] The problem with DualQ is that it fails to properly define what it guarantees, per class (as I understood your arguments) or per-flow fairness (as I understood Koen's arguments) but then  fails hard to deliver any guarantee better than the non LL-queue will get at least 1:15 of the available bandwidth. So if the market shall decide, first start with giving the market the information required to judge and compare different offers, which means give a simple description of what dualQ aims to achieve that will allow to easily predit how dualQ will share bandswidth between fllows of different classes.
	And if you want the market to make a meaningful decision the market needs alternative offers. So you could help by making DualQ/L4S not artificially incompatible with alternative 1/p-type congestion controllers like SCE ;) so that there will be a real open contest. Let the market decide rings a tad hollow given your repeated appeals to stop working on alternatives and rally behind the L4S effort...

> [BB] Outside the forest that your mind is in, there is a wider set of forests where:

	[SM] No, as I stated further below in the email you cited , FQ is not producing an OPTIMAL split of bandwidth between active flows, but as I argue, most/all AQM nodes lack the information to objectively optimize anything anyhow. The charm of FQ is that it also is at the same time highly unlikely to create PESSIMAL bandwidth splits and that it is relatively easy to predict reason about. 
	So, I mentioned this non-optimality quite explicitly, so why do you think that I believe that "FQ attempts to optimally split bandwidth"?

> * in other forests, the DualQ Coupled AQM doesn't even attempt to optimally split bandwidth, depending instead on "the application knows that best" (collectively) 

	[SM] Except we already know that applications do not know best, e.g. gamers that download game content via steam complain about choppy experience while using twitch streaming (either in or out) at the same time. CDNs are known to often optimize for expedited delivery (with non-standard means like IW >> 10), which often ist the desired behaviour, but often it is not and leads to reduced bandwidth/increased latency of flows the user values higher. IMHO it is evident that "anything goes" is not a viable route for many users, we have tried that and found it lacking.

> * and in yet other forests (DualQ with configurable per-flow queue protection and policing), there can be various points on a spectrum between the two

	[SM] Both of these addition will nor fix the brokenness of DualQ's core design though... let's admit it, dualQ is broken BY design, as its description is supposed to appease the FQ-leaning crowd, while at the same time not actually giving any FQ-like guarantees; as you are adamantly opposed to FQ on philosophical grounds. This inner conflict is simply too much for dualQ to straddle. 

*) I note that the L4S drafts even reject to require the implementation of these two features, so in all likelihood they will not even be available...

> 
> L4S signalling was designed to support all these forests.
> But according to your ironic insult, one of the architects of this ecosystem of forests only sees trees. There are more forests than the one you are in.

	[SM] You considered that to be an insult? In that case I would like to apologize, this was not intended to offend. Then again, you are returning in kind, so fair is fair.

> [BB] It is neither appropriate nor necessary for the IETF to decide to assign a codepoint in the IP header that only works well in your forest (FQ).

	[SM] You are mostly missing my arguments, are you. My argument, why the L4S should not currently be elevated to experimental RFC status is not that it is not using my preferred hobby-horse technique under the hood. No, my argument is that L4S components as the dual queue coupled AQM as currently implemented fail to meet the L4S projects own goals and requirements. I guess we agree, that the decision to "assign a codepoint in the IP header" should be made in a prudent way, as there is little margin for error, and as I argue, L4S is not at the point where the trade-offs can be realistically predicted.
	But since you keep draging FQ/SCE into the debate, I also bring these up as a examples of solutions that avoid a number of side-effects that L4S suffers from.

> The whole point of L4S was to broaden the ways extremely low latency could be provided (more forests): from FQ to DualQ with QProt+DualQ in between.

	[SM] That IMHO is a red herring, as there are zero applications** that will work with a median queueing delay of 1ms versus the 5ms default fq_codel already delivers in today's internet. And none of the real-time critical applications that one could think of (surgery by remote controlled robot, combustion control of an engine from a remote controller) really are suitable for deployment over the best effort internet.
	I am not saying fq_codel/cake/FQ/... are the end all of all low latency issues, or that reducing median latency below 5ms and/or reducing latency variance/jitter are not worthy goals, but that the considerable side-effects of the L4S approach need to be carefully weighted against the rather modest latency gains it delivers over the state of the art.
	Note, how I am not talking about the high-temporal fidelity 1/p congestion control here? I fully agree that this will be quite a good thing to deploy. I am just not convinced that the current L4S proposals are a good enough solution. 
	And to close the circle, I believe that SCReAM is an interesting example what can be achieved with high-temporal fidelity 1/p congestion control, but it says next to nothing about the low latency component of L4S (there where no competing flows, and SCReAM opted to stay well below link capacity).

Best Regards
	Sebastian

**) Last time I asked I did not get a real answer to what will be additionally made possible by that fivefold reduction in queueing delay.

>>>>>> it is fair to say that these simple simulations should of course be seen as just a
>>>>>> 
>>>>> snapshot.
>>>>> 
>>>>> 	[SM] Fair enough.
>>>>> 
>>>>> 
>>>>>> Meanwhile, the SCReAM code on github is freely available for anyone who
>>>>>>