Subject: [tsvwg] Comments on draft-ietf-tsvwg-l4s-arch-07

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 2 Nov 2020 10:39:00 +0000

Hi

I have read through  draft-ietf-tsvwg-l4s-arch-07 and have a few comments

Section 4 page 11 on Host mechanisms : RFC6679 is listed for RTP, it is however good to add also  
https://tools.ietf.org/wg/avtcore/draft-ietf-avtcore-cc-feedback-message/ , this feedback format is used with SCReAM. cc-feedback-message is BTW listed in the L4S-ID draft.

Section 6.2 :
+ One key benefit that I see with L4S is that it reduces delay jitter and this in turn makes it possible to tune down the dejjitterbuffers on the video streaming client side. The end result should be a better interactive experience for the end users.

+ Another one, that may not be useful. SCReAM implement a transmission buffer on the sending side. This prevents that traffic bloats the network in case of a fast reduction in link throughput. In addition the transmission buffer can be discarded and the video coder can be requested to create a new key frame. This happens occasionally in the experiments with the RC cars that we run because the video coders in the NVIDIA Jetson Nanos are a bit slow to react to reduced target rate. This mechanism should be improved with addition of L4S support as we can then reduce the time that the video playout freezes as the queue buildup in the network is limited.

Nits : 
Section 8.1 Possible grammar… not sure “as best efforts traffic”, always thought it should be “as best effort traffic” ?. But then I am not a professor in the English language 😐

/Ingemar

================================

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

RESEARCHER

GFTL ER NAP NCM Netw Proto & E2E Perf

Labratoriegränd 11

977 53, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked]

www.ericsson.com <http://www.ericsson.com/> 

Talk about a dream, try to make it real

                  Bruce Springsteen

=================================

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 2 Nov 2020 22:09:56 +0000

Ingemar,

On 02/11/2020 10:39, Ingemar Johansson S wrote:
> https://tools.ietf.org/wg/avtcore/draft-ietf-avtcore-cc-feedback-message/ 
> , this feedback format is used with SCReAM. cc-feedback-message is BTW 
> listed in the L4S-ID draft.
>

Feedback protocols are discussed in the bullet below the one you are 
looking at. I think RTP is still the right thing to reference to 
describe the main protocol (that has things attached to it like 
congestion controls, feedback protocols, etc).

Nonetheless, you are right if we're talking about the following bullet, 
where it says RTCP is sufficient for L4S feedback. That was obviously 
written there a long time ago, and it needs to be updated to recommend 
using avtcore-cc-feedback-message.

Assuming you confirm I'm correct, I'll do that (before tonight's 
deadline, if you get back to me soon enough).

> Section 6.2 :
> + One key benefit that I see with L4S is that it reduces delay jitter 
> and this in turn makes it possible to tune down the dejjitterbuffers 
> on the video streaming client side. The end result should be a better 
> interactive experience for the end users.
>

Added. Thx
>
> + Another one, that may not be useful. SCReAM implement a transmission 
> buffer on the sending side. This prevents that traffic bloats the 
> network in case of a fast reduction in link throughput. In addition 
> the transmission buffer can be discarded and the video coder can be 
> requested to create a new key frame. This happens occasionally in the 
> experiments with the RC cars that we run because the video coders in 
> the NVIDIA Jetson Nanos are a bit slow to react to reduced target 
> rate. This mechanism should be improved with addition of L4S support 
> as we can then reduce the time that the video playout freezes as the 
> queue buildup in the network is limited.
>

I'm not quite sure about this.
* If there is a certain standing queue in the network, at a sudden 
reduction in capacity, the network queue will grow by a certain amount.
* If the standing queue is less because of L4S, if there is the same 
sudden reduction in capacity, the queue will grow by the same amount.
In both cases, to get the network queue back to where it was, the sender 
would have to dump the same amount from its send buffer.
So the send buffer can't be any smaller, just because the network queue 
is smaller.

What am I misunderstanding?

I can see you could reduce the send buffer if you used a virtual queue 
in the network to keep some spare headroom. But without any headroom, I 
don't get what you mean.

> Nits :
> Section 8.1 Possible grammar… not sure “as best efforts traffic”, 
> always thought it should be “as best effort traffic” ?. But then I am 
> not a professor in the English language 😐
>

I usually say best efforts. Not sure whether it's any more correct than 
best effort. I just looked it up. It seems to have come from contract 
law. And usually it's written best efforts, but sometimes best effort. 
I'll leave it.

Thanks for the review (again).

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 3 Nov 2020 08:10:03 +0000

Hi Bob

Please see inline tagged [IJ]

From: Bob Briscoe <[email-masked]> 
Sent: den 2 november 2020 23:10
To: Ingemar Johansson S <[email-masked]>
Cc: De Schepper, Koen (Koen) <[email-masked]>; Greg White <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07

Ingemar,

Hi

Feedback protocols are discussed in the bullet below the one you are looking at. I think RTP is still the right thing to reference to describe the main protocol (that has things attached to it like congestion controls, feedback protocols, etc).

Nonetheless, you are right if we're talking about the following bullet, where it says RTCP is sufficient for L4S feedback. That was obviously written there a long time ago, and it needs to be updated to recommend using avtcore-cc-feedback-message.

Assuming you confirm I'm correct, I'll do that (before tonight's deadline, if you get back to me soon enough).
[IJ] Guess I was a bit late, but no big deal, this is jus a minor. I understand now that bullet b refers to the negotiation of the ECT(1) feedback. I believe though that a reference to https://tools.ietf.org/wg/avtcore/draft-ietf-avtcore-cc-feedback-message/ is sufficient as it itself rerfers to RFC66679 on how to negotiate ECN.

Section 6.2 :

Added. Thx

I'm not quite sure about this. 
* If there is a certain standing queue in the network, at a sudden reduction in capacity, the network queue will grow by a certain amount. 
* If the standing queue is less because of L4S, if there is the same sudden reduction in capacity, the queue will grow by the same amount. 
In both cases, to get the network queue back to where it was, the sender would have to dump the same amount from its send buffer.
So the send buffer can't be any smaller, just because the network queue is smaller.

I can see you could reduce the send buffer if you used a virtual queue in the network to keep some spare headroom. But without any headroom, I don't get what you mean.

[IJ] I agree to some degree. I was more referring to the case where L4S marking is done based on actual (low) queue buildup for instance on the PDCP or RLC layer in a 5G radio base station. This was simulated in the master thesis by Davide Brunello. What we have seen is that with L4S marking and the use of off the shelf video encoders that produce frames that vary in size, the nominal bitrate becomes pushed down a bit to accommodate enough headroom to ensure transmission of the largest frames without large queue buildup in the network. The additional headroom often (but not always) gives less data in flight than the case when L4S is not enabled and therefore a reduced queue build up when link throughput drops. All this can of course be claimed to be specific to how SCReAM behaves when it runs with input from video encoders. Perhaps it also becomes too complex to explain in a comprehensible way. And, true , with a virtual queue the story becomes different but it becomes hard to rate control a media stream over a virtual queue without any form of network feedback.
I attached two graphs for the case that throughput drops from 40 down to 20Mbps in 0.3s (simple bottleneck, no 5G), to illustrate this case. I have set the queue delay target quite low in this case (10ms) to give at least a somewhat fair comparions, in reality this is not realistic as SCReAM would false trigger on scheduling jitter, process jitter, clock drift and what not. In any case it should show that L4S for this case gives a lower delay spike when L4S is enabled in the bottleneck.

Not sure that this is possible to follow … 

 

Nits : 

I usually say best efforts. Not sure whether it's any more correct than best effort. I just looked it up. It seems to have come from contract law. And usually it's written best efforts, but sometimes best effort. I'll leave it.

[IJ] OK 😊

Bob

/Ingemar

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

RESEARCHER

Labratoriegränd 11

977 53, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked] <mailto:[email-masked]> 

                  Bruce Springsteen

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/ <https://protect2.fireeye.com/v1/url?k=00a2330a-5f390a49-00a27391-861fcb972bfc-d186ea0fb30cda50&q=1&e=5e345298-09e4-44b1-ad7b-e7973feacef2&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 4 Nov 2020 22:48:42 +0000

Ingemar,

On 03/11/2020 08:10, Ingemar Johansson S wrote:
> *From:* Bob Briscoe <[email-masked]>
> *Sent:* den 2 november 2020 23:10
> *To:* Ingemar Johansson S <[email-masked]>
> *Cc:* De Schepper, Koen (Koen) <[email-masked]>; Greg 
> White <[email-masked]>; [email-masked]; [email-masked]
> *Subject:* Re: Comments on draft-ietf-tsvwg-l4s-arch-07
>
> Ingemar,
>
[snip]
>
>     + Another one, that may not be useful. SCReAM implement a
>     transmission buffer on the sending side. This prevents that
>     traffic bloats the network in case of a fast reduction in link
>     throughput. In addition the transmission buffer can be discarded
>     and the video coder can be requested to create a new key frame.
>     This happens occasionally in the experiments with the RC cars that
>     we run because the video coders in the NVIDIA Jetson Nanos are a
>     bit slow to react to reduced target rate. This mechanism should be
>     improved with addition of L4S support as we can then reduce the
>     time that the video playout freezes as the queue buildup in the
>     network is limited.
>
>
> In both cases, to get the network queue back to where it was, the 
> sender would have to dump the same amount from its send buffer.
> in the network to keep some spare headroom. But without any headroom, 
> I don't get what you mean.
>
> [IJ] I agree to some degree. I was more referring to the case where 
> L4S marking is done based on actual (low) queue buildup for instance 
> on the PDCP or RLC layer in a 5G radio base station. This was 
> simulated in the master thesis by Davide Brunello. What we have seen 
> is that with L4S marking and the use of off the shelf video encoders 
> that produce frames that vary in size, the nominal bitrate becomes 
> pushed down a bit to accommodate enough headroom to ensure 
> transmission of the largest frames without large queue buildup in the 
> network. The additional headroom often (but not always) gives less 
> data in flight than the case when L4S is not enabled and therefore a 
> reduced queue build up when link throughput drops. All this can of 
> course be claimed to be specific to how SCReAM behaves when it runs 
> with input from video encoders. Perhaps it also becomes too complex to 
> explain in a comprehensible way. And, true , with a virtual queue the 
> story becomes different but it becomes hard to rate control a media 
> stream over a virtual queue without any form of network feedback.
> I attached two graphs for the case that throughput drops from 40 down 
> to 20Mbps in 0.3s (simple bottleneck, no 5G), to illustrate this case. 
> I have set the queue delay target quite low in this case (10ms) to 
> give at least a somewhat fair comparions, in reality this is not 
> realistic as SCReAM would false trigger on scheduling jitter, process 
> jitter, clock drift and what not. In any case it should show that L4S 
> for this case gives a lower delay spike when L4S is enabled in the 
> bottleneck.
>

[BB] In your scenario when radio capacity reduces, I've just had it 
explained to me (off list) why using L4S allows you to have a shorter 
send buffer.

Let's say there's a stable flow, then bottleneck capacity suddenly 
reduces to say 1/3 of what it was. Eventually the flow will stabilize to 
run 1/3 as fast. But the question is, what happens between times. Let's 
come back to the under-capacity case later, and for now focus on the 
well-known case of an AQM configured for a standing queue set for a 
certain target delay in both the starting and ending steady states. 
We'll compare the two well-known variants of this: L4S vs Classic, with 
the target being shallower in the L4S case.

When capacity reduces to 1/3, the *existing* queue takes 3 times longer 
to drain through the bottleneck. So, for the same capacity reduction, if 
you start with a shallower queue (as in the L4S case), you get less 
additional queuing delay (not the same additional queue, which I wrongly 
said) and the additional queue gets back to the target more quickly. 
There is the same amount of data in flight in both cases, it's only the 
smaller amount of data standing in the queue that makes the difference.

In both cases, when the bump of feedback reaches the sender after a 
round trip, the sender will eventually have to stabilize at 1/3 of its 
original rate. Nonetheless, to move between the two quickly, you are 
saying some senders hold a 'sacrificial' send buffer that they can 
mostly ditch if congestion feedback implies that capacity has reduced. 
As explained above, in the L4S case, there's less to ditch over less 
time. So the sender's standing send buffer can be smaller as well. In 
contrast, in the Classic case, there's more queue to drain away for 
longer. So it will probably have to stall to allow this queue to drain 
out of the way before it can get back to a steady state at the lower rate.

So, I get it now.

I think the discussion about leaving headroom was rather a red-herring. 
ECN can be used to keep a flow running with some headroom under the link 
capacity, either:
i) the higher rate periods during larger video frame sizes bump up 
against the shallow ECN threshold, while the average rate runs below the 
capacity;
ii) or the AQM uses a virtual queue to leave some headroom

However, I don't think that's specific to L4S ECN. An AQM can be 
configured to under-run with any form of ECN (L4S or Classic). If a 
Classic AQM is configured to underrun by the same amount as an L4S AQM, 
there won't really be an inherent difference between the two if capacity 
suddenly reduces. A queue will probably build during the transition in 
both cases. Both will have had the same amount of data in flight. 
Neither will aim to have a standing queue at the start or end. So both 
ought to need the sender to shed the same amount of data to clear the 
same temporary queue.

By definition, the Classic queue will vary with greater amplitude, which 
will make it more difficult to get the AQM to run at a small degree of 
underrun without also queuing. That would make a difference. But it's 
not as straightforward as the standing queue cases.

Finally, I'm not sure whether I can face the prospect of trying to 
explain all this in the use-cases section of l4s-arch. I could just 
write that a further advantage of L4S is that the sender doesn't need to 
buffer as much data in case it needs to sacrifice it during a reduction 
in capacity. And not even try to explain why. But I'd only want to do 
that if you have some published experiments I can refer to as well.

What do you think?

Cheers

Bob

>     <mailto:[email-masked]>
>
> Bob Briscoehttp://bobbriscoe.net/  <https://protect2.fireeye.com/v1/url?k=00a2330a-5f390a49-00a27391-861fcb972bfc-d186ea0fb30cda50&q=1&e=5e345298-09e4-44b1-ad7b-e7973feacef2&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 5 Nov 2020 07:42:24 +0000

Hi

Please inline [IJ2]

/Ingemar

Sent: den 4 november 2020 23:49

Ingemar,

Hi Bob

From: Bob Briscoe  <mailto:[email-masked]> <[email-masked]> 
Sent: den 2 november 2020 23:10
To: Ingemar Johansson S  <mailto:[email-masked]> <[email-masked]>
Cc: De Schepper, Koen (Koen)  <mailto:[email-masked]> <[email-masked]>; Greg White  <mailto:[email-masked]> <[email-masked]>; [email-masked] <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> 
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07

Ingemar,

Hi

[snip]

[BB] In your scenario when radio capacity reduces, I've just had it explained to me (off list) why using L4S allows you to have a shorter send buffer. 

Let's say there's a stable flow, then bottleneck capacity suddenly reduces to say 1/3 of what it was. Eventually the flow will stabilize to run 1/3 as fast. But the question is, what happens between times. Let's come back to the under-capacity case later, and for now focus on the well-known case of an AQM configured for a standing queue set for a certain target delay in both the starting and ending steady states. We'll compare the two well-known variants of this: L4S vs Classic, with the target being shallower in the L4S case.

When capacity reduces to 1/3, the *existing* queue takes 3 times longer to drain through the bottleneck. So, for the same capacity reduction, if you start with a shallower queue (as in the L4S case), you get less additional queuing delay (not the same additional queue, which I wrongly said) and the additional queue gets back to the target more quickly. There is the same amount of data in flight in both cases, it's only the smaller amount of data standing in the queue that makes the difference.

[IJ2] Yes, correct

In both cases, when the bump of feedback reaches the sender after a round trip, the sender will eventually have to stabilize at 1/3 of its original rate. Nonetheless, to move between the two quickly, you are saying some senders hold a 'sacrificial' send buffer that they can mostly ditch if congestion feedback implies that capacity has reduced. As explained above, in the L4S case, there's less to ditch over less time. So the sender's standing send buffer can be smaller as well. In contrast, in the Classic case, there's more queue to drain away for longer. So it will probably have to stall to allow this queue to drain out of the way before it can get back to a steady state at the lower rate.

[IJ2] Admittedly  quite specific to SCReAM and how it is designed. The approach with a sender queue has proven beneficial in our RC-Car experiments, because off the shelf video coders can have a slow internal rate control loop. In addition there are occasionally issues with link adaptation that causes transmission to halt for a short while when radio conditions are less optimal. The latter issue will be solved over time but that is in any case something that had to be addressed to make the RC-Car drive well even in bad radio. Nonetheless, it is a SCReAM specific approach.

So, I get it now.

I think the discussion about leaving headroom was rather a red-herring. ECN can be used to keep a flow running with some headroom under the link capacity, either:
i) the higher rate periods during larger video frame sizes bump up against the shallow ECN threshold, while the average rate runs below the capacity;
ii) or the AQM uses a virtual queue to leave some headroom

However, I don't think that's specific to L4S ECN. An AQM can be configured to under-run with any form of ECN (L4S or Classic). If a Classic AQM is configured to underrun by the same amount as an L4S AQM, there won't really be an inherent difference between the two if capacity suddenly reduces. A queue will probably build during the transition in both cases. Both will have had the same amount of data in flight. Neither will aim to have a standing queue at the start or end. So both ought to need the sender to shed the same amount of data to clear the same temporary queue.

By definition, the Classic queue will vary with greater amplitude, which will make it more difficult to get the AQM to run at a small degree of underrun without also queuing. That would make a difference. But it's not as straightforward as the standing queue cases.

Finally, I'm not sure whether I can face the prospect of trying to explain all this in the use-cases section of l4s-arch. I could just write that a further advantage of L4S is that the sender doesn't need to buffer as much data in case it needs to sacrifice it during a reduction in capacity. And not even try to explain why. But I'd only want to do that if you have some published experiments I can refer to as well. 

What do you think?

[IJ2] I think that it is reasonable to skip this, I don’t have any good external reference that explains this and it takes quite some text on paper to explain it properly.

Cheers

Bob

Bob

/Ingemar

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

RESEARCHER

Labratoriegränd 11

977 53, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

                  Bruce Springsteen

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/ <https://protect2.fireeye.com/v1/url?k=1e4a2e13-41d11715-1e4a6e88-86ee86bd5107-03576f53158fd973&q=1&e=c6fff227-abf0-416b-9a1e-f0a6da83bc3b&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 5 Nov 2020 10:20:11 +0000

One other good side effect is that if traffic is more bursty or fast on/off (like per video frame), L4S will give priority to latency instead of throughput, so rather underutilize than keeping the link full (depending on the level of sender pacing, leave space between video frames in this example). This is an additional advantage in case the throughput reduces, as it leaves room between frames to clear-up queues faster, instead of accumulating all the excess rate in the queue (causing the latency spike).

Indeed, not sure if and how we can explain this. Anyway, I think an important aspect of having L4S and Classic in parallel is that we have 2 subclasses in best effort going forward: Classic = optimized for Throughput, willing to sacrifice Latency if needed; L4S = optimized for Latency and willing to sacrifice Throughput if needed. The “if needed” part means if conditions are present where having both is impossible (like rapidly changing throughputs, lots of dynamic traffic). If agreed, this might be a simpler message and important design principle to incorporate in the text? It is how it is working at the moment, so maybe better to explicitly mention this at some point.

Koen.

From: Ingemar Johansson S <[email-masked]>
Sent: Thursday, November 5, 2020 8:42 AM
To: Bob Briscoe <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; Greg White <[email-masked]>; Marcelo Bagnulo <[email-masked]>; [email-masked]; Ingemar Johansson S <[email-masked]>
Subject: RE: Comments on draft-ietf-tsvwg-l4s-arch-07

Hi
Please inline [IJ2]

/Ingemar

From: Bob Briscoe <[email-masked]<mailto:[email-masked]>>
Sent: den 4 november 2020 23:49
To: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Cc: De Schepper, Koen (Koen) <[email-masked]<mailto:[email-masked]>>; Greg White <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07

Ingemar,

From: Bob Briscoe <[email-masked]><mailto:[email-masked]>
Sent: den 2 november 2020 23:10
To: Ingemar Johansson S <[email-masked]><mailto:[email-masked]>
Cc: De Schepper, Koen (Koen) <[email-masked]><mailto:[email-masked]>; Greg White <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07

Ingemar,

So, I get it now.

What do you think?

Cheers

Bob

Bob

/Ingemar

Ericsson Research
RESEARCHER
GFTL ER NAP NCM Netw Proto & E2E Perf
Labratoriegränd 11
977 53, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com<http://www.ericsson.com/>

--

Bob Briscoe                               http://bobbriscoe.net/<https://protect2.fireeye.com/v1/url?k=00a2330a-5f390a49-00a27391-861fcb972bfc-d186ea0fb30cda50&q=1&e=5e345298-09e4-44b1-ad7b-e7973feacef2&u=http%3A%2F%2Fbobbriscoe.net%2F>

--

Bob Briscoe                               http://bobbriscoe.net/<https://protect2.fireeye.com/v1/url?k=1e4a2e13-41d11715-1e4a6e88-86ee86bd5107-03576f53158fd973&q=1&e=c6fff227-abf0-416b-9a1e-f0a6da83bc3b&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 5 Nov 2020 11:25:10 +0100

Hi Bob,

more below in-line, prefixed [SM].

Gesendet: Mittwoch, 04. November 2020 um 23:48 Uhr
Von: "Bob Briscoe" <[email-masked]>
An: "Ingemar Johansson S" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>
Betreff: Re: [tsvwg] Comments on draft-ietf-tsvwg-l4s-arch-07
Ingemar,

From: Bob Briscoe <[email-masked]>[mailto:[email-masked]]
Sent: den 2 november 2020 23:10
To: Ingemar Johansson S <[email-masked]>[mailto:[email-masked]]
Cc: De Schepper, Koen (Koen) <[email-masked]>[mailto:[email-masked]]; Greg White <[email-masked]>[mailto:[email-masked]]; [email-masked][mailto:[email-masked]]; [email-masked][mailto:[email-masked]]
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07
 
Ingemar,

 

 

        [SM] A send buffer on the traffic sender that is not also in control of the bottleneck queue, needs to be spaced based on the expected ingress burst rate, and the expected egress rate, assuming its purpose is to avoid the link7connection falling idle/missing transmit slots. It is unclear how the choice of bottleneck AQM will affect any of these quantities in a manor easy to detect by the sender. Sure, the sender will now whether L4S was negotiated for a link, but can not be sure what type of bottleneck of flow will actually encounter, at least in the general sense of a flow traversing the internet. So even if L4S would allow you to get away with a smaller buffer (by e.g. reducing the burstiness of buffer ingress) no flow is guaranteed to always stay in L4S mode (think packet drops making the L4S-CC revert to Reno-compatibility mode, potentially even from a massively overloaded L4S AQM, because say the egress rate of that AQM just dropped by 66%, but all inflight data still comes in at the original rate), no?

When capacity reduces to 1/3, the *existing* queue takes 3 times longer to drain through the bottleneck.

        [SM] Depending of the amount of data n flight at the original 100% rate these queues will actually take longer than 3 times to drain as you still have to deal with a situation where the ingress stays above the egress for approximately >= 1 RTT, no? Which, if RTT is high enough might be enough to drive your L4S AQM into overload protection mode (which I understand from the Destruction... thesis is considered mandatory), no?

 So, for the same capacity reduction, if you start with a shallower queue (as in the L4S case), you get less additional queuing delay (not the same additional queue, which I wrongly said) and the additional queue gets back to the target more quickly. There is the same amount of data in flight in both cases, it's only the smaller amount of data standing in the queue that makes the difference.

        [SM] the standing queue for the shallow queue is 1ms, for the deep queue is 15ms, so as with the RTT bias issue, for short RTTs the base RTT will make a big difference, for longer RTTs that difference will get less important depending on the size of the transiently grown queue, no?

In both cases, when the bump of feedback reaches the sender after a round trip, the sender will eventually have to stabilize at 1/3 of its original rate. Nonetheless, to move between the two quickly, you are saying some senders hold a 'sacrificial' send buffer that they can mostly ditch if congestion feedback implies that capacity has reduced. 

        [SM] At which time at best the one-way delay from receiver to sender has passed (assuming AQM node in the receiver, nut realistic but certainly a valid theoretical lower bound for the signalling delay), now discarding the existing send buffer obviously only makes sense when the expected rate fluctuations on average are slower that the signalling delay...

As explained above, in the L4S case, there's less to ditch over less time. So the sender's standing send buffer can be smaller as well. 

        [SM] Could you explain how you envision that the sender actually decides how to size the send buffer, please? I still fail to see how the L4S-ness of the path will have actionable consequences for the sender in the generic case.

In contrast, in the Classic case, there's more queue to drain away for longer. So it will probably have to stall to allow this queue to drain out of the way before it can get back to a steady state at the lower rate.

        [SM] Is there any testing data available showing what happens to the two queues on sudden rate reduction steps? This again is one of the issues that would be best addressed by actual data to confirm and support any hypothesis/prediction.

So, I get it now.

        [SM] But that requires that all flows use ECN, no?

        [SM] Which effectively means a rather costly traffic shaper.

However, I don't think that's specific to L4S ECN. An AQM can be configured to under-run with any form of ECN (L4S or Classic). If a Classic AQM is configured to underrun by the same amount as an L4S AQM, there won't really be an inherent difference between the two if capacity suddenly reduces. A queue will probably build during the transition in both cases. Both will have had the same amount of data in flight. Neither will aim to have a standing queue at the start or end. 

        [SM] I thought al AQM effectively allow some standing queue as the alternative is to sacrifice bandwidth? Sure the amount of standibg queue differs, but it is always non-zero?

So both ought to need the sender to shed the same amount of data to clear the same temporary queue.

        [SM] +1; only add this when there is empirical data (preferably from real world mesurements over the existing internet) that confirm this prediction.

Best Regards
        Sebastian

What do you think?

Cheers

Bob

Bob

/Ingemar
 
Ericsson Research
RESEARCHER
GFTL ER NAP NCM Netw Proto & E2E Perf
Labratoriegränd 11
977 53, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked][mailto:[email-masked]]
www.ericsson.com[http://www.ericsson.com/]

 
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/[https://protect2.fireeye.com/v1/url?k=00a2330a-5f390a49-00a27391-861fcb972bfc-d186ea0fb30cda50&q=1&e=5e345298-09e4-44b1-ad7b-e7973feacef2&u=http%3A%2F%2Fbobbriscoe.net%2F]
 
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/[http://bobbriscoe.net/]

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 5 Nov 2020 11:38:46 +0100

Hi Koen,

Gesendet: Donnerstag, 05. November 2020 um 11:20 Uhr
Von: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
An: "Ingemar Johansson S" <[email-masked]>, "Bob Briscoe" <[email-masked]>
Cc: "Marcelo Bagnulo" <[email-masked]>, "[email-masked]" <[email-masked]>
Betreff: Re: [tsvwg] Comments on draft-ietf-tsvwg-l4s-arch-07

Indeed, not sure if and how we can explain this. Anyway, I think an important aspect of having L4S and Classic in parallel is that we have 2 subclasses in best effort going forward: Classic = optimized for Throughput, willing to sacrifice Latency if needed; L4S = optimized for Latency and willing to sacrifice Throughput if needed.

        [SM] Make sure to explicitly add this explanation to the drafts though, as that is decidedly different from the claim "This new class of congestion control can provide low latency for capacity-seeking flows, so applications can achieve both high bandwidth and low latency."
        Also this does not ft to the core of the DualQ where a strict priority scheduler gives the shallow queue 15/16 of the throughput as conmpared to the deep queue, meaning that if push comes to shove the deep queue will not get preferential throughput, but both reduced throughput and increased latency.

 The “if needed” part means if conditions are present where having both is impossible (like rapidly changing throughputs, lots of dynamic traffic). If agreed, this might be a simpler message and important design principle to incorporate in the text? It is how it is working at the moment, so maybe better to explicitly mention this at some point.

       [SM] By all means yes. It is a bit disconcerting that you seem to imply that currently the internets drafts are not a correct description of the core functionality of L4S though. Then again, I have been reporting differences between the claims in the drafts and what the implementation & design deliver for a long period, so I will not complain if you finally agree.

Best Regards
        Sebastian

Koen.

Hi
Please inline [IJ2]
 
/Ingemar

From: Bob Briscoe <[email-masked][mailto:[email-masked]]>
Sent: den 4 november 2020 23:49
To: Ingemar Johansson S <[email-masked][mailto:[email-masked]]>
Cc: De Schepper, Koen (Koen) <[email-masked][mailto:[email-masked]]>; Greg White <[email-masked][mailto:[email-masked]]>; [email-masked][mailto:[email-masked]]; [email-masked][mailto:[email-masked]]
Subject: Re: Comments on draft-ietf-tsvwg-l4s-arch-07
 
Ingemar,

 
Ingemar,

So, I get it now.

What do you think?

Cheers

Bob

Bob

/Ingemar
 
Ericsson Research
RESEARCHER

 
 
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/[https://protect2.fireeye.com/v1/url?k=1e4a2e13-41d11715-1e4a6e88-86ee86bd5107-03576f53158fd973&q=1&e=c6fff227-abf0-416b-9a1e-f0a6da83bc3b&u=http%3A%2F%2Fbobbriscoe.net%2F]

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 5 Nov 2020 14:24:29 +0000

Sebastian,

On 05/11/2020 10:25, Sebastian Moeller wrote:
>          [SM] A send buffer on the traffic sender that is not also in control of the bottleneck queue, needs to be spaced based on the expected ingress burst rate, and the expected egress rate, assuming its purpose is to avoid the link7connection falling idle/missing transmit slots.

[BB] Why? You seem to be applying knowledge about the Internet that came 
with assumption-baggage that is being questioned here.

When we have to consider both latency and throughput, Koen just 
articulated well the different approach behind DCTCP (and earlier 
approaches based on virtual queues):
* When conditions are stable, keep buffers just full enough so that data 
is there to exploit most transmit slots.
* But the more conditions are unstable, the more that any long-running 
flows leave headroom *under* the available capacity they are detecting, 
erring on the side of missing a few transmit slots sometimes. Otherwise, 
the goal of trying to exploit every transmit slot necessarily leads to 
bigger queues whenever capacity reduces.

We can talk next about the merits of buffering in the network vs at the 
sender, but let's take this one step at a time.

> It is unclear how the choice of bottleneck AQM will affect any of these quantities in a manor easy to detect by the sender. Sure, the sender will now whether L4S was negotiated for a link, but can not be sure what type of bottleneck of flow will actually encounter, at least in the general sense of a flow traversing the internet. So even if L4S would allow you to get away with a smaller buffer (by e.g. reducing the burstiness of buffer ingress) no flow is guaranteed to always stay in L4S mode (think packet drops making the L4S-CC revert to Reno-compatibility mode, potentially even from a massively overloaded L4S AQM, because say the egress rate of that AQM just dropped by 66%, but all inflight data still comes in at the original rate), no?

[BB] Of course. But if the sender is in a mode where it believes ECN 
marking is from an L4S bottleneck, and it's not responding to any drop 
in the last RTT, we still have to be allowed space on this list to talk 
about what the sender ought to do.

>          [SM] Depending of the amount of data n flight at the original 100% rate these queues will actually take longer than 3 times to drain as you still have to deal with a situation where the in…

[BB] That's why I highlighted *existing* queue. Meaning the packet at 
the tail of the *existing* queue will reach the head 3 times later. Yes, 
in-flight data piling into the tail of the queue will indeed also drain 
more slowly, so the whole queue will take longer to clear. But the 
effect of in-flight data is the same whether the original queue was 
shallow (L4S) or not (Classic). You can see I said that 2 sentences 
further on.

>          [SM] the standing queue for the shallow queue is 1ms, for the deep queue is 15ms, so as with the RTT bias issue, for short RTTs the base RTT will make a big difference, for longer RTTs tha…

[BB] I think you're assuming a window-based CC with pure 
window-fairness, and no logic to remove RTT bias. In that case, because 
the packet rate r that results from a certain window W is r = W/R, where 
R is the RTT, then certainly the smaller queue will lead to a higher 
packet rate for the same window.

But we're talking here about video, which can be rate equation based. If 
the rate equation was written to precisely mimic a pure classical 
window-based CC, you'd be right. But why would anyone do that? With 
equation-based control, the developer has to do extra monitoring of the 
RTT and include more calculation just to mimic the RTT bias of Reno. 
With equation-based control, if no RTT bias is needed, that's the easy case.

>          [SM] At which time at best the one-way delay from receiver to sender has passed (assuming AQM node in the receiver, nut realistic but certainly a valid theoretical lower bound for the sign…

[BB] It's not quite clear precisely what you're trying to say here. But 
I can sort-of agree that the whole context of this discussion is that 
the reduction in capacity has persisted long enough to need to shift 
down a gear. However, the sender might want to start that process during 
the RTT after congestion signalling started to rise. So it doesn't have 
to do nothing about the data it is buffering for *another* RTT after it 
starts to see more congestion signals.

BTW, in case it's not clear, when I talk about ditching the send buffer, 
the sender might replace that buffered data with different (very low 
quality) video frame(s) to avoid a complete discontinuity in the video, 
before getting back on track with frames encoded for 1/3 the original 
bit rate.

>          [SM] Could you explain how you envision that the sender actually decides how to size the send buffer, please? I still fail to see how the L4S-ness of the path will have actionable conseque…

[BB] Why do you need this info? I ask because you are very demanding on 
everyone's time, but we don't see anything useful in return.

>          [SM] Is there any testing data available showing what happens to the two queues on sudden rate reduction steps? This again is one of the issues that would be best addressed by actual data…

[BB] Presumably you saw the plots that Ingemar sent? 3 Nov.
The idea here is that, if you're interested, you contribute data 
yourself as well. It's not the proper etiquette to continually ask 
others to do work for you without contributing at least as much yourself.

>          [SM] But that requires that all flows use ECN, no?

[BB] Yes. And certainly with Classic ECN there is no mechanism to ensure 
that happens. Whereas the aim with L4S ECN is isolation from non-L4S-ECN 
(whether with DualQ or FQ).

Even so, in an Internet access scenario, one period of activity 
involving a collection of flows between the same endpoints rarely 
coincides with another. Because, even where a number of people (in the 
same family or small business) share an access link, the link is 
*typically* only utilized x% even in the 'peak hour', where x is in the 
low single digits. Mostly this is because it approaches fully utilized 
x% of the time and quiet the rest. Where this is the case, there's low 
likelihood of activity from different flows coinciding.

>          [SM] Which effectively means a rather costly traffic shaper.

[BB] Eh? A virtual queue is extremely simple. It's just a number 
representing the length of the virtual queue. On enqueue of each packet 
to the real queue, the virtual queue is incremented by the packet size. 
And it's decremented at a rate slightly slower than the real link rate 
C, e.g. 99% * C. So a vq indicates what the queue would be if the link 
were slightly slower. Then ECN marking based on the length of the 
virtual queue tends to keep the real link running at (say) 99% of capacity.

>          [SM] I thought al AQM effectively allow some standing queue as the alternative is to sacrifice bandwidth? Sure the amount of standibg queue differs, but it is always non-zero?

[BB] The world isn't always what you thought it was.

I divided this email into the case with different size standing queues 
first, then "leaving headroom" second.
Search for the sentence "leaving headroom was rather a red-herring".

This whole conversation is about how DCTCP specifically (and L4S more 
generally) maintains a small standing queue under stable conditions, but 
automatically makes more capacity available in environments where 
available capacity is less stable  (in this conversation we've been 
focusing on a reduction of *actual* capacity, but *available* capacity 
can also reduce during an increase in the load from short flows, which 
was the motivating traffic mix in the original DCTCP paper).

>          [SM] +1; only add this when there is empirical data (preferably from real world mesurements over the existing internet) that confirm this prediction.

[BB] You are the main person (by a large margin) holding up hundreds of 
people in dozens of companies who are trying to get the IETF to approve 
the L4S experiment, so that we can get measurements like this over the 
real Internet.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 5 Nov 2020 16:32:23 +0100

Hi Bob,

more below in-line prefixed with [SM2].

> Gesendet: Donnerstag, 05. November 2020 um 15:24 Uhr
> An: "Sebastian Moeller" <[email-masked]>
> Cc: "Ingemar Johansson S" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>
> Betreff: Re: Aw: Re: [tsvwg] Comments on draft-ietf-tsvwg-l4s-arch-07
[...]
> bigger queues whenever capacity reduces.

        [SM2] Okay, so you are not talking about capacity seeking flows then?

> about what the sender ought to do.

        [SM2] I believe that is what we are doing right now, talking about it, nobody tried to restrict that discussion, no? I am confused a drop on RTT would mean something got faster, but I think the something gets slower case is more relevant as it will potentially cause more issues, or are you talking about an increase in RTT here?

> window-fairness, and no logic to remove RTT bias.

        [SM2] Well, you have neither mentioned that, nor IMHO proposed a method to remove more RTT bias than the broken DualQ artificially foists on flows in the deep queue. So let's ignore RTT bias for now, please.

 In that case, because 
> packet rate for the same window.

        [SM2] Yes, but the question is how much difference the additional 14ms of the deep queue make, e.g. on a 10ms Path that is going to be noticeable, but on a 100ms path much less so.

> [BB] It's not quite clear precisely what you're trying to say here.

        [SM2] Ah, sorry, if you have a sacrifical buffer, you still want to know whether to drop or to send it, if bottleneck share of a flow fluctuates slower than the signalling delay, than drop on early signs of overload and reduce rate is a winning strategy, but if rate fluctuations are considerably higher that becomes a less clear choice.

> But 
> down a gear. 

        [SM2] In that case I agree that dropping seems reasonable, after all the video consumer at the other end will not want to live with all accumulated delay and probably will drop frames (or transiently reduce frame period) to get back to displaying the most recent frame ASAP (while honouring the requested frame rate).

> However, the sender might want to start that process during 

        [SM] TO be able to do this the sender process needs to also buffer the raw frame for the full duration it is supposed to allow re-encodings, and there needs to be a frame by frame method to create lower quality versions of the frame (or rather information, not all codecs produce full independent frames)

> everyone's time, but we don't see anything useful in return.

         [SM] Because you made an assertion, that I do not understand, and you claim that this is a novel advantage of L4S, and it might make sense to go the long distance and think things through before adding them to the drafts?

> others to do work for you without contributing at least as much yourself.

        [SM] Mmmh I had thought, you would have already had a mental model that you could quickly describe here? And let's not go down the route of asking me to do your homework again, shall we? It is not me who wants the IETF to hand over control of the ECT(1) codepoint, and so I do not need to describe cases in which that codepoint can be put to good uses to convince the IETF. 

> likelihood of activity from different flows coinciding.

        [SM] With long downloads and concurrent UHD streaming I would not be so sure that this assertion holds during the peak hours (I base this on what I learnd from supporting SQM users on the OpenWrt mailing list, anecdotal, sure, but based on real reports).

> And it's decremented at a rate slightly slower than the real link rate 

        [SM] E voila you have a traffic shaper, exactly what I said. The costly part tends to be "the decremented at a rate slightly slower", as you need to measure time, no?

> [BB] The world isn't always what you thought it was.

        [SM] Sure that is why I ask questions, to learn. But responses like this tell me you don't know either, fair enough.

> the L4S experiment, so that we can get measurements like this over the 

        [SM] Bob, you are surely trying flattery right now... But honestly, I have been asking for measurements (in accordance with https://tools.ietf.org/html/rfc5033 guidelines) for some years now, if you only started do measure this because my objections are looking like creating a road-block*, you a) did obviously ignore rfc5033 and b) started way to late. The point, is to perform such initial tests to assess safety over the internet you do not need an experimental track RFC to be published before hand. 
        If you insist that you require experimental track RFCs before the experiment, the least you would need to do is turn L4S into a real experiment, with a hypotheses, proposed measurements and a set of criteria to assess success or failure at the end, and given that you lay claim to a rare IP-header codepoint, a realistic description how to undo L4S roll-out after an unsuccessful experiment ad how to reclaim ECT(1), as there are other proposals how to put that to use.
         I am also not convinced that it is true that "hundreds of people in dozens of companies" could not have whipped up the requested experiments in less than a months if necessary. Or slightly concerning, how did these companies actually asses safety of their code/products, if they did it should have been easy to simply post links to existing dada and analyses. 
        After all, team L4S already established several test harnesses for L4S. Simply adding the requested tests should not have been a multi year-problem, no? 

Best Regards
        Sebastian

*) I see no sign of this being actually true.

---

From: Steven Blake <[email-masked]>
Date: Thu, 05 Nov 2020 23:48:18 -0500

On Thu, 2020-11-05 at 16:32 +0100, Sebastian Moeller wrote:
>         [SM] E voila you have a traffic shaper, exactly what I said.
> The costly part tends to be "the decremented at a rate slightly
> slower", as you need to measure time, no?

It's easy if the service rate of the physical queue the virtual queue
is tracking is predictable, though you do need to measure time between
packet arrivals or departures (whichever one you wish to use to drive
the AQM). It is not so easy if the physical queue is competing for
scheduling opportunities against other queues (e.g., WFQ), or if the
physical queue is competing for service on a shared media link (e.g.,
WiFi). Then you have to figure out how much the virtual queue should
have drained when the physical queue was empty and wasn't getting
scheduled. Not an issue with a strict priority scheduler on a fixed
rate link, obviously.

However, **this** is how to really achieve low peak latency with AQM
without  quashing throughput. Trying to run the AQM with low physical
queue thresholds doesn't allow you to gently slow down senders *and*
maintain low peak latency.

[snip]

>         [SM] Bob, you are surely trying flattery right now... But
> honestly, I have been asking for measurements (in accordance with 
> https://tools.ietf.org/html/rfc5033 guidelines) for some years now,
> if you only started do measure this because my objections are looking
> like creating a road-block*, you a) did obviously ignore rfc5033 and
> b) started way to late. The point, is to perform such initial tests
> to assess safety over the internet you do not need an experimental
> track RFC to be published before hand. 
>         If you insist that you require experimental track RFCs before
> the experiment, the least you would need to do is turn L4S into a
> real experiment, with a hypotheses, proposed measurements and a set
> of criteria to assess success or failure at the end, and given that
> you lay claim to a rare IP-header codepoint, a realistic description
> how to undo L4S roll-out after an unsuccessful experiment ad how to
> reclaim ECT(1), as there are other proposals how to put that to use.
>          I am also not convinced that it is true that "hundreds of
> people in dozens of companies" could not have whipped up the
> requested experiments in less than a months if necessary. Or slightly
> concerning, how did these companies actually asses safety of their
> code/products, if they did it should have been easy to simply post
> links to existing dada and analyses. 
>         After all, team L4S already established several test
> harnesses for L4S. Simply adding the requested tests should not have
> been a multi year-problem, no? 

Agreed.

Regards,

// Steve

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 9 Nov 2020 17:41:17 +0000

Steve,

On 06/11/2020 04:48, Steven Blake wrote:
> WiFi).
[BB] Before I started with CableLabs back in 2017, I took care to write 
up this report that describes a sojourn time-based virtual queue, 
including pseudocode. It is a simple way to address this problem of a 
variable drain rate. It uses the same per-packet state as a regular 
queue measuring sojourn-time, except it stores a few more virtual 
packet's worth of metadata (because the virtual queue is longer than the 
real queue). And there's little extra processing. Specifically, see 
https://arxiv.org/pdf/1904.07079.pdf#subsection.2.3

I published it on Cornell Uni's arXiv, to register it as prior art 
against any subsequent attempt to patent similar ideas.
     https://arxiv.org/abs/1904.07079

> Then you have to figure out how much the virtual queue should
> scheduled.

Indeed. For a variable rate, you can predict the time when the remaining 
virtual queue (vq) will have drained, by assuming the vq will drain at 
the same rate as it did over the sojourn of the last packet before the 
real queue went idle (the virtual queue already holds all the data to 
calculate that) {Note 1}. When the real queue goes idle, you store this 
predicted virtual idle time and the current time. Then, on the next 
dequeue event, you calculate what proportion of that time has elapsed 
and discard that proportion of the vq (which may be all of it).

{Note 1}: Of course, this assumption isn't necessarily precisely 
correct, but the implications of being wrong either way are not dire.

> Not an issue with a strict priority scheduler on a fixed
> rate link, obviously.

[BB] Presumably you've seen the low queue delay we can already achieve 
without a virtual queue.

Bob