Subject: [tsvwg] What does the low queueing delay that L4S offers actually mean for the latency experienced by an application?

From: Sebastian Moeller <[email-masked]>
Date: Mon, 25 Mar 2019 18:09:22 +0100

Dear Bob,
I wonder whether microsecond queueing delay as achieved by L4S does necessarily translate into app responsiveness in the microsecond rang? From my naive interpretation, the low queueing delay mainly emerges by all senders basically "synchronizing" (or rather phase-lock to their respective sending-phase) , but then the latency "just" moves from the queue into the endpoint waiting for the sending time slot, what am I missing? I am sure that this might be better for the endpoint as visibility into the actual sending time might be available, but from an enduser's perspective does this really improve application delay?

Best Regards & sorry for too many rather direct emails

	Sebastian

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 27 Mar 2019 12:48:58 +0000

Hi Sebastian,

> sending-phase) , but then the latency "just" moves from the queue into the
> endpoint waiting for the sending time slot, what am I missing? 

Nothing actually: If you move the queue buildup into the application/end-host,
you make it aware of it--as opposed to hiding it down in the TCP stack through
retransmission and hiding it by increasing latencies.
Example applications that can use such a signal are those having to do real-time
pre-processing (e.g., a drone streaming its camera feed for real-time control,
AR/VR apps). Because they see the queue building up on the endhosts, they can
adapt their behavior accordingly (i.e., do a graceful degradation of the video
frame rate instead of causing a playback pause at the receiver due to buffering).

> this might be better for the endpoint as visibility into the actual sending time
> might be available, but from an enduser's perspective does this really
> improve application delay?

The RTT I currently measure using a fiber connection to reach AMSIX is about
30ms (< 200km from here). About 20ms of those are spent just getting past the
access layer of my ISP (< 50km, first hop consistently replies in less than 4ms,
second in ~20ms).
These latencies are mostly due to deep buffers along the path (transmission 
delays alone to AMSIX account for < 3ms in the RTT, assuming all links are using
some form of optical fiber). Any way to cut down these buffers will drastically 
shorten that RTT. If you use an application that has to exchange data over 
multiple RTTs (e.g., web browsing), then those RTT reduction will add up (esp.,
if some objects cannot be loaded in parallel).

Best,
Olivier

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 27 Mar 2019 14:14:17 +0100

Hi Oliver,

> On Mar 27, 2019, at 13:48, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	Sorry, unless TSQ will create bach-pressure to my application the packets will be hiding in the TCP stack on my end... So sure this is closer to the end-user but it does not actually speed up the data delivery from the point in time an application wants to send something.

> frame rate instead of causing a playback pause at the receiver due to buffering).

	That only accounts for the steady state case when all senders through the bottleneck already spread themselves out, this will change if more flows are added to the bottleneck; since this is not super new, you probably have already modeled that and there is a figure in one of the papers I missed/misinterpreted? And how fast does this bottleneck actually reach steady state if a number of staggered flows are coming and going.

> shorten that RTT.
> If you use an application that has to exchange data over 

	This I want to see empirically, as now the flows need to time their sending "slot" so it is well aligned over multiple hops, is that really realistic?

Thanks for the friendly reply, and for helping me understand the concept better.

Best Regards
	Sebastian

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 27 Mar 2019 14:54:26 +0000

Sebastian,

> 	Sorry, unless TSQ will create bach-pressure to my application the
> packets will be hiding in the TCP stack on my end... So sure this is closer to

An active way for latency-sensitive applications is to read the
tcpi_notsent_bytes variable through TCP_INFO to know the extent of your
send Q, i.e., view the congestion level if there are no queuing delays in the
ntework.  IMHO relying on TSQ to block a sendmsg() means that the 
application is not latency-sensitive.

> the end-user but it does not actually speed up the data delivery from the
> point in time an application wants to send something.

One of the benefit with the queuing delay reduction lies in the reactivity that
is gained by your application due to the shorter RTT. Think of an encoder that
never has more than one full frame in-flight, hence can adapt in real-time to
throughput variation (as opposed to intermediate frames stuck in queues).
Another one is the shorter flow completion times (esp. visible for small object
transfers where the 3WHS has a significant impact).

I would like to stress again that increasing throughput of L4S flows wrt. classic
ones is not the goal here (as it would create unfairness). Hence a raw 
data-delivery speed up should not be expected for large static transfers (other
than reduced latencies for the 3WHS + FIN).
 
> 	That only accounts for the steady state case when all senders
> through the bottleneck already spread themselves out, this will change if
> more flows are added to the bottleneck; since this is not super new, you
> probably have already modeled that and there is a figure in one of the
> papers I missed/misinterpreted? And how fast does this bottleneck actually
> reach steady state if a number of staggered flows are coming and going.
> [...]
> 	This I want to see empirically, as now the flows need to time their
> sending "slot" so it is well aligned over multiple hops, is that really realistic?

The most intuitive way to approach this is to remember that TCP is ACK-clocked
and, depending on multiple parameters [1], will stabilize itself.
In other words, past the initial window, you never send more data unless you 
receive an ACK that lets you slide your sending window.

All data packets that traverse a given link are "spread" by the bottleneck ingress
router (i.e., they are serialized over that bottleneck link, one at a time). These 
packets, even if initially sent at the same time, will now arrive at different times
to their destination (i.e., wake their corresponding TCP stacks at different
moments). This will results in ACKs also sent at different times (and also serialized 
at different times by the bottleneck link). This will cause the next data-packets to 
no longer be sent at the same time, as the ACKs allowing them to be sent will not
be received simultaneously. Note that this is an over-simplification (see e.g., [1,2]
for more thorough analyses of classic TCP and [3] for DCTCP).

Best,
Olivier

[1] Veres, A., & Boda, M. (2000, March). The chaotic nature of TCP congestion control.
In Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. 
Nineteenth Annual Joint Conference of the IEEE Computer and Communications 
Societies (Cat. No. 00CH37064) (Vol. 3, pp. 1715-1723). IEEE.
[2] Hollot, C. V., Misra, V., Towsley, D., & Gong, W. (2002). Analysis and design of 
controllers for AQM routers supporting TCP flows. IEEE Transactions on automatic 
control, 47(6), 945-959.
[3] Alizadeh, M., Javanmard, A., & Prabhakar, B. (2011, June). Analysis of DCTCP: 
stability, convergence, and fairness. In Proceedings of the ACM SIGMETRICS joint 
international conference on Measurement and modeling of computer systems 
(pp. 73-84). ACM.

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 27 Mar 2019 16:21:20 +0100

Hi Oliver,

> On Mar 27, 2019, at 15:54, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Sebastian,
> 

	Fair-enough, it shows though that ultra-low queueing delay is not going to magically improve all applications.

> throughput variation (as opposed to intermediate frames stuck in queues).

	Sorry that does not sound actually true. The encoder might have a better prediction about the path characteristics, but unless it uses an oracle that prediction might not reflect the characteristics the packets will encounter en-route. But sure this is an application where reasonable low RTT can be useful, albeit once your RTT exceeds the inter-frame period this example will not be generally useful, no? 

> than reduced latencies for the 3WHS + FIN).

	Thanks, that is what I meant with "speed" above. not bandwidth.

> and, depending on multiple parameters [1], will stabilize itself.

	Well, I would like to see how you achieve the required microsecond synchronization across multiple flow active at the AQM hop, on a > 20 ms path, in my layman's view the ACK-"clock" is not really precise enough for this. I am sure this works for DCTCP in the datacenter, but there RTT probably is < 1 ms to begin with.

> receive an ACK that lets you slide your sending window.

	Sure, but if say 10 more flows start up and hit the AQM bottleneck, the old send timing will be wrong; my question is how long does it take to re-synchronize all L4S flows again to achieve the empty queue?

> for more thorough analyses of classic TCP and [3] for DCTCP).

	Sure, but that just desynchronizes them, for you scheme to work IMHO you require more than de-phasing, you need each sender to send at that point in time that this packet will basically encounter an empty queue at the AQM and again I am not sure how this is going to work over a partially congested 20ms+ link. If you could point me to data showing how L4S deals with this (preferable gracefully, I hope) I would be thankful. 
BTW, the cited DCTCP paper says "While DCTCP converges slower than TCP, we found that its convergence rate is no more than a factor 1.4 slower than TCP." while I believe that L4S would need something that converges faster than TCP if ultra-low queueing delay is to be achieved reliably and robustly; I guess I am still missing something...

Best Regards
	Sebastian

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 28 Mar 2019 10:35:54 +0000

Hi Sebastian,

> 	Fair-enough, it shows though that ultra-low queueing delay is not
> going to magically improve all applications.

Today's applications "work". Reducing queuing delays may help some of them,
but latency wasn't their limiting factor in the first place (or we wouldn't say they
work).

Providing a LL service over the public Internet however enables a new class of
applications that otherwise require dedicated/ad-hoc infrastructure.
Dynamic applications such as AR/VR, thin clients, or interactive videos are
example use-case where the content displayed to the user depends on his
action. If we don't want the user to complain about input lag, motion sickness, 
... latency as to be brought down significantly (and reliably) without sacrificing
the quality of the displayed content (e.g., going beyond wireframe/low-res).

This new service is part of the rationale why L4S' designers think it can get 
deployment traction with ISPs (as opposed to Today's status quo with 
classic ECN AQMs). Other reasons/use-cases are documented in 
draft-ietf-tsvwg-l4s-arch (e.g., using shallow buffers over high-BDP 
links--stallite--is an economic incentive, complementary to PEPs discussed in
RFC3135/RFC3449)

> 	Sorry that does not sound actually true. The encoder might have a
> better prediction about the path characteristics, but unless it uses an oracle
> that prediction might not reflect the characteristics the packets will
> encounter en-route. 

Which is why we care about the tail latencies in all tests/demos/presentations,
and not the average. Having some bounds on the feedback loop, enables to 
design systems around it. 

> But sure this is an application where reasonable low RTT
> can be useful, albeit once your RTT exceeds the inter-frame period this
> example will not be generally useful, no?

Yes, hence why having bounds on your service quality is key to optimize
its deployment (e.g., how much frame time do you allocate to the transmission
delays, the server-side rendering, ...). Currently, one can only hope that the
locations where he placed his service instances are "close-enough" to the
user (as in: we hope the ISP has reasonable queues wrt. where we deployed 
our service).

> 	Well, I would like to see how you achieve the required microsecond
> synchronization across multiple flow active at the AQM hop, on a > 20 ms
> path, in my layman's view the ACK-"clock" is not really precise enough for
> this. I am sure this works for DCTCP in the datacenter, but there RTT probably
> is < 1 ms to begin with.
>
> 	Sure, but if say 10 more flows start up and hit the AQM bottleneck,
> the old send timing will be wrong; my question is how long does it take to re-
> synchronize all L4S flows again to achieve the empty queue?
> 
> >
> 	Sure, but that just desynchronizes them, for you scheme to work
> IMHO you require more than de-phasing, you need each sender to send at
> that point in time that this packet will basically encounter an empty queue at
> the AQM

I am not certain I understand where this us synchronization requirement came
from, as we're not speaking here of TSN/detnet.

We're not requiring sender to synchronize themselves (although they eventually
will if running in a stable environment, as shown by prior research), we're directly
giving them feedback through CE marks if their packets caused the L4S queue to 
build up (typically using a 1ms step threshold). The L4S queue does not have to
 be empty, it has to contain e.g., less than 10 1500bytes packets on a 120Mbps
link. Any more than that, and the receiver will echo the CE marks when ACKing
the received burst of packets, causing the sender to backoff linearly wrt. the 
number of marks when sending its next burst.
Senders are also required to backoff if they take more than fair share of BW,
i.e., if they receive the random marks due to the buildup of the classic queue.

> again I am not sure how this is going to work over a partially
> congested 20ms+ link. If you could point me to data showing how L4S deals
> with this (preferable gracefully, I hope) I would be thankful.

The throughput you get over a  network path with a "partially congested link"
is the available BW over the most congested link/the one with the smallest rate.
This where the AQM has to be (and is usually the access link).
It is true that converging with 20ms is slower than with 1ms (i.e., DCTCP 
currently end up under-utilizing the link for flows lasting only a few packets, see 
draft-ietf-tsvwg-ecn-l4s-id §A.2.3). 

> BTW, the cited DCTCP paper says "While DCTCP converges slower than TCP,
> we found that its convergence rate is no more than a factor 1.4 slower than
> TCP." while I believe that L4S would need something that converges faster
> than TCP if ultra-low queueing delay is to be achieved reliably and robustly; I
> guess I am still missing something...

This is a known limitation with DCTCP indeed.
See draft-ietf-tsvwg-ecn-l4s-id §A.2.3 The main challenge there is properly 
estimating the bottleneck BW without overshooting (i.e., building up a 
queue).

The key here is that the slower convergence causes under-utilization, not 
queue buildup (cfr. DCTCP stability analysis).

Best,
Olivier

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Mar 2019 13:01:40 +0100

Hi Olivier

> On Mar 28, 2019, at 11:35, Tilmans, Olivier (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	It is not an accident I used the word improve instead of work.

> applications that otherwise require dedicated/ad-hoc infrastructure.

	Build it and they will come, but what if they never come? I ask this in the context of whether to bet the ECT(1) codepoint on L4S making it, other than that I see L4S as at least a worthwile and interesting experiment.

> action.

	Sure, but all of these are RTT limited, so all L4S is going to give you is that the end-points can be a bit further away, this will not magically work from Europe to the west-coast.

> If we don't want the user to complain about input lag, motion sickness, 
> classic ECN AQMs).
	
	An ISP will only do this if it will allow either lower Cap/OpEX or higher revenue from the customers. How exactly is L4S helping with either in a way that the latency reduction already delivered by TCP-friendly ECN marking AQMs do not promise tp do so already? Not being an ISP I might not see the forest for the trees, but I am not seeing a qualitatively different deployment scenario than before.

> Other reasons/use-cases are documented in 

	All true, and not counter to my argument. Also "bounds on the feedback loop" is optimistic, I give you lower average RTT and even less variation, but that is different from a hard bound, no?

> user (as in: we hope the ISP has reasonable queues wrt. where we deployed 

	That really does not change qualitatively, close-enough will increase a bit, sure, but L4S still offers no hard bounds (let me mention the overload protection by tail-dropping in the dual queue RFC).

> from, as we're not speaking here of TSN/detnet.

	The synchronisation requirement comes from the fact, that once Packets bunch up in your queue the later packets will need to wait for the earlier packets to clear; queueing delay is a function of how-many bytes need to be pushed out at a given rate before the current packet gets pushed out (I am simplifying here of course). To guarantee ultra-low queueing delay _you_ need to make sure all L4S senders at the bottleneck spread their packets such that there is minimal bunch-up in the AQM queue. And that requires IMHO a somewhat stricter synchronisation of the senders than the simple de-phasing that will help spread out the packet arrival times from different senders that started fully synchronized.

> link.
> Any more than that, and the receiver will echo the CE marks when ACKing

	Agreed on all of this, it does not address my point though.

> draft-ietf-tsvwg-ecn-l4s-id §A.2.3). 

	So what is the plan to tackle this? Especially in the light of competent AQMs in the path, that might not necessarily show a strong correlation between queue build-up and measurable queueing delay?

> queue buildup (cfr. DCTCP stability analysis).

	Good, thanks for pointing that out.

Best Regards
	Sebastian