Subject: [tsvwg] Request for clarification/removal of section 5.1

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Aug 2021 11:52:07 +0200

Dear list,

https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19#page-22 says:
5.5.  Limiting Packet Bursts from Links Supporting L4S AQMs

   As well as senders needing to limit packet bursts (Section 4.3),
   links need to limit the degree of burstiness they introduce.  In both
   cases (senders and links) this is a tradeoff, because batch-handling
   of packets is done for good reason, e.g. processing efficiency or to
   make efficient use of medium acquisition delay.  Some take the
   attitude that there is no point reducing burst delay at the sender
   below that introduced by links (or vice versa).  However, delay
   reduction proceeds by cutting down 'the longest pole in the tent',
   which turns the spotlight on the next longest, and so on.

   This document does not set any quantified requirements for links to
   limit burst delay, primarily because link technologies are outside
   the remit of L4S specifications.  Nonetheless, it would not make
   sense to implement an L4S AQM that feeds into a particular link
   technology without also reviewing opportunities to reduce any form of
   burst delay introduced by that link technology.  This would at least
   limit the bursts that the link would otherwise introduce into the
   onward traffic, which would cause jumpy feedback to the sender as
   well as potential extra queuing delay downstream.  This document does
   not presume to even give guidance on an appropriate target for such
   burst delay until there is more industry experience of L4S.  However,
   as suggested in Section 4.3 it would not seem necessary to limit
   bursts lower than roughly 10% of the minimum base RTT expected in the
   typical deployment scenario (e.g. 250 us burst duration for links
   within the public Internet).

[SM] It seems unclear what a "link" supporting L4S is in this context. IMHO a link connects two nodes, and AQM is or is not running on those nodes not on the links themselves.
But the bigger issue here is that while an L4S AQM surely can be made to limit its line-rate egress burst duration to what ever duration is deemed appropriate, this seems too late in the game.
Why? Well, it is the L4S AQM that is burst intolerant in that it responds to bursts (in either the L or C queue) with too much marking to keep utilization up, so its is not the bottleneck AQM's egress that should limit bursts, but it is the AQM's ingress that should never see such bursts in the first place. But to achieve that all nodes along the path before the AQM would need to know the bottleneck rate of the connection/at the L4S AQM to be able to limit their bursts to what would be equivalent to 250µs (or whatever other arbitrary number) at the AQM. Given that in the internet's core link rates of 100Gbps and above are not unheard of, while internet access links (a natural location for L4S AQM) still often are well below 100 Mbps (so off by three orders of decimal magnitudes) I fail to see hoe these 250µs are useful guidance for anybody?

Also if 0.25ms are <= 10% of the expected minimum base RTT within the public internet, would that min-base RTT not be >= 2.5ms? That seems quite low for even reaching the next IP-hop along a typical DSL or docsis access link, let alone the "public internet". So I am sure I am misunderstanding what this section is trying to convey. So could this section be either clarified or removed, please?

Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 25 Aug 2021 15:57:10 +0300

> On 25 Aug, 2021, at 12:52 pm, Sebastian Moeller <[email-masked]> wrote:
> 
> Why? Well, it is the L4S AQM that is burst intolerant in that it responds to bursts (in either the L or C queue) with too much marking to keep utilization up, so its is not the bottleneck AQM's egress that should limit bursts, but it is the AQM's ingress that should never see such bursts in the first place. But to achieve that all nodes along the path before the AQM would need to know the bottleneck rate of the connection/at the L4S AQM to be able to limit their bursts to what would be equivalent to 250µs (or whatever other arbitrary number) at the AQM.

This is an important point, related to one I just made to Ingemar.  It is clear from the quoted section that a requirement for low jitter and burstiness on the path *upstream of* the L4S AQM, as well as from the sender itself, is a requirement for good performance, and that at least part of that requirement is acknowledged by the authors (though worded in a way that is difficult for non-experts to decipher).

So if L4S relies on low jitter and burstiness on the path, how do endpoints find out whether those requirements are met on a math over which they wish to communicate, *in advance* of negotiating L4S support over their transport protocol of choice?  There is no provision in the wire protocol for negotiating a downgrade to conventional, more burst-tolerant behaviour, so it must be performed unilaterally at the sender, but so far attempts to detect path characteristics similar to this at the sender have met with failure.

I did not get a satisfactory answer from Ingemar as to how this problem should be solved.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 25 Aug 2021 14:22:23 +0100

Sebastian, see [BB] inline...

> [SM] It seems unclear what a "link" supporting L4S is in this context. IMHO a link connects two nodes, and AQM is or is not running on those nodes not on the links themselves.

[BB] You've quoted the answer to your question.
"it would not make sense to implement an L4S AQM that feeds *into* a 
particular link technology without ..."

> Why? Well, it is the L4S AQM that is burst intolerant in that it responds to bursts (in either the L or C queue) with too much marking to keep utilization up, so its is not the bottleneck AQM's egr…

[BB] The IETF defines technology components. The IETF sometimes provide 
guidance on system design, but system designers tend to congregate in 
other fora, like 3GPP, ITU, etc. I'm not saying that we don't want to 
help with system design, just that the IETF is not the foremost place to 
expend such effort. Yes, upstream links will need to reduce burstiness. 
But the text explains why it does not discuss reducing the burstiness of 
other links on the path. That would be in scope for a different RFC 
solely about reducing burtiness of links, which is an important part of 
the whole programme to reduce delay, but not specific to L4S (also 
equally applicable to any alternative to L4S that you might prefer).

> Also if 0.25ms are <= 10% of the expected minimum base RTT within the public internet, would that min-base RTT not be >= 2.5ms? That seems quite low for even reaching the next IP-hop along a typica…

[BB] In an urban setting with local CDN nodes, 2.5ms is quite feasible 
as a /minimum/ base RTT. The /average/ base RTT to the main global CDNs 
across all the RIPE Atlas probes in S Korea and in Nigeria is 4ms. See 
Appx C of a report I recently uploaded, pulling together global stats 
like this [PI2param]. CDN deployment has moved unbelievably fast over 
the last decade or so in both the developed and developing world. 
Assuming 5G edge compute deployment proceeds as planned, such low base 
RTTs will become even more common.

    [PI2param]
               Briscoe, B., "PI2 Parameters", Technical Report TR-BB-
               2021-001 arXiv:2107.01003 [cs.NI], July 2021,
               <https://arxiv.org/abs/2107.01003  <https://arxiv.org/abs/2107.01003>>.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 25 Aug 2021 13:56:45 +0000

Hi Jonathan

I am not at all certain how to answer this properly. I believe that Bob's post earlier today should give some of the answers you are looking for 
https://mailarchive.ietf.org/arch/msg/tsvwg/s_O8b32zMORNIhlZAgGse2c_q2E/

Delay jitter, due to scheduling or retransmission or something else is something that we are working on and need continue to work on to reduce. It is not only an L4S matter. I suggested that pacing is added to reduce burstiness, some may implement it, some not. Finally I don't see that it is particularly fruitful to delay new technology in IETF just because some particular access technology causes certain problems in certain circumstances today, especially when it is quite obvious that remedies can be implemented to reduce the impact of these problems.

/Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
> Sent: den 25 augusti 2021 14:57
> To: Sebastian Moeller <[email-masked]>
> Cc: TSVWG <[email-masked]>
> Subject: Re: [tsvwg] Request for clarification/removal of section 5.1
> 
> > But the bigger issue here is that while an L4S AQM surely can be made to
> limit its line-rate egress burst duration to what ever duration is deemed
> appropriate, this seems too late in the game.
> > Why? Well, it is the L4S AQM that is burst intolerant in that it responds to
> bursts (in either the L or C queue) with too much marking to keep utilization
> up, so its is not the bottleneck AQM's egress that should limit bursts, but it is
> the AQM's ingress that should never see such bursts in the first place. But to
> achieve that all nodes along the path before the AQM would need to know
> the bottleneck rate of the connection/at the L4S AQM to be able to limit their
> bursts to what would be equivalent to 250µs (or whatever other arbitrary
> number) at the AQM.
> 
> This is an important point, related to one I just made to Ingemar.  It is clear
> from the quoted section that a requirement for low jitter and burstiness on
> the path *upstream of* the L4S AQM, as well as from the sender itself, is a
> requirement for good performance, and that at least part of that
> requirement is acknowledged by the authors (though worded in a way that is
> difficult for non-experts to decipher).
> 
> So if L4S relies on low jitter and burstiness on the path, how do endpoints
> find out whether those requirements are met on a math over which they
> wish to communicate, *in advance* of negotiating L4S support over their
> transport protocol of choice?  There is no provision in the wire protocol for
> negotiating a downgrade to conventional, more burst-tolerant behaviour, so
> it must be performed unilaterally at the sender, but so far attempts to detect
> path characteristics similar to this at the sender have met with failure.
>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Aug 2021 16:24:25 +0200

Hi Bob,

> On Aug 25, 2021, at 15:22, Bob Briscoe <[email-masked]> wrote:
> 
> "it would not make sense to implement an L4S AQM that feeds *into* a particular link technology without ..."

	[SM] Yes, and no. The issue is that the low burst requirement is on the ingress side of an L4S AQM, but you seem to argue for mking the egress side of an L4S AQM less bursty. At that point the horse has left the barn already, no? Sure it might be nice to generally reduce bursts somewhat, but at that point it does not really help any mode....

> [BB] The IETF defines technology components. The IETF sometimes provide guidance on system design, but system designers tend to congregate in other fora, like 3GPP, ITU, etc. I'm not saying that we don't want to help with system design, just that the IETF is not the foremost place to expend such effort. Yes, upstream links will need to reduce burstiness. But the text explains why it does not discuss reducing the burstiness of other links on the path. That would be in scope for a different RFC solely about reducing burtiness of links, which is an important part of the whole programme to reduce delay, but not specific to L4S (also equally applicable to any alternative to L4S that you might prefer).

	[SM] I fully agree that other node's behaviour is out of scope for the L4S drafts, but at the same time burst reduction after the L4S bottleneck seems not really helping, as the damage is already done.

> [BB] In an urban setting with local CDN nodes, 2.5ms is quite feasible as a /minimum/ base RTT.

	[SM] Well, "urban setting with local CDN nodes" ist not equvalent with "public internet", sorry. And many link technologies have first HOP RTTs well above these 2.5ms...

> The /average/ base RTT to the main global CDNs across all the RIPE Atlas probes in S Korea and in Nigeria is 4ms.

	[SM] Not living in  S Korea (which currently has a whopping 18 connected ATLAS probes) or Nigeria (which currently has 0 ATLAS probes connected) I doubt the usefulness of these numbers (for reference Germany has 1641 probes today and is reported with 14ms CDN RTT). Really looking at that list where you average the ATLAS CDN latencies by internet users poer country, without first making sure that the CDN-RTT is truly representative seems statistically problematic... 
Let's look at Nigeria and Germany for a sanity check of your list, both have a comparable population density albeit Nigeria being ~3 times larger and Germany having a 4 times larger GDP. Also Nigeria has very little (currently, 2 disconnected probes and 27 abandoned ones) ATLAS probes while Germany has a considerable amount. I can guarantee that Nigerias 137 million internet users do not live 4ms away from the next CDN on average. You table, I am afraid to say, required more sanity checking....
That also puts the validity of your weighted average CDN-RTT into question. That said, I do not believe that taking a internet-user weighted average CDN-RTT as basis to define a typical RTT as a suitable method.

And if the 34ms are bunk, so is the 34 * 0.22 * 2 = 14.96 -> 15 reference RTT. But that also suffers from the fact that the 0.22 ist also the result of an equally complicated deduction (trying to average between cubic and reno behaviour somehow in proportion to their estimated prevalence IIRC) and finally that the scaling/safety factor of 2 is not based on anything.... As I said before, I would like to see the data that you used you initially to set the reference/target delay for the C-queue in dualQ.... I might be naive, but I probably just would have run a series of latency and utilization tests through an DualQ AQM with different settings for the two respective target values and confirmed that the selected value would deliver the desired throughput and latency under load increase numbers I deem acceptable....

> See Appx C of a report I recently uploaded, pulling together global stats like this [PI2param]. CDN deployment has moved unbelievably fast over the last decade or so in both the developed and developing world. Assuming 5G edge compute deployment proceeds as planned, such low base RTTs will become even more common.

	[SM] The "public internet" includes path over the oceans as well, like remote attending IETF meetings, or exchanging data with collaborators on other continents. I am fine with making access to close servers really sing, but please not at the expense of getting worse access to more remote nodes of the internet.... but that is exactly what L4S currently seems to do: it builds a priority lane for short RTT L4S flows....

>    [PI2param
> ]
>               <
> https://arxiv.org/abs/2107.01003

	Yes, I have seen that, and it reminds on why academia typically requires peer review. As shown above your deduction of the "Typical User to CDN RTT" seem to suffer from bad input data (see the plausibility check for Nigera and Germany above and the issues of taking the ATLAS data without further sanitization).

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Aug 2021 16:33:29 +0200

Hi Ingemar,

> On Aug 25, 2021, at 15:56, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Jonathan
> 
> Delay jitter, due to scheduling or retransmission or something else is something that we are working on and need continue to work on to reduce. It is not only an L4S matter. I suggested that pacing is added to reduce burstiness, some may 

	[SM] How can an intermediate node upstream of a bottleneck know which rate would be appropriate (and would that rate not be actually different for different flows with different destinations). Sure the sender has some idea what "rate"/in flight data it can push into the network, a does the bottleneck, but all other nodes along a path lack the required information for appropriate pacing, no? Well, FQ might hep a bit as it better interleaves flows which at least should spread out intra-flow bursts (if there are multiple flows active), but DualQ being effectively single queue (per class) that does not really help much if these noe better interleaved flows traverse the same L4S AQM.

> implement it, some not. Finally I don't see that it is particularly fruitful to delay new technology in IETF just because some particular access technology causes certain problems in certain circumstances today,

	[SM] Well, IMHO the IETF should delay blessing new technology with RFC status until said new technology is ready for primetime, which L4S does not seem to be. All of the issues demonstrated in the last 2-3 years are all still there, some with failed attempts at remedies (like rfc3168 detection and avoidance), RTT de-biasing... 

> especially when it is quite obvious that remedies can be implemented to reduce the impact of these problems.

	[SM] Yes, but these remedies seem to either require an oracle or a time machine..

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 25 Aug 2021 16:00:02 +0100

Ingemar,

Yes, the general answer is that the operator designs their network as a 
system, and tunes the burstiness of the links to whatever goal it 
settles on.

There is some scope for equipment developers to influence the ingress as 
well as the egress link. For instance, a WiFi access point can control 
the parameters that a host uses to send to it. So, a home gateway built 
to support L4S on the WAN side and WiFi on the LAN side can control the 
WiFi burstiness from the host up to the access point, before traffic 
enters the L4S AQM an onward into the upstream link to the Internet.

We are in the early stages of a shift in focus from throughput to 
latency. As long as throughput is the headline benchmark figure, we will 
see processing resources focused on transmitting packet aggregates in 
larger and larger bursts. As the relative importance of latency against 
bandwidth becomes better understood, equipment developers will use 
faster processors to reduce (or at least not increase) aggregate sizes.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Aug 2021 17:12:19 +0200

Hi Bob,

> On Aug 25, 2021, at 17:00, Bob Briscoe <[email-masked]> wrote:
> 
> Ingemar,
> 
> Yes, the general answer is that the operator designs their network as a system, and tunes the burstiness of the links to whatever goal it settles on.

	[SM] Sure nobody probably perceives bursts as desirable.

> 
> There is some scope for equipment developers to influence the ingress as well as the egress link. For instance, a WiFi access point can control the parameters that a host uses to send to it. So, a home gateway built to support L4S on the WAN side and WiFi on the LAN side can control the WiFi burstiness from the host up to the access point, before traffic enters the L4S AQM an onward into the upstream link to the Internet.

	[SM] Well, IIUC the rationale for aggregation in WiFI is that each transmit unit has a relative high wifi overhead and the bursts come from the fact that APs/stations try to cram as many packets into a transmit unit as possible to beter amortize the overhead, so not aggregation will have a relative steep rate/"utilization" price on WiFi.

> We are in the early stages of a shift in focus from throughput to latency.

	[SM] But your claim ist that L4S gives low latency AND high utilization... which it only does under benign conditions. This is a decent way to alienate your potential user base...

> As long as throughput is the headline benchmark figure, we will see processing resources focused on transmitting packet aggregates in larger and larger bursts. As the relative importance of latency against bandwidth becomes better understood, equipment developers will use faster processors to reduce (or at least not increase) aggregate sizes.

	[SM] On WiFi a faster processor will not help, you would need another MAC layer with less transmit time taken up by "overhead". I am not saying that I am not for such a thing, but that is not something that equipment developers can implement/deploy quickly and unilaterally. This is why the benchmark for L4S needs to be how well it works and delivers on its promises in today's existing internet and not one that might or might not exist in the future.

Best Regards
	Sebastian