Subject: Re: [tsvwg] problematic numbers in draft-ietf-tsvwg-aqm-dualq-coupled

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 Oct 2021 09:47:33 +0100

Sebastian,

On 15/09/2021 10:42, Sebastian Moeller wrote:
> Dear list
>
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled#page-38
>
> contains the following text:
>
> "o  RTT_typ is taken as 34 ms.  This is based on an average CDN
>        latency measured in each country weighted by the number of
>        Internet users in that country to produce an overall weighted
>        average for the Internet [PI2param]."
>
> Which as I have already posted in response to PI2param as being factually wrong, but I had not realized that this number had leaked into the draft.

[BB] As the dualq draft makes clear, quoting:
"Frankly, these factors are educated guesses, but with the emphasis 
closer to 'educated' than to 'guess' (see [PI2param 
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#PI2param>] 
for background investigations):"

Prior to this study, the choices of target in PIE, CoDel and any other 
AQMs were based on no particular knowledge of the global situation at 
all, other than non-scientific anecdote. This study states it is 
applicable to PI2 and PIE. So their settings of target are now educated 
guesses, while CoDel's is just a guess? Given the deployment of CoDel, 
it would be useful to have a study of the utilization resulting from 
CoDel's target being set to 5ms.

More...

>
> The measurement this is based on reports for 2019 and each included country the average latency to the fastest CDN (out of a set of 7 CDNs), not the latency averaged over all CDNs, this alone makes the numbers not suited for Bob's purpose here.
> 	In addition to that issue the numbers are not sufficiently sanity checked and I believe are giving the wrong impression:
>
> For example Google operated a CDN-node (or similar) in Lagos (Nigeria), and the four Ripe ATLAS nodes active at that time in Nigeria where at most 12ms away, with an average of ~4ms, which is the number Bob used, but if one actually calculates the average latency of these 4 ATLAS nodes to the seven CDNs that were included in the study, Nigeria's average CDN-latency increases to 92.4ms. In other words the 34ms number in the dualQ draft is in all likelihood incorrect, because it is based on a) a misunderstanding what the cited study actually measured and a failure to make sure all included per-country-numbers are sufficiently representative (e.g. a sufficient hight number of probes, spread sufficiently wide over a countries area).
> 	I consider it worth mentioning, that Bob took the time to discuss the value for China, which appeared unexpectedly large, but failed to sanity check unexpected low RTTs, like Nigeria's. If this section of the draft is supposed to stand, these numbers will need to be put on more solid ground.

[BB] I hadn't investigated Nigeria's figure although I was surprised by 
it (as I was with a number of the others). Nonetheless, the user 
experience is likely to be closer to that of the best CDN than the 
average CDN. The ISPs in each country will tend to choose the better or 
best CDNs. Taking the average of all seven would not be appropriate, 
because that would include the CDNs without any relationships to the 
ISPs in that country (which is why the average figure you suggest I 
should use is completely inappropriate). I took the figures published in 
that study as an indication of what natural market selection was doing. 
Also, some ISPs use multi-CDN solutions where the lowest latency CDN 
wins. I presume that is also why the authors of the study did it that way.

That should mean the figures could be a little optimistic, but I would 
not expect the difference to be large as long as there are enough 
samples in a country. I ranked all the countries in order of size of 
Internet user population, then once I had reached 90% of Internet users, 
I excluded the smaller user populations as a way to exclude the smaller 
samples.

Most importantly, criticism like this is always useful for improving the 
next study.

>
> But since this exercise actually is intended to give solid backing for the recommended default reference target for the C-queue I would like to propose the following instead:
>
> Heuristically test different values for the C-queue reference latency while looking at:
> a) latency under load increase (queueing delay)
> b) utilization at different RTTs
> c) RTT bias/unfairness if flows of wildly different RTTs share the C-queue

That is what we had done prior to this study (surely you must have read 
the experiments by now, which do exactly what you list above - and 
more). But that doesn't help determine /prevalence/ of the different 
RTTs across the world, which is what the study in [PI2param] adds.

>
> It is well possible that in the light of such data, 15ms might seem like a rational and reasonable compromise, much more so than the current exercise of multiplying three more or less made up numbers and miraculously ending up at the pre-selected 15ms...

[BB] That is not a fair characterization.

Incidentally, I have been updating the analysis in PI2 param over the 
past few weeks. It ends up with a higher global weighted average, but 
then it has to be adjusted down again because there is evidence (from 
speedtest.net) that latency has been reducing year-on-year since the 
data was gathered. So we still end up at about target = 15ms, but with 
the caveat that it could have to be reduced further soon. And other 
caveats - you'll see the new outcome in the dualq draft before Monday's 
draft deadline. The new PI2param study will probably come out after that 
- as soon as it works through ArXiv's review process.

Bob

> Regards
> 	Sebastian

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 Oct 2021 12:42:19 +0200

Hi Bob,
[SM2] below

Tl; dr: the classic-queue reference delay target still seems not better justified than when it was selected a few years ago, and more, the method in PI2 Parameters to deduce that number seems incorrect.
I keep harping on this, because the extreme unfairness of L-traffic over C-traffic at L4S main use case of short RTT paths is a direct consequence of the ratio of the two queue's introduced delay under load. And since the deficient design that is the root cause of this L-bias is not changed, any ms that can be shaved off the L-queue's target is going to help existing traffic once L4S is actually deployed.

My preferred outcome would be to not deploy L4S, but it seems quite likely that I in the rough on that.

> On Oct 23, 2021, at 10:47, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> "Frankly, these factors are educated guesses, but with the emphasis closer to 'educated' than to 'guess' (see [PI2param] for background investigations):"

	[SM] No, Bob, I disagree, after my post(s) in this regards, the 34ms is not an educated guess anymore, but simply inaccurate. I would, naively?, expected a "oh, thanks for digging into this, yes there was a genuine mis-understanding of the data source, here are the adjusted values" response. And no I get, "oh these numbers can be wildly off, we do not care since we did not claim precision to beginn with"?

So, here is an explicit question, will you revise the numbers in the draft, or your theory of how to select the reference target (because, as I stated elsewhere, I believe your elaborate theory of how to select that value to be simply wrong).

> Prior to this study, the choices of target in PIE, CoDel and any other AQMs were based on no particular knowledge of the global situation at all, other than non-scientific anecdote.

	[SM] Interesting, it confirms my suspicion that the 10-20ms reference delay targets in PI(E) RFC was never really scientifically selected.

> This study states it is applicable to PI2 and PIE. So their settings of target are now educated guesses, while CoDel's is just a guess?

	[SM] Given the wild inaccuracy in one of your three numbers, no, 15ms is not an aducated guess, but simply wrong. The Codel interval of 100ms has IIRC been selected empirically, after testing a number of different values with flows at internet scale RTTs...  See https://queue.acm.org/detail.cfm?id=2209336: "Since CoDel’s interval is weakly related to RTT, we tested the effectiveness of a 100-ms setting over a wide range of likely RTTs and report on the range from 10 to 500 ms."

	This, incidentally, is the same approach I keep proposing for PIE, just go and measure the effect of different reference targets on RTT-unfairness, throughput and latency under load, converge on a value that strikes an acceptable compromise between these three dimensions.

> Given the deployment of CoDel, it would be useful to have a study of the utilization resulting from CoDel's target being set to 5ms.

	[SM] No, you seem to misunderstand Codel; the "free" parameter of Codel is the interval, target simply is 5% of interval (see https://datatracker.ietf.org/doc/html/rfc8289#page-14), but as far as I can tell interval 100ms (and hence target 5ms) was selected as a result of actual tests with RTTs ranging from 10 to 500ms... where is delivers acceptable though put...
	In the original PIE paper* a reference target of 5ms was actually tested resulting in a somewhat lower throughput that codel at 5ms (expected given the difference in what is actually controlled by that parameter)
RTT 100ms, traffic limit 10Mbs:
in the linked paper, extracted from page 7 (Figure 11 shows the CDF for queueing delay at 5(a), or 20(b) ms for PIE and Codel):
TCP Traffic: []... For the 20ms target delay case, the throughput for PIE is 9.87Mbps vs. CoDel is 9.94Mbps. For the 5ms target delay case, the throughput for PIE is 9.66Mbps vs. CoDel is 9.84Mbps. CoDel’s throughput is slightly better than PIE."
"Mixture of TCP and UDP Traffic: [...]  In this test, when the target delay equals to 20ms, the throughput for PIE is 9.88Mbps vs. CoDel is 9.91Mbps. When the target delay equals to 5ms, the throughput for PIE is 9.79Mbps vs. CoDel is 9.89Mbps. Throughputs are similar."

So unless your argument that 100-100*9.66/9.87 = 2.1%-age points (or 100-100*9.79/9.88 = 0.9%-age points for the mixed traffic) less throughput is inacceptable, 5ms PIE target does not look like a unreasonable proposition. Now I understand why the original single queue PIE rfc opted for a larger target, ultra low latency was not one of the goals, and hence recovering 2% point more throughput probably seemed attractice. BUT for L4S were the ratio of L -and C-queue target delays has a significant influence on inter-queue fairness this choice should be revisited. 
	I have made this point in the past, and have not been presented any numbers that show where classic target 5ms would be unacceptable, if you have any data to back this up, please post links.

*)  [HPSR-PIE] Pan, R., Natarajan, P., Piglione, C., Prabhu, M.S.,
              Subramanian, V., Baker, F., and B. Ver Steeg, "PIE: A
              lightweight control scheme to address the bufferbloat
              problem", IEEE HPSR, DOI 10.1109/HPSR.2013.6602305, 2013,
              <https://www.researchgate.net/publication/261134127_PIE_A_lightweight_control_scheme_to_address_the_bufferbloat_problem?origin=mail>.

> [BB] I hadn't investigated Nigeria's figure although I was surprised by it (as I was with a number of the others).

	[SM] Yes, but you did take time and effort to discuss the for you unexpected high number for China, do you see a problem with such selective scrutiny?

> Nonetheless, the user experience is likely to be closer to that of the best CDN than the average CDN.

	[SM] That would be true only, if all content would be available via all CDNs. In my experience that is simply not true, especially in parts of the world that only have one close-by CDN.

> The ISPs in each country will tend to choose the better or best CDNs.

	[SM] That is not how the internet works... I want to access content and typically do not care much where that content is hosted (as long as "performance" is acceptable). Note how my ISP is only involved in connecting me to the internet. Yes, I understand that ISPs and CDNs play DNS games to route requests to the nearest DC (and I am fine with that, but that is IMHO an optimization not core functionality).

> Taking the average of all seven would not be appropriate, because that would include the CDNs without any relationships to the ISPs

	[SM] You seem to be looking at the whole thing purely from the business side of ISPs interacting with CDNs (in which context L4S prioritization of short RTT paths over long RTT paths will shift the balance of negotiating power towards the ISPs, I note), but for end-users that is not what the internet is about. The relevant number clearly is the access latency to the content users are interested in, which I admit is hard to deduce, but taking the averafe RTT to the closest CDN clearly is at best a lower bound for the true number.

> in that country (which is why the average figure you suggest I should use is completely inappropriate).

	[SM] Well, is is intriguing that your actaul claim is:
"Beganovi ́c [Beg19] provides the average RTT mea- sured using ICMP ping from probes in each country to sites known to be served by CDNs." which IMHO is exactly  that average figure that you just declared as unfit for purpose. I do however agree that the whole approach is unfit for purpose, but if you commit to such an approach I expect that you at least execute it correctly and that, since misunderstandings do happen, you go and correct the numbers if shown that they are deficient. 

> I took the figures published in that study as an indication of what natural market selection was doing.

	[SM] But you did not describe them as such.

> Also, some ISPs use multi-CDN solutions where the lowest latency CDN wins. I presume that is also why the authors of the study did it that way.

	[SM] I disagree, the study clearly is a show case to demonstrate what can be done with the rich data that RIPE's ATLAS probe net can generate, it is not a scientific paper targeted at answering a specific question in the field. But let's keep ripe out of this, I have no gripes with the study per se, but with your misunderstanding/misrepresenting the results from that study.
Ans again, I am looking at the issue from the perspective of an end-user, not from an ISP's perspective.

> That should mean the figures could be a little optimistic,

	[SM] Or pessimistic, depending on the CDNs an ISP has contracts with... This is really getting absurd, we have simply too little data to make meaningful statements, so let's not make any, okay?

> but I would not expect the difference to be large as long as there are enough samples in a country.

	[SM] But unfdortunatelly you did not even check that, comparing Nigeria with a measly 4 probes with Germany with >1000 (and hence the full set of 50 probes in the study) seems problematic. 

> I ranked all the countries in order of size of Internet user population, then once I had reached 90% of Internet users, I excluded the smaller user populations as a way to exclude the smaller samples.

	[SM] Yes, and I did not really object to this step (I consider the whole approach to be problematic, but that step seems in itself okay).

> Most importantly, criticism like this is always useful for improving the next study.

	[SM] Does this mean you are not going to change the numbers in https://arxiv.org/pdf/2107.01003.pdf and more importantly in https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled-17?

> That is what we had done prior to this study (surely you must have read the experiments by now, which do exactly what you list above - and more).

	[SM] Well, I asked for references, but the only one I brought in myself, ad that seems to indicate even a clasic target of 5ms could be acceptable.But if you have more studies showing how PIE behaves with different delay targets, please post references here. 

> But that doesn't help determine /prevalence/ of the different RTTs across the world, which is what the study in [PI2param] adds. 

	[SM] Except it does not, as A) as demostrated the actual numbers are wrong (or at the very least the description of the numbers is actively misleading), and B) the whole approach is not giving a representative estimate of the prevalence os different RTTs, as it explicitly targets a class of paths that by virtue of the CDN business model are likely to be relative short. 
	I believe there is a reason, why this study is not peer reviewed, and I doubt it would pass muster with attentive reviewers.

> [BB] That is not a fair characterization.

	[SM] What that your 15ms is the result of 3 numbers is pretty much indisputable, and that these numbers have different groundings in reality should also not be surprise:

target = weighted TCP geometry factor ∗ safety factor ∗average RTT 

1) geometry factor: there is a deduction of geometry factors for three different TCP regimes, from which a weighted average is calculted based on some weak numbers of relative prominence.
	While the tests with PIE at 5 and 20ms cited above indicae that getting the geometry factor wrong will have only a small throughput cost, this is not a hrad number but still the best of the three

2) safety factor:  Therefore, a ’safety factor’ is applied to the average RTT between users and CDNs, which has to allow for the statistical distribution of RTTs to CDNs, particularly for users in rural areas [KKFR15], who will be further from the nearest CDN and who are also likely to have least bandwidth and therefore be least willing to see it eaten by under-utilization. The safety factor also has to allow for flows between clients and servers other than in CDNs. As an interim educated guess, we apply the safety factor, f = 2.
	As you state your self that is (at best) and educated guess (without reference what data the "educated" part is based on)

3) average RTT: as I have shown the number in the paper is not what the paper describes the number to be, so either one of the two is plain wrong, which makes this number highly dubious.

So I stand to my assessment that these three numbers are neither robust not reliable. OTOH I was not expecting anything better, since it seems clear that using 2019 numbers in 2020 to justify a choice made years before can not be a reflection of how the choice was made in the past.

> Incidentally, I have been updating the analysis in PI2 param over the past few weeks. It ends up with a higher global weighted average, but then it has to be adjusted down again because there is evidence (from speedtest.net)

	[SM] Which completely lacks information about the spatial distribution of measurement nodes and end-points.... and hence can not even be sanity checked. 

> that latency has been reducing year-on-year since the data was gathered. So we still end up at about target = 15ms,

	[SM] Yes, because the goal of this whole exercise is to retro-actively justify the already committed upon 15ms number... 

	My point is more that the whole approach seems wrong, because we know that target <= 15ms will work reasonably well over the existing  internet with its average RTT >> 34ms, so the whole theory of how to select the classic target simply is wrong.

AND, to repeat myself, it seems that instead of theoretical deduction of questionable relevance, it should be relatively easy to empirically justify a selection of a classic target, simply by running experiments with different classic targets and assess the consequences of the choice in relevant dimensions.

> but with the caveat that it could have to be reduced further soon. And other caveats - you'll see the new outcome in the dualq draft before Monday's draft deadline. The new PI2param study will probably come out after that - as soon as it works through ArXiv's review process.

	[SM] Not sure what ArXiv's reviews here, but it does not look like you are talking about a true peer review of the scientific content here... more of a custodial check whether formal criteria are fulfilled.

Regards
	Sebastian

---

From: <[email-masked]>
Date: Mon, 25 Oct 2021 07:30:19 +0000

Bob,

a clear statement under which networking conditions L4S co-exists well with today’s standard transport protocols as offered by the draft is fair. An operator should make sure, that target customers can be served by the proposed RTT from appropriate CDN/service platforms before starting experimentation, guidance should clearly say that (as an aside note, simple ways to determine per flow RTT for providers are useful). As far as I understand, minimising RTT at economical conditions is the dominating trend of CDN location selection within the last decade. I think, it’s time to start gathering experience with non-buffering default forwarding behaviours and suitable transport protocols on broader scale. A number of upcoming communication services depend on scaling low delay and low jitter network behaviour.

Regards,

Ruediger

Von: tsvwg <[email-masked]> Im Auftrag von Bob Briscoe
Gesendet: Samstag, 23. Oktober 2021 10:48
An: Sebastian Moeller <[email-masked]>; TSVWG <[email-masked]>
Betreff: Re: [tsvwg] problematic numbers in draft-ietf-tsvwg-aqm-dualq-coupled

Sebastian,

Dear list

[BB] As the dualq draft makes clear, quoting:
"Frankly, these factors are educated guesses, but with the emphasis closer to 'educated' than to 'guess' (see [PI2param<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#PI2param>] for background investigations):"

Prior to this study, the choices of target in PIE, CoDel and any other AQMs were based on no particular knowledge of the global situation at all, other than non-scientific anecdote. This study states it is applicable to PI2 and PIE. So their settings of target are now educated guesses, while CoDel's is just a guess? Given the deployment of CoDel, it would be useful to have a study of the utilization resulting from CoDel's target being set to 5ms.

More...

[BB] I hadn't investigated Nigeria's figure although I was surprised by it (as I was with a number of the others). Nonetheless, the user experience is likely to be closer to that of the best CDN than the average CDN. The ISPs in each country will tend to choose the better or best CDNs. Taking the average of all seven would not be appropriate, because that would include the CDNs without any relationships to the ISPs in that country (which is why the average figure you suggest I should use is completely inappropriate). I took the figures published in that study as an indication of what natural market selection was doing. Also, some ISPs use multi-CDN solutions where the lowest latency CDN wins. I presume that is also why the authors of the study did it that way.

That should mean the figures could be a little optimistic, but I would not expect the difference to be large as long as there are enough samples in a country. I ranked all the countries in order of size of Internet user population, then once I had reached 90% of Internet users, I excluded the smaller user populations as a way to exclude the smaller samples.

That is what we had done prior to this study (surely you must have read the experiments by now, which do exactly what you list above - and more). But that doesn't help determine /prevalence/ of the different RTTs across the world, which is what the study in [PI2param] adds.

Incidentally, I have been updating the analysis in PI2 param over the past few weeks. It ends up with a higher global weighted average, but then it has to be adjusted down again because there is evidence (from speedtest.net) that latency has been reducing year-on-year since the data was gathered. So we still end up at about target = 15ms, but with the caveat that it could have to be reduced further soon. And other caveats - you'll see the new outcome in the dualq draft before Monday's draft deadline. The new PI2param study will probably come out after that - as soon as it works through ArXiv's review process.

Bob

Regards

  Sebastian

--