Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

From: Bob Briscoe <[email-masked]>
Date: Mon, 26 Jul 2021 22:19:18 +0100

Jonathan,

On 22/06/2021 20:38, Jonathan Morton wrote:
>> On 22 Jun, 2021, at 5:02 pm, Markku Kojo <[email-masked]> wrote:
> Up front, let me say that I mainly agree with your approach to this.
>
>> And, if the number of flows is increased, then the time between the marks decreases. So, we need to be careful in understanding what we are modelling. Otherwise, if we just play with the formulae, the outcome is just crap.
> So, rather than playing with formulae, I organised an empirical test over the weekend to see what the effects really are.  That accounts for the long pause before replying.
>
> 	https://sce.dnsmgr.net/results/mss-tests/
>
> The test involves three flows with the same CC algorithm running through a common single-queue, single-instance AQM.  The AQM is Codel as an exemplar of time-domain marking, or PIE as an exemplar of packet-mode marking, both with ECN enabled.  The test was repeated for Reno and CUBIC, as exemplars of standards-track CC, and DCTCP as an exemplar of 1/p response.  The path parameters are 50Mbps throughput, 80ms RTT, sufficient to keep CUBIC out of "Reno compatibility" mode.
>
> The three flows differ in terms of packet and segment size.  Flow 1 is a conventional bulk flow using a full 1460-byte MSS.  Flow 2 reduces this to 730-byte MSS.  Flow 3 uses a 1460-byte MSS, but goes through a path with reduced MTU (with PMTUD disabled) causing fragmentation into two packets per segment, so has the same packet rate at the AQM as Flow 2.

[BB] Can you quantify the MTU? So we know the sizes of the first and 
second of each pair of fragments. I see a legend label later with 1280 
in it. Is that the MTU? It doesn't actually matter to the main outcome, 
I'd just like to be able to double check.

> Fragmentation reassembly is as per RFC-3168.
>
> A well-conditioned congestion system should give each flow roughly equal application goodput on average.  Some allowance can be made for bandwidth lost to headers in the smaller packets, but this is a small effect.  But what do we actually get?

[BB] In /this/ exercise, we are not trying to make flows with different 
packet sizes have the same bit-rate, if they didn't have the same 
bit-rate without fragmentation.

While that might be an interesting research project, it is boiling the 
ocean compared to the task in hand. We are only trying to ensure that 
flows that are reframed at a lower layer, then reassembled, end up with 
the same rate as if they had not been reframed  then reassembled (modulo 
any slight differences due to differences in total header sizes, as you 
say). That is the art of designing an experiment that only changes the 
one parameter of interest.

> The theoretically ideal case is DCTCP with time-domain marking, where the expected goodput is exactly equal for all three flows.  The actual observed goodput is not quite equal, but reasonably close and very consistent over a 10-minute period:
>
> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-fq_codel-plot.html

[BB] You started out the email saying CoDel. However, the title of this 
plot (and the file name) says fq_codel. Surely you didn't mean to use FQ 
for this experiment? If the bottleneck includes a /scheduler/ designed 
to give each flow the same bit-rate, it will do that - irrespective of 
packet sizes, fragmentation, framing, or anything else for that matter.

>
> But if we apply packet-mode marking, the smaller packets are markedly disadvantaged compared to larger ones, approximately in proportion to the average packet size, and again this is consistent for 10 minutes straight:
>
> 	https://sce.dnsmgr.net/results/mss-tests/dctcp-pie-plot.html

[BB] This proves that the disadvantage of using small packets has 
nothing to do with fragmentation or reassembly, because both flows 1 & 2 
do not go through fragmentation/reassembly, and flow 2 has half the rate 
of flow 1, because its packets are half the size. This is the well-known 
fact that flows using smaller packets disadvantage themselves. That has 
nothing to do with fragmentation or reassembly.

In fact, it shows exactly what I predicted. Even though flow 3 sends the 
same size packets as flow 1 and is identical in every way, it ends up 
with half the rate, the only difference having been that flow 3 goes 
through RFC3168 fragmentation and reassembly with the AQM marking ECN in 
between.

>
> I do not have running code with which to test a "marked bytes preserving" reassembly rule.  However, such a rule could only possibly influence Flow 3, as that is the only one that's fragmented.  In theory it would be elevated to the level of Flow 1, leaving Flow 2 as the only disadvantaged flow.
>
> You might say that using small MSS is not reasonable for capacity-seeking traffic.  But recall what Paul Vixie pointed out in the other thread about UDP Options, that the MTU over Internet paths could reasonably increase in future.  This implies that "jumbo" TCP segments would, at least for some time, share space with traffic still using today's Ethernet-sized packets, and the latter would be in the analogous position of Flow 2 in this test.  So do not dismiss Flow 2 as irrelevant to your interests.
>
> Moving on to Reno, we see roughly the same effect, though the large Reno sawtooth means we have to peer through a lot of noise to see the trends.  Nevertheless they are similar to DCTCP over a long average:
>
> 	https://sce.dnsmgr.net/results/mss-tests/reno-fq_codel-plot.html
> 	https://sce.dnsmgr.net/results/mss-tests/reno-pie-plot.html
>
> And further likewise for CUBIC:
>
> 	https://sce.dnsmgr.net/results/mss-tests/cubic-fq_codel-plot.html
> 	https://sce.dnsmgr.net/results/mss-tests/cubic-pie-plot.html
>
> These results with standards-track CC confirm those from DCTCP, and the same logic applies.
>
> Current practice on the Internet is to use time-domain ECN marking (because Codel is the most widely deployed AQM with ECN enabled) and RFC-3168 fragment reassembly.  The above results show that this is clearly superior to packet-mode marking with either RFC-3168 or "byte-preserving" reassembly rules.  Hence we should probably try to avoid encouraging the latter behaviour in new Internet specifications.

As above, the PIE results prove exactly what I predicted. And the 
FQ_CoDel results don't say anything about the effect of fragmentation 
and reassembly on flow rate, because flow rate is enforced by the scheduler.

Now, it will be interesting to see what happens if the AQM is CoDel 
rather than FQ-CoDel.

Cheers

Bob

>   - Jonathan Morton

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 27 Jul 2021 04:37:17 +0300

> [BB] Can you quantify the MTU? So we know the sizes of the first and second of each pair of fragments. I see a legend label later with 1280 in it. Is that the MTU? It doesn't actually matter to the main outcome, I'd just like to be able to double check.

Agree that the series labels aren't as clear as they should be.  The third flow uses an MSS of 1460 (same as the first flow), but is sent through a link with an MTU of 1280, with MTU discovery disabled so that it gets fragmented there.  The second flow uses a smaller MSS and is not fragmented.

> [BB] In /this/ exercise, we are not trying to make flows with different packet sizes have the same bit-rate, if they didn't have the same bit-rate without fragmentation.
> 
> While that might be an interesting research project, it is boiling the ocean compared to the task in hand. We are only trying to ensure that flows that are reframed at a lower layer, then reassembled, end up with the same rate as if they had not been reframed  then reassembled (modulo any slight differences due to differences in total header sizes, as you say). That is the art of designing an experiment that only changes the one parameter of interest.

When we already have a simple solution that also solves the larger problem, why not just use it?

> [BB] You started out the email saying CoDel. However, the title of this plot (and the file name) says fq_codel. Surely you didn't mean to use FQ for this experiment? If the bottleneck includes a /scheduler/ designed to give each flow the same bit-rate, it will do that - irrespective of packet sizes, fragmentation, framing, or anything else for that matter.

If you look more carefully, you'll notice that we used fq_codel with the "flows 1" argument, which disables the FQ mechanism and places all traffic in the same FIFO and the same set of AQM state.  So we accurately describe it in prose as just "Codel".  We could just as validly have used Cake with the "besteffort flowblind" arguments to achieve a similar effect.

> [BB] This proves that the disadvantage of using small packets has nothing to do with fragmentation or reassembly, because both flows 1 & 2 do not go through fragmentation/reassembly, and flow 2 has half the rate of flow 1, because its packets are half the size. This is the well-known fact that flows using smaller packets disadvantage themselves. That has nothing to do with fragmentation or reassembly.
> 
> In fact, it shows exactly what I predicted. Even though flow 3 sends the same size packets as flow 1 and is identical in every way, it ends up with half the rate, the only difference having been that flow 3 goes through RFC3168 fragmentation and reassembly with the AQM marking ECN in between.

This is an accurate statement of the problem to be solved.  Time-domain marking (as used by Codel) combined with the existing RFC-3168 reassembly rule solves the problem.  Packet-domain marking exposes the problem on both flows 2 and 3, and your proposed reassembly rule solves it for only flow 3, not for flow 2.

 - Jonathan Morton