Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

From: Markku Kojo <[email-masked]>
Date: Thu, 3 Jun 2021 14:41:40 +0300 (EEST)

Hi David, all,

Catching up ...

I'm afraid there is an open/ongoing discussion on this where we have not 
reached concensus. Last message on the topic is here:

  https://mailarchive.ietf.org/arch/msg/tsvwg/AtEI72QCFhOWOn9d6xNcrssTVzs/

Note that while this discussion is much about splitting IP packets to 
smaller fragments/frames/PDUs and reassembling/decoding these smaller 
PDUs back to larger IP packets, it does, however, relate to the problem 
in ECN encapsulation draft on what to do when framing boundaries do not 
necessarily align with packet boundaries. This is because the case when 
boundaries do not align often INVOLVES also the case where IP packets are 
splitted into smaller L2 PDUs (or several smaller IP packets are gathered 
into a larger L2 PDU). So, the solution should be based on the same 
known-to-work method no matter whether we are fragmenting/reassembling IP 
packets or encoding/decoding IP packets to/from L2 PDUs. As the discussion 
referred in the above message is longish, I'll try summarize the 
problem space in the end of this message.

And, then there is also another thread that Bob initiated and I seemingly 
have not replied although promised (I was away from any IETF work in 
April due to family emergency and only resuming now, my apologies). That 
thread is about the (additonal but not independent) problem where L3 
packet and L2 PDU boundaries do not align. The last message on the thread 
is here:

  https://mailarchive.ietf.org/arch/msg/tsvwg/3la3kG5-JLU2OPx3zGxxmhWGYEo/

I will send my notes on that very issue separately tomorrow.

I believe that the best solution to allow ECN encap draft to move forward 
is that the draft does not say anything on the topic except points to a 
new draft (the one that has been envisioned to handle the IP 
fragmentation problem and would include also the handling of not aligned 
packet/frame boundaries) and we initiate such new draft before ECN encap 
draft gets published.

Below I try to summarize the problem with the two suggested paragraphs 
with two SHOULDs.

1. The two paragraphs (SHOULDs) are contradictory: there is no algorithm 
that has been shown to be able to correctly fulfill both requirements 
(please see the first message referred above where I explain why the 
algorithms that Bob has suggested do not work correctly).

2. There is no actual algorithm in the draft for a potential implementor 
that would allow for a proper implementation (nor is there experimental 
evidence showing any such algorithm would work correctly).
With the relatively vague first SHOULD, it is very unlikely anyone would 
get it right, I think.

3. RFC 7141 has been used as the guideline for the new proposed text
    which makes resolving the issue even more problematic because the
    recommendations in RFC 7141 are the origin of the problem on how
    to handle fragmentation/reassembly as well as encapsulation/
    decapsulation of IP packets/L2 frames when there is an AQM
    marking/dropping fragmented/splitted PDUs. As said, this is not
    independent of the problem with not aligned packet/frame boundaries.
    The problems that we have at hand are as follows:

  a) The original paper on RED suggested byte-mode operation where
     byte-mode dropping/marking would adjust drop/mark probability
     of smaller fragments (to have lower probability) such that with
     the same level of bit-congestion the RED AQM would mark/drop
     approximately at the same point a packet for a flow being
     fragmented and for a flow with full-sized packets, i.e., it
     would treat fragmented and non-fragmented traffic fairly.
     That byte-mode drop together with RFC 3168 reassembly logic
     results in fair and correct behavior for Standards track
     congestion control, loss-based CC included.

  b) Recommendation in RFC 7141 to not do byte-mode drop but
     instead use packet-mode drop (with equal drop/mark probability
     regardless of PDU size) will treat fragmented (splitted)
     traffic unfairly, yielding fragmented traffic suboptimal
     performance (as Bob has indicated several times). Therefore,
     packet-mode drop would need an algorithm at reassembly/decapsulation
     that SHOULD approximately preserve the proportion of PDUs(/bytes)
     with congestion indications arriving and leaving.

     However, there is no known algorithm that could do this correctly
     at reassembly/decapsulation as stated in item 1 above.
     More importantly, even if such algorithm existed it cannot
     work with non-ECT AQMs, i.e., with loss-based congestion control
     for which no adjustment of congestion indications can be done
     at the reassembly/decapsulator. But, one can achieve the correct
     outcome in a single place: in the AQM algorithm itself by employing
     byte-mode dropping/marking; it works correctly also for the majority
     of the traffic today, that is, for traffic employing loss-based
     congestion control.

     Moreover, doing adjustment at the reassembly/decapsulator is
     architecturally not very good solution because we would
     need such an algorithm at several places (at receiving
     enpoints, at tunnel egress, at L2 decapsulator) which
     introduces quite unnecessary complexity in several places.

  c) Recommendation in RFC 7141 is not applicable with AQMs that do
     not use probabilistic dropping/marking. E.g., it would result
     in incorrect behavior with CoDel AQM that employs deterministic
     dropping (please see more detailed explanation in the first
     message referred above) and with any potential new AQM that
     does not employ probabilistic dropping/marking.

  d) Although RFC 7141 is BCP, the recommendation in it is not based
     on any deployed mechanism (or at least I am not aware of any such
     best practice) nor on any published/evaluated algorithm that has
     shown to work. On the other hand, there is quite a bit of
     experimental evaluation on RFC 3168 reassembly & byte-mode
     drop/mark (although more high-quality evaluation would be useful).

Thanks,

/Markku

On Tue, 25 May 2021, David Black wrote:

> 
> As draft shepherd and a WG chair, I believe that these drafts resolve the last of the open issues from WG
> Last Call.
> 
>  
> 
> In the next week or two, I will prepare the shepherd writeups and submit these drafts to our AD for further
> review towards IETF Last Call and publication as RFCs.
> 
>  
> 
> Thanks, --David
> 
>  
> 
> From: Bob Briscoe <[email-masked]>
> Sent: Tuesday, May 25, 2021 10:35 AM
> To: Black, David; [email-masked]
> Cc: Donald Eastlake; John Kaippallimalil
> Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution
> 
>  
> 
> [EXTERNAL EMAIL]
> 
> David, tsvwg list,
> 
> As promised yesterday, we just posted a new rev of ecn-encap-guidelines-16, with text based on your
> (David's) suggestions below. To add the references and to avoid some repetition, I twiddled the order round,
> but otherwise kept the text intact.
> 
> I also took the opportunity to post a new rev of rfc6040update-shim, 'cos I noticed Geneve has been
> published as an RFC. There were also a couple of words edited in my local copy as agreed on the list a few
> months ago.
> 
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-encap-guidelines [datatracker.ietf.org]
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-rfc6040update-shim [datatracker.ietf.org]
> 
> Cheers
> 
> 
> 
> Bob
> 
> On 24/05/2021 13:50, Bob Briscoe wrote:
>
>       David, Thx for bringing this one up. See [BB] inline,
>
>       On 22/05/2021 01:02, Black, David wrote:
>
>       On another topic, I believe that I have good news to pass along on the ECN encapsulation
>       drafts.
>
>        
>
>       The current situation is that the 6040update-shim draft is ready for RFC publication to be
>       requested, but there's an open issue in the ecn-encap draft on the contents of this
>       paragraph in Section 4.6 (Reframing and Congestion Markings),
>       https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-encap-guidelines-15#section-4.6
>       [datatracker.ietf.org]:
>
>        
>
>          Congestion indications SHOULD be propagated on the basis that an
>
>          encapsulator or decapsulator SHOULD approximately preserve the
>
>          proportion of PDUs with congestion indications arriving and leaving.
>
>        
>
>       Digging further, this area appears to be dealt with in greater length and detail by RFC
>       7141 (Byte and Packet Congestion Notification) Section 2.4 (Recommendation on Handling
>       Congestion Indications When Splitting or Merging Packets),
>       https://datatracker.ietf.org/doc/html/rfc7141#section-2.4 [datatracker.ietf.org].  The
>       short summary is that the quoted sentence is generally correct with RFC 7141 containing a
>       more comprehensive discussion including an exception.  As RFC 7141 is a BCP, I suggest
>       treating it as authoritative on this matter for now, leaving redesign in this area to a
>       possible future draft (as we did in the 6040update-shim draft wrt RFC 3168 fragment
>       reassembly requirements).
>
>        
>
>       To carry this out, here's an initial ecn-encap draft text change suggestion (begins with
>       last two sentences in second paragraph of Section 4.6):
>
>        
>
>       OLD
>
>             Where framing boundaries do not necessarily align
>
>          with packet boundaries, the following guidance will be needed.  It
>
>          explains how to propagate ECN markings from layer-2 frame headers
>
>          when they are stripped off and IP PDUs with different boundaries are
>
>          reassembled for forwarding.
>
>        
>
>          Congestion indications SHOULD be propagated on the basis that an
>
>          encapsulator or decapsulator SHOULD approximately preserve the
>
>          proportion of PDUs with congestion indications arriving and leaving.
>
>        
>
>          The mechanism for propagating congestion indications SHOULD ensure
>
>          that any incoming congestion indication is propagated immediately,
>
>          not held awaiting the possibility of further congestion indications
>
>          to be sufficient to indicate congestion on an outgoing PDU.
>
>        
>
>       NEW
>
>             Where framing boundaries do not necessarily align
>
>          with packet boundaries, the provisions of Section 2.4 of RFC 7141
>
>          apply to propagation of ECN markings from layer-2 frame headers
>
>          when they are stripped off and IP PDUs with different boundaries are
> 
>    reassembled for forwarding. Those provisions include: "The general
> 
>    rule to follow is that the number of octets in packets with
> 
>    congestion indications SHOULD be equivalent before and after merging
> 
>    or splitting." See RFC 7141 for the complete provisions and related
> 
>    discussion, including an exception to that general rule.
>
>        
>
>          In addition to adhering to the provisions of RFC 7141 Section 2.4,
>
>          the mechanism for congestion indication propagation SHOULD ensure
>
>          that any incoming congestion indication is propagated immediately,
>
>          and not held awaiting possible arrival of further congestion
>
>          indications sufficient to indicate congestion for all of the octets
>
>          of an outgoing IP PDU.
>
>        
>
>       END
> 
> 
> [BB] OK, this is indeed progress.
> 
>
>        
>
>       RFC 7141 (a BCP) would be added as a normative reference.
> 
> 
> [BB] I'll write that up now. And post a revised draft.
> 
> 
> Bob
> 
> 
>
>        
>
>       Comments?
>
>        
>
>       Thanks, --David (as draft shepherd)
>
>        
>
>       David L. Black, Sr. Distinguished Engineer, Technology & Standards
>
>       Infrastructure Solutions Group, Dell Technologies
>
>       mobile +1 978-394-7754 [email-masked]
>
>        
> 
> 
> 
> -- 
> 
> ________________________________________________________________
> 
> Bob Briscoe                               http://bobbriscoe.net/ [bobbriscoe.net]
> 
> 
> 
> -- 
> 
> ________________________________________________________________
> 
> Bob Briscoe                               http://bobbriscoe.net/ [bobbriscoe.net]
> 
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 4 Jun 2021 04:02:55 +0300

> On 3 Jun, 2021, at 2:41 pm, Markku Kojo <kojo=[email-masked]> wrote:
> 
> I'm afraid there is an open/ongoing discussion on this where we have not reached concensus.

I think I agree with your analysis that preserving the number of bytes marked is probably the wrong thing to do.  The implementation complexity argument is fairly compelling in itself, particularly noting that there isn't any known "running code" which would place some upper bound on what the complexity is.

By contrast the RFC-3168 approach of "if any part of the packet is marked, all of it is marked" is very simple to implement, and this is proved by actual implementations in the wild.  This approach is also consistent with the behaviour when ECN is not in use, since if any fragment of a packet is lost, the packet can't be reassembled and is discarded in whole.

I'd like to try and back this position up with some mathematical rigour from a different direction.

One thing I noticed about Bob's position is that he's trying to preserve the number of bytes marked, but usually the transport is actually sensitive to the number of bytes, or possibly the interval of time, *between* marks.  These are two (or three) very different quantities, and it's important not to get them confused in the debate.

In engineering, a very useful tool for checking whether an answer is correct is "dimensional analysis".  Simply put, work out the units that your inputs and outputs are most naturally expressed as, and check that the formulae linking them are consistent with respect to the units.  In this case, we can look at the units of the AQM's marking function, the units of the transport's congestion control steady state criterion, and whether each approach to resolving ECN marks at reassembly preserves the quality of the marking with respect to those units.

For the purpose of this analysis, we are looking for the quantity that is most nearly preserved if we hold the AQM state constant or the transport in steady state, but vary packet size and/or link bandwidth arbitrarily.

Starting with the AQMs:

1: Codel is a time-domain AQM; its marking function is explicitly a frequency, expressed for performance reasons as a time interval between marks.  For analysis, we can equivalently express this quantity in units of marks/sec or its reciprocal, seconds per mark.  If we change the bandwidth or the packet size while holding Codel's control law constant, the number of bytes in marked packets and/or the number of packets between marks will change, but the time between marks will stay constant.

2: A packet-domain marking function (as used in BLUE and many RED-family AQMs) maintains a probability that any given packet will be marked.  Amortised over a large interval, we can express this as a (fractional) number of marks per packet, or equivalently as packets between marks. If the packet rate increases, so will the number of marks per time interval; if the packet size decreases, so will the number of bytes between marks.

3: A byte-domain marking function, as Markku described for early RED work, is obtained by taking a nominally packet-mode probability function, but multiplying the probability by bytes/packet (and a constant scale factor).  The resulting unit is (fractional) marks per byte, or bytes between marks.

Now the transports:

1: DCTCP's steady state is two marks per RTT.  Since RTT has units of time, the steady-state criterion has units of marks/sec.  This is precisely the same as Codel's marking function.

2: Reno's steady state is BDP/2 RTTs per mark, where BDP has units of bytes (the two time terms in the formula cancel).  The steady-state criterion thus has units of byte-seconds per mark.  This does not exactly match any of the AQMs' control laws, but it seems likely that "seconds per mark" or "bytes between marks" would be a better match than "packets between marks".  Regardless, for constant performance with Reno, we would look for methods that preserve the byte-seconds per mark value.

3: CUBIC's steady state, when not in the Reno compatibility regime, is quoted in RFC-8312 as (BDP/MSS) = 1.054 * RTT^0.75 / p^0.75, where p is the (fractional) number of marks per segment, and the quantity BDP/MSS has units of segments.  But each segment, as seen by the transport, is MSS bytes long, and it is ignorant of any fragmentation that might happen at Layer 3 or 2 along the way - so if that occurs, "segments" is not the same as "packets" as seem by the AQM.  So we can convert p into a byte quantity by dividing it by MSS, yielding (BDP/MSS) = 1.054 * (RTT*MSS / p)^0.75. This reveals that for a constant cwnd, a constant byte-seconds per mark value is required, similar to Reno, and we can probably expect this to be true for other "TCP friendly" congestion controls as well.

So now we can turn to the methods of resolving congestion marks at reassembly.  I'll examine three candidate rules that seem relevant to the debate, though others probably exist:

1: "If any fragment of a packet is marked, the whole packet is marked."  This is the RFC-3168 rule, generalised to also cover the weird Layer 2 fragmentation scheme that sparked this debate.  It is clear that this rule preserves the number of bytes between marks and the time between marks (and therefore also the byte-seconds between marks), but not the number of packets between marks - except for the specific case where a fragment contains portions of more than one packet.  If these packets all belong to the same flow, then this is harmless to Reno and CUBIC since consecutive ECN marks are reported to the sender as a single event.  If multiple flows are involved, then this will stochastically amplify the congestion signal by having more than one flow react to a single mark - which seems tolerable in practice.

2: "Every mark is sacred, every mark is great."  The AQM produces one mark, and exactly one reassembled packet receives that mark; the reassembly process can make a reasonable choice as to which one.  This preserves bytes between marks, time between marks, and byte-seconds between marks *exactly*.

3: "The number of bytes marked should be preserved by reassembly."  This would tend to preserve the number of packets between marks better than the other two options, but that is not useful to any of the transports considered.  The distortion to the bytes between marks and time between marks metrics would be distorted proportionally to the difference between average fragmented and reassembled packet sizes, and the byte-seconds between marks measure would be distorted *quadratically* by that difference.

In summary, the "preserve the number of marked bytes" rule is mathematically unreasonable, as well as being much more complicated to implement than either the existing RFC-3168 rule, or the "sacred mark" rule, which actually preserve the quantities that need to be preserved.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 6 Jun 2021 00:06:14 +0100

Markku, see inline [BB] (I'll reply to Jonathan's email yesterday 
separately)

On 03/06/2021 12:41, Markku Kojo wrote:
> Hi David, all,
>
> Catching up ...
>
> I'm afraid there is an open/ongoing discussion on this where we have 
> not reached concensus. Last message on the topic is here:
>
> PDUs back to larger IP packets, it does, however, relate to the 
> problem in ECN encapsulation draft on what to do when framing 
> boundaries do not necessarily align with packet boundaries. This is 
> because the case when boundaries do not align often INVOLVES also the 
> case where IP packets are splitted into smaller L2 PDUs (or several 
> smaller IP packets are gathered into a larger L2 PDU). So, the 
> solution should be based on the same known-to-work method no matter 
> whether we are fragmenting/reassembling IP packets or 
> encoding/decoding IP packets to/from L2 PDUs. 

[BB] RFC3168 reassembly is not 'known to work', even for fragment 
reassembly that it is intended for. You acknowledge that it suffers from 
the fairness problem unless AQMs all use byte-mode marking (and none do 
- see later). RFC3168 reassembly does not even have any suggestion on 
what to do if framing boundaries overlap packet boundaries or where 
packets are smaller than frames - so it is not 'known-to-work' in any 
case, let alone in every case.

> As the discussion referred in the above message is longish, I'll try 
> summarize the problem space in the end of this message.
>
> And, then there is also another thread that Bob initiated and I 
> seemingly have not replied although promised (I was away from any IETF 
> work in April due to family emergency and only resuming now, my 
> apologies). That thread is about the (additonal but not independent) 
> problem where L3 packet and L2 PDU boundaries do not align. The last 
> message on the thread is here:
>
> I believe that the best solution to allow ECN encap draft to move 
> forward is that the draft does not say anything on the topic except 
> points to a new draft (the one that has been envisioned to handle the 
> IP fragmentation problem and would include also the handling of not 
> aligned packet/frame boundaries) and we initiate such new draft before 
> ECN encap draft gets published.

[BB] Let us now start talking about the content of that.

Before we do, can we make sure we're all on the same page regarding some 
basics that I believe are /facts/ about preserving markings when PDU 
boundaries change. Do you agree with the following table that I asked 
about earlier:

                    | marked    marked
                    | PDUs      bytes
-------------------+------------------
preserving prop'n  |  ==        ==
preserving number  |  !=        ==

IOW, do you agree that the three that are tagged as '==' are equivalent 
ways of expressing the same thing, but different from the one tagged '!=' ?

> 1. The two paragraphs (SHOULDs) are contradictory: there is no 
> algorithm that has been shown to be able to correctly fulfill both 
> requirements (please see the first message referred above where I 
> explain why the algorithms that Bob has suggested do not work correctly).

[BB] Since draft-13 (May 2019) I haven't given any specific algorithms 
(because the chairs requested we don't discuss specifics).

Nonetheless, now that the drafts are progressing, I think we're allowed 
to discuss specifics. Although, let's not discuss implementation 
efficiency quite yet - let's keep to intent for now.

So here's one example of pseudocode for propagating ECN marking when 
frames are derived from an ordered byte stream of larger packets, but 
the frame and packet boundaries do not align:

----------------->+<---------------------------->+<------------------------------>+<----
Pkt1       |               Pkt2           | Pkt3                |
+-------------+-------------+-------------+-------------+-------------+-------------+---
|    Fr1      |     Fr2     |     Fr3     |    Fr4      | Fr5     |    
Fr6      |
+-------------+-------------+-------------+-------------+-------------+-------------+---

/* Algorithm A =======================
  */
#define TIMEOUT 1ms    // (say) as a default for the public Internet
bool marked, pending;

// On frame arrival
marked = ismarked(incoming_frame);
pending = pending || marked;

// On packet departure
if (marked|| (pending && time(now) >= expiry_time) ) {
     mark(outgoing_packet);
     expiry_time = time(now) + TIMEOUT;
}
pending = FALSE;
//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

Explanation:
1) First consider the case where marking is frequent enough that 
outgoing packets are always marked more often than every 1ms. Then as 
each outgoing packet is made ready to depart, it will be marked if the 
last frame to arrive was marked. This is based on the idea Jonathan 
suggested for preserving the proportion of marked PDUs.
2) Now, what if marks arrive infrequently, so that the time since the 
last outgoing mark exceeds the TIMEOUT? Then the outgoing packet will be 
marked if /any/ frame that it consists of was marked, because the timer 
condition and 'pending' will both be true. Because pending toggles on if 
any frame is marked, and pending is cleared after every packet is sent.

The value of TIMEOUT can be argued about for ever, but the intent is to 
ensure it's significantly smaller than most RTTs over the Internet.

Here's an alternative algorithm:

/* Algorithm B =======================
  */
#define TIMEOUT 400μs    // (say) as a default for the public Internet

// On frame arrival
if (ismarked(incoming_frame)) {
     balance += size(incoming_frame);
}

// On packet departure
if (balance > 0) {
     mark(outgoing_packet);
     balance -= size(outgoing_packet);
     expiry_time = time(now) + TIMEOUT;
} elseif (time(now) >= expiry_time) {
         balance /= 2;
}
//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

Explanation:
While marked packets are arriving often enough, this keeps the 
proportion of outgoing marked bytes the same as the proportion incoming 
(it therefore also preserves the proportion of marked PDUs).
But if marks arrive more infrequently, it preserves the /timing/ and 
therefore the /number/ of marks.

More specifically, the algo works is as follows:
* When a marked frame first arrives, the algo marks an outgoing packet 
immediately, even if there weren't enough incoming marked bytes for the 
outgoing packet size. But the algo holds the deficit in balance, so that 
if more marked bytes appear before the TIMEOUT, they will have to eat up 
the deficit before causing another outgoing mark. This is what ensures 
the proportion outgoing balances the proportion incoming.
* However, if the balance has been negative for more than the TIMEOUT, 
it is halved. And after another timeout the deficit is halved again, and 
so on, until eventually it will round down to zero. Then, with 
certainty, the next incoming marked frame, whatever its size, will 
trigger an outgoing mark immediately. In the intervening period, every 
time the deficit is halved, it becomes more likely that an incoming 
marked frame will cause an immediate outgoing mark, by taking the 
balance over zero.

I prefer the approach of gradually transitioning between preserving the 
proportion of marks and preserving the timing of marks, rather than a 
hard cut-off between the two. But the latter could be implemented simply 
by changing '/= 2' to '= 0' and using a longer TIMEOUT. Also note that 
the balance approach works whether some frames are larger than packets 
or some packets are larger than frames.

By definition these algorithms resolve a compromise (between preserving 
fairness and preserving timeliness). So if you're a cup-half-empty sort 
of person, you will always be able to argue that they're not truly 
correct. But if you're a cup-half-full-sort of person you will be able 
to argue that they're never too far off being correct.

>
> 2. There is no actual algorithm in the draft for a potential 
> implementor that would allow for a proper implementation (nor is there 
> experimental evidence showing any such algorithm would work correctly).
> With the relatively vague first SHOULD, it is very unlikely anyone 
> would get it right, I think.

[BB] Can you not remember what happened between draft-13 (May 2019) and 
draft-14 (Nov 2020)? Because we couldn't agree, the chairs asked me to 
remove the specific algorithm that was in the draft, and solely state 
requirements (which eventually resulted in the two SHOULDs). The chairs 
wanted to defer this debate, and not to constrain Layer-2 solutions to a 
particular algorithm in the mean time.

If you don't agree with that approach, don't argue with me. The chair's 
intent was that the question of how exactly to mark PDUs while reframing 
was controversial, so it should be solved in a separate draft, in order 
that this ecn-encap draft could go through quickly (ironic raised eyebrow).

>    independent of the problem with not aligned packet/frame boundaries.

[BB] RFC7141 is not "the origin of the problem". It described current 
AQM practice, which was not what researchers thought (and apparently 
still isn't). It is a BCP and many many people were involved in reaching 
consensus on it during the 6 years it progressed through the IETF. It 
explained why AQM practice at the time was not implementing byte-mode 
drop, and proposed a way to move forward with packet mode drop instead.

It included a survey which found that /none/ of the respondents had 
implemented byte-mode dropping (admittedly only 19% response rate, out 
of 84 vendors contacted. Plus we checked Linux RED, which also did not 
implement byte-mode drop).
See https://datatracker.ietf.org/doc/html/rfc7141#appendix-A

RFC7567 is also a BCP, and in section 4.4 it also recommends against 
byte-mode marking and byte-mode drop.

Since RFC7141 and RFC7567 were published, a new round of AQMs appeared 
(CoDel and PIE). Contrary to what you say in the first linked email 
above, CoDel does not do byte-mode drop or byte-mode marking{Note 1}. 
And PIE doesn't either.

DOCSIS PIE does do a limited form of byte-mode drop{Note 2}, but it does 
not support ECN and it's not under 'IETF-change-control' anyway (the 
authors offered to document it as an informational RFC). The only other 
algorithm I know of that has implemented byte-mode drop is the RED used 
in the ns2 simulator.

Later you say byte-mode drop is common industry practice and that it's 
the way forward. It's not industry practice at all, so it's hardly going 
to be the way forward.

{Note 1}: With CoDel:
* The likelihood of marking a specific packet does not depend on that 
specific packet's size, only on the average size of all packets. That is 
packet-mode marking, not byte-mode marking.
* The control law reduces the time between each mark/drop on a 
preordained schedule, so the likelihood of any one packet being 
dropped/marked is higher and increases faster if the average packet rate 
is lower (which can be because packets are larger on average).
* However, just because CoDel happens to mark packets on a preordained 
time schedule is not why it's important for a decap to propagate a mark 
in a timely fashion. The motivation is not specific to CoDel - it's to 
minimize the delay around the control loop whatever the AQM.

{Note 2}: DOCSIS PIE implements limited dependence on packet size, in 
order to reduce the chance of dropping small control packets (not 
because it considers that drop should depend on packet size). In the 
spec, it apologizes for not following RFC7567, but it includes 
safeguards against the consequential DDoS vulnerability that it opens up.

>     would treat fragmented and non-fragmented traffic fairly.

[BB] Can you give a more precise description of the AQM scenario you 
have in your mind here? The words "at the same point" make me suspicious 
that you might be thinking of something different to byte-mode drop. 
It's possible you're talking about byte-mode queue measurement (see 
RFC7141 for definitions). So please describe exactly how you think this 
AQM is working. I.e. whether there are per-flow queues or a shared 
queue; in what units it measures the queue, whether it marks packets of 
different sizes within the same flow with different likelihoods, etc.

>     congestion control, loss-based CC included.

[BB] Assuming for now that you are indeed talking about byte mode drop,...

The perfect complementarity you claim between byte-mode drop and RFC3168 
reassembly never existed, because no-one implemented byte-mode drop. 
RFC7141 did not /cause/ your perfect world to collapse. It investigated 
and discovered that byte-mode drop didn't exist, worked out why, 
articulated why, and proposed a way forward that would give the desired 
complementarity between network and hosts, without relying on byte-mode 
drop, which was clearly problematic to everyone.

You write the above as if byte-mode was deliberately introduced in RED 
so that it worked well with the fragmentation. I'm afraid that's a 
rewrite of history in reverse. The interaction between ECN and 
fragmentation hadn't even been thought about when byte mode and packet 
mode were included in the original 1993 paper on RED. Byte and packet 
mode were two options, which Sally Floyd left configurable because it 
wasn't completely clear at that time which one was most suitable (see 
the [pktByteEmail] reference in RFC7141 to Sally's 1997 email about this).

6 years later, when ECN first became an experimental RFC, there was 
still no mention or thought of fragmentation in RFC 2481. I became 
involved during the late stages of the update from RFC 2481 to what 
became RFC 3168 - 8 years after the original RED paper. Jon Crowcroft 
and I pointed out that there was no mention of fragmentation, as part of 
our attempt to make a more principled division between the IP and TCP 
parts of RFC3168 - see our review at 
https://bobbriscoe.net/pubs.html#ECN-IP (we were coming at it from a 
real-time media perspective. not just TCP). The fragmentation approach 
that was subsequently added to what became RFC3168 was still highly 
TCP-specific, even though it was at the IP layer. However, I didn't feel 
I should hold up RFC3168 to argue any further about it (unlike the 
present situation in tsvwg, I took the view that I had arrived late at 
the party, so allowing others to make progress was more important than 
me continually making the same point over and over, even tho I thought I 
was right).

Then we reach the 2008-2014 time-frame over which RFC7141 was developed 
- when it was discovered that no-one was implementing byte-mode drop 
anyway (see back where I started the story above).

>     of the traffic today, that is, for traffic employing loss-based

[BB] This is nostalgia for a past that never was. When no-one has done 
what you think they should have done, it is important to try to 
understand why.

>     introduces quite unnecessary complexity in several places.

[BB] Quite the opposite. The whole approach is designed to minimize 
in-network processing, and move it to the end-systems.
See https://datatracker.ietf.org/doc/html/rfc7141#section-3.5

At most decaps (including all tunnels), the framing doesn't change. 
Fragmentation is deprecated. And few networks apply ECN marking where 
there is no IP-awareness.

>     does not employ probabilistic dropping/marking.

[BB] Incorrect. See explanation earlier in this email.

>     drop/mark (although more high-quality evaluation would be useful).

[BB] This is completely the reverse of reality. As explained above, 
RFC7141 was based on a survey of AQM implementations at the time, and it 
recorded industry practice at the time. That practice has continued 
since then in CoDel, PIE and now PI2, DCTCP, etc.

In contrast, AFAICT, the /only/ use of byte-mode marking has been in the 
research community.

Regards

Bob

>> In the next week or two, I will prepare the shepherd writeups and 
>> submit these drafts to our AD for further
>> As promised yesterday, we just posted a new rev of 
>> ecn-encap-guidelines-16, with text based on your

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 6 Jun 2021 06:02:29 +0300

> On 6 Jun, 2021, at 2:06 am, Bob Briscoe <[email-masked]> wrote:
> 
> Before we do, can we make sure we're all on the same page regarding some basics that I believe are /facts/ about preserving markings when PDU boundaries change. Do you agree with the following table that I asked about earlier:
> 
>                   | marked    marked
>                   | PDUs      bytes
> IOW, do you agree that the three that are tagged as '==' are equivalent ways of expressing the same thing, but different from the one tagged '!=' ?

This question just tells me that you are still focusing on the wrong metrics.  The number and/or proportion of marked bytes is IRRELEVANT.  What matters are the number of bytes and/or time interval BETWEEN marks.

I explained this in more detail in my previous post.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 6 Jun 2021 08:21:24 +0300

> RFC3168 reassembly does not even have any suggestion on what to do if framing boundaries overlap packet boundaries or where packets are smaller than frames - so it is not 'known-to-work' in any case, let alone in every case.

Actually, it does offer such a suggestion, if you read between the lines a little bit.

The basic premise that both the marking and reassembly rules are based on is: "if you would have dropped a Not-ECT packet, try marking an ECT packet with CE instead".  If any fragment of a packet is dropped, then the whole packet is effectively lost; this is mirrored in the rules that propagate a CE mark to the whole reassembled packet if any fragment of it was so marked.

Applying the same logic to the case where a single Layer 2 frame may span (parts of) more than one IP packet, dropping that frame would cause all the packets it (partially) spans to be lost.  Therefore, the behaviour implied by RFC-3168 would be to propagate a congestion mark found on the Layer 2 frame to all of the packets that would have been lost if that frame were dropped.

We can debate whether this is better or worse than converting a single Layer 2 mark into a single CE mark, but the logic is at least consistent this way.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 9 Jun 2021 16:07:38 +0000

Not having followed the full discussion/history, for me preserving the probability of marks is important (whether on bytes or packets), and as marking/dropping is usually applied in the network according to a probability in a closed control loop, it is probably the best way forward. Even if marking is applied by time distance and/or number of bytes or a count algorithm, it will converge to the "right" probability. This means any reassembly that preserves the average probability is good enough. Due to multiple bottlenecks and RTT dependencies, marking/dropping probability to rate/throughput is not an exact science anyway (though we should try to approach this as much as possible in a reasonable way).

So for me following simple algorithms would be ok to use:
- only mark if the first (even partial) segment of the reassembled packet is marked (ignore marks on any other segments)
- apply a random probability on the packet based on the ratio of marked bytes in the packet and full packet size

I believe both algorithms keep both byte and packet probability equal. It might not fulfil the immediate requirement, but since an AQM applying a low probability is not an immediate action anyway, it is not so important. An immediate AQM like the step threshold is marking with a high marking probability (100%) a series of packets, which is then also immediately carried forward during decapsulation by both algorithms.

Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
Sent: Sunday, June 6, 2021 5:02 AM
To: Bob Briscoe <[email-masked]>
Cc: [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 9 Jun 2021 22:37:09 +0300

> On 9 Jun, 2021, at 7:07 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

This would mean that any mark applied to a segment that covered only the middle or tail of a packet would reliably have no effect.  I firmly disagree with that approach, due to the possibility of accidentally triggering pathological conditions where a long series of marks is lost.

> - apply a random probability on the packet based on the ratio of marked bytes in the packet and full packet size

I can see where you're going with that, but I maintain - based on my analysis a couple of posts ago - that maintaining the number of marked bytes is simply the wrong approach.

The interval *between* marks, measured in either bytes or time, is the important property to preserve; this is actually easier to achieve and is natural to perform statelessly.  If the AQM device can rely on that behaviour, it can be designed to perform byte-mode or time-domain marking to match, which dovetails nicely with all existing transport implementations (whether with high-fidelity or conventional congestion control).

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 10 Jun 2021 23:35:23 +0000

Inline:

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Wednesday, June 9, 2021 9:37 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Bob Briscoe <[email-masked]>; [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

[K] It could as well mean that one outer packet marks 2 inner packets. Note that AQMs best apply random marking and that any pathological conditions will be quickly broken by the closed loop.

[K] These algorithm preserve the marking probability (whether bytes or packets), which boils down to exactly preserving the interval between marks. Not sure what the difference is?? The interval i=1/p with p the probability. So preserving the probability, preserves the interval... not?

Koen.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 10 Jun 2021 23:43:33 +0000

To clarify the second algorithm, "full packet size" could have 2 meanings: the size of the inner packet or the max packet size, independent of this packet's size.
This results in 3 algorithms:
- only mark if the first byte of the inner (reassembled) packet is marked (ignore marks on any other bytes/segments)
- apply a random probability on the inner packet based on the ratio of marked bytes in the inner packet and the inner packet size
- apply a random probability on the inner packet based on the ratio of marked bytes in the inner packet and the maximum inner packet size

Thinking further on this, it seems that the first 2 algorithms keeps both packet and byte probability, only if there is a strong correlation between the sizes of the inner and outer packets. If there is no correlation, these first two algorithms will carry the outer packet probability to the inner packets independent from their sizes, and the third algorithm will mark the inner depending on their size (gives more marking/dropping probability to larger packets).

If there is a strong correlation of packet sizes and the AQM marks with a byte probability, the second algorithm, will square the byte based probabilities. If the AQM marked with packet probability the tunnel decapsulation will mark with inner packet size dependent probability.

Assuming there is usually a large correlation between packet sizes and that AQMs that apply byte based probabilities are not often used, the first 2 are probably recommended? The third is quite tricky as you need to know exactly what is going on to avoid giving short packets an advantage.

Also taking into account why AQMs in routers would mark byte based or packet based is usually dependent on their capacity limit (being packets or bytes for buffer sizes and packets/second or bits/second for rates). If there is a packet size correlation in the tunnel, then it is logical to pass that through to the original senders, but if there isn't, it is logical to also spread the rate or buffer limit over all flows independent of their sizes (as they don't matter and are reshuffled by the tunnel anyway).

Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Wednesday, June 9, 2021 6:08 PM
To: Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>
Cc: [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
Sent: Sunday, June 6, 2021 5:02 AM
To: Bob Briscoe <[email-masked]>
Cc: [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 11 Jun 2021 08:48:10 +0300

> It could as well mean that one outer packet marks 2 inner packets. Note that AQMs best apply random marking and that any pathological conditions will be quickly broken by the closed loop.

I would class that as a poor workaround, since you are effectively excluding an entire, successful class of AQM algorithms from use with this approach.

> These algorithm preserve the marking probability (whether bytes or packets), which boils down to exactly preserving the interval between marks. Not sure what the difference is?? The interval i=1/p with p the probability. So preserving the probability, preserves the interval... not?

You are (approximately) preserving the number of marked bytes and the interval *in packets* between marks.  That is *not at all* the same thing as preserving the interval *in bytes or in time* between marks, given that the size of each packet changes at reassembly.

To ram this point home - since that is clearly required for understanding - let's take an extreme but realistic example: encapsulating 1500-byte IP packets into 53-byte ATM cells using PPPoA encapsulation (10-byte overhead per IP packet), and transmitting them over a CBR 12Mbps ADSL link (28,300 cells per second).  We will also assume there's a few smaller IP packets mixed in from application-limited flows.  Each ATM cell takes a 48-byte portion of the PPPoA-encapsulated payload, so it requires 32 of them to encapsulate one full-size PPPoA frame, and we can transmit 884 of those per second.  We will further assume, perhaps less realistically, that we have a way to apply congestion signals to individual ATM cells.

In your world, probabilities are sacrosanct.  So at some given AQM state, we apply a 1% marking rate to the ATM cells, and you expect that to be converted into 1% of IP packets being marked.  But that is also converting 283 marked ATM cells per second into just 8.84 marked IP packets per second, and increasing the byte interval between marks from 5300 to 150000.  You can see here that packet intervals are *not* the same as byte or time intervals.

In the world of Reno and CUBIC and Codel that actually makes up much of the Internet today, and even in the world of DCTCP, it is the byte/time interval between marks that is sacrosanct.  Neither of your algorithms above *even approximately* preserves this property in the scenario given (which doesn't even involve frames spanning packets, but the arithmetic is similar).  The RFC-3168 rule, however, *does* preserve it.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 11 Jun 2021 08:59:57 +0000

Did you consider the consequences too? With your approach it means that if you just mark 3% of the ATM cells, 100% of the inner 1500 bytes packets will get marked?
Is that what you intended?
While if there are small packets mixed in the inner stream of say 48 bytes size, they still get 3% marked. Is this what you intended too?

It all depends on how these marks are used by the congestion controller and what the Network congestion is related to: a bit-rate limit or a packet-rate limit.
If the outer packet size is constant (always a full MSS or ATM cell size) it doesn't matter if the network limit is packet- or bit-rate. Then it depends on the congestion control in the end-system. First does it keep a packet based window, or a byte based window (I assume byte based is usually used now). And then are CE's counted in "at least one mark" per RTT as in Reno/Cubic or in % of bytes marked to the total received (as in DCTCP-like).

As long as all combinations are possible, there won't be a good answer. We should decide what is the simplest for the expected way forward (scalable, marked bytes %). As we are going to have multiple marks per RTT or time interval, and those are measured in the ratio (bytes marked)/(bytes received or even bytes non-marked) I propose to keep that as the future recommendation, and decapsulation should keep this constant. In the near overload situation, which your example can be classified if packets get marked 100%, proper AQMs need to switch to drop anyway, which is rather matching your proposal I think.

Koen.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Friday, June 11, 2021 7:48 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Bob Briscoe <[email-masked]>; [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 11 Jun 2021 12:19:07 +0300

> On 11 Jun, 2021, at 11:59 am, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

Let me turn that around and ask you what would happen if 3% of the ATM cells were *lost*.  In that case, very few of the full-size packets could be reassembled, and throughput would drop to almost nothing.  The analogous congestion response from a high percentage of ECN marked packets is a natural and obvious consequence of applying 849 marks per second.

So don't do that, then.  Stop thinking in terms of "probability per packet", and start thinking in bytes or seconds instead.  Everyone else is.

There are two currently known AQM designs which implement this approach:

1: Calculate a time-domain marking schedule, as Codel does.

2: Scale the marking probability by the size of the unit being marked.  This would naturally scale down a "3% per 1500 bytes" marking rate to be more suitable for 53-byte cells.  Small packets would also be marked less often, but a capacity-seeking flow using smaller packets would still receive the same number of marks.

Incidentally, I am getting somewhat weary of having to explain everything from multiple angles separately to each interlocutor.  I already made the above points a few posts ago, and I do wish you would have read and absorbed the argument instead of just ignoring it.

> We should decide what is the simplest for the expected way forward…

The simplest rules to implement happen to be the ones that preserve the time/bytes interval between marks.

> …(scalable, marked bytes %).

The 1/p congestion response is sensitive to time between marks, *not* to the percentage of marked bytes.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 11 Jun 2021 10:39:10 +0000

>> Let me turn that around and ask you what would happen if 3% of the ATM cells were *lost*.

So you are still limit your thinking in terms of CE == DROP, which I thought was accepted not to be the most optimal and useful way forward... We can and MUST use ECN in a much better way if you want ECN to get really deployed AND used...

>> Stop thinking in terms of "probability per packet", and start thinking in bytes or seconds instead.  Everyone else is.

I think you are missing the point here: It is not the AQM that determines the marks. It is the end-system congestion control. If that uses the packet probability it will DRIVE the AQM to whatever it needs to converge. It is a closed loop, and the end system determines the rate/probability. We only can hope the AQM is responsive enough to quickly set that rate or probability. This is where CoDel clearly fails, so not a good example to use. We need to learn from our mistakes and forget "rules from the past" that are not useful...

Koen.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Friday, June 11, 2021 11:19 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Bob Briscoe <[email-masked]>; [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 11 Jun 2021 15:13:01 +0300

> On 11 Jun, 2021, at 1:39 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

Funnily enough, nearly all current ECN deployment is using conventional TCP congestion control (RFC-5681, RFC-8312), a time-domain marking strategy (RFC-8289) and the RFC-3168 reassembly rules, and it seems to work pretty well.  Since that is what's currently standardised in published RFCs, don't knock it until you've actually tried it.

> I think you are missing the point here: It is not the AQM that determines the marks. It is the end-system congestion control.

I agree, that is the correct way around to analyse the system.  Which is what I did in a previous post, using Dimensional Analysis.  That is precisely what I've been trying to get you to read and understand.

What I found, in brief, was that DCTCP has a steady state that is dimensioned in marks per second, while Reno and CUBIC have steady states dimensioned in byte-seconds per mark.  None of them are directly sensitive to packet or byte marking probability - that is just an emergent property of empirical measurement on an inappropriate assumed basis.

> If that uses the packet probability it will DRIVE the AQM to whatever it needs to converge. It is a closed loop, and the end system determines the rate/probability.

You're right that the AQM is effectively driven to the state needed to control the actual transport.  Which is why the AQM in my PPPoA example would never be driven to a 3% cell-marking probability except under extreme circumstances.  Instead, it would select a probability that resulted in a reasonable number of marks per second, or if you prefer, a reasonably large number of bytes between marks.  Codel simply operates directly in the time domain, neatly bypassing a few awkward steps on the way.

> We only can hope the AQM is responsive enough to quickly set that rate or probability. This is where CoDel clearly fails, so not a good example to use. We need to learn from our mistakes and forget "rules from the past" that are not useful...

I'll accept the Codel is not perfect.  However, it's the first AQM that matches with the problem domain well enough to deploy successfully with universal defaults, which I think accounts for its popularity.  If you continue to claim that it "fails", you're going to have to back that up with empirical data.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 11 Jun 2021 17:12:21 +0200

Hi Koen,

> On Jun 11, 2021, at 12:39, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

	[SM] I would agree that it is the endpoints that interpret the marks, but it clearly is the function of the AQM to actually set the marks, no? And for this to work, end-points need to interpret the marks in an agreed upon fashion so that AQMs can expect specific per-flow behavior to marking to occur, so the AQM can achieve its own goals (what ever these are). If push comes to shove all AQMs will need to drop packets (even in a fully L4S world there can be situations with queues exceeding available memory or similar). So marking is a way for an AQM to be nicer/gentler to flows (talking softly) but if the flow (or the aggregate) does not respond quickly enough the AQM will use the big stick it carries and start dropping packets (as dropping is the only solution to a bottleneck node's immediate problem if ingress traffic exceeds egress capacity for too long). 

> If that uses the packet probability it will DRIVE the AQM to whatever it needs to converge. It is a closed loop, and the end system determines the rate/probability.

	[SM] Only in the case of a single flow, other wise you have the interaction of multiple endpoints happening in the bottleneck buffers which are potentially outside of the control of each individual endpoint...

> We only can hope the AQM is responsive enough to quickly set that rate or probability.
> This is where CoDel clearly fails, so not a good example to use. We need to learn from our mistakes and forget "rules from the past" that are not useful...

	[SM] Not sure what you mean with Codel fails, though.My homenet work has been sitting behind a codel-style AQM for almost a decade now (first sfq, then fq_codel, then cake) and generally this works well. Sure for individual (under-responsive) flows codel can be a bit too gentle, but that mostly is ironed out by stochastic fair queueing, which tends to isolate the fall-out from misbehaving flows mostly to those misbehaving flows them selves. 

Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 14 Jun 2021 16:04:31 +0000

>> Which is what I did in a previous post, using Dimensional Analysis. 
Dimensional analysis is what you can use to check the validity of equations and further derivations from them. I didn't see any equation in your explanation. 

But I understand your point about the interval being constant for Classic CCs. The Congestion Controls determine if a "packet/byte probability per RTT" or an "RTT intervals between mark/drops" is used:
- DCTCP-like CCs use the marked byte (or packet) ratio per RTT (or other time interval), but marks are expected during every RTT (or other time interval)
- Classic CCs trigger reduction on occurrence of any number of bytes/packets dropped/marked per RTT (whether it is a packet of only 1 byte that is marked in that RTT or all packets and bytes marked in that RTT, the result is the same), and need a number of RTTs to recover from the reduction. If the flow is split in small packets and those are marked with the same random prob, they get smaller RTT intervals and a lower rate.

There were recommendations for AQMs to marking packet based on their size, to overcome this unfairness, but this is not often applied I understood. Probably better is to solve this again in the end system: by only triggering a congestion event if 1MSS of data is marked. This solves both cases: So it needs to collect at least 1500 bytes of marked packets  which come from either a series of 32 marks in a row (a large 1500b outer packet was marked and all 32 46b inner packets too) or it was NOT encapsulated in a tunnel, and an AQM is used that marks the small packets directly or it was encapsulated and a decapsulation works as a normal AQM and marks the packets with a probability (and in both cases marks 32 times more packets). Assuming the small packets are separately ACKed the 32 factor can be "undone" in the sender by only triggering after 1500 bytes were ACKed CE.

For scalable CCs this isn't needed, as the proportion of marked bytes and total bytes per interval is used. And for Classic it would only be needed if the TCP/QUIC sender is sending small packets (not if they are split in tunnels in the NW, assuming they keep the proportional factor.

I think this could be a simple way out of this otherwise already broken and complex mix of strategies...??

Koen.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Friday, June 11, 2021 2:13 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Bob Briscoe <[email-masked]>; [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 14 Jun 2021 19:45:20 +0300

> On 14 Jun, 2021, at 7:04 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

> Dimensional analysis is what you can use to check the validity of equations and further derivations from them. I didn't see any equation in your explanation. 

So you have not in fact read the post with the full explanation in it, in which the relevant formulae are in fact discussed.  Do I have to quote it in full for you to bother to find and read it?

> - DCTCP-like CCs use the marked byte (or packet) ratio per RTT (or other time interval), but marks are expected during every RTT (or other time interval)

The steady state of DCTCP is a constant number of marks per RTT.  That's what you get when you cancel the measurement of probability against the number of segments per RTT that the measured probability is applied to.  You really could simplify DCTCP's response to "each CE mark removes half a segment from the cwnd", thereby bypassing a lot of unnecessarily complicated fixed-point maths.  I can walk you through *exactly* how the DCTCP response equations simplify if you need to hear it.

At any rate, as the estimated path throughput changes, so does the required interval in packets or bytes between marks, but the required interval in *time* between marks is constant as long as RTT doesn't change (and on a given path, it shouldn't change much).  Hence the dimension that DCTCP is sensitive to is marks/sec, and an infrastructure that preserves that dimension in the congestion signal is desired.

> - Classic CCs trigger reduction on occurrence of any number of bytes/packets dropped/marked per RTT (whether it is a packet of only 1 byte that is marked in that RTT or all packets and bytes marked…

Yes, because when the packets are smaller there are more of them per second and fewer bytes between them.  Keeping a constant packet interval between marks therefore distorts the byte interval and the time interval between marks in the same direction.  Since the byte-seconds per mark dimension is what Reno and CUBIC are sensitive to, this also distorts their response.

> There were recommendations for AQMs to marking packet based on their size, to overcome this unfairness, but this is not often applied I understood.

I'm going to stop you there, and remind you that the predominant deployment of ECN AQMs today is a time-domain algorithm, namely Codel.

There are a number of other AQMs out there that may (or may not) behave as you describe, but are deployed without ECN support and thus can only *drop* packets.  The handling of dropped fragments is unambiguous, since the affected packets cannot be reassembled and must therefore be dropped in whole.  But you are keen to state that dropping and ECN should be different, so let's start by applying marks in the correct way when ECN is supported.

Since DualQ-Coupled and its PI2 algorithm is a new development, you should be in a prime position to make it operate, at least approximately, in either the time-interval or byte-interval domains rather than the packet-interval domain.  As I have already pointed out, this simplifies the requirements on downstream devices considerably.

> Probably better is to solve this again in the end system: by only triggering a congestion event if 1MSS of data is marked. This solves both cases: So it needs to collect at least 1500 bytes of marked packets  which come from either a series of 32 marks in a row (a large 1500b outer packet was marked and all 32 46b inner packets too) or it was NOT encapsulated in a tunnel, and an AQM is used that marks the small packets directly or it was encapsulated and a decapsulation works as a normal AQM and marks the packets with a probability (and in both cases marks 32 times more packets). Assuming the small packets are separately ACKed the 32 factor can be "undone" in the sender by only triggering after 1500 bytes were ACKed CE.

That sounds extremely complicated, compared to just Doing It Right in the first place.

I think it'd be helpful for some *other* WG members to weigh in on this topic for a change.  Does anyone else have an opinion they'd like to share?

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 14 Jun 2021 23:07:09 +0200

Hi Koen,

see below, prefixed [SM].

> On Jun 14, 2021, at 18:04, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> There were recommendations for AQMs to marking packet based on their size, to overcome this unfairness, but this is not often applied I understood. Probably better is to solve this again in the end system: by only triggering a congestion event if 1MSS of data is marked.

	[SM] Question, since the decapsulator node is not guaranteed to be the the endpoint for tunneled flows, how would it know each flow's MSS? Especially since flows can have quite different MSS. Sure, we could keep state per flow, but that seems pretty costly and intensive... 

Regards
	Sebastian

> This solves both cases:

	[SM] But will need an MSS-oracle to work well ;)

> So it needs to collect at least 1500 bytes of marked packets  which come from either a series of 32 marks in a row (a large 1500b outer packet was marked and all 32 46b

	[SM] ATM payload is 48 bytes (in a 53 byte cell)...

> inner packets too) or it was NOT encapsulated in a tunnel, and an AQM is used that marks the small packets directly or it was encapsulated and a decapsulation works as a normal AQM and marks the packets with a probability (and in both cases marks 32 times more packets). Assuming the small packets are separately ACKed the 32 factor can be "undone" in the sender by only triggering after 1500 bytes were ACKed CE.
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 15 Jun 2021 23:25:24 +0100

Jonathan,

I waited before replying to your email below until I had the time to try 
to unpick and deeply understand the way you think about all this. I 
would appreciate it if you would do likewise, and certainly please do 
not snip anything I say (like you did last time I put in a similar 
amount of effort into explaining all this on your terms). Instead, 
please try to understand and address each point.
See [BB] inline.

> 3: A byte-domain marking function, as Markku described for early RED work, is obtained by taking a nominally packet-mode probability function, but multiplying the probability by bytes/packet (and a…

[BB] One doesn't see flow rate formulae for Reno (or DCTCP, Cubic, etc) 
that say, "the rate is x if over PIE; but the rate is y if over CoDel". 
The outcome solely depends on the transports involved, which converge on 
a set of rates that fill the capacity irrespective of which AQM is at 
the bottleneck. Therefore the unit that a particular AQM uses to mark 
packets does not alter the steady-state outcome. This is a critical 
point, so if you disagree, there's no need to move on to the next part 
about transports - please stop here and solely explain why you think 
this point is wrong.

The fact that CoDel inserts an amount of time between marks is a 
distraction. CoDel's control law decreases the time between marks until 
it balances with the transports using it. The fact that RED, BLUE, PIE 
etc use a packet marking probability is also a distraction. They also 
adapt, increasing the marking probability to converge on 'the same 
outcome' as CoDel would, if placed in the same location with the same 
traffic sources.

By 'the same outcome', I mean that if you replaced one AQM at the 
bottleneck with another, the outcome would be:
* the same number of marked packets per second (or equivalently the same 
time between marks)
* AND the same marking probability.

If it existed, byte-mode marking (the early RED option) would have the 
same property, but let's only talk about that if we need to, when we get 
to re-framing, where it does make a difference.

BTW, the time taken to reach the outcome (the dynamic performance) will 
be different for different AQMs, but you're talking here about stable 
outcomes, which is indeed one of the two important considerations (the 
other being timeliness).

> 3: CUBIC's steady state, when not in the Reno compatibility regime, is quoted in RFC-8312 as (BDP/MSS) = 1.054 * RTT^0.75 / p^0.75, where p is the (fractional) number of marks per segment, and the…

[BB] One can substitute parameters with different units into 
steady-state flow rate formulae (as I show at the end, headed "Formula 
Transformations"). So if you happen to choose to express a formula in 
terms of a parameter with certain units, it doesn't prove anything about 
reassembling markings. It just says you have chosen to express the 
formulae in terms of a parameter with certain units.

Dimensional analysis is very useful to check one's work. But you're not 
doing dimensional analysis above (otherwise you'd be checking that the 
two sides of an equation have the same units). You're just expressing 
formulae in terms of a parameter with units that you are comfortable 
with, then convincing yourself that this proves that reassembly should 
use the units that you are comfortable with.

When you say '"segments" is not the same as "packets" as seen by the 
AQM', you are following the logic that the AQM examines the packets and 
knows what time to insert between marks. It doesn't. The AQM inserts a 
first guess at a time between marks, the transport responds, and if it 
doesn't respond enough, the queue doesn't shrink enough, so the AQM 
keeps on inserting less time before the next mark until the time between 
marks from the AQM causes the flow rate to balance at a steady queue 
around the target.

As explained earlier, if you observe a PIE or BLUE AQM from the outside, 
you might believe it is inserting a reducing time between marks. But 
internally, it is reducing a marking probability. Looking from the 
outside, the final probability it reaches can also be considered as the 
final time between marks that it reaches. So (again), the units the AQM 
adjusts internally are irrelevant to the steady state it reaches.

When I said that all these formulae can be expressed in terms of marking 
probability, p, rather than time between marks, t_m, that was not to 
prove that p is the correct unit to preserve on reassembly. It's just to 
prove that each transport's formula can be expressed in either unit, so 
the units chosen for a formula do not imply anything about which units 
should be preserved on reassembly. Your formulae are just convincing 
yourself that you can express formulae in terms of a parameter in units 
that you are comfortable with.

In order to convince anyone else, you're going to need to address the 
question of why your proposal (also in RFC3168) to preserve the time 
between marks leads to roughly x2 unfairness between flows that are just 
below and just above the fragmentation limit. Whereas preserving marking 
probability (or equivalently bytes marked), leads to no unfairness. 
That's a real-life phenomenon that doesn't depend on which units anyone 
chooses to reason about the world.

You can stop here, if you are convinced. If not, go to the end, to work 
through the transformations from formulae expressed in 'time between 
marks' into formulae expressed in 'marking probability'.

> In summary, the "preserve the number of marked bytes" rule is mathematically unreasonable, as well as being much more complicated to implement than either the existing RFC-3168 rule, or the "sacred…

[BB] In March 2020, I posted an example of the unfairness problem if 
reassembly preserves the time between marks. The unfairness is between 
flows of reassembled packets and packets that have not been fragmented. 
And I showed that preserving marking probability (or equivalently bytes 
marked) solves the problem:
You can still go straight to the schematic that I referred to: 
https://bobbriscoe.net/projects/netsvc_i-f/consig/encap/ecn-reassembly.pdf
(and FYI, here's the original posting: 
https://mailarchive.ietf.org/arch/msg/tsvwg/Da0sagcLnvPzh6xKFdHFRUZ9w5o/ )

That is the problem with the RFC3168 approach, and with your approach 
which has the same goal. I have carefully worked through your long 
email, but it doesn't even mention the problem I've levelled against the 
scheme you prefer. That is the whole nub of this argument. Please, you 
cannot continue to keep asserting that you are right, without engaging 
with that problem. And you cannot keep just asserting that I don't know 
what I'm talking about and you do, unless you refute my arguments.

In summary, your long email had three parts:
     #1 AQMs
     #2 Transports
     #3 Schemes to reassemble marking.
Part #3 consists of assertions that the reassembly scheme you assert is 
right is the one that preserve the units that you assert are right. But 
you haven't given any arguments for why, and you haven't even tried to 
support your arguments using the first two parts.

I suspect you think your arguments in #1 & #2 are proofs. So, I have 
explained (and proved):
1) why the units of the parameter that an AQM manipulates are irrelevant.
2) that a flow formulae can be expressed in terms of a parameter with 
units either in (time between marks) or in (marking probability). So 
choosing to express a flow formula in terms of a parameter with one unit 
or the other is just your choice. It proves nothing about which is the 
correct unit to preserve during reassembly. (Supporting maths follows at 
the end)

>   - Jonathan Morton

[BB] Regards

Bob

==Errors in your analysis of transports==

Your analysis of DCTCP is OK, but your analyses of Reno and Cubic 
contain errors:

* Reno: Given you define BDP in bytes, your criterion for RTTs per mark 
should be (BDP/MSS)/2 , not BDP/2.
     and a picky point: the const should be approx 2/3 not 1/2.
* Cubic: Obviously, your two formulae compared below cannot both be true 
(by dimensional analysis, ironically):
     (BDP/MSS) = 1.054 * (RTT / p)^0.75
     (BDP/MSS) = 1.054 * (RTT*MSS / p)^0.75
     p is dimensionless (a fraction) so you cannot just divide it by MSS 
to "get a byte quantity", while not changing the other side of the equation.

==Formula Transformations==
Here, we transform rate formulae in terms of (time between marks) into 
formulae in terms of (marking probability).

For brevity I'll use the following notation, all at steady state, and 
with units shown in [square brackets]:
     R : RTT [s/round]
     S : MSS [B]
     x : flow rate [B/s] = S/R * cwnd
     t_m : time between marked packets [s/marked_pkt]

     r : packet rate [pkt/s] = x/S by definition
     BDP : bandwidth-delay product [B/round]
     p : marking probability [], meaning a dimensionless fraction

The time between marks is the reciprocal of the marks per time:
     t_m = 1/(p*r)
where p*r is the marks per time, because r is packets per time, and each 
packet is marked with probability p.
Expressing this in terms of flow rate rather than packet rate:
     t_m = S / (p*x) (1)

We can then substitute the above in your formulae for rounds per mark:

==DCTCP==
     mark per round = 2
     mark per time = 2 / R
Time per mark is the reciprocal,
     t_m = R/2
Substituting from eqn (1)
     S / (p*x) = R/2
     x = S/R * 2/p
which is the well-known DCTCP formula for flow rate, better known in 
terms of cwnd in units of [packet]:
     cwnd = 2/p.

==Reno ==
    rounds/mark = 2/3 * BDP/S            (corrected from your formula)
     t_m = R* (rounds/mark)
            = 2/3 R*BDP/S
            = 2/3 R*R*x/S
Substituting from eqn (1)
     S / (p*x) = 2/3 R*R*x/S
Rearranging and collecting terms:
     x^2 = 3/2 S^2/(R^2 * p)
     x = S/R sqrt(3/2p)
which is the well-known Reno formula for flow rate, better known in 
terms of cwnd in units of [packet]:
     cwnd = sqrt(3/2p).

==Cubic==
We can transform in the other direction from the Cubic formula that RFC 
8312 expressed in p, to a formula expressed in t_m:
     (BDP/S) = 1.054 * (R / p)^0.75
     x*R/S = 1.054 * (R * t_m * x / S)^0.75
Rearranging and collecting terms:
     x^0.25 = 1.054 * S^0.25 * t_m^0.75 / R^0.25
     x = 1.054^4 * S * t_m^3 / R

In summary, the following generalized formulae can be used to express 
flow rate either in terms of p (marking probability) or in terms of t_m 
(time between marks):
     x = k * S / (p^b * R^c)
     x = k * S * t_m^d / R^e
where:
             b    c    d      e
     ____________________________
     DCTCP   1    1  infty  infty
     Reno   1/2   1    1      2
     Cubic  3/4  1/4   3      1

(the infty values are because DCTCP flow rate, x, does not depend on t_m 
or R, given t_m = R/2, with no dependence on x)

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 16 Jun 2021 12:54:42 +0000

>> So you have not in fact read the post with the full explanation in it, in which the relevant formulae are in fact discussed. 

I saw some equations, but missing the relevant factor here: mss being the packet size. If a flow's rate depends on that packet size, then typically the byte or bit rate r is proportional to mss. Then the other aspect is the probability p measurement. Is that expressed in a per byte probability or per packet probability (which is the same if the sender always send equal packet sizes, but not if there is a certain distribution of packet sizes). So this depends heavily on the exact specific implementation and traffic payload.

>> The steady state of DCTCP is a constant number of marks per RTT.  That's what you get when you cancel the measurement of probability against the number of segments per RTT that the measured probability is applied to.  

DCTCP collects the total ACKed number of "bytes" received in an RTT and the CE-ACKed number of "bytes" received in an RTT. Their ratio is used to update alpha, the reduction factor. So if that ratio changes after decapsulation, you change the rate of DCTCP.
Depending on how this window is used at the sender (if it depends on the detected path MSS or to a reference 1500 bytes MSS), the sender is self-sensitive to path MSS or not. It boils down on if the window controls the MSS packet rate or byte rate. In Linux the window is kept in packets, so it is sensitive to path detected MSS.

>> You really could simplify DCTCP's response to "each CE mark removes half a segment from the cwnd", thereby bypassing a lot of unnecessarily complicated fixed-point maths.  I can walk you through *exactly* how the DCTCP response equations simplify if you need to hear it.

I've tried this before (5 years ago by now?). If you reduce the window by half a packet every mark, it isn't DCTCP anymore. It becomes highly unstable. You need the smoothing. You should try to implement it and run it on a step. Run alone, it becomes Reno, and in a mix of flows, it creates nice patterns 😉.

>> I'm going to stop you there, and remind you that the predominant deployment of ECN AQMs today is a time-domain algorithm, namely Codel.

Then let me remind you too that like we discussed before, it is not the AQM that determines how the CC responds. Any AQM is potentially time-domain, if it applies a deterministic probability (even PI2 can) as it in that case applies a mark for every 1/p-ed packet. The real question is if your codel implementation marks differently based on the packet size of the packet itself, and the intermediate non-marked packets?
So Codel is counting in time between packets marks (but adapts this to the level of congestion independent of what mix of packet sizes there are in between), and then marks a single packet independent of its size. If the time to hit the packet at a specific (even random) time would depend on it's size, then CoDel would mark depending on the size. But I understood that Codel just marks the next packet that arrives after the time-out is passed, so again Codel marks size independent. So bottom line is, that any marker (AQM) can be reduced to a random or deterministic process that marks packets independent or dependent on some properties. It seems you assume Codel does mark packets based on its size (since you claim small packets are marked less than big packets). Even if this would be the case, it seems to be the not recommended case (see RFC7141) and not even solve the purpose you intent (equal throughput, as it needs a squared size proportion probability.

Following RFC7141 the packet marking should be agnostic to the packet size, so encapsulation and decapsulation can also. They should keep the packet marking/drop probability equal.

This results in these 2 methods being a very simple (preferred) and a bit more complex (not even needed) method for decapsulation:
- only mark if the first byte of the inner (reassembled) packet is marked (ignore marks on any other bytes/segments)
- apply a random probability on the inner packet based on the ratio of marked bytes in the inner packet and the inner packet size

Both might "throw away marks" or "create marks" as the count of marked packets will change if the inner to outer size is different, and the packet probability needs to be kept the same statistically.

>> I think it'd be helpful for some *other* WG members to weigh in on this topic for a change.  Does anyone else have an opinion they'd like to share?

Seems all of this has been discussed before and the consensus has been written in RFC7141, which I fully agree with and is still very applicable for L4S going forward. Any bias in the network on packet size would need to work differently for Classic and L4S, so this draft made the right recommendations!

Koen.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Monday, June 14, 2021 6:45 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Bob Briscoe <[email-masked]>; [email-masked]; Markku Kojo <[email-masked]>
Subject: Re: [tsvwg] ECN encapsulation draft - proposed resolution

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 22 Jun 2021 11:02:32 +0100

Jonathan,

Just checking that you have seen my response below. I spent a couple of 
days trying to write it in the way that you seem to be thinking about 
this. So we'll need you to respond if we are going to move forward. I 
appreciate there's a lot here, so it needs some time to work through it all.

Bob

On 15/06/2021 23:25, Bob Briscoe wrote:
> Jonathan,
>
> I waited before replying to your email below until I had the time to 
> try to unpick and deeply understand the way you think about all this. 
> I would appreciate it if you would do likewise, and certainly please 
> do not snip anything I say (like you did last time I put in a similar 
>>> On 3 Jun, 2021, at 2:41 pm, Markku Kojo 
>>> <kojo=[email-masked]> wrote:
>>>
>>> not reached concensus.
>> I think I agree with your analysis that preserving the number of 
>> bytes marked is probably the wrong thing to do.  The implementation 
>> complexity argument is fairly compelling in itself, particularly 
>> noting that there isn't any known "running code" which would place 
>> some upper bound on what the complexity is.
>>
>> By contrast the RFC-3168 approach of "if any part of the packet is 
>> marked, all of it is marked" is very simple to implement, and this is 
>> proved by actual implementations in the wild.  This approach is also 
>> consistent with the behaviour when ECN is not in use, since if any 
>> fragment of a packet is lost, the packet can't be reassembled and is 
>> discarded in whole.
>>
>> One thing I noticed about Bob's position is that he's trying to 
>> preserve the number of bytes marked, but usually the transport is 
>> actually sensitive to the number of bytes, or possibly the interval 
>> of time, *between* marks.  These are two (or three) very different 
>> quantities, and it's important not to get them confused in the debate.
>>
>> In engineering, a very useful tool for checking whether an answer is 
>> correct is "dimensional analysis".  Simply put, work out the units 
>> that your inputs and outputs are most naturally expressed as, and 
>> check that the formulae linking them are consistent with respect to 
>> the units.  In this case, we can look at the units of the AQM's 
>> marking function, the units of the transport's congestion control 
>> steady state criterion, and whether each approach to resolving ECN 
>> marks at reassembly preserves the quality of the marking with respect 
>> to those units.
>>
>> For the purpose of this analysis, we are looking for the quantity 
>> that is most nearly preserved if we hold the AQM state constant or 
>> the transport in steady state, but vary packet size and/or link 
>> bandwidth arbitrarily.
>>
>> Starting with the AQMs:
>>
>> 1: Codel is a time-domain AQM; its marking function is explicitly a 
>> frequency, expressed for performance reasons as a time interval 
>> between marks.  For analysis, we can equivalently express this 
>> quantity in units of marks/sec or its reciprocal, seconds per mark.  
>> If we change the bandwidth or the packet size while holding Codel's 
>> control law constant, the number of bytes in marked packets and/or 
>> the number of packets between marks will change, but the time between 
>> marks will stay constant.
>>
>> 2: A packet-domain marking function (as used in BLUE and many 
>> RED-family AQMs) maintains a probability that any given packet will 
>> be marked.  Amortised over a large interval, we can express this as a 
>> (fractional) number of marks per packet, or equivalently as packets 
>> between marks. If the packet rate increases, so will the number of 
>> marks per time interval; if the packet size decreases, so will the 
>> number of bytes between marks.
>>
>> 3: A byte-domain marking function, as Markku described for early RED 
>> work, is obtained by taking a nominally packet-mode probability 
>> function, but multiplying the probability by bytes/packet (and a 
>> constant scale factor).  The resulting unit is (fractional) marks per 
>> byte, or bytes between marks.
>
> [BB] One doesn't see flow rate formulae for Reno (or DCTCP, Cubic, 
> etc) that say, "the rate is x if over PIE; but the rate is y if over 
> CoDel". The outcome solely depends on the transports involved, which 
> converge on a set of rates that fill the capacity irrespective of 
> which AQM is at the bottleneck. Therefore the unit that a particular 
> AQM uses to mark packets does not alter the steady-state outcome. This 
> is a critical point, so if you disagree, there's no need to move on to 
> the next part about transports - please stop here and solely explain 
> why you think this point is wrong.
>
>
> distraction. CoDel's control law decreases the time between marks 
> until it balances with the transports using it. The fact that RED, 
> BLUE, PIE etc use a packet marking probability is also a distraction. 
> They also adapt, increasing the marking probability to converge on 
> 'the same outcome' as CoDel would, if placed in the same location with 
> the same traffic sources.
>
> * the same number of marked packets per second (or equivalently the 
> same time between marks)
> same property, but let's only talk about that if we need to, when we 
> get to re-framing, where it does make a difference.
>
> BTW, the time taken to reach the outcome (the dynamic performance) 
> will be different for different AQMs, but you're talking here about 
> stable outcomes, which is indeed one of the two important 
> considerations (the other being timeliness).
>
>
>> Now the transports:
>>
>> 1: DCTCP's steady state is two marks per RTT.  Since RTT has units of 
>> time, the steady-state criterion has units of marks/sec.  This is 
>> precisely the same as Codel's marking function.
>>
>> 2: Reno's steady state is BDP/2 RTTs per mark, where BDP has units of 
>> bytes (the two time terms in the formula cancel).  The steady-state 
>> criterion thus has units of byte-seconds per mark. This does not 
>> exactly match any of the AQMs' control laws, but it seems likely that 
>> "seconds per mark" or "bytes between marks" would be a better match 
>> than "packets between marks". Regardless, for constant performance 
>> with Reno, we would look for methods that preserve the byte-seconds 
>> per mark value.
>>
>> 3: CUBIC's steady state, when not in the Reno compatibility regime, 
>> is quoted in RFC-8312 as (BDP/MSS) = 1.054 * RTT^0.75 / p^0.75, where 
>> p is the (fractional) number of marks per segment, and the quantity 
>> BDP/MSS has units of segments.  But each segment, as seen by the 
>> transport, is MSS bytes long, and it is ignorant of any fragmentation 
>> that might happen at Layer 3 or 2 along the way - so if that occurs, 
>> "segments" is not the same as "packets" as seem by the AQM.  So we 
>> can convert p into a byte quantity by dividing it by MSS, yielding 
>> (BDP/MSS) = 1.054 * (RTT*MSS / p)^0.75. This reveals that for a 
>> constant cwnd, a constant byte-seconds per mark value is required, 
>> similar to Reno, and we can probably expect this to be true for other 
>> "TCP friendly" congestion controls as well.
>
> terms of a parameter with certain units, it doesn't prove anything 
> about reassembling markings. It just says you have chosen to express 
> the formulae in terms of a parameter with certain units.
>
> Dimensional analysis is very useful to check one's work. But you're 
> not doing dimensional analysis above (otherwise you'd be checking that 
> the two sides of an equation have the same units). You're just 
> expressing formulae in terms of a parameter with units that you are 
> comfortable with, then convincing yourself that this proves that 
> reassembly should use the units that you are comfortable with.
>
> AQM', you are following the logic that the AQM examines the packets 
> and knows what time to insert between marks. It doesn't. The AQM 
> inserts a first guess at a time between marks, the transport responds, 
> and if it doesn't respond enough, the queue doesn't shrink enough, so 
> the AQM keeps on inserting less time before the next mark until the 
> time between marks from the AQM causes the flow rate to balance at a 
> steady queue around the target.
>
> As explained earlier, if you observe a PIE or BLUE AQM from the 
> outside, you might believe it is inserting a reducing time between 
> marks. But internally, it is reducing a marking probability. Looking 
> from the outside, the final probability it reaches can also be 
> considered as the final time between marks that it reaches. So 
> (again), the units the AQM adjusts internally are irrelevant to the 
> steady state it reaches.
>
>
> When I said that all these formulae can be expressed in terms of 
> marking probability, p, rather than time between marks, t_m, that was 
> not to prove that p is the correct unit to preserve on reassembly. 
> It's just to prove that each transport's formula can be expressed in 
> either unit, so the units chosen for a formula do not imply anything 
> about which units should be preserved on reassembly. Your formulae are 
> just convincing yourself that you can express formulae in terms of a 
> parameter in units that you are comfortable with.
>
> between marks leads to roughly x2 unfairness between flows that are 
> just below and just above the fragmentation limit. Whereas preserving 
> marking probability (or equivalently bytes marked), leads to no 
> unfairness. That's a real-life phenomenon that doesn't depend on which 
> units anyone chooses to reason about the world.
>
> You can stop here, if you are convinced. If not, go to the end, to 
> work through the transformations from formulae expressed in 'time 
> between marks' into formulae expressed in 'marking probability'.
>
>> So now we can turn to the methods of resolving congestion marks at 
>> reassembly.  I'll examine three candidate rules that seem relevant to 
>> the debate, though others probably exist:
>>
>> 1: "If any fragment of a packet is marked, the whole packet is 
>> marked."  This is the RFC-3168 rule, generalised to also cover the 
>> weird Layer 2 fragmentation scheme that sparked this debate.  It is 
>> clear that this rule preserves the number of bytes between marks and 
>> the time between marks (and therefore also the byte-seconds between 
>> marks), but not the number of packets between marks - except for the 
>> specific case where a fragment contains portions of more than one 
>> packet.  If these packets all belong to the same flow, then this is 
>> harmless to Reno and CUBIC since consecutive ECN marks are reported 
>> to the sender as a single event.  If multiple flows are involved, 
>> then this will stochastically amplify the congestion signal by having 
>> more than one flow react to a single mark - which seems tolerable in 
>> practice.
>>
>> 2: "Every mark is sacred, every mark is great."  The AQM produces one 
>> mark, and exactly one reassembled packet receives that mark; the 
>> reassembly process can make a reasonable choice as to which one.  
>> This preserves bytes between marks, time between marks, and 
>> byte-seconds between marks *exactly*.
>>
>> 3: "The number of bytes marked should be preserved by reassembly."  
>> This would tend to preserve the number of packets between marks 
>> better than the other two options, but that is not useful to any of 
>> the transports considered.  The distortion to the bytes between marks 
>> and time between marks metrics would be distorted proportionally to 
>> the difference between average fragmented and reassembled packet 
>> sizes, and the byte-seconds between marks measure would be distorted 
>> *quadratically* by that difference.
>>
>> In summary, the "preserve the number of marked bytes" rule is 
>> mathematically unreasonable, as well as being much more complicated 
>> to implement than either the existing RFC-3168 rule, or the "sacred 
>> mark" rule, which actually preserve the quantities that need to be 
>> preserved.
>
> flows of reassembled packets and packets that have not been 
> fragmented. And I showed that preserving marking probability (or 
> equivalently bytes marked) solves the problem:
> email, but it doesn't even mention the problem I've levelled against 
> the scheme you prefer. That is the whole nub of this argument. Please, 
> you cannot continue to keep asserting that you are right, without 
> engaging with that problem. And you cannot keep just asserting that I 
> don't know what I'm talking about and you do, unless you refute my 
> arguments.
>
>
> Part #3 consists of assertions that the reassembly scheme you assert 
> is right is the one that preserve the units that you assert are right. 
> But you haven't given any arguments for why, and you haven't even 
> tried to support your arguments using the first two parts.
>
> choosing to express a flow formula in terms of a parameter with one 
> unit or the other is just your choice. It proves nothing about which 
> is the correct unit to preserve during reassembly. (Supporting maths 
> follows at the end)
>
>>
>>   - Jonathan Morton
>
> [BB] Regards
>
>
>
> Bob
>
> * Reno: Given you define BDP in bytes, your criterion for RTTs per 
> mark should be (BDP/MSS)/2 , not BDP/2.
> * Cubic: Obviously, your two formulae compared below cannot both be 
> true (by dimensional analysis, ironically):
>     p is dimensionless (a fraction) so you cannot just divide it by 
> MSS to "get a byte quantity", while not changing the other side of the 
> equation.
>
>
> where p*r is the marks per time, because r is packets per time, and 
> each packet is marked with probability p.
> We can transform in the other direction from the Cubic formula that 
> RFC 8312 expressed in p, to a formula expressed in t_m:
> flow rate either in terms of p (marking probability) or in terms of 
> t_m (time between marks):
>     x = k * S / (p^b * R^c)
>     x = k * S * t_m^d / R^e
> where:
>             b    c    d      e
> (the infty values are because DCTCP flow rate, x, does not depend on 
> t_m or R, given t_m = R/2, with no dependence on x)
>

---

From: Markku Kojo <[email-masked]>
Date: Tue, 22 Jun 2021 17:02:09 +0300 (EEST)

Hu Bob,

I am replying separately to the messges you addressed to me. I am not 
going analyse the maths by you or Jonathen more deeply as they 
both contain errors. Just pointing out a few things below which might 
help understanding my points in my other replies which do not contain 
much math.

Please see inline.

On Tue, 22 Jun 2021, Bob Briscoe wrote:

> Jonathan,
>
> Just checking that you have seen my response below. I spent a couple of days 
> trying to write it in the way that you seem to be thinking about this. So 
> we'll need you to respond if we are going to move forward. I appreciate 
> there's a lot here, so it needs some time to work through it all.
>
>
> Bob
>
>
>> I waited before replying to your email below until I had the time to try to 
>> unpick and deeply understand the way you think about all this. I would 
>> appreciate it if you would do likewise, and certainly please do not snip 
>> anything I say (like you did last time I put in a similar amount of effort 
>> into explaining all this on your terms). Instead, please try to understand 
>> and address each point.
>> See [BB] inline.
>> 
>>>> reached concensus.
>>> I think I agree with your analysis that preserving the number of bytes 
>>> marked is probably the wrong thing to do.  The implementation complexity 
>>> argument is fairly compelling in itself, particularly noting that there 
>>> isn't any known "running code" which would place some upper bound on what 
>>> the complexity is.
>>> 
>>> By contrast the RFC-3168 approach of "if any part of the packet is marked, 
>>> all of it is marked" is very simple to implement, and this is proved by 
>>> actual implementations in the wild.  This approach is also consistent with 
>>> the behaviour when ECN is not in use, since if any fragment of a packet is 
>>> lost, the packet can't be reassembled and is discarded in whole.
>>> 
>>> One thing I noticed about Bob's position is that he's trying to preserve 
>>> the number of bytes marked, but usually the transport is actually 
>>> sensitive to the number of bytes, or possibly the interval of time, 
>>> *between* marks.  These are two (or three) very different quantities, and 
>>> it's important not to get them confused in the debate.
>>> 
>>> correct is "dimensional analysis".  Simply put, work out the units that 
>>> your inputs and outputs are most naturally expressed as, and check that 
>>> the formulae linking them are consistent with respect to the units.  In 
>>> this case, we can look at the units of the AQM's marking function, the 
>>> units of the transport's congestion control steady state criterion, and 
>>> whether each approach to resolving ECN marks at reassembly preserves the 
>>> quality of the marking with respect to those units.
>>> 
>>> For the purpose of this analysis, we are looking for the quantity that is 
>>> most nearly preserved if we hold the AQM state constant or the transport 
>>> in steady state, but vary packet size and/or link bandwidth arbitrarily.
>>> 
>>> Starting with the AQMs:
>>> 
>>> frequency, expressed for performance reasons as a time interval between 
>>> marks.  For analysis, we can equivalently express this quantity in units 
>>> of marks/sec or its reciprocal, seconds per mark.  If we change the 
>>> bandwidth or the packet size while holding Codel's control law constant, 
>>> the number of bytes in marked packets and/or the number of packets between 
>>> marks will change, but the time between marks will stay constant.
>>> 
>>> 2: A packet-domain marking function (as used in BLUE and many RED-family 
>>> AQMs) maintains a probability that any given packet will be marked.  
>>> Amortised over a large interval, we can express this as a (fractional) 
>>> number of marks per packet, or equivalently as packets between marks. If 
>>> the packet rate increases, so will the number of marks per time interval; 
>>> if the packet size decreases, so will the number of bytes between marks.
>>> 
>>> 3: A byte-domain marking function, as Markku described for early RED work, 
>>> is obtained by taking a nominally packet-mode probability function, but 
>>> multiplying the probability by bytes/packet (and a constant scale 
>>> factor).  The resulting unit is (fractional) marks per byte, or bytes 
>>> between marks.
>> 
>> that say, "the rate is x if over PIE; but the rate is y if over CoDel". The 
>> outcome solely depends on the transports involved, which converge on a set 
>> of rates that fill the capacity irrespective of which AQM is at the 
>> bottleneck. Therefore the unit that a particular AQM uses to mark packets 
>> does not alter the steady-state outcome. This is a critical point, so if 
>> you disagree, there's no need to move on to the next part about transports 
>> - please stop here and solely explain why you think this point is wrong.

[MK] Actually the flow rate should not depend on the AQM, but in practise 
it does. This is because an AQM is used to control the (virtual) queue 
size on the bottleneck AQM node. It depends on how the AQM is configured, 
that is, at which queue level (queue size) it drops/marks the first packet 
after entering its main operating range as well as how it drops/marks 
after that.

If an AQM drops/marks a packet (on avarage) at a different the queue level 
(queue size) depending on the size of the packets, then the outcome 
depends very heavily on the AQM. Just think a single flow. The queue size 
at the time when the AQM marks/drops a packet controls directly what is 
the cwnd size when the congestion signal arrives. Thereby, a lower queue 
size means lower cwnd size and also lower cwnd size after halving 
the cwnd and thereby lower avg cwnd that results in lower flow rate (if 
cwnd has lower value than BDP after it was halved; if it does not, then 
the AQM has been configured more towards buffer bloat than a reasonable 
virtual queue size). In Reno formula this means that the queue size 
affects the avarage RTT in the Reno formula as well as 'p'.

>> distraction. CoDel's control law decreases the time between marks until it 
>> balances with the transports using it.

[MK] Please stop here thinking a single flow case. CoDel's control law is 
never applied with a single flow in congestion avoidance beçause after the 
first mark/drop cwnd is halved and the sojourn time falls below Target. 
Next time a packet is marked/dropped only after several RTTs once the 
cwnd has been increased enough and the queue level (size) has reached the 
point where CoDel marks/drops a packet (i.e., once sojourn time has been 
above Target for a period of Interval). The decreasing marking/dropping 
interval in CoDels control low allows it to react faster when there are 
enough flows sharing the bottleneck such that a single drop/mark does not 
take sojourn time below Target.

>> The fact that RED, BLUE, PIE etc use 
>> a packet marking probability is also a distraction. They also adapt, 
>> increasing the marking probability to converge on 'the same outcome' as 
>> CoDel would, if placed in the same location with the same traffic sources.

[MK] RED, BLUE, PIE etc. with same configuration should also mark/drop a 
single packat per Std Track congestion control sawtooth cycle. So, the 
timing (number of bytes, not packets) between marks/drops matters.
If you decrease the timing, it results in decreased flow rate. And if 
small packets have the same probability to get marked/dropped as large 
packets, the mark/drop occurs earlier, that is, in shorter time, for 
small packets than for large packets.

>> If it existed, byte-mode marking (the early RED option) would have the same 
>> property, but let's only talk about that if we need to, when we get to 
>> re-framing, where it does make a difference.

[MK] Byte-mode marking gives lower probability for small packets, so it 
postpones marking a small packet such that (on the average) the mark 
occurs approximately after same period of time (after same number of 
bytes) regardless of packet size.

[SNIP]

>> Your analysis of DCTCP is OK, but your analyses of Reno and Cubic contain 
>> errors:
>> 

[MK] Incorrect. The const depends on the amount of (virtual) queueing 
configured and number of flows. If the bottleneck (with or without AQM) 
is configured to allow queue that equals to BDP (i.e., mark occurs when 
BDP amount of bytes have been queued), then a single flow gets full 
utilization of the bottleneck and the max cwnd = W = 2*BDP and cwnd after 
cwnd reduction is W/2. So, the time needed to increase cwnd from W/2 
(=BDP) to W is W/2 = (2*BDP/MSS)/2 = BDP/MSS RTTs. If the bottleneck is 
configures to have no queue, the time needed to increase cwnd from BDP/s 
to BDP is (BDP/MSS)/2 RTTs.

And, if the number of flows is increased, then the time between the marks 
decreases. So, we need to be careful in understanding what we are 
modelling. Otherwise, if we just play with the formulae, the outcome is 
just crap.

Best regards,

/Markku

>>     p is dimensionless (a fraction) so you cannot just divide it by MSS to 
>> "get a byte quantity", while not changing the other side of the equation.
>> 
>> 
>> For brevity I'll use the following notation, all at steady state, and with 
>> units shown in [square brackets]:
>>     R : RTT [s/round]
>>     S : MSS [B]
>> which is the well-known DCTCP formula for flow rate, better known in terms 
>> of cwnd in units of [packet]:
>>     cwnd = 2/p.
>> 
>> ==Reno ==
>> which is the well-known Reno formula for flow rate, better known in terms 
>> of cwnd in units of [packet]:
>>     cwnd = sqrt(3/2p).
>> 
>> ==Cubic==
>> In summary, the following generalized formulae can be used to express flow 
>> rate either in terms of p (marking probability) or in terms of t_m (time 
>> between marks):
>>     x = k * S / (p^b * R^c)
>>     x = k * S * t_m^d / R^e
>> where:
>>             b    c    d      e
>> (the infty values are because DCTCP flow rate, x, does not depend on t_m or 
>> R, given t_m = R/2, with no dependence on x)
>> 
>
> --

---

From: Markku Kojo <[email-masked]>
Date: Tue, 22 Jun 2021 17:13:28 +0300 (EEST)

Bob,

In order to make some progress with this discussion I'll try to take us on 
the same page to begin with as it seems to me I was (again) not able to 
make the point clear on what is important to Standards Track congestion 
control. Please focus on the simplest case, that is, the case of a single 
flow using Standards Track congestion control. That hopefully helps in 
understanding the importance of timing when marking/droping packets.

First, I believe we agree the following:

1.
When the sending bit-rate for a small packet sender is the same as 
for a large packet sender, an AQM using probabilistic packet mode 
dropping/marking will result in dropping/marking more small 
packets(/frames) than large packets(/frames) because random marking 
with the same probability is applied more often to the packet flow of the 
small packet sender than that of the large packet sender.

Consequently, if two senders using Standards Track congestion control are 
sending large packets of the same size and the packets for the sender A 
get splitted in smaller PDUs (e.g., in L2 encapsulator, in a tunnel 
ingress, or fragmented on a router in front of a link with an MTU smaller 
than the packet size) before they arrive at a bottleneck AQM but packets 
for the sender B do not get splitted on the path, then the sender A also 
gets more CE marks (I'll call these excess marks as "extra" marks from 
now on) than sender B. This would be unfair for sender A (bias against 
small PDUs at an AQM) as it systematically results in lower throughput 
for it even though A and B are originally sending packets of the same 
size (e.g., MSS-sized packets for TCP) and A and B are increasing and 
decreasing cwnd exactly with the same AIMD algorithms.

2.
In order to enable fair treatment of A and B above (item 1), the "extra" 
marks for A should be eliminated.

I believe we agree on both items (1. and 2.) above, right?
But at the same time I believe we disagree on *how* to eliminate the 
"extra" marks, or that it matters which of these marks are "extra" the 
marks that should be eliminated.

In addition, I believe we agree that

3.
if the bottleneck AQM in the above example happens to use byte-mode drop, 
then there is no (significant) bias against sender A that gets its 
packets splitted before the AQM, i.e., there is no need for any 
additional measures to eliminate the "extra" marks because byte-mo drop 
already does this (please let's use byte-mode drop/marking here only as a 
reference and refrain frrom arguing whether byte-mode is deployed or not 
{Note 1}).

If someone disagrees with this, please see first my comment inline below 
marked {***} and read carefully the cited note by Sally Floyd on whether 
byte-mode drop gives any advantage for small packets (fragments, or 
similarly to flows splitted to small L2 frames before arriving at a 
bottleneck AQM).

{Note 1} How byte-mode drop works is important to understand when 
discussing how to achieve correct operation when we want to eliminate the 
"extra" marks.

What we don't agree, is

4.
How and where in the network to eliminate these "extra" marks, and

5.
Whether CoDel works effectively pretty much like byte-mode drop(/marking) 
even though it does not use probabilistic dropping/marking but 
deterministic drop/marking (timer-based drop/marking). And here the 
important issues is whether CoDel has bias against small packets like 
probabilistic packet-mode drop mentioned in item 1 above.

My claim is that CoDel does not cause any bias against small packets but 
drops/marks a packet at the same level of congestion (queuing) regardless 
of the packet size. Please see e.g., my example on CoDel behaviour in the 
unanswered email I pointed to in my previous message:

Please particularly consider the importance of the timing with a single 
flow (or with a small number of competing flows) what I try to clarify 
below.

With regard to the item 4 and question "where", let's put it aside for a 
while because we first need to understand what is the correct way of 
eliminating the "extra" marks, i.e., how to do it. Where to do it is 
somewhat related to an architectural debate although there are aspects 
that we need to understand concerning where some functions can 
be implemented correctly.

With regard to the question "how" in the item 4, I now notice that we 
seem to approach the problem from a different perspective. Bob seems to 
think of relatively heavy congestion cases in the first place, i.e., when 
there is likely to be a need for an AQM at the bottleneck to mark packets 
relatively often resulting in marking packets (nearly) every RTT or even 
marking several packets per RTT. If this is the case, the approach Bob has 
been proposing (as I understand it) seems to be pretty much on the correct 
track.

Hovever, the way in which we should eliminate the "extra" marks must work 
also for the case when there is only a limited number of competing flows, 
including the case of a single flow. In such a case, the timing of the 
marks is crucial and it is important to understand why.

To keep it simple as possible, I use only the case of a single flow.

I start with very basics. It is very well understood that a single flow 
operating over a tail drop bottleneck requires BDP worth of queue in the 
bottleneck in order to be able to fully utilize the bottleneck link 
capacity. That is, the sender has reached the maximum cwnd (W=2*BDP) when 
a packet is dropped from a full queue and cwnd is halved. The resulting 
cwnd is W/2=BDP, meaning that after halving the window and starting 
the next congestion control cycle (the famous TCP sawtooth) the flow is 
still able to fully utilize the available link capacity. The cwnd of the 
sender oscillates between W/2 and W, giving the average cwnd of 3/4 W = 
3/2 BDP (note that the resulting flow rate is the bottleneck link rate, 
not 3/2 BDP / nominal RTT, but 3/2 BDP / AWG RTT). If we configure a 
shorter queue at the bottleneck or otherwise cause the queue to drop/mark 
at a lower queue level, the maximum cwnd becomes smaller and the 
packet drop occurs "earlier" in the concestion control cycle, i.e., the 
cycle becomes shorter and the saw tooth "amplitude" is lower. That means 
that when a packet is dropped earlier and cwnd is halved, the resulting 
cwnd < BDP and the flow is not able to fully utilize the available link 
capacity. In other words, the interval between two drops (marks/congestion 
signals) becomes shorter.

Now, if we apply an AQM at the bottleneck, the only difference is that 
that we use AQM to control the (virtual) queue size. Again, in order to 
fully utilize the bottleneck capacity, we should configure the AQM such 
that it (on average) drops/marks the first packet after the previous 
drop/mark (and cwnd reduction) only after cwnd has reached W = 2*BDP. If 
we configure or otherwise make it to drop/mark "earlier", i.e., we reduce 
the (virtual) queue size, the flow is not able to fully utilize the link 
capacity but gets a lower delay = average RTT.

So, if we have only a single flow at the bottleneck, the AQM marks 
only a single packet per the famous congestion control saw tooth cycle. 
Once a TCP sender (and many other senders following Stds Track congestion 
control principles) has reduced cwnd to a half of its previous size, it 
starts increasing cwnd by one MSS per RTT. The AQM at the bottleneck 
typically does not mark any packets during the early rounds of this cycle 
because the sender has just allowed the bottleneck queue to drain by 
halving cwnd. After some RTTs the queue has build enough such that the 
AQM moves to its main operating range (e.g., the sojourn time with CoDel 
exceeds Target, or AWG queue length in RED exceeds min threshold). Any 
AQM that has been configured to follow probabilistic packet-mode drop 
(e.g., RED or PIE), starts applying random drop/mark on each arriving 
packet using the same probability regardless of the packet size. This 
means that if the original packets that the sender is sending are split 
into smaller fragments (or frames) before they arrive at the bottleneck 
AQM, the AQM marks the first small packet (PDU) "earlier" (= too early) 
than a packet for a flow with non-splitted packets.

"Earlier" means after a smaller number of RTTs i.e., a sender that gets 
its packets splitted (sender A) has not had a chance to increase its 
cwnd to as high value as a sender (sender B) that does not get its 
packets splitted. Note the both senders increase their cwnd with an equal 
amount of bytes (one MSS) per RTT. So, this FIRST mark per congestion 
control cycle is IMPORTANT, it must not occur too *early*. It means that 
in order to get fair treatment with small PDUs (fragments) we need to 
*postpone* marking the first small PDU (fragment), instead of marking 
immediately. This also means that the timer that Bob has suggested 
will mark a PDU immediately if there were no marks within the timer 
period and therefore it would not work correctly. Having no timer as in 
the "skeleton algorithm idea" in RFC 7141 and in draft-13 would not work 
either because the counter would have an infinite memory.

Hopefully this clarifies the importance of timing. CoDel would work 
correcly as it would set the first mark at the same level of congestion, 
i.e., the sender A with splitted packets and the sender B with non-splitted
packets would experience the first mark during the same round trip after 
the previous cwnd reduction, so cwnd that they both halve is of equal 
size, resulting in the same flow rate.

See below inline some additonal answer/comments inline.

On Sun, 6 Jun 2021, Bob Briscoe wrote:

> Markku, see inline [BB] (I'll reply to Jonathan's email yesterday separately)
>
>> smaller fragments/frames/PDUs and reassembling/decoding these smaller PDUs 
>> back to larger IP packets, it does, however, relate to the problem in ECN 
>> encapsulation draft on what to do when framing boundaries do not 
>> packets or encoding/decoding IP packets to/from L2 PDUs. 
>
> [BB] RFC3168 reassembly is not 'known to work', even for fragment reassembly 
> that it is intended for. You acknowledge that it suffers from the fairness 
> problem unless AQMs all use byte-mode marking (and none do - see later). 
> RFC3168 reassembly does not even have any suggestion on what to do if framing 
> boundaries overlap packet boundaries or where packets are smaller than frames 
> - so it is not 'known-to-work' in any case, let alone in every case.

[MK] I have not suggested anywhere that RFC3168 reassembly would work 
correctly with probabilistic packet-mode marking/dropping. With 
'known-to-work' method I mean something that have been shown to work 
correctly, i.e., there is an algorithm (or a set of algorithms) that has 
been published and evaluated and there is rough consensus it works 
correctly.

And, yes, I think I made it perfectly clear in the message referred above 
that we need to solve all cases, including the cases where pkt/frame 
boundaries ovelap or where pkt/frame sizes do not match, as well as 
a mix of these cases (and I never suggested RFC 3168 reassemby for 
this, I only stated that RFC 3168 reassemby would work correctly for 
Stds Track congestion control when IP fragments hit an AQM bottleneck that 
implements byte-mode drop).

>> have not replied although promised (I was away from any IETF work in April 
>> due to family emergency and only resuming now, my apologies). That thread 
>> is about the (additonal but not independent) problem where L3 packet and L2 
>> PDU boundaries do not align. The last message on the thread is here:
>> 
>> is that the draft does not say anything on the topic except points to a new 
>> draft (the one that has been envisioned to handle the IP fragmentation 
>> problem and would include also the handling of not aligned packet/frame 
>> boundaries) and we initiate such new draft before ECN encap draft gets 
>> published.
>
> boundaries change. Do you agree with the following table that I asked about 
> earlier:
>
>                   | marked    marked
>                   | PDUs      bytes
> IOW, do you agree that the three that are tagged as '==' are equivalent ways 
> of expressing the same thing, but different from the one tagged '!=' ?

[MK] For standards track congestion control it is important that timing 
of the congestion signal is correct: not postponed inappropriately, nor 
injected too early (see above).

I am unclear here if you are looking only the case where packet size = PDU 
payload, i.e., when the boundaries do not align?

>> 
>> Below I try to summarize the problem with the two suggested paragraphs with 
>> two SHOULDs.
>> 

[MK] There have been two earlier suggestions with brief text describing 
the "skeleton/idea of an algorithm".

The first "algorithm" was the one in draft-13 which is essentially the 
same as what RFC 7141 gives in Sec 2.4. It proposed a counter to 
calculate the balance of incoming and outgoing marked octets and to mark 
the outgoing packets such that the number of octets  with congestion 
indications in incoming and outgoing PDUs are equivalent. Such a counter 
has infinite memory and thereby it would remove tens of CE marks present 
in inner fragments (splitted PDUs) in the worst case, postponing the 
congestion indication unacceptably long. I believe we agreed this was 
wrong. And, In particular, it proposed marking an outgoing packet 
immediately with the smallest positive remainder in the counter. This is 
also wrong, because it results in marking an outgoing packet too early 
(see above).

The second "algorithm idea" was presented here:

  https://mailarchive.ietf.org/arch/msg/tsvwg/vmIXNtXhGplqTEK1H-NIsZ1kTEs/

It modified the first one by suggesting to approximately preserve 
the proportion of PDUs with congestion indications arriving and leaving.
In addition, it added a timer with "typical RTT" to ensure immediate
propagation of congestion indication after a timeout period without any
marking. This algorithm has the problem that it also results in marking an 
outgoing packet too early. With lightly congested conditions Standards 
Track congestion control expects receiving congestion signals every Nth 
RTT where N is relative to the maximum congestion window size (W) in the 
prevailing network conditions (N = W/(2*MSS)). A correctly working 
algorithm would need to be able to remove "extra" CE marks during this 
time period. And a decapsulator does not know what the RTT is, and the 
timer period must be different for the case when there is only one flow 
at the bottleneck, compared to the case when there are several flows 
sharing the bottleneck AQM queue.

The third algorithm is presented below.

> Nonetheless, now that the drafts are progressing, I think we're allowed to 
> discuss specifics. Although, let's not discuss implementation efficiency 
> quite yet - let's keep to intent for now.
>
> So here's one example of pseudocode for propagating ECN marking when frames 
> are derived from an ordered byte stream of larger packets, but the frame and 
> packet boundaries do not align:
>
> 1) First consider the case where marking is frequent enough that outgoing 
> packets are always marked more often than every 1ms. Then as each outgoing 
> packet is made ready to depart, it will be marked if the last frame to arrive 
> was marked. This is based on the idea Jonathan suggested for preserving the 
> proportion of marked PDUs.
> 2) Now, what if marks arrive infrequently, so that the time since the last 
> outgoing mark exceeds the TIMEOUT? Then the outgoing packet will be marked if 
> /any/ frame that it consists of was marked, because the timer condition and 
> 'pending' will both be true. Because pending toggles on if any frame is 
> marked, and pending is cleared after every packet is sent.

[MK] This marks splitted packets too "early" in most Internet network 
settings. See above.

> ensure it's significantly smaller than most RTTs over the Internet.

[MK] Sure, but if TIMEOUT is smaller than most RTTs over the Internet, it 
results in majority of cases not eliminating the "extra" marks for 
splitted packets, resulting in bias against splitted (small) packets with 
probabilistic packet-mode AQMs that is one of the goals. We also need to 
understand that the pace at which a single connection increases its 
offered load (cwnd) is different from the pace at which two or more 
competing flows increase the offered load (aggregate rate). So, the 
decapsulator does not need to estimate only the RTT but also adjust the 
timer when the number of competing flows changes. Seems quite complex to me.

> While marked packets are arriving often enough, this keeps the proportion of 
> outgoing marked bytes the same as the proportion incoming (it therefore also 
> preserves the proportion of marked PDUs).

[MK] So, it does not mitigate the bias against splitted packets when 
packet-mode marking is used. Am I missing something?

> immediately,

[MK] So, it marks too "early" as I explained earlier.

> even if there weren't enough incoming marked bytes for the 
> outgoing packet size. But the algo holds the deficit in balance, so that if 
> more marked bytes appear before the TIMEOUT, they will have to eat up the 
> deficit before causing another outgoing mark. This is what ensures the 
> proportion outgoing balances the proportion incoming.

[MK] But not the correct timing of marks.

> * However, if the balance has been negative for more than the TIMEOUT, it is 
> halved. And after another timeout the deficit is halved again, and so on, 
> until eventually it will round down to zero. Then, with certainty, the next 
> incoming marked frame, whatever its size, will trigger an outgoing mark 
> immediately. In the intervening period, every time the deficit is halved, it 
> becomes more likely that an incoming marked frame will cause an immediate 
> outgoing mark, by taking the balance over zero.
>
> proportion of marks and preserving the timing of marks, rather than a hard 
> cut-off between the two. But the latter could be implemented simply by 
> changing '/= 2' to '= 0' and using a longer TIMEOUT. Also note that the 
> balance approach works whether some frames are larger than packets or some 
> packets are larger than frames.
>
> fairness and preserving timeliness). So if you're a cup-half-empty sort of 
> person, you will always be able to argue that they're not truly correct. But 
> if you're a cup-half-full-sort of person you will be able to argue that 
> they're never too far off being correct.
>
>
>> 
> draft-14 (Nov 2020)? Because we couldn't agree, the chairs asked me to remove 
> the specific algorithm that was in the draft, and solely state requirements 
> (which eventually resulted in the two SHOULDs). The chairs wanted to defer 
> this debate, and not to constrain Layer-2 solutions to a particular algorithm 
> in the mean time.
>
> intent was that the question of how exactly to mark PDUs while reframing was 
> controversial, so it should be solved in a separate draft, in order that this 
> ecn-encap draft could go through quickly (ironic raised eyebrow).

[MK] I am not arguing with you, but with the proposed text in the draft.
So, I oppose publishing text in a BCP that indicates a behaviour of an 
algorithm for which there is no algorithm that has been shown to work.
I also object publishing vague text that leaves too much to interpret 
without citing at least one way to do it correctly (or, in general, that 
thereis a way of implementing it correctly).

> [BB] RFC7141 is not "the origin of the problem". It described current AQM 
> practice, which was not what researchers thought (and apparently still 
> isn't). It is a BCP and many many people were involved in reaching consensus 
> on it during the 6 years it progressed through the IETF. It explained why AQM 
> practice at the time was not implementing byte-mode drop, and proposed a way 
> to move forward with packet mode drop instead.

[MK] Seems that the AQM practise has evolved since the publication of 
RFC 7141 and the AQM practise (RED) it evaluated is not anymore 
recommended at all.
Since then new AQMs have been introduced (actually were introduced 
already before publishing RFC 7141 but not published in RFC series)

> implemented byte-mode dropping (admittedly only 19% response rate, out of 84 
> vendors contacted. Plus we checked Linux RED, which also did not implement 
> byte-mode drop).

[MK] The text in RFC 7141 / appendix A does not cite the survey so all 
details remain unknown. I wonder how many of the 14 vendors that answered 
that they did not implement byte-mode drop did implement byte-mode queue 
measurement? For those who did not, I believe, the question about 
implementing byte-mode drop is not quite relevant given the advise in the 
original RED paper, Section 4:

  "when this option is used [queue is measured in bytes] the
   algorithm would be modified to ensure that the probability
   that a packet is marked is proportional to the packet size
   in bytes."

That is, the advise at the time was only implement byte-mode drop when 
byte-mode queue measerement was implemented.

> Since RFC7141 and RFC7567 were published, a new round of AQMs appeared (CoDel 
> and PIE). Contrary to what you say in the first linked email above, CoDel 
> does not do byte-mode drop or byte-mode marking{Note 1}.

[MK} Ehh. Where I have said CoDel does byte-mode drop? I believe I was 
quite clear in saying that CoDel employs deterministic drop/marking. My 
apologies if my wording has been unclear.

> And PIE doesn't  either.
>
> DOCSIS PIE does do a limited form of byte-mode drop{Note 2}, but it does not 
> support ECN and it's not under 'IETF-change-control' anyway (the authors 
> offered to document it as an informational RFC). The only other algorithm I 
> know of that has implemented byte-mode drop is the RED used in the ns2 
> simulator.
>
> Later you say byte-mode drop is common industry practice and that it's the 
> way forward. It's not industry practice at all, so it's hardly going to be 
> the way forward.

[MK] Ehh, again, where have I said byte-mode drop is common industry 
practice?? My point with the byte-mode drop in the first place is to refer 
to a correctly working approach that has  been designed to eliminate 
bias against small packets (fragments) and has been evaluated to work 
correctly (with some later corrections).

> specific packet's size, only on the average size of all packets.

[MK] Incorrect. CoDel marking is not probabilistic at all There is no 
likelyhood, but the drop/mark is deterministic if the congestion 
persists, pretty similar to why tail drop is deterministic. Because it is 
deterministic and absolutely independent of packet size. When sojourn 
time exceeds Target, CoDel schedules first mark after an Interval (given 
that sojourn time remains above Target for the time priod of Interval). 
During the Interval period, the bottleneck link deliveres the same amount 
of bytes regardless of packet sizes (though the amount of payload bytes 
is different due to different pkt hdr overhead, of course). That is, if 
two senders are sending packets (TCP segments) of the same size (they use 
same MSS value) over a network path with same characteristics, they send 
the same amount of data during the Interval and increase their cwnd at 
the same pace (and they increase cwnd at the same pace also after the 
previous time they halved cwnd and sojourn time fell below Target but 
before the sojourn time again exceeds Target). 
So, the next time an CoDel AQM at the bottleneck marks a packet 
(fragment), it occurs at the "same point" (at the same time) regardless of 
whether or not a sender got its packets fragmented (splitted) before 
they arrive at the the bottleneck AQM. That is, when a fragment or packet 
is marked, cwnd has the same value in both cases and cwnd is reduced 
(halved) to the same value when the CE mark is signalled to the sender 
(pls ignore the slight difference due to pkt hdr overhead).

> That is 

[MK] RED, from where these terms come from, uses probabilistic 
dropping/marking. Byte-mode and packet-mode drop/marking is meaningful 
only with probabilistic drop/marking.

> * The control law reduces the time between each mark/drop on a preordained 
> schedule, so the likelihood of any one packet being dropped/marked is higher 
> and increases faster if the average packet rate is lower (which can be 
> because packets are larger on average).

[MK] Well correct but per packet likelyhood is irrelevant. Yes, there are 
less larger packets in between the marks than smaller packets that the 
AQM sees, but the same amount of bytes (= amount of time). What 
matters is that there are the same amount of *original* packets in 
between the marks that the sender is sending (and sees), regardless 
whether the original packets get splitted and merged (fragmented and 
reassembled) an route. The end-point operates on non-splitted units 
(= original packets/TCP segments) it sends and it does not even know what 
happened at lower layers!

> * However, just because CoDel happens to mark packets on a preordained time 
> schedule is not why it's important for a decap to propagate a mark in a 
> timely fashion. The motivation is not specific to CoDel - it's to minimize 
> the delay around the control loop whatever the AQM.

[MK] Please read carefully what I wrote about the importance of timing 
towards the beginning of this message and how timing affects the 
behaviour and performance. And please think of the simplest scenario with 
a single flow. It must work first, only after that we, of course, need to 
think also the cases with a larger number of competing flows).

> {Note 2}: DOCSIS PIE implements limited dependence on packet size, in order 
> to reduce the chance of dropping small control packets (not because it 
> considers that drop should depend on packet size). In the spec, it apologizes 
> for not following RFC7567, but it includes safeguards against the 
> consequential DDoS vulnerability that it opens up.
>
>
> [BB] Can you give a more precise description of the AQM scenario you have in 
> your mind here? The words "at the same point" make me suspicious that you 
> might be thinking of something different to byte-mode drop. It's possible 
> you're talking about byte-mode queue measurement (see RFC7141 for 
> definitions). So please describe exactly how you think this AQM is working. 
> I.e. whether there are per-flow queues or a shared queue; in what units it 
> measures the queue, whether it marks packets of different sizes within the 
> same flow with different likelihoods, etc.
>
>
> reassembly never existed, because no-one implemented byte-mode drop. RFC7141 
> did not /cause/ your perfect world to collapse. It investigated and 
> discovered that byte-mode drop didn't exist, worked out why, articulated why, 
> and proposed a way forward that would give the desired complementarity 
> between network and hosts, without relying on byte-mode drop, which was 
> clearly problematic to everyone.
>
> You write the above as if byte-mode was deliberately introduced in RED so 
> that it worked well with the fragmentation. I'm afraid that's a rewrite of 
> history in reverse.
> The interaction between ECN and fragmentation hadn't even 
> been thought about when byte mode and packet mode were included in the 
> original 1993 paper on RED. Byte and packet mode were two options, which 
> Sally Floyd left configurable because it wasn't completely clear at that time 
> which one was most suitable (see the [pktByteEmail] reference in RFC7141 to 
> Sally's 1997 email about this).

[MK] {***}
Well, we seem to be reading a different written history ;)
the [pktByteEmail] reference in RFC7141 seems to point to Sally's not 
updated web page at LBL that is missing an additional email from year 
2000 where Sally explains why there is no bias against small packets 
(fragmented packets) when byte-mode drop is used. I fully agree with 
Sally with her analysis there. Please explain if you disagree with it. 
Please see the complete reference in Sally's ICIR web page:

  https://www.icir.org/floyd/REDaveraging.txt

I cannot speak for Sally and Van, but I believe that they quite carefully 
considered the bias agains small packets when they introduced byte-mode 
drop in their RED paper to mitigate the problem. Cannot tell if they 
thought of fragmentation at the time of designing and writing the RED 
paper but it works just perfectly fine in the case of fragmentation. I 
wouldn't be suprised if fragmenttation was one of the reasons for 
introducing byte-mode drop.

ECN was first introduced around the same time frame, and the leading 
principle there was not to give ECN undeserved preference over loss-based 
congestion control. And RFC 3168 fulfills it suprisingly well, and 
particularly well in terms of addressing fragmentation.

> 6 years later, when ECN first became an experimental RFC, there was still no 
> mention or thought of fragmentation in RFC 2481. I became involved during the 
> late stages of the update from RFC 2481 to what became RFC 3168 - 8 years 
> after the original RED paper. Jon Crowcroft and I pointed out that there was 
> no mention of fragmentation, as part of our attempt to make a more principled 
> division between the IP and TCP parts of RFC3168 - see our review at 
> real-time media perspective. not just TCP). The fragmentation approach that 
> was subsequently added to what became RFC3168 was still highly TCP-specific, 
> even though it was at the IP layer. However, I didn't feel I should hold up 
> RFC3168 to argue any further about it (unlike the present situation in tsvwg, 
> I took the view that I had arrived late at the party, so allowing others to 
> make progress was more important than me continually making the same point 
> over and over, even tho I thought I was right).
>
> Then we reach the 2008-2014 time-frame over which RFC7141 was developed - 
> when it was discovered that no-one was implementing byte-mode drop anyway 
> (see back where I started the story above).
>
>
>> 
> [BB] This is nostalgia for a past that never was. When no-one has done what 
> you think they should have done, it is important to try to understand why.

[MK] Well, cannot argue that too many have done it, but the original 
inventors of RED did it, and I believe they somehow understood how 
Standards Track congestion control works and what is the right thing to 
do.

Still, I do not have a good understanding about the actual reasons why 
byte-mide drop seemingly(?) was not much implemented by RED vendors. It 
seems to me that the reason potentially is in complexity of byte-mode in 
general (particulrly byte-mode queue measuring), not the complexity of 
byte-mode drop, and most likely that there was not that good understanding 
of the topic at the time RED was widely implemented, nor was there good 
guidance available either.

> See https://datatracker.ietf.org/doc/html/rfc7141#section-3.5

[MK] When an L2 decapsulator or an tunnel egress has become an end-point 
in the Internet architecture?

Efficiency is one aspect but it is much more important to see first where 
a function can be implemented correctly. An end-point (or any node in 
the Internet) cannot implement correctly a function when it does not know 
whether the function is needed or not. That is, if an end-point (or 
tunnel egress, or decapsulator) does not know what kind of AQM has marked 
the fragments, it cannot try to compensate the bias that the AQM 
potentially did with the fragments, right?

> Fragmentation is deprecated.

[MK] Deprcated?? Which RFC? I thought much of the fragmentation seen in 
the Internet is due to tunnels?? What am I missing, please advise?

> And few networks apply ECN marking where there 
> is no IP-awareness.
>
>
>> 

[MK] You never replied my previeous email that I pointed to and where I 
explained why CoDel does not bias agains small packets. Please read 
carefully the explanation in there and in this message as well, and 
please kindly indicate where my explanation in your view is incorrect.

> [BB] This is completely the reverse of reality. As explained above, RFC7141 
> was based on a survey of AQM implementations at the time, and it recorded 
> industry practice at the time. That practice has continued since then in 
> CoDel, PIE and now PI2, DCTCP, etc.

[MK] I should have pointed clearly to the recommendation I meant (RFC 
7141, Section 2.4):

  One can think of a splitting or merging process as if all the
  incoming congestion-marked octets increment a counter and all the
  outgoing marked octets decrement the same counter.  In order to
  ensure that congestion indications remain timely, even the smallest
  positive remainder in the conceptual counter should trigger the next
  outgoing packet to be marked (causing the counter to go negative).

Has an algorithm that implements this recommendation been described in 
detail, evaluated, analysed, and/or deployed somehere such that we could 
say it is the "best current practise"?

> In contrast, AFAICT, the /only/ use of byte-mode marking has been in the 

[MK] Mayby so, but at least there is a crisp enough description of the 
simple algorithm and quite a bit of evaluation, even such that a small 
error in the calculation of the drop probability was revealed and a 
correction introduced.

Regards,

/Markku

>>> [bobbriscoe.net]
>>> [bobbriscoe.net]

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 22 Jun 2021 22:38:31 +0300

> On 22 Jun, 2021, at 5:02 pm, Markku Kojo <[email-masked]> wrote:

Up front, let me say that I mainly agree with your approach to this.

> And, if the number of flows is increased, then the time between the marks decreases. So, we need to be careful in understanding what we are modelling. Otherwise, if we just play with the formulae, the outcome is just crap.

So, rather than playing with formulae, I organised an empirical test over the weekend to see what the effects really are.  That accounts for the long pause before replying.

	https://sce.dnsmgr.net/results/mss-tests/

The test involves three flows with the same CC algorithm running through a common single-queue, single-instance AQM.  The AQM is Codel as an exemplar of time-domain marking, or PIE as an exemplar of packet-mode marking, both with ECN enabled.  The test was repeated for Reno and CUBIC, as exemplars of standards-track CC, and DCTCP as an exemplar of 1/p response.  The path parameters are 50Mbps throughput, 80ms RTT, sufficient to keep CUBIC out of "Reno compatibility" mode.

The three flows differ in terms of packet and segment size.  Flow 1 is a conventional bulk flow using a full 1460-byte MSS.  Flow 2 reduces this to 730-byte MSS.  Flow 3 uses a 1460-byte MSS, but goes through a path with reduced MTU (with PMTUD disabled) causing fragmentation into two packets per segment, so has the same packet rate at the AQM as Flow 2.  Fragmentation reassembly is as per RFC-3168.

A well-conditioned congestion system should give each flow roughly equal application goodput on average.  Some allowance can be made for bandwidth lost to headers in the smaller packets, but this is a small effect.  But what do we actually get?

The theoretically ideal case is DCTCP with time-domain marking, where the expected goodput is exactly equal for all three flows.  The actual observed goodput is not quite equal, but reasonably close and very consistent over a 10-minute period:

	https://sce.dnsmgr.net/results/mss-tests/dctcp-fq_codel-plot.html

But if we apply packet-mode marking, the smaller packets are markedly disadvantaged compared to larger ones, approximately in proportion to the average packet size, and again this is consistent for 10 minutes straight:

	https://sce.dnsmgr.net/results/mss-tests/dctcp-pie-plot.html

I do not have running code with which to test a "marked bytes preserving" reassembly rule.  However, such a rule could only possibly influence Flow 3, as that is the only one that's fragmented.  In theory it would be elevated to the level of Flow 1, leaving Flow 2 as the only disadvantaged flow.

You might say that using small MSS is not reasonable for capacity-seeking traffic.  But recall what Paul Vixie pointed out in the other thread about UDP Options, that the MTU over Internet paths could reasonably increase in future.  This implies that "jumbo" TCP segments would, at least for some time, share space with traffic still using today's Ethernet-sized packets, and the latter would be in the analogous position of Flow 2 in this test.  So do not dismiss Flow 2 as irrelevant to your interests.

Moving on to Reno, we see roughly the same effect, though the large Reno sawtooth means we have to peer through a lot of noise to see the trends.  Nevertheless they are similar to DCTCP over a long average:

	https://sce.dnsmgr.net/results/mss-tests/reno-fq_codel-plot.html
	https://sce.dnsmgr.net/results/mss-tests/reno-pie-plot.html

And further likewise for CUBIC:

	https://sce.dnsmgr.net/results/mss-tests/cubic-fq_codel-plot.html
	https://sce.dnsmgr.net/results/mss-tests/cubic-pie-plot.html

These results with standards-track CC confirm those from DCTCP, and the same logic applies.

Current practice on the Internet is to use time-domain ECN marking (because Codel is the most widely deployed AQM with ECN enabled) and RFC-3168 fragment reassembly.  The above results show that this is clearly superior to packet-mode marking with either RFC-3168 or "byte-preserving" reassembly rules.  Hence we should probably try to avoid encouraging the latter behaviour in new Internet specifications.

 - Jonathan Morton

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 22 Jun 2021 22:35:48 +0000 (UTC)

Jonathan,
  
  It is indeed good to see measured data.

One point: You are talking about Codel below, but the graphs say fq_codel. Regardless of the merits of fq, here I think it is a bit of a confounder to what you want to  
show. For clarity, could you say if or how fq was in use? (I'm guessing here it might be in use, but configured in such a way as not to isolate these flows...)

Alex

- Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 23 Jun 2021 01:38:05 +0300

> On 23 Jun, 2021, at 1:35 am, [email-masked] wrote:
> 
> One point: You are talking about Codel below, but the graphs say fq_codel. Regardless of the merits of fq, here I think it is a bit of a confounder to what you want to show. For clarity, could you say if or how fq was in use?

We set the number of flow queues to 1, thereby effectively disabling FQ.

 - Jonathan Morton

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Tue, 22 Jun 2021 18:20:48 -0700 (PDT)

> We set the number of flow queues to 1, thereby effectively disabling FQ.

To reduce any confusion here and in the future this "configuration"
should be noted if it is not.  Something like codel (implemented
using fq_codel with flow queues set to 1), or some such.

Regards,

-- 
Rod Grimes                                                 [email-masked]

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 23 Jun 2021 04:35:32 +0300

> On 23 Jun, 2021, at 4:20 am, Rodney W. Grimes <[email-masked]> wrote:
> 

It actually is, in the header of the relevant plots: "fq_codel flows 1".

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 23 Jun 2021 11:02:18 +0200

Hi,

I just want to note that (almost) all of the results Pete posted, used and documented the same fq_codel with single queue method.
Also in Linux both sch_codel (see https://elixir.bootlin.com/linux/latest/source/net/sched/sch_codel.c) 
and sch_fq_codel (see https://elixir.bootlin.com/linux/latest/source/net/sched/sch_fq_codel.c) share  some of the codel implementation via:

#include <net/codel_impl.h>
#include <net/codel_qdisc.h>

so "fq_codel flows 1" seems an acceptable method to get to a single queue codel AQM, and one that loows to tests against multi-quueu/fq_codel with minimal unrelated changes.

Regards
	Sebastiab

> On Jun 23, 2021, at 03:35, Jonathan Morton <[email-masked]> wrote:
>