Subject: [tsvwg] Fwd: New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-02.txt

From: Pete Heist <[email-masked]>
Date: Mon, 08 Mar 2021 11:09:00 +0100

-------- Forwarded Message --------
Subject: New Version Notification for draft-heist-tsvwg-ecn-deployment-
observations-02.txt
Date: Mon, 08 Mar 2021 01:57:07 -0800

> A new version of I-D, draft-heist-tsvwg-ecn-deployment-observations-
> 02.txt
> has been successfully submitted by Peter G. Heist and posted to the
> IETF repository.
> 
> Name:           draft-heist-tsvwg-ecn-deployment-observations
> Revision:       02
> Title:          Explicit Congestion Notification (ECN) Deployment
> Observations
> Document date:  2021-03-08
> Group:          Individual Submission
> Pages:          30
> URL:           
> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.txt
> Status:        
> https://datatracker.ietf.org/doc/draft-heist-tsvwg-ecn-deployment-observations/
> Html:          
> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.html
> Htmlized:      
> https://tools.ietf.org/html/draft-heist-tsvwg-ecn-deployment-observations-02
> Diff:          
> https://www.ietf.org/rfcdiff?url2=draft-heist-tsvwg-ecn-deployment-observations-02
> 
> Abstract:
>    This note presents data gathered at an Internet Service Provider's
>    gateway on the observed deployment and usage of ECN.  Relevant IP
>    counter and flow tracking data was collected and analyzed for TCP  
> and other protocols.

A few updates in this version:

* Improved statistics for the separated "known" and "unknown" AQMs.

* Clarified the detection method for possible AQMs.

* Added possibility of QUIC-ECN traffic in non-TCP data.

Pete

---

From: Pete Heist <[email-masked]>
Date: Mon, 08 Mar 2021 23:19:10 +0100

I'm following up in this thread to one question that's come up- why is
an ISP deploying fq_codel in its backhaul?

In short, this ISP encourages research and experiments by its members,
and this was a way to both see how much congestion exists, and attempt
to manage it with AQM when needed. Experiments with fq_codel were
started a couple of years ago, and I helped add support for it to their
custom router interface early last year. However, it's only actually
been enabled on a couple of routers, because:
- it takes a while to update routers in this environment
- some of the routers run kernels prior to fq_codel support
- congestion doesn't usually last long enough to give it much urgency,
and sometimes it's fixed by just adding capacity

IMO, fq_codel isn't ideal in this situation. When there actually is
congestion to manage, flow-fairness means that one user can dominate
the queue with many flows, which is exactly one of the problems we'd
like to avoid. Host fairness could improve on this, or better yet,
member fairness using the member database, as members can have more
than one IP/MAC. But, fq_codel is what's available now, and generally
usable enough in this case to get started with AQM.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 9 Mar 2021 00:25:40 +0000

Pete,

Thanks for the updated draft a fortnight ago, with the extra columns I 
asked for.

I've been excluding the rows with obvious pathologies.

Of the 90 IPs in your TCP results, 74 have at least one obvious 
pathology, leaving only 16 clean.
And actually, there was a pathology suggested in your slides today that 
isn't in the above,
     If I check for that too, 83 IPs have at least one pathology, 
leaving only 7 clean.

Any idea what is going on?

==Details==

In the TCP results, unlike with QUIC, the reverse feedback is visible. 
So I can check for more pathologies.

I checked for this obvious pathology, that wasn't directly mentioned in 
the slides today:
* Congestion experienced (CE) in one direction but no echo congestion 
experienced (ECE) feedback in the other.
* Or conversely, echo congestion experienced (ECE) feedback in one 
direction but no congestion experienced (CE) in the other.

Of the 90 IPs in the TCP results, 40 from the WAN have this pathology 
and 38 from the LAN
(taking the direction as that of the data)
Only 16 IPs, don't have this pathology in either direction.

I would have thought you guys would have picked up these when checking 
for the ECE/CE >= 2
     'cos otherwise you would have got loads of divide by zero errors 
(and/or zeros).

If I include the pathology you mention where ECE/CE < 2
In all, of the 90 IPs in the TCP results, after I exclude any IP with a 
pathology, I only have 7 IPs left.

Bob

PS. I agree it's v strange for an ISP to share capacity between its 
members by how many flows they are using.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Pete Heist <[email-masked]>
Date: Tue, 09 Mar 2021 02:13:54 +0100

Hi Bob, comment below...

> direction but no congestion experienced (CE) in the other.

I believe I've got your question right, but let's see. From the vantage
point of the gateway, we don't always get to see CE in one direction
and ECE in the other. In fact, it seems a lot of the time we don't.

Say downstream traffic passes through the gateway as ECT(0) and gets
marked in an AQM later with CE, before arriving at the user endpoint.
We haven't seen that CE at the gateway, but we still see the ECEs in
the opposite direction. The same can happen upstream, if CE is marked
somewhere after packets leave the gateway. That's why we don't require
the presence of CE marks to mark an entry "possible AQM", but we use
them if they're present.

Pete

> > > >     gateway on the observed deployment and usage of ECN. 
> > > > Relevant
> > > > IP
> > > >     counter and flow tracking data was collected and analyzed
> > > > for
> > > > TCP
> > > > and other protocols.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Mar 2021 13:37:43 +0100

> On 08/03/2021 22:19, Pete Heist wrote:
[...]
>> usable enough in this case to get started with AQM.

[...]

> On Mar 9, 2021, at 01:25, Bob Briscoe <[email-masked]> wrote:
[...]
> PS. I agree it's v strange for an ISP to share capacity between its members by how many flows they are using.
[...]

I just want to note two things here:

a) even without a flow queueing AQM on the bottleneck, sharding works to get a higher part of the capacity since TCP tends to be more or less fair to itself

b) most ISP put a lid on games like that by simply also enforcing (policing or shaping) each users aggregate capacity to numbers mentioned in the contract...

So fq_codel, while by no means ideal here, does not really make thinks worst that the status quo, it is just that sharding can make pure flow fairness regress to less equitable sharing between classes other than flows.

For a cooperative use case, something like a per-member QFQ instance that equitably shares capacity between members with an fq_codel inside each of the QFQ classes seems like a better fit, no?

Best Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Tue, 09 Mar 2021 14:51:27 +0100

Hi Sebastian, one comment below...

On Tue, 2021-03-09 at 13:37 +0100, Sebastian Moeller wrote:
> a) even without a flow queueing AQM on the bottleneck, sharding works
> to get a higher part of the capacity since TCP tends to be more or less
> fair to itself
> 
> b) most ISP put a lid on games like that by simply also enforcing
> (policing or shaping) each users aggregate capacity to numbers
> mentioned in the contract...
> 
> So fq_codel, while by no means ideal here, does not really make thinks
> worst that the status quo, it is just that sharding can make pure flow
> fairness regress to less equitable sharing between classes other than
> flows.
> 
> For a cooperative use case, something like a per-member QFQ instance
> that equitably shares capacity between members with an fq_codel inside
> each of the QFQ classes seems like a better fit, no?

Separating capacity fairly between members is exactly what they'd want,
yes. In this coop there aren't any limits.

I should have mentioned before that there _is_ actually a crude host
fairness mechanism in place, using u32 hashing filters on the last
octet of the IP address, and 256 fq_codel instances. It's slow when
enabling and not ideal, so hasn't been used thus far, but it works.

fq_codel does make things better than the status quo, it's just that we
can imagine something that's more suited for ISP usage out of the box,
making it easier and more efficient to split capacity than by many tc
classes and filters. This was discussed in the bufferbloat circles at
some point too.

That's the long way of explaining how they started using fq_codel,
coming back to that original question. :) We do know that other ISPs
are doing the same, in various ways, so it's not a unique usage.

Pete

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Mar 2021 15:52:51 +0200

> On 9 Mar, 2021, at 2:37 pm, Sebastian Moeller <[email-masked]> wrote:
> 

This is true IMHO.  FQ still has a significant advantage here, in that latency effects are still isolated between flows; sparse flows, which are typically more latency-sensitive, are not affected by the various saturating flows rushing past them on all sides.  That would not necessarily be true with a single queue.

> For a cooperative use case, something like a per-member QFQ instance that equitably shares capacity between members with an fq_codel inside each of the QFQ classes seems like a better fit, no?

This is similar in effect to what Cake can be configured to do.  Cake does it by weighting the delivery rate of individual flows to produce a constant total weight per host.

Since this seems to be a fairly common use case for ISPs, I'm considering writing a qdisc specialised to do it efficiently.  This could be a drop-in replacement for an fq_codel or HTB+fq_codel deployment, assuming the device needing it is running a new enough kernel.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 9 Mar 2021 15:30:19 +0100

HI Jonathan,

> On Mar 9, 2021, at 14:52, Jonathan Morton <[email-masked]> wrote:
> 

	[SM] Guess where I got my inspiration from ;)?

> Since this seems to be a fairly common use case for ISPs, I'm considering writing a qdisc specialised to do it efficiently.  This could be a drop-in replacement for an fq_codel or HTB+fq_codel depl…

	[SM] Most commercial ISPs tend to sell fixed PIR type of contracts, so would require a per "user" control for the shaper speed, no?

Best Regards
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 9 Mar 2021 14:42:15 +0000

I think I was one of the people wondering this, so thanks for this:

> On 8 Mar 2021, at 22:19, Pete Heist <[email-masked]> wrote:
> 
Interesting, and always helpful to see a little more of why specific deployments take place,

Gorry

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 9 Mar 2021 15:04:49 +0000

Hi Jonathan, Sebastian

If you update Cake, please consider to add the possibility to distinguish
between the ECT(0) and ECT(1) codepoint. A first reasonable step is to make
the algorithm ignore the ECT(1) codepoint. This is brought up in section 5.1
in the L4S opt draft and also suggested by Tom Henderson.

Regards
Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
> Sent: den 9 mars 2021 14:53
> To: Sebastian Moeller <[email-masked]>
> Cc: tsvwg IETF list <[email-masked]>; [email-masked]
> Subject: Re: [tsvwg] New Version Notification for draft-heist-tsvwg-ecn-
> deployment-observations-02.txt
> 
fair to
> itself
(policing or
> shaping) each users aggregate capacity to numbers mentioned in the
> contract...
> >
fairness
> regress to less equitable sharing between classes other than flows.
> 
> This is true IMHO.  FQ still has a significant advantage here, in that
latency
> effects are still isolated between flows; sparse flows, which are
typically
> more latency-sensitive, are not affected by the various saturating flows
> rushing past them on all sides.  That would not necessarily be true with a
> single queue.
> 
> This is similar in effect to what Cake can be configured to do.  Cake does
it by
> weighting the delivery rate of individual flows to produce a constant
total
> weight per host.
> 
> Since this seems to be a fairly common use case for ISPs, I'm considering
> writing a qdisc specialised to do it efficiently.  This could be a drop-in
> replacement for an fq_codel or HTB+fq_codel deployment, assuming the
> device needing it is running a new enough kernel.
> 
>  - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Mar 2021 17:24:54 +0200

> On 9 Mar, 2021, at 5:04 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

No, because that would violate RFC-3168.  Until that is actually deprecated by the IETF, we should continue to retain compliance with it.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 9 Mar 2021 15:42:01 +0000

> No, because that would violate RFC-3168.  Until that is actually deprecated by the IETF, we should continue to retain compliance with it.

I think this is in section 4.1?

I understand that to say: if a router wishes to be compatable with L4S, 
this is one way to do this. The recommendation(s) could be stronger, if 
the community thinks this is what is needed.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Mar 2021 17:44:19 +0200

> On 9 Mar, 2021, at 5:42 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> I understand that to say: if a router wishes to be compatable with L4S, this is one way to do this. The recommendation(s) could be stronger, if the community thinks this is what is needed.

As I tried to point out in my talk, the problem is that a great many people are already running RFC-3168 compliant AQMs with no knowledge of L4S and, therefore, no "wish to be compatible" with a new spec that explicitly violates the spec they built against.

Myself included.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 9 Mar 2021 16:09:32 +0000

> As I tried to point out in my talk, the problem is that a great many people are already running RFC-3168 compliant AQMs with no knowledge of L4S and, therefore, no "wish to be compatible" with a ne…

This is pecisely why we have a standards process to make this change. 
When the work on L4S was first brought to the IETF, I know many people 
weighed the prospects of replacing the RFC3168 behavior.

When the L4S specs are published, this will indeed be a disruptive 
change for some, and people deploying and implementing do need to 
consider this. The place to describe this might be in 
draft-white-tsvwg-l4sops within the tsvwg, if this is adopted (this 
adoption call is on-going currently on the tsvwg list).

Gorry

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 9 Mar 2021 16:33:03 +0000 (UTC)

Gorry, Jonathan, tsvwg

See below  [AB]

    On Tuesday, March 9, 2021, 4:18:23 PM GMT, Gorry Fairhurst <[email-masked]> wrote:  

[AB] Not sure it's a problem, but rfc8311 (sec 4.2) says variation of marking behavior between ECT(0) and ECT(1) must be specified in an experimental RFC. The ops call just made for draft-white-tsvwg-l4sops is to be an informational RFC. 

With hindsight, I'm not sure why rfc8311 (which is standards track) didn't deprecate marking ECT(1) the same way it deprecated endpoints using it, outside experiments.

Alex

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 9 Mar 2021 16:41:23 +0000

On 09/03/2021 16:33, [email-masked] wrote:
> >> On 9 Mar, 2021, at 5:42 pm, Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >>
> >> I understand that to say: if a router wishes to be compatable with 
> L4S, this is one way to do this. The recommendation(s) could be 
> stronger, if the community thinks this is what is needed.
> > As I tried to point out in my talk, the problem is that a great many 
> people are already running RFC-3168 compliant AQMs with no knowledge 
> of L4S and, therefore, no "wish to be compatible" with a new spec that 
> explicitly violates the spec they built against.
> >
> > Myself included.
> >
> >  - Jonathan Morton
>
> [AB] Not sure it's a problem, but rfc8311 (sec 4.2) says variation of 
> marking behavior between ECT(0) and ECT(1) must be specified in an 
> experimental RFC. The ops call just made for draft-white-tsvwg-l4sops 
> is to be an informational RFC.
>
> With hindsight, I'm not sure why rfc8311 (which is standards track) 
> didn't deprecate marking ECT(1) the same way it deprecated endpoints 
> using it, outside experiments.
>
>
> Alex
>
Thanks Alex,

I noted that.

Let's see if there is WG consensus to support the OPS draft for L4S.

If you think we need RFC 2119 language to express the intention of the 
draft, then we'll certainly take this into consideration after the WG 
adopts the ID. we'll figure out the standards status for this document - 
this is largely a Chairs role to steer the documentation/recommendations 
to the appropiate specs, but one where we do appreciate input!

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Mar 2021 19:17:57 +0200

> On 9 Mar, 2021, at 6:33 pm, [email-masked] wrote:
> 

Because there's more than one possible experiment involving the ECN field.  SCE actually prefers that existing networks and endpoints continue to comply with RFC-3168 in these respects, originating ECT(0) (though if they originated ECT(1) it actually wouldn't hurt anything) and treating both ECT codepoints as equivalent.  If we started treating ECT(1) differently in the network, that would make deploying SCE more difficult.

Which is another reason I'm opposed to pre-emptively moving to accommodate L4S.  But even if I were to make that change "today" (which I probably couldn't, because Linux kernel maintainers also like to be RFC compliant), there would be a *long* transition period as existing equipment sits around with old firmware, during which L4S would still need to accommodate existing networks or stay away from them, not the other way around.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Mar 2021 19:20:26 +0200

> This is pecisely why we have a standards process to make this change. 

Yes, it is, and we have a responsibility to conduct it properly.

> weighed the prospects of replacing the RFC3168 behavior.

How many years ago was that?  I'm pretty sure ECN AQM deployment is a lot more widespread now than it was then.  So that decision probably needs to be revisited.

I think there will be a lot of spirited discussion on this front over l4s-ops.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Mar 2021 08:20:27 +0000

Hi Jonathan

I most certainly hope that we don't spend valuable meeting time on
discussing the real or imagined merits with SCE. 

If you find that specs are unclear as to how enable L4S support in Linux
kernels and elsewhere, including how to navigate around the RFC lingo, then
please point out where to fix this. A good starting point is the L4S Ops
draft.

Like things are now my only interpretation of the situation is that you
place your bets on that L4S will fail and you do your best to make this
failure happen. For instance you point out a problem with fq-codel and L4S
but show little interest or even oppose solutions to have this issue
remedied. Please prove me wrong.

/Ingemar

> Sent: den 9 mars 2021 18:18
> To: [email-masked]
> Cc: Gorry Fairhurst <[email-masked]>; tsvwg IETF list
> <[email-masked]>
didn't
> deprecate marking ECT(1) the same way it deprecated endpoints using it,
> outside experiments.
> 
> Because there's more than one possible experiment involving the ECN field.
> SCE actually prefers that existing networks and endpoints continue to
comply
> with RFC-3168 in these respects, originating ECT(0) (though if they
originated
> ECT(1) it actually wouldn't hurt anything) and treating both ECT
codepoints as
> equivalent.  If we started treating ECT(1) differently in the network,
that
> would make deploying SCE more difficult.
> 
> Which is another reason I'm opposed to pre-emptively moving to
> accommodate L4S.  But even if I were to make that change "today" (which I
> probably couldn't, because Linux kernel maintainers also like to be RFC
> compliant), there would be a *long* transition period as existing
equipment
> sits around with old firmware, during which L4S would still need to
> accommodate existing networks or stay away from them, not the other way
> around.
> 
>  - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 10 Mar 2021 14:20:22 +0200

> On 10 Mar, 2021, at 10:20 am, Ingemar Johansson S <[email-masked]> wrote:
> 

We have already been running tests using the published L4S kernel, as was discussed at previous meetings.  Here are some results we added recently, and mentioned in my talk on Monday:

	https://github.com/heistp/l4s-tests/#harm-to-flow-completion-time

> remedied. Please prove me wrong.

The reason we oppose L4S is because we believe it constitutes a danger to the Internet.  That was the thrust of my talk on Monday, supported by data collected recently by Pete and previously by Jake.

The reason I, personally, will not attempt to take pre-emptive actions to "enable" L4S deployment is because I have no desire to make that danger more likely to occur, nor to make deployment of less harmful alternative more difficult.  I consider it likely that upstream kernel maintainers would also reject such actions, were I to propose patches, on the grounds that they reduce compliance with existing published RFCs.  I have already explained this clearly.

We have proposed two methods of eliminating the problems with L4S.  One of them is SCE, first introduced almost exactly two years ago, which has running code that is ready to try out (in the lab) by anyone so inclined.  The other one is similar to SCE, but preserves the "classifier" input in the ECN field; it will require more handshake negotiation than SCE as a result, and has not been developed in detail.

The major benefit of both these proposals is that they shift the risk from "innocent bystanders" to "involved participants" by explicitly prioritising compatibility with existing networks, and only triggering the new congestion responses when a new-type AQM explicitly identifies itself as controlling the bottleneck on the path.  This benefit is not identified or discussed in the L4S drafts' appendices discussing these proposals, which is what prompted the topic of my talk on Monday.

Yet both of these proposals have been rejected outright by the L4S team and their allies, and there has been very little net technical change in L4S since we appeared on the scene.  This is despite changes in the existing Internet environment apparently refuting some of the assumptions that drove certain L4S design decisions.

Honestly, we believe the WG is wasting its time with further discussion of L4S per se, but it's up to the Chairs and the WG at large to make that determination.  I will read the l4s-ops draft again and see whether it approaches adequacy.  If it does not, then I will oppose its adoption and explain why.  It's the only ethical thing I can do here.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 22 Mar 2021 16:53:12 +0000

Pete,

On 09/03/2021 01:13, Pete Heist wrote:
>> * Congestion experienced (CE) in one direction but no echo
>> congestion experienced (ECE) feedback in the other.

[BB] OK that's a feasible explanation... but the numbers still seem weird...

Can you provide more insight into the topology of this ISP. You say it 
is a co-operative wireless ISP. Are all the access links symmetric?

I'm asking because, in the direction from the LAN, it seems highly 
unlikely that 38 of the 90 IPs experienced no ECN marking in the 
upstream access link or the backhaul, but they did experience ECN 
marking somewhere further into the network (which could include another 
access link for p2p traffic).

The results for the 38 IPs backhauled via the FQ-CoDel nodes (known to 
support ECN) are even more unexpected (again taking only the TCP 
results, because they show both directions):
* In the direction from the LAN, 10 of these 38 IPs saw no ECN marking 
from the LAN side (including none from those FQ-CoDel nodes), but they 
did see the feedback of ECN marking from somewhere deeper into the network.
* The results in the direction from the WAN are more what I would 
expect: 30 of the 38 IPs saw no ECN marking from the WAN side, but they 
did see the feedback of ECN marking that must have occurred on the LAN 
side (where the FQ-CoDel nodes are).

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 22 Mar 2021 17:51:45 +0000

Sebastian,

> So fq_codel, while by no means ideal here, does not really make thinks worst that the status quo, it is just that sharding can make pure flow fairness regress to less equitable sharing between clas…

[BB] You seem to be using "the status quo" to mean TCP sharing out the 
capacity between users. But that has never been the status quo - since 
even before TCP congestion control was invented in 1988 per-customer 
scheduling has been the norm for public ISPs {Note 1}. Irrespective of 
access technology , whether DSL, DOCSIS cable, PON, mobile, satellite, 
etc. They all share downstream capacity between customers at a node that 
is either at the head of the bottleneck access link, or logically 
limited to match the capacity of the customer's bottleneck access link.

You might be able to turn a few debating somersaults and claim that the 
problem is the absence of a per-customer scheduler, not the presence of 
an FQ scheduler where this scheduler would normally sit. If that allows 
you to sleep at night, you're welcome to live in your Alice through the 
Looking Glass world.

{Note 1}: I can't immediately find evidence from the late 1980s of this. 
All I can find is Appx B of TR-059 
<http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=6924A1B1458AE2F6FEF15047BC95A7EC?doi=10.1.1.176.7875&rep=rep1&type=pdf> 
from the DSL Forum, but that was only 2003 when QoS was added after 
Diffserv was standardized. The closest I can get to something older is 
the second half of RFC970 written in 1985, which was an early step 
towards per-customer scheduling. You can skip over the first half that 
is trying to solve an early form of bufferbloat that predated TCP 
congestion control. Alternatively, I'm sure many people on this list 
will be able to confirm that per-customer scheduling has 'always' been 
the norm.

> For a cooperative use case, something like a per-member QFQ instance that equitably shares capacity between members with an fq_codel inside each of the QFQ classes seems like a better fit, no?

Well, once you've got the per-member scheduling, an AQM in each member's 
FIFO would suffice. But you can have your FQ-CoDel in there instead if 
you must.

When a node is serving a few thousand members, you don't want to 
unnecessarily have a thousand or so queues per member as well - at least 
not if you can achieve similar performance without them.

For instance, BT's network architect set me the task of simplifying BT's 
QoS architecture, because even 5 queues per customer was considered to 
be the main cause of system complexity. See Section 3.1.2 of this 
report: 
https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 22 Mar 2021 21:48:37 +0100

Hi Bob,

More below, prefixed [SM].

On 22 March 2021 18:51:45 CET, Bob Briscoe <[email-masked]> wrote:
>> So fq_codel, while by no means ideal here, does not really make
> thinks worst that the status quo, it is just that sharding can make
> pure flow fairness regress to less equitable sharing between classes
> other than flows.
> 

[SM] Ah, no, with status quo I meant both a) and b) above. I considered my phrasing unambiguous, but still managed to confuse you, sorry for that.

To recapitulate, my point is sharding is unfortunate but hardly a problem unique to FQ, as of a) sharding will give gains even in the absence of FQ, and as of b) sharding from endusers might not actually be a problem ISP/transit networks need to worry about much, they can treat it as an SEP.

> problem is the absence of a per-customer scheduler, not the presence of an FQ scheduler where this scheduler would normally sit. If that allows you to sleep at night, you're welcome to live in your Alice through theLooking Glass world.

[SM] I probably could, but why would I? But since it seems you misunderstood me, maybe re-read what I wrote?

> will be able to confirm that per-customer scheduling has 'always' been 

[SM] Thanks for digging this up. This nicely confirms my point b). I did not claim that to be a new development, even though I am not sure when ISPs switched from simple policers to user friendlier shapers?

> FIFO would suffice. But you can have your FQ-CoDel in there instead if 

[SM] This is not so much a philosophical point as you might believe. I tried single and fq AQMs on my homelink, and for my use cases FQ works considerably better. And I know because I tried. BUT to do this for ingress I operate a traffic shaper on the wrong end of the bottleneck, if I could get my ISP to run fq-codel on the download side of my link at its BNG, I would be willing to pony up money for.
Again, I tried single queue AQMs and found them lacking. Currently I operate something close the proposed model, only I use cake for both the per-internal-IP isolation (member level) and for fq inside each of the top level aggregates. Works well for a family of five in spite of home schooling and video conferencing and remote desktop control, video streaming and downloads. 
I wish all L4S proponents here would take it upon themselves to try for only a week how the state of the art actually feels, but I digress.

> not if you can achieve similar performance without them.

[SM] That would be great. Data so far indicates however performance is not going to be similar.... at least not similar enough to make this a wort while endeavor. That said, if my ISP would offer even only RED on my ingress per-customer queue I would also already be willing to pay for. I would probably still supplement with FQ in my side, but worst case back-spill into the ISP's upstream buffers would be less of an issue.

> https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf

[SM] I see, and I am not surprised. I also do not think your proposal is that much of a simplification, especially given the open ends like : "• Cons: Still we need convincing evidences that this would work, understand the packet loss trade-off especially for the interactive apps" and "• always plan sufficient capacity for sum of all VASs then no need for a separate priority for each value-added service" and "	• Isolation: between traffic within CVLAN e.g. femtocell, or third-party Wi-Fi network (such as FON), i.e. (customer vs non-customer traffic) can be achieved within one queue ( to be evaluated)" there seems to be an awful lot left to do before the proposal can replace the reference, no?

The head line problem "The use case is about serving HTTP Adaptive Streaming (HAS) in the presence of background TCP flows, when the downstream traffic per line is sufficient to saturate the queue at the BNG" is pretty much solved by cakes per-internal IP fairness as long as the per-IP share of capacity suffices for the HAS demands. If not the only solution is to statically carve out a reservation for that stream or grant priorities. (And nothing else is it you do, you allow the HAS streams to dominate the other TCP flows, which is fine if your users prefer the streams over the other data, but since that is not guaranteed to be true, doing that unconditionally seems not ideal). 
And as I might add, it does so without requiring the use of non-internet compatible transport protocols like DCTCP or TCP Prague (which in all fairness might turn out to be internet worthy, but has not convincingly done so so far).

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 22 Mar 2021 23:43:50 +0200

> On 22 Mar, 2021, at 10:48 pm, Sebastian Moeller <[email-masked]> wrote:
> 
>> I'm sure many people on this list 

Quick question:  if an RFC-3168 compliant FQ-AQM is configured to provide only per-subscriber isolation, relying on TCP-friendly coexistence for managing traffic to a given subscriber, what happens when an L4S flow joins the mix?

I'm sure I don't need to spell it out by now.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Tue, 23 Mar 2021 10:35:00 +0100

Bob,

On Mon, 2021-03-22 at 16:53 +0000, Bob Briscoe wrote:
> > > Thanks for the updated draft a fortnight ago, with the extra
> > > columns
> > > I
> > > asked for.
> > > 
> > > And actually, there was a pathology suggested in your slides
> > > today
> > > that
> > > isn't in the above,
> > > I checked for this obvious pathology, that wasn't directly
> > > mentioned
> > > in
> > > the slides today:

>From what I've seen, many seem to have roughly symmetric rates, but
this can vary. Whereas my connection tends to be about 2:1 download to
upload, there are some members whose upload rates are typically higher
than download. There are also some with fiber.

Pete

> > > In all, of the 90 IPs in the TCP results, after I exclude any IP
> > > with
> > > a
> > > > I'm following up in this thread to one question that's come up-
> > > > why
> > > > is
> > > > started a couple of years ago, and I helped add support for it
> > > > to
> > > > their
> > > > > > has been successfully submitted by Peter G. Heist and
> > > > > > posted to
> > > > > > the
> > > > > > IETF repository.
> > > > > > 
> > > > > >      This note presents data gathered at an Internet
> > > > > > Service
> > > > > > Provider's

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 23 Mar 2021 12:03:40 +0200

> On 22 Mar, 2021, at 6:53 pm, Bob Briscoe <[email-masked]> wrote:
> 
> The results for the 38 IPs backhauled via the FQ-CoDel nodes (known to support ECN) are even more unexpected (again taking only the TCP results, because they show both directions):
> * In the direction from the LAN, 10 of these 38 IPs saw no ECN marking from the LAN side (including none from those FQ-CoDel nodes), but they did see the feedback of ECN marking from somewhere deeper into the network.
> * The results in the direction from the WAN are more what I would expect: 30 of the 38 IPs saw no ECN marking from the WAN side, but they did see the feedback of ECN marking that must have occurred on the LAN side (where the FQ-CoDel nodes are).

Some members of this ISP run servers.  This might be surprising from a UK or US perspective, as the incumbent telcos have tended to prevent people from doing that on a "domestic grade" service contract.  But since this is a cooperative ISP, the rules are different.

These servers are "found" by the background of port scanners that has been a feature of the Internet for many years now.  These port scanners often perform a "fingerprint" analysis of servers they find, which includes a probe of their ECN behaviour.  Each port scan is thus associated with an equal number of "artificial" CE marks and ECE flags.  As described, we filtered patterns like that out of our results post-hoc.

This leaves open the possibility of servers at "our" end which see port-scanning activity, also see little congestion within the ISP, but see a small amount of congestion, occasionally, at the client end of their connections, some small fraction of which are ECN Capable.  This is what we think those cases you refer to probably represent.

One possible explanation for this is clients connecting over wifi, through an AP that supports the Airtime Fairness feature introduced for several popular wifi chipset drivers in Linux some years ago.  This includes ath9k, ath10k, and mt76, which together account for a large fraction of deployed wifi APs.  It would not be strange for a wifi link to sometimes be a lot narrower than some of the ISP members' links.

 - Jonathan Morton