Subject: Re: [tsvwg] a WGLC review and support for draft-ietf-tsvwg-l4s-arch

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Oct 2021 23:44:26 +0100

Neal,

Thank you again for these.
See [BB] inline. No comment means I've silently accepted the request.

On 09/08/2021 04:04, Neal Cardwell wrote:
> Regarding the L4S architecture draft:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/ 
> <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/>
>
> I support moving this draft to RFC status.
>
> Below are some editorial comments from reading the most recent version 
> of the draft, draft-ietf-tsvwg-l4s-arch-10, dated July 1, 2021.
>
> These are all minor editorial or wordsmithing suggestions. I do not 
> have any technical objections to the draft.
>
> Comments:
>
> ---
>
> current draft text:
>   "The L4S architecture is intended to enable
>    _all_ Internet applications to transition away from congestion
>    control algorithms that cause queuing delay"
>
> Taken literally, describing the goal as to "transition away from 
> congestion control algorithms that cause queuing delay" seems perhaps 
> unintentionally strong, since AFAIK any practical congestion control 
> scheme, including L4S, will still cause some non-zero queuing delay. :-)
>
> I'd suggest:
>   "The L4S architecture is intended to enable
>    _all_ Internet applications to transition away from congestion
>    control algorithms that cause large queuing delays
>    (where "large" includes, for example, delays on the order of
>    the unloaded round-trip time or longer)"

[BB] I don't like introducing a word that then has to be explained and 
qualified, when the abstract is meant to be brief.
Thesauraus suggests:
significant
substantial    <<== I'm going for this, without further qualification 
(until the reader gets to the 1st para of the Intro.
considerable

>
> ---
>
> current draft text:
>    It must be said that queuing delay only degrades performance
>    infrequently [Hohlfeld14].  It only occurs when a large enough
>    capacity-seeking (e.g. TCP) flow is running alongside the user's
>    traffic
>
> AFAICT [Hohlfeld14] is based on tests in "laboratory-based testbeds", 
> so I'm not sure it's the best support for a claim that queuing delay 
> only degrades performance infrequently. Agreed, queuing delay only 
> degrades performance in specific conditions, but those specific 
> conditions may be quite common, and only large-scale measurements of 
> real production/user workloads could quantify how frequently those 
> conditions arise. Given the mindshare that the problem of bufferbloat 
> has captured over the past decade, and the numerous AQM and congestion 
> control schemes that have been offered to fight it, I think it's fair 
> to say that queuing delay degrades performance frequently enough for 
> it to be considered a significant concern.
>
> I'd suggest:
>
>    Queuing delay degrades performance [Hohlfeld14] when a large enough
>    capacity-seeking (e.g. TCP) flow is running alongside the user's
>    traffic
>

[BB] Certainly the largest part of [Hohlfeld14] (about QoE from § 
onwards) was in lab. But §3 is about a measurement study of "430 million 
randomly selected TCP/HTTP connections captured at a major Content 
Distribution Network (CDN)."

Mindshare captured isn't a scientific measure. Really bad bufferbloat 
has been found, and really really bad bufferbloat has been demonstrated. 
But none of that is evidence of prevalance of the effect on actual 
users. It could be evidence of hysteria, if there are no really good 
empirical studies of prevalance (I'm not saying it /is/, just that it 
could be).

I've just sent you an email I sent recently to an IAB list about this 
Hohlfeld14 paper (and others that try to establish the prevalence of 
buffer bloat).
The quantification of the size of the delay has 2 flaws (explained in 
the other email) but let's run with it anyway...

 From the headline conclusion, you would imagine that [Hohlfeld14] has 
found that bufferbloat is not a great effect:
     "Two very recent studies [...] concluded that the magnitude of 
bufferbloat is modest. Our results [...] further substantiate these 
findings."
However, I think this could be a crying wolf effect, where all the 
bigging up of the bufferbloat issue has made many researchers imagine 
the sky is falling, so when they quantify the issue, even though it 
looks awful to me, they say it's not so bad (...as they had been led to 
believe).

Here's the most relevant quote from the interpretation of the results 
(you can look at Fig 1 yourselves):
"
But then
Most of the user flows experience a mod-
est amount of queueing; 80% of all the flows experience less than
100ms of delay variation. Only 2.8% (1%) experience excessive
queueing delays of more than 500ms (1000ms). This corresponds
to only 2.5% (2%) of the observed hosts.
"

Whatever, I agree that we should take 'only' out of our sentences. But 
the reason I put this sentence in the draft was to ensure that people 
are aware this effect is intermittent, so I'd like to keep that word, 
which is justified.

    Queuing delay degrades performance
    intermittently [Hohlfeld14].  It occurs when a large enough
    capacity-seeking (e.g. TCP) flow is running alongside the user's
    traffic

> ---
>
> current draft text:
>       Section 4 gives a high level explanation of how FQ and DualQ 
> solutions work
>
> I'd suggest making it more clear that the "FQ" approach here is FQ 
> plus L4S support, e.g.:
>       Section 4 gives a high level explanation of two
>       kinds of L4S network solutions, FQ and DualQ,

[BB] "Section 4 gives a high level explanation of how the per-flow-queue 
(FQ) and DualQ variants of L4S work"

>
> ---
> current draft text:
>
>    2.  L4S Architecture Overview
>
>    There are three main components to the L4S architecture; the AQM in
>    the network, the congestion control on the host, and the protocol
>    between them:
>
>    1) Network:  L4S traffic needs to be isolated from the queuing
>       latency of Classic traffic.
>
> AFAICT the architecture overview section on the network only discusses 
> isolation. It doesn't mention ECN, and doesn't mention that the L4S 
> network component needs to do some kind of shallow-threshold ECN 
> marking, both of which IMHO seem important enough to mention in this 
> section.

[BB] You're right.
I had just fixed exactly these omissions in response to Phil Eardley's 
review (probably with a memory of your review in my background memory).

>
> ---
> current draft text:
>    4.  L4S Architecture Components
>
> In section 4 on "L4S Architecture Components" I was expecting to see 
> (probably in section 4.2., Network Components) an explicit definition 
> of what L4S ECN marking behavior or semantics are allowed in (or 
> considered consistent with) the L4S architecture. Or, put another way, 
> in this doc it's not explicitly clear to me what an ECN mark "means" 
> in the L4S architecture. The draft text mentions queuing delay of 
> roughly 1ms several times: "Very low queuing latency means less than 1 
> millisecond (ms) on average and less than about 2 ms at the 99th 
> percentile."; "the additional queuing delay of roughly 1 ms"; "The 
> consequent smaller amplitude sawteeth fit between an empty queue and a 
> very shallow marking threshold (~1 ms in the public Internet)". 
> However, I didn't notice any text that specified that ECN marks must 
> or may happen when queuing delay reached the 1ms level. Nor did I see 
> text saying that bottlenecks may use any marking threshold they want, 
> or should use a threshold in some range, etc. Similarly I didn't see 
> any text that characterized whether randomized/smoothed marks were 
> required or permitted or disallowed, or whether step marking is 
> required or permitted or disallowed, and whether step marking may or 
> may not be based on sojourn time, queue length in octets, etc. Maybe I 
> missed these aspects. Or maybe this document is not where that is 
> intended to be defined; but in that case I still expected to see a 
> clear sentence pointing the reader to another document that would 
> define these aspects.

[BB] Defining the meaning of a mark is definitely the job of ecn-l4s-id 
(which it does relative to a drop):
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19#section-5.2

Regarding marking algos, I've added the following text at the end of the 
network bullet of the Architecture Overview (after it refers to the FQ 
and DualQ explanations and the DualQ draft):

"A specific marking algorithm is not mandated for L4S AQMs. Appendices 
of [I-D.ietf-tsvwg-aqm-dualq-coupled] give non-normative examples that 
have been implemented and evaluated, and give recommended default 
parameter settings. It is expected that L4S experiments will improve 
knowledge of parameter settings and whether the set of marking 
algorithms needs to be limited."

The following section in the DualQ draft gives some of what you're asking:
Normative Requirements for a DualQ Coupled AQM
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled#section-2.5

Most of the functional requirements are about the coupled marking.
In the management requirements it suggests certain parameters and their 
units.
But the primary approach is by example using existing implementations.

>
> ---
> current draft text:
>    c.  A network operator might wish to include certain unresponsive,
>        non-L4S traffic in the L4S queue if it is deemed to be smoothly
>        enough paced and low enough rate not to build a queue. ...
>        This traffic would need to be tagged with specific identifiers,
>        e.g. a low latency Diffserv Codepoint such as Expedited
>        Forwarding (EF [RFC3246]), Non-Queue-Building
>        (NQB [I-D.ietf-tsvwg-nqb]), or operator-specific identifiers.
>
> For tagging such traffic with specific identifiers, is marking as 
> ECT(1) also an option? That would seem to be more practical than DSCP 
> to carry end-to-end. But maybe that has gotchas. Either way it might 
> be nice to mention this option, if only to clarify why it's not a good 
> idea.

[BB] There's an extensive section in ecn-l4s-id all about this:
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id#section-5.4

A sender shouldn't tag a packet with ECT unless it is prepared to 
respond to congestion. If it will, then that is indeed going to be a 
worthwhile strategy.
But if it's unresponsive traffic, it should use one of the other 
identifiers.
In DOCSIS the marking of packets makes no difference to the queue 
protection algorithm.

The most relevant text to what you say is:

    A packet that carries one of these non-ECN identifiers to classify it
    into the L queue would not be subject to the L4S ECN marking
    treatment, unless it also carried an ECT(1) or CE codepoint.  The
    specification of an L4S AQM MUST define the behaviour for packets
    with unexpected combinations of codepoints, e.g. a non-ECN-based
    classifier for the L queue, but ECT(0) in the ECN field (for examples
    see section 2.5.1.1 of [I-D.ietf-tsvwg-aqm-dualq-coupled  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id#ref-I-D.ietf-tsvwg-aqm-dualq-coupled>]).

    For clarity, non-ECN identifiers, such as the examples itemized
    above, might be used by some network operators who believe they
    identify non-L4S traffic that would be safe to mix with L4S traffic.
    They are not alternative ways for a host to indicate that it is
    sending L4S packets.  Only the ECT(1) ECN codepoint indicates to a
    network element that a host is sending L4S packets (and CE indicates
    that it could have originated as ECT(1)).  Specifically ECT(1)
    indicates that the host claims its behaviour satisfies the
    prerequisite transport requirements inSection 4  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id#section-4>.

>
> ---
> current draft text:
>    Figure 1: Components of an L4S Solution:
>
> I'd suggest phrasing this in a way that explicitly mentions it's for a 
> DualQ bottleneck and differentiates it from the FQ-based L4S 
> bottleneck, since the diagram seems to only apply to DualQ:
>
>    Figure 1: Components of a DualQ L4S Solution:

[BB] Good point (my XML editor fits figure captions in a tiny box, which 
makes me prone to overlooking what they actually say).

>
> ---
> current draft text:
>        within each queue of an FQ-CoDel
>        system, as well as a CoDel AQM, there is typically also ECN
>        marking at an immediate (unsmoothed) shallow threshold to support
>        use in data centres
>
> Since the ce_threshold feature defaults to disabled in Linux, I'd 
> suggest wording this as:
>
>        within each queue of an FQ-CoDel
>        system, as well as a CoDel AQM, there is typically also
>        the option of ECN
>        marking at an immediate (unsmoothed) shallow threshold to support
>        use in data centres
>
> ---
> current draft text:
>    4.3.  Host Mechanisms
>
>    The L4S architecture includes two main mechanisms in the end host
>    that we enumerate next:
>
>    a.  Scalable Congestion Control at the sender: Data Center TCP is the
>        most widely used example.  It has been documented as an
>        informational record of the protocol currently in use in
>        controlled environments [RFC8257].
>
> Given this is the meat of an architecture document, I would suggest 
> starting 4.3.a with a high-level definition of what a scalable 
> congestion control is, then perhaps pointers to the docs giving a more 
> detailed definition, before mentioning that DCTCP is an example.

[BB] I've now put a definition of scalable CC in the host part of the 
Architecture Overview (taken from the Terminology section).
But I've also summarized it in the host component part. And I've 
referred to the full requirements of a scalable CC in §4.3 of ecn-l4s-id.

> ---
> current draft:
>
>   Figure 2: Likely location of DualQ (DQ) Deployments in common access
>                                   topologies
>
> In this figure it would be nice to include a depiction of a case where 
> there is a downstream DQ deployed in the wifi AP, since I imagine 
> there are many homes with a downstream DOCSIS link that is faster than 
> the wifi, so the downstream bottleneck is the wifi link.

[BB] I've added a sentence - getting too late to start ASCII art drawing.

Thanks for all these points. All very useful.

Bob

> ---
> best regards,
> neal

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/