Subject: Re: [tsvwg] explanation of option 3 choice

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 18 May 2020 08:01:58 +0000

Hi Tom

I limit my comments to the part around jitter and the viability of an L4S
solution and stick to the access that I understand reasonably well in my
reasoning (4G/5G access). I write here L4S as I am personally in favor of
L4S and see in the poll that option 1 has strong support.

3GPP systems have continuously reduced the delay jitter from 100s of
milliseconds in 2G (GPRS-EDGE) down to low 10s of milliseconds in 5G (NR)
with potential to go down to sub milliseconds jitter. 
So from that perspective I don't see network jitter as a show stopper for
L4S, network vendors will continuously work to reduce network jitter in 3GPP
systems with optimized power control, adjusted RLC timers etc, shorter
TTIs, URLLC... 
This work should however be done in parallel with L4S, L4S is an important
tool to reduce the network queue delay, besides queue delay there are other
sources of delay such as retransmission on MAC and (possibly RLC layer),
handover that is solved with other configuration work.

With regards to burstiness, my observation is that packet pacing is becoming
a more and more a best current practice, BBR implements it, can also mention
that SCReAM does it too. Packet pacing is IMHO a good too to reduce
burstiness in networks.

So, given the above reasoning. I don't see the reason why concerns over
network jitter should be an argument for option 3 (more testing).

Also.. L4S can actually be helpful in the cases where non-congestion related
jitter may obscure the queue delay. 
To take one example: SCReAM (FC8298) is designed to cope with the natural
delay jitter in LTE systems, up to quite high load levels, this is achieved
with an acceptance to this delay jitter in the delay sensitive part of the
algorithm. The downside is that one need to trade the sensitivity to
increased queue delay against robustness to delay jitter to get a high
throughput. The delay jitter in turn depends on the configuration of the
LTE/5G system which means that SCReAM had to be designed with something of a
worst case jitter. 
L4S is helpful here because the congested nodes can clearly indicate that
they are congested, after all, they know when queues are about to increase,
therefore endpoints do not need to infer congestion from implicit metrics
like delay and packet loss.
  
/Ingemar

> 
> Message: 4
> Date: Sun, 17 May 2020 10:06:46 -0700
> From: Tom Henderson <[email-masked]>
> To: tsvwg IETF list <[email-masked]>
> Subject: [tsvwg] explanation of option 3 choice
> Message-ID: <[email-masked]>
> Content-Type: text/plain; charset=utf-8; format=flowed
> 
> I support the L4S goals to deploy scalable congestion controls and achieve
low
> latency while maintaining high throughput when possible.  I believe that a
clear
> input signal to the network will be needed, so I lean towards ECT(1)
because it is
> the most likely to survive tunnel encapsulation and avoid bleaching.
However,
> the discussion has raised several performance and architectural concerns
that I
> believe that the WG should address, so I believe that option 3 best aligns
with
> my current view.
> 
> Selecting between L4S and SCE involves tradeoffs, but the decision seems
to be
> clouded by skepticism about the viability of deploying a low-latency
service even
> if we had enough bits to support two clear input signals and two output
signals,
> and space to carry these different signals back to the sender.  How robust
will
> the congestion controls be with a shallow threshold in the presence of
jitter or
> burstiness, which is likely for multiple access wireless networks?  How
low can
> the marking threshold be configured, and what are the tradeoffs (and how
does
> the performance degrade when assumptions are violated)?  For a given
latency
> threshold, what limits on RTT, burstiness, or bottleneck link rate may
exist, and
> should the congestion controllers try to detect those conditions with
heuristics?
> If one were to try to design policers to prevent abuse, what would be the
> conformant flow behavior to police, and what if flows cannot be
distinguished
> due to tunneling, or if in-network aggregation makes the flow
non-conformant?
> 
> Assuming that the long-term issues inherent with a low-threshold ECN mark
can
> be worked out even in the presence of enough signaling bits, then both the
L4S
> and SCE proposals present further questions.  For L4S, aside from RFC 3168
> detection, how does the lack of an overload signal (other than packet
loss)
> inhibit the design of a congestion controller, and how does slow start to
> congestion avoidance transition occur?  For SCE, the reliance on
DSCP-based
> classification will cause legacy/classic and low latency flows to end up
more
> often in the same queue, and SCE-enabled flows could end up with worse
> performance than legacy/classic flows.  Can SCE results be presented that
show
> the performance of competing flows (legacy/classic and SCE) in shared
queues,
> but with the SCE classification disabled?
> 
> My suggestion to move forward is for the WG to better define the desired
> properties, behavior, and scope of low-latency ECN semantics assuming (in
the
> long run) two inputs and two outputs, and to expand the evaluation
framework
> to take into account the suggestions that others have made to include
short
> flows, bidirectional traffic, asymmetry, jitter and network aggregation,
and
> tunnels.  Constraints introduced by SCE and L4S could also be evaluated in
the
> same framework.  If the WG were to reach consensus that, with enough bits,
> the objective design works well enough, but the constraints do not, then
> perhaps more effort could be devoted to finding more bits.  Jake Holland
made
> one such proposal (ECT(1) to ECT(0), plus existing CE).  If low latency
service is
> within reach but is blocked by some existing decapsulation rules that
could be
> revised, why shouldn?t the WG try for that?  Appendix B of the AccECN
draft
> hypothesizes that there may be creative ways to send both a low-level
> congestion and overload signal back to the sender.
> 
> - Tom
> 
> 
> 
> ------------------------------
> 
> Message: 5
> Date: Sun, 17 May 2020 10:52:19 -0700
> From: Dave Taht <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Subject: Re: [tsvwg] Consensus call on ECT(1)
> Message-ID:
> 	<CAA93jw7BEkZuSSaU90nT7AizUc-
> vsC1Fp=[email-masked]>
> Content-Type: text/plain; charset="UTF-8"
> 
> If I had to make a choice, today, I would vote for option 2.
> ECT(1) as a network output. The present conception of SCE "fails safe"
> in far more circumstances than the present conception of L4S appears to.
> 
> The internet as a whole needs to retain and improve upon, the existing
strong
> signal of congestion that the RFC3168 compliant CE indicator brings to it,
and
> furthermore, more fully deploy the existing IETF AQMs in more places where
it
> matters.
> 
> However, I vote for option 3, with a rationale to follow. (The chair's
question
> also wanted a list of tests, but that got too long for this vote, so I am
moving
> that to the
> rationale)
> 
> --
> "For a successful technology, reality must take precedence over public
relations,
> for Mother Nature cannot be fooled" - Richard Feynman
> 
> [email-masked] <Dave T?ht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> 
> 
> 
> ------------------------------
> 
> Message: 6
> Date: Sun, 17 May 2020 11:14:42 -0700
> From: Dave Taht <[email-masked]>
> To: tsvwg IETF list <[email-masked]>
> Subject: [tsvwg] dave taht's rationale for option 3 on ect(1) (more
> 	testing)
> Message-ID:
> 	<CAA93jw5HHY6xA1NOxrqad-
> [email-masked]>
> Content-Type: text/plain; charset="UTF-8"
> 
> The codebases, simulations, etc, for the l4s and SCE approaches are still
in a very
> raw state. With these early implementations, many problems have been
> identified, and only a few resolved. The internet drafts and codebases are
out of
> alignment, with new bugs landing in the codebases and operating systems
every
> time someone takes them out for a spin.
> 
> Nearly all the tests bufferbloat.net considered basic in the development
of the
> original pie, codel, and fq_codel aqms in the aqm working group as of 2004
have
> yet to be run.
> 
> Participants may weight the results differently, but adversarial tests of
all kinds
> need to be run, regularly, in response to every change to the
specification and
> code, in order to retain an accurate picture of the problems found, fixed,
or
> ignored.
> 
> (I confess that much of my interest in lossless congestion control comes
from
> longer  RTT near earth orbit, geo, and gto - and even more distant
space-born
> applications,  which is a from-orbit viewpoint I suspect few here share)
> 
> To the greatest extent possible, tests to be performed in simulation AND
on a
> variety of real hardware, AND in virtualized environments.
> 
> The recent bugs found in encapsulations need to be evaluated against more
> tunnel types and operating systems. More shipping switches and hardware
> based need to be tested and configured using ect(1) -> CE transitions, in
> particular, to see what happens.
> 
> Problems with legacy uses of the IP stack, for private and industrial and
> embedded r/t control use, have not been explored either. Examples include
> 6lowpan, and other relatively unknown services in the ietf you can find in
all the
> other ethernet packet types known to man.
> 
> The potential side effects on classic traffic, have been barely explored,
if at all.
> The impact on conventional udp and classic gaming, voice, and DNS traffic
of
> the pie side of the l4s solution, (with the initial exploration of the
rrul_be test
> showing up bugs galore), is largely unknown.
> 
> The true effects on the internet of senders such as the increasingly
available
> BBRv1 with RFC3168 ecn accidentally enabled deployment needs analysis, as
do
> the effects of a modestly or truly lying senders.
> 
> The costs and methods required to mitigate ECN-enabled DDOS attacks, also
is
> absent from analysis.
> 
> The side effects of essentially deprioritizing classic routing protocols
and other
> essential oam and maintenance packet types (such as rip v2 (used on
cable),
> ospf, Babel, ISIS, RA, ND and ARP), needs to be explored.
> 
> The effects on existing deployed RFC3168 compliant multicast file
transports
> such as uftp[1], needs to be tested.
> 
> Recognising ?loss and mark? as an even stronger signal than either, not
thought
> about.
> There's a long list of other more trivial things - dynamically reducing
mss size at
> cwnd 2 or less, improving packet pacing to cope with sub cwnds, more fully
> exploring l4s-style behaviors on unreliable, jittery networks such as wifi
and lte,
> and chirping.
> 
> In short:
> 
> I see no reason for the IVTF to allocate the ECT(1) codepoint at this time
without
> a ton more simulation and testing.
> 
> With a lot more robust, repeatable testing, good software engineering
> practices, complete with regression tests, and continuous integration, and
> source code that matches the internet drafts (and vice versa), perhaps we
can all
> find a way forward closer to the goals of low latency and high throughput
and
> the most viable, safest, use for this last half a bit in the IP header.
> 
> 
> --
> "For a successful technology, reality must take precedence over public
relations,
> for Mother Nature cannot be fooled" - Richard Feynman
> 
> [email-masked] <Dave T?ht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> 
> 
> 
> End of tsvwg Digest, Vol 193, Issue 15
> **************************************

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 18 May 2020 08:42:25 -0700 (PDT)

Hello Ingemar,
	Some comments inline, some text reflowed to couple response
with original statement.  [rwg]

Regards,
Rod

> systems with optimized power control, adjusted RLC timers etc, shorter

 [RWG] Have you looked closely at the data presented on what happens to L4S
when presented with Jitter?  One can not deploy something today based
on what "may" become the future, I suspect doing so would create enough
pain for operators that L4S would end up bleached.

> that SCReAM does it too. Packet pacing is IMHO a good too to reduce

 [RWG] Again, depending on something coming in the future that may fix the
shortcomings is not a robost plan.  Further, IMHO, pacing is not
going to do all that some feel it may.  It is a good idea, and it
shall help some things, but it seems as if pacing proponents are
ignoring that packet origination timing is only one very small
source of burstiness in the network.  I do not see pacing fixing
the issue that occurs as you blend traffic at aggrigation points
as it travels towards the core of the internet, and even less
effective as that traffic deaggregates coming out of the core.

> network jitter should be an argument for option 3 (more testing).

 [RWG] So lets not test and analyze these effects, simply use our crystal
ball to guess it might be ok?  It has already been demonstrated that the
current L4S algorithms fail misserable with respect to the current
Jitter and Burstiness of the internet.

> they are congested, after all, they know when queues are about to increase,

Now your saying L4S can predict the future?  Would you care to explain how
L4S can predict queue growth?

-- 
Rod Grimes                                                 [email-masked]

---

From: Tom Henderson <[email-masked]>
Date: Mon, 18 May 2020 08:58:24 -0700

> network jitter should be an argument for option 3 (more testing).

Hi Ingemar, thank you for the response.  I listed jitter as a concern 
mainly because of my familiarity with Wi-Fi and the tension that exists 
between trying to obtain larger TXOPs to achieve higher data rate, at 
the expense of access time variability.  As one tries to increase Wi-Fi 
throughput through larger TXOPs, burstiness will increase, leading to 
bursts arriving at a shallow threshold, and possibly working against the 
congestion control which will reduce rate in response to marks.  This is 
a guess; it would be nice to back it up with some test results.  Perhaps 
the scheduling capabilities in newer Wi-Fi standards will alleviate 
this.  It may be that the system behavior is such that, if the threshold 
is set too low in relation to the jitter, it preserves latency at the 
expense of end-to-end throughput, but it would be nice to confirm this 
property.

As to how this relates to selecting option 1 vs. 2 at this time, it is 
perhaps not the main issue in that decision, but just one of the items I 
listed in response to option 3 requesting a list of desired tests.  My 
selection of option 3 is not a suggestion to change the current WG 
position on ECT(1) as input, but more to signal a general level of 
concern that I have about needing to understand the system better and 
resolving the reported concerns with the current drafts before taking 
decisions.

- Tom

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 18 May 2020 18:32:25 +0000

Hi

Please see inline [IJ]

/Ingemar

> -----Original Message-----
> From: Rodney W. Grimes <[email-masked]>
> Sent: den 18 maj 2020 17:42
> To: Ingemar Johansson S <[email-masked]>
> Cc: [email-masked]; Ingemar Johansson S
> <[email-masked]>
> Subject: Re: [tsvwg] explanation of option 3 choice
> 
> Hello Ingemar,
> 	Some comments inline, some text reflowed to couple response with
> original statement.  [rwg]
> 
> Regards,
> Rod
> 
> > Hi Tom
> >
> > I limit my comments to the part around jitter and the viability of an
> > L4S solution and stick to the access that I understand reasonably well
> > in my reasoning (4G/5G access). I write here L4S as I am personally in
> > favor of L4S and see in the poll that option 1 has strong support.
> >
> > milliseconds in 2G (GPRS-EDGE) down to low 10s of milliseconds in 5G
> > (NR) with potential to go down to sub milliseconds jitter.
> > So from that perspective I don't see network jitter as a show stopper
> > for L4S, network vendors will continuously work to reduce network
> > jitter in 3GPP systems with optimized power control, adjusted RLC
> > timers etc, shorter TTIs, URLLC...
> 
> when presented with Jitter?  One can not deploy something today based on
> what "may" become the future, I suspect doing so would create enough pain
for
> operators that L4S would end up bleached.

[IJ] Yes, but I see that as room for improvement and not as an indicator
that the L4S experiment should be halted.

> 
> > This work should however be done in parallel with L4S, L4S is an
> > important tool to reduce the network queue delay, besides queue delay
> > there are other sources of delay such as retransmission on MAC and
> > (possibly RLC layer), handover that is solved with other configuration
work.
> >
> > With regards to burstiness, my observation is that packet pacing is
> > becoming a more and more a best current practice, BBR implements it,
> > can also mention that SCReAM does it too. Packet pacing is IMHO a good
> > too to reduce burstiness in networks.
> 
> shortcomings is not a robost plan.  Further, IMHO, pacing is not going to
do all
> that some feel it may.  It is a good idea, and it shall help some things,
but it
> seems as if pacing proponents are ignoring that packet origination timing
is only
> one very small source of burstiness in the network.  I do not see pacing
fixing the
> issue that occurs as you blend traffic at aggrigation points as it travels
towards
> the core of the internet, and even less effective as that traffic
deaggregates
> coming out of the core.

[IJ] It does not seem like a fruitful plan to wait for the future either,
what I can observe is that delay jitter is going down at least in 3GPP
access and the drive towards that becomes stronger with the introduction of
services that require low latency, what implies that delay jitter will be an
unsolvable problem in other access  types ? 
I find it quite meaningless to take the flaws in today's networks as an
argument against L4S. 

> 
> >
> > So, given the above reasoning. I don't see the reason why concerns
> > over network jitter should be an argument for option 3 (more testing).
> 
ball to
> guess it might be ok?  It has already been demonstrated that the current
L4S
> algorithms fail misserable with respect to the current Jitter and
Burstiness of the
> internet.

[IJ] That is you interpretation. 
What I mean is that testing and improvement is of course needed over time
and this will be done. But as said above, I don't see the rationale why we
should wait for the future to happen before we continue with the L4S
experiments, we can do L4S and network optimization in parallel. Otherwise
it becomes quite likely that we'll land a man on one of Jupiter´s moons
before we have agreed to do something decent with the ECN bits.

> 
> > Also.. L4S can actually be helpful in the cases where non-congestion
> > related jitter may obscure the queue delay.
> > To take one example: SCReAM (FC8298) is designed to cope with the
> > natural delay jitter in LTE systems, up to quite high load levels,
> > this is achieved with an acceptance to this delay jitter in the delay
> > sensitive part of the algorithm. The downside is that one need to
> > trade the sensitivity to increased queue delay against robustness to
> > delay jitter to get a high throughput. The delay jitter in turn
> > depends on the configuration of the LTE/5G system which means that
> > SCReAM had to be designed with something of a worst case jitter.
> > L4S is helpful here because the congested nodes can clearly indicate
> > that they are congested, after all, they know when queues are about to
> > increase,
> 
L4S
> can predict queue growth?

[IJ] I'll put is short. Given that the L4S marking is done in the eNBs and
gNBs it becomes possible to look at other metrics than queue delay alone.
You can call it prediction of the future if you prefer. 

> 
> > therefore endpoints do not need to infer congestion from implicit
> > metrics like delay and packet loss.
> >
> > /Ingemar
> >
> >
> > >
> > > Message: 4
> > > If one were to try to design policers to prevent abuse, what would
> > > be the conformant flow behavior to police, and what if flows cannot
> > > be
> > distinguished
> > > and SCE proposals present further questions.  For L4S, aside from
> > > RFC 3168 detection, how does the lack of an overload signal (other
> > > than packet
> > loss)
> > > inhibit the design of a congestion controller, and how does slow
> > > start to congestion avoidance transition occur?  For SCE, the
> > > reliance on
> > DSCP-based
> > > often in the same queue, and SCE-enabled flows could end up with
> > > worse performance than legacy/classic flows.  Can SCE results be
> > > presented that
> > show
> > > My suggestion to move forward is for the WG to better define the
> > > desired properties, behavior, and scope of low-latency ECN semantics
> > > assuming (in
> > the
> > > same framework.  If the WG were to reach consensus that, with enough
> > > bits, the objective design works well enough, but the constraints do
> > > not, then perhaps more effort could be devoted to finding more bits.
> > > Jake Holland
> > made
> > > hypothesizes that there may be creative ways to send both a
> > > low-level congestion and overload signal back to the sender.
> > >
> > > - Tom
> > >
> > >
> > >
> > > The recent bugs found in encapsulations need to be evaluated against
> > > more tunnel types and operating systems. More shipping switches and
> > > hardware based need to be tested and configured using ect(1) -> CE
> > > transitions, in particular, to see what happens.
> > >
> > > Problems with legacy uses of the IP stack, for private and
> > > industrial and embedded r/t control use, have not been explored
> > > either. Examples include 6lowpan, and other relatively unknown
> > > services in the ietf you can find in
> > all the
> > > cwnd 2 or less, improving packet pacing to cope with sub cwnds, more
> > > fully exploring l4s-style behaviors on unreliable, jittery networks
> > > such as wifi
> > and lte,
> > > and chirping.
> > >
> > > In short:
> > >
> > > With a lot more robust, repeatable testing, good software
> > > engineering practices, complete with regression tests, and
> > > continuous integration, and source code that matches the internet
> > > drafts (and vice versa), perhaps we
> > can all

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 18 May 2020 18:40:01 +0000

Hi
Please see inline

/Ingemar

> Sent: den 18 maj 2020 17:58
> To: Ingemar Johansson S <[email-masked]>;
> [email-masked]
work.
> Hi Ingemar, thank you for the response.  I listed jitter as a concern
mainly
> because of my familiarity with Wi-Fi and the tension that exists between
trying
> to obtain larger TXOPs to achieve higher data rate, at the expense of
access
> time variability.  As one tries to increase Wi-Fi throughput through
larger TXOPs,
> burstiness will increase, leading to bursts arriving at a shallow
threshold, and
> possibly working against the congestion control which will reduce rate in
> response to marks.  This is a guess; it would be nice to back it up with
some test
> results.  Perhaps the scheduling capabilities in newer Wi-Fi standards
will
> alleviate this.  It may be that the system behavior is such that, if the
threshold is
> set too low in relation to the jitter, it preserves latency at the expense
of end-to-
> end throughput, but it would be nice to confirm this property.

[IJ] Personally I have poor understanding of WiFi so I cannot say for
certain what this community does to combat delay jitter. I do however guess
that WiFi faces the same desire for low latency as 3GPP does.

> As to how this relates to selecting option 1 vs. 2 at this time, it is
perhaps not the
> main issue in that decision, but just one of the items I listed in
response to
> option 3 requesting a list of desired tests.  My selection of option 3 is
not a
> suggestion to change the current WG position on ECT(1) as input, but more
to
> signal a general level of concern that I have about needing to understand
the
> system better and resolving the reported concerns with the current drafts
before
> taking decisions.

[IJ] It is fair to raise concerns around delay jitter and the impact on L4S
performance, I don't however believe that IETF can solve all these issues.
For instance delay jitter issues in 4G/5G, that are not only related to L4S,
have to be solved in 3GPP, possibly in liaison with IETF. That are the
standards related matters, then we have the implementation aspects and that
is a story on its own.