Subject: Re: Unified ACK frame encoding

From: "Gorry (erg)" <[email-masked]>
Date: Sun, 2 Sep 2018 13:27:44 +0100

See below 

> On 31 Aug 2018, at 17:18, Mirja Kühlewind <[email-masked]> wrote:
> 
> Hi!
> 
> On 28.08.2018 15:58, Kazuho Oku wrote:
>>> I think usage of a single ECT code point are going to work fine. The
>>> reason has to do with how the ECT(0) and ECT(1) codepoints are defined
>>> for the experimental usage. In legacy the two ECT codepoint are the
>>> same, but by default a QUIC stack will mark only with ECT(0). Thus,
>>> reception of ECT(1) means a non-compliant path that does remarking that
>>> should not occur. ECT(1) will be used for alternative behaviours,
>>> including different ECN marking schemes such as in L4S. For a  QUIC
>>> endpoint to use ECT(1) as marking when sending it needs to have
>>> awareness of the experiment and implement the correct response
>>> functions. It also needs to agree with the receiver about this
>>> capability. Although possible it could be limited to that the receiver
>>> only needs to know that ECT marking will be ECT(1) and that should be
>>> reported. However it might imply additional receiver behaviour changes.
>>> Thus, I fully expect that any such experiment in QUIC will require
>>> either a version or a Transport Parameter.
> 
>> To clarify, my point is that if we could have an ACK frame that can
>> transmit all 5 modes (4 ECN types + gaps) precisely, the sender of the
>> packets and the middleboxes can use the ECN flags in the way they
>> want, without requiring an update on the receiver.
> 
> Yes, I think it is important to have a feedback scheme that would make it possible to experiment with the use of ECT(1) without changing the receiver. So if we don't find a way to send separate feedback for ECT(1), I would rather encode ECT(0) and ECT(1) with the same type (instead of ECT(1) and Not-ECT).
> No matter what, if you don't send ECT(1) feedback separately, you will not be able to detect path mangling which either changes ECT(0) to ECT(1) or the other way around.
> 
Yes - that is important. RFC4774 does allow many behaviours, and senders do need feedback to understand the network marking behaviour - or probably more accurately: to know that the received ECN  code points are consistent with their expectations.

> Note that while ECT(0) and ECT(1) should not (anymore) be mixed up in the same connection

Although that is still permitted by RFCs when alternate semantics are identified by the DSCP.

> , it can still be a valid case to send some of your packets as Not-ECT (also probably not recommended for QUIC but e.g. TCP does mark control packets and retransmission as Not-ECT).

I think this should be allowed. And I really think there should at least be summary information reporting the number of each ECN codepoint seen - and at least accurate feedback for any CE mark.

Gorry 

> Mirja
> <mirja_kuehlewind.vcf>

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 4 Sep 2018 10:36:08 +0900

2018年8月29日(水) 3:47 Christian Huitema <[email-masked]>:
>
> On 8/28/2018 7:27 AM, Eric Rescorla wrote:
>
> >
> > Correction: Ian pointed out to me that because of the two varint bits,
> > we actually pay an extra byte whenever the range length would fit in 6
> > bits, but not 4, which is a lot worse, because lengths in the range
> > 17-64 are going to be quite common.
> >
> > -Ekr
>
> There is another cost, but it is only borne if the implementation
> actually supports ECN. Suppose the node receives a string of packets
> with alternating CE marking, something like
> ECT(1)-ECT(1)-CE-ECT(1)-ECT(1)-CE-ECT(1)-ECT(1)-CE, etc. This is not a
> theoretical case: in DC-TCP, equilibrium is reached when the queue size
> at the bottleneck hovers around the low water mark, and the algorithm is
> specifically designed to produced frequent feedback. Now, if you have
> that, you end up encoding a series of short ranges in the ACK, which
> makes it way longer than just saying "I received everything but 3
> packets were marked as CE".
>
> This inflates not just the size of the ACKs, but also the number of
> ranges in the "SACK dashboard" that nodes have to keep to manage the
> sending of ACKs. When packets are received out of order, ranges can
> normally be coalesced, but with ECN they can only be coalesced if all
> packets had the same ECN marking.
>
> So there is indeed a cost. I also have some sympathy for EKR's argument
> about future extensions. ECN is just one of the signals that peers may
> want to use to manage congestion and recovery. If nodes want to make
> thoughtful decisions, they will certainly also look at RTT or one way
> delays, and maybe at out-of-order deliveries. Would we want to use bits
> in the "gap-range-length" to indicate that for packet number 73 the one
> way delay was longer than usual, or would we use some other extension
> mechanism? And if we have another extension mechanism, why not use it
> for ECN as well?

My preference goes to either concentrating on the ordinary use of ECN,
or adopting an encoding that can encode *any* use-case of the ECN
flags. I do not think that we should optimize for a particular
non-standard use-case of ECN flags in QUICv1.

#1706 concentrates on the ordinary use of ECN, and I think that it is
in good balance.

If we are to choose an encoding that can encode any type of use, we
cannot lose any signal (e.g., by "counting" the flags associated to
each packet). The simple solution would therefore be to use a variant
of #1706 that uses 5-mode encoding compressed into 2 bits.

I agree that the 5-mode encoding is not optimal in some cases. But I
do not think that the wire overhead or the size of the SACK dashboard
will be a deal blocker.

> -- Christian Huitema

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 4 Sep 2018 08:42:38 +0200

Any feedback on the encoding I proposed a few mails back where you get full
information at the expense of alternating ACK and GAP, and without encoding
the length variable with special bits?

Mikkel

On 4 September 2018 at 03.36.40, Kazuho Oku ([email-masked]) wrote:

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Tue, 4 Sep 2018 17:46:50 +1000

You would need to encode a 0-length gap, which is unfortunate, but not
fatal.  The mode switching would be much more expensive.  It's
basically the same as the suggestion ekr made, with the hack being
that you use the very long integer values (the 62-bit varint encoding)
to encode the type.  That means you need special parsing logic to
catch the markers and not route to a varint decoder.
On Tue, Sep 4, 2018 at 4:42 PM Mikkel Fahnøe Jørgensen
<[email-masked]> wrote:
>
> Any feedback on the encoding I proposed a few mails back where you get full information at the expense of alternating ACK and GAP, and without encoding the length variable with special bits?
>
> Mikkel
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 4 Sep 2018 10:10:04 +0200

A 0 length gap should not be necessary, although it would be valid.
This is because the type prefix overrides the presence of 0 gaps -
otherwise alternating CE and ECT would be expensive.
That aside, I do agree that it is a tradeoff that is only worthwhile if you
want full information.

On 4 September 2018 at 09.47.03, Martin Thomson ([email-masked])
wrote:

<[email-masked]> wrote:
>
> Any feedback on the encoding I proposed a few mails back where you get
full information at the expense of alternating ACK and GAP, and without
encoding the length variable with special bits?

---

From: Christian Huitema <[email-masked]>
Date: Tue, 4 Sep 2018 12:38:32 -0700

> #1706 concentrates on the ordinary use of ECN, and I think that it is

I guess that's where I disagree. #1706 looks efficient because it
concentrates on the "standard" use of ECN, in which ECN-CE marks as
substitute for packet losses, and thus not expected to happen very
often. But that same property makes ECN a poor feedback mechanism,
especially for high speed transmission. Systems that want efficient
congestion control at high speed need feedback at least once per RTT, if
not more often. In practice today, they get that feedback by using
delay-based algorithms, because delay samples can be obtained at every
ACK -- BBR is an example of such algorithms.

If you want to use ECN marks for efficient feedback, you need to
redefine the marking strategy so that marks happen many times per RTT --
much like DCTCP does, or better yet Bob Briscoe's proposals.

> do not think that the wire overhead or the size of the SACK dashboard

I am concerned that we are optimizing for a specific property, "knowing
which packet was marked in what way", when there is no supporting
evidence that this property is useful. We have two documented ECN based
congestion algorithms, the standard one and the DCTCP experiment, and
none of them uses that property.

Standard TCP currently reports standard ECN with one bit of feedback per
RTT per RFC 3168: a CE flag, cleared when the CWR flag is received from
the peer. It does not provide the any indication about which packet was
affected. So if our goal is to support standard ECN in QUIC, I would
argue that we only need to define a way to convey the CE and CWR flags.
We can do that without redefining the format of ranges.

Non-standard TCP such as DCTCP (RFC 8257) also uses the CE bit, but
changes the behavior. The sender estimates the congestion state by
competing a running average of the fraction of data acknowledged by ACKs
in which the CE bit is marked. DCTCP does not use the CWR bit, but sets
it in accordance with RFC 3168 for compatibility.

I feel that #1706 is a premature optimization. It works fine if the
network uses standard ECN, but is overkill there because TCP-similar
RENO would just need a single bit. It causes a significant overhead if
the network experiments with non standard ECN as in DCTCP, when in fact
the DCTCP congestion algorithm only requires on bit of signalling per
ACK, the same as standard TCP.

-- Christian Huitema

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 5 Sep 2018 15:44:05 +0200

Hi,

I think Christian has a valid point about what the design goals of the 
optimization is. From my perspective I am considering several factors 
and try to consider the trade-off between them. I think there is a 
benefit of a unified reporting form that include ECN information. I also 
want it to be usable with L4S and DCTCP style usages. I expect we will 
see QUICv1 based implementation with sender side only congestion control 
optimizations to such environments. Because of this I do see a point of 
doing an optimization that works reasonable for all cases, no ECN, 
legacy ECN and "new" ECN.

So to support DCTCP and L4S using TCP what has been proposed there. 
There is a requirements document: 
https://datatracker.ietf.org/doc/rfc7560/ and a experimental proposal 
for accurate ECN reporting: 
https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ the later 
is actually a byte counter based proposal. That combines an TCP option 
that reports the least 24-bytes of the counters for ECT(0), ECT(1) and 
CE. The number of CE marked packets are also reported using a three bit 
field in the TCP header. To make this somewhat robust it requires that a 
receiver send one ACK one every 2 (SHOULD) to 6 (Max) increments of the 
CE packet counter.

The requirements document states that preferably one should have byte 
accounting on what is marked. Regarding accuracy in section 4:

       In the best case, a sender that sees more accurate ECN feedback
       information would be able to reconstruct the occurrence of any of
       the four codepoints (Not-ECT, CE, ECT(0), ECT(1)).  However,
       assuming the sender marks all data packets as ECN-capable and uses
       a default setting of ECT(0) (as with [RFC3168]), solely feeding
       back the occurrence of CE and ECT(1) might be sufficient.  Because
       the sender can keep account of the transmitted segments with any
       of the three ECN codepoints, conveying any two of these back to
       the sender is sufficient for it to reconstruct the third as
       observed by the receiver.  Thus, a more accurate ECN feedback
       scheme should at least provide information on two of these
       signals, e.g., CE and ECT(1).

The current packet counters do not provide byte accurate accounting of 
the different markings, but it provides a clear proportional marking 
signal that is very useful for DCTCP or L4S style usages.

The Unified ACKing proposal enables byte accurate accounting. However, 
using a one byte-ack per change results in that when the frequency of 
the changes between ECT and CE gets higher than 12 (common case) in the 
ACK vector coverage they are less efficient. The ACK vector coverage 
will, if reliable ACKing is used, cover at least 1 RTT of in flight 
packets and likely will contain around 1.5 RTT worths of packet.

If I interpret the marking probabilities correctly in this paper for L4S 
http://www.bobbriscoe.net/projects/latency/dctth_preprint.pdf with dual 
queue. A single flow filling the bottleneck using L4S would see a bit 
more than 10% marking probability. If 10 L4S TCP flows compete with 10 
Cubic flows, the L4S flow will see more than 80% of its packets marked, 
while the cubic flow will see around 1% drop rates in its queue. Thus 
evaluating overhead considerations for L4S cases might actually need to 
include the pattern that every second packet is CE marked as worst case 
scenario. Even at 10% marking rate the bandwidth delay products where 
counters are more efficient are actually quite low. So lets assume 4 
bytes counters per type, then 12 changes to give equal overhead, means 
BDP of less than 120 packets in a good case, and 24 packets in worst 
case. Looking at the BDP figures I published in the discussion:
https://github.com/quicwg/base-drafts/issues/1439

The worst case is around 250 mbps at 1 ms RTT and 5 mbps on 50 ms. The 
good case is a bit over 1Gbps on a 1 ms RTT and just under 25 mbps on 50 
ms RTT. The conclusion one can draw is that no DCTCP or L4S use cases at 
bitrates that can be expected to exist both inside a data center 
(1Gbps+) or for more demanding applications across actual access 
networks can be efficiently supported using per packet explicit marking. 
In fact the ACK blocks will blow through the MTU in high rate applications.

I think if one want a unified ACK vector, then maybe one should look at 
making the counters a bit shorter by only reporting the lowest part of 
the counters. Where the receiver chooses if it uses the 6-bits, 14-bits, 
30-bits or 62-bits worth of counter based on how quickly it is changing. 
I would assume that if one has more bits than what wraps in a couple of 
RTTs this will be robust. Even without unified this could be good to 
reduce the overhead for long lived flows and making it only dependent on 
the amount of in-flight packets.

Cheers

Magnus

Den 2018-09-04 kl. 21:38, skrev Christian Huitema:
>> or adopting an encoding that can encode*any*  use-case of the ECN
> congestion control at high speed need feedback at least once per RTT, 
> if not more often. In practice today, they get that feedback by using 
> redefine the marking strategy so that marks happen many times per RTT 
> -- much like DCTCP does, or better yet Bob Briscoe's proposals.
>
>
> I am concerned that we are optimizing for a specific property, 
> "knowing which packet was marked in what way", when there is no 
> supporting evidence that this property is useful. We have two 
> documented ECN based congestion algorithms, the standard one and the 
> DCTCP experiment, and none of them uses that property.
>
> Standard TCP currently reports standard ECN with one bit of feedback 
> per RTT per RFC 3168: a CE flag, cleared when the CWR flag is received 
> from the peer. It does not provide the any indication about which 
> packet was affected. So if our goal is to support standard ECN in 
> QUIC, I would argue that we only need to define a way to convey the CE 
> and CWR flags. We can do that without redefining the format of ranges.
>
> competing a running average of the fraction of data acknowledged by 
> ACKs in which the CE bit is marked. DCTCP does not use the CWR bit, 
> but sets it in accordance with RFC 3168 for compatibility.
>
> the network experiments with non standard ECN as in DCTCP, when in 
> fact the DCTCP congestion algorithm only requires on bit of signalling 
> per ACK, the same as standard TCP.
>
> -- Christian Huitema

-- 

Magnus Westerlund

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Sep 2018 10:02:55 -0400

On Wed, Sep 5, 2018 at 9:44 AM Magnus Westerlund <
[email-masked]> wrote:

> optimization is. From my perspective I am considering several factors and
> try to consider the trade-off between them. I think there is a benefit of a
> unified reporting form that include ECN information. I also want it to be
> usable with L4S and DCTCP style usages. I expect we will see QUICv1 based
> implementation with sender side only congestion control optimizations to
> such environments. Because of this I do see a point of doing an
> optimization that works reasonable for all cases, no ECN, legacy ECN and
> "new" ECN.
>
> So to support DCTCP and L4S using TCP what has been proposed there. There
> is a requirements document: https://datatracker.ietf.org/doc/rfc7560/ and
> a experimental proposal for accurate ECN reporting:
> is actually a byte counter based proposal. That combines an TCP option that
> reports the least 24-bytes of the counters for ECT(0), ECT(1) and CE. The
> number of CE marked packets are also reported using a three bit field in
> the TCP header. To make this somewhat robust it requires that a receiver
> send one ACK one every 2 (SHOULD) to 6 (Max) increments of the CE packet
> counter.
>
> The current packet counters do not provide byte accurate accounting of the
> different markings, but it provides a clear proportional marking signal
> that is very useful for DCTCP or L4S style usages.
>

I actually missed this, I was thinking these counters were in bytes, not
packets.  If all we want to do is fix this, it's fairly straightforward.

> using a one byte-ack per change results in that when the frequency of the
> changes between ECT and CE gets higher than 12 (common case) in the ACK
> vector coverage they are less efficient. The ACK vector coverage will, if
> reliable ACKing is used, cover at least 1 RTT of in flight packets and
> likely will contain around 1.5 RTT worths of packet.
>
> queue. A single flow filling the bottleneck using L4S would see a bit more
> than 10% marking probability. If 10 L4S TCP flows compete with 10 Cubic
> flows, the L4S flow will see more than 80% of its packets marked, while the
> cubic flow will see around 1% drop rates in its queue. Thus evaluating
> overhead considerations for L4S cases might actually need to include the
> pattern that every second packet is CE marked as worst case scenario. Even
> at 10% marking rate the bandwidth delay products where counters are more
> efficient are actually quite low. So lets assume 4 bytes counters per type,
> then 12 changes to give equal overhead, means BDP of less than 120 packets
> in a good case, and 24 packets in worst case. Looking at the BDP figures I
> published in the discussion:

To me, this sounds like a pretty bad situation for L4S.  The ACK block
format is optimized for typical loss patterns, which are typically bursty
even when the loss rate is high, but I can imagine L4S creating much more
even marking, and hence getting very inefficient encoding.

> good case is a bit over 1Gbps on a 1 ms RTT and just under 25 mbps on 50 ms
> RTT. The conclusion one can draw is that no DCTCP or L4S use cases at
> bitrates that can be expected to exist both inside a data center (1Gbps+)
> or for more demanding applications across actual access networks can be
> efficiently supported using per packet explicit marking. In fact the ACK
> blocks will blow through the MTU in high rate applications.
>

> making the counters a bit shorter by only reporting the lowest part of the
> counters. Where the receiver chooses if it uses the 6-bits, 14-bits,
> 30-bits or 62-bits worth of counter based on how quickly it is changing. I
> would assume that if one has more bits than what wraps in a couple of RTTs
> this will be robust. Even without unified this could be good to reduce the
> overhead for long lived flows and making it only dependent on the amount of
> in-flight packets.
>

This seems like a potentially worthwhile optimization, especially if you
move to byte based counting.

> or adopting an encoding that can encode **any** use-case of the ECN
> substitute for packet losses, and thus not expected to happen very often.
> But that same property makes ECN a poor feedback mechanism, especially for
> high speed transmission. Systems that want efficient congestion control at
> high speed need feedback at least once per RTT, if not more often. In
> practice today, they get that feedback by using delay-based algorithms,
> because delay samples can be obtained at every ACK -- BBR is an example of
> such algorithms.
>
> If you want to use ECN marks for efficient feedback, you need to redefine
> the marking strategy so that marks happen many times per RTT -- much like
> DCTCP does, or better yet Bob Briscoe's proposals.
>
>
> which packet was marked in what way", when there is no supporting evidence
> that this property is useful. We have two documented ECN based congestion
> algorithms, the standard one and the DCTCP experiment, and none of them
> uses that property.
>
> RTT per RFC 3168: a CE flag, cleared when the CWR flag is received from the
> peer. It does not provide the any indication about which packet was
> affected. So if our goal is to support standard ECN in QUIC, I would argue
> that we only need to define a way to convey the CE and CWR flags. We can do
> that without redefining the format of ranges.
>
> competing a running average of the fraction of data acknowledged by ACKs in
> which the CE bit is marked. DCTCP does not use the CWR bit, but sets it in
> accordance with RFC 3168 for compatibility.
>
> network uses standard ECN, but is overkill there because TCP-similar RENO
> would just need a single bit. It causes a significant overhead if the
> network experiments with non standard ECN as in DCTCP, when in fact the
> DCTCP congestion algorithm only requires on bit of signalling per ACK, the
> same as standard TCP.
>
> -- Christian Huitema
>
>
> --
>
> Magnus Westerlund
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 5 Sep 2018 11:34:36 -0500

Posting as a curious TSV AD, who can be wrong about things ... to a mailing
list of folks who shouldn't be shy about saying "you're wrong!"

On Wed, Sep 5, 2018 at 9:03 AM Ian Swett <ianswett=
[email-masked]> wrote:

>> is a requirements document: https://datatracker..ietf.org/doc/rfc7560/
>> <https://datatracker.ietf.org/doc/rfc7560/> and a experimental proposal
>> the different markings, but it provides a clear proportional marking signal

"Creating much more marking" is certainly my understanding of the way L4S
is intended to work, FWIW.

I'm still having some chats with TSVWG chairs and Bob about when L4S would
be requested for publication, but the plan of record right now is that L4S
and DualQ are targeting September 2018 (see
https://datatracker.ietf.org/wg/tsvwg/about/ in case this changes in the
next five minutes). The intended Status is Experimental, but it would be
nice to know what we should be experimenting with ...

I know that Bob is involved with the QUIC ECN design team, but don't
actually know whether L4S/DualQ came up as part of that discussion.

This might be a really good conversation to have, fairly soon ...

Thanks,

Spencer

---

From: Martin Thomson <[email-masked]>
Date: Thu, 6 Sep 2018 09:53:45 +1000

On Thu, Sep 6, 2018 at 2:34 AM Spencer Dawkins at IETF
<[email-masked]> wrote:
> "Creating much more marking" is certainly my understanding of the way L4S is intended to work, FWIW.

If the problem with congestion controllers is that feedback from loss
is too rare, then clearly a setting where markings are equivalent to
loss is not a great improvement, so yay.

The question I have is whether this is something that can be deployed
unilaterally.  It appears from the l4s architecture doc
(https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-02#section-6.3.2
specifically) that unilateral deployment is possible.  That could
guide the design toward something that allows for L4S (and its ilk).

But L4S remains an experiment and what we're talking about here is an
encoding that is merely suboptimal in terms of bandwidth efficiency
for those settings.  It is otherwise workable.

That makes me very much inclined to suggest that - for this experiment
and others like it - we don't go out of our way to design for it and
to leave this as a problem for later versions of this working group to
ponder.  Designing for the QUIC we have rather than the one we might
want is already firmly in our charter and when we have to accommodate
too many use cases, that's where things go pear-shaped.

---

From: Ian Swett <[email-masked]>
Date: Wed, 5 Sep 2018 21:22:20 -0400

Agreed that we should design for the world as it exists today.  Though to
be a pragmatist, that means we should optimize for DCTCP, not classic ECN.

The current drafts have a design that works well for L4S and all other
known users of ECN, so I think the bar for replacing it is fairly high.
Certainly moving from marked packets to marked bytes and allowing a more
efficient encoding for long connections are potential improvements, but
even without them, the current feedback mechanism is fairly good.

Some of the simplification of unifying the frames in your PR is from not
having to discuss ACK and ACK_ECN as separate frame types.  We could also
declare them a single type and reserve a bit to indicate whether the ECN
section was present or not.

I've heard of a few experimental uses of ECN that have the property that a
large fraction(ie: 50%) of packets are marked, so I don't think it's unique
to L4S.  Also, despite the 'experiment' label, I was under the impression
that L4S is fairly far along, so not considering it would be fairly short
sighted, but Spencer or Bob know better than I do.

I'm also trying to figure out if it makes sense to work timestamps into
this unified format in a future version of QUIC, or it's easier to add a
separate timestamp section, similar to the extra ECN section.  So far, I'm
leaning towards a separate section, and I think understanding the answer to
that question would also make me more confident about the direction of this
decision.

TLDR: I'm currently leaning against this unified design, but I also
appreciate its benefits.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 5 Sep 2018 22:26:09 -0500

Just to be clear on something I could have been substantially clearer on
...

> sighted, but Spencer or Bob know better than I do.

So, my understanding, is that TSVWG did
https://datatracker.ietf.org/doc/rfc8311/ and
https://datatracker.ietf.org/doc/status-change-ecn-signaling-with-nonces-to-historic/,
which basically means

   - we are going to experiment with Ideas Like L4S, and we're reclaiming
   the ECT(1) bit for those experiments;
   - if you only use ECT(0), ECN will continue to work the way you think it
   works now,
   - L4S won't be the only experiment we do with ECT(1), so don't spend a
   lot of attention trying to interpret ECT(1) until we have some experiments,
   and some experimental results, unless you're intentionally participating in
   one (or more) of these experiments.

And, of course, for extra credit, we're looking at multiple ways of using
the ECT(1) bit, but we can't standardize more than one way to use the same
bit, so if we're doing experiments using ECT(1) with ideas that won't work
for you, you're not going to do better than classic ECN using ECT(0) when
we standardize use of ECT(1).

I defer to the QUIC working group on what the right thing to do for QUIC
is, but as the responsible AD for TSVWG, I'd appreciate it if people who
see problems with L4S/DualQ for QUIC say so in TSVWG, sooner rather than
later. It would be silly if L4S blew up in IETF Last Call because of
concerns from QUIC that hadn't surfaced before IETF Last Call, but could
have surfaced earlier.

Does that help?

And thanks to all of you for thinking this through.

Spencer

---

From: Christian Huitema <[email-masked]>
Date: Wed, 5 Sep 2018 21:40:39 -0700

On 9/5/2018 6:22 PM, Ian Swett wrote:

> Agreed that we should design for the world as it exists today.  Though
> to be a pragmatist, that means we should optimize for DCTCP, not
> classic ECN.
>
> known users of ECN, so I think the bar for replacing it is fairly
> high.  Certainly moving from marked packets to marked bytes and
> allowing a more efficient encoding for long connections are potential
> improvements, but even without them, the current feedback mechanism is
> fairly good.

My point exactly. The ACK-ECN solution in draft-14 may not be the most
elegant, but it does tick a number of check-boxes: no overhead at all
when ECN is not used; works OK with standard ECN; works fine, better
than current TCP with DCTCP or L4S, or in fact pretty much any
experiment that desires frequent congestion feedback with ECN marks. The
proposed unified encoding does OK with standard ECN, but creates
substantial overhead with L4S style ECN, to the point that it might
become unusable if the bandwidth*delay product requires large windows
and thus large number of ranges.

So at this point my inclination would be to just keep what we have.

>
> Some of the simplification of unifying the frames in your PR is from
> not having to discuss ACK and ACK_ECN as separate frame types.  We
> could also declare them a single type and reserve a bit to indicate
> whether the ECN section was present or not..

Yep. For example, define ACK as 0x1c -- 0x1F, with bit 0x01 set if ECN
counters are present and bit 0x02 set if one way delays are present.

>
> I've heard of a few experimental uses of ECN that have the property
> that a large fraction(ie: 50%) of packets are marked, so I don't think
> it's unique to L4S.  Also, despite the 'experiment' label, I was under
> the impression that L4S is fairly far along, so not considering it
> would be fairly short sighted, but Spencer or Bob know better than I do.
>
> I'm also trying to figure out if it makes sense to work timestamps
> into this unified format in a future version of QUIC, or it's easier
> to add a separate timestamp section, similar to the extra ECN
> section.  So far, I'm leaning towards a separate section, and I think
> understanding the answer to that question would also make me more
> confident about the direction of this decision.
>

If we want to do something like LEDBAT, having one way delays would be
really nice. I think it is enough for LEDBAT style control to have just
one extra time stamp, indicating the receiver's local clock at the time
the ACK was sent. From that, sender can distinguish between variations
of the send delay and variation of the return path delay. But I am sure
that a large enough committee can come up with something more grandiose.

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 6 Sep 2018 15:12:35 +0900

It would be sad to not have efficient support for L4S just because we do
not want a separate ACK_ECN frame.

But the inexact nature of counters also adds complications. What about
Kazuho’s bitmap proposal?
Keep ACK’s as they are and have a separate optional bitmap vector if you
need ECN bits?

On 6 September 2018 at 06.41.12, Christian Huitema ([email-masked])
wrote:

-- Christian Huitema

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 6 Sep 2018 10:40:21 +0200

Hi Mikkel

Den 2018-09-06 kl. 08:12, skrev Mikkel Fahnøe Jørgensen:
> It would be sad to not have efficient support for L4S just because we 
> do not want a separate ACK_ECN frame.

Agreed.
> Keep ACK’s as they are and have a separate optional bitmap vector if 
> you need ECN bits?
>
So, a bit vector proposal will at least require one bit, possibly two if 
one like to represent the ECN states accurately. Thus also here the 
bandwidth delay product for the not acked acks will be an issue. A pure 
ACK packet of minimal MTU will only have room for some 1200 bytes, i.e. 
4800 or 9600 packets worth of markings. That is not sufficient for high 
Bandwidth delay products. In these cases the counters can represent this 
accurately.

The lack of per packet exact mapping for the ECN mapping appear not to 
be a significant issue as I see it. For the congestion controllers that 
people are looking at the proportion of markings are what matters. For 
the failure detection it is the lack of markings on some received 
packets that matters. Thus, I really want to see more exact problem 
statements why per packet information of the ECN markings are so 
important. If it was easy enough to include and had no downside I think 
it would be fine. But, now we have arrived at a point where we see that 
there is a real trade-off in attempting to support per packet ECN 
information.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 6 Sep 2018 11:02:01 +0200

Hi,

This answer is in regards to byte counting and the representation of the 
ECN counters in the ACK.

Den 2018-09-05 kl. 16:02, skrev Ian Swett:
>
> On Wed, Sep 5, 2018 at 9:44 AM Magnus Westerlund 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> I actually missed this, I was thinking these counters were in bytes, 
> not packets.  If all we want to do is fix this, it's fairly 
> straightforward.

So personally I don't argue for byte counters. I think having a sender 
side moving average of sent packets are sufficient to respond correctly 
enough to congestion signals as counted in number of sent bytes.

Part of my argument here is that most application will send with mostly 
full sized packets. Thus, the difference between an average and the 
actual size will be low. Only applications with highly varying packet 
sizes that contains frame which are included in congestion controller's 
counting may have any significant error here.

>
> This seems like a potentially worthwhile optimization, especially if 
> you move to byte based counting.

The cost of going to byte counters even with the proposed optimization 
is that most flows will still spend 4-bytes per counter, as the expected 
packet sizes will take 10-14 (1024-8192 bytes) to represent. Thus, any 
flow having more than one to eight packets outstanding will require to 
use the 30-bit representation, i.e. each counter will consume 4 bytes 
for a total of 12-bytes per ACK. Using packet counters a very large 
proportions of the flow will not need more than 2 bytes per counter per 
ACK. Thus resulting in that the overhead for the ECN counters with the 
optimization will be 3 or 6 bytes for flows with no more than 64 or 
16384 packets outstanding since last ack of ack.

But if anyone have a congestion controller that really benefits from 
precise knowledge of how many bytes it would be good to hear about it now.

To conclude, I am of the opinion that byte counters as feedback is not 
necessary. But, the optimization can be worth while as it for a quite 
modest implementation efforts will prevent the ACK counters 
representation from being dependent on flow length and only depend on 
bandwidth delay product. The saving expected is 6 bytes or more per ACK.

Cheers

Magnus Westerlund

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 6 Sep 2018 11:10:17 +0200

Hi Ian, hi all,

two quick comments/clarifications below.

On 05.09.2018 16:02, Ian Swett wrote:
> <[email-masked] <mailto:[email-masked]>> 
> wrote:
> 
>     Hi,
> 
>     I think Christian has a valid point about what the design goals of
>     the optimization is. From my perspective I am considering several
>     factors and try to consider the trade-off between them. I think
>     there is a benefit of a unified reporting form that include ECN
>     information. I also want it to be usable with L4S and DCTCP style
>     usages. I expect we will see QUICv1 based implementation with sender
>     side only congestion control optimizations to such environments.
>     Because of this I do see a point of doing an optimization that works
>     reasonable for all cases, no ECN, legacy ECN and "new" ECN.
> 
>     https://datatracker..ietf.org/doc/rfc7560/
>     <https://datatracker.ietf.org/doc/rfc7560/> and a experimental
>     proposal for accurate ECN reporting:
>     https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ the
>     later is actually a byte counter based proposal. That combines an
>     TCP option that reports the least 24-bytes of the counters for
>     ECT(0), ECT(1) and CE. The number of CE marked packets are also
>     reported using a three bit field in the TCP header. To make this
>     somewhat robust it requires that a receiver send one ACK one every 2
>     (SHOULD) to 6 (Max) increments of the CE packet counter.
> 
>     The requirements document states that preferably one should have
>     byte accounting on what is marked. Regarding accuracy in section 4:
> 
>     The current packet counters do not provide byte accurate accounting
>     of the different markings, but it provides a clear proportional
>     marking signal that is very useful for DCTCP or L4S style usages.
> 
> 

The counters in the options are in bytes (only the CE counter in the 
header is in packet but that is optimized for use of 3 bits only; in TCP 
we only count the TCP payload, so there are 0-length control packets 
that do should up in the packet counter but not the byte counter. So 
that's an important bit of additional information to have in the TCP case).

> 
> 
>     The Unified ACKing proposal enables byte accurate accounting.
>     However, using a one byte-ack per change results in that when the
>     frequency of the changes between ECT and CE gets higher than 12
>     (common case) in the ACK vector coverage they are less efficient.
>     The ACK vector coverage will, if reliable ACKing is used, cover at
>     least 1 RTT of in flight packets and likely will contain around 1.5
>     RTT worths of packet.
> 
>     If I interpret the marking probabilities correctly in this paper for
>     L4S http://www.bobbriscoe.net/projects/latency/dctth_preprint.pdf
>     with dual queue. A single flow filling the bottleneck using L4S
>     would see a bit more than 10% marking probability. If 10 L4S TCP
>     flows compete with 10 Cubic flows, the L4S flow will see more than
>     80% of its packets marked, while the cubic flow will see around 1%
>     drop rates in its queue. Thus evaluating overhead considerations for
>     L4S cases might actually need to include the pattern that every
>     second packet is CE marked as worst case scenario. Even at 10%
>     marking rate the bandwidth delay products where counters are more
>     efficient are actually quite low. So lets assume 4 bytes counters
>     per type, then 12 changes to give equal overhead, means BDP of less
>     than 120 packets in a good case, and 24 packets in worst case.
>     Looking at the BDP figures I published in the discussion:
> format is optimized for typical loss patterns, which are typically 
> bursty even when the loss rate is high, but I can imagine L4S creating 
> much more even marking, and hence getting very inefficient encoding.
> 
> 
> 
>     The worst case is around 250 mbps at 1 ms RTT and 5 mbps on 50 ms.
>     The good case is a bit over 1Gbps on a 1 ms RTT and just under 25
>     mbps on 50 ms RTT. The conclusion one can draw is that no DCTCP or
>     L4S use cases at bitrates that can be expected to exist both inside
>     a data center (1Gbps+) or for more demanding applications across
>     actual access networks can be efficiently supported using per packet
>     explicit marking. In fact the ACK blocks will blow through the MTU
>     in high rate applications. 

This is dependent on your ACK rate.

Mirja

> 
> 
>     I think if one want a unified ACK vector, then maybe one should look
>     at making the counters a bit shorter by only reporting the lowest
>     part of the counters. Where the receiver chooses if it uses the
>     6-bits, 14-bits, 30-bits or 62-bits worth of counter based on how
>     quickly it is changing. I would assume that if one has more bits
>     than what wraps in a couple of RTTs this will be robust. Even
>     without unified this could be good to reduce the overhead for long
>     lived flows and making it only dependent on the amount of in-flight
>     packets.
> 
> 
>>>     or adopting an encoding that can encode**any**  use-case of the ECN
>>     concentrates on the "standard" use of ECN, in which ECN-CE marks
>>     as substitute for packet losses, and thus not expected to happen
>>     very often. But that same property makes ECN a poor feedback
>>     mechanism, especially for high speed transmission. Systems that
>>     want efficient congestion control at high speed need feedback at
>>     least once per RTT, if not more often. In practice today, they get
>>     that feedback by using delay-based algorithms, because delay
>>     samples can be obtained at every ACK -- BBR is an example of such
>>     algorithms.
>>
>>     redefine the marking strategy so that marks happen many times per
>>     RTT -- much like DCTCP does, or better yet Bob Briscoe's proposals.
>>
>>
>>     documented ECN based congestion algorithms, the standard one and
>>     the DCTCP experiment, and none of them uses that property.
>>
>>     Standard TCP currently reports standard ECN with one bit of
>>     feedback per RTT per RFC 3168: a CE flag, cleared when the CWR
>>     flag is received from the peer. It does not provide the any
>>     indication about which packet was affected. So if our goal is to
>>     support standard ECN in QUIC, I would argue that we only need to
>>     define a way to convey the CE and CWR flags. We can do that
>>     without redefining the format of ranges.
>>
>>     Non-standard TCP such as DCTCP (RFC 8257) also uses the CE bit,
>>     but changes the behavior. The sender estimates the congestion
>>     state by competing a running average of the fraction of data
>>     acknowledged by ACKs in which the CE bit is marked. DCTCP does not
>>     use the CWR bit, but sets it in accordance with RFC 3168 for
>>     compatibility.
>>
>>     I feel that #1706 is a premature optimization. It works fine if
>>     the network uses standard ECN, but is overkill there because
>>     TCP-similar RENO would just need a single bit. It causes a
>>     significant overhead if the network experiments with non standard
>>     ECN as in DCTCP, when in fact the DCTCP congestion algorithm only
>>     requires on bit of signalling per ACK, the same as standard TCP.
>>
>>     -- Christian Huitema
> 
>     -- 
> 
>     Magnus Westerlund
> 
>     SE-164 80 Stockholm, Sweden | mailto:[email-masked] <mailto:[email-masked]>

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 6 Sep 2018 11:19:34 +0200

The thing that I like about the new ECN encoding proposal is that you 
cannot not implement the ECN part. I think that is the most important 
step to enable ECN experimentation.

Further I think it is okay to optimize for current use of ECN as long as 
experimentation is possible at all. Therefore I would support the new 
design if ECT(0) and ECT(1) are encoded in the same type because that 
leaves it as a sender-side only decision to start an experiment. Of 
course feedback behavior will not be optimal for e.g. the L4S 
experimentation but I would say it is part of the experiment to figure 
out what we would need to optimize for (and then when we know we can 
later on design a new extension frame that is optimized for that case).

Mirja

On 06.09.2018 03:22, Ian Swett wrote:
> to be a pragmatist, that means we should optimize for DCTCP, not classic 
> ECN.
> 
> having to discuss ACK and ACK_ECN as separate frame types.  We could 
> also declare them a single type and reserve a bit to indicate whether 
> the ECN section was present or not..
> 
> I've heard of a few experimental uses of ECN that have the property that 
> a large fraction(ie: 50%) of packets are marked, so I don't think it's 
> unique to L4S.  Also, despite the 'experiment' label, I was under the 
> impression that L4S is fairly far along, so not considering it would be 
> fairly short sighted, but Spencer or Bob know better than I do.
> 
> separate timestamp section, similar to the extra ECN section.  So far, 
> I'm leaning towards a separate section, and I think understanding the 
> answer to that question would also make me more confident about the 
> direction of this decision.
> 
> On Wed, Sep 5, 2018 at 7:54 PM Martin Thomson <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 6 Sep 2018 10:47:35 +0000

Hi,

On 2018-9-6, at 12:19, Mirja Kühlewind <[email-masked]> wrote:
> The thing that I like about the new ECN encoding proposal is that you cannot not implement the ECN part.

I'm not sure that's fully accurate. Both the -14 ACK_ECN frame and any new unified frame will be mandatory to parse on RX for compliance. But there is no enforceable requirement to act on the content.

Likewise, on TX, it's not enforceable that a stack must obtain the ECN bits from the kernel and encode them. With -14, it would simply always encode ACK frames (and never encode ACK_ECN frames), and with any unified approach, it could simply always encode "nothing marked".

There is maybe a slight nudge towards doing ECN properly with a unified approach, since you need to encode something for ECN, and so you might as well do the right thing. But it remains fully possible to not support ECN, with either approach. (Unfortunately.)

Lars

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 6 Sep 2018 12:49:16 +0200

Hi,

> step to enable ECN experimentation.

I think this is mostly optics. Yes one need to implement processing the 
ECT and CE type if they ever occur in an ACK. However, an end-point that 
don't want to implement ECN can simply never attempt to set or read the 
ECN bits. By reporting all packets as not-ect it doesn't care what the 
other endpoint does. And it should never receive an ECT or CE marked ack 
block except for the highly unlikely remarking from not-ect to ECT/CE.

 From this perspective the actual implementation burden for someone who 
don't like to support ECN is basically the same. Also a non-ECN enabled 
endpoint should consider what it does if it receives an ACK-ECN in the 
draft-14 way. The simplest way is to process it like an ACK with the 
additional skipping of the three counters.

If that is the optics one is after, having a signalling bit inside the 
ACK frame instead of as the frame-type would provide a very similar optics.

>
> Further I think it is okay to optimize for current use of ECN as long 
> as experimentation is possible at all. Therefore I would support the 
> new design if ECT(0) and ECT(1) are encoded in the same type because 
> that leaves it as a sender-side only decision to start an experiment. 
> Of course feedback behavior will not be optimal for e.g. the L4S 

Yes, that is a possibility as the only use cases that appears to not 
really work is the really high bandwidth delay product, like 
inter-continental high band inter data center transfers, when using high 
marking probability schemes. My view is that counters still are more 
robust and efficient without any short comings that actually affects a 
usage likely to actually be experimented with.

Cheers

Magnus Westerlund