Subject: Q and L loss bits: which is which?

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 19 Nov 2019 09:32:19 -0500

Hello,

I've read the loss bits draft [1] and the basic functionality (a single
counter per connection, ignoring the per-CID or per-tuple nuances) is
very doable.  I can set up a separate public endpoint that generates
these bits for experimentation.

One stumbling block is that the draft does not specify which short header
R bit [2] is L and which is Q.

  - Dmitri.

1. https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-02
2. https://tools.ietf.org/html/draft-ietf-quic-transport-24#section-17.3

---

From: <[email-masked]>
Date: Tue, 19 Nov 2019 15:49:08 +0100

Hello Dmitri,

Thanks for your offer: experimentations are very welcome !
You're right we forgot to just define which is what, we'll fix the
draft shortly. We'll choose: 0x10 for Q and 0x08 for L.
This means the first byte layout becomes:
 |0|1|S|Q|L|K|P P|

Note the WG meeting yesterday concluded these should be controlled by a
negotiated option. But in the meantime, that does not prevent you from
running an experiment as long as you also modify the client, so that it
does not enforce the drafted constraint on the reserved bits.

> 2. https://tools.ietf.org/html/draft-ietf-quic-transport-24#section-1

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 19 Nov 2019 10:08:29 -0500

> This means the first byte layout becomes:

Thank you!

I also assume that these two bits would now be outside of the header
protection?  That is, the short header header protection mask [1] is
now 0x7 instead of 0x1F?  This is not mentioned in your draft, but I
don't see how an observer could get at those bits otherwise.

  - Dmitri.

1. https://tools.ietf.org/html/draft-ietf-quic-tls-24#section-5.4.1

---

From: <[email-masked]>
Date: Tue, 19 Nov 2019 18:19:04 +0100

Yes, you're right again. Those bits are definitely to be sent in clear:
     |0|1|S|Q|L|E E E|
Of course, the whole header remains part of the AD so is integrity
protected. Hence Q&L are no more subject to middlebox tampering than
the rest.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 19 Nov 2019 14:26:23 -0500

I added a loss bits server on

    http3-test.litespeedtech.com:4440

Feel free to give it a shot!

Both lsquic server and client can be compiled with loss bits support
by doing

    EXTRA_CLFAGS=-DLSQUIC_LOSS_BITS=1 cmake ....

The code is on a branch for now:

    https://github.com/litespeedtech/lsquic/commit/0b83dfa304bcd9750b67e6297416ce493d9fb5b1

  - Dmitri.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 Nov 2019 07:19:09 +0800

I am willing to do the same in Picoquic, but I would really like to get
the transport parameter defined first.

Also, I would like some clarification. The loss counter is supposedly
incremented using Quic's loss detection machinery, but the loss
detection mechanism sometimes err on the false positive side. This is
detected if an ack arrive late for a packet that was supposed lost, and
the loss is classified as spurious.

Any guidance on how to handle spurious loss detection?

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Nov 2019 23:56:05 +0000

Christian, I actually thought about this, and decided that the recommendation is not to decrement L counter.  It is in the -tsvwg- version of the draft but somehow did not have it into the -quic- version (keeping two versions in sync is apparently hard), sorry. (https://tools.ietf.org/html/draft-ferrieuxhamchaoui-tsvwg-lossbits-02#section-3.2)

"
If the protocol is able to rescind the loss determination later, the Unreported Loss counter SHOULD NOT be decremented due to the rescission.
"

I thought this rule would limit the confusion of the observer who is trying to correlate Q and L bits. I'm happy to hear alternative opinions on this.

- Igor

> -----Original Message-----
> From: Christian Huitema <[email-masked]>
> Sent: Wednesday, November 20, 2019 7:19 AM
> To: [email-masked]; IETF QUIC WG <[email-masked]>
> Subject: Re: Q and L loss bits: which is which?
> 
> I am willing to do the same in Picoquic, but I would really like to get the
> transport parameter defined first.
> 
> incremented using Quic's loss detection machinery, but the loss detection
> mechanism sometimes err on the false positive side. This is detected if an ack
> arrive late for a packet that was supposed lost, and the loss is classified as
> spurious.
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 20 Nov 2019 08:16:28 +0800

Hi Igor, all,

Sent from my iPhone

> On 20 Nov 2019, at 07:56, Lubashev, Igor <[email-masked]> wrote:
> 

FWIW I agree that these are the right semantics.

With these semantics, if I follow correctly, the L bit is basically signaling that the sender has detected a loss _to which it will react_ (or rather, to which it would react if it were already network limited), by reducing the congestion window, updating its loss estimate or capacity model, or whatever sending-rate-relevant parameters its congestion control and avoidance scheme uses. 

This makes the condition it is signaling pretty close to the CWR bit (if the CWR bit covered all such reactions, not just those triggered by ECN), which is basically the proposal I (poorly and cryptically) alluded to in the issue and the mic.

(Two nice properties of exposing loss reaction as opposed to raw loss: one, you’re closer to measuring what you actually care about — poor transport performance — and two, since the signal is correlated with the time series of the sending rate, statistical techniques are applicable to verifying the signal: it’s harder to lie.)

Cheers, B

---

From: <[email-masked]>
Date: Wed, 20 Nov 2019 01:18:51 +0100

> the transport parameter defined first.

Sure, working on it :)
By the way, in what shape is the IANA registry announced for TPs ? Is
it opened yet ?

> Any guidance on how to handle spurious loss detection?

One way would be to just decrement the "yet unreported loss"  counter
whenever the stack decides a loss was spurious. This may when the
counter is zero, bringing it to negative values. This does not affect
the rule for setting L=1, namely that the counter be >0. This way,
assuming spurious events are a fraction of genuine ones, the total
count at the end of the connection is accurate (except for spurious
after the last genuine).

A refinement would be to somehow delay sending L=1, to let a chance to
the counter to become >=0 again. The problem being, of course, that we
can't guarantee to have enough outgoing packets before the end of a
connection, leading to underestimation of loss at the end; this may be
workable in practice, as the Q halfperiod granularity has a similar
effect for upstream loss anyway.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 Nov 2019 08:25:27 +0800

> 
> On Nov 20, 2019, at 8:16 AM, Brian Trammell (IETF) <[email-masked]> wrote:
> 
> Hi Igor, all,
> 
> Sent from my iPhone
> 

OK, but if I determine that a loss is spurious I revert the congestion control state to what it would be if the loss did not happen. 

>> I thought this rule would limit the confusion of the observer who is trying to correlate Q and L bits. I'm happy to hear alternative opinions on this.

I am in the rescind case. But maintaining a single counter is simpler, so I will do that.

-- Christian

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 20 Nov 2019 00:37:01 +0000

> On Wed, Nov 20, 2019 at 8:25 AM, Christian Huitema <[email-masked]> wrote:
>
> OK, but if I determine that a loss is spurious I revert the congestion control
> state to what it would be if the loss did not happen.

Ok, I think there is a better middle ground.  If the counter is > 0, sure, decrement it (since the loss has not yet been reported).  But if the counter is already 0, do not let it go negative, because then you would not be able to report an actual future loss event (and an actual future congestion controller response).

- Igor

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 19 Nov 2019 20:48:50 -0500

> the transport parameter defined first.

Once there is a TP and a draft, we plan to include this option into
main line of lsquic.

  - Dmitri.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 20 Nov 2019 02:06:11 +0000

> On Wed, Nov 20, 2019 at 9:49 AM, Dmitri Tikhonov <[email-masked]> wrote:
> 
> > I am willing to do the same in Picoquic, but I would really like to
> > get the transport parameter defined first.
> 
> Once there is a TP and a draft, we plan to include this option into main line of
> lsquic.

Thanks, Dmitri and Christian.  We are getting great feedback now, which we are absorbing.  Plus, there are privacy guidelines to carefully consider and refine.  So I'd expect a draft in about 2-3 weeks.

- Igor

---

From: Lars Eggert <[email-masked]>
Date: Thu, 21 Nov 2019 08:42:31 +0200

Hi,

I'm wondering whether specifying a CONEX mapping for QUIC would satisfy the needs that quic-lossbits attempts to address? We could reuse an existing piece of IETF work.

See https://tools.ietf.org/html/rfc7713 for an overview and https://tools.ietf.org/html/rfc7786 for the TCP mapping.

Lars

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 21 Nov 2019 14:44:11 +0800

hi Lars,

This is kind of where I was going with the "expose CWR-like semantics for loss" thing; "just do conex" is a much better shorthand for that.

Thanks, cheers,

Brian

> On 21 Nov 2019, at 14:42, Lars Eggert <[email-masked]> wrote:
> 
> Hi,
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 21 Nov 2019 07:23:06 +0000

A few things about the ConEx vs QL-lossbits

1. ConEx has a mechanism that is similar to L-bit (we mention ConEx RFC7713 in the draft).  But it has no equivalent of the Q-bit signal.  It is very hard to tell upstream loss from downstream loss with ConEx.

2. ConEx reports bytes, while QL-loss reports packets.  There is a trade-off here, as bytes may be more valuable, but reporting bytes can be imprecise, if the packets marked are of different size from the packets lost.  Overall, both work well enough.

3. ConEx requires more bits than what we have in QUICv1, so it would be a significant disruption to QUICv1 header format to properly implement all ConEx bits.  The Q- and L- bits are enough for the purpose.

4. CWR is an interesting signal, and I actually have a separate CWR bit (I call it "E-bit") in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-tsvwg-lossbits-02#section-5 -- the -tsvwg- version of the draft.  However, it is not very closely related to loss for networks with ECN.  On an ECN-capable network for ECN-capable flows, downstream loss would become indistinguishable from downstream congestion that marks CE.  Ultimately, it seemed that given just two bits to work with, Q-bit plus a pure loss bit seemed preferable to Q-bit plus "loss-or-CE_echo" bit.  But I am happy to discuss this!

- Igor

> On Thu, Nov 21, 2019 at 2:44 PM, Brian Trammell (IETF) <[email-masked]> wrote:
> 
> hi Lars,
> 
> This is kind of where I was going with the "expose CWR-like semantics for loss"
> thing; "just do conex" is a much better shorthand for that.
> 
> Thanks, cheers,
> 
> Brian
> 
> > I'm wondering whether specifying a CONEX mapping for QUIC would satisfy
> the needs that quic-lossbits attempts to address? We could reuse an existing
> piece of IETF work.
> >
> > See https://tools.ietf.org/html/rfc7713 for an overview and
> https://tools.ietf.org/html/rfc7786 for the TCP mapping.
> >
> > Lars
> >

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 21 Nov 2019 18:07:08 +0800

> On 21 Nov 2019, at 15:23, Lubashev, Igor <[email-masked]> wrote:
> 

Yep, at least when I talk about this proposal I mean "replace L with ConEx". We can leave Q as it is. (The question as to the independent utility of Q and L is a separate discussion).

> 2. ConEx reports bytes, while QL-loss reports packets.  There is a trade-off here, as bytes may be more valuable, but reporting bytes can be imprecise, if the packets marked are of different size f…

IIRC ConEx spent an inordinate amount of time on the bytes versus packets bikeshed so let's not :)

> 3. ConEx requires more bits than what we have in QUICv1, so it would be a significant disruption to QUICv1 header format to properly implement all ConEx bits.  The Q- and L- bits are enough for the…

What I'd like to look at is a one-bit encoding for L that exposes the same signals ConEx did. It turns out this might be about the same thing as L as currently defined when L doesn't count losses determined to be spurious. 

> 4. CWR is an interesting signal, and I actually have a separate CWR bit (I call it "E-bit") in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-tsvwg-lossbits-02#section-5 -- the -tsvwg- version of the draft.  However, it is not very closely related to loss for networks with ECN.

This raises the other question: whether what we really want to measure is "this packet is lost" (which network operations activities use loss as an indication of lower-layer issues they need to fix) or "this packet caused the transport difficulty" (which is more of an indication of endpoint-visible performance, interesting to a different department in network operations). Or, "if a packet is dropped in the woods, and nobody cares about it, does it make a sound?"

On most networks with most deployed CC algorithms these generate basically the same metric, so the distinction is maybe philosophical. It is less true for nonelastic traffic and for newer CC algorithms.

Cheers,

Brian

>  On an ECN-capable network for ECN-capable flows, downstream loss would become indistinguishable from downstream congestion that marks CE.  Ultimately, it seemed that given just two bits to work with, Q-bit plus a pure loss bit seemed preferable to Q-bit plus "loss-or-CE_echo" bit.  But I am happy to discuss this!
> 
> - Igor
>

---

From: <[email-masked]>
Date: Thu, 21 Nov 2019 13:16:44 +0100

On Thu, 2019-11-21 at 18:07 +0800, Brian Trammell (IETF) wrote:
> What I'd like to look at is a one-bit encoding for L that exposes the
> same signals ConEx did. It turns out this might be about the same
> thing as L as currently defined when L doesn't count losses
> determined to be spurious.

It might very well :)

Any other signal than loss is out of scope in many real-life scenarii:
a router that loses packets well below capacity due to internal
implementation limitations (read "bugs") will certainly *not* CE-mark
any packet; it will just drop in sheer panic. The "pre-ECN world" is
not dead.

Regarding spurious losses: you're right, it would be better if we had a
sure-fire method to eliminate them from L. Attempts to do that were
recently mentioned here. A reasonable, conservative one was proposed by
Igor yesterday:

"Ok, I think there is a better middle ground.  If the counter is > 0,
sure, decrement it (since the loss has not yet been reported).  But if
the counter is already 0, do not let it go negative, because then you
would not be able to report an actual future loss event (and an actual
future congestion controller response)."

To judge the importance of this issue, it would be good to have an idea
of the frequency of such (DSACK-like) spurious events under varied
assumptions.