Subject: QUIC Invariants and QUIC-LB

From: Martin Duke <[email-masked]>
Date: Wed, 20 May 2020 16:28:09 -0700

TL;DR We are trying to make QUIC-LB as independent of QUIC version as
possible, but there are a few gaps in the invariants that make that harder.
If any of these are oversights, it would be great to add it to the
invariants.

*******
A few draft versions ago we did a pass on QUIC-LB to make it truly
QUIC-version invariant, except for the Retry Services portion.

After a little more thought, I added this section to the draft:
https://quicwg.org/load-balancers/draft-ietf-quic-load-balancers.html#name-version-invariance-of-quic-
which describes a bunch of QUIC behaviors that are not explicitly described
in the invariants draft, but that if changed might cause QUIC-LB not to be
fully robust to new versions. An obvious one is a version that described a
maximum CID length too small for some of the encodings.

A more subtle one is the handling of "non-compliant CIDs" (Sec 4). These
are CIDs that are (1) too short for the configured algorithm, (2) fail
authentication, or (3) don't map to a valid server ID. In v1, these will
only occur in the client's first flight of Initial and 0RTT packets. The
rules we have for version-independent LBs to process packets with
non-compliant CIDs are as follows:
* if a short header, drop it.
* if a long header, use any hash algorithm that will consistently get you
the same result over short time scales (i.e. a couple of RTTs, by which
time a server-generated CID will be in use)
Compliant CIDs are always processed using the decoding in the spec.

There are some assumptions baked into this advice.
(1) Short headers never contain client-generated CIDs. [QUIC-LB servers
will always provide new CIDs when given the opportunity, but a new version
could have short 0RTT headers or something]
(2) Use of client-generated DCIDs is limited to a few RTTs at most (any
more and you run the risk of the number of servers changing in this
interval, which will generally cause the hash to get a different result)
(3) There's a bunch of stuff that remains constant in the packet headers
over this short interval. I probably need to tighten up the language on
what is a suitable input for the LB's hash, but in v1 you're currently safe
using some combination of Client IP, Client Port, SCID, and DCID. If there
of these that we're willing to lock down in the invariants, I could rewrite
the guidance accordingly.

For example, if the invariants could include a statement like "A client
will use no more than one DCID that it didn't receive from the server",
that would simplify the problem space.

****

I'm fine with having this text in the QUIC-LB draft, if need be, but if
there are things that should be in the invariants but we didn't think about
them, that would be even better.

Martin

---

From: Ian Swett <[email-masked]>
Date: Thu, 21 May 2020 10:31:28 -0400

Thanks for pointing these out.  To me, these constraints seem like natural
consequences of using QUIC with a load balancer.  But there are use cases
for QUIC where a load balancer isn't required(ie: P2P and many datacenter
use cases), so I'd prefer not to add more constraints to the invariants.

On Wed, May 20, 2020 at 7:28 PM Martin Duke <[email-masked]> wrote:

> which describes a bunch of QUIC behaviors that are not explicitly
> described in the invariants draft, but that if changed might cause QUIC-LB
> not to be fully robust to new versions. An obvious one is a version that
> described a maximum CID length too small for some of the encodings.
>

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 22 May 2020 09:40:21 +1000

That's a good point.

The set of assumptions here are fairly small, which is good.  Essentially, these turn into requirements for a new version of QUIC that is intended to replace v1, if that version also needs to work with a load balancer (which seems quite likely).  As the set is so small, and as I don't think that any replacement would be likely to make changes at so fundamental a level, it seems like those requirements wouldn't end up being too onerous.

Basically, this boils down to whether you think the ability to load balance is invariant, and while it might be inconceivable for something like HTTP, it is not generally so.

On Fri, May 22, 2020, at 00:31, Ian Swett wrote:
> Thanks for pointing these out. To me, these constraints seem like 
> natural consequences of using QUIC with a load balancer. But there are 
> use cases for QUIC where a load balancer isn't required(ie: P2P and 
> many datacenter use cases), so I'd prefer not to add more constraints 
> to the invariants.
> 
> > TL;DR We are trying to make QUIC-LB as independent of QUIC version as possible, but there are a few gaps in the invariants that make that harder. If any of these are oversights, it would be great to add it to the invariants.
> > 
> > *******
> > A few draft versions ago we did a pass on QUIC-LB to make it truly QUIC-version invariant, except for the Retry Services portion.
> > 
> > which describes a bunch of QUIC behaviors that are not explicitly described in the invariants draft, but that if changed might cause QUIC-LB not to be fully robust to new versions. An obvious one is a version that described a maximum CID length too small for some of the encodings.
> > 
> > A more subtle one is the handling of "non-compliant CIDs" (Sec 4). These are CIDs that are (1) too short for the configured algorithm, (2) fail authentication, or (3) don't map to a valid server ID. In v1, these will only occur in the client's first flight of Initial and 0RTT packets. The rules we have for version-independent LBs to process packets with non-compliant CIDs are as follows:
> > * if a long header, use any hash algorithm that will consistently get you the same result over short time scales (i.e. a couple of RTTs, by which time a server-generated CID will be in use)
> > (1) Short headers never contain client-generated CIDs. [QUIC-LB servers will always provide new CIDs when given the opportunity, but a new version could have short 0RTT headers or something] 
> > (2) Use of client-generated DCIDs is limited to a few RTTs at most (any more and you run the risk of the number of servers changing in this interval, which will generally cause the hash to get a different result)
> > (3) There's a bunch of stuff that remains constant in the packet headers over this short interval. I probably need to tighten up the language on what is a suitable input for the LB's hash, but in v1 you're currently safe using some combination of Client IP, Client Port, SCID, and DCID. If there of these that we're willing to lock down in the invariants, I could rewrite the guidance accordingly.
> > 
> > For example, if the invariants could include a statement like "A client will use no more than one DCID that it didn't receive from the server", that would simplify the problem space.
> > 
> > ****
> > 
> > I'm fine with having this text in the QUIC-LB draft, if need be, but if there are things that should be in the invariants but we didn't think about them, that would be even better.
> > 
> > Martin

---

From: Martin Duke <[email-masked]>
Date: Thu, 21 May 2020 17:00:06 -0700

I don't think we're going to get a 100% version-invariant QUIC-LB, though
we're pretty close.

It would be nice to give LBs somewhat more specific guidance for what
fields they can use for packets with non-compliant DCIDs. For example, if
we could at least safely assume the DCID won't change until the server
provides one, that would be way better than forcing them to make
assumptions about IP and UDP fields remaining stable.

On Thu, May 21, 2020 at 4:41 PM Martin Thomson <[email-masked]> wrote:

> That's a good point.
>
> The set of assumptions here are fairly small, which is good.  Essentially,
> these turn into requirements for a new version of QUIC that is intended to
> replace v1, if that version also needs to work with a load balancer (which
> seems quite likely).  As the set is so small, and as I don't think that any
> replacement would be likely to make changes at so fundamental a level, it
> seems like those requirements wouldn't end up being too onerous.
>
> Basically, this boils down to whether you think the ability to load
> balance is invariant, and while it might be inconceivable for something
> like HTTP, it is not generally so.
>
> > > A more subtle one is the handling of "non-compliant CIDs" (Sec 4).
> These are CIDs that are (1) too short for the configured algorithm, (2)
> fail authentication, or (3) don't map to a valid server ID. In v1, these
> will only occur in the client's first flight of Initial and 0RTT packets.
> The rules we have for version-independent LBs to process packets with
> > > * if a long header, use any hash algorithm that will consistently get
> you the same result over short time scales (i.e. a couple of RTTs, by which
> > > (1) Short headers never contain client-generated CIDs. [QUIC-LB
> servers will always provide new CIDs when given the opportunity, but a new
> version could have short 0RTT headers or something]
> > > (2) Use of client-generated DCIDs is limited to a few RTTs at most
> (any more and you run the risk of the number of servers changing in this
> > > (3) There's a bunch of stuff that remains constant in the packet
> headers over this short interval. I probably need to tighten up the
> language on what is a suitable input for the LB's hash, but in v1 you're
> currently safe using some combination of Client IP, Client Port, SCID, and
> DCID. If there of these that we're willing to lock down in the invariants,
> I could rewrite the guidance accordingly.
> > >
> > > For example, if the invariants could include a statement like "A
> client will use no more than one DCID that it didn't receive from the
> server", that would simplify the problem space.
> > >
> > > ****
> > >
> > > I'm fine with having this text in the QUIC-LB draft, if need be, but
> if there are things that should be in the invariants but we didn't think
> about them, that would be even better.
> > >
> > > Martin
>
>

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 22 May 2020 13:11:47 +1000

On Fri, May 22, 2020, at 10:00, Martin Duke wrote:
> I don't think we're going to get a 100% version-invariant QUIC-LB, 
> though we're pretty close.

Yep.

Based on this discussion, what I think is happening here is that we're defining a new set of invariants: the set of things that we can't really change between versions that support the same load balancers as QUIC version 1.  That this is a superset of the invariants work is a bit unfortunate, but the difference is small enough.

> fields they can use for packets with non-compliant DCIDs. For example, 
> if we could at least safely assume the DCID won't change until the 
> server provides one, that would be way better than forcing them to make 

Based on invariants + LB-invariants, I think that you have some good advice already:

If you didn't create the DCID, then apply some sort of "default" routing.  That routing will rely on the stability of a subset of fields.  I would select from (source IP, source port, destination IP, destination port, DCID) and no others.  Nothing else in the QUIC packet is good, except *perhaps* the version number.  Without understanding the version, SCID is totally out as it isn't guaranteed to be stable.  Add anything that isn't stable and you might get problems, so while I can see a case for including something like the IPv6 flow label, that's risky.

Generally speaking, it should be enough to route based on DCID alone, or to route just on the source address, but the more (stable) fields you include the more you allow concurrent connection attempts from the same address.

Finally, to speak to the first assumption, if it is a short header, and you decide to use the DCID for routing, then you are OK throwing the packet away, because you don't know how long the DCID is.  Similarly, if the DCID is empty and you need it, then throwing it out is justified.  But you can do QUICv1 without either if you only use addressing information, so that wouldn't need to be the same for every load balancer.

---

From: Martin Duke <[email-masked]>
Date: Fri, 22 May 2020 08:52:39 -0700

Filed  https://github.com/quicwg/load-balancers/issues/22 to use Martin's
recommendation on fields likely to be stable.

On Thu, May 21, 2020 at 8:12 PM Martin Thomson <[email-masked]> wrote:

> Based on this discussion, what I think is happening here is that we're
> defining a new set of invariants: the set of things that we can't really
> change between versions that support the same load balancers as QUIC
> version 1.  That this is a superset of the invariants work is a bit
> unfortunate, but the difference is small enough.
>
> If you didn't create the DCID, then apply some sort of "default" routing.
> That routing will rely on the stability of a subset of fields.  I would
> select from (source IP, source port, destination IP, destination port,
> DCID) and no others.  Nothing else in the QUIC packet is good, except
> *perhaps* the version number.  Without understanding the version, SCID is
> totally out as it isn't guaranteed to be stable.  Add anything that isn't
> stable and you might get problems, so while I can see a case for including
> something like the IPv6 flow label, that's risky.
>
> Generally speaking, it should be enough to route based on DCID alone, or
> to route just on the source address, but the more (stable) fields you
> include the more you allow concurrent connection attempts from the same
> address.
>
> Finally, to speak to the first assumption, if it is a short header, and
> you decide to use the DCID for routing, then you are OK throwing the packet
> away, because you don't know how long the DCID is.  Similarly, if the DCID
> is empty and you need it, then throwing it out is justified.  But you can
> do QUICv1 without either if you only use addressing information, so that
> wouldn't need to be the same for every load balancer.
>
>