Subject: Facebook's QUIC Deployment

From: Matt Joras <[email-masked]>
Date: Wed, 21 Oct 2020 09:30:43 -0700

All,

I'm excited to finally publicly share some of the work we've been doing on
QUIC in recent years. You can read about it on the Facebook Engineering
blog here
<https://engineering.fb.com/networking-traffic/how-facebook-is-bringing-quic-to-billions/>
.

The headline is that 75% of Facebook's Internet traffic is now QUIC and
HTTP/3. In real terms this amounts to QUIC being fully enabled in the
Facebook and Instagram mobile apps. We have had production traffic using
QUIC on the Internet for over two years, but have dramatically increased
the volume through 2020. We have been able to do this because QUIC has
consistently shown significant empirical improvements over TCP. Importantly
I'd like to stress that these aren't simply "performance" improvements, but
rather measurable improvements to the application experience.

A few things to note:
- QUIC is enabled for all content in both apps, and is utilized for media
upload as well.
- The baseline HTTP for the apps is either HTTP/2 (for the Facebook app) or
HTTP/1.1 (for Instagram).
- The baseline transport server-side is vanilla Linux TCP + BBR with
sensible TCP tuning. Our QUIC implementation doesn't diverge majorly from
the drafts except for the use of BBR as the CCA.
- Our TCP baseline uses TLS 1.3 and early_data, with the vast majority of
connections resuming.
- We are not currently using 0-RTT for QUIC.
- We are not currently using connection migration.
- We use a default UDP payload size of 1232 or 1252 depending on the IP
version, and do not yet have our DPLPMTUD implementation enabled.
- None of the quoted metrics include browser traffic, though we have had
QUIC enabled on Facebook and Instagram web endpoints for about a year. As
of today about 56% of Chrome stable traffic towards our servers is using
QUIC, though we do not control the sampling so we cannot make very
definitive statements about improvements.

We believe several factors contribute to QUIC's significant advantage. A
major factor which is often overlooked on paper is QUIC's ability to
utilize modern loss recovery and congestion control undisturbed by the
myriad middleboxes on Internet networks. The others we all like to mention
in theory clearly play a role in practice as well.

The linked post is targeted at a wide audience, so it naturally may leave
people with questions. I'm happy to answer and provide more detail in this
forum.

I'd also like to thank everyone for your tireless efforts on this protocol
over the last few years. You're a remarkable group of people and I hope
this serves as one among many votes of confidence in QUIC as the future of
Internet transports.

Thanks,
Matt Joras

---

From: Ian Swett <[email-masked]>
Date: Thu, 22 Oct 2020 02:30:27 -0400

Thanks for the update Matt, that's quite a success story.

You note that you're not currently using 0-RTT in QUIC.  Can you share
why?  Chrome won't have 0-RTT enabled in Stable until M87(November), so we
don't have reliable metrics yet, but we're expecting improvements.

Thanks, Ian

On Wed, Oct 21, 2020 at 12:31 PM Matt Joras <[email-masked]> wrote:

> - The baseline HTTP for the apps is either HTTP/2 (for the Facebook app)
> or HTTP/1.1 (for Instagram).

---

From: Matt Joras <[email-masked]>
Date: Thu, 22 Oct 2020 10:39:36 -0700

We've had 0-RTT experiments on the Internet as long as we've had QUIC
traffic on the Internet. The somewhat awkward truth is that they
consistently show regressions relative to 1-RTT QUIC. We've had difficulty
figuring out why, as it seems to be functioning well for our internal
network traffic where we dogfood both HTTP/3 and a non-HTTP protocol. Since
it didn't just work out of the box we prioritized going "wide" with 1-RTT
QUIC first, and so 0-RTT hasn't gotten much attention. I highly suspect our
0-RTT problem is mvfst-specific, and it's in our plan to get that fixed
next, followed by finishing connection migration.

If Chrome + Facebook servers see 0-RTT regressions when you turn it on,
that would be a good signal it's a server-side problem, and in that case we
can always disable it for non-FB clients until it's fixed.

Matt Joras

On Wed, Oct 21, 2020, 11:30 PM Ian Swett <[email-masked]> wrote:

>> I'm excited to finally publicly share some of the work we've been doing
>> on QUIC in recent years. You can read about it on the Facebook Engineering
>> blog here
>> I'd also like to thank everyone for your tireless efforts on this
>> protocol over the last few years. You're a remarkable group of people and I
>> hope this serves as one among many votes of confidence in QUIC as the
>> future of Internet transports.
>>
>> Thanks,
>> Matt Joras
>>
>

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 23 Oct 2020 19:02:14 -0700

Thanks for this update, Matt, this is super exciting to hear about!
I loved reading about this: "What was most puzzling was that, despite QUIC
being enabled only for dynamic requests, we observed increased error rates
for static content downloaded with TCP."

Nicely done, both on the deployment and on the post!
- jana

On Thu, Oct 22, 2020 at 10:41 AM Matt Joras <[email-masked]> wrote:

>>> - QUIC is enabled for all content in both apps, and is utilized for
>>> media upload as well.
>>> - Our TCP baseline uses TLS 1.3 and early_data, with the vast majority
>>> of connections resuming.
>>> The linked post is targeted at a wide audience, so it naturally may
>>> leave people with questions. I'm happy to answer and provide more detail in
>>> this forum.
>>>

---

From: Dirkjan Ochtman <[email-masked]>
Date: Sun, 25 Oct 2020 22:11:04 +0100

On Wed, Oct 21, 2020 at 6:31 PM Matt Joras <[email-masked]> wrote:
> The linked post is targeted at a wide audience, so it naturally may leave people with questions. I'm happy to answer and provide more detail in this forum.

I was curious about this line:

> We also optimized the data structures and algorithms that handle unacknowledged QUIC data.

Any high-level insights on the work you did here/data structures and
algorithms that improved performance?

Kind regards,

Dirkjan