Subject: UDP send costs in Linux

From: Ian Swett <[email-masked]>
Date: Wed, 04 Apr 2018 00:20:08 +0000

One challenge with QUIC at the moment is the increased CPU cost of sending
UDP packets vs TCP payloads.  I've seen this across every platform Google
has deployed QUIC on, so it's a widespread issue.

Here's an excellent presentation on what's causing the increased CPU
consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf

And while you're thinking of CPU usage, it's worth looking at the
presentation on timing wheel based packet pacing(which is minimum release
time based) and is ideal for QUIC(and TCP for that matter):
https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel.pdf

-Ian

---

From: Subodh Iyengar <[email-masked]>
Date: Wed, 4 Apr 2018 02:50:31 +0000

Thanks for sharing this Ian.

This definitely matches some of the observations we've seen as well in the UDP write path. Some of the other paths that we saw that added overhead was the route table lookup in linux udp stack. Connected UDP sockets did amortize that.

I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there any indication of the timeline for these patches to make it to linux? Would be happy to try any of these out to help iron out the API.

Subodh

________________________________
From: QUIC <[email-masked]> on behalf of Ian Swett <ianswett=[email-masked]>
Sent: Tuesday, April 3, 2018 5:20:08 PM
To: IETF QUIC WG
Subject: UDP send costs in Linux

One challenge with QUIC at the moment is the increased CPU cost of sending UDP packets vs TCP payloads.  I've seen this across every platform Google has deployed QUIC on, so it's a widespread issue.

Here's an excellent presentation on what's causing the increased CPU consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf

And while you're thinking of CPU usage, it's worth looking at the presentation on timing wheel based packet pacing(which is minimum release time based) and is ideal for QUIC(and TCP for that matter): https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=tgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE&s=87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY&e=>

-Ian

---

From: Subodh Iyengar <[email-masked]>
Date: Wed, 4 Apr 2018 02:50:31 +0000

Subodh

-Ian

---

From: Ian Swett <[email-masked]>
Date: Wed, 04 Apr 2018 13:10:53 +0000

I hope some of these patches will be available soon, but I'm not sure if
soon is a month or 6.

On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <[email-masked]> wrote:

> This definitely matches some of the observations we've seen as well in the
> UDP write path. Some of the other paths that we saw that added overhead was
> the route table lookup in linux udp stack. Connected UDP sockets did
> amortize that.
>
>
> I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there
> any indication of the timeline for these patches to make it to linux? Would
> be happy to try any of these out to help iron out the API.
>
>
> Subodh
> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Ian Swett
> <ianswett=[email-masked]..org>
> *Sent:* Tuesday, April 3, 2018 5:20:08 PM
> *To:* IETF QUIC WG
> *Subject:* UDP send costs in Linux
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=tgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE&s=87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY&e=>
>
> -Ian
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 08:32:58 -0700

I have no data to add on the Linux UDP stack, but another issue is the lack
of netmap support in cloud hosting environments.
I have not yet been working with this, but have looked into the problem and
asked around.

netmap is default in FreeBSD and optional in Linux. But neither works
efficiently without a hypervisor patch that is also available for netmap.
With support for netmap, the user space application can send directly to
the network adapter with very little overhead. There is also dpdk and some
other interfaces that might be slightly faster but more vendor specific.

Assuming an application has access to optimized netmap, the only hurdle is
address lookup, but if the application also manages that, or at least does
the caching, there shouldn’t be much in the way of OS interference.

Of course, netmap blocks the entire network stack, so no PING or SSH.
CloudFlare added a netmap patch so only some traffic would be routed fra
the network interface to netmap, and netmap also supports efficient packet
forwarding to the OS or other applications.

None of this works well in general, but for a cloud host that can be bootet
automatically and destroyed rather than serviced, there is some opportunity.

but only if cloud service providers starts adding support their supported
images and hypervisors. Not sure if any are working on this now.

Mikkel

On 4 April 2018 at 15.11.17, Ian Swett (ianswett=[email-masked])
wrote:

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 08:32:58 -0700

None of this works well in general, but for a cloud host that can be bootet
automatically and destroyed rather than serviced, there is some opportunity..

Mikkel

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Apr 2018 12:28:47 -0400

I would not worry too much at this point.

The reason we want to be able to work at the application level is backwards
compatibility. It has to be possible to deploy QUIC on any machine even
without OS support or it won't be deployable.

It does not have to be performant on every platform. If people are using
QUIC, whatever needs to be moved into the kernel for performance reasons
will move there.

On Wed, Apr 4, 2018 at 11:32 AM, Mikkel Fahnøe Jørgensen <[email-masked]

> I have no data to add on the Linux UDP stack, but another issue is the
> lack of netmap support in cloud hosting environments.
> I have not yet been working with this, but have looked into the problem
> and asked around.
>
> None of this works well in general, but for a cloud host that can be
> bootet automatically and destroyed rather than serviced, there is some
> opportunity.
>
> On 4 April 2018 at 15.11.17, Ian Swett (ianswett=40google.com@dmarc.
> ietf.org) wrote:
>
>> This definitely matches some of the observations we've seen as well in
>> the UDP write path. Some of the other paths that we saw that added overhead
>> was the route table lookup in linux udp stack. Connected UDP sockets did
>> amortize that.
>>
>>
>> I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is
>> there any indication of the timeline for these patches to make it to linux?
>> Would be happy to try any of these out to help iron out the API.
>>
>>
>> Subodh
>> One challenge with QUIC at the moment is the increased CPU cost of
>> sending UDP packets vs TCP payloads.  I've seen this across every platform
>> Google has deployed QUIC on, so it's a widespread issue.
>>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Apr 2018 12:28:47 -0400

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 17:57:22 +0000

The need to worry is not on the client side – at least not immediately. 802.11ad, 802.11ax, and 5G LTE will bring Gigabit rates to client side so it is certainly going to become a problem longer term.

The current worry is on the server side. A large part of the work we do is performance optimization of the network stack. Google’s Sigcomm QUIC paper shows a 2x CPU increase going from TCP to QUIC. Most web services will NOT be able to make that trade off to get latency improvements for 90th percentile.

IMO improving UDP performance and hardware offloads are absolutely a necessity for QUIC to become widely adopted (not just by the biggest corporations with a large budget). We have work under way on both these fronts and I am very happy to see Linux also investing here in preparation for what’s coming. We need to do our best democratize this technology - multiple implementations and performance will both play a big role.

From: QUIC [mailto:[email-masked]] On Behalf Of Phillip Hallam-Baker
Sent: Wednesday, April 4, 2018 9:29 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Subodh Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>; Ian Swett <ianswett=[email-masked]>
Subject: Re: UDP send costs in Linux

The reason we want to be able to work at the application level is backwards compatibility. It has to be possible to deploy QUIC on any machine even without OS support or it won't be deployable.

It does not have to be performant on every platform. If people are using QUIC, whatever needs to be moved into the kernel for performance reasons will move there.

On Wed, Apr 4, 2018 at 11:32 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
I have no data to add on the Linux UDP stack, but another issue is the lack of netmap support in cloud hosting environments.
I have not yet been working with this, but have looked into the problem and asked around.

netmap is default in FreeBSD and optional in Linux. But neither works efficiently without a hypervisor patch that is also available for netmap. With support for netmap, the user space application can send directly to the network adapter with very little overhead. There is also dpdk and some other interfaces that might be slightly faster but more vendor specific.

Assuming an application has access to optimized netmap, the only hurdle is address lookup, but if the application also manages that, or at least does the caching, there shouldn’t be much in the way of OS interference.

Of course, netmap blocks the entire network stack, so no PING or SSH. CloudFlare added a netmap patch so only some traffic would be routed fra the network interface to netmap, and netmap also supports efficient packet forwarding to the OS or other applications.

None of this works well in general, but for a cloud host that can be bootet automatically and destroyed rather than serviced, there is some opportunity.

but only if cloud service providers starts adding support their supported images and hypervisors. Not sure if any are working on this now.

Mikkel

On 4 April 2018 at 15.11.17, Ian Swett (ianswett=[email-masked]<mailto:ianswett=[email-masked]>) wrote:
I hope some of these patches will be available soon, but I'm not sure if soon is a month or 6.

On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <[email-masked]<mailto:[email-masked]>> wrote:

Subodh

________________________________
From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Ian Swett <ianswett=[email-masked]..org<mailto:ianswett=[email-masked]..org>>
Sent: Tuesday, April 3, 2018 5:20:08 PM
To: IETF QUIC WG
Subject: UDP send costs in Linux

Here's an excellent presentation on what's causing the increased CPU consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.kernel.org%2Fnetconf2017_files%2Frx_hardening_and_udp_gso.pdf&data=02%7C01%7Cpravb%40microsoft.com%7C103a6e81f3684c67382608d59a492c2e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636584561449872935&sdata=FPe0mS5aG%2B88%2FSXvIVSO3yWnXh9tOhQfKkH6XnQDxvE%3D&reserved=0>

And while you're thinking of CPU usage, it's worth looking at the presentation on timing wheel based packet pacing(which is minimum release time based) and is ideal for QUIC(and TCP for that matter): https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel..pdf<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf%26d%3DDwMFaQ%26c%3D5VD0RTtNlTh3ycd41b3MUw%26r%3Dh3Ju9EBS7mHtwg-wAyN7fQ%26m%3DtgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE%26s%3D87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY%26e%3D&data=02%7C01%7Cpravb%40microsoft.com%7C103a6e81f3684c67382608d59a492c2e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584561449872935&sdata=hSXXvTODvljI6niBLmMxGUFS9g%2FqOiHskUcNcgHtE6k%3D&reserved=0>

-Ian

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 17:57:22 +0000

Mikkel

Subodh

-Ian

---

From: Frederick Kautz <[email-masked]>
Date: Wed, 4 Apr 2018 11:25:06 -0700

I apologize to anyone here who gets a repeat of this message, the mailing
list ate my earlier reply because my mail provider changed my default email
address during a transition. Here is the original message:

I agree with this sentiment. If we see traction, then we should see better
kernel support and offloading.

It should be trivial to get this working in something like VPP which can
use DPDK to hw offload, entirely skipping the kernel in the data path.

There are also techniques to bypass the initial memory allocation in the
kernel such as ebpf used by cilium and memif used by vpp.

My main concern at this point would be with cloud native environments with
limited hw offloading support. E.g. if we run in AWS or GCE, better kernel
support will probably be necessary.

On Wed, Apr 4, 2018 at 10:57 AM, Praveen Balasubramanian <
pravb=[email-masked]> wrote:

> The need to worry is not on the client side – at least not immediately.
> 802.11ad, 802.11ax, and 5G LTE will bring Gigabit rates to client side so
> it is certainly going to become a problem longer term.
>
>
>
> The current worry is on the server side. A large part of the work we do is
> performance optimization of the network stack. Google’s Sigcomm QUIC paper
> shows a 2x CPU increase going from TCP to QUIC. Most web services will NOT
> be able to make that trade off to get latency improvements for 90th
> percentile.
>
>
>
> IMO *improving UDP performance and hardware offloads are absolutely a
> necessity for QUIC to become widely adopted* (not just by the
> biggest corporations with a large budget). We have work under way on both
> these fronts and I am very happy to see Linux also investing here in
> preparation for what’s coming. We need to do our best democratize this
> technology - multiple implementations and performance will both play a big
> role.
>
>
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Phillip
> Hallam-Baker
> *Sent:* Wednesday, April 4, 2018 9:29 AM
> *To:* Mikkel Fahnøe Jørgensen <[email-masked]>
> *Cc:* Subodh Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>; Ian
> Swett <ianswett=[email-masked]>
> *Subject:* Re: UDP send costs in Linux
>
>
>
> The reason we want to be able to work at the application level is
> backwards compatibility. It has to be possible to deploy QUIC on any
> machine even without OS support or it won't be deployable.
>
>
>
> On Wed, Apr 4, 2018 at 11:32 AM, Mikkel Fahnøe Jørgensen <
> [email-masked]> wrote:
>
> *From:* QUIC <[email-masked]> on behalf of Ian Swett <
> ianswett=[email-masked]..org>
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.kernel.org%2Fnetconf2017_files%2Frx_hardening_and_udp_gso.pdf&data=02%7C01%7Cpravb%40microsoft.com%7C103a6e81f3684c67382608d59a492c2e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636584561449872935&sdata=FPe0mS5aG%2B88%2FSXvIVSO3yWnXh9tOhQfKkH6XnQDxvE%3D&reserved=0>
>
>
>
> https://conferences.sigcomm.org/sigcomm/2017/
> files/program/ts-9-4-carousel..pdf
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf%26d%3DDwMFaQ%26c%3D5VD0RTtNlTh3ycd41b3MUw%26r%3Dh3Ju9EBS7mHtwg-wAyN7fQ%26m%3DtgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE%26s%3D87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY%26e%3D&data=02%7C01%7Cpravb%40microsoft.com%7C103a6e81f3684c67382608d59a492c2e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636584561449872935&sdata=hSXXvTODvljI6niBLmMxGUFS9g%2FqOiHskUcNcgHtE6k%3D&reserved=0>
>
>
>
> -Ian
>
>
>

---

From: Frederick Kautz <[email-masked]>
Date: Wed, 4 Apr 2018 11:25:06 -0700

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 20:31:53 +0200

Got it before, but the reason why netmap and DPDK is important in contrast
to kernel support is the very rich application interfaces that a custom
user space QUIC implementation provides. Spending the next 10 years
discussing epoll vs poll for QUIC is not going to cut it.

OS support that makes it easier to use netmap and friends concurrently with
other network traffic would be welcome though.
Question is what hardware offload support can be given to a custom QUIC
implementation, but one thing that comes to mind is pre-decryption on
netmap where the OS first intercepts the packet and forwards the decrypted
and DoS filtered packet to the app which then handles all the detailed
framing, ACK, pacing etc.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 4 April 2018 at 20.25.07, Frederick Kautz ([email-masked]) wrote:

On Wed, Apr 4, 2018 at 10:57 AM, Praveen Balasubramanian <
pravb=[email-masked]> wrote:

> *From:* QUIC [mailto:[email-masked]] *On Behalf Of* Phillip
> Hallam-Baker

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 4 Apr 2018 20:31:53 +0200

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 18:39:36 +0000

Hardware support is not orthogonal to DPDK and kernel bypass even user mode can take advantage of it. DPDK has a lot of other issues including no battle hardened TCPIP stacks. IMO we need to make progress on better performance regardless of user or kernel implementations. Especially for client devices user space implementations cannot be given direct hardware access for security reasons. For cloud VMs there is SR-IOV for direct hardware access which is now supported by more and more cloud providers.

From: QUIC [mailto:[email-masked]] On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Wednesday, April 4, 2018 11:32 AM
To: Frederick Kautz <[email-masked]>; Praveen Balasubramanian <pravb=[email-masked]>
Cc: Subodh Iyengar <[email-masked]>; Ian Swett <[email-masked]>; Phillip Hallam-Baker <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: UDP send costs in Linux

Got it before, but the reason why netmap and DPDK is important in contrast to kernel support is the very rich application interfaces that a custom user space QUIC implementation provides. Spending the next 10 years discussing epoll vs poll for QUIC is not going to cut it.

OS support that makes it easier to use netmap and friends concurrently with other network traffic would be welcome though.
Question is what hardware offload support can be given to a custom QUIC implementation, but one thing that comes to mind is pre-decryption on netmap where the OS first intercepts the packet and forwards the decrypted and DoS filtered packet to the app which then handles all the detailed framing, ACK, pacing etc.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 4 April 2018 at 20.25.07, Frederick Kautz ([email-masked]<mailto:[email-masked]>) wrote:
I apologize to anyone here who gets a repeat of this message, the mailing list ate my earlier reply because my mail provider changed my default email address during a transition. Here is the original message:

I agree with this sentiment. If we see traction, then we should see better kernel support and offloading.

It should be trivial to get this working in something like VPP which can use DPDK to hw offload, entirely skipping the kernel in the data path.

There are also techniques to bypass the initial memory allocation in the kernel such as ebpf used by cilium and memif used by vpp.

My main concern at this point would be with cloud native environments with limited hw offloading support. E.g. if we run in AWS or GCE, better kernel support will probably be necessary.
[https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif]

On Wed, Apr 4, 2018 at 10:57 AM, Praveen Balasubramanian <pravb=[email-masked]<mailto:pravb=[email-masked]>> wrote:
The need to worry is not on the client side – at least not immediately. 802.11ad, 802.11ax, and 5G LTE will bring Gigabit rates to client side so it is certainly going to become a problem longer term.

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Phillip Hallam-Baker
Sent: Wednesday, April 4, 2018 9:29 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Cc: Subodh Iyengar <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>>
Subject: Re: UDP send costs in Linux

Mikkel

On 4 April 2018 at 15.11.17, Ian Swett (ianswett=[email-masked]..org<mailto:ianswett=[email-masked]>) wrote:
I hope some of these patches will be available soon, but I'm not sure if soon is a month or 6.

Subodh

Here's an excellent presentation on what's causing the increased CPU consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf<https://na01.safelinks.protection.outlook..com/?url=http%3A%2F%2Fvger.kernel.org%2Fnetconf2017_files%2Frx_hardening_and_udp_gso.pdf&data=02%7C01%7Cpravb%40microsoft.com%7C103a6e81f3684c67382608d59a492c2e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636584561449872935&sdata=FPe0mS5aG%2B88%2FSXvIVSO3yWnXh9tOhQfKkH6XnQDxvE%3D&reserved=0>

-Ian

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 4 Apr 2018 18:39:36 +0000

Kind Regards,
Mikkel Fahnøe Jørgensen

Mikkel

Subodh

-Ian

---

From: Frederick Kautz <[email-masked]>
Date: Wed, 04 Apr 2018 16:45:41 +0000

On Wed, Apr 4, 2018, 9:28 AM Phillip Hallam-Baker <[email-masked]>
wrote:

>> Assuming an application has access to optimized netmap, the only hurdle
>> is address lookup, but if the application also manages that, or at least
>> does the caching, there shouldn’t be much in the way of OS interference.
>>
>> On 4 April 2018 at 15.11.17, Ian Swett (
>> ianswett=[email-masked]) wrote:
>>
>>> https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel..pdf