Subject: Re: Impact of hardware offloads on network stack performance

From: craigt <[email-masked]>
Date: Thu, 10 May 2018 13:50:11 +0100

Has anyone experimented with the zerocopy socket interfaces for exactly
this use case?
  https://netdevconf.org/2.1/papers/netdev.pdf

It would seem that is a more natural fit to the assertions made by several
of the authors surrounding packetisation.
I'm unaware if Windows/BSD have similar interfaces/implementations.

c

On 30 April 2018 at 21:37, Roberto Peon <[email-masked]> wrote:

> QUIC (mostly) doesn’t do that (rexmit a packet verbatim)—you’d need a new
> packet# for the rexmitted packet.
>
> -=R
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Watson Ladd <
> [email-masked]>
> *Date: *Monday, April 23, 2018 at 1:56 PM
> *To: *Mike Bishop <[email-masked]>
> *Cc: *Praveen Balasubramanian <[email-masked]>, "Lubashev, Igor" <
> [email-masked]>, Ian Swett <ianswett=[email-masked]>,
> Mikkel Fahnøe Jørgensen <[email-masked]>, IETF QUIC WG <[email-masked]>,
> "Deval, Manasi" <[email-masked]>
> *Subject: *Re: Impact of hardware offloads on network stack performance
>
>
>
>
>
>
>
> On Mon, Apr 23, 2018 at 1:06 PM, Mike Bishop <[email-masked]> wrote:
>
> I agree – the point of offloading the crypto is not only to spare that
> cost on the CPU, though that can be useful, it’s to enable further offloads
> that can’t be done on the encrypted packet.
>
>
>
> LRO that could simply group packets by CID and deliver a single batch of
> packets from a single CID in a single invocation seems like a great first
> step, and doesn’t even require crypto offload.  Once you have crypto
> offload, of course those packets per CID could be delivered already
> decrypted.  I’d want to see data on whether there’s value in making the
> received data look like one huge packet when there’s not a single
> bytestream, but that’s not something we need to specify here.
>
>
>
> LSO is more interesting, and the challenge there is that the transport
> stack still needs to know what data was in which packet so that it can do
> loss recovery properly.  QUIC explicitly disallows fragmentation of the UDP
> datagram, for the “painful and potentially non-deterministic” reasons you
> referenced.
>
>
>
> Forgive my complete ignorance, but doesn't it just need to know what
> packet it sent?
>
> More explicitly the sender would maintain a queue of sendable packets, and
> mark which have been sent. Acknowledged packets are removed from the queue.
> In event of loss, they simply retransmit the same packet. It's up to higher
> layers to pick the right size of packet and take data from the individual
> QUIC streams. If you wanted to expose the stream structure, that sounds a
> lot more complicated.
>
>
> Sincerely,
>
> Watson
>
>
>
>
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 10 May 2018 06:30:56 -0700

Thanks for the pointer.

I doubt you will see similar gains (92%) in UDP because there is going to
be a lot of kernel context switches, one for each packet vis sendmsg.
Copying is probably not the dominating factor here.

I do wish there was an easier way to use the netmap interface without
kernel patches and without taking over the entire network card, e.g. by
binding netmap to a single UDP port.

https://www.unix.com/man-page/freebsd/4/netmap/

Mikkel

On 10 May 2018 at 14.50.13, craigt ([email-masked]) wrote:

c

On 30 April 2018 at 21:37, Roberto Peon <[email-masked]> wrote:

> *From:* QUIC <[email-masked]> on behalf of Watson Ladd <
> [email-masked]>
> *Date:* Monday, April 23, 2018 at 1:56 PM
> *To:* Mike Bishop <[email-masked]>
> *Cc:* Praveen Balasubramanian <[email-masked]>, "Lubashev, Igor" <
> *Subject:* Re: Impact of hardware offloads on network stack performance
>
>
>
>
>
>
>

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 10 May 2018 13:33:31 +0000

On 2018-5-10, at 14:30, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> I do wish there was an easier way to use the netmap interface without kernel patches and without taking over the entire network card, e.g. by binding netmap to a single UDP port.

You can bind netmap to individual NIC rings.

Lars

---

From: "Deval, Manasi" <[email-masked]>
Date: Thu, 10 May 2018 13:38:24 +0000

The current socket scheme uses the kernel stack to implement the Ethernet – IP – UDP stack processing for QUIC packets. The AF_XDP provides a raw frame so one would need a policy on where to run the protocol stack processing.

Another comment on offloads -
Thanks for the insightful point of how crypto offload enables other offloads like segmentation. A segmentation offload for QUIC would need to increment packet numbers for each of the individual packets put on the wire. The transmitted packet numbers for a transmit scatter gather list would be deterministic. Therefore, this should not adversely impact any retransmits.

Thanks,
Manasi

From: craigt [mailto:[email-masked]]
Sent: Thursday, May 10, 2018 5:50 AM
To: Roberto Peon <[email-masked]>
Cc: Watson Ladd <[email-masked]>; Mike Bishop <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Lubashev, Igor <[email-masked]>; Ian Swett <ianswett=[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; IETF QUIC WG <[email-masked]>; Deval, Manasi <[email-masked]>
Subject: Re: Impact of hardware offloads on network stack performance

Has anyone experimented with the zerocopy socket interfaces for exactly this use case?
  https://netdevconf.org/2.1/papers/netdev.pdf

It would seem that is a more natural fit to the assertions made by several of the authors surrounding packetisation.
I'm unaware if Windows/BSD have similar interfaces/implementations.

c

On 30 April 2018 at 21:37, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
QUIC (mostly) doesn’t do that (rexmit a packet verbatim)—you’d need a new packet# for the rexmitted packet.
-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Watson Ladd <[email-masked]<mailto:[email-masked]>>
Date: Monday, April 23, 2018 at 1:56 PM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Cc: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>, "Lubashev, Igor" <[email-masked]<mailto:[email-masked]>>, Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>>, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>, "Deval, Manasi" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Impact of hardware offloads on network stack performance

On Mon, Apr 23, 2018 at 1:06 PM, Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
I agree – the point of offloading the crypto is not only to spare that cost on the CPU, though that can be useful, it’s to enable further offloads that can’t be done on the encrypted packet.

LRO that could simply group packets by CID and deliver a single batch of packets from a single CID in a single invocation seems like a great first step, and doesn’t even require crypto offload.  Once you have crypto offload, of course those packets per CID could be delivered already decrypted.  I’d want to see data on whether there’s value in making the received data look like one huge packet when there’s not a single bytestream, but that’s not something we need to specify here.

LSO is more interesting, and the challenge there is that the transport stack still needs to know what data was in which packet so that it can do loss recovery properly.  QUIC explicitly disallows fragmentation of the UDP datagram, for the “painful and potentially non-deterministic” reasons you referenced.

Forgive my complete ignorance, but doesn't it just need to know what packet it sent?

More explicitly the sender would maintain a queue of sendable packets, and mark which have been sent. Acknowledged packets are removed from the queue. In event of loss, they simply retransmit the same packet. It's up to higher layers to pick the right size of packet and take data from the individual QUIC streams. If you wanted to expose the stream structure, that sounds a lot more complicated.

Sincerely,
Watson

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 10 May 2018 06:46:01 -0700

On 10 May 2018 at 15.33.58, Eggert, Lars ([email-masked]) wrote:

Yes, but is is not exactly simple since deployment gets rather elaborate.

---

From: Boris Pismenny <[email-masked]>
Date: Thu, 10 May 2018 16:57:29 +0300

On 5/10/2018 4:46 PM, Mikkel Fahnøe Jørgensen wrote:
> On 10 May 2018 at 15..33.58, Eggert, Lars ([email-masked] 
> <mailto:[email-masked]>) wrote:

It is possible to use DPDK without taking over the entire network card:
https://www.slideshare.net/LF_DPDK/lfdpdkmellanox-bifurcated-driver-model

Also, is there a reason to prefer netmap over other userspace network 
stacks like DPDK/VPP/etc?

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 10 May 2018 14:03:24 +0000

On 2018-5-10, at 14:57, Boris Pismenny <[email-masked]> wrote:
> Also, is there a reason to prefer netmap over other userspace network stacks like DPDK/VPP/etc?

For me personally, the netmap API is a lot more minimal and therefore easier to integrate. DPDK really wants to take over scheduling, memory management and all kinds of other stuff besides packet I/O. YMMV.

Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 10 May 2018 07:57:35 -0700

Netmap is has a fairly low line count and complexity for what it achieves,
it is not tied to any vendor specific hardware, it patches do seem to
arrive for various cards.

I recently did a user space emulation of netmap where separate user threads
drains and fills the tx and rx rings respectively. I’m not sure this is
worth it in terms of performance since it adds more synchronization (netmap
makes certain assumptions because some updates are in the kernel). However,
it does provide a uniform buffering layer and it makes it easier to plug in
proper netmap when available. It does move some per message latency out of
the way.

On 10 May 2018 at 16.03.28, Eggert, Lars ([email-masked]) wrote:

For me personally, the netmap API is a lot more minimal and therefore
easier to integrate. DPDK really wants to take over scheduling, memory
management and all kinds of other stuff besides packet I/O. YMMV.

Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 10 May 2018 08:02:20 -0700

Some of you probably have inside knowledge of this:

https://www.bbc.co.uk/rd/blog/2018-04-high-speed-networking-open-source-kernel-bypass

On 10 May 2018 at 16.57.35, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Lars

---

From: craigt <[email-masked]>
Date: Thu, 10 May 2018 16:39:36 +0100

The BBC folk involved in that work aren't on this list. If there are
questions about the implementation you would like answered, get in touch
with me and I'll try to get some time.

c

On 10 May 2018 at 16:02, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:

> I recently did a user space emulation of netmap where separate user
> threads drains and fills the tx and rx rings respectively. I’m not sure
> this is worth it in terms of performance since it adds more synchronization
> (netmap makes certain assumptions because some updates are in the kernel).
> However, it does provide a uniform buffering layer and it makes it easier
> to plug in proper netmap when available. It does move some per message
> latency out of the way.
>
>