Subject: [nvo3] Fwd: New Version Notification for draft-herbert-vxlan-rco-00.txt

From: Tom Herbert <[email-masked]>
Date: Tue, 2 Dec 2014 09:47:38 -0800

This is a proposal for remote checksum offload in VXLAN. We use a
compressed format for the remote checksum offload data to support
offload of UDP and TCP inner checksums. This allocates a reserved bit
(#10 which would be immediately after version bits in VXLAN-GPE) and
uses the 8 bits after VNI for data.

I have implemented this in Linux stack and posted a first cut on
netdev (http://permalink.gmane.org/gmane.linux.network/339847). It
works as expected, eliminating the need to perform host checksum
computation over a packet when encapsulating TCP over VXLAN between
hosts using plain NICs.  In simple throughput tests this reduced CPU
usage by ~20% on transmit, and using outer UDP checksum reduces CPU on
receive by almost 50%.

Comments are appreciated.

Thanks,
Tom

---------- Forwarded message ----------
From:  <[email-masked]>
Date: Tue, Dec 2, 2014 at 8:13 AM
Subject: New Version Notification for draft-herbert-vxlan-rco-00.txt
To: Tom Herbert <[email-masked]>

A new version of I-D, draft-herbert-vxlan-rco-00.txt
has been successfully submitted by Tom Herbert and posted to the
IETF repository.

Name:           draft-herbert-vxlan-rco
Revision:       00
Title:          Remote checksum offload for VXLAN
Document date:  2014-12-01
Group:          Individual Submission
Pages:          6
URL:
http://www.ietf.org/internet-drafts/draft-herbert-vxlan-rco-00.txt
Status:         https://datatracker.ietf.org/doc/draft-herbert-vxlan-rco/
Htmlized:       http://tools.ietf.org/html/draft-herbert-vxlan-rco-00

Abstract:
   This specification describes remote checksum offload for VXLAN.
   Remote checksum offload is a mechanism that provides checksum offload
   of transport checksums in encapsulated packets using rudimentary
   offload capabilities found in most Network Interface Card (NIC)
   devices. The outer UDP checksum is enabled on transmit and, with some
   additional meta data, a receiver is able to deduce the checksum to be
   set in an encapsulated packet. Effectively this offloads the
   computation of the inner checksum which can be a significant
   performance optimization. Enabling the UDP checksum has the
   additional advantage that it covers more of the packet including the
   IP pseudo header and virtual network identifier.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Tue, 2 Dec 2014 20:34:47 +0000

setting it to 0 will reduce it by 100%

-D
On Dec 2, 2014, at 9:47 AM, Tom Herbert <[email-masked]> wrote:

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 2 Dec 2014 17:15:56 -0800

Hi Tom,

It looks like you're using 1's complement math, so you may find this
helpful:
https://tools.ietf.org/html/rfc1624

Anoop

---

From: Tom Herbert <[email-masked]>
Date: Mon, 8 Dec 2014 08:20:10 -0800

> setting it to 0 will reduce it by 100%
I assume you mean setting the outer UDP checksum to zero. The purpose
of remote checksum offload is to offload an inner encapsulated
checksum. For example, to offload the TCP checksum in an
IP/UDP/VXLAN/Eth/IP/TCP encapsulation. Many commonly deployed NICs do
not support checksum offload for such an encapsulation, which means if
a host is sourcing or receiving encapsulated TCP packets they will
need to compute TCP checksum in the CPU-- this is an expensive per
byte cost.

Remote checksum offload addresses this by enabling the outer UDP
checksum and inferring the value of the inner checksum based on that
on receive. The ability to do checksum offload for UDP/IP packets is
pretty ubiquitous in deployed NICs. With UDP checksum enabled and
remote checksum offload we generally see much better performance in
host to host encapsulation than setting UDP checksum to zero.

As I also mentioned in the draft, enabling the UDP checksum covers
more of the packet including IP pseudo header and virtual network ID.
Covering the pseudo header is especially important in IPv6 since there
is no header checksum in IPv6 (there is ongoing discussion about the
importance of UDP checksum in tunneling with IPv6).

Thanks,
Tom

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 9 Dec 2014 06:22:41 +0000

> -----Original Message-----
> From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
> Sent: Monday, December 8, 2014 9:50 PM
> To: Darrel Lewis (darlewis)
> Cc: [email-masked]
> Subject: Re: [nvo3] New Version Notification for draft-herbert-vxlan-rco-
> 00.txt
> 
> I assume you mean setting the outer UDP checksum to zero. The purpose of
> remote checksum offload is to offload an inner encapsulated checksum. For
> example, to offload the TCP checksum in an IP/UDP/VXLAN/Eth/IP/TCP
> encapsulation. Many commonly deployed NICs do not support checksum
> offload for such an encapsulation, which means if a host is sourcing or
> receiving encapsulated TCP packets they will need to compute TCP checksum
> in the CPU-- this is an expensive per byte cost.

[PG] Legacy NICs don't support large send offload or VMQ/VMDq/NetQueue
for encapsulated packets either and those are far more important from performance
and/or CPU overhead reduction perspective than checksum offload. I am not sure if it
is practical to even use NICs in network virtualization scenarios without support
of these offloads, given how poor the performance is, at least in 10G environment
(which should be common in datacenters). So I am not sure how much value there is
In optimizing for this.

> checksum and inferring the value of the inner checksum based on that on
> receive. The ability to do checksum offload for UDP/IP packets is pretty
> ubiquitous in deployed NICs. With UDP checksum enabled and remote
> checksum offload we generally see much better performance in host to host
> encapsulation than setting UDP checksum to zero.
> 
> As I also mentioned in the draft, enabling the UDP checksum covers more of
> the packet including IP pseudo header and virtual network ID.
> Covering the pseudo header is especially important in IPv6 since there is no
> header checksum in IPv6 (there is ongoing discussion about the importance
> of UDP checksum in tunneling with IPv6).
> 
> Thanks,
> Tom
> 
> > -D
> >> usage by ~20% on transmit, and using outer UDP checksum reduces CPU
> >> on receive by almost 50%.
> >>
> >> A new version of I-D, draft-herbert-vxlan-rco-00.txt has been
> >> successfully submitted by Tom Herbert and posted to the IETF
> >> repository.
> >>
> >> Please note that it may take a couple of minutes from the time of
> >> submission until the htmlized version and diff are available at
> tools.ietf.org.
> >>
> >> The IETF Secretariat
> >>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Dec 2014 08:00:58 -0800

> of these offloads, given how poor the performance is, at least in 10G environment

Hi Pankaj,

>From my experience it is practical. We already have a large scaled
deployment that does not use any specialized NIC virtualization
features. It would be an enormous cost to upgrade all of the NICs to
support virtualization (e.g. host bypass like you mentioned), and even
if we wanted to do that I don't believe there are currently any
available that would meet our security and control requirements, nor
do we have an IETF protocol standard for nvo3 encapsulation so when
that materializes I worry that we'd need change again. If we can
leverage the existing features in legacy NICs and otherwise optimize
the data path, performance actually can come pretty close to that of
native networking.

I suspect for a long time there will be other people in the same boat
in running virtualization in their DC on legacy hardware. VXLAN is not
specifically a hardware protocol, so I think it's reasonable to
implement open techniques to make it work well as possible in these
environments.

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 9 Dec 2014 18:22:04 +0000

Inline.

> From: Tom Herbert [mailto:[email-masked]]
> Sent: Tuesday, December 9, 2014 9:31 PM
> To: Pankaj Garg
> Cc: Darrel Lewis (darlewis); [email-masked]
> >> Subject: Re: [nvo3] New Version Notification for
> >> draft-herbert-vxlan-rco- 00.txt
> >>
> >> not support checksum offload for such an encapsulation, which means
> >> if a host is sourcing or receiving encapsulated TCP packets they will
> >> need to compute TCP checksum in the CPU-- this is an expensive per byte
> cost.
> >
> > for encapsulated packets either and those are far more important from
> > performance and/or CPU overhead reduction perspective than checksum
> > offload. I am not sure if it is practical to even use NICs in network
> > virtualization scenarios without support of these offloads, given how
> > poor the performance is, at least in 10G environment
> 
> Hi Pankaj,
> 
> deployment that does not use any specialized NIC virtualization features. It
> would be an enormous cost to upgrade all of the NICs to support
> virtualization (e.g. host bypass like you mentioned), and even if we wanted
> to do that I don't believe there are currently any available that would meet
> our security and control requirements, nor do we have an IETF protocol
> standard for nvo3 encapsulation so when that materializes I worry that we'd
> need change again. If we can leverage the existing features in legacy NICs
> and otherwise optimize the data path, performance actually can come pretty
> close to that of native networking.

[PG] My point was that I am not sure if just reducing checksum overhead would
be sufficient to make network virtualization performance acceptable to most people.
In my experience with Hyper-V in both Azure and private clouds, it wasn't acceptable,
since the performance penalty was just too high and we had to "opportunistically" use 
other techniques like NAT with legacy NICs to simulate "network virtualization" with 
acceptable performance.

If legacy NICs are important, then I feel it would be better to think of an overlay solution
that can work with such NICs and still provide reasonable performance e.g. something like
STT, maybe? As I am not sure if the gains we get with remove checksum offload moves
the needle much (without LSO and VMQ) but then, if you _have_ a reason to use VXLAN
with legacy NICs, then yes, it would save some CPU cycles.

> 
> I suspect for a long time there will be other people in the same boat in
> running virtualization in their DC on legacy hardware. VXLAN is not specifically
> a hardware protocol, so I think it's reasonable to implement open techniques
> to make it work well as possible in these environments.
> 
> Tom
> 
> > (which should be common in datacenters). So I am not sure how much
> > value there is In optimizing for this.
> >
> >>
> >> Covering the pseudo header is especially important in IPv6 since
> >> there is no header checksum in IPv6 (there is ongoing discussion
> >> about the importance of UDP checksum in tunneling with IPv6).
> >>
> >> Thanks,
> >> Tom
> >>
> >> > -D
> >> >> (#10 which would be immediately after version bits in VXLAN-GPE)
> >> >> and uses the 8 bits after VNI for data.
> >> >>
> >> >> computation over a packet when encapsulating TCP over VXLAN
> >> >> between hosts using plain NICs.  In simple throughput tests this
> >> >> reduced CPU usage by ~20% on transmit, and using outer UDP
> >> >> checksum reduces CPU on receive by almost 50%.
> >> >>