Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

From: Lizhong Jin <[email-masked]>
Date: Tue, 24 Sep 2013 21:58:42 +0800

---------- Forwarded message ----------
From: Lucy yong <[email-masked]>
To: Jeff Wheeler <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Date: Mon, 23 Sep 2013 17:06:43 +0000
Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks
Jeff,

I agree that supporting LAG/ECMP at the NIC->host interface is the issue
for both VXLAN/NVGRE. However if NVGRE uses gre-in-udp, it will make future
NIC enhancement for this capability much simpler since both encapsulations
use udp src port for the flow entropy that is used by LB.  Why do you think
that inner payload inspection is necessary for ECMP LB?

[Lizhong] the NIC LB is quite different with the router/switch. The NIC LB
is called flow steering, and it does not rely on a hash value to steer
traffic, but rely on a flow table to steer traffic to a specific CPU core.
Usually the key of flow table is the 4 tuple of payload, src/dst IP add and
UDP/TCP port. That's why the NIC always has to parse payload header. What I
have to point out is, how to steer the VXLAN/NVGRE traffic is still under
development, and maybe different with the one I described. Hope the
explanation helps.

Regards
Lizhong

Thank,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jeff Wheeler
Sent: Sunday, September 22, 2013 1:13 PM
Cc: [email-masked]
Subject: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

On Sat, Sep 21, 2013 at 11:57 AM, Lucy yong <[email-masked]> wrote:
> *         Requirement: For performance reasons, multipath over LAG and
ECMP
> paths SHOULD be supported.
>
> VXLAN supports common five tuple based LB. NVGRE requests LB to use
> GRE header, which is not commonly supported by underlying IP network.

I'd like to note that it is becoming increasingly common for the underlying
L3 or L2 network to support load-balancing based on GRE inner-header
fields.  In fact, this can be largely relied on for high-bandwidth GRE
tunnels over the public Internet (!) today, and within many datacenter
networks.

Where this is not so reliable is in the NIC->Host interface.  This is an
issue facing NVGRE, and in some circumstance, VXLAN also.

For example, the most popular 10GE server NIC chipsets will deliver all
NVGRE packets to a host DMA ring buffer based on the GRE outer-header
only.  One example is the Intel 82599, which everyone must be familiar with
to have an informed opinion on vRouter-related topics.  Even if the NIC is
configured with several DMA rings, and the vRouter is able to use different
CPU cores to service those rings for distribution of the work, all NVGRE
traffic (having same outer-header) will arrive at only one DMA ring, and
other CPU cores may not be utilized.

If vRouters are used to support network-heavy applications, NIC vendors
must be encouraged to support additional header inspection which may then
be used for load-balancing across host DMA rings.  One would hope same
vendors will eventually implement hardware offload of VXLAN/NVGRE entirely,
and these two requirements share a common underlying need for the NIC to
understand the VXLAN/NVGRE encapsulation.  In other words, if a NIC vendor
plans to support deeper header inspection for load-balancing, they have
already done some of the work needed to support hardware offload.

This list is very focused on procedural items and support in the networks;
but nearly zero attention is paid to NIC->Host interface issues.  For
overlay technologies to deliver acceptable performance for network-heavy
workloads, NIC->Host interface must be more intelligent than it is today.
Next-generation NICs must implement new capabilities.  It is therefore
useful to consider NICs in any discussion of load-balancing across the
datacenter network, if for no other reason than to foster greater
understanding of this challenge.
--
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts
_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Sep 2013 14:11:43 +0000

Lizhong,

Thank you very much for the detail explanation.

The explanation further confirms that using gre-in-udp for NVGRE will simplify NIC LB implementation. It requires no change on the flow table structure.

Thanks,
Lucy

From: Lizhong Jin [mailto:[email-masked]]
Sent: Tuesday, September 24, 2013 8:59 AM
To: Lucy yong
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

---------- Forwarded message ----------
From: Lucy yong <[email-masked]<mailto:[email-masked]>>
To: Jeff Wheeler <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date: Mon, 23 Sep 2013 17:06:43 +0000
Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks
Jeff,

I agree that supporting LAG/ECMP at the NIC->host interface is the issue for both VXLAN/NVGRE. However if NVGRE uses gre-in-udp, it will make future NIC enhancement for this capability much simpler since both encapsulations use udp src port for the flow entropy that is used by LB.  Why do you think that inner payload inspection is necessary for ECMP LB?

[Lizhong] the NIC LB is quite different with the router/switch. The NIC LB is called flow steering, and it does not rely on a hash value to steer traffic, but rely on a flow table to steer traffic to a specific CPU core. Usually the key of flow table is the 4 tuple of payload, src/dst IP add and UDP/TCP port. That's why the NIC always has to parse payload header. What I have to point out is, how to steer the VXLAN/NVGRE traffic is still under development, and maybe different with the one I described. Hope the explanation helps.

Regards
Lizhong

Thank,
Lucy

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jeff Wheeler
Sent: Sunday, September 22, 2013 1:13 PM
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

On Sat, Sep 21, 2013 at 11:57 AM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
> *         Requirement: For performance reasons, multipath over LAG and ECMP

I'd like to note that it is becoming increasingly common for the underlying L3 or L2 network to support load-balancing based on GRE inner-header fields.  In fact, this can be largely relied on for high-bandwidth GRE tunnels over the public Internet (!) today, and within many datacenter networks.

Where this is not so reliable is in the NIC->Host interface.  This is an issue facing NVGRE, and in some circumstance, VXLAN also.

For example, the most popular 10GE server NIC chipsets will deliver all NVGRE packets to a host DMA ring buffer based on the GRE outer-header only.  One example is the Intel 82599, which everyone must be familiar with to have an informed opinion on vRouter-related topics.  Even if the NIC is configured with several DMA rings, and the vRouter is able to use different CPU cores to service those rings for distribution of the work, all NVGRE traffic (having same outer-header) will arrive at only one DMA ring, and other CPU cores may not be utilized.

If vRouters are used to support network-heavy applications, NIC vendors must be encouraged to support additional header inspection which may then be used for load-balancing across host DMA rings.  One would hope same vendors will eventually implement hardware offload of VXLAN/NVGRE entirely, and these two requirements share a common underlying need for the NIC to understand the VXLAN/NVGRE encapsulation.  In other words, if a NIC vendor plans to support deeper header inspection for load-balancing, they have already done some of the work needed to support hardware offload.

This list is very focused on procedural items and support in the networks; but nearly zero attention is paid to NIC->Host interface issues.  For overlay technologies to deliver acceptable performance for network-heavy workloads, NIC->Host interface must be more intelligent than it is today.  Next-generation NICs must implement new capabilities.  It is therefore useful to consider NICs in any discussion of load-balancing across the datacenter network, if for no other reason than to foster greater understanding of this challenge.
--
Jeff S Wheeler <[email-masked]<mailto:[email-masked]>>
Sr Network Operator  /  Innovative Network Concepts _______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Lizhong Jin <[email-masked]>
Date: Tue, 24 Sep 2013 22:26:47 +0800

On Sep 24, 2013 10:12 PM, "Lucy yong" <[email-masked]> wrote:
> The explanation further confirms that using gre-in-udp for NVGRE will
simplify NIC LB implementation. It requires no change on the flow table
structure.
[Lizhong] there maybe some misunderstanding here. The key of flow table is
the customer payload 4 tuple, not tunnel header 4 tuple. I don't think the
discussion confirms anything.

Regards
Lizhong

>
> [Lizhong] the NIC LB is quite different with the router/switch. The NIC
LB is called flow steering, and it does not rely on a hash value to steer
Jeff Wheeler
ECMP
> I'd like to note that it is becoming increasingly common for the
underlying L3 or L2 network to support load-balancing based on GRE
inner-header fields.  In fact, this can be largely relied on for
high-bandwidth GRE tunnels over the public Internet (!) today, and within
many datacenter networks.
>
> This list is very focused on procedural items and support in the
networks; but nearly zero attention is paid to NIC->Host interface issues.
For overlay technologies to deliver acceptable performance for
network-heavy workloads, NIC->Host interface must be more intelligent than
it is today.  Next-generation NICs must implement new capabilities.  It is
therefore useful to consider NICs in any discussion of load-balancing
across the datacenter network, if for no other reason than to foster
greater understanding of this challenge.

---

From: Lucy yong <[email-masked]>
Date: Tue, 24 Sep 2013 14:32:05 +0000

So this is about payload flow forwarding.  Will a payload flow be forwarded to any CPU core, or must be designated to one of cores?
Lucy

From: Lizhong Jin [mailto:[email-masked]]
Sent: Tuesday, September 24, 2013 9:27 AM
To: Lucy yong
Cc: [email-masked]; [email-masked]
Subject: RE: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

> The explanation further confirms that using gre-in-udp for NVGRE will simplify NIC LB implementation. It requires no change on the flow table structure.
[Lizhong] there maybe some misunderstanding here. The key of flow table is the customer payload 4 tuple, not tunnel header 4 tuple. I don't think the discussion confirms anything.

Regards
Lizhong

>
>
>
> Thanks,
>
> Lucy
>
>
>
> From: Lizhong Jin [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
>

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 30 Sep 2013 09:17:49 -0400

On Tue, Sep 24, 2013 at 10:32 AM, Lucy yong <[email-masked]> wrote:
> So this is about payload flow forwarding.  Will a payload flow be forwarded
> to any CPU core, or must be designated to one of cores?

"It depends on the NIC," as well as configuration / NIC driver.  The
popular Intel 82599 will deliver all GRE traffic with same outer SRC &
DST to one Rx queue, which will be served by only one CPU core.
GRE-in-UDP should have L4 entropy, as you've mentioned, so the entropy
information will allow traffic to be distributed to multiple queues /
CPUs.

It's true that this NIC (and others) can store flow-to-queue affinity
state on-die, but in truth, this feature is not useful for NVO3 unless
the guest OS is trusted to perform its own NVO3 encapsulation
(violating many folks' security model.)

This is really not a problem today, because it is not ideal for the
hypervisor implementing NVO3 to be executed on all CPUs anyway --
otherwise the presence of much network traffic will starve the guest
OS and its applications of CPU, cause the CPU caches to be affected
badly (probably thrashing most of the L1 ICACHE each time a packet
arrives), and so on.

When it becomes a problem is when NIC vendors decide to implement NVO3
offloading.  At that point, multiple CPUs can be used effectively as
long as sufficient information can be parsed from the NVO3 transport
header, or the inner-header, to determine which DMA queue should
receive each packet.  A "random" scheme (we are throwing around the
word entropy) is stupid, because in many cases, multiple guest VMs
will be running on one physical host (hypervisor) and these VMs can
have affinity to certain CPU cores.

In these cases, it is desirable to direct traffic for VM#1 to CPU#1,
VM#2 to CPU#2, and so on, depending on how the systems folks have
allocated resources.

Entropy here is really not smart all the time.  It is a useful option
for some scenarios, but others will require a fixed L4 header per NVO3
termination, so the appropriate VM/CPU can receive packets from the
NIC efficiently, without executing a hypervisor at all.

---

From: Lucy yong <[email-masked]>
Date: Mon, 30 Sep 2013 16:23:52 +0000

Hi Jeff,

Please see inline.

-----Original Message-----
From: Jeff Wheeler [mailto:[email-masked]] 
Sent: Monday, September 30, 2013 8:18 AM
To: Lucy yong
Cc: Lizhong Jin; [email-masked]
Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

On Tue, Sep 24, 2013 at 10:32 AM, Lucy yong <[email-masked]> wrote:
> So this is about payload flow forwarding.  Will a payload flow be 
> forwarded to any CPU core, or must be designated to one of cores?

"It depends on the NIC," as well as configuration / NIC driver.  The popular Intel 82599 will deliver all GRE traffic with same outer SRC & DST to one Rx queue, which will be served by only one CPU core.
GRE-in-UDP should have L4 entropy, as you've mentioned, so the entropy information will allow traffic to be distributed to multiple queues / CPUs.

It's true that this NIC (and others) can store flow-to-queue affinity state on-die, but in truth, this feature is not useful for NVO3 unless the guest OS is trusted to perform its own NVO3 encapsulation (violating many folks' security model.)

This is really not a problem today, because it is not ideal for the hypervisor implementing NVO3 to be executed on all CPUs anyway -- otherwise the presence of much network traffic will starve the guest OS and its applications of CPU, cause the CPU caches to be affected badly (probably thrashing most of the L1 ICACHE each time a packet arrives), and so on.

When it becomes a problem is when NIC vendors decide to implement NVO3 offloading.  At that point, multiple CPUs can be used effectively as long as sufficient information can be parsed from the NVO3 transport header, or the inner-header, to determine which DMA queue should receive each packet.  A "random" scheme (we are throwing around the word entropy) is stupid, because in many cases, multiple guest VMs will be running on one physical host (hypervisor) and these VMs can have affinity to certain CPU cores.
[Lucy] We should distinct flow forwarding and ECMP. This is about flow forwarding, not ECMP. Does NIC implementing NVO3 offloading mean VXLAN/NVGRE encap/decap and tunnel function or other as well? When implementing NVO3 offloading, does that mean that hypervisor implementing vSwitching need to run on one CPU and all the packets need to pass this hypervisor first?

In these cases, it is desirable to direct traffic for VM#1 to CPU#1,
VM#2 to CPU#2, and so on, depending on how the systems folks have allocated resources.

Entropy here is really not smart all the time.  It is a useful option for some scenarios, but others will require a fixed L4 header per NVO3 termination, so the appropriate VM/CPU can receive packets from the NIC efficiently, without executing a hypervisor at all.
[Lucy] Flow entropy relates to ECMP LB process, does not apply to flow forwarding, i.e. flow entropy can't be used in flow forwarding. Thus, the thread title and discussed content are difference topics. 

Thank you very much for the explanation.

Lucy

---

From: Jeff Wheeler <[email-masked]>
Date: Mon, 30 Sep 2013 13:02:13 -0400

> [Lucy] We should distinct flow forwarding and ECMP. This is about flow forwarding, not ECMP. Does NIC implementing NVO3 offloading mean VXLAN/NVGRE encap/decap and tunnel function or other as well?…

If the NIC has NVO3 offloading, the hypervisor will *not* need to
interact with the traffic -- as long as the use of NVO3 on the VMs are
within the design limits of the NIC's offloading implementation.  The
NIC itself would encapsulate / decapsulate.  This is the advantage of
NIC offloading -- removing the CPU expense of executing the hypervisor
for encapsulation / decapsulation.

If the NIC has this capability, it would be able to direct packets to
the appropriate DMA queue (which can have affinity with a CPU) for the
guest VM.  In addition, the hypervisor would not need to be executed
at all to receive packets (and maybe not even to transmit; depending
on security needs.)  This provides ideal resource utilization.

> [Lucy] Flow entropy relates to ECMP LB process, does not apply to flow forwarding, i.e. flow entropy can't be used in flow forwarding. Thus, the thread title and discussed content are difference to…

These are actually related, if the entropy information is carried in a
part of the outer-packet (like L4 field) that may be used by NICs.

I will explain a scenario below.  My point is not really to highlight
this specific case, but to highlight the fact that there is very, very
little discussion in the area of NIC/host interfaces on the NVO3
mailing list; and insufficient consideration given to this area.  This
might be because many of the posters are unfamiliar with this subject.

Imagine that there is no NIC offloading of NVO3, and a hypervisor uses
2 or more CPUs, CPU#1 and CPU#2, for its various functions, including
NIC Rx/Tx.  Two different traffic flows arrive, called F and G, and
these flows are both destined for guest VM "A" which executes on one
or more CPUs, for simplicity, call this CPU#3.

If the NIC happens to deliver flow F to the hypervisor DMA queue for
CPU#1, and flow G to the hypervisor DMA queue for CPU#2, and also, the
guest VM happens to receive these flows in the same bounce-buffer (the
VM might only have one such buffer) then, as packets are received in
the order F-G-F-G-F-G the CPU#1 and CPU#2 may be flushing
bounce-buffer descriptor rows from their L1 caches if the descriptors
share cachelines (implementation dependent.)  CPU#3 will also need to
be reading these same cachelines, most likely causing traffic to the
L3 cache (often the only memory interface among distant CPU cores on a
single die.)  In modern CPUs this can mean dozens of additional Hz
wasted per each packet receipt just waiting on the cache coherency
machinery to allow memory to be read or written.

So the above situation means that having two CPU cores for receiving
network traffic, performing hypervisor / vSwitch-based encapsulation &
decapsulation, and bouncing traffic into the guest VMs could actually
perform worse than using only one CPU core -- because the NIC might
use flow entropy information which is carried in L4 headers and used
in its hashing algorithm -- but the guest VM may not have multiple Rx
queues to take advantage of this.

The same entropy information which benefits ECMP/LAG hashing in the
datacenter network might hinder the NIC -> hypervisor -> guest VM
interface, rather than help it, depending on a variety of
implementation-specific issues.  For example, the whole problem above
can be mitigated if the hypervisor -> guest VM packet descriptors do
not span across cachelines; or if the NIC is configured to ignore L4
information in Rx DMA ring hashing, or if the L4 fields identify the
guest VM instead of containing entropy from the inner-flow, or if the
guest VM has multiple different bounce-buffers for packet receipt, at
least one per hypervisor CPU that might be handling Rx network
traffic, etc.

Entropy data for LAG/ECMP is good and necessary for the datacenter
network, but its affect on the end-hosts should be considered and
discussed.  Such discussion may be helpful to software folks, NIC
vendors, etc. as they decide to implement NVO3.
-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Lucy yong <[email-masked]>
Date: Mon, 30 Sep 2013 18:04:14 +0000

Jeff,

To implement NVO3 offloading, NIC MUST be able to support VXLAN/NVGRE/STT encap/decap and tunnel termination, i.e. inspect the protocol type and demulexing TCP/UDP ports first. The Flow entropy of these usages is embedded in src port. NIC does not have to use it and can use inner header directly for flow steering. 

Your point that ECMP may hinder NIC->hypervior-> guest VM interface design is true. No free lunch. At least, ingress tunnel point needs to generate the flow entropy and encode on the packets, and egress point needs to take it off. However, ECMP support becomes an essential to underlying network (see nvo3 data plane requirement). This is just matter what we gain via where we spend money on. 

I do not work on NIC design. Just through my 2cent.

Regards,
Lucy

-----Original Message-----
From: Jeff Wheeler [mailto:[email-masked]] 
Sent: Monday, September 30, 2013 12:02 PM
To: Lucy yong
Cc: Lizhong Jin; [email-masked]
Subject: Re: [nvo3] LAG/ECMP load-balancing problems facing overlay networks

> [Lucy] We should distinct flow forwarding and ECMP. This is about flow forwarding, not ECMP. Does NIC implementing NVO3 offloading mean VXLAN/NVGRE encap/decap and tunnel function or other as well?…

If the NIC has NVO3 offloading, the hypervisor will *not* need to interact with the traffic -- as long as the use of NVO3 on the VMs are within the design limits of the NIC's offloading implementation.  The NIC itself would encapsulate / decapsulate.  This is the advantage of NIC offloading -- removing the CPU expense of executing the hypervisor for encapsulation / decapsulation.

If the NIC has this capability, it would be able to direct packets to the appropriate DMA queue (which can have affinity with a CPU) for the guest VM.  In addition, the hypervisor would not need to be executed at all to receive packets (and maybe not even to transmit; depending on security needs.)  This provides ideal resource utilization.

> [Lucy] Flow entropy relates to ECMP LB process, does not apply to flow forwarding, i.e. flow entropy can't be used in flow forwarding. Thus, the thread title and discussed content are difference to…

These are actually related, if the entropy information is carried in a part of the outer-packet (like L4 field) that may be used by NICs.

I will explain a scenario below.  My point is not really to highlight this specific case, but to highlight the fact that there is very, very little discussion in the area of NIC/host interfaces on the NVO3 mailing list; and insufficient consideration given to this area.  This might be because many of the posters are unfamiliar with this subject.

Imagine that there is no NIC offloading of NVO3, and a hypervisor uses
2 or more CPUs, CPU#1 and CPU#2, for its various functions, including NIC Rx/Tx.  Two different traffic flows arrive, called F and G, and these flows are both destined for guest VM "A" which executes on one or more CPUs, for simplicity, call this CPU#3.

If the NIC happens to deliver flow F to the hypervisor DMA queue for CPU#1, and flow G to the hypervisor DMA queue for CPU#2, and also, the guest VM happens to receive these flows in the same bounce-buffer (the VM might only have one such buffer) then, as packets are received in the order F-G-F-G-F-G the CPU#1 and CPU#2 may be flushing bounce-buffer descriptor rows from their L1 caches if the descriptors share cachelines (implementation dependent.)  CPU#3 will also need to be reading these same cachelines, most likely causing traffic to the
L3 cache (often the only memory interface among distant CPU cores on a single die.)  In modern CPUs this can mean dozens of additional Hz wasted per each packet receipt just waiting on the cache coherency machinery to allow memory to be read or written.

So the above situation means that having two CPU cores for receiving network traffic, performing hypervisor / vSwitch-based encapsulation & decapsulation, and bouncing traffic into the guest VMs could actually perform worse than using only one CPU core -- because the NIC might use flow entropy information which is carried in L4 headers and used in its hashing algorithm -- but the guest VM may not have multiple Rx queues to take advantage of this.

The same entropy information which benefits ECMP/LAG hashing in the datacenter network might hinder the NIC -> hypervisor -> guest VM interface, rather than help it, depending on a variety of implementation-specific issues.  For example, the whole problem above can be mitigated if the hypervisor -> guest VM packet descriptors do not span across cachelines; or if the NIC is configured to ignore L4 information in Rx DMA ring hashing, or if the L4 fields identify the guest VM instead of containing entropy from the inner-flow, or if the guest VM has multiple different bounce-buffers for packet receipt, at least one per hypervisor CPU that might be handling Rx network traffic, etc.

Entropy data for LAG/ECMP is good and necessary for the datacenter network, but its affect on the end-hosts should be considered and discussed.  Such discussion may be helpful to software folks, NIC vendors, etc. as they decide to implement NVO3.
--
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts