Subject: [nvo3] performance limitations with virtual switch as the nvo3 end point

From: David LE GOFF <[email-masked]>
Date: Wed, 29 Aug 2012 15:16:10 +0200

Hi Folks,

Did anyone experienced some performance limitations in Labs with the
virtual switch function as the bottleneck when dealing with network
overlays?

I mean with the tunnel end point located on the hypervisor (virtual
switch), setting up Tagging, QoS, ACL, encryption/decryption, etc. require
significant CPUs.

I know there is not yet official nvo3 implementation there, though VSphere
5 announced it with VXLAN recently but at any chance if some studies have
been done, I would be glad to read those.

I know STT has been built to overcome such challenges thanks to the NIC
offload capabilities…

These studies may also drive the brainstorming about which protocol we may
use/build?

Thank you!

david le goff.

---

From: "smith, erik" <[email-masked]>
Date: Wed, 29 Aug 2012 18:47:51 -0400

Hi David, a few months ago we did some basic performance testing with OVS and were pretty happy with the results.  For one reason or another we were under the impression that using OVS to encap/decap would limit our total throughput to 4-6 Gbps and this turned out to not be the case.  In our configuration, we were able to demonstrate 20 Gbps over a bonded pair of 10GbE NICs using STT for the overlay.  Our testing wasn't exactly scientific but I also found an interesting blog post by Martin Cassado that our limited testing seems to corroborate.

I haven't done any testing with VMware and VXLAN.  However, if you're experiencing limited performance with OVS on <insert your favorite Linux distro here>, I would suggest playing around with Jumbo frames (starting from within the guest) and working your way out to the physical interfaces.

For additional information, refer to the following:

1)      Martin Cassado's blog: ( http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/ )

2)      I posted something to my blog a bit less detailed (but with diagrams) earlier this week ( http://brasstacksblog.typepad.com/brass-tacks/2012/08/network-virtualization-networkings-21st-century-equivalent-to-the-space-race.html )  Specifically, the final three diagrams..

Erik

From: [email-masked] [mailto:[email-masked]] On Behalf Of David LE GOFF
Sent: Wednesday, August 29, 2012 9:16 AM
To: [email-masked]
Subject: [nvo3] performance limitations with virtual switch as the nvo3 end point

Hi Folks,

Did anyone experienced some performance limitations in Labs with the virtual switch function as the bottleneck when dealing with network overlays?
I mean with the tunnel end point located on the hypervisor (virtual switch), setting up Tagging, QoS, ACL, encryption/decryption, etc. require significant CPUs.

I know there is not yet official nvo3 implementation there, though VSphere 5 announced it with VXLAN recently but at any chance if some studies have been done, I would be glad to read those.
I know STT has been built to overcome such challenges thanks to the NIC offload capabilities...

These studies may also drive the brainstorming about which protocol we may use/build?

Thank you!

david le goff.

---

From: Somesh Gupta <[email-masked]>
Date: Wed, 29 Aug 2012 17:19:15 -0700

I assume that majority of the NIC vendors will support the stateless offloads for
VxLAN and NvGRE by sometime next year - so they should all be on equal footing
from that point of view.

the additional overhead of encap/decap compared to the overhead of copying date between
the VM and the hypervisor should be minimal.

From: [email-masked] [mailto:[email-masked]] On Behalf Of smith, erik
Sent: Wednesday, August 29, 2012 3:48 PM
To: David LE GOFF; [email-masked]
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

Hi Folks,

Thank you!

david le goff.

---

From: Jon Hudson <[email-masked]>
Date: Wed, 29 Aug 2012 17:25:36 -0700

Totally agreed, but that will require all new hardware.

One of the gains of STT was the use of current off the self performance
capabilities.

On Wed, Aug 29, 2012 at 5:19 PM, Somesh Gupta <[email-masked]> wrote:

> I assume that majority of the NIC vendors will support the stateless
> offloads for****
>
> VxLAN and NvGRE by sometime next year – so they should all be on equal
> footing****
>
> from that point of view.****
>
> ** **
>
> the additional overhead of encap/decap compared to the overhead of copying
> date between****
>
> the VM and the hypervisor should be minimal.****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *smith, erik
> *Sent:* Wednesday, August 29, 2012 3:48 PM
> *To:* David LE GOFF; [email-masked]
> *Subject:* Re: [nvo3] performance limitations with virtual switch as the
> nvo3 end point****
>
> ** **
>
> Hi David, a few months ago we did some basic performance testing with OVS
> and were pretty happy with the results.  For one reason or another we were
> under the impression that using OVS to encap/decap would limit our total
> throughput to 4-6 Gbps and this turned out to not be the case.  In our
> configuration, we were able to demonstrate 20 Gbps over a bonded pair of
> 10GbE NICs using STT for the overlay.  Our testing wasn’t exactly
> scientific but I also found an interesting blog post by Martin Cassado that
> our limited testing seems to corroborate.****
>
> ** **
>
> I haven’t done any testing with VMware and VXLAN.  However, if you’re
> experiencing limited performance with OVS on <insert your favorite Linux
> distro here>, I would suggest playing around with Jumbo frames (starting
> from within the guest) and working your way out to the physical interfaces.
>   ****
>
> ** **
>
> For additional information, refer to the following:****
>
> **1)      **Martin Cassado’s blog: (
> http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/)
> ****
>
> **2)      **I posted something to my blog a bit less detailed (but with
> diagrams) earlier this week (
> http://brasstacksblog.typepad.com/brass-tacks/2012/08/network-virtualization-networkings-21st-century-equivalent-to-the-space-race.html)  Specifically, the final three diagrams..
> ****
>
> ** **
>
> Erik****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *David LE GOFF
> *Sent:* Wednesday, August 29, 2012 9:16 AM
> *To:* [email-masked]
> *Subject:* [nvo3] performance limitations with virtual switch as the nvo3
> end point****
>
> ** **
>
> Hi Folks,****
>
> ** **
>
> overlays?****
> significant CPUs.****
> been done, I would be glad to read those.****
>
> offload capabilities…****
>
> ** **
>
> use/build?****
> ** **
> Thank you!****
> ** **
> david le goff.****
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>
>

-- 
"Do not lie. And do not do what you hate."

---

From: Somesh Gupta <[email-masked]>
Date: Wed, 29 Aug 2012 17:29:59 -0700

Agree although some of the NICs can do some or most of the offloads even today.
But in my opinion STT introduces reassembly state in the receiver which would
prevent gateway implementations in ASICs - for  a short term gain.

From: Jon Hudson [mailto:[email-masked]]
Sent: Wednesday, August 29, 2012 5:26 PM
To: Somesh Gupta
Cc: [email-masked]
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

One of the gains of STT was the use of current off the self performance capabilities.
On Wed, Aug 29, 2012 at 5:19 PM, Somesh Gupta <[email-masked]<mailto:[email-masked]>> wrote:
I assume that majority of the NIC vendors will support the stateless offloads for
VxLAN and NvGRE by sometime next year - so they should all be on equal footing
from that point of view.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of smith, erik
Sent: Wednesday, August 29, 2012 3:48 PM
To: David LE GOFF; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of David LE GOFF
Sent: Wednesday, August 29, 2012 9:16 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] performance limitations with virtual switch as the nvo3 end point

Hi Folks,

Thank you!

david le goff.

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Martin Casado <[email-masked]>
Date: Wed, 29 Aug 2012 17:30:12 -0700

Not to mention that LOM is likely an additional 8-12 months from the 
chipsets hitting the market.

On 8/29/12 5:25 PM, Jon Hudson wrote:
>     VxLAN and NvGRE by sometime next year -- so they should all be on
>     equal footing
>
>     from that point of view.
>
>     *From:[email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] *On
>     Behalf Of *smith, erik
>     *To:* David LE GOFF; [email-masked] <mailto:[email-masked]>
>     *Subject:* Re: [nvo3] performance limitations with virtual switch
>     as the nvo3 end point
>
>     Hi David, a few months ago we did some basic performance testing
>     with OVS and were pretty happy with the results.  For one reason
>     or another we were under the impression that using OVS to
>     encap/decap would limit our total throughput to 4-6 Gbps and this
>     turned out to not be the case.  In our configuration, we were able
>     to demonstrate 20 Gbps over a bonded pair of 10GbE NICs using STT
>     for the overlay. Our testing wasn't exactly scientific but I also
>     found an interesting blog post by Martin Cassado that our limited
>     testing seems to corroborate.
>
>     I haven't done any testing with VMware and VXLAN.  However, if
>     you're experiencing limited performance with OVS on <insert your
>     favorite Linux distro here>, I would suggest playing around with
>     Jumbo frames (starting from within the guest) and working your way
>     out to the physical interfaces.
>
>     1)Martin Cassado's blog: (
>     2)I posted something to my blog a bit less detailed (but with
>     http://brasstacksblog.typepad.com/brass-tacks/2012/08/network-virtualization-networkings-21st-century-equivalent-to-the-space-race.html
>     )  Specifically, the final three diagrams..
>
>     Erik
>
>     *From:[email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] *On
>     Behalf Of *David LE GOFF
>     *To:* [email-masked] <mailto:[email-masked]>
>     *Subject:* [nvo3] performance limitations with virtual switch as
>     the nvo3 end point
>
>     Hi Folks,
>
>     Did anyone experienced some performance limitations in Labs with
>     the virtual switch function as the bottleneck when dealing with
>     network overlays?
>
>     I mean with the tunnel end point located on the hypervisor
>     (virtual switch), setting up Tagging, QoS, ACL,
>     encryption/decryption, etc. require significant CPUs.
>
>
>     I know there is not yet official nvo3 implementation there, though
>     VSphere 5 announced it with VXLAN recently but at any chance if
>     some studies have been done, I would be glad to read those.
>
>     [email-masked] <mailto:[email-masked]>

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Martin Casado
Nicira Networks, Inc.
www.nicira.com
cell: 650-776-1457
~~~~~~~~~~~~~~~~~~~~~~~~~~~

---

From: Jon Hudson <[email-masked]>
Date: Wed, 29 Aug 2012 17:32:24 -0700

very true

On Wed, Aug 29, 2012 at 5:29 PM, Somesh Gupta <[email-masked]> wrote:

> Agree although some of the NICs can do some or most of the offloads even
> today.****
>
> But in my opinion STT introduces reassembly state in the receiver which
> would****
>
> prevent gateway implementations in ASICs – for  a short term gain.****
>
> ** **
>
> *From:* Jon Hudson [mailto:[email-masked]]
> *Sent:* Wednesday, August 29, 2012 5:26 PM
> *To:* Somesh Gupta
> *Cc:* [email-masked]
>
> Totally agreed, but that will require all new hardware. ****
>
> ** **
>
> capabilities. ****
> 1)      Martin Cassado’s blog: (
> 2)      I posted something to my blog a bit less detailed (but with
> https://www.ietf.org/mailman/listinfo/nvo3****
>
>
>
> ****
>
> ** **
>
> --
> "Do not lie. And do not do what you hate."****
>

---

From: Martin Casado <[email-masked]>
Date: Wed, 29 Aug 2012 17:37:05 -0700

Agree here too.  STT is optimized for when both ends are x86 (although 
one of those ends could certainly be used as a gateway).

On 8/29/12 5:29 PM, Somesh Gupta wrote:
> prevent gateway implementations in ASICs -- for  a short term gain.
>
> *From:*Jon Hudson [mailto:[email-masked]]
> *Subject:* Re: [nvo3] performance limitations with virtual switch as 
> the nvo3 end point
>
> offloads for
> *Subject:* Re: [nvo3] performance limitations with virtual switch as 
> the nvo3 end point
>
> Hi David, a few months ago we did some basic performance testing with 
> OVS and were pretty happy with the results.  For one reason or another 
> we were under the impression that using OVS to encap/decap would limit 
> our total throughput to 4-6 Gbps and this turned out to not be the 
> case.  In our configuration, we were able to demonstrate 20 Gbps over 
> a bonded pair of 10GbE NICs using STT for the overlay.  Our testing 
> wasn't exactly scientific but I also found an interesting blog post by 
> Martin Cassado that our limited testing seems to corroborate.
>
> I haven't done any testing with VMware and VXLAN.  However, if you're 
> experiencing limited performance with OVS on <insert your favorite 
> Linux distro here>, I would suggest playing around with Jumbo frames 
> (starting from within the guest) and working your way out to the 
> physical interfaces.
>
> http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/ )
>
> *Subject:* [nvo3] performance limitations with virtual switch as the 
> nvo3 end point
>
> Hi Folks,
>
> switch), setting up Tagging, QoS, ACL, encryption/decryption, etc. 
> require significant CPUs.
>
>
> VSphere 5 announced it with VXLAN recently but at any chance if some 
> studies have been done, I would be glad to read those.
>

---

From: Haoweiguo <[email-masked]>
Date: Thu, 30 Aug 2012 10:02:17 +0000

Offloading to the NIC is a simple solution that don't need signal protocol between NVE and TES.so the solution will have large impact on the architecture of NVO3.

But offloading to TOR has it's specific advantage.TOR can realize better ACL/QOS etc feature than the NIC.

So I think It's not clear we should offload to NIC or offload to TOR.

Regards

Weiguo

________________________________

发件人: Somesh Gupta [[email-masked]]
发送时间: 2012年8月30日 8:19
到: [email-masked]
主题: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

I assume that majority of the NIC vendors will support the stateless offloads for
VxLAN and NvGRE by sometime next year – so they should all be on equal footing
from that point of view.

Hi David, a few months ago we did some basic performance testing with OVS and were pretty happy with the results.  For one reason or another we were under the impression that using OVS to encap/decap would limit our total throughput to 4-6 Gbps and this turned out to not be the case.  In our configuration, we were able to demonstrate 20 Gbps over a bonded pair of 10GbE NICs using STT for the overlay.  Our testing wasn’t exactly scientific but I also found an interesting blog post by Martin Cassado that our limited testing seems to corroborate.

I haven’t done any testing with VMware and VXLAN.  However, if you’re experiencing limited performance with OVS on <insert your favorite Linux distro here>, I would suggest playing around with Jumbo frames (starting from within the guest) and working your way out to the physical interfaces.

1)      Martin Cassado’s blog: ( http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/ )

Erik

Hi Folks,

I know there is not yet official nvo3 implementation there, though VSphere 5 announced it with VXLAN recently but at any chance if some studies have been done, I would be glad to read those.
I know STT has been built to overcome such challenges thanks to the NIC offload capabilities…

Thank you!

david le goff.

---

From: David LE GOFF <[email-masked]>
Date: Thu, 30 Aug 2012 12:09:52 +0200

Third way to go:
Dedicate cores for processing this workload and "offload" them from
application/VM cores.

Are you aware of the Intel DPDK (Data Plane Development Kit)?

David.

On Thu, Aug 30, 2012 at 12:02 PM, Haoweiguo <[email-masked]> wrote:

>  Offloading to the NIC is a simple solution that don't need signal
> protocol between NVE and TES.so the solution will have large impact on the
> architecture of NVO3.
>

---

From: Jon Hudson <[email-masked]>
Date: Thu, 30 Aug 2012 03:26:49 -0700

And what is the reason behind not offering a path to both?

On Thu, Aug 30, 2012 at 3:02 AM, Haoweiguo <[email-masked]> wrote:

> ------------------------------
>
>  *发件人:* Somesh Gupta [[email-masked]]
> *发送时间:* 2012年8月30日 8:19
> *到:* [email-masked]
> *主题:* Re: [nvo3] performance limitations with virtual switch as the nvo3
> end point
>
> footing
> date between
> our limited testing seems to corroborate.
>
>
>
> end point

---

From: Somesh Gupta <[email-masked]>
Date: Thu, 30 Aug 2012 06:20:12 -0700

In the offloading to NIC discussion, none of the vswitch functionality is
offloaded to the NIC – only TCP checksum, LSO, LRO, inbound load spreading etc
(SR-IOV would be a different story although that has very limited
deployment).

From: Haoweiguo [mailto:[email-masked]]
Sent: Thursday, August 30, 2012 3:02 AM
To: Somesh Gupta; [email-masked]
Subject: 答复: [nvo3] performance limitations with virtual switch as the nvo3 end point

Regards

Weiguo

Erik

Hi Folks,

Thank you!

david le goff.

---

From: "Ayandeh, Siamack" <[email-masked]>
Date: Thu, 30 Aug 2012 14:25:26 +0000

Hi Erik,

Thanks for the post. Do you by any chance have any data on impact of packet loss on STT performance if application is running TCP? Would the application resend the entire segment!?

Thanks,

Siamack
From: [email-masked] [mailto:[email-masked]] On Behalf Of smith, erik
Sent: Wednesday, August 29, 2012 6:48 PM
To: David LE GOFF; [email-masked]
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

Hi Folks,

Thank you!

david le goff.

---

From: "Garcia Del Rio, Diego (Diego)" <[email-masked]>
Date: Thu, 30 Aug 2012 16:47:00 +0200

I don't agree that offloading to the ToR will give you better QoS or ACLs. A lot of the merchant silicon out there would not scale to the needed number of ACLs as they have very restricted tcams. QoS-wise, very few perform real ingress queuing.

Diego Garcia del Rio
Product Management, IPD
Mountain View,CA
+1 (415) 439-9420

From: Haoweiguo [mailto:[email-masked]]
Sent: Thursday, August 30, 2012 12:02 PM
To: Somesh Gupta <[email-masked]>; [email-masked] <[email-masked]>
Subject: [nvo3] 答复: performance limitations with virtual switch as the nvo3 end point

Regards

Weiguo

Erik

Hi Folks,

Thank you!

david le goff.

---

From: Martin Casado <[email-masked]>
Date: Thu, 30 Aug 2012 08:47:44 -0700

For the most part, drop performance impact is similar to TSO today 
(partial coalescing can be done on the receive side).  Here is a 
relevant snippet from a discussion on this at 
(http://networkheresy.com/2012/03/04/network-virtualization-encapsulation-and-stateless-tcp-transport-stt/)

" the semantics are very similar (to TSO) in that received packets can 
be batched in consecutive sequences and passed to the guest as 
legitimate TCP frames (just like TSO today).

However, with STT, the outer frame is what is segmented, where with 
other tunneling protocols presumably it would be the inner TCP frame. 
There are clear trade-offs between the two approaches. With STT, if the 
first packets drops, then we're hosed. On the other hand, segmenting the 
inner header (with L2) would likely require duplicating the TCP header 
in each packet which would be less efficient byte-for-byte."

Again, it's worth pointing out that STT was designed for use when x86 is 
on both ends.  The header is 32 bit aligned, the spec isn't parsimonious 
with bit sizes in header fields, and the fields are opaque based on the 
assumption that they'll be interpreted by software on either side that 
is evolving relatively quickly.  This makes sense in some environments, 
and not in others.

 From our (Nicira's) standpoint, using a more flexible encap makes sense 
when we own both sides of the communication since we are often often 
evolving our control plane (header bits are useful for all sorts of 
stuff, datapath state versioning, multi-hop logical topologies, carrying 
additional information like logical inport, or logical output port, 
etc.).   Also, it is generally only deployed in the datacenter fabric, 
so abusing TCP isn't a huge issue since no middleboxes should be on 
route.  For deployment environments with middleboxe, GRE is clearly more 
suitable (and we support that too).

Of course, whenever an end point is an ASIC or a third party device we 
don't control, clearly something like VXLAN or NVGRE is preferable.

In general, I think it is a good idea to decouple the control plane and 
the encap so there is more flexibility to map the right technology to 
the right deployment environment.

.m

On 8/30/12 7:25 AM, Ayandeh, Siamack wrote:
>
> Hi Erik,
>
> Thanks for the post. Do you by any chance have any data on impact of 
> packet loss on STT performance if application is running TCP? Would 
> the application resend the entire segment!?
>
> Thanks,
>
> Siamack
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *smith, erik
> *Sent:* Wednesday, August 29, 2012 6:48 PM
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *David LE GOFF

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 30 Aug 2012 11:37:14 -0500

The NIC support for VXLAN and NVGRE came up earlier this week at VMworld 2012. During a VMware VXLAN presentation their PLM mentioned VXLAN optimized hardware will be available for NICs soon. On the Exhibition floor one startup said they have already NIC HW optimized for VXLAN processing, the established NIC vendor said they will get it by YE 2013. Heard similar statements about NVGRE from the same people.

Agree it will take a bit longer to get new hardware in real life deployments, we need to understand though whether the whole timing justifies standardizing interim solutions in NVO3. Granted we did it for some solutions in the past (see the Ethernet PW raw and VLAN options) and it caused some interop pain for both VPWS and VPLS.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Wednesday, August 29, 2012 5:30 PM
To: Jon Hudson
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

From: Jon Hudson [mailto:[email-masked]]
Sent: Wednesday, August 29, 2012 5:26 PM
To: Somesh Gupta
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

Hi Folks,

Thank you!

david le goff.

---

From: Somesh Gupta <[email-masked]>
Date: Thu, 30 Aug 2012 09:37:27 -0700

totally agree about decoupling the encap and the control plane -
probably needs some abstraction of mappings to achieve that.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Martin Casado
Sent: Thursday, August 30, 2012 8:48 AM
To: Ayandeh, Siamack
Cc: David LE GOFF; [email-masked]; smith, erik
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

For the most part, drop performance impact is similar to TSO today (partial coalescing can be done on the receive side).  Here is a relevant snippet from a discussion on this at (http://networkheresy.com/2012/03/04/network-virtualization-encapsulation-and-stateless-tcp-transport-stt/)

" the semantics are very similar (to TSO) in that received packets can be batched in consecutive sequences and passed to the guest as legitimate TCP frames (just like TSO today).

However, with STT, the outer frame is what is segmented, where with other tunneling protocols presumably it would be the inner TCP frame. There are clear trade-offs between the two approaches. With STT, if the first packets drops, then we're hosed. On the other hand, segmenting the inner header (with L2) would likely require duplicating the TCP header in each packet which would be less efficient byte-for-byte."

Again, it's worth pointing out that STT was designed for use when x86 is on both ends.  The header is 32 bit aligned, the spec isn't parsimonious with bit sizes in header fields, and the fields are opaque based on the assumption that they'll be interpreted by software on either side that is evolving relatively quickly.  This makes sense in some environments, and not in others.

>From our (Nicira's) standpoint, using a more flexible encap makes sense when we own both sides of the communication since we are often often evolving our control plane (header bits are useful for all sorts of stuff, datapath state versioning, multi-hop logical topologies, carrying additional information like logical inport, or logical output port, etc.).   Also, it is generally only deployed in the datacenter fabric, so abusing TCP isn't a huge issue since no middleboxes should be on route.  For deployment environments with middleboxe, GRE is clearly more suitable (and we support that too).

Of course, whenever an end point is an ASIC or a third party device we don't control, clearly something like VXLAN or NVGRE is preferable.

In general, I think it is a good idea to decouple the control plane and the encap so there is more flexibility to map the right technology to the right deployment environment.

.m

Thanks,

Siamack
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of smith, erik
Sent: Wednesday, August 29, 2012 6:48 PM
To: David LE GOFF; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of David LE GOFF
Sent: Wednesday, August 29, 2012 9:16 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] performance limitations with virtual switch as the nvo3 end point

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

www.nicira.com<http://www.nicira.com>

cell: 650-776-1457

---

From: Somesh Gupta <[email-masked]>
Date: Thu, 30 Aug 2012 09:40:15 -0700

By the time we come up with a spec we won't have to worry about hardware support :)
(unless we pick something other than VxLAN or NvGRE)

From: Balus, Florin Stelian (Florin) [mailto:[email-masked]]
Sent: Thursday, August 30, 2012 9:37 AM
To: Somesh Gupta; Jon Hudson
Cc: [email-masked]
Subject: RE: [nvo3] performance limitations with virtual switch as the nvo3 end point

Erik

Hi Folks,

Thank you!

david le goff.

---

From: Somesh Gupta <[email-masked]>
Date: Thu, 30 Aug 2012 09:46:43 -0700

BTW, should the standard support multiple encapsulation types? And should/can a single L2-CUG
support multiple encapsulation types?

From: [email-masked] [mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Thursday, August 30, 2012 9:37 AM
To: Martin Casado; Ayandeh, Siamack
Cc: smith, erik; [email-masked]; David LE GOFF
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

.m

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

cell: 650-776-1457

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 30 Aug 2012 12:09:23 -0500

In-line...

Of course, whenever an end point is an ASIC or a third party device we don't control, clearly something like VXLAN or NVGRE is preferable.
[FB>] If we are to talk about what will work on the interim/a gateway optimized soln: you forgot MPLS VPN encap, still one of the encaps under consideration in NVO3.  Ideal for an existing VPN/VLAN gateway, VPN providers will definitely love it, it's already standardized, available in merchant silicon and widely deployed on the WAN side in all the Telco networks. And to sing a friendly note, Openflow/SDN can program it... :)

.m

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

cell: 650-776-1457

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 30 Aug 2012 17:20:30 +0000

Agreed re Florin's MPLS comments, and that suggests 'Yes' is a reasonable anwser to this question from Somesh.  "BTW, should the standard support multiple encapsulation types? And should/can a single L2-CUG support multiple encapsulation types?"

Peter

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Balus, Florin Stelian (Florin)
Sent: Thursday, August 30, 2012 1:09 PM
To: Martin Casado; Ayandeh, Siamack
Cc: smith, erik; [email-masked]; David LE GOFF
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

In-line...

.m

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

cell: 650-776-1457

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 30 Aug 2012 12:33:49 -0500

Multiple types of service header/encapsulations create issues for the usual IETF (VPN) control plane; the VPN one was designed for uniform encapsulation at VPN demux layer although various tunneling could be used (MPLS/IP GRE)...
We need some new thinking here in IETF but we are probably getting ahead of ourselves. Good discussion to have after we sort out the data and control plane requirement drafts...

From: [email-masked] [mailto:[email-masked]] On Behalf Of AshwoodsmithPeter
Sent: Thursday, August 30, 2012 10:21 AM
To: Balus, Florin Stelian (Florin); Martin Casado; Ayandeh, Siamack
Cc: David LE GOFF; [email-masked]; smith, erik
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

Peter

________________________________
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Balus, Florin Stelian (Florin)
Sent: Thursday, August 30, 2012 1:09 PM
To: Martin Casado; Ayandeh, Siamack
Cc: smith, erik; [email-masked]<mailto:[email-masked]>; David LE GOFF
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

In-line...

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Martin Casado
Sent: Thursday, August 30, 2012 8:48 AM
To: Ayandeh, Siamack
Cc: David LE GOFF; [email-masked]<mailto:[email-masked]>; smith, erik
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

.m

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

cell: 650-776-1457

---

From: Jon Hudson <[email-masked]>
Date: Thu, 30 Aug 2012 10:54:39 -0700

" to be great you have to do things others would not"

=)

On Aug 30, 2012, at 10:33 AM, "Balus, Florin Stelian (Florin)" <[email-masked]> wrote:

> Multiple types of service header/encapsulations create issues for the usual IETF (VPN) control plane; the VPN one was designed for uniform encapsulation at VPN demux layer although various tunneling could be used (MPLS/IP GRE)…
> We need some new thinking here in IETF but we are probably getting ahead of ourselves. Good discussion to have after we sort out the data and control plane requirement drafts…
>  
> Agreed re Florin’s MPLS comments, and that suggests ‘Yes’ is a reasonable anwser to this question from Somesh.  “BTW, should the standard support multiple encapsulation types? And should/can a single L2-CUG support multiple encapsulation types?”
>  
> Peter
>  
> In-line…
> However, with STT, the outer frame is what is segmented, where with other tunneling protocols presumably it would be the inner TCP frame. There are clear trade-offs between the two approaches. With STT, if the first packets drops, then we’re hosed. On the other hand, segmenting the inner header (with L2) would likely require duplicating the TCP header in each packet which would be less efficient byte-for-byte."
> 
> [FB>] If we are to talk about what will work on the interim/a gateway optimized soln: you forgot MPLS VPN encap, still one of the encaps under consideration in NVO3.  Ideal for an existing VPN/VLAN gateway, VPN providers will definitely love it, it’s already standardized, available in merchant silicon and widely deployed on the WAN side in all the Telco networks. And to sing a friendly note, Openflow/SDN can program it… J
>

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Thu, 30 Aug 2012 20:46:56 +0200

Yes & Yes. It’s like MPLS/VPN where a PE-router can reach some PE routers through an LSP, others through MPLS-over-GRE encapsulation.

We will probably need a control-plane mechanism to indicate supported/preferred encapsulation methods for each peer NVE.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Thursday, August 30, 2012 6:47 PM
To: [email-masked]
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

totally agree about decoupling the encap and the control plane – 

.m

Hi Erik,

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 30 Aug 2012 21:14:33 +0200

> From our (Nicira's) standpoint, using a more flexible encap 
> makes sense when we own both sides of the communication since 
> we are often often evolving our control plane (header bits 
> are useful for all sorts of stuff, datapath state versioning, 
> multi-hop logical topologies, carrying additional information 
> like logical inport, or logical output port, etc.).   Also, 
> it is generally only deployed in the datacenter fabric, so 
> abusing TCP isn't a huge issue since no middleboxes should be 
> on route.

For what it is worth, from a transport layer point of view abusing TCP can cause harm even without middleboxes. For instance, mis-configured STT could inject erroneous data into existing TCP connections on non-STT hosts if they happen to use the same 5-tuple. If the SEQ and ACK fields are modified ("header bits are useful for all sorts of stuff"), this could even be a kind of scan for the relevant window range in that connection. The risk might be small in fully controlled environments, but apparently STT can have similar effects like spoofed TCP segment attacks. 

Michael (TCPM co-chair)

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 30 Aug 2012 15:19:38 -0500

Actually this is different: the service encapsulation varies not the tunnel. In MPLS VPNs the MPLS service label is always present.

To address the multiple service encapsulations the control plane in charge with service Instance ad, rib and fib must indeed have a mechanism to indicate the type of encap used.

On Aug 30, 2012, at 11:47 AM, "Ivan Pepelnjak" <[email-masked]<mailto:[email-masked]>> wrote:

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Thursday, August 30, 2012 6:47 PM
To: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

From: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]> [<mailto:[email-masked]>mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Thursday, August 30, 2012 9:37 AM
To: Martin Casado; Ayandeh, Siamack
Cc: smith, erik; <mailto:[email-masked]> [email-masked]<mailto:[email-masked]>; David LE GOFF
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

From: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]> [<mailto:[email-masked]>mailto:[email-masked]] On Behalf Of Martin Casado
Sent: Thursday, August 30, 2012 8:48 AM
To: Ayandeh, Siamack
Cc: David LE GOFF; <mailto:[email-masked]> [email-masked]<mailto:[email-masked]>; smith, erik
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

For the most part, drop performance impact is similar to TSO today (partial coalescing can be done on the receive side).  Here is a relevant snippet from a discussion on this at (<http://networkheresy.com/2012/03/04/network-virtualization-encapsulation-and-stateless-tcp-transport-stt/>http://networkheresy.com/2012/03/04/network-virtualization-encapsulation-and-stateless-tcp-transport-stt/)

.m

Thanks,

Siamack
From: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]> [<mailto:[email-masked]>mailto:[email-masked]] On Behalf Of smith, erik
Sent: Wednesday, August 29, 2012 6:48 PM
To: David LE GOFF; <mailto:[email-masked]> [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

1)     Martin Cassado’s blog: ( <http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/> http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/ )

2)     I posted something to my blog a bit less detailed (but with diagrams) earlier this week ( <http://brasstacksblog.typepad.com/brass-tacks/2012/08/network-virtualization-networkings-21st-century-equivalent-to-the-space-race.html> http://brasstacksblog.typepad.com/brass-tacks/2012/08/network-virtualization-networkings-21st-century-equivalent-to-the-space-race.html )  Specifically, the final three diagrams..

Erik

From: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]> [<mailto:[email-masked]>mailto:[email-masked]] On Behalf Of David LE GOFF
Sent: Wednesday, August 29, 2012 9:16 AM
To: <mailto:[email-masked]> [email-masked]<mailto:[email-masked]>
Subject: [nvo3] performance limitations with virtual switch as the nvo3 end point

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

<mailto:[email-masked]>[email-masked]<mailto:[email-masked]>

<https://www.ietf.org/mailman/listinfo/nvo3>https://www.ietf.org/mailman/listinfo/nvo3

--

Martin Casado

Nicira Networks, Inc.

<http://www.nicira.com>www.nicira.com<http://www.nicira.com>

cell: 650-776-1457

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 31 Aug 2012 08:31:29 +0000

Himanshu and I had ever discussed extending our IS-IS VPLS proposal so as to support other data encapsulation formats such as VXLAN and NVGRE, in addition to the existing MAC-in-MPLS-in-IP/GRE/UDP encapsulation formats as described in this draft. As you may have already noticed before, one major concern about the deployment of VXLAN in a multi-tenant cloud data center environment is its requirement for a strong multicast capability in the IP underlying networks. With some IS-IS extensions, the DCVPN membership could be discovered automatically and as a result, the DCVPN solution could easily support ingress replication capability. In this way, DC operators would have enough flexibility to make a tradeoff between states and bandwidth on a per tenant basis according to the particular conditions of each tenant (e.g., the amount of PE/NVE nodes, the volume of BUM traffic and so on). Besides, the special encapsulation format could be signaled automatically between PE/NVE nodes. Finally, in the case where multiple tunnel endpoint addresses are required to facilitate the load-balancing of DC VPN traffic across the IP core networks (e.g., an additional IPsec encapsulation may be required after the VXLAN encapsulation due to security considerations in some cases, or NVGRE may need such capability so as to add more entropy info to the DCVPN traffic between a PE/NVE node pair), IS-IS extension could be used as well to advertise multiple tunnel endpoint addresses in the form of one or more prefixes.

We are considering extending our IS-IS VPLS proposal for the above purposes now, any comments and suggestions are appreciated.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Balus, Florin Stelian (Florin)
发送时间: 2012年8月31日 4:20
收件人: Ivan Pepelnjak
抄送: [email-masked]; Somesh Gupta
主题: Re: [nvo3] performance limitations with virtual switch as the nvo3 end point

.m

Thanks,

Siamack

Erik

Hi Folks,

Thank you!

david le goff.

nvo3 mailing list

--

Martin Casado

Nicira Networks, Inc.

cell: 650-776-1457