Subject: Re: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

From: Lucy yong <[email-masked]>
Date: Tue, 3 Sep 2013 16:37:08 +0000

Sorry for late reply due to travel and holiday. Please see my reply inline w/ [Lucy1]

From: Pankaj Garg [mailto:[email-masked]]
Sent: Monday, August 26, 2013 1:50 PM
To: Lucy yong; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: comments on draft-sridharan-virtualization-nvgre-03

My comments are inline.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, August 22, 2013 4:46 PM
To: Pankaj Garg; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

Hi Pankaj,

Please see my reply in-line below.

Regards,
Lucy

From: Pankaj Garg [mailto:[email-masked]]
Sent: Thursday, August 22, 2013 4:18 PM
To: Lucy yong; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: comments on draft-sridharan-virtualization-nvgre-03

Hi Lucy,

My comments are inline.

Thanks
Pankaj

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Tuesday, August 20, 2013 2:49 PM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

Hi Authors,

I read this new version and have a major concern on the ECMP support.

*         The new version adds ECMP support section (4.1) and require switches and routers SHOULD provide ECMP on the NVGRE packet using the outer frame fields and entire Key field (32-bit).  This means that the solution not only requires the change on the endpoint devices but also the change on the intermediate switches and routers, which put the high bar for the network. Rare switches and routers today support GRE key based load balance.
[PG] - This requirement has always been there as without using the key field, there is not enough entropy for ECMP in the network. There are two options for GRE based network virtualization, either use the outer frame fields and the key field or calculate entropy based on the inner packet fields. Many of the newer current generation switches and router silicon supports ECMP hashing for NVGRE packets using outer frame fields and key field. However, the older devices would not be able to do that.
[Lucy] There is another option beside these two. That is described in the draft mentioned below, which has no change request on intermediate routers/switches. Many older devices are there now. I don't see the reason to replace them for supporting this feature. In addition, both described options create complex logic to existing LB process. Today switches/routers widely support 5 tuple based LB, which we should leverage instead of adding unnecessary complex to it.
[Lucy1] We should promote this approach as part of NVGRE solution?

*         To support GRE Key based load balancing, the hardware has to check the GRE Key presence and the protocol type to avoid a mis-operation on other GRE usages, which is bad.

*         Using 8 bits in the Key field for flow entropy is not sufficient for all the applications.
[PG] - The entropy is defined by 4 fields in NVGRE, outer source IP, outer destination IP, VSID and FlowId. The VSID is the only thing that is static for one virtual network. All other fields can change and that can provide additional entropy needed in the large networks.
[Lucy] I mean that one virtual network may contain many and many flows, mapping all the flows into 8 bits flow entropy does not give a good granularity, which may potentially cause uneven load balancing issue. Hashing based load balancing is based on statistical. Without a large amount of flow entropy, it will not work well.
[PG] This is incorrect. The entropy is defined by outer source IP, destination IP, VSID and FlowId. In a specific virtual network, only VSID is constant, hence the flow would be mapped based on source IP, destination IP and FlowId. Did you mean to say that between two NVE sending traffic on same VSID, all flows would map based on 8-bit entropy? If yes, then I agree, however different source IP can be used to add additional entropy in such cases if necessary.
[Lucy1] Yes, I mean to say that two NVE sending traffic on same VSID, all flows would map based on 8-bit entropy. That may not provide good flow granularity for the load balance in core. You can use difference source IP addresses, but it adds the complex in many aspects and not scale. Source IP address is used as the tunnel EP identification and has some special properties.

*         Directly disclosing VSID to underlying network can be a security concern in some cases.
[PG] - In theory, it is possible to use IPsec as the transport to protect the GRE header, however then you lose ECMP etc. I am also not sure why this is a concern since the virtual network ID is disclosed in the network in all network virtualization tunneling proposals out there, correct?
[Lucy] My point is that the fields that are processed by underlying network (intermediate nodes) should not be the fields belonging to the virtual network. If we want the overlay virtual network is decoupled from the physical infrastructure, we should do that consistently. To let overlay and underlying operate on the same field on packets  is bed idea, IMHO. It creates some dependency and restriction for both networks. For example, what if virtual networks span across several WAN networks, and some WAN operator want to create its own flow entropy by rehashing some fields on the packets? Your two options do not give that flexibility. IMHO, it is important to have flow entropy field on packet header for the underlying network and allow the end point to fill the flow entropy there.
[PG] Yes, this is something not allowed in the spec i.e. modification of GRE key field by transit devices.
[Lucy1] if we separate GRE key field and flow entropy field. Underlying operator can manipulate the flow entropy field if they want to.

*         The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex.
[PG] - There is some operational overhead, but I believe it is minimum as one can assign multiple source IP to each NVE to gain additional entropy for ECMP. The tunneling operation can select the source IP based on the hash of inner packet headers.
[Lucy] I don't believe that is simple to manage.
[PG] Actually it is not hard, because all you need to do is assign more than one IP to the NVE.  However, it depends upon the operation model. In Hyper-V Network Virtualization, the network virtualization policy is handled by a central controller and hence it is easy to configure the NVE to provide more hashing if needed.
[Lucy1] As I mentioned about, outer IP address is used as Tunnel EP identification. It is routed in underling network. Having many source IP or destination IP address for flow entropy purpose, it may add unnecessary routing burden for core networks. It may also adds complex to other functions such as OAM.

We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support.
[PG] - Thanks, I would take a look at the proposal, however I treat this as another tunneling protocol in addition to three existing ones i.e. NVGRE, VxLAN and STT. What are the specific advantages of this over other protocols? Also many of current generation NIC hardware support offloads for the existing three protocols. Wouldn't using GRE-IN-UDP break those offloads?
[Lucy] If NIC hardware support VxLAN which use UDP port for the flow entropy, it should be easy to support gre-in-udp encaps as well. Will be great  to see chip vendor comment on this. Please take a look at the gre-in-udp encapsulation proposal (below), welcome comment on it.
[PG] Sure, I would take a look at it.
[Lucy1] Have you look at it? It is a simple draft.
Lucy

Regards,
Lucy

Here is the draft. The TSVWG will adopt it as WG draft soon.

http://datatracker.ietf.org/doc/draft-yong-tsvwg-gre-in-udp-encap/

Regards,
Lucy

---

From: "Pat Thaler" <[email-masked]>
Date: Wed, 18 Sep 2013 23:41:52 +0000

Dear Lucy,

I'm responding to your question to chip vendors about using NVGRE with your UDP encap draft.

It isn't that supporting any one packet format in hardware is a problem. The issue is the number of formats that need to be supported by a chip - each may be small but the number adds up. Adding another packet format variant isn't justified and would be burdensome for the chips.

The VXLAN packet format already provides for use cases where a flow field in the UDP header is desired. There is no need to duplicate that capability by adding an NVGRE in UDP format. We would be strongly against udp encapsulation of NVGRE.

Regards,
Pat

Regards,
Lucy

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Thu, 19 Sep 2013 01:11:16 +0000

From: Pat Thaler [mailto:[email-masked]]
Sent: Wednesday, September 18, 2013 6:42 PM
To: Lucy yong; Pankaj Garg; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: comments on draft-sridharan-virtualization-nvgre-03

Dear Lucy,

It isn't that supporting any one packet format in hardware is a problem. The issue is the number of formats that need to be supported by a chip - each may be small but the number adds up. Adding another packet format variant isn't justified and would be burdensome for the chips.
[Lucy] My suggestion is not about adding a new format. It is to enhance existing NVGRE protocol.

The VXLAN packet format already provides for use cases where a flow field in the UDP header is desired. There is no need to duplicate that capability by adding an NVGRE in UDP format. We would be strongly against udp encapsulation of NVGRE.
[Lucy] This is not about duplicate that capability, it is about use of a common way in supporting underlying ECMP. Do you support both NVGRE and VXLAN encapsulation, or only VXLAN encapsulation?

Regards,
Lucy

Regards,
Pat

Regards,
Lucy

Regards,
Lucy

---

From: "Pat Thaler" <[email-masked]>
Date: Fri, 20 Sep 2013 21:34:49 +0000

From: Lucy yong [mailto:[email-masked]]
Sent: Wednesday, September 18, 2013 6:11 PM
To: Pat Thaler; Pankaj Garg; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: comments on draft-sridharan-virtualization-nvgre-03

To: Lucy yong; Pankaj Garg; [email-masked]<mailto:[email-masked]>

Dear Lucy,

>From the point of view of hardware handling of the frames, putting the UDP header in between the IP and NVGRE headers makes it a new format for us.

The VXLAN packet format already provides for use cases where a flow field in the UDP header is desired. There is no need to duplicate that capability by adding an NVGRE in UDP format. We would be strongly against udp encapsulation of NVGRE.
[Lucy] This is not about duplicate that capability, it is about use of a common way in supporting underlying ECMP. Do you support both NVGRE and VXLAN encapsulation, or only VXLAN encapsulation?
There are Broadcom authors on both the VXLAN and NVGRE I-Ds. We support both formats going forward.

Regards,
Lucy

Regards,
Pat

Regards,
Lucy

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Sat, 21 Sep 2013 15:09:16 +0000

Hi Pat,

Technology evolution does not stop because of many technologies existence. It continues because existing technologies let people see the better technologies.

The NVGRE/VXLAN overlay technology principle is about the separation of virtual networks from the physical transport network, i.e. virtual network traffic encapsulated in IP data grams and just likes another IP application. Using UDP src port for flow entropy is clear way to support underlying ECMP without breaking this principle and requests no change to underlying network (ECMP becomes the common underlying network capability). Yes, VXLAN encapsulation adopts this mechanism at first. IMO: current NVGRE encapsulation does not align with this principle well. It is a particular IP application (GRE), not like another IP application (TCP/UDP); it requests the underlying ECMP LB to use GRE header, which ties overlay network and underlying network together in some degree. This is why we propose gre-in-udp encapsulation to improve gre as of a generic encapsulation for an overlay technology. The semantics has the key components for an overlay technology.  NVGRE is designed for virtual network overlay and uses gre header. Thus, IMO, it is merit for NVGRE to use gre-in-udp format.

Although VXLAN is good to specify the use of UDP initially, but VXLAN header does not have protocol type field and only allow carrying Ethernet payload. It is designed for one purpose only as its name. So it can't apply to other overlay application without enhancement. NVGRE uses GRE header that has a protocol type field so it can easily extend for other overlay applications.  If you think that gre-in-udp for NVGRE or adding a protocol type field in VXLAN is new format and strongly against them. You soon will see a new encapsulation proposal to enable other overlay application such as service function chaining.  I don't see you have a way to get around that.

We should be able to specify one generic overlay encapsulation format that can support an overlay application.  The nvgre and vxlan enhancements we proposed makes them to be equally good to serve this purpose because both will contain the key components required for an overlay application and align with the overlay principle .  It would be great if IETF can standardize just one, but given existing deployment situation, this may not be practical. We leave the choice to the community.

Regards,
Lucy

Dear Lucy,

Regards,
Lucy

Regards,
Pat

Regards,
Lucy

Regards,
Lucy