Subject: [nvo3] Sugggested addition-replacement to draft-narten-nvo3-arch-01

From: Linda Dunbar <[email-masked]>
Date: Mon, 18 Nov 2013 20:10:30 +0000

Dear Authors,
I posted those comments and suggested text changes during the WG Adoption process. Since I haven't heard any replies from the authors, I am re-posting my suggested addition and change of text to draft-narten-nvo3-arch-01 plus a few more suggestions to clear out issues being discussed on the mailing list.

Issues with the current writing of Distributed Gateway (Section 5.4):
As described in Section 5.3, a Gateway does many things. However, I don't think that a NVE, if taking on the responsibility of a distributed Gateway, will do all the things that a conventional Gateway does (or the list of items mentioned in the Section 5.3).
First, it might be too much to ask a NVE based gateway (especially Hypervisor based NVE) to

*        relay traffic off the virtual networks, i.e. perform gateway function to reach destinations outside the local VNs,

*        serve as IPSec gateway to external  (i.e. out of Virtual Networks),

*        perform NAT on the source virtual addresses, or

*        relay traffic to a VN that doesn't have any hosts attached to the NVE  (it is debatable if a NVE based distributed Gateway should take this responsibility)
The meaningful functions performed by NVE, if designated as "distributed gateway", are more like Inter-VN relay (instead of full blown Gateway function).
Second, when host "a" in VN-1 sends traffic to "b" in VN-2, the data packet's Ethernet header has "MAC-DA = Gateway-MAC & MAC-SA= a-MAC & VLAN= VN-1".  Most implementations (Microsoft Window 8 and VM NSX) allocate a "fake MAC" for all the NVE based distributed gateways to share, so that host "a" can use the same Gateway-MAC when moved to another NVE. This again is different from the conventional gateways.
Third, the issue of conventional gateway (i.e. a->b traffic to be routed at gateway even if "a" & "b" are attached to the same NVE) is traffic "hairpinning", instead of triangular routing.
Therefore, I suggest rewriting Section 5.4 as below:
5.4 Distributed Gateway (Re-write)
The relaying of traffic from one VN to another, especially when Source and Destination are attached to the same NVE, deserves special consideration. With conventional Gateways, the traffic between TSes on different VNs has to be traversed to the gateway, even if the Source and Destination are attached to the save NVE, which causes traffic hairpinning and wasted bandwidth.
As an optimization, it is desirable for individual NVEs to take over the inter-VN relay responsibilities that are traditionally done by conventional gateways to reduce or eliminate hairpinning issue. In order for NVEs to perform inter-VN relay, the NVEs must have access to the policy information needed to determine whether inter-VN communication is allowed. Those inter-VN communication policies are most likely to come from NVA.
However, it is not practical for NVEs to take over all functions of conventional gateways.
In particular, it might be too much to ask a NVE based gateway (especially Hypervisor based NVE) to

*        relay traffic off the virtual networks, i.e. perform gateway function to reach destinations outside the local VNs,

*        serve as IPSec gateway to external  traffic

*        perform NAT on the source virtual addresses, or

*        relay traffic to a VN that doesn't have any hosts attached to the NVE.
The NVEs that are capable of performing inter-VN replaying are called "Distributed Gateway" in this document. (Note: Inter-VN relaying capable NVE is a more accurate term).
The NVO3 architecture should support distributed gateways, at least allowing some NVEs, if not all, supporting the inter-VN relaying function, especially when both source and destination are attached to the same NVE.  Such support requires the NVO3 control protocols include mechanisms for the maintenance and distribution of the inter-VN policy information to the NVEs that are capable of performing inter-VN communications.

There are many emails on the list with regard to the necessity of having NVE<->NVE control plane protocol. All those emails discussion warrants a subsection under Section 4 to explain why/how the consequence of not having NVE<->NVE control plane protocol is a minor issue (or not worth addressing) in the environment addressed by NVO3. Here is my suggested text:

4.4 Is it necessary to have Inter-NVE control plane protocol? (Suggested new sub-section)
There could be various reasons, link failure, node failure, or others, causing egress NVEs not reachable. Without any NVE<->NVE control plane protocol, the ingress NVE is not aware of the reachability of egress NVE causing encapsulated packets to dropped somewhere in the underlay network.
 If most TSes are only attached to a single NVE and traffic to NVEs are not aggregated flows, then the NVE-NVE control plane protocol doesn't provide any benefits.   Under this environment, the difference between having "NVE-NVE control protocol" is whether the packets being dropped at the Ingress NVE or somewhere in the underlay network when egress NVE is not reachable.
In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big.
However, in an environment where TSes are connected to multiple NVEs, then it might be worthwhile to consider the Inter-NVE control plane protocol, so that ingress NVE can choose different egress NVE for a given target.

Thanks for considering my suggested text.

Linda Dunbar

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 12:25:21 -0500

Linda,

Linda Dunbar <[email-masked]> writes:

> I posted those comments and suggested text changes during the WG
>  Adoption process. Since I haven’t heard any replies from the
>  authors, I am re-posting my suggested addition and change of text
>  to draft-narten-nvo3-arch-01 plus a few more suggestions to clear
>  out issues being discussed on the mailing list.

Apologies for not getting back sooner. These had not been forgotten,
but they also require more involved response.

> As described in Section 5.3, a Gateway does many things. However, I
> don’t think that a NVE, if taking on the responsibility of a
> distributed Gateway, will do all the things that a conventional
> Gateway does (or the list of items mentioned in the Section 5.3).

To be clear, Section 5.4 "Distribute Gateways" was really about
Inter-VN gateways. The heading could have been clearer.

> ·        relay traffic off the virtual networks, i.e. perform gateway function to reach destinations outside the local VNs, 
> ·        serve as IPSec gateway to external  (i.e. out of Virtual Networks),
> ·        perform NAT on the source virtual addresses, or
> ·        relay traffic to a VN that doesn’t have any hosts attached to the NVE  (it is debatable if a NVE based distributed Gateway should take this responsibility)
> 
> The meaningful functions performed by NVE, if designated as
> “distributed gateway”, are more like Inter-VN relay (instead of full
> blown Gateway function).

Mostly agree. However, the purpose of the architecture is not to say
what implementations MUST do, it's about what the architecture must
allow, in order for implementations (and solutions) have the option of
implementing something. So at this point, we shouldn't get too caught
up in exact details of what would and would not make sense to
distribute.

> Second, when host “a” in VN-1 sends traffic to “b” in VN-2, the data
> packet’s Ethernet header has “MAC-DA = Gateway-MAC & MAC-SA= a-MAC &
> VLAN= VN-1”.  Most implementations (Microsoft Window 8 and VM NSX)
> allocate a “fake MAC” for all the NVE based distributed gateways to
> share, so that host “a” can use the same Gateway-MAC when moved to
> another NVE. This again is different from the conventional
> gateways.

Sure.

> Third, the issue of conventional gateway (i.e. a->b traffic to be
>  routed at gateway even if “a” & “b” are attached to the same NVE)
>  is traffic “hairpinning”, instead of triangular routing.

Sure. It's a special case of triangular routing.

> Therefore, I suggest rewriting Section 5.4 as below:

I took some of your text, but think it would be better to pull out the
policy stuff into its own section.

Here is proposed text:

      <section title="Distributed Inter-VN Gateways">
	<t>
      	  The relaying of traffic from one VN to another deserves
	  special consideration.  Whether traffic is permitted to flow
	  from one VN to another is a matter of policy, and would not
	  (by default) be allowed unless explicitely enabled. In
	  addition, NVAs are the logical place to maintain policy
	  information about allowed inter-VN communication.  Policy
	  enforcement for inter-VN communication can be handled in (at
	  least) two different ways. Explicit gateways could be the
	  central point for such enforcement, with all inter-VN
	  traffic forwarded to such gateways for
	  processing. Alternatively, the NVA can provide such
	  information directly to NVEs, by either providing a mapping
	  for a target TS on another VN, or indicating that such
	  communication is disallowed by policy.
	</t>

above is new text.	

	<t>
          When inter-VN gateways are centralized, traffic between TSes
	  on different VNs can take suboptimal paths, i.e., triangular
	  routing results in paths that always traverse the
	  gateway. In the worst case, traffic between two TSes
	  connected to the same NVE can be hairpinned through an
	  external gateway. As an optimization, individual NVEs can be
	  part of a distributed gateway that performs such relaying,
	  reducing or completely eliminating triangular routing. In a
	  distributed gateway, each ingress NVE can perform such
	  relaying activity directly, so long as it has access to the
	  policy information needed to determine whether cross-VN
	  communication is allowed. Having individual NVEs be part of
	  a distributed gateway allows them to tunnel traffic directly
	  to the destination NVE without the need to take suboptimal
	  paths.
	</t>

Mostly same as in existing doc, just added "hairpinning" text.

	<t>
	  The NVO3 architecture must support distributed gateways for
	  the case of inter-VN communication. Such support requires
	  that NVO3 control protocols include mechanisms for the
	  maintenance and distribution of policy information about
	  what type of cross-VN communication is allowed so that NVEs
	  acting as distributed gateways can tunnel traffic from one
	  VN to another as appropriate.
	</t>

Cleaned up and just said "must"

	<t>
	  Distributed gateways could also be used to to distribute
	  other traditional router services to individual NVEs. The
	  NVO3 architecture does not preclude such implementations,
	  but does not define or require them as they are outside the
	  scope of NVO3.
	</t>

Above is to basically say distributed functionality that is non-NVO3
related is out of scope for NVO3 to have an opinion on. It means
implementations can do more or less what they want, NVO3 doesn't
really need to take a postion.

I'll respond to your other points in a subsequent message.

Thomas