Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture

From: Zu Qiang <[email-masked]>
Date: Thu, 24 Oct 2013 19:12:58 +0000

Hello,
Sorry to jump into this discussion. A few questions on the Distributed Gateways definition. 
- There is Gateway defined in section 5.3. Do we still need a Gateway when Distributed Gateways are enabled in the NVEs? Maybe yes. Please clarify it in the draft.
- Assuming the Distributed Gateway is defined for L3 service, right? Please clarify it in the draft.
- For L3 service, does the Distributed Gateway support routing or forwarding or both? There is no routing protocol running between the Distributed Gateways, right? I assume it is a Yes as it is "relaying" function only. Maybe the Distributed Gateways can be renamed to Distributed Forwarding. Or a clarification needs to be added.
- The text in 5.4 implicitly say that the forwarding policies are updated by the NVA. This may be ok if user plane routing is not in the scope. If there is a vR installed in a VM as an user plane router, there may be routing communications between the vR and the Gateway (or Distributed Gateways) which may have an impact on the forwarding policies. Do we expect any forwarding policies updates due to above data plane routing communications? I hope it is a No. Maybe it is better to have it clarified in the draft. 

Have a nice day
Zu Qiang

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Larry Kreeger (kreeger)
>Sent: Friday, October 18, 2013 6:00 PM
>To: Thomas Narten; Lucy yong
>Cc: [email-masked]
>Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
>document]
>
>Hi Thomas and Lucy,
>
>The WG needs to think hard about this one.
>
>Support of a distributed L3 gateway function between L2 VNs is a significant
>increase in scope of the NVA, and the NVE to NVA protocol.  Where we had
>previously stated L2 service or L3 service and pretty much left a combined
>L2/L3
>service as an exercise for the reader, we would now be adding   whatever
>mechanisms are needed to the protocols.  We will need to add cases for L2
>service, L3 service and L2/L3 service.  We no longer have simple inner to outer
>mappings, but now need NVEs to do MAC rewrites, local NVE ARP termination,
>and multiple lookups depending on the destination MAC address (first L2,
>then potentially L3).  We will also need to distribute two different VN
>identifiers (one for L2 and one for L3), and somehow convey the containment
>relationship between the two (multiple L2 VNs within one
>L3 VN).  While I think this is all very useful, I just want to make sure the WG
>agrees to this since I feel it is a significant change/increase in scope from my
>perspective.
>
>Thanks, Larry
>
>
>
>On 10/18/13 2:52 PM, "Thomas Narten" <[email-masked]> wrote:
>
>>Hi Lucy.
>>
>>Lucy yong <[email-masked]> writes:
>>
>>> Section 5.3 describes gateways. IMO: it misses an important use
>>> case. A Gateway, say overlay gateway, may be used to interconnect
>>> two or more overlay VNs. In this case, the traffic traversing
>>> between two overlay VNs must go through the gateway where the  policy
>>> can be enforced. Furthermore, it is possible to implement
>>> centralized or distributed overlay gateway. The latter has overlay
>>> gateway function implemented on NVEs. Thus, it requests the  cross-VN
>>> policies to be distributed to NVEs.
>>
>>> Current section seems very focus on overlay VN interconnect a
>>> non-overlay network, which centralized gateway architecture is
>>> practical. But in overlay networks, both centralized or distributed
>>> are possible and depend on the applications.
>>
>>Agreed. I propose adding a new section after 5.3 that says:
>>
>>      <section title="Distributed Gateways">
>>	<t>
>>	  The relaying of traffic from one VN to another deserves
>>	  special consideration. The previous section described
>>	  gateways performing this function. If such gateways are
>>	  centralized, traffic between TSes on different VNs can take
>>	  suboptimal paths, i.e., triangular routing results in paths
>>	  that always traverse the gateway. As an optimization,
>>	  individual NVEs can be part of a distributed gateway that
>>	  performs such relaying, reducing or completely eliminating
>>	  triangular routing. In a distributed gateway, each ingress
>>	  NVE can perform such relaying activity directly, so long as
>>	  it has access to the policy information needed to determine
>>	  whether cross-VN communication is allowed. Having individual
>>	  NVEs be part of a distributed gateway allows them to tunnel
>>	  traffic directly to the destination NVE without the need to
>>	  take suboptimal paths.
>>	</t>
>>	<t>
>>	  The NVO3 architecture should [must? or just say it does?]
>>	  support distributed gateways. Such support requires that
>>	  NVO3 control protocols include mechanisms for the
>>	  maintenance and distribution of policy information about
>>	  what type of cross-VN communication is allowed so that NVEs
>>	  acting as distributed gateways can tunnel traffic from one
>>	  VN to another as appropriate.
>>	</t>
>>      </section>
>>
>>Thoughts?
>>
>>Thomas
>>
>>_______________________________________________
>>nvo3 mailing list
>>[email-masked]
>>https://www.ietf.org/mailman/listinfo/nvo3
>
>_______________________________________________
>nvo3 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Black, David" <[email-masked]>
Date: Thu, 24 Oct 2013 16:00:33 -0400

> - There is Gateway defined in section 5.3. Do we still need a Gateway when
> Distributed Gateways are enabled in the NVEs? Maybe yes. Please clarify it in
> the draft.

Gateway = function, Distributed Gateway = implementation of Gateway function.

> - Assuming the Distributed Gateway is defined for L3 service, right? Please

Wrong - it applies to both L2 and L3 service.  The typical L3VPN implementation
distributes the gateway via the routing infrastructure, so most of the list
discussion attention has been on L2 service and distribution of the gateway to
avoid triangle or trombone routing.

> - For L3 service, does the Distributed Gateway support routing or forwarding
> or both? There is no routing protocol running between the Distributed
> Gateways, right? I assume it is a Yes as it is "relaying" function only. Maybe
> the Distributed Gateways can be renamed to Distributed Forwarding. Or a
> clarification needs to be added.

For L3 service, please consult some material on how BGP/MPLS L3VPNs work; the
answers to your questions can be found there.

> - The text in 5.4 implicitly say that the forwarding policies are updated by
> the NVA. This may be ok if user plane routing is not in the scope. If there is
> a vR installed in a VM as an user plane router, there may be routing
> communications between the vR and the Gateway (or Distributed Gateways) which
> may have an impact on the forwarding policies. Do we expect any forwarding
> policies updates due to above data plane routing communications? I hope it is
> a No. Maybe it is better to have it clarified in the draft.

Ok, good catch - I agree that this topic should be noted, and the question on
forwarding policy updates over the virtualized data plane is one for the WG to
discuss, IMHO, even though I'd also like to start from a "No" answer.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Zu
> Qiang
> Sent: Thursday, October 24, 2013 3:13 PM
> To: Larry Kreeger (kreeger); Thomas Narten; Lucy yong
> Cc: [email-masked]
> - There is Gateway defined in section 5.3. Do we still need a Gateway when
> Distributed Gateways are enabled in the NVEs? Maybe yes. Please clarify it in
> the draft.
> - For L3 service, does the Distributed Gateway support routing or forwarding
> or both? There is no routing protocol running between the Distributed
> Gateways, right? I assume it is a Yes as it is "relaying" function only. Maybe
> the Distributed Gateways can be renamed to Distributed Forwarding. Or a
> clarification needs to be added.
> - The text in 5.4 implicitly say that the forwarding policies are updated by
> the NVA. This may be ok if user plane routing is not in the scope. If there is
> a vR installed in a VM as an user plane router, there may be routing
> communications between the vR and the Gateway (or Distributed Gateways) which
> may have an impact on the forwarding policies. Do we expect any forwarding
> policies updates due to above data plane routing communications? I hope it is
> a No. Maybe it is better to have it clarified in the draft.
> 
> Have a nice day
> Zu Qiang
> 
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Thu, 24 Oct 2013 22:43:42 +0000

Hi David and Zu,

My responses are below, marked with LK>.

 - Larry

>Gateway = function, Distributed Gateway = implementation of Gateway

LK> While I agree with the statement above, section 5.3 "Gateways" focuses
mostly on gateways that connect VNs to non-overlay networks.  E.g. Connect
a L2 VN to a VLAN, connect an L2 or L3 VN to the internet.  It does have
one sentence "Gateways could also forward traffic between a virtual
network and
   other hosts on the data center network or relay traffic between
different VNs." which mentions relaying traffic between VNs. I am
presuming that VN to physical gateways are not distributed gateways, but
VN to VN gateways could be distributed.

>
>> - Assuming the Distributed Gateway is defined for L3 service, right?
>>Please
>> clarify it in the draft.
>

LK> But is there a requirement to connect two L2 VNs at L2?  I was
assuming that was an uninteresting case.

>discuss, IMHO, even though I'd also like to start from a "No" answer.

LK> I would also like to start from "No".

>>Zu
>> - Assuming the Distributed Gateway is defined for L3 service, right?
>>Please
>> clarify it in the draft.
>> >service, L3 service and L2/L3 service.  We no longer have simple inner
>>to
>> outer
>> >L3 VN).  While I think this is all very useful, I just want to make
>>sure the
>> WG
>> >agrees to this since I feel it is a significant change/increase in
>>scope from
>> my
>> >perspective.
>> >
>> >Thanks, Larry
>> >
>> >
>> >

---

From: "Black, David" <[email-masked]>
Date: Thu, 24 Oct 2013 20:35:38 -0400

> I am

I would not presume the former, although this discussion has been primarily
about the latter.  In particular it is definitely possible that different
NVEs in a data center could have different preferred routes to the public
Internet, so that a distributed implementation of the default gateway for
IP could result in better usage of that external connectivity to the public
Internet.

> assuming that was an uninteresting case.

I would agree that it is uninteresting but there are two different L3 concepts
in this discussion:

"L3 service" provided by VNs envisions an L3VPN class of implementation where
there are no L2 adjacencies.  L3VPN does have a concept of connectivity among
groups, and that connectivity implementation is fully distributed via the
routing infrastructure that is used to convey mappings to the NVEs.

For "L2 service" the L3 gateway implementation among L2 VNs may be distributed
among the NVEs.

I believe that the Distributed Gateway discussion will need to encompass both
the "L2 service" and "L3 service" cases, and that was the basis of my statement
that Distributed Gateways apply to both types of service.

Thanks,
--David

> From: Larry Kreeger (kreeger) [mailto:[email-masked]]
> Sent: Thursday, October 24, 2013 6:44 PM
> To: Black, David; Zu Qiang
> Cc: [email-masked]

---

From: Zu Qiang <[email-masked]>
Date: Fri, 25 Oct 2013 14:13:57 +0000

Hello,
	First of all I agree that the gateway or distributed gateway may support both L3 routing and L2 switching. Even the latter is less interesting. But it is possible.  This discussion is primarily about how to support  L3 routing with the distributed gateway in NVO3 architecture. Let's take a few implementation alternatives for our discussion.
	Alt. a) Let's assume each distributed gateway supports L3 routing/forwarding function. Among the distributed gateways, a L3VPN can be setup. 
	Alt. b) Let's assume there is a logic L3 default gateway setup for a tenant. The RIB is managed by the default gateway. Based on the routing policies, the default gateway will install a FIB in each distributed L3 gateways. In this case, routing is supported by the default gateway, and forwarding is supported in each distributed gateway function.
	Alt. c) NVA-NVE protocol is used to configure the L3 mapping table in each distributed gateway enabled NVE. No NVE-NVE control plane is used. In this case, only L3 forwarding is supported in each distributed gateway function. There is no routing function supported in the NVO3.
	In all the above three alternatives, all the distributed gateways are grouped together as one (or more) L3 default gateway in the tenant view.  However, the different is that both a) and b) are supporting dynamic routing updates, e.g. if there is user plane routing function enabled. Alternative c) is more like a support of static L3 forwarding function with only FIB configured in the NVEs. Both alternative a) and b) require a NVE-NVE control plane protocol for RIB and FIB updating, unless the routing protocol is considered as data plane protocol which is out of NVO3 scope. Alternative c) only needs NVA-NVE control plane for the NVE configurations. 
	My reading from the architecture draft is that there is no support of a NVE-NVE control plane. I see three options for the WG to discuss:
	1. make it clear in the architecture draft that only L3 forwarding function is supported with the distributed gateway
	2. allows both Alt. a) and Alt. b), with the considerations that the L3 routing protocol as data plane protocol. The NVA-NVE protocol is used for initial configuration of the distributed gateway only. 
	3. defining a NVE-NVE control plane to support dynamic routing function

	L3 forwarding needs to be supported with the distributed gateway anyway. So I would propose to the WG to support option 1 for the first step of the architecture work. Both option 2 and option 3 can be a long term discussion. 

Have a nice day
Zu Qiang

>>> - There is Gateway defined in section 5.3. Do we still need a Gateway
>>>when  Distributed Gateways are enabled in the NVEs? Maybe yes. Please
>>>clarify it in  the draft.
>>
>LK> While I agree with the statement above, section 5.3 "Gateways"
>LK> focuses
>a L2 VN to a VLAN, connect an L2 or L3 VN to the internet.  It does have one
>sentence "Gateways could also forward traffic between a virtual network and
>   other hosts on the data center network or relay traffic between different
>VNs." which mentions relaying traffic between VNs. I am presuming that VN
>to physical gateways are not distributed gateways, but VN to VN gateways
>could be distributed.
>
>>
>>Wrong - it applies to both L2 and L3 service.  The typical L3VPN
>>implementation distributes the gateway via the routing infrastructure,
>>so most of the list discussion attention has been on L2 service and
>>distribution of the gateway to avoid triangle or trombone routing.
>
>>> - For L3 service, does the Distributed Gateway support routing or
>>>forwarding  or both? There is no routing protocol running between the
>>>Distributed  Gateways, right? I assume it is a Yes as it is "relaying"
>>>function only. Maybe  the Distributed Gateways can be renamed to
>>>Distributed Forwarding. Or a  clarification needs to be added.
>>
>>For L3 service, please consult some material on how BGP/MPLS L3VPNs
>>work; the answers to your questions can be found there.
>>
>>> - The text in 5.4 implicitly say that the forwarding policies are
>>>updated by  the NVA. This may be ok if user plane routing is not in
>>>the scope. If there is  a vR installed in a VM as an user plane
>>>router, there may be routing  communications between the vR and the
>>>Gateway (or Distributed Gateways) which  may have an impact on the
>>>forwarding policies. Do we expect any forwarding  policies updates due
>>>to above data plane routing communications? I hope it is  a No. Maybe
>>>it is better to have it clarified in the draft.
>>
>>Ok, good catch - I agree that this topic should be noted, and the
>>question on forwarding policy updates over the virtualized data plane
>>is one for the WG to discuss, IMHO, even though I'd also like to start
>>from a "No" answer.
>
>>> From: [email-masked] [mailto:[email-masked]] On Behalf
>>>Of Zu  Qiang
>>> - There is Gateway defined in section 5.3. Do we still need a Gateway
>>>when  Distributed Gateways are enabled in the NVEs? Maybe yes. Please
>>>clarify it in  the draft.
>>> - For L3 service, does the Distributed Gateway support routing or
>>>forwarding  or both? There is no routing protocol running between the
>>>Distributed  Gateways, right? I assume it is a Yes as it is "relaying"
>>>function only. Maybe  the Distributed Gateways can be renamed to
>>>Distributed Forwarding. Or a  clarification needs to be added.
>>> - The text in 5.4 implicitly say that the forwarding policies are
>>>updated by  the NVA. This may be ok if user plane routing is not in
>>>the scope. If there is  a vR installed in a VM as an user plane
>>>router, there may be routing  communications between the vR and the
>>>Gateway (or Distributed Gateways) which  may have an impact on the
>>>forwarding policies. Do we expect any forwarding  policies updates due
>>>to above data plane routing communications? I hope it is  a No. Maybe
>>>it is better to have it clarified in the draft.
>>>
>>> Have a nice day
>>> Zu Qiang
>>>
>>>
>>> >From: [email-masked] [mailto:[email-masked]] On Behalf
>>> >Of Larry Kreeger (kreeger)
>>> >mechanisms are needed to the protocols.  We will need to add cases
>>> >for
>>>L2
>>> >then potentially L3).  We will also need to distribute two different
>>> >VN identifiers (one for L2 and one for L3), and somehow convey the
>>>containment
>>> >>> centralized or distributed overlay gateway. The latter has
>>> >>> overlay gateway function implemented on NVEs. Thus, it requests
>>> >>> the
>>>cross-VN
>>> >>> practical. But in overlay networks, both centralized or
>>> >>> distributed are possible and depend on the applications.
>>> >>