Subject: [nvo3] NVO3 Architecture document

From: Thomas Narten <[email-masked]>
Date: Fri, 11 Oct 2013 09:04:01 -0400

draft-narten-nvo3-arch-00.txt is intended to become a WG document, but
to date we've received limited feedback with specific actions.  If
folk have changes/additions/clarifications they'd like to see, or
disagree with particular aspects, please speak up.

For example, Section's 6-8 talk about NVAs and a federated
model. These concepts have been discussed several times over the last
year and appear to have general support. That said, some of the
sections have "notes" specifically asking for further
discussion/feedback.

Thanks!

Thomas

---

From: Lizhong Jin <[email-masked]>
Date: Sat, 12 Oct 2013 10:53:16 +0800

Hi Thomas,
In section 4, you describe two NVE models. The model name "NVE Co-located
With Server Hypervisor" is not so accurate, and I suggest to change to
"NVE Co-located With Server" or "NVE integrated in Server". In the
implementations, part of NVE function will be performed by the adaptor,
not hypervisor. The function performed by adaptor will be more than
checksum offload and LRO/TSO, it is even possible in the future that the
full NVE is embedded in adaptor.

Regards
Lizhong

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]]
Sent: 2013年10月11日 21:04
To: [email-masked]
Subject: [nvo3] NVO3 Architecture document

draft-narten-nvo3-arch-00.txt is intended to become a WG document, but to
date we've received limited feedback with specific actions. If folk have
changes/additions/clarifications they'd like to see, or disagree with
particular aspects, please speak up.

For example, Section's 6-8 talk about NVAs and a federated model. These
concepts have been discussed several times over the last year and appear
to have general support. That said, some of the sections have "notes"
specifically asking for further discussion/feedback.

Thanks!

Thomas

---

From: Zu Qiang <[email-masked]>
Date: Wed, 16 Oct 2013 17:42:01 +0000

Hello,
	I have one comment which I have sent in the control plane draft discussion. But it may have an impact on the architecture draft as well. In section 7.1, there are two alternatives described. However it shall be possible for an implementation to use NVE-NVE protocol for inner-outer mapping updating. Using NVE-NVE protocol may not be the best solution. But as an architecture document, it may be good to list all the possibilities. And it may be better to have some analysis text on the pros and cons of each alternatives. I'm not sure if this is the right place to do this comparison. If yes, I do have some security considerations text of each alternatives in the security requirements draft [http://tools.ietf.org/html/draft-zu-nov3-security-requirements-01] . hope it can give some help for this discussion. 

Have a nice day
Zu Qiang

>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Thomas Narten
>Sent: Friday, October 11, 2013 9:04 AM
>To: [email-masked]
>changes/additions/clarifications they'd like to see, or disagree with particular
>aspects, please speak up.
>
>concepts have been discussed several times over the last year and appear to
>have general support. That said, some of the sections have "notes" specifically
>asking for further discussion/feedback.
>
>Thanks!
>
>Thomas
>
>_______________________________________________
>nvo3 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nvo3

---

From: Lucy yong <[email-masked]>
Date: Wed, 16 Oct 2013 21:08:40 +0000

Hi Thomas,

Here are some comments.

Section 5.3 describes gateways. IMO: it misses an important use case. A Gateway, say overlay gateway, may be used to interconnect two or more overlay VNs. In this case, the traffic traversing between two overlay VNs must go through the gateway where the policy can be enforced. Furthermore, it is possible to implement centralized or distributed overlay gateway. The latter has overlay gateway function implemented on NVEs. Thus, it requests the cross-VN policies to be distributed to NVEs. 

Current section seems very focus on overlay VN interconnect a non-overlay network, which centralized gateway architecture is practical. But in overlay networks, both centralized or distributed are possible and depend on the applications.

The draft barely touches Multicast/broadcast supports in describing NVE state in section 4.3. But it does not describe in section 7 at all. How to obtain the mapping for multicast traffic is much more complex. Simply statement " Since the NVA has current knowledge of which NVE each Tenant System is connected to, the NVA can proactively push updates out to the NVEs when they occur." is not enough for multicast transport.

It seems valuable to describe that NVA may send NVE a default route/gateway instead of tenant/location mapping and NVA can perform the route distribution control.

Regards,
Lucy  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, October 11, 2013 8:04 AM
To: [email-masked]
Subject: [nvo3] NVO3 Architecture document

draft-narten-nvo3-arch-00.txt is intended to become a WG document, but to date we've received limited feedback with specific actions.  If folk have changes/additions/clarifications they'd like to see, or disagree with particular aspects, please speak up.

For example, Section's 6-8 talk about NVAs and a federated model. These concepts have been discussed several times over the last year and appear to have general support. That said, some of the sections have "notes" specifically asking for further discussion/feedback.

Thanks!

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 17 Oct 2013 16:54:53 -0400

Lizhong,

> full NVE is embedded in adaptor.

I think it is important in the wording at issue above to include the
word "hypervisor" because Section 4.1 specifically addresses the case
of server virtualization. This is in contrast to Section 5.1, which
talks about the case of Network Service Appliances, where there is no
hypervisor, but the NVE is also implented on the server.

With that in mind, I'm a bit bit puzzled by the request. I don't think
it really changes Section 4.1 in any way. That makes me wonder if I'm
not understanding the real concern behind your request.

The second paragraph of Section 4.1 specifically talks about adaptor
offloads, making it clear that some functionality may be offloaded
onto the adaptor for performance reasons. If your concern is really
about properly addressing adaptor offloads, is there specific wording
in the second paragraph that you think needs tweaking?

Thanks!

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Thu, 17 Oct 2013 17:29:21 -0400

Zu Qiang <[email-masked]> writes:

> I have one comment which I have sent in the control plane draft
> discussion. But it may have an impact on the architecture draft as
> well. In section 7.1, there are two alternatives described. However
> it shall be possible for an implementation to use NVE-NVE protocol
> for inner-outer mapping updating. Using NVE-NVE protocol may not be
> the best solution. But as an architecture document, it may be good
> to list all the possibilities.

Agreed. That is why both are mentioned.

> And it may be better to have some
> analysis text on the pros and cons of each alternatives.

That I would like to avoid. The architecture document should describe
the direction the WG has chosen to go in. I don't think we should
document the pros/cons of various approaches (that is more what the
framework does).

Also, in the specific case above, I'm not sure listing pros/cons of
the two approaches helps. I don't expect folk to pick one or the other
based on the pros/cons. One of the starting premises is that there are
already existing orchestration systems that provide the functionality
of the NVA, and existing deployments use them. I don't expect
implementations already using such a facility to change. I do think we
will need to coexist with such approaches, hence they need to be  part
of the architecture.

Does that make sense?

Thomas

---

From: Zu Qiang <[email-masked]>
Date: Fri, 18 Oct 2013 13:12:44 +0000

Hello, Thomas
	Thanks for your email. see inline.

Have a nice day
Zu Qiang

>Sent: Thursday, October 17, 2013 5:29 PM
>To: Zu Qiang
>Cc: [email-masked]
>Subject: Re: [nvo3] NVO3 Architecture document
>
>> well. In section 7.1, there are two alternatives described. However it
>> shall be possible for an implementation to use NVE-NVE protocol for
>> inner-outer mapping updating. Using NVE-NVE protocol may not be the
>> best solution. But as an architecture document, it may be good to list
>> all the possibilities.
>
[Zu Qiang] I see three alternatives which are all described in the draft [http://tools.ietf.org/html/draft-zu-nov3-security-requirements-01]
A) If either the NVA is collocated with the VM Orchestration Systems, or there is an interface between the VM Orchestration Systems and NVA, it can be assumed that the NVA may learn the VN connection / disconnection and vNIC association / disassociation from the VM Orchestration Systems. Then the NVA configures the attached NVE with the VN context of the connected / disconnected VN and the associated / disassociated vNIC addresses. The next step is that the NVA updates the inner-outer address mapping table of the VN at both the attached NVE and all the participating remote NVEs.
B) The attached NVE is informed by the hypervisor using the Hypervisor-NVE protocol at VN connection / disconnection and vNIC association / disassociation. Then the attached NVE notifies the NVA with the received VN updating information. The next step is that the NVA updates the inner-outer address mapping table of the VN at both the attached NVE and all the participating remote NVEs.
C) It is also possible to use a NVE-NVE control plane protocol to update the peer NVEs' inner-outer address mapping table timely at VN connection / disconnection or vNIC association / disassociation. However, this approach may require the NVE-NVE control plane packets to be flooded to all NVEs when no mapping exists.
In implementation, maybe two of the alternatives or three alternatives are needed in some cases. So hybrid approach might be desirable in some development cases.
A) is described in section 7.1 in the architecture draft. B) is described as an alternative in section 7.1 in the architecture draft. However, I don't see alternative C). Do I miss anything in my reading?

>That I would like to avoid. The architecture document should describe the
>direction the WG has chosen to go in. I don't think we should document the
>pros/cons of various approaches (that is more what the framework does).
[Zu Qiang] So your suggestion is to keep the analysis text in different document? For instance, we may have the security aspect analysing in security requirement document. Does that make sense?

>
>Also, in the specific case above, I'm not sure listing pros/cons of the two
>approaches helps. I don't expect folk to pick one or the other based on the
>pros/cons. One of the starting premises is that there are already existing
>orchestration systems that provide the functionality of the NVA, and existing
>deployments use them. I don't expect implementations already using such a
>facility to change. I do think we will need to coexist with such approaches,
>hence they need to be  part of the architecture.
>
>Does that make sense?
[Zu Qiang] I agree that it's not the intention to change any existing implementations. The analysis text is just to show the considerations of each implementations. For instance, one alternative may have additional security requirements, which mean different security solutions may be needed. 
So I think analysis text is needed. But we shall avoid to give any conclusion of which alternative is better than others.

---

From: "Reith, Lothar" <[email-masked]>
Date: Fri, 18 Oct 2013 17:09:35 +0000

Thomas,

maybe to avoid the risk of misunderstanding, as the term "collocation" could mean:
- in the same physical building or in the physical room (that is the meaning of  co-location in the telco world, when a CLEC co-locates at a LEC building).
- on the same physical server, but at the same level as the hypervisor, which means in case there are 2 hypervisors configured on the server, one NVE may serve both. 
- on top of the hypervisor/ as a part of the hypervisor/ integrated with the server hypervisor/ i.e. the NVE is kind of a plug-in to the hypervisor or to an V-Switch that is controlled by the hypervisor. This means, that if there are 2 hypervisors installed on the Server, than each must have its own NVE.

I think that either one or both last meanings are meant, but it may be could to make it more explicit what is meant.

Lothar

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Thomas Narten
Gesendet: Donnerstag, 17. Oktober 2013 22:55
An: Lizhong Jin
Cc: [email-masked]
Betreff: Re: [nvo3] NVO3 Architecture document

Lizhong,

Thanks!

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 18 Oct 2013 17:52:30 -0400

Hi Lucy.

Lucy yong <[email-masked]> writes:

> Section 5.3 describes gateways. IMO: it misses an important use
>  case. A Gateway, say overlay gateway, may be used to interconnect
>  two or more overlay VNs. In this case, the traffic traversing
>  between two overlay VNs must go through the gateway where the
>  policy can be enforced. Furthermore, it is possible to implement
>  centralized or distributed overlay gateway. The latter has overlay
>  gateway function implemented on NVEs. Thus, it requests the
>  cross-VN policies to be distributed to NVEs.

> Current section seems very focus on overlay VN interconnect a
> non-overlay network, which centralized gateway architecture is
> practical. But in overlay networks, both centralized or distributed
> are possible and depend on the applications.

Agreed. I propose adding a new section after 5.3 that says:
      
      <section title="Distributed Gateways">
	<t>
	  The relaying of traffic from one VN to another deserves
	  special consideration. The previous section described
	  gateways performing this function. If such gateways are
	  centralized, traffic between TSes on different VNs can take
	  suboptimal paths, i.e., triangular routing results in paths
	  that always traverse the gateway. As an optimization,
	  individual NVEs can be part of a distributed gateway that
	  performs such relaying, reducing or completely eliminating
	  triangular routing. In a distributed gateway, each ingress
	  NVE can perform such relaying activity directly, so long as
	  it has access to the policy information needed to determine
	  whether cross-VN communication is allowed. Having individual
	  NVEs be part of a distributed gateway allows them to tunnel
	  traffic directly to the destination NVE without the need to
	  take suboptimal paths.
	</t>
	<t>
	  The NVO3 architecture should [must? or just say it does?]
	  support distributed gateways. Such support requires that
	  NVO3 control protocols include mechanisms for the
	  maintenance and distribution of policy information about
	  what type of cross-VN communication is allowed so that NVEs
	  acting as distributed gateways can tunnel traffic from one
	  VN to another as appropriate.
	</t>
      </section>

Thoughts?

Thomas

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 18 Oct 2013 22:00:23 +0000

Hi Thomas and Lucy,

The WG needs to think hard about this one.

Support of a distributed L3 gateway function between L2 VNs is a
significant
increase in scope of the NVA, and the NVE to NVA protocol.  Where we had
previously
stated L2 service or L3 service and pretty much left a combined L2/L3
service as an exercise for the reader, we would now be adding   whatever
mechanisms are needed to the protocols.  We will need to add cases for L2
service, L3 service and L2/L3 service.  We no longer have simple inner to
outer mappings, but now need NVEs to do MAC rewrites, local NVE ARP
termination, and multiple lookups depending on the destination MAC address
(first L2, then potentially L3).  We will also need to distribute two
different VN identifiers (one for L2 and one for L3), and somehow convey
the containment relationship between the two (multiple L2 VNs within one
L3 VN).  While I think this is all very useful, I just want to make sure
the WG agrees to this since I feel it is a significant change/increase in
scope from my perspective.

Thanks, Larry

---

From: Lucy yong <[email-masked]>
Date: Fri, 18 Oct 2013 22:05:49 +0000

Proposed text is good to me.
Thanks,
Lucy

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]] 
Sent: Friday, October 18, 2013 4:53 PM
To: Lucy yong
Cc: [email-masked]
Subject: Distributed Gateways [was Re: [nvo3] NVO3 Architecture document]

Hi Lucy.

Lucy yong <[email-masked]> writes:

> Section 5.3 describes gateways. IMO: it misses an important use  case. 
> A Gateway, say overlay gateway, may be used to interconnect  two or 
> more overlay VNs. In this case, the traffic traversing  between two 
> overlay VNs must go through the gateway where the  policy can be 
> enforced. Furthermore, it is possible to implement  centralized or 
> distributed overlay gateway. The latter has overlay  gateway function 
> implemented on NVEs. Thus, it requests the  cross-VN policies to be 
> distributed to NVEs.

      

Thoughts?

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 18 Oct 2013 18:10:56 -0400

Hi Zu.

OK, I see where you are coming from now.

> [Zu Qiang] I see three alternatives which are all described in the
>  draft
>  [http://tools.ietf.org/html/draft-zu-nov3-security-requirements-01]

> A) If either the NVA is collocated with the VM Orchestration
> Systems, or there is an interface between the VM Orchestration
> Systems and NVA, it can be assumed that the NVA may learn the VN
> connection / disconnection and vNIC association / disassociation
> from the VM Orchestration Systems. Then the NVA configures the
> attached NVE with the VN context of the connected / disconnected VN
> and the associated / disassociated vNIC addresses. The next step is
> that the NVA updates the inner-outer address mapping table of the VN
> at both the attached NVE and all the participating remote NVEs.

> B) The attached NVE is informed by the hypervisor using the
> Hypervisor-NVE protocol at VN connection / disconnection and vNIC
> association / disassociation. Then the attached NVE notifies the NVA
> with the received VN updating information. The next step is that the
> NVA updates the inner-outer address mapping table of the VN at both
> the attached NVE and all the participating remote NVEs.

> C) It is also possible to use a NVE-NVE control plane protocol to
> update the peer NVEs' inner-outer address mapping table timely at VN
> connection / disconnection or vNIC association /
> disassociation. However, this approach may require the NVE-NVE
> control plane packets to be flooded to all NVEs when no mapping
> exists.

C) is another possible approach. In some sense, this is a highly
distributed NVA where the NVEs are really part of the NVA. I.e., NVEs
are talking directly to other NVEs to obtain and exchange information,
rather than relying on an NVA. In this model, I think we lose a clean
separation between NVEs and NVAs, since NVEs are implementing NVA
functionality.

While I agree this is one model, I don't think we should put it in the
architecture unless the WG decides it wants to support such a
model. Remember, the goal of the architecture document is not to show
all possible alternatives, but to show the one NVO3 will support.

That said, another possible interaction model has NVEs sending
information to other NVEs that one might consider part of the control
plane, such as an egress NVE responding with a "target VM not here" or
"target VM has moved to NVE XXX", etc. Such messages could be treated
as authoritative, in which case they are accepted as updates. Or, they
could be treated as hints to go check with the NVA for updated
information. The two approaches have very different trust assumptions.

Note that draft-sridharan-virtualization-nvgre-ext-00.txt defines
UNREACHABLE and REDIRECT messages between NVEs.

As a strawman for discussion, I'd suggest that the architecture:

1) Support messages between NVEs that provide hints that an NVE has
   stale information and should go back to the NVA for
   updated/authoritative information. and

2) That the architecture does not merge NVE/NVA functionality in such
   a way that the NVE effectively is part of the NVA.

Thoughts?

Thomas

---

From: Lucy yong <[email-masked]>
Date: Fri, 18 Oct 2013 22:21:34 +0000

Larry,

Distributed L3 gateway is very useful and some vendors already implement that. Current framework also includes L2/L3 service as Marc agrees to add the text I proposed although we did not explicitly define as a service type.

Current architecture clear states NVE and NVA roles, i.e. NVE performs forwarding, and NVA performs routing. If NVA is not able to distribute routing policy to NVE at all, I do not know how NVA can perform route distribution control? If NVA only supports simple inner to outer mappings, how can NVE get information to perform local forwarding?

Thanks,
Lucy

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Friday, October 18, 2013 5:00 PM
To: Thomas Narten; Lucy yong
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Thomas and Lucy,

Support of a distributed L3 gateway function between L2 VNs is a significant increase in scope of the NVA, and the NVE to NVA protocol.  Where we had previously stated L2 service or L3 service and pretty much left a combined L2/L3
service as an exercise for the reader, we would now be adding   whatever
mechanisms are needed to the protocols.  We will need to add cases for L2 service, L3 service and L2/L3 service.  We no longer have simple inner to outer mappings, but now need NVEs to do MAC rewrites, local NVE ARP termination, and multiple lookups depending on the destination MAC address (first L2, then potentially L3).  We will also need to distribute two different VN identifiers (one for L2 and one for L3), and somehow convey the containment relationship between the two (multiple L2 VNs within one
L3 VN).  While I think this is all very useful, I just want to make sure the WG agrees to this since I feel it is a significant change/increase in scope from my perspective.

Thanks, Larry

On 10/18/13 2:52 PM, "Thomas Narten" <[email-masked]> wrote:

>> between two overlay VNs must go through the gateway where the  policy 
>> can be enforced. Furthermore, it is possible to implement  
>> gateway function implemented on NVEs. Thus, it requests the  cross-VN 
>> policies to be distributed to NVEs.
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 18 Oct 2013 23:38:56 +0000

Lucy,

See inline with LK>.

Thanks, Larry

On 10/18/13 3:21 PM, "Lucy yong" <[email-masked]> wrote:

>Larry,
>
>Distributed L3 gateway is very useful and some vendors already implement
>that. 

I agree.

>Current framework also includes L2/L3 service as Marc agrees to add the
>text I proposed although we did not explicitly define as a service type.

I think we will need to because the control plane requirements will likely
be different for a hybrid service.

>
>Current architecture clear states NVE and NVA roles, i.e. NVE performs
>forwarding, and NVA performs routing.

I'm not sure what you mean by routing.  Are you referring to L3 service?
If so, I don't agree.

>If NVA is not able to distribute routing policy to NVE at all, I do not
>know how NVA can perform route distribution control?

You lost me.  Are you referring to a hybrid L2/L3 service?

> If NVA only supports simple inner to outer mappings, how can NVE get
>information to perform local forwarding?

Inner to outer mapping resolution works fine for pure L2 or pure L3
service.  Local forwarding doesn¹t need mappings, the local NVE knows what
VAPs the TSI are connected to.

>significant increase in scope of the NVA, and the NVE to NVA protocol.
>Where we had previously stated L2 service or L3 service and pretty much
>left a combined L2/L3
>termination, and multiple lookups depending on the destination MAC
>address (first L2, then potentially L3).  We will also need to distribute
>two different VN identifiers (one for L2 and one for L3), and somehow
>convey the containment relationship between the two (multiple L2 VNs
>within one

---

From: Lucy yong <[email-masked]>
Date: Mon, 21 Oct 2013 03:20:20 +0000

Hi Larry,

Please see inline with [Lucy]

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Friday, October 18, 2013 6:39 PM
To: Lucy yong; Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lucy,

See inline with LK>.

Thanks, Larry

On 10/18/13 3:21 PM, "Lucy yong" <[email-masked]> wrote:

>Larry,
>
>Distributed L3 gateway is very useful and some vendors already 
>implement that.

I agree.

>text I proposed although we did not explicitly define as a service type.

I think we will need to because the control plane requirements will likely be different for a hybrid service.
[Lucy] Do you mean that we should define an l2/L3 service type? I full agree and hope see  more people support that too. We can either define it in the existing framework, or in the framework addition draft I submitted while ago.

I'm not sure what you mean by routing.  Are you referring to L3 service?
If so, I don't agree.
[Lucy] Sorry to make you confuse. The routing here does not mean L3 services. I should state that NVE performs data plan forwarding, NVA performs control plane routing, which, IMO, it is not just simple DB to have inner to outer mappings. NVA may get the routing policy from operators or customer, then interpret the policy, then generates the mapping of tenant/next-hop location and send to NVEs. An NVE receives the mapping in which, if the location is the same as itself, it translates to a tenant/VAP mapping; If not, it installs as an inner/outer mapping. Thus, it works regardless whether sender tenant and destination tenant are one the same NVE or on different NVEs. The mapping between tenant/location is fully controlled under NVA. In this way, operator only needs to input the routing policies to NVA. NVE simply performs the forwarding accordingly. This is also my view about the SDN based architecture. 

You lost me.  Are you referring to a hybrid L2/L3 service?
[Lucy] Let me know above explanation help or not. No, not particulate to L2/L3 service but that certainly applies to L2/L3 service.

>information to perform local forwarding?

Inner to outer mapping resolution works fine for pure L2 or pure L3 service.  Local forwarding doesn¹t need mappings, the local NVE knows what VAPs the TSI are connected to.
[Lucy] does pure L3 service means an L3VPN w/o any policy? For an L3 service, you can implement it w/o policy or w policy. IMO: NVE is not the entity to enforce the policy. NVA is the entity to enforce the policy regardless the tenant locations. Again, Network virtualization overlay has to address how to support the policies and tenant mobility. IMO: current architecture draft is vague in or lacks of describing this but it is important to architect this when having data plan and control plane separated on different entities.

Thanks,
Lucy

>mechanisms are needed to the protocols.  We will need to add cases for 
>L2 service, L3 service and L2/L3 service.  We no longer have simple 
>inner to outer mappings, but now need NVEs to do MAC rewrites, local 
>NVE ARP termination, and multiple lookups depending on the destination 
>MAC address (first L2, then potentially L3).  We will also need to 
>distribute two different VN identifiers (one for L2 and one for L3), 
>and somehow convey the containment relationship between the two 
>(multiple L2 VNs within one
>L3 VN).  While I think this is all very useful, I just want to make 
>sure the WG agrees to this since I feel it is a significant 
>change/increase in scope from my perspective.
>
>Thanks, Larry
>
>
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Mon, 21 Oct 2013 08:28:18 +0000

Lucy,

See inline with LK2>.

 - Larry

On 10/20/13 8:20 PM, "Lucy yong" <[email-masked]> wrote:

>I think we will need to because the control plane requirements will
>likely be different for a hybrid service.
>[Lucy] Do you mean that we should define an l2/L3 service type? I full
>agree and hope see  more people support that too. We can either define it
>in the existing framework, or in the framework addition draft I submitted
>while ago.

LK2> Yes, if the group is to seriously address a hybrid L2/L3 service, I
think we need to define it as we will need to identify its requirements
independently from a pure L2 or pure L3 service.

>[Lucy] Sorry to make you confuse. The routing here does not mean L3
>services. I should state that NVE performs data plan forwarding, NVA
>performs control plane routing, which, IMO, it is not just simple DB to
>have inner to outer mappings. NVA may get the routing policy from
>operators or customer, then interpret the policy, then generates the
>mapping of tenant/next-hop location and send to NVEs. An NVE receives the
>mapping in which, if the location is the same as itself, it translates to
>a tenant/VAP mapping; If not, it installs as an inner/outer mapping.
>Thus, it works regardless whether sender tenant and destination tenant
>are one the same NVE or on different NVEs. The mapping between
>tenant/location is fully controlled under NVA. In this way, operator only
>needs to input the routing policies to NVA. NVE simply performs the
>forwarding accordingly. This is also my view about the SDN based
>architecture. 
>
>[Lucy] Let me know above explanation help or not. No, not particulate to
>L2/L3 service but that certainly applies to L2/L3 service.

LK2> OK, if it isn't specific to L2/L3 service, than I'd rather leave it
out of the discussion for now.  And yes, the above helped me understand
what you meant.  The term "routing" is very overloaded.

>service.  Local forwarding doesn¹t need mappings, the local NVE knows
>what VAPs the TSI are connected to.
>[Lucy] does pure L3 service means an L3VPN w/o any policy? For an L3
>service, you can implement it w/o policy or w policy.

LK2> Yes, I was thinking of pure L3 service without adding policy.  I'm
pretty sure you need more than just inner to outer mappings to implement
policy.

>IMO: NVE is not the entity to enforce the policy. NVA is the entity to
>enforce the policy regardless the tenant locations. Again, Network
>virtualization overlay has to address how to support the policies and
>tenant mobility. IMO: current architecture draft is vague in or lacks of
>describing this but it is important to architect this when having data
>plan and control plane separated on different entities.

LK2> I'm not sure if I agree with you about the NVE not being the entity
to enforce policy, but  I think it is something better discussed in a
conversation (not email).

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Mon, 21 Oct 2013 10:29:56 +0000

Allow me to take a step back here.

It seems that Lucy is saying that in all cases, the NVA is the
control-plane, and the NVEs are implementing the data plane.  I believe
this is doesn't have to be the case for all deployments, as a matter of
fact this separation of control plane versus data plane is merely a
deployment model rather than an architectural foundation.

The NVA is more like a 'last-resort' function.  If the NVE has the correct
mappings and policies locally, it can just use those.  Only in the case
where it doesn't , it should pull them from the NVA.

On the other hand the NVA doesn't always need to 'provisioned' via some
sort of northbound interface regarding mappings and policies.  An NVE can
also 'feed' the NVA with information it discovers at the TS facing side of
it.

With regards to the 'new service type' of a combined L2/L3 NVE,
http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unified-00
 describes what you are referring to , but i am not sure if it needs to be
called a new service-type.  I see it more as a deployment choice where you
choose to deploy distributed gateways with uniform mac/IP addressing
across the DC, collocate them with the L2 overlay, and do some traffic
steering to make sure IP traffic gets sent to the L3 overlay (at all
times), and non-IP traffic gets sent across the L2 overlay.

Yves

---

From: Zu Qiang <[email-masked]>
Date: Mon, 21 Oct 2013 13:38:37 +0000

Hello, Thomas
	First of all, I'm not in favour of alternative C. As I said in my email, hybrid approach might be desirable in some development cases. I do think your proposed text make sense. It is more link a NVE-NVE control plane plus NVA-NVE control plane. In additional, the architecture shall also have a clear statement if alternative C is not supported by NVO3. 

Have a nice day
Zu Qiang

>Sent: Friday, October 18, 2013 6:11 PM
>To: Zu Qiang
>Cc: [email-masked]
>Subject: NVE-NVE control plane interactions [was Re: [nvo3] NVO3
>Architecture document]
>
>Hi Zu.
>
>> A) If either the NVA is collocated with the VM Orchestration Systems,
>> or there is an interface between the VM Orchestration Systems and NVA,
>> it can be assumed that the NVA may learn the VN connection /
>> disconnection and vNIC association / disassociation from the VM
>> Orchestration Systems. Then the NVA configures the attached NVE with
>> the VN context of the connected / disconnected VN and the associated /
>> disassociated vNIC addresses. The next step is that the NVA updates
>> the inner-outer address mapping table of the VN at both the attached
>> NVE and all the participating remote NVEs.
>
>> connection / disconnection or vNIC association / disassociation.
>> However, this approach may require the NVE-NVE control plane packets
>> to be flooded to all NVEs when no mapping exists.
>
>C) is another possible approach. In some sense, this is a highly distributed NVA
>where the NVEs are really part of the NVA. I.e., NVEs are talking directly to
>other NVEs to obtain and exchange information, rather than relying on an
>NVA. In this model, I think we lose a clean separation between NVEs and
>NVAs, since NVEs are implementing NVA functionality.
>
>architecture unless the WG decides it wants to support such a model.
>Remember, the goal of the architecture document is not to show all possible
>alternatives, but to show the one NVO3 will support.
>
>That said, another possible interaction model has NVEs sending information to
>other NVEs that one might consider part of the control plane, such as an
>egress NVE responding with a "target VM not here" or "target VM has moved
>to NVE XXX", etc. Such messages could be treated as authoritative, in which
>case they are accepted as updates. Or, they could be treated as hints to go
>check with the NVA for updated information. The two approaches have very
>different trust assumptions.
>

---

From: Eric Gray <[email-masked]>
Date: Mon, 21 Oct 2013 15:16:39 +0000

Thomas,

                A couple of subtle points about this discussion:

1) if the intent is to document the architecture agreed to within the NVO3 working

     group, it would be clearer if we cleaned up the first paragraph in the Introduction

     so that it is consistent with this goal.  In particular, the first sentence in the section

     indicates that the draft "presents a high-level overview of a possible architecture"

     (which begs a question - "have other possibilities been considered/addressed?").

2) if we want to ensure that we have an architecture likely to have any long-term

     viability, we should use some care in making sure that it doesn't break under some

     set of reasonable use-cases we are discussing already.

This said, I believe the fact that - even with extensive colocation of NVA functionality

- unless we contemplate requiring this, there will be a need to define a standard way

for one vendor's NVE to conduct transactions with another vendor's NVA (and vice

versa).

So the thing we might want to ensure is that we do not impose any unreasonable

architectural restrictions that would preclude solutions that rely on (or support) some

arbitrary degree of co-location of (for instance) NVE and NVA functionality.

I am pretty sure we could do exactly that, if we insist on making the architecture simpler

than it needs to be.

--

Eric

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, October 18, 2013 6:11 PM
To: Zu Qiang
Cc: [email-masked]
Subject: [nvo3] NVE-NVE control plane interactions [was Re: NVO3 Architecture document]

Hi Zu.

> to be flooded to all NVEs when no mapping exists.

C) is another possible approach. In some sense, this is a highly distributed NVA where the NVEs are really part of the NVA. I.e., NVEs are talking directly to other NVEs to obtain and exchange information, rather than relying on an NVA. In this model, I think we lose a clean separation between NVEs and NVAs, since NVEs are implementing NVA functionality.

While I agree this is one model, I don't think we should put it in the architecture unless the WG decides it wants to support such a model. Remember, the goal of the architecture document is not to show all possible alternatives, but to show the one NVO3 will support.

That said, another possible interaction model has NVEs sending information to other NVEs that one might consider part of the control plane, such as an egress NVE responding with a "target VM not here" or "target VM has moved to NVE XXX", etc. Such messages could be treated as authoritative, in which case they are accepted as updates. Or, they could be treated as hints to go check with the NVA for updated information. The two approaches have very different trust assumptions.

Note that draft-sridharan-virtualization-nvgre-ext-00.txt defines UNREACHABLE and REDIRECT messages between NVEs.

Thoughts?

Thomas

nvo3 mailing list

[email-masked]<mailto:[email-masked]>

---

From: Thomas Narten <[email-masked]>
Date: Mon, 21 Oct 2013 11:43:04 -0400

Hi Eric.

> 1) if the intent is to document the architecture agreed to within
>      the NVO3 working group, it would be clearer if we cleaned up
>      the first paragraph in the Introduction so that it is
>      consistent with this goal.  In particular, the first sentence
>      in the section indicates that the draft "presents a high-level
>      overview of a possible architecture" (which begs a question –
>      "have other possibilities been considered/addressed?").

Agreed, and I've removed it. 

> 2) if we want to ensure that we have an architecture likely to have
>      any long-term viability, we should use some care in making sure
>      that it doesn't break under some set of reasonable use-cases we
>      are discussing already.

Agreed!

> This said, I believe the fact that – even with extensive colocation
> of NVA functionality – unless we contemplate requiring this, there
> will be a need to define a standard way for one vendor's NVE to
> conduct transactions with another vendor's NVA (and vice versa).

I assume that when you say "colocation" you mean colocation of the NVE
and NVA functionality on the same device, and specifically, an edge
device that connects to a TS and does the NVE functions such as
encap/decap.

> So the thing we might want to ensure is that we do not impose any
> unreasonable architectural restrictions that would preclude
> solutions that rely on (or support) some arbitrary degree of
> co-location of (for instance) NVE and NVA functionality.

I think you are now actually getting into internal NVA design
(something we haven't talked about much). If one wants to implement
NVAs in a more distributed fashion, with more NVA functions co-located
on the same edge device as the NVE, that is one design. But it seems
to me that one of the design benefits of an NVA is that it doesn't
need to have a component reside on every edge device housing an
NVE. That would presumably have some scalability issues compared to a
model where the NVA at a site can be implemented on a relatively small
number of nodes, and certainly an order of magnitude fewer nodes than
NVEs.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Mon, 21 Oct 2013 20:32:27 +0000

Hi Yves,

To clarify, this is not I say it, it is the architecture described in the arch. draft. Here is text from section 6.

   Before sending to and receiving traffic from a virtual network, an
   NVE must obtain the information needed to build its internal
   forwarding tables and state as listed in Section 4.3.  An NVE obtains
   such information from a Network Virtualization Authority.

   The Network Virtualization Authority (NVA) is the entity that
   provides address mapping and other information to NVEs.  NVEs
   interact with an NVA to obtain any required information they need in
   order to properly forward traffic on behalf of tenants.  The term NVA
   refers to the overall system, without regards to its scope or how it
   is implemented.

Section 3 has the similar statement. 

I agree that it is possible to implement NVO without the separation of data plane and control plane. In that architecture, the focus will be the protocol between NVA/NVE -NVA/NVE as suggested in another thread. The WG needs decide if we should document one architecture or both architectures; if one, which one as the standard recommendation. I thought that the WG already had the decision and pointed the team to document it, wasn't?

Thanks,
Lucy 

-----Original Message-----
From: Yves Hertoghs (yhertogh) [mailto:[email-masked]] 
Sent: Monday, October 21, 2013 5:30 AM
To: Larry Kreeger (kreeger); Lucy yong; Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

It seems that Lucy is saying that in all cases, the NVA is the control-plane, and the NVEs are implementing the data plane.  I believe this is doesn't have to be the case for all deployments, as a matter of fact this separation of control plane versus data plane is merely a deployment model rather than an architectural foundation.

The NVA is more like a 'last-resort' function.  If the NVE has the correct mappings and policies locally, it can just use those.  Only in the case where it doesn't , it should pull them from the NVA.

On the other hand the NVA doesn't always need to 'provisioned' via some sort of northbound interface regarding mappings and policies.  An NVE can also 'feed' the NVA with information it discovers at the TS facing side of it.

With regards to the 'new service type' of a combined L2/L3 NVE,
http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unified-00
 describes what you are referring to , but i am not sure if it needs to be called a new service-type.  I see it more as a deployment choice where you choose to deploy distributed gateways with uniform mac/IP addressing across the DC, collocate them with the L2 overlay, and do some traffic steering to make sure IP traffic gets sent to the L3 overlay (at all times), and non-IP traffic gets sent across the L2 overlay.

Yves

On 21/10/13 10:28, "Larry Kreeger (kreeger)" <[email-masked]> wrote:

>>>Current framework also includes L2/L3 service as Marc agrees to add 
>>>the text I proposed although we did not explicitly define as a service type.
>>
>>agree and hope see  more people support that too. We can either define 
>>it in the existing framework, or in the framework addition draft I 
>>submitted while ago.
>
>LK2> Yes, if the group is to seriously address a hybrid L2/L3 service, 
>LK2> I
>>>Current architecture clear states NVE and NVA roles, i.e. NVE 
>>>performs forwarding, and NVA performs routing.
>>
>>performs control plane routing, which, IMO, it is not just simple DB 
>>to have inner to outer mappings. NVA may get the routing policy from 
>>mapping of tenant/next-hop location and send to NVEs. An NVE receives 
>>the mapping in which, if the location is the same as itself, it 
>>translates to a tenant/VAP mapping; If not, it installs as an inner/outer mapping.
>>tenant/location is fully controlled under NVA. In this way, operator 
>>only needs to input the routing policies to NVA. NVE simply performs 
>>the forwarding accordingly. This is also my view about the SDN based 
>>architecture.
>>
>>>If NVA is not able to distribute routing policy to NVE at all, I do 
>>>not know how NVA can perform route distribution control?
>>
>LK2> OK, if it isn't specific to L2/L3 service, than I'd rather leave 
>LK2> it
>pretty sure you need more than just inner to outer mappings to 
>implement policy.
>
>>tenant mobility. IMO: current architecture draft is vague in or lacks 
>>of describing this but it is important to architect this when having 
>>data plan and control plane separated on different entities.
>
>LK2> I'm not sure if I agree with you about the NVE not being the 
>LK2> entity
>>>Where we had previously stated L2 service or L3 service and pretty 
>>>much left a combined L2/L3
>>>NVE ARP termination, and multiple lookups depending on the 
>>>destination MAC address (first L2, then potentially L3).  We will 
>>>also need to distribute two different VN identifiers (one for L2 and 
>>>one for L3), and somehow convey the containment relationship between 
>>>the two (multiple L2 VNs within one
>>>>> practical. But in overlay networks, both centralized or 
>>>>> distributed are possible and depend on the applications.
>>>>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Mon, 21 Oct 2013 23:47:39 +0000

Hi Yves,

See my responses inline with LK>.

 - Larry

>where it doesn't , it should pull them from the NVA.

LK> I agree.  Just because we have an NVA to facilitate the running of the
overlays, it doesn't mean that the NVA needs to control the minutiae of
everything an NVE does.  I don't recall separation of control and data
planes being a goal of NVO3 (I believe that is what FORCES is doing).

>also 'feed' the NVA with information it discovers at the TS facing side of

LK> I agree with you about mappings.  I'm less sure about NVAs feeding
policies to the NVA.

>times), and non-IP traffic gets sent across the L2 overlay.

LK> I'll give you an example of why I think the protocol requirements will
be different for L2 vs L3 vs a combined L2/L3 service.  For an L2 VN, the
VN needs to be identified (e.g. with a Name or ID).  For an L3 VN, it
similarly needs to be identified.  For a combined L2/L3 service, I think
the NVEs need to know both the L3 VN identity (one), and all the L2 VN
identities that are part of the L3 VN.  When doing distributed L3
forwarding between a TS on one L2 VN to one on another L2 VN, it will need
to know not just the mapping of inner to outer address, but the mapping of
inner L3 address to destination L2 VN and MAC address (so it can rewrite
the MAC and the L2 VN).

>>>it
>>>the
>>>only

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 22 Oct 2013 00:06:53 +0000

Hi Lucy,

I think you are interpreting the sections you quoted below differently
from how they were intended.  I'm not aware of any mention of separation
of forwarding from control in the architecture document.  Where it talks
about obtaining information from the NVA for building its internal
forwarding tables, this does not mean that all internal forwarding table
information comes from the NVA.  An NVE can certainly build local
forwarding state all on its own.  Forwarding between locally connected TS
on the same NVE should not necessarily require the NVA.

 - Larry

On 10/21/13 1:32 PM, "Lucy yong" <[email-masked]> wrote:

>Hi Yves,
>
>To clarify, this is not I say it, it is the architecture described in the
>arch. draft. Here is text from section 6.
>
>I agree that it is possible to implement NVO without the separation of
>data plane and control plane. In that architecture, the focus will be the
>protocol between NVA/NVE -NVA/NVE as suggested in another thread. The WG
>needs decide if we should document one architecture or both
>architectures; if one, which one as the standard recommendation. I
>thought that the WG already had the decision and pointed the team to
>document it, wasn't?
>
>Thanks,
>Lucy 
>
>
>The NVA is more like a 'last-resort' function.  If the NVE has the
>correct mappings and policies locally, it can just use those.  Only in
>the case where it doesn't , it should pull them from the NVA.
>
>also 'feed' the NVA with information it discovers at the TS facing side
>of it.
>
> describes what you are referring to , but i am not sure if it needs to
>be called a new service-type.  I see it more as a deployment choice where
>you choose to deploy distributed gateways with uniform mac/IP addressing

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Tue, 22 Oct 2013 08:00:46 +0000

Lucy,

That still doesn't read as separation of control and data plane. These
paragraphs just illustrate that NVE's can request information from the
NVA.  There might be control plane interactions between the NVE and NVA
that can allow a scenario where the NVE feeds information into the NVA.

I also don't believe the WG has decided on a particular architectural
model ? Can you point me to a pointer in case i missed it ??

Yves

---

From: Lucy yong <[email-masked]>
Date: Tue, 22 Oct 2013 14:56:12 +0000

Larry and Yves,

It is great that an architecture clearly points out the protocol need between two entities and does not deal with the entity implementation. However, IMO, it is necessary to define the role of each entity in order to be clear what are needed in the protocol. Without it, we will later argue this in the protocol development and may end up a heavy and complex protocol to support different implementations; or make it clear later during the protocol development. Isn't it that the architecture draft had intention in staying long in WG for this reason? I am fine that people have different views and interpretations of the description. In the end, we will see the consequences and make consensus one way or another.    

Thanks,
Lucy   

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Monday, October 21, 2013 7:07 PM
To: Lucy yong; Yves Hertoghs (yhertogh); Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lucy,

I think you are interpreting the sections you quoted below differently from how they were intended.  I'm not aware of any mention of separation of forwarding from control in the architecture document.  Where it talks about obtaining information from the NVA for building its internal forwarding tables, this does not mean that all internal forwarding table information comes from the NVA.  An NVE can certainly build local forwarding state all on its own.  Forwarding between locally connected TS on the same NVE should not necessarily require the NVA.

 - Larry

On 10/21/13 1:32 PM, "Lucy yong" <[email-masked]> wrote:

>Hi Yves,
>
>To clarify, this is not I say it, it is the architecture described in 
>the arch. draft. Here is text from section 6.
>
>data plane and control plane. In that architecture, the focus will be 
>the protocol between NVA/NVE -NVA/NVE as suggested in another thread. 
>The WG needs decide if we should document one architecture or both 
>sort of northbound interface regarding mappings and policies.  An NVE 
>can also 'feed' the NVA with information it discovers at the TS facing 
>side of it.
>
>http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unifie
>d-0
>0
>be called a new service-type.  I see it more as a deployment choice 
>where you choose to deploy distributed gateways with uniform mac/IP 
>addressing across the DC, collocate them with the L2 overlay, and do 
>some traffic steering to make sure IP traffic gets sent to the L3 
>overlay (at all times), and non-IP traffic gets sent across the L2 overlay.
>
>Yves
>
>>>[Lucy] Do you mean that we should define an l2/L3 service type? I 
>>>full agree and hope see  more people support that too. We can either 
>>>define it in the existing framework, or in the framework addition 
>>>draft I submitted while ago.
>>
>>think we need to define it as we will need to identify its 
>>requirements independently from a pure L2 or pure L3 service.
>>
>>>
>>>>
>>>Thus, it works regardless whether sender tenant and destination 
>>>tenant are one the same NVE or on different NVEs. The mapping between 
>>out of the discussion for now.  And yes, the above helped me 
>>understand what you meant.  The term "routing" is very overloaded.
>>
>>>
>>>> If NVA only supports simple inner to outer mappings, how can NVE 
>>>>get information to perform local forwarding?
>>>
>>LK2> Yes, I was thinking of pure L3 service without adding policy.  
>>LK2> I'm
>>>IMO: NVE is not the entity to enforce the policy. NVA is the entity 
>>>to enforce the policy regardless the tenant locations. Again, Network 
>>>>much left a combined L2/L3 service as an exercise for the reader, we 
>>>>would now be adding whatever mechanisms are needed to the protocols.  
>>>>We will need to add cases for
>>>>>> centralized or distributed overlay gateway. The latter has 
>>>>>> overlay gateway function implemented on NVEs. Thus, it requests 
>>>>>> the cross-VN policies to be distributed to NVEs.
>>>>>

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Tue, 22 Oct 2013 14:57:05 +0000

Larry,

A VN could be dynamically created on an NVE as a result of a move of a
certain TS.  The NVE could notify the NVA about the new location of the TS.

Yves

On 22/10/13 01:47, "Larry Kreeger (kreeger)" <[email-masked]> wrote:

>>correct
>>of
>>http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unified-
>>0
>>0
>>be
>>>>>not
>>>>of

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Tue, 22 Oct 2013 14:59:56 +0000

Lucy,

If the document allows for different interpretations, we need to improve
the document in my opinion.
But i haven't really seen where the current text is implying what you are
saying with regards to control / data plane separation.

Yves

On 22/10/13 16:56, "Lucy yong" <[email-masked]> wrote:

>Larry and Yves,
>
>It is great that an architecture clearly points out the protocol need
>between two entities and does not deal with the entity implementation.
>However, IMO, it is necessary to define the role of each entity in order
>to be clear what are needed in the protocol. Without it, we will later
>argue this in the protocol development and may end up a heavy and complex
>protocol to support different implementations; or make it clear later
>during the protocol development. Isn't it that the architecture draft had
>intention in staying long in WG for this reason? I am fine that people
>have different views and interpretations of the description. In the end,
>we will see the consequences and make consensus one way or another.
>
>Thanks,
>Lucy   
>

---

From: Lucy yong <[email-masked]>
Date: Tue, 22 Oct 2013 15:08:56 +0000

NVO3 has data plane and control plane requirements. In your opinion, where do they fit in this architecture?  

Lucy 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Tuesday, October 22, 2013 10:00 AM
To: Lucy yong; Larry Kreeger (kreeger); Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lucy,

If the document allows for different interpretations, we need to improve the document in my opinion.
But i haven't really seen where the current text is implying what you are saying with regards to control / data plane separation.

Yves

On 22/10/13 16:56, "Lucy yong" <[email-masked]> wrote:

>However, IMO, it is necessary to define the role of each entity in 
>order to be clear what are needed in the protocol. Without it, we will 
>later argue this in the protocol development and may end up a heavy and 
>complex protocol to support different implementations; or make it clear 
>later during the protocol development. Isn't it that the architecture 
>draft had intention in staying long in WG for this reason? I am fine 
>that people have different views and interpretations of the 
>description. In the end, we will see the consequences and make consensus one way or another.
>
>Thanks,
>Lucy   
>
>from how they were intended.  I'm not aware of any mention of 
>separation of forwarding from control in the architecture document.  
>Where it talks about obtaining information from the NVA for building 
>its internal forwarding tables, this does not mean that all internal 
>forwarding table information comes from the NVA.  An NVE can certainly 
>build local forwarding state all on its own.  Forwarding between 
>locally connected TS on the same NVE should not necessarily require the NVA.
>
> - Larry
>
>>control-plane, and the NVEs are implementing the data plane.  I 
>>believe this is doesn't have to be the case for all deployments, as a 
>>matter of fact this separation of control plane versus data plane is 
>>merely a deployment model rather than an architectural foundation.
>>
>>On the other hand the NVA doesn't always need to 'provisioned' via 
>>some sort of northbound interface regarding mappings and policies.  An 
>>NVE can also 'feed' the NVA with information it discovers at the TS 
>>facing side of it.
>>
>> describes what you are referring to , but i am not sure if it needs 
>>to be called a new service-type.  I see it more as a deployment choice 
>>>LK2> Yes, if the group is to seriously address a hybrid L2/L3 
>>>LK2> service, I
>>>>mapping of tenant/next-hop location and send to NVEs. An NVE 
>>>>receives the mapping in which, if the location is the same as 
>>>>itself, it translates to a tenant/VAP mapping; If not, it installs 
>>>>as an inner/outer mapping.
>>>>tenant are one the same NVE or on different NVEs. The mapping 
>>>>between tenant/location is fully controlled under NVA. In this way, 
>>>>operator only needs to input the routing policies to NVA. NVE simply 
>>>>performs the forwarding accordingly. This is also my view about the 
>>>>SDN based architecture.
>>>>
>>>>service.  Local forwarding doesn¹t need mappings, the local NVE 
>>>>knows what VAPs the TSI are connected to.
>>>>to enforce the policy regardless the tenant locations. Again, 
>>>>Network virtualization overlay has to address how to support the 
>>>>policies and tenant mobility. IMO: current architecture draft is 
>>>>vague in or lacks of describing this but it is important to 
>>>>architect this when having data plan and control plane separated on different entities.
>>>
>>>>>much left a combined L2/L3 service as an exercise for the reader, 
>>>>>we would now be adding whatever mechanisms are needed to the protocols.
>>>>>inner to outer mappings, but now need NVEs to do MAC rewrites, 
>>>>>local NVE ARP termination, and multiple lookups depending on the 
>>>>>also need to distribute two different VN identifiers (one for L2 
>>>>>and one for L3), and somehow convey the containment relationship 
>>>>>between the two (multiple L2 VNs within one
>>>>>>> case. A Gateway, say overlay gateway, may be used to 
>>>>>>> interconnect two or more overlay VNs. In this case, the traffic 
>>>>>>> traversing between two overlay VNs must go through the gateway 
>>>>>>> where the policy can be enforced. Furthermore, it is possible to 
>>>>>>> implement centralized or distributed overlay gateway. The latter 
>>>>>>> has overlay gateway function implemented on NVEs. Thus, it 
>>>>>>> requests the cross-VN policies to be distributed to NVEs.
>>>>>>

---

From: Lizhong Jin <[email-masked]>
Date: Wed, 23 Oct 2013 10:31:44 +0800

[Lizhong] I tend to agree with Yves, it is more of a deployment choice.
The behavior you described for L3 forwarding between two L2 VPN is more of
a standard L3 forwarding function, looking up with destination IP address,
then getting the destination MAC, source MAC, VLAN and NVO3 tunnel from
the table. The MAC and VLAN are resolved from the next hop within same L3
VN. We write a combined L2/L3 forwarding for NVO3 last year, just for
reference http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00.
BTW, we may update it soon.

Thanks
Lizhong

>>>service. Local forwarding doesnšt need mappings, the local NVE knows

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 23 Oct 2013 03:04:49 +0000

Yves,

Absolutely, but I'm not sure what statement I made you were responding to.

 - Larry

On 10/22/13 7:57 AM, "Yves Hertoghs (yhertogh)" <[email-masked]> wrote:

>>>can
>>>>I
>>>>>to
>>>>implement
>>>>entity
>>>>>>destination
>>>>>>>>distributed

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 23 Oct 2013 03:06:33 +0000

Lucy,

Pretty much everything has a data plane (e.g. IPv4, IPv6) and a control
plane (e.g. OSPF, IS-IS), but that doesn't mean they are separated into
different devices.

 - Larry

On 10/22/13 8:08 AM, "Lucy yong" <[email-masked]> wrote:

>Yves Hertoghs (yhertogh)

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 23 Oct 2013 03:26:48 +0000

Hi Lizong,

Let me give more details on my thinking.

A pure L2 service only requires inner MAC address to outer IP address mappings. When a frame is delivered between two TS, the source and dest MAC addresses have not been changed.

A pure L3 service only requires inner IP to outer IP address mappings.  There is no expectation that two TS are on the same L2 segment, and the TS would always ARP for a router to reach another TS.  When a frame is delivered between the two TS, both the source and dest MAC will have been rewritten.  TTL will have been decremented at least once.

I'm expecting a hybrid L2/L3 service to preserve L2 semantics within a subnet.  TS ARP for the other TS on its subnet instead of the router.  The source and dest MAC are not rewritten.  TTL is not decremented.  From the perspective of the two TS on the same subnet, they are connected to the same L2 segment.  To achieve this, the NVEs must behave differently when forwarding IP packets within the same L2 VN vs between different L2 VNs.  For example, the NVE would need to respond to ARP, not with its own MAC address, but with the remote TSs MAC address if the TS are in the same L2 VN.  Furthermore, as I wrote below, there needs to be a mapping of the set of L2 VNs that belong to the L3 VN.

So, I still think NVEs will need some additional information for a hybrid service, and I think we will need to describe the behavior of a hybrid service beyond a sentence in the framework saying it is possible.  Otherwise, I think it is quite likely that two different hybrid implementations will not interoperate.

 - Larry

From: Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, October 22, 2013 7:31 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, "Yves Hertoghs (yhertogh)" <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Thanks
Lizhong

>
>Yves
>
>On 21/10/13 10:28, "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>> wrote:
>
>>Lucy,
>>
>>See inline with LK2>.
>>
>> - Larry
>>
>>On 10/20/13 8:20 PM, "Lucy yong" <[email-masked]<mailto:[email-masked]>> wrote:
>>
>>>Hi Larry,
>>>
>>>From: Larry Kreeger (kreeger) [mailto:[email-masked]<mailto:[email-masked]>]
>>>Cc: [email-masked]<mailto:[email-masked]>
>>>On 10/18/13 3:21 PM, "Lucy yong" <[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>>>Larry,
>>>>
>>>>From: Larry Kreeger (kreeger) [mailto:[email-masked]<mailto:[email-masked]>]
>>>>Cc: [email-masked]<mailto:[email-masked]>
>>>>On 10/18/13 2:52 PM, "Thomas Narten" <[email-masked]<mailto:[email-masked]>> wrote:
>>>>
>>>>>Hi Lucy.
>>>>>
>>>>>Lucy yong <[email-masked]<mailto:[email-masked]>> writes:
>>>>>

---

From: Lucy yong <[email-masked]>
Date: Wed, 23 Oct 2013 05:24:54 +0000

Agree with Larry. If DC operator expands so many resources to support hybrid service, they will charge more compared to a pure L2 or pure L3 service. Besides followings, they also need implement some policies/firewalls.

Lucy

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Tuesday, October 22, 2013 10:27 PM
To: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: Lucy yong <[email-masked]>
Date: Wed, 23 Oct 2013 05:24:51 +0000

Larry,

I am surprised to see your answer. Does nvo3 control plane requirement draft also covers underlying control plane requirement? We are talking about the control plane for overlay virtual network! Yes, every communication in overlay data plane and control plane relies on underlying network. Traditional underlying networks do not have control plane and data plane separation. Recently OF or SDN based architecture uses the concept of control plane and data plane separation.

In addition, I use the word entity, not device. They are not equivalent in my opinion.

Lucy

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Tuesday, October 22, 2013 10:07 PM
To: Lucy yong; Yves Hertoghs (yhertogh); Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lucy,

Pretty much everything has a data plane (e.g. IPv4, IPv6) and a control plane (e.g. OSPF, IS-IS), but that doesn't mean they are separated into different devices.

 - Larry

On 10/22/13 8:08 AM, "Lucy yong" <[email-masked]> wrote:

>But i haven't really seen where the current text is implying what you 
>are saying with regards to control / data plane separation.
>
>Yves
>
>>later argue this in the protocol development and may end up a heavy 
>>and complex protocol to support different implementations; or make it 
>>clear later during the protocol development. Isn't it that the 
>>architecture draft had intention in staying long in WG for this 
>>reason? I am fine that people have different views and interpretations 
>>of the description. In the end, we will see the consequences and make 
>>consensus one way or another.
>>
>>Thanks,
>>Lucy   
>>
>>>I agree that it is possible to implement NVO without the separation 
>>>of data plane and control plane. In that architecture, the focus will 
>>>be the protocol between NVA/NVE -NVA/NVE as suggested in another thread.
>>>correct mappings and policies locally, it can just use those.  Only 
>>>in the case where it doesn't , it should pull them from the NVA.
>>>
>>>some sort of northbound interface regarding mappings and policies.  
>>>An NVE can also 'feed' the NVA with information it discovers at the 
>>>TS facing side of it.
>>>
>>>http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unif
>>>i
>>>e
>>>d-0
>>>0
>>>to be called a new service-type.  I see it more as a deployment 
>>>choice where you choose to deploy distributed gateways with uniform 
>>>mac/IP addressing across the DC, collocate them with the L2 overlay, 
>>>and do some traffic steering to make sure IP traffic gets sent to the 
>>>L3 overlay (at all times), and non-IP traffic gets sent across the L2 
>>>overlay.
>>>
>>>Yves
>>>
>>>>>>Current framework also includes L2/L3 service as Marc agrees to 
>>>>>>add the text I proposed although we did not explicitly define as a 
>>>>>>service type.
>>>>>
>>>>>full agree and hope see  more people support that too. We can 
>>>>>either define it in the existing framework, or in the framework 
>>>>>addition draft I submitted while ago.
>>>>
>>>>>services. I should state that NVE performs data plan forwarding, 
>>>>>NVA performs control plane routing, which, IMO, it is not just 
>>>>>simple DB to have inner to outer mappings. NVA may get the routing 
>>>>>policy from operators or customer, then interpret the policy, then 
>>>>>generates the mapping of tenant/next-hop location and send to NVEs. 
>>>>>An NVE receives the mapping in which, if the location is the same 
>>>>>as itself, it translates to a tenant/VAP mapping; If not, it 
>>>>>installs as an inner/outer mapping.
>>>>>operator only needs to input the routing policies to NVA. NVE 
>>>>>simply performs the forwarding accordingly. This is also my view 
>>>>>about the SDN based architecture.
>>>>>
>>>>>>If NVA is not able to distribute routing policy to NVE at all, I 
>>>>>>do not know how NVA can perform route distribution control?
>>>>>
>>>>LK2> OK, if it isn't specific to L2/L3 service, than I'd rather 
>>>>LK2> leave it
>>>>>[Lucy] does pure L3 service means an L3VPN w/o any policy? For an 
>>>>>L3 service, you can implement it w/o policy or w policy.
>>>>
>>>>to enforce policy, but  I think it is something better discussed in 
>>>>a conversation (not email).
>>>>
>>>>>
>>>>>Thanks,
>>>>>Lucy
>>>>>
>>>>>>
>>>>>>Thanks,
>>>>>>Lucy
>>>>>>
>>>>>>L2 service, L3 service and L2/L3 service.  We no longer have 
>>>>>>simple inner to outer mappings, but now need NVEs to do MAC 
>>>>>>rewrites, local NVE ARP termination, and multiple lookups 
>>>>>>depending on the destination MAC address (first L2, then 
>>>>>>potentially L3).  We will also need to distribute two different VN 
>>>>>>identifiers (one for L2 and one for L3), and somehow convey the 
>>>>>>containment relationship between the two (multiple L2 VNs within 
>>>>>>one
>>>>>>L3 VN).  While I think this is all very useful, I just want to 
>>>>>>make sure the WG agrees to this since I feel it is a significant 
>>>>>>>> where the policy can be enforced. Furthermore, it is possible 
>>>>>>>> to implement centralized or distributed overlay gateway. The 
>>>>>>>> latter has overlay gateway function implemented on NVEs. Thus, 
>>>>>>>> it requests the cross-VN policies to be distributed to NVEs.
>>>>>>>

---

From: Lizhong Jin <[email-masked]>
Date: Wed, 23 Oct 2013 13:42:08 +0800

Hi Larry,

See inline below.

Regards

Lizhong

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: 2013年10月23日 11:27
To: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

Hi Lizong,

A pure L2 service only requires inner MAC address to outer IP address
mappings. When a frame is delivered between two TS, the source and dest MAC
addresses have not been changed.

A pure L3 service only requires inner IP to outer IP address mappings.
 There is no expectation that two TS are on the same L2 segment, and the TS
would always ARP for a router to reach another TS.  When a frame is
delivered between the two TS, both the source and dest MAC will have been
rewritten.  TTL will have been decremented at least once.

I'm expecting a hybrid L2/L3 service to preserve L2 semantics within a
subnet.  TS ARP for the other TS on its subnet instead of the router.  The
source and dest MAC are not rewritten.  TTL is not decremented.  From the
perspective of the two TS on the same subnet, they are connected to the
same L2 segment.  To achieve this, the NVEs must behave differently when
forwarding IP packets within the same L2 VN vs between different L2 VNs.
 For example, the NVE would need to respond to ARP, not with its own MAC
address, but with the remote TSs MAC address if the TS are in the same L2
VN.

[Lizhong] the ARP request with broadcast MAC address will not be responded
by NVE, but be flooded to remote TS within same subnet. Only when ARP proxy
is enabled on the NVE, the NVE will respond all the ARP requests. The NVE
will do IP forwarding for packets destined to its own MAC, and do MAC
forwarding for packets with other MAC.

Furthermore, as I wrote below, there needs to be a mapping of the set of L2
VNs that belong to the L3 VN.

[Lizhong] I agree, from the control point of view, NVE need additional
information for combined L2&L3 solution.

So, I still think NVEs will need some additional information for a hybrid
service, and I think we will need to describe the behavior of a hybrid
service beyond a sentence in the framework saying it is possible.
 Otherwise, I think it is quite likely that two different hybrid
implementations will not interoperate.

[Lizhong] I try to say combined L2&L3 is not a service, but a solution, and
that’s why the draft-nvo3-dataplane does not define it as a service. I
agree we should add some description for the control plane in framework or
other draft. There is another draft described the solution,
http://tools.ietf.org/id/draft-sajassi-l2vpn-evpn-ipvpn-interop-02.txt.

 - Larry

From: Lizhong Jin <[email-masked]>
Date: Tuesday, October 22, 2013 7:31 PM
To: Larry Kreeger <[email-masked]>, "Yves Hertoghs (yhertogh)" <
[email-masked]>, Lucy yong <[email-masked]>, Thomas Narten <
[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

Thanks
Lizhong

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 23 Oct 2013 06:38:43 +0000

发件人: [email-masked] [mailto:[email-masked]] 代表 Larry Kreeger (kreeger)
发送时间: 2013年10月23日 11:27
收件人: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]; [email-masked]
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

If so, what’s the difference between hybrid L2/L3 service and IRB (Integrated Routing and Bridging)? In addition, I believe the hybrid L2/L3 service you mentioned above is different from what Yves meant by “combined L2/L3”. The former is  about bridging for intra-subnet traffic and routing for inter-subnet traffic while the latter is about routing for (both intra-subnet and inter-subnet) IP traffic while bridging for non-IP traffic.

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 08:31:53 +0000

Larry, apologies for not being more clear.  I've cleaned up the thread
below to indicate what i was responding to.
And after reading your answer and my answer again, i agree with you :-)
Mappings can be fed from NVE to NVA, polcies not likely!

Yves

---

From: "Reith, Lothar" <[email-masked]>
Date: Wed, 23 Oct 2013 09:15:59 +0000

Hi Larry,

excellent explanation of L2, L3 and hybrid L2/L3 service, however, I tend to have a slightly different view regarding the hybrid L2/L3 service, which is also based on my analysis of
http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unified-00

First let me introduce my Neandertal abstraction of L2 and L3 networking as follows:

Assuming Neandertal people sitting around a fireplace each havea  local name in the language of that Neandertal tribe (AKA MAC Addresses) , they can speak to each other including unicast, multicast and Broadcast (to all on the fireplace). This is layer 2 networking, using layer2 endpoint identifiers in a local addressing scheme - assume for an example the source tribe members have French MAC addresses such as Francois and the destination tribes speaks Italian, and all members have Italian names such as  Susanna.

Assuming different Neantertal people found a way to communicate, using an inter-neandertal tribe endpoint identification system, where they agree on first name and family name in englisch language as their Layer3 endpoint addressing scheme, where the family name identifies the fireplace (network portion of IP Adress) and the first name identifies the individual endpoint i.e. the member of the tribe (aka Host part of IP-address).

So we have 2 communication partners: Susanna  from a tribe speaking only Italian, and Francois speaking only French in his tribe. The layer 3 addresses given are Susan (Hostpart) Italiantribefireplace (Network part), and Frank (host part) and Frenchtribefireplace (network part).

Your interpretation seems to be, that when Francois wants to communicate with Susan, he needs to ARP to find out her local identity (MAC) address "Susanna".

According to my analysis of above draft, this is not the case.

Rather, in hybrid L2/L3 mode Neandertal communications, Francois only needs to know the English name "Susan Italiantribefireplace", the NVA adds the additional local name "Susanna" (as an additional RLOC -routing locator) and therefore, there is no ARP necessary when the packet arrives at the Italian fireplace, as the NVE will forward it to Susanna without requiring ARP to resolve the layer 3 endpoint Hostpart "Susan" to the layer2 endpoint ID "Susanna".

 Back to your statements:

You stated: From the perspective of the two TS on the same subnet, they are connected to the same L2 segment.
Based on above introduction, my view is: no, according to above draft, it appears that the perspective of the two TS is that they are on different subnets, however, the burden of ARP is taken away with a little help of a  friend called "NVA", which is helping the gateways on the path and eliminates the need for performing an ARP in the destination subnet in order to reach the destination station (more exactly destination station MSAP - MAC Service Access Point - as endpoint) in the destination subnet.

Would you agree?

Best Regards, Lothar

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Larry Kreeger (kreeger)
Gesendet: Mittwoch, 23. Oktober 2013 05:27
An: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Betreff: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 09:36:29 +0000

Larry,

You are right to point out that there are behavioural changes inside the NVE when it comes to things like ARP and intersubnet and intrasubnet IP forwarding, as well as how e.g. Default gateway mac and IP addresses are distributed across the architecture.

But a 'service' is really what the TS'es are seeing, and from their point of view the intrasubnet service just looks like it did before, and the intersubnet service just looks like it did before.

So I agree that describing the behaviour of the unified L2/L3 NVE is important, while i dont think the 'service' (from the point of view of the TS'es) has to change.

Yves

From: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 05:26
To: Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 09:50:03 +0000

Lucy,

The tenants would not see a difference in user experience.  The service does not change from their point of view.  Why on earth would they pay more, and why on earth would the NVO3-provider have to implement firewalls/policies in order to achieve that ?  The unified/hybrid/combined L2/L3 NVE would need to be described from a behavioural way , on the side of the TS, and that is simple: it just would look exactly like a default gateway for IP intersubnet traffic, and it would be transparent for IP intrasubnet and non-IP traffic i.e. Exactly the same as a typical L2/L3 switch (except that now the switch is 'distributed' in every location that the Tses want to move to).

On the other hand, the real behavioural change NVO3 would want to document is INSIDE the network, i.e. What is needed to offer that 'distributed' L2/L3 switch behaviour to tenants, as well as how the control plane needs to evolve to support the model.

Yves

From: Lucy yong <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 07:24
To: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lucy

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 09:54:10 +0000

Inline [yves]

From: Xuxiaohu <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 08:38
To: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: 答复: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Larry Kreeger (kreeger)
发送时间: 2013年10月23日 11:27
收件人: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

If so, what’s the difference between hybrid L2/L3 service and IRB (Integrated Routing and Bridging)? In addition, I believe the hybrid L2/L3 service you mentioned above is different from what Yves meant by “combined L2/L3”. The former is  about bridging for intra-subnet traffic and routing for inter-subnet traffic while the latter is about routing for (both intra-subnet and inter-subnet) IP traffic while bridging for non-IP traffic.
[yves] It isn't really 'routing' as such for intrasubnet forwarding, as you want to keep the MAC headers intact, as you dont want to confuse the TSes.  So what Larry (and me ) mean , is a 3rd model, none of the two above.

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 23 Oct 2013 10:16:11 +0000

It said in http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-unified-00#page-8:
   “Note that , within this document, it is assumed that a unified L2/L3
   NVE is deployed and therefore all IP traffic will be forwarded using
   the L3 overlay, even intra-subnet traffic.”

Or did you mean the other thing than the above definition by “combined L2/l3”?

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Yves Hertoghs (yhertogh)
发送时间: 2013年10月23日 17:54
收件人: Xuxiaohu; Larry Kreeger (kreeger); Lizhong Jin; Lucy yong; Thomas Narten
抄送: [email-masked]; [email-masked]
主题: Re: [nvo3] 答复: Distributed Gateways [was Re: NVO3 Architecture document]

Inline [yves]

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 10:29:23 +0000

Great to see such analogies being made on an IETF mailing list :-)
More inline with [yves]

From: <Reith>, Lothar <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 11:15
To: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: AW: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

Assuming Neandertal people sitting around a fireplace each havea  local name in the language of that Neandertal tribe (AKA MAC Addresses) , they can speak to each other including unicast, multicast and Broadcast (to all on the fireplace). This is layer 2 networking, using layer2 endpoint identifiers in a local addressing scheme – assume for an example the source tribe members have French MAC addresses such as Francois and the destination tribes speaks Italian, and all members have Italian names such as  Susanna.

[yves] Allthough i like the analogy, in my draft I assume all MAC-addresses are part of the same language (Dutch :).  Otherwise non-IP traffic will break, unless you want to re-invent the stuff in the 90's that involved bridging token ring to ethernet.

Your interpretation seems to be, that when Francois wants to communicate with Susan, he needs to ARP to find out her local identity (MAC) address “Susanna”.

Rather, in hybrid L2/L3 mode Neandertal communications, Francois only needs to know the English name “Susan Italiantribefireplace”, the NVA adds the additional local name “Susanna” (as an additional RLOC –routing locator) and therefore, there is no ARP necessary when the packet arrives at the Italian fireplace, as the NVE will forward it to Susanna without requiring ARP to resolve the layer 3 endpoint Hostpart “Susan” to the layer2 endpoint ID “Susanna”.
[yves]In reality, a TS will (need to) know the other TS MAC and IP address, or at least it would try finding out the MAC through ARP.  You seem to say that the TS'es would need to know about the mechanics of the overlay service in order to communicate with other Tses, which is essentially what the draft is trying to avoid.

 Back to your statements:

You stated: From the perspective of the two TS on the same subnet, they are connected to the same L2 segment.
Based on above introduction, my view is: no, according to above draft, it appears that the perspective of the two TS is that they are on different subnets, however, the burden of ARP is taken away with a little help of a  friend called “NVA”, which is helping the gateways on the path and eliminates the need for performing an ARP in the destination subnet in order to reach the destination station (more exactly destination station MSAP – MAC Service Access Point - as endpoint) in the destination subnet.
[yves]End stations still use ARP.  You wouldn't want to change that behaviour.   ARPs will be responded to, with the 'real' MAC address of the destination.  But the way the ARP is dealt with inside the network is different.

Would you agree?
[yves]I dont :-)

Best Regards, Lothar

Von: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] Im Auftrag von Larry Kreeger (kreeger)

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Reith, Lothar" <[email-masked]>
Date: Wed, 23 Oct 2013 13:05:44 +0000

Hi Yves,

thanks for explanation, and your adaptation of the analogy to Dutch ;-)

Lothar

Von: Yves Hertoghs (yhertogh) [mailto:[email-masked]]
Gesendet: Mittwoch, 23. Oktober 2013 12:29
An: Reith, Lothar; Larry Kreeger (kreeger); Lizhong Jin; Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Betreff: Re: AW: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

 Back to your statements:

Would you agree?
[yves]I dont :-)

Best Regards, Lothar

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 13:35:57 +0000

I'd agree that i didn't go into the semantics of TTL and MAC addresses in the draft, which i should have done.
There are different implementations which could lead to the same behaviour from the point of view of the TS, which wants to communicate with another TS part of the same subnet:

1.  You bridge the traffic.  This requires an L2 overlay and a MAC-lookup.  This also means you need to extend the broadcast domain across all locations of the subnet.
2. You make sure all ARPs are responded to by the default gateway, even the ones which are 'intrasubnet' from the point of view of the TS.  In reality, the NVO3 infrastructure does not use subnets as such, it only uses /32 (/128) prefixes.  If you would check the ARP tables in the Tses, all of them would point to the default gateways MAC address, even the one intrasubnet.  The NVO3 infrastructure routes the intrasubnet traffic, as everything is destined to the default gateway from a MAC perspective.  You would want to make sure that Tses can only talk to other Tses via the default gateway, even intrasubnet.  Non-IP traffic still requires an L2 overlay.  The NVE is the default gateway, and it has the same identity at all locations.
3. You make sure that ARPs are responded to , and that the ARPs contain the REAL mac-address of the IP hosts that the TS wanted to resolve to.  You still make sure that TS'es cannot direcly communicate with eachohter, they always need to go (transparently) across the NVE.  NVE's do an IP lookup for all IP traffic, even intrasubnet and route the traffic across the overlay if needed.  But at the egress point they make sure they insert the source MAC of the original host into the packet if its part of the same subnet).

I would argue that models 2 and 3 can both be implemented in draft-hertoghs

Yves

From: Xuxiaohu <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 12:16
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [nvo3] 答复: 答复: Distributed Gateways [was Re: NVO3 Architecture document]

Best regards,
Xiaohu

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Yves Hertoghs (yhertogh)
发送时间: 2013年10月23日 17:54
收件人: Xuxiaohu; Larry Kreeger (kreeger); Lizhong Jin; Lucy yong; Thomas Narten
抄送: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] 答复: Distributed Gateways [was Re: NVO3 Architecture document]

Inline [yves]

发件人:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Larry Kreeger (kreeger)
发送时间: 2013年10月23日 11:27
收件人: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: Lucy yong <[email-masked]>
Date: Wed, 23 Oct 2013 13:59:19 +0000

Yves,

IMO: Service is not defined based on individual tenant system to see the difference or not. The service is defined based on what networking capability offers. Sometimes SP may define different services or sometimes may define a service with optional features. IMO:  in this case, it is better to define as a new service type because of the given reasons.

IRB capability is implemented on the devices and used in physical network but it does not be implemented in the context of IP/MPLS VPNs. To implement in VPN environment, you need additional capabilities.

Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Wednesday, October 23, 2013 4:50 AM
To: Lucy yong; Larry Kreeger (kreeger); Lizhong Jin; Thomas Narten
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lucy,

Yves

Lucy

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 14:04:56 +0000

Inline with [yves]

From: Lucy yong <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 23 October 2013 15:59
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Yves,

[yves]Maybe you and I have a different definition of 'service'.  You have providers of the service, and consumers of the service.  In an NVO3 context these map to tenants, and their tenant end systems , which are consuming the service, and we have the NVO3 network, which is providing the network service.  The way the service needs to be described is ALWAYS from the service consumer point of view.

IRB capability is implemented on the devices and used in physical network but it does not be implemented in the context of IP/MPLS VPNs. To implement in VPN environment, you need additional capabilities.
[yves] such as ?  And why do you talk about IP/MPLS VPNs ? Have we decided to use IP/MPLS VPNs in NVO3 as the only viable technology? We haven't decided on any technology afaik.

Lucy

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)

Lucy,

Yves

Lucy

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: Lucy yong <[email-masked]>
Date: Wed, 23 Oct 2013 14:26:04 +0000

Snip..

IRB capability is implemented on the devices and used in physical network but it does not be implemented in the context of IP/MPLS VPNs. To implement in VPN environment, you need additional capabilities.
[yves] such as ?  And why do you talk about IP/MPLS VPNs ? Have we decided to use IP/MPLS VPNs in NVO3 as the only viable technology? We haven't decided on any technology afaik.
[Lucy] Because there is a key difference in between. In this community, many people are familiar with IP/MPLS VPN definition.  A tenant network in NVo3 is also a VPN network, isn't it? To be clarify, I do not have intention to say that IP/MPLS VPN  is the technology for nvo3. (although nvo3 framework describes that way, look at L3 service type definition in the framework).

Lucy

Lucy

Lucy,

Yves

Lucy

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Wed, 23 Oct 2013 15:02:12 +0000

Lucy,

The reference to RFC4364 is to describe the similarity of  a PE to an L3 NVE in terms of service providing characteristics, nothing more, nothing less, it seems you are reading things between the lines. RFC6830 is another technology that can provide segmentation as well at the IP layer.

Now , reading the framework draft again, i can see where all this service type bickering is coming from.
The model in draft-hertoghs assumes a unified L2/L3 NVE i.e. It doesn't work without it.  A unifed L2/L3 really does exactly what a L2 NVE and a L3 NVE does, but the main catch is that BOTH *have* to be exist on the device in order for the unified model to work.

Yves

Date: Wednesday 23 October 2013 16:26

Snip..

IRB capability is implemented on the devices and used in physical network but it does not be implemented in the context of IP/MPLS VPNs. To implement in VPN environment, you need additional capabilities.
[yves] such as ?  And why do you talk about IP/MPLS VPNs ? Have we decided to use IP/MPLS VPNs in NVO3 as the only viable technology? We haven't decided on any technology afaik.
[Lucy] Because there is a key difference in between. In this community, many people are familiar with IP/MPLS VPN definition.  A tenant network in NVo3 is also a VPN network, isn’t it? To be clarify, I do not have intention to say that IP/MPLS VPN  is the technology for nvo3. (although nvo3 framework describes that way, look at L3 service type definition in the framework).

Lucy

Lucy

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)

Lucy,

Yves

Lucy

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: Lucy yong <[email-masked]>
Date: Wed, 23 Oct 2013 16:15:34 +0000

FYI, here is another document for you to reference.

http://tools.ietf.org/html/draft-yong-nvo3-frwk-dpreq-addition-00

Lucy

Sent: Wednesday, October 23, 2013 10:02 AM

Lucy,

Yves

Snip..

Lucy

Lucy

Lucy,

Yves

Lucy

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 23 Oct 2013 23:34:06 +0000

Hi Lothar,

While your analogy was amusing, it didn't clarify things – at least for me.  See my response inline with LK>, below.

Thanks, Larry

From: <Reith>, Lothar <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, October 23, 2013 2:15 AM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, "Yves Hertoghs (yhertogh)" <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: AW: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

LK> If I understand your analogy, Francois and Susanna are on different L2 domains/subnets.  In that case, no, Francois will not ARP for Susanna, but instead for the gateway from his tribes fireplace (sorry If I didn't follow the analogy perfectly).

LK>I'm pretty sure ARP is needed no matter what since we are assuming there is an L2 network between Francois and his tribe's fireplace to English gateway.

 Back to your statements:

Would you agree?

LK> Not really.

Best Regards, Lothar

Hi Lizong,

 - Larry

Thanks
Lizhong

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 23 Oct 2013 23:47:44 +0000

Hi Lucy,

I'm afraid we both may be misunderstanding each other.  When I read your
statement "NVO3 has data plane and control plane requirements. In your
opinion, where do they fit in this architecture?" I interpreted this to
mean that because we have separate documents for data plane requirements
from control plane requirements, than that leads to the assumption that
they are separated.  I apologize if I misinterpreted you.

So far, IMO, NVO3 has not been about the OpenFlow architecture, nor SDN
(whatever that means, let's leave that do the marketers and SDNRG), nor
about control/data plane separation (FORCES).  The architecture calls for
a logically centralized NVA to resolve inner to outer address mappings
(among other things), and based on recent list discussions, NVEs may send
messages directly to each other to help optimize the correction of a stale
mapping.

Maybe we can have a conversation about this more easily when we are in
Vancouver.

Thanks, Larry

On 10/22/13 10:24 PM, "Lucy yong" <[email-masked]> wrote:

>Larry,
>
>I am surprised to see your answer. Does nvo3 control plane requirement
>draft also covers underlying control plane requirement?
> We are talking about the control plane for overlay virtual network! Yes,
>every communication in overlay data plane and control plane relies on
>underlying network. Traditional underlying networks do not have control
>plane and data plane separation. Recently OF or SDN based architecture
>uses the concept of control plane and data plane separation.
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Thu, 24 Oct 2013 00:32:15 +0000

I've now read through this whole thread  twice.  I see now that there may be some confusion between an L2/L3 service (what tenant's see) and an L2/L3 solution (how NVO3 provides the service).

Prior to this thread, I may have agreed that we don't need to define a hybrid/combined/unified L2/L3 "service", but now I'm less sure. Just be be clear, I was not calling so much for a definition of the service, but that we need:

1) A name for the service (lets's pick one, is it hybrid, combined or unified?)
2) To address it from a solution point of view (e.g. are packets forwarded based on MAC or based on IP when source and dest are in the same subnet?).

As I stated before, I believe the control plane requirements will be different for a combined service than for a pure L2 or pure L3 service, and that we if don't start specifically calling out the control plane requirements for a combined service (from a solution point of view), we will have a very low likelihood of interoperable solutions as each implementer  fills in the gaps in their own way.

Does the WG agree that we should do 1) and 2) above?

I would also not object if we had a short description in the framework of what the combined service looks like compared to a pure L2 or pure L3 service, just so we can level set for everyone.

Thanks, Larry

Date: Tuesday, October 22, 2013 11:38 PM

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 24 Oct 2013 09:54:48 +0800

Hi Larry,

1) A name for the service (lets's pick one, is it hybrid, combined or
unified?)

[Lizhong] IRB, which have been used before. We could change it to virtual
IRB.

2) To address it from a solution point of view (e.g. are packets forwarded
based on MAC or based on IP when source and dest are in the same subnet?).

[Lizhong] Yes, we should do that, and there are already some drafts there (
http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00). For IP
forwarding for intra subnet traffic:

1.       I don’t think it could be wildly accepted in next two or three
years to have a pure IP datacenter. Pure Ethernet traffic and MAC
forwarding will be there anyway.

2.       From the dataplane point, you have to design larger IP routing
table, and possibly larger TCAM (for prefix entry), and higher cost
consequently. What’s more, should we even design aging for IP entry like
MAC entry?

3.       From the control plane point, you need to change the mapping entry
rule, and may have new interoperable issue with legacy L2/L3 VPN.

I don’t see much benefit. Could anyone give me some hints?

Thanks

Lizhong

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: 2013年10月24日 8:32
To: Xuxiaohu; Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas
Narten
Cc: [email-masked]; [email-masked]
Subject: Re: 答复: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

I've now read through this whole thread  twice.  I see now that there may
be some confusion between an L2/L3 service (what tenant's see) and an L2/L3
solution (how NVO3 provides the service).

Prior to this thread, I may have agreed that we don't need to define a
hybrid/combined/unified L2/L3 "service", but now I'm less sure. Just be be
clear, I was not calling so much for a definition of the service, but that
we need:

1) A name for the service (lets's pick one, is it hybrid, combined or
unified?)

2) To address it from a solution point of view (e.g. are packets forwarded
based on MAC or based on IP when source and dest are in the same subnet?).

As I stated before, I believe the control plane requirements will be
different for a combined service than for a pure L2 or pure L3 service, and
that we if don't start specifically calling out the control plane
requirements for a combined service (from a solution point of view), we
will have a very low likelihood of interoperable solutions as each
implementer  fills in the gaps in their own way.

I would also not object if we had a short description in the framework of
what the combined service looks like compared to a pure L2 or pure L3
service, just so we can level set for everyone.

Thanks, Larry

From: Xuxiaohu <[email-masked]>
Date: Tuesday, October 22, 2013 11:38 PM
To: Larry Kreeger <[email-masked]>, Lizhong Jin <[email-masked]>,
"Yves Hertoghs (yhertogh)" <[email-masked]>, Lucy yong <
[email-masked]>, Thomas Narten <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: 答复: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

发件人: [email-masked] [mailto:[email-masked]] 代表 Larry Kreeger
(kreeger)
发送时间: 2013年10月23日 11:27
收件人: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]; [email-masked]
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

VN.  Furthermore, as I wrote below, there needs to be a mapping of the set
of L2 VNs that belong to the L3 VN.

If so, what’s the difference between hybrid L2/L3 service and IRB
(Integrated Routing and Bridging)? In addition, I believe the hybrid L2/L3
service you mentioned above is different from what Yves meant by “combined
L2/L3”. The former is  about bridging for intra-subnet traffic and routing
for inter-subnet traffic while the latter is about routing for (both
intra-subnet and inter-subnet) IP traffic while bridging for non-IP traffic.

Best regards,

Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Lizhong Jin" <[email-masked]>
Date: Thu, 24 Oct 2013 10:22:50 +0800

One more negative point for IP forwarding for intra subnet traffic: if IPv6 is applied, you have to use much larger table to do IP forwarding than MAC forwarding.

Lizhong

From: Lizhong Jin [mailto:[email-masked]] 
Sent: 2013年10月24日 9:55
To: Larry Kreeger (kreeger); Xuxiaohu; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

[Lizhong] IRB, which have been used before. We could change it to virtual IRB.

[Lizhong] Yes, we should do that, and there are already some drafts there (http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00). For IP forwarding for intra subnet traffic:

1.       I don’t think it could be wildly accepted in next two or three years to have a pure IP datacenter. Pure Ethernet traffic and MAC forwarding will be there anyway.

2.       From the dataplane point, you have to design larger IP routing table, and possibly larger TCAM (for prefix entry), and higher cost consequently. What’s more, should we even design aging for IP entry like MAC entry?

3.       From the control plane point, you need to change the mapping entry rule, and may have new interoperable issue with legacy L2/L3 VPN.

Thanks

Lizhong

From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: 2013年10月24日 8:32
To: Xuxiaohu; Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
Cc: [email-masked]; [email-masked]
Subject: Re: 答复: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Thanks, Larry

To: Larry Kreeger <[email-masked]>, Lizhong Jin <[email-masked]>, "Yves Hertoghs (yhertogh)" <[email-masked]>, Lucy yong <[email-masked]>, Thomas Narten <[email-masked]>

Hi Lizong,

I'm expecting a hybrid L2/L3 service to preserve L2 semantics within a subnet.  TS ARP for the other TS on its subnet instead of the router.  The source and dest MAC are not rewritten.  TTL is not decremented.  >From the perspective of the two TS on the same subnet, they are connected to the same L2 segment.  To achieve this, the NVEs must behave differently when forwarding IP packets within the same L2 VN vs between different L2 VNs.  For example, the NVE would need to respond to ARP, not with its own MAC address, but with the remote TSs MAC address if the TS are in the same L2 VN.  Furthermore, as I wrote below, there needs to be a mapping of the set of L2 VNs that belong to the L3 VN.

Best regards,

Xiaohu

 - Larry

To: Larry Kreeger <[email-masked]>, "Yves Hertoghs (yhertogh)" <[email-masked]>, Lucy yong <[email-masked]>, Thomas Narten <[email-masked]>

Thanks
Lizhong

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 24 Oct 2013 03:23:41 +0000

发件人: Lizhong Jin [mailto:[email-masked]]
发送时间: 2013年10月24日 9:55
收件人: Larry Kreeger (kreeger); Xuxiaohu; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]; [email-masked]
主题: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

[Xiaohu] Regarding whether it is good enough to support IP traffic only in some data centers, I would like to hear more voices from real data center operators and vendors with customers who have asked for Layer3 overlay solutions. By the way, even there is non-IP traffic in some data centers, if non-IP traffic can be separated from IP traffic on the originating hosts by using different interfaces or VLANs for each of them, the interface/VLAN containing IP traffic can be mapped to a Layer3 overlay while the interface/VLAN containing non-IP traffic can be mapped to a Layer2 overlay. Otherwise, the ingress NVEs themselves could distinguish non-IP traffic from IP traffic through internal implementations then forward them to Layer 2 and Layer3 overlays respectively.

[Xiaohu] With the map&encap mechanism and the pull mode, the routing table scalability issue can be addressed well. As for the aging issue, it depends on how the IP entries are learnt, e.g., if they are learnt for ARP, the existing ARP aging mechanism is good enough to reflect the status of the IP entry. If they are learnt from an orchestration system, the orchestration system will deal with the aging issue. There is no much difference from the aging mechanism for MAC entries, IMHO.

[Xiaohu] What’s the new interoperable issue with legacy L3VPN? Can you say more?

Best regards,

Xiaohu

Thanks

Lizhong

Thanks, Larry

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] 代表 Larry Kreeger (kreeger)
发送时间: 2013年10月23日 11:27
收件人: Lizhong Jin; Yves Hertoghs (yhertogh); Lucy yong; Thomas Narten
抄送: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lizong,

Best regards,

Xiaohu

 - Larry

Thanks
Lizhong

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 24 Oct 2013 03:37:18 +0000

Agree. However, there is no free lunch in reality. BTW, in the distributed gateway deployment scenario, the NVEs have already installed either IPv4 or IPv6 host routes.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Lizhong Jin
发送时间: 2013年10月24日 10:23
收件人: 'Larry Kreeger (kreeger)'; Xuxiaohu; 'Yves Hertoghs (yhertogh)'; Lucy yong; 'Thomas Narten'
抄送: [email-masked]; [email-masked]
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lizhong

Hi Larry,

Thanks

Lizhong

Thanks, Larry

Hi Lizong,

Best regards,

Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 24 Oct 2013 09:26:51 +0000

Larry,

Yes agreed we do need to add some words to the framework that describes these type of NVE's

It seems we might need two new types of NVEs (note I am not using the word 'service'

  1.  An integrated L2/L3 NVE that provides discrete LAN-Emulation and IP-VRF type services.  This NVE is a logical combination of an L2NVE sub-funciton and L3NVE sub-function.  A node implementing this service type has VAPs connecting to ports or VLANs, and VAPs are aggregated through bridge-domains.  The discrete L2NVE sub-function attaches to the bridge-domain, allowing for per tenant Ethernet Emulation services.  A virtual Routing Interface also attaches to the bridge-domain, allowing for inter-VLAN routing.  The Virtual Routing Interface is part of the tenants VRF, and the VRF is extended across the network by the discrete L3NVE sub-function. (Note: i dont think this service type will change the control plane requirements, as it is merely a nodal disposition of functions).
  2.  A Unified L2/L3 NVE that provides per protocol

From: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
Date: Thursday 24 October 2013 02:32
To: Xuxiaohu <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: 答复: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Thanks, Larry

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 24 Oct 2013 09:35:46 +0000

Sorry hit send to soon.

Larry,

It seems we might need two new types of NVEs (note I am not using the word 'service')

  1.  An Integrated L2/L3 NVE that provides discrete LAN-Emulation and IP-VRF type services.  This NVE is a logical combination of an L2NVE sub-funciton and L3NVE sub-function.  A node implementing this service type has VAPs connecting to ports or VLANs, and VAPs are aggregated through bridge-domains.  The discrete L2NVE sub-function attaches to the bridge-domain, allowing for per tenant Ethernet Emulation services.  A virtual Routing Interface also attaches to the bridge-domain, allowing for inter-VLAN routing.  The Virtual Routing Interface is part of the tenants VRF, and the VRF is extended across the network by the discrete L3NVE sub-function. (Note: i dont think this service type will change the control plane requirements, as it is merely a nodal disposition of functions, although there might be value in unifying the control plane and data plane for the integration).
  2.  A Unified L2/L3 NVE that provides IP optimised forwarding and LAN Emulation type service.  This NVE builds on the Integrated NVE, but this time all IP traffic, independent of subnet/prefix boundaries is routed through either the Virtual Routing Interface, or through the L3 Overlay provided by the L3NVE sub function.  Non-IP traffic is still subject to the forwarding rules of the L2NVE sub function. (This NVE type could change the control plane requirements, as IP hosts will have to be discovered through their IP AND MAC addresses.  The data plane could be optimised to allow for L2/L3 integration.)

An additional note: switching between NVE types MUST be supported.

Yves

Thanks, Larry

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 24 Oct 2013 11:24:11 +0000

inline

From: Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Date: Thursday 24 October 2013 03:54
To: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Larry,

[yves] read draft-hertoghs-nvo3-lisp-unified-control-plane

Yves

Thanks

Lizhong

Thanks, Larry

Hi Lizong,

Best regards,

Xiaohu

 - Larry

Thanks
Lizhong

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 24 Oct 2013 23:06:02 +0800

Hi Yves,
Are you referring the benefits below:
Effectively this allows the 'spread' of
   subnets across the Datacenter(s) without leading to network wide
   broadcast and associated failure domains, while allowing free
   mobility of the end-stations.

MAC based forwarding will also not lead to network wide broadcast, if all
the MAC mapping entries are generated by NVA. I remember this has already
been discussed on NVO3 list last year.

The only benefit I could see is, for an IP only data center, you only need
to have IP forwarding.

Regards
Lizhong

On Oct 24, 2013 7:24 PM, "Yves Hertoghs (yhertogh)" <[email-masked]>
wrote:
> To: "Larry Kreeger (kreeger)" <[email-masked]>, Xuxiaohu <
[email-masked]>, Yves Hertoghs <[email-masked]>, Lucy yong <
[email-masked]>, Thomas Narten <[email-masked]>
document]
unified?)
>> 2) To address it from a solution point of view (e.g. are packets
forwarded based on MAC or based on IP when source and dest are in the same
subnet?).
>>
>> [Lizhong] Yes, we should do that, and there are already some drafts
there (http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00). For
>>
>> 3.       From the control plane point, you need to change the mapping
entry rule, and may have new interoperable issue with legacy L2/L3 VPN.
Narten
document]
>>
>>
>>
>> I've now read through this whole thread  twice.  I see now that there
may be some confusion between an L2/L3 service (what tenant's see) and an
L2/L3 solution (how NVO3 provides the service).
unified?)
>>
>> 2) To address it from a solution point of view (e.g. are packets
forwarded based on MAC or based on IP when source and dest are in the same
subnet?).
>> I would also not object if we had a short description in the framework
of what the combined service looks like compared to a pure L2 or pure L3
service, just so we can level set for everyone.
>> Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]
>
document]
>>
>>
>>
>> Best regards,
>>
>> Xiaohu
>>
>>
>>
>> So, I still think NVEs will need some additional information for a
hybrid service, and I think we will need to describe the behavior of a
hybrid service beyond a sentence in the framework saying it is possible.
 Otherwise, I think it is quite likely that two different hybrid
implementations will not interoperate.
>> Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]
>
document]

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 24 Oct 2013 15:54:11 +0000

Inline

From: Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Date: Thursday 24 October 2013 17:06
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>
Cc: Lucy yong <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Yves,

MAC based forwarding will also not lead to network wide broadcast, if all the MAC mapping entries are generated by NVA.

Since when is that a requirement for the NVA ?  An NVA will store the mappings, not generate it.  The mapping generation can be done through orchestration of the NVA, or through the NVE to NVA interactions as a result of the NVE detection new Tenant Systems.
Or are you saying that, independent of how the NVA knows about them, tenant broadcast will only get flooded within the scope of the VNI? If yes, why would this prevent from sending L2 broadcasts to all participating NVEs, and attached TS'es?  There might be some optimisations in the underlay to make this work 'better' , but it doesn't stop those ugly L2 broadcast packets from hitting all TS'es.  A L2 overlay creates a L2 failure domain, in the context of broadcast and multicast.  Nothing you can do about it.

I remember this has already been discussed on NVO3 list last year.

The only benefit I could see is, for an IP only data center, you only need to have IP forwarding.

Regards
Lizhong

---

From: Lucy yong <[email-masked]>
Date: Thu, 24 Oct 2013 16:11:01 +0000

Larry,

Yes, let's talk more in YVR.

Lucy

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Wednesday, October 23, 2013 6:48 PM
To: Lucy yong; Yves Hertoghs (yhertogh); Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Lucy,

I'm afraid we both may be misunderstanding each other.  When I read your statement "NVO3 has data plane and control plane requirements. In your opinion, where do they fit in this architecture?" I interpreted this to mean that because we have separate documents for data plane requirements from control plane requirements, than that leads to the assumption that they are separated.  I apologize if I misinterpreted you.

So far, IMO, NVO3 has not been about the OpenFlow architecture, nor SDN (whatever that means, let's leave that do the marketers and SDNRG), nor about control/data plane separation (FORCES).  The architecture calls for a logically centralized NVA to resolve inner to outer address mappings (among other things), and based on recent list discussions, NVEs may send messages directly to each other to help optimize the correction of a stale mapping.

Maybe we can have a conversation about this more easily when we are in Vancouver.

Thanks, Larry

On 10/22/13 10:24 PM, "Lucy yong" <[email-masked]> wrote:

> We are talking about the control plane for overlay virtual network! 
>Yes, every communication in overlay data plane and control plane relies 
>on underlying network. Traditional underlying networks do not have 
>control plane and data plane separation. Recently OF or SDN based 
>architecture uses the concept of control plane and data plane separation.
>
>>>order to be clear what are needed in the protocol. Without it, we 
>>>will later argue this in the protocol development and may end up a 
>>>heavy and complex protocol to support different implementations; or 
>>>make it clear later during the protocol development. Isn't it that 
>>>the architecture draft had intention in staying long in WG for this 
>>>reason? I am fine that people have different views and 
>>>interpretations of the description. In the end, we will see the 
>>>consequences and make consensus one way or another.
>>>
>>>Thanks,
>>>Lucy   
>>>
>>>I think you are interpreting the sections you quoted below 
>>>differently from how they were intended.  I'm not aware of any 
>>>mention of separation of forwarding from control in the architecture document.
>>>forwarding table information comes from the NVA.  An NVE can 
>>>certainly build local forwarding state all on its own.  Forwarding 
>>>between locally connected TS on the same NVE should not necessarily 
>>>require the NVA.
>>>
>>> - Larry
>>>
>>>>To clarify, this is not I say it, it is the architecture described 
>>>>in the arch. draft. Here is text from section 6.
>>>>
>>>>of data plane and control plane. In that architecture, the focus 
>>>>will be the protocol between NVA/NVE -NVA/NVE as suggested in 
>>>>another thread.
>>>>believe this is doesn't have to be the case for all deployments, as 
>>>>a matter of fact this separation of control plane versus data plane 
>>>>is merely a deployment model rather than an architectural foundation.
>>>>
>>>>L3 overlay (at all times), and non-IP traffic gets sent across the 
>>>>L2 overlay.
>>>>
>>>>Yves
>>>>
>>>>>>>add the text I proposed although we did not explicitly define as 
>>>>>>>a service type.
>>>>>>
>>>>>>I think we will need to because the control plane requirements 
>>>>>>will likely be different for a hybrid service.
>>>>>>[Lucy] Sorry to make you confuse. The routing here does not mean 
>>>>>>L3 services. I should state that NVE performs data plan 
>>>>>>forwarding, NVA performs control plane routing, which, IMO, it is 
>>>>>>not just simple DB to have inner to outer mappings. NVA may get 
>>>>>>the routing policy from operators or customer, then interpret the 
>>>>>>policy, then generates the mapping of tenant/next-hop location and send to NVEs.
>>>>>>between tenant/location is fully controlled under NVA. In this 
>>>>>>way, operator only needs to input the routing policies to NVA. NVE 
>>>>>>Inner to outer mapping resolution works fine for pure L2 or pure 
>>>>>>L3 service.  Local forwarding doesn¹t need mappings, the local NVE 
>>>>>>IMO: NVE is not the entity to enforce the policy. NVA is the 
>>>>>>entity to enforce the policy regardless the tenant locations. 
>>>>>>Again, Network virtualization overlay has to address how to 
>>>>>>support the policies and tenant mobility. IMO: current 
>>>>>>architecture draft is vague in or lacks of describing this but it 
>>>>>>is important to architect this when having data plan and control 
>>>>>>plane separated on different entities.
>>>>>
>>>>>>>Where we had previously stated L2 service or L3 service and 
>>>>>>>pretty much left a combined L2/L3 service as an exercise for the 
>>>>>>>reader, we would now be adding whatever mechanisms are needed to 
>>>>>>>the protocols.
>>>>>>>potentially L3).  We will also need to distribute two different 
>>>>>>>VN identifiers (one for L2 and one for L3), and somehow convey 
>>>>>>>>> Section 5.3 describes gateways. IMO: it misses an important 
>>>>>>>>> use case. A Gateway, say overlay gateway, may be used to 
>>>>>>>>> interconnect two or more overlay VNs. In this case, the 
>>>>>>>>> traffic traversing between two overlay VNs must go through the 
>>>>>>>>> gateway where the policy can be enforced. Furthermore, it is 
>>>>>>>>> possible to implement centralized or distributed overlay 
>>>>>>>>> gateway. The latter has overlay gateway function implemented 
>>>>>>>>> on NVEs. Thus, it requests the cross-VN policies to be distributed to NVEs.
>>>>>>>>

---

From: Thomas Narten <[email-masked]>
Date: Thu, 24 Oct 2013 16:57:29 -0400

OK, I'll bite. :)

> 1) A name for the service (lets's pick one, is it hybrid, combined

We do a need a name. "combined L2/L3" service seems about right to me.

> forwarded based on MAC or based on IP when source and dest are in
> the same subnet?). 

Right. We need a clear definition of what this means as a service
(i.e., facing the Tenant). How one implements it is a secondary discussion.

Is it enough to say something like:

  If the packet is IP, it's treated as an L3 packet requiring L3
  service.

  If it's a non-IP packet (except for things like ARP), it gets
  treated as an L2 packet to be processed using L2 service semantics.

Can it be that simple?

Thomas

---

From: Lucy yong <[email-masked]>
Date: Thu, 24 Oct 2013 21:24:16 +0000

We do a need a name. "combined L2/L3" service seems about right to me.
[Lucy] "Combined L2/L3 service", or "L2/L3 service"

> forwarded based on MAC or based on IP when source and dest are in the 
> same subnet?).

Right. We need a clear definition of what this means as a service (i.e., facing the Tenant). How one implements it is a secondary discussion.

Can it be that simple?
[Lucy] If the motivation of this service type is 1) no change on VM behavior 2) VM may communicate with VMs that may be in the same or different subnet. IMO: the packet from/to a VM is always Ethernet frame, and the dMAC address on the packet should be the VM MAC, and NVE has to support with ARP. I am fine with two "if".

Lucy

Thomas

---

From: Lucy yong <[email-masked]>
Date: Thu, 24 Oct 2013 21:46:25 +0000

I am glad that we get more support to add a new service type in the framework.

This new service type not only adds some additional requirements to control plan, but also to data plane requirement. For example,  NVE needs to terminate the L2 frame from  a tenant system, and add L2 header on an IP packet prior to forwarding the packet to a VAP.  NVE needs to support ARP.  Regarding the packet delivered between NVEs should be ip-in-ip or mac-in-ip, IMO: we should not make the decision now.

Lucy

Sent: Wednesday, October 23, 2013 7:32 PM

Thanks, Larry

Hi Lizong,

Best regards,
Xiaohu

 - Larry

Thanks
Lizhong

---

From: "Lizhong Jin" <[email-masked]>
Date: Fri, 25 Oct 2013 10:33:25 +0800

See inline.

From: Yves Hertoghs (yhertogh) [mailto:[email-masked]] 
Sent: 2013年10月24日 23:54
To: Lizhong Jin
Cc: Lucy yong; [email-masked]; Thomas Narten; Xuxiaohu; [email-masked]; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Inline

From: Lizhong Jin <[email-masked]>
Date: Thursday 24 October 2013 17:06
To: Yves Hertoghs <[email-masked]>
Cc: Lucy yong <[email-masked]>, "[email-masked]" <[email-masked]>, Thomas Narten <[email-masked]>, Xuxiaohu <[email-masked]>, "[email-masked]" <[email-masked]>, "Larry Kreeger (kreeger)" <[email-masked]>
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi Yves,

[Lizhong] Sorry for the confusion, I mean the MAC mapping of remote will be got from NVA, and MAC mapping of local will be got from NVA or generated by itself. I did not invent new things, and all of that is described in draft-narten-nvo3-arch-01. The NVE will reply the ARP from local TS if corresponding MAC mapping is already existed. For EVPN, broadcast only happen when there is no MAC-IP bindings locally, and that works well. There maybe some benefits existed from the multicast view, and I should think about it.

Thanks

Lizhong

Regards
Lizhong

On Oct 24, 2013 7:24 PM, "Yves Hertoghs (yhertogh)" <[email-masked]> wrote:
> To: "Larry Kreeger (kreeger)" <[email-masked]>, Xuxiaohu <[email-masked]>, Yves Hertoghs <[email-masked]>, Lucy yong <[email-masked]>, Thomas Narten <[email-masked]>
>

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Fri, 25 Oct 2013 10:07:15 +0000

Lizhong,

My main point is that , in the absence of non-IP traffic, there is a lot of value of doing IP based forwarding for all traffic, as this effectively isolates the failure domains to the L2 domains underneath the NVE's.

Yves

From: Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Date: Friday 25 October 2013 04:33
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>
Cc: Lucy yong <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, 'Thomas Narten' <[email-masked]<mailto:[email-masked]>>, 'Xuxiaohu' <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

See inline.

From: Yves Hertoghs (yhertogh) [mailto:[email-masked]]
Sent: 2013年10月24日 23:54
To: Lizhong Jin
Cc: Lucy yong; [email-masked]<mailto:[email-masked]>; Thomas Narten; Xuxiaohu; [email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Inline

Hi Yves,

Thanks
Lizhong

Regards
Lizhong

---

From: "Lizhong Jin" <[email-masked]>
Date: Fri, 25 Oct 2013 22:52:56 +0800

Yves,

In that scenario, I agree. But my concern is, is it realistic for a datacenter with only IP traffic. I am not sure. But at least as I know, if you want to deploy RoCE, there will be non-IP traffic.

Lizhong

From: Yves Hertoghs (yhertogh) [mailto:[email-masked]] 
Sent: Friday, October 25, 2013 6:07 PM
To: Lizhong Jin
Cc: 'Lucy yong'; [email-masked]; 'Thomas Narten'; 'Xuxiaohu'; [email-masked]; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lizhong,

Yves

Cc: Lucy yong <[email-masked]>, "[email-masked]" <[email-masked]>, 'Thomas Narten' <[email-masked]>, 'Xuxiaohu' <[email-masked]>, "[email-masked]" <[email-masked]>, "Larry Kreeger (kreeger)" <[email-masked]>

See inline.

Inline

Hi Yves,

Thanks

Lizhong

Regards
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Fri, 25 Oct 2013 15:05:08 +0000

True, but the unified L2/L3 solution gives you the choice on a per VNI basis whether you want to do 'enhanced IP forwarding' or 'Ethernet based forwarding'.

I also wouldn't call RDMA over Converged Ethernet a prime use case for a multi tenant DC.  Typically this would be deployed standalone and across a few ethernet switches.

Yves

Date: Friday 25 October 2013 16:52

Yves,

Lizhong

From: Yves Hertoghs (yhertogh) [mailto:[email-masked]]
Sent: Friday, October 25, 2013 6:07 PM
To: Lizhong Jin
Cc: 'Lucy yong'; [email-masked]<mailto:[email-masked]>; 'Thomas Narten'; 'Xuxiaohu'; [email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Lizhong,

Yves

See inline.

Inline

Hi Yves,

Thanks
Lizhong

Regards
Lizhong

---

From: "Xialiang (C)" <[email-masked]>
Date: Sat, 26 Oct 2013 09:42:14 +0000

Hi all,
Distributed Gateway is a very interesting topic. It would also be valuable to solve some challenge introduced by network virtualization.

To support the intra/inter-subnet connections through the same or different NVEs in one tenant, the Distributed Gateway should implement the bridging function and routing function at the same time.

But, there are still some points need to be studied and solved. Below are some examples (NVE1 and NVE2 are all Distributed Gateway. Hosts of subnet 1.1.1.0/24 and 2.2.2.0/24 are attached to them):
                     -------                    /
              -------       ------             /
            //                    \\          /
          //                        \\       /
        //                            \\    /
       /                                \  /
      /         +--------+      +-------+\/Internet
     /          |        |      |       |/\
    /           | CORE   |------+Gateway/  \
   /            |        |      |       |   \
  |             +--/----\*      +-------+    |
  |              //       \\                 |
|             //           \\                |
|           //               \\              |
|     +----/----+         +----\----+        |
|      |         |         |         |         |
|      |  NVE1             |  NVE2   |         |
|      |  /   \  |         |  /   \  |         |
|      +-/-----\-+         +-/-----\-+         |
|       /      \            /     \          |
|      /        \          /       \         |
|      ---        ----     /       \         |
  |   //   \\     /    \   /  --     \ ---   |
  |  |1.1.1.0|   |      |   //  \\    /   \  |
   \ |       |   |2.2.2.0  |      |  2.2.2.0/
    \ \\   //     \    /   |1.1.1.0   \   //
     \  ---        ----    |      |    ---/
      \                     \\  //       /
       \                      --        /
        \\                            //
          \\                        //
            \\                    //
              -------      -------

Problems 1: If we configure the same gateway address (i.e. 1.1.1.1) for the two subnets--1.1.1.0/24 attached to NVE1 and NVE2, and there is intra-subnet connection between them, it will result in an gateway address collision in one subnet. If we configure different gateway address for same subnet attached to different NVEs to solve the address collision problem, it will bring a huge configuration workload and waste of address space, and also it will limit the VM mobility.

Problem 2: How to aggregate or decrease the address number learned by NVEs or hosts?

Problem 3: How to deal with multi-tenants?

In addition to above problems, there are maybe more points around the Distributed Gateway issue.
I think the distributed gateway will solve those problems. But before we define any functions of it, we need a problem statement document for those scenarios firstly.

Your comments?

B.R.
Frank

From: [email-masked] [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Friday, October 25, 2013 11:05 PM
To: Lizhong Jin
Cc: 'Thomas Narten'; [email-masked]; [email-masked]; Lucy yong; Xuxiaohu; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Yves

Yves,

Lizhong

Lizhong,

Yves

See inline.

Inline

Hi Yves,

Thanks
Lizhong

Regards
Lizhong

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Sat, 26 Oct 2013 10:12:07 +0000

Inline.

From:  "Xialiang   (C)" <[email-masked]>
Date:  Saturday 26 October 2013 11:42
To:  Yves Hertoghs <[email-masked]>, Lizhong Jin <[email-masked]>
Cc:  'Thomas Narten' <[email-masked]>, "[email-masked]"
<[email-masked]>, "[email-masked]" <[email-masked]>, Lucy yong
<[email-masked]>, Xuxiaohu <[email-masked]>, "Larry Kreeger
(kreeger)" <[email-masked]>
Subject:  RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

>Hi all,
>Distributed Gateway is a very interesting topic. It would also be
>valuable to solve some challenge introduced by network virtualization.
> 
>To support the intra/inter-subnet connections through the same or
>different NVEs in one tenant, the Distributed Gateway should implement
>the bridging function and routing function at the same time.
> 
>But, there are still some points need to be studied and solved. Below are
>some examples (NVE1 and NVE2 are all Distributed Gateway. Hosts of subnet
>1.1.1.0/24 and 2.2.2.0/24 are attached to them):
>                     -------                    /
>              -------       ------             /
>            //                    \\          /
>          //                        \\       /
>        //                            \\    /
>       /                                \  /
>Problems 1: If we configure the same gateway address (i.e. 1.1.1.1) for
>the two subnets--1.1.1.0/24 attached to NVE1 and NVE2, and there is
>intra-subnet connection between them, it will result in an gateway
>address collision
> in one subnet. If we configure different gateway address for same subnet
>attached to different NVEs to solve the address collision problem, it
>will bring a huge configuration workload and waste of address space, and
>also it will limit the VM mobility.

Why ? You are thinking that the default gateway has subnets configured,
feeding into its routing table.  It doesn't.  It learns all hosts as
host-routes, with optional aggregation in case the host-routing table can
be aggregated on bit boundaries.  Host might have mask/prefix-lengths
configured, as that is how they are operating today, but in reality these
denote an address-pool rather than a subnet/prefix.  For IP traffic these
two parts of a subnet are not interconnected by a L2 link.  Surely there
might be a L2 overlay spanning both NVE's, but that overlay does not
switch IP packets.

>Problem 2: How to aggregate or decrease the address number learned by
In any case you will perform the same or better than when learning
MAC-addresses. With a MAC address namespace you cannot aggregate.  With an
IP namespace and host-route learning there is at least a change of
aggregation, either at the NVE, or higher up in the data centre.  And
actually the aggregation function isn't really a part of the NVE, it can
be a function of the NVA to aggregate address-space, if a contiguous block
of host-routes are all pointing to the same NVE.

>Problem 3: How to deal with multi-tenants?

This is orthogonal.  The NVE learns about hosts and their IP addresses
within the NV instance. I dont see why multi tenancy is a problem ?
 
>I think the distributed gateway will solve those problems. But before we
>define any functions of it, we need a problem statement document for
>those scenarios firstly.
I think the problem statement of how the solution should look like from a
user's perspective is already there.  A distributed gateway, with a
unified L2/L3 NVE on it, is an option to solve those problems.

>True, but the unified L2/L3 solution gives you the choice on a per VNI
>basis whether you want to do 'enhanced IP forwarding' or 'Ethernet based
>forwarding'.
>
> 
>
>I also wouldn't call RDMA over Converged Ethernet a prime use case for a
>multi tenant DC.  Typically this would be deployed standalone and across
> a few ethernet switches.
>
> 
>
>Yves
>
> 
>
>From:
>Cc: Lucy yong <[email-masked]>, "[email-masked]"
><[email-masked]>, 'Thomas Narten' <[email-masked]>,
> 'Xuxiaohu' <[email-masked]>, "[email-masked]" <[email-masked]>,
>"Larry Kreeger (kreeger)" <[email-masked]>
>>In that scenario, I agree. But my concern is, is it realistic for a
>>datacenter with only IP traffic. I am not sure. But at least as I know,
>>if
>> you want to deploy RoCE, there will be non-IP traffic.
>> 
>>Lizhong
>> 
>>My main point is that , in the absence of non-IP traffic, there is a lot
>>of value of doing IP based forwarding for all traffic, as this
>>effectively
>> isolates the failure domains to the L2 domains underneath the NVE's.
>>
>> 
>>
>>Yves
>>
>> 
>>
>>From:
>>Cc: Lucy yong <[email-masked]>, "[email-masked]"
>><[email-masked]>, 'Thomas Narten' <[email-masked]>,
>> 'Xuxiaohu' <[email-masked]>, "[email-masked]" <[email-masked]>,
>>"Larry Kreeger (kreeger)" <[email-masked]>
>>>Sent: 2013年10月24日
>>> 23:54
>>>Cc: Lucy yong <[email-masked]>, "[email-masked]"
>>><[email-masked]>, Thomas Narten <[email-masked]>,
>>> Xuxiaohu <[email-masked]>, "[email-masked]" <[email-masked]>,
>>>"Larry Kreeger (kreeger)" <[email-masked]>
>>>>MAC based forwarding will also not lead to network wide broadcast, if
>>>>all the MAC mapping entries are generated by NVA.
>>>>
>>>>
>>>>
>>>Since when is that a requirement for the NVA ?  An NVA will store the
>>>mappings, not generate it.  The mapping generation can be done through
>>>orchestration
>>> of the NVA, or through the NVE to NVA interactions as a result of the
>>>NVE detection new Tenant Systems.
>>>
>>>Or are you saying that, independent of how the NVA knows about them,
>>>tenant broadcast will only get flooded within the scope of the VNI? If
>>>yes,
>>> why would this prevent from sending L2 broadcasts to all participating
>>>NVEs, and attached TS'es?  There might be some optimisations in the
>>>underlay to make this work 'better' , but it doesn't stop those ugly L2
>>>broadcast packets from hitting all TS'es.  A
>>> L2 overlay creates a L2 failure domain, in the context of broadcast
>>>and multicast.  Nothing you can do about it.
>>> 
>>>[Lizhong] Sorry for the confusion, I mean the MAC mapping of remote
>>>will be got from NVA, and MAC mapping of local will be got from NVA or
>>>generated
>>> by itself. I did not invent new things, and all of that is described
>>>in draft-narten-nvo3-arch-01. The NVE will reply the ARP from local TS
>>>if corresponding MAC mapping is already existed. For EVPN, broadcast
>>>only happen when there is no MAC-IP bindings locally,
>>> and that works well. There maybe some benefits existed from the
>>>multicast view, and I should think about it.
>>> 
>>>Thanks
>>>Lizhong
>>>
>>> 
>>>
>>>>> To: "Larry Kreeger (kreeger)" <[email-masked]>, Xuxiaohu
>>>>><[email-masked]>, Yves Hertoghs <[email-masked]>, Lucy yong
>>>>><[email-masked]>,
>>>> Thomas Narten <[email-masked]>
>>>>>
>>>>>>there 
>>>>>>(http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00). For
>>>>>> 1.       I don’t think it could be wildly accepted in next two or
>>>>>>three years to have a pure IP datacenter. Pure Ethernet traffic and
>>>>>>MAC forwarding will be there anyway.
>>>>>>
>>>>>> 2.       From the dataplane point, you have to design larger IP
>>>>>>routing table, and possibly larger TCAM (for prefix entry), and
>>>>>>higher cost consequently. What’s more, should we even design aging
>>>>>>for IP entry like MAC entry?
>>>>>>
>>>>>> 3.       From the control plane point, you need to change the
>>>>>>mapping entry rule, and may have new interoperable issue with legacy
>>>>>>L2/L3 VPN.
>>>>>>
>>>>>> Sent: 2013年10月24日
>>>> 8:32
>>>>>> Cc: [email-masked];
>>>>[email-masked] <mailto:[email-masked]>
>>>>>> I've now read through this whole thread  twice.  I see now that
>>>>>>there may be some confusion between an L2/L3 service (what tenant's
>>>>>>see) and an L2/L3 solution (how NVO3 provides the service).
>>>>>>
>>>>>>  
>>>>>>
>>>>>> Prior to this thread, I may have agreed that we don't need to
>>>>>>define a hybrid/combined/unified L2/L3 "service", but now I'm less
>>>>>>sure. Just be be clear, I was not calling so much for a definition
>>>>>>of the service, but that we need:
>>>>>>
>>>>>>  
>>>>>>
>>>>>> As I stated before, I believe the control plane requirements will
>>>>>>be different for a combined service than for a pure L2 or pure L3
>>>>>>service, and that we if don't start specifically calling out the
>>>>>>control plane requirements for a combined service (from a
>>>> solution point of view), we will have a very low likelihood of
>>>>interoperable solutions as each implementer  fills in the gaps in
>>>>their own way.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> I would also not object if we had a short description in the
>>>>>>framework of what the combined service looks like compared to a pure
>>>>>>L2 or pure L3 service, just so we can level set for everyone.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> Thanks, Larry
>>>>>>
>>>>>>  
>>>>>>
>>>>>> To: Larry Kreeger <[email-masked]>, Lizhong Jin
>>>>>><[email-masked]>, "Yves Hertoghs (yhertogh)"
>>>>>><[email-masked]>, Lucy yong <[email-masked]>,
>>>> Thomas Narten <[email-masked]>
>>>>>> 发件人:
>>>>[email-masked] [mailto:[email-masked]]
>>>>代表 Larry Kreeger (kreeger)
>>>>>> 发送时间: 2013年10月23日
>>>> 11:27
>>>>>> 抄送:
>>>>[email-masked]; 
>>>>[email-masked] <mailto:[email-masked]>
>>>>>> A pure L2 service only requires inner MAC address to outer IP
>>>>>>address mappings. When a frame is delivered between two TS, the
>>>>>>source and dest MAC addresses have not been changed.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> A pure L3 service only requires inner IP to outer IP address
>>>>>>mappings.  There is no expectation that two TS are on the same L2
>>>>>>segment, and the TS would always ARP for a router to reach another
>>>>>>TS.  When a frame is delivered between the two TS, both the
>>>> source and dest MAC will have been rewritten.  TTL will have been
>>>>decremented at least once.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> I'm expecting a hybrid L2/L3 service to preserve L2 semantics
>>>>>>within a subnet.  TS ARP for the other TS on its subnet instead of
>>>>>>the router.  The source and dest MAC are not rewritten.  TTL is not
>>>>>>decremented.  From the perspective of the two TS on the same
>>>> subnet, they are connected to the same L2 segment.  To achieve this,
>>>>the NVEs must behave differently when forwarding IP packets within the
>>>>same L2 VN vs between different L2 VNs.  For example, the NVE would
>>>>need to respond to ARP, not with its own MAC address,
>>>> but with the remote TSs MAC address if the TS are in the same L2 VN.
>>>>Furthermore, as I wrote below, there needs to be a mapping of the set
>>>>>>(Integrated Routing and Bridging)? In addition, I believe the hybrid
>>>>>>L2/L3 service you mentioned above is different from what Yves meant
>>>>>>by “combined L2/L3”. The former is  about bridging
>>>> for intra-subnet traffic and routing for inter-subnet traffic while
>>>>the latter is about routing for (both intra-subnet and inter-subnet)
>>>>IP traffic while bridging for non-IP traffic.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Xiaohu
>>>>>>
>>>>>>  
>>>>>>
>>>>>>hybrid service, and I think we will need to describe the behavior of
>>>>>>a hybrid service beyond a sentence in the framework saying it is
>>>>>>possible.  Otherwise, I think it is quite likely that
>>>> two different hybrid implementations will not interoperate.
>>>>>>
>>>>>>  
>>>>>>
>>>>>>  - Larry
>>>>>>
>>>>>>  
>>>>>>
>>>>>> To: Larry Kreeger <[email-masked]>, "Yves Hertoghs (yhertogh)"
>>>>>><[email-masked]>, Lucy yong <[email-masked]>, Thomas Narten
>>>> <[email-masked]>
>>>>>> reference 
>>>>http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00
>>>><http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00>.
>>>>>>add
>>>>>>will
>>>>>>either
>>>>>>service,
>>>>>>L3
>>>>>>NVA
>>>>>>receives
>>>>>>between
>>>>>> >>>[Lucy] Let me know above explanation help or not. No, not
>>>>>>particulate
>>>>>> >>>to
>>>>>>leave
>>>>>>L3
>>>>>>Network
>>>>>>pretty
>>>>>>simple
>>>>>>local
>>>>>>make
>>>>>>use
>>>>>>interconnect

---

From: "Black, David" <[email-masked]>
Date: Sun, 27 Oct 2013 18:09:38 -0400

>might be a L2 overlay spanning both NVE's, but that overlay does not

Right - "Doctor it hurts when I do <this>".  "Don't do <that>!" :-),

i.e., don't connect subnets across NVEs.  That's the simplest approach,

but when it's not sufficient ...

... this particular L2 problem was addressed a long time ago, e.g., by

VRRP.  See RFC 5798, and note that the original specification of VRRP

was over a decade earlier, RFC 2338.

Thanks,

--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754
----------------------------------------------------
________________________________
From: [email-masked] [[email-masked]] On Behalf Of Yves Hertoghs (yhertogh) [[email-masked]]
Sent: Saturday, October 26, 2013 6:12 AM
To: Xialiang (C); Lizhong Jin
Cc: 'Thomas Narten'; [email-masked]; [email-masked]; Lucy yong; Xuxiaohu; Larry Kreeger (kreeger)
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Inline.

>>>>>> >>>service. Local forwarding doesn?t need mappings, the local NVE
>>>>>>knows

---

From: Jon Hudson <[email-masked]>
Date: Sun, 27 Oct 2013 16:44:34 -0700

<inline>

> On Oct 27, 2013, at 3:09 PM, "Black, David" <[email-masked]> wrote:
> 
> but when it's not sufficient

> ...

+1 

And there have been some very nice solutions in the market built on top of VRRP for a while now. 

> [email-masked]        Mobile: +1 (978) 394-7754

---

From: "Xialiang (Frank)" <[email-masked]>
Date: Mon, 28 Oct 2013 02:49:25 +0000

Hi Yves,
Thanks for your response.
Please see my response with [Frank].

B.R.
Frank

> Sent: Saturday, October 26, 2013 6:12 PM
> <[email-masked]>, Xuxiaohu <[email-masked]>, "Larry
> Kreeger (kreeger)" <[email-masked]>
> >But, there are still some points need to be studied and solved. Below
> >are some examples (NVE1 and NVE2 are all Distributed Gateway. Hosts of
> >subnet
> >address collision  in one subnet. If we configure different gateway
> >address for same subnet attached to different NVEs to solve the address
> >collision problem, it will bring a huge configuration workload and
> >waste of address space, and also it will limit the VM mobility.
> 
> feeding into its routing table.  It doesn't.  It learns all hosts as host-routes,
> with optional aggregation in case the host-routing table can be aggregated
> on bit boundaries.  Host might have mask/prefix-lengths configured, as that
> is how they are operating today, but in reality these denote an address-pool
> rather than a subnet/prefix.  For IP traffic these two parts of a subnet are
> not interconnected by a L2 link.  Surely there might be a L2 overlay spanning
> both NVE's, but that overlay does not switch IP packets.
> 
[Frank]: Maybe my description is not so clear. What I mean is every host in the subnet configures its gateway for IP forwarding. For the same subnet distributed in different NVEs, Same gateway address located in different NVEs can exist. When these gateways in different NVEs send gratuitous ARP messages in the subnet, or several VRRP groups in same subnet send VRRP messages in the subnet, the IP address collision between gateways will be detected. How to solve this kind of problem?

> MAC-addresses. With a MAC address namespace you cannot aggregate.
> With an IP namespace and host-route learning there is at least a change of
> aggregation, either at the NVE, or higher up in the data centre.  And actually
> the aggregation function isn't really a part of the NVE, it can be a function of
> the NVA to aggregate address-space, if a contiguous block of host-routes are
> all pointing to the same NVE.
> 
[Frank]: Right, I agree your analysis about address aggregation. Actually, what I am concerned is the possibility to decrease or need not to learn all host's address by some gateway-like method.

> >I think the distributed gateway will solve those problems. But before
> >we define any functions of it, we need a problem statement document for
> >those scenarios firstly.
> user's perspective is already there.  A distributed gateway, with a unified
> L2/L3 NVE on it, is an option to solve those problems.
> 
[Frank]: It depends on how many problems we find by discussion and come to the agreement. We can keep on discussing.

> >basis whether you want to do 'enhanced IP forwarding' or 'Ethernet
> >based forwarding'.
> >
> >
> >
> >I also wouldn't call RDMA over Converged Ethernet a prime use case for
> >a multi tenant DC.  Typically this would be deployed standalone and
> >across  a few ethernet switches.
> >
> >
> >
> >Yves
> >
> >
> >
> >From:
> 'Xuxiaohu'
> ><[email-masked]>, "[email-masked]" <[email-masked]>, "Larry Kreeger
> >>datacenter with only IP traffic. I am not sure. But at least as I
> >>know, if  you want to deploy RoCE, there will be non-IP traffic.
> >>
> >>Lizhong
> >>
> >>My main point is that , in the absence of non-IP traffic, there is a
> >>lot of value of doing IP based forwarding for all traffic, as this
> >>effectively  isolates the failure domains to the L2 domains underneath
> >>the NVE's.
> >>
> >>
> >>
> >>Yves
> >>
> >>
> >>
> >>From:
> 'Xuxiaohu'
> >><[email-masked]>, "[email-masked]" <[email-masked]>, "Larry
> Kreeger
> Xuxiaohu
> >>><[email-masked]>, "[email-masked]" <[email-masked]>, "Larry
> >>>Kreeger (kreeger)" <[email-masked]>
> >>>>MAC based forwarding will also not lead to network wide broadcast,
> >>>>if all the MAC mapping entries are generated by NVA.
> >>>>
> >>>>
> >>>>
> >>>mappings, not generate it.  The mapping generation can be done
> >>>through orchestration  of the NVA, or through the NVE to NVA
> >>>interactions as a result of the NVE detection new Tenant Systems.
> >>>
> >>>tenant broadcast will only get flooded within the scope of the VNI?
> >>>If yes,  why would this prevent from sending L2 broadcasts to all
> >>>participating NVEs, and attached TS'es?  There might be some
> >>>optimisations in the underlay to make this work 'better' , but it
> >>>doesn't stop those ugly L2 broadcast packets from hitting all TS'es.
> >>>A
> >>>will be got from NVA, and MAC mapping of local will be got from NVA
> >>>or generated  by itself. I did not invent new things, and all of that
> >>>is described in draft-narten-nvo3-arch-01. The NVE will reply the ARP
> >>>from local TS if corresponding MAC mapping is already existed. For
> >>>EVPN, broadcast only happen when there is no MAC-IP bindings locally,
> >>>>><[email-masked]>, Yves Hertoghs <[email-masked]>, Lucy
> >>>>>yong <[email-masked]>,
> >>>>>> [Lizhong] Yes, we should do that, and there are already some
> >>>>>>drafts there
> >>>>>>(http://tools.ietf.org/html/draft-jin-nvo3-virb-centralized-00).
> >>>>>>For IP forwarding for intra subnet traffic:
> >>>>>>
> >>>>>>three years to have a pure IP datacenter. Pure Ethernet traffic
> >>>>>>and MAC forwarding will be there anyway.
> >>>>>>
> >>>>>>the router.  The source and dest MAC are not rewritten.  TTL is
> >>>>>>not decremented.  From the perspective of the two TS on the same
> >>>> subnet, they are connected to the same L2 segment.  To achieve
> >>>>this, the NVEs must behave differently when forwarding IP packets
> >>>>within the same L2 VN vs between different L2 VNs.  For example, the
> >>>>NVE would need to respond to ARP, not with its own MAC address,  but
> >>>>with the remote TSs MAC address if the TS are in the same L2 VN.
> >>>>Furthermore, as I wrote below, there needs to be a mapping of the
> >>>>set of L2 VNs that belong to the L3 VN.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>L2/L3 service you mentioned above is different from what Yves
> >>>>>>meant by “combined L2/L3”. The former is  about bridging
> >>>>>> So, I still think NVEs will need some additional information for
> >>>>>>a hybrid service, and I think we will need to describe the
> >>>>>>behavior of a hybrid service beyond a sentence in the framework
> >>>>>>saying it is possible.  Otherwise, I think it is quite likely that
> >>>>>>>http://tools.ietf.org/html/draft-hertoghs-nvo3-lisp-controlplane-
> >>>>>>>uni
> >>>>>>>fie
> >>>>>> >d-0
> >>>>>> >0
> >>>>>> >be called a new service-type. I see it more as a deployment
> >>>>>> >choice where you choose to deploy distributed gateways with
> >>>>>> >uniform mac/IP addressing across the DC, collocate them with the
> >>>>>> >L2 overlay, and
> >>>>>>do
> >>>>>> >some traffic steering to make sure IP traffic gets sent to the
> >>>>>> be different for L2 vs L3 vs a combined L2/L3 service. For an L2
> >>>>>>VN, the  VN needs to be identified (e.g. with a Name or ID). For
> >>>>>>an L3 VN, it  similarly needs to be identified. For a combined
> >>>>>>L2/L3 service, I think  the NVEs need to know both the L3 VN
> >>>>>>identity (one), and all the L2 VN  identities that are part of the
> >>>>>>L3 VN. When doing distributed L3  forwarding between a TS on one
> >>>>>>L2 VN to one on another L2 VN, it will need  to know not just the
> >>>>>>mapping of inner to outer address, but the mapping of  inner L3
> >>>>>>address to destination L2 VN and MAC address (so it can rewrite
> >>>>>>the MAC and the L2 VN).
> >>>>>>
> >>>>>> The behavior you described for L3 forwarding between two L2 VPN
> >>>>>>is more of  a standard L3 forwarding function, looking up with
> >>>>>>destination IP address,  then getting the destination MAC, source
> >>>>>>MAC, VLAN and NVO3 tunnel from  the table. The MAC and VLAN
> are
> >>>>>>resolved from the next hop within same L3  VN. We write a
> combined
> >>>>>>L2/L3 forwarding for NVO3 last year, just for  reference
> >>>>>> >>>>Distributed L3 gateway is very useful and some vendors
> >>>>>> >>>>already implement that.
> >>>>>> >>>
> >>>>>> >>>I agree.
> >>>>>> >>>
> >>>>>> >>>[Lucy] Do you mean that we should define an l2/L3 service
> >>>>>> >>>type? I full agree and hope see more people support that too.
> >>>>>> >>>We can
> >>>>>>either
> >>>>>> >>>define it in the existing framework, or in the framework
> >>>>>> >>>to have inner to outer mappings. NVA may get the routing
> >>>>>> >>>policy
> >>>>>>from
> >>>>>> >>>operators or customer, then interpret the policy, then
> >>>>>> >>>generates
> >>>>>>the
> >>>>>> >>>the mapping in which, if the location is the same as itself,
> >>>>>> >>>it translates to a tenant/VAP mapping; If not, it installs as
> >>>>>> >>>an inner/outer mapping.
> >>>>>> >>>Thus, it works regardless whether sender tenant and
> >>>>>> >>>destination tenant are one the same NVE or on different NVEs.
> >>>>>> >>>The mapping
> >>>>>>between
> >>>>>> >>>>If NVA is not able to distribute routing policy to NVE at
> >>>>>> >>>>all,
> >>>>>>I do
> >>>>>> >>>tenant mobility. IMO: current architecture draft is vague in
> >>>>>> >>>or
> >>>>>>lacks
> >>>>>> >>LK2> I'm not sure if I agree with you about the NVE not being
> >>>>>> >>LK2> the entity
> >>>>>> >>>>Support of a distributed L3 gateway function between L2 VNs
> >>>>>> >>>>is a significant increase in scope of the NVA, and the NVE to
> >>>>>> >>>>NVA
> >>>>>>protocol.
> >>>>>> >>>>destination MAC address (first L2, then potentially L3). We
> >>>>>> >>>>will also need to distribute two different VN identifiers
> >>>>>> >>>>(one for
> >>>>>>L2 and
> >>>>>> >>>>>> two or more overlay VNs. In this case, the traffic
> >>>>>> >>>>>> traversing between two overlay VNs must go through the
> >>>>>> >>>>>> gateway where the policy can be enforced. Furthermore, it
> >>>>>> >>>>>> is possible to
> >>>>>>implement
> >>>>>> >>>>>> Current section seems very focus on overlay VN
> >>>>>> >>>>>> interconnect a non-overlay network, which centralized
> >>>>>> >>>>>> gateway architecture
> >>>>>>is
> >>>>>> >>>>> <section title="Distributed Gateways"> <t> The relaying of
> >>>>>> >>>>> traffic from one VN to another deserves special
> >>>>>> >>>>> consideration. The previous section described gateways
> >>>>>> >>>>> performing this function. If such gateways are centralized,
> >>>>>> >>>>> traffic between TSes on different VNs can take suboptimal
> >>>>>> >>>>> paths, i.e., triangular routing results in paths that
> >>>>>> >>>>> always traverse the gateway. As an optimization, individual
> >>>>>> >>>>> NVEs can be part of a distributed gateway that performs
> >>>>>> >>>>> such relaying, reducing or completely eliminating
> >>>>>> >>>>> whether cross-VN communication is allowed. Having
> >>>>>> >>>>> individual NVEs be part of a distributed gateway allows
> >>>>>> >>>>> them to tunnel traffic directly to the destination NVE
> >>>>>> >>>>> without the need to take suboptimal paths.
> >>>>>> >>>>> </t>
> >>>>>> >>>>> <t>

---

From: "Xialiang (Frank)" <[email-masked]>
Date: Mon, 28 Oct 2013 03:21:26 +0000

Hi David,
Interest explanation~~
Please see my response inline.

B.R.
Frank

From: Black, David [mailto:[email-masked]]
Sent: Monday, October 28, 2013 6:10 AM
To: Yves Hertoghs (yhertogh); Xialiang (Frank)
Cc: [email-masked]
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

[Frank] : If I understand you correctly, you mean these gateways shared the same address, but located in different NVEs, can compose a VRRP group to avoid the IP address collision problem. Right?

If so, I agree it may be a possible solution. But I think it’s not the conventional VRRP protocol because all the gateways negotiate their VRRP messages in the interfaces different to their hosts-side interface used for ARP and data plane. And only one router in the VRRP is active will result in some hosts send their traffic to a remote gateway to form a suboptimal path.

Thanks,
--David
From: [email-masked]<mailto:[email-masked]> [[email-masked]] On Behalf Of Yves Hertoghs (yhertogh) [[email-masked]]
Cc: 'Thomas Narten'; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Lucy yong; Xuxiaohu; Larry Kreeger (kreeger)

From:  "Xialiang   (C)" <[email-masked]<mailto:[email-masked]>>
Date:  Saturday 26 October 2013 11:42
To:  Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Cc:  'Thomas Narten' <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
<[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Lucy yong
<[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "Larry Kreeger
(kreeger)" <[email-masked]<mailto:[email-masked]>>
Subject:  RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture
document]

>Cc: 'Thomas Narten'; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Lucy yong;
>Xuxiaohu; Larry Kreeger (kreeger)
>From:
>Lizhong Jin <[email-masked]<mailto:[email-masked]>>
>Cc: Lucy yong <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
><[email-masked]<mailto:[email-masked]>>, 'Thomas Narten' <[email-masked]<mailto:[email-masked]>>,
> 'Xuxiaohu' <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>,
>"Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
>>Cc: 'Lucy yong'; [email-masked]<mailto:[email-masked]>; 'Thomas Narten'; 'Xuxiaohu';
>>[email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger)
>>From:
>>Lizhong Jin <[email-masked]<mailto:[email-masked]>>
>>Cc: Lucy yong <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
>><[email-masked]<mailto:[email-masked]>>, 'Thomas Narten' <[email-masked]<mailto:[email-masked]>>,
>> 'Xuxiaohu' <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>,
>>"Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
>>>Cc: Lucy yong; [email-masked]<mailto:[email-masked]>; Thomas Narten; Xuxiaohu;
>>>[email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger)
>>>From:
>>>Lizhong Jin <[email-masked]<mailto:[email-masked]>>
>>>Cc: Lucy yong <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
>>><[email-masked]<mailto:[email-masked]>>, Thomas Narten <[email-masked]<mailto:[email-masked]>>,
>>> Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>,
>>>"Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
>>>>On Oct 24, 2013 7:24 PM, "Yves Hertoghs (yhertogh)"
>>>><[email-masked]<mailto:[email-masked]>> wrote:
>>>>>
>>>>> inline
>>>>>
>>>>> To: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>, Xuxiaohu
>>>>><[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong
>>>>><[email-masked]<mailto:[email-masked]>>,
>>>> Thomas Narten <[email-masked]<mailto:[email-masked]>>
>>>>>
>>>>> Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
>>>>><[email-masked]<mailto:[email-masked]>>
>>>>>> Cc: [email-masked]<mailto:[email-masked]>;
>>>>[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
>>>>>> To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, Lizhong Jin
>>>>>><[email-masked]<mailto:[email-masked]>>, "Yves Hertoghs (yhertogh)"
>>>>>><[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>,
>>>> Thomas Narten <[email-masked]<mailto:[email-masked]>>
>>>>>> Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
>>>>>><[email-masked]<mailto:[email-masked]>>
>>>>[email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
>>>>[email-masked]<mailto:[email-masked]>;
>>>>[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
>>>>>> To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, "Yves Hertoghs (yhertogh)"
>>>>>><[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Thomas Narten
>>>> <[email-masked]<mailto:[email-masked]>>
>>>>>> Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>"
>>>>>><[email-masked]<mailto:[email-masked]>>
>>>>>>VN, the

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Mon, 28 Oct 2013 10:32:45 +0000

David,

Inline w [yves]

From: <Black>, David <[email-masked]<mailto:[email-masked]>>
Date: Monday 28 October 2013 00:09
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, "Xialiang (C)" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

I dont think you are fully understanding what is being proposed.  The described solution DOES allow to connect subnets across NVE's.  It just doesn't use L2 to achieve that.
It was all fine to use L2 in order to span subnets across two distribution switches, but here the requirements is to span the subnet across a set of arbitrary locations…totally different problem statement imo.  Kneejerk reaction seems to be: oh, lets do what we know: span the broadcast domain across all of the locations.  The fact that there is an overlay in-between the locations seems to allow the knee to jerk even more and use MAC-addresses for forwarding across the L2 overlay.  My point is: lets take a step back and look at what the requirements are:

  *   TS'es dont have  to understand  or need to be exposed to the mechanics of the overlay
  *   IP is the main protocol type
  *   Hosts expect to be part of a subnet and will ARP for one another
  *   Avoid L2 domains spanning the entire infrastructure

VRRP leads to all sort of weird traffic tromboning issues when subnets are spanned across locations using L2.

Yves

Thanks,

--David
From: [email-masked]<mailto:[email-masked]> [[email-masked]<mailto:[email-masked]>] On Behalf Of Yves Hertoghs (yhertogh) [[email-masked]<mailto:[email-masked]>]

Inline.

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Mon, 28 Oct 2013 10:39:20 +0000

Frank,

inline

On 28/10/13 03:49, "Xialiang (Frank)" <[email-masked]> wrote:

>>host-routes,
>[Frank]: Maybe my description is not so clear. What I mean is every host
>in the subnet configures its gateway for IP forwarding. For the same
>subnet distributed in different NVEs, Same gateway address located in
>different NVEs can exist. When these gateways in different NVEs send
>gratuitous ARP messages in the subnet, or several VRRP groups in same
>subnet send VRRP messages in the subnet, the IP address collision between
>gateways will be detected. How to solve this kind of problem?
In the topology drawn by you i dont see any L2 interconnection between
NVE1 and NVE2.  You dont even need VRRP is this topology.  You are again
assuming a distributed subnets equals one L2 domain.  It doesn't need to
be like this, as long as all hosts are treated as host-routes on the NVEs
i.e. The NVE is not configured for a certain TS prefix on an interface.
It dynamically installs the host-routes based on Tses registering (via VQP
or some other means) and pushes them to the NVA.  The subnet is no longer
a subnet, it is merely an address-pool.

>>actually
>[Frank]: Right, I agree your analysis about address aggregation.
>Actually, what I am concerned is the possibility to decrease or need not
>to learn all host's address by some gateway-like method.
>
>> >
>>unified
>[Frank]: It depends on how many problems we find by discussion and come
>to the agreement. We can keep on discussing.
Haven't seen any problem yet, as this model is implemented already.  The
biggest problem is that people dont fully understand it, and perhaps we
need a separate draft to discuss this in detail.  Draft-hertoghs does
explain it, but the LISP machinery around it might confuse people i.e the
distributed, unified L2/L3 NVE is orthogonal to the NVE-NVA CP being used.

Yves
>> >Lizhong Jin <[email-masked]>
>> >>Lizhong Jin <[email-masked]>
>> >>>Lizhong Jin <[email-masked]>
>> >>>><[email-masked]> wrote:
>> >>>>>
>> >>>>> inline
>> >>>>>

---

From: "Black, David" <[email-masked]>
Date: Mon, 28 Oct 2013 10:00:57 -0400

> [Frank] : If I understand you correctly, you mean these gateways shared the same address, but located in different NVEs,

> can compose a VRRP group to avoid the IP address collision problem. Right?

That’s a start ...

> If so, I agree it may be a possible solution. But I think it’s not the conventional VRRP protocol because all the gateways negotiate their VRRP

> messages in the interfaces different to their hosts-side interface used for ARP and data plane. And only one router in the VRRP is active will

> result in some hosts send their traffic to a remote gateway to form a suboptimal path.

I’m sorry to be blunt, but please go *read* RFC 5798, including section 4.2
- the statement that “only one router in the VRRP is active” misses the
important distinction between virtual routers and physical routers.  That
said, dynamic optimality in gateway usage as routes shift and loads change
may involve a routing protocol, and that’s also discussed
in RFC 5798.

There is a problem here beyond what VRRP does.  VRRP does a fine job when
the routers (NVEs) are fully substitutable (i.e., it doesn’t matter which one
is used) – putting a pair of NVEs in ToR switches in the same rack is an
example.  OTOH, when it does matter, one approach is to make the inter-NVE
portion of the VN a default-free zone.  This involves picking off traffic for
the default gateway at the first NVE it encounters so that neither the MAC
nor IP of the default gateway shows up as either a source or destination
address in encapsulated traffic among NVEs – this also requires blocking
default gateway MAC/IP propagation across the intra-subnet connection that
spans NVEs (or VRRP groups of NVEs) so that the default gateway MAC/IP
pair always points at one NVE.

Beyond that, Yves and I appear to be talking past each other:

> I don’t think you are fully understanding what is being proposed.  The described
> solution DOES allow to connect subnets across NVE's.  It just doesn't use L2 to
> achieve that.

I do understand what you’re proposing - an L3VPN-class approach that offers
L3 service and uses host routes to sort things out.  OTOH, Frank was talking
about an L2 approach that offers L2 service where connecting the subnets
across NVEs would be at L2.

> It was all fine to use L2 in order to span subnets across two distribution switches,
> but here the requirements is to span the subnet across a set of arbitrary locations

That’s an interesting “requirement” ;-).  Please consider the repeated use
of the terms “data center” and DCVPN in the nvo3 WG charter:

http://datatracker.ietf.org/wg/nvo3/charter/

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xialiang (Frank)
Sent: Sunday, October 27, 2013 11:21 PM
To: Black, David; Yves Hertoghs (yhertogh)
Cc: [email-masked]
Subject: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Hi David,
Interest explanation~~

B.R.
Frank

Thanks,
--David

---

From: "Reith, Lothar" <[email-masked]>
Date: Mon, 28 Oct 2013 14:30:36 +0000

Yves,

you stated:

1.     Do you propose that the connectivity of Hosts aggregated into a pseudo-subnet have a “split horizon” connectivity, i.e. equivalent to being member of an ANTI-CUG group  which prohibits that they can talk to each other via layer 2 (only via the NVE/Gateway und thus via Layer 3)?

Do we need a discussion of the term subnet?

You seem to have the view, that a host is not attached via a subnet. My view is, that hosts at least in IP4 context are attached via /32 subnets, similar for IPv6. These subnets are from a mathematical point of view a singularity, where the IP-address of the subnet itself (the host route), the IP address of the local interface (at the NVE facing the host), and the IP address of the remote interface (at the host facing the NVE)  are all identical, because they are collapsed, and still, depending on the position and context, it is still clear what to do with an incoming packet at the host interface or NVE interface – just forward it to the other side of the collapsed subnet, and in routing context, it is clear that this address is referring to the collapsed subnet, because routing only deals with subnets, not with hosts, even though the term “host route” may mislead folks this way. In routing, the subnet role of the 3 roles of the IP-address is used.

I believe this multiple roles of the collapsed subnet “host route” is a key reason for the confusion, also the ambiguity regarding the definition what a subnet is.

But this is just my personal view, and may not be aligned with popular wisdom. However, popular wisdom is confusing, because the definition of subnet is a bit confusing. Is an aggregated subnet not a subnet? Is a hostroute not a subnet?

The term subnet seems to have a dual meaning:
– one view is that a subnet is just the address space.

-        Another view is that the subnet is the binding of an address space to an underlying layer 2 network (or to multiple underlying layer 2 networks in case of aggregation).

 I tend to prefer the second view in the current context. But many folks seem to be confused because for them the object “subnet” identifies just an address space, and in many contexts this interpretation is sufficient, but not in all.

So, I understand now, that what Yves is trying to say is, that the hosts are not in a conventional subnet, which is the binding between one contigous address range and one layer 2 domain, but in a non-conventional subnet, which is the binding between one contiguous address space and multiple layer 2 domains, in particular these multiple layer 2 domains are collapsed subnets which he refers to as host routes.
But maybe there are better well known terms for this situation, and I welcome anybody to speak up and correct me.

With reference to my earlier presented Neandertal abstraction of Layer3 and Layer2  communications: Such a collapsed subnet is essentially a fireplace with 2 people sitting on it, one NVE as gateway and one Host, and the name of the fireplace (misleadingly called hostroute) which may get advertised in routing comprises just a family name, no first name, where the family name is starting with some special digit say “1” to make it a speaking family name with the semantic that all family names starting with 1 refer to  single member families (fireplaces with just one seat besides the gateway seat). Therefore, the name of the single family member’s interface to the fireplace, the name of the fireplace, and the name of the NVE (Gateway) interface to the fireplace can be identical and are identical.

In closing, may I ask: Would a well known MAC-address for the NVE/gateway interface help?   This way, the hosts would not even need to ARP, and moving the VM would be much easier, as the MAC address of the gateway interface at the other side of  a single seat fireplace does not matter. It also does not matter if one talks to him via unicast, multicast or broadcast, only one frame will arrive anyway.

Lothar

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Yves Hertoghs (yhertogh)
Gesendet: Montag, 28. Oktober 2013 11:33
An: Black, David; Xialiang (C)
Cc: [email-masked]
Betreff: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

David,

Inline w [yves]

Yves

Thanks,
--David

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Mon, 28 Oct 2013 15:00:58 +0000

David,

inline

From: <Black>, David <[email-masked]<mailto:[email-masked]>>
Date: Monday 28 October 2013 16:00
To: "Xialiang (Frank)" <[email-masked]<mailto:[email-masked]>>, Yves Hertoghs <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

That’s a start ...

Fair enough that Frank's initial question perhaps was the latter rather than the former, so hence we are answering from our own point of view what Franks question actually was :-)

Not quite.  It doesn't offer a L3 service as such, as the service consumers (TS'es) which are part of the same subnet dont need to be aware that they are attached to a service that happens to do lookups at L3.  They arp for other hosts which are part of the 'subnet'.  Its the NVE that is replying, but with the MAC address of the real host.  Just to keep the host happy.  IP traffic between hosts is done via L3 lookups on the NVE, without looking at destination mac-addresses.  And in order to get 'off' the subnet, you send traffic to the default gateway, which is distributed across all NVE's.

http://datatracker.ietf.org/wg/nvo3/charter/
I dont see how the word DCVPN used in the charter is in conflict with my proposed solution, unless you think i mean the world with  'a set of arbitrary locations'.  Obviously i meant to say an arbitrary set of locations within the scope of the DCVPN.

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Xialiang (Frank)

Hi David,
Interest explanation~~

B.R.
Frank

Thanks,
--David
From:[email-masked]<mailto:[email-masked]> [[email-masked]<mailto:[email-masked]>] On Behalf Of Yves Hertoghs (yhertogh) [[email-masked]<mailto:[email-masked]>]

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Mon, 28 Oct 2013 15:21:51 +0000

Lothar,

inline

From: <Reith>, Lothar <[email-masked]<mailto:[email-masked]>>
Date: Monday 28 October 2013 16:30
To: Yves Hertoghs <[email-masked]<mailto:[email-masked]>>, "Black, David" <[email-masked]<mailto:[email-masked]>>, "Xialiang (C)" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: AW: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

Yves,

you stated:

Yes,as is stipulated in draft-hertoghs

I believe this multiple roles of the collapsed subnet “host route” is a key reason for the confusion, also the ambiguity regarding the definition what a subnet is.
You might be on to something here.  In IPv6 this is solved afaik, as you can tag a prefix as 'off-link', which effectively disables multicast RA's to that end system, but rather does unicast RA's.  That is why i keep on saying: the subnet is seen merely as an address-pool.  The TS'es 'think' they are part of a /24, but in reality they all end up as host-routes in the overlay.

Yes that would be good, as i might be confusing the hell out of people :-)

In closing, may I ask: Would a well known MAC-address for the NVE/gateway interface help?   This way, the hosts would not even need to ARP, and moving the VM would be much easier, as the MAC address of the gateway interface at the other side of  a single seat fireplace does not matter. It also does not matter if one talks to him via unicast, multicast or broadcast, only one frame will arrive anyway.
Yes it does, and it is also stipulated in draft-hertoghs.  They would still ARP (you can't stop them from ARPing I'm afraid).  So they would also ARP for the default gateway, and they always get the same response independent of their location !

Yves

Lothar

Von: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] Im Auftrag von Yves Hertoghs (yhertogh)

David,

Inline w [yves]

Yves

Thanks,
--David

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 29 Oct 2013 02:21:35 +0000

发件人: [email-masked] [mailto:[email-masked]] 代表 Yves Hertoghs (yhertogh)
发送时间: 2013年10月28日 18:33
收件人: Black, David; Xialiang (Frank)
抄送: [email-masked]
主题: Re: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

David,

Inline w [yves]

It’s a good rethink.
Xiaohu

Yves

Thanks,
--David

---

From: "Xialiang (Frank)" <[email-masked]>
Date: Tue, 29 Oct 2013 03:41:21 +0000

Hi Yves,
inline

> Sent: Monday, October 28, 2013 6:39 PM
> To: Xialiang (Frank)
> >> >different NVEs in one tenant, the Distributed Gateway should
> >> >implement the bridging function and routing function at the same time.
> >> >
> >> >But, there are still some points need to be studied and solved.
> >> >Below are some examples (NVE1 and NVE2 are all Distributed Gateway.
> >> >Hosts of subnet
> >> >Problems 1: If we configure the same gateway address (i.e. 1.1.1.1)
> >> >for the two subnets--1.1.1.0/24 attached to NVE1 and NVE2, and there
> >> >is intra-subnet connection between them, it will result in an
> >> >gateway address collision  in one subnet. If we configure different
> >> >gateway address for same subnet attached to different NVEs to solve
> >> >the address collision problem, it will bring a huge configuration
> >> >workload and waste of address space, and also it will limit the VM
> mobility.
> >>
> >> Why ? You are thinking that the default gateway has subnets
> >>configured,  feeding into its routing table.  It doesn't.  It learns
> >>all hosts as host-routes,  with optional aggregation in case the
> >>host-routing table can be aggregated  on bit boundaries.  Host might
> >>have mask/prefix-lengths configured, as that  is how they are
> >>operating today, but in reality these denote an address-pool  rather
> >>than a subnet/prefix.  For IP traffic these two parts of a subnet are
> >>not interconnected by a L2 link.  Surely there might be a L2 overlay
> >>spanning  both NVE's, but that overlay does not switch IP packets.
> >>
> >[Frank]: Maybe my description is not so clear. What I mean is every
> >host in the subnet configures its gateway for IP forwarding. For the
> >same subnet distributed in different NVEs, Same gateway address located
> >in different NVEs can exist. When these gateways in different NVEs send
> >subnet send VRRP messages in the subnet, the IP address collision
> >between gateways will be detected. How to solve this kind of problem?
> assuming a distributed subnets equals one L2 domain.  It doesn't need to be
> like this, as long as all hosts are treated as host-routes on the NVEs i.e. The
> NVE is not configured for a certain TS prefix on an interface.
> It dynamically installs the host-routes based on Tses registering (via VQP or
> some other means) and pushes them to the NVA.  The subnet is no longer a
> subnet, it is merely an address-pool.
> 
[Frank]: Your can treat all the hosts as routes. But gateways are still needed for all the hosts for inter-subnet connections. And these gateways are configured and located in NVEs and within same L2 domain of hosts. That's why I think the IP address collision between gateways exists. In addition, I think there can be several options of how to treat and handle hosts' traffic, just like the previous discussion in this thread. For example, we have Non-IP traffic and intra-subnet traffic. I think switching function is suitable for them.

> >> With an IP namespace and host-route learning there is at least a
> >>change of  aggregation, either at the NVE, or higher up in the data
> >>centre.  And actually  the aggregation function isn't really a part of
> >>the NVE, it can be a function of  the NVA to aggregate address-space,
> >>if a contiguous block of host-routes are  all pointing to the same
> >>NVE.
> >>
> >Actually, what I am concerned is the possibility to decrease or need
> >not to learn all host's address by some gateway-like method.
> >
> >> >
> >> This is orthogonal.  The NVE learns about hosts and their IP
> >> addresses within the NV instance. I dont see why multi tenancy is a
> problem ?
> >>
> >> >
> >> >I think the distributed gateway will solve those problems. But
> >> >before we define any functions of it, we need a problem statement
> >> >document for those scenarios firstly.
> >> I think the problem statement of how the solution should look like
> >>from a  user's perspective is already there.  A distributed gateway,
> >>with a unified
> biggest problem is that people dont fully understand it, and perhaps we need
> a separate draft to discuss this in detail.  Draft-hertoghs does explain it, but
> the LISP machinery around it might confuse people i.e the distributed, unified
> L2/L3 NVE is orthogonal to the NVE-NVA CP being used.
> 
[Frank] By my understanding, distributed gateways are new functions needed to support in NVE and can work together with NVE-NVA CP well.

> >> >True, but the unified L2/L3 solution gives you the choice on a per
> >> >VNI basis whether you want to do 'enhanced IP forwarding' or
> >> >'Ethernet based forwarding'.
> >> >
> >> >
> >> >
> >> >I also wouldn't call RDMA over Converged Ethernet a prime use case
> >> >for a multi tenant DC.  Typically this would be deployed standalone
> >> >and across  a few ethernet switches.
> >> >
> >> >
> >> >
> >> >Yves
> >> >
> >> >
> >> >
> >> >From:
> >> >>My main point is that , in the absence of non-IP traffic, there is
> >> >>a lot of value of doing IP based forwarding for all traffic, as
> >> >>this effectively  isolates the failure domains to the L2 domains
> >> >>underneath the NVE's.
> >> >>
> >> >>
> >> >>
> >> >>Yves
> >> >>
> >> >>
> >> >>
> >> >>From:
> >> >>>>MAC based forwarding will also not lead to network wide
> >> >>>>broadcast, if all the MAC mapping entries are generated by NVA.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>Since when is that a requirement for the NVA ?  An NVA will store
> >> >>>the mappings, not generate it.  The mapping generation can be done
> >> >>>Or are you saying that, independent of how the NVA knows about
> >> >>>them, tenant broadcast will only get flooded within the scope of the
> VNI?
> >> >>> L2 overlay creates a L2 failure domain, in the context of
> >> >>>broadcast and multicast.  Nothing you can do about it.
> >> >>>
> >> >>>[Lizhong] Sorry for the confusion, I mean the MAC mapping of
> >> >>>remote will be got from NVA, and MAC mapping of local will be got
> >> >>>from NVA or generated  by itself. I did not invent new things, and
> >> >>>all of that is described in draft-narten-nvo3-arch-01. The NVE
> >> >>>will reply the ARP from local TS if corresponding MAC mapping is
> >> >>>already existed. For EVPN, broadcast only happen when there is no
> >> >>>MAC-IP bindings locally, and that works well. There maybe some
> >> >>>benefits existed from the multicast view, and I should think about it.
> >> >>>
> >> >>>Thanks
> >> >>>Lizhong
> >> >>>
> >> >>>
> >> >>>
> >> >>>>>> 2) To address it from a solution point of view (e.g. are
> >> >>>>>>packets forwarded based on MAC or based on IP when source and
> >> >>>>>>dest are in the same subnet?).
> >> >>>>>>
> >> >>>>>>higher cost consequently. What’s more, should we even design
> >> >>>>>>aging for IP entry like MAC entry?
> >> >>>>>>
> >> >>>>>> I've now read through this whole thread  twice.  I see now
> >> >>>>>>that there may be some confusion between an L2/L3 service (what
> >> >>>>>>tenant's
> >> >>>>>>define a hybrid/combined/unified L2/L3 "service", but now I'm
> >> >>>>>>less sure. Just be be clear, I was not calling so much for a
> >> >>>>>>definition of the service, but that we need:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> 2) To address it from a solution point of view (e.g. are
> >> >>>>>>packets forwarded based on MAC or based on IP when source and
> >> >>>>>>dest are in the same subnet?).
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> As I stated before, I believe the control plane requirements
> >> >>>>>>will be different for a combined service than for a pure L2 or
> >> >>>>>>pure L3 service, and that we if don't start specifically
> >> >>>>>>calling out the control plane requirements for a combined
> >> >>>>>>service (from a
> >> >>>>>> 主题: Re: [nvo3] Distributed Gateways [was Re: NVO3
> Architecture
> >> >>>>>>document]
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Hi Lizong,
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>mappings.  There is no expectation that two TS are on the same
> >> >>>>>>L2 segment, and the TS would always ARP for a router to reach
> >> >>>>>>another TS.  When a frame is delivered between the two TS, both
> >> >>>>>>the
> >> >>>> source and dest MAC will have been rewritten.  TTL will have
> >> >>>>been decremented at least once.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>within a subnet.  TS ARP for the other TS on its subnet instead
> >> >>>>>>of the router.  The source and dest MAC are not rewritten.  TTL
> >> >>>>>>is not decremented.  From the perspective of the two TS on the
> >> >>>>>>same
> >> >>>>within the same L2 VN vs between different L2 VNs.  For example,
> >> >>>>the NVE would need to respond to ARP, not with its own MAC
> >> >>>>address,  but with the remote TSs MAC address if the TS are in the
> same L2 VN.
> >> >>>>>> If so, what’s the difference between hybrid L2/L3 service and
> >> >>>>>>IRB (Integrated Routing and Bridging)? In addition, I believe
> >> >>>>>>the hybrid
> >> >>>> for intra-subnet traffic and routing for inter-subnet traffic
> >> >>>>while the latter is about routing for (both intra-subnet and
> >> >>>>inter-subnet) IP traffic while bridging for non-IP traffic.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Best regards,
> >> >>>>>>
> >> >>>>>> Xiaohu
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> So, I still think NVEs will need some additional information
> >> >>>>>>for a hybrid service, and I think we will need to describe the
> >> >>>>>> > describes what you are referring to , but i am not sure if
> >> >>>>>> >it
> >> >>>>>>needs to
> >> >>>>>> be different for L2 vs L3 vs a combined L2/L3 service. For an
> >> >>>>>>L2 VN, the  VN needs to be identified (e.g. with a Name or ID).
> >> >>>>>>For an L3 VN, it  similarly needs to be identified. For a
> >> >>>>>>combined
> >> >>>>>>L2 VN to one on another L2 VN, it will need  to know not just
> >> >>>>>>the mapping of inner to outer address, but the mapping of
> >> >>>>>>inner L3 address to destination L2 VN and MAC address (so it
> >> >>>>>>can rewrite the MAC and the L2 VN).
> >> >>>>>>
> >> >>>>>> The behavior you described for L3 forwarding between two L2
> >> >>>>>>VPN is more of  a standard L3 forwarding function, looking up
> >> >>>>>>with destination IP address,  then getting the destination MAC,
> >> >>>>>>source MAC, VLAN and NVO3 tunnel from  the table. The MAC
> and
> >> >>>>>>VLAN
> >> are
> >> >>>>>> >On 21/10/13 10:28, "Larry Kreeger (kreeger)"
> >> >>>>>> ><[email-masked]>
> >> >>>>>>wrote:
> >> >>>>>> >
> >> >>>>>> >>Lucy,
> >> >>>>>> >>
> >> >>>>>> >>See inline with LK2>.
> >> >>>>>> >>
> >> >>>>>> >> - Larry
> >> >>>>>> >>
> >> >>>>>> >>LK2> Yes, if the group is to seriously address a hybrid
> >> >>>>>> >>LK2> L2/L3
> >> >>>>>>service,
> >> >>>>>> >>LK2> I
> >> >>>>>> >>>>Current architecture clear states NVE and NVA roles, i.e.
> >> >>>>>> >>>>NVE performs forwarding, and NVA performs routing.
> >> >>>>>> >>>
> >> >>>>>> >>>I'm not sure what you mean by routing. Are you referring to
> >> >>>>>> >>>L3
> >> >>>>>> service?
> >> >>>>>> >>>If so, I don't agree.
> >> >>>>>> >>>the mapping in which, if the location is the same as
> >> >>>>>> >>>itself, it translates to a tenant/VAP mapping; If not, it
> >> >>>>>> >>>the forwarding accordingly. This is also my view about the
> >> >>>>>> >>>SDN
> >> >>>>>>based
> >> >>>>>> >>>architecture.
> >> >>>>>> >>>
> >> >>>>>> >>LK2> OK, if it isn't specific to L2/L3 service, than I'd
> >> >>>>>> >>LK2> rather
> >> >>>>>>leave
> >> >>>>>> >>LK2> it
> >> >>>>>> >>>> If NVA only supports simple inner to outer mappings, how
> >> >>>>>> >>>> can
> >> >>>>>>NVE
> >> >>>>>> >>>service. Local forwarding doesnšt need mappings, the local
> >> >>>>>> >>>NVE
> >> >>>>>>knows
> >> >>>>>> >>>[Lucy] does pure L3 service means an L3VPN w/o any policy?
> >> >>>>>> >>>For
> >> >>>>>>an L3
> >> >>>>>> >>pretty sure you need more than just inner to outer mappings
> >> >>>>>> >>to implement policy.
> >> >>>>>> >>
> >> >>>>>> >>>tenant mobility. IMO: current architecture draft is vague
> >> >>>>>> >>>in or
> >> >>>>>>lacks
> >> >>>>>> >>>of describing this but it is important to architect this
> >> >>>>>> >>>when
> >> >>>>>>having
> >> >>>>>> >>LK2> I'm not sure if I agree with you about the NVE not
> >> >>>>>> >>LK2> being the entity
> >> >>>>>> >>to enforce policy, but I think it is something better
> >> >>>>>> >>discussed
> >> >>>>>>in a
> >> >>>>>> >>>>Support of a distributed L3 gateway function between L2
> >> >>>>>> >>>>VNs is a significant increase in scope of the NVA, and the
> >> >>>>>> >>>>NVE to NVA
> >> >>>>>>protocol.
> >> >>>>>> >>>>would now be adding whatever mechanisms are needed to
> the
> >> >>>>>>protocols.
> >> >>>>>> >>>>destination MAC address (first L2, then potentially L3).
> >> >>>>>> >>>>We will also need to distribute two different VN
> >> >>>>>> >>>>identifiers (one for
> >> >>>>>>L2 and
> >> >>>>>> >>>>one for L3), and somehow convey the containment
> >> >>>>>> >>>>relationship
> >> >>>>>>between
> >> >>>>>> >>>>sure the WG agrees to this since I feel it is a
> >> >>>>>> >>>>significant change/increase in scope from my perspective.
> >> >>>>>> >>>>
> >> >>>>>> >>>>Thanks, Larry
> >> >>>>>> >>>>
> >> >>>>>> >>>>
> >> >>>>>> >>>>
> >> >>>>>> >>>>On 10/18/13 2:52 PM, "Thomas Narten"
> <[email-masked]>
> >> wrote:
> >> >>>>>> >>>>
> >> >>>>>> >>>>>Hi Lucy.
> >> >>>>>> >>>>>
> >> >>>>>> >>>>>> gateway where the policy can be enforced. Furthermore,
> >> >>>>>> >>>>>> it is possible to
> >> >>>>>>implement
> >> >>>>>> >>>>>> centralized or distributed overlay gateway. The latter
> >> >>>>>>requests
> >> >>>>>> >>>>> <section title="Distributed Gateways"> <t> The relaying
> >> >>>>>> >>>>> of traffic from one VN to another deserves special
> >> >>>>>> >>>>> performing this function. If such gateways are
> >> >>>>>> >>>>> centralized, traffic between TSes on different VNs can
> >> >>>>>> >>>>> take suboptimal paths, i.e., triangular routing results
> >> >>>>>> >>>>> in paths that always traverse the gateway. As an
> >> >>>>>> >>>>> optimization, individual NVEs can be part of a
> >> >>>>>> >>>>> distributed gateway that performs such relaying,
> >> >>>>>> >>>>> reducing or completely eliminating triangular routing.
> >> >>>>>> >>>>> In a distributed gateway, each ingress NVE can perform
> >> >>>>>> >>>>> such relaying activity directly, so long as it has
> >> >>>>>> >>>>> access to the policy information needed to determine
> >> >>>>>> >>>>> The NVO3 architecture should [must? or just say it
> >> >>>>>> >>>>> does?] support distributed gateways. Such support
> >> >>>>>> >>>>> requires that
> >> >>>>>> >>>>> what type of cross-VN communication is allowed so that
> >> >>>>>> >>>>> NVEs acting as distributed gateways can tunnel traffic
> >> >>>>>> >>>>> from one VN to another as appropriate.
> >> >>>>>> >>>>> </t>
> >> >>>>>> >>>>> </section>
> >> >>>>>> >>>>>
> >> >>>>>> >>>>>Thoughts?
> >> >>>>>> >>>>>
> >> >>>>>> >>>>>Thomas
> >> >>>>>> >>>>>

---

From: "Xialiang (Frank)" <[email-masked]>
Date: Tue, 29 Oct 2013 06:02:18 +0000

Hi David
Please see inline.

From: Black, David [mailto:[email-masked]]
Sent: Monday, October 28, 2013 10:01 PM
To: Xialiang (Frank); Yves Hertoghs (yhertogh)
Cc: [email-masked]
Subject: RE: [nvo3] Distributed Gateways [was Re: NVO3 Architecture document]

That’s a start ...

[Frank] I know your mean is that we can have several virtual routers for load-sharing. But it also needs several different virtual routers’ address for a L2 domain. Which goes back to the impact I have mentioned in the previous mail, I copy it below:

“If we configure different gateway address for same subnet attached to different NVEs to solve the address collision problem, it will bring a huge configuration workload and waste of address space, and also it will limit the VM mobility.”

So, I think whether method we choose, there are always some problems need to be solved.

Thanks,
--David

Hi David,
Interest explanation~~

B.R.
Frank

Thanks,
--David

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Tue, 29 Oct 2013 08:17:05 +0000

On 29/10/13 04:41, "Xialiang (Frank)" <[email-masked]> wrote:

>[Frank]: Your can treat all the hosts as routes. But gateways are still
>needed for all the hosts for inter-subnet connections. And these gateways
>are configured and located in NVEs and within same L2 domain of hosts.
>That's why I think the IP address collision between gateways exists. In
>addition, I think there can be several options of how to treat and handle
>hosts' traffic, just like the previous discussion in this thread. For
>example, we have Non-IP traffic and intra-subnet traffic. I think
>switching function is suitable for them.

Frank, why do you keep on insisting that there is a L2 overlay connecting
the NVE's for IP traffic? In my proposal there isn't.  Non-IP traffic is
switches across the L2 overlay, all IP traffic (including intra-subnet) is
routed across the L3 overlay.  NVE's will respond to all ARP's, but with
the 'right' MAC address.  Intersubnet traffic is send to the default
gateway, which has the same identity across the entire DC.  Obviously the
default gateway IP address is not part of the L3 overlay routing table.

Yves
>>need
>[Frank] By my understanding, distributed gateways are new functions
>needed to support in NVE and can work together with NVE-NVA CP well.
>
>> Yves
>> >
>> >> >
>> >> >Your comments?
>> >> >
>> >> >B.R.
>> >> >Frank
>> >> >
>> >> >