Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

From: "Bocci, Matthew (Matthew)" <[email-masked]>
Date: Wed, 13 Nov 2013 13:57:45 +0000

This email begins a two week poll to help the chairs judge if there is consensus  to adopt draft-narten-nvo3-arch-01.txt as an NVO3 working group draft.

Please respond to this email on the list with 'support' or 'do not support'.

Please also send any comments on the draft to the NVO3 list.

Please consider whether this draft takes the right basic approach, and is a good basis for the work going forward (and potential future rechartering). It does not have to be perfect at this stage.

Coincidentally, we are also polling for knowledge of any IPR that applies to this draft, to ensure that IPR has been disclosed in compliance with IETF IPR rules (see RFCs 3979, 4879, 3669 and 5378 for more details).

If you are listed as a document author or contributor please respond to this email whether or not you are aware of any relevant IPR. The draft will not be adopted until a response has been received from each author and contributor.

If you are on the NVO3 WG email list but are not listed as an author or contributor, then please explicitly respond only if you are aware of any IPR that has not yet been disclosed in conformance with IETF rules.

This poll closes on Friday 29th November 2013.

Regards

Matthew and Benson

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Wed, 13 Nov 2013 17:37:54 +0000

Support as a co-author. I am not aware of any relevant IPR.

Marc

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Wednesday, November 13, 2013 2:58 PM
To: [email-masked]
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 13 Nov 2013 09:56:55 -0800

> Please respond to this email on the list with 'support' or 'do not support'.

Support. 

> Please consider whether this draft takes the right basic approach, and is a good basis for the work going forward (and potential future rechartering). It does not have to be

The document is heading in the right direction IMO. 

Dino

---

From: Lucy yong <[email-masked]>
Date: Wed, 13 Nov 2013 18:12:11 +0000

Support!
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Wednesday, November 13, 2013 7:58 AM
To: [email-masked]
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: David Meyer <[email-masked]>
Date: Wed, 13 Nov 2013 10:32:22 -0800

Support this, and yes, looks to be moving in the right direction. --dmm

> This email begins a two week poll to help the chairs judge if there is
> consensus  to adopt draft-narten-nvo3-arch-01.txt as an NVO3 working group
> draft.
>
>
>
> Please consider whether this draft takes the right basic approach, and is a
> good basis for the work going forward (and potential future rechartering).
> It does not have to be perfect at this stage.
>
>
>
> Coincidentally, we are also polling for knowledge of any IPR that applies to
> this draft, to ensure that IPR has been disclosed in compliance with IETF
> IPR rules (see RFCs 3979, 4879, 3669 and 5378 for more details).
>
>
>
> If you are listed as a document author or contributor please respond to this
> email whether or not you are aware of any relevant IPR. The draft will not
> be adopted until a response has been received from each author and
> contributor.
>
>
>
> If you are on the NVO3 WG email list but are not listed as an author or
> contributor, then please explicitly respond only if you are aware of any IPR
> that has not yet been disclosed in conformance with IETF rules.
>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: ramki Krishnan <[email-masked]>
Date: Wed, 13 Nov 2013 10:35:43 -0800

Support.

Thanks,
Ramki

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Wednesday, November 13, 2013 5:58 AM
To: [email-masked]
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: Sharon <[email-masked]>
Date: Wed, 13 Nov 2013 12:17:58 -0800

Support and conquer with the lispers experience..
NVE-NVAs underlay-peering one another for both 
transport and lookup is both elegant and scalable.

--szb

> On Nov 13, 2013, at 5:57, "Bocci, Matthew (Matthew)" <[email-masked]> wrote:
>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 13 Nov 2013 21:45:18 +0000

Support (as a co-author). I am not aware of an IPR related to this draft.

 - Larry

From: <Bocci>, "Matthew (Matthew)" <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, November 13, 2013 6:57 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: Jon Hudson <[email-masked]>
Date: Wed, 13 Nov 2013 23:27:27 -0800

Support (as co-author). No IPR related to this draft that I am aware of.

J

> On Nov 13, 2013, at 1:45 PM, "Larry Kreeger (kreeger)" <[email-masked]> wrote:
> 
> From: <Bocci>, "Matthew (Matthew)" <[email-masked]>
> To: "[email-masked]" <[email-masked]>

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 14 Nov 2013 08:57:32 +0000

Hi all,

In the current arch draft, there is no mention of NVE-NVE protocol. Does it mean that there is no need for direct exchange of VN and/or address mapping information between NVEs? If so, Does it mean that the control plane mechanisms used by TRILL or SPB which depend on the NVE-NVE interaction are not suitable for multi-tenant data center networks anymore, leaving aside whether the underlay is IP or not.

IMHO, the NVE-NVE protocol is still useful in some small and medium sized multi-tenant data center networks. AFAIK, most tenants within public cloud data centers are small and medium sized enterprises which usually don’t need a lot of VMs. That means the number of NVEs for most VNs would not be very large especially in the case where the NVE is deployed at physical switches, rather than hypervisors/servers. In this case, the VN membership can be discovered via IGP flooding and the address mapping information of a given VN could be directly exchanged among NVEs of that VN, without a need for a dedicated NVA.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Bocci, Matthew (Matthew)
发送时间: 2013年11月13日 21:58
收件人: [email-masked]
主题: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: "Black, David" <[email-masked]>
Date: Thu, 14 Nov 2013 08:32:40 -0500

Support (as a co-author).  I am not aware of any IPR that applies to this draft.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bocci, Matthew (Matthew)
Sent: Wednesday, November 13, 2013 8:58 AM
To: [email-masked]
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: "Shah, Himanshu" <[email-masked]>
Date: Thu, 14 Nov 2013 09:40:10 -0500

Support.
Yes - it is taking the right approach and is headed in right direction, IMO.
/himanshu

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 15:59:18 +0000

I share my view.

The current architecture document is more focusing on NVE-NVA interface and assumes that NVA is able to obtain all VN and/or address mapping information’s that an NVE needs. That does implicitly indicate that no control plane protocol is needed between NVEs. (NVE-NVE data plane protocol is still needed).  From the architecture perspective, if it allows the control plane protocol exist both between NVE-NVA and NVE-NVE, it may lead very complex solution and many operation issue; it has to resolve which information NVE should trust or use, i.e. from NVA or from NVE.

Weather this means excluding TRILL or SPB, it is debatable. The current charter clearly states that NVO3 targets network virtualization over IP network. Beyond this point, TRILL has directory based solution, which fits into NVE-NVA architecture.  SPB also have SPB-EVPN solution that also aligns with NVE-NVA architecture.

IMO: NVE-NVA based control plane architecture and NVE-NVE based control plane architecture are both possible for NVO3, but not the combined architecture. As you said, it is true that NVE-NVE based architecture is useful in some small applications.  Since TRILL and SBP already address it, NVO3 should only focus on the NVE-NVA based architecture. One of NVO3 goal is the scalability.

Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, November 14, 2013 2:58 AM
To: Bocci, Matthew (Matthew); [email-masked]
Subject: [nvo3] 答复: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi all,

Best regards,
Xiaohu

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Bocci, Matthew (Matthew)
发送时间: 2013年11月13日 21:58
收件人: [email-masked]<mailto:[email-masked]>
主题: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Thu, 14 Nov 2013 17:07:30 +0000

Support.  

I believe this document takes the right basic approach.

-Darrel

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 14 Nov 2013 18:15:11 +0000

I disagree with the need for an NVE to NVE control plane.  That doesn't mean that you can't colocate a portion of the distributed NVA with every NVE, which is the model that e.g. BGP/L3VPN or ISIS/TRILL uses.

Yves

From: Lucy yong <[email-masked]<mailto:[email-masked]>>
Date: Thursday 14 November 2013 16:59
To: Xuxiaohu <[email-masked]<mailto:[email-masked]>>, Matthew Bocci <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

I share my view.

Lucy

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, November 14, 2013 2:58 AM
To: Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>
Subject: [nvo3] 答复: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi all,

Best regards,
Xiaohu

发件人:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表Bocci, Matthew (Matthew)
发送时间: 2013年11月13日 21:58
收件人:[email-masked]<mailto:[email-masked]>
主题: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 18:19:38 +0000

Yves,

From: [email-masked] [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Thursday, November 14, 2013 12:15 PM
To: Lucy yong; Xuxiaohu; Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

I disagree with the need for an NVE to NVE control plane.
[Lucy]  do you think we need NVE-NVE control plane? I don’t think this is what you mean from the following statement.
 That doesn't mean that you can't colocate a portion of the distributed NVA with every NVE, which is the model that e.g. BGP/L3VPN or ISIS/TRILL uses.
[Lucy] Agreed. NVA can collocate w/ NVE. (partially or entire).

Lucy

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 14 Nov 2013 18:23:26 +0000

inline

From: Lucy yong <[email-masked]<mailto:[email-masked]>>
Date: Thursday 14 November 2013 19:19
To: Cisco Hertoghs <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, Matthew Bocci <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Yves,

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Thursday, November 14, 2013 12:15 PM
To: Lucy yong; Xuxiaohu; Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

I disagree with the need for an NVE to NVE control plane.
[Lucy]  do you think we need NVE-NVE control plane? I don’t think this is what you mean from the following statement.
No we dont need an NVE to NVE control plane.
 That doesn't mean that you can't colocate a portion of the distributed NVA with every NVE, which is the model that e.g. BGP/L3VPN or ISIS/TRILL uses.
[Lucy] Agreed. NVA can collocate w/ NVE. (partially or entire).
And as a result there is only a need for a control plane between the NVE function and the NVA function.

Yves

Lucy

Yves

I share my view.

Lucy

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, November 14, 2013 2:58 AM
To: Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>
Subject: [nvo3] 答复: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 18:27:45 +0000

Yves, We are on the same boat.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Yves Hertoghs (yhertogh)
Sent: Thursday, November 14, 2013 12:23 PM
To: Lucy yong; Xuxiaohu; [email-masked]
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

inline

Yves,

Yves

Lucy

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: "Fedyk, Don" <[email-masked]>
Date: Thu, 14 Nov 2013 18:46:36 +0000

Hi

Support.  This draft takes the right basic approach IMHO.

Regards,
Don

Regards

Matthew and Benson

---

From: "Fedyk, Don" <[email-masked]>
Date: Thu, 14 Nov 2013 19:33:17 +0000

Hi Xiaohu

I don’t see the architecture precluding the use of IS-IS for SPB or Trill within the data center.
If you have used SPBM in the data center you could use draft-ietf-l2vpn-spbm-evpn as part of the NVA for L2, for example.

Cheers,
Don
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, November 14, 2013 3:58 AM
To: Bocci, Matthew (Matthew); [email-masked]
Subject: [nvo3] 答复: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 14 Nov 2013 22:24:32 +0000

Support WG adoption if the following comments can be addressed:

Issues with the current writing of Distributed Gateway (Section 5.4):
As described in Section 5.3, a Gateway does many things. However, I don't think that a NVE, if taking on the responsibility of a distributed Gateway, will do all the things that a conventional Gateway does (or the list of items mentioned in the Section 5.3).

First, it might be too much to ask a NVE based gateway (especially Hypervisor based NVE) to

*        relay traffic off the virtual networks, i.e. perform gateway function to reach destinations outside the local VNs,

*        Interface with external VPN domain (i.e. out of Virtual Networks),

*        perform NAT on the source virtual addresses, or

*        relay traffic to a VN that doesn't have any hosts attached to the NVE  (it is debatable if a NVE based distributed Gateway should take this responsibility)
The real functions performed by NVE, if designated as "distributed gateway", is more like Inter-VN relay (instead of full blown Gateway function).

Second, when host "a" in VN-1 sends traffic to "b" in VN-2, the data packet's Ethernet header has "MAC-DA = Gateway-MAC & MAC-SA= a-MAC & VLAN= VN-1".  Most implementations (Microsoft Window 8 and VM NSX) allocate a "fake MAC" for all the NVE based distributed gateways to share, so that host "a" can use the same Gateway-MAC when moved to another NVE. This again is different from the conventional gateways.

Third, the issue of conventional gateway (i.e. a->b traffic to be routed at gateway even if "a" & "b" are attached to the same NVE) is traffic "hairpinning", instead of triangular routing.

Therefore, I suggest rewriting Section 5.4 as below:

5.4 Distributed Gateway (Re-write)
The relaying of traffic from one VN to another, especially when Source and Destination are attached to the same NVE, deserves special consideration. With conventional Gateways, the traffic between TSes on different VNs has to be traversed to the gateway, even if the Source and Destination are attached to the save NVE, which causes traffic hairpinning and wasted bandwidth.

As an optimization, it is desirable for individual NVEs to take over the inter-VN relay responsibilities that are traditionally done by conventional gateways to reduce or eliminate hairpinning issue. In order for NVEs to perform inter-VN relay, the NVEs must have access to the policy information needed to determine whether inter-VN communication is allowed. Those inter-VN communication policies are most likely to come from NVA.

However, it is not practical for NVEs to take over all functions of conventional gateways.
In particular, it might be too much to ask a NVE based gateway (especially Hypervisor based NVE) to

*        relay traffic off the virtual networks, i.e. perform gateway function to reach destinations outside the local VNs,

*        perform NAT on the source virtual addresses, or

*        relay traffic to a VN that doesn't have any hosts attached to the NVE.
The NVEs that are capable of performing inter-VN replaying are called "Distributed Gateway" in this document. (Note: Inter-VN relaying capable NVE is a more accurate term).

The NVO3 architecture should support distributed gateways, at least allowing some NVEs, if not all, supporting the inter-VN relaying function, especially when both source and destination are attached to the same NVE.  Such support requires the NVO3 control protocols include mechanisms for the maintenance and distribution of the inter-VN policy information to the NVEs that are capable of performing inter-VN communications.

----------------------------------------
Thanks for considering my suggested text.

Linda Dunbar

Regards

Matthew and Benson

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 14 Nov 2013 15:07:06 -0800

Hi folks,

In my view the distributed Gateway cannot do a lot of the centralized
functions required.

We need a single place to terminate/ start IPsec connections - the most
common way to connect to a cloud. A lot of the control plane to talk to the
external world also needs to be centralized.

Thanks,
Vishwas

On Thu, Nov 14, 2013 at 2:24 PM, Linda Dunbar <[email-masked]>wrote:

> As described in Section 5.3, a Gateway does many things. However, I don’t
> think that a NVE, if taking on the responsibility of a distributed Gateway,
> will do all the things that a conventional Gateway does (or the list of
> items mentioned in the Section 5.3).
>
>
>
> ·        relay traffic off the virtual networks, i.e. perform gateway
> function to reach destinations outside the local VNs,
>
> ·        Interface with external VPN domain (i.e. out of Virtual
> Networks),
>
> ·        perform NAT on the source virtual addresses, or
>
> ·        relay traffic to a VN that doesn’t have any hosts attached to
> the NVE  (it is debatable if a NVE based distributed Gateway should take
> this responsibility)
>
> The real functions performed by NVE, if designated as “distributed
> gateway”, is more like *Inter-VN relay *(instead of full blown Gateway
> function).
>
>
>
> Second, when host “a” in VN-1 sends traffic to “b” in VN-2, the data
> packet’s Ethernet header has “MAC-DA = Gateway-MAC & MAC-SA= a-MAC & VLAN=
> VN-1”.  Most implementations (Microsoft Window 8 and VM NSX) allocate a
> “fake MAC” for all the NVE based distributed gateways to share, so that
> host “a” can use the same Gateway-MAC when moved to another NVE. This again
> is different from the conventional gateways.
>
>
>
> Third, the issue of conventional gateway (i.e. a->b traffic to be routed
> at gateway even if “a” & “b” are attached to the same NVE) is traffic
> “hairpinning”, instead of triangular routing.
>
>
>
> *5.4 Distributed Gateway (Re-write)*
>
> The relaying of traffic from one VN to another, especially when Source and
> Destination are attached to the same NVE, deserves special consideration.
> With conventional Gateways, the traffic between TSes on different VNs has
> to be traversed to the gateway, even if the Source and Destination are
> attached to the save NVE, which causes traffic hairpinning and wasted
> bandwidth.
>
>
>
> As an optimization, it is desirable for individual NVEs to take over the
> inter-VN relay responsibilities that are traditionally done by conventional
> gateways to reduce or eliminate hairpinning issue. In order for NVEs to
> perform inter-VN relay, the NVEs must have access to the policy information
> needed to determine whether inter-VN communication is allowed. Those
> inter-VN communication policies are most likely to come from NVA.
>
>
>
> ·        relay traffic off the virtual networks, i.e. perform gateway
> function to reach destinations outside the local VNs,
>
> ·        perform NAT on the source virtual addresses, or
>
> ·        relay traffic to a VN that doesn’t have any hosts attached to
> the NVE.
>
> The NVEs that are capable of performing inter-VN replaying are called
> “Distributed Gateway” in this document. (Note: *Inter-VN relaying capable
> NVE* is a more accurate term).
>
>
>
> The NVO3 architecture should support distributed gateways, at least
> allowing some NVEs, if not all, supporting the inter-VN relaying function,
> especially when both source and destination are attached to the same NVE.
> Such support requires the NVO3 control protocols include mechanisms for the
> maintenance and distribution of the inter-VN policy information to the NVEs
> that are capable of performing inter-VN communications.
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Bocci, Matthew (Matthew)
>
> *Sent:* Wednesday, November 13, 2013 7:58 AM
> *To:* [email-masked]
> *Subject:* [nvo3] Poll for WG adoption and IPR check for
> draft-narten-nvo3-arch-01.txt
>
>
>
> Please consider whether this draft takes the right basic approach, and is
> a good basis for the work going forward (and potential future
> rechartering). It does not have to be perfect at this stage.
>
>
>
> Coincidentally, we are also polling for knowledge of any IPR that applies
> to this draft, to ensure that IPR has been disclosed in compliance with
> IETF IPR rules (see RFCs 3979, 4879, 3669 and 5378 for more details).
>
>
>
> If you are listed as a document author or contributor please respond to
> this email whether or not you are aware of any relevant IPR. The draft will
> not be adopted until a response has been received from each author and
> contributor.
>
>
>
> contributor, then please explicitly respond only if you are aware of any
> IPR that has not yet been disclosed in conformance with IETF rules.
>
>
>

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 23:16:31 +0000

Hi Vishwas,

I do not disagree with what you said. A lot of applications do need a centralized gateway, especially a VN interconnecting with WAN and Internet. The nvo3-nve draft gives the recommendations on the choice between a gateway and distributed gateway. VN within DC can benefit a lot from a distributed gateway. Some vender already implement these.

Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Vishwas Manral
Sent: Thursday, November 14, 2013 5:07 PM
To: Linda Dunbar
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi folks,

In my view the distributed Gateway cannot do a lot of the centralized functions required.

We need a single place to terminate/ start IPsec connections - the most common way to connect to a cloud. A lot of the control plane to talk to the external world also needs to be centralized.

Thanks,
Vishwas
On Thu, Nov 14, 2013 at 2:24 PM, Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:
Support WG adoption if the following comments can be addressed:

Linda Dunbar

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Bocci, Matthew (Matthew)

Sent: Wednesday, November 13, 2013 7:58 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 14 Nov 2013 15:40:08 -0800

Hi Lucy,

I see the VPC as a branch of a bigger Infrastructure (in many cases), and
the gateway that connects the same as an edge device.

We need to connect this VPC securely - and IPsec as the technology when
connecting over the internet. I dont know if you can distribute IPsec
functionality. I know you need tecnologies like firewalls etc too, which I
know can be centrally managed yet be enforced in a distributed manner (not
the general way to do it however). Same with the control plane that
connects to the peers in the Enterprise (we need a single control plane).

I guess I am missing the point.

Thanks,
Vishwas

On Thu, Nov 14, 2013 at 3:16 PM, Lucy yong <[email-masked]> wrote:

>  Hi Vishwas,
>
>
>
> I do not disagree with what you said. A lot of applications do need a
> centralized gateway, especially a VN interconnecting with WAN and Internet.
> The nvo3-nve draft gives the recommendations on the choice between a
> gateway and distributed gateway. VN within DC can benefit a lot from a
> distributed gateway. Some vender already implement these.
>
>
>
> Lucy
>
>
>
> Of *Vishwas Manral
> *Sent:* Thursday, November 14, 2013 5:07 PM
> *To:* Linda Dunbar
> *Cc:* Bocci, Matthew (Matthew); [email-masked]
> *Subject:* Re: [nvo3] Poll for WG adoption and IPR check for

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 14 Nov 2013 15:41:54 -0800

Infact even between tiers of an Application you need firewall etc. So
direct access is allowed to the Web tier but not to the mid tier etc etc.

-Vishwas

On Thu, Nov 14, 2013 at 3:40 PM, Vishwas Manral <[email-masked]>wrote:

>> The relaying of traffic from one VN to another, especially when Source
>> and Destination are attached to the same NVE, deserves special
>> consideration. With conventional Gateways, the traffic between TSes on
>> different VNs has to be traversed to the gateway, even if the Source and
>> Destination are attached to the save NVE, which causes traffic hairpinning
>> and wasted bandwidth.
>>
>>
>>
>> “Distributed Gateway” in this document. (Note: *Inter-VN relaying
>> capable NVE* is a more accurate term).
>>
>>
>>

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 23:45:10 +0000

Hi Vishwas,

From: [email-masked] [mailto:[email-masked]] On Behalf Of Vishwas Manral
Sent: Thursday, November 14, 2013 5:40 PM
To: Lucy yong
Cc: Bocci, Matthew (Matthew); [email-masked]; Linda Dunbar
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi Lucy,

I see the VPC as a branch of a bigger Infrastructure (in many cases), and the gateway that connects the same as an edge device.
[Lucy] VPN is  the case for a VN to interconnect with external network (enterprise) via WAN. For such case, a gateway is more proper. The architecture give you option to use centralized gateway or distributed gateway depending the applications. There are a lot of internal VNs in DC, when they need interconnect, distributed GW make sense. As Linda said, both VMare and Microsoft already implement these. If not use case, why do you do that?

Lucy

We need to connect this VPC securely - and IPsec as the technology when connecting over the internet. I dont know if you can distribute IPsec functionality. I know you need tecnologies like firewalls etc too, which I know can be centrally managed yet be enforced in a distributed manner (not the general way to do it however). Same with the control plane that connects to the peers in the Enterprise (we need a single control plane).

Thanks,
Vishwas
On Thu, Nov 14, 2013 at 3:16 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
Hi Vishwas,

Lucy

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Vishwas Manral
Sent: Thursday, November 14, 2013 5:07 PM
To: Linda Dunbar
Cc: Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi folks,

Thanks,
Vishwas

Linda Dunbar

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Thu, 14 Nov 2013 23:46:47 +0000

Agree. NVX implement the distributed firewall too. It is also described in nvo3-nve draft. Like to see your feedback on that draft.
Lucy

From: Vishwas Manral [mailto:[email-masked]]
Sent: Thursday, November 14, 2013 5:42 PM
To: Lucy yong
Cc: Linda Dunbar; Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Infact even between tiers of an Application you need firewall etc. So direct access is allowed to the Web tier but not to the mid tier etc etc.

-Vishwas
On Thu, Nov 14, 2013 at 3:40 PM, Vishwas Manral <[email-masked]<mailto:[email-masked]>> wrote:
Hi Lucy,

Thanks,
Vishwas

Lucy

Hi folks,

Thanks,
Vishwas

Linda Dunbar

Regards

Matthew and Benson

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 14 Nov 2013 15:52:25 -0800

Hi Lucy/ Linda,

What I am saying is that you will need a centralized gateway even in cases
you do some of the functionality that is distributed.

We can talk of advantages of a disrtibuted + centralized model that are
well known. Ta distributed Layer-3 gateway allows traffic within the same
tier can be routed directly.

Thanks,
Vishwas

On Thu, Nov 14, 2013 at 3:45 PM, Lucy yong <[email-masked]> wrote:

> *Sent:* Thursday, November 14, 2013 5:40 PM
> *To:* Lucy yong
> *Cc:* Bocci, Matthew (Matthew); [email-masked]; Linda Dunbar
>
> *[Lucy] VPN is  the case for a VN to interconnect with external network
> (enterprise) via WAN. For such case, a gateway is more proper. The
> architecture give you option to use centralized gateway or distributed
> gateway depending the applications. There are a lot of internal VNs in DC,
> when they need interconnect, distributed GW make sense. As Linda said, both
> VMare and Microsoft already implement these. If not use case, why do you do
> that?*
>
>
>
> *Lucy*
>
>
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 15 Nov 2013 00:30:14 +0000

WARNING: contains banned part

Subject: RE: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt
From: Lucy yong <[email-masked]>
To: Vishwas Manral <[email-masked]>
Cc: "Bocci, Matthew (Matthew)" <[email-masked]>, "[email-masked]" <[email-masked]>, Linda Dunbar <[email-masked]>
Date: Fri, 15 Nov 2013 00:30:14 +0000

Agree. This is described in nvo3-nve draft. Figure 6 in the draft is shown below.

It describes L2VNx interconnecting with several networks. I am glad that we are on the same page.

Lucy

From: Vishwas Manral [mailto:[email-masked]]
Sent: Thursday, November 14, 2013 5:52 PM
To: Lucy yong
Cc: Bocci, Matthew (Matthew); [email-masked]; Linda Dunbar
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi Lucy/ Linda,

What I am saying is that you will need a centralized gateway even in cases you do some of the functionality that is distributed.

We can talk of advantages of a disrtibuted + centralized model that are well known. Ta distributed Layer-3 gateway allows traffic within the same tier can be routed directly.

Thanks,
Vishwas
On Thu, Nov 14, 2013 at 3:45 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
Hi Vishwas,

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Vishwas Manral
Sent: Thursday, November 14, 2013 5:40 PM
To: Lucy yong
Cc: Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>; Linda Dunbar

Hi Lucy,

Lucy

Thanks,
Vishwas

Lucy

Hi folks,

Thanks,
Vishwas

Linda Dunbar

Regards

Matthew and Benson

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 15 Nov 2013 01:04:58 +0000

The framework mentioned two approaches (see below):

In order to get reachability information, NVEs may exchange
       information directly between themselves via a protocol. In this

    Lasserre, et al.        Expires May 12, 2014                  [Page 9]

 <http://tools.ietf.org/html/draft-ietf-nvo3-framework-04#page-10>
    Internet-Draft  Framework for DC Network Virtualization      November
    2013

       case, a control plane module resides in every NVE. This is how
       routing control plane modules are implemented in routers for
       instance.

       It is also possible for NVEs to communicate with an external Network
       Virtualization Authority (NVA) to obtain reachability and forwarding
       information. In this case, a protocol is used between NVEs and
       NVA(s) to exchange information. OpenFlow [OF] is one example of such
       a protocol.

Now the arch draft mentioned only the NVE-NVA interaction approach. Does it mean the information contained in the framework draft is stale?

Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Yves Hertoghs (yhertogh)
发送时间: 2013年11月15日 2:15
收件人: Lucy yong; Xuxiaohu; Bocci, Matthew (Matthew); [email-masked]
主题: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 15 Nov 2013 01:39:05 +0000

[Xiaohu] In the case where the distribute NVA is deployed at each NVE, what’s the meaning of defining the NVA-NVE protocol? IMHO, almost all the description within the arch draft doesn’t consider the above case and therefore conflict with that case.  If the WG consensus is that the above case is valid as well, it should at least mention that case explicitly somewhere in the draft although the whole draft is focused  on the centralized controller(i.e, NVA)-based architecture.

Xiaohu

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 15 Nov 2013 01:49:23 +0000

发件人: Lucy yong
发送时间: 2013年11月15日 2:20
收件人: Yves Hertoghs (yhertogh); Xuxiaohu; Bocci, Matthew (Matthew); [email-masked]
主题: RE: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Yves,

[Xiaohu] If the above argument was correct, does it mean that we could say that the Internet only supports source routing since the destination-based routing is a specific case of the source routing (i.e., the destination address is the only explicitly specified hop of the explicit path)?  IMHO, over-abstract will result in unnecessary confusions.

Xiaohu

Lucy

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 15 Nov 2013 03:39:17 +0000

Hi Yves,
See inline. - Larry

From: "Yves Hertoghs (yhertogh)" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, November 14, 2013 11:15 AM
To: Lucy yong <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "Bocci, Matthew (Matthew)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

LK> Yves, in section 3.2.5.2 of draft-hertoghs-nvo3-lisp-controlplane-unified-00 (near the bottom of page 11), it describes an ETR sending a Solicit Map Request message to an ITR directly.  Given this, why do you disagree since your own draft describes an NVE sending a control message to another NVE?

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Fri, 15 Nov 2013 08:33:47 +0000

Larry,

I guess this is where the abstraction that Xiahou is referring to can lead to confusion.  In LISP the NVA function is mostly logically centralised, however the local cache at the xTR's are authoritative as well i.e. They can be seen as an extension of the NVA function.  This is how DDT operates, the xTR's can be part of the DDT lookup (very similar to a DNS).

So perhaps my question is now: are the NVE and NVA entitities that we are talking about nodes, or functions within a node?
Based on my understanding it is the latter, hence my interpretation that the pure NVE 'function' does not need a control-plane to interact with the NVE function on another node, although it could interact with the NVA function resident on the same node.

Yves

From: "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>>
Date: Friday 15 November 2013 04:39
To: Cisco Hertoghs <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, Matthew Bocci <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi Yves,
See inline. - Larry

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Fri, 15 Nov 2013 16:50:50 +0000

If we have following text in the framework, it will make the framework and architecture doc align well.

       It is possible for NVEs to communicate with an external Network
       Virtualization Authority (NVA) to obtain reachability and forwarding
       information. In this case, a protocol is used between NVEs and
       NVA(s) to exchange information. OpenFlow [OF] is one example of such
       a protocol.

      It is also possible for NVA partial or entire module co-locates with every NVE. This is how
       routing control plane modules are implemented in routers for instance.

Lucy

From: Xuxiaohu
Sent: Thursday, November 14, 2013 7:05 PM
To: Yves Hertoghs (yhertogh); Lucy yong; Bocci, Matthew (Matthew); [email-masked]
Subject: 答复: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Xiaohu

发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Yves Hertoghs (yhertogh)
发送时间: 2013年11月15日 2:15
收件人: Lucy yong; Xuxiaohu; Bocci, Matthew (Matthew); [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Fri, 15 Nov 2013 17:14:59 +0000

Xiaohu,

[Lucy] please see inline below.

[Xiaohu] In the case where the distribute NVA is deployed at each NVE, what’s the meaning of defining the NVA-NVE protocol?

[Lucy] I agree what you said. If NVA are co-located with each NVE, there is no need for IETF to work on NVA-NVE protocol. But that is not the all cases, right? If NVA and NVE are separated, We need NVE-NVA protocol, right. This is the same as when NVE-TS are co-located, no need to specify the protocol in between, it is all internal implementation. But NVO3 also have to support NVE-TS are separated, where hypervisor-NVE protocol is needed.

 IMHO, almost all the description within the arch draft doesn’t consider the above case and therefore conflict with that case.  If the WG consensus is that the above case is valid as well, it should at least mention that case explicitly somewhere in the draft although the whole draft is focused  on the centralized controller(i.e, NVA)-based architecture.

[Lucy] IMO: the architecture document focuses on the NVE and NVA separation from the functional perspective. Therefore, a protocol between NVE-NVA is necessary in the case. However, this separation can be interpreted as data plane and control plane separation, or the control plane separation, i.e. some control plane function on NVE, some on NVA, which can be the case of BGP w/RR, or fully distributed NVA. Thus the architecture document allows all these solutions. For the last two cases, it is obviously that we do not need to develop a new protocol, but need to exam if existing protocol can fit NVO3 well, where need to enhance and where need to imply.

Lucy

Xiaohu

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Lucy yong <[email-masked]>
Date: Fri, 15 Nov 2013 17:16:55 +0000

Xiaohu,

Sorry, I don’t get what you say below.

Lucy

Yves,

Xiaohu

Lucy

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 15 Nov 2013 17:37:59 +0000

Hi Yves,

In my interpretation, the NVA is a logically centralized function, and if that function happens to be co-resident on the same device as the NVE, then it is still the NVA (not the NVE).  See other responses below.

 - Larry

From: "Yves Hertoghs (yhertogh)" <[email-masked]<mailto:[email-masked]>>
Date: Friday, November 15, 2013 1:33 AM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "Bocci, Matthew (Matthew)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Larry,

I guess this is where the abstraction that Xiahou is referring to can lead to confusion.  In LISP the NVA function is mostly logically centralised, however the local cache at the xTR's are authoritative as well i.e. They can be seen as an extension of the NVA function.

LK> I would not consider the caches in the NVEs to be part of the NVA.

 This is how DDT operates, the xTR's can be part of the DDT lookup (very similar to a DNS).

LK> I admit, I've never looked into the details of DDT, but I was under the impression that the protocol between xTRs and MS/MRs was decoupled from whatever protocol MS/MRs use between themselves.  e.g. you could use BGP tunnels, or DDT within the mapping system, but xTRs always use the same LISP protocol to the MS/MR and are oblivious to how the mapping system works internally (BGP tunnels or DDT).

LK> I still think it is useful for NVEs to be able to communicate directly with other NVEs in some cases, with the most useful example, where an NVE essentially sends the equivalent of a Host Unreachable ICMP message back to a source NVE when the destination TS is no longer connected to the decapsulating NVE.  This doesn't necessarily communicate mapping information directly, but instead a notification that the source NVE may have a stale cache, or an indication that the source NVE should use an alternate mapping if they have another available (which may be useful during a VM migration event).

Yves

Hi Yves,
See inline. - Larry

Yves

I share my view.

Lucy

Hi all,

Best regards,
Xiaohu

Regards

Matthew and Benson

---

From: Thomas Narten <[email-masked]>
Date: Fri, 15 Nov 2013 17:19:11 -0500

NVO3 does not need an NVE to NVE control protocol.

Calling this out explicitly, as it is consistent with the current
architecture document. There is no need for an NVE to NVE control
protocol, for the purpose of maintaining/populating the mapping
tables. There may well be interactions between NVEs, such as setting
up tunnels, creating security associations for protecting data plane
traffic, or providing error indications (e.g., equivalent of ICMP "TS
unreachable" responses).

If folk disagree, now would be a good time to have that conversation.

"Yves Hertoghs (yhertogh)" <[email-masked]> writes:
 
> > [Lucy] do you think we need NVE-NVE control plane? I don’t think
> >  this is what you mean from the following statement.
> 
> >> That doesn't mean that you can't colocate a portion of the
> >> distributed NVA with every NVE, which is the model that
> >> e.g. BGP/L3VPN or ISIS/TRILL uses.
> 

Thomas

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 15 Nov 2013 22:54:22 +0000

Hi Thomas,

I agree with you about the types of NVE to NVE interactions, but if this
isn't called a control protocol, what do we call it?  Do we need
requirements for it?

Thanks, Larry

On 11/15/13 2:19 PM, "Thomas Narten" <[email-masked]> wrote:

>> > [Lucy] do you think we need NVE-NVE control plane? I donâ€™t think

---

From: Lucy yong <[email-masked]>
Date: Fri, 15 Nov 2013 23:07:18 +0000

I guess, some of them belong to OAM. others?

-----Original Message-----
From: Larry Kreeger (kreeger) [mailto:[email-masked]] 
Sent: Friday, November 15, 2013 4:54 PM
To: Thomas Narten; Yves Hertoghs (yhertogh)
Cc: [email-masked]; Lucy yong
Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi Thomas,

I agree with you about the types of NVE to NVE interactions, but if this isn't called a control protocol, what do we call it?  Do we need requirements for it?

Thanks, Larry

On 11/15/13 2:19 PM, "Thomas Narten" <[email-masked]> wrote:

>protocol, for the purpose of maintaining/populating the mapping tables. 
>There may well be interactions between NVEs, such as setting up 
>tunnels, creating security associations for protecting data plane 
>> >> distributed NVA with every NVE, which is the model that e.g. 
>> >> BGP/L3VPN or ISIS/TRILL uses.
>>

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 15 Nov 2013 15:40:02 -0800

> NVO3 does not need an NVE to NVE control protocol.

How about so an NVE knows a path to another NVE is up? Because if it is not, it could choose another NVE that supports the end-host.

> traffic, or providing error indications (e.g., equivalent of ICMP "TS

Then don't worry about naming semantics. Let's just agree that NVEs need to talk to each other, other than just encapsulating to each other.

> If folk disagree, now would be a good time to have that conversation.

I know you may not be suggesting using ICMP unreachables or the equivalent, but remember they just tell you when something "goes unreachable". They don't tell you when something has become reachable (not just become reachable but in a timely fashion), so you'll need some NVE-to-NVE interaction.

Dino

---

From: Sharon <[email-masked]>
Date: Fri, 15 Nov 2013 15:41:22 -0800

100% agree.

There is however a real risk of a control signaling ("NVOi") creeping in,
Wrecking some the distributed-emergent elegance & resilience of the spec,
and creating long lived states and dependencies which are sizeof(mesh).

It always comes up at some point. So you should probably prepare for it.
This prep can be in the form of optional overlay OAM or Echo spec.
This is be used to measure RTT and such IFF peer NVEs choose to.

Per each Echo option that will come up, and guarantee to be many,
there should be a simple litmus test which is - can it be mapped.
Only if not, then it should be considered.

--szb

> On Nov 15, 2013, at 14:19, Thomas Narten <[email-masked]> wrote:
>

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 15 Nov 2013 23:56:38 +0000

Dino, 

Are you talking about one TS connected to two different NVEs?

Then, there are much other issues, such as will you allow multiple uplinks to NVEs being active, do traffic between "a" to "b" have to be symmetric? Etc.   many of them being discussed in TRILL. 

Linda

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Dino Farinacci
> Sent: Friday, November 15, 2013 5:40 PM
> To: Thomas Narten
> Cc: [email-masked]; Lucy yong; Yves Hertoghs (yhertogh)
> Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for
> WG adoption and IPR check for draft-narten-nvo3-arch-01.txt
> 
> How about so an NVE knows a path to another NVE is up? Because if it is
> not, it could choose another NVE that supports the end-host.
> 
> Then don't worry about naming semantics. Let's just agree that NVEs
> need to talk to each other, other than just encapsulating to each other.
> 
> I know you may not be suggesting using ICMP unreachables or the
> equivalent, but remember they just tell you when something "goes
> unreachable". They don't tell you when something has become reachable
> (not just become reachable but in a timely fashion), so you'll need
> some NVE-to-NVE interaction.
> 
> Dino
> 
> >
> >>> [Lucy] do you think we need NVE-NVE control plane? I don't think

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 15 Nov 2013 16:07:27 -0800

> Are you talking about one TS connected to two different NVEs?

No, I was talking about any NVE talking to any other NVE, regardless of the number of TSes attached to it.

> Then, there are much other issues, such as will you allow multiple uplinks to NVEs being active, do traffic between "a" to "b" have to be symmetric? Etc.   many of them being discussed in TRILL. 

;-)

Dino

---

From: Linda Dunbar <[email-masked]>
Date: Sat, 16 Nov 2013 00:14:39 +0000

> the number of TSes attached to it.

Aren't NVEs the end hosts to the underlay network? It is the underlay network's responsibility to forward (encapsulated) packets to other NVEs. If the destination NVEs don't exist, those packets go to black hole. 

Linda

> >>>> And as a result there is only a need for a control plane between
> the
> >>>> NVE function and the NVA function.
> >>>
> >>> Thomas
> >>>

---

From: Florin Balus <[email-masked]>
Date: Fri, 15 Nov 2013 16:21:18 -0800

Not sure I understand what you are asking. Do you really mean to say there
is no need for a control plane protocol period in between NVEs?

Or are you stating there is no need for an additional control plane
protocol? The existing one(s) would cut it.

---

From: Lucy yong <[email-masked]>
Date: Sat, 16 Nov 2013 00:21:56 +0000

Tunnel EPs have OAM tools to monitor each other. 

Lucy

-----Original Message-----
From: Linda Dunbar 
Sent: Friday, November 15, 2013 6:15 PM
To: Dino Farinacci
Cc: Thomas Narten; [email-masked]; Lucy yong; Yves Hertoghs (yhertogh)
Subject: RE: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Linda

> >> From: [email-masked] [mailto:[email-masked]] On 
> >> Behalf
> Of
> >> Dino Farinacci
> >> How about so an NVE knows a path to another NVE is up? Because if 
> >> it
> is

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 15 Nov 2013 16:22:24 -0800

> Aren't NVEs the end hosts to the underlay network? It is the underlay network's responsibility to forward 

Don't go there and confuse matters.

> (encapsulated) packets to other NVEs. If the destination NVEs don't exist, those packets go to black hole. 

Yes, it is. But a NVE could be unreachable and if the encapsulating NVE has a choice to select another NVE to encapsulate to, it should. This makes the architecture more robust.

Dino

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 18 Nov 2013 05:57:44 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Thomas
> Narten
> 发送时间: 2013年11月16日 6:19
> 收件人: Yves Hertoghs (yhertogh)
> 抄送: [email-masked]; Lucy yong
> 主题: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG adoption
> and IPR check for draft-narten-nvo3-arch-01.txt
> 
> Calling this out explicitly, as it is consistent with the current architecture
> document. There is no need for an NVE to NVE control protocol, for the purpose
> of maintaining/populating the mapping tables. There may well be interactions

I'd like to know the reason behind that decision.

Xiaohu

> between NVEs, such as setting up tunnels, creating security associations for
> protecting data plane traffic, or providing error indications (e.g., equivalent of
> ICMP "TS unreachable" responses).
> 
> > > [Lucy] do you think we need NVE-NVE control plane? I don鈥檛 think

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 18 Nov 2013 11:58:24 +0000

Thomas,

The way that you have phrased this sentence is confusing and needs clarification.

Various control protocols can be used between NVEs, as you indicated.

If you meant to say that there is no need for NVEs to exchange address mapping information between themselves, this is also debatable.
An external NVA for such a function is not always required in nvo3. When the NVA address mapping function is colocated with an NVE, NVEs appear as exchanging address mapping information (just looking at the wire). IGPs within NVEs also exchange topological/address mapping information.

Marc

> Behalf Of Thomas Narten
> Sent: Friday, November 15, 2013 11:19 PM
> To: Yves Hertoghs (yhertogh)
> Subject: [nvo3] No need for NVE-NVE control plane [was Re: 
> Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt
> 
> Calling this out explicitly, as it is consistent with the 
> current architecture document. There is no need for an NVE to 
> NVE control protocol, for the purpose of 
> maintaining/populating the mapping tables. There may well be 
> interactions between NVEs, such as setting up tunnels, 
> creating security associations for protecting data plane 
> traffic, or providing error indications (e.g., equivalent of 
> > And as a result there is only a need for a control plane 
> between the

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 18 Nov 2013 15:37:30 +0000

Dino, 

>If the destination NVEs don't
> exist, those packets go to black hole.
> 
> Yes, it is. But a NVE could be unreachable and if the encapsulating NVE
> has a choice to select another NVE to encapsulate to, it should. This
> makes the architecture more robust.

You don't need control plane protocol to achieve what you are asking here. A simple TCP session among NVEs can do the job. When a NVE can't reach the egress NVE to which the target is attached, drop the packet. 

Linda

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 18 Nov 2013 09:09:38 -0800

Dino, 
> You don't need control plane protocol to achieve what you are asking here. A simple TCP session among NVEs can do the job. When a NVE can't reach the egress NVE to which the target is attached, dro…

That is a form of a control-plane protocol. And you will have to same scaling challenges with TCP and arguably it is worse, since TCP, inherently, does not have keepalives.

Dino

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 18 Nov 2013 17:43:24 +0000

Dino, 

So you are asking for a simple keep alive among NVEs (i.e. simpler than TCP) for NVEs to be aware if the target egress NVEs are not reachable. 

That might be good to have, depending on the environment. 

With this so called "control protocol", the packets will be dropped at the Ingress NVE when egress NVE is not reachable.  
Without this so called "control protocol, the packets will be dropped somewhere in the network when the egress NVE is not reachable.  

In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big. 

But in an environment where NVEs have aggregated traffic from many sources/flows, then it is important to have this "control protocol". In this environment, it might even warrant a control plane for NVE to notify the source when the egress is not reachable, so the source can choose a different ingress NVE. 

Bottom-line: I truly think that NVO3 needs a better boundary on the problem domain. If NVO3 is targeted for solving problems for entire worldwide internet, you need many control plane protocols. If NVO3 is targeted for solving problems in data centers, then many control plane protocols becomes unnecessary. 

Linda

> > You don't need control plane protocol to achieve what you are asking
> here. A simple TCP session among NVEs can do the job. When a NVE can't
> reach the egress NVE to which the target is attached, drop the packet.
> 
> That is a form of a control-plane protocol. And you will have to same
> scaling challenges with TCP and arguably it is worse, since TCP,
> inherently, does not have keepalives.
> 
> Dino
> 
> >
> >
> > Linda
> >

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 18 Nov 2013 09:47:08 -0800

> With this so called "control protocol", the packets will be dropped at the Ingress NVE when egress NVE is not reachable.  

Only if *all* the NVEs for the EID? (what do you guys call the things behind the NVE) are unreachable.

> Without this so called "control protocol, the packets will be dropped somewhere in the network when the egress NVE is not reachable.  

Right.

> In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where th…

You have to make this work for UDP applications.

> But in an environment where NVEs have aggregated traffic from many sources/flows, then it is important to have this "control protocol". In this environment, it might even warrant a control plane fo…

No comment.  ;-) But that works naturally in MPTCP by the way.

Dino

---

From: Sharon <[email-masked]>
Date: Mon, 18 Nov 2013 10:15:44 -0800

> On Nov 18, 2013, at 9:47 AM, Dino Farinacci <[email-masked]> wrote:
> 

These things behind the NVE (in the outer-lay segments) need a name. Not just their general EID, but their connection to a given NVE aggregation right now.

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Mon, 18 Nov 2013 18:15:10 +0000

On 11/18/13 9:43 AM, "Linda Dunbar" <[email-masked]> wrote:

>Dino, 
>
>So you are asking for a simple keep alive among NVEs (i.e. simpler than
>TCP) for NVEs to be aware if the target egress NVEs are not reachable.
>
>With this so called "control protocol", the packets will be dropped at
>the Ingress NVE when egress NVE is not reachable.
>Without this so called "control protocol, the packets will be dropped
>somewhere in the network when the egress NVE is not reachable.
>
>In data center environment where most communications are among
>applications, most likely a source will not send more packets without
>acknowledgment from the destination. Then the impact of where the packet
>is dropped is not that big.
>
>But in an environment where NVEs have aggregated traffic from many
>sources/flows, then it is important to have this "control protocol". In
>this environment, it might even warrant a control plane for NVE to notify
>the source when the egress is not reachable, so the source can choose a
>different ingress NVE.
>
>Bottom-line: I truly think that NVO3 needs a better boundary on the
>problem domain. If NVO3 is targeted for solving problems for entire
>worldwide internet, you need many control plane protocols. If NVO3 is
>targeted for solving problems in data centers, then many control plane
>protocols becomes unnecessary.

LK> I think the charter is clear that NVO3 is targeted for data center
solutions.

---

From: Ken Gray <[email-masked]>
Date: Mon, 18 Nov 2013 19:24:54 +0000

I don't think the logical association of the NVA with an NVE negates Thomas basic point.  The key word you use here is "appears" ... in the end, it's not the NVE that other NVEs are exchanging mapping information with, it is the NVA that is co-located.

+1 for Thomas proposal.  YES, you can make it more explicit in the architecture that "mapping" is what you are talking about instead of the generic "control".

We can beat NVE-NVE liveliness to death with other truncheons.
________________________________________
From: [email-masked] <[email-masked]> on behalf of LASSERRE, MARC (MARC) <[email-masked]>
Sent: Monday, November 18, 2013 6:58 AM
To: Thomas Narten; Yves Hertoghs (yhertogh)
Cc: [email-masked]; Lucy yong
Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG      adoption and IPR check for draft-narten-nvo3-arch-01.txt

Thomas,

Marc

---

From: Ken Gray <[email-masked]>
Date: Mon, 18 Nov 2013 19:54:22 +0000

Unless it's built at a lower layer than a piggy-back app, it's going to be a scale monster (IMO, if you plan for success, then ultimately you have highly connected NVEs - hello scale monster).  Something like CFM for overlays (logical ethernet, logical continuity check built into your header as a special packet type, punt to momma NVE process ...not some cousin, etc?)?  =8^)
________________________________________
From: [email-masked] <[email-masked]> on behalf of Dino Farinacci <[email-masked]>
Sent: Monday, November 18, 2013 12:09 PM
To: Linda Dunbar
Cc: Thomas Narten; [email-masked]; Lucy Yong; Yves Hertoghs (yhertogh)
Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG      adoption and IPR check for draft-narten-nvo3-arch-01.txt

Dino,

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 18 Nov 2013 14:06:16 -0800

> Unless it's built at a lower layer than a piggy-back app, it's going to be a scale monster (IMO, if you plan for success, then ultimately you have highly connected NVEs - hello scale monster).  Som…

Definitely agree Ken. See LISP echo-noncing. I pulled some text out of RFC 6830 and pasted below.

Dino

------

6.3.1.  Echo Nonce Algorithm

   When data flows bidirectionally between Locators from different
   sites, a data-plane mechanism called "nonce echoing" can be used to
   determine reachability between an ITR and ETR.  When an ITR wants to
   solicit a nonce echo, it sets the N- and E-bits and places a 24-bit
   nonce [RFC4086] in the LISP header of the next encapsulated data
   packet.

   When this packet is received by the ETR, the encapsulated packet is
   forwarded as normal.  When the ETR next sends a data packet to the
   ITR, it includes the nonce received earlier with the N-bit set and
   E-bit cleared.  The ITR sees this "echoed nonce" and knows that the
   path to and from the ETR is up.

   The ITR will set the E-bit and N-bit for every packet it sends while
   in the echo-nonce-request state.  The time the ITR waits to process
   the echoed nonce before it determines the path is unreachable is
   variable and is a choice left for the implementation.

   If the ITR is receiving packets from the ETR but does not see the
   nonce echoed while being in the echo-nonce-request state, then the
   path to the ETR is unreachable.  This decision may be overridden by
   other Locator reachability algorithms.  Once the ITR determines that
   the path to the ETR is down, it can switch to another Locator for
   that EID-Prefix.

   Note that "ITR" and "ETR" are relative terms here.  Both devices MUST
   be implementing both ITR and ETR functionality for the echo nonce
   mechanism to operate.

   The ITR and ETR may both go into the echo-nonce-request state at the
   same time.  The number of packets sent or the time during which echo
   nonce requests are sent is an implementation-specific setting.
   However, when an ITR is in the echo-nonce-request state, it can echo
   the ETR's nonce in the next set of packets that it encapsulates and
   subsequently continue sending echo-nonce-request packets.

   This mechanism does not completely solve the forward path
   reachability problem, as traffic may be unidirectional.  That is, the
   ETR receiving traffic at a site may not be the same device as an ITR
   that transmits traffic from that site, or the site-to-site traffic is
   unidirectional so there is no ITR returning traffic.

   The echo-nonce algorithm is bilateral.  That is, if one side sets the
   E-bit and the other side is not enabled for echo-noncing, then the
   echoing of the nonce does not occur and the requesting side may
   erroneously consider the Locator unreachable.  An ITR SHOULD only set
   the E-bit in an encapsulated data packet when it knows the ETR is
   enabled for echo-noncing.  This is conveyed by the E-bit in the
   Map-Reply message.

   Note that other Locator reachability mechanisms are being researched
   and can be used to compliment or even override the echo nonce
   algorithm.  See the next section for an example of control-plane
   probing.

------

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 19 Nov 2013 02:21:02 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> LASSERRE, MARC (MARC)
> 发送时间: 2013年11月18日 19:58
> 收件人: Thomas Narten; Yves Hertoghs (yhertogh)
> 主题: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG
> adoption and IPR check for draft-narten-nvo3-arch-01.txt
> 
> Thomas,
> 
> If you meant to say that there is no need for NVEs to exchange address mapping
> information between themselves, this is also debatable.
> An external NVA for such a function is not always required in nvo3. When the
> NVA address mapping function is colocated with an NVE, NVEs appear as
> exchanging address mapping information (just looking at the wire). IGPs within
> NVEs also exchange topological/address mapping information.

Fully agree.

Xiaohu

> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Thomas Narten

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 19 Nov 2013 12:36:52 +0000

NVE-NVE control plane is needed for operations that don't necessarily need NVA interaction or where NVE-NVE interaction can provide better performance. 

For example, if the "mappings" are stale on an NVE, then the receiver NVE can send a control plane message to the sender NVE to update its mappings. This can happen due to events where the VM live migrated from one host to another but the sender NVE has stale mappings due to delay in getting those mappings updated via NVA. Think of this as a hint for cache flush from the receiver NVE to the sender NVE. 

The control plane between NVE-NVE can also be used for protocol negotiation in case of multi-protocol support at NVE.

Pankaj

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Ken Gray
Sent: Tuesday, November 19, 2013 12:55 AM
To: LASSERRE, MARC (MARC); Thomas Narten; Yves Hertoghs (yhertogh)
Cc: [email-masked]; Lucy yong
Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Thomas,

Marc

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 19 Nov 2013 16:09:11 +0000

Pankaj, 

Of course, NVE-NVE protocol can add benefits, but it also adds cost. A good design is not about if there is nothing to add, but about if there is nothing that can be removed. 
If most TSes are only attached to a single NVE and traffic to NVEs are not aggregated flows, then the NVE-NVE control plane protocol doesn’t provide much benefits.   Under this environment, the difference between having “NVE-NVE control protocol" is whether the packets being dropped at the Ingress NVE or somewhere in the underlay network when egress NVE is not reachable.  
In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big.
 
Therefore, I think it is worthwhile to have a subsection in the architecture document to describe this trade-off. It is not clear-cut "must have" or "must not have". 
Here is my suggested subsection:
--------------------------------------------------

4.4 Is it necessary to have Inter-NVE control plane protocol? (Suggested new sub-section)
There could be various reasons, link failure, node failure, or others, causing egress NVEs not reachable. Without any NVE<->NVE control plane protocol, the ingress NVE is not aware of the reachability of egress NVE causing encapsulated packets to dropped somewhere in the underlay network. 
 If most TSes are only attached to a single NVE and traffic to NVEs are not aggregated flows, then the NVE-NVE control plane protocol doesn’t provide much benefits.   Under this environment, the difference between having “NVE-NVE control protocol" is whether the packets being dropped at the Ingress NVE or somewhere in the underlay network when egress NVE is not reachable.  
In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big. 
However, in an environment where TSes are connected to multiple NVEs, then it might be worthwhile to consider the Inter-NVE control plane protocol, so that ingress NVE can choose different egress NVE for a given target.

Linda 

> From: nvo3 [mailto:[email-masked]] On Behalf Of Pankaj Garg
> Sent: Tuesday, November 19, 2013 6:37 AM
> To: Ken Gray; LASSERRE, MARC (MARC); Thomas Narten; Yves Hertoghs
> (yhertogh)
> NVE-NVE control plane is needed for operations that don't necessarily
> need NVA interaction or where NVE-NVE interaction can provide better
> performance.
> 
> For example, if the "mappings" are stale on an NVE, then the receiver
> NVE can send a control plane message to the sender NVE to update its
> mappings. This can happen due to events where the VM live migrated from
> one host to another but the sender NVE has stale mappings due to delay
> in getting those mappings updated via NVA. Think of this as a hint for
> cache flush from the receiver NVE to the sender NVE.
> 
> The control plane between NVE-NVE can also be used for protocol
> negotiation in case of multi-protocol support at NVE.
> 
> Pankaj
> 
> I don't think the logical association of the NVA with an NVE negates
> Thomas basic point.  The key word you use here is "appears" ... in the
> end, it's not the NVE that other NVEs are exchanging mapping
> information with, it is the NVA that is co-located.
> 
> +1 for Thomas proposal.  YES, you can make it more explicit in the
> architecture that "mapping" is what you are talking about instead of
> the generic "control".
> 
> From: [email-masked] <[email-masked]> on behalf of
> LASSERRE, MARC (MARC) <[email-masked]>
> If you meant to say that there is no need for NVEs to exchange address
> mapping information between themselves, this is also debatable.
> An external NVA for such a function is not always required in nvo3.
> When the NVA address mapping function is colocated with an NVE, NVEs
> appear as exchanging address mapping information (just looking at the
> wire). IGPs within NVEs also exchange topological/address mapping
> information.
> 
> Marc
>

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 19 Nov 2013 19:19:44 +0000

Hi Linda - I agree that it is not a must have, but definitely a SHOULD. I am not sure I understand the point about most TS connected to single NVE, wouldn't the case be where most TSes would be connected to different NVE as in different hypervisors that are doing encap/decap?

Just to elaborate the example I gave earlier, in the live migration (or vMotion) or cluster failover scenario, the CA address to PA address mapping changes and an NVA may not be involved in the decision of that mapping change. The cluster failover can happen automatically without NVA intervention. In this case, if there is existing communication going on between two VMs, to keep the promise of continuous availability, the packet flow must recover with-in N seconds. If NVA is down during this time, we would need an inter-NVE control channel to keep packet flowing. 

I suggest few changes to the text you suggest, below.

4.4 Is it necessary to have Inter-NVE control plane protocol? (Suggested new sub-section) There could be various reasons, link failure, node failure, or others, causing egress NVEs not reachable. Without any NVE<->NVE control plane protocol, the ingress NVE is not aware of the reachability of egress NVE causing encapsulated packets to dropped somewhere in the underlay network. 
>>[PG] - An NVE<->NVE control plane protocol can provide faster recovery from such failures. 

>>[PG] - I think we can strikethrough this as the text above seems sufficient to describe the problem and value of NVE<->control plane. 

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, November 19, 2013 9:39 PM
To: Pankaj Garg; Ken Gray; LASSERRE, MARC (MARC); Thomas Narten; Yves Hertoghs (yhertogh)
Cc: [email-masked]; Lucy yong
Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Pankaj, 

4.4 Is it necessary to have Inter-NVE control plane protocol? (Suggested new sub-section) There could be various reasons, link failure, node failure, or others, causing egress NVEs not reachable. Without any NVE<->NVE control plane protocol, the ingress NVE is not aware of the reachability of egress NVE causing encapsulated packets to dropped somewhere in the underlay network. 
 If most TSes are only attached to a single NVE and traffic to NVEs are not aggregated flows, then the NVE-NVE control plane protocol doesn’t provide much benefits.   Under this environment, the difference between having “NVE-NVE control protocol" is whether the packets being dropped at the Ingress NVE or somewhere in the underlay network when egress NVE is not reachable.  
In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big. 
However, in an environment where TSes are connected to multiple NVEs, then it might be worthwhile to consider the Inter-NVE control plane protocol, so that ingress NVE can choose different egress NVE for a given target.

Linda 

> Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll 
> for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt
> 
> mappings. This can happen due to events where the VM live migrated 
> from one host to another but the sender NVE has stale mappings due to 
> delay in getting those mappings updated via NVA. Think of this as a 
> hint for cache flush from the receiver NVE to the sender NVE.
> 
> Subject: Re: [nvo3] No need for NVE-NVE control plane [was Re: Poll 
> for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt
> 
> > traffic, or providing error indications (e.g., equivalent of ICMP 
> > "TS unreachable" responses).
> >

---

From: Florin Balus <[email-masked]>
Date: Tue, 19 Nov 2013 11:52:48 -0800

I agree with Marc, the question is open to interpretation: are you talking
about underlay or overlay control plane? For underlay there is definitely
an IGP/BGP component that might be needed at least in some NVEs,
especially when the NVE function is running on a Gateway (ToR/PE etc).

>From the perspective of an NVE service overlay there are already multiple
proposals in different WGs (IP VPN, L2VPN, LISP etc) designed to provide
control plane for VXLAN and NVGRE-based L2/L3 services: I.e. BGP EVPN and
BGP VPN are specifically designed to provide NVE2NVE distribution for
certain types of NVEs. Proposals are also covering NVA2NVE or even NVA2NVA
distribution. So the question is pointless IMHO, there is already a lot of
traction behind building overlay/underlay control plane for NVO3 based
encapsulations and there is solution work in progress elsewhere.

Unless you are asking whether there is a need to do some control plane
work in NVO3? If this is the question then IMHO it is going in the right
direction. Although I would formulate it more generic: is NVO3 is the
right place to do any solution analysis or development at this stage?

On 11/18/13, 3:58 AM, "LASSERRE, MARC (MARC)"
<[email-masked]> wrote:

>An external NVA for such a function is not always required in nvo3. When
>the NVA address mapping function is colocated with an NVE, NVEs appear as
>exchanging address mapping information (just looking at the wire). IGPs
>within NVEs also exchange topological/address mapping information.
>
>Marc
>

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 12:07:22 -0500

Linda,

> If most TSes are only attached to a single NVE and traffic to NVEs
>  are not aggregated flows, then the NVE-NVE control plane protocol
>  doesn’t provide much benefits.

I want to flag the first part of the sentence above because this
question has been raised before. It suggests that a TS might connect
to more than one NVE at a time.

>From an architectural perpsective, I think we should say "no". Life is
much simpler if a TS connects to a VN through exactly one NVE. More
precisely, a TSI always connects to one NVE. If we allowed connections
to multple NVEs simultaneously, it would raise all sorts of questions
as to when to use one over the other, etc.

I don't think I'm actually saying anything new with this... I've
always assumed this, documents are written with this in mind, and it
hasn't come up (other than in passing) on the list.

A TS can still connect to multiple different VNs through different
network interfaces, and hence different TSIs, but I think we should
have a TSI always connect to exactly one NVE.

Any other opinions?

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 20 Nov 2013 09:24:37 -0800

> as to when to use one over the other, etc.

Thomas, just be aware that in the physical case, a server is dual attached to multiple TORs. If each TOR is going to be an NVE, because you want to save state in the data center core layer, then you'll have to do multi-homing.

Dino

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 12:31:22 -0500

Dino,

> Thomas, just be aware that in the physical case, a server is dual
>  attached to multiple TORs. If each TOR is going to be an NVE,
>  because you want to save state in the data center core layer, then
>  you'll have to do multi-homing.

I suspect we need to dive a bit deeper into this. i.e., what does
"dual attached" mean?

Do you mean the NVE has two physical connections to the ToR? If so,
than the question presumably is about allowing/supporting multi-homed
NVEs.

If the question is about the TS having two attachements to the ToR,
wouldn't that be modeled (architecturally) as two distinct
interfaces/ports, in which case each one can connect to its own NVE,
in which case we are good?

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 20 Nov 2013 12:34:07 -0500

Most data center diagram sshow the servers (platforms hosting the TS if 
I understand correctly) as dual-homed to the network.  This is necessary 
for robustness.

How does that interact with asserting that TS are always connected 
through the VN through exactly one NVE?

Yours,
Joel

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 20 Nov 2013 17:46:24 +0000

Thomas, 

Since there are quite a few people stating that NVE-NVE control plane is needed for NVO3, the architecture document should at least have a sub-section describing the trade-off of having NVE-NVE protocol. 

Even with one TS being connected to one NVE, there are people saying that one NVE could have two different uplink ports to underlay network, therefore might have two different IP addresses. My counter-argument is that NVE should use its loopback IP address and can have different MAC addresses for its different uplink ports or use NIC-Teaming for its multiple ports. 

I think that my suggested subsection is necessary to describe the trade-off and to make WG feel that their comments are well addressed.  

Granted that NVE-NVE protocol does provide some value, like notifying each other the directed attached TSes when NVA is not reachable, etc. 
NVO3 architecture doesn't have to "require this protocol", but doesn't have to " explicitly exclude this protocol " either.  

Again here is my suggested new subsection (with minor change to address Pankaj's comments:
-----------------------------------

4.4 The trade-off of Inter-NVE control plane protocol (Suggested new sub-section) 

There could be various reasons, link failure, node failure, or others, causing egress NVEs (or even NVA) not reachable. Without any NVE<->NVE control plane protocol, the ingress NVE is not aware of the reachability of egress NVE causing encapsulated packets to dropped somewhere in the underlay network. 

If most TSes are only attached to a single NVE and traffic to NVEs are not aggregated flows, then the value provided by NVE-NVE control plane protocol is not significant compared to the extra cost added to NVE.  When one NVE has multiple uplink ports to the underlay network, it is recommended to use NVE's loopback address instead of port addresses to avoid the complication on ingress node in determining which address to use. 
Under this environment, the difference between having “NVE-NVE control protocol" is whether the packets being dropped at the Ingress NVE or somewhere in the underlay network when egress NVE is not reachable.  
In data center environment where most communications are among applications, most likely a source will not send more packets without acknowledgment from the destination. Then the impact of where the packet is dropped is not that big. 
However, in an environment where TSes are connected to multiple NVEs or high chance of connection failure between NVE & NVA, then it might be worthwhile to consider the Inter-NVE control plane protocol, so that ingress NVE can choose different egress NVE for a given target.

---------------------------

Linda

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Wednesday, November 20, 2013 11:07 AM
> To: Linda Dunbar
> Cc: [email-masked]
> Subject: TS conects to VN through one NVE [was Re: [nvo3] No need for
> NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for
> draft-narten-nvo3-arch-01.txt]
> 
> Linda,
> 
> >  doesn鈥檛 provide much benefits.
>

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 20 Nov 2013 10:10:51 -0800

"Dino,
> I suspect we need to dive a bit deeper into this. i.e., what does

A bare-metal server has 2 physical connections to different boxes (where a box is a swtich or a router). In some cases, the 2 connections are active/active, but mostly active/backup. There is some assumption from the server OSes that the upstream boxes are layer-2 switches. But it doesn't have to be the case, they can be layer-3 routers.
 
> Do you mean the NVE has two physical connections to the ToR? If so,

No, that would be the NVE being able to get ECMP support from the underlay. I am talking about the NVE in the TOR, as one example.

> than the question presumably is about allowing/supporting multi-homed

That is right Thomas.

> in which case we are good?

The questions is about the return traffic coming to the TS. Do you want it load-split across all the NVEs associated with the TS. The answer is a definite yes, as Joel stated for robustness and to take advantage of all cross-sectional cheap bandwidth in a data center.

Dino

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 14:59:36 -0500

Hi Dino.

> A bare-metal server has 2 physical connections to different boxes
>  (where a box is a swtich or a router). In some cases, the 2
>  connections are active/active, but mostly active/backup. There is
>  some assumption from the server OSes that the upstream boxes are
>  layer-2 switches. But it doesn't have to be the case, they can be
>  layer-3 routers.

I think there are multiple scenarios here to consider, and the details
matter.

Scenario 1: Bare metal server has 2 separate connections to two
different boxes. How are these two connections presented to the
(bare-metal) TS?

1A) two different interfaces are visible to the TS, both with their
own separate IP addresses. This is not an issue for NVO3, since this
is modeled as two interfaces to two separate VNs. (Right?)

B) TS sees only one interface, the CNA hides the details of two
physical uplinks from TS. Links could be active/active or
active/backup. In this case, the TS has one IP adddress (I presume)
with the CNA hiding the details of multiple uplinks from the TS.

This is a messier case... Do both uplinks have to  connect to the same
NVE? In the standard L2 case, don't both L2 uplinks have to connect to
the "same L2 network"? That only means that both ToRs would need to
support the same VLANs and connect to the same LAN domains. (Right?)

If above we allow the two uplinks to go to two different NVEs, the
implication is that from an NVO3 perspective, a given TS mapping is
reachable through different NVE addresses. That raises all sorts of
interesting questions. :-)

> No, that would be the NVE being able to get ECMP support from the
>  underlay. I am talking about the NVE in the TOR, as one example.

Right, NVE is doing IP and each uplink would be to different L3 dest.

> That is right Thomas.

Yikes. I guess we need to have that discussion. :)

> The questions is about the return traffic coming to the TS. Do you
> want it load-split across all the NVEs associated with the TS. The
> answer is a definite yes, as Joel stated for robustness and to take
> advantage of all cross-sectional cheap bandwidth in a data center.

Fundamentally, it means that address mappings (inner to outer) are not
one-to-one, but are one-to-many.

Is this something we need to support? I agree it has some
benefits. But it also adds some complications...

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 20 Nov 2013 13:55:38 -0800

> I think there are multiple scenarios here to consider, and the details

Agree.

> different boxes. How are these two connections presented to the

I am going to fail at describing this with the NVO3 terminology. A TS to me is a site, that has lots of hosts, switches, routers in it. They are addressable. I call those addresses EIDs. So we are talking about a single EID. An IP address assigned to a bare-metal server that has two connections, one to each physical NVE.

> is modeled as two interfaces to two separate VNs. (Right?)

I can't answer this because I am not sure you and I agree on what a TS is. I am not disagreeing how NVO3 defines a TS but I think it is not specific enough for this context.

> support the same VLANs and connect to the same LAN domains. (Right?)

Yes, they have to connect to the same L2 network.

> interesting questions. :-)

This called multi-homing.  ;-)  That is equivalent to a LISP EID that has a locator-set. The reason there is a locator-set is because there can be multiple RLOCs (read: NVEs) used reach the EID.

> Do you mean the NVE has two physical connections to the ToR? If so,

No, I am simply generalizing it to this topology:

            EIDs
           /   \
          /     \
        NVE     NVE
        /         \
       /           \

Where there are multiple from the EID to 1-or-more NVEs. And the NVEs have multiple paths leading to the core.

> one-to-one, but are one-to-many.

That is correct. And many-to-one as well.

> benefits. But it also adds some complications...

It depends how close topologically the NVE is to the endpoint. If the NVE was in the application, then for the same IP address there would be multiple NVEs. So the point could be taken to an extreme.

Dino

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 17:08:59 -0500

> I am going to fail at describing this with the NVO3 terminology. A
>  TS to me is a site, that has lots of hosts, switches, routers in
>  it. They are addressable. I call those addresses EIDs. So we are
>  talking about a single EID. An IP address assigned to a bare-metal
>  server that has two connections, one to each physical NVE.

A TS is Tenant System. E.g., a VM on virtualized host, or the bare
metal server in your original example. 

The NVE is normally very close to the TS, i.e., on the same server, or
offloaded to the ToR.

> I can't answer this because I am not sure you and I agree on what a
> TS is. I am not disagreeing how NVO3 defines a TS but I think it is
> not specific enough for this context.

TS is a single host endpoint.  It connects directly to a virtual
network through an NVE.

> Yes, they have to connect to the same L2 network.

The difference is in the L2 case, you can send a packet up either
uplink and things generally work (you may still have to deal with MAC
flapping concerns). In the NVO3 case, if the two upstream NVEs are
different, they will have their own IP addresses and the tunnel source
for packets they send to remote NVEs will be different depending on
which NVE is the sender.

> This called multi-homing.  ;-) That is equivalent to a LISP EID that
>  has a locator-set. The reason there is a locator-set is because
>  there can be multiple RLOCs (read: NVEs) used reach the EID.

Right.

The WG just hasn't had much discussion about whether NVEs need to be
multi-homed. So we don't really have a position yet.

> It depends how close topologically the NVE is to the endpoint. If
> the NVE was in the application, then for the same IP address there
> would be multiple NVEs. So the point could be taken to an extreme.

In the virtualized case, the NVE is on the same server as the
VM/TS. In the bare metal case, the NVE is offloaded onto an adjacent
device (e.g., ToR), but is still quite close.

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 20 Nov 2013 14:27:17 -0800

> The NVE is normally very close to the TS, i.e., on the same server, or

I know of deployments where the NVE equivalent is at the end-of-row router.

> TS is a single host endpoint.  It connects directly to a virtual

What do you call a multi-tenant rack? In deployments, a tenant is a customer of the provider where there are many 'systems' in the tenant, each with one or more IP addresses assigned to them.

> for packets they send to remote NVEs will be different depending on

Sending is not the interesting case and is a bit off topic. It is return packets and which (or both) NVEs is used as the target of encapsulation.

> multi-homed. So we don't really have a position yet.

Yes, right, why we are bringing it up now.

> device (e.g., ToR), but is still quite close.

You are assuming a specific implementation of an NVE and where it resides. That should be be assumed in the architecture IMO.

Dino

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Nov 2013 17:36:07 -0500

> I know of deployments where the NVE equivalent is at the end-of-row

But lets deal with the common/expected case. That starts with NVEs
co-located with the hypervisors.

One important reason that NVEs tend to be close to the TS is that you
can deploy things that way with little or no change to the underlay.

> What do you call a multi-tenant rack? In deployments, a tenant is a
>  customer of the provider where there are many 'systems' in the
>  tenant, each with one or more IP addresses assigned to them.

Different kind of tenant. If you have a tenant (say coke) that has a
virtual network with (say) 100 VMs on it that talk to each other, each
VM is a TS and connects to an NVE.

> Sending is not the interesting case and is a bit off topic. It is
>  return packets and which (or both) NVEs is used as the target of
>  encapsulation.

Agree...

> You are assuming a specific implementation of an NVE and where it
>  resides. That should be be assumed in the architecture IMO.

You presumably mean "should not be assumed".

Well, the truth is that NVEs have been assumed to be close to the TS
from day one... And one reason is that if you push the NVE out further
and further away from the TS, it adds complications. We shouldn't do
that unless necessary.

Thomas

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 20 Nov 2013 14:58:40 -0800

> co-located with the hypervisors.

Why is that an expected case? I guess host vendors or software-only vendors would appreciate that but I'm sure equipment vendors would not.

> can deploy things that way with little or no change to the underlay.

At the cost of lost aggregation. There are pros and cons either way.

> VM is a TS and connects to an NVE.

It is is a VPN with say 10 sites, each with your 10 VMs, then each site could share an NVE.

> You presumably mean "should not be assumed".

Yes, that was a typo.

> and further away from the TS, it adds complications. We shouldn't do

You can see by now is that my point is that architecturally you shouldn't say where NVEs are or how they are implemented (in software, hardware or combo software/hardware).

I will argue that if you push an NVE further out there are less of them to manage so you get OpEx benefits.

Dino

---

From: "Reith, Lothar" <[email-masked]>
Date: Thu, 21 Nov 2013 00:03:22 +0000

Dear friends,

are we sure everybody is on the same page? Please make sure we are all talking about use case a) below when we use the term "multi-homing", and not referring to use cases b),c) or even d).

Two physical links from a bare metal server representing the equivalent of a single VM interfacing to two different physical devices can be implemented in the following use case scenarios:

a). Layer 3 Multihoming with two different underlying layer 2 networks (i.e. the VM or bare metal server has two different TSIs each interfacing to a different Gateway IP)

b). Layer 3 VRRP based Gateway redundancy with both physical links being part of the same layer 2 network, which has 3 endpoints, however 2 of them share the same MAC and IP address which floats between the two physical devices as virtual IP address and virtual MAC address of one virtual gateway that is redundant because it is distributed/split across two physical devices. In order not to be confused with the term distributed gateway, this redundant gateway can be one local portion of a distributed gateway in the previously discussed sense, where the portions of the distributed gateway may even be distributed across multiple autonomous systems (if I understand this correctly).

c). Layer 2 bonding/Link aggregation based with LACP controlled active/passive mode or less likely also supporting active/active mode: Both Tenant system sided physical link endpoints are a member of a normal LAG (bonding group), while both corresponding NVE sided physical link endpoints are also bonded together to form a single MAC-Service Access Point using a potentially proprietary multichassis LAG implementation, where the control plane for cross-synchronization between both NVE sided endpoints may be via Layer3 or via Layer2, and where implementations may be vendor specific and differ with respect to whether the NVE is one logical device distributed across two physical devices, or two logical NVE devices (one per physical device). In any case both physical devices are cooperating somehow to pretend to the other side of the point to point layer 2 network that they implement a single aggregated layer 2 interface (most likely only supporting active/passive mode where only one of the two physical links is active carrying user plane traffic at any one time).

d). For completeness: Layer 1 bonding is also possible, but might be neglected here.

Lothar

-----Ursprüngliche Nachricht-----
Von: nvo3 [mailto:[email-masked]] Im Auftrag von Dino Farinacci
Gesendet: Mittwoch, 20. November 2013 19:11
An: Thomas Narten
Cc: [email-masked]; Linda Dunbar
Betreff: Re: [nvo3] TS conects to VN through one NVE [was Re: No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt]

"Dino,

That is right Thomas.

Dino

---

From: "Reith, Lothar" <[email-masked]>
Date: Thu, 21 Nov 2013 01:05:28 +0000

Just wanted to add that the terms "NIC-teaming", NIC-bonding","port-channel" "LAG" "LACP" are all referring to case c) when combined with two physical links towards two different physical devices implementing the NVE side (potentially vendor dependant either one NVE distributed across 2 physical devices or two NVE.  In any case, the single NVE or the both NVEs terminate the two physical links and hide the fact that they are two physical interfaces from their upper layers, by pretending they can be accessed like a single physical interface, due to the fact that a single MAC-Service Access Point is presented to the upper layers, which may switch locations when the active/passive sub-state changes.

Kind of magic, but I am aware of at least 3 or 4 different vendor implementations of such multi-chassis LAG behavior. 

And I like to note that some legacy OSS systems may have trouble coping with logical devices distributed across multiple physical devices, making the IT for provisioning and assurance quite expensive or suboptimal due to workarounds being applied, which complicate things over time.

Please keep in mind that at the end of the day, operators need to map the NVE in their inventory systems, correlate alarms for assurance etc. It would be a nightmare, if different vendors implement the obviously required NVE redundancy in fundamentally different ways such as some vendors implementing an NVE in a way that his has one location, and another vendor where an NVE has multiple locations. This may even lead to vendor specific assurance processes - an absolute No-Go for carriers if they are not insane.

Lothar

-----Ursprüngliche Nachricht-----
Von: nvo3 [mailto:[email-masked]] Im Auftrag von Reith, Lothar
Gesendet: Donnerstag, 21. November 2013 01:03
An: Dino Farinacci; Thomas Narten
Cc: [email-masked]; Linda Dunbar
Betreff: Re: [nvo3] TS conects to VN through one NVE [was Re: No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt]

Dear friends,

Lothar

-----Ursprüngliche Nachricht-----
Von: nvo3 [mailto:[email-masked]] Im Auftrag von Dino Farinacci
Gesendet: Mittwoch, 20. November 2013 19:11
An: Thomas Narten
Cc: [email-masked]; Linda Dunbar
Betreff: Re: [nvo3] TS conects to VN through one NVE [was Re: No need for NVE-NVE control plane [was Re: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt]

"Dino,

That is right Thomas.

Dino

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 21 Nov 2013 06:00:28 -0500

Doesn't your item 2 assume that what is being delivered is an L3 
service?  What if the TS is multi-connected to two NVEs, and is getting 
L2 service to the same tenant bridged space from both of them?  Even if 
the service is layer 3, I would expect the tenant to want all IP 
addresses to work even if on NVE connection fails.  To some degree what 
you seem to be saying below is that we should always treat multiple 
TS-NVE connections as if they were separate TS.  That seems to be very 
restrictive on the redundancy models people expect.

Yes, as you comment in another note, if we assume the NVE is in the 
hyperviso then this is a non-problem.  But I was under the impression 
that we were not to make that assumption.

Yours,
Joel

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 22 Nov 2013 10:37:36 -0800

> and further away from the TS, it adds complications. We shouldn't do

If we are saying that architecturally the TS is separate from the NVE 
with a network in between (to support implementations like off-loading 
NVE functionality to a TOR), then the question about multihoming will 
naturally arise. Today there are physical servers which are multihomed 
(using techniques like active-standby NIC teaming or active-active 
multi-chassis LAG), and it would seem odd of the NVO3 architecture were 
to preclude that.

BTW: I think the subject like is incorrect. It it the TS that is 
multihomed (to different NVEs) and the subject talks about multihomed NVEs.

    Erik

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 22 Nov 2013 11:24:58 -0800

> BTW: I think the subject like is incorrect. It it the TS that is multihomed (to different NVEs) and the subject talks about multihomed NVEs.

Erik, just an FYI, I was referring to both when I brought up the issue. That is, you need to have generality between the TS and the NVE (a network, as you stated, either an L2 or L3 network) and a general network from the NVE out to other NVEs (again a network, mostly likely only L3, but L2 should just fall out).

Dino

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 22 Nov 2013 13:10:56 -0800

Dino,

I was thinking of L3 between NVEs, in which case L3 routing including 
ECMP takes care of NVEs that are multiply connected.

You're correct that if L2 used between NVEs than we need to make sure 
that the architecture doesn't preclude using existing L2 techniques for 
multihoming and multipathing.

Regards,
    Erik

---

From: Thomas Narten <[email-masked]>
Date: Mon, 25 Nov 2013 11:41:52 -0500

Support as co-author and am not not aware of IPR that applies to this
draft.

Thomas

---

From: Pankaj Garg <[email-masked]>
Date: Tue, 26 Nov 2013 04:33:08 +0000

I agree, NVE-NVE control should be included in the arch document. It provides additional value and is required for optimal functioning of VNs in certain cases such as cluster failover or other events where a logically centralized entity like NVA may not be available.

Pankaj

________________________________
From: [email-masked] <[email-masked]> on behalf of Xuxiaohu <[email-masked]>
Sent: Thursday, November 14, 2013 14:27
To: Bocci, Matthew (Matthew); [email-masked]
Subject: [nvo3] ??: Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Hi all,

IMHO, the NVE-NVE protocol is still useful in some small and medium sized multi-tenant data center networks. AFAIK, most tenants within public cloud data centers are small and medium sized enterprises which usually don't need a lot of VMs. That means the number of NVEs for most VNs would not be very large especially in the case where the NVE is deployed at physical switches, rather than hypervisors/servers. In this case, the VN membership can be discovered via IGP flooding and the address mapping information of a given VN could be directly exchanged among NVEs of that VN, without a need for a dedicated NVA.

Best regards,
Xiaohu

???: [email-masked] [mailto:[email-masked]] ?? Bocci, Matthew (Matthew)
????: 2013?11?13? 21:58
???: [email-masked]
??: [nvo3] Poll for WG adoption and IPR check for draft-narten-nvo3-arch-01.txt

Regards

Matthew and Benson