Subject: [nvo3] Draft NVO3 WG Charter

From: Thomas Narten <[email-masked]>
Date: Fri, 17 Feb 2012 09:51:55 -0500

Below is a draft charter for this effort. One detail is that we
started out calling this effort NVO3 (Network Virtualization Over L3),
but have subsequently realized that we should not focus on just "over
L3". One goal of this effort is to develop an overlay standard that
works over L3, but we do not want to restrict ourselves only to "over
L3". The framework and architecture that we are proposing to work on
should be applicable to other overlays as well (e.g., L2 over
L2). This is (hopefully) captured in the proposed charter.

Comments?

Thomas

NVO: Network Virtualization Overlays 

Support for multi-tenancy has become a core requirement of data
centers, especially in the context of data centers which include
virtualized servers known as virtual machines (VMs).  With
multi-tenancy, a data center can support the needs of many thousands
of individual tenants, ranging from individual groups or departments
within a single organization all the way up to supporting thousands of
individual customers.  A key multi-tenancy requirement is traffic
isolation, so that a tenant's traffic (and internal address usage) is
not visible to any other tenant and does not collide with addresses
used within the data center itself.  Such isolation can be achieved by
creating and assigning one or more virtual networks to each tenant
such that traffic within a virtual network is isolated from traffic in
other virtual networks.

Tenant isolation is primarily achieved today within data centers using
Ethernet VLANs. But the 12-bit VLAN tag field isn't large enough to
support existing and future needs. A number of approaches to extending
VLANs and scaling L2s have been proposed or developed, including IEEE
802.1ah Shortest Path Bridging (SPB) and TRILL (with the proposed
fine-grained labeling extension).  At the L3 (IP) level, VXLAN and
NVGRE have also been proposed. As outlined in
draft-narten-nvo3-overlay-problem-statement-01.txt, however, existing
L2 approaches are not satisfactory for all data center operators,
e.g., larger data centers that desire to keep L2 domains small or push
L3 further into the data center (e.g., all the way to top-of-rack
switches). Furthermore, there is a desire to decouple the
configuration of the data center network from the configuration
associated with individual tenant applications and to seamlessly and
rapidly update the network state to handle live VM migrations or fast
spin-up and spin-down of new tenant VMs (or servers). Such tasks are
complicated by the need to simultaneously reconfigure and update data
center network state (e.g., VLAN settings on individual switches).

This WG will develop an approach to multi-tenancy that does not rely
on any underlying L2 mechanisms to support multi-tenancy. In
particular, the WG will develop an approach where multitenancy is
provided at the IP layer using an encapsulation header that resides
above IP. This effort is explicitly intended to leverage the interest
in L3 overlay approaches as exemplified by VXLAN
(draft-mahalingam-dutt-dcops-vxlan-00.txt) and NVGRE
(draft-sridharan-virtualization-nvgre-00.txt).

Overlays are a form of "map and encap", where an ingress node maps the
destination address of an arriving packet (e.g., from a source tenant
VM) into the address of an egress node to which the packet can be
tunneled to. The ingress node then encapsulates the packet in an outer
header and tunnels it to the egress node, which decapsulates the
packet and forwards the original (unmodified) packet to its ultimate
destination (e.g., a destination tenant VM). All map-and-encap
approaches must address two issues: the encapsulation format (i.e.,
the contents of the outer header) and how to distribute and manage the
mapping tables used by the tunnel end points.

The first area of work concerns encapsulation formats. This WG will
develop requirements and desirable properties for any encapsulation
format. Given the number of already existing encapsulation formats,
it is not an explicit goal of this effort to choose exactly one format
or to develop yet another new one.

A second work area is in the control plane, which allows an ingress
node to map the "inner" (tenant VM) address into an "outer"
(underlying transport network) address in order to tunnel a packet
across the data center. We propose to develop two control planes. One
control plane will use a learning mechanism similar to IEEE 802.1D
learning, and could be appropriate for smaller data centers. A second,
more scalable control plane would be aimed at large sites, capable of
scaling to hundreds of thousands of nodes. Both control planes will
need to handle the case of VMs moving around the network in a dynamic
fashion, meaning that they will need to support tunnel endpoints
registering and deregistering mappings as VMs change location and
ensuring that out-of-date mapping tables are only used for short
periods of time. Finally, the second control plane must also be
applicable to geographically dispersed data centers.

Although a key objective of this WG is to produce a solution that
supports an L2 over L3 overlay, an important goal is to develop a
"layer agnostic" framework and architecture, so that any specific
overlay approach can reuse the output of this working group. For
example, there is no inherent reason why the same framework could not
be used to provide for L2 over L2 or L3 over L3. The main difference
would be in the address formats of the inner and outer headers and the
encapsulation header itself.

Finally, some work may be needed in connecting an overlay network with
traditional L2 or L3 VPNs (e.g., VPLS). One approach appears straight
forward, in that there is a clear boundary between a VPN device and
the edge of an overlay network. Packets forwarded across the boundary
would simply need to have the tenant identifier on the overlay side
mapped into a corresponding VPN identifier on the VPN
side. Conceptually, this would appear to be analogous to what is done
already today when interfacing between L2 VLANs and VPNs.

The specific deliverables for this group include:

1) Finalize and publish the overall problem statement as an
Informational RFC (basis:
draft-narten-nvo3-overlay-problem-statement-01.txt)

2) Develop requirements and desirable properties for any encapsulation
format, and identify suitable encapsulations. Given the number of
already existing encapsulation formats, it is not an explicit goal of
this effort to choose exactly one format or to develop a new one.

3) Produce a Standards Track control plane document that specifies how
to build mapping tables using a "learning" approach. This document is
expected to be short, as the algorithm itself will use a mechanism
similar to IEEE 802.1D learning.

4) Develop requirements (and later a Standards Track protocol) for a
more scalable control plane for managing and distributing the mappings
of "inner" to "outer" addresses. We will develop a reusable framework
suitable for use by any mapping function in which there is a need to
map "inner" to outer addresses. Starting point:
draft-kreeger-nvo3-overlay-cp-00.txt

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 07:00:50 -0800

Thomas,

This basically is a re-statement of what is done by L3/L2 VPNs.  It might be useful to do a gap analysis of these existing technologies, in particular E-VPNs (http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04), before asserting that something new is required.

Thanks,

John

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Narten
> Sent: Friday, February 17, 2012 6:52 AM
> To: [email-masked]
> Subject: [nvo3] Draft NVO3 WG Charter
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 11:04:11 -0500

John,

> This basically is a re-statement of what is done by L3/L2 VPNs.  It'
> might be useful to do a gap analysis of these existing technologies,
> in particular E-VPNs (http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04),
> before asserting that something new is required.
BGP and MPLS are non-starters for a lot of datacenter-internal networks.
Some of the more important NVO deployment scenarios involve map-and-encap
in a hypervisor software network switch.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754
----------------------------------------------------
________________________________
From: [email-masked] [[email-masked]] On Behalf Of John E Drake [[email-masked]]
Sent: Friday, February 17, 2012 10:00 AM
To: Thomas Narten; [email-masked]
Cc: Ronald Bonica; Nitin Bahadur; Adrian Farrel
Subject: Re: [nvo3] Draft NVO3 WG Charter

Thomas,

Thanks,

John

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 08:13:34 -0800

Comments inline

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, February 17, 2012 8:04 AM
> To: John E Drake; [email-masked]; [email-masked]
> Subject: RE: [nvo3] Draft NVO3 WG Charter
> 
> John,
> 

[JD]  This is an assertion.  It is also the misses the fact that MPLS is only required to mux/demux packets at the edges of the VPN network. 

> Some of the more important NVO deployment scenarios involve map-and-
> encap in a hypervisor software network switch.

[JD]  Your point eludes me.   

> This basically is a re-statement of what is done by L3/L2 VPNs.  It
> might be useful to do a gap analysis of these existing technologies, in
> particular E-VPNs (http://tools.ietf.org/html/draft-raggarwa-sajassi-
> l2vpn-evpn-04), before asserting that something new is required.
> 
> Thanks,
> 
> John
>

---

From: Thomas Narten <[email-masked]>
Date: Fri, 17 Feb 2012 11:10:40 -0500

John,

> This basically is a re-statement of what is done by L3/L2 VPNs.

I think this comment is a bit unfair. We had a session in Taipei where
this issue was discussed. The push for this is coming with support
from data center operators themselves, not from the VPN side (and the
"provider-provisioned" VPN side in particular, which is where both
L2VPN and L3VPN have historically worked from and what their current
charters continue to heavily emphasize is their starting point).

> It might be useful to do a gap analysis of these existing
>  technologies, in particular E-VPNs
>  (http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04),

I think it has been established that while existing technologies could
be made to work in data centers in a *theoretical* sense, the Taipei
session showed that there were operators who are not entirely happy
with that approach and wanted the IETF to pursuse what we are
proposing.

Thomas

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 08:20:08 -0800

Comments inline

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Friday, February 17, 2012 8:11 AM
> To: John E Drake
> Cc: [email-masked]; Ronald Bonica; Nitin Bahadur; Adrian Farrel

[JD]  I was describing what the technologies provide, not who is deploying them.  The data center operators are the ones who should be doing the gap analysis I suggested, below.

> with that approach and wanted the IETF to pursuse what we are

[JD]  I don't recall anything close to consensus on this statement.

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 11:31:03 -0500

Hi John,

> [JD]  This is an assertion.  It is also the misses the fact that MPLS
> is only required to mux/demux packets at the edges of the VPN network.

Indeed it is, but I stand by it.  The interesting "edges of the VPN

network" for NVO include datacenter ToR switches, datacenter access

switches and hypervisor softswitches - there are plenty of examples of

these for which MPLS and BGP are non-starters.

I suggest reading the NVGRE and VXLAN drafts for more context:

   http://tools.ietf.org/html/draft-sridharan-virtualization-nvgre-00

   http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-00

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754
----------------------------------------------------
________________________________
From: John E Drake [[email-masked]]
Sent: Friday, February 17, 2012 11:13 AM
To: Black, David; [email-masked]; [email-masked]
Cc: Ronald Bonica; Nitin Bahadur; Adrian Farrel
Subject: RE: [nvo3] Draft NVO3 WG Charter

Comments inline

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Feb 2012 08:40:44 -0800

< metaphor mix >

there is more than one way to skin a cat6.  let's not argue over what
the elephant is, please.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Feb 2012 17:49:35 +0100

Hi John,

> is only required to mux/demux packets at the edges of the VPN

Let me observe that at least in some data centers the "edge of VPN" is 
likely to be the hypervisor (with OVS) on the host machine. The reason 
being that VMs residing on such host may belong to different customers.

Could you describe how existing technologies would work in such case ? 
Are you proposing to turn each host to be a PE ? Or are you worried that 
host has not sufficient forwarding power to act as PE so one would need 
a dedicated hardware to be a PE ?

Best regards,
R.

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 09:02:06 -0800

Robert,

As an example, please see: http://datatracker.ietf.org/doc/draft-marques-l3vpn-end-system/

Thanks,

John  

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Friday, February 17, 2012 8:50 AM
> To: John E Drake
> Cc: [email-masked]; [email-masked]; [email-masked]; Ronald
> Bonica; Nitin Bahadur; Adrian Farrel

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Feb 2012 18:08:28 +0100

Hi John,

But that is perfect example of the draft for nvo3 ! Especially since 
L3VPN WG is apparently closing down ....

Since you brought it ... are you and your affiliation support 
draft-marques-l3vpn-end-system ?

Regards,
R.

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 09:08:09 -0800

David,

I have read all of the NV and DC drafts that have been published to this point and the acting assumption in virtually all of them is that something new is needed.  I know it is much more fun to work on something new, but a gap analysis of existing technologies should be undertaken before blithely proceeding.

Thanks,

John 

> Sent: Friday, February 17, 2012 8:31 AM
> It'
> > [email-masked]<mailto:[email-masked]>        Mobile: +1
> (978)
> > 394-7754
> > From: [email-masked] [[email-masked]] On Behalf Of John
> E
> > Drake [[email-masked]]
> in
> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > Of
> > > Thomas Narten

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Feb 2012 18:21:32 +0100

>>      http://tools.ietf.org/html/draft-sridharan-virtualization-nvgre-00

Actually to be fare on this one I have to agree with John.

10 authors and 16 pages to define a GRE key is an amazing achievement.

Especially since GRE header with MPLS label behind it is used 
effectively for many years now to achieve the same level of 
virtualization in number of shipping applications.

Reference: RFC4023

Regards,
R.

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 09:24:27 -0800

Robert,

Comments inline.

Thanks,

John

> Sent: Friday, February 17, 2012 9:08 AM
> To: John E Drake
> Cc: [email-masked]; Ronald Bonica; [email-masked];
> [email-masked]; Adrian Farrel; Nitin Bahadur
> But that is perfect example of the draft for nvo3 !

[JD]  I'm not sure what you mean by nv03.  If you mean a yet to be chartered WG group that develops enhancements to existing IETF technologies to address requirements unique to the data center, then I would agree with you.  Btw, are you aware of the VM mobility draft:  http://datatracker.ietf.org/doc/draft-raggarwa-data-center-mobility/? 

> Especially since

[JD]  The draft has a Juniper co-author.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Feb 2012 18:53:09 +0100

Hi John,

> Btw, are you aware of the VM mobility
> draft:http://datatracker.ietf.org/doc/draft-raggarwa-data-center-mobility/?

The most interesting there is the section 6.2 attempting to propose ways 
to avoid inter-dc triangular routing after VM migration.

However I am not keen on host routes injection even in the limited 
radius (which may not be possible in hybrid cloud cases) nor NHRP 
proposal from section 6.2.3.

IMHO much better class of solutions are already documented in LISP 
and/or ILNP documents. They solve the same problem in much more elegant 
way. Note that those can be deployed within one's administration domain 
boundaries only or extend beyond given domain (for example to given 
domain's customers).

Many thx,
R.

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 13:15:51 -0500

Hi John,

And I'm likewise familiar with the marques-l3vpn-end-system draft.

Perhaps we can tone this down a bit.  I read your initial message as
effectively asserting that there is nothing to do because VPN technology
already solves all the problems in this entire space.  If you'll agree
that this wasn't what you meant, that ought to be a basis for a more
reasonable discussion.

In turn, I'll offer to lead off with some of the gaps in the -marques-
draft (yes, there has been some gap analysis of that draft), plus some
more explanation of Robert's point about how the proposed nvo (or nvo3)
work is complementary to that draft.

Can we do that?

Thanks,
--David
Sent: Friday, February 17, 2012 12:08 PM

David,

Thanks,

John

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 13:26:59 -0500

Hi Robert,

> Actually to be fare on this one I have to agree with John.
I think you meant "fair" as opposed to "fare", although I'll be happy to collect
the latter from you ... or your spell-checker :-).

> 10 authors and 16 pages to define a GRE key is an amazing achievement.

I referenced both drafts for context, and I'm not going to argue with that
"major achievement" characterization.  Both drafts are -00 versions that will
need more work to fill in details.  The reason to look at those drafts now
is that they provide additional insight into the problem space, which may
help illuminate why a "just use a VPN" approach doesn't solve all the problems
of interest.

Thanks,
--David

________________________________
From: Robert Raszuk [[email-masked]]
Sent: Friday, February 17, 2012 12:21 PM
To: Black, David
Cc: John E Drake; [email-masked]; [email-masked]; Ronald Bonica; Nitin Bahadur; Adrian Farrel
Subject: Re: [nvo3] Draft NVO3 WG Charter

Reference: RFC4023

Regards,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Feb 2012 19:35:56 +0100

Hi,

<Apologies for past and future spelling mistakes>

> Both drafts are -00 versions that will need more work to fill in details.  The reason to look at those drafts now

Actually maybe nvo3 could even sign under "just use a VPN" .. however I 
think that the term VPN has a very broad definition. And as said before 
I am with you that there is area to be covered where current L2VPN and 
L3VPN WG documents do not well cover it.

The comment about nvgre is not to dismiss the need for new WG. It is 
solely to point out that we should reuse as much as possible in anything 
we do. That said I think multiplexing and demultiplexing with GRE 
encapsulation and some tag or label behind it IMHO is a solved problem 
which we may not need to reinvent again.

Many thx,
R.

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 10:34:55 -0800

Robert,

Although we might disagree wrt the applicability of certain technologies, such as LISP, I think you are reinforcing my point that there are lots of existing IETF technologies that should be evaluated.

Thanks,

John 

> Sent: Friday, February 17, 2012 9:53 AM
> To: John E Drake

---

From: John E Drake <[email-masked]>
Date: Fri, 17 Feb 2012 10:44:10 -0800

David,

Please re-read my original email.

What I said was that the charter looked very much like the L3/L2 VPN charters and that it would be useful to perform a gap analysis of these technologies against the DC specific requirements before starting to work on something new.

I didn't say there was no new work to be done.

Thanks,

John

> Sent: Friday, February 17, 2012 10:16 AM
> I have read all of the NV and DC drafts that have been published to
> this point and the acting assumption in virtually all of them is that
> something new is needed.  I know it is much more fun to work on
> something new, but a gap analysis of existing technologies should be
> undertaken before blithely proceeding.
>
> Thanks,
>
> John
>
> > > > within a single organization all the way up to supporting
> thousands
> > > of
> > > > isolation, so that a tenant's traffic (and internal address
> usage)
> > is
> > > > used within the data center itself.  Such isolation can be
> achieved
> > > by
> > > > such that traffic within a virtual network is isolated from
> traffic
> > > in
> > > > other virtual networks.
> > > >
> > > > Overlays are a form of "map and encap", where an ingress node
> maps
> > > the
> > > > the contents of the outer header) and how to distribute and
> manage
> > > the
> > > > more scalable control plane would be aimed at large sites,
> capable
> > of
> > > > would be in the address formats of the inner and outer headers
> and
> > > the
> > > > already existing encapsulation formats, it is not an explicit
> goal
> > of
> > > > 3) Produce a Standards Track control plane document that
> specifies
> > > how
> > > > to build mapping tables using a "learning" approach. This
> document
> > is
> > > > 4) Develop requirements (and later a Standards Track protocol)
> for
> > a

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 17 Feb 2012 19:10:11 +0000

> L2). This is (hopefully) captured in the proposed charter.

This worries me. It is going to be difficult to avoid getting into
a situation of boiling oceans here, and we need to make sure we
start in the right place. That said I think that if a simple general
solution can be designed as opposed to yet another application
specific encapsulation, that would be a great service to the industry
over the long term.
> (draft-sridharan-virtualization-nvgre-00.txt).

The WG will need to consider the operations that it wants to
encode in the "transport" layer. Encapsulation, delivery and
multiplexing are compulsory, but are there others?

I am NOT proposing to push MPLS here, but think for a little
while about the subtly of that small header which encodes
not (encap + delivery + mux), but a set of opaque
instructions agreed between peers. So think very carefully
about what you want hard coded and self describing and
what flexibility you want to provide.

> scaling to hundreds of thousands of nodes.
The WG clearly needs to solve both problems, but I think that
it is too early to say whether you need two control planes or not
for scaling. However the concept of mandating that the encapsulation
layer be decoupled from the control protocol adds significantly
to the utility of the encapsulation. The WG needs to bare in mind
that there may in the long run be many reasons to create
additional control protocols besides scaling.

> Both control planes will

I think that we need to start by figuring out the properties that are
needed and the figure out whether we need one, two of some other
number of control protocols.

> already today when interfacing between L2 VLANs and VPNs.
Remember that we broke this up into a number of work packages
in order to scale the IETF to the problem.
> draft-narten-nvo3-overlay-problem-statement-01.txt)
OK

However consider producing a framework next so that
work can proceed in parallel on the next topics with the
framework acting as glue to hold the independent components
together.
> this effort to choose exactly one format or to develop a new one.
OK. When we get here we will know whether we use one or
more existing or a new one. We need to work really hard to
make sure the decision here is objective.
> similar to IEEE 802.1D learning.
This is not how we should define this in the charter "similar to
802.1D" sets all sorts of expectations concerning the design
and takes us into all sorts of territory we do not want to go to.

At this stage only the first sentence is needed.
> draft-kreeger-nvo3-overlay-cp-00.txt
Conceptually the first sentence is correct, although you assume a
lot by talking about inner and outer addresses.
The WG will decide the starting point when it is formed.

For the purposes of going forward we need to talk about scalability
in the control plane, but I am not sure exactly how much solution
oriented detail is appropriate at this stage.

- Stewart

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 14:23:13 -0500

John,

Thank you - beyond the operational concerns about extending MPLS
and BGP into data centers, here are a few gaps in the
marques-l3vpn-end-system draft:

(1) No multicast support.  Among the important examples for VMs is
that the network load balancer in at least Windows 2000 and 2003
is preferably run using multicast (otherwise it wants to use
duplicate unicast MACs on the same LAN/VLAN).  There are other
examples.

(2) A "Host OS" is required in the hypervisor to solve some
routing problems.  At least one important hypervisor, ESXi,
doesn't have a "Host OS" - the closest location for this
functionality is the softswitch in the hypervisor.  I suspect that
the routing functionality proposed for the Host OS can be moved
into the softswitch, so this may be a very minor gap.

(3) The use of a single link-local address may cause problems
with the usual technique of issuing a gratuitous ARP (or RARP)
after a VM move - the design intent of that is to get the L2
bridges to learn that the VM's network interfaces have moved.
In particular, the draft asserts the following without
explaining how it is done:

   When a particular VM "moves" from one physical end-system
   to another, its respective VPN address will be advertised
   by the new system and that notification propagated to all
   attachment points of that VPN.

This is why NVO(3) is potentially complementary.  Defining
a protocol that can help by communicating the VM "move" to a
VPN end point that is in a softswitch or an external switch is
among the things that the NVO effort proposes to undertake.  It
would be desirable for this change communication protocol to
be common across all (or at least a good number of) the involved
networking technologies, as opposed to restricted to some L3VPNs,
so that the protocol only has to be implemented once for each
hypervisor.

Thanks,
--David

Sent: Friday, February 17, 2012 1:44 PM

David,

Thanks,

John

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 17 Feb 2012 13:39:27 -0800

Dave,

>    attachment points of that VPN.

Please read section 5 of the draft.

> so that the protocol only has to be implemented once for each

Wrt the claim that "NVO(3) is potentially complementary", let me
point out that the currently proposed charter has plenty of other
stuff, besides defining the protocol to communicate the VM "move".

Wrt the protocol that would allow to communicate the VM "move" to
the network, I agree with you that it is indeed desirable. Having
said this, should we consider the work going on now in IEEE on VDP ?
Should we consider XMPP (along the lines proposed by
marques-l3vpn-end-system draft) ?

Yakov.

> What I said was that the charter looked very much like the L3/L2 VPN charters
 and that it would be useful to perform a gap analysis of these technologies ag
ainst the DC specific requirements before starting to work on something new.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 17 Feb 2012 16:58:59 -0500

> specific encapsulation, that would be a great service to the industry

Developing a general solution is exactly the thinking. There is
absolutely no intention at all to boil the ocean. That is the last
thing I want to be involved in.

But note that we have the challenge that if we narrowly say we are
doing just L2 over L3, we get sent over to L2VPN right away. Or, if we
say just L3 over L3, L3VPN applies.

Personally (though I'm probably not speaking for everyone) I'd like to
see this effort explicitely support both (and only) L2 over L3 and L3
over L3. I think that most workloads today (and going forward) are
entirely IP based. For those, why not just dispense with L2 assume all
traffic will be IP?

But there will still be some workloads that do depend on L2
communication. For those, L2 over L3 needs to be supported.

But IMO a common approach should be able to support both.

Clearly there will be some differences in the control plane (e.g.,
different address familiies) but most of what is needed for either
solution can be the same. And of course packet encapsulations would
also depend on the specific technology.

So when I generalize about which layers to use, I'm not at all looking
to do "anything over anything". But I don't want to develop technology
that isn't easily extendible or reusable by those that have a need to
do so (if justified).

But such extensibility would be out of scope for this WG>

> multiplexing are compulsory, but are there others?

As far as I know at this point, I'm not aware of any others. The key
one is demultiplexing, i.e., identifying the tenant the packet belongs
to.

> what flexibility you want to provide.

Would welcome some further discussions or pointers as to what in MPLS
you specifically are referring to.

> additional control protocols besides scaling.

At least among the folk I have talked to that are interested in this
work, it seems generally recognized that just using "learning" to
distribute address mappings is not viewed as being
reliable/predictable/robust enough for very large data centers. One of
the goals here is scaling to at least 100K physical machines, and if
we assume 10 VMs per physical machine (a low estimate going forward),
that's a million edge VMs.

Can we make a learning based approach scale to that size? Some folk
will probably argue yes, but others will say "maybe, but why bother
when we know we can do better."

> number of control protocols.

I would hope we don't need more than two!

> framework acting as glue to hold the independent components

OK.

> make sure the decision here is objective.

Yes.

> At this stage only the first sentence is needed.

OK. But when we talk about learning, it is the 802.1D learning that
defines the approach. If we just saying "learning", do folk generally
know what that means?

> oriented detail is appropriate at this stage.

FYI, draft-kreeger-nvo3-overlay-cp-00.txt says:

   5.   Highly scalable.  This means scaling to hundreds of thousands of
        OBPs and several million VNs within a single administrative
        domain.  As the number of OBPs and/or VNs within a data center
        grows, the protocol overhead at any one OBP should not increase
        significantly.

with OBP and VNs defined as:

   OBP:  Overlay Boundary Point.  This is a network entity that is on
      the edge boundary of the overlay.  It performs encapsulation to
      send packets to other OBPs across an Underlying Network for
      decapsulation.  An OBP could be implemented as part of a virtual
      switch within a hypervisor, a physical switch or router, a Network
      Service Appliance or even be embedded within an End Station.

   VN:  Virtual Network.  This is one instance of a virtual overlay
      network.  Two Virtual Networks are isolated from one another and
      may use overlapping addresses.

Thomas

---

From: Larry Kreeger <[email-masked]>
Date: Fri, 17 Feb 2012 14:10:41 -0800

Hi Thomas,

Thanks for getting the ball rolling on this.  I do have a few comments which
are inline below.

Thanks, Larry

> center network state (e.g., VLAN settings on individual switches).

Regarding the last two sentences above, I just want to be clear that there
is a difference between data center network "configuration" (typically
meaning manual human action), and dynamic network state that is established
using a protocol exchange.  I bring this up because I don't think the goals
of this effort are to necessarily remove all knowledge/state/visibility of
the virtual networks from the data center networking equipment (although
this could be done).  To me, the goals of the protocols/protocol reqs in
this charter must allow for the ability of the data center networking
equipment to terminate the overlay tunnels on the behalf of attached devices
that either cannot perform the encap/decap function, or if they can, do so
at significantly reduced performance.  These protocols are needed to
eliminate the manual configuration of the data center network equipment as
the virtual network requirements dynamically change.  This ties to my next
comment further below.

> draft-kreeger-nvo3-overlay-cp-00.txt

This starting point draft lists protocol requirements beyond the "inner" to
"outer" address mappings mentioned in point 4 above.  The remainder of the
protocol functions in the referenced draft are important for allowing the
data center networking equipment to be aware of the dynamically changing
virtual network requirements and other information needed to allow the
networking equipment to perform the encap/decap function on the behalf of
the end devices which need access the virtual networks - without resorting
to manual configuration.  I think the other protocol functions mentioned in
the draft should be explicitly called out in the WG charter in addition to
the important "inner" to "outer" mapping protocol.

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 17 Feb 2012 15:26:25 -0800 (PST)

I think this is a great charter, but, do we really need to bother with L2 
over L2 type of overlays here? Does anybody actually plan on building 
something like that? 

-igor

On Fri, 17 Feb 2012, Thomas Narten wrote:

:: Below is a draft charter for this effort. One detail is that we
:: started out calling this effort NVO3 (Network Virtualization Over L3),
:: but have subsequently realized that we should not focus on just "over
:: L3". One goal of this effort is to develop an overlay standard that
:: works over L3, but we do not want to restrict ourselves only to "over
:: L3". The framework and architecture that we are proposing to work on
:: should be applicable to other overlays as well (e.g., L2 over
:: L2). This is (hopefully) captured in the proposed charter.
:: 
:: Comments?
:: 
:: Thomas
:: 
:: NVO: Network Virtualization Overlays 
:: 
:: Support for multi-tenancy has become a core requirement of data
:: centers, especially in the context of data centers which include
:: virtualized servers known as virtual machines (VMs).  With
:: multi-tenancy, a data center can support the needs of many thousands
:: of individual tenants, ranging from individual groups or departments
:: within a single organization all the way up to supporting thousands of
:: individual customers.  A key multi-tenancy requirement is traffic
:: isolation, so that a tenant's traffic (and internal address usage) is
:: not visible to any other tenant and does not collide with addresses
:: used within the data center itself.  Such isolation can be achieved by
:: creating and assigning one or more virtual networks to each tenant
:: such that traffic within a virtual network is isolated from traffic in
:: other virtual networks.
:: 
:: Tenant isolation is primarily achieved today within data centers using
:: Ethernet VLANs. But the 12-bit VLAN tag field isn't large enough to
:: support existing and future needs. A number of approaches to extending
:: VLANs and scaling L2s have been proposed or developed, including IEEE
:: 802.1ah Shortest Path Bridging (SPB) and TRILL (with the proposed
:: fine-grained labeling extension).  At the L3 (IP) level, VXLAN and
:: NVGRE have also been proposed. As outlined in
:: draft-narten-nvo3-overlay-problem-statement-01.txt, however, existing
:: L2 approaches are not satisfactory for all data center operators,
:: e.g., larger data centers that desire to keep L2 domains small or push
:: L3 further into the data center (e.g., all the way to top-of-rack
:: switches). Furthermore, there is a desire to decouple the
:: configuration of the data center network from the configuration
:: associated with individual tenant applications and to seamlessly and
:: rapidly update the network state to handle live VM migrations or fast
:: spin-up and spin-down of new tenant VMs (or servers). Such tasks are
:: complicated by the need to simultaneously reconfigure and update data
:: center network state (e.g., VLAN settings on individual switches).
:: 
:: This WG will develop an approach to multi-tenancy that does not rely
:: on any underlying L2 mechanisms to support multi-tenancy. In
:: particular, the WG will develop an approach where multitenancy is
:: provided at the IP layer using an encapsulation header that resides
:: above IP. This effort is explicitly intended to leverage the interest
:: in L3 overlay approaches as exemplified by VXLAN
:: (draft-mahalingam-dutt-dcops-vxlan-00.txt) and NVGRE
:: (draft-sridharan-virtualization-nvgre-00.txt).
:: 
:: Overlays are a form of "map and encap", where an ingress node maps the
:: destination address of an arriving packet (e.g., from a source tenant
:: VM) into the address of an egress node to which the packet can be
:: tunneled to. The ingress node then encapsulates the packet in an outer
:: header and tunnels it to the egress node, which decapsulates the
:: packet and forwards the original (unmodified) packet to its ultimate
:: destination (e.g., a destination tenant VM). All map-and-encap
:: approaches must address two issues: the encapsulation format (i.e.,
:: the contents of the outer header) and how to distribute and manage the
:: mapping tables used by the tunnel end points.
:: 
:: The first area of work concerns encapsulation formats. This WG will
:: develop requirements and desirable properties for any encapsulation
:: format. Given the number of already existing encapsulation formats,
:: it is not an explicit goal of this effort to choose exactly one format
:: or to develop yet another new one.
:: 
:: A second work area is in the control plane, which allows an ingress
:: node to map the "inner" (tenant VM) address into an "outer"
:: (underlying transport network) address in order to tunnel a packet
:: across the data center. We propose to develop two control planes. One
:: control plane will use a learning mechanism similar to IEEE 802.1D
:: learning, and could be appropriate for smaller data centers. A second,
:: more scalable control plane would be aimed at large sites, capable of
:: scaling to hundreds of thousands of nodes. Both control planes will
:: need to handle the case of VMs moving around the network in a dynamic
:: fashion, meaning that they will need to support tunnel endpoints
:: registering and deregistering mappings as VMs change location and
:: ensuring that out-of-date mapping tables are only used for short
:: periods of time. Finally, the second control plane must also be
:: applicable to geographically dispersed data centers.
:: 
:: Although a key objective of this WG is to produce a solution that
:: supports an L2 over L3 overlay, an important goal is to develop a
:: "layer agnostic" framework and architecture, so that any specific
:: overlay approach can reuse the output of this working group. For
:: example, there is no inherent reason why the same framework could not
:: be used to provide for L2 over L2 or L3 over L3. The main difference
:: would be in the address formats of the inner and outer headers and the
:: encapsulation header itself.
:: 
:: Finally, some work may be needed in connecting an overlay network with
:: traditional L2 or L3 VPNs (e.g., VPLS). One approach appears straight
:: forward, in that there is a clear boundary between a VPN device and
:: the edge of an overlay network. Packets forwarded across the boundary
:: would simply need to have the tenant identifier on the overlay side
:: mapped into a corresponding VPN identifier on the VPN
:: side. Conceptually, this would appear to be analogous to what is done
:: already today when interfacing between L2 VLANs and VPNs.
:: 
:: The specific deliverables for this group include:
:: 
:: 1) Finalize and publish the overall problem statement as an
:: Informational RFC (basis:
:: draft-narten-nvo3-overlay-problem-statement-01.txt)
:: 
:: 2) Develop requirements and desirable properties for any encapsulation
:: format, and identify suitable encapsulations. Given the number of
:: already existing encapsulation formats, it is not an explicit goal of
:: this effort to choose exactly one format or to develop a new one.
:: 
:: 3) Produce a Standards Track control plane document that specifies how
:: to build mapping tables using a "learning" approach. This document is
:: expected to be short, as the algorithm itself will use a mechanism
:: similar to IEEE 802.1D learning.
:: 
:: 4) Develop requirements (and later a Standards Track protocol) for a
:: more scalable control plane for managing and distributing the mappings
:: of "inner" to "outer" addresses. We will develop a reusable framework
:: suitable for use by any mapping function in which there is a need to
:: map "inner" to outer addresses. Starting point:
:: draft-kreeger-nvo3-overlay-cp-00.txt
:: 
:: _______________________________________________
:: nvo3 mailing list
:: [email-masked]
:: https://www.ietf.org/mailman/listinfo/nvo3
:: 

--------------------+----------------------+------------------
   Igor Gashinsky   | Network Architecture | Yahoo! Inc.
 [email-masked] |  cell 917.807.2213   | Do You... Yahoo?
--------------------+----------------------+------------------

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 18 Feb 2012 00:43:24 +0100

Hi Igor,

While I agree that we should focus on L3 for transport I guess QinQ has 
some applicability for both traditional networks as well as OF based. I 
would keep it in the charter.

Best,
R.

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 17 Feb 2012 16:58:29 -0700

L2 over L2 overlay already exist in some DC's via SPBm. 

--
Paul Unbehagen

Sent from my iPhone

---

From: <[email-masked]>
Date: Fri, 17 Feb 2012 19:05:46 -0500

There was also interest expressed @ the Taipei L2VPN meeting in data center usage of SPB/PBB.  That's another potential reason to leave L2 over L2 in scope.

Thanks, --David +++ Sent from BlackBerry +++

----- Original Message -----
From: Robert Raszuk [mailto:[email-masked]]
Sent: Friday, February 17, 2012 06:43 PM
To: Igor Gashinsky <[email-masked]>
Cc: Thomas Narten <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: [nvo3] Draft NVO3 WG Charter

Hi Igor,

Best,
R.

---

From: Larry Kreeger <[email-masked]>
Date: Fri, 17 Feb 2012 16:22:07 -0800

Yakov,

> said this, should we consider the work going on now in IEEE on VDP ?

I'm not an expert on VDP, but my understanding is that (since it is IEEE),
it would only work for a layer 2 "inner" address.  Since we want the
protocol to be layer agnostic (i.e. also support L3 VM addresses), it would
need to be extended in some way...and it is not an IETF protocol to extend.

> marques-l3vpn-end-system draft) ?

Consider away.  One reason we put a long list of protocol characteristic
requirements within draft-kreeger-nvo3-overlay-cp-00.txt was so that all
proposed candidate protocols could be evaluated against them so we can
choose the most appropriate one.  I don't claim to know much about XMPP
either, but my first reaction is...what does an protocol designed for chat
have to do with this problem space, and isn't it a bit heavy weight for the
purposes?

---

From: Ping Pan <[email-masked]>
Date: Sat, 18 Feb 2012 00:36:52 +0000

> have to do with this problem space, and isn't it a bit heavy weight for the

Not at all. While many seem to be re-inventing the wheels in labels and L2, we see it as the need to have the applications talking to the existing network gears to forward the IP flows. XMPP is just one of the mature application-friendly protocols.

Regards,

Ping

---

From: "Pat Thaler" <[email-masked]>
Date: Sat, 18 Feb 2012 01:14:30 +0000

802.1Q is probably a better example for a reference on what the "learning" might be like. 802.1D didn't have any concept of a virtual network and flooded to the whole network. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, February 17, 2012 1:59 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Draft NVO3 WG Charter

OK.

Yes.

Thomas

---

From: "Pat Thaler" <[email-masked]>
Date: Sat, 18 Feb 2012 01:34:44 +0000

VDP (in IEEE P802.1Qbg) does not assume that the VM is defined by a layer 2 "inner" address. It identifies a VSI by a VSI ID which can be an IP address (v4 or v6), MAC address, UUID (RFC 4122) or locally defined format. It can also carry filter information - several formats are currently defined for filter info - some of them have a 32-bit GroupID to cover the cases where a 12 bit VID isn't large enough to identify the virtual network. That was added based on the need to support a larger identifier when PBB was used, but it could serve for other 24 bit tenant identifiers.

VDP is a protocol that runs between the end system (e.g. a hypervisor) to an adjacent bridge so it serves to communicate from the end system to the device on the edge of the network that a VM is moving, but it doesn't cover how to propagate news of that move across the network. It might be a piece of the broader solution but it isn't a complete solution.

Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Larry Kreeger
Sent: Friday, February 17, 2012 4:22 PM
To: Yakov Rekhter; David Black
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] Draft NVO3 WG Charter

Yakov,

>> John,
<snip>

---

From: Larry Kreeger <[email-masked]>
Date: Fri, 17 Feb 2012 18:03:50 -0800

Hi Pat,

Thanks for educating me about VDP.  Based on what you wrote, it sounds like
it might indeed by useful for the VM attach/detach protocol.  One concern is
that you seem to say the VSI ID can be one of a list of things you mention
that sound promising, but can it support multiple of these?  For example,
the encap/decap device may want to know a Virtual Network's UUID and the
VM's MAC and the VM's IP.

Thanks, Larry

On 2/17/12 5:34 PM, "Pat Thaler" <[email-masked]> wrote:

> VDP (in IEEE P802.1Qbg) does not assume that the VM is defined by a layer 2
> "inner" address. It identifies a VSI by a VSI ID which can be an IP address
> (v4 or v6), MAC address, UUID (RFC 4122) or locally defined format. It can
> also carry filter information - several formats are currently defined for
> filter info - some of them have a 32-bit GroupID to cover the cases where a 12
> bit VID isn't large enough to identify the virtual network. That was added
> based on the need to support a larger identifier when PBB was used, but it
> could serve for other 24 bit tenant identifiers.
> 
> VDP is a protocol that runs between the end system (e.g. a hypervisor) to an
> adjacent bridge so it serves to communicate from the end system to the device
> on the edge of the network that a VM is moving, but it doesn't cover how to
> propagate news of that move across the network. It might be a piece of the
> broader solution but it isn't a complete solution.
> 
> Pat
>

---

From: Roger Jørgensen <[email-masked]>
Date: Sat, 18 Feb 2012 13:50:52 +0100

> L2). This is (hopefully) captured in the proposed charter.

Plenty, much have been said in other mails in this thread.

I've only experienced this problem in a small scale, we (the VM/server
guys) cause major headache for the network group when they realized
what virtualization was, that a machine could be running anywhere in
the network, or in any of the datacenters involved. They had no
control of the traffic flow, and freaked even more out when they
realized there were several clusters of VM running from different
datacenters and different part of the network exchanging lots of
traffic.
That the traffic crossed the backbone up to 8times or more (yes it's
true) for a group of 4 VMs made the network group consider the server
group insane... some of these problem can be solved for VMware but I
haven't seen the same being solved for other virtualization platforms.

The other side of the VM vs network problem I meet daily are VM in the
same group/LAN running from different datacenters...  plenty of new
tools, standard, products and technologies out there, but none that
make it easy. Not to mention single/group of VM moving between
datacenters.

LISP, TRILL, VXLAN, different type of VPN's solve some problems, but
they don't solve The Big Picture problem. Especial not the one with
VM's moving random around in your entire datacenter(s).
Most of these also have another problem, MTU size. Add something on
top of something else and you use up packetsize...

So the problem is real, and it need to consider both L2 and L3, who
said we will always be running IP?

Just to add another aspect, who said the IP the VMs have configured
need to be the same as they use outside their own interconnect? Maybe
they can have one IP range configured for all communication between
them but use another set of IP's toward everyone else, a bit like
LISP's concept of EID but where the EID's ain't known for the outside.

... in my current job I'm lucky because we have enough fibre in the
ground to support multi-site datacenters with virtual unlimited
bandwidth between them, however I still have to solve the split-brain
issue that _will_ happen sooner or later with a special set of
outages.

--- Roger J ---

-- 

Roger Jorgensen           |
[email-masked]          | - IPv6 is The Key!
http://www.jorgensen.no   | [email-masked]

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Sat, 18 Feb 2012 11:00:11 -0600

Thomas:

I think that draft-kreeger does a very good job in describing 
the problems, However, it identifies some additional problems that 
are not clearly stated in the WG Charter below. Specifically, it identifies
the need for a connect/disconnect notification between edge devices
and networks in order to cover the case where encapsulations are not
necessarily implemented in the same node.

Quoting from the draft:

" When an OBP is external, a protocol is needed between a client device
   making use of the external OBP and the OBP itself in order to make
   the OBP aware of the changing VN membership requirements of the
   client device.  A key driver for using a protocol rather than using
   static configuration of the exernal OBP is because the VN
   connectivity requirements can change frequently as VMs are brought
   up, moved and brought down on various hypervisors throughout the data
   center."

I believe that this should be one of the issues tackled for a complete
end-to-end solution.

Thanks,

Dimitri

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Sat, 18 Feb 2012 11:07:30 -0600

+1 on Larry's comments below, and I stated the same in my previous mail
about the charter ignoring a very important problem.

Dimitri

> Larry Kreeger
> Sent: Friday, February 17, 2012 2:11 PM

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 20 Feb 2012 06:30:32 -0800

David,

> these for which MPLS and BGP are non-starters.

What are the specific *technical* reason(s) why MPLS over GRE is
a "non-starter" for (a) ToR switches, (b) datacenter access switches,
and (c) hypervisor softswitches ?

Yakov.

> [JD]  This is an assertion.  It is also the misses the fact that MPLS is only
 required to mux/demux packets at the edges of the VPN network.

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 20 Feb 2012 10:05:04 -0800

> defines the approach. If we just saying "learning", do folk generally

"learning from the data path" should do it.

Anoop

---

From: <[email-masked]>
Date: Mon, 20 Feb 2012 15:10:49 -0500

Yaakov,

> and (c) hypervisor softswitches ?

Sure, that was on my list of things to do, so thanks for asking, but

> > > > BGP and MPLS are non-starters for a lot of datacenter-internal

Let's start with one of the more important problems that has motivated
interest in overlays for virtual networking.  From the proposed charter:

   Support for multi-tenancy has become a core requirement of data
   centers, especially in the context of data centers which include
   virtualized servers known as virtual machines (VMs).  

In these datacenters, there is a sizeable population of virtual machines
running using VLANs.  In a bit more detail, that means:
	- Data Plane: TCP/IP, Ethernet VLANs
	- Control Plane: IP Routing based on IGPs (e.g., OSPF), VLAN
		configuration, LLDP, etc.
Beyond that, management, operational practices and network admin skills
are matched to the environment.

Again from the proposed charter:

   Tenant isolation is primarily achieved today within data centers using
   Ethernet VLANs. But the 12-bit VLAN tag field isn't large enough to
   support existing and future needs.

This is an incremental growth problem - the datacenter is running fine
with VLANs, but VLAN address space is being exhausted.  The solution
should be incremental in impact and incrementally deployable.

Taking a look at MPLS and BGP, and assuming that the gaps previously
pointed out in the marques-l3vpn-end-system draft are addressed, I see
the following:
	- Introduce new data plane: MPLS
	- Introduce new control plane: BGP
	- Significant changes to management and operational practices
	- New network admin skills required
That's not the best incremental impact story.  The last one is particularly
important.

Incremental deployment also leaves a bit to be desired, as the
approach described in the marques-l3vpn-end-system does not work with
existing VM live migration implementations, because the VM's IP addressing
("VM route table" in the draft) has to be modified.  This has some
consequences:

	- Existing live VM migration implementations won't be able to move
		a VM between VLAN and MPLS-BGP environments because they
		don't reconfigure the VM route table.
	- New live VM migration implementations that want to support this
		sort of cross-environment migration will need additional
		functionality (e.g., may need to coax the VM to go renew
		its DHCP lease to discover what happened, and hope it copes).
	- Cold migration of a VM between VLAN and MPLS-BGP environments
		requires reconfiguration to change the VM route table. 
		Similarly, use of a common VM template across both environments
		requires an additional reconfiguration step.

If you'll pardon the double-negative, this isn't to say that an MPLS-BGP
approach can't be made to work, and I'd definitely like to see a common
protocol between end systems and the network for new live migration 
implementations that can be used by any relevant technology.  Rather, the
deployment and operational pain of the above makes it a non-starter
courtesy of its impacts (data plane, control plane, management, operational
practices, required netadmin skills), and operational drawbacks wrt
existing live VM migration deployments.

I believe that with enough engineering and design work, something workable
will emerge here, but I'm concerned that for a significant portion of the
problem space, this is akin to trying to turn a hammer into a powered
screwdriver.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Yakov Rekhter
> Sent: Monday, February 20, 2012 9:31 AM
> To: Black, David
> > [email-masked]        Mobile: +1 (978) 394-7754

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 20 Feb 2012 21:40:07 +0100

Hi Yakov,

> and (c) hypervisor softswitches ?

I don't think this is right question to ask. I personally see no 
technical reasons why MPLS over GRE could not be used in (a) or (b) or 
(c). In fact as one could notice from my former posts I think this is a 
very good idea to use it.

However the right question here is the choice of signalling and control 
plane such that either (a) or (b) or (c) knows how to build GRE header 
or what value of MPLS label to impose. I think this is the crux of focus 
of this potential WG.

IMHO the better questions to ask this WG are ...

* If we choose draft-marques-l3vpn-end-system to be the signalling 
protocol for nvo3 would we get general agreement that all (a) ToR 
switches, (b) datacenter access switches and (c) hypervisor 
soft-switches would support it ?

* Do we need one signalling protocol for all (a) or (b) or (c) or would 
we be fine with more then one depending on the level of virtualization 
boundary

* Do we in general agree that p2mp nature of BGP distribution (even with 
RTC) which is in core of draft-marques-l3vpn-end-system proposal is a 
good fit for DC L2 and L3 virtualization requirements ?

* For short lived VM based virtualization requirements (job processing 
application) especially across DCs are we really convinced that dynamics 
of today's BGP (where per prefix priority is simply not implemented 
anywhere and where fair treatment to all prefixes is preferred) is the 
best mechanism we could come up today to this problem space ?

Best regards,
R.

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 21 Feb 2012 01:35:47 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> [email-masked]
> 发送时间: 2012年2月21日 4:11
> 收件人: [email-masked]
> 抄送: [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> 主题: Re: [nvo3] Draft NVO3 WG Charter
> 
> Yaakov,
> 

Hi David,

Compared to the encapsulation format proposed by either existing NVo3 proposal (e.g., VXLAN), MPLS (i.e., MPLS in GRE) should be deemed as an existing data plane, rather than a new data plane, especially when you consider to deploy the encap/decap funcations on physical switches/routers.

> 	- Introduce new control plane: BGP

Using MPLS in GRE on the data plane doesn't mean we must use BGP on the control plane. Take VPLS as an example, if you believe BGP is not scalable or a bit heavy-weight in the data center environment while MPLS in GRE is acceptable, we can attempt to just define a light-weight control plane, for example, extend IGP a bit to deliver VPLS. 

> 	- Significant changes to management and operational practices

If you consider those cloud data centers which are operated by ISPs, you will come to an opposite conclusion since they have deployed and operated their MPLS/IP networks for many years.

> That's not the best incremental impact story.  The last one is particularly

The same as the above.

Best regards,
Xiaohu

> Yakov Rekhter
> > Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> > [email-masked]

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 21 Feb 2012 03:31:41 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Robert
> Raszuk
> 发送时间: 2012年2月18日 0:50
> 收件人: John E Drake
> 抄送: [email-masked]; Ronald Bonica; [email-masked];

In addition to the concern about not sufficient forwarding power, host-based L2 overlay will more heavily depends on the core multicast trees to deliver uu/mc/bc traffic since the number of PE nodes will increase compared to the ToR-based L2 overly without doubt. However, keeping too many multicast groups within the core implies a serious scaling challenge on both the control plane and the data plane. Otherwise(if head-end replication is used), host-based L2 overlay will consume more bandwidth compared to ToR-based L2 overlay. Unless there is no mc and bc, and even no uu traffic within data centers...

Best regards,
Xiaohu

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 21 Feb 2012 05:31:34 -0800

David,

>From the proposed charter:

Developing a new (Standards Track) protocol would introduce:

   - new control plane 
   - significant changes to management and operational practices
   - new network admin skills required

So, developing a new protocol, as proposed in the charter, is
certainly "not the best incremental impact story". Thus, following
your line of reasoning, item 4 (developing a Standards Track protocol)
should be removed from the charter. However, I don't recall seeing
an e-mail from you suggesting to remove item 4 from the charter.
Did I miss it ?

Yakov.

---

From: <[email-masked]>
Date: Tue, 21 Feb 2012 11:38:43 -0500

Yakov,

> However, I don't recall seeing

Nope.  The original question was about why BGP and MPLS might be non-starters
for some data centers.  I explained in terms of the level of disruption for data
centers that aren't running either BGP or MPLS (and there are a lot of them).

Now, you seem to be advocating that a data center that can't accept the
disruptive introduction of BGP and MPLS can't accept any level of disruption
(independent of the attempt to ascribe that reasoning to me, it's not what I
wrote).  Needless to say, that's silly.  Are you really advocating that position?

The disruption under discussion is not binary, but ranges across a scale of impact
depending on the degree of change.  There are likely to be alternative approaches
to the scaling problem in charter item 4 that are less disruptive than
introduction of BGP/MPLS, and degree of disruption is something that should be
germane to looking at alternative approaches to this problem area.

For example, Pedro appears to have just confirmed that the l3vpn-end-system draft
made a design choice that breaks existing live VM migration implementations in
favor of improving network convergence.  A data center currently running one of
those existing VM live migration implementations may well prefer a less disruptive
scaling approach that doesn't have that side effect.

Thanks,
--David

> Sent: Tuesday, February 21, 2012 8:32 AM
> To: Black, David

---

From: Pedro Marques <[email-masked]>
Date: Tue, 21 Feb 2012 09:34:08 -0800

David,

On Tue, Feb 21, 2012 at 8:38 AM,  <[email-masked]> wrote:
> favor of improving network convergence.

You misunderstood my message. My point is that for live migration L2
transparency is not relevant but convergence time is quite critical.

I've not understood why you believe that the proposal "breaks live
migration" specially after i pointed out that the routing
configuration on the VM is a non-issue.

>  A data center currently running one of

It is unclear to me whether you the "existing VM live migration
implementations" that you are referring to perform storage access via
direct L2 protocols. Direct L2 storage access is indeed a tradeoff
versus scaling.

That tradeoff maybe attractive to small scale deployments while
completely unacceptable to large scale ones. That is one of the things
to keep in mind when the IETF considers data from "existing
implementations".

>> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked];
>  > [email-masked]

---

From: <[email-masked]>
Date: Tue, 21 Feb 2012 13:25:08 -0500

Pedro,

> configuration on the VM is a non-issue.

Could you supply the details of how it's a non-issue, including the
network response to the gratuitous ARPs and RARPs issued by existing
hypervisors?  Those details are not in the draft, and you might want
to put them into a revised version of the draft.

My interest is not so much in the requirements of VMs in principle,
but how this will co-exist in practice with existing hypervisor deployments.

> direct L2 protocols. Direct L2 storage access is indeed a tradeoff

They generally do not use direct L2 storage access.  An FCoE initiator
in a VM would perform direct L2 storage access, but that's not a good
idea because most hypervisor softswitches do not support the DCB Ethernet
functionality required by FCoE.  Hypervisor use of FCoE is typically a
static configuration - a moved VM uses the destination hypervisor's FCoE
to access its storage, and does not move its FCoE session/connection
across hypervisors.  iSCSI, NFS and CIFS all use IP.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Pedro Marques
> Sent: Tuesday, February 21, 2012 12:34 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]

---

From: "Pat Thaler" <[email-masked]>
Date: Wed, 29 Feb 2012 22:26:37 +0000

Larry, 

The VSIID field can carry one thing that uniquely identifies the VSI but it isn't the only information in the TLV. Perhaps it would help to expand on the purpose of VDP and the information it carries to support that purpose. VDP assumes that there is some form of virtualization management system that holds profiles for Virtual Station Interfaces (VSI), i.e. the virtual network port of a virtual machine. Often a profile is created for the characteristics needed by a VM of a certain type and multiple VMs can use that profile. VDP is intended to carry enough information to identify the profile and the particular VM so that a bridge (or other network device connected to the end node) can use the information to get the profile and determine whether it can support the configuration for that VM. 

A PDU for VDP carries TLVs. The first TLV in the PDU is a VSI manager TLV which can carry an IPv6 address for the manager holding the profiles for the rest of the TLVs in the PDU.

The rest of the TLVs are VDP association TLVs (and organizationally defined TLVs). 

The VDP association TLV has a type which indicates pre-association (i.e. the VM isn't moving here now, but fetch the profile and see if it can be supported for later), associate and de-associate. It carries the following information

VSI Type ID: 3  octets; identifies a type of VSI - i.e. a profile which could apply to multiple VSI instances
VSI Type Version: 1 octet; allows for multiple versions of a type.
VSSIID: 16 octets; uniquely identifies the VSI by IP address, MAC address, UUID or locally defined format

Filter info: this is information that can be used to filter frames to identify the VSI. An 8-bit filter info format field identifies the format of the filter info field. 4 formats are defined in 802.1Qbg. All of the currently defined filter formats start with a number of entries field - the VSI might have more than one.

The formats currently defined are: 
	VID (VLAN ID)
	MAC/VID
	GroupID/VID
	GroupID/MAC/VID

GroupID is 4 octets and is intended for cases where the network is too large for 12-bit VIDs to scale. For example, it can carry a 24-bit tenant ID appropriate to the network (e.g. PBB I-CID, NVGRE TNI, etc.). The station can put the null VID value in and the bridge can fill it in with a VID that it will map to the tenant ID. The mapping could be local to that physical port.

The currently defined format can carry the VMs MAC address(es) in addition to its IP address or UUID. If one wanted to get both a UUID and an IP address, currently it would have to come from the VM manager along with the profile, e.g. the bridge sends the Type ID and the UUID from the VDP to the virtualization manager and it sends back the profile and the IP address.  Another possibility is that IEEE 802.1 could define additional filter info formats in the future if there is a need.

There are two status bits in the VDP TLV that may be helpful for migration. The M-bit = 1 indicates that the VM is migrating to this port. The S-bit = 1 indicates that the VM is suspended.

Regards,
Pat

-----Original Message-----
From: Larry Kreeger [mailto:[email-masked]] 
Sent: Friday, February 17, 2012 6:04 PM
To: Pat Thaler; Yakov Rekhter; David Black
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] Draft NVO3 WG Charter

Hi Pat,

Thanks, Larry