Subject: [nvo3] draft-rekhter-nvo3-vm-mobility-issues-00.txt

From: Yakov Rekhter <[email-masked]>
Date: Mon, 20 Aug 2012 12:49:03 -0700

fyi, and comments...
------- Forwarded Message

Date:    Mon, 20 Aug 2012 12:44:08 -0700
From:    [email-masked]
To:      [email-masked]
Subject: I-D Action: draft-rekhter-nvo3-vm-mobility-issues-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : Network-related VM Mobility Issues
	Author(s)       : Yakov Rekhter
                          Wim Henderickx
                          Ravi Shekhar
                          Luyuan Fang
                          Linda Dunbar
                          Ali Sajassi
	Filename        : draft-rekhter-nvo3-vm-mobility-issues-00.txt
	Pages           : 10
	Date            : 2012-08-20

Abstract:
   This document describes a set of network-related issues presented by
   the desire to support seamless Virtual Machine mobility in the data
   center and between data centers. In particular, it looks at the
   implications of meeting the requirements for "seamless mobility".

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-rekhter-nvo3-vm-mobility-issues

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-rekhter-nvo3-vm-mobility-issues-00

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

------- End of Forwarded Message

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 21 Aug 2012 17:10:04 +0200

Dear all,

A few comments on an otherwise excellent draft:

2. Introduction
===============
You might want to differentiate between cold VM mobility - VM is stopped or paused, moved to another hypervisor/cluster/DC and restarted/resumed - where IP address preservation is desired or required, and hot VM mobility, where a running VM is moved to another location (hypervisor/cluster/DC). Fundamental differences between the two are:

* Convergence time requirements - obviously highly relevant to hot VM mobility but not for the cold case;
* ARP cache expiration - after a cold VM move, the hypervisor can bounce the VM virtual LAN interface (the TCP sessions are gone anyway) hopefully clearing the ARP cache.

You might also want to mention storage-related issues (the moved VM has to have access to the same virtual disk), obviously in the "out-of-scope" part of the introduction.

2.1 Terminology
===============
You might want to make the ToR definition more precise in case of port/fabric extenders. Reference to "controlling bridge" from EVB and 802.1BR would probably make most sense.

Also, when defining L2 CUG, it would make sense to specify whether a VM participating in multiple L2 CUGs has multiple (logical) interfaces (one per L2 CUG), requiring simple VPNs, or one interface in multiple L2 CUGs, requiring overlapping VPNs. 

The rest of the text implies a VLAN-per-CUG approach, which translates into logical interface per L2 CUG, but it might be worth spelling out.

3.1 VLAN IDs
============
I would assume that most people familiar with network virtualization come to an immediate conclusion that you either need multiple (non-tagged) interfaces per VM or a VLAN-tagged VM interface, but these conclusions might be worth documenting.

3.4 Optimal IP routing
======================
There might be cases where the only L3-7 networking device between a VM and the outside world is a router (in which case the optimal IP routing is the biggest problem you have), but in most cases the headaches are caused by stateful devices in the path (load balancers, address translators and/or firewalls). 

Solving optimal IP routing is "easy" (at least we have demonstrable solutions), network device state transfer and session preservation on hot VM move is a much harder problem (with the exception of a hypervisor-based firewall).

It would make sense to either expand the problem statement to include stateful devices, or make them explicitly out-of-scope.

Kind regards,
Ivan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Yakov Rekhter
> Sent: Monday, August 20, 2012 9:49 PM
> To: [email-masked]
> Subject: [nvo3] draft-rekhter-nvo3-vm-mobility-issues-00.txt
> 
> fyi, and comments...
> Internet-Draft directories: http://www.ietf.org/shadow.html or
> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> 
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Gary Berger (gaberger)" <[email-masked]>
Date: Tue, 21 Aug 2012 21:48:44 +0000

Very good document.. A few comments

Live Migration 

I agree with Ivan, If mobility is required across a site boundary, the L2
CUG Site must already exist due to the need to access shared storage (I.e.
VMFS, ISCSI, etc..). This knowledge needs to be known a priori (I.e.
Before the operator tries to migrate the host). Under some live migration
scenarios all data structures and state remain intact with usually only a
gratuitous ARP Reply to signal something changed..obviously complicating
the optimal routing scenario.

Terminology

The definition of ToR is also confusing IMHO and we are missing the more
noticeable virtual switch in this design architecture. Because of the
limitations in mobility (the target of this draft) under hypervisor bypass
and IOV implementations there must be a virtual edge switch described.

Usage of VLAN IDs

VM's that belong to the same L2-CUG and are in different sites MAY use the
same or different VLAN-Ids. I think there is an issue in defining the
L2-CUG in terms of a locally significant namespace (I.e. VLAN-ID). This
will make policy enforcement much more complicated as the VM's need to be
re-associated with a new VLAN-ID. Wouldn't it be more appropriate to
utilize a large enough non-conflicting namespace which is associated with
the L2-CUG even across L2-sites managed by the same autonomous entity? Yes
this is similar to what is implemented in NVP and also represented as the
RLOC in LISP. I think I would have expected some conversations to that
effect in this draft.

Mobility in General

Why do we presume that the only purpose for mobility is for VM to VM and
not just any generalized service which is bound to an IP address? The more
generalized problem is the dynamic-multihoming support which is not
restricted by physical topology and locally significant name spaces like
VLANs. For instance, what if I am Google and I have fully populated my
service exhausting the ports associated with my L2-CUG. There are free
ports on an adjacent rack but they are stranded because I can't expand my
service due to the L2 adjacency problem. The very benefit of the overlay
and a new namespace (aka Network Virtualization) allows for the decoupling
of macs-to-vlans-to-ports allowing for the mobility without reconfiguring
the underlying physical network.

Tks

-g

On 8/21/12 11:10 AM, "Ivan Pepelnjak" <[email-masked]> wrote:

>You might want to differentiate between cold VM mobility - VM is stopped
>or paused, moved to another hypervisor/cluster/DC and restarted/resumed -
>where IP address preservation is desired or required, and hot VM
>mobility, where a running VM is moved to another location
>(hypervisor/cluster/DC). Fundamental differences between the two are:
>
>* ARP cache expiration - after a cold VM move, the hypervisor can bounce
>the VM virtual LAN interface (the TCP sessions are gone anyway) hopefully
>clearing the ARP cache.
>
>You might also want to mention storage-related issues (the moved VM has
>to have access to the same virtual disk), obviously in the "out-of-scope"
>part of the introduction.
>
>2.1 Terminology
>===============
>You might want to make the ToR definition more precise in case of
>port/fabric extenders. Reference to "controlling bridge" from EVB and
>802.1BR would probably make most sense.
>
>Also, when defining L2 CUG, it would make sense to specify whether a VM
>participating in multiple L2 CUGs has multiple (logical) interfaces (one
>per L2 CUG), requiring simple VPNs, or one interface in multiple L2 CUGs,
>requiring overlapping VPNs.
>
>The rest of the text implies a VLAN-per-CUG approach, which translates
>into logical interface per L2 CUG, but it might be worth spelling out.
>
>3.1 VLAN IDs
>============
>I would assume that most people familiar with network virtualization come
>to an immediate conclusion that you either need multiple (non-tagged)
>interfaces per VM or a VLAN-tagged VM interface, but these conclusions
>might be worth documenting.
>
>3.4 Optimal IP routing
>======================
>There might be cases where the only L3-7 networking device between a VM
>and the outside world is a router (in which case the optimal IP routing
>is the biggest problem you have), but in most cases the headaches are
>caused by stateful devices in the path (load balancers, address
>translators and/or firewalls).
>
>Solving optimal IP routing is "easy" (at least we have demonstrable
>solutions), network device state transfer and session preservation on hot
>VM move is a much harder problem (with the exception of a
>hypervisor-based firewall).
>
>It would make sense to either expand the problem statement to include
>stateful devices, or make them explicitly out-of-scope.
>
>Kind regards,
>Ivan
>

---

From: Somesh Gupta <[email-masked]>
Date: Wed, 22 Aug 2012 04:07:02 -0700

Hopefully this is not a repeat of something hashed out earlier.

It is not clear to me why an L2-CUG would be tied to a VLAN-ID
in an nvo3 network?

Is this document discussing the problem as is today without
the changes nvo3 would bring? If that is the case, fine.

Thanks
Somesh

---

From: "Black, David" <[email-masked]>
Date: Wed, 22 Aug 2012 15:19:35 -0400

> It is not clear to me why an L2-CUG would be tied to a VLAN-ID

Uhm, it's not, but the draft could be much clearer on this topic.

IMHO, the draft uses the term "L2 site" in a rather non-intuitive
fashion for nvo3.  Here's the definition from Section 2 of the draft:

   In this document the term "L2 site" refers to a collection of
   interconnected devices that perform forwarding based on the
   information carried in the Ethernet header. In a non-trivial L2 site
   (site that contains multiple forwarding entities) forwarding could be
   provided by such layer 2 technologies as Spanning Tree Protocol
   (STP), etc...  Note that any multi-chassis LAG is treated as a single
   L2 site - a given multi-chassis LAG has to be contained within a
   single L2 site.  This document assumes that a layer 2 access domain
   is an L2 site.

It's easy to mis-read that definition and infer than an "L2 site" is a
data center, but that's not what's going on.  For example, consider the
scenario in which the NVEs are located in the hypervisors (and possibly
also the DCBRs) - the "L2 site" adjacent to a VM then consists of one or
more virtual links in the hypervisor, and none of the physical network
components in the data center are part of any "L2 site" (as that term
is defined in the text quoted above).  Among the interesting consequences
for this scenario is that the following requirement (from Section 3.1 of
the draft) is far less restrictive than it may initially appear:

   This document assumes that VMs that belong to the same L2-based CUG,
   and are in the same L2 site MUST use the same VLAN-ID.

If every "L2 site" is a single virtual link within a hypervisor, that
requirement is rather easy to meet, especially for virtual links that
aren't VLAN-tagged ;-).

When the NVEs are in the ToR switches, the "L2 site" may be as small as
a single Ethernet link between the physical server and the ToR switch.
There's a much longer discussion of this scenario in Section 4.1 of
draft-dunbar-nvo3-overlay-mobility-issues-00.txt .

In any case, I suggest using a word other than "site" in "L2 site" for
clarity.

Following up on Ivan's comment about tagged vs. untagged network
interfaces in VMs, the first paragraph in Section 3.1 of this draft
has an unfortunate mixture of VLAN-IDs assigned by the network/hypervisor
with VLAN-IDs used by VMs. While both are VLAN-IDs, their management is
sufficiently different that they should be discussed separately.  In
particular, I suggest splitting the last sentence of that paragraph
out into a separate paragraph on VLAN-tagged traffic to/from tagged
network interfaces in VMs and providing a longer discussion somewhere
earlier that contrasts hypervisor assignment of VLAN-IDs for traffic
to/from untagged VM network interfaces with VLAN-ID usage by tagged
VM network interfaces.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Somesh
> Gupta
> Sent: Wednesday, August 22, 2012 7:07 AM
> To: 'Yakov Rekhter'; [email-masked]
> Subject: Re: [nvo3] Comments on Live Migration and VLAN-IDs
>

---

From: Lucy yong <[email-masked]>
Date: Wed, 22 Aug 2012 20:29:42 +0000

Snip..

[[LY]] Yes, I agree. The draft should discuss the scenarios separately.

Thanks,
--David

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 24 Aug 2012 20:20:33 +0000

Ivan, 

Thank you very much for the comments. 
Responses to your comments are inserted below:

> Ivan Pepelnjak
> Sent: Tuesday, August 21, 2012 10:10 AM
> Subject: [nvo3] Comments to draft-rekhter-nvo3-vm-mobility-issues-
> 00.txt
> 
> Dear all,
> 
> You might want to differentiate between cold VM mobility - VM is
> stopped or paused, moved to another hypervisor/cluster/DC and
> restarted/resumed - where IP address preservation is desired or
> required, and hot VM mobility, where a running VM is moved to another
> location (hypervisor/cluster/DC). Fundamental differences between the
> two are:
> 

[Linda] Can you be specific on which issues described by the draft are impacted by the "convergence time"? 
 	If VMs maintain the same IP addresses in the new location, regardless of "Hot" or "Cold" with different convergence time, all the issues described in the draft still exists: e.g.  Usage of VLAN-IDs, L2 extension, and the Optimal routing (in-bound & out-bound), etc

> * ARP cache expiration - after a cold VM move, the hypervisor can
> bounce the VM virtual LAN interface (the TCP sessions are gone anyway)
> hopefully clearing the ARP cache.

[Linda] With Hot migration, most, it not all,  hypervisors today send out a gratuitous ARP when the new VMs are instantiated in the new place. So the Cache would have been refreshed. 
Besides, you can't depend on various ARP cache timer in different OSs. Therefore you can't say that because it is "cold" migration therefore no need to address wrong cache entries. 

> to have access to the same virtual disk), obviously in the "out-of-
> scope" part of the introduction.

[Linda] I thought the draft stated that "storage issues are out of the scope" in the last paragraph of "Introduction" 

> 2.1 Terminology

[Linda] Do you refer to IEEE802.1Qbg? The TOR in this draft doesn't mean the "controlling bridge". The ToR is the first external switch connecting to servers. 
Of course, servers could be embedded with blade switch or virtual switch. The "controlling bridge" have different meaning. 

> participating in multiple L2 CUGs has multiple (logical) interfaces
> (one per L2 CUG), requiring simple VPNs, or one interface in multiple
> L2 CUGs, requiring overlapping VPNs.

[Linda] The draft states " If a given VM is a member of more than one L2-based CUG, this VM would have multiple IP addresses, one per each such CUG.". 

If network administrator needs to map multiple "subnets" to "one VPN", they can. They can also map one "subnet" to "one VPN". 

> I would assume that most people familiar with network virtualization
> come to an immediate conclusion that you either need multiple (non-
> tagged) interfaces per VM or a VLAN-tagged VM interface, but these
> conclusions might be worth documenting.

[Linda] Do you mean each VM have multiple (non-tagged) physical interfaces? 

Thanks, Linda 

> solutions), network device state transfer and session preservation on
> hot VM move is a much harder problem (with the exception of a

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 24 Aug 2012 21:11:21 +0000

Agree with David on the meaning of L2 site. If you read carefully the definition of "L2 site" in the draft (instead of various connotations of L2 site under other context), the definition is very precise and correct for the usage of this document. 

Agree with David that "If every "L2 site" is a single virtual link within a hypervisor, that requirement is rather easy to meet, especially for virtual links that aren't VLAN-tagged ;-)."

But most, if not all, data centers today don't have the Hypervisors which can encapsulate the NVo3 defined header. The deployment to all 100% NVo3 header based servers won't happen overnight. One thing for sure that you will see data centers with mixed types of servers for very long time. 

If NVEs are in the ToR, you will see mixed scenario of blade servers, servers with simple virtual switches, or even IEEE802.1Qbg's VEPA. So it is necessary for NVo3 to deal with the "L2 Site" defined in this draft. 

Linda Dunbar

> Black, David
> Sent: Wednesday, August 22, 2012 2:20 PM
> To: [email-masked]

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 24 Aug 2012 21:22:02 +0000

Gary,

Response to your comments are inserted below:

> Gary Berger (gaberger)
> Sent: Tuesday, August 21, 2012 4:49 PM
> To: Ivan Pepelnjak; 'Yakov Rekhter'; [email-masked]
> Subject: [nvo3] Comments on Live Migration and VLAN-IDs
> 

[Linda] Most likely, if not all, the storage interfaces are separate from front end access. Very often they are even on different physical ports of a server. So a VM's ISCSI port and VLAN is different from the VM's front-end interface to other peers in same CUG or different CUGs. 

The draft deals with the VMs front-end IP addresses, VLANs, etc for communication between End Stations (or peers).

The draft states clearly that the how far VMs can move and back-end interfaces are out of the scope of draft. 

> and IOV implementations there must be a virtual edge switch described.

[Linda] The "TOR" in the draft refers to a physical switch connected to servers on a rack. The serves can be hosting one OS, or multiple OSs (in the form of VM). The servers can have embedded virtual switches, VEPA, or none at all. 

> the L2-CUG even across L2-sites managed by the same autonomous entity?

[Linda] Yes, it might be better with different identifier other than VLAN. But we have to deal with existing OSs sending out IEEE802.1Q VID encoded data frames, and virtual switches deployed today attaching IEEE802.1Q VIDs. 

Linda Dunbar

> Yes
> stopped
> bounce
> (one
> come
> hot

---

From: Somesh Gupta <[email-masked]>
Date: Fri, 24 Aug 2012 14:27:34 -0700

I see the following in the draft

"We say that an L2 site contains a given VM (or that a given VM is in
a given L2 site), if the server presently hosting this VM is
connected to a ToR that is part of that site."

and it seems to be at odds with "L2 site" being a virtual link
in the hypervisor. Or maybe it is just my ability to understand :-)

Somesh

> Sent: Friday, August 24, 2012 2:11 PM
> To: Black, David; [email-masked]
> Agree with David on the meaning of L2 site. If you read carefully the
> definition of "L2 site" in the draft (instead of various connotations
> of L2 site under other context), the definition is very precise and
> correct for the usage of this document.
> 
> 
> Agree with David that "If every "L2 site" is a single virtual link
> within a hypervisor, that requirement is rather easy to meet,
> especially for virtual links that aren't VLAN-tagged ;-)."
> 
> But most, if not all, data centers today don't have the Hypervisors
> which can encapsulate the NVo3 defined header. The deployment to all
> 100% NVo3 header based servers won't happen overnight. One thing for
> sure that you will see data centers with mixed types of servers for
> very long time.
> 
> 
> If NVEs are in the ToR, you will see mixed scenario of blade servers,
> servers with simple virtual switches, or even IEEE802.1Qbg's VEPA. So
> it is necessary for NVo3 to deal with the "L2 Site" defined in this
> draft.
> 
> Linda Dunbar
> 
> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > Of Somesh
> > > Gupta

---

From: Ivan Pepelnjak <[email-masked]>
Date: Sat, 25 Aug 2012 16:37:50 +0200

>> [... deleted ...]
>   	If VMs maintain the same IP addresses in the new location, regardless of "Hot" or "Cold" with different convergence time, all the issues described in the draft still exists: e.g.  Usage of VLAN-IDs, L2 extension, and the Optimal routing (in-bound&  out-bound), etc

Regular IP routing is more than good enough for cold VM migration. We 
might need something with guaranteed convergence time for hot VM 
migration (or decide on a technique where the source NVO3 node 
temporarily forwards packets to destination NVO3 node).

> Besides, you can't depend on various ARP cache timer in different OSs. Therefore you can't say that because it is "cold" migration therefore no need to address wrong cache entries.

The VM you're migrating has ARP cache in its TCP stack. Under the most 
generic case the hypervisor cannot influence the contents of that cache. 
Do I have to spell out the consequences?

> Of course, servers could be embedded with blade switch or virtual switch. The "controlling bridge" have different meaning.

What is the ToR switch in a scenario where a server is connected to a 
Nexus 2000 Fabric Extender, which is in turn connected to a Nexus 5000? 
How about a virtual NIC that uses 802.1Qbg S-component to access a switch?

> [Linda] The draft states " If a given VM is a member of more than one L2-based CUG, this VM would have multiple IP addresses, one per each such CUG.".

I was not asking about CUG-to-subnet mappings, I got that. I was asking 
about CUG-to-interface mappings.

> [Linda] Do you mean each VM have multiple (non-tagged) physical interfaces?

That's how VMs belonging to multiple subnets (load balancers, firewalls, 
web caches ...) are commonly deployed today. And unless you use 
hypervisor bypass (which is rarely used unless you work with SR-IOR or 
something similar) the VMs have virtual not physical interfaces.

Ivan

---

From: Ivan Pepelnjak <[email-masked]>
Date: Sat, 25 Aug 2012 17:22:39 +0200

On 8/24/12 11:11 PM, Linda Dunbar wrote:
[...]

> it is necessary for NVo3 to deal with the "L2 Site" defined in this

There are two hypothetical ways of implementing NVO3: existing layer-2 
technologies (with well-known scaling properties that prompted the 
creation of NVO3 working group) or something-over-IP encapsulation.

I might be myopic, but from what I see most data centers today (at least 
based on market shares of individual vendors) don't have ToR switches 
that would be able to encapsulate MAC frames or IP datagrams in UDP, GRE 
or MPLS envelopes. I am not familiar enough with the commonly used 
merchant silicon hardware to understand whether that's a software or 
hardware limitation. In any case, I wouldn't expect switch vendors to 
roll out NVO3-like something-over-IP solutions any time soon.

On the hypervisor front, VXLAN is shipping for months, NVGRE is included 
in the next version of Hyper-V and MAC-over-GRE is available (with Open 
vSwitch) for both KVM and Xen. Open vSwitch is also part of standard 
Linux kernel distribution and thus available to any other Linux-based 
hypervisor product.

So: all major hypervisors have MAC-over-IP solutions, each one using a 
proprietary encapsulation because there's no standard way of doing it, 
and yet we're spending time discussing and documenting the history of 
evolution of virtual networking. Maybe we should be a bit more 
forward-looking, acknowledge the world has changed, and come up with a 
relevant hypervisor-based solution.

Furthermore, performing something-in-IP encapsulation in the hypervisors 
greatly simplifies the data center network, removes the need for 
bridging (each ToR switch can be a L3 switch) and all associated 
bridging kludges (including large-scale bridging solutions). Maybe we 
should remember that "Perfection is achieved, not when there is nothing 
more to add, but when there is nothing left to take away" along with a 
few lessons from RFC 3439.

I am positive a decade from now we'll see ancient servers still using 
VLAN-only hypervisor switches (or untagged interfaces), so there might 
definitely be an need for an NVO3-to-VLAN gateway, but we shouldn't 
continuously focus our efforts on something that's probably going to be 
a rare corner case a few years from now.

... or I may be completely wrong. Wouldn't be the first time.
Ivan

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 25 Aug 2012 17:54:50 +0200

Ivan,

 > ... or I may be completely wrong.

I think you are actually right on the spot correct.

However I am afraid authors of this document are not likely to admit 
that TOR switches should be just basic IP nodes providing only transport 
between servers.

Likewise they will not likely to admit that all logic of encapsulation 
should happen on the hyper-visors as they are simply not in that 
technology space.

Similarly I very much agree and support providing clear distinction 
between "cold" and "hot" VM mobility cases and perhaps even further 
provide number of sub-classes hot VM mobility can be accomplish today - 
clearly there is more then one way.

Also as you have pointed out storage discussion can not be just swapped 
under carpet and addressed by quote: "storage issues are out of the scope".

While Linda was perhaps right to say that today most storage today is 
coming to servers via backend this is what I would call very inefficient 
and legacy way. If we are to think ahead one needs to observe how the 
industry advances in storage virtualization via front-end IP very often 
not co-located with the compute racks.

In my view network related mobility discussion is not about TOR or about 
VLANs. It is about an IP layer above IP transport which would carry all 
necessary information of the actual location of the VMs and which in 
fact would play the main role in shortening or eliminating the 
triangular routing problem.

Rgs,
R.

---

From: "Black, David" <[email-masked]>
Date: Sun, 26 Aug 2012 02:01:31 -0400

Robert,

> under carpet and addressed by quote: "storage issues are out of the scope".

I agree ... and that looks like my cue to say something ... e.g., see the
domain part of my email address ;-).

iSCSI and NFS use TCP/IP in the storage stack and hence will run fine over
all of the data encapsulations being discussed here.  If the iSCSI initiator
or NFS client is in the VM, that's most of the discussion.  That's not always
the case for a number of reasons - the obvious one is that a hypervisor
iSCSI initiator or NFS client is required if the VM's executable image is being
loaded and/or paged using one of those protocols.  It's also the case that
many hypervisors simplify the storage interface presented to VMs so that it
looks like direct attached or internal disk drives), and map those disks to
networked storage using a hypervisor iSCSI initiator or NFS client.  Ensuring
that the VM migration destination hypervisor has appropriate connectivity to
storage is mostly a configuration concern.  The upshot is that iSCSI and NFS
run fine over nvo3-encapsulated networks.

In contrast, as I said at the microphone at the nvo3 BOF in Paris, I suggest
that the WG not initially consider FCoE, in order to defer spending time on
discussing how to deliver DCB Ethernet service/behavior (required by FCoE -
ordinary non-DCB Ethernet isn't sufficient for FCoE because FCoE is *very*
sensitive to drops) through the encapsulation(s).

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert
> Raszuk
> Sent: Saturday, August 25, 2012 11:55 AM
> To: Ivan Pepelnjak
> Cc: Black, David; [email-masked]; Linda Dunbar
> Subject: Re: [nvo3] Let's refocus on real world
> 
> Ivan,
>

---

From: Aldrin Isaac <[email-masked]>
Date: Sun, 26 Aug 2012 20:47:25 -0400

AFAIK, scale out cloud storage software such as Ceph do not rely on FC,
FCoE, NFS or iSCSI on the VM.  Ceph storage appears to the VM as local
storage and does not depend on network virtualization.  VM migration is not
an issue for Ceph since the VM itself does not need to connect to a storage
server over the network.  So as far as real cloud storage is concerned
nothing is being swept under the rug.

-- aldrin

On Sunday, August 26, 2012, Black, David wrote:

> > From: [email-masked] <javascript:;> [mailto:[email-masked]<javascript:;>]
> On Behalf Of Robert
> > Raszuk
> > Cc: Black, David; [email-masked] <javascript:;>; Linda Dunbar
> > > [email-masked] <javascript:;>
> > [email-masked] <javascript:;>
> [email-masked] <javascript:;>

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Mon, 27 Aug 2012 04:55:08 +0000

Ivan:

I agree and at the same time disagree with some of the statements
below. I would like to understand your view.

See inline:

>relevant hypervisor-based solution.

Correct, and here is where IETF as a standard body fails. There is no
easy way (any time soon) for a VXLAN based solution to talk to an NVGRE
or MAC/GRE, or Cloudstack MAC/GRE or STT  (you forgot this one), based
solution. 
Proprietary approaches that drive enterprises to vendor lock ins. And
instead 
of trying to address the first problem that is about "interoperability",
we completely throw it under the rug as "not important". And by the time
we are done with NVO3, there will be a controller lock in as well, and the
death of interoperability.. If I was on the deployment side of the
solution, 
that's the number one flexibility I would like to see. I don't want to be
forced to buy all my hypervisors from a given vendor, given that not all
applications are served equally by all hypervisors (for several $$$
reasons 
I might add, that can be related with the licensing options of different
OSes on top of different hypervisors).

>a rare corner case a few years from now.

You are absolutely correct.
I think that if the gateways were trying to solve the interoperability with
legacy servers problem, then obviously they are doomed since they have a
limited
life time as you correctly point out.

But I would argue that the only reason for the gateways is rather
different,
and it has to do with the point of separation of trust. I believe that
people
have several use cases in mind, where there is no hypervisor involved.
Some examples: ARM/Low power servers where the unit of computation is the
processor and there is no hypervisor, offers of "bare metal servers as as
service", where the handover is the physical wire and what the server puts
on the wire cannot be trusted, etc.

Another real problem described in the last Openstack conference
by one of the panelists:"We want test&dev and QA systems to run over VMs
and production systems to run over the same data center network, but on
bare metal." 
Obviously, they want to scale bare metal usage the same way as VMs
depending
on load etc (i.e. drop down the available test&dev resources when
production needs it).
Same separation problems etc exist.

So yes, spending too much time worrying about VMs moving around and doing
encapsulations on ToRs is probably a waste of time. But spending a lot of
time
to understand interoperability between a hypervisor based environments and
use
cases such as the above that require gateways, I think is a real world
problem.

Dimitri

---

From: Jon Hudson <[email-masked]>
Date: Mon, 27 Aug 2012 01:48:41 -0700

oh man, this is getting good....

First I will say I think Ivan is dead on.

Second to Roberts input

> However I am afraid authors of this document are not likely to admit that
TOR >switches should be just basic IP nodes providing only transport
between >servers.
> Likewise they will not likely to admit that all logic of encapsulation
should >happen on the hyper-visors as they are simply not in that
technology space.

this has been a fundamental issue for a while now. The people who will
decide which method prevails will be the Virtualization Admins not the
Network admins. And what the Virt Admins want is: point -> click -> drag ->
drop = poof virtual magic network overlay.

Since most Virt Admins don't have access to ToR or any other network
hardware, I would argue that if control and encap/decap doesn't happen in
the hypervisor then NVo3 won't see much adoption. Especially since this is
what is happening now in the proprietary offerings.

And then Dimitri!

>death of interoperability..

THANK YOU

Pure hypervisors environments are not going to exist very often. You will
have a mixed model of vmware, xen & hyperV (kvm?) in most if not all shops.

This makes the current environment of VXLAN/NVGRE/STT to be very very
scary. Do I really want to run each tech for each hypervisor? Do people
really assume that whatever the NVo3 group produces will result in all
those vendors just dropping their versions and running with open arms to
the standard?

Offering a method of interoperability between the three would be HUGE. That
would be a reason for the user community to really get excited about NVo3.
Right now many just look at is as a standards version of what they can
already get (correct or incorrect as that vision may be)

And finally a quick comment on the storage issue. I won't say much as I
think others have said it well. But I will restate the most important part.
To say that storage is out of scope is just insane. Taking a forward
looking view of how things might be (as Robert suggested) would be very
wise.

-- 
"Do not lie. And do not do what you hate."

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 09:21:35 +0200

Dimitri,

We're more in agreement than it might seem. I might have my doubts about 
the operational viability of the OpenStack-to-baremetal use case you 
described below, but I'm positive someone will try to do that as well.

In any case, regardless of whether we're considering VMs or bare-metal 
servers, in the simplest scenario the server-to-NVE connection is a 
point-to-point link, usually without VLAN tagging.

In the VM/hypervisor case, NVE is implemented in the hypervisor soft 
switch; in the baremetal server case, it has to be implemented in the 
ToR switch.

It's important to keep in mind the limitations of the ToR switches to 
ensure whatever solution we agree upon will be implementable in ToR 
switches as well, but it makes absolutely no sense to assume NVE will 
not be in the hypervisor (because someone wants to support a customer 
having a decade-old VLAN-only hypervisor soft switch).

As for ToR switch capabilities, Dell has demonstrated NVGRE support and 
Arista is right now showing off a hardware VXLAN VTEP prototype, so I 
guess it's safe to assume next-generation merchant silicon will support 
GRE- and UDP-based encapsulations well before we'll agree on what NVO3 
solution should be.

Finally, can at least some of us agree that the topology that makes most 
sense is a direct P2P link between (VM or bare-metal) server and NVE 
using VLAN tagging only when a server participating in multiple L2 CUGs 
has interface limitations?

Kind regards,
Ivan

On 8/27/12 6:55 AM, Stiliadis, Dimitrios (Dimitri) wrote:
> On 8/25/12 8:22 AM, "Ivan Pepelnjak"<[email-masked]>  wrote:
>

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 28 Aug 2012 06:53:45 -0700

> Sent: Tuesday, August 28, 2012 12:22 AM
> To: Stiliadis, Dimitrios (Dimitri)
> Subject: Re: [nvo3] Let's refocus on real world (was: Comments on Live
> Migration and VLAN-IDs)
> 
> Dimitri,
> 

  This is certainly only today's restriction. If nov3 takes off, there
  certainly could be a pseudo-driver in Linux that could implement the
  NVE (like a VLAN driver) without much additional overhead.

---

From: "Ayandeh, Siamack" <[email-masked]>
Date: Tue, 28 Aug 2012 15:02:21 +0000

"To say that storage is out of scope is just insane. Taking a forward looking view of how things might be (as Robert suggested) would be very wise. "
Storage has and continues to move to the front end network through I/O consolidation using 10GE. Even when there is a separate storage fabric in place; the edge i.e. server to the first hop switch, may be shared. This trend is continuing with x20GE and 40GE to the server.

One may take the position that FC and its derivative FCoE would not be part of any cloud offering. On the other hand if FCoE is in scope; it will put a requirement on the underlay service interface for lossless delivery (per David's comments). This may not be as bad as it looks. DCB may be available; the scope may be limited to the edge with hand off to a separate fabric; even with "TOR >switches should be just basic IP nodes providing only transport between >servers" the links will be Ethernet! And in the time frame of this working group; lossless Ethernet will likely be supported across the datacenter. So looking forward even IP based storage e.g. iSCSI can benefit from the lossless feature of the underlay and use it to scale.

I don't see any harm in having storage, in its broadest sense,  in scope given the time frames and evolution of datacenter and storage networking. If not at least we should clearly articulate why it was left out or limited in scope.

Siamack

From: [email-masked] [mailto:[email-masked]] On Behalf Of Jon Hudson
Sent: Monday, August 27, 2012 4:49 AM
To: Stiliadis, Dimitrios (Dimitri)
Cc: Ivan Pepelnjak; Black, David; [email-masked]; Linda Dunbar
Subject: Re: [nvo3] Let's refocus on real world (was: Comments on Live Migration and VLAN-IDs)

Second to Roberts input

> However I am afraid authors of this document are not likely to admit that TOR >switches should be just basic IP nodes providing only transport between >servers.
>
> Likewise they will not likely to admit that all logic of encapsulation should >happen on the hyper-visors as they are simply not in that technology space.
this has been a fundamental issue for a while now. The people who will decide which method prevails will be the Virtualization Admins not the Network admins. And what the Virt Admins want is: point -> click -> drag -> drop = poof virtual magic network overlay.

Since most Virt Admins don't have access to ToR or any other network hardware, I would argue that if control and encap/decap doesn't happen in the hypervisor then NVo3 won't see much adoption. Especially since this is what is happening now in the proprietary offerings.

And then Dimitri!

THANK YOU

Pure hypervisors environments are not going to exist very often. You will have a mixed model of vmware, xen & hyperV (kvm?) in most if not all shops.

This makes the current environment of VXLAN/NVGRE/STT to be very very scary. Do I really want to run each tech for each hypervisor? Do people really assume that whatever the NVo3 group produces will result in all those vendors just dropping their versions and running with open arms to the standard?

Offering a method of interoperability between the three would be HUGE. That would be a reason for the user community to really get excited about NVo3. Right now many just look at is as a standards version of what they can already get (correct or incorrect as that vision may be)

And finally a quick comment on the storage issue. I won't say much as I think others have said it well. But I will restate the most important part. To say that storage is out of scope is just insane. Taking a forward looking view of how things might be (as Robert suggested) would be very wise.

On Sun, Aug 26, 2012 at 9:55 PM, Stiliadis, Dimitrios (Dimitri) <[email-masked]<mailto:[email-masked]>> wrote:
Ivan:

See inline:

Dimitri

>[email-masked]<mailto:[email-masked]>

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 28 Aug 2012 15:11:29 +0000

Robert, 

The only difference I see between "Cold" migration and "Hot" migration is the distance on how far VMs can be moved from their original server racks. 

For "Cold" migration, VMs can be shut down, wait for their needed storage to be moved to the new location, and then the VMs can be re-instantiated again. 

For "Hot" migration, the VMs can only be moved to racks which still have access to their image and data. Most likely they are not too far. 

Linda Dunbar

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Saturday, August 25, 2012 10:55 AM
> To: Ivan Pepelnjak
> Cc: Linda Dunbar; Black, David; [email-masked]

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 28 Aug 2012 15:13:44 +0000

>  NVE (like a VLAN driver) without much additional overhead.

 That doesn't work if you assume that tenants and DC operators are
different
 entities. The DC operator cannot rely on the tenant to do the right
 encapsulation. Different administrative and trust domains. That's why
 in my original email I was talking about "trust boundaries".

Dimitri

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 28 Aug 2012 15:17:54 +0000

Ivan, 

See my comments inserted below:

> From: Ivan Pepelnjak [mailto:[email-masked]]
> Sent: Saturday, August 25, 2012 10:23 AM
> To: Linda Dunbar
> Cc: Black, David; [email-masked]
> Subject: Let's refocus on real world (was: [nvo3] Comments on Live
> Migration and VLAN-IDs)
> 

[Linda] I have to say this is incorrect. ToR switches are capable of doing VxLAN and GRE encapsulations. 

> a rare corner case a few years from now.

[Linda] For applications which requires more than one "computing engine" to accomplish a task might need dedicated servers instead of VMs.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 28 Aug 2012 15:20:28 +0000

>point-to-point link, usually without VLAN tagging.

 Unfortunately not always. If the NVE is the ToR and the server is part
 of a blade system, the server-to-NVE connection is actually multiplexed.
 Same holds for "fabric extender" type of architectures.
 One could require that the NVE must always be one hop away and in a
 p2p connection, but this would limit options.

 Also, if you want to consider multi-homed servers to dual NVEs, there
 are some additional complexities to consider, especially if we are
 looking for active/active configurations.

>has interface limitations?
 
 Yes, VLAN hand offs are perfectly fine and cover most cases, and I don't
 think they require a p2p link.

Dimitri

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 28 Aug 2012 15:26:05 +0000

Somesh, 

You are correct that this description does imply that TOR has physical links to the server which host the VMs. 

We will find out a way to change the wording to include the Virtual links case. 

Thanks, Linda

> Somesh Gupta
> Sent: Friday, August 24, 2012 4:28 PM
> To: [email-masked]
> > Of
> > >    (site that contains multiple forwarding entities) forwarding
> could
> > > be
> > > It's easy to mis-read that definition and infer than an "L2 site"
> is
> > a
> > > also the DCBRs) - the "L2 site" adjacent to a VM then consists of
> one
> > > or
> > > for this scenario is that the following requirement (from Section
> 3.1
> > > of
> > > When the NVEs are in the ToR switches, the "L2 site" may be as
> small
> > as
> > > with VLAN-IDs used by VMs. While both are VLAN-IDs, their
> management
> > is

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 28 Aug 2012 15:37:51 +0000

David, 

I agree that the first phase of NVo3 should not consider encapsulating  FCoE traffic. 

That means servers need to have separate physical ports for FCOE traffic so that NVE won't need any mechanism to recognize the FCoE traffic. 

If servers don't have separate physical ports for FCoE traffic, they use some VLANs for FCoE traffic instead. Then NVEs have to be configured not to process those VLANs.  Is it correct?  

How about iSCSI & NFS traffic? Should they be handled in the same way as FCoE Traffic? 

Linda

> Sent: Sunday, August 26, 2012 1:02 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [nvo3] Storage (part of: Let's refocus on real world)
> 
> Robert,
>

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 28 Aug 2012 12:40:37 -0400

The question regarding FCoE is whether overlay solutions need to transport
it.  I think the answer is no.  If something operates at the underlay level
than it isn't in scope for NVo3, including DCB.

On Tuesday, August 28, 2012, Somesh Gupta wrote:

> On Behalf Of

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 18:44:04 +0200

In sane real-life designs the virtual network overlay solution would not 
transport FCoE. I'm also positive someone will come up with exactly that 
requirement sooner rather than later :D

On 8/28/12 6:40 PM, Aldrin Isaac wrote:
> The question regarding FCoE is whether overlay solutions need to 
> transport it.  I think the answer is no.  If something operates at the 
> underlay level than it isn't in scope for NVo3, including DCB.
>
>
>     > From: [email-masked] <javascript:;>
>     [mailto:[email-masked] <javascript:;>] On Behalf Of
>     > Ivan Pepelnjak
>     > >>> servers with simple virtual switches, or even IEEE802.1Qbg's
>     VEPA.
>     > So
>     > >> I might be myopic, but from what I see most data centers
>     today (at
>     > least
>     > >> that would be able to encapsulate MAC frames or IP datagrams
>     in UDP,
>     > GRE
>     > >> hardware limitation. In any case, I wouldn't expect switch
>     vendors
>     > to
>     > >> in the next version of Hyper-V and MAC-over-GRE is available
>     (with
>     > Open
>     > >> So: all major hypervisors have MAC-over-IP solutions, each
>     one using
>     > a
>     > >> proprietary encapsulation because there's no standard way of
>     doing
>     > it,
>     > >> and yet we're spending time discussing and documenting the
>     history
>     > of
>     > >> forward-looking, acknowledge the world has changed, and come
>     up with
>     > a

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 18:51:23 +0200

Exactly right. There are good reasons we're using VLANs and untagged 
server interfaces today.

I wouldn't trust my servers to choose which virtual network they want to 
participate in, let alone my customers' servers.

Ivan

On 8/28/12 5:13 PM, Stiliadis, Dimitrios (Dimitri) wrote:
[...]

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 28 Aug 2012 16:55:04 +0000

FCoE is clearly not a requirementŠ

but, there is something to be said about storage (and I should have
responded in the
other email about this), but in general storage isolation is done at the
storage 
level and not the network layer. So, we can ignore.

If we take a a storage server that exports a file system that is mounted
by a 
hypervisor and multiple tenants have their VMs in this file system, then a
single
network connection between hypervisor and storage device could potentially
lead to head of line blocking and allow one tenant to influence the
performance
of another tenant. If my memory serves me correct, VMware for example can
only 
use two or four iSCSI initiators that have be to shared by the different
VMs 
of the hypervisor, and thus traffic from multiple tenants is multiplexed
on the same
network flow .. This means that storage drivers/devices have to take care
of 
traffic isolation. And this can be perfectly fine in point-to-point
situations, but
it can get interesting in multiplexed scenarios Š

(but we just don't want the storage guys to blame the network guys for
performance issues ;)

Dimitri

On 8/28/12 9:44 AM, "Ivan Pepelnjak" <[email-masked]> wrote:

>> From: 
>[email-masked] <javascript:;> [mailto:[email-masked]
><javascript:;>] On Behalf Of
>> Ivan Pepelnjak
>> Cc: Black, David;
>[email-masked] <javascript:;>; Linda Dunbar

---

From: Jon Hudson <[email-masked]>
Date: Tue, 28 Aug 2012 09:58:36 -0700

That's not enough resolution

Cold migration means that I am not maintaining tcp sessions. It may mean I even have the opportunity to bring interfaces up and down to apply new ipaddr/netmask/gateway info. It may mean that i reboot the VM. It may also mean the VM comes up in a "quarantine" area for validation before promotion back to a production state.

And yes it means my distance limitation goes from 10ms to 350ms which is a big deal. But the above details matter much as it opens many options.

---> sidebar

As a side note, VMs would not be shut down to wait for data sync. That data sync with no seed could take 12hrs or more.

An asynchronous type copy/snapshot type technology is employed to get "most" of the data to the new site while the VM is still running. Then you can do a sync->suspend->true-up->unsuspend on other side type maneuver or a shutdown->trueup->bring up on the other side. In many cases you will h e been distributing snapshots to many locations for a long time giving you the chance to just reappear somewhere else as a clone even if VM' fell into a black hole.

---

From: Jon Hudson <[email-masked]>
Date: Tue, 28 Aug 2012 10:10:19 -0700

Dead on. 

Anytime you have a fan-in-fan-out type traffic flow with filesystem info one person can ruin the party for everyone. FCoE is a perfect example where a pause frame sent on a aggregation link can end up impacting many initiators. Or even at a controller level of any array you can get a traffic jam of sorts on poorly designed and layed out subsystems. Or too few lines for food at an IETF social.

Lots can be done with queues etc to mitigate the issue, but it is always something to be mindful of. Especially if your remote filesystem is not just a mounted LUN but the mainsystem/boot LUN and you have windows paging over the wire.

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 19:25:21 +0200

If I understood vSphere manuals and discussions on various blogs/forums 
correctly, VMware solved most of this problem a long time ago with I/O 
shares and a few other features ... but don't trust a networking guy to 
know anything about storage :)

On 8/28/12 7:10 PM, Jon Hudson wrote:
> On Aug 28, 2012, at 9:55 AM, "Stiliadis, Dimitrios (Dimitri)"<[email-masked]>  wrote:
>
>> On 8/28/12 9:44 AM, "Ivan Pepelnjak"<[email-masked]>  wrote:
>>
>>>
>>>
>>> [email-masked]<javascript:;>  [mailto:[email-masked]
>>> [email-masked]<javascript:;>; Linda Dunbar

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 19:28:03 +0200

I'm so glad to hear that; solves a major problem one of my customers is 
having.

Could you share with me which (shipping) ToR switch in particular has 
hardware VXLAN termination?

Thank you!
Ivan

On 8/28/12 5:17 PM, Linda Dunbar wrote:
[...]
>>  In any case, I wouldn't expect switch vendors to

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 28 Aug 2012 10:27:37 -0700

The trust boundaries will certainly be important. If a tenant is
allowed to load their own OS on bare metal, it will probably be outside
the trust zone of the provider's overlay network.

1. Consider a case of an enterprise where it has full control over
   what is loaded onto a server - in this case regardless of whether
   a hypervisor is on the bare-metal or a non-virtualized OS, it
   makes sense to have the NVE in the OS

2. Similarly a cloud sp may use non-virtualized servers to offer
   certain services such as storage, hadoop, queuing etc. It makes
   sense to have the NVE in the OS

3. Lastly a case where a service provide lets a tenant load their
   own OS onto the server - i.e. some sort of managed hosting. In
   that case the servers dedicated to the tenant will be in
   a separate trust zone. They will most likely be on a different
   VLAN altogether - they may even be physically located in
   a different rack in a data center.

   In this case you are looking at more than beyond simple encap/decap
   to provide connectivity between the two trust zones.

regarding the point about using untagged interfaces today, I think
having two different sets of admins - server admins and network
admins - and who owns setting  up VLANs - may have a lot to do with
that as well. Which set of admins will own encap/decap? I think the roles will change.

Somesh

> Sent: Tuesday, August 28, 2012 9:51 AM
> Cc: Somesh Gupta; Black, David; [email-masked]; Linda Dunbar

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 19:30:30 +0200

So you think your server (in case #1 or #2) will never be pwned? Good luck!

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 28 Aug 2012 17:32:11 +0000

;) see my "multiplexed" comment below (i.e. two servers mounting the same
file system).

IO shares work on the server side ..

Cheers,

Dimitri

On 8/28/12 10:25 AM, "Ivan Pepelnjak" <[email-masked]> wrote:

>> Anytime you have a fan-in-fan-out type traffic flow with filesystem
>>info one person can ruin the party for everyone. FCoE is a perfect
>>example where a pause frame sent on a aggregation link can end up
>>impacting many initiators. Or even at a controller level of any array
>>you can get a traffic jam of sorts on poorly designed and layed out
>>subsystems. Or too few lines for food at an IETF social.
>>
>> Lots can be done with queues etc to mitigate the issue, but it is
>>always something to be mindful of. Especially if your remote filesystem
>>is not just a mounted LUN but the mainsystem/boot LUN and you have
>>windows paging over the wire.
>>
>> On Aug 28, 2012, at 9:55 AM, "Stiliadis, Dimitrios
>>(Dimitri)"<[email-masked]>  wrote:
>>
>>>>>>> that would be able to encapsulate MAC frames or IP datagrams in
>>>>>>>UDP,
>>>>> GRE
>>>>>>> So: all major hypervisors have MAC-over-IP solutions, each one
>>>>>>>using
>>>>> a
>>>>>>> forward-looking, acknowledge the world has changed, and come up
>>>>>>>with
>>>>> a

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 19:38:42 +0200

Storage I/O control (SIOC) is supposed to work on datastore level across 
multiple hypervisors (previous disclaimer still applies :) )

On 8/28/12 7:32 PM, Stiliadis, Dimitrios (Dimitri) wrote:
> On 8/28/12 10:25 AM, "Ivan Pepelnjak"<[email-masked]>  wrote:
>

---

From: Jon Hudson <[email-masked]>
Date: Tue, 28 Aug 2012 10:48:55 -0700

No you are right. VMware has done a lot of work through things like VAAI
and Storage DRS etc to mitigate and in some cases eliminate many of the
issues. However, not everyone runs VMware. (while I must admit, being at
VMworld this week, I wouldn't tell any of these folks that ;-)

On Tue, Aug 28, 2012 at 10:38 AM, Ivan Pepelnjak <[email-masked]>wrote:

>> On 8/28/12 10:25 AM, "Ivan Pepelnjak"<ipepelnjak@gmail.**com<[email-masked]>>
>>  wrote:
>>
>>>> (Dimitri)"<dimitri.stiliadis@**alcatel-lucent.com<[email-masked]>>
>>>>   wrote:
>>>>
>>>>> On 8/28/12 9:44 AM, "Ivan Pepelnjak"<ipepelnjak@gmail.**com<[email-masked]>>
>>>>>   wrote:
>>>>>
>>>>>
>>>>>> [email-masked]<**javascript:;>   [mailto:[email-masked]
>>>>>>>> On 8/25/12 8:22 AM, "Ivan Pepelnjak"<ipepelnjak@gmail.**com<[email-masked]>>
>>>>>>>>    wrote:
>>>>>>>>
>>>>>>>>>> which can encapsulate the NVo3 defined header. The deployment to
>>>>>>>>>>
>>>>>>>>> all
>>>>>>>
>>>>>>>> 100% NVo3 header based servers won't happen overnight. One thing
>>>>>>>>>>
>>>>>>>>> for
>>>>>>>
>>>>>>>>>> If NVEs are in the ToR, you will see mixed scenario of blade
>>>>>>>>>>
>>>>>>>>> servers,
>>>>>>>
>>>>>>>> servers with simple virtual switches, or even IEEE802.1Qbg's VEPA.
>>>>>>>>>>
>>>>>>>>> So
>>>>>>>
>>>>>>>>> There are two hypothetical ways of implementing NVO3: existing
>>>>>>>>>
>>>>>>>> layer-2
>>>>>>>
>>>>>>>>> I might be myopic, but from what I see most data centers today (at
>>>>>>>>>
>>>>>>>> least
>>>>>>>
>>>>>>>> based on market shares of individual vendors) don't have ToR
>>>>>>>>>
>>>>>>>> switches
>>>>>>>
>>>>>>>>> hardware limitation. In any case, I wouldn't expect switch vendors
>>>>>>>>>
>>>>>>>> to
>>>>>>>
>>>>>>>>> On the hypervisor front, VXLAN is shipping for months, NVGRE is
>>>>>>>>>
>>>>>>>> included
>>>>>>>
>>>>>>>> in the next version of Hyper-V and MAC-over-GRE is available (with
>>>>>>>>>
>>>>>>>> Open
>>>>>>>
>>>>>>>>> Linux kernel distribution and thus available to any other Linux-
>>>>>>>>>
>>>>>>>> based
>>>>>>>
>>>>>>>> hypervisor product.
>>>>>>>>>
>>>>>>>> proprietary encapsulation because there's no standard way of doing
>>>>>>>>>
>>>>>>>> it,
>>>>>>>
>>>>>>>> and yet we're spending time discussing and documenting the history
>>>>>>>>>
>>>>>>>> of
>>>>>>>
>>>>>>>> easy way (any time soon) for a VXLAN based solution to talk to an
>>>>>>>>
>>>>>>> NVGRE
>>>>>>>
>>>>>>>> or MAC/GRE, or Cloudstack MAC/GRE or STT  (you forgot this one),
>>>>>>>>
>>>>>>> based
>>>>>>>
>>>>>>>> solution.
>>>>>>>> of trying to address the first problem that is about
>>>>>>>>
>>>>>>> "interoperability",
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>  ______________________________**_________________
>>>>> nvo3 mailing list
>>>>> [email-masked]
>>>>> https://www.ietf.org/mailman/**listinfo/nvo3<https://www.ietf.org/mailman/listinfo/nvo3>
>>>>>
>>>> ______________________________**_________________
>>> nvo3 mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/**listinfo/nvo3<https://www.ietf.org/mailman/listinfo/nvo3>
>>>
>>

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 28 Aug 2012 11:45:36 -0700

Sorry, not sure what pwned is?

> Sent: Tuesday, August 28, 2012 10:31 AM
> To: Somesh Gupta
> Cc: Stiliadis, Dimitrios (Dimitri); Black, David; [email-masked]; Linda
> Dunbar
> >> I wouldn't trust my servers to choose which virtual network they
> want
> >> to
> >>>>    certainly could be a pseudo-driver in Linux that could
> implement
> >> the
> >>>    encapsulation. Different administrative and trust domains.
> That's
> >> why

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 28 Aug 2012 12:16:01 -0700

Thanks for clarifying what pwned is :-)

The problem remains whether you run a hypervisor or
a non-virtualized OS image that you control.

Trusting the hypervisor or any other entity - (including
the switch or gateways - who programmed the switch dynamically?) - is
an going to be an issue that needs to be addressed.

BTW, why is a Linux based storage appliance e.g. (or running some
other trusted application under the control of the same administrator) -
more risky than a hypervisor?

As I said, if you don't trust the OS (or the person loading the OS) or
the application, it will likely be isolated physically or by VLAN
and connect through gateways.

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 28 Aug 2012 21:29:37 +0200

My layman intuition (which, as Bruce Schneier pointed out in a recent 
blog post, is a dangerous thing when discussing security) tells me that 
a system running application software is more likely to be subverted 
than a more tightly controlled system, and that a system being 
accessible from untrusted third parties is more likely to be subverted 
than a system protected with proper infrastructure ACLs. But this is all 
irrelevant ...

A Linux version of nvo3 NVE will be available in one for or another 
(after all, KVM, Xen and a few other virtualization solutions run on top 
of Linux), and if someone decides to deploy that on an application 
server or an appliance, that's not our problem.

In retrospect, NVE running on a bare metal Linux server is no different 
from our perspective than a hypervisor running a single VM. Let's move 
on - we have to decide whether an nvo3 domain will be a single trust 
zone (like MPLS/VPN) or whether we want to complicate things and allow 
untrusted NVEs to join the domain in a controlled manner.

Ivan

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 28 Aug 2012 15:11:15 -0700

> That means servers need to have separate physical ports for FCOE traffic so that NVE won't need any mechanism to recognize the FCoE traffic.

Not necessary to have separate physical ports as noted below.

> If servers don't have separate physical ports for FCoE traffic, they use some VLANs for FCoE traffic instead. Then NVEs have to be configured not to process those VLANs.  Is it correct?

Yes, FCoE would typically use its own VLAN(s) and those wouldn't be
configured for forwarding to any of the tunnels at the NVE.

Anoop

---

From: "Ayandeh, Siamack" <[email-masked]>
Date: Wed, 29 Aug 2012 12:35:45 +0000

You can partition the traffic, but would you then lose the properties of NVO3 such as overlapping address space and mobility for storage traffic?

Siamack

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, August 28, 2012 6:11 PM
To: Linda Dunbar
Cc: Black, David; [email-masked]; [email-masked]
Subject: Re: [nvo3] Storage (part of: Let's refocus on real world)

Anoop

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Wed, 29 Aug 2012 19:31:11 +0200

From: Stiliadis, Dimitrios (Dimitri) [mailto:[email-masked]]
[...]
> of a blade system, the server-to-NVE connection is actually multiplexed.

Ignoring for the moment that data centers that would benefit most from NVO3 supposedly don't over-appreciate blade servers - if the server-to-ToR switch connection is multiplexed, then there must be an active element in the blade enclosure. Usually it's a switch. Is there a good reason that switch couldn't be a NVE, but a ToR switch could be?

>  Same holds for "fabric extender" type of architectures.

Fabric extenders using 802.1BR or the S-component of 802.1Qbg present every server NIC as a separate interface to the controlling bridge (ToR switch).

>  One could require that the NVE must always be one hop away and in a  p2p
> connection, but this would limit options.

It would also make the whole setup a lot more secure, as the NVE could reliably enforce per-VM/server security policy. Once you mix traffic sourced from multiple VMs/servers into a VLAN, it's impossible to enforce reliable per-server security.

> looking for active/active configurations.

So maybe we'd finally get a standard version of MLAG? Would be about time ;)

[...] 
> think they require a p2p link.

See above.
Ivan

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 30 Aug 2012 07:37:59 +0000

I fully agree that FCoE should not be considered in the scope of NVo3 due to the following reasons: 1) FCoE is just one option for network consolidation, more importantly, there are other more promising alternatives such as iSCSI which have less requirements on the networks compared to FCoE, e.g., no need for lossless ; 2) Even in the case where FCoE is chosen as a network consolidation, DC operators may prefer to use single-hop FCoE, rather than multi-hop FCoE since the former is easier to achieve “lossless”. In the single-hop FCoE solution, there is no need to transport the FCoE traffic across the tunnels since the FCoE traffic will leave the ToR where it had entered.

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 Aldrin Isaac
发送时间: 2012年8月29日 0:41
收件人: Somesh Gupta
抄送: Ivan Pepelnjak; Black, David; [email-masked]; Stiliadis, Dimitrios (Dimitri); Linda Dunbar
主题: Re: [nvo3] Let's refocus on real world (was: Comments on Live Migration and VLAN-IDs)

The question regarding FCoE is whether overlay solutions need to transport it.  I think the answer is no.  If something operates at the underlay level than it isn't in scope for NVo3, including DCB.

On Tuesday, August 28, 2012, Somesh Gupta wrote:

> From: [email-masked]<javascript:;> [mailto:[email-masked]<javascript:;>] On Behalf Of
> Ivan Pepelnjak
> Cc: Black, David; [email-masked]<javascript:;>; Linda Dunbar