Subject: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

From: Benson Schliesser <[email-masked]>
Date: Sat, 16 Jun 2012 20:01:19 -0500

Dear NVO3 Participants -

This message begins a two week Call for Adoption of http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement-02 by the NVO3 working group, ending on 30-June-2012.

Please respond to the NVO3 mailing list with any statements of approval or disapproval, along with any additional comments that might explain your position. Also, if any NVO3 participant is aware of IPR associated with this draft, please inform the mailing list and/or the NVO3 chairs.

Thanks,
-Benson & Matthew

---

From: Dhruv Dhody <[email-masked]>
Date: Tue, 19 Jun 2012 09:46:42 +0530

Hi, 

Support! 

Here are a few comments/queries ...

(1) Sec 2.2
"A key requirement for live migration is that a VM retain 
its IP address(es) and MAC address(es) in its new location 
(to avoid tearing down existing communication)."

IP address must be retained but couldn't live migration be still supported
if the MAC address of the VM changes in the routed semantics? 

(2) Sec 3.3
"The first component consists of a back-end "oracle" that
is responsible for distributing and maintaining the mapping
information for the entire overlay system."

Should we use the word "oracle" in the ID? IMHO using a more generic term
would be much better. 

Regards,
Dhruv

****************************************************************************
***
Dhruv Dhody, System Architect, Huawei Technologies, Bangalore, India, Ph.
+91-9845062422

This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Benson Schliesser
Sent: Sunday, June 17, 2012 6:31 AM
To: [email-masked]
Cc: Matthew (Matthew) Bocci;
[email-masked]
Subject: [nvo3] call for adoption:
draft-narten-nvo3-overlay-problem-statement-02

Dear NVO3 Participants -

This message begins a two week Call for Adoption
of http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement-02
 by the NVO3 working group, ending on 30-June-2012.

Please respond to the NVO3 mailing list with any statements of approval or
disapproval, along with any additional comments that might explain your
position. Also, if any NVO3 participant is aware of IPR associated with this
draft, please inform the mailing list and/or the NVO3 chairs.

Thanks,
-Benson & Matthew

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 09:30:51 -0400

Hi Dhruv,

Thank you for your comments and support.

> if the MAC address of the VM changes in the routed semantics?

In full generality, no.  For example, some VMs contain software whose
license is keyed by the VM's MAC address.  If the MAC address changes, the
software stops working.

There are some VMs for which a MAC address change may work, but the
MAC address change is visible to the VM's OS - the device driver for the
virtual NIC has to reconfigure, and that's potentially disruptive and
non-transparent depending on what that device driver and OS can cope with.

> Should we use the word "oracle" in the ID? IMHO using a more generic term

Please suggest an alternative word.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dhruv
> Dhody
> Sent: Tuesday, June 19, 2012 12:17 AM
> To: 'Benson Schliesser'; [email-masked]
> Cc: 'Matthew (Matthew) Bocci'; draft-narten-nvo3-overlay-problem-
> [email-masked]
> Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-
> statement-02
> 
> Hi,
> 
> Support!
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 19 Jun 2012 09:35:18 -0500

>Please suggest an alternative word.

I believe the original proposals of this concept used the term 
"directory service" (see VL2 paper etc).

Dimitri

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 14:35:56 -0400

Hmm ... I suspect that people who are interested in use of routing protocols to
distribute the mapping information might object to that term.

Thanks,
--David

> From: Stiliadis, Dimitrios (Dimitri) [mailto:dimitri.stiliadis@alcatel-
> lucent.com]
> Sent: Tuesday, June 19, 2012 10:35 AM
> To: Black, David; [email-masked]; [email-masked]; [email-masked]
> Cc: Bocci, Matthew (Matthew); draft-narten-nvo3-overlay-problem-
> [email-masked]
> Subject: RE: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-
> statement-02
>

---

From: Dhruv Dhody <[email-masked]>
Date: Wed, 20 Jun 2012 11:17:16 +0530

There is term used in the ID itself - "centralized directory-based database
server"

Or

Lets come up with a new term say - 

Network Virtualization Map Server (NVMS) 
Overlay Map Server (OMS) 

[The framework document could be updated to mention it as well :)]

Regards,
Dhruv

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, June 20, 2012 12:06 AM
> To: [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Cc: [email-masked]; draft-narten-nvo3-overlay-problem-
> [email-masked]
> Hmm ... I suspect that people who are interested in use of routing
protocols
> to
> > > Please respond to the NVO3 mailing list with any statements of
approval
> or
> > > position. Also, if any NVO3 participant is aware of IPR associated
with
> this

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 21 Jun 2012 00:44:06 -0700

Benson,

> Dear NVO3 Participants -
> 
> This message begins a two week Call for Adoption of 
> http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement-02 
> Please respond to the NVO3 mailing list with any statements of approval 
> or disapproval, along with any additional comments that might explain 
> your position. Also, if any NVO3 participant is aware of IPR associated 
> with this draft, please inform the mailing list and/or the NVO3 chairs.

Before considering adoption of draft-narten-nvo3-overlay-problem-statement,
we should consider how this draft is related to
draft-fang-vpn4dc-problem-statement (see below) ? Do these two drafts 
cover the same topic ? Are they complementary with each other ?
Should both of these drafts be merged into a single draft, before
being adopted by the NVO3 working group ?

Yakov.

-----------------------------------------------------------------------
Date:    Wed, 20 Jun 2012 10:50:53 CDT
To:      <[email-masked]>
cc:      Pedro Marques <[email-masked]>, [email-masked],
	 "Dennis Cai \(dcai\)" <[email-masked]>
From:    "Luyuan Fang (lufang)" <[email-masked]>
Subject: [nvo3] FW: New Version Notification fordraft-fang-vpn4dc-problem-sta
     ***tement-01.txt

Dear colleagues,

We have uploaded a new updated version of draft-fang-vpn4dc-problem-statement
-01.txt (on 6/12), by Maria Napierala, Dennis Cai, and myself.
Appreciate your review and comments.

Thanks,
Luyuan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, June 12, 2012 10:48 PM
To: Luyuan Fang (lufang)
Cc: [email-masked]; Dennis Cai (dcai)
Subject: New Version Notification fordraft-fang-vpn4dc-problem-statement-01.t
xt

A new version of I-D, draft-fang-vpn4dc-problem-statement-01.txt
has been successfully submitted by Luyuan Fang and posted to the
IETF repository.

Filename:	 draft-fang-vpn4dc-problem-statement
Revision:	 01
Title:		 IP-VPN Data Center Problem Statement and Requirements
Creation date:	 2012-06-12
WG ID:		 Individual Submission
Number of pages: 18
URL:             http://www.ietf.org/internet-drafts/draft-fang-vpn4dc-proble
m-statement-01.txt
Status:          http://datatracker.ietf.org/doc/draft-fang-vpn4dc-problem-st
atement
Htmlized:        http://tools.ietf.org/html/submission.filename }}-01
Diff:            http://tools.ietf.org/rfcdiff?url2=draft-fang-vpn4dc-problem
-statement-01

Abstract:
   Network Service Providers commonly use BGP/MPLS VPNs [RFC 4364] as
   the control plane for virtual networks. This technology has proven
   to scale to a large number of VPNs and attachment points, and it is
   well suited for Data Center connectivity, especially when
   supporting all IP applications.

   The Data Center environment presents new challenges and imposes
   additional requirements to IP VPN technologies, including multi-
   tenancy support, high scalability, VM mobility, security, and
   orchestration. This document describes the problems and defines the
   new requirements.

The IETF Secretariat

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 21 Jun 2012 11:05:24 +0100

> being adopted by the NVO3 working group ?

Yakov

How long will you need to do the analysis that you are proposing?

Stewart

---

From: John E Drake <[email-masked]>
Date: Thu, 21 Jun 2012 03:51:49 -0700

Benson,

The WG has yet to have its first meeting and we have yet to see the drafts which will be presented at it.  Generally, requirements and framework drafts adopted by the WG as WG drafts are a synthesis of many individual drafts,

So, it is extremely premature to be adopting the subject draft as a WG draft.

Thanks,

John

Sent from my iPhone

>Yakov Rekhter
>Sent: Thursday, June 21, 2012 12:44 AM
>To: Benson Schliesser
>Cc: Matthew (Matthew) Bocci; Luyuan Fang (lufang); [email-masked]; Dennis
>Cai (dcai); Pedro Marques; draft-narten-nvo3-overlay-problem-
>[email-masked]; [email-masked]
>Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-
>problem-statement-02
>
>Benson,
>
>> Dear NVO3 Participants -
>>
>> http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement
>> -02 by the NVO3 working group, ending on 30-June-2012.
>>
>> approval or disapproval, along with any additional comments that might
>> explain your position. Also, if any NVO3 participant is aware of IPR
>> associated with this draft, please inform the mailing list and/or the
>NVO3 chairs.
>
>we should consider how this draft is related to draft-fang-vpn4dc-
>problem-statement (see below) ? Do these two drafts cover the same topic
>? Are they complementary with each other ?
>Should both of these drafts be merged into a single draft, before being
>adopted by the NVO3 working group ?
>
>Yakov.
>

---

From: Lucy yong <[email-masked]>
Date: Thu, 21 Jun 2012 14:45:12 +0000

Fully agree.
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of John E Drake
Sent: Thursday, June 21, 2012 5:52 AM
To: Yakov Rekhter; Benson Schliesser
Cc: Matthew (Matthew) Bocci; Luyuan Fang (lufang); [email-masked]; Dennis Cai (dcai); Pedro Marques; [email-masked]; [email-masked]
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

Benson,

Thanks,

John

Sent from my iPhone

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jun 2012 12:16:05 -0800

On 6/21/12 2:51 AM, John E Drake wrote:
> The WG has yet to have its first meeting and we have yet to see the
> drafts which will be presented at it.  Generally, requirements and
> framework drafts adopted by the WG as WG drafts are a synthesis of
> many individual drafts,

???  Generally, I don't think so.  From time to time, to be sure,
but "generally?"  No.

At any rate I think this draft is a very good starting point and it
seems to me that unless there are *specific* reasons related to
content or structure not to adopt the draft, I don't see a reason not
to.  Progressing a working group draft is a collaborative process and
one, I think, that seems healthier and more likely to be able to
result in a consensus to publish than having what's been referred to
elsewhere as a "contest of champions."

Do you have reasons to not want to adopt that have something to do with
content?

Melinda

---

From: John E Drake <[email-masked]>
Date: Thu, 21 Jun 2012 14:22:30 -0700

Comments inline.

Sent from my iPhone

>Melinda Shore
>Sent: Thursday, June 21, 2012 1:16 PM
>To: [email-masked]
>???  Generally, I don't think so.  From time to time, to be sure, but
>"generally?"  No.

JD:  I'm sorry, 'generally' is within the context of the Routing Area, going back to the early days of MPLS.

>seems to me that unless there are *specific* reasons related to content
>or structure not to adopt the draft, I don't see a reason not to.

JD:    I disagree.  Other requirements drafts, e.g., draft-fang-vpn4dc-problem-statement, have been published and others are in the works.  What is so special about draft-narten?

>Progressing a working group draft is a collaborative process and one, I
>think, that seems healthier and more likely to be able to result in a
>consensus to publish than having what's been referred to elsewhere as a
>"contest of champions."

JD:  I expect a multiplicity of requirements and framework drafts that will be synthesized into a single requirements and a single framework draft as the WG proceeds.  After a time, these will be adopted as WG drafts.

>Do you have reasons to not want to adopt that have something to do with

JD:  When did you decide it was your prerogative to specify the basis on which to adopt a draft as a WG draft?

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jun 2012 13:31:10 -0800

On 6/21/12 1:22 PM, John E Drake wrote:
> JD:    I disagree.  Other requirements drafts, e.g.,
> draft-fang-vpn4dc-problem-statement, have been published and others
> are in the works.  What is so special about draft-narten?

It's the one that has been proposed as the basis for the deliverable.
I don't understand why the others weren't, if that was why they
were written.

> on which to adopt a draft as a WG draft?

I'm just trying to understand your objection, to see if there's
something there I can agree with.  Part of that is clarifying whether
your complaint is technical or political.

Melinda

---

From: John E Drake <[email-masked]>
Date: Thu, 21 Jun 2012 14:48:11 -0700

Comments inline

Sent from my iPhone

>From: Melinda Shore [mailto:[email-masked]]
>Sent: Thursday, June 21, 2012 2:31 PM
>To: John E Drake
>Cc: [email-masked]
>I don't understand why the others weren't, if that was why they were
>written.

JD:  That is completely incorrect.  From the charter:

"The NVO3 WG will write the following informational RFCs, which must have completed Working Group Last Call before rechartering can be considered: 

Problem Statement 
Framework document 
Control plane requirements document 
Data plane requirements document 
Operational Requirements 
Gap Analysis"

>your complaint is technical or political.

JD:  Process.  Draft-narten was written by a group of people with a particular perspective.  In particular, they are blissfully unaware of most of the technologies developed by the IETF over the past fifteen years.  (Also technologies developed by the IEEE, but I digress.)

Other drafts will be written by groups with other perspectives, and the expectation is that by considering all of these drafts on an *equal* basis, we will have a more complete understanding of the problem space.

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jun 2012 14:35:55 -0800

On 6/21/12 1:48 PM, John E Drake wrote:
> JD:  Process.  Draft-narten was written by a group of people with a
> particular perspective.  In particular, they are blissfully unaware
> of most of the technologies developed by the IETF over the past
> fifteen years.  (Also technologies developed by the IEEE, but I
> digress.)

Okay, *that's* the basis for an objection, not that it's too soon
(although I am grateful to have seen a complaint that the IETF is
moving too quickly on something - I may never see another, and I'm
cherishing the moment).

Look, if the draft is irredeemable or there's a more suitable one
available now, say so.  My tendency is to go into these things
recognizing that drafts are changed *substantially* between the
time that one is adopted and the time it enters WG last call, and
that's normal, healthy, and desirable.  If an editor is unwilling
to accept a working group decision on something, they're fired.

I just don't think "too soon" is an acceptable or even reasonable
criterion for not adopting a document and the "too soon" argument
is exceptionally unlikely to lead me to agree with you.  You seem
to be suggesting that you want a draft to be close to being able
to get into WG LC before adopting it.  Seriously - what is the
problem with the *document* (calling the authors uninformed is not
the same thing as saying that the document is flawed), and why is
it so serious as to preclude adoption by the working group?

Melinda

---

From: John E Drake <[email-masked]>
Date: Fri, 22 Jun 2012 05:05:49 -0700

Melinda,

My original email indicated that there were other requirements drafts in the works and that it was premature to make draft-narten a WG draft prior to seeing them.

If we really, really, have to have a requirements draft selected as a WG draft immediately, and I have seen no reason expressed for doing this, let alone a compelling reason for doing this, I would propose draft-fang-vpn4dc-problem-statement.

It is better written, has a better understanding of the problem space, and has at least a passing familiarity with technologies developed in the IETF and IEEE.

Thanks,

John 

Sent from my iPhone

>Sent: Thursday, June 21, 2012 3:36 PM
>To: John E Drake
>Cc: [email-masked]
>> particular perspective.  In particular, they are blissfully unaware of
>> most of the technologies developed by the IETF over the past fifteen
>> years.  (Also technologies developed by the IEEE, but I
>> digress.)
>
>(although I am grateful to have seen a complaint that the IETF is moving
>too quickly on something - I may never see another, and I'm cherishing
>the moment).
>
>recognizing that drafts are changed *substantially* between the time
>that one is adopted and the time it enters WG last call, and that's
>normal, healthy, and desirable.  If an editor is unwilling to accept a
>working group decision on something, they're fired.
>
>criterion for not adopting a document and the "too soon" argument is
>exceptionally unlikely to lead me to agree with you.  You seem to be
>suggesting that you want a draft to be close to being able to get into
>WG LC before adopting it.  Seriously - what is the problem with the
>*document* (calling the authors uninformed is not the same thing as
>saying that the document is flawed), and why is it so serious as to
>preclude adoption by the working group?
>
>Melinda

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 22 Jun 2012 07:57:25 -0700

Benson,
I object to the document on the following points:

1) It doesn't appear to recognize that there are multiple type of data-center deployments with different requirements.
2) It appears to "dismiss" RFC4364 based on the requirements of section 2.7.
3) Does not discuss the requirements for inter-CUG traffic.

On the first point:
  - One can place data-center design approaches on a continuum where at one extreme one encounters data-centers that in general:
       o Have a small number of physical servers per cluster (orchestration system scope).
       o Rely on hardware resiliency.
       o Use a single VM per application tier.
       o The VM lifespan is very large.
       o Use a SAN or local disks for storage.
       o Use a oversubscribed network that is L2 based.
       o Tend to exchange traffic within a "VLAN"

    At the other end of the spectrum you have data-centers in which the design points are in general:
      o Very large number of physical servers.
      o Rely on software resiliency.
      o Use multiple VMs per app tier, often 10s or 100s.
      o The VM lifespan is small since VMs are created and deleted according to load requirements.
      o Use a distributed block storage and distributed NAS file-systems.
      o Use a non-oversubscribed network from the TOR up that is L3 only and typically does not support multicast.
      o Most of the traffic crosses "closed user group" boundaries.

Between these two extremes there is often non common elements: server hardware, storage, software architectures and network are completely different. The document in question seems to be focusing on solutions at the left end of the spectrum (first approach) and does not at all reflect the requirements of data-ceterns at the middle or at the right end of the spectrum (the latter approach).

On the second point:
  - Section 2.7 of the document states that the overlay technology must take into account:
    1. Highly distributed systems
    2. Many highly distributed virtual networks with sparse membership.
    3. Highly dynamic end systems.
    4. Work with existing […] routers and switches.
    5. administered by a single administrative domain

L3VPN has proven to work in highly distributed systems, with many distributed virtual networks with space membership and with highly dynamic rechability information. It can be encapsulated as MPLS over GRE which implies IP packets and will work with any existing routers and switches. It can be administered by a single administrative domain.

Among the related work that is cited, i don't believe that trill, .1aq and arms can claim to work with existing routers and swathes. And none of the approaches has proven the same sort of route scaling as l3vpn.

thank you,
  Pedro.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Fri, 22 Jun 2012 10:35:40 -0500

I believe that Section 3.1 needs to be significantly revised since it has some confusing
statements. I cannot support this draft without a revision of this Section:

" VRF's are a pure routing construct and do not have end-to-end
   significance in the sense that the data plane carries a VRF indicator
   on an end-to-end basis."

-          Why is this a problem? The control plane can propagate the end-to-end

significance and there is nothing that says that the identifier needs to be global.

(ie why is a global identifier a requirement? Note, that the framework draft

makes this clear).

"  Instead, the VRF is derived at each hop
   using a combination of incoming interface and some information in the
   frame (e.g., local VLAN tag).  Furthermore, the VRF model has
   typically assumed that a separate control plane governs the
   population of the forwarding table within that VRF."

-          Obviously RFC 4364 does NOT require multiple control planes, (ie one for

each VRF as it is suggested in the statement above. The protocol is designed

for multi-tenancy. In an earlier statement, this is clarified, but then it is dismissed

here.

"  Thus, a
   traditional VRF model assumes multiple, independent control planes
   and has no specific tag within a data frame to identify the VRF of
   the frame."

-          Both of these statements are incorrect.  The VRF model does NOT assume

multiple independent control planes, and it has a tag that identifies frames

that belong in a VRF (as Pedro mentioned it can be an MPLS label in MPLS

over GRE).

There could be other reasons (deployment, familiarity, etc) that RFC4364 cannot be
the only solution to this problem space. However, none of the technical reasons stated
above make any sense and they cannot be part of the problem statement.

Dimitri

From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Saturday, June 16, 2012 6:01 PM
To: [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Jun 2012 08:05:42 -1000

> 2) It appears to "dismiss" RFC4364 based on the requirements of

isn't that for the nov2 or nov2.5 wg?

randy

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 10:25:47 -0400

Hi Pedro,

Could you provide some examples of the requirements differences between
the types of data centers? I'm not sure that a data center taxonomy is
an important addition to the draft, but I'm definitely interested in
what may be missing in area of requirements.  Specific requirements
belong in requirements drafts, but I want to understand how you see
the differences affecting what the WG needs to support.

I don't understand your assertion that Section 2.7 appears to "dismiss"
RFC 4364, when the latter part of your message appears to describe how
RFC 4364 (BGP/MPLS VPNs) meets the requirements in that section.  What
did I miss?

I will admit that I don't expect to see BGP implemented/deployed in
hypervisor softswitches (although OSPF as the edge protocol for BGP/MPLS
VPNs seems more realistic for that implementation location), and hence
I'm interested in a standard protocol for NVEs to talk to an "oracle"
which could be a set of network nodes that use a routing protocol to
distribute information among themselves .  That possibility of a distributed
structure is not easy to infer from the current draft, so we probably
should add text to describe it ... and knock out the one parenthetical
remark that the "oracle" is directory-based.  I don't care how the
"oracle" is structured internally (a routing protocol is a fine way
to distribute information), although NVEs that don't directly participate
in the "oracle" probably view the "oracle" as some sort of directory on
which NVEs perform address lookups.

Inter-VN traffic (what you refer to as inter-CUG traffic) is handled by
a straightforward application of IP routing to the inner IP headers; this
is similar to the well-understood application of IP routing to forward
traffic across VLANs.  We should talk about VRFs as something other than
a limitation of current approaches - for VLANs, VRFs (separate instances
of routing) are definitely a feature, and I expect this to carry forward
to nvo3 VNs.  In addition, we need to make changes to address Dimitri's
comments about problems with the current VRF text.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Pedro Roque Marques
Sent: Friday, June 22, 2012 10:57 AM
To: Benson Schliesser
Cc: Matthew (Matthew) Bocci; [email-masked]; [email-masked]
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

Benson,

On the first point:

On the second point:
    4. Work with existing [...] routers and switches.

thank you,
  Pedro.

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Pedro Roque Marques <[email-masked]>
Date: Tue, 26 Jun 2012 08:35:40 -0700

On Jun 26, 2012, at 7:25 AM, <[email-masked]> <[email-masked]> wrote:

> the types of data centers?

Differences in aggregate bandwidth and locality of data access drive very different network designs.

To give you one example: large scale/large aggregate throughput DC networks typically do not support any form of broadcast / multicast. Solutions that rely on broadcast / IP multicast for learning are not very useful for this category of DCs.

> I’m not sure that a data center taxonomy is
> an important addition to the draft, but I’m definitely interested in

The WG needs to support 2 (perhaps more) sets of requirements that are opposite in many respects. Or split into multiple WGs.

Another example: "traditional / legacy" DCs may require the transport of unmodified ethernet frames; large scale/low cost DCs do not have any use for ethernet headers since applications are not allowed to send / receive non IP packets.

It is also unclear to me whether the requirements of section 2.7 do satisfy a realistic scenario. For instance, there are several proposals that are being supported by vendors in the "legacy DC" space that clearly do not scale up… but that may not be a problem for a DC in which the cluster size is 30 machines. 

>  
> I don’t understand your assertion that Section 2.7 appears to “dismiss”

quote:
  "They are not
   necessarily seen as supporting the characteristics outlined in
   Section 2.7."

>  
> I will admit that I don’t expect to see BGP implemented/deployed in
> hypervisor softswitches

That isn't what is being proposed in http://datatracker.ietf.org/doc/draft-marques-l3vpn-end-system/.

> (although OSPF as the edge protocol for BGP/MPLS
> I’m interested in a standard protocol for NVEs to talk to an “oracle”
> remark that the “oracle” is directory-based.  I don’t care how the
> “oracle” is structured internally (a routing protocol is a fine way
> to distribute information), although NVEs that don’t directly participate
> in the “oracle” probably view the “oracle” as some sort of directory on

There are several reasons to have multiple and interoperable "oracles":
  - multiple oracles means multiple failure domains;
  - it means the possibility of operation across multiple administrative domains (or even multiple clusters).
  - and interoperability always ends up creating a bigger ecosystem which in the end benefits network operators.

> traffic across VLANs.

That is where again the differences between different types of data-centers do play in. If for instance 90% of a VMs traffic happens to be between the Host OS and a network attached storage file system run as-a-Service (with the appropriate multi-tenent support) then the question of where are the routers becomes a very important issue. In a large scale data-center where the Host VM and the CPU that hosts the filesystem block can be randomly spread where is the router ?
Is every switch a router ? Does it have all the CUGs present ?
In some DC designs the problem to solve is the inter-CUG traffic. With L2 headers being totally irrelevant.

>   We should talk about VRFs as something other than
> to nvo3 VNs.  In addition, we need to make changes to address Dimitri’s

My overall comment is that the attempt to build a problem statement for all types of DCs is the fundamental issue. There are DCs that have opposite design goals. This results in different server hardware, software, management processes, application architectures, storage solutions. An attempt to create a single unified problem statement is, imho, counter-productive.

Reasonable people, solving totally different problems with opposite goals will necessarily disagree.

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 13:10:34 -0400

Pedro,

A couple of high level responses ...

-- Different requirements

> The WG needs to support 2 (perhaps more) sets of requirements that are
> opposite in many respects. Or split into multiple WGs.

> My overall comment is that the attempt to build a problem statement for
> all types of DCs is the fundamental issue.

That may be.  One of my biggest reactions in reading the VPN4DC problem
statement draft was that I wished that it contained an applicability statement
expanding on (quoting from Section 3's first paragraph) "the assumption that
applications require IP connectivity but no Layer 2 direct adjacencies" as
opposed to the draft's current attempts to argue that a layer 2 service
that preserves such adjacencies is not needed.  As an aside - starting from
that quoted assumption, the absence of a need for such a service is effectively
a foregone conclusion ;-).

I have examples where MAC addressing and MAC adjacencies are required
(and where use of a single or very limited number of MAC addresses for
all virtual machines is unworkable), but they involve enterprise-centric
use cases and typically existing deployed-and-unmodifiable applications
and/or functionality that could be out-of-scope for a well-crafted
VPN4DC applicability statement (e.g., one that picks up some of the
examples from your email).

-- Section 2.7

  >  Section 2.7<http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement-02#section-2.7>."

Ah, I'll need to check w/my co-authors, but I suggest just deleting that
sentence, as it seems to be adding a lot of heat and not much light to the
discussion.

> It is also unclear to me whether the requirements of section 2.7 do satisfy a
> realistic scenario. For instance, there are several proposals that are being
> supported by vendors in the "legacy DC" space that clearly do not scale up...
> but that may not be a problem for a DC in which the cluster size is 30 machines.

Right, and doing something about some of those proposals is one of the opportunities
that I see for nvo3 to contribute.  An obvious example is that flooding-based
data plane learning has scaling limitations, but one can preserve the interface
to the end system even while doing something smarter at the NVE.

Thanks,
--David

From: Pedro Roque Marques [mailto:[email-masked]]
Sent: Tuesday, June 26, 2012 11:36 AM
To: Black, David
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

On Jun 26, 2012, at 7:25 AM, <[email-masked]<mailto:[email-masked]>> <[email-masked]<mailto:[email-masked]>> wrote:

Hi Pedro,

I'm not sure that a data center taxonomy is

It is also unclear to me whether the requirements of section 2.7 do satisfy a realistic scenario. For instance, there are several proposals that are being supported by vendors in the "legacy DC" space that clearly do not scale up... but that may not be a problem for a DC in which the cluster size is 30 machines.

quote:
  "They are not

   Section 2.7<http://tools.ietf.org/html/draft-narten-nvo3-overlay-problem-statement-02#section-2.7>."

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Pedro Roque Marques
Sent: Friday, June 22, 2012 10:57 AM
To: Benson Schliesser
Cc: Matthew (Matthew) Bocci; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

Benson,

On the first point:

On the second point:

thank you,
  Pedro.

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: John E Drake <[email-masked]>
Date: Tue, 26 Jun 2012 11:00:05 -0700

Snipped, comment inline

Sent from my iPhone

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, June 26, 2012 7:26 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

Hi Pedro,

distribute information among themselves.

JD:  Interestingly enough, OSPF v2 & v3 are standard CE-PE interfaces in L3VPNs.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 20:10:29 +0200

Hi John/David,

Where do you see CE and PE there ?

Is hypervisor softswitch a CE in your opinion ? And all VMs running on 
it belong to the same tenant ?

Is it a vrf-lite CE and you are proposing to run as many OSPF instances 
as number of VNs ?

On the other hand if this is a PE I am afraid neither OSPF v2 nor v3 
allow you for easy tenant multiplexing and scalable control plane 
signalling between such PE and the "oracle".

Thx,
R.

> Snipped, comment inline
>
> Sent from my iPhone
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Tuesday, June 26, 2012 7:26 AM
> *To:* [email-masked]
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [nvo3] call for adoption:

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 14:37:34 -0400

Robert,

John will have to answer for CE and PE, as he introduced those acronyms to
this discussion ;-).

My comment is that (IMHO, your O will likely be different), OSPF seems
to be a more reasonable candidate for hypervisor softswitch implementation
by comparison to BGP.  This is mostly in the realm of futures, as hypervisor
softswitches start out as handling L2 traffic with no IP routing protocol
support.

If the "oracle" were based on a routing protocol or protocols for information
distribution with OSPF being in use closest to the NVEs, I could see an approach
where the NVEs directly participate in OSPF.  OTOH, my preference would be
to put OSPF on network nodes and use some other protocol to do the address
mapping lookups from the NVEs to the network nodes, because this preserves OSPF's
current scaling properties/expectations wrt the scale of the physical network.
In contrast, OSPF in softswitches turns every server into an OSPF participant,
thereby changing OSPF's scaling properties wrt the physical network infrastructure
by at least an order of magnitude.

Thanks,
--David

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Tuesday, June 26, 2012 2:10 PM
> To: John E Drake
> Cc: Black, David; [email-masked]; [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 20:45:18 +0200

Hi David,

> by comparison to BGP.

IMHO neither of those is a good idea if we are talking about choice of 
vehicle to carry control plane propagation to hypervisor.

I am of the opinion that XMPP as proposed in l3vpn-end-systems draft is 
a much better choice there.

> by at least an order of magnitude.

My take is that while "oracle(s)" can participate in dynamic routing 
towards the core (data center and or wan) on the other side it would be 
much more convenient to ask them to speak the language which hosts are 
the most familiar with. Extensible xml schema seems at least as one 
possible candidate there. I am sure there could be more, but that a good 
start I think.

Rgs,
R.

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 14:53:01 -0400

Robert,

We need to distinguish "hypervisor" as manager of the VM from the
embedded "softswitch" as a networking component.  The l3vpn-end-systems
draft proposes XMPP for communication that would be between the hypervisor
(as or on behalf of the Tenant End System [TES]) and the NVE.  The mention
of OSPF in this thread has been for NVE access to address mapping information
from the "oracle", and this would be different from use of OSPF to route
traffic on the underlying network (using the outer IP addresses in an
nvo3 encapsulation).

Please keep these use cases separate as they have rather different
characteristics and requirements.  IEEE VDP is also a candidate protocol
for the NVE-TES communication.  One thing that is of some concern to me
about XMPP is that there are attach/detach storms (e.g., call center VDI
startup when all the workers show up at the start of the shift), and I
wonder whether one wants to be parsing XML to deal with every event in
such a storm.

Thanks,
--David

> Sent: Tuesday, June 26, 2012 2:45 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 21:03:36 +0200

David,

> (as or on behalf of the Tenant End System [TES]) and the NVE.

Actually I do not see that those two are need to be decoupled. Linux 
kernel with some additional enhancement module can act as NVE. Tenants 
are just connected to the NVE over normal bridge interfaces by the 
co-located hypervisor.

My reading of l3vpn-end-systems is that XMPP is used between the NVE and 
the "oracle" and NVE = to kernel module = analogy to softswitch (but the 
last is a bit too broad of a term these days).

I am sure Pedro can authoritatively clarify that.

 > and I wonder whether one wants to be parsing XML to deal with every
 > event in such a storm.

There is no storm I envision to happen. By proper filtering of required 
only information to be delivered by given host the filtering happens on 
the oracle instead. RT-Constrain seems perfect to address that task here.

Rgs,
R.

> The mention

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 15:15:25 -0400

> are just connected to the NVE over normal bridge interfaces by the

The salient question is not what's possible, but what's actually being
done ("running code"), as nvo3 can't succeed if it requires major
hypervisor rewrites.

Linux is not the system that supports hypervisors.  The Cisco Nexus
1000V softswitches are examples where there is a strong decoupling
and that softswitch is available for at least two non-Linux hypervisors,
suggesting similar decoupling in the system architectures for those
hypervisors.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert
> Raszuk
> Sent: Tuesday, June 26, 2012 3:04 PM
> To: Black, David

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 21:24:21 +0200

David,

Just as a data point ..

All of current running production data centers and those which I am 
working on to be shortly deployed assume NVE (Network Virtualization 
Edge) like functionality is embedded within the end-host.

It can be a form of prioprietary softswitch, opensource softswitch or 
any other flavor of kernel module acting as such softswitch.

I am not stating everyone does the same .. some may use TOR as a 
softswitch or even furthere though.

But as one requirement I am very interested in making sure the real use 
case of embedded softswitch in the end system is captured and documented 
in the problem statement document.

Thx,
R.

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 15:32:24 -0400

Robert,

We may be in semi-violent agreement ...

> Edge) like functionality is embedded within the end-host.

So far, so good.  As you're using the terminology, the Nexus 1000V is
software that deployed in the end-host.

Nonetheless, the Nexus 1000V a logically separate networking component
(network admin views it as a layer 2 switch that is easily distinguishable
from the hypervisor in which it runs), and the same is true to a lesser
extent of the native softswitches in at least the VMware and Hyper-V
hypervisors (they tend to be more oriented towards server admin management).

> in the problem statement document.

Depending on what you mean by "embedded", we may be in semi-violent
agreement.

Please define "embedded" as you're using the term.

Thanks,
--David

> Sent: Tuesday, June 26, 2012 3:24 PM
> To: Black, David

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 21:48:11 +0200

Hi David,

Honestly I think we are getting more into implementation then 
standardization debate.

What I call a softswitch can be either part of the linux kernel entirely 
as a loadable module, can be partially in the kernel and partially in 
the user space or if you follow NETMAP's direct memory mapping from 
interface to user space could sit completely there.

Examples of softswitches which I am experimenting with is as you said 
one from VMWare, the other most popular is OVS, there is LINK released 
recently and there is at least few more in the works which would be 
partially sitting in the end-system kernel or the kernel and user space.

At least this is what I meant by using the term "embedded".

The main point I think is that the L2/L3 virtualization/separation will 
be happening in all of the above cases which share one very important 
common characteristic - thay are all residing on the end host.

Best regards,
R.

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 15:59:27 -0400

Then I think we're in semi-violent agreement, as I strongly concur that those
softswitches are important NVE locations.  This is captured in the narten
problem statement draft, but if you have specific text changes to suggest,
please do.

I do want to see protocol interfaces on both sides of these softswitch NVEs -
VM-facing for attach/detach, etc. and "oracle"-facing for address mapping
lookup.  Designing these as protocol interfaces results in the same structure
of functional interactions when the NVE is offboard on a ToR switch or the like.  

Thanks,
--David

> Sent: Tuesday, June 26, 2012 3:48 PM
> To: Black, David
> Cc: [email-masked]
> Subject: Re: [nvo3] Softswitches
> 
> Hi David,
> 
> Robert

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 26 Jun 2012 22:05:35 +0200

> problem statement draft, but if you have specific text changes to suggest,

Great.

> of functional interactions when the NVE is offboard on a ToR switch or the like.

I am not sure we need to spend much time on the NVE to VM protocol 
interface. I think the orchestration either directly or via hypervisor 
control should be sufficient for VM instantiation as well as you 
mentioned attach/detach messaging.

NVE-2-oracle and oracle-2-rest_of_the_network I highly agree. But I am 
not sure if one size will fit all there.

R.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 26 Jun 2012 15:20:29 -0500

> mentioned attach/detach messaging.

If the NVE is on a ToR, then there is a need for an attach/detach
protocol method hypervisors and ToRs.

> not sure if one size will fit all there.

I think that David has a valid point trying to separate the two and we 
need both.

Dimitri

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 26 Jun 2012 21:29:40 +0000

> protocol method hypervisors and ToRs.

[Linda] Does it assume that hypervisors will need to add this new attach/detach function? 

IEEE802.1Qbg defines a Virtual Interface Discovery Protocol (VDP) which require Hypervisors to send Port Profile upon a new VM is added. However, the challenge is that Data Center operators may not be able to dictate what servers/hypervisors being deployed in their data centers. 

If you can't dictate any changes to deployed hypervisors, "ARP/ND" might be able to be used as a way to announce the attachment, and timeout to declare the "detachment".

My two cents. 

Linda Dunbar

---

From: Melinda Shore <[email-masked]>
Date: Tue, 26 Jun 2012 13:54:47 -0800

On 6/26/12 11:03 AM, Robert Raszuk wrote:
> kernel with some additional enhancement module can act as NVE.

Hi, Robert:  You're describing one implementation.  They really
are two separate logical entities.

Melinda

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 19:16:40 -0400

Hi Linda,

> IEEE802.1Qbg defines a Virtual Interface Discovery Protocol (VDP) which
> require Hypervisors to send Port Profile upon a new VM is added. However, the
> challenge is that Data Center operators may not be able to dictate what
> servers/hypervisors being deployed in their data centers.

Yes, VDP is a candidate for the attach/detach protocol.

Thanks,
--David

> From: Linda Dunbar [mailto:[email-masked]]
> Sent: Tuesday, June 26, 2012 5:30 PM
> To: Stiliadis, Dimitrios (Dimitri); [email-masked]; Black, David
> Cc: [email-masked]
> Subject: RE: [nvo3] Softswitches
> 
> 
> >
> IEEE802.1Qbg defines a Virtual Interface Discovery Protocol (VDP) which
> require Hypervisors to send Port Profile upon a new VM is added. However, the
> challenge is that Data Center operators may not be able to dictate what
> servers/hypervisors being deployed in their data centers.
> 
> If you can't dictate any changes to deployed hypervisors, "ARP/ND" might be
> able to be used as a way to announce the attachment, and timeout to declare
> the "detachment".
> 
> 
> My two cents.
> 
> Linda Dunbar

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Tue, 26 Jun 2012 18:20:36 -0500

> [Linda] Does it assume that hypervisors will need to add this new

There are different implementations possible for this. It could be
hypervisor driven or orchestration driven. My understanding is that this
can be done today in all current hypervisors, although by using different 
methods (thus the need for a standard).

> require Hypervisors to send Port Profile upon a new VM is added. However, the

This is one possible approach, but not the only one. It hasn't been
adopted by several hypervisor vendors for various reasons.

> able to be used as a way to announce the attachment, and timeout to declare

I don't believe there is enough information in ARP to do this attachment in 
a multi-tenant environment. Who send the ARP and why should I trust them. 

Dimitri

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 27 Jun 2012 16:00:14 +0000

Dimitri, 

There are much (much) more applications/Guest OSs being deployed than switches/routers. It is hard for any data centers to throw away all their deployed servers/hypervisors in order to deploy new protocols coming from IETF. As you said that there are issues associated with IEEE802.1Qbg, how do you guarantee that new protocols coming out won't have any issues? 

ARP/ND may not be dependable, but most Guest OSs (except very small percentage of Non-IP applications) do send them out. 

Linda 

> Sent: Tuesday, June 26, 2012 6:21 PM
> To: Linda Dunbar; [email-masked]; [email-masked]
> Cc: [email-masked]
> > > > NVE-2-oracle and oracle-2-rest_of_the_network I highly agree. But
> I
> > > am
> > > > >> What I call a softswitch can be either part of the linux
> kernel
> > > entirely
> > > > >> as a loadable module, can be partially in the kernel and
> partially
> > > in
> > > > >> Examples of softswitches which I am experimenting with is as
> you
> > > said
> > > > >> recently and there is at least few more in the works which
> would
> > > be
> > > > >> partially sitting in the end-system kernel or the kernel and
> user
> > > space.
> > > > >>
> > > > >> The main point I think is that the L2/L3
> virtualization/separation
> > > will

---

From: John E Drake <[email-masked]>
Date: Thu, 28 Jun 2012 08:39:36 -0700

Robert,

I expect the CE-PE interface, aka VPN service interface, will be implemented between a multiplicity of devices.  However, my point was a different one.

Viz, L3/L2 VPNs do not require BGP as the CE-PE interface.  So, rather than BGP being a 'non-starter', it is invisible.  L3VPNs also support OSPF v2 & V3, as well as others I have probably forgotten.  E-VPNs support STP, LAG, LLDP, etc.  There are also Pedro's proposal and the IEEE's VDP.

Thanks,

John

Sent from my iPhone

>Sent: Tuesday, June 26, 2012 11:10 AM
>To: John E Drake
>Cc: [email-masked]; [email-masked]; [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 28 Jun 2012 18:09:08 +0200

> Robert,
>
> I expect the CE-PE interface, aka VPN service interface, will be
> implemented between a multiplicity of devices.  However, my point was
> a different one.

And my point is that I do not really see a clear need to run any PE-CE 
dynamic routing protocol if we assume that PE is running on the end host.

I am also not actually sure if this is in scope of NVO3 to worry about 
PE-CE dynamic routing protocol as CEs would most likely be VMs 
regardless where the PE is and therefor to simplify the architecture 
they may just default to the PE. Such default would be provisioned by 
the orchestration system just like the IP address of the VM interface so 
IMHO out of scope of this work.

Thx,
R.

> Viz, L3/L2 VPNs do not require BGP as the CE-PE interface.  So,
> rather than BGP being a 'non-starter', it is invisible.  L3VPNs also
> support OSPF v2 & V3, as well as others I have probably forgotten.
> E-VPNs support STP, LAG, LLDP, etc.  There are also Pedro's proposal
> and the IEEE's VDP.
>
> Thanks,
>
> John

---

From: Lucy yong <[email-masked]>
Date: Thu, 28 Jun 2012 16:24:13 +0000

Robert,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Thursday, June 28, 2012 11:09 AM
To: John E Drake
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] call for adoption: draft-narten-nvo3-overlay-problem-statement-02

[[LY]] I pretty agree on the view. VM was typically configured with its IP address, GW IP address, DNS address. Here seems no longer need to configure GW IP address in an VN, default to the NVE. I think this should be within the scope of nv03.

Lucy

Thx,
R.

---

From: John E Drake <[email-masked]>
Date: Thu, 28 Jun 2012 10:25:32 -0700

Okay, thanks for clarifying.

Sent from my iPhone

>Sent: Thursday, June 28, 2012 9:09 AM
>To: John E Drake

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:47:56 -0400

> Should we use the word "oracle" in the ID? IMHO using a more generic term

I don't particularly like the term "oracle" myself, but that was the
best I was able to come up with. Other suggestions welcome.

Note: the whole point of using a term like "oracle" is that it says
nothing about how the oracle itself is implemented. As others have
mentioned, other terms (like "directory based") have "bagggage"
associated with them that could be read to imply particular solution
approach.

The oracle itself could be implemented as a directory or database of
some sort (single instance, distributed, or whatever).

Or it could be implemented via an existing (or modified) routing
protocol (BGP, IS-IS, etc.).

Or it could be implemented as part of the orchestration systems used
to manage/migrate VMs.

Because there are a range of plausible approaches for implementing the
oracle, it seems desirable to use a standardized protocol for the
NVE-oracle interaction. The NVE can then use a single protocol to
query the oracle for the mappings it doesn't have, or to push mappings
to the oracle for any VM's the NVE supports without caring how the
oracle itself is implemented/architected.

If we have a separate protocol, we aren't tying the NVE to a
particular implementation/approach for the oracle.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:48:30 -0400

Robert,

> in the problem statement document.

Can you clarify your terminology a bit when you say "softswitch in the
end system"?

Do you mean "end system" as in TES (per the framework document
terminology), i.e., within the VM itself?

Or do you mean as part of a hypervisor supporting TES/VMs?

Or something else?

Thanks!

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:52:04 -0400

> 1) It doesn't appear to recognize that there are multiple type of 
> data-center deployments with different requirements.

I'd like to hear more from others on this point.

I don't see right off how discussions on different data center
deployments types actually changes the problem statement
document. Yes, there are many different ones. But trying to describe
them in any sort of detail will be challenging, as ARMD discovered.

But more importantly, one of the reasons for doing overlays is to
separate the details of the underlay network from the overlay. i.e.,
just layer an overlay on top of whatever underlay deployment a
particular DC has chosen to use. That way one can change/tweak the
underlay more easily.

Please provide some examples of differences in DC deployments and
*how* they impact NVO3, either from a problem statement perspective or
from a framework perspective.

In another message:

> > Hi Pedro,
> >
> > Could you provide some examples of the requirements differences
> > between the types of data centers?

> To give you one example: large scale/large aggregate throughput DC 
> networks typically do not support any form of broadcast / multicast. 
> Solutions that rely on broadcast / IP multicast for learning are not 
> very useful for this category of DCs.

I agree that support for multicast in the underlay cannot be
assumed. Pretty much everyone I've asked on this point says the same
thing. So I think one of the requirements will need to be emulation of
multicast (at the VN level) across underlays that do not support (or
want to enable) multicast. But that is a requirements discussion, not
a problem statement issue.

Section 4.2.3 of the framework draft covers this.

I also agree that solutions that rely on multicast for learning are
only useful for smallish data centers. But we've been saying that all
along.  That is, one of the key work areas (as described in the
problem statement) is having a separate oracle that can be queried for
mappings precisely so NVE's do not have to do learning.

> Another example: "traditional / legacy" DCs may require the
> transport of unmodified ethernet frames; large scale/low cost DCs do
> not have any use for ethernet headers since applications are not
> allowed to send / receive non IP packets.

I believe it is perfectly reasonably for an overlay to provide two
almost identical services. L2 services (which preserves ethernet
frames at an e2e level) and L3 service, for those cases where a
deployment is happy with IP only emulation.

While there clearly are some differences in those two service models,
I think both can be accomodated by the same NVO3 framework (and
problem statement). Indeed, that is IMO very much a goal of this
WG. Earlier versions of the charter had explicit text about this, that
was later removed. But I see no reason why it should not continue to
be a goal of this WG, even though not explicitly called out in the
current charter.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:53:02 -0400

> 2) It appears to "dismiss" RFC4364 based on the requirements of section 

OK. This is clearly a contentious point and will need to be deferred
to when we do a gap analysis.

Proposed new text:

	  There are a number of VPN approaches that provide some if
	  not all of the desired semantics of virtual networks. A gap
	  analysis will be needed to assess how well existing
	  approaches satisfy the requirements.

To replace:

	  There are number of VPN approaches that provide some of the
	  desired semantics of virtual networks
	  (e.g., <xref target="RFC4364"></xref>). But VPN approaches
	  have traditionally been deployed across WANs and have not
	  seen widespread deployment within enterprise data
	  centers. They are not necessarily seen as supporting the
	  characteristics outlined in Section 2.7.

Does that work?

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:55:59 -0400

> 3) Does not discuss the requirements for inter-CUG traffic.

Given that the problem statement is not supposed to be the
requirements document,, what exactly should the problem statement say
about this topic?

<[email-masked]> writes:

> Inter-VN traffic (what you refer to as inter-CUG traffic) is handled
> by a straightforward application of IP routing to the inner IP
> headers; this is similar to the well-understood application of IP
> routing to forward traffic across VLANs.  We should talk about VRFs
> as something other than a limitation of current approaches - for
> VLANs, VRFs (separate instances of routing) are definitely a
> feature, and I expect this to carry forward to nvo3 VNs.  In
> addition, we need to make changes to address Dimitri's comments
> about problems with the current VRF text.

Pedro Roque Marques <[email-masked]> writes:

> That is where again the differences between different types of
>  data-centers do play in. If for instance 90% of a VMs traffic
>  happens to be between the Host OS and a network attached storage
>  file system run as-a-Service (with the appropriate multi-tenent
>  support) then the question of where are the routers becomes a very
>  important issue. In a large scale data-center where the Host VM and
>  the CPU that hosts the filesystem block can be randomly spread
>  where is the router ?

Where is what router? Are you assuming the Host OS and NAS are in the
different VNs? And hence, traffic has to (at least conceptually) exit
one VN and reenter another whenever there is HostOS - NAS traffic?

> Is every switch a router ? Does it have all the CUGs present ?

The underlay can be a mixture of switches and routers... that is not
our concern. So long as the underlay delivers traffic sourced by an
ingress NVE to the appropriate egress NVE, we are good.

If there are issues with the actual path taken being suboptimal in
some sense, that is an underlay problem to solve, not for the overlay.

> In some DC designs the problem to solve is the inter-CUG
> traffic. With L2 headers being totally irrelevant.

There is an underlying assumption in NVO3 that isolating tenants from
each other is a key reason to use overlays. If 90% of the traffic is
actually between different tenants, it is not immediately clear to me
why one has set up a system with a lot of "inter tenant" traffic. Is
this is a case we need to focus on optimizing?

But in any case, if one does have inter-VN traffic, that will have to
get funneled through a "gateway" between VNs, at least conceptually, I
would assume that an implementation of overlays would provide at least
one, and likely more such gateways on each VN. How many and where to
place them will presumably depend on many factors but would be done
based on traffic patterns and network layout. I would not think every
NVE has to provide such functionality.

What do you propose needs saying in the problem statement about that?

Thomas

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 29 Jun 2012 23:56:56 +0200

Hi Thomas,

 >> case of embedded softswitch in the end system is captured and
 >> documented in the problem statement document.
 >

No. I see no need for soft-switch within the tenant VMs itself.

 > Or do you mean as part of a hypervisor supporting TES/VMs?

Yes it could be part of hypervisor. It could also be part of kernel 
(assuming those two are not integrated). It could also be just running 
in the user space. I have see various designs recently - with quite 
reasonable performance numbers for all three above - so I think we need 
to consider all. But that really does not have that much of impact to 
the architecture IMHO.

Thx,
R.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 29 Jun 2012 17:59:52 -0400

> > IP address must be retained but couldn't live migration be still
> > supported if the MAC address of the VM changes in the routed
> > semantics?

> In full generality, no.  For example, some VMs contain software
> whose license is keyed by the VM's MAC address.  If the MAC address
> changes, the software stops working.

> MAC address change is visible to the VM's OS - the device driver for
> the virtual NIC has to reconfigure, and that's potentially
> disruptive and non-transparent depending on what that device driver
> and OS can cope with.

Goodpoints. Proposed new text (including also change from Vishwas'
comments):

	  A key benefit of server virtualization is virtual machine
	  (VM) mobility. A VM can be migrated from one server to
	  another, live, i.e., while continuing to run and without
	  needing to shut it down and restart it at the new
	  location. A key requirement for live migration is that a VM
	  retain critical network state at its new location, including
	  its IP and MAC address(es).  Preservation of MAC addresses
	  is necessary when software licences are bound to MAC
	  addresses. More generally, any change in the VM's MAC
	  addresses resulting from a move would be visible to the VM
	  and thus potentially result in unexpected disruptions.
	  Retaining IP addresses after a move is necessary to prevent
	  existing transport (e.g., TCP) connections from breaking and
	  needing to be restarted.
	
	  In traditional data centers, servers are assigned IP
	  addresses based on their physical location, for example
	  based on the Top of Rack (ToR) switch for the server rack or
	  the VLAN configured to the server.  Servers can only move to
	  other locations within the same IP subnet. This constraint
	  is not problematic for physical servers, which move
	  infrequently, but it restricts the placement and movement of
	  VMs within the data center.  Any solution for a scalable
	  multi-tenant data center must allow a VM to be placed (or
	  moved to) anywhere within the data center, without being
	  constrained by the subnet boundary concerns of the host
	  servers.
   
Thomas

---

From: <[email-masked]>
Date: Fri, 29 Jun 2012 19:02:27 -0400

> this is a case we need to focus on optimizing?

A single tenant may have multiple virtual networks with routing used to
provide/control access among them.  The crucial thing is to avoid assuming
that a tenant or other administrative entity has a single virtual network
(or CUG in Pedro's email).  For example, consider moving a portion of 
a single data center that uses multiple VLANs and routers to selectively
connect them into an nvo3 environment - each VLAN gets turned into a virtual
network, and the routers now route among virtual networks instead of VLANs.

One of the things that's been pointed out to me in private is that the level
of importance that one places on routing across virtual networks may depend
on one's background.  If one is familiar with VLANs and views nvo3 overlays
providing VLAN-like functionality, IP routing among virtual networks is a
straightforward application of IP routing among VLANs (e.g., the previous
mention of L2/L3 IRB functionality that is common in data center network
switches).  OTOH, if one is familiar with VPNs where access among
otherwise-closed groups has to be explicitly configured, particularly
L3 VPNs where one cannot look to L2 to help with grouping the end systems,
this sort of cross-group access can be a significant area of functionality.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas
> Narten
> Sent: Friday, June 29, 2012 5:56 PM
> To: Pedro Roque Marques
> Cc: [email-masked]
> Subject: [nvo3] inter-CUG traffic [was Re: call for adoption: draft-narten-
> nvo3-overlay-problem-statement-02]
>

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 16:08:22 -0700

Thomas,

> from a framework perspective.

1) Whether L2 headers are preserved (including for instance VLAN tags) or L2 headers are rewritten an only IP packets are delivered.
2) Whether the problem statement is about building a "bigger vlan" or about routing between IP "subnets" where there is an ACL that controls traffic exchanges between subnets.
3) Whether the underlying network can be assumed to support broadcast/multicast services.
4) Whether it is assumed that the hypervisor switch can or not be modified.
5) Whether "unknown unicast" frames are broadcasted or communication requires the end-points to be known a-priori.

These factors depend on the DC design.

> thing.

My understanding is that there is a significant number of proposals in these space that are based on data-plane learning. These tend to assume an underlying multicast service.
Are you suggesting that these need to also support ingress replication ? How is the membership discovered ? How does ingress replication "fit" with scaling requirements ?

The problem space that i'm personally interested in is not one where the mac-in-mac style solutions is applicable. There are however DC designs where this type of solutions is valid and i suspect that the proponents of these style of solutions will disagree with your statement.

> So I think one of the requirements will need to be emulation of

I'm not sure i agree. If you define the problem as supporting reasonable small scale DCs that follow a rather conventional architecture, which some participants in this WG do (reasonably so) then emulation of multicast is no longer an issue.

> deployment is happy with IP only emulation.

These are very different problems. An L2 service is expected to solve complex issues relating with interaction with IEEE bridging semantics.
Solutions such as TRILL, etc spend a very considerable effort in dealing with this interaction.
An L3 service on the other hand is mostly about dealing with traffic that crosses subnets.

My understanding  is that DC operators that are focused on an L3 service have no need for an L2 service and its associated complexity. Saddling a possible solution with the need to provide an L2 service is imho highly counter-productive.

> problem statement).

I completely disagree. You can write a problem statement that contains both but it will impose unnecessary complexity in the solutions.

> Indeed, that is IMO very much a goal of this
> WG.

That is unfortunate. 

> Earlier versions of the charter had explicit text about this, that

Is it possible that it was removed because it is actually counter-productive to attempt to build both an L2 overlay and an L3 overlay with the same solution ? I'm under the impression that there is enough history in the IETF around the desire for different approaches. Specifically since an L2 overlay is not really about just the ethernet headers  (which no end-system really care about) but IEEE bridging semantics. The problem space where an L2 overlay is desired is  one where there is the expectation of interoperability between a bridged network (e.g. the part of the network that has not been "upgraded") and the overlay itself.

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 16:28:51 -0700

> requirements document,, what exactly should the problem statement say

The problem statement does not acknowledge that in a very relevant segment of DC designs most traffic crosses subnets/CUGs. Since in this type of  data-centers the elements of a subnet can be anywhere, the assumption the there is a router that interconnects the subnets is not reasonable.
Thus the problem statement misses a fundamental part of the problem.

> some sense, that is an underlay problem to solve, not for the overlay.

That statement misses the "problem". The actual path being optimal is a 1st order problem to solve. It is a more important issue than other aspects currently discussed in the document.
This problem cannot be delegated to the "underlay".

> why one has set up a system with a lot of "inter tenant" traffic.

As i pointed out, it is very common to run database or file system services to multiple tenants. As you can imagine these tend to dominate the traffic.
A "subnet"/CUG is not necessarily a tenant. It can be an application tier. That is standard practice in several DC deployments and supported for instance by Amazon's VPC service.

> Is

It is more important than intra-subnet/CUG traffic. And the problem statement does not reflect this.

> What do you propose needs saying in the problem statement about that?

I propose that the problem statement that focuses on large scale DCs (which should probably be a different document than the one for small scale) should clearly state upfront that most of the traffic crosses CUGs. And that a solution should be able to provide optimal inter-CUG traffic without packets crossing the    "DC  underlay" multiple times.

Conceptually, that means that the functionality of the "gateway" should be implemented at the overlay ingress and egress points, rather than requiring a mid-box. Whenever a mid-box is present it becomes a scaling and provisioning nightmare. This is relevant in DCs where the compute load can be spread beyond a single rack (i.e. the old aggregation switch boundary).

---

From: John E Drake <[email-masked]>
Date: Fri, 29 Jun 2012 16:32:24 -0700

Pedro,

An excellent email.  Thanks for pointing out some of the real issues.

John

Sent from my iPhone

>Pedro Roque Marques
>Sent: Friday, June 29, 2012 4:29 PM
>To: Thomas Narten
>Cc: [email-masked]
>Subject: Re: [nvo3] inter-CUG traffic [was Re: call for adoption: draft-
>narten-nvo3-overlay-problem-statement-02]
>
>
>The problem statement does not acknowledge that in a very relevant
>segment of DC designs most traffic crosses subnets/CUGs. Since in this
>type of  data-centers the elements of a subnet can be anywhere, the
>assumption the there is a router that interconnects the subnets is not
>reasonable.
>>> VLANs, VRFs (separate instances of routing) are definitely a feature,
>>> and I expect this to carry forward to nvo3 VNs.  In addition, we need
>>> to make changes to address Dimitri's comments about problems with the
>>> current VRF text.
>>
>>> data-centers do play in. If for instance 90% of a VMs traffic happens
>>> to be between the Host OS and a network attached storage file system
>>> run as-a-Service (with the appropriate multi-tenent
>>> the CPU that hosts the filesystem block can be randomly spread where
>>> is the router ?
>>
>That statement misses the "problem". The actual path being optimal is a
>1st order problem to solve. It is a more important issue than other
>aspects currently discussed in the document.
>>> In some DC designs the problem to solve is the inter-CUG traffic.
>>> With L2 headers being totally irrelevant.
>>
>As i pointed out, it is very common to run database or file system
>services to multiple tenants. As you can imagine these tend to dominate
>the traffic.
>A "subnet"/CUG is not necessarily a tenant. It can be an application
>tier. That is standard practice in several DC deployments and supported
>for instance by Amazon's VPC service.
>
>> Is
>I propose that the problem statement that focuses on large scale DCs
>(which should probably be a different document than the one for small
>scale) should clearly state upfront that most of the traffic crosses
>CUGs. And that a solution should be able to provide optimal inter-CUG
>traffic without packets crossing the    "DC  underlay" multiple times.
>
>Conceptually, that means that the functionality of the "gateway" should
>be implemented at the overlay ingress and egress points, rather than
>requiring a mid-box. Whenever a mid-box is present it becomes a scaling
>and provisioning nightmare. This is relevant in DCs where the compute
>load can be spread beyond a single rack (i.e. the old aggregation switch
>boundary).
>
>> Thomas
>>

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 16:37:44 -0700

On Jun 29, 2012, at 4:02 PM, <[email-masked]> wrote:

> switches).

Can you please describe an example of how you could set up such straightforward routing, assuming two Hosts belong to different "CUGs" such that these can be randomly spread across the DC  ? My question is where is the "gateway", how is it provisioned and  how can traffic paths be guaranteed to be optimal.

>  OTOH, if one is familiar with VPNs where access among

Considering that in a VPN one can achieve inter-CUG traffic exchange without an gateway in the middle via policy, it is unclear why you suggest that "look to L2" would help.

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 16:46:30 -0700

Thomas,

> 	  characteristics outlined in Section 2.7.

"not necessarily seen as supporting the characteristics outlined in Section 2.7" does not work for me unless you refute the arguments put forward by several people that in fact it does support these characteristics. In fact more so than other referenced technologies.

"have not seen widespread deployment" is also not something that looks like it would belong here. They have seen deployment. And i doubt that the intent is to look at market share since any new solution that gets proposed has 0 market share by definition. Should we also discount any of the other technologies  discussed in the doc               since clearly nothing has significant market share compared to VLANs ?

---

From: <[email-masked]>
Date: Fri, 29 Jun 2012 19:54:10 -0400

Pedro,

> Can you please describe an example of how you could set up such
> straightforward routing, assuming two Hosts belong to different "CUGs" such
> that these can be randomly spread across the DC  ? My question is where is the
> "gateway", how is it provisioned and  how can traffic paths be guaranteed to
> be optimal.

Ok, I see your point - the routing functionality is straightforward to move over,
but ensuring optimal pathing is significantly more work, as noted in another one
of your messages:

> Conceptually, that means that the functionality of the "gateway" should be
> implemented at the overlay ingress and egress points, rather than requiring
> a mid-box.

Thanks,
--David

> Sent: Friday, June 29, 2012 7:38 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]
> Can you please describe an example of how you could set up such
> straightforward routing, assuming two Hosts belong to different "CUGs" such
> that these can be randomly spread across the DC  ? My question is where is the
> "gateway", how is it provisioned and  how can traffic paths be guaranteed to
> be optimal.
> 
> Considering that in a VPN one can achieve inter-CUG traffic exchange without
> an gateway in the middle via policy, it is unclear why you suggest that "look
> to L2" would help.
> 
> >
> > Thanks,
> > --David
> >

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 29 Jun 2012 20:26:45 -0400

Depending upon what portion of the traffic needs inter-region handling 
(inter vpn, inter-vlan, ...) it is not obvious that "optimal" is an 
important goal.  As a general rule, perfect is the enemy of good.

Yours,
Joel

---

From: <[email-masked]>
Date: Fri, 29 Jun 2012 20:36:26 -0400

Pedro,

You and Thomas are in violent agreement about what to delete - Thomas put the new text *above* the text to be replaced (!).

Thanks, --David +++Sent from Blackberry

----- Original Message -----
From: Pedro Roque Marques [mailto:[email-masked]]
Sent: Friday, June 29, 2012 07:46 PM
To: Thomas Narten <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [nvo3] RFC4364 text in problem statement [was Re: call for	adoption: draft-narten-nvo3-overlay-problem-statement-02]

Thomas,

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 18:40:51 -0700

Joel,
A very common model currently is to have a 3 tier app where each tier is in its VLAN. You will find that web-servers for instance don't actually talk much to each other… although they are on the same VLAN 100% of their traffic goes outside VLAN. Very similar story applies to app logic tier. The database tier may have some replication traffic within its VLAN but hopefully than is less than the requests that it serves.

There isn't a whole lot of intra-CUG/subnet traffic under that deployment model. A problem statement that assumes (implicitly) that most or a significant part of the traffic stays local to a VLAN/subnet/CUG is not a good match for the common 3-tier application model. Even if you assume that web and app tiers use a  VLAN/subnet/CUG per tenant (which really is an application in enterprise) the database is typically common for a large number of apps/tenants.

  Pedro.

On Jun 29, 2012, at 5:26 PM, Joel M. Halpern wrote:

> Depending upon what portion of the traffic needs inter-region handling (inter vpn, inter-vlan, ...) it is not obvious that "optimal" is an important goal.  As a general rule, perfect is the enemy of good.
> 
> Yours,
> Joel
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 29 Jun 2012 22:31:34 -0400

I would not be so bold as to insist that all deployments can safely 
ignore inter-VPN intra-data-center traffic.  But there are MANY cases 
where that is not an important part of the traffic mix.
So I was urging that we not mandate optimal inter-subnet routing as part 
of the NVO3 requirements.
I would not want to prohibit it either, as there are definitely cases 
where it matters, some along the lines you alude to.

Yours,
Joel

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 30 Jun 2012 03:00:17 +0000

Still take the 3-tier app as an example, if the app servers each could be configured with two interfaces with one located in the VLAN/subnet of the web servers while the another located in the VLAN/subnet of the DB servers, the network only needs to care about the optimization for intra-subnet traffic (e.g., web<->app intra-subnet traffic and app<->DB intra-subnet traffic).

Of course, if the above demand can't be met due to some reason, the network should consider the optimization for inter-subnet traffic, and as had been pointed out by someone before, one practical solution is to deploy the default gateway functions as close as possible to the servers, e.g., inside the NVEs.

Best regards,
Xiaohu 

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Pedro
> Roque Marques
> 发送时间: 2012年6月30日 9:41
> 收件人: Joel M. Halpern
> 抄送: [email-masked]; [email-masked]; [email-masked]
> 主题: Re: [nvo3] inter-CUG traffic [was Re: call for adoption:
> draft-narten-nvo3-overlay-problem-statement-02]
> 
> Joel,
> A very common model currently is to have a 3 tier app where each tier is in its
> VLAN. You will find that web-servers for instance don't actually talk much to
> each other… although they are on the same VLAN 100% of their traffic goes
> outside VLAN. Very similar story applies to app logic tier. The database tier may
> have some replication traffic within its VLAN but hopefully than is less than the
> requests that it serves.
> 
> There isn't a whole lot of intra-CUG/subnet traffic under that deployment
> model. A problem statement that assumes (implicitly) that most or a significant
> part of the traffic stays local to a VLAN/subnet/CUG is not a good match for the
> common 3-tier application model. Even if you assume that web and app tiers
> use a  VLAN/subnet/CUG per tenant (which really is an application in
> enterprise) the database is typically common for a large number of
> apps/tenants.
> 
>   Pedro.
> 
> > Depending upon what portion of the traffic needs inter-region handling (inter
> vpn, inter-vlan, ...) it is not obvious that "optimal" is an important goal.  As a
> general rule, perfect is the enemy of good.
> >
> > Yours,
> > Joel
> >

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Fri, 29 Jun 2012 22:53:09 -0500

> 
> Still take the 3-tier app as an example, if the app servers each could be
> configured with two interfaces with one located in the VLAN/subnet of the web
> servers while the another located in the VLAN/subnet of the DB servers, the
> network only needs to care about the optimization for intra-subnet traffic
> (e.g., web<->app intra-subnet traffic and app<->DB intra-subnet traffic).

This would violate several security best practices. It would make the VM 
an easy entry point to the sensitive business logic and/or DB networks.
A vulnerability in the VM OS and/or apps would allow anyone to gain access
to the VM and through there the internal network. That's why people
are using DMZs. 

A router/firewall has a much lower code footprint and most likely
is more resilient to such vulnerabilities. So, not only does one 
need L3 isolation, but most often a real firewall in 
between the web-servers and the rest of the applications.

> 
> Of course, if the above demand can't be met due to some reason, the network
> should consider the optimization for inter-subnet traffic, and as had been
> pointed out by someone before, one practical solution is to deploy the default
> gateway functions as close as possible to the servers, e.g., inside the NVEs.
> 
> Best regards,
> Xiaohu
> 
> > -----邮件原件-----
> (inter
> > >> Ok, I see your point - the routing functionality is straightforward to
> move
> > over,
> > >> but ensuring optimal pathing is significantly more work, as noted in
> another
> > one
> > >> of your messages:
> > >>

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 30 Jun 2012 03:58:46 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Thomas Narten
> 发送时间: 2012年6月30日 5:48
> 收件人: Dhruv Dhody
> 抄送: [email-masked]
> 主题: [nvo3] Oracle Terminology [was: Re: call for adoption:

Fully agree that it would be much better to use a standardized protocol for the NVE-oracle interaction if appropriate. 

As an example of using BGP, a standardized routing protocol for the NVE-oracle interaction, section 3.6.3. "RIB Reduction on PE Routers" of Virtual Subnet (http://tools.ietf.org/html/draft-xu-virtual-subnet-07) briefly explains how to use BGP tricks to push and query mappings between the NVE (i.e., L3VPN PE) and the oracle (i.e., Route Reflector) . Note that one of the advantage of using BGP as NVE-oracle interaction protocol is that the mappings cached on the NVEs (e.g., L3VPN PEs) could be updated in a timely fashion, which has been listed as one of the basic NVo3 control plane characteristics in the NVo3 control-plane requirement doc.

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 30 Jun 2012 04:10:12 +0000

> -----邮件原件-----
> 发件人: Stiliadis, Dimitrios (Dimitri)
> [mailto:[email-masked]]
> 发送时间: 2012年6月30日 11:53
> 收件人: Xuxiaohu; Pedro Roque Marques; Joel M. Halpern
> 主题: RE: [nvo3] inter-CUG traffic [was Re: call for adoption:

Yes. If security is highly concerned, a dedicated firewall/router would have to be placed somewhere (e.g., attached to the aggregation switches) between different tiers. In the way, the path optimization for inter-subnet traffic would have to be left aside:)

Best regards,
Xiaohu

> Pedro
> > > >>> straightforward routing, assuming two Hosts belong to different
> "CUGs"
> > > such
> > > >>> that these can be randomly spread across the DC  ? My question is
> where
> > > is the
> > > >>>> connect them into an nvo3 environment - each VLAN gets turned into
> a
> > > virtual
> > > >>>> network, and the routers now route among virtual networks instead
> of
> > > VLANs.
> > > >>>>
> > > >>> straightforward routing, assuming two Hosts belong to different
> "CUGs"
> > > such
> > > >>> that these can be randomly spread across the DC  ? My question is
> where
> > > is the
> > > >>>>> From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > > Of
> > > >>> Thomas
> > > >>>>> Narten
> > > >>>>> requirements document,, what exactly should the problem
> statement
> > > say
> > > >>>>> about this topic?
> > > >>>>>
> > > >>>>> one, and likely more such gateways on each VN. How many and
> where
> > > to