Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

From: Benson Schliesser <[email-masked]>
Date: Mon, 18 Jun 2012 16:51:20 -0500

Dear NVO3 Participants -

This message begins a two week Call for Adoption of http://tools.ietf.org/html/draft-lasserre-nvo3-framework-02 by the NVO3 working group, ending on 02-July-2012.

Please respond to the NVO3 mailing list with any statements of approval or disapproval, along with any additional comments that might explain your position. Also, if any NVO3 participant is aware of IPR associated with this draft, please inform the mailing list and/or the NVO3 chairs.

Thanks,
-Benson & Matthew

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 18 Jun 2012 18:07:10 -0400

I sent the comment below to the authors, upon reviewing the diffs from 
the previous version of this draft.  I would appreciate clarification on 
this issue before the WG adopts this document as a basis for further work:

In looking at the latest revision of this draft, the text seems to have 
moved from describing the devices at the ToR as switches / routers to 
refering to them as just switches.  I can not tell if this change is 
because the authors understand switch to include IP forwarding device 
(possibly with IP routing protocol support), or if there is a change in 
capabilities envisioned.
If the former, it should be stated explicitly, since it is an unusual usage.
If the later, I am confused as the document then very clearly states 
that the data center interconnect devices (now referred to in section 
1.3 as switches) are L3 capable devices.  In fact, the premise of the 
document requires such L3 capable devices (usually known as routers.) 
Thus, teh sentence "Core switches are usually Ethernet switches, but can 
also support routing capabilities" seems very strange.  switches != 
routers.  And this document and the WG charter requires those devices to 
support L3 capabilities.

Thank you,
Joel M. Halpern

On 6/18/2012 5:51 PM, Benson Schliesser wrote:
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 18 Jun 2012 18:20:15 -0500

Hi, Joel.

I would like for the authors to respond with their own comments. But speaking only for myself (as an individual):

I think that common usage of the unqualified term "ToR" generally refers to a "ToR switch". While the term "ToR" literally refers to a location, and could be used to describe a "ToR router" or "ToR storage array" etc, in my experience the definition in the framework draft is fairly accurate. (And moreover, "switch" isn't necessarily limited to L2... forwarding != routing, and encap / tunneling makes this even more confusing.)

But regardless, I think the definition of "ToR" is more-or-less inconsequential to the framework. We should get it right, of course. But it's more important that we define the NVE correctly. And the NVE could perhaps be resident in many types of device, including a device that is not exactly a router but does have L3 interface(s).

In the draft, the ToR concept is introduced in an "example of multi-tier DC network architecture". I know from experience that there are many possible variations on where the access and aggregation layers are located. Do you think the authors should make the example more generic, perhaps change ToR to "access" or something like that? It's not clear to me what's best here - suggestions would be appreciated.

Cheers,
-Benson

On Jun 18, 2012, at 5:07 PM, Joel M. Halpern wrote:

> I sent the comment below to the authors, upon reviewing the diffs from the previous version of this draft.  I would appreciate clarification on this issue before the WG adopts this document as a basis for further work:
> 
> In looking at the latest revision of this draft, the text seems to have moved from describing the devices at the ToR as switches / routers to refering to them as just switches.  I can not tell if this change is because the authors understand switch to include IP forwarding device (possibly with IP routing protocol support), or if there is a change in capabilities envisioned.
> If the later, I am confused as the document then very clearly states that the data center interconnect devices (now referred to in section 1.3 as switches) are L3 capable devices.  In fact, the premise of the document requires such L3 capable devices (usually known as routers.) Thus, teh sentence "Core switches are usually Ethernet switches, but can also support routing capabilities" seems very strange.  switches != routers.  And this document and the WG charter requires those devices to support L3 capabilities.
> 
> Thank you,
> Joel M. Halpern
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 18 Jun 2012 19:26:52 -0400

I probably would have sent a private comment, but not bothered the list, 
if it was just ToR entities.  But the document has changed what the ToR 
entities are connect to from being switches / routers to being switches. 
  It is that change which concerns me, and for which I seek explanation.

Yours,
Joel

PS: I actually agree that the common usage is ToR switch, and the common 
deployment is to put L2 devices in that place in the topology.

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 19 Jun 2012 12:06:56 +0200

It was certainly not a deliberate change to imply that L3 was not needed...

Could you suggest which sentence(s) need clarification?

Thanks,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Tuesday, June 19, 2012 1:27 AM
To: Benson Schliesser
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Yours,

Joel

nvo3 mailing list

[email-masked]

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Tue, 19 Jun 2012 12:17:19 +0200

It’s the classic “what is a SWITCH” confusion caused by some marketing whiz more than a decade ago. I’m not too familiar with what you can or cannot do within an ID/RFC, but the logical thing to do would be to define ...

Switch = a network device performing packet forwarding based on L2 or L3 headers, or a combination of both

ToR = ToR switch (unless indicated otherwise)

... or something similar in the General Terminology section.

Kind regards,

Ivan

From: [email-masked] [mailto:[email-masked]] On Behalf Of LASSERRE, MARC (MARC)
Sent: Tuesday, June 19, 2012 12:07 PM
To: Joel M. Halpern; Benson Schliesser
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

It was certainly not a deliberate change to imply that L3 was not needed…

Thanks,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Tuesday, June 19, 2012 1:27 AM
To: Benson Schliesser
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Yours,

Joel

 

nvo3 mailing list

[email-masked]

---

From: Dhruv Dhody <[email-masked]>
Date: Tue, 19 Jun 2012 17:04:43 +0530

Hi,

Support! 

But I think in section 4.2, along with control plane driven and data plane
driven, Management plane driven issues and consideration would make the ID
complete. 

Regards,
Dhruv

****************************************************************************
***
Dhruv Dhody, System Architect, Huawei Technologies, Bangalore, India, Ph.
+91-9845062422

This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Benson Schliesser
> Sent: Tuesday, June 19, 2012 3:21 AM
> To: [email-masked]
> Cc: Matthew (Matthew) Bocci; [email-masked]
> Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
> 
> Dear NVO3 Participants -
> 
> This message begins a two week Call for Adoption of
> http://tools.ietf.org/html/draft-lasserre-nvo3-framework-02 by the NVO3
> working group, ending on 02-July-2012.
> 
> Please respond to the NVO3 mailing list with any statements of approval or
> disapproval, along with any additional comments that might explain your
> position. Also, if any NVO3 participant is aware of IPR associated with
this
> draft, please inform the mailing list and/or the NVO3 chairs.
> 
> Thanks,
> -Benson & Matthew
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 19 Jun 2012 10:14:39 -0400

Please do not aggravate the mess marketing produced by redefining switch 
to include router.

Yours,
Joel

On 6/19/2012 6:17 AM, Ivan Pepelnjak wrote:
> It’s the classic “what is a SWITCH” confusion caused by some marketing
> whiz more than a decade ago. I’m not too familiar with what you can or
> cannot do within an ID/RFC, but the logical thing to do would be to
> define ...
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf
> Of *LASSERRE, MARC (MARC)
> *Sent:* Tuesday, June 19, 2012 12:07 PM
> *To:* Joel M. Halpern; Benson Schliesser
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
>
> From: [email-masked] <mailto:[email-masked]>
> [mailto:[email-masked]] On Behalf Of Joel M. Halpern
> Cc: [email-masked] <mailto:[email-masked]>
>  > I would like for the authors to respond with their own comments. But
> speaking only for myself (as an individual):
>
>  >
>
>  > I think that common usage of the unqualified term "ToR" generally
> refers to a "ToR switch". While the term "ToR" literally refers to a
> location, and could be used to describe a "ToR router" or "ToR storage
> array" etc, in my experience the definition in the framework draft is
> fairly accurate. (And moreover, "switch" isn't necessarily limited to
> L2... forwarding != routing, and encap / tunneling makes this even more
> confusing.)
>
>  >
>
>  > But regardless, I think the definition of "ToR" is more-or-less
> inconsequential to the framework. We should get it right, of course. But
> it's more important that we define the NVE correctly. And the NVE could
> perhaps be resident in many types of device, including a device that is
> not exactly a router but does have L3 interface(s).
>
>  >
>
>  > In the draft, the ToR concept is introduced in an "example of
> multi-tier DC network architecture". I know from experience that there
> are many possible variations on where the access and aggregation layers
> are located. Do you think the authors should make the example more
> generic, perhaps change ToR to "access" or something like that? It's not
> clear to me what's best here - suggestions would be appreciated.
>
>  >
>
>  > Cheers,
>
>  > -Benson
>
>  >
>
>  >
>
>  >> I sent the comment below to the authors, upon reviewing the diffs
> from the previous version of this draft.  I would appreciate
> clarification on this issue before the WG adopts this document as a
> basis for further work:
>
>  >>
>
>  >> In looking at the latest revision of this draft, the text seems to
> have moved from describing the devices at the ToR as switches / routers
> to refering to them as just switches.  I can not tell if this change is
>  >>> Please respond to the NVO3 mailing list with any statements of
> approval or disapproval, along with any additional comments that might
> explain your position. Also, if any NVO3 participant is aware of IPR
> associated with this draft, please inform the mailing list and/or the
> NVO3 chairs.
>
>  >>>
>
>  >>> Thanks,
>
>  >>> -Benson & Matthew
>
>  >>>
>
>  >>> [email-masked] <mailto:[email-masked]>
>
>  >> [email-masked] <mailto:[email-masked]>
>
> [email-masked] <mailto:[email-masked]>
>

---

From: Paul Unbehagen <[email-masked]>
Date: Tue, 19 Jun 2012 09:06:15 -0600

Ivan is making a good point that switch's acting as ToR's have routing capability in them as well.   This should be documented as a state of reality. 

--
Paul Unbehagen

On Jun 19, 2012, at 8:14 AM, "Joel M. Halpern" <[email-masked]> wrote:

> Please do not aggravate the mess marketing produced by redefining switch to include router.
> 
> Yours,
> Joel
> 
>

---

From: Joel jaeggli <[email-masked]>
Date: Tue, 19 Jun 2012 08:22:59 -0700

> Ivan is making a good point that switch's acting as ToR's have routing capability in them as well.   This should be documented as a state of reality. 

pretty much all the devices in the network stack are capable of making
l2/l3 forwarding decisions it's not therefore a point of differentiation.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 19 Jun 2012 11:25:22 -0400

If you want to make that point, then go back to the text as it was in 
the previous version.  Do not make the point by hiding it in a confusing 
definition.

The ToR devices were not what I was concerned about.  I can live with 
saying they may be routers.  I can live with describing them as 
switches.  It was the Intra-DC devices that I was concerned about.

Adding marketing confusion to the definitions would create a new issue 
for me to be concerned about.

Yours,
Joel

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 19 Jun 2012 17:33:18 +0200

As a clarification, the text which specified "switches/routers" in -01 was the proper text and got lost by mistake in -02.
It will be fixed to say "switches/routers" instead of "switches" only.

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Tuesday, June 19, 2012 5:25 PM
To: Paul Unbehagen
Cc: Ivan Pepelnjak; Benson Schliesser; [email-masked]; LASSERRE, MARC (MARC)
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

If you want to make that point, then go back to the text as it was in the previous version.  Do not make the point by hiding it in a confusing definition.

The ToR devices were not what I was concerned about.  I can live with saying they may be routers.  I can live with describing them as switches.  It was the Intra-DC devices that I was concerned about.

Adding marketing confusion to the definitions would create a new issue for me to be concerned about.

Yours,
Joel

On 6/19/2012 11:06 AM, Paul Unbehagen wrote:
>>> It's the classic "what is a SWITCH" confusion caused by some 
>>> marketing whiz more than a decade ago. I'm not too familiar with 
>>> what you can or cannot do within an ID/RFC, but the logical thing to 
>>> do would be to define ...
>>>
>>> *From:[email-masked] [mailto:[email-masked]] *On 
>>> Behalf Of *LASSERRE, MARC (MARC)
>>> location, and could be used to describe a "ToR router" or "ToR 
>>> storage array" etc, in my experience the definition in the framework 
>>> draft is fairly accurate. (And moreover, "switch" isn't necessarily 
>>> limited to L2... forwarding != routing, and encap / tunneling makes 
>>> this even more
>>> confusing.)
>>>
>>>>
>>>
>>> inconsequential to the framework. We should get it right, of course. 
>>> But it's more important that we define the NVE correctly. And the 
>>> NVE could perhaps be resident in many types of device, including a 
>>> device that is not exactly a router but does have L3 interface(s).
>>>
>>>>
>>>
>>> multi-tier DC network architecture". I know from experience that 
>>> there are many possible variations on where the access and 
>>> aggregation layers are located. Do you think the authors should make 
>>> the example more generic, perhaps change ToR to "access" or 
>>> something like that? It's not clear to me what's best here - suggestions would be appreciated.
>>>
>>>>
>>>
>>>> Cheers,
>>>
>>>> -Benson
>>>
>>>>
>>>
>>>>
>>>
>>> have moved from describing the devices at the ToR as switches / 
>>> routers to refering to them as just switches.  I can not tell if 
>>> this change is because the authors understand switch to include IP 
>>> forwarding device (possibly with IP routing protocol support), or if 
>>> there is a change in capabilities envisioned.
>>>
>>> 1.3 as switches) are L3 capable devices.  In fact, the premise of 
>>> the document requires such L3 capable devices (usually known as 
>>> routers.) Thus, teh sentence "Core switches are usually Ethernet 
>>> switches, but can also support routing capabilities" seems very 
>>> strange.  switches != routers.  And this document and the WG charter 
>>> requires those devices to support L3 capabilities.
>>>
>>>>>
>>>
>>>>> Thank you,
>>>
>>>>> Joel M. Halpern
>>>
>>>>>
>>>
>>> http://tools.ietf.org/html/draft-lasserre-nvo3-framework-02 by the 
>>> NVO3 working group, ending on 02-July-2012.
>>>
>>>>>>
>>>
>>> approval or disapproval, along with any additional comments that 
>>> might explain your position. Also, if any NVO3 participant is aware 
>>> of IPR associated with this draft, please inform the mailing list 
>>> and/or the
>>> NVO3 chairs.
>>>
>>>>>>
>>>
>>>>>> Thanks,
>>>
>>>>>> -Benson & Matthew
>>>
>>>>>>
>>>

---

From: Paul Unbehagen <[email-masked]>
Date: Tue, 19 Jun 2012 09:48:55 -0600

Thats a fair point. So if the definition was more descriptive of the place in the network and not of the functionality would that address your concern?

--
Paul Unbehagen

---

From: Paul Unbehagen <[email-masked]>
Date: Tue, 19 Jun 2012 09:49:53 -0600

Thanks Marc, I agree that text would be better.

--
Paul Unbehagen

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 19 Jun 2012 11:30:37 -0400

I'll try:
Bridge: preserves the L2 header. It may add headers (tunnel) and may use routing technology (eg. IS-IS) to find the destination.
Router: preserves the L3 header (with permitted modification, such as TTL). Destroys the original L2 header.

--
Jakob Heitz.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 19 Jun 2012 12:00:30 -0400

Thank you Marc.  With this change, I support adoption of the document by 
the working group.

Yours,
Joel

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 20 Jun 2012 08:43:44 +0000

Hi,

I support the WG adoption of this draft.

The following are some editorial and technical comments for consideration:

1. In section 2.1. Generic Reference Model, it said“… The following diagram shows a DC reference model for network virtualization using Layer3 overlays…”

[Xiaohu] Layer3 overlay -> overlays over layer 3 underlay? Otherwise, the Layer3 overlay could be mistakenly interpreted as a overlay which offer Layer3 connectivity service:)

2. In section 3.1.4. Tunnel Overlays and Encapsulation options,  it said“…   Different IP tunneling options (GRE/L2TP/IPSec) and tunneling options (BGP VPN, PW, VPLS) can be used are available for tunneling both Ethernet and IP formats.”

[Xiaohu] The IP tunneling options could also include IP (e.g., MPLS-in-IP) and UDP (e.g., MPLS-in-UDP) tunnel technologies, so it would be better to remain the word "e.g.," (formerly used in the -01 version). BTW, what did the latter tunneling options mean by listing BGP VPN, PW and VPLS as examples?

3. In section 3.1.5.2. Address advertisement and tunnel mapping,  it said “…As traffic reaches an ingress NVE, a lookup is performed to determine which tunnel the packet needs to be sent to. It is then encapsulated with a tunnel header containing the destination address of the egress overlay node.

[Xiaohu] “egress NVE”(formerly used in the -01 version) is better than “egress overlay node”, IMHO.

4. In section 3.1.5.2. Address advertisement and tunnel mapping,  it said “…Furthermore, a control plane protocol that carries both MAC and IP addresses and associated MACs eliminates the need for ARP, and
   hence addresses one of the issues with explosive ARP handling.”

[Xiaohu] The statement of " eliminates the need for ARP " is too strong (may cause some confusion) and the "explosive ARP handling issue" is only associated with Layer2 NVEs, IMHO. Hence how about say something like“…Furthermore, a control plane protocol that carries MAC and IP pairs reduces the ARP broadcast flood across NVEs , and hence addresses the explosive ARP handling issues associated with those Layer2 NVEs.”

5. In section 4.1. Pros & Cons,  it said “…o Unicast tunneling state management is handled at the edge of the network. Intermediate transport nodes are unaware of such state. Note that this is not often the case when multicast is enabled in the core network.

[Xiaohu] if “this” represents the“unicast tunneling state management ”issue, this is STILL the case even multicast is enabled in the core;)  How about say something like“…However, if multicast needs to be enabled in the core network,  both edge and intermediate nodes need to maintain multicast states”

6. In section 4.1. Pros & Cons,  it said “… o Support of a large number of virtual network identifiers. Identifiers”

[Xiaohu] virtual network identifiers ->virtual network instances?

7. In section 4.1. Pros & Cons,  it said “…  o Multicast service scalability. Multicast support may be
        required in the overlay network to address for each tenant
        flood containment or efficient multicast handling.”

[Xiaohu] in the overlay network -> in the underlay network?

8. In section 4.1. Pros & Cons,  it said “…   o Load balancing may not be optimal as the hash algorithm may not
        work well due to the limited number of combinations of tunnel
        source and destination addresses”

[Xiaohu] The number of combinations of tunnel source and destination addresses is just one of the factors which affect the load-balancing effect. Hence how about say something like “…Load balancing in the core may not be optimal as the hash algorithm performed by the core nodes could not calculate out enough entropy values to distinguish different tenant traffic flows among a given NVE pairs”

9. In section 4.2.1. Data plane vs Control plane driven, “…It should be noted that the amount of state to be distributed is a function of the number of virtual machines. Different forms of caching can also be utilized to minimize state distribution among between the various elements.

[Xiaohu] Caching is just one option, IMHO. How about say something like “…different FIB or RIB reduction approaches  can be utilized to address the routing/forwarding table scalability issues on the data plane and even on the control plane.”

Best regards,
Xiaohu

发件人: [email-masked] [mailto:[email-masked]] 代表 LASSERRE, MARC (MARC)
发送时间: 2012年6月19日 18:07
收件人: Joel M. Halpern; Benson Schliesser
抄送: [email-masked]
主题: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Thanks,
Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Tuesday, June 19, 2012 1:27 AM
To: Benson Schliesser
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Yours,
Joel

---

From: Lucy yong <[email-masked]>
Date: Wed, 20 Jun 2012 15:02:39 +0000

Hi Ben and Matt,

I want to say that the draft is well written and thanks to authors. I support this work.

I also want to raise a question regarding the WG process.

Last IETF (IETF83) had NVo3 BOF.(I heard that was well done)  NVo3 WG is just formed on May 1 2012. The framework draft 00 was just published in March 2012. The 02 uploaded recently has fair amount of changes from 01. 

Why do the WG want rapidly adopting the draft into WG draft. Are there some DC operator want this badly? It seems the time line in the charter is not the reason. Frankly, this set of works are pretty new to IETF people. It should give people more time to think and contribute.

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: Thomas Narten <[email-masked]>
Date: Wed, 20 Jun 2012 17:03:50 -0400

Lucy,

> Last IETF (IETF83) had NVo3 BOF.(I heard that was well done) NVo3 WG
> is just formed on May 1 2012. The framework draft 00 was just
> published in March 2012. The 02 uploaded recently has fair amount of
> changes from 01.

There is no changes section, but the main changes are adding
terminology (pulling it out of draft-kreeger-nvo3-overlay-cp-00.txt to
be specific) and then making changes to align up the terminology. It
was a goal to align the  terminology in all three of the drafts:

  draft-kreeger-nvo3-overlay-cp-00.txt
  draft-narten-nvo3-overlay-problem-statement-02.txt
  draft-lasserre-nvo3-framework-02.txt

I don't think one can really argue that there are major/substantive
changes to the framework document.

> Why do the WG want rapidly adopting the draft into WG draft. Are
> there some DC operator want this badly? It seems the time line in
>  the charter is not the reason. Frankly, this set of works are
>  pretty new to IETF people. It should give people more time to think
>  and contribute.

Urgency in getting started? Perhaps not. Desire to move slowly?
Absolutely not. ;-)

The IETF moves pretty slow on the whole, and as it is, it will take us
until the end of the year/early next year to get to the point of
rechartering (according to the WG milestones).

The call for adopting the document as a WG document isn't saying the
document is finished or won't change. What it would say is that the
framework document is the one that WG will use as basis for moving
forward, as opposed to, say, developing a completely different
document, or waiting for additional submissions and then picking from
them or figuring out how to merge them into one.

IMO, the framework doc is a good base to start with and I support the
WG adopting it.  If there are specific issues with the document that
anyone thinks needs addressing, please raise them. It is perfectly OK
to raise issues with the document and at the same time support making
the document a WG document.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Thu, 21 Jun 2012 15:23:08 +0000

Tom,

Snipped..

Urgency in getting started? Perhaps not. Desire to move slowly?
Absolutely not. ;-)
[[LY]] the latter is not the case absolutely in this WG. The WG want to adopt two drafts (or more?) without having a first WG meeting yet.

[[LY]] I will post some comments in separated e-mail.

Regards,
Lucy

Thomas

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jun 2012 12:18:21 -0800

On 6/21/12 7:23 AM, Lucy yong wrote:
> [[LY]] the latter is not the case absolutely in this WG. The WG want
> to adopt two drafts (or more?) without having a first WG meeting
> yet.

Why does the meeting matter?

Melinda

---

From: Lucy yong <[email-masked]>
Date: Thu, 21 Jun 2012 22:24:53 +0000

Some people especially operators may not have time to study the documents until the meeting time. The meeting gives authors opportunity to explain and have f2f discussions which may help the understanding.  Otherwise, why does IETF have regularly meetings yearly?

This is very new piece of work for many people, and other people want to think and contribute. Such rapid processing is seen as intentionally steered by a small set of people. That is the point.
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Melinda Shore
Sent: Thursday, June 21, 2012 3:18 PM
To: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Melinda

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 22 Jun 2012 10:05:13 +0100

Lucy

The primary method of communication in the IETF is through discussion on 
the list, not through the meetings.

The meetings are intended to progress exceptional issues, previously 
discussed on the list, which need the additional bandwidth of a face to 
face meeting.

If you have questions regarding the text in the framework, please ask them.

If you wish to propose additional or alternate text, please post it to 
the list, so it can be discussed.

If there is consensus on proposed changes to a WG draft those changes 
will be incoprorated in that draft.

Stewart

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Lucy yong <[email-masked]>
Date: Fri, 22 Jun 2012 14:54:04 +0000

Stewart,

I understand that the list is primary channel for the communication and the importance of the meetings.

I have no problem to make a proposal via the list.

My original concern is that the pace of the WG process seems too fast for a new subject.

Lucy

-----Original Message-----
From: Stewart Bryant [mailto:[email-masked]] 
Sent: Friday, June 22, 2012 4:05 AM
To: Lucy yong
Cc: Melinda Shore; [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Lucy

Stewart

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 22 Jun 2012 18:47:59 +0200

Hi Lucy,

Thanks for supporting this draft.

If you do have specific concerns about the draft content, we can discuss these on the list. But this would contradict your first sentence below...
Note that changes in -02 are only related to terminology (expanded section) and the alignment of such terminology in the rest of the draft.

I won't comment on your last emails about the process (which Stewart alrady addressed in his mail).
But I'd ask that you change the subject of future emails wrt process in order not to confuse the list.

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, June 20, 2012 5:03 PM
To: Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Ben and Matt,

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: Lucy yong <[email-masked]>
Date: Fri, 22 Jun 2012 16:53:33 +0000

Yes, I should change subject title in the first mail. Sorry to make the confuse.
I am very busy with other stuff now and will send the comments ASAP.
lucy

-----Original Message-----
From: LASSERRE, MARC (MARC) [mailto:[email-masked]] 
Sent: Friday, June 22, 2012 11:48 AM
To: Lucy yong; Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Lucy,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, June 20, 2012 5:03 PM
To: Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Ben and Matt,

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Jun 2012 08:03:58 -1000

> My original concern is that the pace of the WG process seems too fast

i will paste that on my wall

---

From: Lucy yong <[email-masked]>
Date: Fri, 22 Jun 2012 23:11:25 +0000

Marc,

Here are some comments on this draft:

1) The VN (or VNI?) in the doc. is an overlay network that is transported over the tunnels. These tunnels are provided by the underlying network. Why describe the tunnel as Tunnel Overlays or Tunneling Overlay in the doc.? It should be the tunnel or the tunnel provided by underlying network.

2) section 2.3, why call it NVE service type? Should we call it VN (or VNI) service type? One NVE may terminate both L2 VNI and L3 VNI. NVE is equivalent to PE in L2VPN/L3VPN. We did not have service type for PE and had service type for VPN.

3) Suggest to illustrate a VN packet structure in data plane and state:
  
  the tunneling may in turn be tunneled over other intermediate
  tunnels. It is also possible that intra DC and inter DC tunnels are stitched together to form an end-
  to-end tunnel between two NVEs.

       +-------------------------------------------+ 
       |         L2/L3 Tenant Payload              | 
       +-------------------------------------------+     Overlay network
       |               VN Context                  | 
       +-------------------------------------------+    ---------
       |          Tunnel Source Addresses          |  
       +-------------------------------------------+     Underlying network
       |          Tunnel Destination Address       | 
       +-------------------------------------------+ 
       
 4) Text: Different IP tunneling options (GRE/L2TP/IPSec) and tunneling
   options (BGP VPN, PW, VPLS) are available for both Ethernet and IP
   formats.  Comments: should also mention LSP tunnel too. 

 5) It is not clear to me when an NVE resides on DC GW, the VNI on the NVE associates to a logical interface on a port facing WAN. Do we describe the interface as VAP or not? In this case, the logical interface does not connect to end system.

 6) For the pros of overlay network, it should state that the overlay architecture eliminates IP
   subnet constraints in the underlying network when VMs move. This makes VM mobility easier.
  
 7) It is important to state that the major difference between other overlay network technology and NVO3 is that
   the client edges of the NVO3 network are individual virtualized hosts and not network sites.

 8) The doc. often refers NVO3 as a service. It may not true for DC. The service in DC is about compute, storage, networking, and applications. In this case, NVO3 provides the networking capability but not a service itself.  

 9) Consider a section about overlay network OAM.

 10) overlay node is not defined but used in text. I think it should be NVE.

Cheers,
Lucy

-----Original Message-----
From: LASSERRE, MARC (MARC) [mailto:[email-masked]] 
Sent: Friday, June 22, 2012 11:48 AM
To: Lucy yong; Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Lucy,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, June 20, 2012 5:03 PM
To: Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Ben and Matt,

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 25 Jun 2012 16:23:09 +0200

Hi Lucy,

Thanks for your comments.
See my comments below prefixed with [ml].

Marc 

-----Original Message-----
From: Lucy yong [mailto:[email-masked]] 
Sent: Saturday, June 23, 2012 1:11 AM
To: LASSERRE, MARC (MARC); Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew)
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Marc,

[ml] The VN and VNI are defined in the terminoly section. The overlay network is realized via NVE edge nodes over an L3 underlay network. The overlay network defines a L2 or L3 service, with its own encapsulation header. The underlay use IP and/or MPLS to provide connectivity between NVEs.
Let me know whether specific sentences in the draft need clarification.

[ml] Like with L2 and L3 PEs, there are L2 and L3 NVEs.

[ml] Thanks for this suggestion. We can certainly add such a figure in a future revision.

[ml] BGP VPNs, and PW/VPLS use MPLS labels for service demuxing and can use either MPLS or IP/GRE tunneling. Hence MPLS is implicit in the text.

If the NVE function were to reside on a DC GW, the same concepts still apply i.e. VAPs logical interfaces where VM traffic arrives are attached to DC GWs.

[ml] Overlays do not solve VM mobility.

[ml] NVO3 can cope with both virtualized and non-virtualized hosts.

[ml] NVO3 NVEs do provide a service as L2 and L3 PEs do. NVEs provide per-tenant VN instance services.

[ml] Good point for a future revision.

[ml] Yes, there have been used interchangibly. I will change the text to only mention NVE instead.

Cheers,
Lucy

-----Original Message-----
From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
Sent: Friday, June 22, 2012 11:48 AM
To: Lucy yong; Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Lucy,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, June 20, 2012 5:03 PM
To: Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Ben and Matt,

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: Lucy yong <[email-masked]>
Date: Mon, 25 Jun 2012 15:52:20 +0000

Marc,

Please see inline below with [[LY]].

Lucy

-----Original Message-----
From: LASSERRE, MARC (MARC) [mailto:[email-masked]] 
Sent: Monday, June 25, 2012 9:23 AM
To: Lucy yong; Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew)
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Lucy,

Marc 

-----Original Message-----
From: Lucy yong [mailto:[email-masked]] 
Sent: Saturday, June 23, 2012 1:11 AM
To: LASSERRE, MARC (MARC); Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew)
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Marc,

[ml] The VN and VNI are defined in the terminoly section. The overlay network is realized via NVE edge nodes over an L3 underlay network. The overlay network defines a L2 or L3 service, with its own encapsulation header. The underlay use IP and/or MPLS to provide connectivity between NVEs.
Let me know whether specific sentences in the draft need clarification.
[[LY]] Section 3.1.4 title is "tunnel overlays and encapsulation options", Tunnel overlay also used in the reference model. Consider replacing with "tunnel".

[ml] Like with L2 and L3 PEs, there are L2 and L3 NVEs.
[[LY]] Should we consider this as VN service type? I understand the description but have trouble with the title of NVE service type.

       

[ml] BGP VPNs, and PW/VPLS use MPLS labels for service demuxing and can use either MPLS or IP/GRE tunneling. Hence MPLS is implicit in the text.
[[LY]] It is better to explicit LSP tunnel in the text. Because VN may be over LSP tunnel directly without VPN,PW, or VPLS. It has difference.

If the NVE function were to reside on a DC GW, the same concepts still apply i.e. VAPs logical interfaces where VM traffic arrives are attached to DC GWs.
[[LY]] My indication is more about the traffic of VMs in a VN from/to Internet, i.e. the use case in section 4.1 of use case draft.

[ml] Overlays do not solve VM mobility.
[[LY]] Right, VM mobility requires other. But the overlay does move away this restriction. IMO: it is worth to mention.

[ml] NVO3 can cope with both virtualized and non-virtualized hosts.
[[LY]] Yeah, that is more precise. The point is that the client edge is not network sites like CEs in a VPN.

[ml] NVO3 NVEs do provide a service as L2 and L3 PEs do. NVEs provide per-tenant VN instance services.
[[LY]] In DC, both NVEs and VMs may be on the same hardware and managed by the same operators who offers cloud services. Such service provides virtualized compute, storage, networking, and applications. In this case, NV03 provides the networking functions but not a service itself from DC operator perspective. The doc. should capture both cases, i.e. NVo3 as a service and NVo3 for the networking of a service.

--end

Cheers,
Lucy

-----Original Message-----
From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
Sent: Friday, June 22, 2012 11:48 AM
To: Lucy yong; Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Lucy,

Marc

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, June 20, 2012 5:03 PM
To: Benson Schliesser; [email-masked]
Cc: Bocci, Matthew (Matthew); [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Ben and Matt,

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
Sent: Monday, June 18, 2012 4:51 PM
To: [email-masked]
Cc: Matthew (Matthew) Bocci; [email-masked]
Subject: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Dear NVO3 Participants -

Thanks,
-Benson & Matthew

---

From: <[email-masked]>
Date: Mon, 25 Jun 2012 13:14:21 -0400

Extracting a couple of items for additional comment:

> 2) section 2.3, why call it NVE service type? Should we call it VN (or VNI)
> service type? One NVE may terminate both L2 VNI and L3 VNI. NVE is equivalent
> to PE in L2VPN/L3VPN. We did not have service type for PE and had service type
> for VPN.
> 
> [[LY]] Should we consider this as VN service type? I understand the
> description but have trouble with the title of NVE service type.

I think VN service type makes sense, as an NVE could conceivably support both
L2 and L3 services for the same end system.  OTOH, mixing L2 and L3 service
types on the same VN requires that the encapsulation indicate the service
type (for correct decap processing), and that would also be useful to note.

> network sites like CEs in a VPN.

Actually the NVE can be in a network node.  See Figure 3 and the discussion of
"traditional physical server" in the last paragraph on p.10, including the
statement that the NVE can be in the ToR.  FWIW, the Storage Systems example
in that paragraph is fine, although one can envision storage systems that
contain NVEs.

Thanks,
--David

> Sent: Monday, June 25, 2012 11:52 AM
> 1) The VN (or VNI?) in the doc. is an overlay network that is transported over
> the tunnels. These tunnels are provided by the underlying network. Why
> describe the tunnel as Tunnel Overlays or Tunneling Overlay in the doc.? It
> should be the tunnel or the tunnel provided by underlying network.
> 
> [ml] The VN and VNI are defined in the terminoly section. The overlay network
> is realized via NVE edge nodes over an L3 underlay network. The overlay
> network defines a L2 or L3 service, with its own encapsulation header. The
> underlay use IP and/or MPLS to provide connectivity between NVEs.
> [[LY]] Section 3.1.4 title is "tunnel overlays and encapsulation options",
> Tunnel overlay also used in the reference model. Consider replacing with
> "tunnel".
> 
> 2) section 2.3, why call it NVE service type? Should we call it VN (or VNI)
> service type? One NVE may terminate both L2 VNI and L3 VNI. NVE is equivalent
> to PE in L2VPN/L3VPN. We did not have service type for PE and had service type
> for VPN.
> 
> [[LY]] Should we consider this as VN service type? I understand the
> description but have trouble with the title of NVE service type.
> 
> [ml] BGP VPNs, and PW/VPLS use MPLS labels for service demuxing and can use
> either MPLS or IP/GRE tunneling. Hence MPLS is implicit in the text.
> [[LY]] It is better to explicit LSP tunnel in the text. Because VN may be over
> LSP tunnel directly without VPN,PW, or VPLS. It has difference.
> 
>  5) It is not clear to me when an NVE resides on DC GW, the VNI on the NVE
> associates to a logical interface on a port facing WAN. Do we describe the
> interface as VAP or not? In this case, the logical interface does not connect
> to end system.
> 
> If the NVE function were to reside on a DC GW, the same concepts still apply
> i.e. VAPs logical interfaces where VM traffic arrives are attached to DC GWs.
> [[LY]] My indication is more about the traffic of VMs in a VN from/to
> Internet, i.e. the use case in section 4.1 of use case draft.
> 
> [[LY]] Right, VM mobility requires other. But the overlay does move away this
> restriction. IMO: it is worth to mention.
> 
>  8) The doc. often refers NVO3 as a service. It may not true for DC. The
> service in DC is about compute, storage, networking, and applications. In this
> case, NVO3 provides the networking capability but not a service itself.
> 
> [[LY]] In DC, both NVEs and VMs may be on the same hardware and managed by the
> same operators who offers cloud services. Such service provides virtualized
> compute, storage, networking, and applications. In this case, NV03 provides
> the networking functions but not a service itself from DC operator
> perspective. The doc. should capture both cases, i.e. NVo3 as a service and
> NVo3 for the networking of a service.
> 
> --end
> 
> If you do have specific concerns about the draft content, we can discuss these
> on the list. But this would contradict your first sentence below...
> Note that changes in -02 are only related to terminology (expanded section)
> and the alignment of such terminology in the rest of the draft.
> 
> Last IETF (IETF83) had NVo3 BOF.(I heard that was well done)  NVo3 WG is just
> formed on May 1 2012. The framework draft 00 was just published in March 2012.
> The 02 uploaded recently has fair amount of changes from 01.
> 
> Why do the WG want rapidly adopting the draft into WG draft. Are there some DC
> operator want this badly? It seems the time line in the charter is not the
> reason. Frankly, this set of works are pretty new to IETF people. It should
> give people more time to think and contribute.
> 
> Regards,
> Lucy
> 
> 
> 
> position. Also, if any NVO3 participant is aware of IPR associated with this

---

From: Lucy yong <[email-masked]>
Date: Mon, 25 Jun 2012 17:32:46 +0000

Comments inline.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, June 25, 2012 12:14 PM
To: Lucy yong; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

[[LY]] it may explicitly encapsulate the indication or implicitly indicate by the VN context. For the later when creating a VN, it specifies it as L2 or L3. 

[[LY]] Yes, NVE can be in a network node. A traditional physical server can connect to a ToR port and let ToR provide NVE capability. In this case, the server does not provide the networking function, which is the point, i.e. it is not a network site.

Lucy

Thanks,
--David

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 26 Jun 2012 10:11:11 +0200

Hi Dave,

See my comments below.

Thanks,
Marc

> Sent: Monday, June 25, 2012 7:14 PM
> To: [email-masked]; LASSERRE, MARC (MARC); [email-masked]
> Cc: [email-masked]
> > 2) section 2.3, why call it NVE service type? Should we 
> call it VN (or 
> > VNI) service type? One NVE may terminate both L2 VNI and L3 
> VNI. NVE 
> > is equivalent to PE in L2VPN/L3VPN. We did not have service 
> type for 
> > PE and had service type for VPN.
> > 
> L2 and L3 services for the same end system.  OTOH, mixing L2 
> and L3 service types on the same VN requires that the 
> encapsulation indicate the service type (for correct decap 
> processing), and that would also be useful to note.

Agreed. I'll change the text accordingly.

> 
> >  7) It is important to state that the major difference 
> between other 
> > overlay network technology and NVO3 is that
> > [[LY]] Yeah, that is more precise. The point is that the 
> client edge 
> > is not network sites like CEs in a VPN.
> 
> Actually the NVE can be in a network node.  See Figure 3 and 
> the discussion of "traditional physical server" in the last 
> paragraph on p.10, including the statement that the NVE can 
> be in the ToR.  FWIW, the Storage Systems example in that 
> paragraph is fine, although one can envision storage systems 
> that contain NVEs.

I think that Lucy was refering to clients that are attached to NVEs.
I think that the text is clear about the fact that such clients can either be virtualized or non-virtualized.
As far as NVEs, as you said, there are also several paragraphs explaining where such NVE function can reside.

> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Lucy yong
> > From: LASSERRE, MARC (MARC) 
> [mailto:[email-masked]]
> > 1) The VN (or VNI?) in the doc. is an overlay network that is 
> > transported over the tunnels. These tunnels are provided by the 
> > underlying network. Why describe the tunnel as Tunnel Overlays or 
> > Tunneling Overlay in the doc.? It should be the tunnel or 
> the tunnel provided by underlying network.
> > 
> > [ml] The VN and VNI are defined in the terminoly section. 
> The overlay 
> > network is realized via NVE edge nodes over an L3 underlay network. 
> > The overlay network defines a L2 or L3 service, with its own 
> > encapsulation header. The underlay use IP and/or MPLS to 
> provide connectivity between NVEs.
> > [[LY]] Section 3.1.4 title is "tunnel overlays and encapsulation 
> > options", Tunnel overlay also used in the reference model. Consider 
> > replacing with "tunnel".
> > 
> > 2) section 2.3, why call it NVE service type? Should we 
> call it VN (or 
> > VNI) service type? One NVE may terminate both L2 VNI and L3 
> VNI. NVE 
> > is equivalent to PE in L2VPN/L3VPN. We did not have service 
> type for 
> > PE and had service type for VPN.
> > 
> >   tunnels. It is also possible that intra DC and inter DC 
> tunnels are 
> > stitched together to form an end-
> Overlay network
> Underlying network
> > [ml] Thanks for this suggestion. We can certainly add such 
> a figure in 
> > a future revision.
> > 
> > [ml] BGP VPNs, and PW/VPLS use MPLS labels for service demuxing and 
> > can use either MPLS or IP/GRE tunneling. Hence MPLS is 
> implicit in the text.
> > [[LY]] It is better to explicit LSP tunnel in the text. 
> Because VN may 
> > be over LSP tunnel directly without VPN,PW, or VPLS. It has 
> difference.
> > 
> >  5) It is not clear to me when an NVE resides on DC GW, the 
> VNI on the 
> > NVE associates to a logical interface on a port facing WAN. Do we 
> > describe the interface as VAP or not? In this case, the logical 
> > interface does not connect to end system.
> > 
> > If the NVE function were to reside on a DC GW, the same 
> concepts still 
> > apply i.e. VAPs logical interfaces where VM traffic arrives 
> are attached to DC GWs.
> >  6) For the pros of overlay network, it should state that 
> the overlay 
> > architecture eliminates IP
> > [[LY]] Right, VM mobility requires other. But the overlay does move 
> > away this restriction. IMO: it is worth to mention.
> > 
> >  7) It is important to state that the major difference 
> between other 
> > overlay network technology and NVO3 is that
> > [[LY]] Yeah, that is more precise. The point is that the 
> client edge 
> > is not network sites like CEs in a VPN.
> > 
> >  8) The doc. often refers NVO3 as a service. It may not 
> true for DC. 
> > The service in DC is about compute, storage, networking, and 
> > applications. In this case, NVO3 provides the networking 
> capability but not a service itself.
> > 
> > [ml] NVO3 NVEs do provide a service as L2 and L3 PEs do. 
> NVEs provide 
> > per- tenant VN instance services.
> > [[LY]] In DC, both NVEs and VMs may be on the same hardware and 
> > managed by the same operators who offers cloud services. 
> Such service 
> > provides virtualized compute, storage, networking, and 
> applications. 
> > In this case, NV03 provides the networking functions but 
> not a service 
> > itself from DC operator perspective. The doc. should capture both 
> > cases, i.e. NVo3 as a service and
> > [ml] Yes, there have been used interchangibly. I will 
> change the text 
> > to only mention NVE instead.
> > 
> > Cheers,
> > Lucy
> > 
> > From: LASSERRE, MARC (MARC) 
> [mailto:[email-masked]]
> > Cc: Bocci, Matthew (Matthew); 
> > [email-masked]
> > If you do have specific concerns about the draft content, we can 
> > discuss these on the list. But this would contradict your 
> first sentence below...
> > Note that changes in -02 are only related to terminology (expanded 
> > section) and the alignment of such terminology in the rest 
> of the draft.
> > 
> > I won't comment on your last emails about the process 
> (which Stewart 
> > alrady addressed in his mail).
> > But I'd ask that you change the subject of future emails 
> wrt process 
> > in order not to confuse the list.
> > 
> > Marc
> > 
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Lucy yong
> > Cc: Bocci, Matthew (Matthew); 
> > [email-masked]
> > I want to say that the draft is well written and thanks to 
> authors. I 
> > support this work.
> > 
> > is just formed on May 1 2012. The framework draft 00 was 
> just published in March 2012.
> > Why do the WG want rapidly adopting the draft into WG 
> draft. Are there 
> > some DC operator want this badly? It seems the time line in the 
> > charter is not the reason. Frankly, this set of works are 
> pretty new 
> > to IETF people. It should give people more time to think 
> and contribute.
> > 
> > Regards,
> > Lucy
> > 
> > 
> > 
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Benson Schliesser
> > Cc: Matthew (Matthew) Bocci; 
> > [email-masked]
> > associated with this draft, please inform the mailing list 
> and/or the NVO3 chairs.
> > 
> > Thanks,
> > -Benson & Matthew
> >

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 26 Jun 2012 10:31:40 +0200

Hi Aldrin,

See my comments below.

Thanks,
Marc

> From: Aldrin Isaac [mailto:[email-masked]] 
> Sent: Monday, June 25, 2012 8:23 PM
> To: LASSERRE, MARC (MARC)
> Cc: Lucy yong; [email-masked]; [email-masked]
> Hi Marc,
> Looking for more clarity on the following...
> 
> Should we assume that a VNI can only support L2VNs (VNI = 
> L2VNI) or L3VNs (VNI = L3VNI) but not both?  
> 
> draft-mity-nvo3-use-cases describes a "virtual network 
> instance interconnecting interface" (VNIF) to interconnect 
> VNI of different type.  An example of a VNIF would be an IRB 
> (integrated routing and bridging) interface that is common on 
> "L3 switches".  Is this something that draft-lasserre captures?

While the current draft focuses on L2VNs and L3VNs as distinct services, no assumptions have been made about the ability to support IRB.
A couple of sentences could be added to highlight this possibility.

> 
> Are there any restriction to the topology of a VN or topology 
> created using a set of VNs.  I.e. can (1) a VNI be a member 
> of multiple VN and (2) can a VN be either hub-and-spoke or 
> full-mesh?  Does draft-lasserre support (1) and (2)?

A VN is represented by a VNI created within NVEs that participate in this VN. Hence there is a one-to-one mapping between a VN and VNI.

Hub-and-spoke is discussed briefly in this draft. More details are provided in draft-bl-nvo3-dataplane-requirements.

> 
> Best -- aldrin
> 
> 
> On Jun 25, 2012, at 1:14 PM, <[email-masked]> wrote:
> 
> >> 2) section 2.3, why call it NVE service type? Should we call it VN 
> >> (or VNI) service type? One NVE may terminate both L2 VNI 
> and L3 VNI. 
> >> NVE is equivalent to PE in L2VPN/L3VPN. We did not have 
> service type 
> >> for PE and had service type for VPN.
> >> 
> > L2 and L3 services for the same end system.  OTOH, mixing L2 and L3 
> > service types on the same VN requires that the 
> encapsulation indicate 
> > the service type (for correct decap processing), and that 
> would also be useful to note.
> > 
> > Actually the NVE can be in a network node.  See Figure 3 and the 
> > discussion of "traditional physical server" in the last 
> paragraph on 
> > p.10, including the statement that the NVE can be in the 
> ToR.  FWIW, 
> > the Storage Systems example in that paragraph is fine, although one 
> > can envision storage systems that contain NVEs.
> > 
> > Thanks,
> > --David
> > 
> >> 2) section 2.3, why call it NVE service type? Should we call it VN 
> >> (or VNI) service type? One NVE may terminate both L2 VNI 
> and L3 VNI. 
> >> NVE is equivalent to PE in L2VPN/L3VPN. We did not have 
> service type 
> >> for PE and had service type for VPN.
> >> 
> >> stitched together to form an end-  to-end tunnel between two NVEs.
> >> 
> a figure 
> >> in a future revision.
> >> [[LY]] It is better to explicit LSP tunnel in the text. Because VN 
> >> may be over LSP tunnel directly without VPN,PW, or VPLS. 
> It has difference.
> >> 
> >> If the NVE function were to reside on a DC GW, the same concepts 
> >> still apply i.e. VAPs logical interfaces where VM traffic 
> arrives are attached to DC GWs.
> >> In this case, NV03 provides the networking functions but not a 
> >> service itself from DC operator perspective. The doc. 
> should capture 
> >> both cases, i.e. NVo3 as a service and
> >> there some DC operator want this badly? It seems the time 
> line in the 
> >> of IPR associated with this draft, please inform the 
> mailing list and/or the NVO3 chairs.
> >> 
> >> Thanks,
> >> -Benson & Matthew
> >>

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 26 Jun 2012 09:23:31 -0400

Hi Marc,

Thanks for the clarifications.

Wrt my first question, should we assume that there isn't yet a consensus on
whether a VNI can support more than one type of VN?

Regarding one-to-one mapping of VNI to VN, this implies that an end-station
that needs to be a part of more than one VN will require a separate
interface into each associated VNI.  This implies that the end-station will
require a potentially complex routing table (depending on how aggregateable
the address space within a VN is).  Unfortunately as an operator I don't
find this an acceptable trade off.  Can we please address the issue of
managing multiple interfaces and complex tables in end-systems? I believe
the problem statement needs to capture this.

Best -- aldrin

On Tuesday, June 26, 2012, LASSERRE, MARC (MARC) wrote:

> > From: Aldrin Isaac [mailto:[email-masked] <javascript:;>]
> > Cc: Lucy yong; [email-masked] <javascript:;>; [email-masked]<javascript:;>
> While the current draft focuses on L2VNs and L3VNs as distinct services,
> no assumptions have been made about the ability to support IRB.
> A VN is represented by a VNI created within NVEs that participate in this
> VN. Hence there is a one-to-one mapping between a VN and VNI.
>
> Hub-and-spoke is discussed briefly in this draft. More details are
> provided in draft-bl-nvo3-dataplane-requirements.
>
> >
> > Best -- aldrin
> >
> >

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 26 Jun 2012 15:43:05 +0200

Hi Aldrin,

See my comments below.

Thanks,
Marc

________________________________
From: Aldrin Isaac [mailto:[email-masked]]
Sent: Tuesday, June 26, 2012 3:24 PM
To: LASSERRE, MARC (MARC)
Cc: Aldrin Isaac; [email-masked]; [email-masked]; Lucy yong
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Marc,

Wrt my first question, should we assume that there isn't yet a consensus on whether a VNI can support more than one type of VN?
It does not preclude it. I'd like to hear how much interest there is to support this capability.

Regarding one-to-one mapping of VNI to VN, this implies that an end-station that needs to be a part of more than one VN will require a separate interface into each associated VNI.  This implies that the end-station will require a potentially complex routing table (depending on how aggregateable the address space within a VN is).  Unfortunately as an operator I don't find this an acceptable trade off.  Can we please address the issue of managing multiple interfaces and complex tables in end-systems? I believe the problem statement needs to capture this.
This is not any different than with existing VPN technologies. Supporting multiple interfaces to connect to different subnets and/or nvo3 domains does not make routing really complex.
Have you had specific problems with past technologies?

Best -- aldrin

See my comments below.

Thanks,
Marc

> From: Aldrin Isaac [mailto:[email-masked]<javascript:;>]
> Cc: Lucy yong; [email-masked]<javascript:;>; [email-masked]<javascript:;>

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 09:51:46 -0400

Hi Aldrin,

I observe that for IRB to work on a VNI, the VNI has to provide layer 2 service.  What
I mean by this is that IRB assumes that the Ethernet header is present on all traffic -
bridging uses that Ethernet header, whereas the routing portion of IRB (logically)
strips the Ethernet header and operates on the IP header.  To apply IRB to nvo3 traffic,
the inner headers (Ethernet and IP) are involved after decapsulation (again, logically).

The reason for "(logically)" is that there are optimization possibilities when the
same headers will be present outbound (e.g., when routing between two VNIs that use
the same encapsulation, an integrated implementation may modify the headers in place
- as long as the outbound traffic is correctly formatted, it doesn't matter whether
the headers were stripped and reapplied internally vs. modified in place).

In order to mix layer 2 service (inner Ethernet header present) and layer 3 service
(no inner Ethernet header present) on the same VNI, the encapsulation needs to indicate
whether the first header after the encapsulation header is Ethernet or IP.  Obviously,
there are ways to do that (GRE comes to mind), but I wonder whether there's a strong
requirement for multiple nvo3 encapsulation formats on a single VNI vs. forcing use of
a separate VNI if there's a desire for more than one encapsulation format.

Thanks,
--David

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Tuesday, June 26, 2012 9:24 AM
To: LASSERRE, MARC (MARC)
Cc: Aldrin Isaac; Black, David; [email-masked]; Lucy yong
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

---

From: Lucy yong <[email-masked]>
Date: Tue, 26 Jun 2012 16:56:22 +0000

Marc,

Snipped..

[[LY]] It is good to point out that in most of cases, the clients are not network sites. There are already some overlay technologies, it is important to distinct the differences between other overlay's and nvo3. 

Thanks,
Lucy

---

From: Randy Bush <[email-masked]>
Date: Tue, 26 Jun 2012 07:08:00 -1000

excuse the top-posting and excuse the meta-post.  but do folk here
realize that some of us use classic mail readers and it is almost
impossible to decode what is being quoted and what is actually
being said?

i append what i see (not picking on you, marc.  a lot of folk are doing
this).  and putting [me] in the middle of a mass of text is very little
help to the reader.  do folk's mail user agents really not have the
classic > quoting capability?

i spend a minute trying to decode and then give up and hit delete.

randy

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
To: Aldrin Isaac <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>,
 Lucy yong <[email-masked]>, Aldrin Isaac <[email-masked]>
Date: Tue, 26 Jun 2012 15:43:05 +0200

Hi Aldrin,

See my comments below.

Thanks,
Marc

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 26 Jun 2012 13:21:32 -0400

> impossible to decode what is being quoted and what is actually

If you use Outlook, try this:
http://home.in.tum.de/~jain/software/outlook-quotefix/

-- 
Jakob Heitz.

---

From: Randy Bush <[email-masked]>
Date: Tue, 26 Jun 2012 07:24:39 -1000

> http://home.in.tum.de/~jain/software/outlook-quotefix/

what's outlook?  though i have heard people say i have a bad outlook.

i use wanderlust under emacs in x windows

randy

---

From: János Farkas <[email-masked]>
Date: Fri, 29 Jun 2012 15:35:26 +0200

Hi,

I have some comments to the draft. Please find them below under the 
section header they belong to.

1. Introduction

“Limited VLAN space”
The 12-bit overlay network ID limitation of the Ethernet header is not 
there any more for several years. There are different types of VLANs. 
The I-SID can be considered as a 24-bit VLAN ID, which is sometimes 
called I-VLAN. Furthermore, there exist B-VLANs and S-VLANs aside the 
C-VLANs that were defined at the first place. The combination of these 
VLANs provides flexibility for network virtualization; altogether the 
Ethernet header today provides 60 bits for network virtualization.
It would be better to remove this because the limitation is not there today.

“FIB explosion due to handling of large number of MACs/IP addresses”
FIB explosion might be there in a L2 network if the already available L2 
network virtualization tools are not used. FIB explosion is not likely 
in a PBB network.
It would be better to update the bullet accordingly or remove it.

“Spanning Tree limitations”
L2 networks are not limited to a single spanning tree for several years. 
Shortest Path Bridging (SPB) provides shortest path forwarding and uses 
physical links that might have been blocked by a single spanning tree 
instance. Moreover, MSTP also provides multiple spanning tree instances, 
which allow the use of the physical links.
It would be better to remove the bullet.

“Broadcast storms”
Broadcast storms are not necessarily there in today’s L2 networks, e.g. 
there are no broadcast storms at all in an SPBM network. Furthermore, 
applying the existing L2 network virtualization techniques, the 
broadcast due to unknown destination is kept within the virtual network, 
thus it is not a broadcast storm.
It would be better to update the bullet such that it exactly specifies 
the problem or remove it.

“Inefficient Broadcast/Multicast handling”
It is not clear what is meant here, it would be better to be more 
specific. (If it is about L2, then the statement is not valid. Handling 
of multicast is very efficient in L2, both shortest path trees and 
shared threes can be used today.)

“Limited mobility/portability support”
It is not clear what is meant here. Is it about VM migration? If so, 
then the statement is not exact for L2, VDP supports VM migration. 
(Furthermore, L2 networks typically automatically learn the new location 
of a station after its movement.)

“Lack of service auto-discovery"
If it is about L2, then that is not the case. SPB has in-built service 
auto-discovery.
It would be better to be more specific in the bullet or remove it.

The issue list is introduced as “Existing virtual network models used 
for data center networks have known limitations, specifically in the 
context of multiple tenants. These issues can be summarized as”
As the comments to the bullets show the list is not accurate given the 
existing virtual network models available for data centers today. 
Therefore, it would be better to bring this issue list up to date to 
today’s networking technologies.
(I understand that the WG aims to provide a solution over L3. 
Nevertheless, it seems to me that it would be better to give other 
motivation for it than L2 has issues, given that no issue has been 
pointed out that is still valid for Ethernet today.)

4.1. Pros & Cons

“Unicast tunneling state management is handled at the edge of the 
network. Intermediate transport nodes are unaware of such state. Note 
that this is not the case when multicast is enabled in the core network.”
This statement may depend on the solution applied or the terms used here 
might not be entirely clear. Even if the core only provides point to 
point tunnels, then those tunnels have to be established in the core, 
hence maintenance of some states is required in the core nodes. If the 
core provides multipoint tunnels as well, then of course more state is 
required to maintain a multipoint tunnel in the core than a point to 
point tunnel. The type of connectivity that needs to be provided by the 
core depends on the actual location of VMs belonging to the same 
group/tenant. If VMs of a group are only behind two NVEs, then it is 
enough to provide a point to point connectivity/tunnel by the core 
independently of whether the traffic among the VMs is unicast or 
multicast. If VMs of a group are behind more than two NVEs, then the 
core has to provide a multipoint connectivity between those NVEs; the 
way it is provided is solution dependent.

“Load balancing may not be optimal as the hash algorithm may not work 
well due to the limited number of combinations of tunnel source and 
destination addresses”
There are other possibilities of load balancing besides hash.
It would be better to update the sentence to “Hash-based load balancing 
may not be optimal as the hash algorithm may not work well due to the 
limited number of combinations of tunnel source and destination addresses”

4.2.1. Data plane vs Control plane driven

“Multicasting in the core network for dynamic learning can lead to 
significant scalability limitations.”
Multicast should be kept within the virtual network even while it is 
transmitted in the core, which can be aided by service auto-discovery. 
Having these features, the scalability limitations should not be 
significant.

“Specific forwarding rules must be enforced to prevent loops from 
happening. This can be achieved using a spanning tree protocol or a 
shortest path tree, or using a split-horizon mesh.”
“a spanning tree protocol” is not the same category as “a shortest path 
tree” or “split horizon mesh” here. The first one is a protocol, while 
the latter two are forwarding rules as said in the beginning of the 
sentence.
It would be better to remove “protocol” from the sentence or resolve it 
another way.

“the amount of state to be distributed is a function of the number of 
virtual machines.”
This ‘function’ is not that easy to draw, it depends on a lot of 
factors, furthermore it is most likely that the state to be maintained 
does not depend on the way the state is installed, i.e. it is the same 
both for the control and the data plane driven cases.
For example, there is no need to maintain any state if the connectivity 
is point to point through the core, i.e. a group of VMs are only behind 
two NVEs.
We can take a look on mapping tables e.g. by having an example of three 
NVEs: NVE-A, NVE-B and NVE-C providing the connectivity through the core 
for a group of VMs. If VM1 behind NVE-A communicates with VN2 behind 
NVE-B, then the mapping table in NVE-A is VN2->NVE-B, i.e. NVE-A has to 
address the packets to NVE-B for the transmission through the core if 
any VM behind NVE-A sends them to VN-2. Similarly the mapping table is 
VN1->NVE-A in NVE-B. It does not matter how these mapping tables are 
maintained, the same states should be there. That is, states are the 
same both for data and control plane driven mapping tables.
If state here means a mapping state, then it would be better to update 
the cited sentence. Moreover, maybe this section is not the best 
location for it. One way out is to remove the paragraph. Alternatively 
it can be updated, e.g. “the amount of state to be distributed depends 
on the network scenario, the grouping and the number of virtual machines.”
If the state means something else, then it would be useful to clarify 
what exactly meant here.

4.2.6. Interaction between network overlays and underlays

Is it really the case that the DC provider wants to give visibility of 
its infrastructure to its Tenants? Isn’t it that the Tenant gets is 
overlay and the underlay should be completely hidden? For example, if 
the Tenant buys L2aaS, does/should it bother with the fact that L2 
overlay happens to be provided by an L3 underlay? Maybe the direction of 
SLAs would be more appropriate here than providing visibility of the 
underlay.

Best regards,
János

On 6/18/2012 11:51 PM, Benson Schliesser wrote:
> -Benson&  Matthew

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 29 Jun 2012 17:17:32 +0200

Janos,

It is well understood that PBB and SPBM address traditional Ethernet limitations.

However, our intent is not to focus the nvo3 framework towards specific L2 technologies.
In fact, nvo3 targets IP and/or MPLS as the underlying transport and not Ethernet-based transport.

Thanks,
Marc

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of János Farkas
> Sent: Friday, June 29, 2012 3:35 PM
> To: [email-masked]
> I have some comments to the draft. Please find them below 
> under the section header they belong to.
> 
> 
> 1. Introduction
> 
> "Limited VLAN space"
> "FIB explosion due to handling of large number of MACs/IP addresses"
> "Spanning Tree limitations"
> "Broadcast storms"
> Broadcast storms are not necessarily there in today's L2 
> networks, e.g. 
> "Inefficient Broadcast/Multicast handling"
> "Limited mobility/portability support"
> "Lack of service auto-discovery"
> The issue list is introduced as "Existing virtual network models used 
> context of multiple tenants. These issues can be summarized as"
> today's networking technologies.
> "Unicast tunneling state management is handled at the edge of the 
> that this is not the case when multicast is enabled in the 
> core network."
> "Load balancing may not be optimal as the hash algorithm may not work 
> destination addresses"
> It would be better to update the sentence to "Hash-based load 
> balancing 
> limited number of combinations of tunnel source and 
> destination addresses"
> 
> 
> "Multicasting in the core network for dynamic learning can lead to 
> significant scalability limitations."
> "Specific forwarding rules must be enforced to prevent loops from 
> shortest path tree, or using a split-horizon mesh."
> "a spanning tree protocol" is not the same category as "a 
> shortest path 
> tree" or "split horizon mesh" here. The first one is a 
> protocol, while 
> It would be better to remove "protocol" from the sentence or 
> resolve it 
> another way.
> 
> "the amount of state to be distributed is a function of the number of 
> virtual machines."
> This 'function' is not that easy to draw, it depends on a lot of 
> it can be updated, e.g. "the amount of state to be 
> distributed depends 
> on the network scenario, the grouping and the number of 
> virtual machines."
> its infrastructure to its Tenants? Isn't it that the Tenant gets is 
> http://tools.ietf.org/html/draft-lasserre-nvo3-framework-02 
> by the NVO3 working group, ending on 02-July-2012.
> >
> > Please respond to the NVO3 mailing list with any statements 
> of approval or disapproval, along with any additional 
> comments that might explain your position. Also, if any NVO3 
> participant is aware of IPR associated with this draft, 
> please inform the mailing list and/or the NVO3 chairs.
> >
> > Thanks,
> > -Benson&  Matthew
> >

---

From: János Farkas <[email-masked]>
Date: Fri, 29 Jun 2012 17:42:12 +0200

Marc,

> It is well understood that PBB and SPBM address traditional Ethernet limitations.

PBB and SPB are approved and deployed standards. This is Ethernet today. 
With respect to Ethernet, all I suggest is to remove statements on 
Ethernet limitations that are not valid today.

> In fact, nvo3 targets IP and/or MPLS as the underlying transport and not Ethernet-based transport.

As I wrote below:
"I understand that the WG aims to provide a solution over L3. 
Nevertheless, it seems to me that it would be better to give other
motivation for it than L2 has issues, given that no issue has been 
pointed out that is still valid for Ethernet today."

I'm not against any L3 solution and clearly understand that the goal 
here is to provide one. I guess it has other motivation than L2 is poor. 
All I pointed out in my comments to the introduction is that I think it 
would be better to explain that motivation in the introduction.
Note further, that my comments to the document are not more L2 specific 
than the text itself in the draft.

Best regards,
Janos

>> 4.1. Pros&  Cons

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 29 Jun 2012 11:42:51 -0400

There appears to be a mis-communication here.
You note that the existence of PBB and SPBM is well known, and that they 
address traditional Ethernet limitations.
The framework draft has quite a bit of wording that is written as if the 
limitations of traditional Ethernet are a major problem that without 
NVO3 could not be addressed.
There are lots of good reasons for doing the NVO3 work.
Let's not have one of our key drafts making mis-statements (much less 
claiming such mis-statements as driving motivation) about the state of 
the art.

Yours,
Joel

---

From: Lucy yong <[email-masked]>
Date: Fri, 29 Jun 2012 16:32:37 +0000

I agree that NVO3 should not restrict the underlying network solution only layer 3. Nvo3 works on the overlay and virtualized network solution, it does not work on the underlying network solution and should leave the choice flexible. The framework just needs to mention when different underlying network solutions are used, encapsulation/decapsulation is necessary at the tunnel interworking point.

Some operators also express this opinion, this is why we show L3 tunneling and L2 tunneling in the use case draft and describe the tunneling interworking.

Thanks,
Lucy  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Friday, June 29, 2012 10:43 AM
To: LASSERRE, MARC (MARC)
Cc: [email-masked]; János Farkas
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Yours,
Joel

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 29 Jun 2012 18:35:32 +0200

Hi Janos, 

See my comments below.

Thanks,
Marc
> From: János Farkas [mailto:[email-masked]] 
> Sent: Friday, June 29, 2012 5:42 PM

Understood but these technologies are not that common in data centers yet, even if available from some vendors.

> With respect to Ethernet, all I suggest is to remove 
> statements on Ethernet limitations that are not valid today.

The limitations stated are still valid in most Ethernet-based (albeit traditional) DCs.

> > In fact, nvo3 targets IP and/or MPLS as the underlying 
> transport and not Ethernet-based transport.
> 
> As I wrote below:
> Nevertheless, it seems to me that it would be better to give 
> other motivation for it than L2 has issues, given that no 
> issue has been pointed out that is still valid for Ethernet today."
> 
> I'm not against any L3 solution and clearly understand that 
> the goal here is to provide one. I guess it has other 
> motivation than L2 is poor. 

True, it is also motivated by the fact that some DC operators prefer an IP-based solution.
I could make this clearer in the draft.

> All I pointed out in my comments to the introduction is that 
> I think it would be better to explain that motivation in the 
> introduction.
> Note further, that my comments to the document are not more 
> L2 specific than the text itself in the draft.
> 
> Best regards,
> Janos
> 
> 
> >
> > Thanks,
> > Marc
> >
> >> Of János Farkas
> >> The 12-bit overlay network ID limitation of the Ethernet header is 
> >> not there any more for several years. There are different types of 
> >> VLANs.
> >> C-VLANs that were defined at the first place. The combination of 
> >> these VLANs provides flexibility for network virtualization; 
> >> altogether the Ethernet header today provides 60 bits for network 
> >> virtualization.
> >> It would be better to remove this because the limitation 
> is not there 
> >> today.
> >>
> >> FIB explosion might be there in a L2 network if the 
> already available 
> >> L2 network virtualization tools are not used. FIB explosion is not 
> >> likely in a PBB network.
> >> Shortest Path Bridging (SPB) provides shortest path forwarding and 
> >> uses physical links that might have been blocked by a 
> single spanning 
> >> tree instance. Moreover, MSTP also provides multiple spanning tree 
> >> instances, which allow the use of the physical links.
> >> Broadcast storms are not necessarily there in today's L2 networks, 
> >> e.g.
> >> broadcast due to unknown destination is kept within the virtual 
> >> network, thus it is not a broadcast storm.
> >> It would be better to update the bullet such that it exactly 
> >> specifies the problem or remove it.
> >>
> >> specific. (If it is about L2, then the statement is not valid. 
> >> Handling of multicast is very efficient in L2, both shortest path 
> >> trees and shared threes can be used today.)
> >>
> >> (Furthermore, L2 networks typically automatically learn the new 
> >> location of a station after its movement.)
> >>
> >> If it is about L2, then that is not the case. SPB has in-built 
> >> service auto-discovery.
> >> As the comments to the bullets show the list is not accurate given 
> >> the existing virtual network models available for data 
> centers today.
> >> that this is not the case when multicast is enabled in the core 
> >> network."
> >> This statement may depend on the solution applied or the 
> terms used 
> >> here might not be entirely clear. Even if the core only provides 
> >> point to point tunnels, then those tunnels have to be 
> established in 
> >> the core, hence maintenance of some states is required in the core 
> >> nodes. If the core provides multipoint tunnels as well, then of 
> >> course more state is required to maintain a multipoint 
> tunnel in the 
> >> core than a point to point tunnel. The type of connectivity that 
> >> needs to be provided by the core depends on the actual location of 
> >> VMs belonging to the same group/tenant. If VMs of a group are only 
> >> behind two NVEs, then it is enough to provide a point to point 
> >> connectivity/tunnel by the core independently of whether 
> the traffic 
> >> among the VMs is unicast or multicast. If VMs of a group 
> are behind 
> >> more than two NVEs, then the core has to provide a multipoint 
> >> connectivity between those NVEs; the way it is provided is 
> solution 
> >> dependent.
> >>
> >> balancing may not be optimal as the hash algorithm may not 
> work well 
> >> due to the limited number of combinations of tunnel source and 
> >> destination addresses"
> >>
> >>
> >> "a spanning tree protocol" is not the same category as "a shortest 
> >> path tree" or "split horizon mesh" here. The first one is 
> a protocol, 
> >> while the latter two are forwarding rules as said in the 
> beginning of 
> >> the sentence.
> >> It would be better to remove "protocol" from the sentence 
> or resolve 
> >> it another way.
> >>
> >> factors, furthermore it is most likely that the state to be 
> >> maintained does not depend on the way the state is 
> installed, i.e. it 
> >> is the same both for the control and the data plane driven cases.
> >> For example, there is no need to maintain any state if the 
> >> connectivity is point to point through the core, i.e. a 
> group of VMs 
> >> are only behind two NVEs.
> >> NVEs: NVE-A, NVE-B and NVE-C providing the connectivity 
> through the 
> >> core for a group of VMs. If VM1 behind NVE-A communicates with VN2 
> >> behind NVE-B, then the mapping table in NVE-A is VN2->NVE-B, i.e.
> >> NVE-A has to
> >> any VM behind NVE-A sends them to VN-2. Similarly the 
> mapping table 
> >> is
> >> If state here means a mapping state, then it would be better to 
> >> update the cited sentence. Moreover, maybe this section is not the 
> >> best location for it. One way out is to remove the paragraph.
> >> Alternatively
> >> it can be updated, e.g. "the amount of state to be distributed 
> >> depends on the network scenario, the grouping and the number of 
> >> virtual machines."
> >> Is it really the case that the DC provider wants to give 
> visibility 
> >> of its infrastructure to its Tenants? Isn't it that the 
> Tenant gets 
> >> is overlay and the underlay should be completely hidden? 
> For example, 
> >> if the Tenant buys L2aaS, does/should it bother with the 
> fact that L2 
> >> overlay happens to be provided by an L3 underlay? Maybe 
> the direction 
> >> of SLAs would be more appropriate here than providing 
> visibility of 
> >> the underlay.
> >>
> >>
> >> Best regards,
> >> János
> >>
> >>
> >>
> comments that

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 29 Jun 2012 18:37:40 +0200

Joel,

This was certainly not the intent.
If you have a specific sentence(s) that you'd like to see reworded, could you point them to me?

Thanks,
Marc 

> From: Joel M. Halpern [mailto:[email-masked]] 
> Sent: Friday, June 29, 2012 5:43 PM
> Cc: János Farkas; [email-masked]
> You note that the existence of PBB and SPBM is well known, 
> and that they address traditional Ethernet limitations.
> The framework draft has quite a bit of wording that is 
> written as if the limitations of traditional Ethernet are a 
> major problem that without
> Let's not have one of our key drafts making mis-statements 
> (much less claiming such mis-statements as driving 
> motivation) about the state of the art.
> 
> Yours,
> Joel
>

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 29 Jun 2012 18:14:29 +0000

A comment on section 4.2.2 "Coordination between data plane and control plane"

   "Often a combination of data plane and control based learning is
   necessary."

I think this statement is too strong since in a solution proposed in draft-marques-l3vpn-end-system, for example, there is no data plane learning in a virtual network. Maybe it should be explain when such combination is necessary.

Maria

---

From: <[email-masked]>
Date: Fri, 29 Jun 2012 17:48:10 -0400

There are some limits to what will work with draft-marques-l3vpn-end-system;
I'm hoping that the authors of the VPN4DC draft will add some applicability
statement language that will help to clarify what works w/an L3VPN approach
in data centers and what those limits are (e.g., there are some applications
and VMs for which the MAC address does matter).

Data plane learning is what networks currently do to work with existing
hypervisors, and supporting it on the tenant side of the NVE helps with
some incremental deployment scenarios.

As has been pointed out a number of times, pure data plane learning leads
to a lot of BUM traffic flooding, so a combination of data plane and control
plane can work better with existing systems and improve scalability.

Thanks,
--David

> NAPIERALA, MARIA H
> Sent: Friday, June 29, 2012 2:14 PM
> To: [email-masked]
> I think this statement is too strong since in a solution proposed in draft-
> marques-l3vpn-end-system, for example, there is no data plane learning in a
> virtual network. Maybe it should be explain when such combination is
> necessary.
> 
> Maria
> 
>

---

From: Pedro Roque Marques <[email-masked]>
Date: Fri, 29 Jun 2012 17:06:21 -0700

> and VMs for which the MAC address does matter).

There are DC designs in which, by policy, the MAC address does not matter. That is not a recent development. Many DCs have application deployment rules such that applications are only allowed to use IP sockets  (easily enforced by the OS, since looking at the ethernet header requires root privileges).
DCs in which the app development team puts the app in staging and the IT team deploys it into production do to my knowledge enforce this type of security policies.

> some incremental deployment scenarios.

Data plane learning is orthogonal to L2 header transparency.
Interoperability with a pure bridged network is yet another distinct requirement.

> plane can work better with existing systems and improve scalability.

Data plane learning, l2 header transparency, bridging interoperability are all very reasonable requirements for one type of data-centers. But also an unacceptable burden for a different class of DC designs. In my view you and Maria are just looking at different types of DC designs. They are different problems.

   Pedro.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Sat, 30 Jun 2012 00:18:28 +0000

> Data plane learning, l2 header transparency, bridging interoperability
> are all very reasonable requirements for one type of data-centers. But
> also an unacceptable burden for a different class of DC designs. In my
> view you and Maria are just looking at different types of DC designs.
> They are different problems.

Precisely. And my comment was that the sentence in draft: "Often a combination of data plane and control based learning is necessary" seems to be assuming that pretty much all data centers must use (some) data plane learning. I have suggested to clarify it.

Maria

---

From: János Farkas <[email-masked]>
Date: Sat, 30 Jun 2012 09:36:19 +0200

Hi Marc,

 > The limitations stated are still valid in most Ethernet-based (albeit 
traditional) DCs.

I'm afraid the word "traditional" (and phrases alike) is not well 
defined, specific and exact. I would avoid using it in a standard 
specification.

 > Understood but these technologies are not that common in data centers 
yet, even if available from some vendors.

I'm afraid I also fail to translate this sentence to be applicable in a 
standard.

As far as I understand these type of statements are not exact. Does not 
seem to mean much more than DCs built upon 20th century technology may 
have some issues, and one has to do something if aims to get rid of 
these issues. In order to achieve that, an upgrade is needed for such 
DCs, e.g. to the NVO3 solution, which may be not that common in data 
centers yet, even if it is available from some vendors.

Please do not misunderstand me. All I'm trying to point on in my 
comments is to have exact, specific and correct statements in a standard 
as much as possible. I'm not arguing for any specific solution. This is 
a framework document; I think it should be quite generic and solution 
independent, and in-line with the specification of the charter.

Regards,
Janos