Subject: [nvo3] Comment of draft-narten-nvo3-arch-00

From: Lizhong Jin <[email-masked]>
Date: Thu, 18 Jul 2013 00:10:20 +0800

Hi Authors,
After reading this draft, I find it still combines many possible options
(e.g, dataplane). Hope to see a draft with much more clear opinion in next
version.

Several comments below:
1. Since TS is the end of the network, flow (e.g, 5 tuple) based VN
forwarding would bring much network flexibility. The L2/L3 VN service is
the traditional service, but I believe the VN should have the capability to
provide a flow based forwarding service.
2. Is there a standard interface between NVA and VM orchestration, or a
private interface? Hope to see more description in detail.
3. If push and pull are supported, then how to combine the pull and push
mode should be described, which is closely related with NVE-NVA protocol.
4. section 12, why dataplane definition is not considered as the main area
of work?

Regards
Lizhong

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Jul 2013 12:52:33 -0400

Hi Lizhong.

> (e.g, dataplane). Hope to see a draft with much more clear opinion in next

Not sure what you mean here when you say "combines many possible
options (e.g., dataplane)."

One thing that became apparently in writing this document is that the
WG hasn't actually (on the record) decided number of things. To me,
some of the decisions are obvious (in the sense that they make sense
to me and others I've talked to), but that is not the same as the WG
actually weighing in making a choice.

One of the key differences between the architecture document and the
framework document is that the framework is a bit more open ended in
terms of possible approaches, whereas the architecture will reflect
the architectural choices the WG has made for NVO3.

So, an important goal in putting together this strawman is to get
feedback and see where folk seem to say "yes", and where they say
"actually, we should do something different..."

I'll respond to your specific questions in separate threads, so that
it's a bit easier to track/follow the issues.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Jul 2013 12:53:43 -0400

> provide a flow based forwarding service.

I'm not at all sure what it would mean for the VN itself (that is
NVO3) to provide "flow based forwarding service". What did you have in
mind? What benefits would that have?

In my mind, how traffic is forwarded by the underlay network is
entirely the business of the underlay. Flow based forwarding would be
fine, if desirable. But, that would be completely orthogonal to NVO3
and the virtual network itself. I don't see right off the need for
NVO3 itself to be involved in this.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Jul 2013 12:57:12 -0400

> mode should be described, which is closely related with NVE-NVA

I agree, more detail is needed here. But that said, I expect that
there will be some implementations that want to go heavy on the "push"
side while others might be heavier on the "pull" side. I don't see it
as a requirement of the architecture to define exactly where in
spectrum between pull (only) and push (only) an optimal spot
is. Rather, I think the architecture should support a range of
behaviors, so that an NVE would work equally with an NVA that is "push
heavy" as with one that is "pull heavy".

So in that sense, I don't see the architecture document as being the
place that "chooses" or "recommends" a "best choice", or makes a lot
of arguments about the pros an cons of each choice.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Jul 2013 12:55:30 -0400

> private interface? Hope to see more description in detail.

Today, there are already existing VM orchestration systems that embed
some or all of the functionality of the NVA. I don't see those systems
as changing or wanting to implement a standard interface just for the
sake of doing so. So, today, there are private interfaces that do that
and they will presumably continue to exist in the future.

However, IMO there is still a need to have a standard interface so
that entities other than hypervisors can interact with the
NVA. Existing VM orchestration systems that have an embedded NVA may
then find it useful to implement such a standard interface, but I
would expect they would continue to also support their private
interface for backwards compatability with other components of their
systems.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 24 Jul 2013 12:58:24 -0400

> 4. section 12, why dataplane definition is not considered as the main area

Let me just quote the entirety of Section 12, and ask others in the WG
to comment. My sense is that there is little reason to do any
additional data plane work. There are existing data planes that are
"good enough" and defining yet one more doesn't provide enough
benefit. But this is something that the WG needs to make a decision
on.

>From the draft:

> 10.  NVO3 Data Plane Encapsulation
> 
>    When tunneling tenant traffic, NVEs add encapsulation header to the
>    original tenant packet.  The exact encapsulation to use for NVO3 does
>    not seem to be critical.  The main requirement is that the
>    encapsulation support a Context ID of sufficient size
>    [I-D.ietf-nvo3-dataplane-requirements].  A number of encapsulations
>    already exist that provide a VN Context of sufficient size for NVO3.
>    For example, VXLAN [I-D.mahalingam-dutt-dcops-vxlan] has a 24-bit
>    VXLAN Network Identifier (VNI).  NVGRE
>    [I-D.sridharan-virtualization-nvgre] has a 24-bit Tenant Network ID
>    (TNI).  MPLS-over-GRE provides a 20-bit label field.  While there is
>    widespread recognition that a 12-bit VN Context would be too small
>    (only 4096 distinct values), it is generally agreed that 20 bits (1
>    million distinct values) and 24 bits (16.8 million distinct values)
>    are sufficient for a wide variety of deployment scenarios.
> 
>    [Note: the following paragraph is included for WG discussion.  Future
>    versions of this document may omit this text.]
> 
>    While one might argue that a new encapsulation should be defined just
>    for NVO3, no compelling requirements for doing so have been
>    identified yet.  Moreover, optimized implementations for existing
>    encapsulations are already starting to become available on the market
>    (i.e., in silicon).  If the IETF were to define a new encapsulation
>    format, it would take at least 2 (and likely more) years before
>    optimized implementations of the new format would become available in
>    products.  In addition, a new encapsulation format would not likely
>    displace existing formats, at least not for years.  Thus, there seems
>    little reason to define a new encapsulation.  However, it does make
>    sense for NVO3 to support multiple encapsulation formats, so as to
>    allow NVEs to use their preferred encapsulations when possible.  This
>    implies that the address dissemination protocols must also include an
>    indication of supported encapsulations along with the address mapping
>    details.
>

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 24 Jul 2013 19:37:04 +0000

Thomas, 

Even though TRILL has detailed directory mechanism document, TRILL's Directory Framework document includes the following basic requirement for Push/Pull:

Push: When the incoming data frame's DA is not found in entries being pushed down, policies need to be configured on the edge: 
	
      - simply drop the data packet,

      - flood it to the core, or

      - start the pull process to get information from pull directory server(s) (i.e. NVA in NVO3's terminology)

Pull: 
 - need to specify if the pulled entries can be cached, or timer for the cache
 - need policy to specify if the data frame can be flooded across core if no-reply is received
 - Directory (NVA) needs to notify all edges (NVEs) when the pulled entries become stale. 

Hope the NVO3 architecture document can also include those necessary requirement for Push and Pull. 

Linda 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Narten
> Sent: Wednesday, July 24, 2013 11:57 AM
> To: Lizhong Jin
> Cc: [email-masked]
> Subject: [nvo3] Arch: push vs. pull
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 24 Jul 2013 19:47:26 +0000

I would think that the interface between NVA and VM orchestration is out of the NVO3's scope (if not out of IETF's scope), isn't it?

Linda 

> Sent: Wednesday, July 24, 2013 11:56 AM
> To: Lizhong Jin
> Cc: [email-masked]
> Subject: [nvo3] Arch: NVA - VM Orchestration interface
>

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 25 Jul 2013 23:42:50 +0800

> options (e.g., dataplane)."
[Lizhong] In section 10, I see at least three dataplane ecapsulation
approach (vxlan, nvgre, mpls over gre). I hope the WG will not standardize
so many kinds of encapsulations, otherwise it would be a nightmare for
vendors. The arch document should weight among the candidates, and gives
out the standardization direction, or at least should point out the reason
why we need three encapsulations.

>From the market point, I haven't heard of any potential NIC supporting MPLS
over GRE. Anyone heard of that?

Regards
Lizhong

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 25 Jul 2013 23:51:30 +0800

> mind? What benefits would that have?
[Lizhong] flow based forwarding will not have influence to the underlay,
but the interface between NVE and NVA needs to exchange the flow entry, not
L2/L3 forwarding entry. There would be some scenario for flow based
forwarding, e.g, firewall needs to forward different flow to different VM.
The benefits is more finely control of the flow distribution.

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 25 Jul 2013 23:54:45 +0800

> benefit. But this is something that the WG needs to make a decision
[Lizhong] I hope to see less dataplane approach to be standardized. The
IETF WG should try its best to have one standardization solution, there are
many examples, e.g, MPLS-TP OAM.

Regards
Lizhong

---

From: Lizhong Jin <[email-masked]>
Date: Fri, 26 Jul 2013 00:02:10 +0800

> heavy" as with one that is "pull heavy".
[Lizhong] I share the same technical view with you here. And it would be
valuabe to give out the system model example for "push heavy" or "pull
heavy". Anyway, this is only a suggestion.

Lizhong

---

From: Lizhong Jin <[email-masked]>
Date: Fri, 26 Jul 2013 00:09:22 +0800

On Thu, Jul 25, 2013 at 3:47 AM, Linda Dunbar <[email-masked]>wrote:

> I would think that the interface between NVA and VM orchestration is out
> of the NVO3's scope (if not out of IETF's scope), isn't it?
>
[Lizhong] I don't know which standardization body should work with IETF to
standardize this interface (maybe WG Chair or AD should care about that).
But from a standardized solution point of view, it would have potential
benefit if NVA could be a standalone element, and have standardized
interface with other elements.

Regards
Lizhong

---

From: "Black, David" <[email-masked]>
Date: Sun, 28 Jul 2013 04:38:44 -0400

The question that I have is: How does your notion of flow-based forwarding relate to
the notions of L2 and L3 service in the problem statement and framework drafts?

We've had a prior discussion of IRB (combined L2/L3 implementation) and my impression
of the conclusion of that discussion is that the service is still L2, but the L3
implementation has been optimized.  Is something similar going on here?

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lizhong Jin
Sent: Thursday, July 25, 2013 11:52 AM
To: Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Arch: Flow based forwarding service

I'm not at all sure what it would mean for the VN itself (that is
NVO3) to provide "flow based forwarding service". What did you have in
mind? What benefits would that have?
[Lizhong] flow based forwarding will not have influence to the underlay, but the interface between NVE and NVA needs to exchange the flow entry, not L2/L3 forwarding entry. There would be some scenario for flow based forwarding, e.g, firewall needs to forward different flow to different VM. The benefits is more finely control of the flow distribution.

Thomas

---

From: "Black, David" <[email-masked]>
Date: Sun, 28 Jul 2013 04:41:23 -0400

Lizhong,

We can certainly talk about what the WG will or won't standardize, but
(IMHO), it's necessary to pay attention to what's in use - this is a
specific case of looking at "running code" in addition to "rough consensus".

All three of those encapsulations are deployed and in use, today.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lizhong Jin
Sent: Thursday, July 25, 2013 11:43 AM
To: Thomas Narten
Cc: [email-masked]
Subject: Re: [nvo3] Comment of draft-narten-nvo3-arch-00

On Thu, Jul 25, 2013 at 12:52 AM, Thomas Narten <[email-masked]<mailto:[email-masked]>> wrote:
Hi Lizhong.

Not sure what you mean here when you say "combines many possible
options (e.g., dataplane)."
[Lizhong] In section 10, I see at least three dataplane ecapsulation approach (vxlan, nvgre, mpls over gre). I hope the WG will not standardize so many kinds of encapsulations, otherwise it would be a nightmare for vendors. The arch document should weight among the candidates, and gives out the standardization direction, or at least should point out the reason why we need three encapsulations.

>From the market point, I haven't heard of any potential NIC supporting MPLS over GRE. Anyone heard of that?

Regards
Lizhong

Thomas

---

From: Jon Hudson <[email-masked]>
Date: Sun, 28 Jul 2013 12:17:40 +0200

+1 "However, IMO there is still a need to have a standard interface"

---

From: Jon Hudson <[email-masked]>
Date: Sun, 28 Jul 2013 12:28:53 +0200

Have to agree on the need to support both. 

I have been honestly a little baffled by all the "push vs pull" discussion. 

In a past life of developing monitoring tools for datacenters/networks we quickly saw that that one model never met all needs. Too many odd scenarios (one being how do you ask someone if they are dead yet)

So a hybrid where remote nodes would push but a pull could be done whenever needed or something along those lines would always win out.

While I understand the need to lock down certain design ideas, I'm not sure push vs pull needs to be as Boolean as we have made it.

And to Thomas's point some implementations may choose to go heavy one way or another as a competitive advantage toward their in house skills. We need to give that room.

---

From: Lizhong Jin <[email-masked]>
Date: Sun, 28 Jul 2013 23:19:36 +0800

Hi David,
Flow based refer to forward the traffic based on not only MAC/VLAN(L2) and
Dest-IP(L3), but also, e.g, 5 tuple, including TCP/UDP port. This is useful
for some network service, e.g, firewall.

Regards
Lizhong
 On Jul 28, 2013 4:39 PM, "Black, David" <[email-masked]> wrote:

> The question that I have is: How does your notion of flow-based forwarding
> relate to****
>
> the notions of L2 and L3 service in the problem statement and framework
> drafts?****
>
> ** **
>
> We’ve had a prior discussion of IRB (combined L2/L3 implementation) and my
> impression****
>
> of the conclusion of that discussion is that the service is still L2, but
> the L3****
>
> implementation has been optimized.  Is something similar going on here?***
> *
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Lizhong Jin
> *Sent:* Thursday, July 25, 2013 11:52 AM
> *To:* Thomas Narten
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Arch: Flow based forwarding service****
>
> ** **
>
> ****
> mind? What benefits would that have?****
>
> The benefits is more finely control of the flow distribution.****
>
>  ****
>
>
> Thomas****
> ** **

---

From: Lizhong Jin <[email-masked]>
Date: Sun, 28 Jul 2013 23:26:24 +0800

Thanks David for the information. Then standardizing three dataplane
options makes me dissappointed.

Regards
Lizhong
On Jul 28, 2013 4:41 PM, "Black, David" <[email-masked]> wrote:

> Lizhong,****
>
> ** **
>
> We can certainly talk about what the WG will or won’t standardize, but****
>
> (IMHO), it’s necessary to pay attention to what’s in use – this is a****
>
> specific case of looking at “running code” in addition to “rough
> consensus”.****
>
> ** **
>
> All three of those encapsulations are deployed and in use, today.****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *Sent:* Thursday, July 25, 2013 11:43 AM
> *To:* Thomas Narten
> *Cc:* [email-masked]
> *Subject:* Re: [nvo3] Comment of draft-narten-nvo3-arch-00****
>
> ** **
>
> ** **
>
> Hi Lizhong.****
> > version.****
> options (e.g., dataplane)."****
>
> why we need three encapsulations.****
>
> ** **
>
> From the market point, I haven't heard of any potential NIC supporting
> MPLS over GRE. Anyone heard of that?****
>
> ** **
>
> Regards****
>
> Lizhong****
>
> ** **
>
>

---

From: Linda Dunbar <[email-masked]>
Date: Sun, 28 Jul 2013 17:14:27 +0000

> In a past life of developing monitoring tools for datacenters/networks
> we quickly saw that that one model never met all needs. Too many odd
> scenarios (one being how do you ask someone if they are dead yet)
> 
> So a hybrid where remote nodes would push but a pull could be done
> whenever needed or something along those lines would always win out.
> 

[Linda] 100% agree. There could be situations where mapping for some Virtual network Instances need to be pushed whereas others need to be pulled.