Subject: Re: [nvo3] draft-­-pang-­-nvo3-­-vxlan-­-path-­-detection-­-01

From: "Dacheng Zhang" <[email-masked]>
Date: Wed, 04 Nov 2015 09:52:38 +0800

This draft actually proposes a mechanism where the intermediates are
required to recognize the vxlan oam packets. If this assumption is broken,
the solutions proposed in this draft may not be effective.

Cheers

Dacheng

发件人:  nvo3 <[email-masked]> on behalf of Shahram Davari
<[email-masked]>
日期:  2015年11月4日 星期三 上午9:33
至:  "[email-masked]" <[email-masked]>
主题:  [nvo3] draft-‐pang-‐nvo3-‐vxlan-‐path-‐detection-‐01

Hi,
 
This draft needs to address how intermediate L3 routers are going to see
these VXLAN OAM packets, since L3 routers just do L3 routing and don’t look
at the payload to see it is VXLAN and then see that these are PD OAM
packets. The only option I can think of is TTL expiry, otherwise it won’t
work, the way it is defined now,
 
Thx
Shahram
_______________________________________________ nvo3 mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/nvo3

---

From: Shahram Davari <[email-masked]>
Date: Wed, 4 Nov 2015 01:58:49 +0000

The only text I can see is the following:

“Each network device receives the Path Tracking packet from its
   upstream device, makes a copy of it and passes the copy to its CPU.
“

I don’t see any mechanism being proposed in the draft.

Thx
SD

From: Dacheng Zhang [mailto:[email-masked]]
Sent: Tuesday, November 03, 2015 5:53 PM
To: Shahram Davari; [email-masked]
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

This draft actually proposes a mechanism where the intermediates are required to recognize the vxlan oam packets. If this assumption is broken, the solutions proposed in this draft may not be effective.

Cheers

Dacheng

发件人: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of Shahram Davari <[email-masked]<mailto:[email-masked]>>
日期: 2015年11月4日 星期三 上午9:33
至: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
主题: [nvo3] draft-‐pang-‐nvo3-‐vxlan-‐path-‐detection-‐01

Hi,

This draft needs to address how intermediate L3 routers are going to see these VXLAN OAM packets, since L3 routers just do L3 routing and don’t look at the payload to see it is VXLAN and then see that these are PD OAM packets. The only option I can think of is TTL expiry, otherwise it won’t work, the way it is defined now,

Thx
Shahram
_______________________________________________ nvo3 mailing list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Shahram Davari <[email-masked]>
Date: Wed, 4 Nov 2015 02:02:17 +0000

I think your assumption is broken. But you have an alternative method and that is using TTL expiry.

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: "Dacheng Zhang" <[email-masked]>
Date: Wed, 04 Nov 2015 10:24:53 +0800

I can give a simple example. A OpenFlow switch can easily be used to support
this function. But of course it is an example, we do not try to argue that
we should  use Openflow.

发件人:  nvo3 <[email-masked]> on behalf of Shahram Davari
<[email-masked]>
日期:  2015年11月4日 星期三 上午9:58
至:  dacheng de <[email-masked]>, "[email-masked]"
<[email-masked]>
主题:  Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

 
 
Thx
SD

Cheers

Dacheng

日期: 2015年11月4日星期三上午9:33

Hi,

Thx
Shahram

---

From: Shahram Davari <[email-masked]>
Date: Wed, 4 Nov 2015 02:28:28 +0000

But the L3 device in the core of a data center  is a router not an open flow switch that can arbitrarily match on any field. And this draft is in IETF and not ONF.

Thx
SD

From: nvo3 [mailto:[email-masked]] On Behalf Of Dacheng Zhang
Sent: Tuesday, November 03, 2015 6:25 PM
To: Shahram Davari; [email-masked]
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

I can give a simple example. A OpenFlow switch can easily be used to support this function. But of course it is an example, we do not try to argue that we should  use Openflow.

发件人: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of Shahram Davari <[email-masked]<mailto:[email-masked]>>
日期: 2015年11月4日 星期三 上午9:58
至: dacheng de <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
主题: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Thx
SD

From: Dacheng Zhang [mailto:[email-masked]]
Sent: Tuesday, November 03, 2015 5:53 PM
To: Shahram Davari; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 3 Nov 2015 18:29:07 -0800

I expressed the same concern at last IETF meeting, as Shahram raised here.
Haven’t gotten the  explanation yet.

If TTL expiry mechanism is used, then the definition of IP TTL will have to be redefined in order to make a copy and forward to next hop.
But if L3 devices have to read into VXLAN header to determine OAM bit is set, they need to implement DPI for the same.

Secondly, imagine when there exists a loop. In fact, they do exist even in controller based networks. 

Speaking as an operator, as mentioned yesterday, this will cause packet storm and unintended consequences.

Why are we solving the problem when it doesn’t exist?

-sam

> On Nov 3, 2015, at 6:02 PM, Shahram Davari <[email-masked]> wrote:
> 
> 发件人: nvo3 <[email-masked] <mailto:[email-masked]>> on behalf of Shahram Davari <[email-masked] <mailto:[email-masked]>>
> 至: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> _______________________________________________ nvo3 mailing list [email-masked] <mailto:[email-masked]>https://www.ietf.org/mailman/listinfo/nvo3 <https://www.ietf.org/mailman/listinfo/nvo3>_______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Dapeng Liu <[email-masked]>
Date: Wed, 4 Nov 2015 10:36:56 +0800

Hi Shahram,

We can add text to explain the mechanism in the next version.

Another example is using ACL to match the PD packet and hardware support it.

Regards,
Dapeng Liu

2015-11-04 9:58 GMT+08:00 Shahram Davari <[email-masked]>:

> *From:* Dacheng Zhang [mailto:[email-masked]]
> *Sent:* Tuesday, November 03, 2015 5:53 PM
> *To:* Shahram Davari; [email-masked]
> *Subject:* Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01
>
>
>
> *发件人**: *nvo3 <[email-masked]> on behalf of Shahram Davari <
> [email-masked]>
> *日期**: *2015年11月4日 星期三 上午9:33
> *至**: *"[email-masked]" <[email-masked]>
> *主题**: *[nvo3] draft-‐pang-‐nvo3-‐vxlan-‐path-‐detection-‐01
>
>
>
> Hi,
>
>
>

-- 

------
Best Regards,
Dapeng Liu

---

From: "Dacheng Zhang" <[email-masked]>
Date: Wed, 04 Nov 2015 10:44:39 +0800

There are various deployment scenarios. So, we should only say what will
happen if our assumption is broken. In addition, compared with TTL, our
solution will be more efficient in checking the path connectivity of complex
networks.

发件人:  nvo3 <[email-masked]> on behalf of Shahram Davari
<[email-masked]>
日期:  2015年11月4日 星期三 上午10:02
至:  dacheng de <[email-masked]>, "[email-masked]"
<[email-masked]>
主题:  Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

I think your assumption is broken. But you have an alternative method and
that is using TTL expiry.
 
Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Dapeng Liu <[email-masked]>
Date: Wed, 4 Nov 2015 10:47:04 +0800

2015-11-04 10:29 GMT+08:00 Sam Aldrin <[email-masked]>:

> If TTL expiry mechanism is used, then the definition of IP TTL will have
> to be redefined in order to make a copy and forward to next hop.
> But if L3 devices have to read into VXLAN header to determine OAM bit is
> set, they need to implement DPI for the same.
>

As respond to  Shahram's question, there are various ways to implement
this. ACL etc..

> controller based networks.

The PD packet has TTL also to avoid this.

> storm and unintended consequences.

We are also operator, we do not see problems here.  Maybe we need more
clarification and offline discussion to make sure our solution be
understood correctly.

Thanks for the comments,
Dapeng Liu

> *From:* Dacheng Zhang [mailto:[email-masked]
> <[email-masked]>]
> [email-masked]://www.ietf.org/mailman/listinfo/nvo3

-- 

------
Best Regards,
Dapeng Liu

---

From: Shahram Davari <[email-masked]>
Date: Wed, 4 Nov 2015 03:02:41 +0000

By TTL expiry I didn't mean change TTL expiry behavior. I meant set TT=1 for first DP packet then set TTL=2 for 2nd DP packet and so forth. And for each TTL just one router that detects TTL expiry will respond.

Regards,
Shahram

On Nov 4, 2015, at 11:47 AM, Dapeng Liu <[email-masked]<mailto:[email-masked]>> wrote:

2015-11-04 10:29 GMT+08:00 Sam Aldrin <[email-masked]<mailto:[email-masked]>>:
I expressed the same concern at last IETF meeting, as Shahram raised here.
Haven’t gotten the  explanation yet.

As respond to  Shahram's question, there are various ways to implement this. ACL etc..

We are also operator, we do not see problems here.  Maybe we need more clarification and offline discussion to make sure our solution be understood correctly.

Thanks for the comments,
Dapeng Liu

-sam

On Nov 3, 2015, at 6:02 PM, Shahram Davari <[email-masked]<mailto:[email-masked]>> wrote:

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram
_______________________________________________ nvo3 mailing list [email-masked]<mailto:[email-masked]>https://www.ietf.org/mailman/listinfo/nvo3
_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

--

------
Best Regards,
Dapeng Liu

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 3 Nov 2015 19:04:43 -0800

> On Nov 3, 2015, at 6:47 PM, Dapeng Liu <[email-masked]> wrote:
> 
> 
> 
> 2015-11-04 10:29 GMT+08:00 Sam Aldrin <[email-masked] <mailto:[email-masked]>>:
When there is text related to this, we could discuss. Without that, no point speculating how it could be done.
> The PD packet has TTL also to avoid this.
Having TTL doesn’t avoid, it may minimize the implosion.
> We are also operator, we do not see problems here.  Maybe we need more clarification and offline discussion to make sure our solution be understood correctly.
Please talk to your co-authors. I did spend quite a bit of time to explain on why it doesn’t work.

My take on this is, If you have specific needs and vendors are willing to make it work for you, please go ahead by all means, and you do not need to standardize that.
If you really need to standardize it, ensure that it doesn’t break other networks deployments. And VXLAN or similar protocols within NVo3 or  IETF at large, are meant to be used elsewhere as well, ex: DCI etc.
I haven’t seen technical merits or strong use cases for the intended method.

Lastly, OAM DT team could consider this into the discussion.

-sam
>> On Nov 3, 2015, at 6:02 PM, Shahram Davari <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> From: Dacheng Zhang [mailto:[email-masked] <mailto:[email-masked]>] 
>> To: Shahram Davari; [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/nvo3 <https://www.ietf.org/mailman/listinfo/nvo3>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nvo3 <https://www.ietf.org/mailman/listinfo/nvo3>
> 
> 
> 
> 
> -- 
> 
> ------
> Best Regards,
> Dapeng Liu

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Wed, 4 Nov 2015 03:10:56 +0000

Hi Shahram/Sam,

This solution is hardware centric with controller and policy needs to be created on each hop.
This solution is not applicable for all scenarios.

Policy example
Match peer vtep ip == destination ip of packet destination  port 4789, pd bit action punt and drop.
Match peer vtep ip!=destination ip destination port == 4789, pd bit action punt and forward.

Now drop takes care of leak scenario from leafs.

Now controller eats up the packet so no issue of loop.
Also in network packet is going as data packet as per vxlan rule of max ttl so not sure where's loop.
If loop is there oam and data both will suffer.

Loop with controller can be avoided but that's outside the scope.

Alibaba is also operator and using this data center for cloud services.

I agree Ttl expiry will also work but that's software solution and separate draft not this draft intention.

On Concern of policy application controller will apply the policy and if network is not hardware oam capable they won't initiate it and use software oam method.

We evaluated multiple Asic and found out solution can be done on multiple broadcom and custom Asic and Alibaba network is running on 2 different Broadcom Asic.

Thanks
Deepak

Sent from my iPhone

On Nov 4, 2015, at 11:29 AM, Sam Aldrin <[email-masked]<mailto:[email-masked]>> wrote:

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 3 Nov 2015 19:41:43 -0800

Hi Deepak,

What you are describing is very narrow scenario, which has its own pitfalls.
Inline for my comments.
> On Nov 3, 2015, at 7:10 PM, Deepak Kumar (dekumar) <[email-masked]> wrote:
> 
> Hi Shahram/Sam,
> 
If you want to employ policy for every vtep and on every device in the network, IMO, a bad design to start with.
> Also in network packet is going as data packet as per vxlan rule of max ttl so not sure where's loop.
You mean there cannot be loops in n/w, just because TTL is used? (loop life is dependent on ttl)

> If loop is there oam and data both will suffer.
Yes both will suffer. You use OAM to detect whether data plane has problem or not. With this, it will compound the problem.
> I agree Ttl expiry will also work but that's software solution and separate draft not this draft intention.
If you already have a solution, why invent a new one? Are you saying controller is not efficient and cannot perform oam efficiently with existing ttl mechanism? :D

> On Concern of policy application controller will apply the policy and if network is not hardware oam capable they won't initiate it and use software oam method.
Well, you have the answer right there. 
In other words, if a device cannot support your proposed solution, you will revert back to ttl solution. why don’t you just use that solution instead?
> We evaluated multiple Asic and found out solution can be done on multiple broadcom and custom Asic and Alibaba network is running on 2 different Broadcom Asic.
And your point being? :D

-sam
> 
> Thanks 
> Deepak
> 
> Sent from my iPhone
> 
> On Nov 4, 2015, at 11:29 AM, Sam Aldrin <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Wed, 4 Nov 2015 04:03:29 +0000

HI Sam,

This is good discussion and we are bringing this draft as informatiinal draft for narrow scenario for some operators but not for other operators.

Ttl solution is too slow at scale and instead of argument we can give data of how much time it takes but for some operator that amount of time is okay but for some they have will want it to complete it quickly. As this being informational solution it's brought to working group as hardware driven controller controlled scenario and make its language may and should so all the issues it may cause to software vtep can be fixed.

Why can't software based and hardware based solution co-exist when information draft won't force everyone to implement it.

Thanks
Deepak

Sent from my iPhone

On Nov 4, 2015, at 12:41 PM, Sam Aldrin <[email-masked]<mailto:[email-masked]>> wrote:

Hi Deepak,

What you are describing is very narrow scenario, which has its own pitfalls.
Inline for my comments.
On Nov 3, 2015, at 7:10 PM, Deepak Kumar (dekumar) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

???: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of Shahram Davari <[email-masked]<mailto:[email-masked]>>
??: 2015?11?4? ??? ??9:33
?: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
??: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Hi,

Thx
Shahram

---

From: Dapeng Liu <[email-masked]>
Date: Wed, 4 Nov 2015 13:10:03 +0800

> Please talk to your co-authors. I did spend quite a bit of time to explain
I guess we have answered your question and it indeed works.

> My take on this is, If you have specific needs and vendors are willing to
> make it work for you, please go ahead by all means, and you do not need to
> standardize that.
> If you really need to standardize it, ensure that it doesn’t break other
> networks deployments. And VXLAN or similar protocols within NVo3 or  IETF
> at large, are meant to be used elsewhere as well, ex: DCI etc.
>

This is an informational document. Why can't hardware and software
solutions both coexist.

> Lastly, OAM DT team could consider this into the discussion.

There is a need for more efficient tool.

Dapeng Liu

-- 

------
Best Regards,
Dapeng Liu

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 3 Nov 2015 22:09:12 -0800

Hi Deepak,

Aren’t you or aren’t you not changing the packet format by introducing PD flag bit in the reserved field. i.e changing RFC7348?
If so, how can you claim to be informational? Is it because RFC is informational?

For ex, VXLAN-GPE is in standards track, although it is now in expired state.

Irrespective of technical differences, if a specific format is being changed, it will impact existing future deployments as well, informational or not.
Being informational does not avoid that.

-sam
> On Nov 3, 2015, at 8:03 PM, Deepak Kumar (dekumar) <[email-masked]> wrote:
> 
> HI Sam,
> 
> On Nov 4, 2015, at 12:41 PM, Sam Aldrin <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> Hi Deepak,
>> 
>>> On Nov 3, 2015, at 7:10 PM, Deepak Kumar (dekumar) <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Hi Shahram/Sam,
>>>

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 3 Nov 2015 22:17:01 -0800

> On Nov 3, 2015, at 9:10 PM, Dapeng Liu <[email-masked]> wrote:
> 
> 
> 
> 2015-11-04 11:04 GMT+08:00 Sam Aldrin <[email-masked] <mailto:[email-masked]>>:
> 
>> On Nov 3, 2015, at 6:47 PM, Dapeng Liu <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 
Disagree. As pointed out, draft text doesn’t provide complete details either. 

> This is an informational document. Why can't hardware and software solutions both coexist.
The point is about behavior being changed, not about co-existence.
If you are solving the issue, without introducing any changes to VXLAN header and behavior, then informational is the right way to pursue, IMO.
> There is a need for more efficient tool.
There are many ways to achieve that. Just that the proposed method is not the way to achieve that, IMO.

-sam

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Wed, 4 Nov 2015 06:51:34 +0000

HI Sam,

Vxlan field that is used is reserved field and so existing Asic based hardware won't add this in transmit but receiving packet with reserved bit set has no side effect.
If hardware is programmable their is no issue even in transmit.

Can you give me example of any Asic implementation which will have problem, we can add text for user to be careful before turning  on the solution.

We can even call this extension of vxlan with pd bit.

Thanks,
Deepak

Sent from my iPhone

On Nov 4, 2015, at 3:09 PM, Sam Aldrin <[email-masked]<mailto:[email-masked]>> wrote:

Hi Deepak,

-sam
On Nov 3, 2015, at 8:03 PM, Deepak Kumar (dekumar) <[email-masked]<mailto:[email-masked]>> wrote:

HI Sam,

Thanks
Deepak

Sent from my iPhone

Hi Deepak,

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Haoweiguo <[email-masked]>
Date: Wed, 4 Nov 2015 12:31:13 +0000

Hi Sam,

The extra bit in VXLAN reserved field has no side effect on regular VXLAN forwarding process. The hardware requirements for intermediate nodes is also low, the intermediate nodes only need to grab the data packets with the OAM flag to control plane using regular ACL, most current commertial chipsets can support this behavior.

Thanks,

weiguo

________________________________
From: Deepak Kumar (dekumar) [[email-masked]]
Sent: Wednesday, November 04, 2015 14:51
To: Sam Aldrin
Cc: Shahram Davari; [email-masked]; Dacheng Zhang
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

HI Sam,

Thanks,
Deepak

Sent from my iPhone

Hi Deepak,

-sam

HI Sam,

Thanks
Deepak

Sent from my iPhone

Hi Deepak,

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Tom Herbert <[email-masked]>
Date: Thu, 5 Nov 2015 07:05:29 +0900

On Wed, Nov 4, 2015 at 9:31 PM, Haoweiguo <[email-masked]> wrote:
> Hi Sam,
>
> The extra bit in VXLAN reserved field has no side effect on regular VXLAN
> forwarding process. The hardware requirements for intermediate nodes is also
> low, the intermediate nodes only need to grab the data packets with the OAM
> flag to control plane using regular ACL, most current commertial chipsets
> can support this behavior.
>

"The extra bit in VXLAN reserved field has no side effect on regular
VXLAN forwarding process."-- I think this is an assumption based on
what implementations are probably doing currently as opposed to what
is specified by the protocol. The VXLAN RFC does not specify a
forwarding process AFAICT. And, as I mentioned yesterday, VXLAN is not
extensible in its current definition.  Unknown reserved bits must be
ignored on reception which means that a receiver that doesn't support
PD bit will assume that the payload is an Ethernet packet and attempt
to deliver it as such.

Given that this is not the first time someone has tried to extend
VXLAN and it's wide deployment, it might be prudent to try to define a
method to extend the protocol without breaking compatibility. The
pseudo packet (cannot just be called a header) trick will work if the
packet is created such that it is guaranteed to be dropped at a
receiver that doesn't understand the extensions (e.g. put a zero in
total length field in case of IP). Obviously any method should support
more than just one extension. This implies that fields must be
precisely ordered, so for an implementation to parse field, it must be
able to calculate the offset of the field. So the order that fields
are defined in VXLAN becomes important and needs to be spelled out.
The current placement of the PD bit probably would have to change for
instance. The easiest thing to do is define the flag bits left to
right, where to process the n'th bit an implementation must understand
the first n-1 bits (see GRE or GUE), but the n+1 bits and beyond can
still be ignored.

Tom

The pseudo header trick for extensibility might work
> Vxlan field that is used is reserved field and so existing Asic based
> hardware won't add this in transmit but receiving packet with reserved bit
> set has no side effect.
> Can you give me example of any Asic implementation which will have problem,
> we can add text for user to be careful before turning  on the solution.
>
> On Nov 4, 2015, at 3:09 PM, Sam Aldrin <[email-masked]> wrote:
>
> Hi Deepak,
>
> Aren’t you or aren’t you not changing the packet format by introducing PD
> flag bit in the reserved field. i.e changing RFC7348?
> Irrespective of technical differences, if a specific format is being
> changed, it will impact existing future deployments as well, informational
> or not.
> This is good discussion and we are bringing this draft as informatiinal
> draft for narrow scenario for some operators but not for other operators.
>
> Ttl solution is too slow at scale and instead of argument we can give data
> of how much time it takes but for some operator that amount of time is okay
> but for some they have will want it to complete it quickly. As this being
> informational solution it's brought to working group as hardware driven
> controller controlled scenario and make its language may and should so all
> the issues it may cause to software vtep can be fixed.
>
> Why can't software based and hardware based solution co-exist when
> information draft won't force everyone to implement it.
>
> Thanks
> Deepak
>
> Sent from my iPhone
>
> On Nov 4, 2015, at 12:41 PM, Sam Aldrin <[email-masked]> wrote:
>
> Hi Deepak,
>
> If you want to employ policy for every vtep and on every device in the
> network, IMO, a bad design to start with.
>
>
> Yes both will suffer. You use OAM to detect whether data plane has problem
> or not. With this, it will compound the problem.
>
>
> If you already have a solution, why invent a new one? Are you saying
> controller is not efficient and cannot perform oam efficiently with existing
> ttl mechanism? :D
>
>
> On Concern of policy application controller will apply the policy and if
> network is not hardware oam capable they won't initiate it and use software
> oam method.
>
> In other words, if a device cannot support your proposed solution, you will
> revert back to ttl solution. why don’t you just use that solution instead?
>
>
> We evaluated multiple Asic and found out solution can be done on multiple
> broadcom and custom Asic and Alibaba network is running on 2 different
> Broadcom Asic.
>
> And your point being? :D
>
> -sam
>
>
> Thanks
> Deepak
>
> Sent from my iPhone
>
> On Nov 4, 2015, at 11:29 AM, Sam Aldrin <[email-masked]> wrote:
>
> If TTL expiry mechanism is used, then the definition of IP TTL will have to
> be redefined in order to make a copy and forward to next hop.

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Thu, 5 Nov 2015 00:43:29 +0000

Hi,

I double checked with all of my hardware teams, Reserved bit are usually ignored on reception so there’s no side effect to vxlan hardware, I believe TCP also was extended to support ECN this way. Also we are keeping this within network so server/vm won’t see this traffic.

Now when new encapsulation which are hardware friendly comes (Geneve/GUe/GPE we will respin the ASIC if required) and we will have hardware OAM solution even more powerful as we will be able to generate oam packet from ingress pipeline with OAM bit set so we verify actual datapath with QoS at initiating switch.

Hardware based oam solution for hardware vtep is necessary as TTL or existing solutions can’t address all the problem.

1.
As I said TTL expiry I like but it doesn’t verify the exact datapath as real vxlan traffic travels on the Leaf so it’s more of something better than nothing solution.
TTL pipeline in ASIC based on few implementation I know from our ASIC occur(s) before the path for vxlan packet and it doesn’t verify the exact datapath, vni to vlan/bd mapping, etc.
When packet is received on the core facing interface from the core side platform adds their own header, decrement the TTL and that cause exception, so complete packet is sent to software slow path.
When packet is getting forwarded in hardware from core port <—-> core —vtep—bridge — Edge port.
So steps
1. Add platform specific header
2. check for forwarding packet, if it’s for our (due to ip matching the vtep)
2. Decrement TTL
3. Check exception of TTL, if exception punt to software slow path
4. As it’s matching peer vtep ip, de-cap the packet and use VNI to add vlan header in the inner packet.
5. Now bridge the packet towards host on that vlan.

Even new hardware has resilient ECMP where software and hardware ECMP are not the same, so TTL expiry method will be little hard even in underlay to get egress interface.

2.
Also Leaf are deployed in resilent manner so traffic flow is not just vtep - vtep in same path.
-------
L3 core
-------
  |
L1—L2   —> Both leafs acting as Virtual Vtep IP.
\  /
 \/
 VM

Now what happen is traffic to VM can be hashed using ECMP from L1 or L2 both, Now if link between VM and L1 goes down for some reason this information is local and not known to the core and packet will still be delivered to both L1 and L2, but traffic to VM is delivered over link connecting L1 -> L2 -> VM.

3. Even on intitiator leaf without having VM’s traffic profile VM path can’t be traced.
Lets for example VM does ping between 2 endpoints it doesn’t cover the same path as real traffic as hardware at initiator does L3 hashing instead of L4 hashing for udp/tcp flows.

Now VM hashing is required to get the sport and after that if packet is delivered from software we need to find the right egress interface which require outer header and in this scenario due to tunnel this hashing is different and implementation specific but usually different than l3 hashing in the core.
In case of vxlan gpe, we won’t have to do all this software tricks as we can insert the packet right from the ingress pipeline and it will cover the right path as data, so we can move the bit position for gpe, geneve, gue as appropriately.

Thanks,
Deepak

From: Haoweiguo <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, November 4, 2015 at 4:31 AM
To: dekumar <[email-masked]<mailto:[email-masked]>>, Sam Aldrin <[email-masked]<mailto:[email-masked]>>
Cc: Shahram Davari <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Dacheng Zhang <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Hi Sam,

Thanks,

weiguo

________________________________
From: Deepak Kumar (dekumar) [[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, November 04, 2015 14:51
To: Sam Aldrin
Cc: Shahram Davari; [email-masked]<mailto:[email-masked]>; Dacheng Zhang
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

HI Sam,

Thanks,
Deepak

Sent from my iPhone

Hi Deepak,

-sam

HI Sam,

Thanks
Deepak

Sent from my iPhone

Hi Deepak,

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Liyizhou <[email-masked]>
Date: Thu, 5 Nov 2015 01:46:02 +0000

RFC7348 VxLAN itself is an informational document. IMO, being informational does not necessarily mean no new bit can be defined. The document can be improved to clearly state the applicable scenario and compatibility considerations. However the historical procedural issue should not be the only or major reason to block the progressing of such kind of documents. I do not see anything wrong to proceed it with informational in this case given there exists a clear requirement and reasonable deployment from it.

Thanks,
Yizhou

-----邮件原件-----
发件人: nvo3 [mailto:[email-masked]] 代表 Tom Herbert
发送时间: 2015年11月5日 6:05
收件人: Haoweiguo
抄送: Sam Aldrin; Shahram Davari; [email-masked]; Dacheng Zhang; Deepak Kumar (dekumar)
主题: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

On Wed, Nov 4, 2015 at 9:31 PM, Haoweiguo <[email-masked]> wrote:
> Hi Sam,
>
> The extra bit in VXLAN reserved field has no side effect on regular 
> VXLAN forwarding process. The hardware requirements for intermediate 
> nodes is also low, the intermediate nodes only need to grab the data 
> packets with the OAM flag to control plane using regular ACL, most 
> current commertial chipsets can support this behavior.
>

"The extra bit in VXLAN reserved field has no side effect on regular VXLAN forwarding process."-- I think this is an assumption based on what implementations are probably doing currently as opposed to what is specified by the protocol. The VXLAN RFC does not specify a forwarding process AFAICT. And, as I mentioned yesterday, VXLAN is not extensible in its current definition.  Unknown reserved bits must be ignored on reception which means that a receiver that doesn't support PD bit will assume that the payload is an Ethernet packet and attempt to deliver it as such.

Given that this is not the first time someone has tried to extend VXLAN and it's wide deployment, it might be prudent to try to define a method to extend the protocol without breaking compatibility. The pseudo packet (cannot just be called a header) trick will work if the packet is created such that it is guaranteed to be dropped at a receiver that doesn't understand the extensions (e.g. put a zero in total length field in case of IP). Obviously any method should support more than just one extension. This implies that fields must be precisely ordered, so for an implementation to parse field, it must be able to calculate the offset of the field. So the order that fields are defined in VXLAN becomes important and needs to be spelled out.
The current placement of the PD bit probably would have to change for instance. The easiest thing to do is define the flag bits left to right, where to process the n'th bit an implementation must understand the first n-1 bits (see GRE or GUE), but the n+1 bits and beyond can still be ignored.

Tom

> Subject: Re: [nvo3] 
> draft--pang--nvo3--vxlan--path--detection--01
>
> HI Sam,
>
> hardware won't add this in transmit but receiving packet with reserved 
> bit set has no side effect.
> Can you give me example of any Asic implementation which will have 
> problem, we can add text for user to be careful before turning  on the solution.
>
> Aren’t you or aren’t you not changing the packet format by introducing 
> PD flag bit in the reserved field. i.e changing RFC7348?
> changed, it will impact existing future deployments as well, 
> informational or not.
> This is good discussion and we are bringing this draft as 
> informatiinal draft for narrow scenario for some operators but not for other operators.
>
> Ttl solution is too slow at scale and instead of argument we can give 
> data of how much time it takes but for some operator that amount of 
> time is okay but for some they have will want it to complete it 
> quickly. As this being informational solution it's brought to working 
> group as hardware driven controller controlled scenario and make its 
> language may and should so all the issues it may cause to software vtep can be fixed.
>
> Yes both will suffer. You use OAM to detect whether data plane has 
> problem or not. With this, it will compound the problem.
>
>
> controller is not efficient and cannot perform oam efficiently with 
> existing ttl mechanism? :D
>
>
> On Concern of policy application controller will apply the policy and 
> if network is not hardware oam capable they won't initiate it and use 
> software oam method.
>
> In other words, if a device cannot support your proposed solution, you 
> will revert back to ttl solution. why don’t you just use that solution instead?
>
>
> We evaluated multiple Asic and found out solution can be done on 
> multiple broadcom and custom Asic and Alibaba network is running on 2 
> different Broadcom Asic.
>
> And your point being? :D
>
> -sam
>
>
> Thanks
> Deepak
>
> Sent from my iPhone
>
> If TTL expiry mechanism is used, then the definition of IP TTL will 
> have to be redefined in order to make a copy and forward to next hop.
> But if L3 devices have to read into VXLAN header to determine OAM bit 
> is set, they need to implement DPI for the same.
>
> Speaking as an operator, as mentioned yesterday, this will cause 
> packet storm and unintended consequences.
>
> Subject: Re: [nvo3] 
> draft--pang--nvo3--vxlan--path--detection--01
>
> required to recognize the vxlan oam packets. If this assumption is 
> broken, the solutions proposed in this draft may not be effective.
>
> Cheers
>
> Dacheng
>
> This draft needs to address how intermediate L3 routers are going to 
> see these VXLAN OAM packets, since L3 routers just do L3 routing and 
> don’t look at the payload to see it is VXLAN and then see that these 
> are PD OAM packets. The only option I can think of is TTL expiry, 
> otherwise it won’t work, the way it is defined now,
>
> Thx
> Shahram

---

From: "Dacheng Zhang" <[email-masked]>
Date: Thu, 05 Nov 2015 09:47:35 +0800

Fully agree. ^_^

在 15-11-5 上午9:46， "Liyizhou" <[email-masked]> 写入:

>RFC7348 VxLAN itself is an informational document. IMO, being
>informational does not necessarily mean no new bit can be defined. The
>document can be improved to clearly state the applicable scenario and
>compatibility considerations. However the historical procedural issue
>should not be the only or major reason to block the progressing of such
>kind of documents. I do not see anything wrong to proceed it with
>informational in this case given there exists a clear requirement and
>reasonable deployment from it.
>
>
>Thanks,
>Yizhou
>
>
>-----邮件原件-----
>VXLAN forwarding process."-- I think this is an assumption based on what
>implementations are probably doing currently as opposed to what is
>specified by the protocol. The VXLAN RFC does not specify a forwarding
>process AFAICT. And, as I mentioned yesterday, VXLAN is not extensible in
>its current definition.  Unknown reserved bits must be ignored on
>reception which means that a receiver that doesn't support PD bit will
>assume that the payload is an Ethernet packet and attempt to deliver it
>as such.
>
>Given that this is not the first time someone has tried to extend VXLAN
>and it's wide deployment, it might be prudent to try to define a method
>to extend the protocol without breaking compatibility. The pseudo packet
>(cannot just be called a header) trick will work if the packet is created
>such that it is guaranteed to be dropped at a receiver that doesn't
>understand the extensions (e.g. put a zero in total length field in case
>of IP). Obviously any method should support more than just one extension.
>This implies that fields must be precisely ordered, so for an
>implementation to parse field, it must be able to calculate the offset of
>the field. So the order that fields are defined in VXLAN becomes
>important and needs to be spelled out.
>instance. The easiest thing to do is define the flag bits left to right,
>where to process the n'th bit an implementation must understand the first
>n-1 bits (see GRE or GUE), but the n+1 bits and beyond can still be
>ignored.
>
>Tom
>
>
>

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 4 Nov 2015 18:13:33 -0800

Just so you know, VXLAN RFC was not WG document, rather independent
submission.
As you are bringing this to NVo3 WG and asking for comments, they are given.
There is nothing wrong in either direction, unless you feel it that way.

-sam

On Wed, Nov 4, 2015 at 5:46 PM, Liyizhou <[email-masked]> wrote:

> "The extra bit in VXLAN reserved field has no side effect on regular VXLAN
> forwarding process."-- I think this is an assumption based on what
> its current definition.  Unknown reserved bits must be ignored on reception
> which means that a receiver that doesn't support PD bit will assume that
> the payload is an Ethernet packet and attempt to deliver it as such.
>
> and it's wide deployment, it might be prudent to try to define a method to
> extend the protocol without breaking compatibility. The pseudo packet
> understand the extensions (e.g. put a zero in total length field in case of
> IP). Obviously any method should support more than just one extension. This
> implies that fields must be precisely ordered, so for an implementation to
> parse field, it must be able to calculate the offset of the field. So the
> order that fields are defined in VXLAN becomes important and needs to be
> spelled out.
> n-1 bits (see GRE or GUE), but the n+1 bits and beyond can still be ignored.
>
> Tom
>
>
>

---

From: Jeff Tantsura <[email-masked]>
Date: Thu, 5 Nov 2015 02:16:29 +0000

Hi,

Going back to more fundamental problem – unpredictability.
It is not uncommon to have 64-128 ways ECMP between source and destination VTEP’s across IP fabric.
In order to exercise every possible path enough entropy (in this case – UDP source port) has to be created by the sending VTEP, while possible – hashing is local to the devices applying it, so there’s high probability that some paths would never be exercised.
Exposing hashing semantics to a controller in a multivendor environment sounds a bit unrealistic.

Cheers,
Jeff

From: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of "Deepak Kumar (dekumar)" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, November 5, 2015 at 09:43
To: Haoweiguo <[email-masked]<mailto:[email-masked]>>, Sam Aldrin <[email-masked]<mailto:[email-masked]>>
Cc: Shahram Davari <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Dacheng Zhang <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Hi,

1.

2.

Thanks,
Deepak

Hi Sam,

Thanks,

weiguo

HI Sam,

Thanks,
Deepak

Sent from my iPhone

Hi Deepak,

-sam

HI Sam,

Thanks
Deepak

Sent from my iPhone

Hi Deepak,

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 4 Nov 2015 18:50:18 -0800

Hi Deepak,

I'd rather have this discussion steered towards the issues at hand. Please
do consider users of VXLAN beyond controlled environment that includes
different administrative domains, multitenant networks, etc.

1. Are you requesting the device behave the way draft describes, when
packet received has PD bit set? I am not asking how it works in your
customer environment, rather, what this is mandating in behavior change.
2. It doesn't matter whether it is enhancing VXLAN or new encaps, if the
packet is received with bit set, has to be punt+replicate, the receiver has
no control over it (as silicon has that feature baked into it). The only
way to control that, as per your replies, is by configuring policies on the
devices. This do not fly as it is changing the default behavior and
receivers of the packet have no idea whether it needs the policy config or
not (complexity in configuring policies is different beast altogether).
3. How do you determine if the dataplane is broken or device doesn't
support? IOW, what is your ask in the draft for capability and support?

These are few to start with.

General questions/observations
1. What is the ask by this draft to WG? as it was repeatedly pointed that
it is informational, is it to allocate code point by reserving this bit ? I
mean, is the draft mandating it? If so, how can it become informational?
2. Why do you want WG to invest time into this, as we moved on have adopted
3 new DP protocols and VXLAN is not one of them?
3. This proposal is not the only way to solve in h/w. So, the question is
not about h/w vs s/w.

thanks
-sam

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Thu, 5 Nov 2015 03:02:35 +0000

Hi Jeff,

There are 2 scenarios

1. vtep - Vtep
2. VM - VM  (Now this scenario has no issues of vtep-vtep algorithm)

Now for 1.

If we didn’t had controller I would have agree with your point.
Datacenter network by east - west traffic design with leaf - spine or leaf, spine and super spine architecture all the paths can be achieved. Anyway controller knows all the path even in complex network and if all the path can’t be achieved newer algorithm can be exercised. If sending packet on full sport range also doesn’t give all the path then there’s definitely something wrong with box ecmp.

In this solution all original packet is sent back to controller so controller and ops guys are even knowing how efficient hashing or load balancing for each switch is working in the network. Also it learns from this distribution and can change the algorithm by sending different vm-vm traffic to cover all the path.

If all else fails we can work on solution like done in George swallow mpls draft to learn the sport by quering each node but that’s outside the scope of this document.

Thanks,
Deepak

From: Jeff Tantsura <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, November 4, 2015 at 6:16 PM
To: dekumar <[email-masked]<mailto:[email-masked]>>, Haoweiguo <[email-masked]<mailto:[email-masked]>>, Sam Aldrin <[email-masked]<mailto:[email-masked]>>
Cc: Shahram Davari <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Dacheng Zhang <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Hi,

Cheers,
Jeff

Hi,

1.

2.

Thanks,
Deepak

Hi Sam,

Thanks,

weiguo

HI Sam,

Thanks,
Deepak

Sent from my iPhone

Hi Deepak,

-sam

HI Sam,

Thanks
Deepak

Sent from my iPhone

Hi Deepak,

Hi Shahram/Sam,

Policy example

-sam

Thanks
Deepak

Sent from my iPhone

-sam

Thx
SD

Cheers

Dacheng

Hi,

Thx
Shahram

---

From: Liyizhou <[email-masked]>
Date: Thu, 5 Nov 2015 04:18:09 +0000

WG do not own the base encapsulation so they can't even work on it. That seems the implication?

Yizhou

发件人: Sam Aldrin [mailto:[email-masked]]
发送时间: 2015年11月5日 10:14
收件人: Liyizhou
抄送: Tom Herbert; Haoweiguo; Shahram Davari; [email-masked]; Dacheng Zhang; Deepak Kumar (dekumar)
主题: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Just so you know, VXLAN RFC was not WG document, rather independent submission.
As you are bringing this to NVo3 WG and asking for comments, they are given.
There is nothing wrong in either direction, unless you feel it that way.

-sam

On Wed, Nov 4, 2015 at 5:46 PM, Liyizhou <[email-masked]<mailto:[email-masked]>> wrote:
RFC7348 VxLAN itself is an informational document. IMO, being informational does not necessarily mean no new bit can be defined. The document can be improved to clearly state the applicable scenario and compatibility considerations. However the historical procedural issue should not be the only or major reason to block the progressing of such kind of documents. I do not see anything wrong to proceed it with informational in this case given there exists a clear requirement and reasonable deployment from it.

Thanks,
Yizhou

-----邮件原件-----
发件人: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] 代表 Tom Herbert
发送时间: 2015年11月5日 6:05
收件人: Haoweiguo
抄送: Sam Aldrin; Shahram Davari; [email-masked]<mailto:[email-masked]>; Dacheng Zhang; Deepak Kumar (dekumar)
主题: Re: [nvo3] draft--pang--nvo3--vxlan--path--detection--01

Tom

> From: Dacheng Zhang [mailto:[email-masked]<mailto:[email-masked]>]
> 发件人: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of Shahram Davari
> <[email-masked]<mailto:[email-masked]>>
> [email-masked]://www.ietf.org/mailman/listinfo/nvo3<http://www.ietf.org/mailman/listinfo/nvo3>

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 4 Nov 2015 20:23:28 -0800

No. It was more in reference to your comment about blocking this document.
 If a draft is brought into WG, it goes through regular process.

> On Nov 4, 2015, at 8:18 PM, Liyizhou <[email-masked]> wrote:
> 
> On Wed, Nov 4, 2015 at 5:46 PM, Liyizhou <[email-masked] <mailto:[email-masked]>> wrote:
> 发件人: nvo3 [mailto:[email-masked] <mailto:[email-masked]>] 代表 Tom Herbert
> 发送时间: 2015年11月5日 6:05
> 收件人: Haoweiguo
> 抄送: Sam Aldrin; Shahram Davari; [email-masked] <mailto:[email-masked]>; Dacheng Zhang; Deepak Kumar (dekumar)
> On Wed, Nov 4, 2015 at 9:31 PM, Haoweiguo <[email-masked] <mailto:[email-masked]>> wrote:
> > Hi Sam,
> >
> > From: Deepak Kumar (dekumar) [[email-masked] <mailto:[email-masked]>]
> > Cc: Shahram Davari; [email-masked] <mailto:[email-masked]>; Dacheng Zhang
> >
> > Subject: Re: [nvo3]
> > On Nov 4, 2015, at 3:09 PM, Sam Aldrin <[email-masked] <mailto:[email-masked]>> wrote:
> >
> > Hi Deepak,
> >
> > On Nov 3, 2015, at 8:03 PM, Deepak Kumar (dekumar) <[email-masked] <mailto:[email-masked]>>
> > wrote:
> >
> > HI Sam,
> >
> > 发件人: nvo3 <[email-masked] <mailto:[email-masked]>> on behalf of Shahram Davari
> > <[email-masked] <mailto:[email-masked]>>
> > [email-masked]://www.ietf.org/mailman/listinfo/nvo3 <http://www.ietf.org/mailman/listinfo/nvo3>