Subject: Re: Request a WG adoption poll for draft-xu-virtual-subnet-10

From: <[email-masked]>
Date: Mon, 1 Jul 2013 13:13:42 +0000

Hello Xuxiaohu,

Before eventually starting a discussion on the adoption on this 
document, we chairs would like to see more discussion and have the 
feedback from more readers (during the 2 previous meetings, only 5 to 10 
people raised their hand when the room was polled on having read the 
document).

Regards,

-Thomas and Martin,
l3vpn co-chairs

2013-07-01, Xuxiaohu:
> Hi Thomas and Martin,
>
> I haven't received any response from any of you. Hence I resend it.
>
> Best regards,
> Xiaohu
>
>> -----邮件原件-----
>> 发件人: Xuxiaohu
>> 发送时间: 2013年6月20日 15:07
>> 收件人: [email-masked]'; [email-masked]';
>> [email-masked]'
>> 抄送: L3VPN
>> 主题: Request a WG adoption poll for draft-xu-virtual-subnet-10
>>
>> Hi WG co-chairs,
>>
>> As mentioned before, this draft
>> (http://tools.ietf.org/html/draft-xu-virtual-subnet-10) describes how to reuse
>> the proven L3VPN technology to realize subnet extension for data center
>> interconnect, with the aid of many concrete experiments and verifications.
>>
>> We co-authors believe the information contained in this draft would be valuable
>> for cloud service providers to evaluate the feasibility of adopting the proven
>> L3VPN technology for data center interconnection (by the way, some people
>> had shown their interests on the scheme described in this draft when visiting
>> our implementation demo of this scheme at the Bits-N-Bites event of IETF86).
>> Hence we would like co-chairs to start a WG adoption poll for this draft.
>>
>> Best regards,
>> Xiaohu (on behalf of all co-authors)

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 2 Jul 2013 06:52:07 +0000

Hi all,

Till now, it seems that too much attention has been paid to the use of L2VPN for data center interconnect. Although there are still a few clustering applications which rely on non-IP or non-routable IP communications (e.g., heartbeat messages between cluster nodes), more and more cluster vendors are offering clustering applications based on Layer 3 interconnection. In addition, geographical high-availability clusters is not a must in many data center interconnect cases.

As L3VPN has many distinct advantages over L2VPN for data center interconnect such as: scaling forwarding tables of data center switches, path optimization, unknown unicast and ARP suppression... why not try to use the proven L3VPN technology for data center interconnect in the scenario where the limitations as mentioned above don't exist?

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]]
> 发送时间: 2013年7月1日 21:14
> 收件人: Xuxiaohu
> 抄送: [email-masked]; [email-masked];
> L3VPN
> 主题: Re: Request a WG adoption poll for draft-xu-virtual-subnet-10
> 
> Hello Xuxiaohu,
> 
> ________________________________________________________________
> _________________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Tue, 2 Jul 2013 12:06:48 +0000

Hi Xiaohu-

  Let me first say that I agree that L3 is the way things should be built.  But I'm not sure that your draft adds much to existing methods, and it seems to make a number of assumptions.

   "PE routers (i.e.,
   PE-1 and PE-2) which are used for interconnecting these two data
   centers create host routes for their local CE hosts respectively and
   then advertise them via L3VPN signaling.... In fact, such subnet is a virtual      
   subnet which is emulated by using host routes."

So if I have a /24 I will end up with (up to) 253 host addresses in the network?

How does a PE know which hosts exist?  Does it wait until it sees local ARP in order to determine that there is a valid /32?  

What happens if Host A wants to reach Host C, homed off of PE-3, but PE-3 does not know about Host C because it's never participated in ARP?  Do I need to relay the ARP from PE-3 to the other PEs for the VPN?

What scale benefits are there if I have to advertise all the host routes on my local portion of the subnet?  Specifically, can you compare the scale benefits of advertising a bunch of /32s to, say, EVPN, where MAC addresses are basically /48 host routes?

In section 3.6.1:

---
the ingress PE router will forward the packet to the
      RR according to one of the VP routes learnt from the RR, which in
      turn forwards the packet to the relevant egress PE router
      according to the host route learnt from that egress PE router.
---

This makes me nervous, as you've now made a Route Reflector (often a box with lots of CPU and memory but not necessarily much forwarding throughput) into basically a Broadcast/Unknown Multicast server.  

The whole of Section 3.6 feels a little like LISP or NHRP or something.  If this is really what you're after you may want to look at how they approached this problem and show what's different about your solution.

eric

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Xuxiaohu
> Sent: Tuesday, July 02, 2013 2:52 AM
> To: L3VPN
> Cc: [email-masked]
> Subject: [nvo3] Why not use L3VPN for data center interconnect?//re:
> Request a WG adoption poll for draft-xu-virtual-subnet-10
> 
> Hi all,
> 
> Till now, it seems that too much attention has been paid to the use of
> L2VPN for data center interconnect. Although there are still a few
> clustering applications which rely on non-IP or non-routable IP
> communications (e.g., heartbeat messages between cluster nodes), more
> and more cluster vendors are offering clustering applications based on
> Layer 3 interconnection. In addition, geographical high-availability
> clusters is not a must in many data center interconnect cases.
> 
> As L3VPN has many distinct advantages over L2VPN for data center
> interconnect such as: scaling forwarding tables of data center switches,
> path optimization, unknown unicast and ARP suppression... why not try to
> use the proven L3VPN technology for data center interconnect in the
> scenario where the limitations as mentioned above don't exist?
> 
> Best regards,
> Xiaohu
> 
> > -----邮件原件-----
> > >> (http://tools.ietf.org/html/draft-xu-virtual-subnet-10) describes
> how to
> > reuse
> > >> We co-authors believe the information contained in this draft would
> be
> > valuable
> > >> had shown their interests on the scheme described in this draft
> when
> > visiting
> > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce
> > message par erreur, veuillez le signaler
> > Orange decline toute responsabilite si ce message a ete altere,
> deforme ou
> > falsifie. Merci.
> >
> > If you have received this email in error, please notify the sender and
> delete this
> > message and its attachments.
> > As emails may be altered, Orange is not liable for messages that have
> been
> > modified, changed or falsified.
> > Thank you.
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 2 Jul 2013 14:44:26 +0000

> why not try to use the proven L3VPN technology for data center
>interconnect...

We presented draft-fang-l3vpn-data-center-interconnect in Orlando, IETF 86.
http://tools.ietf.org/html/draft-fang-l3vpn-data-center-interconnect-01

Comments from WGs are welcome.

Thanks,
Luyuan

-----Original Message-----
From: Xuxiaohu <[email-masked]>
Date: Tuesday, July 2, 2013 2:52 AM
To: L3VPN <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Why not use L3VPN for data center interconnect?//re: Request a
WG	adoption poll for draft-xu-virtual-subnet-10

>communications (e.g., heartbeat messages between cluster nodes), more and
>more cluster vendors are offering clustering applications based on Layer
>3 interconnection. In addition, geographical high-availability clusters
>is not a must in many data center interconnect cases.
>
>> Orange decline toute responsabilite si ce message a ete altere, deforme
>>ou
>> falsifie. Merci.
>>

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 2 Jul 2013 14:57:09 +0000

> The whole of Section 3.6 feels a little like LISP or NHRP or something.  If this

Very good analogy. NHRP based solution e.g. DMVPN, which is commonly used by enterprises to build site-to-site CE-based overlay (with or without L3VPN inside the enterprise network), fits the bill here (beyond just DCI, strictly speaking) very well.

Having said that, NHRP is not perceived to scale as well as that of BGP.

Cheers,
Rajiv

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Eric Osborne (eosborne)
> Sent: Tuesday, July 02, 2013 8:07 AM
> To: Xuxiaohu; L3VPN
> Cc: [email-masked]
> Subject: RE: [nvo3] Why not use L3VPN for data center interconnect?//re:
>   Let me first say that I agree that L3 is the way things should be built.  But
> I'm not sure that your draft adds much to existing methods, and it seems to
> make a number of assumptions.
> 
>    "PE routers (i.e.,
> How does a PE know which hosts exist?  Does it wait until it sees local ARP
> in order to determine that there is a valid /32?
> 
> What happens if Host A wants to reach Host C, homed off of PE-3, but PE-3
> does not know about Host C because it's never participated in ARP?  Do I
> need to relay the ARP from PE-3 to the other PEs for the VPN?
> 
> What scale benefits are there if I have to advertise all the host routes on my
> local portion of the subnet?  Specifically, can you compare the scale
> benefits of advertising a bunch of /32s to, say, EVPN, where MAC addresses
> are basically /48 host routes?
> 
> 
> 
> In section 3.6.1:
> 
> ---
> This makes me nervous, as you've now made a Route Reflector (often a box
> with lots of CPU and memory but not necessarily much forwarding
> throughput) into basically a Broadcast/Unknown Multicast server.
> 
> The whole of Section 3.6 feels a little like LISP or NHRP or something.  If this
> is really what you're after you may want to look at how they approached this
> problem and show what's different about your solution.
> 
> 
> 
> 
> eric
> 
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Xuxiaohu
> > interconnect such as: scaling forwarding tables of data center
> > switches, path optimization, unknown unicast and ARP suppression...
> > interconnect in the scenario where the limitations as mentioned above
> don't exist?
> >
> > Best regards,
> > Xiaohu
> >
> > > -----邮件原件-----
> > > feedback from more readers (during the 2 previous meetings, only 5
> > > to
> > 10
> > > >> for cloud service providers to evaluate the feasibility of
> > > >> adopting
> > the proven
> > > >> L3VPN technology for data center interconnection (by the way,
> > > >> some
> > people
> > > >> our implementation demo of this scheme at the Bits-N-Bites event
> > > >> of
> > > IETF86).
> > > >> Hence we would like co-chairs to start a WG adoption poll for
> > > >> this
> > draft.
> > > >>
> > > >> Best regards,
> > > confidentielles ou privilegiees et ne doivent donc pas etre
> > > diffuses, exploites ou copies sans autorisation. Si vous avez
> > recu ce
> > > message par erreur, veuillez le signaler a l'expediteur et le
> > > detruire ainsi que les pieces jointes. Les
> > messages
> > > electroniques etant susceptibles d'alteration, Orange decline toute
> > > responsabilite si ce message a ete altere,
> > deforme ou
> > > falsifie. Merci.
> > >
> > > information that may be protected by law; they should not be
> > > distributed, used or copied without authorisation.

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 3 Jul 2013 01:10:47 +0000

Hi Eric,

Thanks for your comments. Please see my reply inline.

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Eric
> Osborne (eosborne)
> 发送时间: 2013年7月2日 20:07
> 收件人: Xuxiaohu; L3VPN
> 抄送: [email-masked]
> 主题: Re: [nvo3] Why not use L3VPN for data center interconnect?//re:

That's great.

> So if I have a /24 I will end up with (up to) 253 host addresses in the network?

Yes.

> How does a PE know which hosts exist?  Does it wait until it sees local ARP in
> order to determine that there is a valid /32?

ARP is just one possible approach for CE host discovery. You can also use other approaches (e.g., an orchestration system) to fill this goal.

> need to relay the ARP from PE-3 to the other PEs for the VPN?

The basic assumption of this draft is that local PE routers could timely discover their own local CE hosts through a given approach. However, the approach you proposed may be worthwhile to be considered in this draft.

> local portion of the subnet?  Specifically, can you compare the scale benefits of
> advertising a bunch of /32s to, say, EVPN, where MAC addresses are basically
> /48 host routes?

Almost the same from the forwarding table size perspective. And both can suppress the unknown uncast flooding. However, from the maturity degree of technology and product and the operation experience perspectives, L3VPN is absolutely a better choice, IMHO.

> with lots of CPU and memory but not necessarily much forwarding throughput)
> into basically a Broadcast/Unknown Multicast server.

As mentioned just after the text you quoted, 

     "... Note that in the case where the RR is
      not available for transferring L3VPN traffic between PE routers
      for some reason (e.g., the RR is implemented on a server, rather
      than a router), the APR function could actually be performed by a
      given PE router other than the RR as long as that PE router has
      installed all host routes belonging to the virtual subnet into its
      FIB."

> The whole of Section 3.6 feels a little like LISP or NHRP or something.  If this is
> really what you're after you may want to look at how they approached this

Your observation is correct. To some extent, LISP can be looked as another type of L3VPN approach. Furthermore, LISP-mobility is another type of host route based subnet extension approach just like the one defined in this draft. It's up to the cloud providers to make a choice between L3VPN and LISP. BTW, if you were a cloud provider, which one would you choose in practice?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 3 Jul 2013 01:13:00 +0000

> Your observation is correct. To some extent, LISP can be looked as another type
> of L3VPN approach. Furthermore, LISP-mobility is another type of host route
> based subnet extension approach just like the one defined in this draft. It's up
> to the cloud providers to make a choice between L3VPN and LISP. BTW, if you
> were a cloud provider, which one would you choose in practice?

s/LISP-mobility/LISP VM-Mobility/

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 3 Jul 2013 01:15:50 +0000

> -----邮件原件-----
> 发件人: Rajiv Asati (rajiva) [mailto:[email-masked]]
> 发送时间: 2013年7月2日 22:57
> 收件人: Eric Osborne (eosborne); Xuxiaohu; L3VPN
> 抄送: [email-masked]
> 主题: RE: [nvo3] Why not use L3VPN for data center interconnect?//re:
> Very good analogy. NHRP based solution e.g. DMVPN, which is commonly used
> by enterprises to build site-to-site CE-based overlay (with or without L3VPN
> inside the enterprise network), fits the bill here (beyond just DCI, strictly
> speaking) very well.
> 

Fully agree.

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 3 Jul 2013 01:26:25 +0000

Hi Rajiv

> 发送时间: 2013年7月2日 23:00
> Hi Xiaohu,
> Which notification message do you allude to in section 3.5?  And how would
> the router receive such a message?
> 
> //
>    3.5. CE Host Mobility
> 
>    During the VM migration process, the PE router to which the moving
>    VM is now attached would create a host route for that CE host upon
>    receiving a notification message of VM attachment while the PE
>    router to which the moving VM was previously attached would withdraw
> //

Gratuitous ARP/ND message is one example. 

Best regards,
Xiaohu

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Wed, 3 Jul 2013 05:24:22 +0000

> > What happens if Host A wants to reach Host C, homed off of PE-3, but
> > PE-3 does not know about Host C because it's never participated in
> > ARP?  Do I need to relay the ARP from PE-3 to the other PEs for the VPN?

I would say - No, since that would equate to flooding layer 2 frames in an L3VPN environment.

The traffic to an unknown host should be eventually dropped (closer to the source point) in any L3 environment, IMO.

Cheers,
Rajiv

> Sent: Tuesday, July 02, 2013 9:11 PM
> To: Eric Osborne (eosborne); L3VPN
> Cc: [email-masked]
> Subject: re: [nvo3] Why not use L3VPN for data center interconnect?//re:
> > I'm not sure that your draft adds much to existing methods, and it
> > seems to make a number of assumptions.
> >
> >    "PE routers (i.e.,
> > How does a PE know which hosts exist?  Does it wait until it sees
> > local ARP in order to determine that there is a valid /32?
> 
> ARP is just one possible approach for CE host discovery. You can also use
> other approaches (e.g., an orchestration system) to fill this goal.
> 
> > What happens if Host A wants to reach Host C, homed off of PE-3, but
> > PE-3 does not know about Host C because it's never participated in
> > ARP?  Do I need to relay the ARP from PE-3 to the other PEs for the VPN?
> 
> The basic assumption of this draft is that local PE routers could timely
> discover their own local CE hosts through a given approach. However, the
> approach you proposed may be worthwhile to be considered in this draft.
> 
> > What scale benefits are there if I have to advertise all the host
> > routes on my local portion of the subnet?  Specifically, can you
> > compare the scale benefits of advertising a bunch of /32s to, say,
> > EVPN, where MAC addresses are basically
> > /48 host routes?
> 
> Almost the same from the forwarding table size perspective. And both can
> suppress the unknown uncast flooding. However, from the maturity degree
> of technology and product and the operation experience perspectives,
> L3VPN is absolutely a better choice, IMHO.
> 
> > In section 3.6.1:
> >
> > ---
> > This makes me nervous, as you've now made a Route Reflector (often a
> > box with lots of CPU and memory but not necessarily much forwarding
> > The whole of Section 3.6 feels a little like LISP or NHRP or
> > something.  If this is really what you're after you may want to look
> > at how they approached this problem and show what's different about
> your solution.
> 
> Your observation is correct. To some extent, LISP can be looked as another
> type of L3VPN approach. Furthermore, LISP-mobility is another type of host
> route based subnet extension approach just like the one defined in this
> draft. It's up to the cloud providers to make a choice between L3VPN and
> LISP. BTW, if you were a cloud provider, which one would you choose in
> practice?
> 
> Best regards,
> Xiaohu
> 
> > eric
> >
> >
> > > Till now, it seems that too much attention has been paid to the use
> > > of L2VPN for data center interconnect. Although there are still a
> > > few clustering applications which rely on non-IP or non-routable IP
> > > communications (e.g., heartbeat messages between cluster nodes),
> > > more and more cluster vendors are offering clustering applications
> > > based on Layer 3 interconnection. In addition, geographical
> > > high-availability clusters is not a must in many data center interconnect
> cases.
> > >
> > > > people raised their hand when the room was polled on having read
> > > > the document).
> > > >
> > > > Regards,
> > > >
> > > > -Thomas and Martin,
> > > > l3vpn co-chairs
> > > >
> > > >
> > > > 2013-07-01, Xuxiaohu:
> > > > > Hi Thomas and Martin,
> > > > >
> > > > >> the proven L3VPN technology to realize subnet extension for
> > > > >> data
> > > center
> > > > >> our implementation demo of this scheme at the Bits-N-Bites
> > > > >> event of
> > > > IETF86).
> > > > electroniques etant susceptibles d'alteration, Orange decline
> > > > toute responsabilite si ce message a ete altere,
> > > deforme ou
> > > > falsifie. Merci.
> > > >

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 15 Jul 2013 09:15:16 +0000

Hi all,

For the convenience of finding this draft, I have submitted this draft to the L3VPN WG (see http://tools.ietf.org/html/draft-xu-l3vpn-virtual-subnet-00). More comments and suggestions are welcome.

By the way, it seems that there has been a implementation of using BGP/L3VPN host routes for subnet extension as proposed in the above draft (see
http://www.networkcomputing.com/next-generation-data-center/commentary/cisco-accelerates-sdn-strategy-with-dyna/240157709#disqus_thread).

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Xuxiaohu
> 发送时间: 2013年7月3日 9:11
> 收件人: Eric Osborne (eosborne); L3VPN
> 抄送: [email-masked]
> in
> ARP is just one possible approach for CE host discovery. You can also use other
> approaches (e.g., an orchestration system) to fill this goal.
> 
> The basic assumption of this draft is that local PE routers could timely discover
> their own local CE hosts through a given approach. However, the approach you
> proposed may be worthwhile to be considered in this draft.
> 
> suppress the unknown uncast flooding. However, from the maturity degree of
> technology and product and the operation experience perspectives, L3VPN is
> absolutely a better choice, IMHO.
> 
> > In section 3.6.1:
> >
> > ---
> throughput)
> is