Subject: [Idr] Shortest Path Routing Extensions for BGP Protocol

From: Robert Raszuk <[email-masked]>
Date: Fri, 8 Jul 2016 17:51:56 +0200

Hi,

I have reviewed your proposal.

Turning path vector or distance vector protocol into link state carrier is
no doubt a bold idea :).

Effectively what you are proposing is to use BGP TCP sessions to propagate
link state database creating first "link vector protocol" !

For start I have few questions:

Q1:

RFC7752 has gone via lot of efforts (especially sections 3.2 and up) to
include number of OSPF or ISIS specific encodings. In your proposal you
mentioned OSPF twice and not even once ISIS. Does it mean that you are not
going to use all encoding for specific IGPs as defined in RFC7752 ?

Q2:

Who creates and maintains LSDB in each BGP speaker ? Are you planning to
run OSPF and or ISIS except disable it to establish any adjaciencies ?

Q3:

Currently there are already to models to build DCs with BGP ... one uses
BGP to create only lean underlay the other is to use BGP for both underlay
and tenants (example project Calico for the latter). With that scale wise I
think your proposal will work great for the former. However I do have
concerns about using your model for the latter where say 10,000 or 100,000
/32s or /128s from each VMs are injected and you need to construct SPT with
all of those.

Q4:

Related to Q3 in your model and say flat DC routing each compute node other
then just injecting 10s of /32s and being "done" now becomes an IGP node.
Since your document explicitely targets Massively Scaled Data Centers
(MSDCs) I am concerned that having 100,000+ IGP nodes and in many case much
more is not the best idea.

Q5:

Have you considered just proposing an OSPF route reflector instead without
stuffing BGP into the mix ? As some of you perhaps remember the work on
this started around year 2000 to optimize PE-CE CSC deployments :) It seems
to me very reusable for this goal.

Best regards,
Robert.

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Fri, 8 Jul 2016 16:22:29 +0000

Hi Robert,

Thanks for engaging…

From: <[email-masked]<mailto:[email-masked]>> on behalf of Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Date: Friday, July 8, 2016 at 11:51 AM
To: "Keyur Patel (keyupate)" <[email-masked]<mailto:[email-masked]>>, Acee Lindem <[email-masked]<mailto:[email-masked]>>, "Abhay Roy (akr)" <[email-masked]<mailto:[email-masked]>>, "Derek Man-Kit Yeung (myeung)" <[email-masked]<mailto:[email-masked]>>, "Venu Venugopal (venuv)" <[email-masked]<mailto:[email-masked]>>
Cc: IDR List <[email-masked]<mailto:[email-masked]>>
Subject: Shortest Path Routing Extensions for BGP Protocol

Hi,

Turning path vector or distance vector protocol into link state carrier is no doubt a bold idea :).

“Fortune befriends the bold.” -Emily Dickinson

Effectively what you are proposing is to use BGP TCP sessions to propagate link state database creating first "link vector protocol" !

Q1:

RFC7752 has gone via lot of efforts (especially sections 3.2 and up) to include number of OSPF or ISIS specific encodings. In your proposal you mentioned OSPF twice and not even once ISIS. Does it mean that you are not going to use all encoding for specific IGPs as defined in RFC7752 ?

We are using the BGP Protocol-ID defined for BGP-EPE. The IANA request will be generalized from “BGP-EPE” to “BGP” in support of Segment Routing and other enhancements. The BGP-LS NLRI are specific BGP and not either IS-IS or OSPF.

Q2:

Who creates and maintains LSDB in each BGP speaker ? Are you planning to run OSPF and or ISIS except disable it to establish any adjaciencies ?

I’ve seen designs like this in my time but have never been of a fan of them ;^). BGP will do the SPF directly and maintain the SPT. You’ll note that a simplified SPF is already done for ORR.

Q3:

Currently there are already to models to build DCs with BGP ... one uses BGP to create only lean underlay the other is to use BGP for both underlay and tenants (example project Calico for the latter). With that scale wise I think your proposal will work great for the former. However I do have concerns about using your model for the latter where say 10,000 or 100,000 /32s or /128s from each VMs are injected and you need to construct SPT with all of those.

Similar to those designs, the SPT could be limited to the underlay. However, if there is no requirement for the benefits of L2 or L3 VPNs, I see no reason why these 100Ks of leafs VM prefixes in DC center couldn’t be supported.

Q4:

Related to Q3 in your model and say flat DC routing each compute node other then just injecting 10s of /32s and being "done" now becomes an IGP node. Since your document explicitely targets Massively Scaled Data Centers (MSDCs) I am concerned that having 100,000+ IGP nodes and in many case much more is not the best idea.

100K+ switches in a single DC fabric (i.e., BGP routing domain)? I have some experience in link-state protocols and I can tell you that OSPF is I/O bound mainly due to the flooding. If done right, the SPF calculation can be done with minimal computation. While BGP-LS isn’t the world's most compact encoding, it is completely incremental.

Q5:

Have you considered just proposing an OSPF route reflector instead without stuffing BGP into the mix ? As some of you perhaps remember the work on this started around year 2000 to optimize PE-CE CSC deployments :) It seems to me very reusable for this goal.

We looked at lots of alternatives and this one seemed like the best one. Please pass me a pointer to the work you mention.

Thanks,
Acee

Best regards,
Robert.

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 9 Jul 2016 21:49:12 +0200

Hi AC,

Many thx for responding.

As far as your question 100K+ was about number of compute nodes, virtual
compute nodes in flat routing (enabled via SR-IOV) & switches together as
in your proposal each of them is a node in the SPT. Unless you are planning
to introduce analogy to ospf areas here :)

In any case the most powerful property of BGP is hierarchy and
(multi-level) indirection via next hop. In normal BGP deployments next hops
which are carried in BGP are resolved in IGP. In DC cases where eBGP is
used between all fabric stages next hops resolves to connected routes.

In your case in flat routing you no longer have such recursion.

However I assume your proposal can coexist with other SAFIs and say
external to the DC desitnations will be still carried in 1/1 or 2/1 and
their next hops will be resolved by routes inserted into RIB by running SPF
correct?

Last .. LFA is a great technology however I am not sure if enabling it in
topologies where you have 10s of ECMP active paths and where you are ready
by design for a failure handling is a right thing to do. What are other
real use cases for giving up on pure BGP best path selection and normal BGP
multipath ?

Best regards,
R.

On Fri, Jul 8, 2016 at 6:22 PM, Acee Lindem (acee) <[email-masked]> wrote:

> Hi Robert,
>
> Thanks for engaging…
>
> From: <[email-masked]> on behalf of Robert Raszuk <[email-masked]>
> To: "Keyur Patel (keyupate)" <[email-masked]>, Acee Lindem <
> [email-masked]>, "Abhay Roy (akr)" <[email-masked]>, "Derek Man-Kit Yeung
> (myeung)" <[email-masked]>, "Venu Venugopal (venuv)" <[email-masked]>
> Cc: IDR List <[email-masked]>
> We are using the BGP Protocol-ID defined for BGP-EPE. The IANA request
> will be generalized from “BGP-EPE” to “BGP” in support of Segment Routing
> and other enhancements. The BGP-LS NLRI are specific BGP and not either
> IS-IS or OSPF.
>
>
> Q2:
>
> I’ve seen designs like this in my time but have never been of a fan of
> them ;^). BGP will do the SPF directly and maintain the SPT. You’ll note
> that a simplified SPF is already done for ORR.
>
>
> Q3:
>
> Similar to those designs, the SPT could be limited to the underlay.
> However, if there is no requirement for the benefits of L2 or L3 VPNs, I
> see no reason why these 100Ks of leafs VM prefixes in DC center couldn’t be
> supported.
>
>
> Q4:
>
> Related to Q3 in your model and say flat DC routing each compute node
> other then just injecting 10s of /32s and being "done" now becomes an IGP
> node. Since your document explicitely targets Massively Scaled Data Centers
> 100K+ switches in a single DC fabric (i.e., BGP routing domain)? I have
> some experience in link-state protocols and I can tell you that OSPF is I/O
> bound mainly due to the flooding. If done right, the SPF calculation can be
> done with minimal computation. While BGP-LS isn’t the world's most compact
> encoding, it is completely incremental.
>
>
> Q5:
>
> We looked at lots of alternatives and this one seemed like the best one.
> Please pass me a pointer to the work you mention.
>
> Thanks,
> Acee
>
>
>
> Best regards,
> Robert.
>
>

---

From: "Keyur Patel (keyupate)" <[email-masked]>
Date: Mon, 11 Jul 2016 18:24:17 +0000

Hi Robert,

Comment inlined #Keyur.

From: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Date: Saturday, July 9, 2016 at 12:49 PM
To: "Acee Lindem (acee)" <[email-masked]<mailto:[email-masked]>>
Cc: Keyur Patel <[email-masked]<mailto:[email-masked]>>, "Abhay Roy (akr)" <[email-masked]<mailto:[email-masked]>>, "Derek Man-Kit Yeung (myeung)" <[email-masked]<mailto:[email-masked]>>, "Venu Venugopal (venuv)" <[email-masked]<mailto:[email-masked]>>, idr wg <[email-masked]<mailto:[email-masked]>>
Subject: Re: Shortest Path Routing Extensions for BGP Protocol

Hi AC,

Many thx for responding.

As far as your question 100K+ was about number of compute nodes, virtual compute nodes in flat routing (enabled via SR-IOV) & switches together as in your proposal each of them is a node in the SPT. Unless you are planning to introduce analogy to ospf areas here :)

In any case the most powerful property of BGP is hierarchy and (multi-level) indirection via next hop. In normal BGP deployments next hops which are carried in BGP are resolved in IGP. In DC cases where eBGP is used between all fabric stages next hops resolves to connected routes.

However I assume your proposal can coexist with other SAFIs and say external to the DC desitnations will be still carried in 1/1 or 2/1 and their next hops will be resolved by routes inserted into RIB by running SPF correct?

#Keyur: Yep. Other SAFIs can totally use the draft extensions to recurse their next hops.

Regards,
Keyur

Last .. LFA is a great technology however I am not sure if enabling it in topologies where you have 10s of ECMP active paths and where you are ready by design for a failure handling is a right thing to do. What are other real use cases for giving up on pure BGP best path selection and normal BGP multipath ?

Best regards,
R.

On Fri, Jul 8, 2016 at 6:22 PM, Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Robert,

Thanks for engaging…

Hi,

Q1:

Q2:

Q3:

Q4:

Q5:

Thanks,
Acee

Best regards,
Robert.

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Mon, 11 Jul 2016 22:59:08 +0000

Hi Robert,
See inline.

From: <[email-masked]<mailto:[email-masked]>> on behalf of Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Date: Saturday, July 9, 2016 at 3:49 PM
To: Acee Lindem <[email-masked]<mailto:[email-masked]>>
Cc: "Keyur Patel (keyupate)" <[email-masked]<mailto:[email-masked]>>, "Abhay Roy (akr)" <[email-masked]<mailto:[email-masked]>>, "Derek Man-Kit Yeung (myeung)" <[email-masked]<mailto:[email-masked]>>, "Venu Venugopal (venuv)" <[email-masked]<mailto:[email-masked]>>, IDR List <[email-masked]<mailto:[email-masked]>>
Subject: Re: Shortest Path Routing Extensions for BGP Protocol

Hi AC,

Many thx for responding.

No areas here… If you are running BGP on the 100K+ compute nodes as I believe you are suggesting with the reference to SR-IOV, you would probably not want flat routing. In many of today DCs, the compute nodes are merely hosts on ToR subnets and these prefixes can be easily computed using SPF optimizations.

As Keyur responded, SPF routes could be used for resolution of other SAFIs.

I agree that the value of LFA is limited if you have prevalent ECMP and a data plane that does fast ECMP rehashing in the event of failure. The other inherent advantage of SPF-based computation is that a single failure only results in the directly impacted NLRI being advertised as opposed to all the impacted prefixes having to propagate hop-by-hop. This should improve convergence and even scaling.

Thanks,
Acee

Best regards,
R.

Thanks for engaging…

Hi,

Q1:

Q2:

Q3:

Q4:

Q5:

Thanks,
Acee

Best regards,
Robert.