Subject: [GROW] Review request for draft-szarecki-grow-abstract-nh-scaleout-peering-00

From: Rafal Jan Szarecki <[email-masked]>
Date: Thu, 28 Feb 2019 04:23:07 +0000

GROW-ers,

Here is draft describing abstract Next-Hop concept and it’s use in scale-out peering. I would appreciate reviews and comments.
https://datatracker.ietf.org/doc/draft-szarecki-grow-abstract-nh-scaleout-peering/

--
Rafal Szarecki

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 28 Feb 2019 10:48:11 +0100

Hi Rafał,

It seems to me that all you are doing here is setting next hop self on
ASBRs (pretty well know) + tracking this next hop based on the EBGP
sessions state you are applying it to when advertising intra-domain (say to
your RRs).

Is there anything else new here ?

Question:

How are you going to deal with the case where say 3 paths received from
your peering ASto a given ASBR will contain different AIGP attribute (or
any other attribute for that matter) ? Will you just propagate best path
with ANH and hide the information contained in other two paths to the rest
of your network ?

Kind regards,
R.

On Thu, Feb 28, 2019 at 7:16 AM Rafal Jan Szarecki <rjszarecki=
[email-masked]> wrote:

> GROW-ers,
>
>
>
> Here is draft describing abstract Next-Hop concept and it’s use in
> scale-out peering. I would appreciate reviews and comments.
>
>
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: Rafal Jan Szarecki <[email-masked]>
Date: Thu, 28 Feb 2019 18:42:52 +0000

Robert,

See inline.

Rafal Szarecki JNCIE
Routing Solution Architect
Juniper Networks.
+14086809604

On Feb 28, 2019, at 1:48 AM, Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:

Hi Rafał,

It seems to me that all you are doing here is setting next hop self on ASBRs (pretty well know) + tracking this next hop based on the EBGP sessions state you are applying it to when advertising intra-domain (say to your RRs).

Not sure if I can ack your understanding.
Let me put this way - ABR insert ANH route to IGP conditionally - if session is up and EoR have been received.
Note that multiple session may be tracked for given ANH, wit OR logic.

Question:

How are you going to deal with the case where say 3 paths received from your peering ASto a given ASBR will contain different AIGP attribute (or any other attribute for that matter) ? Will you just propagate best path with ANH and hide the information contained in other two paths to the rest of your network ?

Each ASBR will propagate its best route to on-site RR and then to on-site CR (add path). So CR will run normal selection to pick best. CR shall run multipath (but necessary relaxed one). But if AIGP or other attributes differs, normal selection apply. So CR creates FIB as per best path(s).

Now RR is advertising single best path to other sites.
With another ANH’ (RR change BGP NH). So other sites see one path. It could happen that as result of failure this path need to be updated by other path, but since they have same BGP NH, other site routers do not need to re-program FIB.

Now, update over WAN can take time. During this time previous routing information is used, but since BGP NH == ANH’ it will come to same (set) of CR on originating site. And it will have accurate up-to date FIB to send traffic out vis best ASBR(s) at this time.

If at RR, best and 2nd best path have same attributes, and only diff is BGP NH, since it is set to ANH’, there is no need to send update to other sites.

Kind regards,
R.

On Thu, Feb 28, 2019 at 7:16 AM Rafal Jan Szarecki <rjszarecki=[email-masked]<mailto:[email-masked]>> wrote:
GROW-ers,

Here is draft describing abstract Next-Hop concept and it’s use in scale-out peering. I would appreciate reviews and comments.
https://datatracker.ietf.org/doc/draft-szarecki-grow-abstract-nh-scaleout-peering/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dszarecki-2Dgrow-2Dabstract-2Dnh-2Dscaleout-2Dpeering_&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Hjhzvcy3RXY7GgnXtof0rgOeWXlbs83hVb3_12LdlBA&m=1hGKA2FSZ8xSV4kVP14dhAHrPEiyW0unU1IECnicMVM&s=Tg02rNI9DSW_s8w8SatgtMG8FyAUmPh6Mgt1Dblywrw&e=>

--
Rafal Szarecki
_______________________________________________
GROW mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/grow<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_grow&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Hjhzvcy3RXY7GgnXtof0rgOeWXlbs83hVb3_12LdlBA&m=1hGKA2FSZ8xSV4kVP14dhAHrPEiyW0unU1IECnicMVM&s=JcXO6_VFWVcFcfhlja-7rKIgMuSnb5a_sDWznzas2l8&e=>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 28 Feb 2019 20:41:11 +0100

Rafał,

Let me put this way - ABR insert ANH route to IGP conditionally - if
> session is up and EoR have been received.

OK - we all agree that this conditional injection is not a bad thing. And I
guess you can do it today easily with some implementations (for example
using object tracking).

Each ASBR will propagate its best route to on-site RR

That is precisely the moment when I think we need to seriously consider
consequences.

point #1 - more and more stuff in BGP is opaque to BGP and plays no role in
best path selection. So if someone really needs any of those information he
must not use this solution and that should be spelled out very clearly as
this information will be lost.

point #2 - what is the real problem we are solving ? Full table takes
depending on the implementation of BGP anywhere from 300-450 MB of RAM.
Extra path would be another 150 MB. This is all control plane memory so
pretty cheap and happily fits any x86 box to be placed to act as RR.

Now you made two observations:

-A- I have a weak router which is fine from bw pov but does not have steam
to handle 5M paths - My take is - ok send him 1 or 2 paths from RRs and be
done.

-B- The withdraw of all BGP routes takes soo long - well let's observe that
we can withdraw all routes from a given peer with single BGP message using
techniques as described in
https://tools.ietf.org/html/draft-raszuk-aggr-withdraw-00

Bottom line I think using Abstract Next Hop can help for specific SAFis in
specific topologies to reduce the amount of control plane if that is ever
an issue. Yes dealing with BGP control plane handling is implementation
specific and some code does it more efficiently then other.

Best,
R.

---

From: Rafal Jan Szarecki <[email-masked]>
Date: Fri, 1 Mar 2019 00:45:40 +0000

Robert,

THX for picking it up.
See inline.
--
Rafal Szarecki

That is precisely the moment when I think we need to seriously consider consequences.

point #1 - more and more stuff in BGP is opaque to BGP and plays no role in best path selection. So if someone really needs any of those information he must not use this solution and that should be spelled out very clearly as this information will be lost.

[RJS] Well, basic BGP rule is that only single path per prefix is advertised. Unless you enable ADD-PATH. Right? So Vanilla behavior of ASBR is select one best path and advertise it to RR.  Quite commonly, BGP NH is changed to IP of ASBR’s loopback. In my solution is changed to IP od ANH. I do not see how proposed solution is going to hide information that are otherwise available.

So let’s consider situation when I would like export all path from ASBR to some kind of “Route Controller” for purpose of analytics or EPE. If this controller is not my on-site RR, I can still do this w/o any problem via BMP or using ADD-PATH. Just do not set BGP NH to ANH in export policy.

Finally let assume we want ADD-PATH from ASBR to RR, for whatever reason. In this case indeed BGP NH become important. In proposed solution value of BGP NH (ANH) depends on what session given path was learned. So is eBGP session1 has associated ANH1 nad eBGP session2 has associated ANH2, then ADD-PATH from ASBR to on-site RR will give both path with unique BGP NH values.
Then let assume is eBGP session3 has associated ANH1. In this case indeed only path learned form session1 xor session3 will be advertised to RR (plus path from session2). Yes, some information is suppressed now. Operator can control what could be suppressed by associating same ANH with given set of eBGP session, or not. That is configurable. The corner case will be 1:1 mapping eBGP session to ANH, what is very similar to keeping BGP NH unchanged (peer IP). With notable difference that we can remove ANH from IGP regardless of interface state, base on eBGP session state.

point #2 - what is the real problem we are solving ? Full table takes depending on the implementation of BGP anywhere from 300-450 MB of RAM. Extra path would be another 150 MB. This is all control plane memory so pretty cheap and happily fits any x86 box to be placed to act as RR.

[RJS] I agree. Memory footprint at RR is not an issue. Convergence at scale is.
Let assume at site 1 I have 4 ASBRS connected to AS_2 each with 1 sessions, and this ASBR learns 300k prefixes form AS_2 and all of then are best from each ASBR POV. So 300k path per ASBR, 300k pfx per ASBR, 1 path per prefix per ASBR.
The RR gets 4 x 300k pfx with BGP NH set to ASBR1-2-3-4 loopbacks. And send it w/ ADD-PAT to on-site CR
When eBGP session of one of ASBR (say ASBR1) fails, it has to withdraw 300k path from RR, and RR need to withdraw 300k path form CR. Untill this is done CR will keep sending ¼ oftraffic to ASBR1, and BGP NH == loopback is reachable.
Now if ANH is used, CR sees 4 path per prefix with BGP NH == ANH1-2-3-4 respectively. When eBGP session of one of ASBR (say ASBR1) fails, it removes ANH1 form IGP and start to withdraw 300k path from RR, and RR need to withdraw 300k path form CR. As soon as CR sees IGP update (ANH1 unreachable) it can mark all 300k path that have BGP NH == ANH1 unusable. And stop forwarding to ASBR1. If CR runs BGP PIC EDGE it could be sub-second.

The inter-site operation – advertising only one path w/ BGP NH representing “set of eBGP sessions from set of ASBRS” is just one more optimization. Let call this SP_ANH (Site-Peer ANH in contrast to above discussed ASBR-Peer ANH).

  *   If RR advertise to other sites only one path and BGP NH is loopback of one of ASBRs (or ASBR-Peer ANH), then what is convergence in case of this ASBR failure?  RR has to send 300k path w/ new BGP NH. Until this is done, remote sites will send traffic somewhere elsw. Not best egress site.
  *   If RR advertise to other sites only all 4  path and BGP NH is loopback of one of ASBRs (or ASBR-Peer ANH), then IGP update removing this address will allow for quick restoration (as other 3 path are available everywhere). But in multi-path scenario, we just sreated 2-3 level of ECMP structure on remote BGP speakers:
prefix--> (list of 4 BGP NH) --> each BGP NH --> list of IGP ECMP neighbours. That costly to manage in S/W and in HW
  *   If RR advertise to other sites only one path and BGP NH is Site-Peer ANH as in this proposal, then Site-Peer ANH is not removed form IGP (as other ASBRs has session with Peer). Re,mote sites keep sending traffic using pre-failure data until BGP update from RR comes. End when it comes, it will have same BGP NH as pre-failure path. So there will be no need to update FIB. Also FIB structure will be simpler and less costly
prefix--> one  BGP NH --> list of IGP ECMP neighbours.
Some merchant chips have really limited ECMP capability…

-A- I have a weak router which is fine from bw pov but does not have steam to handle 5M paths - My take is - ok send him 1 or 2 paths from RRs and be done.

-B- The withdraw of all BGP routes takes soo long - well let's observe that we can withdraw all routes from a given peer with single BGP message using techniques as described in https://tools.ietf.org/html/draft-raszuk-aggr-withdraw-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Draszuk-2Daggr-2Dwithdraw-2D00&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Hjhzvcy3RXY7GgnXtof0rgOeWXlbs83hVb3_12LdlBA&m=iJUN0T42tjxfPWsB6SlnCrzunCWTBLFel6FkdLOBPvU&s=zc62mnhXDVusW-Gi9_0csHmhhApgkVDLBhiCprjmzPY&e=>

[RJS] ACK. We can always extend protocol to do something. Or develop new one. I think I also saw proposal of new NLRI  to advertise “NH invalidation”.
This proposal do not requires any change to BGP. So it can interoperate with virtually any implementations.

Bottom line I think using Abstract Next Hop can help for specific SAFis in specific topologies to reduce the amount of control plane if that is ever an issue. Yes dealing with BGP control plane handling is implementation specific and some code does it more efficiently then other.

[RJS] Agree. Draft is focused specific architecture/topology – scale-out peering – where operator want and assume ECMP of egress traffic among N x ASBRs existing at given site.
Not that ANH has no other possible uses, but this draft is about this use case.

Best,
R.