Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation

From: "Dhananjaya Rao (dhrao)" <[email-masked]>
Date: Wed, 2 Mar 2022 12:56:54 +0000

Hello Jeff, co-chairs,

Thank you for posting this question.

Taking the presented use-case as example, the goal is to ensure that routes via specific originating domain border nodes be made available at an ingress PE to provide it with a choice of forwarding diversity into the egress domain.
 
In the CAR solution, this indicates specific intent and hence is represented by originating routes of different colors. Specifically, and taking the topology in Slide 9 of the attached interim presentation as reference, if ABR 231 and ABR 232 are intended to provide diverse points of entry into the egress domain 3, then there will be two BGP-CAR routes for an egress PE E3, one with Color C1 with ABR 231 as the primary/preferred next-hop, and another with color C2 with ABR 232 as the primary/preferred next-hop. (Note the figure does not illustrate C2)

Accordingly, service routes from the PE that desire this diversity intent will have colors C1 or C2 or both attached, as required.
 
It should be clear that with this design, both routes (E3, C1) and (E3, C2) are made available at any ingress PE in any domain, providing the desired forwarding diversity.
 
In addition to the desired forwarding diversity, the solution also inherently provides the local fast convergence and churn isolation between domains when one of the egress domain ABRs fails, as illustrated in the slide.
 
In the CAR solution, this design is identical regardless of the different scenarios where routes may be originated from the egress PE, or from the domain border nodes.
 
To dispel a related misconception that’s been persistent, there is no need for ADD-PATH at inter-AS EBGP boundaries to convey this originating egress domain border router diversity in CAR, since the color in the NLRI defines a unique BGP route.

This design has another benefit – the presence of colors C1 and C2 in the service route enables automated steering with desired priority/preference between the paths at ingress PEs. Further, user policy is also simpler to specify, just in terms of color.

Since this scenario was initially raised by the BGP-CT proponents as one where the CAR NLRI design has limitations, we feel compelled to point out that the BGP-CT design of using separate RDs for these different originating routes of the same color cannot provide both forwarding diversity and localized fast convergence at the same time.
 
As we have described in the session and on the list, the BGP-CT design with different RDs for routes originated by each ABR (231 and 232) forces an ABR failure to be propagated all the way to ingress PEs in other domains, causing slower traffic re-convergence, as well as exposing the failure churn to all other domains.
 
A workaround has been proposed in a recent update where the RD is stripped at every hop to expose the ABRs IP prefix, but that then loses the claimed forwarding diversity while still propagating failure churn of individual ABRs beyond the local domain.

Even in BGP-CT, the simplest design to address this use-case optimally without caveats will be to use routes with different colors, thereby rendering the presence of RD for this purpose irrelevant.
 
We hope that we have addressed this question satisfactorily.
 
Please let us know if any additional clarifications would be helpful.

Regards,
-Dhananjaya

On 2/18/22, 9:16 PM, "Idr on behalf of Jeffrey Haas" <[email-masked] on behalf of [email-masked]> wrote:

    As promised during the recent IDR interim, we want to start a set of threads
    to the IDR mailing list to clarify points discussed during that interim
    about the BGP routes with color proposals.  These questions are directed to
    the authors of those proposals.  The responses to those questions will be
    used to drive Working Group discussion about the solution space.

    Question 2: Route origination and propagation.

    A point of strong discussion during the recent IDR interim covering BGP
    routes with color was related to where routes within a given proposal were
    originated, and what their expected propagation was within the proposal.

    Two examples during the discussion:
    1. Origination of a route with color at an Autonomous System Border Router
    (ASBR) for a domain containing an endpoint address.

    2. Origination at the Provider Edge device hosting the endpoint.

    There may be other scenarios.

    A route-selection "pinch point" is any place in the BGP topology wherein
    multiple paths for the same NLRI may be received, only one is selected, and
    is propagated.  There is a loss of forwarding-diversity at such points under
    some circumstances.  Mechanisms that permit additional BGP routes to be
    propagated can address such loss of forwarding diversity.

    With regard to propagation through route-selection pinch-points, the interim
    discussion illustrated that there were differences of opinion as to when it
    made sense at an ASBR to use next-hop-self or propagate more than one path
    across the peering session.

    The above inputs to the scenario(s) culminate in an ingress device having
    one or more than one instances of the received path.

    What is the default scenario you expect your proposal to use?

    Are there situations where you may want to have origination or propagation
    behave differently than your default scenario?  If so, how does that work?

    Please illustrate the scenarios with a topology for discussion.

    -- Jeff

    _______________________________________________
    Idr mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/idr

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Thu, 3 Mar 2022 00:49:48 +0000

> To dispel a related misconception that’s been persistent, there is no need for ADD-PATH at inter-AS EBGP boundaries to convey this originating egress domain border router diversity in CAR, since th…

What about when E is an anycast address in the egress domain? Or when a unicast E participates in multiple forwarding slices.

About your point on BGP-CT – it is true that simultaneously a route cannot be given forwarding-diversity and local-convergence, because they are opposite requirements. 😊 But the point to be noted is, BGP CT protocol mechanisms can provide both forwarding-diversity and local-convergence, for different BGP-CT prefixes, as use-case demands. Prefix E1 can get forwarding-diversity and E2 can get local convergence. With CAR, forwarding diversity is not possible.

Please consider the use-case where multiple forwarding-slices may exist within a Color, how a distinct label can be advertised for those slices? Because CAR uses just (E, C prefix) doing such things is not possible.

In CT, because it uses (RD:E prefix and C as attribute) such use-cases are possible.

Thanks
Kaliraj

From: Idr <[email-masked]> on behalf of Dhananjaya Rao (dhrao) <dhrao=[email-masked]>
Date: Wednesday, March 2, 2022 at 4:59 AM
To: Jeffrey Haas <[email-masked]>, [email-masked] <[email-masked]>
Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation
[External Email. Be cautious of content]

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

    -- Jeff

    _______________________________________________
    Idr mailing list
    [email-masked]
    https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!V69e63Jy8eXWI-IgjSk37_MtusvOEaJPN6rmriGp1KrluLR57Xy_MP-CW8z04Tep$<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!V69e63Jy8eXWI-IgjSk37_MtusvOEaJPN6rmriGp1KrluLR57Xy_MP-CW8z04Tep$>

Juniper Business Use Only

---

From: "Dhananjaya Rao (dhrao)" <[email-masked]>
Date: Mon, 7 Mar 2022 13:28:36 +0000

It is unfortunate that with BGP-CT, an operator has to make a choice between forwarding diversity and faster convergence/failure churn localization.

We’ve already described below how BGP CAR provides both necessary diversity and fast convergence/churn avoidance by using color.

Regarding anycast, for a normal anycast address, regular routing will select the closest endpoint so there isn’t anything to be done.

I suppose you’re referring to a scenario where one may want paths to multiple anycast endpoints be visible at an ingress PE. Again, the simplest way to achieve that is to assign a different color to each anycast endpoint (or egress domain as the case may be). This design is being used in deployments today.

It’s not clear what the references to the forwarding slices mean.

In general, should a clear need for some other variant or requirement arise, the CAR SAFI is designed to be extensible enough to accommodate it.

Regards,
-Dhananjaya

From: Kaliraj Vairavakkalai <kaliraj=[email-masked]>
Date: Thursday, March 3, 2022 at 6:20 AM
To: "Dhananjaya Rao (dhrao)" <[email-masked]>, Jeffrey Haas <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation

Thanks
Kaliraj

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

    -- Jeff

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Tue, 8 Mar 2022 00:12:44 +0000

Multiple Forwarding-slices within a Color: think of it as simply a use-case that requires advertising multiple distinct labels for the same “EP, C”. One use-case where this is helpful is “Per ingress-node/ASBR traffic statistics”.

BGP-CT can advertise different labels L1, L2 for “RD1:EP1” and “RD2:EP1” resp. until the ingress/BN where such distinction is required. E.g. an Egress-PE may want to gather traffic-stats per ingress ASBR in it’s domain.

With CAR, “EP:C” does not provide any way to do so. From your statement, I see you don’t consider this a ‘necessary diversity’.

Also, this deficiency forces customers to use a different color for the anycast-cases for better network-wide load-balancing, like you note below.

So in summary: CT supports all the variations supported by CAR, and additional modes that CAR doesn’t support (yet). And yes, customers may need to choose (e.g. policy config) which mode they need for a use-case. I don’t see why you see it as unfortunate. 😊

Right, CAR may have to introduce a distinguisher like RD for such use-cases in future. Consider, SRTE has a ‘distinguisher’ field after-all.

Thanks
Kaliraj
From: Dhananjaya Rao (dhrao) <dhrao=[email-masked]>
Date: Monday, March 7, 2022 at 5:29 AM
To: Kaliraj Vairavakkalai <[email-masked]>, Jeffrey Haas <[email-masked]>, [email-masked] <[email-masked]>
Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation
[External Email. Be cautious of content]

Regards,
-Dhananjaya

Thanks
Kaliraj

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

    -- Jeff

---

From: <[email-masked]>
Date: Tue, 8 Mar 2022 10:23:08 +0800 (CST)

Hi Kaliraj,

Sorry to interrupt.

As you said "BGP-CT can advertise different labels L1, L2 for “RD1:EP1” and “RD2:EP1” resp."

combined with the following description from BGP-CT document:
"
 The IP prefix in the
      transport RIB context (IP-prefix, Transport-Class) is used as the
      key to do per-prefix label allocation. 
"

That means for the discussion case we should also use different Transport-Class for the same prefix. Right ?

Regards,
PSF

------------------原始邮件------------------
发件人：KalirajVairavakkalai
收件人：Dhananjaya Rao (dhrao);Jeffrey Haas;[email-masked];
日 期 ：2022年03月08日 08:13
主 题 ：Re: [Idr] BGP routes with color, Question 2: Route origination and propagation
_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

Thanks
Kaliraj

Regards,
-Dhananjaya

Thanks
Kaliraj

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

-- Jeff

_______________________________________________
Idr mailing list
[email-masked]
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!V69e63Jy8eXWI-IgjSk37_MtusvOEaJPN6rmriGp1KrluLR57Xy_MP-CW8z04Tep$

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Tue, 8 Mar 2022 04:53:53 +0000

Hi Peng Shaofu,

These are following label allocation modes that can be used in BGP-CT.

  1.  Per prefix. Where RD1:EP1 and RD2:EP1 get distinct labels L1, L2, for same color C1
  2.  Per transport-class prefix. Where RD1:EP1 and RD2:EP1 get same label L1, for a color C1.
  3.  Per nexthop label-alocation mode. Where label allocation is based on nexthops.

Most of these are similar to BGP-LU, nothing new. Mode (b) is a new mode introduced by BGP-CT.

The draft suggests mode (b) as the default mode, as you quote below.

> As you said "BGP-CT can advertise different labels L1, L2 for “RD1:EP1” and “RD2:EP1” resp."

This statement talks about mode (a).

Either modes (a) or (c) can be enabled by explicit policy config.

I will take your and Dhananjay’s input and clarify these details in the draft, so it is clearer to understand.

Thanks
Kaliraj

From: [email-masked] <[email-masked]>
Date: Monday, March 7, 2022 at 6:23 PM
To: Kaliraj Vairavakkalai <[email-masked]>
Cc: [email-masked] <[email-masked]>, [email-masked] <[email-masked]>, [email-masked] <[email-masked]>
Subject: Re:[Idr] BGP routes with color, Question 2: Route origination and propagation
[External Email. Be cautious of content]

Hi Kaliraj,

Sorry to interrupt.

Regards,
PSF

------------------原始邮件------------------
发件人：KalirajVairavakkalai
收件人：Dhananjaya Rao (dhrao);Jeffrey Haas;[email-masked];
日 期 ：2022年03月08日 08:13
主 题 ：Re: [Idr] BGP routes with color, Question 2: Route origination and propagation
_______________________________________________
Idr mailing list
[email-masked]
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!UOp3a1ALtCMQ-AqyiQ8HbD9gtgYIw_euKdp0fDzd5xHjJXGbxUDPQPJXLxARpWmn$<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!UOp3a1ALtCMQ-AqyiQ8HbD9gtgYIw_euKdp0fDzd5xHjJXGbxUDPQPJXLxARpWmn$>

Thanks
Kaliraj

Regards,
-Dhananjaya

Thanks
Kaliraj

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

-- Jeff

---

From: "Dhananjaya Rao (dhrao)" <[email-masked]>
Date: Fri, 18 Mar 2022 00:26:24 +0000

An observation on examples such as this – regardless of solution, there is a cost incurred both in control and data plane in terms of route and forwarding (label) entries, so I would expect an operator to be judicious in deploying such scenarios. Also, a side note that I don’t think referring to this as a forwarding slice is accurate.

But regardless, for the sake of this discussion, BGP CAR can easily address such a requirement by use of a distinct color for each such distinct domain entry point. Once again, the necessary forwarding diversity is achieved by use of a single construct (color). It also simplifies related configuration or policies for a user.

Color is a 32-bit space hence provides wide flexibility for such use-cases.

There is certainly no need having to resort to overloading semantics of VPN RDs that are opaque.

There’s the additional operational burden on the operator. In your given example, when the routes originate from a PE, since your intent is to pin each route to traverse via a specific ingress border node, if both routes have the same color, then the user needs to use the route’s RD to configure the appropriate policies, which changes for every PE. Or introduce additional attributes such as a community. It can be done, but the additional constructs are simply unnecessary.

But in the end, it is important to reiterate the limitation in the CT design that you’re trying to obscure as a user choice.

The use of RDs fails to provide fast localized convergence upon an ingress node failure for a very basic deployment scenario, and propagates the failure churn all the way to ingress PEs in all domains. This is not a matter of choice, but a critical flaw.

The workaround you’ve documented in the draft of stripping the RDs at each hop is a major hack, that by it’s definition merges the LSP in the dataplane (and hence loses the forwarding diversity you were claiming to provide), but still propagates both individual routes in the control plane so fails to suppress the churn from other domains.

Not to mention the unnecessary overhead of VPN import procedures on every underlay BGP hop.

So, there’s only pain without any gain by using RDs for the transport routes.

Regards,
-Dhananjaya

From: Kaliraj Vairavakkalai <kaliraj=[email-masked]>
Date: Tuesday, March 8, 2022 at 5:43 AM
To: "Dhananjaya Rao (dhrao)" <[email-masked]>, Jeffrey Haas <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation

Thanks
Kaliraj

Regards,
-Dhananjaya

Thanks
Kaliraj

Hello Jeff, co-chairs,

Regards,
-Dhananjaya

    -- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 19 Mar 2022 17:46:35 -0400

Kaliraj,

On Sat, Feb 19, 2022 at 01:26:26AM +0000, Kaliraj Vairavakkalai wrote:
> The subsequent text in draft (sections 10, 11) explain propagation of BGP-CT routes thru ASBRs, RRs. Some key points listed below:
> 
> ·         Regular BGP procedures for route advertisement is followed.
> ·         RD is used to pass thru path selection pinch points, like RR.
> ·         IBGP-Addpath is also used at RR, to not hide redundant BNs (transport ASBRs/ABRs).
> ·         RFC 8212 is followed, preventing unintended propagation across EBGP boundaries.
> ·         Rules of RFC4684 (RTC) work with BGP-CT routes also. Thus providing “on demand nexthop”.

Section 10 has the following text:
:    Border node readvertising Classful Transport route with nexthop self:
: [...]
:       The label SHOULD be allocated with "per-prefix" label allocation
:       semantics.  RD is stripped from the BGP CT NLRI prefix when a BGP
:       CT route is leaked to a Transport RIB.  The IP prefix in the
:       transport RIB context (IP-prefix, Transport-Class) is used as the
:       key to do per-prefix label allocation.  This helps in avoiding BGP
:       CT route churn through out the CT network when a failure happens
:       in a domain.  The failure is not propagated further than the BN
:       closest to the failure.

In the above text, "RD is stripped" - does this mean that a new RD is
allocated for the route for propagation?  Or does the -CT route keep the
same NLRI but the label selection procedure simply proceeds in the context
of the transport RIB in question?

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 19 Mar 2022 17:50:40 -0400

Dhananjaya,

The following is from two different message, but seems to be a related
point:

> We’ve already described below how BGP CAR provides both necessary diversity and fast convergence/churn avoidance by using color.
[...]

> There is certainly no need having to resort to overloading semantics of VPN RDs that are opaque.

Reading the back and forth between you and Kaliraj, would it be fair to
summarize for -CAR, "If you need forwarding diversity, diversity comes from
multiple colors"?

-- Jeff

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Sun, 20 Mar 2022 07:12:31 +0000

Hi Jeff,

There is no change in RD.

The CT route keeps the same NLRI but the label selection procedure simply proceeds in the context
of the transport RIB in question.

Thanks
Kaliraj
From: Jeffrey Haas <[email-masked]>
Date: Saturday, March 19, 2022 at 2:46 PM
To: Kaliraj Vairavakkalai <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Idr] BGP routes with color, Question 2: Route origination and propagation
[External Email. Be cautious of content]

Kaliraj,

-- Jeff

---

From: "Dhananjaya Rao (dhrao)" <[email-masked]>
Date: Mon, 21 Mar 2022 10:01:44 +0000

Hi Jeff,

Not sure about possible interpretations of “forwarding diversity”. But as the example use-cases indicated a specific intent for forwarded traffic, color would be the way to provide the distinction for the corresponding routes.

Regards,
-Dhananjaya

On 3/20/22, 3:21 AM, "Jeffrey Haas" <[email-masked]> wrote:

    Dhananjaya,

    [...]

    -- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 21 Mar 2022 09:12:39 -0400

Dhananjaya,

> Not sure about possible interpretations of “forwarding diversity”. But as the example use-cases indicated a specific intent for forwarded traffic, color would be the way to provide the distinction…

I believe we're in accordance.  For clarity, my meaning for "forwarding
diversity" in this context is situations where forwarding properties such as
BGP Nexthops, labels, SIDS, etc. are desired to be carried through to the
ingress.  If, for a given endpoint, one desires such behavior to be carried
to the ingress, it's a different "intent" from a -CAR perspective, and thus
would be a different color.

-- Jeff