Subject: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

From: "Aijun Wang" <[email-masked]>
Date: Wed, 26 Jan 2022 10:16:37 +0800

Hi, All:

As Peter's example and Acee's suggestions, let's focus on the following
problem to think how to solve it efficiently and reasonably:

Scenario: 100 areas each with 1000 PEs (100K total PEs) with 2 ABRs per area

Problem: Overlay services(BGP or Tunnel) that rely on the IGP needs to be
notified immediately when the remote Peer failed, to assist such overlay
service accomplish fast switchover(how to switchover is out of the
discussion)

Potential Solutions:

   There are now mainly four categories of the solutions, as described below
and their brief analysis:

   Category A: PUA/PULSE. Utilizes the existing IGP mechanism to
transport/flooding the notification message.

   Category B: Detail/Important Prefixes Leaks. Bypass the summary
side-effect for some detailed/important prefixes by leaking/not summarize
them into each area.

   Category C: BGP based solution: Utilize the existing BGP infrastructure
to transport the notification message 

   Category D: OOB Solution. Design some new OOB protocol to transport the
notification message.

Because we are in LSR WG, and people are all IGP experts. After the intense
discussion, can we now focus on the Category A/B?

It is very curious that LSR WG will and should produce some BGP or OOB based
solution. I think they may be feasible, but should be evaluated/discussed by
other WGs.

Or else, I think we can't converge to one standard solution.

>From the POV of the operator, we prefer to the IGP based solution. If there
is no unsolvable concerns, let's accept it. I think there is enough
interests and experts to accomplish this task.

Best Regards

Aijun Wang

China Telecom

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 25 Jan 2022 19:06:04 -0800

Hi Aijun,
I believe that under Option D you can add multihop BFD per RFC 5883. No new
protols needed.

Regards,
Greg

On Tue, Jan 25, 2022, 18:17 Aijun Wang <[email-masked]> wrote:

> Hi, All:
>
>
>
> As Peter’s example and Acee’s suggestions, let’s focus on the following
>    There are now mainly four categories of the solutions, as described
> below and their brief analysis:
>
> Because we are in LSR WG, and people are all IGP experts. After the
> intense discussion, can we now focus on the Category A/B?
>
> It is very curious that LSR WG will and should produce some BGP or OOB
> based solution. I think they may be feasible, but should be
> evaluated/discussed by other WGs.
>
> Or else, I think we can’t converge to one standard solution.
>
>
>
> From the POV of the operator, we prefer to the IGP based solution. If
> there is no unsolvable concerns, let’s accept it. I think there is enough
> _______________________________________________
> Lsr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lsr
>

---

From: "Aijun Wang" <[email-masked]>
Date: Wed, 26 Jan 2022 11:50:11 +0800

Hi, Greg:

Yes. I think so. If we select the “OOB solution“ category, RFC 5883 is one existing option,  and has no new connection states introduced within the network devices.

The reason that we prefer to the IGP solution is that we want just to relieve from the configuration/operational overhead for these “OOB solution” in the mentioned scenarios.

Best Regards

Aijun Wang

China Telecom

From: [email-masked] <[email-masked]> 
Sent: Wednesday, January 26, 2022 11:06 AM
To: Aijun Wang <[email-masked]>
Cc: lsr <[email-masked]>
Subject: Re: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

Hi Aijun,

I believe that under Option D you can add multihop BFD per RFC 5883. No new protols needed.

Regards, 

Greg

On Tue, Jan 25, 2022, 18:17 Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, All:

As Peter’s example and Acee’s suggestions, let’s focus on the following problem to think how to solve it efficiently and reasonably:

Problem: Overlay services(BGP or Tunnel) that rely on the IGP needs to be notified immediately when the remote Peer failed, to assist such overlay service accomplish fast switchover(how to switchover is out of the discussion)

Potential Solutions:

   There are now mainly four categories of the solutions, as described below and their brief analysis:

   Category A: PUA/PULSE. Utilizes the existing IGP mechanism to transport/flooding the notification message.

   Category B: Detail/Important Prefixes Leaks. Bypass the summary side-effect for some detailed/important prefixes by leaking/not summarize them into each area.

   Category C: BGP based solution: Utilize the existing BGP infrastructure to transport the notification message 

   Category D: OOB Solution. Design some new OOB protocol to transport the notification message.

Because we are in LSR WG, and people are all IGP experts. After the intense discussion, can we now focus on the Category A/B?

It is very curious that LSR WG will and should produce some BGP or OOB based solution. I think they may be feasible, but should be evaluated/discussed by other WGs.

>From the POV of the operator, we prefer to the IGP based solution. If there is no unsolvable concerns, let’s accept it. I think there is enough interests and experts to accomplish this task.

Best Regards

Aijun Wang

China Telecom

_______________________________________________
Lsr mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/lsr

---

From: Tony Li <[email-masked]>
Date: Tue, 25 Jan 2022 20:15:57 -0800

Hi all,

I suggest that we all agree to disagree.

Since we cannot make progress, we simply drop all proposals, adopt none of them, and give up.

Tony

> On Jan 25, 2022, at 6:16 PM, Aijun Wang <[email-masked]> wrote:
> 
> Hi, All:
>  
> https://www.ietf.org/mailman/listinfo/lsr <https://www.ietf.org/mailman/listinfo/lsr>

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Wed, 26 Jan 2022 05:11:40 +0000

Greg –

With 100K PE scale, we are talking about 100K BFD sessions/PE and close to 5 million BFD sessions network-wide.

Eliminating one of the options we are discussing is admittedly a small step, but still worthwhile.

However, If you still want to continue to advocate for BFD, I will say no more.

   Les

From: Lsr <[email-masked]> On Behalf Of Greg Mirsky
Sent: Tuesday, January 25, 2022 7:06 PM
To: Aijun Wang <[email-masked]>
Cc: lsr <[email-masked]>
Subject: Re: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

Hi Aijun,

Regards,
Greg

On Tue, Jan 25, 2022, 18:17 Aijun Wang <[email-masked]<mailto:[email-masked]>> wrote:
Hi, All:

Best Regards

Aijun Wang
China Telecom

_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 25 Jan 2022 21:41:05 -0800

Les,
if we assume that this is a real case, it appears to me that it is a rear
one. Thus, I don't see a good enough  reason standardizing something that
solves a rear case when there's already a standard-based solution that
addresses 80%, if not 90% of cases. Yes, it is interesting problem but, in
my opinion, not for an SDO.

Regards,
Greg

On Tue, Jan 25, 2022, 21:11 Les Ginsberg (ginsberg) <[email-masked]>
wrote:

> *From:* Lsr <[email-masked]> *On Behalf Of * Greg Mirsky
> *Sent:* Tuesday, January 25, 2022 7:06 PM
> *To:* Aijun Wang <[email-masked]>
> *Cc:* lsr <[email-masked]>
> *Subject:* Re: [Lsr] How to forward the solutions for "Prefixes
> Unreachable Notification" problem
>
>
>
> Hi Aijun,
>

---

From: Christian Hopps <[email-masked]>
Date: Wed, 26 Jan 2022 17:14:40 -0500

"Les Ginsberg (ginsberg)" <ginsberg=[email-masked]> writes:

> Greg –
>
> With 100K PE scale, we are talking about 100K BFD sessions/PE and
> close to 5 million BFD sessions network-wide.
>

Hang on a sec. :)

We are starting off with this GINORMOUS network with 100,000 PE routers! Why would 5 million sessions of anything over this gigantic network of routers be a reason to disregard it as a solution? (How many total routers are there BTW?)

If you build something gignatic *everything* is going to scale way up. To use an oldie but a goodie: TANSTAAFL.

Thanks,
Chris.

>     As Peter’s example and Acee’s suggestions, let’s focus on the
>     following problem to think how to solve it efficiently and
>     reasonably:
>
>     Problem: Overlay services(BGP or Tunnel) that rely on the IGP
>     needs to be notified immediately when the remote Peer failed, to
>     assist such overlay service accomplish fast switchover(how to
>     switchover is out of the discussion)
>
>     Potential Solutions:
>
>        There are now mainly four categories of the solutions, as
>     described below and their brief analysis:
>
>        Category B: Detail/Important Prefixes Leaks. Bypass the
>     summary side-effect for some detailed/important prefixes by
>     leaking/not summarize them into each area.
>
>        Category C: BGP based solution: Utilize the existing BGP
>     infrastructure to transport the notification message
>
>     Because we are in LSR WG, and people are all IGP experts. After
>     the intense discussion, can we now focus on the Category A/B?
>
>     It is very curious that LSR WG will and should produce some BGP
>     or OOB based solution. I think they may be feasible, but should
>     be evaluated/discussed by other WGs.
>
>     >From the POV of the operator, we prefer to the IGP based
>     solution. If there is no unsolvable concerns, let’s accept it. I
>     think there is enough interests and experts to accomplish this
>     task.
>
>
>
>     Best Regards
>
>
>
>     Aijun Wang
>
>     China Telecom
>
>
>

---

From: Christian Hopps <[email-masked]>
Date: Wed, 26 Jan 2022 17:22:40 -0500

"Aijun Wang" <[email-masked]> writes:

> Hi, Greg:
>
>
>
> Yes. I think so. If we select the “OOB solution“ category, RFC 5883
> is one existing option,  and has no new connection states introduced
> within the network devices.
>
> The reason that we prefer to the IGP solution is that we want just to
> relieve from the configuration/operational overhead for these “OOB
> solution” in the mentioned scenarios.

And here it is again, the "configuration is too hard" reasoning ... is not valid. Configurations are built algorithmically and deployed automatically more recently from service models using databases etc. It's not a good justification for not using something.

Thanks,
Chris.

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Thu, 27 Jan 2022 00:49:00 +0000

Chris -

The scale request comes from real customers. So, it is understandable for you to be "aghast" - but it is a real request.

As far as BFD goes, my opinion is this won’t scale. There is a significant difference between operating sessions which continuously monitor liveness in a full mesh versus using some approach which only triggers network-wide traffic when some topology change is locally detected. There are multiple approaches being discussed which do the latter - but BFD is not one of them.

You can disagree - or - as Greg has done - say we don’t really have to consider this scale. I am not going to try to convince you otherwise.
But if so you aren’t solving the problem we have been asked to solve.

   Les

> -----Original Message-----
> From: Christian Hopps <[email-masked]>
> Sent: Wednesday, January 26, 2022 2:15 PM
> To: Les Ginsberg (ginsberg) <[email-masked]>
> Cc: Greg Mirsky <[email-masked]>; Aijun Wang
> <[email-masked]>; [email-masked]
> We are starting off with this GINORMOUS network with 100,000 PE routers!
> Why would 5 million sessions of anything over this gigantic network of
> routers be a reason to disregard it as a solution? (How many total routers are
> there BTW?)
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 27 Jan 2022 02:06:24 +0100

Hi Les,

Yes you are correct. It is a classic pull vs push model.

Push gives you notification about state. That's it.

Pull gives you much more as it includes e2e elements of the data plane - of
course for a bit higher cost.

I would not disregard any of the above.

We have been having similar discussions in the past - I am sure some of us
still remember MARP proposal from Alvaro and Russ inspired by David Oran's
idea. While it was L2 switch centric I mention it here to highlight how a
cool idea went nowhere ... ref:
https://tools.ietf.org/id/draft-retana-marp-02.txt and how the end to end
pull model overwhelmed it.

At least we could learn from it for the WAN now. Perhaps fully
coincidentally it is also very similar to proposed by Tony pub-sub model.

Thx,
R.

On Thu, Jan 27, 2022 at 1:49 AM Les Ginsberg (ginsberg) <ginsberg=
[email-masked]> wrote:

> Chris -
>
> The scale request comes from real customers. So, it is understandable for
> you to be "aghast" - but it is a real request.
>
> As far as BFD goes, my opinion is this won’t scale. There is a significant
> difference between operating sessions which continuously monitor liveness
> in a full mesh versus using some approach which only triggers network-wide
> traffic when some topology change is locally detected. There are multiple
> approaches being discussed which do the latter - but BFD is not one of them.
>
> You can disagree - or - as Greg has done - say we don’t really have to
> consider this scale. I am not going to try to convince you otherwise.
> > If you build something gignatic *everything* is going to scale way up.
> To use
> > an oldie but a goodie: TANSTAAFL.
> >
> > Thanks,
> > Chris.
> >
> >
> > >
> > >

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 27 Jan 2022 09:09:47 +0800

Hi, Chris:

Keeping asking whether the peer is healthy or not is obviously not efficient let the proxy(ABR) to notify others when the peer is unhealthy. 
Imaging that there may be thousands of such nodes are keeping to ask each other node, and almost all of nodes are healthy in almost all of the time.

Don't you think such approach is so noisy and inefficient?

Best Regards

Aijun Wang
China Telecom

-----Original Message-----
From: [email-masked] <[email-masked]> On Behalf Of Christian Hopps
Sent: Thursday, January 27, 2022 6:23 AM
To: Aijun Wang <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

"Aijun Wang" <[email-masked]> writes:

> Hi, Greg:
>
>
>
> Yes. I think so. If we select the “OOB solution“ category, RFC 5883 is 
> one existing option,  and has no new connection states introduced 

Thanks,
Chris.

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 26 Jan 2022 17:10:27 -0800

Les,
it appears that you bear some misconceptions about multihop BFD that make
you believe that by its nature it cannot scale to the level your customers
need. I'd point out that all objections to using BFD, as I see them can be
grouped as follows:

   - operational cost - configuration and management;
   - extra state information on PEs;
   - extra traffic load.

If my summary is accurate, none of these concerns are related to the BFD
protocol itself. Can we agree? Furthermore, as Chris has noted,
configuration and management of network functions are automatable. In RFC
9127 anyone can find the BFD YANG data model (RFC 9127bis would not change
it beyond recognition) As far as extra traffic is concerned, a single BFD
session that would *guarantee* the detection of the network failure between
two PEs in 3 sec (one BFD Control message per second from each PE) adds
mere 250 bytes (or so) in IPv6 and even less in IPv4 network. As for the
amount of BFD state information and ability of an implementation to handle
it, I call it the Art of Implementation.

I hope I clarified some of the questions you might have regarding the use
of BFD in the scenario we're discussing.

Regards,
Greg

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 27 Jan 2022 09:53:25 +0800

Hi, Greg:

I think PUA/PULSE solution can eliminate all of the your mentioned concerns for BFD based solution. 

And it is reasonable to find the more optimal solution.

Best Regards

Aijun Wang

China Telecom

From: [email-masked] <[email-masked]> 
Sent: Thursday, January 27, 2022 9:10 AM
To: Les Ginsberg (ginsberg) <[email-masked]>
Cc: Christian Hopps <[email-masked]>; Aijun Wang <[email-masked]>; [email-masked]
Subject: Re: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

Les,

it appears that you bear some misconceptions about multihop BFD that make you believe that by its nature it cannot scale to the level your customers need. I'd point out that all objections to using BFD, as I see them can be grouped as follows:

*       operational cost - configuration and management;

*       extra state information on PEs;

*       extra traffic load.

If my summary is accurate, none of these concerns are related to the BFD protocol itself. Can we agree? Furthermore, as Chris has noted, configuration and management of network functions are automatable. In RFC 9127 anyone can find the BFD YANG data model (RFC 9127bis would not change it beyond recognition) As far as extra traffic is concerned, a single BFD session that would guarantee the detection of the network failure between two PEs in 3 sec (one BFD Control message per second from each PE) adds mere 250 bytes (or so) in IPv6 and even less in IPv4 network. As for the amount of BFD state information and ability of an implementation to handle it, I call it the Art of Implementation.

I hope I clarified some of the questions you might have regarding the use of BFD in the scenario we're discussing.

Regards,

Greg

On Wed, Jan 26, 2022 at 4:49 PM Les Ginsberg (ginsberg) <[email-masked] <mailto:[email-masked]> > wrote:

Chris -

   Les

> From: Christian Hopps <[email-masked] <mailto:[email-masked]> >
> To: Les Ginsberg (ginsberg) <[email-masked] <mailto:[email-masked]> >
> Cc: Greg Mirsky <[email-masked] <mailto:[email-masked]> >; Aijun Wang
> <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
> "Les Ginsberg (ginsberg)" <ginsberg=[email-masked] <mailto:[email-masked]> > writes:
> 
> > Greg –
> >
> > From: Lsr <[email-masked] <mailto:[email-masked]> > On Behalf Of Greg Mirsky
> > To: Aijun Wang <[email-masked] <mailto:[email-masked]> >
> > Cc: lsr <[email-masked] <mailto:[email-masked]> >

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 27 Jan 2022 07:13:14 +0100

Right, I also saw req' up to 0.5M nodes in flat IGP core by some customers
thinking they'll have the money & need for such things (though I know only
one company in the world right now that runs anything deployed of this
scale give it 2-3 multiplier of total networking devices including switches
;-). Now, generally IME those are customers that did not work with IGPs
extensively since the beginning and put lots assumptions forward which may
look good on paper but it is not clear whether they can be built with the
current technology available (or maybe ever given we're starting to talk
gin-ormous ;-) amounts of state needing synchronization over what is
basically a broadcast). My dry observation was that we either need to break
up the network (theoretically, with enough areas that should be possible
but @ this IGP scale nothing is obvious ;-) or the other approach is to
build it in a very regular graph and use the properties of the graph to
reduce the amount of state in smart ways so in short not use "traditional"
IGPs really.

Now, because someone wants a huge IGP and lots of scale problems are being
hit, the answer is not to make it go "faster" by increasing flooding & push
more and more stuff into the broadcast, especially with flat addresses. If
that "strategy" would work we would not have IP but the planet would be
bridged L2 ;-)

Sounds abstract until the ugliness in the field overruns you (and the
assertions of "rate limited" and "timer expiring" and such stuff is
basically assumptions which will be by experience invalidated by customers
fiddling knobs or the 'unexpected' meltdown that will expose the one huge
blast radius this all encompasses).

IME at this scale you are really best served by a subscribe/publish
mechanism or to put it differently scoped queries on the network. Or you
limit the domain by e.g. carrying nhops in their own dedicated IBGP
reflected or some such thing (roughly stuff Robert was ruminating on).  And
@ publish/subscribe architectural fork in the road I truly do think that
IGPs role should be limited to exposing the SSAPs (roughly Tony's stuff). A
possible variation on the theme is mp2mp substrate that is a S-PMSI at the
same time but that's probably too far out for people's thinking of today
;-)

The discussion is currently bogged down as couple folks observed, given the
variety of assumptions and experience levels & claims extended happily
(where I often ask myself how they have been confabulated) I don't see much
progress will be made. A good option may be to observe that there is no
ultimate need for IGP to get involved here and things can be solved by an
overlay solution of PEs just fine and with that move on to practical
problems we can agree IGP must deal with (as others did already).

--- tony

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 27 Jan 2022 15:28:57 +0800

Not each information carried within the LSP will be consumed by every Node within the IGP, and the PUA/PULSE message doesn’t trigger the SPF calculation.

What are the melt down effect that you are worrying then?

PUB/SUB model introduces again the connection states to the network devices, especially on the ABR. Is this should be considered seriously at the large scale network?

And, we are discussing the control plane message transport, not the forwarding plane flooding. Then why bridged L2 can be used to replace the IP network?

Best Regards

Aijun Wang

China Telecom

From: [email-masked] <[email-masked]> 
Sent: Thursday, January 27, 2022 2:13 PM
To: Les Ginsberg (ginsberg) <ginsberg=[email-masked]>
Cc: Christian Hopps <[email-masked]>; [email-masked]; Greg Mirsky <[email-masked]>; Aijun Wang <[email-masked]>
Subject: Re: [Lsr] How to forward the solutions for "Prefixes Unreachable Notification" problem

Right, I also saw req' up to 0.5M nodes in flat IGP core by some customers thinking they'll have the money & need for such things (though I know only one company in the world right now that runs anything deployed of this scale give it 2-3 multiplier of total networking devices including switches ;-). Now, generally IME those are customers that did not work with IGPs extensively since the beginning and put lots assumptions forward which may look good on paper but it is not clear whether they can be built with the current technology available (or maybe ever given we're starting to talk gin-ormous ;-) amounts of state needing synchronization over what is basically a broadcast). My dry observation was that we either need to break up the network (theoretically, with enough areas that should be possible but @ this IGP scale nothing is obvious ;-) or the other approach is to build it in a very regular graph and use the properties of the graph to reduce the amount of state in smart ways so in short not use "traditional" IGPs really. 

Now, because someone wants a huge IGP and lots of scale problems are being hit, the answer is not to make it go "faster" by increasing flooding & push more and more stuff into the broadcast, especially with flat addresses. If that "strategy" would work we would not have IP but the planet would be bridged L2 ;-) 

Sounds abstract until the ugliness in the field overruns you (and the assertions of "rate limited" and "timer expiring" and such stuff is basically assumptions which will be by experience invalidated by customers fiddling knobs or the 'unexpected' meltdown that will expose the one huge blast radius this all encompasses). 

IME at this scale you are really best served by a subscribe/publish mechanism or to put it differently scoped queries on the network. Or you limit the domain by e.g. carrying nhops in their own dedicated IBGP reflected or some such thing (roughly stuff Robert was ruminating on).  And @ publish/subscribe architectural fork in the road I truly do think that IGPs role should be limited to exposing the SSAPs (roughly Tony's stuff). A possible variation on the theme is mp2mp substrate that is a S-PMSI at the same time but that's probably too far out for people's thinking of today ;-) 

The discussion is currently bogged down as couple folks observed, given the variety of assumptions and experience levels & claims extended happily (where I often ask myself how they have been confabulated) I don't see much progress will be made. A good option may be to observe that there is no ultimate need for IGP to get involved here and things can be solved by an overlay solution of PEs just fine and with that move on to practical problems we can agree IGP must deal with (as others did already). 

--- tony 

 

On Thu, Jan 27, 2022 at 1:49 AM Les Ginsberg (ginsberg) <ginsberg=[email-masked] <mailto:[email-masked]> > wrote:

Chris -

   Les

---

From: Christian Hopps <[email-masked]>
Date: Thu, 27 Jan 2022 02:10:26 -0500

> The scale request comes from real customers. So, it is understandable for you to be "aghast" - but it is a real request.

I'm not aghast. The exclamation point was just a period scaled to the 100K PE level. :)

BTW, earlier in the thread I asked if every PE actually would need the liveness detection for every other PE (full-mesh) and Peter indicated that this was not the case. So 5M, while not unreasonable for a network of this size, is additionally a pathological case. I think it's important to consider this less-than-full-mesh attribute when talking about this deployment.

Finally, for this model deployment I'll ask again, how many *total* routers are there in this network?

Thanks,
Chris.
[as wg member]

> difference between operating sessions which continuously monitor liveness in a
> full mesh versus using some approach which only triggers network-wide traffic
> when some topology change is locally detected. There are multiple approaches
> being discussed which do the latter - but BFD is not one of them.
>