Subject: Re: [Rift] RIFT

From: Antoni Przygienda <[email-masked]>
Date: Fri, 12 Apr 2019 17:44:40 +0000

Hey Kris, great to see you engaging back ;-) I cc: rift mailing list for posterity

1) yepp, very nice python implementation, especially if one wants to understand RIFT as running protocol rather than paper spec ;-)
2) Yepp, multi-planes did lead to lots of discussions in core-team meetings around acceptable solutions and how we'd explain it properly. Explanation largely due to Pascal's and Ilya's work, it takes a bit to soak the ASCII pictures but once you start to grok the concept of "crossbars crossbaring crossbars" being Clos ;-) it's very easy to think through the stuff.
3) A knob to basically "always disaggregate southbound" is  as simple thing to do really. Just like Bruno has it in his computation first phase mostly decides _which_ nodes need disaggregation. The result can be simply replaced by all southbound nodes & then disaggregation happens naturally. Observe that you still want the default origination since a PoD doesn't see other PoDs except via spine and disaggregation is _not_ transitive (I'm talking positive now). There are other cases you want to advertise southobuond some prefixes beside default and it's a normal thing to do, nothing says that you only advertise default southbound anywhere.
4) Observe that we do NOT have a single ring! A ring is only as long as the #planes you have. No'one will have 1000 planes ;-) So let's say you have 64 switches in each plane and 4 planes. You will have 64 rings of lengh 4. Obviously you can double-ring or ring within the plane as well to improve reliability but basically the topology is coherent until 2! links in the same ring break.
5) Always enabling disaggregation: That's point 3. Observe that does NOT solve your multi-plane problem on breakages since positive disaggregation is NOT transitive. Yes, you can turn southbound PGP on and blast whole fabric with all prefixes which basically makes your blast radius uncontained on any change (kind of flat IGP or rather IGP with DV prefixes ;-) and a single link coming/going may lead to massive amount of convergence traffic due to prefix reachability changes. Moreover, all your leafs (which is servers in extreme case) need FIB size of the size of fabric host routes ...

Spec authorship is still open ;-) so if you feel like improving/adding to draft, just let me know your moniker on bitbucket since thath's where the newest spec versions live ...

https://bitbucket.org/riftrfc/rift_draft/src/master/

--- tony

________________________________
From: Kris Price <[email-masked]>
Sent: Friday, April 12, 2019 10:22 AM
To: Antoni Przygienda; [email-masked]
Subject: RIFT

Hey guys,

... couple of thoughts for you:

I see someone much better at explaining things than me talked to you since we last spoke :-) so you've caveated the case with how middle tiers in a Clos don't get full visibility of all nodes at that tier via the reflection because by very nature of the Clos they're not fully striped to that lower tier (that is the essence of how Clos topologies scale so is present in any large fabric). You've covered that under the description as "hyper planes" and work around is to use a ring at those tiers to pass control plane traffic.

Would it be possible to somehow /safely/ have a 'knob' to turn off the aggregation at a tier so it'll always advertise all prefixes southbound? Particularly this is useful in a network that might migrate to this protocol that does not want to go back and cable a ring. And in some cases cabling a ring will be undesirable to some potential users anyway. (In some real world networks that could mean cabling a ring connecting >1000 switches that form the "spine columns". That’s a long brittle ring.)

Larger question, would it be possible to disable aggregation network wide? (For someone that might be interested in using RIFT, but for reasons other than the aggregation capability and where that capability may be seen as undesirable.)

There's a semi-related issue at the very bottom of the fabric but it's a bit difficult to explain, and I'm not sure it's really RIFTs problem to solve, (in part really its due to a bad network design choice that exists) so I might draw that up later.

Cheers
Kris

---

From: Bruno Rijsman <[email-masked]>
Date: Wed, 17 Apr 2019 11:47:10 -0300

Hi Kris,

Thank you very much for your pull request; I merged it into the main repo.  I am very happy to see you actively contributing code!

I added the request for a “don’t aggregate south-bound knob” to my (internal) TODO list.  It will be a while before I get to it, because I would like to finish the base implementation (security envelope, east-west rings, etc.) first.

— Bruno

> On Apr 12, 2019, at 2:44 PM, Antoni Przygienda <[email-masked]> wrote:
> 
> https://bitbucket.org/riftrfc/rift_draft/src/master/ <https://bitbucket.org/riftrfc/rift_draft/src/master/>
> 
> --- tony 
> 
>

---

From: Kris Price <[email-masked]>
Date: Thu, 18 Apr 2019 13:28:14 -0400

Hey Tony,

On the rings: Ahh! I get it, okay that makes it better. I was also
wondering if some kind of designated 'S-TIE' reflector / virtual links
/ or explicitly configured multi-hop adjacencies solution could be
used (the issue being one of how do you route these packets between
the peers without needing to do something like source route multiple
hops southbound before being default routed northbound).

Back on the subject of disaggregation:

The other reason for asking for the always disaggregate option is to
prevent the transient congestion that can occur on link failures. But
I do see now on rereading the draft you've called this out in the
second to last paragraph of 5.2.5.1., but it's left it as an
implementation specific problem to solve.

It seems this would arise frequently at the bottom two tiers of the
network. Any loss of any single link to any rack (tier 0) would result
in all other nodes at tier 1 disaggregating the prefix(es) for that
rack and causing the potential transient incast-like congestion. I'm a
bit concerned that this may be a noticeable event in some cases (e.g.
a storage row/cluster or maybe where RoCE is in use), and one that
would be fairly annoying to debug and remedy post transition to RIFT
if you didn't foresee it and have the tools (knobs) in place to
prevent it from happening without a PR and s/w upgrade. Should
implementations have a conscious solution in advance for this, and
what's the best way to ensure that? The 'always-disaggregate' knob is
one. Another might be something like a 'min-next-hops' option where
the local RIFT instance on tier 0 won't install a prefix unless it has
received it from a minimum number of up streams.

Both of these do run counter to the low-configuration nature of RIFT.
Another might be a protocol change, something like nodes
disaggregating prefixes by default until they know they are more than
1 hop from the bottom of fabric? (This may run into other convergence
issues during fabric bring up and cold start and maybe there are other
issues with it that need doodling out.)

/2c

Cheers :-)
Kris

---

From: Antoni Przygienda <[email-masked]>
Date: Thu, 18 Apr 2019 18:14:09 +0000

Hey, Kris, inline

________________________________
From: Kris Price <[email-masked]>
Sent: Thursday, April 18, 2019 10:28 AM
To: Antoni Przygienda
Cc: [email-masked]; [email-masked]
Subject: Re: RIFT

Hey Tony,

good, I know it takes bit to grok the stuff. We did the best we could with ASCII and language but the concepts need some chewing for sure, even if you have been around big fabrics for a bit ;-) So, nothing like route reflectors and so on, within a plane normal south reflection takes care of sync'ing up all you need, outside the plane the ring takes care of sync'ing up planes (for flooding horizontal links below ToF are south and @ ToF level north basically and with that you have all the topology to figure out negative disaggregation.  I explicitly killed any "virtual link" suggestions, I went through this particular hell in my life more than once and don't want to visit it anymore ;-) ...

well, yes, no free lunch, either you gum up your fabric with all stuff and suffer large blast radius or you dig the beauty of having minimum blast radius and minimal topology info everywhere but on massive failures stuff needs to be sloshed around so e'one has enough info to not blackhole. Finely enough, today's networks, especially fabrics, allow insane flooding rates without breaking half a sweat (first thing I played with when thinking about RIFT design ;-) and I learned here some lessons from looking @ p2p networks BTW. If you run my free package you'll see easily convergence rate of 7-10+K TIEs in the database per second and that's the "usable rate" in the sense that there is much more flooding on the links and it's the "best TIEs in LSDB" rate already. UDP is really quite phenomenal and with a bit of additional help (look @ the packet number & the "you flood too fast" indications) you can dynamically adjust flooding to walk the edge of having losses. Brave new world ...

prevent it from happening without a PR and s/w upgrade.

yepp, you call the spade but you're a bit too pesimistic me thinks. Let's assume 2 ToRs dual-homing a rack or couple racks of servers. if you loose a link in a multi-homed server you basically end up having the other ToR de-aggregating just this server prefix to other servers (even if you run some kubernetes @ scale you may have 100 prefixes or so I'd say, I can't imagine a server hosting thousands really) ... Then, if you think about the ToRs on top of PoD then it's not as bad as you think. If you loose a single ToR in a PoD towards a spine (I'm loose with terminology here) then you will NOT see disaggregation as long the other ToRs in the PoD are still connected to the PoD. Draw pictures & run the public consumption package ;-)  More interesting discussions are bandwdith balancing on link losses (which I think we solved well northbound) and whether it even should be done southbound since notion of "available bandwidth southbound" is confounding ... Spec doesn't forbid it (the beauty of loop-free valley-free routing that gives you insane amount of lee-way how you choose to forward) BTW if somone is smart enough to figure that out ;-) ...

Should
received it from a minimum number of up streams

The always disaggregate knob is something you can do per level if you desire but it's basically a big hammer buying you much bigger blast radius in normal operation. And if you pull RIFT onto servers in multi-plane fabrics your FIB may blow up if you do that (unless we think server adapters with 2M FIB size, probably ain't gonna happen ;-).

The other idea I don't grok, you have to explain in more detail.

Yeah, doodle, I'm not concerned about the convergence and interested in your ideas. ZTP has no beef with prefixes, will work irregardless. So far I saw no indications that size of the fabric up to any reasonable bound of nodes will prevent it to cold-boot properly. ZTP FSM has no timers for that reason BTW (another no, no I put down there ;-) and flooding only has a single retransmit timer.

And, yes, the more stuff like forced disaggregation you start to fiddle with, the more you loose ZTP (depending whether you want it or not, RIFT will work either way). BTW, same with security, the more security you desire the less ZTP you get ;-)  Nature of the beast ...

--- tony

---

From: Kris Price <[email-masked]>
Date: Thu, 18 Apr 2019 16:07:08 -0400

Hey Tony, inline:

[snip]
> good, I know it takes bit to grok the stuff. We did the best we could with ASCII and language but the concepts need some chewing for sure, even if you have been around big fabrics for a bit ;-) So,…

[KP]: I'm a bit skeptical of buy in to rings as a solution, but if you
have customer's buying into that then that's cool. (I omitted
describing the physical *shudder* when I wrote "virtual links" ;-))

[snip]
> yepp, you call the spade but you're a bit too pesimistic me thinks. Let's assume 2 ToRs dual-homing a rack or couple racks of servers. if you loose a link in a multi-homed server you basically end…

[KP]: You're not the first to describe me as a pessimist. :-) I don't
follow the 2x ToRs and multi-homed servers part, I haven't seen that
used in a very long time, and granted I've been out of the game for a
bit but is anyone still multihoming servers at scale? Maybe certain
enterprise use cases, but they're not pushing the boundaries of scale
so don't need the aggregation anyway.

[KP]: A top of rack switch ("tier-1" lets say) may be connected to 8
or 16 switches (or more) northbound (naturally let's call that next
tier "tier-2"). If any single link between a tier-1 and tier-2 switch
goes down (let's say between tier-1-1 and tier-2-1), all other nodes
in tier-2 (that is tier-2-2, tier-2-3, to tier-2-n) will determine
that tier-2-1 no longer has southbound reachability for tier-1-1's
prefixes and that they each need to disagregate these to prevent
tier-1-2..n from sending any traffic for tier-1-1 via tier-2-1 (which
would then need to forward up to tier-3 and back down).

[KP]: With positive disagregation we can introduce transient
congestion if there's a lot of traffic from tier-1-2..n to tier-1-1
because a switch may get the prefix from one upstream node first and
install that before getting it from the remaining upstream nodes. (So
we could for a brief instant go from 8 paths ECMP to 1 path then back
up to 7 paths.) On the other hand if all prefixes are disaggregated,
and when the link between tier-1-1 and tier-2-1 fails, tier-2-1 is now
only announcing a withdraw for the affected prefixes to tier-1-2..n,
we can avoid generating this temporary incast-like scenario by design.

[KP]: It's a preference for more deterministic behavior of the fabric
over less deterministic behavior.

> The always disaggregate knob is something you can do per level if you desire but it's basically a big hammer buying you much bigger blast radius in normal operation. And if you pull RIFT onto serve…

[KP]: Blast radius doesn't seem bigger to me. FIB explosion is a
design consideration for anyone before thinking about routes from
servers. At a small number selectively it's fine and practiced, e.g.
advertising prefixes from servers that are doing software load
balancing.

[KP]: But advertising them say for every VM so you can move VMs
anywhere... that's still going to have impacts on your network design.
With FIB sizes as they are these days most people below the top 5 (or
so) are going to be fine. And anyone in the top 5 (or so) are still
going to be running into trouble. And if you do something like use the
same switch for top of rack layer as at further layers in the Clos,
then this RIFT scaling feature doesn't apply to you anyway as you have
the same FIB size at all tiers.

[KP]: In any case disaggregation at the bottom two tiers where this is
much more likely to be a problem, still permits aggregation higher.

> The other idea I don't grok, you have to explain in more detail.

[KP]: As an alternative to disaggregation and announcing a withdraw
when the link between tier-1-1 and tier-2-1 goes down. It could be
that we have all the RIFT instances on tier-1 configured to know that
they should not install a prefix *unless* they have seen it advertised
from some minimum number of tier-2 nodes. E.g. if there are 8 tier-2
nodes, we might set that to say 4. Now we somewhat avoid the incast
scenario where the switch installs the disaggregated prefix with one
next hop into it's FIB. Instead it'll wait until it has a minimum of
four next-hops. (This is spit balling, it may open other problems.)

Cheers :-)
Kris

---

From: Bruno Rijsman <[email-masked]>
Date: Thu, 18 Apr 2019 19:10:18 -0300

Kris,

What is your opinion is on negative aggregation as a solution for the transient incast-like congestion after a failure with positive disaggregation?

— Bruno

> On Apr 18, 2019, at 5:07 PM, Kris Price <[email-masked]> wrote:
> 
> Hey Tony, inline:
> 
> [snip]

---

From: Bruno Rijsman <[email-masked]>
Date: Thu, 18 Apr 2019 19:11:48 -0300

Typo: make that negative *disaggregation*

> On Apr 18, 2019, at 7:10 PM, Bruno Rijsman <[email-masked]> wrote:
> 
> Kris,
>

---

From: Antoni Przygienda <[email-masked]>
Date: Thu, 18 Apr 2019 23:10:40 +0000

________________________________
From: Kris Price <[email-masked]>
Sent: Thursday, April 18, 2019 1:07 PM
To: Antoni Przygienda
Cc: [email-masked]; [email-masked]
Subject: Re: RIFT

Hey Tony, inline:

[snip]

We spent a lot of time chewing different design points. Your choices are either "flat host routes everywhere" or "in case of failures your servers may become top of your fabric" in case of multi-plane ... Or you ring on top in multi-plane (which BTW some top 5 already do and generally, 90% of people are happy with single-plane where you don't need any rings @ ToF ;-) ...  You missed the core team discussions that have been had for weeks and months ;-) Look @ recordings pls ...

[snip]

Modern architectures I see will be moving to good extent to ROTH IMO due to micro-segmentation and tunnel origination on servers.

I think we have a disconnect here. ToF level will only disaggregate if a ToF looses _all_ ToP connections to a PoD in a single plane design so I don't follow your argument. If you run multi-plane design you should multi-home each Pod multiple times into your plane as well. If you don't, dugh, you must disaggregate since the plane will blackhole.

Well, having blast radius of whole fabric is in a sense deterministic with every server changing/rebooting shaking whole fabric. I wouldn't call it optimal though.

Far more helpful than "deterministic" is in control system theory (https://en.wikipedia.org/wiki/Stability_theory) to think about "stability" where desireable positive stability is correlated with minimal blast radius. The more inputs shake more of your system the less "stability" you have.
[https://upload.wikimedia.org/wikipedia/commons/3/3b/Stability_Diagram.png]<https://en.wikipedia.org/wiki/Stability_theory>

Stability theory - Wikipedia<https://en.wikipedia.org/wiki/Stability_theory>
In mathematics, stability theory addresses the stability of solutions of differential equations and of trajectories of dynamical systems under small perturbations of initial conditions. The heat equation, for example, is a stable partial differential equation because small perturbations of initial data lead to small variations in temperature at a later time as a result of the maximum principle.
en.wikipedia.org

But again, if you want to disaggreagate e'thing all the time, RIFT won't stop you and you still will be benefiting from flood reduction and N-flooding-only in RIFT which makes for about 25% of normal flooding volume based on empirical data here ...

Then we disconnect. Think about flat host routing & what rebooting one server does to you in terms of flooding & resulting computaton and so on and what RIFT blast radius is. There is a world of difference.

Obviously, your choice. If all servers in a PoD want to have all server prefixes disaggregated, RIFT won't prevent you if you get the implementaion knob.

yes, it's always-negative-disaggregation which is possible, however much harder to implement and you would somehow need to ring the ToP to have all the necesasry topolgoy information to achieve that (that's why we ring ToF in multi-plane design). Argument has been made before, we spent tons time with Pascal going pro and cons until the current design was found the best choice.

---

From: Kris Price <[email-masked]>
Date: Sat, 20 Apr 2019 12:16:27 -0400

Hi Bruno,

Not sure if I followed negative disaggregation correctly. Is this used
at levels below the top of fabric or was it something that was
discussed as a possibility?

Cheers
Kris

---

From: Antoni Przygienda <[email-masked]>
Date: Sat, 20 Apr 2019 16:57:56 +0000

Kris, negative disaggreagtion is used if and only if

a) fabric has mutliple planes
b) a node gets completely separated in terms of cross-sectional bandwidth from a plane, call that "fallen leafs"

Negative disaggregation is transitive only to the level where the breakage is healed

Sections 5.2.5.2, 6.5 explains that all in nice details and specs out the mechanisms necessary. All in protocol since a bit now ...

thanks

--- tony
________________________________
From: Kris Price <[email-masked]>
Sent: Saturday, April 20, 2019 9:16 AM
To: Bruno Rijsman
Cc: Antoni Przygienda; [email-masked]
Subject: Re: RIFT

Hi Bruno,

Cheers
Kris

---

From: Kris Price <[email-masked]>
Date: Sat, 20 Apr 2019 14:58:41 -0400

Inline:

[snip]

> [Prz] Modern architectures I see will be moving to good extent to ROTH IMO due to micro-segmentation and tunnel origination on servers.

[KP]: Will respond to this further down.

> [Prz] I think we have a disconnect here. ToF level will only disaggregate if a ToF looses _all_ ToP connections to a PoD in a single plane design so I don't follow your argument. If you run multi-plane design you should multi-home each Pod multiple times into your plane as well. If you don't, dugh, you must disaggregate since the plane will blackhole.

[KP]: I think the disconnect is due my not using RIFT labels for
devices. I'm not talking about top of fabric. In RIFT labels I'm
describing a PoD, where the Leafs are top of rack switches. Then when
a Top of PoD<->Leaf link fails, the other Top of PoD switches will
disaggregate the prefixes on and below that Leaf leading to the incast
problem described. (This is described in the draft.)

[snip]

> [Prz] Well, having blast radius of whole fabric is in a sense deterministic with every server changing/rebooting shaking whole fabric. I wouldn't call it optimal though.

[KP]: Will respond to this further down.

> Far more helpful than "deterministic" is in control system theory (https://en.wikipedia.org/wiki/Stability_theory) to think about "stability" where desireable positive stability is correlated with…

[KP]: Absolutely, I'm not a mathematician, but reducing the amount of
change under small perturbations is a concern at the back of my head
when I described a preference for more deterministic behavior. Aside
from addressing the incast concern, it was an intuition that adding
routes, and subtracting routes as they come and go would be less
change than mass adds/removals when disagregating. So e.g. sticking
with the example of one PoD, where the link between a top of rack and
top of PoD switch goes down. That means one top of PoD device
withdraws one route (or maybe 1*many routes if routing on host is
happening), and that was less of a churn than say 7 other top of PoD
switches advertising 7*many new routes.

> [Prz] But again, if you want to disaggreagate e'thing all the time, RIFT won't stop you and you still will be benefiting from flood reduction and N-flooding-only in RIFT which makes for about 25% of normal flooding volume based on empirical data here ...

[KP]: That's good, that was my hope. There are benefits to RIFT beyond
it's positive disaggregation feature.

[snip]

> [Prz] Then we disconnect. Think about flat host routing & what rebooting one server does to you in terms of flooding & resulting computaton and so on and what RIFT blast radius is. There is a world of difference.

[KP]: I don't fully follow the statements about routing on the host
and shaking the fabric. Sure it would be a bad idea[tm] to do this in
a flat OSPF domain. As I understood it in RIFT with it's link state
up, distance vector down design, if we have a route come and go at the
edge then it will be propagated all the way northbound, but will not
be propagated southbound. If we were disaggregated between the Top of
Rack switches and the Top of PoD switches, then a route appearing or
dissapearing does propagate back down from the Top of PoD to the Top
of Rack, but that's it. It doesn't shake the wider fabric any more
than without disaggreation. If we were disagregated between the Top of
Fabric and Top of PoD routes would be propagated back down to the
other Top of PoDs, but not below.

[KP]: WRT to the routing on host it also seems in contradiction with
concerns about fabric stability. If fabric stability is a concern I
would think you still want addressing hierarchy and to use another
layer of indirection to achieve service mobility, to keep the fabric
unaware of the services constantly popping in and out of existence.
Yes, selectively this works, e.g. in the software load balancer
scenario, tunnel ingest, etc. That is fine and widely practiced within
limits. But the way you're describing it sounds like it's expected to
be used generously, with every host announcing prefixes, and there's
an expectation to move those prefixes such that you end up with a
random distribution. (Which is fine, I am probably out of touch with
fashion.) So with dissagregation in the PoD servers being single homed
would still see just the default. But all Top of Rack switches will
see the prefixes from other servers in the PoD, vs. when aggregation
is in effect and they'll only see the default (plus any disaggregated
due to a failure). The Top of PoD will have all prefixes in the PoD
and below, and further up in higher layers they'll all need to scale
up their FIB requirements to see all fabric routes. That's the same in
all cases with RIFT due to link state up.

[KP]: From purely the scaling perspective the aggregation feature is
useful where the number of routes produced by the servers in a PoD can
overwhelm the top of rack switches in that PoD but not the Top of PoD
switches, and so on up higher layers in the fabric. The FIB available
in devices these days would seem to preclude that.

[snip]

> yes, it's always-negative-disaggregation which is possible, however much harder to implement and you would somehow need to ring the ToP to have all the necesasry topolgoy information to achieve that (that's why we ring ToF in multi-plane design). Argument has been made before, we spent tons time with Pascal going pro and cons until the current design was found the best choice.[snip]

[KP]: It looks like negative disaggregation could be an elegant
protocol level solution if feasible and reliable.

[KP]: I think the answer is that RIFT has looked into the problem and
prospective operators are happy with the risks and trade off so I
accept that. :-)

Thanks!
Kris

---

From: Tony Przygienda <[email-masked]>
Date: Sat, 20 Apr 2019 18:35:52 -0700

On Sat, Apr 20, 2019 at 11:58 AM Kris Price <[email-masked]> wrote:

> ...

Let's look @ at a figure

              .                                  [A,B,C,D]
              .                                  [E]
              .             +-----+      +-----+
 Level 2      .             |  E  |      |  F  | A/32 @ [C,D]
              .             +-+-+-+      +-+-+-+ B/32 @ [C,D]
              .               | |          | |   C/32 @ C
              .               | |    +-----+ |   D/32 @ D
              .               | |    |       |
              .               | +------+     |
              .               |      | |     |
 Level 1      .       [A,B] +-+---+  | | +---+-+ [A,B]
              .       [D]   |  C  +--+ +-+  D  | [C]
              .             +-+-+-+      +-+-+-+
              .  0/0  @ [E,F] | |          | |   0/0  @ [E,F]
              .  A/32 @ A     | |    +-----+ |   A/32 @ A
              .  B/32 @ B     | |    |       |   B/32 @ B
              .               | +------+     |
              .               |      | |     |
              .             +-+---+  | | +---+-+
              .             |  A  +--+ +-+  B  |
 Level 0      . 0/0 @ [C,D] +-----+      +-----+ 0/0 @ [C,D]

Let's call A ToR and it's holding 8 server addresses. If you loos D-A the
only disaggregation you will see is C disaggregating to B the 8 addresses.
This is unavoidable. I assume we agree.

>
> > [Prz] I think we have a disconnect here. ToF level will only
> disaggregate if a ToF looses _all_ ToP connections to a PoD in a single
> plane design so I don't follow your argument. If you run multi-plane design
> you should multi-home each Pod multiple times into your plane as well. If
> you don't, dugh, you must disaggregate since the plane will blackhole.
>

right. It's really the only possible choice between having aggreation and
having to react to link failure by disaggregating.

>
> > Far more helpful than "deterministic" is in control system theory (
> https://en.wikipedia.org/wiki/Stability_theory) to think about
> "stability" where desireable positive stability is correlated with minimal
> blast radius. The more inputs shake more of your system the less
> "stability" you have.
>

if that's your preference you can simply configure all your Level 1
switches to always disaggregate @ the cost of extra flooding on every
address change and FIB size in Level 0.

> > advertising prefixes from servers that are doing software load

Your blast radius is somewhat bigger. Every server coming will affect all
Level 0s in the PoD.

> other Top of PoDs, but not below.

Yes, I meant if you run a flat OSPF domain with host routes (as people
actually do if the scale holds up).

Otherwise, yes, we agree. We just needed to talk in same words about same
things ;-)

> unaware of the services constantly popping in and out of existence.

yes and no. Depends what you need. if you want to multi-home servers (since
impact on your services is non-negligible if you loose e.g. a ToR) and need
automatic bandwidth balancing north (nice thing), no need for MC-LAG in L2,
tunnel origination on server without stitching, automatic disaggregation on
failures, view of full topology on top of fabric and so on this has lots of
appeal. And then, if you start to do things like running BIRD on host &
then redistribute a default route and so on you don't have lots of these
capabilites and on top another layer/protocol isntance to manage.

> fashion.)

yupp, that is the expectation (i.e. RIFT is designed to be able to support
that if needed). Look @ mobility section ;-)  Then you really have a
"fabric" vs. a "network", i.e. something that gives you bandwidth the same
way chips give you RAM. You don't think on which RAM bank your allocation
has to reside to work, why should you be all concerned where and how you
hook stuff up and whether your services move addresses if all you need is
just "more bandwidth".

> So with dissagregation in the PoD servers being single homed
> would still see just the default.

if your server is single homed running any kind of routing protocol seems a
waste really (unless you statically provision addreses and want them
carried through rather than using DHCP and so on). You can as well point a
static out,  it's not like you can load balance, react to failures or
anything much.

> But all Top of Rack switches will

Only ToF needs all routes (which is level 2 in 5-stage folded) in case of
single plane fabric. In multi-plane fabric things are more complex. Any
reasonable failure should be healed by negative disaggregation in levels
higher up but one could construct completely pathological scenarios where
you have to propagate all the way down since if a server can reach another
server through certain planes only, it must know which planes to avoid to
prevent a up-fabric/down-fabric/up-fabric again effectively turning other
servers into ToF (which we call "fabric inversion" and seems extremely
undesirable, BTW, in such scenarios your flooding on normal protocols also
has to go up/down/up so once that happens you really don't have any kind of
"hierarchical fabric" but bunch of nodes & links where traffic tries to get
places somehow). I think the draft explains that decently well.

> in devices these days would seem to preclude that.
Right, so it's an interesting discussion and you're very focused on the way
you prefer to deploy it and then that all makes sense. But if you have for
reasons above pull RIFT all way down into multi-home server you realize
that your FIB is small and that storing your underlay routes competes
directly with your overlay routes which are the ones paying the bills so
the dynamic changes. If your ToRs are originating overlays (as in EVPN
e.g.) you'll face the same calculus. RIFT is agnostic, run it to the ToR,
disaggregate servers if you want, will woirk fine but it allows you to pull
all the way to ROTH and fast mobility of addresses, or run EVPN origination
on the ToRs and use all-active or MC-LAG or whatever from servers, it will
all work.

So, we have an applicability document pending and this kind of stuff should
all go into it IMO. Feel free to drum up the crowd and start/massage it.

> [snip]
>
> > yes, it's always-negative-disaggregation which is possible, however much
> harder to implement and you would somehow need to ring the ToP to have all
> the necesasry topolgoy information to achieve that (that's why we ring ToF
> in multi-plane design). Argument has been made before, we spent tons time
> with Pascal going pro and cons until the current design was found the best
> choice.[snip]
>

we spec'ed it our solid me thinks & you find examples and so on in the
spec. Implementation doesn't look very challenging, most intersting is the
recursive FIB hole punching in case of negative disaggregates but in
fabrics it seems very unlikely people will carry lots aggregates together
with more specific so then the problem doesn't even exist. Silicon is
oblivious to it BTW, it all happens in control plane. If you read that and
have further input, all interested in that ...

nice you're drilling, I think lots people looked @ the stuff over last
year+ and we closed all the holes and discussed out all the design choices
but one more pair of experienced eyes never hurts

--- tony