Subject: [Rift] Question about intent of RIFT spec: adjacency between two non-leaf nodes, more than 1 level apart

From: Bruno Rijsman <[email-masked]>
Date: Fri, 10 Aug 2018 22:13:34 +0000

Hi Tony and list,

I have a question about RIFT, specifically the scenario where we have two nodes, both of them hard-configured to be a non-leaf, and with level values that are more than 1 level apart.

Consider the following topology:

#  +-----------+
#  | node1     |
#  | (level 3) |  <<< More than 1 level difference between hard-configured level 3
#  +-----------+      on node1 and hard-configured level 1 on node2.
#        | if1        Furthermore, neither node1 nor node2 is a leaf.
#        |            Hence, the adjacency should not come up to 3-way.
#        | if1
#  +-----------+
#  | node2     |
#  | (level 1) |
#  +-----------+

Question 1: Does the above text accurately describe the intent of the spec? In other words, is it the intent of the spec that the adjacency should *NOT* reach 3-way?

If so, I am trying to understand the intent of the following clause:

 if [...] OR (lie’s level
 is not leaf AND its difference is more than one from my
 level) then CLEANUP, PUSH UpdateZTPOffer, PUSH
 UnacceptableHeader else

Question 2: When you say "UpdateZTPOffer", which if the following do you intend to happen?

(a) ZTP to treat this as a REMOVE_OFFER, since the LIE message was rejected.

OR

(b) ZTP to treat this is a Valid Offered Level (VOL) despite the fact that the LIE message
was rejected. Process the offer in the LIE as an UPDATE_OFFER, and consider it for the
computation of HAL. (But not for HAT, because the adjacency will not reach 3-way).

Question 3: Is this the sitation that you have in mind when you said "passing all checks
for adjacency formation while disregarding all clauses involving level values" in the following
section?

 Valid Offered Level (VOL): A neighbor’s level received on a valid
 LIE (i.e. passing all checks for adjacency formation while
 disregarding all clauses involving level values) persisting for
 the duration of the holdtime interval on the LIE.

---

From: Antoni Przygienda <[email-masked]>
Date: Fri, 10 Aug 2018 22:24:11 +0000

Question 1: correct, no 3-way is formed
Question 2: b), reason being that the offer that the node accepted may go away and then you want to re-converge faster than waiting for a LIE that you accept.  Definition of VOL is very precise, after packet passed all checks except the level checks it is considered VOL
Question 3: yepp, and it seems to compute

The reasoning WHY it would have to be that way is kind of long’ish and came out of experimentation, lots whiteboard, going back and forth on running code. The latest discussion on BAL after hackathon (best available level rather than HAL) that we had shows e.g. that it’s pretty easy to build loops or “invert the fabric upside down” (and/or slow convergence) with the best intentions. Waiting for 3-ways to accept offers also sounds logical unless one realizes that that way a) convergence is pretty slow b) it shakes the fabric a lot since 3-ways that are already formed tend to be torn when it shows that better offers are to be had. 3-way up/down is flooding and SPF and all the nasty stuff, hence the offers are carried below 3-way for max. speed & minimum shake. For the thing to work well clear gradient has to be built (starts from top and diffuses and everyone tries to hook up to the highest available source [i.e. closest to the ultimate source of truth]). That’s what the spec does.  I expect we’ll have discussion on finer points once you run your stuff in regard to convergence speed on initial convergence vs. purge on loss of all super-spines. Then we’ll take the trade-offs taken ;-)

No free lunch as usual …

--- tony

Thus spake <[email-masked]<mailto:[email-masked]>>:

Hi Tony and list,

#  +-----------+
#  | node1     |

 if [...] OR (lie’s level

OR

---

From: Bruno Rijsman <[email-masked]>
Date: Fri, 10 Aug 2018 22:32:49 +0000

Okay, got it.

That is exactly the way my code behaves.

When I saw the behavior (LIE rejected, but offer in it accepted as VOL for HAL anyway), I was surprised and I suspected a bug in my code.

I went back to the spec, confirmed that yes indeed, that is what is written there.

That triggered me to verify the behavior with you, and you confirmed that this is indeed what is supposed to happen.

Your "reasons why” sound plausible an I will probably grok them better once I get to big topologies.

For now, I am just writing automated system tests for simple 2 and 3 node scenarios, and that is already exposing lots of “interesting” corner cases.

Having automated reproducible tests is really necessary to catch those rare timing-related corner cases. The most interesting so far was that a slow IGMP join caused the appearance of a uni-directional link failure during boot up.

Anyways, this is all just a long-winded excuse for not yet having completed the ZTP-interop testing :-)  Still a work in progress, but getting closer.

— Bruno

On Aug 10, 2018, at 3:24 PM, Antoni Przygienda <[email-masked]<mailto:[email-masked]>> wrote:

No free lunch as usual …

--- tony

Hi Tony and list,

#  +-----------+
#  | node1     |

 if [...] OR (lie’s level

OR

 Valid Offered Level (VOL): A neighbor’s level received on a valid
 LIE (i.e. passing all checks for adjacency formation while
 disregarding all clauses involving level values) persisting for
 the duration of the holdtime interval on the LIE.
_______________________________________________
RIFT mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rift

---

From: Antoni Przygienda <[email-masked]>
Date: Fri, 10 Aug 2018 22:42:33 +0000

;-)  Havin’ seen the tests you wrote, like I told you on the weekly call, I had a chuckle given how very similar it looked to what I do normally in principle albeit not necessarily in scale ;-)

And yeah, 25 years into designing asynchronous protocols I still rely on this kind of stuff to jump surprises on me and prove that “intuition” counts nothing in heavily distributed systems design and proofs are hard to do and mostly fragile. ;-)  It’s like solving riddles where the pieces are never where you put them away last time ;-)

I will have pretty solid confidence in the spec BTW once you went the curve & both implementations grok’ each other happily ;-) After that flooding will be the last delicate piece, the flood reduction and other pieces are on purpose designed in an unsynchronized way, i.e. it doesn’t matter how many nodes support it and whether they even run precisely same distributed algorithms (unless you start to implement non-leaf including de-aggreation and south prefix origination, there’s good amount of interesting stuff right there again).

Thanks for the work, I hope it’s fun & walking the curve here makes for short days for you ;-)

--- tony

Okay, got it.

— Bruno

No free lunch as usual …

--- tony

Hi Tony and list,

#  +-----------+
#  | node1     |

 if [...] OR (lie’s level

OR

https://www.ietf.org/mailman/listinfo/rift<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_rift&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=maKXfKzgRTpiLitqHnJiww&m=48Ivbu0IhmnxAcL3S0X7v48wVof-cWwA3K8W2RStpUw&s=4sBtBetWbytlWusghDrDlUcKLWzkoG9sxuGzVlJdPVc&e=>