Subject: Re: [Rift] [RIFT] Routing in Fat Trees "chaos testing" hackathon at IETF 104 in Prague

From: Tony Przygienda <[email-masked]>
Date: Sun, 3 Mar 2019 11:05:20 -0800

Preparing stuff here along instructions, ran it on a t2.micro (spot request
is cheap and works well)

b) no command line completion on interactive ;-{
c) the meta config file restricts to 3 levels only (5 stage folded), good
enough for now but testing more than 3 levels is very interesting
proposition for the future
d) running the example I see tons of those

2019-03-03 18:35:08,355:ERROR:node.if.tx:[spine-2-2:veth-105g-4e] Error
[Errno 111] Connection refused sending IPv6 TIE ProtocolPacket(

and then

2019-03-03 18:35:08,274:ERROR:node.if.tx:[super-4:veth-4e-105g] Error
[Errno 111] Connection refused sending IPv4 TIDE ProtocolPacket

but I see no UDP losses

Udp:
    16103 packets received
    88 packets to unknown port received
    0 packet receive errors
    16224 packets sent
    0 receive buffer errors
    0 send buffer errors

Consequently I see on start  often 2-3 pairs of leafs that can't ping each
other, I did not analyze further, every time different pairs, sometimes all
come up.

monkey testing works fine ... Next iteration will be of course doing that
once positive disaggregation is implemented in python RIFT ;-)

I think @ hackathon first thing we have to write a simple script that
collects the kernel routing tables & compares them for # expected entries
since leaf-2-leaf has such a wide path diversity it's hard to debug,
observe that due to ECMP hashing you can get a leaf being able to ping 2
other leafs in another pod but not the third since it takes different
spine/superspine combination. Generally, to test connectivity on fabric
well, you need a fan-out of UDP source/sink pairs that lead to exhaustive
ECMP coverage (as well as tests quality of load balancing once we get to
the bandwidth balancing in python RIFT)

Yes, connectivity to spine loopbacks and so on has been discussed on the
core team list and is in the draft (-04 is in publication queue) ...
Generally, most robust IP fabric architecture is leaf-2-leaf since relying
on every leaf being able to reach every spine leads unavoidably into
exposing more and more topology towards the leafs with according
scalabiltiy problems on the leafs. So, a fabric where every leaf can reach
every leaf is in a sense "working fine" even if links are lost or some
spines crash. That modulo bandwidth balancing since we want to account for
load being balanced (unsurprisingly, RIFT does all that ;-) if we talk a
single prefix (loopback) on a spine/superspine then PGP and even simpler
stuff can work well but if we assume that specific superspines terminate
lots prefixes and rely on leafs being able to reach them then the fabric
"inverts" and scalbility is lost. Variants of the problem exist by e.g.
having superspines exposing anycast addresses. Once deployment experience
starts to come in I can see a "operational deployment" draft coming up (I
think someone already suggested to write one) ...

thanks

--- tony

On Mon, Feb 25, 2019 at 5:15 PM Bruno Rijsman <[email-masked]>
wrote:

> Hi everyone,
>
> At the Routing In Fat Trees (RIFT) IETF 104 hackathon in Prague we will be
> working on "chaos monkey" testing of two RIFT implementations: the
> RIFT-Python open source implementation and a vendor RIFT implementation.
>
> This hackathon is inspired by the concept of chaos engineering and in
> particular by the chaos monkey tool, which intentionally and randomly shuts
> down servers in a production environment. This is intended to build
> confidence in the system's capability to withstand turbulent and unexpected
> conditions.
>
> In this hackathon we will do something similar for RIFT networks. We will
> build a large fat tree network of RIFT routers. Then we will randomly
> insert failures into the network and make sure that the network behaves
> correctly. We will make sure the RIFT routers don't crash or otherwise
> behave badly. And we will make sure that the routers forwarding tables and
> link-state databases converge correctly.
>
> We will use (and possibly enhance) the chaos testing framework as well as
> the visualization and verification tools that are already part of the
> RIFT-Python open source project.
>
> Even if you are not much interested in the RIFT routing protocol, but you
> are interested in testing large network topologies using container-like
> technology, or in visualization tools, or in chaos testing, this might be
> the right hackathon for you.
>
> *A detailed description of the planned activities at the hackathon is
> available at: http://bit.ly/rift-hackathon-ietf-104
> <http://bit.ly/rift-hackathon-ietf-104>*
>
> The project is listed on the IETF 104 hackathon wiki page:
> https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon (info copied
> below)
>
> Feel free to join us.
>
> Bruno Rijsman
>
>
> Routing In Fat Trees (RIFT) Chaos Monkey Testing
>
> * Champions
>   * Tony Przygienda <prz at juniper.net> (will join in person)
>   * Bruno Rijsman <brunorijsman at gmail.com> (will join remotely)
> * Project(s)
>   * ​RIFT is a STD track WG working on a modern IP fabric/DC routing
> protocol
>   * ​Detailed description of the planned activities at the hackathon at
>     http://bit.ly/rift-hackathon-ietf-104
>   * We have two implementations now:
>     * RIFT-Python open source
>     * Vendor implementation
>     * The open source one has a chaos monkey framework in place we can use
>       to stress protocol implementations and interoperability between
> them.
>       Unusual hackathon but that will be very, very useful for further
>       development and quality of implementations while breaking completely
>       new grounds (chaos testing on model based protocol specifications).
> * Preconditions:
>   * Python knowledge
>   * Familiarity with the open source RIFT code on ​
> https://github.com/brunorijsman/rift-python
>   * RIFT Specification in some detail ​RIFT draft
>   * Familiarity with creating virtual machines in AWS
>
> _______________________________________________
> RIFT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rift
>

---

From: Bruno Rijsman <[email-masked]>
Date: Fri, 22 Mar 2019 13:15:34 -0300

See >>> inline for responses.

Question for you: is it possible / easy to run RIFT-Vendor as a standalone node in a network namespace in AWS?  I assume it is, but  have not yet tried it.  That we we can do Python - Vendor interop testing for larger topologies.  This is something we should try in the hackathon.

> On Mar 3, 2019, at 4:05 PM, Tony Przygienda <[email-masked]> wrote:
> 
> Preparing stuff here along instructions, ran it on a t2.micro (spot request is cheap and works well) 
> 

>>> Right, this is pretty annoying and it is on my TODO list to fix, but bigger fish to fry first.

> c) the meta config file restricts to 3 levels only (5 stage folded), good enough for now but testing more than 3 levels is very interesting proposition for the future

>>> Good point. I added "Support generating 4-level / 7-state topologies.” to my TODO list. But first, I need to get 3-level / 5-stage working more reliable. Currently there there is a pretty persistent issue with ping failures and we need to get to the bottom of that before we go bigger.  (Discussed in more detail below).

> 2019-03-03 18:35:08,355:ERROR:node.if.tx:[spine-2-2:veth-105g-4e] Error [Errno 111] Connection refused sending IPv6 TIE ProtocolPacket(
> 
> and then 
> 
> 2019-03-03 18:35:08,274:ERROR:node.if.tx:[super-4:veth-4e-105g] Error [Errno 111] Connection refused sending IPv4 TIDE ProtocolPacket

>>> Yes, this issue was discussed in detail on the RIFT core-team mailing list. Search your inbox for an e-mail dated 6-Dec-2018 and titled "[RIFT-TEAM] UDP send error issue (was Update: SPF is code-complete)”. I explained what the cause of the issue was and your response was "Not an issue, implementation can do what it wants ;-) flooding retx’s”. I have an item on my TODO list to open the UDP RX socket earlier in the FSM (as described in the email) but I have not yet gotten around to implementing that.

> Consequently I see on start  often 2-3 pairs of leafs that can't ping each other, I did not analyze further, every time different pairs, sometimes all come up. 

>>> I see that as well.  This the main thing I / we need to concentrate on next. Clearly, there is some pretty persistent issue with convergence.  Today (Friday), I will be enhancing the generated check.sh script to check more things than just pings: I will add check to make sure that all adjacencies are up, all LSDBs have the correct contents, all routes are present, etc. etc. etc.  That way we can more easily get to the root of the issue.  This is something that we could work on more during the hackathon.  We will need these more sophisticated tests are part of the chaos monkey testing infrastructure before we move on to larger 5-level / 7-stage topologies (those are simply too large to check by hand)>

> 
> monkey testing works fine ... Next iteration will be of course doing that once positive disaggregation is implemented in python RIFT ;-)

>>> Yep. After IETF-104, disaggregation (positive and negative) is the last remaining big ticket item (along with cleaning up a 100 or so small TODO items).  Once we have the enhanced automated checks in place that I mentioned in the previous point, re-running and debugging the chaos monkey tests with extra features (e.g. disaggregation) should be pretty straightforward.

> 
> I think @ hackathon first thing we have to write a simple script that collects the kernel routing tables & compares them for # expected entries since leaf-2-leaf has such a wide path diversity it's hard to debug, observe that due to ECMP hashing you can get a leaf being able to ping 2 other leafs in another pod but not the third since it takes different spine/superspine combination. Generally, to test connectivity on fabric well, you need a fan-out of UDP source/sink pairs that lead to exhaustive ECMP coverage (as well as tests quality of load balancing once we get to the bandwidth balancing in python RIFT)

>>> Agree 100%.  This would be part of the “enhanced automated checks” that I mentioned previously.
> 
> Yes, connectivity to spine loopbacks and so on has been discussed on the core team list and is in the draft (-04 is in publication queue) ... Generally, most robust IP fabric architecture is leaf-2-leaf since relying on every leaf being able to reach every spine leads unavoidably into exposing more and more topology towards the leafs with according scalabiltiy problems on the leafs. So, a fabric where every leaf can reach every leaf is in a sense "working fine" even if links are lost or some spines crash. That modulo bandwidth balancing since we want to account for load being balanced (unsurprisingly, RIFT does all that ;-) if we talk a single prefix (loopback) on a spine/superspine then PGP and even simpler stuff can work well but if we assume that specific superspines terminate lots prefixes and rely on leafs being able to reach them then the fabric "inverts" and scalbility is lost. Variants of the problem exist by e.g. having superspines exposing anycast addresses. Once deployment experience starts to come in I can see a "operational deployment" draft coming up (I think someone already suggested to write one) ... 

>>> I will mentioned “lack of connectivity to spine loopbacks” as one of the lessons learned in my slide deck.  Whether or not that is a problem should be a point of discussion during the RIFT WG meeting.

>>> Looking forward to “seeing” everyone tomorrow at the hackathon. Reminder: please setup a zoom meeting or something similar; I will be remote (Argentina).

> 
> 
> 
> 
> 
> On Mon, Feb 25, 2019 at 5:15 PM Bruno Rijsman <[email-masked] <mailto:[email-masked]>> wrote:
> Hi everyone,
>  
> At the Routing In Fat Trees (RIFT) IETF 104 hackathon in Prague we will be working on "chaos monkey" testing of two RIFT implementations: the RIFT-Python open source implementation and a vendor RIFT implementation.
> 
> This hackathon is inspired by the concept of chaos engineering and in particular by the chaos monkey tool, which intentionally and randomly shuts down servers in a production environment. This is intended to build confidence in the system's capability to withstand turbulent and unexpected conditions.
> 
> In this hackathon we will do something similar for RIFT networks. We will build a large fat tree network of RIFT routers. Then we will randomly insert failures into the network and make sure that the network behaves correctly. We will make sure the RIFT routers don't crash or otherwise behave badly. And we will make sure that the routers forwarding tables and link-state databases converge correctly.
> 
> We will use (and possibly enhance) the chaos testing framework as well as the visualization and verification tools that are already part of the RIFT-Python open source project.
> 
> Even if you are not much interested in the RIFT routing protocol, but you are interested in testing large network topologies using container-like technology, or in visualization tools, or in chaos testing, this might be the right hackathon for you.
> 
> A detailed description of the planned activities at the hackathon is available at: http://bit.ly/rift-hackathon-ietf-104 <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbit.ly%2Frift-hackathon-ietf-104&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811943278&sdata=Xt1ChJNF7JJEJ0Kfjn16f3lX0zpkUlr89kQKtizyFZA%3D&reserved=0>
> 
> The project is listed on the IETF 104 hackathon wiki page: https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrac.ietf.org%2Ftrac%2Fietf%2Fmeeting%2Fwiki%2F104hackathon&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811953283&sdata=n21RZh4p91Lmq7%2FB4iv8LyAD1m3ZiKziBlinMVkhq5o%3D&reserved=0> (info copied below)
>  
> Feel free to join us.
> 
> Bruno Rijsman
> 
> 
>   * Tony Przygienda <prz at juniper.net <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fjuniper.net&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811963295&sdata=eyUD%2Fuz8kHhDRKzLyh1UTYijM%2F68D7AC6ytWsWGQWQs%3D&reserved=0>> (will join in person)
>   * Bruno Rijsman <brunorijsman at gmail.com <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgmail.com&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811973306&sdata=FDsJ4hH8QByjd8Q%2BstIkScJTqsVHWuej6Wrf6dluI38%3D&reserved=0>> (will join remotely)
> * Project(s)
>   * ​RIFT is a STD track WG working on a modern IP fabric/DC routing protocol
>     http://bit.ly/rift-hackathon-ietf-104 <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbit.ly%2Frift-hackathon-ietf-104&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811983299&sdata=miXwSMnpPXXQcqQ7gPomj4uCpxYpBYU%2BiorFWBgLS5g%3D&reserved=0>
>       to stress protocol implementations and interoperability between them. 
>   * Familiarity with the open source RIFT code on ​https://github.com/brunorijsman/rift-python <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbrunorijsman%2Frift-python&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367811993322&sdata=uPFe4Ith8R5LK2ujuWwXwDryLP98lqCPoFznJnh7y7E%3D&reserved=0>
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/rift <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Frift&data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367812003309&sdata=ADlAx6zuVWqqlnf1cqVoTbh7dVV3UeR6naQOTEXvDO0%3D&reserved=0>
> hackathon mailing list
> [email-masked]
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fhackathon&amp;data=02%7C01%7C%7Ca04585e2044e4aa6bbd108d6a00b514e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636872367812033330&amp;sdata=onHIVdyEd8jrhEzwvC1Fx1CKAGMFg1fV%2B6Fn%2BolnqhM%3D&amp;reserved=0

---

From: Bruno Rijsman <[email-masked]>
Date: Fri, 22 Mar 2019 15:02:46 -0300

>>     88 packets to unknown port received

PS: Maybe you already did this, but just to be sure: when you look at the UDP statistics, make sure you are looking at the correct network namespace, for example:

(env) root@ip-xx-xx-xx-xx:~/rift-python# ip netns exec netns-1001 netstat --statistics --udp
IcmpMsg:
    InType0: 160
    InType3: 265
    InType8: 160
    InType11: 438
    OutType0: 160
    OutType3: 137
    OutType8: 160
Udp:
    11793 packets received
    204 packets to unknown port received
    0 packet receive errors
    6019 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
IpExt:
    InMcastPkts: 3365
    OutMcastPkts: 3372
    InOctets: 2386337
    OutOctets: 1525221
    InMcastOctets: 646036
    OutMcastOctets: 643080
    InNoECTPkts: 6577

---

From: Tony Przygienda <[email-masked]>
Date: Fri, 22 Mar 2019 23:05:12 +0100

On Fri, Mar 22, 2019 at 5:15 PM Bruno Rijsman <[email-masked]>
wrote:

> Question for you: is it possible / easy to run RIFT-Vendor as a standalone
> node in a network namespace in AWS?  I assume it is, but  have not yet
> tried it.  That we we can do Python - Vendor interop testing for larger
> topologies.  This is something we should try in the hackathon.
>
>
not bringing it to hackathon. python itself will be enough work. we can get
to the interop later

> >>> Good point. I added "Support generating 4-level / 7-state topologies.”
> to my TODO list. But first, I need to get 3-level / 5-stage working more
> reliable. Currently there there is a pretty persistent issue with ping
> failures and we need to get to the bottom of that before we go bigger.
>  (Discussed in more detail below).
>
> >>> Yes, this issue was discussed in detail on the RIFT core-team mailing
> list. Search your inbox for an e-mail dated 6-Dec-2018 and titled "*[RIFT-TEAM]
> UDP send error issue (was Update: SPF is code-complete)*”. I explained
> what the cause of the issue was and your response was *"Not an issue,
> implementation can do what it wants ;-) flooding retx’s”*. I have an item
> on my TODO list to open the UDP RX socket earlier in the FSM (as described
> in the email) but I have not yet gotten around to implementing that.
>
>
we need to talk that through Rtx'ing is fine but if the other side doesn't
have a socket open we shouldn't be getting the "refused sending". Let's
chat tommorow ... I'll come in afternoon around 2.30PM

> >>> I see that as well.  This the main thing I / we need to concentrate on
> next. Clearly, there is some pretty persistent issue with convergence.
> Today (Friday), I will be enhancing the generated check.sh script to check
> more things than just pings: I will add check to make sure that all
> adjacencies are up, all LSDBs have the correct contents, all routes are
> present, etc. etc. etc.  That way we can more easily get to the root of the
> issue.  This is something that we could work on more during the hackathon.
> We will need these more sophisticated tests are part of the chaos monkey
> testing infrastructure before we move on to larger 5-level / 7-stage
> topologies (those are simply too large to check by hand)>
>
>
agreed, I think that's what we'll focus on first.

> >>> Yep. After IETF-104, disaggregation (positive and negative) is the
> last remaining big ticket item (along with cleaning up a 100 or so small
> TODO items).  Once we have the enhanced automated checks in place that I
> mentioned in the previous point, re-running and debugging the chaos monkey
> tests with extra features (e.g. disaggregation) should be pretty
> straightforward.
>
>
> >>> I will mentioned “lack of connectivity to spine loopbacks” as one of
> the lessons learned in my slide deck.  Whether or not that is a problem
> should be a point of discussion during the RIFT WG meeting.
>
>
> thanks
>
> --- tony
>
>
> >>> Looking forward to “seeing” everyone tomorrow at the hackathon.
> Reminder: please setup a zoom meeting or something similar; I will be
> remote (Argentina).
>
>
I'll open zoom as soon I come in from my flight ...