Subject: [Anima] ANIMA when there is a system-wide issue

From: Brian E Carpenter <[email-masked]>
Date: Tue, 1 Dec 2020 11:02:28 +1300

"AWS reveals it broke itself by exceeding OS thread limits"

https://www.theregister.com/2020/11/30/aws_outage_explanation/

Especially:
"The TIFU-like post also outlines why Amazon's dashboards offered only scanty info about the incident – because they, too, depend on a service that depends on Kinesis."

Perhaps there is something we should specify in ANIMA to prevent the ANIMA infrastructure falling into this sort of trap: when there is a system-wide issue (such as hitting an O/S resource limit everywhere at the same time) it also prevents the autonomic mechanisms from working.
 
Regards
   Brian Carpenter

---

From: Michael Richardson <[email-masked]>
Date: Mon, 30 Nov 2020 18:36:16 -0500

Brian E Carpenter <[email-masked]> wrote:
    > Perhaps there is something we should specify in ANIMA to prevent the
    > ANIMA infrastructure falling into this sort of trap: when there is a
    > system-wide issue (such as hitting an O/S resource limit everywhere at
    > the same time) it also prevents the autonomic mechanisms from working.

I think you mean, that it should not also prevent the autonomic system?
I think that key is:
  1) allocate resources (including threads) up-front
  2) do not dynamically allocate threads per interface, but rather use async
    routines.

But, I don't know what we could write into the specification to make this
happen.  It seems that we really just need smart implementers.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 1 Dec 2020 15:37:06 +1300

> happen.  It seems that we really just need smart implementers.

I think the IETF could do better on "Implementation Considerations" in
general. For now, I'll make a note to add something in the "Robustness"
section of the ASA Guidelines document.

Regards
   Brian