Subject: Re: [109attendees] [109all] NOC update

From: Lucy Lynch <[email-masked]>
Date: Tue, 17 Nov 2020 17:10:44 -0800

> On Nov 17, 2020, at 4:57 PM, Sean Croghan <[email-masked]> wrote:
> 
> 
> 
> 
> I have an update for those of you affected by the outage in yesterdays IABOPEN session. We have isolated this to a interrupt to the virtual machines network interface. We currently have no explanation for this outage. We have engaged the hardware and network team with Azure to determine the cause of this event but do not have an explanation at this time. 
> 

So I’ve noticed an odd thing which may or may not be relevant - 

I joined all the relevant jabber streams for my favorite WGs early using my iPad client -
When I join the meetecho stream I’m also spliced into the chat / jabber

Both seem to work but I wonder if distributing the joins across VMs would cause some kind of OAuth r e condition? 

I will note that people seem to complain about losing services piece by piece so there is some kind of fracture or mis-match in play.

Don’t know enough to be smart so feel free to write this off as a stupid remark but I will note that token flows are sucking hard

> I will provide an update when we have received more information. 
> 
> 
> For those interested in details:
> 
> At 07:56:36 UTC the network interface (eth0) went link down and the interface was removed from the VM
> At 08:00:28 UTC then a new interface was added to the VM
> At 08:00:29 UTC (eth1) went link up 
> 
> Yes the VM added a new interface. The servers were provisioned with SR-IOV and we suspect that a migration event occurred that moved the VM to different hardware causing the NIC driver to be reloaded. We have found some evidence that would support our theory that a migration or unscheduled maintenance event occurred and are working to verify if that happened during this event. We have removed SR-IOV from the network interfaces on all servers. 
> 
> I hope you are having a good and productive week 
> 
> 
> — The IEFT NOC Team
> 
> -- 
> 109all mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/109all