Power failed to a large part of the SIO Campus, in a way that caused the generators to not kick on. Our facility was without power for about 90 minutes, and when power was restored it took a while to get services back online. Most services were restored 12 hours later at around noon.
The biggest user facing issue we had was that DNS services were only running at the SIOCOLO. We are working on a way to redistribute those services to more datacenters, as the failover nodes didn’t do as there were supposed to.