HPWREN Server/Network Outage
Postmortem

Power failed to a large part of the SIO Campus, in a way that caused the generators to not kick on. Our facility was without power for about 90 minutes, and when power was restored it took a while to get services back online. Most services were restored 12 hours later at around noon.

The biggest user facing issue we had was that DNS services were only running at the SIOCOLO. We are working on a way to redistribute those services to more datacenters, as the failover nodes didn’t do as there were supposed to.

Posted Mar 29, 2023 - 17:54 PDT

Resolved
This incident has been resolved.
Posted Mar 22, 2023 - 15:18 PDT
Update
All HPWREN services have been restored.
Posted Mar 22, 2023 - 15:17 PDT
Update
HPWREN website has been restored
Posted Mar 22, 2023 - 11:30 PDT
Investigating
UCSD Campus suffered a power outage last night. Power has been restored and we are working on restoring server/network access for HPWREN services.
Posted Mar 22, 2023 - 11:23 PDT
This incident affected: Web (Main Web Site, Camera Image Acquistion), Network (Backbone, DNS), and Sensors.