From HPWREN's point of view, traffic has been stable for the past few hours. We will continue to monitor CENIC as they make progress towards a permanent resolution, but as of now HPWREN traffic no longer appears to be affected.
Posted Mar 02, 2023 - 19:27 PST
Update
Update from CENIC:
CENIC has disabled multicast and reenabled one backbone circuit along with services to sites experiencing loss of connectivity. Other circuits remain disabled and CENIC is monitoring the device for stability. If the router remains stable, additional services will be turned up gradually tonight after midnight. Juniper TAC is still working with its core team to identify a permanent fix.
Posted Mar 02, 2023 - 18:56 PST
Monitoring
Update from CENIC:
Juniper TAC has identified a multicast-related bug on our Los Angeles aggregation router. Currently, all services are disabled on the router to avoid any further impact to the backbone.
CENIC is starting maintenance to disable multicast on the device and then re-enable one backbone circuit along with services to sites experiencing loss of connectivity. Other circuits will remain disabled while CENIC monitors the device for stability. If the router remains stable, additional services will be turned up gradually tonight after midnight. Juniper TAC is currently working with their core team to identify a permanent fix.
Posted Mar 02, 2023 - 17:52 PST
Update
CENIC engineers continue to troubleshoot with vendor, Juniper, to further investigate the root cause of the backbone device outage. We will provide you with more information as it becomes available.
Posted Mar 02, 2023 - 15:27 PST
Identified
Our upstream network provider, CENIC, is suffering from major service interruptions today, starting at 12:34 PM Pacific. Traffic in and out of the HPWREN network is affected.