Power loss across multiple systems
Incident Report for Pawsey Supercomputing Centre
Postmortem

This outage was caused by the total load on one of the switchboards (that provides static UPS backed power to the supercompute cell) exceeding the adjustable trip limit during ‘burn in’ testing of the new Garrawarla cluster.

This has been remediated by switching the new Garrawarla compute nodes to an alternative supply, and the adjustable limit will be increased during the upcoming September maintenance window, which will require a shutdown of all equipment connected to that distribution board, and any services that depend on that equipment.

Posted Aug 20, 2020 - 15:36 AWST

Resolved
This incident has been resolved.
Posted Jul 08, 2020 - 10:44 AWST
Monitoring
Supercomputing clusters: Magnus, Galaxy, Zeus, and Topaz, along with lustre filesystems, have returned to operation.
Posted Jul 03, 2020 - 16:41 AWST
Identified
One of the lustre filesystems (/astro) isn't coming back online cleanly. A call has been lodged with the vendor and we're awaiting an update from them before proceeding. Supercompute services will remain offline until the filesystem issue is addressed.
Posted Jul 02, 2020 - 21:05 AWST
Update
All the affected services are fed via a single sub-board. CSIRO facilities staff are on their way to site to investigate cause and Pawsey staff will commence service recovery when given the all-clear
Posted Jul 02, 2020 - 16:25 AWST
Update
We are continuing to investigate this issue.
Posted Jul 02, 2020 - 15:57 AWST
Investigating
We are investigating a possible power loss to some parts of the Pawsey infrastructure, it seems to be services located in the supercompute cell. Staff are investigating and will update shortly. See also https://support.pawsey.org.au/documentation/display/US/I-2020-07-02-Pawsey
Posted Jul 02, 2020 - 15:34 AWST
This incident affected: Zeus (Zeus login node, Galaxy ingest nodes), Lustre filesystems (/scratch filesystem, /group filesystem, /astro filesystem, /askapbuffer filesystem), Galaxy (Galaxy login nodes), Magnus (Magnus login nodes), and Storage Systems (Hierarchical Storage Management Systems, CASDA Nodes).