Two OSSes on /scratch have crashed

Incident Report for Pawsey Supercomputing Research Centre

Resolved

Scratch appears to be nominal. We are waiting on a RCA from HPE.

Posted Apr 26, 2024 - 08:18 AWST

Monitoring

The two OSTs which refused to reassemble (avenger style) have been put back together and have a filesystem check performed on them.

Scratch is now fully function, and staff will resume drained nodes on Setonix on Friday (26th April 2024).

Posted Apr 25, 2024 - 11:05 AWST

Identified

HPE are onsite and have replaced failed hardware. They are currently bringing the OSTs back and will perform a filesystem check when they are ready.

Posted Apr 24, 2024 - 19:26 AWST

Investigating

HPE are working on it

Posted Apr 24, 2024 - 15:27 AWST

This incident affected: Lustre filesystems (/scratch filesystem (new)).