/scratch has problems (again)

Incident Report for Pawsey Supercomputing Research Centre

Resolved

/scratch has been "operational" for almost five days. HPE are still investigating what caused the issue.

Posted Apr 02, 2024 - 08:49 AWST

Monitoring

HPE R&D confirmed that mounting clients with 'localflock' should help. This has been implemented on /software and /scratch on Setonix and /scratch on Garrawarla.

The reservations have been removed off Setonix and Garrawarla.

Posted Mar 28, 2024 - 13:57 AWST

Identified

HPE have informed Pawsey that the local file locking option will have be re-implemented. Pawsey has implemented the change in the configuration system and is rebooting Setonix.

Posted Mar 28, 2024 - 09:09 AWST

Investigating

Both meta data servers on /scratch have appeared to have failed. Pawsey will raise a ticket with HPE.

Posted Mar 27, 2024 - 18:53 AWST

This incident affected: Lustre filesystems (/scratch filesystem (new)).