Scratch File System Performance Degradation

Incident Report for Pawsey Supercomputing Research Centre

Resolved

All OSTs in scratch are operational and we will continue to monitor the filesystem over the weekend.

The issue may have been caused by the flash pool becoming critically full.

Please remember that the scratch filesystem is a shared resource for the temporary storage of results of simulations and data processing.

Please be mindful of other researchers, and do not store unnecessary data.
Posted Apr 17, 2025 - 15:40 AWST

Monitoring

The problematic OSTs have been checked and /scratch is fully operational.
Posted Apr 17, 2025 - 14:04 AWST

Update

HPE have reported two of the OSTs have "journal errors" are are performing a check.
Posted Apr 17, 2025 - 11:14 AWST

Update

Two OSSes have failed.
Posted Apr 17, 2025 - 11:00 AWST

Investigating

There is performance degradation in relation to the "/scratch" filesystem
* This is related to SSD storage OST pools being close to full capacity
Posted Apr 17, 2025 - 10:35 AWST
This incident affected: Lustre filesystems (/scratch filesystem).
← Current Status Powered by Atlassian Statuspage