Flash pool usage has returned to less concerning levels and we are monitoring the levels and implementing more systems to migrate data that has not been used in a while to disk to free up the highest speed storage. Pawsey remains impressed with our colleagues ability to generate data on Setonix at a rate faster than we can deal with it and acknowledge that under normal circumstances we'd be celebrating it rather than panicking. Thank you for your patience.
Posted May 16, 2025 - 08:50 AWST
Investigating
There are performance / usability issues pertaining to "/scratch" filesystem * Flash Storage pools are approaching Max capacity * This affects the overall performance / usability of scratch * Data generation is exceeding data migration to Non-flash pools (Mitigation efforts)
If you have unnecessary files on /scratch, please remove them ASAP
Posted May 15, 2025 - 15:23 AWST
This incident affected: Lustre filesystems (/scratch filesystem).