Completed
All services were returned to production on Thursday (3rd September 2026).
The Slurm controller on Setonix has been operating normally over the weekend.
With the Slurm upgrade there are changes in how Slurm allocates resources for jobs running on GPU nodes. As a result, researchers need to update their job submission scripts to ensure continued compatibility. Please read through the documentation page prepared by Pawsey staff (https://pawsey.atlassian.net/wiki/spaces/US/pages/1965031425/SLURM+25.11.7+changes).
If you have any questions, please reach out to the Service Desk (help@pawsey.org.au).
Thank you for your patience. And thank you to the Pawsey staff who resolved the issue.
Posted Sep 07, 2026 - 09:22 AWST
Update
We are continuing to verify the maintenance items.
Posted Sep 03, 2026 - 11:26 AWST
Update
Pawsey staff have tweaked the configuration of the Slurm controller for Setonix and brought more of the system back online. We will be providing NVIDIA additional logs and will continue to monitor the system.
Posted Sep 03, 2026 - 10:53 AWST
Update
We are continuing to investigate the issue.
Posted Sep 02, 2026 - 09:17 AWST
Update
The Pawsey team have altered the network connections on the scheduler to make it more responsive.
A ticket is being lodged with NVIDIA, who provide L3 Slurm support.
Posted Sep 01, 2026 - 19:12 AWST
Update
Setonix passed our internal testing, but once the maintenance reservation was removed there appeared to be communication issues between the Slurm controller and nodes.
We are investigating the issue.
Posted Sep 01, 2026 - 17:37 AWST
Verifying
Vis services are back up in Production.
The Setonix workflow nodes are being looked at.
Windows updates are being applied to the Nebula nodes.
Posted Sep 01, 2026 - 16:09 AWST
Update
Acacia Projects is back up in Production.
Acacia MWA is back up in Production.
Core services have been patched.
Posted Sep 01, 2026 - 15:49 AWST
Update
Acacia Ingest is back up in Production.
Banksia is back up in Production.
Mediaflux is back up in Production.
Posted Sep 01, 2026 - 14:34 AWST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Sep 01, 2026 - 08:00 AWST
Scheduled
Maintenance on Pawsey systems will be carried out on Tuesday the 1st September to apply required patches and updates to improve the systems stability, security, and performance. This maintenance window will also be used to undertake other tasks which require down-time to achieve.
Planned work for this window includes:
• Patching of visualisation services will be undertaken
• Mediaflux server will have the latest bug and security fixes from Rocky Linux
• Finalisation of the control plane of Acacia Ingest
• Acacia compliance runs
• Banksia's servers will have the latest bug and security fixes from Rocky Linux
• Banksia will be upgraded to the latest version of ScoutAM
• Patching of core Pawsey services will be undertaken
• Setonix will have the latest bug and security fixes from SLES 15 applied
• Setonix will be upgraded to a supported version of Slurm
• ASKAP Ingest will have the latest bug and security fixes from SLES 15 applied
• ASKAP Ingest will be upgraded to a supported version of Slurm
With the Slurm upgrade there are changes in how Slurm allocates resources for jobs running on GPU nodes. As a result, researchers need to update their job submission scripts to ensure continued compatibility. Please read through the documentation page prepared by Pawsey staff (https://pawsey.atlassian.net/wiki/spaces/US/pages/1965031425/SLURM+25.11.7+changes).
If you have any questions, please contact help@pawsey.org.au.
Posted Aug 25, 2026 - 09:00 AWST
This scheduled maintenance affected: ASKAP (ASKAP ingest nodes, ASKAP service nodes), Central Services (Authentication and Authorization, License Server, Application Portal, Origin, Central Slurm Database), Storage Systems (Acacia Ingest, Acacia MWA, Acacia Projects, Banksia, Data Portal Systems, MWA ASVO), Setonix (Login nodes, Data-mover nodes, Slurm scheduler, Setonix work partition, Setonix debug partition, Setonix long partition, Setonix copy partition, Setonix askaprt partition, Setonix highmem partition, Setonix gpu partition, Setonix gpu high mem partition, Setonix gpu debug partition), and Visualisation Services (Remote Vis, Vis scheduler, Setonix vis nodes, Nebula vis nodes, Visualisation Lab, Reservation, CARTA - Stable, CARTA - Test, Pawsey Remote VR).