Pawsey Supercomputing Research Centre Status

Setonix Login nodes experiencing Slingshot issues

Update - setonix-04 reported a C_EC_CRIT error yesterday. It is not in the login pool, but HPE are stumped at why this is happening.
Nov 07, 2025 - 13:40 AWST

Monitoring - HPE rebooted a number of Slingshot switches during maintenance.

We haven't observed any Slingshot errors on the login, data mover or visualisation nodes for 48 hours.

We will continue to monitor.
Nov 06, 2025 - 12:29 AWST

Update - HPE have provided no new information
Oct 31, 2025 - 08:11 AWST

Update - HPE have provided no new information.
Oct 27, 2025 - 10:59 AWST

Update - HPE have provided no new information.
Oct 24, 2025 - 21:01 AWST

Update - HPE have provided no new information.

setonix-08 has slingshot issues. Pawsey is rebooting it.
Oct 20, 2025 - 13:25 AWST

Update - setonix-02 and setonix-03 have been added back to the RR DNS.
Oct 16, 2025 - 14:09 AWST

Investigating - There appears to be an issue will the Slingshot interfaces in the login nodes in Setonix. We appear to be down to 1 login node in the normal pool of login nodes.

We have had a case open with HPE for weeks, but they appear to be no closer to providing any kind of solution.

Please, please, please, please don't run any computational intensive operations on the login nodes. We have lovely compute nodes for that.

Please be aware that you can log into setonix-workflow.pawsey.org.au and get access to additional "workflow" nodes.
Oct 16, 2025 - 12:02 AWST

Setonix Operational

Data-mover nodes Operational

Slurm scheduler Operational

Setonix work partition Operational

Setonix debug partition Operational

Setonix long partition Operational

Setonix copy partition Operational

Setonix askaprt partition Operational

Setonix highmem partition Operational

Setonix gpu partition Operational

Setonix gpu high mem partition Operational

Setonix gpu debug partition Operational

Lustre filesystems Operational

/scratch filesystem Operational

/software filesystem Operational

/askapbuffer filesystem Operational

/askapingest filesystem Operational

Storage Systems Operational

Acacia Ingest Operational

Acacia MWA Operational

Acacia Projects Operational

Banksia Operational

Data Portal Systems Operational

MWA ASVO Operational

ASKAP Operational

ASKAP ingest nodes Operational

ASKAP service nodes Operational

Central Services Operational

Authentication and Authorization Operational

Service Desk Operational

License Server Operational

Application Portal Operational

Origin Operational

/home filesystem Operational

/pawsey filesystem Operational

Central Slurm Database Operational

Documentation Operational

Visualisation Services Operational

Remote Vis Operational

Vis scheduler Operational

Setonix vis nodes Operational

Nebula vis nodes Operational

Visualisation Lab Operational

Reservation Operational

CARTA - Stable Operational

CARTA - Test Operational

Pawsey Remote VR Operational

The Australian Biocommons Operational

Fgenesh++ Operational

Nimbus - Legacy Operational

Ceph storage Operational

Nimbus instances Operational

Nimbus dashboard Operational

Nimbus APIs Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

Allocated Cores (Setonix)

Fetching

Allocated Nodes (Setonix work partition)

Fetching

Allocated nodes (Setonix askaprt partition)

Fetching

Past Incidents

Dec 16, 2025

No incidents reported today.

Dec 15, 2025

No incidents reported.

Dec 14, 2025

No incidents reported.

Dec 13, 2025

No incidents reported.

Dec 12, 2025

No incidents reported.

Dec 11, 2025

No incidents reported.

Dec 10, 2025

No incidents reported.

Dec 9, 2025

No incidents reported.

Dec 8, 2025

No incidents reported.

Dec 7, 2025

No incidents reported.

Dec 6, 2025

No incidents reported.

Dec 5, 2025

No incidents reported.

Dec 4, 2025

No incidents reported.

Dec 3, 2025

No incidents reported.

Dec 2, 2025

Pawsey Scheduled Maintenance (December)

Completed - All Pawsey systems have been returned to service.

Please note the following changes were *not* implemented
• Updating the cli-filter on Setonix to support future GPU power control operations.
• Moving Banksia to a new VLAN.

We would like to remind researchers about the Pawsey login nodes policy, which you can review here (https://pawsey.atlassian.net/wiki/spaces/US/pages/51931338/Login+nodes+policy). We kindly ask all researchers to adhere to the policy.

If you have any problems, please e-mail help@pawsey.org.au.
Dec 2, 18:00 AWST

Verifying - Acacia Ingest is having its last verification and testing.

All other systems have been returned to service.

We will provide a final update at 6PM.

Thank you very much for your patience.
Dec 2, 17:09 AWST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 2, 09:00 AWST

Scheduled - Maintenance will be carried out on Pawsey systems on Tuesday the 2nd December to apply required patches and updates to improve the systems stability, security, and performance. This maintenance window will also be used to undertake other tasks which require down-time to achieve.

Planned work for this window includes:
• The firewall will have security profiles applied to increase visibility and threat prevention coverage.
• Firewall security policy clean up.
• Preemptive high availability election on firewalls will be enabled.
• HPE will re-cable the temperature sensor in rack x1001 of Setonix.
• HPE will perform coolant sampling of Setonix.
• Update to the cli-filter on Setonix to support future GPU power control operations.
• Install CPU and GPU versions of NAMD 3.0.2 on Setonix.
• /home on ASKAP Ingest and Ella will be replaced with a NetApp (as well as /software on ASKAP Ingest).
• Block port 5000 externally on Acacia Projects.
• Acacia Ingest will continue migration work off Puppet infrastructure.
• Banksia will be moving to a new VLAN.
• Change over to new Kafka Production server for event notifications on Banksia.
• Patching of visualisation services will be undertaken.
• Patching of core Pawsey services will be undertaken.

If you have any questions, please contact help@pawsey.org.au.
Nov 25, 09:49 AWST