Pawsey Supercomputing Research Centre Status

Setonix Login nodes experiencing Slingshot issues

Update - HPE made change to the Global Flow Control on scratch and software during January's maintenance, as well as a modification to the configuration of the LAG ports in the Slingshot fabric.

We haven't seen any C_EC_CRIT errors on the login nodes since maintenance, and are continuing to monitor them like hawks.
Jan 19, 2026 - 08:21 AWST

Update - setonix-04 reported a C_EC_CRIT error yesterday. It is not in the login pool, but HPE are stumped at why this is happening.
Nov 07, 2025 - 13:40 AWST

Monitoring - HPE rebooted a number of Slingshot switches during maintenance.

We haven't observed any Slingshot errors on the login, data mover or visualisation nodes for 48 hours.

We will continue to monitor.
Nov 06, 2025 - 12:29 AWST

Update - HPE have provided no new information
Oct 31, 2025 - 08:11 AWST

Update - HPE have provided no new information.
Oct 27, 2025 - 10:59 AWST

Update - HPE have provided no new information.
Oct 24, 2025 - 21:01 AWST

Update - HPE have provided no new information.

setonix-08 has slingshot issues. Pawsey is rebooting it.
Oct 20, 2025 - 13:25 AWST

Update - setonix-02 and setonix-03 have been added back to the RR DNS.
Oct 16, 2025 - 14:09 AWST

Investigating - There appears to be an issue will the Slingshot interfaces in the login nodes in Setonix. We appear to be down to 1 login node in the normal pool of login nodes.

We have had a case open with HPE for weeks, but they appear to be no closer to providing any kind of solution.

Please, please, please, please don't run any computational intensive operations on the login nodes. We have lovely compute nodes for that.

Please be aware that you can log into setonix-workflow.pawsey.org.au and get access to additional "workflow" nodes.
Oct 16, 2025 - 12:02 AWST

Setonix Degraded Performance

Data-mover nodes Operational

Slurm scheduler Operational

Setonix work partition Operational

Setonix debug partition Operational

Setonix long partition Operational

Setonix copy partition Operational

Setonix askaprt partition Operational

Setonix highmem partition Degraded Performance

Setonix gpu partition Operational

Setonix gpu high mem partition Operational

Setonix gpu debug partition Operational

Lustre filesystems Operational

/scratch filesystem Operational

/software filesystem Operational

/askapbuffer filesystem Operational

/askapingest filesystem Operational

Storage Systems Operational

Acacia Ingest Operational

Acacia MWA Operational

Acacia Projects Operational

Banksia Operational

Data Portal Systems Operational

MWA ASVO Operational

ASKAP Operational

ASKAP ingest nodes Operational

ASKAP service nodes Operational

Central Services Operational

Authentication and Authorization Operational

Service Desk Operational

License Server Operational

Application Portal Operational

Origin Operational

/home filesystem Operational

/pawsey filesystem Operational

Central Slurm Database Operational

Documentation Operational

Visualisation Services Operational

Remote Vis Operational

Vis scheduler Operational

Setonix vis nodes Operational

Nebula vis nodes Operational

Visualisation Lab Operational

Reservation Operational

CARTA - Stable Operational

CARTA - Test Operational

Pawsey Remote VR Operational

The Australian Biocommons Operational

Fgenesh++ Operational

Nimbus - Legacy Operational

Ceph storage Operational

Nimbus instances Operational

Nimbus dashboard Operational

Nimbus APIs Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

Allocated Cores (Setonix)

Fetching

Allocated Nodes (Setonix work partition)

Fetching

Allocated nodes (Setonix askaprt partition)

Fetching

Past Incidents

Jan 26, 2026

No incidents reported today.

Jan 25, 2026

No incidents reported.

Jan 24, 2026

No incidents reported.

Jan 23, 2026

No incidents reported.

Jan 22, 2026

No incidents reported.

Jan 21, 2026

No incidents reported.

Jan 20, 2026

No incidents reported.

Jan 19, 2026

Unresolved incident: Setonix Login nodes experiencing Slingshot issues.

Jan 18, 2026

No incidents reported.

Jan 17, 2026

No incidents reported.

Jan 16, 2026

No incidents reported.

Jan 15, 2026

No incidents reported.

Jan 14, 2026

No incidents reported.

Jan 13, 2026

Pawsey Scheduled Maintenance (January)

Completed - All Pawsey services have been returned to service. If you have any problems, please e-mail help@pawsey.org.au.

Mandatory annual testing of the High Voltage equipment at the Pawsey Centre will be performed on the 10th February 2026.

Pawsey will shutdown all services housed in the Pawsey Centre starting at 12 PM on the 9th February 2026. This is to preserve the integrity of the data in our storage system and reduce the risk of damage to equipment.

During this time all Pawsey Supercomputing Centre services will be unavailable.

We will start returning services as soon as we get the all clear from CBIS, starting on Wednesday, 11th February 2026.
Jan 13, 17:08 AWST

Verifying - Banksia has been returned to service and is currently being tested by System Owners.

Acacia has been returned to service, and running compliance checking which should be non-disruptive.

Current ETA for closure of maintenance is 4:30 PM (AWST).
Jan 13, 15:37 AWST

Update - Setonix has been returned to service.

Patching of core Pawsey services is complete.

Please note the firewall is still being updated, so there might be the occasional connectivity issue with Pawsey services.
Jan 13, 13:13 AWST

Update - Firewall updates have commenced.

Setonix login, data mover and visualisation nodes have been re-provisioned.

We have handed the CPU and GPU nodes to HPE so they can replace two of the bladder tanks.
Jan 13, 09:15 AWST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 13, 07:00 AWST

Update - A number of Power Distribution Units (PDUs) in Numbus are going to be moved from one circuit to another to isolate them from other systems. This should not impact services on Nimbus, but the service should be considered at risk during maintenance.
Jan 9, 13:41 AWST

Scheduled - Maintenance will be carried out on Pawsey systems on Tuesday the 13th January to apply required patches and updates to improve the systems stability, security, and performance. This maintenance window will also be used to undertake other tasks which require down-time to achieve.

Planned work for this window includes:
• The firewall will be upgraded.
• HPE will replace the bladder tanks for two of the Cooling Distribution Units (CDU).
• Setonix will have the latest bug and security fixes applied from SUSE Linux Enterprise Server 15 SP6.
• HPE will be modifying the Slingshot settings for LAG ports.
• "singularityce" module on Setonix will be hidden, current users of the module have been informed separately and all other singularity modules on the system will remain available
• Update to the cli-filter on Setonix to support future GPU power control operations.
• Acacia Projects will commence migration work off Puppet infrastructure.
• Acacia Projects will have automation compliance tests performed.
• Acacia MWA will have the new network traffic monitoring tool deployed.
• Banksia will have a ScoutAM upgrade performed.
• Banksia will be moving to a new VLAN.
• Change over to new Kafka Production server for event notifications on Banksia.
• Patching of core Pawsey services will be undertaken.

If you have any questions, please contact help@pawsey.org.au.
Jan 6, 09:00 AWST

Jan 12, 2026

No incidents reported.