CARTA Service Access Issues
Investigating - We are currently investigating an issue affecting access to the CARTA visualisation service on Setonix.

Users may experience a "502 Bad Gateway" error when accessing https://vis.pawsey.org.au/carta/. Further updates will be provided as information becomes available.
Nov 10, 2026 - 17:28 AWST
Banksia - shared filesystem issues affecting service
Update - Banksia has been running again under synthetic staging load for most of today and it appears the workarounds we put in place are working. Suffice to say Banksia is again available for use. If you do happen to strike any issues please log a helpdesk ticket.
Nov 10, 2026 - 15:58 AWST
Update - Further work on Banksia has continued with our external support team. While yesterday’s testing showed progress, Banksia remains unstable under load and is not yet able to be returned to service. Given the ongoing fibre channel issues, we are reaching out to additional experts with specialist experience in fibre channel problems to assist with diagnosis and resolution. Work is continuing and we will provide further updates as information comes to hand. Thank you for your continued patience.
Nov 10, 2026 - 07:36 AWST
Update - Work continues of Banksia today with our external support team, we have noted that there was an unstable fibre channel state which we resolved. We are busy trying to bring up Banksia and ensure it remains stable for use.
Nov 09, 2026 - 14:32 AWST
Update - We have worked to determine there's a block device issue with one of the DDN arrays for the shared filesystem. Work is continuing by our support partner over the weekend to test and find the failed unit parts so we can either implement a workaround or replace them with spare new unit parts and get the Banksia system operating for use. Monday will be when you may expect the update on this item.
Nov 06, 2026 - 17:44 AWST
Update - Banksia update - all of Banksia has been rolled back to the pre-maintenance state, unfortunately shared filesystem problems persist. Exact cause identification (to lead to a fix) work continues with the assistance and guidance of our Support Providers on both the filesystem software side and shared storage hardware side. Another update will be provided today.
Nov 06, 2026 - 13:42 AWST
Update - Good Morning, IBM Support have come back with further information overnight and Pawsey Staff will undertake further work on parts of Banksia that include the DDN block storage and FibreChannel switches and another update will be provided later today after this work is executed. Thankyou for your continued patience.
Nov 06, 2026 - 08:38 AWST
Update - Testing two fixes and rolling back the system to pre-maintenance is underway. We have attended to the banksia servers and they are done, however the issues with the shared filesystem remain. We have attempted to roll back the fibre channel switche OS version also however performing this rollback has hit a snag so a P1 case has been opened with IBM USA to escalate to the SAN Switch specialist team for prompt assistance, diagnosis and rollback.
Nov 05, 2026 - 18:07 AWST
Investigating - Banksia is again experiencing an issue with the shared filesystem storage on one node again. We are working on this presently - we need to do further work with our external support partner to work on a permanent fix to return Banksia to service.
Nov 05, 2026 - 07:23 AWST
Setonix Degraded Performance
Login nodes Operational
Data-mover nodes Operational
Slurm scheduler Operational
Setonix work partition Operational
Setonix debug partition Operational
Setonix long partition Operational
Setonix copy partition Operational
Setonix askaprt partition Operational
Setonix highmem partition Operational
Setonix gpu partition Degraded Performance
Setonix gpu high mem partition Operational
Setonix gpu debug partition Operational
Lustre filesystems Operational
/scratch filesystem Operational
/software filesystem Operational
/askapbuffer filesystem Operational
/askapingest filesystem Operational
Storage Systems Operational
Acacia Ingest Operational
Acacia MWA Operational
Acacia Projects Operational
Banksia Operational
Data Portal Systems Operational
MWA ASVO Operational
ASKAP Operational
ASKAP ingest nodes Operational
ASKAP service nodes Operational
Central Services Operational
Authentication and Authorization Operational
Service Desk Operational
License Server Operational
Application Portal Operational
Origin Operational
/home filesystem Operational
/pawsey filesystem Operational
Central Slurm Database Operational
Documentation Operational
Visualisation Services Partial Outage
Remote Vis Operational
Vis scheduler Operational
Setonix vis nodes Operational
Nebula vis nodes Operational
Visualisation Lab Operational
Reservation Operational
CARTA - Stable Partial Outage
CARTA - Test Operational
Pawsey Remote VR Operational
The Australian Biocommons Operational
Fgenesh++ Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
Allocated Cores (Setonix)
Fetching
Allocated Nodes (Setonix work partition)
Fetching
Allocated nodes (Setonix askaprt partition) ?
Fetching

Nov 11, 2026

No incidents reported today.

Nov 10, 2026

Unresolved incidents: CARTA Service Access Issues, Banksia - shared filesystem issues affecting service.

Nov 9, 2026

Nov 8, 2026

No incidents reported.

Nov 7, 2026

No incidents reported.

Nov 6, 2026

Nov 5, 2026

Nov 4, 2026

Banksia - shared filesystem issues affecting service
Resolved - We have brought Banksia back into service. We are monitoring it closely to ensure its back to stable operation.
Nov 4, 17:09 AWST
Investigating - Banksia is presently experiencing an issue with a shared filesystem storage device. We are working on this presently - we have identified the device with the issue with assistance from our external support partner and working on a fix to return Banksia to service.
Nov 4, 09:40 AWST

Nov 3, 2026

Pawsey Scheduled Maintenance (November)
Completed - Good afternoon everyone,

Today’s maintenance work is complete, and the following services have been returned to service.

Setonix
HPE successfully applied the required Slingshot configuration change. The Cray Programming Environment (CPE) has been upgraded to version 26.03, and the 2026.11 software stack is now the default. The latest SLES 15 bug fixes and security updates have also been applied.

Important notes:
Pytorch
We have also identified that the new PyTorch software module pytorch/2.13.0-rocm7.2.4 is failing test with an illegal shader instruction error. This is being investigated, researchers that used PyTorch module on the system will be notified about resolution.

Scratch filesystem
During our return to service testing we have noted some minor degradation in performance of the flash partition of the scratch file system. Pawsey staff will continue to monitor and investigate the cause.

LD_LIBRARY_PATH
A new Lmod CVE fix requires upgrading to Lmod 9.4.3, which changes how LD_LIBRARY_PATH is handled. Pawsey Singularity modules have been updated accordingly and testing was successful, but any custom modules that directly read LD_LIBRARY_PATH may need review and modification. Please raise Help Desk ticket if you encounter any issues.

For details of the software stack changes and recommended user actions, please see November 2026 Software Update – Important Information.

ASKAP ingest
ASKAP Ingest has been returned to service following installation of the latest SLES 15 bug fixes and security updates.

Core services
General patching of all core services has been completed.

Acacia object Storage
Acacia has been returned to service following completion of security patching.

Banksia and tape storage
Banksia and tape storage have been returned to service. Banksia’s servers have received the latest bug fixes and security updates, ScoutAM has been upgraded to the latest version, and the Fibre Channel switches have received the latest firmware updates.

The upgrade from ScoutGW to VersityGW was not implemented and will be re-visited next month. VSS-6 will remain out of service.

Visualisation services
Visualisation services have been returned to service following patching.

Mediaflux
Mediaflux has been returned to service following an upgrade and patching in accordance with vendor recommendations.

Data Portal
The Data Portal has been returned to service. Data Portal now supports Acacia allocations on the Projects cluster.

Thank you for your patience during today’s maintenance. If you have any questions or encounter any issues, please contact help@pawsey.org.au.
Nov 3, 15:27 AWST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 3, 07:00 AWST
Scheduled - Maintenance on Pawsey systems will be carried out on Tuesday the 3rd November to apply required patches and updates to improve the systems stability, security, and performance. This maintenance window will also be used to undertake other tasks which require down-time to achieve.

HPE will be reapplying the Slingshot configuration change they made in October to perform troubleshooting in conjunction with resources in the US.

Planned work for this window includes:
• Patching of visualisation services will be undertaken
• Mediaflux server will have the latest bug and security fixes from Rocky Linux
• Mediaflux will be upgraded
• Data Portal will be upgraded
• Data Portal support for Acacia allocations will be rolled out to all projects with allocations on the Acacia
• Acacia will have some insecure packages removed
• Banksia's servers will have the latest bug and security fixes from Rocky Linux
• Banksia will be upgraded to the latest version of ScoutAM
• Banksia fibre switches will have firmware updates
• Patching of core Pawsey services will be undertaken
• Setonix will be upgraded to the Cray Programming Environment (CPE) version 26.03
• New software stack 2026.11 will be set as default on Setonix
• Setonix will have the latest bug and security fixes from SLES 15 applied
• ASKAP Ingest will have the latest bug and security fixes from SLES 15 applied

For more information about the software stack changes on Setonix, please see https://pawsey.atlassian.net/wiki/spaces/US/pages/2132049952/November+2026+Software+Update+-+Important+Information.

If you have any questions, please contact help@pawsey.org.au.
Oct 27, 09:02 AWST

Nov 2, 2026

No incidents reported.

Nov 1, 2026

No incidents reported.

Oct 31, 2026

No incidents reported.

Oct 30, 2026

No incidents reported.

Oct 29, 2026

No incidents reported.

Oct 28, 2026

No incidents reported.

Incident History Powered by Atlassian Statuspage