Acacia Ingest degraded with some operations failing

Incident Report for Pawsey Supercomputing Research Centre

Resolved

This incident has been resolved.

Posted May 29, 2025 - 12:48 AWST

Monitoring

Firewall rules have been reinstated, and all daemons are responding correctly. The service is restored to operation.

The wrong cluster was initially identified - this is for the Acacia Ingest cluster, not Acacia Projects. We apologise for the confusion.

Posted May 29, 2025 - 12:17 AWST

Update

We are continuing to work on a fix for this issue.

Posted May 29, 2025 - 12:07 AWST

Identified

Inadvertently changed firewall rules stopped the machines from communicating with each other. We are fixing the rules.

Posted May 29, 2025 - 12:06 AWST

Investigating

Storage daemons on a number of machines for Acacia Ingest are not operating properly. Some read and write operations still succeed, others others fail if they are directed toward those machines.

We are investigating the cause.

Posted May 29, 2025 - 11:51 AWST

This incident affected: Storage Systems (Acacia - Ingest).