Acacia Ingest has been operating normally since we implemented a change to the load balancer architecture.
Posted Mar 08, 2024 - 16:10 AWST
Monitoring
We have made changes to the architecture of the primary load balancer, and are seeing a significant increase in throughput.
Posted Mar 06, 2024 - 15:20 AWST
Update
We are working with MWA and CASDA to investigate the cause of the slow read performance.
There was originally some concern that the MWA ingest was impacting the CASDA read performance, however MWA halved their ingest workload this morning and this did not show a noticeable change to the read behaviour. This was true in reverse also - MWA increased ingest from half to full yesterday morning, and the read misbehaviour didn't show until around 1pm.
At this point in the investigation, it appears that we are hitting some limits in overall Acacia (both clusters) throughput, due to increased usage. Although they are separate clusters, they share some infrastructure. We are preparing some changes to address this, but these will not be able to be implemented this week.
Posted Mar 01, 2024 - 12:41 AWST
Investigating
We have received reports of very slow read workloads on the Ingest cluster, and are investigating.
Posted Feb 29, 2024 - 17:23 AWST
This incident affected: Storage Systems (Acacia - Ingest).