Overnight the staging queue reduced quite significantly, and is still reducing this morning so this issue and risk can be closed off. Thankyou for your patience during this busy time on Banksia.
Posted Jan 19, 2024 - 07:18 AWST
Monitoring
the leader was restarted and staging is operating again as evidenced by tape load and unload operations but the queue statistics are changing (more jobs running, queue length reducing).
Posted Jan 18, 2024 - 10:12 AWST
Identified
The issue appears to the scoutam services are stuck on vss nodes. so the leader will be restarted.
Posted Jan 18, 2024 - 10:10 AWST
Investigating
Files queued via ScoutAM REST API for staging are not progressing on Tape Library 1. This is causing the staging queue to be blocked. ScoutAM cannot communicate with Library 1 although Library 1 is up on. Checking continues.
Posted Jan 18, 2024 - 09:20 AWST
This incident affected: Storage Systems (Banksia, Data Portal Systems, MWA ASVO).