The extreme slowness issues with Banksia last week appear now be resolved after vendor worked to clean up the system and implement a new software patch.
Posted May 25, 2022 - 11:25 AWST
Update
Banksia queue are now idle and the system is ready for use again. The Vendor has remedid short reads and dealt with one tape in particular that was causing a queue blockage. We will monitor for a few days and if all is operating optimally, we will look to closing this incident.
Posted May 23, 2022 - 09:53 AWST
Update
Banksia has been patched and files are once again staging from tape. The stage queue length has subsequently halved. The Vendor is completing further cleanup work and verification work on the filesystem cache which may slow things down over the next day or so, but the vast majority of queued up stage requests are now running. We will monitor this over today and over the weekend. We expect that the system should settle back down to normal by Monday.
Posted May 20, 2022 - 09:59 AWST
Update
The vendor is working on a new software release 2.6.3 for Banksia to remedy current issues that meant the tape queue scheduler was idled. This release is currently undergoing testing before it is scheduled to be rolled into production. More details on this date will be provided after testing is completed in a few days or so. Online files remain available.
Posted May 19, 2022 - 09:45 AWST
Update
Overnight the vendor has needed to idle the staging scheduler so presently files requested from tape won’t be retrieved and will be queued, however files already online are still available. The scheduler will be resumed after the next patch, due out shortly.
Posted May 18, 2022 - 08:49 AWST
Update
Banksia load has reduced further and the system state is looking better today. There still remains a volume of files to be recalled from tapes but this is a manageable number.
Posted May 17, 2022 - 09:46 AWST
Update
Pawsey and the Vendor are continuing to monitor for further issues, the system is still very busy post-cleanup and it is expected to be such for some days.
Posted May 16, 2022 - 09:04 AWST
Monitoring
The Vendor has resumed the jobs to stage files online, but there is quite a long tail of items to process and retrieve, mostly Pawsey Data Portal file requests. So the slow state may persist for the time being. Another update will be provided tomorrow.
Posted May 14, 2022 - 08:50 AWST
Update
Vendor has completed their initial work. The Staging Scheduler has been stopped so the system can process the backlog. This is unlikely to complete today.
Posted May 13, 2022 - 11:56 AWST
Update
Banksia has slowed down further. We have met with the vendor and have a robust plan to address all known issues.
Posted May 13, 2022 - 11:41 AWST
Update
Banksia is under high load, operating extremely slowly, so although it is online but essentially unavailable. The vendor is investigating and we will provide an update later today.
Posted May 13, 2022 - 09:53 AWST
Investigating
The Banksia system continues to experience intermittent periods of high load, nodes being ejected from the cluster and issues that affect the staging of some files from tape.
Work continues to further tune the system to ensure all migrated files stage successfully.
Please email help@pawsey.org.au for files that you require if they do not stage back after a day.
Posted May 11, 2022 - 13:38 AWST
This incident affected: Storage Systems (Banksia).