Resolved -
The extreme slowness issues with Banksia last week appear now be resolved after vendor worked to clean up the system and implement a new software patch.
May 25, 11:25 AWST
Update -
Banksia queue are now idle and the system is ready for use again. The Vendor has remedid short reads and dealt with one tape in particular that was causing a queue blockage. We will monitor for a few days and if all is operating optimally, we will look to closing this incident.
May 23, 09:53 AWST
Update -
Banksia has been patched and files are once again staging from tape. The stage queue length has subsequently halved. The Vendor is completing further cleanup work and verification work on the filesystem cache which may slow things down over the next day or so, but the vast majority of queued up stage requests are now running. We will monitor this over today and over the weekend. We expect that the system should settle back down to normal by Monday.
May 20, 09:59 AWST
Update -
The vendor is working on a new software release 2.6.3 for Banksia to remedy current issues that meant the tape queue scheduler was idled. This release is currently undergoing testing before it is scheduled to be rolled into production. More details on this date will be provided after testing is completed in a few days or so. Online files remain available.
May 19, 09:45 AWST
Update -
Overnight the vendor has needed to idle the staging scheduler so presently files requested from tape won’t be retrieved and will be queued, however files already online are still available. The scheduler will be resumed after the next patch, due out shortly.
May 18, 08:49 AWST
Update -
Banksia load has reduced further and the system state is looking better today. There still remains a volume of files to be recalled from tapes but this is a manageable number.
May 17, 09:46 AWST
Update -
Pawsey and the Vendor are continuing to monitor for further issues, the system is still very busy post-cleanup and it is expected to be such for some days.
May 16, 09:04 AWST
Monitoring -
The Vendor has resumed the jobs to stage files online, but there is quite a long tail of items to process and retrieve, mostly Pawsey Data Portal file requests. So the slow state may persist for the time being. Another update will be provided tomorrow.
May 14, 08:50 AWST
Update -
Vendor has completed their initial work. The Staging Scheduler has been stopped so the system can process the backlog. This is unlikely to complete today.
May 13, 11:56 AWST
Update -
Banksia has slowed down further.
We have met with the vendor and have a robust plan to address all known issues.
May 13, 11:41 AWST
Update -
Banksia is under high load, operating extremely slowly, so although it is online but essentially unavailable.
The vendor is investigating and we will provide an update later today.
May 13, 09:53 AWST
Investigating -
The Banksia system continues to experience intermittent periods of high load, nodes being ejected from the cluster and issues that affect the staging of some files from tape.
Work continues to further tune the system to ensure all migrated files stage successfully.
Please email help@pawsey.org.au for files that you require if they do not stage back after a day.
May 11, 13:38 AWST