RCA: Gale Academic OneFile Collection Deletion, August 2021
Introduction
This document serves as a Root Cause Analysis (RCA) for the reported incident on June 15, 2021.The goal of this document is to share our findings regarding the reported incident, specify the root cause analysis, handling of the event, outline actions to be taken for mitigation, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
Date |
Activity |
June 8, 2021 |
The collection Gale Academic OneFile was deleted from Alma CZ |
June 15, 2021 |
The incident was reported on the Alma Listserv as well as via support cases indicating that the collection was deleted from Alma CZ |
June 16, 2021 |
First Ex Libris communication to the Alma Listserv, updating the issue is being investigated with highest priority |
June 20, 2021 |
Communication update to the Alma Listserv, acknowledging the deletion and updating restore is in process |
June 23, 2021 |
First attempt to restore the Gale Collection to Alma CZ |
June 28, 2021 |
Communication update to the Alma Listserv regarding the restoration of the collection to Alma CZ and an update that IZ activation restoration will be done automatically by Ex Libris and expected to take affect over the coming days |
June 29, 2021 |
Communication update to the Alma Listserv with clarifications on the expected fix |
July 15, 2021
|
Communication update to the Alma Listserv regarding IZ restoration testing status including an indication that institutions who wish to, can start manual activation of the collection Article published to track updates on the status of the collection and related fixes |
July 18, 2021 |
Issue identified with initial restoration of the collection to CZ, causing the collection to be activated with wrong number of portfolios for institutions that performed manual activation |
July 20, 2021
|
Collection was restored again to Alma CZz Communication update sent to Alma listserv including detailed clear step by step instructions: Alma CZ: Gale Academic OneFile - July 2021 |
July 22, 2021
|
Following risk analysis, it was decided to restore IZ activations only for some of the institutions, based on activation type:
Communication update to the Alma Listserv |
July 25, 2021 |
Targeted communication sent to institutions where the Gale collection is activated with the wrong number of portfolios with instructions on how to fix the issue |
July 25, 2021 - today |
Monitoring institutions' restoration status |
Root Cause Analysis
Ex Libris investigated this event to determine the impact and root cause analysis with the following results: On June 8th, the collection was deleted by Content Operations.
Findings
Ex Libris investigated this event and determined the following:
-
The collection was manually deleted from Alma CZ, outside of the regular procedure
-
Due to the above, a special procedure was required to restore the full collection in Alma CZ
-
Such a procedure was not available as it is the first time such a deletion happened
-
First attempt to restore the collection on Alma CZ was incomplete
-
-
After thorough analysis, Ex Libris attempt to automatically restore institution data was withdrawn due to solution complexity and high risk
-
Insufficient communication on status and timeline
-
Time to resolution does not meet Ex Libris targets
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
-
Enhance prevention capabilities to perform actions outside of the routine procedure to significantly reduce the risk of manual error
-
We have reviewed and updated existing procedure for collection deletion (Done)
-
We have initiated a process to redefine roles and privileges for all teams (End of August 2021)
-
We will provide new users in all teams with corrected roles and privileges (End of August 2021)
-
-
Change management process analysis for high impact changes
-
Monitor and alert on high impact changes once occurs
-
Quick recovery procedure for deleted content
-
Solution recommendations should be ready by end of September 2021
-
-
Deletion procedure
-
We have refreshed the Deletion procedure for all team members (Done)
-
We have ensured the procedure is well documented and available for all team members (Done)
-
We have ensured the procedure is part of new employee onboarding (Done)
-
We have scheduled quarterly knowledge refresh with all team members (Done)
-
Plan to revisit deletion procedures (manual and automated) overall including portfolios (End of September 2021)
-
-
Enhance communication policy
-
Alma October 2021 release will include a new feature called ‘Alma Announcement’. This new feature allows Ex Libris to update users about of high impact issues, unusual occurrences, or major feature rollouts, as well as ad hoc updates and announcements.Content Operations will use this feature to update customers directly on ad-hoc issues, their status, and resolutions. Using Alma Announcement will ensure timely communication that is directed to relevant staff at the effected institutions. (October)
-
Conclusion
Ex Libris treats this incident as high priority with evaluation, assessment, mitigation processes and lessons learned. We are determined to improve the level of CZ quality and the value it provides to our customers.