Skip to main content
ExLibris
  • Subscribe by RSS
  • Ex Libris Knowledge Center

    RCA: Gale Academic OneFile Collection Deletion, August 2021

    Introduction

    This document serves as a Root Cause Analysis (RCA) for the reported incident on June 15, 2021.The goal of this document is to share our findings regarding the reported incident, specify the root cause analysis, handling of the event, outline actions to be taken for mitigation, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

     

    Event Timeline

    Date

    Activity

    June 8, 2021

    The collection Gale Academic OneFile was deleted from Alma CZ

    June 15, 2021

    The incident was reported on the Alma Listserv as well as via support cases indicating that the collection was deleted from Alma CZ

    June 16, 2021

    First Ex Libris communication to the Alma Listserv, updating the issue is being investigated with highest priority

    June 20, 2021

    Communication update to the Alma Listserv, acknowledging the deletion and updating restore is in process

    June 23, 2021

    First attempt to restore the Gale Collection to Alma CZ

    June 28, 2021

    Communication update to the Alma Listserv regarding the restoration of the collection to Alma CZ and an update that IZ activation restoration will be done automatically by Ex Libris and expected to take affect over the coming days

    June 29, 2021

    Communication update to the Alma Listserv with clarifications on the expected fix

    July 15, 2021

     

    Communication update to the Alma Listserv regarding IZ restoration testing status including an indication that institutions who wish to, can start manual activation of the collection 

    Article published to track updates on the status of the collection and related fixes

    July 18, 2021

    Issue identified with initial restoration of the collection to CZ, causing the collection to be activated with wrong number of portfolios for institutions that performed manual activation

    July 20, 2021

     

    Collection was restored again to Alma CZz

    Communication update sent to Alma listserv including detailed clear step by step instructions: Alma CZ: Gale Academic OneFile - July 2021

    July 22, 2021

     

    Following risk analysis, it was decided to restore IZ activations only for some of the institutions, based on activation type:

    • Institutions that had the collection only for CDI search (Active for search in CDI) had the collection restored by Ex Libris on July 22nd, 2021
    • Institutions who had the collection active in Alma were asked to follow step by step instructions (Alma CZ: Gale Academic OneFile - July 2021) or submit an SFC to Alma support for Ex Libris to perform the reactivation

    Communication update to the Alma Listserv 

    July 25, 2021

    Targeted communication sent to institutions where the Gale collection is activated with the wrong number of portfolios with instructions on how to fix the issue

    July 25, 2021 -  today

    Monitoring institutions' restoration status

     

    Root Cause Analysis

    Ex Libris investigated this event to determine the impact and root cause analysis with the following results: On June 8th, the collection was deleted by Content Operations.

     

    Findings

    Ex Libris investigated this event and determined the following:

    • The collection was manually deleted from Alma CZ, outside of the regular procedure

    • Due to the above, a special procedure was required to restore the full collection in Alma CZ

    • Such a procedure was not available as it is the first time such a deletion happened

      • First attempt to restore the collection on Alma CZ was incomplete  

    •  After thorough analysis, Ex Libris attempt to automatically restore institution data was withdrawn due to solution complexity and high risk  

    • Insufficient communication on status and timeline

    • Time to resolution does not meet Ex Libris targets   

     

    Technical Action Items and Preventive Measures

    Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

    • Enhance prevention capabilities to perform actions outside of the routine procedure to significantly reduce the risk of manual error

      • We have reviewed and updated existing procedure for collection deletion (Done)

      • We have initiated a process to redefine roles and privileges for all teams (End of August 2021)

      • We will provide new users in all teams with corrected roles and privileges (End of August 2021)

    • Change management process analysis for high impact changes

      • Monitor and alert on high impact changes once occurs

      • Quick recovery procedure for deleted content

      • Solution recommendations should be ready by end of September 2021

    • Deletion procedure

      • We have refreshed the Deletion procedure for all team members (Done)

      • We have ensured the procedure is well documented and available for all team members (Done)

      • We have ensured the procedure is part of new employee onboarding (Done)

      • We have scheduled quarterly knowledge refresh with all team members (Done)

      • Plan to revisit deletion procedures (manual and automated) overall including portfolios (End of September 2021)

    • Enhance communication policy

      • Alma October 2021 release will include a new feature called ‘Alma Announcement’. This new feature allows Ex Libris to update users about of high impact issues, unusual occurrences, or major feature rollouts, as well as ad hoc updates and announcements.Content Operations will use this feature to update customers directly on ad-hoc issues, their status, and resolutions. Using Alma Announcement will ensure timely communication that is directed to relevant staff at the effected institutions. (October)

     

    Conclusion

    Ex Libris treats this incident as high priority with evaluation, assessment, mitigation processes and lessons learned. We are determined to improve the level of CZ quality and the value it provides to our customers.

     

    • Was this article helpful?