RCA: JSTOR Incorrect Linking Syntax, August 2021
Introduction
This document serves as a Root Cause Analysis (RCA) for the reported incident on July 27, 2021. The goal of this document is to share our findings regarding the reported incident, specify the root cause analysis, handling of the event, outline actions to be taken for mitigation, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
| Date | Activity |
| July 26, 2021 | 31 of JSTOR collections were released with incorrect linking syntax configuration, resulting in invalid links and failure to access JSTOR content on the provider's site |
| July 26, 2021 | The incident was reported on the Alma Listserv as well as via support cases demonstrating the incorrect linking configuration and resulted impact |
| July 27, 2021 | Analysis completed followed by code fix introduction |
| July 27, 2021 | Ex Libris communicated update to the Alma Listserv, informing that the issue was resolved and to will be released on August 1st |
| August 2, 2021 | Release was applied on Community Zone triggering "Synchronize Changes from CZ" process |
| August 2, 2021 | Ex Libris communicated update to the Alma Listserv, informing that the process "Synchronize Changes from CZ" was initiated and expected to complete within 3 days |
| August 9, 2021 | Process completion and fix is fully applied |
Root Cause Analysis
Ex Libris investigated this event to determine the impact and root cause analysis with the following results: as part of operational maintenance work on JSTOR content, incorrect changes to linking syntax were made by Content Operations resulting in invalid links.
Findings
Ex Libris investigated this event and determined the following:
- Changes to linking syntax configuration made by Content Operations, using inappropriate procedure for the change performed
- Automated QA validation did not detect the issue and therefore no alert notification was sent as expected in such scenario
- Initial communication updating on expected resolution timeline was missing the duration of the "Synchronize Changes from CZ" process
- Long duration took the fix to be fully applied
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- Enhance prevention capabilities to significantly reduce the risk of human error
- Bug detected in the automated QA routine was fixed, tested, and applied (Done)
- Enhanced auditing capabilities to better track and control configuration changes (Done)
- Add new automated QA routine to ensure validation of critical data elements (End of September)
- Overall review of all automatic QA routines to ensure they perform as expected (End of September)
- Reduce duration for applying fixes for severe disruption of service
- Initiated a discussion and analysis for a quick recovery procedure
- Solution recommendations should be ready by September
- Manual checks procedure
- Verified procedure overview is part of new employee onboarding process (Done)
- Scheduled quarterly knowledge refresh with all team members (Done)
- Review manual checks procedure and extended usage for additional scenarios (End of August)
- Enhance communication policy
- Factored into consideration duration of Synchronize Changes from CZ process when communicating expected resolution time (Done)
- Alma October 2021 release will include a new feature called ‘Alma Announcement’. This new feature allows Ex Libris to update users about of high impact issues, unusual occurrences, or major feature rollouts, as well as ad hoc updates and announcements. Content Operations will use this feature to update customers directly on ad-hoc issues, their status, and resolutions. Using Alma Announcement will ensure timely communication that is directed to relevant staff at the effected institutions. (October)
Conclusion
Ex Libris treats the incident in high priority with evaluation, assessment, and mitigation processes and lessons learned. We are determined to improve the level of CZ quality and the value it provides to our customers.