Confidential Information, Disclaimer and Trade Marks
This document serves as a Root Cause Analysis for the Primo Central service interruption experienced by Ex Libris customers.
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Service interruption was experienced by Ex Libris customers served by the Primo Central during the following hours:
February 2nd, 2018 from 2:59 PM until 1:11 AM UTC time.
During the event, the service was unavailable.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:
Due to a malfunction in the configuration tables update, two tables were updated with invalid values (null).
As a result, any query against these tables got stuck until there was no thread left in the pool to serve the end user searches.
Identifying the cause of the issue was complicated because there were many different symptoms that caused different components of the system to behave incorrectly. In addition, during the course of the event, the thread pool was cleared and the system recovered and provided service – until the thread pool filled up once again and the service was unavailable. This behavior made us believe during the event that we had fixed the problem, although after a short period of time, we understood that we had solved only the symptoms and not the root cause.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- During the event, the malfunction in the tables was fixed by importing them from a backup system
- Development will create tools for sensitive table updates, which will include all required validations
- Ex Libris will add a special analysis procedure that should reduce the 'time to resolution'
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/
These updates are automatically sent as emails to registered customers.
|February 13, 2018||Initial Publication|