Skip to main content
ExLibris
  • Subscribe by RSS
  • Ex Libris Knowledge Center

    Primo Central - RCA - February 2, 2018

    Confidential Information, Disclaimer and Trade Marks

    Introduction

    This document serves as a Root Cause Analysis for the Primo Central service interruption experienced by Ex Libris customers.

    The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

    Event Timeline

    Service interruption was experienced by Ex Libris customers served by the Primo Central during the following hours: 

    February 2nd, 2018 from 2:59 PM until 1:11 AM UTC time.

    During the event, the service was unavailable.

     

    Root Cause Analysis 

    Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

    Due to a malfunction in the configuration tables update, two tables were updated with invalid values (null).

    As a result, any query against these tables got stuck until there was no thread left in the pool to serve the end user searches.

    Identifying the cause of the issue was complicated because there were many different symptoms that caused different components of the system to behave incorrectly. In addition, during the course of the event, the thread pool was cleared and the system recovered and provided service – until the thread pool filled up once again and the service was unavailable. This behavior made us believe during the event that we had fixed the problem, although after a short period of time, we understood that we had solved only the symptoms and not the root cause.

     

    Technical Action Items and Preventive Measures

    Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

    • During the event, the malfunction in the tables was fixed by importing them from a backup system
    • Development will create tools for sensitive table updates, which will include all required validations
    • Ex Libris will add a special analysis procedure that should reduce the 'time to resolution'

    Customer Communication

    ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/

    These updates are automatically sent as emails to registered customers.

     

     

    Publication History

    Date Publication History
    February 13, 2018 Initial Publication

     

    • Was this article helpful?