Primo MT NA01 RCA September 8 and 10 and 25 2015
Confidential Information, Disclaimer and Trade Marks
Introduction
This document serves as a Root Cause Analysis for the Primo service interruption experienced by Ex Libris customers on September 8 and 10 and 25, 2015.
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
Service interruption was experienced by Ex Libris customers served by the Primo MT NA01 instance at the North America Data Center during the following hours:
September 08, 2015 from 01:16 PM until 01:44 pm U.S. Central Time.
September 10, 2015 from 01:44 pm until 02:46 pm U.S. Central Time.
September 25, 2015 from 03:05 pm until 03:30 pm U.S. Central Time.
During the events, Primo environment was not accessible.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:
An unexpected load on search engines and on the Front Ends took place. The extensive load on the system elements caused slowness and in extreme cases caused the system to build a high queue that prevented it entirely from operating.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- To accommodate the load, the environment had been updated with the following :
o Additional computing unit (CPU) were added to the Search Engines
o A code correction was added to the environment to allow better performance of the front ends. - Monitoring had been set to have a more sensitive thresholds to allow identification of possible slowness earlier
Customer Communication
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/