Primo MT EU01 - BO impact RCA - September 25-29, 2016
Confidential Information, Disclaimer and Trade Marks
Introduction
This document serves as a Root Cause Analysis for the Primo service interruption experienced by Ex Libris customers onSeptember 25 - 29, 2016
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
Service interruption was experienced by Ex Libris customers served by thePrimo MT EU01 instance at the Europe Data Center during the following hours:
September 25h though 29th daily from 10:00 AM until 4:00PM Amsterdam time zone
During the timeframes listed, the environment back office was Occasionally unavailable for a duration of 15 minutes.
in addition, from time to time RTA (real time availability) was slow or not responding, this had impacted the availability status of an item (a book) in the FE search result page.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:
A configuration setting in the back office was indented to be set to a value that does not match the needed configuration of the system at this point.
The configuration has impacted the available memory cache. The incorrect value had impacted the ability of the back office to function as expected.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- A configuration change has been applied to the environment. The configuration change requiered a code change. Once verified in a testing environment, the correction has been applied to the environment.
Customer Communication
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/