Primo MT EU Data Center RCA May 15 2015

Introduction

This document serves as a Root Cause Analysis for the Primo service interruption experienced by Ex Libris customers on May 15, 2015.

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Event Timeline

Service interruption was experienced by Ex Libris customers served by the Europe Data Center during the following hours: 

May 15, 2015 from 11:23AM until 6:58PM Central European Time (CET).

During the event, the service was unavailable for the environment.

Root Cause Analysis

Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

• The service interruption was a result of an application defect that impacted CPU consumption on search engine.
• The defect was triggered due to configuration changes related to search scope.

• Technical Action Items and Preventive Measures

Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

• Ex Libris' Engineers revert the configuration in order to recover the service.
• Permanent fix for the issue is being worked on and is planned for July 2015 release.