Primo MT EU01 RCA November 10, 2015

Introduction

This document serves as a Root Cause Analysis for the Primo service interruption experienced by Ex Libris customers on November 10, 2015.

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Event Timeline

Service interruption was experienced by Ex Libris customers served by the Primo MT EU01 instance at the Europe Data Center during the following hours: 

November 10,2015  from 3:54 PM until 4:20 PM Central Europe time zone (CET).

During this interruption, Primo was extremely slow and became not available for
customers on the European multitenant EU01 environment.

Root Cause Analysis

Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

The environment had received an extreme load. This extreme load was not identified to be
coming from a specific source. This situation is not blocked by the existing preventing
mechanism built into the application.

Technical Action Items and Preventive Measures

Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

• Following investigation done, Ex Libris had identified a code defect related to search scope filtering method. It had been identified that the existing logic is causing an increased consumption of CPU and it impacts the ability of the search engines to handle large number of queries. A code correction for this issue will be implemented in the environment in the next week.
• Ex Libris is working on a significant change to the application safety and blocking mechanism which will allow the application to block abnormal searches from the same session. This change will be implemented on the February release.
• Monitoring enhancements are being continuously done to allow quicker identification of the issues.

Customer Communication

ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/