Introduction

This document serves as a Root Cause Analysis for the North America  performance issues experienced by Ex Libris customers on  February 23-24, 2016

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Event Timeline

Periodic service disruption were experienced by Ex Libris customers on the North America Data center during the following hours: 

February 23, 2015 from 04:40 PM until 06:00 PM Chicago time zone

February 23, 2015 from 08:55 PM until 10:00 PM Chicago time zone

During these times slowness, timeout and disconnects were experienced.

Root Cause Analysis

Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

Ex Libris engineers found that the issue was caused by high amount packet loss on the Firewall. The packet loss had been identified to be related to a defect in the firewall version. 

Technical Action Items and Preventive Measures

Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

• During the event, Ex Libris cloud and Network engineers had worked closely with the Firewall vendor to identify the cause and potential solutions. Based on the commendation of the Firewall vendor, Ex Libris engineers have upgraded the firewall version - working hand in hand with the experts from the firewall vendor.

   

  • Following the upgrade close monitoring of all NA systems had been done and is still being conducted. No issues have been identified since the upgrade.

   

Customer Communication

ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/