Primo MT EU01 - RCA - November 14,15,16 - 2016

Introduction

This document serves as a Root Cause Analysis for the Primo service interruption experienced by Ex Libris customers on November 14,15,16  - 2016

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Event Timeline

Service interruption was experienced by Ex Libris customers served by the Primo MT EU01 instance at the Eurpoe Data Center during the following hours: 

November 14, 2016 from 2:56 PM until 3:36 PM Amsterdam time zone

November 15, 2016 from 10:56 AM until 11:14 AM Amsterdam time zone

November 15, 2016 from 11:23 AM until 11:29 AM Amsterdam time zone

November 15, 2016 from 11:59 AM until 12:20 PM Amsterdam time zone

November 16, 2016 from 7:56 PM until 8:11 AM Amsterdam time zone

During the event, the service was unavailable for the environment. Events were identified by the 24x7 monitoring

Root Cause Analysis

Ex Libris continues to experience DoS attempts that aim to prevent legitimate users from accessing services by increasing the load on the system. With these types of attacks, it can be difficult to find the source, since they are initiated from multiple proxy sources.

Please note that there has been no impact to any data, and no security breach was identified!

Ex Libris' security controls continue to protect our systems and prevent attacks on a daily basis. Unfortunately we still experience changes in the attack patterns, which may result in short disruptions to system performance.

Technical Action Items and Preventive Measures

Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

The Ex Libris security team and our technical experts continue to investigate and research new solutions to further improve the security of our hosted solutions. This is accomplished by continuously monitoring our systems both internally and externally. We continue deploying and configuring to Ex Libris' Cloud – best-of-breed security technologies to address these attacks.

Ex Libris is working with cyber security experts to review our existing multi-layered security protection system and to ensure that we continue to adopt new protection measurements as they are introduced.

Customer Communication

ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/

These updates are automatically sent as emails to registered customers.

Publication History