Introduction

This document serves as a Root Cause Analysis for the service interruption experienced by Ex Libris customers of the following Products and services:

• 360 services

• RapidILL

• Legacy RefWorks

• HEP – only institutions using the Ex Libris Identity Service

• AquaBrowser

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future

Event Timeline

Service interruption was experienced by 360, RapidILL, Legacy RefWorks, HEP (for a small number of institutions using the Ex Libris Identity Service) and AquaBrowser customers,

on July 19, 2024 from 05:30 until 21:00 UTC time. 

During this time frame customers experienced service interruption.

Root Cause Analysis

Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

The event impacted Ex Libris products and services that are using Windows servers that are running CrowdStrike Falcon.

These products and services were affected by the defective Falcon content update for Windows hosts. See here additional details.

Our engineers and subject experts  quickly understood the magnitude of the event and acted with sense of urgency to locate all affected services and apply relevant remediation steps, and communicated this to customers through our status page and support channels.

Technical Action Items and Preventive Measures

Ex Libris has taken the following measures to improve our event management in case of such wide scale outages.

• Internal procedures have been fine-tuned and updated.
• We have implemented improvements to internal monitoring tools, flows and processes.

Customer Communication

Ex Librisis committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/

These updates are automatically sent as emails to registered customers