EU02 - RCA - April 9, 2024

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Introduction

This document serves as a Root Cause Analysis for the service interruption experienced by Ex Libris customers.
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Effected Products

HEP EU02

Event Timeline

Service interruption was experienced by Ex Libris customers served by the Alma EU02 instance during the following date and time:

April 9, 2024, from 06:28 until 08:02 Frankfurt time (94 min).

During this time frame login via SSO was not working however users which were already logged-in were not affected, and Primo was available for guest users.

Root Cause Analysis

Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:

After the April release, Ex Libris engineers initiated the phased rollout of security upgrades designed to enhance the browser's detection and prevention of cross-site scripting and other malicious activities. These enhancements are detailed here. However, during the phased implementation of Content Security Policy Level 2, an unforeseen bug emerged that obstructed the display of the institution's SSO login page in the browser.

This enhancement was deployed first on EU02 and was rollbacked immediately as detected.

Technical Action Items and Preventive Measures

Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:

The bug has been fixed
We’ve added more testing to avoid such bugs in the future.
We plan to improved our internal communication processes regarding, ensuring that all relevant parties are informed when such features are being rolled out.

Customer Communication

ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/

These updates are automatically sent as emails to registered customers.