Introduction

This document serves as a Root Cause Analysis for the CDI Search Update service interruptions experienced by Ex Libris customers during two windows: February 8 to 27, March 17 to 29.

The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the degradation, as well as preventive measures Ex Libris is taking to avoid similar cases in future.

Affected Products

• Summon, Primo, CDI

Background

CDI (Central Discovery Index) relies on the orchestration of multiple large scale technology systems. The overall CDI system accommodates the storage, merging and indexing of several billion records, and handles millions of content transactions (adds, updates, deletes) per day. The CDI Search Index is scheduled to be fully refreshed twice per week.

Event Timeline

Service disruption was experienced by Ex Libris CDI and Summon customers during the following timeframe:

• February 8 to 27
  • Data corruption detected preventing CDI Search Update process
  • Catalog additions were uninterrupted
  • Near Realtime Record Corrections continued to update CDI
• March 17 to 29
  • Storage hardware failure preventing CDI Search Update process
  • Catalog additions were uninterrupted
  • Near Realtime Record Corrections continued to update CDI

Root Cause Analysis

Ex Libris engineers detected data corruption during preparation to expand the content storage system. The system expansion was paused and data recovery process was initiated. Since the system data maintains multiple copies, the corruption did not cause data loss. 

The same content data storage system experienced hardware failure 2 weeks following the data recovery process completion. The hardware replacement and system recovery was initiated.
 

Technical Action Items and Preventive Measures

The CDI R&D team compiled the following list of improvements to be addressed, to both reduce the risk in future, and expedite the recovery to reduce missed CDI data updates:

• Significantly expand hardware and process monitoring and alerting for the systems that suffered degradation (data corruption and hardware failure). (80% Completed)
• Exercise failure scenarios in a Test environment to validate and improve recovery options. (Underway)
• Improve the CDI update process to allow index updates even when a server is offline due to hardware failure. (90% Completed)
• Significantly improve content data distribution and optimize cleanup to minimize the volume of data involved in system recovery. (Underway)

Customer Communication

ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/

These updates are automatically sent as emails to registered customers.