CampusM AP01 - RCA - January 2 and January 3, 2022
Introduction
This document serves as a Root Cause Analysis for the service interruption experienced by Ex Libris customers on CampusM AP01 instance
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
A service interruption was experienced by Ex Libris customers served by the CampusM AP01 instance at the Singapore Data Center during the following time periods:
-
January 2, 2022 from 14:50 until 15:40 Singapore time
-
January 3, 2022 from 08:52 until 09:30 Singapore time
During this time there was a service disruption.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause with the following results:
We at Ex Libris are constantly trying to improve our network environment and to provide a better and optimal services to our customers. As part of these improvements, we have recently redesigned and upgraded part of the Singapore DC network and are in process of transitioning to advanced technologies.
We chose to do this activity during the holidays and in between semesters in order to minimize the impact on our customers.
After an in-depth investigation done by Ex Libris engineers to understand the root cause of the recent events, we found that the latest events were a result of an unexpected load in one of the network components.
On December 20th, after the first event a malfunction issue was identified in a specific network device that was addressed together with the vendor, we enabled the redundant network device to bring the service back up again Even after the service was up and running again, we decided to continue monitoring the environment to make sure the issue won't occur again.
After the January 3, 2022 event we found that the malfunction was only one symptom and the source of the problem was a load created on the network which impacted the network devices. It was discovered that before each service disruption, there was a huge load peak of traffic from a specific component which caused the service disruption. The root cause of the issue was fixed and, in addition, some protections were applied to reduce the chances of this issue reoccurring.
We will continue monitoring the environment to make sure the issue is resolved.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- The root of the load was fixed and optimized
- Improving and optimizing the network environment
- Reduce the buffering queue to minimize the chance of traffic spikes
- Ex Libris engineers are working with the Vendor support to optimize the network device configuration
- Automatic fail over to the redundant network was reviewed and improved
Customer Communication
Ex Libris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/
These updates are automatically sent as emails to registered customers.