This document serves as a Root Cause Analysis for the service interruption experienced by Ex Libris customers.
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Service interruption was experienced by Ex Libris customers served by the European Data Center during the following hours:
September 24, 2017 from 12:00 AM until 05:00 AM Amsterdam time.
During the event, all services were unavailable for the environment.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause with the following results:
During a storage infrastructure upgrade planned within a scheduled maintenance window time frame in order to reduce customer impact, which was prior tested in the lab and performed successfully on other DCs, a defect related to the storage vendor upgrade process caused the servers related to this infrastructure to lose their connection, and required that all virtual servers affected be rebooted.
The vendor has provided their initial RCA and identified that several parameters were changed erroneously during the automated upgrade process.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- The defect that was encountered was fixed by the Vendor under Ex Libris engineer supervision
- All production servers and the application were restarted by Ex Libris engineers within the maintenance window
- Ex Libris engineers are currently in contact with the Vendor to obtain the final RCA for this event and identified defect.
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/
These updates are automatically sent as emails to registered customers.
|October 1, 2017||Initial Publicaiton|