Legacy RefWorks - RCA – July 14, 2018
Confidential Information, Disclaimer and Trade Marks
Introduction
This document serves as a Root Cause Analysis for the Legacy RefWorks interruption experienced by Ex Libris customers on July 14th, 2018
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event and identify preventive measures Ex Libris is taking to avoid similar cases in future.
Event Timeline
Service interruption was experienced by Ex Libris customers served by Legacy RefWorks during the following hours:
July 14th , 2018 from 12.00 AM until July 17th 5.00 PM US Pacific time.
During the event, the following services encountered data lost issues:
- Legacy Refworks
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:
As part of ongoing upgrades to our legacy RefWorks servers, we are migrating our data stores from old to new servers. One of those migrations happened on Saturday July 14th. A few days (Tuesday July 17th) after the migration, we received notification that some users' citations were missing. This was due to the migrating data being out of sync between the servers
We decided to roll back the migration and revert the changes to a known good state. This resulted in data changes made during this period (Saturday July 14th – Tuesday July 17th) to be lost.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- Suspend all work related to data migration immediately
- Review all our migration steps to ensure data integrity during migration
Customer Communication
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/
These updates are automatically sent as emails to registered customers.
Publication History
| Date | Publication History |
|---|---|
| July 26, 2018 | Initial Publication |