Ex Libris Cloud Services BCP

Definitions

Business Continuity Plan (BCP) - a document describing a set of arrangements, resources, and sufficient procedures that enable an organization to respond to a disaster and resume its critical operations within pre-defined time frame without incurring unacceptable operational impacts.

Disaster Recovery Plan (DRP) – a technical document describing the processes, policies, and procedures related to implementing precautionary measures and preparing for the recovery, continuation, or resumption of services in the event a catastrophic event occurs.

Disaster – a sudden, unplanned calamitous event that causes a complete loss or significant disruption in customer’s mission critical services. The primary objective of the plan is to minimize the risk of low-level events and minimize the impact of major high-level events.

Business Continuity Plan Objectives

The principal objective of the business continuity plan is to develop, test and document a well-structured and easily understood plan which will help Ex Libris cloud services recover as quickly and effectively as possible from an unforeseen disaster or emergency which interrupts Ex Libris cloud services and business operations. 

The objectives of this document are:

• Develop a Business Continuity Plan structure for managing a disaster that affects the Ex Libris cloud services.
• Document critical information and procedures as required for the implementation of the Business Continuity Plan.
• Present a course of action for restoring critical cloud services within a minimum number of days of initiation of the plan
• Provide guidelines with an escalation plan for a disaster declaration that will result in the execution of this Business Continuity Plan.
• Describe an organizational structure for carrying out the plan and ensure that all employees fully understand their duties in implementing such a plan.
• Ensure an orderly recovery after a disaster occurs, minimizing risk of lost production or services

Business Continuity Plan Policy

Ex Libris management has approved the following policy statement: 

• The company shall develop a comprehensive Business Continuity Plan. 
• A formal risk assessment shall be undertaken to determine the requirements for the Business Continuity Plan. 
• The Business Continuity Plan should cover all essential and critical infrastructure elements, systems and services, in accordance with key business activities. 
• The Business Continuity Plan should be periodically tested to ensure that it can be implemented in emergency situations and that the management and staff understand how it is to be executed. 
• All Ex Libris cloud services staff must be made aware of the Business Continuity Plan and their own respective roles. 
• The Business Continuity Plan is to be kept up to date to take into account changing circumstances. 

Assumptions of the Plan

The Business Continuity Plan has been developed and maintained based on the following assumptions:

• This document plans for the major/worst case disaster. However, if an outage of services occurs to a lesser degree, this plan will cover the incident.
• The situation that caused the disaster is localized to one of Ex Libris data center locations (Chicago, Amsterdam or Singapore).
• Ex Libris cloud services are utilizing a co-location agreement with a leading data center facilities provider (Equinix). The facilities provider will provide the baseline infrastructure needs in the event of an emergency. 

BCP Team Descriptions and Responsibilities

• BCP Management Team - Responsible for the overall direction, decision-making, and approvals required to implement the Business Continuity Plan. The team is comprised of the Ex Libris Chief Operating Officer (COO), and cloud services directors who are responsible for leadership within their respective areas. 
• Business Continuity Coordinator (BCC) – A member of the BCP Management Team with responsibility for the development, coordination, training, testing and implementation of the Business Continuity Plan.
• BCP Team Leaders - Responsible for carrying out the tasks and provisions of the Business Continuity Plan including assigning tasks to staff, obtaining offsite data backups, contacting vendors, monitoring work progress and reporting the status to the BCP Management Team.  The team is comprised of all Ex Libris cloud services team leaders and managers.
• Emergency Operations Center (EOC) – A location established by the BCP Management Team for central coordination during the recovery efforts. This location will typically be established at Ex Libris group headquarter offices.

Preventive Measures

Redundancy Strategies

The following are the redundancy strategies available in the Ex Libris Group Cloud Services environment:

• Active/Active Load balancing - Traffic is split evenly between 2 or more servers for critical servers. Upon failure of one of the servers, traffic is shifted automatically and seamlessly to the working server.
• Active/Active Anti-virus – traffic is split between 2 or more anti-virus scan devices for balancing. The traffic is shifted automatically and seamlessly to the working server.
• Active/Passive Network – The External Network comes from a primary dedicated line with automatic failover into a secondary dedicated line. The internal network is redundant at all layers.
• Active/Passive Multiply ISP providers – Multi ISP connected to the primary site using Managed Internet Route Optimizer with automatic fail over.
• Active/Passive Firewall – Network traffic uses primary Firewall with automatic failover to a secondary Firewall.
• Active/Active Storage – Data is split between 2 or more storage controllers for balancing and take over in case of fail.
• Available on-site servers’ capacity – Also referred to as “Onsite Cold Equipment”. In the event a catastrophic hardware failure takes place and cannot be resolved in a timely manner, the virtualized instance of the customer’s hosted environment will be mounted from an existing standby server that has been preloaded with the appropriate OS and administrative applications. In case needed, the customer’s data restoration would then be performed from onsite or from offsite backups.

Backup Strategies

Ex Libris has a well-developed backup plan consisting of multiple daily snapshots including a full daily backup. The backups are made to a separate set of disks which offers a much more reliable fast retrieve backup media, and is stored at the site and in a remote secured location over a private dedicated fast secured line. This guarantees that at any point in time, in case of a disaster, Ex Libris holds copies of the data onsite and in a remote and secured disk backup.  On a regular basis, Ex Libris performs a system backup to back up application files, database files, and storage files. The privacy controls in practice at the company apply as well to all backup files. All backup files are subject to the privacy controls in practice at Ex Libris. The restore procedures are tested on an ongoing basis to ensure rapid restoration in case of data loss.

•  On-site backup – Full backup for OS platform, application, and customer data are performed at least daily (multiple snapshots during the day for critical services/systems) using storage snapshot technology. The backups are kept for one week on-site at a separated set of disks. The snapshots are automatically mounted with specific access restriction values seen by the operating system in a special set of directories allowing for an easy and immediate restore at any time by Ex Libris authorized personnel.
• Off-site backup – Full backup for OS platform, application, and customer data are performed daily using snap mirror technology over a private dedicated fast secured network connection from the primary data centre to an off-site backup location using the same storage technology as the storage at the primary location. Subject to the privacy controls in practice at Ex Libris, Ex Libris maintains the off-site backup locations in the same territory (NA, EMEA, and APAC) as the primary locations with a sufficient best practice physical distance. The backups can be retrieved back to the main data centre 24/7 by Ex Libris authorized personnel. The backups are kept at the off-site backup managed locations.

Disaster Notification

Whoever detects the disaster should notify the Ex Libris Cloud Operations Director or the Ex Libris Cloud Engineering Director. In addition to providing some fault tolerance in the initial response, this role sharing enables effective use of shifts during the disaster recovery process.

The Ex Libris Cloud Operations Director or Ex Libris Cloud Engineering Director will establish the Emergency Operations Center (EOC) and monitor the evolving situation and, if appropriate, will then notify the BCP Management Team. The complete emergency contact list for the Ex Libris cloud services is included in Appendix A.                       

Normally, the facilities provider’s Network Operation Center (NOC) and/or the local law enforcement receive the initial alarm notice through their monitoring system capabilities. If the emergency does not activate a normal alarm system, these two parties should immediately be notified by the Ex Libris Cloud Operations Director or Ex Libris Cloud Engineering Director.

Determine Personnel Status

One of the Ex Libris Cloud Operations Director or Ex Libris Cloud Engineering Director’s important early duties is to determine the status of personnel working at the time of the disaster. Safety personnel on site after the disaster will affect any rescues or first aid necessary to people caught in the disaster. However, the director should produce a list of the able-bodied people who will be available to aid in the recovery process. Taking care of people is a very important task and should receive the highest priority immediately following the disaster. While we will have a huge technical task of restoring computer and network operations ahead of us, we can't lose sight of the human interest at stake.

Damage Assessment

To determine how the business continuity plan will be implemented following a sever disruption to service, it is essential to assess the nature and extent of the damage to the system.

Once the appropriate facilities provider’s contacts have been notified, the BCP Team Leaders will be contacted so that a preliminary determination can be made whether an onsite damage assessment is required or feasible. 

Damage assessment is intended to establish the extent of damage to mission critical computing devices and the facility that houses them as quickly as the given conditions permit, with personnel safety remaining the highest priority.

The following areas should be addressed:

• Cause of the disaster or disruption
• Potential for additional disruptions or damage
• Area affected by the disaster
• Status of physical infrastructure (e.g., structural integrity of data center, condition of electric power, telecommunications, and heating and ventilation/environmental conditions)
• Inventory and functional condition of Ex Libris equipment
• Type of damage to equipment or data (e.g., water, fire, physical impact, electrical surge)
• Estimated time to restore normal services

Disaster Determination

The key outcome from the Damage Assessment process is to determine the severity of the disaster and to estimate the amount of time required to restore the Ex Libris cloud services back to normal operations.

The Ex Libris cloud services group has classified disasters and emergencies into the following three levels – minor, major and catastrophic:

• Minor Disaster - A minor disaster will be characterized by an expected downtime of no more than 48 hours. Damage can be to hardware, software, and/or operating environment. Ex Libris cloud services could be restored to normal operations at the primary site and repairs can be started as soon as possible:
• Major Disaster - A major disaster will be characterized by an expected downtime of more than 48 hours but less than 7 days. A major disaster will normally have extensive damage to system hardware, software, networks, and/or operating environment. Ex Libris cloud services could be restored to normal operation with the assistance of certain recovery teams who will be called to direct restoration of normal operations at the primary site.
• Catastrophic Disaster - A catastrophic disaster will be characterized by expected downtime of greater than 7 days. The facility is destroyed to the extent that an alternate facility must be used. Damage to the system hardware, software, and/or operating environment requires total replacement / renovation of all impacted systems. The implementation of the Disaster Recovery Plan in a remote recovery site is required to restore Ex Libris cloud services to normal operation.

DR Strategies for Minor & Major Disasters

DR Strategy for Catastrophic Disaster

This section details the activities undertaken for restoration of cloud services due to catastrophic disaster at the facilities level:

• The BCP Team Leaders, in conjunction with the facilities provider, will perform an assessment of the extent of the facilities loss.
• In the event the primary facility will be out of service for an extended period of time (longer than 7 days), customer notification would be updated at Ex Libris Status Portal.
• An assessment of the viability of retrieving Ex Libris owned equipment from the disaster site will be performed.
• Simultaneously, a predetermined alternate facilities provider will be notified and engaged.
• Any hardware that can be retrieved will be reclaimed by Ex Libris for use in the designated recovery site.
• Procurement activities for replacement equipment to replace non-retrievable equipment will be initiated.
• A plan and timeline for implementation of the recovery site will be finalized and distributed to the Ex Libris customer stakeholders.
• The implementation plan will be executed.
• The Ex Libris and customer stakeholders will be notified of resumption of service at the alternative hosting facility.

Locate and Salvage Data and Equipment

Early efforts are targeted at protecting and preserving the salvageable computer and networking equipment (any hardware that can be retrieved will be reclaimed by Ex Libris for use in the Recovery Site). In particular, any backup storage media (hard drives, backup tapes) are identified and either protected from the elements or removed to a clean, dry environment away from the disaster site.

Designate Recovery Site

An inspection of the data center and telecommunication closets scene is done by BCP Team Leaders  to estimate the amount of time required to put the salvageable equipment back into working order, providing there are adequate facilities to work with. A decision is then made whether to use a designated remote location where computing and networking capabilities can be temporarily restored until the primary site is available. If estimates from this process indicate that recovery at the original site will require more than 7 days, migration to the remote recovery site is initiated by notifying and engaging with a predetermined alternate facilities provider.

Systems and Data Recovery

Ex Libris will use salvageable equipment if possible. If equipment is damaged beyond repair, the Ex Libris Procurement Team will work with our vendors (listed in Appendix B) to expedite replacement equipment.

Data recovery is performed using backups retrieved from the disaster site from the offsite backup locations. Backups can take on various forms of media including hard drives and magnetic tapes. After identifying salvageable equipment, early data recovery efforts first focus on restoring the operating system(s) for each system. Next, mission critical system data is restored. After system data is restored, individual customer data is restored.

Move Back to Restored Primary Sites

If the recovery process has taken place at an alternate remote site, physical restoration of the primary data center will have begun. When the data center is ready for occupancy, the systems assembled at the alternate remote site are to be moved back to their permanent home.

This section outlines the logistical planning to be undertaken in order to transfer services back to the primary hosting site in the event that services needed to be moved to a designated recovery site to mitigate a catastrophic site disaster.

Operational readiness at the original primary site must be verified prior to the execution of this step. Once verified, the following steps will be initiated:

• Migration schedule availability at the facilities supplier
• Migration schedule availability with the customer
• Cloud Services preparation for migration
• Migration execution
• Systems Acceptance Test (SAT) and User Acceptance Test (UAT) completion
• Notice to Operations of migration

Business Continuity Coordinator (BCC)

The Business Continuity Coordinator has overall responsibility for the design, development, coordination, implementation, administration, training, awareness programs, and maintenance of the Business Continuity Plan. The BCC will follow the best practices established by the DRI International Professional Practices for Business Continuity Planners (see www.drii.org for the latest version).

In accordance with the DRII Professional Practices, the Business Continuity Coordinator has the following responsibilities:

• Provide BCP project coordination and management.
• Perform risk evaluation and mitigation as required.
• Develop and obtain approval for the Business Continuity Strategy (ies).
• Develop and implement the Business Continuity Plan.
• Develop, maintain, coordinate, exercise, and evaluate the BCP.

Business Continuity Plan Maintenance

The plan will be annually evaluated and updated. All portions of the plan will be reviewed by the Ex Libris COO and Cloud Engineering Director. If it is deemed that portions need to changed or be rewritten or reviewed by other cloud teams, the COO will assign that task to the appropriate team. In addition the plan will be tested on a regular basis and any faults will be corrected. The BCP Management Team has the responsibility of overseeing the individual components and files and ensuring that they meet standards consistent with the rest of the plan.

Exercising (Testing) the BCP

The Business Continuity Coordinator is responsible for conducting periodic exercises of the Business Continuity Plan using different methodologies (structured walk-through exercise, tactical exercise, and technical exercise for the BCP Team Leaders) or combination of these methodologies. A report will be submitted to the BCP Management Team after the completion of the exercise that will detail the success and/or failure of the exercise. A discussion surrounding any improvements to the plan will occur. Any revisions to the document based upon the results of the test and the discussion in management will be integrated into the document.