CampusM Uptime Report for CampusM AP01 Instance (APAC) - Q1 2018

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Uptime Quarterly Reports are published each quarter to provide a comprehensive view of our uptime performance as measured over the last three (3) months and over the last twelve (12) months. The report measures our performance as defined in the Service Level Agreement.

This document details the uptime report for the CampusM AP01 in APAC.

The uptime measured on a rolling 12 month basis April 2017 - March 2018 is 99.93%

The uptime measured on a rolling 3 month basis Jan 2018 - March 2018 is 99.88%

Please note our System Uptime Status page, which allows our customers to view the current status of their instance/environment at any time, using the following link: http://status.exlibrisgroup.com

Unscheduled downtime incidents in Q1 2018

Date	Start time [SGT]	End time [SGT]	Duration (minutes)	Description
February 27, 2018	07:31 AM	09:29 AM	118	During the first event on February 27th, the Ex Libris engineers discovered the root cause for the service disruption was a combination of two issues: Firewall issue Pileup on the thread pool causing its exhaustion Both issues caused slowness in outgoing traffic that led to performance degradation and eventually complete system disruption. Following this event, Ex Libris implemented the following: Disabling the problematic configuration in the firewall that initiated the bug Removing the problematic configuration across all firewall devices Triple the application thread pool The second event on March 5th indicated that the previous thread pool change did not completely resolve the issue, but only delayed the thread pileup and hanging threads and Ex Libris manually removed the hanging thread to restore the system back to its normal service. Following this event, Ex Libris implemented the following: Adding a safety system that identifies and removes hanging threads During the third event on March 6th, while there was no complete service disruption (system down), the application still suffered performance degradation. The Ex Libris engineers determined that the safety system introduced created in cases false-positive alerts causing removal of long sessions that were not actually hanging, just taking longer. Following this event, Ex Libris implemented the following: Adjust and remove all false-positive alerts and triggers by fine tuning the safety system to identify and remove only hanging sessions
March 05, 2018	09:45 AM	10:17 AM	32

Date

Start time

[SGT]

End time

[SGT]

Duration
(minutes)

Description

February 27, 2018

07:31 AM

09:29 AM

118

During the first event on February 27th, the Ex Libris engineers discovered the root cause for the service disruption was a combination of two issues:

Firewall issue
Pileup on the thread pool causing its exhaustion

Both issues caused slowness in outgoing traffic that led to performance degradation and eventually complete system disruption.
Following this event, Ex Libris implemented the following:

Disabling the problematic configuration in the firewall that initiated the bug
Removing the problematic configuration across all firewall devices
Triple the application thread pool

The second event on March 5th indicated that the previous thread pool change did not completely resolve the issue, but only delayed the thread pileup and hanging threads and Ex Libris manually removed the hanging thread to restore the system back to its normal service.
Following this event, Ex Libris implemented the following:

Adding a safety system that identifies and removes hanging threads

During the third event on March 6th, while there was no complete service disruption (system down), the application still suffered performance degradation. The Ex Libris engineers determined that the safety system introduced created in cases false-positive alerts causing removal of long sessions that were not actually hanging, just taking longer.
Following this event, Ex Libris implemented the following:

Adjust and remove all false-positive alerts and triggers by fine tuning the safety system to identify and remove only hanging sessions

March 05, 2018

09:45 AM

10:17 AM

Scheduled downtimes during maintenance windows in Q1 2018

Start Date	Start Time [SGT]	End Time [SGT]	Duration (Minutes)
March 17, 2018	22:00 PM	00:00 AM	240

Start Date

Start Time

[SGT]

End Time

[SGT]

Duration (Minutes)

March 17, 2018

22:00 PM

00:00 AM

240

Total unscheduled downtime minutes during past 12 months

Quarter	Total unscheduled downtime in Quarter (minutes)
Q2 2017	180
Q3 2017	0
Q4 2017	0
Q1 2018	150

How is Uptime Calculated?

The uptime calculation is based on the following calculation (as defined in the Service Level Agreement):

"Uptime" means the total period in minutes during which the Service is available for access and use during this period.

“Uptime Percentage” means Uptime expressed as a percentage, calculated in accordance with the following formula:

Uptime Percentage = X /(Y–Z) × 100

Where:

X = Uptime

Y = Last 12 months period

Z = The duration (in minutes) of any SLA Exclusions* during these 12 months

*SLA exclusions are defined in the contract SLA (e.g. Scheduled maintenance, etc.)

Further Information

If you have any queries on the information within this report please contact Ex Libris through your usual channel.