Connect Layer Troubleshooting

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Product: campusM
Operating system: iOS, Android

This article covers the list of troubleshooting steps you may carry out on the Tomcat server (Connect Layer) in order to attempt to identify where an issue may be occurring. This article is relevant when you are experiencing one or both of the following symptoms:

Unable to log into the App / Web Portal via an authenticate profile (consistently or intermittently).
Integrations (e.g. Timetable, Email, Library ...) intermittently not responding

Step 1: Identify whether the WSDL is publicly accessible

The first step of troubleshooting would be to identify whether the WSDL is publicly accessible. To do this you would need to enter the following URL in a web browser:

https://<basic_auth_username>:<basic_auth_password>@<hostname>/<war_filename>/services/CampusMUniversityService?wsdl

The basic authentication credentials can be located on the Tomcat server at $TOMCAT_HOME/conf/tomcat-users.xml

Alternatively, you may also gather basic auth details via the App Manager: App Manager > App Settings > Application Servers > (Select relevant server e.g. Production / Live)

Step 2: Check the SSL certificate

Identify whether there are any issues with the SSL certificate installation on the Tomcat server by using the website below:

https://www.sslshopper.com/ssl-checker.html

Enter WSDL URL into sslshopper: https://<hostname>/<war_filename>/services/CampusMUniversityService?wsdl

Step 3: Check Connect layer

On the Connect Layer server, identify whether Tomcat is up and running by using the following command:

ps –ef | grep tomcat

If the output of this command contains the whole path to the tomcat folder then it is running.

Alternatively, depending on if Tomcat has been configured to run as a service, you can use the following command:

service tomcat status

Step 4: Identify whether the WSDL is accessible locally

On the server itself, run the following curl command (or enter into a browser if you have a desktop session to the server):

http://<basic_auth_username>:<basic_auth_password>@<hostname>:<port_8080_depends_on_setup>/<war_file_name>/services/CampusMUniversityService?wsdl

If you are able to see a WSDL response after running the above URL but not able to access the WSDL publicly please check your load balancer / firewall configuration has not changed.

Step 5: Check the Tomcat logs

Check the catalina.out and localhost logs for the following (errors which would indicate a system down):

java.io.FileNotFoundException Too many open files
Exceeded MaxThreads exception

Step 6: Identify whether the connections have built up on the Connect layer

On the server, run the following command:

netstat -a

If you suspect a specific data source hostname is causing the build-up you can filter traffic using the following command:

netstat -a | grep ‘hostname’

Step 7: Diagnose which services are responding slowly

Run manual curl requests for the pertinent integrations to determine whether the API or database connections are responding slowly.

See https://knowledge.exlibrisgroup.com/campusM/Knowledge_Articles/Creating_a_Curl_Requests_for_Testing

Step 8: Check ALL Tomcat nodes behind the load balancer

If you are experiencing intermittent outages this could be pointing to an issue with one or more of your Connect layer nodes not responding correctly. Please repeat the checks from from Step 3 onwards to diagnose if a particular tomcat node is at fault.

Firewall / Load Balancer

Please inform us beforehand if you are going to make any changes to your networking and firewall rules. If the changes have been misconfigured, it may result in the application becoming unresponsive. For example, there may be an issue with routing between the load balancer and Tomcat server or between the Tomcat server and the backend systems with which campusM communicates and integrates.

Article last edited: DD-Mmm-YYYY