Some SIPS in the submission job hang for several hours unprocessed
- Article Type: General
- Product: Rosetta
Problem Symptoms:
1. Some SIPS in the submission job are processed, but other hang for several hours.
2. Processing errors (e.g.):
SIP 7682 deposit processing took long time (~7.5 hours):
ERROR | wrapper | 2013/09/26 08:51:21 | Shutdown failed: Timed out waiting for the JVM to terminate.
ERROR | wrapper | 2013/09/26 08:51:21 | JVM did not exit on request, terminated
3. Workers are thrown out:
2013-09-18 03:33:01,185 ERROR [org.hibernate.util.JDBCExceptionReporter] (DEPOSIT_WORK_QUEUE Queue Job Receiver 358) [] ORA-24010: QUEUE V2GR_ROS00.DEPOSIT_WORK_QUEUE does not exist
Cause:
Not enough jboss.maxmemory in global.properties
Resolution:
Update global.properties file to increase the maximum allocated memory from jboss.maxmemory=4000 to jboss.maxmemory=6000
1. dps_conf
2. global.properties [e.g. increase jboss.maxmemory=6000]
3. dps_stop
4. Run set_globals
>dps_bin
>set_globals.sh
5. dps_start
Additional Information
If the above doesn?t solve the issue it?s possible that the sshd process is still consuming lots of CPU.
In this case check the Heap Dump file from the day the SIP ran to see how many GB were used.
It may be possible that ~1.2GB (e.g.) is being used up by copies of the full METS.
The stack trace will likely show that the out of memory happens when the system replaces the file path in the METS, it has to duplicate the String (METS).
In this case the dps_log/server.log will show ?java.lang.string.replaceFirst? for the PERMANENT_WORK_QUEUE
Its possible to save a few hundred MBs by replacing the String in place.
However, Rosetta will still use most of the allowed 4G, so a larger METS might cause an out of memory exception, so consider increasing the memory (increased heap size) anyway.
Category: Deposit - Rosetta
Subject: Server Processing - Rosetta
- Article last edited: 12/4/2013