Process J000 died, see its trace file;... kkjcre1p: unable to spawn jobq slave process

Description

The Oracle ALERT LOG has repeated occurrencs of these errors:

Process J000 died, see its trace file
 Thu Jan 05 16:42:09 2017
 kkjcre1p: unable to spawn jobq slave process
 Thu Jan 05 16:42:09 2017
 Errors in file /exlibris/app/oracle/diag/rdbms/aleph23/aleph23/trace/aleph23_cjq
 0_16604.trc:
 Process J000 died, see its trace file
 <etc.>

Resolution

This is an Ex Libris hosted server.  The DBA Team diagnosed the problem as follows:

Symptoms first appeared on Tue Jan 03 15:57:52 2017 and then aggregated on Wed Jan 04 10:18:06 2017 and ever since. process m000 died and then j000 could not be spawned. This looks like a resource starvation issue.

oracle@aio0103(aleph23)rdbms/aleph23$free -m
 total used free shared buffers cached
 Mem: 15948 15669 279 871 7 1097
 -/+ buffers/cache: 14565 1383
 Swap: 1023 877 146

Further check the system settings found that /etc/sysctl.conf was not properly configured based on the physical RAM and the server type (Standalone with Oracle and Aleph application) 

[root@aio0103 ~]# more /etc/sysctl.conf | grep shm
 kernel.shmmax = 68719476736
 kernel.shmall = 4294967296

For this I have adjusted the above and took effect:
[root@aio0103 ~]# sysctl -a | grep shm
 kernel.shmmax = 8589934592
 kernel.shmall = 2097152

Also updated /etc/security/limits.d/exlibris.conf with these 2 lines but need to wait till next DB start to take effect.
 @dba soft memlock 8388608
 @dba hard memlock 8388608

One change made in the database is increase PGA target size from 97M to 160M - for the minimum.
 sys@ALEPH23> alter system set pga_aggregate_target=160M scope=both;
 System altered.

If symptom continues, please consider adding more RAM for this server (current: 15G) and would be nice to have 24G).

[Note:  Taken from Jira DBA-1385 -- which is closed.]  

Article last edited: 14-Mar-2017