/exlibris file_system fills up

• Article Type: General
• Product: Aleph
• Product Version: 20

Description:
Some process is filling up the /exlibris file_system.

Resolution:
Indexing jobs, such as p_manage_01 or p_manage_02, can use large amounts of space. Is such a job running? (Check $alephe_scratch and the xxx01 $data_scratch to see.) The "top" command (or "prstat" where "top" is not available) gives you the 15 processes with the highest current cpu usage.

A runaway ue or batch process could be below this current threshold but still be accumulating CPU. If it does show up, the high value in its "TIME" column is a good clue of a problem. (You type "o" in the top display and then "time" to get the highest-cumulative-CPU-time processes.)

If df -k /aleph (or df -k /exlibris) shows 100% for the space used, you should do "cd $LOGDIR" and delete some old logs (to immediately make at least a little space). Then do "df -k " several times in a row. If the Available space is rapidly decreasing, you need to determine what is using the space.

There are at least two things you can do:

1) This series of commands:
cd /aleph [or cd /exlibris] du > /tmp/firstdu du > /tmp/seconddu cd /tmp diff firstdu seconddu

And then examine the diff output for cases of large directories with a large difference between the first instance and the second -- indicating a rapid increase. Then cd to the directory and do "ls -lrt" to check for large, rapidly increasing files.

2) ps -ef and look for ue processes with high CPU times. (The cpu time is the second time.) And then stop the process: You can try the normal util e menu option (such as util e/2). If that doesn't work, then you will need to stop it with the "kill" command. One repeated culprit has been runaway ue_01 processes -- occurring because the ue_01 has not been stopped by the nightly aleph_shutdown. (Note: The util c/1 display shows you when each process was started.) It's also possible that some batch job (rather than a ue process) has run away. To check this, check for large, increasing job logs in $alephe_scratch.