Archive log files filling up /exlibris2 filesystem
- Article Type: General
- Product: Aleph
- Product Version: 20
Description:
1) OPAC down: error message:
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
2) Cannot connect to library when via clients
3) We *can* do "ssh"
Resolution:
df -k" showed that the /exlibris2 filesystem (occupied by oradata) was 100% full.
The following grep in abc10 $data_scratch shows that the abc10 ue_01 processed 120,313 z07's (indexing requests):
aleph@libcat(a20_1) ABC10> grep -c 'HANDLING DOC NO' run_e_01.8215
120313
and caused 219,000 z105's to be written, which were processed by the abc50 ue_11 daemon:
aleph@libcat(a20_1) ABC50> grep -c ABC10 run_e_11.1265
219139
This caused /exlibris2 to become filled up with /exlibris2/oradata/alephp/arch files at 22:10 last night. And, as can be seen in the /exlibris1/backup/logs/ora_cold_a1_Detail_110216_0015.log, the backup job failed with the same error:
specification does not match any archived log in the recovery catalog
...
Oracle error: handle_connection
ORA-00257: archiver error. Connect internal only, until freed.
I have done util w/5 to Toggle the System down. (Thus preventing the ue daemons and online activity from writing additional arch records. This will need to be undone later, once things are straightened out.)
It seems to me that we need to run a successful cold backup. Though the messages don't show a clear connection, we strongly suspect that the cold backup failure was due to the lack of space.
[Later:]
They deleted the /exlibris2/oradata/alephp/arch files (to make space) and did a cold backup. This worked. But, as described in KB 16384-35137, the following night's (cold) backup failed: **the deleted archivelog files needed to be removed from RMAN's catalog.**
- Article last edited: 10/8/2013