Incremental ETL duration too long
- Article Type: General
- Product: Aleph
- Product Version: 2
Problem Symptoms:
* the incremental ETL takes several hours and takes almost as long as the full ETL
Cause:
Configuration of ./xxx50/tab/arc_conf_table.conf
Resolution:
In order to reduce the duration for the ETL load, reduce the amount of data included into the load.
Check / Correct the settings of table .xxx50tabarc_conf_table.conf to include less data:
[ARC HISTORY] period for all history files (z30h, z36h, z37h)
[OPAC] period for OPAC events (z69)
[EVENTS] period for events (z35)
Example for configuration including all (history) data since 01.01.2010
./xxx50/tab/arc_conf_table.conf
[ARC HISTORY]
2010
[OPAC]
20100101
[EVENTS]
20100101
Additional Information
In ARC 2.0 the Incremental Load differs from the Full Load only this way, that the events (z35) and OPAC events (z69) are loaded incremental. All other files are fully loaded during ETL.
Starting with ARC 3.0 / Aleph 21 a Full Complete Incremental Load is available.
For details please see ALEPH Configuration for ARC 3.0.pdf
The following additional options are available starting with ARC 3.0:
[PATRON-DATA]
eliminate_data=N
[CIRC-LOGGER]
from date = 20070101
- Article last edited: 06-Feb-2017