ARC table is partially loaded, error message "File size too large"
- Article Type: General
- Product: Aleph
- Product Version: 2
Problem Symptoms:
Some ARC tables are partially created and the following error messages appears in the logs:
Table mrr_fact_items Loaded 589300 of 15221170 - Not all records were loaded
I/O error : file '/exlibris/aleph/u20_1/XXX50/files/mrr_fact_items.seq.14'
error code: 9/194 (ANS74), pc=0, call=1, seg=0
194 File size too large
Cause:
The "File size too large" message is related to the cycle size.
Resolution:
1. Reduce the cycle size for the ARC ETL by adding a line to your ./xxx50/prof_library file:
setenv p_arc_01_loop_length 10000
The default size is 50000, we suggest changing this value to 10000.
2. Rerun ETL and check the results.
Additional Information
There are a few tables where the ARC ETL process uses a poor algorithm to divide the data. In most cases, it reads the number of records that are specified in the cycle size. For instance, if the cycle size is 50,000, it reads 50,000 bib records at a time, with each set going into a separate file. There are a few cases, however, where the cycle size is used for two different tables, based on their relationships, and it means that the cycle size is not very helpful. In the case where the file gets too large, the program is looking for loan records related to a chunk of item records. An item record can only have one loan record, but it can have many loan history records. That works correctly, but then it uses the same chunk of item records to look for loan history records. In this case the file size can get maxed out by the massive number of loan history records related to that range of items.
The solution therefore is to reduce the cycle size for the ARC ETL, so all of the parts of the ETL will be affected, producing more, smaller files.
Category: ETL (ARC)
Subject: Extract
- Article last edited: 10/8/2013