Voyager to Alma AutoExtract Migration
Requirements
- Migration Tool path on MFT server
- Credentials for file download
- Links to the following documents:
- Voyager Migration Guide
- Getting Ready for Alma and Discovery Implementation
- This document
- APAC customers: migration-dc05.hosted.exlibrisgroup.com (APAC-DC05) or migration-dc07.hosted.exlibrisgroup.com (APAC-DC07). The relevant username should be supplied by your Professional Services.
- EU customers: migration-dc03.hosted.exlibrisgroup.com (EU-DC03) or migration-dc06.hosted.exlibrisgroup.com (EU-DC03) . The relevant username should be supplied by Professional Services.
- NA customers: migration-dc01.hosted.exlibrisgroup.com (NA-DC01) or migration-dc04.hosted.exlibrisgroup.com (NA-DC04) . The relevant username should be supplied by Professional Services.
- CANADA customers: migration-dc82.hosted.exlibrisgroup.com
- CHINA customers: migration-dc81.hosted.exlibrisgroup.com.cn
Downloading and Installing the AutoExtract Tool
- Download the tool from MFT, as described in the email sent from Ex Libris and move it under the desired Voyager database under the following path: /m1/voyager/xxxdb, where xxxdb is name of the Voyager database, for example: voyagerdb.
- Unzip the general package:
>> unzip Voyager2Alma.zipIf it is not the first time you are downloading and installing the AutoExtract kit, the following message is displayed:replace unzip_package.sh? [y]es, [n]o, [A]ll, [N]one, [r]ename:
- Enter A to replace all relevant migration programs with the new programs.
- Unzip the package:
>> ./unzip_package.sh
- Confirm your Voyager database:
You are connected to the xxxdb.Is this the correct db (Y/N)?If this is not the Voyager database that you want to extract, enter No, exit the menu, and move to the Voyager database that you want to extract.After confirming the Voyager database, the installation continues until the prompt is displayed.During the installation, two new directories are created under the following path /m1/voyager/xxxdb:
- migration_conf
- migration_scripts
- Do not exit while the installation is running or you will have to re-install the tool. If you encounter a problem during the installation process, contact Ex Libris.
- If you downloaded and installed the package previously, you can place that version of the Voyager Migration Form on the server. Use the migration menu to update the previously used migration form. See Updating an Existing Migration Form.
6. Update Java environment to OpenJDK (optional)
The Voyager extract kit will use the existing Java installation. However, if you would like to change this and run the tool with OpenJDK, please add the path to your OpenJDK to the file startup_OpenJDK.txt under migration_scripts directory. This must be done after downloading the tool, but before starting/using the tool.
1. Using any text editor, open /m1/voyager/dbname/migration_scripts/startup_OpenJDK.txt
2. Enter the full path to your OpenJDK's installation directory (this directory holds, among others, the 'bin' sub-directory).
3. Save and close the file, and then open the Voyager menu as normal.
This OpenJDK file is overwritten every time you re-install the Voyager kit, so it needs to be updated each time.
Perform a Test Extract (optional)
1. Extract all areas
2. Extract with no Acquisition
3. Extract with partial Acquisition (without Invoice)
4. Extract Fulfillment only (Patron/Request/Loan)
5. Extract invoice only
Select one of the extraction options for your test extract.
Creating a New Migration Form
- Go to the migration_scripts directory of the Voyager database where the kit is installed.
- Run the following script:
>> ./check_database_connection.shIn case the connection cannot be determined, the following message is displayed, see the following example:The connection test to your database server failed. Enter your database server name :
il-voy2urm02
Enter you database port number:
1521If the connection test is successful, the following message is displayed:Connection test success – working against the database {host}:{port}. Proceed to the menu. - Run the kit:
>>./Pextract_menu
- The Voyager-Alma Extract menu is displayed:
The db name is: voyagerdb
*************************************************************
Voyager-Alma Extract Menu
*************************************************************
1. Create "dummy" Migration Form
2. Define extract areas
3. UB Databases option
4. Create and validate the Voyager Migration Form
5. Validate the P2E file
6. Extract the Data for defined areas
7. Check the extract progress
8. Package the data
9. Transfer the packaged data
10. Check the transfer progress
11. Extract the historical loans
-------------------------------
b. Back
e. Exit
> Please enter your selection: - Select 2. Define extract areas. This does not extract the data, it only defines the areas to extract so that the migration form can be created appropriately. The following menu is displayed:
1. Extract all areas
2. Extract with no Acquisition
3. Extract with partial Acquisition (without Invoice)
4. Extract Fulfillment only (Patron/Request/Loan)
5. Extract invoice onlySelect which type of extract you are performing, according to your contractual agreement with Ex Libris regarding Acquisitions. Extract Fulfillment only is typically for cutover extracts. - If you are a Univeral Borrowing (UB) customer, select 3. UB Databases option
Use this option if you will select one of the following on the migration form for UB_CUSTOMER question
and you also want to link Master and Stub patrons together in Alma.
-Yes migrate all UB
-Yes migrate patron, loans and fines only
-Yes combine into one institution in Alma
1. Yes, we will be answering UB_CUSTOMER with one of the above and we want to link patrons
2. No (either we are not using one of the options, or we do not want to link patrons)
Select either Yes or No, depending on your response to the UB_CUSTOMER question in the Migration form.If you are not a Universal Borrowing customer, you may ignore this question.
- Select 4. Create and validate the Voyager Migration Form. The following menu is displayed:
1. Create the Voyager Migration Form
2. Validate the Voyager Migration Form - Select 1. Create the Voyager Migration Form. The creation of the migration form begins. The following is displayed:
The process is now running in the background. Please wait till the process is complete.
- After the process is complete, the following is displayed:
The migration form is located at: /m1/voyager/xxxdb/migration_conf/ xxxdb_generated_
.xlsm
Press Enter to continue. - The created migration form, named xxxdbgenerated_
_.xlsm can be found under the migration_conf directory of the current Voyager database. - After the process is complete, the following is displayed:
The log of this process is located at: /m1/voyager/xxxdb/migration_scripts/exl_logs/voyager_migration_log.
- After creating the form, fill in the questions and map your data according to the instructions. For more information on how to fill in the migration form, see the Voyager to Alma Migration Guide.
Updating an Existing Migration Form
- Transfer the most recent version of the Voyager migration form to the Voyager server, in the directory /m1/voyager/xxxdb/migration_conf. The migration form may already be present from the testload.
- Name the previous migration form should xxxdb.xlsm. For example, if your database is yourschooldb, then the file should be yourschooldb.xlsm.
- Follow the steps as described in Creating a New Migration Form to create a new migration form. Since the old migration form is in the directory, the toolkit uses the existing form and adds to it.
- After creating the new form, download the form to your PC again, answer the questions, and map your data according to the instructions. For more information on how to fill in the migration form, see the Voyager to Alma Migration Guide.
- Pay particular attention to any newly added locations, fiscal periods, reporting codes, and possible new questions on the Questionnaire tab.
Validating the Migration Form
- Place the Voyager migration form under the Voyager database/migration_conf directory and re-name it xxxdb.xlsm.
- Go to the migration_scripts directory of the Voyager database where the kit is installed.
- Run the kit:
>> ./Pextract_menu
- Select 4. Create and Validation Migration Form.
The following menu is displayed:1. Create the Voyager Migration Form
2. Validate the Voyager Migration Form - Select 2. Validate Migration Form. The validation of the migration form begins and the following message is displayed:
The process is now running in the background. Please wait till the process is complete.After the process is complete, the following is displayed:The validation report of the migration form is located at: /m1/voyager/xxxdb/migration_scripts/exl_logs/mf_validation.log.If the form is found to be invalid, the following message is displayed:~~Number of errors: 1
~~Number of warnings: 3
~~The validation report of the migration form is located at: /m1/voyager/xxxdb/migration_scripts/exl_logs/mf_validation.log
Please correct the errors and review the warnings listed in the validation report. - Review the validation report of the migration form that is located under the migration_scripts directory of the current Voyager database named mf_validation.log.
- The validation report is divided in two sections: MIGRATION FORM ERRORS and MIGRATION FORM WARNINGS. Each line in the report refers to a specific error or warning found in the migration form.
For example:ERROR: 2014-04-07 12:05:44 --- MIGRATION FORM ERRORS ---
Sheet: Questionnaire Institution Id (as determined by the Alma Migration Team) cannot be empty
WARN: 2014-04-07 12:05:44 --- MIGRATION FORM WARNINGS ---
Sheet: Reporting Codes Fund: GN is defined in database and is not defined in the spreadsheet
Sheet: Reporting Codes Fund: LEXIS is defined in database and is not defined in the spreadsheet
Sheet: Reporting Codes Fund: CDHist is defined in database and is not defined in the spreadsheet
2014-04-07 12:05:44 --- Migration Form validation is finished. Status: FAILURE --- - Review the warnings and correct the errors listed in the validation report. It is recommended to re-validate the corrected form after your changes.
- Send the validated form to the Alma migration team for approval two weeks before Testload and a week before Cutover before proceeding with the extract.
Validate the P2E File
- Create the file according to the instructions in the Voyager to Alma Migration Guide.
- Place the file under the Voyager xxxdb/migration_conf directory and name it xxxdb.csv
-
The form must be in all lowercase.
-
The Voyager migration form must be present and completed (all tabs mapped)
- Change the directory to migration_scripts: ../migration_scripts and run the kit ./Pextract_menu.
- Select option 5. Validate the P2E file from the main menu.
After the process is complete, if there are errors, the following is displayed:
The validation report of the p2e file is located at: /m1/voyager/xxxdb/migration_scripts/exl_logs/p2e_validation.log
3. Correct any errors regarding the format or the electronic type in the file, for example:
Not allowed type in line 2: Xortfolio
Extracting the Data
- Place the approved and validated migration form under the Voyager xxxdb/migration_conf directory and name it xxxdb.xlsm.
The form name must be in all lower case, even if your database name has some upper case letters in it.
- Place the approved and validated e-resource P2E file under the Voyager xxxdb/migration_conf directory and name it xxxdb.csv.
- Go to the migration_scripts directory of the Voyager database where the kit is installed.
- Run the following script:
>> ./check_database_connection.shIf the connection cannot be determined, the following message is displayed, for example:The connection test to your database server failed. Enter your database server name:
il-voy2urm02
Enter you database port number:
1521If the connection test is successful, the following message is displayed:Connection test success. Proceed to the menu. - If a secondary item file is used, place it in migration_scripts/secondary_item/secondary_item.csv prior to running the extract. See the Voyager to Alma Migration Guide, Appendix C, for more information on generating this file.
- Run the kit:
>> ./Pextract_menu
- Select 2. Define extract areas. This does not extract the data, it only defines the areas to extract prior to extracting. The following menu is displayed:
1. Extract all areas
2. Extract with no Acquisition
3. Extract with partial Acquisition (without Invoice)
4. Extract Fulfillment only (Patron/Request/Loan)
5. Extract invoice only - Select which type of extract you are performing, according to your contractual agreement with Ex Libris regarding Acquisitions. Extract Fulfillment only is typically for cutover extracts.
- Select option 6. Extract the Data for defined areas.
- Define the directory (with enough prerequisite disk space) in which to place the extracted data. (A default directory is recommended.):
The default root path for the extracted output files is: /m1/voyager/xxxdb/voyagerdb_extract. Press enter to confirm or enter a different root path:The Voyager migration form validation process extract starts running after you answer the questions above. The validation report is produced at the end of the process. If the form is invalid, the extract does not proceed. If the form is valid, the extract begins automatically.
- Indicate if this is a testload or a cutover extract.
Is this a testload or cutover extract? [T/C]:For testload extracts, the site may choose to allow staff to continue working during the data extract. At the end of extracting each data element, the migration program compares the number of records extracted vs the number of records in the database. If the numbers do not match, for example if a record was added in the middle of the extract because staff are working, the extract program assumes an error. To correct the potential error, the program will attempt to re-extract - but only for cutover loads. The program will not attempt to re-extract for testloads.
- Jobs running in the background will slow the extract process and may stop it. Therefore, the extract reports if any Voyager jobs are running.
If the extract encounters a running job or a scheduled job (within 48 hours) the list of jobs is displayed. You are asked if you want to continue or to exit the extract in order to disable the jobs.
- After checking , the extract begins and the following message is displayed:
Voyager extract is now running in the background. Press ENTER to go back to the main menu.
- Press ENTER to go back to the Voyager-Alma Extract menu and select option 7. Check the extract progress to monitor the process.
Monitoring the Extract Process
- Go to migration_scripts directory of the Voyager database where the kit is installed.
- Run the kit:
>> ./Pextract_menu
- Select option 7. Check the extract progress. The following message is displayed:
Monitoring the extracted output files located at the following root path: /m1/voyager/xxxdb/voyagerdb_extract/<date&time>
The monitoring output is refreshed every 1 minuteThe screen refreshes every minute, so you can see the progress for each area:AREA EXTRACTED PROGRESS
==== ========= ========
PATRONS 206 100.0%
LOANS 347 100.0%
REQUESTS 21 100.0%
Checking the extract after completion
Checking the extract logs post-extract is mandatory.
Customers must check the final extract counts for their data, to compare what they think is migrating to what was actually extracted. Logs may be found in the extract directory. Your complete extract may be found here:
/m1/voyager/yyydb/voyagerdb_extract/20190615095136
where the directory name is the date/time of extract. Files of interest are:
1. logs_status.log:
- a summary of record count differences: database count vs extracted count
- errors reported (if any)
- corrupted zip files (if any)
2. reports.xlsx and reports.txt
- a Excel spreadsheet and parallel text file reporting the record counts for all data elements extracted (these two contain the same information, just different formats)
- Note: elements with rejected counts greater than zero have corresponding *_rejected.csv files with record details in extract directory
3. to review errors in the job possibly unrelated to data, you may wish to review /m1/voyager/xxxdb/migration_scripts/exl_logs/voyager_migration_log.
Customers are advised to review these files for extract accuracy.
Package and Transfer Extracted Files
- Open the Voyager-Alma Extract menu and select option 8. Package the data. The following message is displayed:
Packaging the extracted data. This may take several minutes. Please wait.
Data packaged into /m1/voyager/xxxdb/voyagerdb_extract /migration_transferAfter the packaging is completed, the packaged extract data is placed in the transfer directory. This directory is located under the directory in which the extracted data was placed, as entered after selecting option 6. Extract the Data. This directory is located under the default location /m1/voyager/xxxdb/voyagerdb_extract /migration_transfer - Open the Voyager-Alma Extract menu and select option 9. Transfer the packaged data.
- Select the MFT server to which you want the data transferred, according to your region:
Select the mft server by the optional regions:
1. APAC
2. EUR
3. USA
4. CANADA
5. CHINA
Please select your region:If your region is APAC, EU, or NA, you are asked to select one of the data centers in your region, according to Professional Services. You are asked to enter your MFT user name. Copy and paste the key from the link sent to you by Ex Libris. Below is an example of the key you are asked to insert.Enter your password:
-----BEGIN RSA PRIVATE KEY-----
MIIEoAIBAAKCAQEAz+h+RByvWo5aLNxpXS0hNh2DKlipYA81Ur+pVzoILXiA3EWW
h6zfCOxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxbP0UPtbKojfhIgCbAJrtwa7jszxu
XW9vn95ulDUNNuoyKGyyVeve7InsFx0NTzlLaJeJq3vfPCVXvO8InRY7giX/zCE0
yxsOal+Wsf5vn+9L8XGEdcaG37vPWGEBfaNYywIBIwKCAQBNORj0ujKAuIfkxuVO
fnoMySIXCv0VDPaMcxMDJC7sUVRvEoheR4YKnIVxq+edr12hLcdp7g5VjihxmxVP
-----END RSA PRIVATE KEY-----
The following message is displayed:Verifying the connection to the MFT server requested can be established. Please wait.If the connection cannot be established, the following message is displayed:Cannot establish connection to the MFT server requested. Do you want to try again resetting your password? (Y/N) [N]When the connection is established the data transfer begins and the following message is displayed:Data is being transferred.The data is transferred securely using MFT. - Open the Voyager-Alma Extract menu and select option 10. Check the transfer progress. The following message is displayed:
NN% files of the extract was transferred
- Check the progress again by selecting option 10. Check the transfer progress. The transfer is considered completed when the following message is displayed:
The transfer is complete.The data transfer process can take up to 48 hours to complete, depending on the size of your data and your Internet connection speed. The functioning of Voyager is not affected by this process running in the background, and your normal work in Voyager can proceed in parallel. If the process does not complete within 60 hours, contact Ex Libris.
- Inform Ex Libris when the data transfer process is completed.
Fulfillment Cutover
- Move to the migration_scripts directory under the desired Voyager database.
- Run the following script:
>> ./check_database_connection.shIf the connection cannot be determined, the following message is displayed, for example:The connection test to your database server failed. Enter your database server name:
il-voy2urm02
Enter you database port number:
1521If the connection test is successful, the following message is displayed:Connection test success. Proceed to the menu. - Open the Voyager-Alma Extract menu by entering the following:
>>./Pextract_menu
- Select option 2. Define Extra areas from the Voyager-Alma Extract menu. The following is displayed:
1. Extract all areas
2. Extract with no Acquisition
3. Extract with partial Acquisition (without Invoice)
4. Extract Fulfillment only (Patron/Request/Loan)
5. Extract invoice only - Select option 4. Fulfillment only (Patron/Request/Loan).
- Define the directory (with enough prerequisite disk space) in which the extracted data will be placed. (A default directory is recommended.)
The default root path for the extracted output files is: /m1/voyager/xxxdb/voyagerdb_extract. Press enter to confirm or enter a different root path :
- Select option 6. Extract the data for the defined areas.
- Monitor the process via the monitoring screen as described in the section above.
- After the extract is completed, package the data and transfer it as described in the sections above.
Extract Historical Loans
Troubleshooting
Unable to generate a migration form
If you are unable to generate a migration form, make sure you have not modified, or modified the name of, the form: migration_conf/VoyagerMigrationForm.xlsm. This is used as a basis for a new form and is required for a new migration form to be generated.
Copying the MFT SSH_KEY
In the extract process, you are asked to copy the ssh_key given to you by your Ex Libris representative. If you do not copy the entire key, the data is not packaged and transferred. If this situation occurs, do the following:
- Close the extract menu.
- Copy the entire content of the ssh_key (including the first and last "BEGIN"/"END" lines) and paste it into a text file in Windows.
- Save the file with the following name: your generated UserName + _urm_sftp_ssh_key (without any suffix). For example, if your user name = MyGenUsrName, the ssh_key file is named: MyGenUsrName_urm_sftp_ssh_key.
- Copy the ssh_key file to the Voyager server in the location where the kit was placed in the ../migration_conf directory.
- Open a putty session to your Voyager server, and go to the ../migration_conf directory. Set permissions to your ssh_key file: chmod 600 MyGenUsrName_urm_sftp_ssh_key.
- Rerun the extract menu. This time you are not asked to insert the ssh_key.