Parallel Processing of Pipes
The Parallel Processing of Pipes Mode parameter under the Installation subsystem on the General Configuration page allows you to choose from the following types of pipe modes:
-
Harvesting only – This mode was available in Primo V3, but it only allows parallel processing through the harvesting stage. This is the out-of-the-box option.
-
Harvesting, NEP, Dedup-FRBR – This mode allows parallel processing of pipes through the Dedup and FRBR stages. Only one pipe per institution can run at a time.
Harvesting, NEP, Dedup-FRBR ModeThis mode is intended for multi-institution sites where there is no need to Dedup and FRBRize the entire database, and Dedup and FRBR are relevant only within the records that belong to a specific institution. Such sites previously prevented Dedup and FRBR between institutions by adding the institution code to the Dedup and FRBR keys. With this mode, adding the institution code is no longer necessary because the system only groups records that belong to the same institution.The institution must be the owner the data source and the pipe. If you have a scenario in which a single data source belongs to multiple institutions, dedup and FRBR will occur across these institutions. If you want to prevent this, you should add the institution code to the Dedup and FRBR keys. -
Harvesting, NEP – This mode has been added to allow parallel processing of pipes through the NEP (Normalization, Enrichment, and Persistence – that is load to the database) stage. In this mode, the Dedup and FRBR stages must be run separately with a new pipe, which is called Dedup_Frbr. For more information, see Dedup_Frbr Pipe.If you want a pipe’s records to be excluded from the Dedup and/or FRBR processes, clear the Include DEDUP and/or Include FRBR check boxes on the Define Pipe page.
Harvesting, NEP ModeIf you are using the Harvesting, NEP mode for a multi-institutional environment, Dedup and FRBR will process all pipes even if another institution’s pipe does not have Dedup/FRBR defined. If you want to limit Dedup and FRBR to specific institutions, use either of the following methods:- setting the Parallel Processing of Pipes Mode parameter to Harvesting, NEP, DEdup/FRBR in the General Configuration Wizard
- adding the institution code to the Dedup and FRBR keys
Pipe Restrictions
Pipes cannot run in parallel with the following processes:
-
Indexing and Hotswapping
-
Delete Pipe – which runs after a successful hotswap
-
Tools – except for the Generate Site Map tool
-
Dedup_Frbr pipe
The failure of a single pipe will not prevent the running of another pipe, indexing, or the Dedup_Frbr pipe.
Configuring the Maximum Number of Pipes
The number of pipes that can run in parallel depends on the system resources of the Back Office server (such as number of processors and cores). If there are not enough resources available, the system will not start a pipe.
Each pipe requires four threads in order to run. If more threads are available, the pipe runs faster.
Out of the box, the Primo allows two pipes to run in parallel. On-premises customers can modify this setting with the primo.process.ParallelProcessing.NumThreads parameter under the Installation subsystem in the General Configuration Wizard. For more information, see General Configuration Wizard.
Changing Parallel Processing Modes
In most cases, your installation will switch from the Harvesting only mode (which is the out-of-the-box mode) to either the Harvesting, NEP or Harvesting, NEP, Dedup-FRBR mode, but it is also possible to switch between the modes. The following table summarizes the actions required after changing the parallel processing mode. For more information on the modes, refer to Parallel Processing of Pipes.
| Current Mode | New Mode | Actions |
|---|---|---|
|
Harvesting only
|
Harvesting, NEP
|
Change the mode and then deploy the System Configuration option on the Primo Home > Deploy All page.
|
|
Harvesting only or Harvesting, NEP
|
Harvesting, NEP, Dedup-FRBR
|
If you have not run dedup/FRBR for any of the data sources, or you prevented cross-institution grouping by adding something (such as the institution code) to the Dedup/FRBR keys, you will need to deploy the System Configuration option on the Primo Home > Deploy All page after changing the mode.
Otherwise, you will have to create the database from scratch. Consult with Ex Libris support before making this type of change.
|
|
Harvesting, NEP, Dedup-FRBR
|
Harvesting only or Harvesting, NEP
|
Change the mode and then deploy the System Configuration option on the Primo Home > Deploy All page.
Run an update pipe on all data sources to ensure that dedup/FRBR is cross-institution.
|
Dedup_Frbr Pipe
When the parallel processing mode is set to Harvesting, NEP, the system will add the Dedup_Frbr pipe to the pipes list (see Pipes List). It runs like any other pipe, and it can be executed manually or scheduled. It should be run before indexing and hot-swapping are executed.
Pipes List
The Define Pipe page allows you to schedule the pipe to be run:
Define Pipe Page
Starvation Mode
If pipes run for a long time, indexing or the execution of the new Dedup_Frbr pipe may be delayed. To prevent this from happening, the Starvation Mode parameter has been added to the Installation subsystem on the General Configuration page. The following options are available:
-
By Schedule – This is the default option. Indexing and the Dedup/FRBR Pipe will wait until all of the scheduled pipes have finished. Note that a failed a pipe will not prevent indexing or the execution of the Dedup/FRBR pipe.
-
Force – Prevents scheduled pipes from starting if indexing or the Dedup-FRBR pipe is waiting to be executed.
It is not necessary to perform a deploy or restart the Back Office after changing this parameter.