Duplication Detection (Dedup)

The duplication detection or matching process is based on creating a dedup vector for every Alma record (and all imported records from external sources). The vector includes all the data required by the Duplication Detection algorithm to determine if two records are equivalent. The vectors include one or more keys that identify the record. Keys are based on the metadata fields in the source record (MARC or DC records).

In the search results, records that are marked as duplicates are displayed as a single record. The metadata is displayed from the first record in the results set, and the delivery related information is created from all the records in the group.

FRBR

Primo VE records are grouped using the principles in the Functional Requirements for Bibliographic Records published by the IFLA Study Group on the Functional Requirements for Bibliographic Records. The grouping process creates a vector for every Alma record (and all imported records from external sources). This vector includes one or more keys that identify the group it represents. Keys are based on the metadata fields in the source record (MARC or DC records).

Records that have a matching key (Primo VE attempts to match all keys in the record) are added to a FRBR group and assigned a FRBR ID. Because records can belong to one FRBR group only, Primo VE terminates the grouping process for a record as soon as it is matched with a group.

The group IDs are indexed and used during searches to group search results by the same group ID. A record that belongs to a FRBR group displays the Multiple versions exist. See all versions link in the brief results, which enables users to see all records that have been grouped together. The record that displays in the brief results can be configured to display either information for a preferred record or generic information that pertains to all records in the group. After performing a search, results are grouped by FRBR ID, and after selecting the Multiple versions exist. See all versions link, the results are grouped by Dedup ID.

How Dedup and FRBR Work Together

The Dedup and FRBR matching processes are very similar, but the content of the keys is different. The system first looks for duplicate records and then looks for FRBR groups. Because the Dedup ID is used as a key for the FRBR process, a record that is Deduped is also FRBRized.

Calculating the Keys

The Dedup and FRBR keys are used to determine whether records match and are generated from specific data elements from the bibliographic record and then combined into keys. The sections below first describe how the key information in the MARC21 fields are mapped to the database in Primo VE and then describe how the elements are combined to create the Dedup and FRBR keys.

Calculating the Group ID

The system hashes the keys into a number and then adds in a coefficient, which is based on the key's priority, to ensure that keys with higher priorities have larger numbers. After a group ID is chosen for a specific record, all the keys that were calculated from that record are saved with the group ID to ensure that all the keys point to that group ID. This allows giving records that have identical keys for the same group ID.

Dedup/FRBR Process Flow

The system performs the following steps for both Dedup and FRBR processes, starting with the Dedup process:

1. Calculate all the keys from the source record.

2. Find all the keys that were calculated in step 1 that already exist in the database (from previous calculations). If there are keys, the system selects the group ID that is associated with the key that has the highest value (this prioritizes keys with higher priority). If the record matches more than one group and the keys have the same priority, the system selects the group ID randomly. If there is no existing key in the database, the system selects the key with the highest value and uses it as the group ID.

3. For all keys calculated in step 1, the system stores the selected group ID. This enables other records to connect to recently processed records.

The algorithm is not fully transitive, which means that two records may have a matching common record but are not considered matches themselves. Because of this, the order in which the records are processed can affect the groupings. For more information, see Dedup and Transitivity.

Dedup and Transitivity

1. Record B is added to the database and no match is found.
2. Record C is added to the database and none of its keys match record B.
3. Record A is added to the database and one of its keys matches a key in record B first. Because a match was found, the system stops processing the record and returns the following groups even if record A also matches record C: AB and C.

Suppressing FRBR/Dedup Groups

You can use the following methods to prevent records from being deduped or FRBRized:

• Define suppression rules, which are based on types of records (such as images), which are applied during re-indexing when new records are loaded or when an existing record's metadata is modified. For more details, see Suppressing Groups of Records from Dedup and FRBR. If you want to suppress existing records, see the next option.

• Run the Prevent FRBR and/or Dedup in Discovery job on a set of bibliographic records (Admin > Manage Jobs and Sets > Run a Job). For more information, see Running Manual Jobs on Defined Sets.

If you want to prevent the display of FRBR and Dedup groups for all records without changing any rules or running any jobs, you can disable FRBR/Dedup by configuring the Enable Dedup and Enable FRBR fields on the Brief Results tab on the View Configuration page (Configuration Menu > Discovery > Display Configuration > Configure Views). For more information, see Brief Results Tab.

Recalculating Dedup and FRBR Groups

FRBR and Dedup keys, which are built initially at record creation and/or migration, may need to be recalculated after you modify a bibliographic record. For information on how to recalculate your keys, see Recalculate Dedup and FRBR Groups.